From c38a471042cefbc27f0acee9b2a4f5f4ca05b5a8 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 14 Nov 2026 20:14:47 -0600 Subject: [PATCH] perf(server): download stale agent files before the agent row lock (#21694) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Durable agent files live in object storage. Each server keeps a local copy (the projection), marked with the head generation it matches > - When the projection is stale, the next locked read or write restores it: it downloads every file from object storage. This happens for the first run of each agent on a new server (every deploy on hosted ephemeral servers) and after another server wrote > - The download ran inside the transaction that holds the `agents` row lock (`FOR UPDATE`), on the main pool for the run-start read (`readCommittedForRuntime`) and on the 2-connection file-storage pool for the store. A download can take up to 30 s > - After a deploy at 1,000 agents, this is one long row lock and one pinned connection per agent, all at once, and every writer of that agent waits behind it > - This pull request downloads the files outside the transaction and takes the lock again only to swap them in > - The benefit is that the row lock is held for milliseconds instead of the download time (0 ms observed instead of 321 ms with a 300 ms download in the test), and writers do not wait for a download ## Linked Issues or Issue Description **What existing behavior does this improve?** The restore of a stale local copy of an agent's durable files (the first run of each agent on a new server, or after another server wrote). **Subsystem affected** Server: durable agent files (`agent-file-durability.ts`, `agent-file-store.ts`, `agent-instruction-revisions.ts`). **Current behavior** The restore downloads every file from object storage inside the transaction that holds the `agents` row lock (`FOR UPDATE`) and a pooled connection. A download can take up to 30 s, and every writer of that agent waits behind it. **Proposed behavior** Under the lock, detect that a download is needed and roll back. Download the head's files, verified by digest, into a private staging directory with no lock. Run the transaction again, and swap the staged files in only when the head generation is unchanged. After two changed heads, restore under the lock as before. **Reason and benefit** After a deploy at 1,000 agents, every agent's first run restores at once. The row lock and connection are now held for milliseconds instead of the download time, and writers do not wait for a download. **Additional context** Refs the 1,000-concurrent-agents plan (`doc/plans/2026-11-13-thousand-concurrent-agents.md`). The save path had the same problem for uploads and was fixed earlier with uploads before the lock (`uploadAgentFilesBeforeCommit`); this is the restore side. Reading the head before every transaction was considered; it adds a query to every call, while the deferral costs nothing when the projection is current. ## What Changed - `agent-file-durability.ts`: the download loop moves into `downloadAgentFiles`, unchanged. `withAgentFilesRestoredBeforeLock(work)` runs `work` in a scope. Inside the scope, the first restore of an attempt that needs a download throws a private deferral error, which rolls back the transaction. The wrapper stages the download in `.agent-files-` next to the root and runs `work` again. `restoreDurableAgentFiles` uses a staged directory only when its generation equals the head it reads under the lock. Staged directories are removed when the scope ends. - Only the first restore of each attempt can defer, so nested restores of other agents keep their old behavior. Outside the scope, the behavior is unchanged. - `agent-file-store.ts` (`locked`, so every store read, write, apply and working-copy preparation) and `agent-instruction-revisions.ts` (`readCommittedForRuntime`, `readCurrent`, `materializeCurrent`) run their transactions in the scope. ## Verification - New test `server/src/__tests__/agent-file-restore-locking.test.ts` on embedded PostgreSQL, with an object-storage fixture: - Hold time, with a 300 ms download, through `readCommittedForRuntime` (the run-start read). Before: the download ran with the agent row locked, and the row was locked 321 ms of 322 ms. After: the download ran with the row unlocked, and a sampler saw the row locked for 0 ms. - Concurrency: a board write of the same file runs while the reader's download is blocked. Before: the writer waited for the reader's download (5 s timeout). After: the writer commits a new head during the download. The reader then discards its stale staged copy and returns the writer's content. The projection is marked at the writer's generation, and no staging directory is left. - Failure: when object storage fails, the read fails once with the storage error, after one attempt made without the row lock, and leaves no staging directory. - Progress: when the head generation changes during each of two downloads, the third attempt restores under the lock as before (3 downloads), and the projection matches the final head. - `npx vitest run` for agent-file-checkpoint-locking, agent-file-checkpoints, agent-file-durability, agent-file-restore-locking, agent-instruction-revisions, agent-instruction-working-copies, agent-instructions-service, agent-directory-working-copies and agent-instructions-routes: 287 passed, 1 skipped. - `npx tsc --noEmit -p server/tsconfig.json`: passes. ## Risks - A deferred attempt runs the transaction body again. The deferral happens at the first restore, before the body writes files, and the transaction rollback undoes database work, so the second attempt starts from the same state. - A staged download uses disk next to the agent root until the scope ends, the same as the temporary directory of the old download. - If the head changes during every download, the third attempt downloads under the lock as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../agent-file-restore-locking.test.ts | 199 ++++++++++++++++++ server/src/services/agent-file-durability.ts | 144 +++++++++---- server/src/services/agent-file-store.ts | 6 +- .../services/agent-instruction-revisions.ts | 13 +- 4 files changed, 309 insertions(+), 53 deletions(-) create mode 100644 server/src/__tests__/agent-file-restore-locking.test.ts diff --git a/server/src/__tests__/agent-file-restore-locking.test.ts b/server/src/__tests__/agent-file-restore-locking.test.ts new file mode 100644 index 00000000000..e804283d0d3 --- /dev/null +++ b/server/src/__tests__/agent-file-restore-locking.test.ts @@ -0,0 +1,199 @@ +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { randomUUID } from "node:crypto"; +import { Readable } from "node:stream"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { and, eq, sql } from "drizzle-orm"; +import { agentFileRoots, agents, authUsers, closeFileStorageDb, companies, companyMemberships, createDb, principalPermissionGrants } from "@paperclipai/db"; +import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { agentFileStore, fileHash } from "../services/agent-file-store.js"; +import { agentInstructionRevisionService } from "../services/agent-instruction-revisions.js"; +import { resolveManagedInstructionsRoot } from "../services/agent-instructions.js"; + +const objects = vi.hoisted(() => ({ + bytes: new Map(), + downloads: [] as Array<{ objectKey: string; agentRowLocked: boolean | null }>, + probe: null as null | (() => Promise), + gate: null as null | { opened: Promise; reached: () => void }, + delayMs: 0, +})); +vi.mock("../storage/index.js", () => ({ getStorageService: () => ({ + provider: "s3", + async putFile(input: { companyId: string; namespace: string; body: Readable }) { + const chunks: Buffer[] = []; + for await (const chunk of input.body) chunks.push(Buffer.from(chunk)); + const objectKey = `${input.companyId}/${input.namespace}/${randomUUID()}`; + objects.bytes.set(objectKey, Buffer.concat(chunks)); + return { objectKey }; + }, + async getObject(_companyId: string, objectKey: string) { + const body = objects.bytes.get(objectKey); + if (!body) throw new Error("fixture object missing"); + objects.downloads.push({ objectKey, agentRowLocked: objects.probe ? await objects.probe() : null }); + const gate = objects.gate; + if (gate) { objects.gate = null; gate.reached(); await gate.opened; } + if (objects.delayMs) await new Promise(resolve => setTimeout(resolve, objects.delayMs)); + return { stream: Readable.from([body]) }; + }, +}) })); + +describe("agent file restore locking", () => { + let database: Awaited>; + let db: ReturnType; + let observer: ReturnType; + let home: string, companyId: string, agentId: string, userId: string, root: string; + const entryFile = "AGENTS.md"; + const board = () => ({ type: "board" as const, userId, source: "session" as const }); + const target = () => ({ companyId, agentId }); + + async function agentRowLocked(): Promise { + try { + await observer.transaction(async tx => { await tx.execute(sql`select id from agents where id = ${agentId} for update nowait`); }); + return false; + } catch (error) { + const code = (error as { code?: string; cause?: { code?: string } }).code ?? (error as { cause?: { code?: string } }).cause?.code; + if (code === "55P03") return true; + throw error; + } + } + async function head() { + const [row] = await db.select().from(agentFileRoots).where(and(eq(agentFileRoots.companyId, companyId), eq(agentFileRoots.agentId, agentId))); + return row!; + } + async function loseLocalProjection() { + await fs.rm(root, { recursive: true, force: true }); + await fs.rm(path.join(path.dirname(root), "file-sync", "durable-generation"), { force: true }); + } + async function stagingLeftovers() { + return (await fs.readdir(path.dirname(root))).filter(name => name.startsWith(".agent-files-")); + } + + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("agent-file-restore-locking-"); + db = createDb(database.connectionString); + observer = createDb(database.connectionString); + }, 90_000); + afterAll(async () => { + await observer?.$client.end({ timeout: 1 }); + if (db) { await closeFileStorageDb(db, 1); await db.$client.end({ timeout: 1 }); } + await database?.cleanup(); + }); + beforeEach(async () => { + home = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "agent-file-restore-locking-"))); + vi.stubEnv("PAPERCLIP_HOME", home); + vi.stubEnv("PAPERCLIP_DURABLE_AGENT_FILES", "true"); + objects.bytes.clear(); objects.downloads.length = 0; objects.probe = null; objects.gate = null; objects.delayMs = 0; + companyId = randomUUID(); agentId = randomUUID(); userId = randomUUID(); + root = resolveManagedInstructionsRoot({ id: agentId, companyId, name: "Owner", adapterConfig: {} }); + await db.insert(companies).values({ id: companyId, name: "Restore locking", issuePrefix: randomUUID().slice(0, 8) }); + await db.insert(authUsers).values({ id: userId, name: "Editor", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() }); + await db.insert(agents).values({ id: agentId, companyId, name: "Owner", + adapterConfig: { instructionsBundleMode: "managed", instructionsRootPath: root, instructionsEntryFile: entryFile } }); + await db.insert(companyMemberships).values([ + { companyId, principalType: "user", principalId: userId, membershipRole: "operator" }, + { companyId, principalType: "agent", principalId: agentId, membershipRole: "member" }, + ]); + await db.insert(principalPermissionGrants).values({ companyId, principalType: "user", principalId: userId, + permissionKey: "agents:configure", scope: { agentIds: [agentId] } }); + await fs.mkdir(root, { recursive: true }); + await fs.writeFile(path.join(root, entryFile), "# Policy\n"); + expect((await agentInstructionRevisionService(db).readCommittedForRuntime(target()))?.content).toBe("# Policy\n"); + expect((await head()).entries.map(entry => entry.path)).toEqual([entryFile]); + }); + afterEach(async () => { + vi.unstubAllEnvs(); + await fs.rm(home, { recursive: true, force: true }); + }); + + it("downloads a stale projection without holding the agent row lock", async () => { + await loseLocalProjection(); + objects.probe = agentRowLocked; + objects.delayMs = 300; + let lockedMs = 0; + let sampling = true; + const sampler = (async () => { + let last = performance.now(); + while (sampling) { + const locked = await agentRowLocked(); + const now = performance.now(); + if (locked) lockedMs += now - last; + last = now; + await new Promise(resolve => setTimeout(resolve, 10)); + } + })(); + const startedAt = performance.now(); + let current: Awaited["readCommittedForRuntime"]>>; + let elapsedMs: number; + try { + current = await agentInstructionRevisionService(db).readCommittedForRuntime(target()); + elapsedMs = performance.now() - startedAt; + } finally { + sampling = false; + await sampler; + } + console.info(JSON.stringify({ metric: "agent_file_restore_row_lock", downloads: objects.downloads.length, + downloadsWhileLocked: objects.downloads.filter(download => download.agentRowLocked).length, lockedMs: Math.round(lockedMs), elapsedMs: Math.round(elapsedMs) })); + expect(current?.content).toBe("# Policy\n"); + expect(objects.downloads).toEqual([{ objectKey: expect.any(String), agentRowLocked: false }]); + expect(lockedMs).toBeLessThan(150); + expect(await fs.readFile(path.join(path.dirname(root), "file-sync", "durable-generation"), "utf8")).toBe((await head()).generation); + expect(await stagingLeftovers()).toEqual([]); + }); + + it("lets a writer commit during the download and then serves the writer's head, not the stale download", async () => { + await loseLocalProjection(); + let open!: () => void, reached!: () => void; + const opened = new Promise(resolve => { open = resolve; }); + const reachedPromise = new Promise(resolve => { reached = resolve; }); + objects.gate = { opened, reached }; + const before = (await head()).generation; + const reader = agentInstructionRevisionService(db).readCommittedForRuntime(target()); + await reachedPromise; + try { + const written = await Promise.race([ + agentFileStore(db).write({ companyId, agentId, path: entryFile, bytes: Buffer.from("# Policy v2\n"), baseHash: fileHash(Buffer.from("# Policy\n")) }, board()), + new Promise((_, reject) => setTimeout(() => reject(new Error("the writer waited for the reader's download")), 5_000)), + ]); + expect(written).toMatchObject({ changed: true }); + } finally { open(); } + const current = await reader; + const after = await head(); + expect(after.generation).not.toBe(before); + expect(current?.content).toBe("# Policy v2\n"); + expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe("# Policy v2\n"); + expect(await fs.readFile(path.join(path.dirname(root), "file-sync", "durable-generation"), "utf8")).toBe(after.generation); + expect(await stagingLeftovers()).toEqual([]); + }); + + it("fails once, without the row lock and without retrying, when the staged download fails", async () => { + await loseLocalProjection(); + const attempts: boolean[] = []; + objects.probe = async () => { + attempts.push(await agentRowLocked()); + throw new Error("object storage unavailable"); + }; + await expect(agentInstructionRevisionService(db).readCommittedForRuntime(target())).rejects.toThrow("object storage unavailable"); + expect(attempts).toEqual([false]); + expect(await stagingLeftovers()).toEqual([]); + }); + + it("restores under the lock as before when the head keeps changing during every download", async () => { + await loseLocalProjection(); + const store = agentFileStore(db); + let version = 0; + const realGet = objects.probe; + objects.probe = async () => { + if (version < 2) { + version++; + await db.update(agentFileRoots).set({ generation: randomUUID() }) + .where(and(eq(agentFileRoots.companyId, companyId), eq(agentFileRoots.agentId, agentId))); + } + return realGet ? realGet() : false; + }; + expect((await store.read(companyId, agentId, entryFile, board()))?.toString("utf8")).toBe("# Policy\n"); + expect(objects.downloads).toHaveLength(3); + expect(await fs.readFile(path.join(path.dirname(root), "file-sync", "durable-generation"), "utf8")).toBe((await head()).generation); + expect(await stagingLeftovers()).toEqual([]); + }); +}); diff --git a/server/src/services/agent-file-durability.ts b/server/src/services/agent-file-durability.ts index 058175f6c19..4ba1f60e058 100644 --- a/server/src/services/agent-file-durability.ts +++ b/server/src/services/agent-file-durability.ts @@ -1,3 +1,4 @@ +import { AsyncLocalStorage } from "node:async_hooks"; import { createHash, randomUUID } from "node:crypto"; import fs from "node:fs/promises"; import { constants } from "node:fs"; @@ -104,52 +105,25 @@ export async function restoreDurableAgentFiles(tx: Tx, owner: Owner, root: strin throw error; }); if (generation === head.generation && await fs.stat(root).then(s => s.isDirectory()).catch(() => false)) return true; + const scope = restoreScope.getStore(); + const deferrable = scope?.deferring === true && !scope.used; + if (scope) scope.used = true; + const stageKey = `${owner.companyId}:${owner.id}:${root}`; + const staged = scope?.staged.get(stageKey); + if (staged?.generation === head.generation) { + scope!.staged.delete(stageKey); + try { + await invalidateAgentFileProjection(root); + await fs.rm(root, { recursive: true, force: true }); + await fs.rename(staged.directory, root); + await mark(root, head.generation); + return true; + } finally { await fs.rm(staged.directory, { recursive: true, force: true }); } + } + if (deferrable) throw new AgentFilesRestoreDeferred(owner, root, head); const temporary = path.join(path.dirname(root), `.agent-files-${randomUUID()}`); - await fs.mkdir(temporary, { recursive: true, mode: 0o700 }); - const failed = new AbortController(); - const signal = AbortSignal.any([AbortSignal.timeout(30_000), failed.signal]); try { - // Bound open streams and file handles while avoiding one network round trip - // per file in series. Await every worker before removing a failed projection. - const entries = head.entries; - let next = 0; - const restoreEntry = async (entry: DurableAgentFileEntry) => { - signal.throwIfAborted(); - const relative = instructionPath(entry.path); - if (entry.kind === "dir") { - await assertInstructionPathSafe(temporary, `${relative}/.path-check`); - await fs.mkdir(path.join(temporary, relative), { recursive: true, mode: 0o700 }); return; - } - const filename = await assertInstructionPathSafe(temporary, relative); - await fs.mkdir(path.dirname(filename), { recursive: true, mode: 0o700 }); - const handle = await fs.open(filename, constants.O_WRONLY | constants.O_CREAT | constants.O_EXCL, entry.mode & 0o777); - try { - const hash = createHash("sha256"); - let size = 0; - if (entry.objectKey !== null) { - const object = await service.getObject(owner.companyId, entry.objectKey, { signal }); - await pipeline(object.stream, new Transform({ transform(chunk: Buffer, _encoding, callback) { - size += chunk.length; - if (size > entry.size) return callback(new Error("Agent file exceeds its committed size")); - hash.update(chunk); callback(null, chunk); - } }), handle.createWriteStream(), { signal }); - } - if (size !== entry.size || hash.digest("hex") !== entry.sha256) throw new Error("Agent file does not match its committed digest"); - } finally { await handle.close(); } - }; - const workers = Array.from({ length: Math.min(8, entries.length) }, async () => { - try { - while (next < entries.length) { - signal.throwIfAborted(); - await restoreEntry(entries[next++]!); - } - } catch (error) { - failed.abort(error); - throw error; - } - }); - await Promise.allSettled(workers); - signal.throwIfAborted(); + await downloadAgentFiles(service, owner, head.entries, temporary); await invalidateAgentFileProjection(root); await fs.rm(root, { recursive: true, force: true }); await fs.rename(temporary, root); @@ -158,6 +132,88 @@ export async function restoreDurableAgentFiles(tx: Tx, owner: Owner, root: strin } finally { await fs.rm(temporary, { recursive: true, force: true }); } } +async function downloadAgentFiles(service: ReturnType, owner: Pick, + entries: DurableAgentFileEntry[], temporary: string) { + await fs.mkdir(temporary, { recursive: true, mode: 0o700 }); + const failed = new AbortController(); + const signal = AbortSignal.any([AbortSignal.timeout(30_000), failed.signal]); + let next = 0; + const restoreEntry = async (entry: DurableAgentFileEntry) => { + signal.throwIfAborted(); + const relative = instructionPath(entry.path); + if (entry.kind === "dir") { + await assertInstructionPathSafe(temporary, `${relative}/.path-check`); + await fs.mkdir(path.join(temporary, relative), { recursive: true, mode: 0o700 }); return; + } + const filename = await assertInstructionPathSafe(temporary, relative); + await fs.mkdir(path.dirname(filename), { recursive: true, mode: 0o700 }); + const handle = await fs.open(filename, constants.O_WRONLY | constants.O_CREAT | constants.O_EXCL, entry.mode & 0o777); + try { + const hash = createHash("sha256"); + let size = 0; + if (entry.objectKey !== null) { + const object = await service.getObject(owner.companyId, entry.objectKey, { signal }); + await pipeline(object.stream, new Transform({ transform(chunk: Buffer, _encoding, callback) { + size += chunk.length; + if (size > entry.size) return callback(new Error("Agent file exceeds its committed size")); + hash.update(chunk); callback(null, chunk); + } }), handle.createWriteStream(), { signal }); + } + if (size !== entry.size || hash.digest("hex") !== entry.sha256) throw new Error("Agent file does not match its committed digest"); + } finally { await handle.close(); } + }; + const workers = Array.from({ length: Math.min(8, entries.length) }, async () => { + try { + while (next < entries.length) { + signal.throwIfAborted(); + await restoreEntry(entries[next++]!); + } + } catch (error) { + failed.abort(error); + throw error; + } + }); + await Promise.allSettled(workers); + signal.throwIfAborted(); +} + +class AgentFilesRestoreDeferred extends Error { + constructor(readonly owner: Owner, readonly root: string, readonly head: Head) { super("agent_files_restore_deferred"); } +} +type StagedAgentFiles = { generation: string; directory: string }; +type RestoreScope = { staged: Map; deferring: boolean; used: boolean }; +const restoreScope = new AsyncLocalStorage(); +const RESTORE_DEFERRALS = 2; + +export async function withAgentFilesRestoredBeforeLock(work: () => Promise): Promise { + const scope: RestoreScope = { staged: new Map(), deferring: true, used: false }; + try { + for (let attempt = 0; ; attempt++) { + scope.deferring = attempt < RESTORE_DEFERRALS; + scope.used = false; + try { + return await restoreScope.run(scope, work); + } catch (error) { + if (!(error instanceof AgentFilesRestoreDeferred)) throw error; + const stageKey = `${error.owner.companyId}:${error.owner.id}:${error.root}`; + const previous = scope.staged.get(stageKey); + scope.staged.delete(stageKey); + if (previous) await fs.rm(previous.directory, { recursive: true, force: true }); + const directory = path.join(path.dirname(error.root), `.agent-files-${randomUUID()}`); + try { + await downloadAgentFiles(storage(), error.owner, error.head.entries, directory); + scope.staged.set(stageKey, { generation: error.head.generation, directory }); + } catch (failure) { + await fs.rm(directory, { recursive: true, force: true }); + throw failure; + } + } + } + } finally { + await Promise.all([...scope.staged.values()].map(staged => fs.rm(staged.directory, { recursive: true, force: true }))); + } +} + /** Objects uploaded before the agent row lock, by content (`sha256:size`). */ export type PreparedAgentFileObjects = ReadonlyMap; const contentKey = (sha256: string, size: number) => `${sha256}:${size}`; diff --git a/server/src/services/agent-file-store.ts b/server/src/services/agent-file-store.ts index a5813d54317..06074ce50b9 100644 --- a/server/src/services/agent-file-store.ts +++ b/server/src/services/agent-file-store.ts @@ -16,7 +16,7 @@ import { authorizeInstructionCommit, authorizeInstructionRead } from "./agent-in import { assertInstructionPathSafe, instructionPath, instructionBytes, materializeInstructionBytes, readInstructionBytes, MAX_INSTRUCTION_BYTES } from "./agent-instruction-files.js"; import { agentInstructionsBundleMode, deriveBundleState, resolveManagedInstructionsRoot } from "./agent-instructions.js"; import type { AuthorizationActor } from "./authorization.js"; -import { agentFileProjectionCurrent, durableAgentFilesEnabled, invalidateAgentFileProjection, persistDurableAgentFiles, restoreDurableAgentFiles, uploadAgentFilesBeforeCommit, type PreparedAgentFileObjects } from "./agent-file-durability.js"; +import { agentFileProjectionCurrent, durableAgentFilesEnabled, invalidateAgentFileProjection, persistDurableAgentFiles, restoreDurableAgentFiles, uploadAgentFilesBeforeCommit, withAgentFilesRestoredBeforeLock, type PreparedAgentFileObjects } from "./agent-file-durability.js"; type Tx = Parameters[0]>[0]; type Agent = typeof agents.$inferSelect; @@ -195,7 +195,7 @@ export function agentFileStore(db: Db) { async function locked(companyId: string, agentId: string, actor: AuthorizationActor, write: boolean, fn: (tx: Tx, agent: Agent, root: string, bound: AuthorizationActor, onRollback: (restore: () => Promise) => void) => Promise, texts: string[] = [], prepared?: PreparedAgentFileObjects) { - return db.transaction(async tx => { + return withAgentFilesRestoredBeforeLock(() => db.transaction(async tx => { if (write) await assertAgentInteraction(tx, { companyId, actorAgentId: actor.type === "agent" ? actor.agentId : null, operation: "configure", targetAgentId: agentId, lockAgentIds: texts.flatMap(extractAgentMentionIds), lock: "update" }); const [agent] = await tx.select().from(agents).where(and(eq(agents.companyId, companyId), eq(agents.id, agentId))).for("update"); @@ -212,7 +212,7 @@ export function agentFileStore(db: Db) { await restore?.(); throw error; } - }); + })); } async function audit(tx: Tx, agent: Agent, actor: AuthorizationActor, details: Record) { const origin = currentPluginInteractionOrigin(); diff --git a/server/src/services/agent-instruction-revisions.ts b/server/src/services/agent-instruction-revisions.ts index babbbc11e4d..cffe2bca42e 100644 --- a/server/src/services/agent-instruction-revisions.ts +++ b/server/src/services/agent-instruction-revisions.ts @@ -4,6 +4,7 @@ import { assertAgentInteraction } from "./agent-interaction-policy.js"; import { updateAgentConfigurationInTransaction } from "./agent-configuration-transaction.js"; import { createHash } from "node:crypto"; import { adoptAgentFiles, agentFileToken, isAgentFileSaveRecorded } from "./agent-file-store.js"; +import { withAgentFilesRestoredBeforeLock } from "./agent-file-durability.js"; import { invalidateAgentFileProjection, persistDurableAgentFiles } from "./agent-file-durability.js"; import { and, desc, eq, getTableColumns, lt, or, sql } from "drizzle-orm"; import { @@ -193,28 +194,28 @@ export function agentInstructionRevisionService(db: Db) { target: InstructionTarget, actor: AuthorizationActor, ): Promise { - return db.transaction(async (tx) => { + return withAgentFilesRestoredBeforeLock(() => db.transaction(async (tx) => { const state = await lockTarget(tx, target, actor); const bound = await authorizeRead(tx, actor, target); const row = await currentFile(tx, target, state, bound); return row ? snapshot(row) : null; - }); + })); } /** Trusted orchestration read for the already authorized run target. This * never seeds or writes, and is not exposed through HTTP or model tools. */ async function readCommittedForRuntime(target: InstructionTarget): Promise { - return db.transaction(async (tx) => { + return withAgentFilesRestoredBeforeLock(() => db.transaction(async (tx) => { const state = await lockTarget(tx, target); const current = await currentFile(tx, target, state); return current ? snapshot(current) : null; - }); + })); } /** Rebuild disk from the current committed head under the same lock as commits. Safe after restart. */ async function materializeCurrent(target: InstructionTarget): Promise { - await db.transaction(async (tx) => { + await withAgentFilesRestoredBeforeLock(() => db.transaction(async (tx) => { const state = await lockTarget(tx, target); await adoptAgentFiles(tx, state.agent); - }); + })); } async function commitInternal( input: