mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Durable agent files live in object storage. Each server keeps a local copy (the projection), marked with the head generation it matches > - When the projection is stale, the next locked read or write restores it: it downloads every file from object storage. This happens for the first run of each agent on a new server (every deploy on hosted ephemeral servers) and after another server wrote > - The download ran inside the transaction that holds the `agents` row lock (`FOR UPDATE`), on the main pool for the run-start read (`readCommittedForRuntime`) and on the 2-connection file-storage pool for the store. A download can take up to 30 s > - After a deploy at 1,000 agents, this is one long row lock and one pinned connection per agent, all at once, and every writer of that agent waits behind it > - This pull request downloads the files outside the transaction and takes the lock again only to swap them in > - The benefit is that the row lock is held for milliseconds instead of the download time (0 ms observed instead of 321 ms with a 300 ms download in the test), and writers do not wait for a download ## Linked Issues or Issue Description **What existing behavior does this improve?** The restore of a stale local copy of an agent's durable files (the first run of each agent on a new server, or after another server wrote). **Subsystem affected** Server: durable agent files (`agent-file-durability.ts`, `agent-file-store.ts`, `agent-instruction-revisions.ts`). **Current behavior** The restore downloads every file from object storage inside the transaction that holds the `agents` row lock (`FOR UPDATE`) and a pooled connection. A download can take up to 30 s, and every writer of that agent waits behind it. **Proposed behavior** Under the lock, detect that a download is needed and roll back. Download the head's files, verified by digest, into a private staging directory with no lock. Run the transaction again, and swap the staged files in only when the head generation is unchanged. After two changed heads, restore under the lock as before. **Reason and benefit** After a deploy at 1,000 agents, every agent's first run restores at once. The row lock and connection are now held for milliseconds instead of the download time, and writers do not wait for a download. **Additional context** Refs the 1,000-concurrent-agents plan (`doc/plans/2026-11-13-thousand-concurrent-agents.md`). The save path had the same problem for uploads and was fixed earlier with uploads before the lock (`uploadAgentFilesBeforeCommit`); this is the restore side. Reading the head before every transaction was considered; it adds a query to every call, while the deferral costs nothing when the projection is current. ## What Changed - `agent-file-durability.ts`: the download loop moves into `downloadAgentFiles`, unchanged. `withAgentFilesRestoredBeforeLock(work)` runs `work` in a scope. Inside the scope, the first restore of an attempt that needs a download throws a private deferral error, which rolls back the transaction. The wrapper stages the download in `.agent-files-<uuid>` next to the root and runs `work` again. `restoreDurableAgentFiles` uses a staged directory only when its generation equals the head it reads under the lock. Staged directories are removed when the scope ends. - Only the first restore of each attempt can defer, so nested restores of other agents keep their old behavior. Outside the scope, the behavior is unchanged. - `agent-file-store.ts` (`locked`, so every store read, write, apply and working-copy preparation) and `agent-instruction-revisions.ts` (`readCommittedForRuntime`, `readCurrent`, `materializeCurrent`) run their transactions in the scope. ## Verification - New test `server/src/__tests__/agent-file-restore-locking.test.ts` on embedded PostgreSQL, with an object-storage fixture: - Hold time, with a 300 ms download, through `readCommittedForRuntime` (the run-start read). Before: the download ran with the agent row locked, and the row was locked 321 ms of 322 ms. After: the download ran with the row unlocked, and a sampler saw the row locked for 0 ms. - Concurrency: a board write of the same file runs while the reader's download is blocked. Before: the writer waited for the reader's download (5 s timeout). After: the writer commits a new head during the download. The reader then discards its stale staged copy and returns the writer's content. The projection is marked at the writer's generation, and no staging directory is left. - Failure: when object storage fails, the read fails once with the storage error, after one attempt made without the row lock, and leaves no staging directory. - Progress: when the head generation changes during each of two downloads, the third attempt restores under the lock as before (3 downloads), and the projection matches the final head. - `npx vitest run` for agent-file-checkpoint-locking, agent-file-checkpoints, agent-file-durability, agent-file-restore-locking, agent-instruction-revisions, agent-instruction-working-copies, agent-instructions-service, agent-directory-working-copies and agent-instructions-routes: 287 passed, 1 skipped. - `npx tsc --noEmit -p server/tsconfig.json`: passes. ## Risks - A deferred attempt runs the transaction body again. The deferral happens at the first restore, before the body writes files, and the transaction rollback undoes database work, so the second attempt starts from the same state. - A staged download uses disk next to the agent root until the scope ends, the same as the temporary directory of the old download. - If the head changes during every download, the third attempt downloads under the lock as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
441 lines
29 KiB
TypeScript
441 lines
29 KiB
TypeScript
import { updateAgentConfigurationInTransaction } from "./agent-configuration-transaction.js";
|
|
import { currentPluginInteractionOrigin } from "./plugin-interaction-context.js";
|
|
import { extractAgentMentionIds } from "@paperclipai/shared";
|
|
import { assertAgentInteraction } from "./agent-interaction-policy.js";
|
|
import { createHash } from "node:crypto";
|
|
import fs from "node:fs/promises";
|
|
import { cachedAgentFileManifest, checkpointSnapshot, type AgentFileManifest } from "./agent-file-checkpoints.js";
|
|
import { constants } from "node:fs";
|
|
import { Readable } from "node:stream";
|
|
import path from "node:path";
|
|
import { and, desc, eq, isNull, sql } from "drizzle-orm";
|
|
import { agents, activityLog, agentInstructionHeads, agentInstructionRevisions, resourceLifecycleEvents, fileStorageDb, type Db } from "@paperclipai/db";
|
|
import { captureDirectorySnapshot, mergeDirectoryWithBaseline, type DirectorySnapshot } from "@paperclipai/adapter-utils/workspace-restore-merge";
|
|
import { HttpError, conflict, notFound, unprocessable } from "../errors.js";
|
|
import { authorizeInstructionCommit, authorizeInstructionRead } from "./agent-instruction-authorization.js";
|
|
import { assertInstructionPathSafe, instructionPath, instructionBytes, materializeInstructionBytes, readInstructionBytes, MAX_INSTRUCTION_BYTES } from "./agent-instruction-files.js";
|
|
import { agentInstructionsBundleMode, deriveBundleState, resolveManagedInstructionsRoot } from "./agent-instructions.js";
|
|
import type { AuthorizationActor } from "./authorization.js";
|
|
import { agentFileProjectionCurrent, durableAgentFilesEnabled, invalidateAgentFileProjection, persistDurableAgentFiles, restoreDurableAgentFiles, uploadAgentFilesBeforeCommit, withAgentFilesRestoredBeforeLock, type PreparedAgentFileObjects } from "./agent-file-durability.js";
|
|
|
|
type Tx = Parameters<Parameters<Db["transaction"]>[0]>[0];
|
|
type Agent = typeof agents.$inferSelect;
|
|
export const AGENT_FILES_CONTRACT = "paperclip.agent-files.v1";
|
|
export const MAX_AGENT_FILE_BYTES = 256 * 1024 * 1024;
|
|
export const MAX_AGENT_DIRECTORY_BYTES = 2 * 1024 * 1024 * 1024;
|
|
export const MAX_AGENT_DIRECTORY_ENTRIES = 100_000;
|
|
export const fileHash = (bytes: Uint8Array) => createHash("sha256").update(bytes).digest("hex");
|
|
|
|
export class AgentFileLimitError extends HttpError {
|
|
constructor(message: string) {
|
|
super(422, message, { code: "AGENT_FILES_LIMIT_EXCEEDED" });
|
|
}
|
|
}
|
|
function assertFileSize(size: number, relative: string) {
|
|
if (size > MAX_AGENT_FILE_BYTES) throw new AgentFileLimitError(`Agent file "${relative}" exceeds the 256 MiB per-file limit`);
|
|
}
|
|
function assertDirectorySize(size: number, count: number) {
|
|
if (size > MAX_AGENT_DIRECTORY_BYTES) throw new AgentFileLimitError("Agent directory exceeds the 2 GiB total storage limit");
|
|
if (count > MAX_AGENT_DIRECTORY_ENTRIES) throw new AgentFileLimitError("Agent directory exceeds the 100,000-entry limit (files and folders)");
|
|
}
|
|
export function agentStorageWarning(detail: string) {
|
|
return `Agent storage is full. ${detail}. Runs can continue; remove or shrink files in AGENT_HOME to free space. Changes exceeding the storage limits will not be saved.`;
|
|
}
|
|
function storageWarning(size: number, count: number, fullFile?: string) {
|
|
if (size >= MAX_AGENT_DIRECTORY_BYTES) return agentStorageWarning("The agent folder has reached its 2 GiB limit");
|
|
if (count >= MAX_AGENT_DIRECTORY_ENTRIES) return agentStorageWarning("The agent folder has reached its 100,000-entry limit");
|
|
if (fullFile) return agentStorageWarning(`Agent file "${fullFile}" has reached its 256 MiB limit`);
|
|
return null;
|
|
}
|
|
|
|
export function agentFilePath(value: string): string {
|
|
const relative = instructionPath(value);
|
|
if (relative.split("/").includes(".paperclip-runtime") || relative === "promptTemplate.legacy.md") {
|
|
throw unprocessable(`${relative} is reserved and cannot be used for agent files`);
|
|
}
|
|
return relative;
|
|
}
|
|
|
|
/** Compatibility ETag for clients whose old schema requires a UUID. This is a
|
|
* content token, not a revision ID: no snapshot or history row is created. */
|
|
export function agentFileToken(bytes: Uint8Array): string {
|
|
return agentFileTokenFromHash(fileHash(bytes));
|
|
}
|
|
export function agentFileTokenFromHash(h: string): string {
|
|
return `${h.slice(0, 8)}-${h.slice(8, 12)}-8${h.slice(13, 16)}-a${h.slice(17, 20)}-${h.slice(20, 32)}`;
|
|
}
|
|
|
|
/** Call under the agent row lock. Restore shared authority first; legacy local
|
|
* adoption is allowed only on the server that actually owns the old directory. */
|
|
export async function adoptAgentFiles(tx: Tx, agent: Agent): Promise<string> {
|
|
if (agentInstructionsBundleMode(agent) === "external") throw unprocessable("External instructions must be migrated to managed storage first");
|
|
const root = resolveManagedInstructionsRoot(agent);
|
|
if (await restoreDurableAgentFiles(tx, agent, root)) return root;
|
|
const storedRoot = agent.adapterConfig.instructionsRootPath;
|
|
if (durableAgentFilesEnabled() && typeof storedRoot === "string"
|
|
&& (path.resolve(storedRoot) !== root || !await readInstructionBytes(root, deriveBundleState(agent).entryFile))) {
|
|
throw new HttpError(503, "Migrate managed agent files on their current server before replacing its filesystem", { code: "AGENT_FILES_MIGRATION_REQUIRED" });
|
|
}
|
|
const markerRoot = path.join(path.dirname(root), "file-sync");
|
|
const marker = "adopted.json";
|
|
await assertInstructionPathSafe(root, deriveBundleState(agent).entryFile);
|
|
await fs.mkdir(root, { recursive: true, mode: 0o700 });
|
|
const filename = await assertInstructionPathSafe(markerRoot, marker);
|
|
const adopted = await fs.readFile(filename, "utf8").catch((error: NodeJS.ErrnoException) => {
|
|
if (error.code === "ENOENT") return null;
|
|
throw error;
|
|
});
|
|
if (adopted !== null) {
|
|
if (JSON.parse(adopted).schema !== AGENT_FILES_CONTRACT) throw new Error("Unsupported agent file contract");
|
|
await persistDurableAgentFiles(tx, agent, root, true);
|
|
return root;
|
|
}
|
|
const legacy = await tx.select({ entryFile: agentInstructionHeads.entryFile, contentBase64: agentInstructionRevisions.contentBase64 })
|
|
.from(agentInstructionHeads).innerJoin(agentInstructionRevisions, eq(agentInstructionHeads.revisionId, agentInstructionRevisions.id))
|
|
.where(and(eq(agentInstructionHeads.companyId, agent.companyId), eq(agentInstructionHeads.agentId, agent.id)));
|
|
for (const entry of legacy) await materializeInstructionBytes(root, entry.entryFile, Buffer.from(entry.contentBase64, "base64"));
|
|
await materializeInstructionBytes(markerRoot, marker, Buffer.from(JSON.stringify({ schema: AGENT_FILES_CONTRACT })));
|
|
await persistDurableAgentFiles(tx, agent, root, true);
|
|
return root;
|
|
}
|
|
|
|
async function openAgentFile(root: string, relative: string) {
|
|
const filename = await assertInstructionPathSafe(root, relative);
|
|
const handle = await fs.open(filename, constants.O_RDONLY | constants.O_NOFOLLOW).catch((error: NodeJS.ErrnoException) => {
|
|
if (error.code === "ENOENT") return null;
|
|
throw error;
|
|
});
|
|
if (!handle) return null;
|
|
try {
|
|
const stat = await handle.stat();
|
|
if (!stat.isFile() || stat.nlink !== 1) throw unprocessable("Agent files must be regular files without links");
|
|
assertFileSize(stat.size, relative);
|
|
return { handle, size: stat.size };
|
|
} catch (error) { await handle.close(); throw error; }
|
|
}
|
|
|
|
/** Hash large files incrementally. Only editor-sized content is retained in RAM. */
|
|
export async function inspectAgentFile(root: string, relative: string, bufferLimit = MAX_INSTRUCTION_BYTES) {
|
|
const opened = await openAgentFile(root, relative);
|
|
if (!opened) return null;
|
|
const hash = createHash("sha256");
|
|
let chunks: Buffer[] | null = opened.size <= bufferLimit ? [] : null;
|
|
let size = 0;
|
|
try {
|
|
for await (const chunk of opened.handle.createReadStream()) {
|
|
size += chunk.length;
|
|
assertFileSize(size, relative);
|
|
hash.update(chunk);
|
|
if (size > bufferLimit) chunks = null;
|
|
chunks?.push(chunk);
|
|
}
|
|
return { size, hash: hash.digest("hex"), bytes: chunks ? Buffer.concat(chunks) : null };
|
|
} finally { await opened.handle.close(); }
|
|
}
|
|
|
|
export async function readAgentFile(root: string, relative: string): Promise<Buffer | null> {
|
|
return (await inspectAgentFile(root, relative, MAX_AGENT_FILE_BYTES))?.bytes ?? null;
|
|
}
|
|
|
|
/** Validate before staging and again after provider stop; never follow links or
|
|
* silently skip an unsupported file. Bounds apply to bytes, including binaries. */
|
|
async function scanAgentFiles(root: string, enforceLimits = true) {
|
|
await assertInstructionPathSafe(root, ".path-check");
|
|
let size = 0, count = 0;
|
|
const entries = new Set<string>();
|
|
let fullFile: string | undefined;
|
|
async function walk(dir: string) {
|
|
for (const item of await fs.readdir(path.join(root, dir), { withFileTypes: true })) {
|
|
const relative = agentFilePath(dir ? `${dir}/${item.name}` : item.name);
|
|
const stat = await fs.lstat(path.join(root, relative));
|
|
count++;
|
|
if (enforceLimits) assertDirectorySize(size, count);
|
|
entries.add(relative);
|
|
if (stat.isDirectory()) await walk(relative);
|
|
else if (stat.isFile() && stat.nlink === 1) {
|
|
size += stat.size;
|
|
if (stat.size >= MAX_AGENT_FILE_BYTES) fullFile ??= relative;
|
|
if (enforceLimits) {
|
|
assertFileSize(stat.size, relative);
|
|
assertDirectorySize(size, count);
|
|
}
|
|
} else throw unprocessable("Agent directories support regular files and directories, without links or special files");
|
|
}
|
|
}
|
|
await walk("");
|
|
return { size, entries, storageWarning: storageWarning(size, count, fullFile) };
|
|
}
|
|
|
|
export async function inspectAgentDirectory(root: string, enforceLimits = true) {
|
|
const usage = await scanAgentFiles(root, enforceLimits);
|
|
return { snapshot: await captureDirectorySnapshot(root), storageWarning: usage.storageWarning };
|
|
}
|
|
export async function snapshotAgentFiles(root: string, enforceLimits = true): Promise<DirectorySnapshot> {
|
|
return (await inspectAgentDirectory(root, enforceLimits)).snapshot;
|
|
}
|
|
|
|
export async function isAgentFileSaveRecorded(tx: Tx, agent: Agent, relative: string, contentHash: string | null): Promise<boolean> {
|
|
if (agent.status === "pending_approval" || agent.status === "terminated") return true;
|
|
const [latest] = await tx.select({ id: resourceLifecycleEvents.id }).from(resourceLifecycleEvents)
|
|
.where(and(eq(resourceLifecycleEvents.companyId, agent.companyId), eq(resourceLifecycleEvents.resourceType, "agent_runtime"), eq(resourceLifecycleEvents.resourceId, agent.id)))
|
|
.orderBy(desc(resourceLifecycleEvents.id)).limit(1);
|
|
if (!latest) return false;
|
|
const hash = sql<string>`${activityLog.details}->>'contentHash'`;
|
|
const [recorded] = await tx.select({ id: activityLog.id }).from(activityLog)
|
|
.innerJoin(resourceLifecycleEvents, and(eq(resourceLifecycleEvents.id, latest.id), eq(activityLog.createdAt, resourceLifecycleEvents.createdAt))).where(and(
|
|
eq(activityLog.companyId, agent.companyId), eq(activityLog.entityType, "agent"), eq(activityLog.entityId, agent.id), eq(activityLog.action, "agent.files_updated"),
|
|
eq(sql<string>`${activityLog.details}->>'agentRuntimeRevision'`, String(latest.id)),
|
|
eq(sql<string>`${activityLog.details}->>'path'`, relative), contentHash === null ? isNull(hash) : eq(hash, contentHash),
|
|
)).limit(1);
|
|
return Boolean(recorded);
|
|
}
|
|
|
|
export function agentFileStore(db: Db) {
|
|
db = fileStorageDb(db);
|
|
async function locked<T>(companyId: string, agentId: string, actor: AuthorizationActor, write: boolean,
|
|
fn: (tx: Tx, agent: Agent, root: string, bound: AuthorizationActor, onRollback: (restore: () => Promise<void>) => void) => Promise<T>, texts: string[] = [],
|
|
prepared?: PreparedAgentFileObjects) {
|
|
return withAgentFilesRestoredBeforeLock(() => db.transaction(async tx => {
|
|
if (write) await assertAgentInteraction(tx, { companyId, actorAgentId: actor.type === "agent" ? actor.agentId : null,
|
|
operation: "configure", targetAgentId: agentId, lockAgentIds: texts.flatMap(extractAgentMentionIds), lock: "update" });
|
|
const [agent] = await tx.select().from(agents).where(and(eq(agents.companyId, companyId), eq(agents.id, agentId))).for("update");
|
|
if (!agent) throw notFound("Agent not found");
|
|
const bound = write ? await authorizeInstructionCommit(tx, actor, agent) : await authorizeInstructionRead(tx, actor, agent);
|
|
const root = await adoptAgentFiles(tx, agent);
|
|
if (write) await invalidateAgentFileProjection(root);
|
|
let restore: (() => Promise<void>) | undefined;
|
|
try {
|
|
const result = await fn(tx, agent, root, bound, callback => { restore = callback; });
|
|
if (write) await persistDurableAgentFiles(tx, agent, root, false, prepared);
|
|
return result;
|
|
} catch (error) {
|
|
await restore?.();
|
|
throw error;
|
|
}
|
|
}));
|
|
}
|
|
async function audit(tx: Tx, agent: Agent, actor: AuthorizationActor, details: Record<string, unknown>) {
|
|
const origin = currentPluginInteractionOrigin();
|
|
await tx.insert(activityLog).values({ companyId: agent.companyId, actorType: origin ? "agent" : actor.type === "board" ? "user" : "agent",
|
|
actorId: origin?.agentId ?? (actor.type === "board" ? actor.userId : actor.agentId)!, agentId: origin?.agentId ?? (actor.type === "agent" ? actor.agentId : null),
|
|
runId: origin ? origin.runId : actor.runId, responsibleUserId: origin ? null : actor.type === "board" ? actor.userId : actor.onBehalfOfUserId,
|
|
action: "agent.files_updated", entityType: "agent", entityId: agent.id, details });
|
|
}
|
|
/** The write authorization of `locked`, without the row lock or the file
|
|
* tree, in a transaction that always rolls back. It writes nothing: an
|
|
* accepted change confirmation stays unconsumed, so only the locked commit
|
|
* of a save consumes it. It is enough for a save that changes no file, and
|
|
* it stops a save that will be refused before that save uploads any bytes. */
|
|
async function authorizeSave(companyId: string, agentId: string, actor: AuthorizationActor) {
|
|
const rollback = Symbol("authorization preview");
|
|
await db.transaction(async tx => {
|
|
await assertAgentInteraction(tx, { companyId, actorAgentId: actor.type === "agent" ? actor.agentId : null,
|
|
operation: "configure", targetAgentId: agentId });
|
|
const [agent] = await tx.select().from(agents).where(and(eq(agents.companyId, companyId), eq(agents.id, agentId)));
|
|
if (!agent) throw notFound("Agent not found");
|
|
await authorizeInstructionCommit(tx, actor, agent);
|
|
throw rollback;
|
|
}).catch(error => { if (error !== rollback) throw error; });
|
|
}
|
|
/**
|
|
* The save of a run that changed no file. When the locked file-tree step
|
|
* would change nothing (see `agentFileProjectionCurrent`), only the
|
|
* read-only authorization runs, without the row lock. Otherwise the locked
|
|
* write runs as for any save, so it adopts, restores or heals the durable
|
|
* head exactly as before.
|
|
*/
|
|
async function saveUnchanged(companyId: string, agentId: string, actor: AuthorizationActor) {
|
|
const [agent] = await db.select().from(agents).where(and(eq(agents.companyId, companyId), eq(agents.id, agentId)));
|
|
if (agent && agentInstructionsBundleMode(agent) !== "external"
|
|
&& await agentFileProjectionCurrent(db, agent, resolveManagedInstructionsRoot(agent))) {
|
|
await authorizeSave(companyId, agentId, actor);
|
|
return;
|
|
}
|
|
await locked(companyId, agentId, actor, true, async () => undefined);
|
|
}
|
|
return {
|
|
locked,
|
|
authorizeSave,
|
|
saveUnchanged,
|
|
download: async (companyId: string, agentId: string, relative: string, actor: AuthorizationActor) => {
|
|
const opened: { file: Awaited<ReturnType<typeof openAgentFile>> } = { file: null };
|
|
try {
|
|
const file = await locked(companyId, agentId, actor, false, async (_tx, _agent, root) => {
|
|
opened.file = await openAgentFile(root, instructionPath(relative));
|
|
return opened.file;
|
|
});
|
|
// Transfer outside the database lock. The open descriptor pins the file
|
|
// across concurrent atomic replacements; end bounds concurrent growth.
|
|
if (!file) return null;
|
|
if (file.size === 0) { await file.handle.close(); return { size: 0, stream: Readable.from([]) }; }
|
|
return { size: file.size, stream: file.handle.createReadStream({ end: file.size - 1 }) };
|
|
} catch (error) { await opened.file?.handle.close(); throw error; }
|
|
},
|
|
read: (companyId: string, agentId: string, relative: string, actor: AuthorizationActor) =>
|
|
locked(companyId, agentId, actor, false, (_tx, _agent, root) => readAgentFile(root, instructionPath(relative))),
|
|
rename: (input: { companyId: string; agentId: string; path: string; renameTo: string; bytes: Buffer; baseHash: string | null }, actor: AuthorizationActor) =>
|
|
locked(input.companyId, input.agentId, actor, true, async (tx, agent, root, bound, onRollback) => {
|
|
const source = agentFilePath(input.path);
|
|
const target = agentFilePath(input.renameTo);
|
|
if (source === target) throw unprocessable("Choose a different file name");
|
|
instructionBytes(input.bytes);
|
|
const previous = await inspectAgentFile(root, source);
|
|
const isRoot = deriveBundleState(agent).entryFile === source;
|
|
if ((!previous && !isRoot) || (previous?.hash ?? null) !== input.baseHash) {
|
|
throw conflict("This file changed since it was read. Reload before renaming.", { code: "AGENT_FILE_CONFLICT", path: source, currentHash: previous?.hash ?? null });
|
|
}
|
|
if (previous && !previous.bytes) throw unprocessable("This file is too large to rename in the editor");
|
|
const snapshot = await scanAgentFiles(root);
|
|
if ([...snapshot.entries].some(name => name.toLowerCase() === target.toLowerCase())) throw conflict("A file with this name already exists");
|
|
const newEntries = target.split("/").map((_part, i, parts) => parts.slice(0, i + 1).join("/")).filter(name => !snapshot.entries.has(name)).length;
|
|
assertDirectorySize(snapshot.size - (previous?.size ?? 0) + input.bytes.length, snapshot.entries.size - (previous ? 1 : 0) + newEntries);
|
|
await assertAgentInteraction(tx, { companyId: input.companyId, actorAgentId: bound.type === "agent" ? bound.agentId : null,
|
|
operation: "mention", texts: [input.bytes.toString("utf8")] });
|
|
const sourcePath = await assertInstructionPathSafe(root, source);
|
|
const targetPath = await assertInstructionPathSafe(root, target);
|
|
onRollback(async () => {
|
|
if (previous?.bytes) await materializeInstructionBytes(root, source, previous.bytes);
|
|
await fs.rm(targetPath, { force: true });
|
|
await invalidateAgentFileProjection(root);
|
|
});
|
|
await materializeInstructionBytes(root, target, input.bytes);
|
|
if (previous) await fs.unlink(sourcePath);
|
|
if (isRoot) {
|
|
await updateAgentConfigurationInTransaction(tx as unknown as Db, agent.id, {
|
|
adapterConfig: { ...agent.adapterConfig, instructionsBundleMode: "managed",
|
|
instructionsRootPath: root, instructionsEntryFile: target, instructionsFilePath: path.join(root, target) },
|
|
}, { allowPendingApprovalConfigUpdate: true, recordRevision: {
|
|
createdByAgentId: bound.type === "agent" ? bound.agentId : null,
|
|
createdByUserId: bound.type === "board" ? bound.userId : null,
|
|
source: "instructions_rename",
|
|
} });
|
|
}
|
|
await audit(tx, agent, bound, { path: source, contentHash: null, renamedTo: target });
|
|
await audit(tx, agent, bound, { path: target, contentHash: fileHash(input.bytes), created: true, renamedFrom: source });
|
|
}, [input.bytes.toString("utf8")]),
|
|
write: (input: { companyId: string; agentId: string; path: string; bytes: Buffer | null; baseHash: string | null }, actor: AuthorizationActor) =>
|
|
locked(input.companyId, input.agentId, actor, true, async (tx, agent, root, bound) => {
|
|
const relative = agentFilePath(input.path);
|
|
if (input.bytes) assertFileSize(input.bytes.length, relative);
|
|
const previous = await inspectAgentFile(root, relative);
|
|
const currentHash = previous?.hash ?? null;
|
|
const incomingHash = input.bytes === null ? null : fileHash(input.bytes);
|
|
if (currentHash === incomingHash) {
|
|
if (!await isAgentFileSaveRecorded(tx, agent, relative, incomingHash)) await audit(tx, agent, bound, { path: relative, contentHash: incomingHash });
|
|
return { contentHash: currentHash, changed: false };
|
|
}
|
|
if (input.bytes && input.bytes.length <= MAX_INSTRUCTION_BYTES) {
|
|
await assertAgentInteraction(tx, { companyId: input.companyId, actorAgentId: bound.type === "agent" ? bound.agentId : null,
|
|
operation: "mention", texts: [input.bytes.toString("utf8")] });
|
|
}
|
|
if (currentHash !== input.baseHash) throw conflict("This file changed since it was read. Reload before saving.", { code: "AGENT_FILE_CONFLICT", path: relative, currentHash });
|
|
if (input.bytes === null && relative === deriveBundleState(agent).entryFile) throw unprocessable("The configured instruction entry cannot be deleted");
|
|
if (input.bytes !== null) {
|
|
if (relative === deriveBundleState(agent).entryFile) instructionBytes(input.bytes);
|
|
// A quota check needs metadata only. Do not hash unrelated large
|
|
// files while holding the editor's row lock for a one-file save.
|
|
const snapshot = await scanAgentFiles(root);
|
|
const total = snapshot.size + input.bytes.length - (previous?.size ?? 0);
|
|
const newEntries = relative.split("/").map((_part, i, parts) => parts.slice(0, i + 1).join("/")).filter(name => !snapshot.entries.has(name)).length;
|
|
assertDirectorySize(total, snapshot.entries.size + newEntries);
|
|
}
|
|
if (input.bytes === null) await fs.unlink(await assertInstructionPathSafe(root, relative));
|
|
else await materializeInstructionBytes(root, relative, input.bytes);
|
|
await audit(tx, agent, bound, { path: relative, contentHash: incomingHash, created: previous === null && input.bytes !== null });
|
|
return { contentHash: incomingHash, changed: true };
|
|
}, input.bytes && input.bytes.length <= MAX_INSTRUCTION_BYTES ? [input.bytes.toString("utf8")] : []),
|
|
apply: async (input: { companyId: string; agentId: string; sourceDir: string; baseline: DirectorySnapshot; checkpoint?: AgentFileManifest }, actor: AuthorizationActor) => {
|
|
const incoming = input.checkpoint ? checkpointSnapshot(input.checkpoint) : await snapshotAgentFiles(input.sourceDir);
|
|
const [target] = await db.select().from(agents).where(and(eq(agents.companyId, input.companyId), eq(agents.id, input.agentId)));
|
|
if (!target) throw notFound("Agent not found");
|
|
const requestedEntry = deriveBundleState(target).entryFile;
|
|
const changedMentions = new Map<string, { hash: string; text: string }>();
|
|
if (actor.type === "agent" || currentPluginInteractionOrigin()) {
|
|
for (const [name, entry] of incoming.entries) {
|
|
if (entry.kind !== "file" || JSON.stringify(entry) === JSON.stringify(input.baseline.entries.get(name))) continue;
|
|
if ((await fs.stat(await assertInstructionPathSafe(input.sourceDir, name))).size > MAX_INSTRUCTION_BYTES) continue;
|
|
const bytes = (await inspectAgentFile(input.sourceDir, name))?.bytes;
|
|
const mentions = bytes ? extractAgentMentionIds(bytes.toString("utf8")) : [];
|
|
if (mentions.length) changedMentions.set(name, { hash: fileHash(bytes!), text: mentions.map(id => `[@agent](agent://${id})`).join(" ") });
|
|
}
|
|
}
|
|
// Upload the run's changed bytes before the row lock, after a preview of
|
|
// the same write authorization. The commit below runs that authorization,
|
|
// validates the tree and records only objects with matching digests.
|
|
if (input.checkpoint) await authorizeSave(input.companyId, input.agentId, actor);
|
|
const prepared = input.checkpoint ? await uploadAgentFilesBeforeCommit(db, target, input.sourceDir, input.checkpoint.entries
|
|
.filter(([name, entry]) => entry.kind === "file" && JSON.stringify(incoming.entries.get(name)) !== JSON.stringify(input.baseline.entries.get(name)))
|
|
.map(([name, entry]) => ({ path: name, size: entry.size!, sha256: entry.hash! }))) : undefined;
|
|
return locked(input.companyId, input.agentId, actor, true, async (tx, agent, root, bound) => {
|
|
if (deriveBundleState(agent).entryFile !== requestedEntry) throw conflict("Agent instructions entry changed; retry the file synchronization");
|
|
const entryPath = deriveBundleState(agent).entryFile;
|
|
if (incoming.entries.get(entryPath)?.kind !== "file") throw unprocessable("The configured instruction entry cannot be deleted");
|
|
if (!input.checkpoint || JSON.stringify(incoming.entries.get(entryPath)) !== JSON.stringify(input.baseline.entries.get(entryPath))) {
|
|
const entry = await readInstructionBytes(input.sourceDir, entryPath);
|
|
if (entry === null) throw unprocessable("The configured instruction entry cannot be deleted");
|
|
instructionBytes(entry);
|
|
}
|
|
// Previously saved/imported bytes must remain readable, including when
|
|
// over quota. Enforce limits on the incoming and resulting tree so a
|
|
// run can delete files to recover instead of being locked out forever.
|
|
const current = input.checkpoint ? checkpointSnapshot(await cachedAgentFileManifest(root)) : await snapshotAgentFiles(root, false);
|
|
const texts = [...changedMentions].flatMap(([name, value]) => {
|
|
const prior = current.entries.get(name);
|
|
return prior?.kind === "file" && prior.hash === value.hash ? [] : [value.text];
|
|
});
|
|
await assertAgentInteraction(tx, { companyId: input.companyId, actorAgentId: bound.type === "agent" ? bound.agentId : null,
|
|
operation: "mention", texts });
|
|
// Rebase only the run's changed paths onto the current tree. This makes
|
|
// same-file edits/deletions last-sync-wins while untouched files retain
|
|
// changes from other runs. Ancestors may need recreating after a writer
|
|
// replaced a directory with a file.
|
|
const entries = new Map(input.baseline.entries);
|
|
const changed = [...new Set([...input.baseline.entries, ...incoming.entries].map(([name]) => name))]
|
|
.filter(name => JSON.stringify(input.baseline.entries.get(name)) !== JSON.stringify(incoming.entries.get(name)));
|
|
for (const name of changed) {
|
|
const present = current.entries.get(name);
|
|
if (present) entries.set(name, present); else entries.delete(name);
|
|
if (incoming.entries.has(name)) {
|
|
for (let parent = path.posix.dirname(name); parent !== "."; parent = path.posix.dirname(parent)) {
|
|
if (current.entries.get(parent)?.kind === "dir") continue;
|
|
const before = current.entries.get(parent);
|
|
if (before) entries.set(parent, before); else entries.delete(parent);
|
|
}
|
|
}
|
|
}
|
|
const applyBaseline = { ...input.baseline, entries };
|
|
const finalEntries = new Map([...current.entries].map(([name, value]) => [name, { value, root }]));
|
|
for (const name of changed) if (!incoming.entries.has(name)) finalEntries.delete(name);
|
|
for (const [name, value] of incoming.entries) {
|
|
if (JSON.stringify(input.baseline.entries.get(name)) === JSON.stringify(value)) continue;
|
|
if (value.kind !== "dir") for (const child of finalEntries.keys()) if (child.startsWith(`${name}/`)) finalEntries.delete(child);
|
|
finalEntries.set(name, { value, root: input.sourceDir });
|
|
for (let parent = path.posix.dirname(name); parent !== "."; parent = path.posix.dirname(parent)) {
|
|
if (finalEntries.get(parent)?.value.kind !== "dir") finalEntries.set(parent, { value: { kind: "dir" }, root: input.sourceDir });
|
|
}
|
|
}
|
|
// An unchanged file created by another run can keep a removed folder
|
|
// alive. Count those surviving parent directories in quota preflight.
|
|
for (const name of finalEntries.keys()) {
|
|
for (let parent = path.posix.dirname(name); parent !== "."; parent = path.posix.dirname(parent)) {
|
|
if (!finalEntries.has(parent)) finalEntries.set(parent, { value: { kind: "dir" }, root });
|
|
}
|
|
}
|
|
let total = 0;
|
|
let fullFile: string | undefined;
|
|
for (const [name, item] of finalEntries) if (item.value.kind === "file") {
|
|
const size = (await fs.stat(path.join(item.root, name))).size;
|
|
assertFileSize(size, name);
|
|
if (size >= MAX_AGENT_FILE_BYTES) fullFile ??= name;
|
|
total += size;
|
|
}
|
|
assertDirectorySize(total, finalEntries.size);
|
|
await mergeDirectoryWithBaseline({ ...input, baseline: applyBaseline, targetDir: root,
|
|
...(input.checkpoint ? { snapshots: { source: incoming, current } } : {}) });
|
|
await audit(tx, agent, bound, { sourceRunId: currentPluginInteractionOrigin()?.runId ?? actor.runId, contract: AGENT_FILES_CONTRACT });
|
|
return { storageWarning: storageWarning(total, finalEntries.size, fullFile) };
|
|
}, [...changedMentions.values()].map(value => value.text), prepared);
|
|
},
|
|
};
|
|
}
|