mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 20:29:18 +01:00
feat(agents): persist agent files across tasks without revision history (#14420)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An agent needs personal files across tasks and sessions. > - AGENTS.md is one file in that directory. Supporting files need the same persistence. > - The Instructions Editor and agent runs must share one current directory. > - Concurrent runs should apply only the files they change. The last sync of the same file wins. > - This pull request uses existing file transport and removes temporary copies after sync. > - Old instruction-only sessions keep their restore contract. New saves do not create revision history. ## Linked Issues or Issue Description Refs #14325. This replaces its revision-oriented design with persistent agent files. Keep #14325 unmerged. Transport prerequisite #14416 merged first at `d172197117a14b80a1eb2d2835a0e7cce2679656`. This PR now targets master and remains below 100 changed files. Related work: #4513 and #8798 cover instruction tooling. This change handles run synchronization, cross-task personal files, browser editing, and old-session restoration. ## What Changed - Keep one current directory per company and agent. Point AGENT_HOME at a temporary working copy for each active run. Keep task files and provider HOME separate. - Restore text, binary files, and nested folders through workspace transport. Exclude remote agent files from task Git snapshots with a self-ignoring file inside the reserved runtime directory; never write through repository-controlled Git metadata. - Collect after the provider and child processes have stopped. Keep resumable conversation state. - Apply changed and deleted files under the agent lock. The last sync wins for the same file. Unrelated concurrent changes survive. - Remove temporary copies after successful sync, rejected sync, and staging failure. Register ownership before copying so restart recovery can remove interrupted preparation. Retry transient synchronization up to three times. Preserve the original remote lease reference until deletion succeeds; restart cleanup never acquires a replacement sandbox. Do not create captured directories or a conflict-review queue for new runs. - Keep browser editing, stale-draft protection, and streaming binary downloads. Keep the instruction entry and text editor limited to 1 MiB. - Keep historical agent-folder sync failures on their affected runs instead of repeating them above current saved instructions. Preserve legacy candidate review and current browser-save errors. Avoid duplicate quota warnings while retaining separate sync failures when they describe a different problem. - Require target-scoped caller grants for peer instruction access, while preserving self edits, responsible-user checks, and protected-change consent. - Treat full storage as a nonblocking run warning, never an agent pause or run-admission failure. Restore already-over-quota saved folders so ordinary agent cleanup can recover; warn on each run until cleanup. The run detail view shows the warning. - Allow 256 MiB per file, 2 GiB per directory, and 100,000 entries. Hash large files as streams. Check editor-save quotas with metadata instead of hashing unrelated files. - Preserve old native inputs, instruction-only copies, paths, digests, and pending legacy candidates. Adopt old revision heads once. New writes do not append history rows. - Add idempotent migration 0287 and verify upgrades from the preview tables and receipts. - Add nine interactive stories under **Agents / Persistent files**, including automatic incoming edits, stale browser drafts, and storage-limit diagnostics. ## Verification - Merge candidate: `4f5390107ec6ffd80a76d1d2e85530e66f21d079`, after merging current master and the landed transport prerequisite. Integration required no manual conflict resolution; the feature remains 99 changed files. Full workspace typecheck, production build, token gates, and 715 focused tests passed on this merge candidate. Fresh Greptile review is 5/5 with no unresolved findings. All 55 checks passed, with four conditional skips, including the build, typecheck, browser E2E, and canary dry run. A single retry recovered four jobs interrupted by runner shutdowns; no source changes were required. - Historical-warning UI fix: all 6,834 UI tests across 640 files passed, including regression coverage for three old failures, legacy preserved edits, and warnings scoped to the affected run. Full workspace typecheck, production build, Storybook build, and token gates passed. Browser-verified Storybook playtests passed for Historical Failures After Successful Save, Storage Limit, and Full Storage Run Warning. - Review follow-ups at `4e20c9fb2`: all 18 focused tests passed, including external Git directories, linked worktrees, symlinks, hardlinks, and distinct I/O failures alongside storage warnings. Server and UI typechecks, token gates, and the production build passed. - Storage warning regressions at `0724f3012`: all 33 directory tests and all five heartbeat-list tests passed, with no skips in their successful runs. They cover repeated runs while full, an already-over-quota saved folder, cleanup, warnings retained after unrelated save failures, and bounded warnings in large result JSON. Server typecheck passed after the final warning fixes. - Full workspace typecheck, production build, and token gates passed during this follow-up. Product E2E harness: 631 tests passed across 52 files; harness typecheck passed. Earlier native session/context and directory/legacy collection suites passed 537 tests; Runner unit/transport suites passed 329 tests. - **Real E2E at `0724f3012` (before this follow-up):** legacy local Codex and native Daytona Codex each passed six tasks, one server restart, seven independent assertions, and cleanup verification. Both prove browser-to-agent edits, agent-to-browser edits, nested/binary restoration, per-file last-sync-wins, a successful run after an oversized save rejection, and cleanup clearing the warning. - Native local Codex also passed the six-task quota flow before the final warning-retention fixes. That pass began at `918d1ed02` while the bounded-result warning fix was being edited, so it is not claimed as exact-final-head evidence. Its final-head rerun failed during embedded PostgreSQL bootstrap before any provider run: the macOS host had 87,365 of 87,381 SysV semaphores occupied. No unrelated services or kernel limits were changed. - The final-source report intentionally records **2/3 cells passed**, preserving the blocked native-local attempt: `tests/runner-e2e/results/agent-files-quota-final-20260928-report/`. Earlier failed attempts and provenance notes remain under `tests/runner-e2e/results/agent-files-quota-final-20260928-input/` and the original campaign directories. - Daytona used immutable image `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:5643f0d801417cae3581833a1a3bc6715b325e028602738d2652c44cac5dc6bf` and its exact Linux runner binary. Controller source is `0724f3012`; image source is recorded separately. - Legacy-session compatibility and all three ACP Stop/resume browser regressions passed on the prior validated feature head `169fab46d5af21caa2269b4c1b29b69c933a6951`. They assert the same provider session is retained and interrupted writes are not replayed. Migration upgrade tests also passed earlier. - Nine interactive stories are under **Agents / Persistent files**, including **Full Storage Run Warning**. Its playtest and visual browser inspection passed; the warning states that runs continue and the editor remains available. - Prior-head checks on `4e20c9fb2`: 55 passed, two conditional jobs skipped, no failures or pending checks. All eight browser E2E shards and their aggregate passed. Fresh Greptile review is 5/5 with no findings; all review threads are resolved, the security scan passed, and GitHub reports no merge conflicts. - The broad local follow-up test run was interrupted after host semaphore exhaustion affected isolated PostgreSQL instances. It also encountered the existing macOS long-path fixture failure and two timeout failures. This is not a claim that the full local suite passed. Logs are retained; focused storage/warning tests passed. ## Risks - A later sync can overwrite an earlier edit to the same file, including a saved browser edit. There is no text merge or retained version. This is the intended last-sync-wins policy. - A save that exceeds a storage limit is rejected and its temporary copy is discarded. The run itself continues normally, and later runs restore the last saved files with a warning until cleanup. Transient sync failures get bounded retries. An I/O failure partway through a sync can leave some files updated; a failed receipt does not claim whole-folder success. - Larger folders increase copy time, network traffic, and temporary disk usage. Active runs still need working copies. Terminal runs do not accumulate archives. Operators must provision disk for agents and configured concurrency; these limits are not company-wide quotas. - A restored old native session remains instruction-only until a fresh session starts. Its original conflict fence and existing pending candidates remain compatible. - Provider processes close at the collection boundary. Conversation resume remains available, but warm process reuse is lost. - Backups must include the instance filesystem and database. External bundles keep their existing behavior until explicitly moved to managed storage. ## Model Used OpenAI Codex, GPT-6 family. The session does not expose a more specific model ID or context-window size. Reasoning, code execution, and browser tools assisted this change. Real provider E2E uses `gpt-5.6-sol`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Fry (Paperclip) <noreply@paperclip.ing>
This commit is contained in:
1 parent
d172197117
commit
3ca196b0a6
99 files changed
+7948
-261
No files matched your search
@@ -0,0 +1,539 @@
|
||||
import fs from "node:fs/promises";
|
||||
import { execFile as execFileCallback } from "node:child_process";
|
||||
import { promisify } from "node:util";
|
||||
import * as executionTargetTools from "@paperclipai/adapter-utils/execution-target";
|
||||
import * as ssh from "@paperclipai/adapter-utils/ssh";
|
||||
const execFile = promisify(execFileCallback);
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { agentFileStore, fileHash, inspectAgentFile, snapshotAgentFiles, MAX_AGENT_FILE_BYTES, MAX_AGENT_DIRECTORY_BYTES, MAX_AGENT_DIRECTORY_ENTRIES } from "../services/agent-file-store.js";
|
||||
import { agents, companies, authUsers, companyMemberships, principalPermissionGrants, heartbeatRuns, environmentLeases, environments, agentInstructionWorkingCopies, agentInstructionRevisions, agentInstructionHeads, createDb } from "@paperclipai/db";
|
||||
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { agentInstructionRevisionService } from "../services/agent-instruction-revisions.js";
|
||||
import { agentInstructionWorkingCopyService, instructionWorkingCopyGuidance } from "../services/agent-instruction-working-copies.js";
|
||||
import { resolveManagedInstructionsRoot } from "../services/agent-instructions.js";
|
||||
import { buildNativeRuntimeContext } from "../services/native-runtime/runtime-context.js";
|
||||
import type { EnvironmentRuntimeService } from "../services/environment-runtime.js";
|
||||
import { remoteTerminationReceipt } from "../services/remote-execution-termination.js";
|
||||
|
||||
describe("persistent agent directories", () => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let copies: ReturnType<typeof agentInstructionWorkingCopyService>;
|
||||
let revisions: ReturnType<typeof agentInstructionRevisionService>;
|
||||
const previousHome = process.env.PAPERCLIP_HOME;
|
||||
let home: string;
|
||||
let companyId: string, agentId: string, userId: string, root: string;
|
||||
const entryFile = "policy/INSTRUCTIONS.txt";
|
||||
const initial = "\uFEFF# Original\r\n☃\n";
|
||||
const target = () => ({ companyId, agentId });
|
||||
const board = () => ({ type: "board" as const, userId, source: "session" as const });
|
||||
async function run() {
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand", responsibleUserId: userId });
|
||||
return (await copies.prepare({ ...target(), runId, cwd: home }))!;
|
||||
}
|
||||
beforeAll(async () => {
|
||||
home = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "instruction-working-copies-")));
|
||||
process.env.PAPERCLIP_HOME = home;
|
||||
database = await startEmbeddedPostgresTestDatabase("instruction-copies-db-");
|
||||
db = createDb(database.connectionString);
|
||||
copies = agentInstructionWorkingCopyService(db);
|
||||
revisions = agentInstructionRevisionService(db);
|
||||
}, 90_000);
|
||||
afterAll(async () => {
|
||||
if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome;
|
||||
await database?.cleanup();
|
||||
if (home) {
|
||||
const writable = async (dir: string) => {
|
||||
await fs.chmod(dir, 0o700);
|
||||
for (const entry of await fs.readdir(dir, { withFileTypes: true })) if (entry.isDirectory()) await writable(path.join(dir, entry.name));
|
||||
};
|
||||
await writable(home);
|
||||
await fs.rm(home, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
beforeEach(async () => {
|
||||
companyId = randomUUID(); agentId = randomUUID(); userId = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name: "Instruction tests", issuePrefix: randomUUID().slice(0, 8) });
|
||||
await db.insert(authUsers).values({ id: userId, name: "Editor", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() });
|
||||
root = resolveManagedInstructionsRoot({ ...target(), id: agentId, name: "Target", adapterConfig: {} });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Target", adapterConfig: { instructionsBundleMode: "managed", instructionsRootPath: root, instructionsEntryFile: entryFile } });
|
||||
await db.insert(companyMemberships).values([
|
||||
{ companyId, principalType: "user", principalId: userId, membershipRole: "operator" },
|
||||
{ companyId, principalType: "agent", principalId: agentId, membershipRole: "member" },
|
||||
]);
|
||||
await db.insert(principalPermissionGrants).values({ companyId, principalType: "user", principalId: userId, permissionKey: "agents:configure", scope: { agentIds: [agentId] } });
|
||||
await fs.mkdir(path.dirname(path.join(root, entryFile)), { recursive: true });
|
||||
await fs.writeFile(path.join(root, entryFile), initial);
|
||||
});
|
||||
|
||||
it.each([".paperclip-runtime/state", "notes/.paperclip-runtime/state", "promptTemplate.legacy.md"])("rejects reserved board path %s before mutation", async (reserved) => {
|
||||
await expect(agentFileStore(db).write({ ...target(), path: reserved, bytes: Buffer.from("reserved"), baseHash: null }, board())).rejects.toMatchObject({ status: 422 });
|
||||
await expect(fs.stat(path.join(root, reserved))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect(await run()).toBeTruthy();
|
||||
});
|
||||
|
||||
it("round trips nested text, empty directories and binary bytes independently of task files", async () => {
|
||||
const first = await run();
|
||||
expect(first.localRoot.startsWith(path.join(home, ".paperclip-runtime"))).toBe(false);
|
||||
await fs.mkdir(path.join(first.localRoot, "notes", "empty"), { recursive: true });
|
||||
await fs.writeFile(path.join(first.localRoot, "notes", "fact.txt"), "remember me");
|
||||
const bytes = Buffer.from([0, 255, 17, 128, 9]);
|
||||
await fs.writeFile(path.join(first.localRoot, "image.bin"), bytes);
|
||||
await fs.writeFile(path.join(home, "task-only.txt"), "not personal");
|
||||
expect((await copies.collectStopped({ companyId, runId: first.runId }))?.state).toBe("saved");
|
||||
copies = agentInstructionWorkingCopyService(db);
|
||||
const next = await run();
|
||||
expect(await fs.readFile(path.join(next.localRoot, "notes", "fact.txt"), "utf8")).toBe("remember me");
|
||||
expect(await fs.readFile(path.join(next.localRoot, "image.bin"))).toEqual(bytes);
|
||||
expect((await fs.stat(path.join(next.localRoot, "notes", "empty"))).isDirectory()).toBe(true);
|
||||
await expect(fs.stat(path.join(next.localRoot, "task-only.txt"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect(await db.select().from(agentInstructionRevisions).where(eq(agentInstructionRevisions.agentId, agentId))).toHaveLength(0);
|
||||
});
|
||||
|
||||
async function sparseFile(filename: string, size: number) {
|
||||
const handle = await fs.open(filename, "w");
|
||||
try { await handle.truncate(size); } finally { await handle.close(); }
|
||||
}
|
||||
|
||||
it("saves and restores files beyond the former file and folder limits, with streaming inspection and download", async () => {
|
||||
const first = await run();
|
||||
const size = 66 * 1024 * 1024;
|
||||
await sparseFile(path.join(first.localRoot, "large.bin"), size);
|
||||
expect((await copies.collectStopped({ companyId, runId: first.runId }))?.state).toBe("saved");
|
||||
await copies.release(companyId, first.runId);
|
||||
const next = await run();
|
||||
const restored = await inspectAgentFile(next.localRoot, "large.bin");
|
||||
expect(restored).toMatchObject({ size, bytes: null });
|
||||
const store = agentFileStore(db);
|
||||
const download = (await store.download(companyId, agentId, "large.bin", board()))!;
|
||||
expect(download.size).toBe(size);
|
||||
// The lock is released before consuming the stream. An editor replacement
|
||||
// must not change the already opened download's bytes.
|
||||
await store.write({ ...target(), path: "large.bin", bytes: Buffer.from("replacement"), baseHash: restored!.hash }, board());
|
||||
const hash = createHash("sha256");
|
||||
let downloaded = 0;
|
||||
for await (const chunk of download.stream) { downloaded += chunk.length; hash.update(chunk); }
|
||||
expect(downloaded).toBe(size);
|
||||
expect(hash.digest("hex")).toBe(restored!.hash);
|
||||
expect(await fs.readFile(path.join(root, "large.bin"), "utf8")).toBe("replacement");
|
||||
await store.write({ ...target(), path: "empty.bin", bytes: Buffer.alloc(0), baseHash: null }, board());
|
||||
const empty = (await store.download(companyId, agentId, "empty.bin", board()))!;
|
||||
expect(empty.size).toBe(0);
|
||||
for await (const _chunk of empty.stream) throw new Error("Empty download must have no chunks");
|
||||
}, 30_000);
|
||||
|
||||
it("accepts the exact 256 MiB file boundary without retaining a text buffer", async () => {
|
||||
expect(MAX_AGENT_FILE_BYTES).toBe(256 * 1024 * 1024);
|
||||
await sparseFile(path.join(root, "boundary.bin"), MAX_AGENT_FILE_BYTES);
|
||||
expect(await inspectAgentFile(root, "boundary.bin")).toMatchObject({ size: MAX_AGENT_FILE_BYTES, bytes: null });
|
||||
});
|
||||
|
||||
it("checks quota for a small editor save without reading unrelated file bytes", async () => {
|
||||
const asset = path.join(root, "large.bin");
|
||||
await sparseFile(asset, MAX_AGENT_FILE_BYTES);
|
||||
const open = vi.spyOn(fs, "open");
|
||||
try {
|
||||
await agentFileStore(db).write({ ...target(), path: "note.txt", bytes: Buffer.from("small edit"), baseHash: null }, board());
|
||||
expect(open.mock.calls.some(([filename]) => filename === asset)).toBe(false);
|
||||
expect(await fs.readFile(path.join(root, "note.txt"), "utf8")).toBe("small edit");
|
||||
} finally { open.mockRestore(); }
|
||||
});
|
||||
|
||||
it("reports an oversized run file without a partial save and removes its temporary copy", async () => {
|
||||
const copy = await run();
|
||||
await sparseFile(path.join(copy.localRoot, "too-large.bin"), MAX_AGENT_FILE_BYTES + 1);
|
||||
await fs.writeFile(path.join(copy.localRoot, entryFile), "changed instructions");
|
||||
const failed = await copies.collectStopped({ companyId, runId: copy.runId });
|
||||
expect(failed).toMatchObject({ state: "unavailable", errorCode: "AGENT_FILES_LIMIT_EXCEEDED", candidateHash: null, nextAttemptAt: null, attempts: 1 });
|
||||
expect(failed?.receipt?.storageWarning).toContain("Runs can continue");
|
||||
expect(failed?.errorMessage).toContain('"too-large.bin" exceeds the 256 MiB');
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(initial);
|
||||
await expect(fs.stat(path.join(root, "too-large.bin"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect((await copies.reportUnavailable(companyId, copy.runId))?.errorCode).toBe("AGENT_FILES_LIMIT_EXCEEDED");
|
||||
await copies.release(companyId, copy.runId);
|
||||
await expect(fs.stat(copy.localRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect((await copies.get(companyId, copy.runId))?.receipt?.baseline).toBeUndefined();
|
||||
const next = await run();
|
||||
expect(await fs.readFile(path.join(next.localRoot, entryFile), "utf8")).toBe(initial);
|
||||
await fs.writeFile(path.join(next.localRoot, "next-task.txt"), "still working");
|
||||
expect((await copies.collectStopped({ companyId, runId: next.runId }))?.state).toBe("saved");
|
||||
expect(await fs.readFile(path.join(root, "next-task.txt"), "utf8")).toBe("still working");
|
||||
});
|
||||
|
||||
it("warns on each run at the file limit and clears the warning after ordinary agent cleanup", async () => {
|
||||
await sparseFile(path.join(root, "full.bin"), MAX_AGENT_FILE_BYTES);
|
||||
const first = await run();
|
||||
expect(first.receipt?.storageWarning).toContain("256 MiB");
|
||||
expect(instructionWorkingCopyGuidance(first)).toContain("Runs can continue");
|
||||
const unchanged = await copies.collectStopped({ companyId, runId: first.runId });
|
||||
expect(unchanged).toMatchObject({ state: "unchanged", errorCode: null });
|
||||
expect(unchanged?.receipt?.storageWarning).toContain("Agent storage is full");
|
||||
copies = agentInstructionWorkingCopyService(db);
|
||||
const second = await run();
|
||||
expect(second.receipt?.storageWarning).toContain("Agent storage is full");
|
||||
await fs.unlink(path.join(second.localRoot, "full.bin"));
|
||||
await fs.writeFile(path.join(second.localRoot, "task-output.txt"), "work continues");
|
||||
expect(await copies.collectStopped({ companyId, runId: second.runId })).toMatchObject({ state: "saved", receipt: { storageWarning: null } });
|
||||
const next = await run();
|
||||
expect(next.receipt?.storageWarning).toBeNull();
|
||||
expect(await fs.readFile(path.join(next.localRoot, "task-output.txt"), "utf8")).toBe("work continues");
|
||||
}, 30_000);
|
||||
|
||||
it("starts successive runs with an already oversized saved file and lets the agent remove it", async () => {
|
||||
await sparseFile(path.join(root, "old-large.bin"), MAX_AGENT_FILE_BYTES + 1);
|
||||
const first = await run();
|
||||
expect(first.state).toBe("prepared");
|
||||
expect(first.receipt?.storageWarning).toContain("256 MiB");
|
||||
expect(await fs.readFile(path.join(first.localRoot, entryFile), "utf8")).toBe(initial);
|
||||
const stopped = await copies.collectStopped({ companyId, runId: first.runId });
|
||||
expect(stopped).toMatchObject({ state: "unavailable", errorCode: "AGENT_FILES_LIMIT_EXCEEDED" });
|
||||
await expect(fs.stat(first.localRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
copies = agentInstructionWorkingCopyService(db);
|
||||
const cleanup = await run();
|
||||
expect(cleanup.state).toBe("prepared");
|
||||
await fs.unlink(path.join(cleanup.localRoot, "old-large.bin"));
|
||||
expect(await copies.collectStopped({ companyId, runId: cleanup.runId })).toMatchObject({ state: "saved", receipt: { storageWarning: null } });
|
||||
expect((await run()).receipt?.storageWarning).toBeNull();
|
||||
}, 30_000);
|
||||
|
||||
it("starts when the existing folder exceeds its total quota and accepts agent cleanup", async () => {
|
||||
for (let index = 0; index < 8; index++) await fs.writeFile(path.join(root, `quota-part-${index}.bin`), "fixture");
|
||||
const lstat = fs.lstat.bind(fs);
|
||||
// Keep this regression small on disk while exercising the real filesystem,
|
||||
// snapshots, database receipts, restore, and merge with 2 GiB of metadata.
|
||||
const sizes = vi.spyOn(fs, "lstat").mockImplementation(async (...args: Parameters<typeof fs.lstat>) => {
|
||||
const stat = await lstat(...args);
|
||||
if (path.basename(String(args[0])).startsWith("quota-part-")) Object.assign(stat, { size: MAX_AGENT_FILE_BYTES });
|
||||
return stat;
|
||||
});
|
||||
try {
|
||||
const first = await run();
|
||||
expect(first.receipt?.storageWarning).toContain("2 GiB");
|
||||
expect((await copies.collectStopped({ companyId, runId: first.runId }))?.errorCode).toBe("AGENT_FILES_LIMIT_EXCEEDED");
|
||||
const cleanup = await run();
|
||||
for (let index = 0; index < 8; index++) await fs.unlink(path.join(cleanup.localRoot, `quota-part-${index}.bin`));
|
||||
await fs.writeFile(path.join(cleanup.localRoot, "small.txt"), "recovered");
|
||||
expect(await copies.collectStopped({ companyId, runId: cleanup.runId })).toMatchObject({ state: "saved", receipt: { storageWarning: null } });
|
||||
expect((await run()).receipt?.storageWarning).toBeNull();
|
||||
} finally { sizes.mockRestore(); }
|
||||
});
|
||||
|
||||
it("rejects more than 2 GiB in aggregate before hashing or saving any file", async () => {
|
||||
expect(MAX_AGENT_DIRECTORY_BYTES).toBe(2 * 1024 * 1024 * 1024);
|
||||
const copy = await run();
|
||||
for (let index = 0; index < 8; index++) await sparseFile(path.join(copy.localRoot, `part-${index}.bin`), MAX_AGENT_FILE_BYTES);
|
||||
// The eight allowed files exactly fill the quota; the entry is additional.
|
||||
const result = await copies.collectStopped({ companyId, runId: copy.runId });
|
||||
expect(result).toMatchObject({ state: "unavailable", errorCode: "AGENT_FILES_LIMIT_EXCEEDED" });
|
||||
expect(result?.errorMessage).toContain("2 GiB total storage limit");
|
||||
expect(await fs.readdir(root)).toEqual(["policy"]);
|
||||
});
|
||||
|
||||
it("rejects excessive entry count before snapshotting", async () => {
|
||||
const directory = path.join(home, "many-entries");
|
||||
await fs.mkdir(directory);
|
||||
await fs.writeFile(path.join(directory, "empty"), "");
|
||||
const entries = await fs.readdir(directory, { withFileTypes: true });
|
||||
// Exercise the walker bound without creating 100,001 physical files.
|
||||
const readdir = vi.spyOn(fs, "readdir").mockResolvedValue(Array(MAX_AGENT_DIRECTORY_ENTRIES + 1).fill(entries[0]));
|
||||
try {
|
||||
await expect(snapshotAgentFiles(directory)).rejects.toMatchObject({ status: 422, message: expect.stringContaining("100,000-entry limit") });
|
||||
} finally { readdir.mockRestore(); }
|
||||
}, 30_000);
|
||||
|
||||
it("keeps the entry's 1 MiB limit and cleans up failed synchronization", async () => {
|
||||
const copy = await run();
|
||||
await fs.writeFile(path.join(copy.localRoot, entryFile), "a".repeat(1024 * 1024 + 1));
|
||||
const result = await copies.collectStopped({ companyId, runId: copy.runId });
|
||||
expect(result?.state).toBe("unavailable");
|
||||
await expect(fs.stat(copy.localRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect(result?.errorMessage).toContain("at most 1 MiB");
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(initial);
|
||||
});
|
||||
|
||||
it("merges independent changes and uses the last synchronization for same-file edits", async () => {
|
||||
const a = await run(), b = await run();
|
||||
await fs.writeFile(path.join(a.localRoot, "a.txt"), "A");
|
||||
await fs.writeFile(path.join(b.localRoot, "b.txt"), "B");
|
||||
await Promise.all([a, b].map(copy => copies.collectStopped({ companyId, runId: copy.runId })));
|
||||
expect(await fs.readFile(path.join(root, "a.txt"), "utf8")).toBe("A");
|
||||
expect(await fs.readFile(path.join(root, "b.txt"), "utf8")).toBe("B");
|
||||
const c = await run(), d = await run();
|
||||
await fs.writeFile(path.join(c.localRoot, "a.txt"), "C");
|
||||
await fs.writeFile(path.join(d.localRoot, "a.txt"), "D");
|
||||
await fs.writeFile(path.join(c.localRoot, "b.txt"), "new B");
|
||||
expect((await copies.collectStopped({ companyId, runId: c.runId }))?.state).toBe("saved");
|
||||
expect((await copies.collectStopped({ companyId, runId: d.runId }))?.state).toBe("saved");
|
||||
expect(await fs.readFile(path.join(root, "a.txt"), "utf8")).toBe("D");
|
||||
expect(await fs.readFile(path.join(root, "b.txt"), "utf8")).toBe("new B");
|
||||
expect(await copies.list(companyId, agentId, board())).toEqual([]);
|
||||
for (const copy of [a, b, c, d]) await expect(fs.stat(path.dirname(copy.localRoot))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect(await fs.readdir(path.join(path.dirname(root), "file-sync", "runs"))).toEqual([]);
|
||||
});
|
||||
|
||||
it("makes concurrent edit/delete races last-sync-wins and preserves unrelated new files", async () => {
|
||||
await fs.mkdir(path.join(root, "notes"));
|
||||
await fs.writeFile(path.join(root, "notes", "old.txt"), "old");
|
||||
const deleting = await run(), editing = await run();
|
||||
await fs.rm(path.join(deleting.localRoot, "notes"), { recursive: true });
|
||||
await fs.writeFile(path.join(editing.localRoot, "notes", "old.txt"), "edited");
|
||||
await fs.writeFile(path.join(editing.localRoot, "notes", "new.txt"), "unrelated");
|
||||
await copies.collectStopped({ companyId, runId: editing.runId });
|
||||
expect((await copies.collectStopped({ companyId, runId: deleting.runId }))?.state).toBe("saved");
|
||||
await expect(fs.stat(path.join(root, "notes", "old.txt"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect(await fs.readFile(path.join(root, "notes", "new.txt"), "utf8")).toBe("unrelated");
|
||||
const remove = await run(), change = await run();
|
||||
await fs.unlink(path.join(remove.localRoot, "notes", "new.txt"));
|
||||
await fs.writeFile(path.join(change.localRoot, "notes", "new.txt"), "restored by later edit");
|
||||
await copies.collectStopped({ companyId, runId: remove.runId });
|
||||
await copies.collectStopped({ companyId, runId: change.runId });
|
||||
expect(await fs.readFile(path.join(root, "notes", "new.txt"), "utf8")).toBe("restored by later edit");
|
||||
});
|
||||
|
||||
it("handles a concurrently replaced parent and keeps the later file change", async () => {
|
||||
await fs.mkdir(path.join(root, "notes"));
|
||||
await fs.writeFile(path.join(root, "notes", "fact.txt"), "old");
|
||||
const replacing = await run(), editing = await run();
|
||||
await fs.rm(path.join(replacing.localRoot, "notes"), { recursive: true });
|
||||
await fs.writeFile(path.join(replacing.localRoot, "notes"), "now a file");
|
||||
await fs.writeFile(path.join(editing.localRoot, "notes", "fact.txt"), "new fact");
|
||||
await copies.collectStopped({ companyId, runId: replacing.runId });
|
||||
expect(await fs.readFile(path.join(root, "notes"), "utf8")).toBe("now a file");
|
||||
expect((await copies.collectStopped({ companyId, runId: editing.runId }))?.state).toBe("saved");
|
||||
expect(await fs.readFile(path.join(root, "notes", "fact.txt"), "utf8")).toBe("new fact");
|
||||
});
|
||||
|
||||
it("cleans stopped copies after a crash during cleanup without touching active copies", async () => {
|
||||
const finished = await run(), active = await run();
|
||||
await db.update(agentInstructionWorkingCopies).set({ state: "saved", processStoppedAt: new Date() }).where(eq(agentInstructionWorkingCopies.runId, finished.runId));
|
||||
copies = agentInstructionWorkingCopyService(db);
|
||||
await copies.recoverCaptured();
|
||||
await expect(fs.stat(finished.localRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect((await fs.stat(active.localRoot)).isDirectory()).toBe(true);
|
||||
});
|
||||
|
||||
it("cleans up a staging failure before any provider starts", async () => {
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand", responsibleUserId: userId });
|
||||
const shell = vi.spyOn(executionTargetTools, "runAdapterExecutionTargetShellCommand").mockResolvedValue({ exitCode: 1, signal: null, timedOut: false, stdout: "", stderr: "fixture failure" });
|
||||
try {
|
||||
await expect(copies.prepare({ ...target(), runId, cwd: home, target: { kind: "remote", transport: "ssh", environmentId: randomUUID(), remoteCwd: "/fixture/task",
|
||||
spec: { host: "unused.invalid", port: 22, username: "test", remoteCwd: "/fixture/task" } } })).rejects.toThrow("Could not exclude");
|
||||
const row = (await copies.get(companyId, runId))!;
|
||||
expect(row.state).toBe("unavailable");
|
||||
expect(row.receipt?.baseline).toBeUndefined();
|
||||
await expect(fs.stat(path.dirname(row.localRoot))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
} finally { shell.mockRestore(); }
|
||||
});
|
||||
|
||||
it.each(["succeeded", "failed", "cancelled", "timed_out", "interrupted"])("registers staging ownership before copying and cleans preparation for a %s run after restart", async (status) => {
|
||||
const original = fs.cp.bind(fs);
|
||||
const copy = vi.spyOn(fs, "cp").mockImplementationOnce(async (source, destination, options) => {
|
||||
const rows = await db.select().from(agentInstructionWorkingCopies).where(eq(agentInstructionWorkingCopies.agentId, agentId));
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0]).toMatchObject({ state: "preparing", localRoot: destination });
|
||||
return original(source, destination, options);
|
||||
});
|
||||
const interrupted = await run();
|
||||
copy.mockRestore();
|
||||
const active = await run();
|
||||
for (const row of [interrupted, active]) {
|
||||
await db.update(agentInstructionWorkingCopies).set({ state: "preparing", baseHash: "preparing", receipt: { schema: "paperclip.agent-files.v1" } }).where(eq(agentInstructionWorkingCopies.runId, row.runId));
|
||||
}
|
||||
await db.update(heartbeatRuns).set({ status }).where(eq(heartbeatRuns.id, interrupted.runId));
|
||||
copies = agentInstructionWorkingCopyService(db);
|
||||
await copies.recoverStopped();
|
||||
expect((await copies.get(companyId, interrupted.runId))?.state).toBe("unavailable");
|
||||
await expect(fs.stat(path.dirname(interrupted.localRoot))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect((await fs.stat(active.localRoot)).isDirectory()).toBe(true);
|
||||
});
|
||||
|
||||
it("retains remote cleanup across restart and failed retries until the original lease is cleaned", async () => {
|
||||
const copy = await run();
|
||||
const environmentId = randomUUID(), leaseId = randomUUID(), remoteCwd = "/fixture/task";
|
||||
const executionRoot = path.posix.join(remoteCwd, ".paperclip-runtime", "agent-files", agentId, copy.runId);
|
||||
await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "sandbox" });
|
||||
await db.insert(environmentLeases).values({ id: leaseId, companyId, environmentId, heartbeatRunId: copy.runId, provider: "daytona", providerLeaseId: "original-sandbox" });
|
||||
await db.update(agentInstructionWorkingCopies).set({ state: "saved", processStoppedAt: new Date(), location: `remote:${environmentId}`, executionRoot,
|
||||
receipt: { ...copy.receipt, cleanup: { leaseId, remoteCwd } } }).where(eq(agentInstructionWorkingCopies.runId, copy.runId));
|
||||
copies = agentInstructionWorkingCopyService(db);
|
||||
await copies.recoverCaptured();
|
||||
const pending = (await copies.get(companyId, copy.runId))!;
|
||||
expect(pending.receipt).toMatchObject({ cleanupPending: true, cleanup: { leaseId, remoteCwd } });
|
||||
expect(pending.receipt?.baseline).toBeUndefined();
|
||||
await expect(fs.stat(copy.localRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
const execute = vi.fn().mockRejectedValueOnce(new Error("provider temporarily unavailable")).mockResolvedValue({ exitCode: 0, stdout: "", stderr: "" });
|
||||
copies = agentInstructionWorkingCopyService(db, { environmentRuntime: { execute } as unknown as EnvironmentRuntimeService });
|
||||
for (const expectedPending of [true, false]) {
|
||||
await db.update(agentInstructionWorkingCopies).set({ nextAttemptAt: null }).where(eq(agentInstructionWorkingCopies.runId, copy.runId));
|
||||
await copies.recoverCaptured();
|
||||
expect((await copies.get(companyId, copy.runId))?.receipt?.cleanupPending).toBe(expectedPending);
|
||||
}
|
||||
expect(execute).toHaveBeenCalledTimes(2);
|
||||
expect(execute).toHaveBeenLastCalledWith(expect.objectContaining({ lease: expect.objectContaining({ id: leaseId, providerLeaseId: "original-sandbox" }),
|
||||
command: "rm", args: ["-rf", "--", executionRoot], bypassSession: true }));
|
||||
await copies.recoverCaptured();
|
||||
expect(execute).toHaveBeenCalledTimes(2);
|
||||
await copies.release(companyId, copy.runId);
|
||||
expect((await copies.get(companyId, copy.runId))?.receipt?.cleanupPending).toBe(false);
|
||||
expect(execute).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("finishes remote cleanup from a destruction receipt after the environment is deleted", async () => {
|
||||
const copy = await run();
|
||||
const environmentId = randomUUID(), leaseId = randomUUID(), remoteCwd = "/fixture/task";
|
||||
const lease = { id: leaseId, companyId, environmentId, heartbeatRunId: copy.runId, provider: "daytona", providerLeaseId: "destroyed-sandbox" };
|
||||
await db.insert(environments).values({ id: environmentId, name: environmentId, driver: "sandbox" });
|
||||
await db.insert(environmentLeases).values({ ...lease, status: "released", releasedAt: new Date(), cleanupStatus: "success",
|
||||
metadata: { remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "destroyed" }) } });
|
||||
await db.update(agentInstructionWorkingCopies).set({ state: "saved", processStoppedAt: new Date(), location: `remote:${environmentId}`,
|
||||
executionRoot: path.posix.join(remoteCwd, ".paperclip-runtime", "agent-files", agentId, copy.runId),
|
||||
receipt: { ...copy.receipt, cleanupPending: true, cleanup: { leaseId, remoteCwd } } }).where(eq(agentInstructionWorkingCopies.runId, copy.runId));
|
||||
await db.delete(environments).where(eq(environments.id, environmentId));
|
||||
copies = agentInstructionWorkingCopyService(db);
|
||||
await copies.recoverCaptured();
|
||||
expect((await copies.get(companyId, copy.runId))?.receipt?.cleanupPending).toBe(false);
|
||||
await expect(fs.stat(copy.localRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
});
|
||||
|
||||
it.each([false, true])("rejects symlinks without saving any part of the tree, preserving an existing full-storage warning: %s", async (full) => {
|
||||
if (full) await sparseFile(path.join(root, "full.bin"), MAX_AGENT_FILE_BYTES);
|
||||
const copy = await run();
|
||||
await fs.writeFile(path.join(copy.localRoot, "innocent.txt"), "changed");
|
||||
await fs.symlink(path.join(home, "outside"), path.join(copy.localRoot, "escape"));
|
||||
const result = await copies.collectStopped({ companyId, runId: copy.runId });
|
||||
expect(result?.state).toBe("unavailable");
|
||||
expect(result?.errorCode).toBe("AGENT_FILES_SAVE_FAILED");
|
||||
expect(result?.receipt?.storageWarning).toBe(copy.receipt?.storageWarning);
|
||||
if (full) expect(result?.receipt?.storageWarning).toContain("Agent storage is full");
|
||||
await expect(fs.stat(copy.localRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await expect(fs.stat(path.join(root, "innocent.txt"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
}, 30_000);
|
||||
|
||||
it("adopts the last deployed head once and bridges an old collector without appending history", async () => {
|
||||
const id = randomUUID();
|
||||
const content = "last deployed revision";
|
||||
await db.insert(agentInstructionRevisions).values({ id, ...target(), entryFile, contentBase64: Buffer.from(content).toString("base64"), contentHash: fileHash(Buffer.from(content)), byteLength: content.length, source: "board" });
|
||||
await db.insert(agentInstructionHeads).values({ ...target(), entryFile, revisionId: id });
|
||||
const current = await revisions.readCurrent(target(), board());
|
||||
expect(current?.content).toBe(content);
|
||||
await revisions.commit({ ...target(), entryFile, content: "new directory contents", baseRevisionId: id, source: "cleanup" }, board());
|
||||
await revisions.materializeCurrent(target());
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe("new directory contents");
|
||||
await expect(revisions.commit({ ...target(), entryFile, content: "stale legacy run", baseRevisionId: id, source: "cleanup" }, board())).rejects.toMatchObject({ status: 409 });
|
||||
expect(await db.select().from(agentInstructionRevisions).where(eq(agentInstructionRevisions.agentId, agentId))).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("ordinary file edits leave the loaded instruction digest unchanged", async () => {
|
||||
const make = async () => {
|
||||
const copy = await run();
|
||||
const context = await buildNativeRuntimeContext({ db, agent: { id: agentId, companyId, name: "Target", adapterConfig: { instructionsBundleMode: "managed", instructionsRootPath: root, instructionsEntryFile: entryFile } },
|
||||
runId: copy.runId, runtimeConfig: {}, runtimeSkillEntries: [], instructionWorkingCopy: { rootPath: copy.executionRoot, entryPath: entryFile, kind: "agent_files" } });
|
||||
return { copy, context };
|
||||
};
|
||||
const before = await make();
|
||||
await fs.writeFile(path.join(before.copy.localRoot, "notes.txt"), "new personal knowledge");
|
||||
await copies.collectStopped({ companyId, runId: before.copy.runId });
|
||||
const after = await make();
|
||||
expect(after.context.aggregateDigest).toBe(before.context.aggregateDigest);
|
||||
expect(after.context.instructions.bundle.fileCount).toBe(1);
|
||||
});
|
||||
it("uses the workspace transport to restore after destruction of the remote filesystem", async () => {
|
||||
const remoteCwd = path.join(home, "remote-task");
|
||||
await fs.mkdir(remoteCwd, { recursive: true });
|
||||
await execFile("git", ["init", remoteCwd]);
|
||||
const runner: import("@paperclipai/adapter-utils/command-managed-runtime").CommandManagedRuntimeRunner = {
|
||||
execute: async input => {
|
||||
const startedAt = new Date().toISOString();
|
||||
const env = { ...process.env, ...input.env };
|
||||
const args = [...(input.args ?? [])];
|
||||
if (input.stdin != null && (args[0] === "-c" || args[0] === "-lc")) {
|
||||
env.PAPERCLIP_TEST_STDIN = input.stdin;
|
||||
args[1] = `printf '%s' "$PAPERCLIP_TEST_STDIN" | (${args[1]})`;
|
||||
}
|
||||
try {
|
||||
const result = await execFile(input.command, args, { cwd: input.cwd, env, timeout: input.timeoutMs, maxBuffer: 32 * 1024 * 1024 });
|
||||
return { exitCode: 0, signal: null, timedOut: false, stdout: result.stdout, stderr: result.stderr, pid: null, startedAt };
|
||||
} catch (error) {
|
||||
const e = error as { code?: number; signal?: NodeJS.Signals; stdout?: string; stderr?: string };
|
||||
return { exitCode: typeof e.code === "number" ? e.code : 1, signal: e.signal ?? null, timedOut: false, stdout: e.stdout ?? "", stderr: e.stderr ?? "", pid: null, startedAt };
|
||||
}
|
||||
},
|
||||
};
|
||||
const executionTarget = { kind: "remote" as const, transport: "sandbox" as const, environmentId: randomUUID(), remoteCwd, runner };
|
||||
const prepare = async () => {
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand", responsibleUserId: userId });
|
||||
return (await copies.prepare({ ...target(), runId, cwd: home, target: executionTarget }))!;
|
||||
};
|
||||
await fs.mkdir(path.join(root, "build"));
|
||||
await fs.writeFile(path.join(root, "build", "personal.txt"), "cache-like names are still agent files");
|
||||
const first = await prepare();
|
||||
expect(await fs.readFile(path.join(first.executionRoot, "build", "personal.txt"), "utf8")).toBe("cache-like names are still agent files");
|
||||
await fs.mkdir(path.join(first.executionRoot, "notes"));
|
||||
await fs.writeFile(path.join(first.executionRoot, "notes", "bytes.bin"), Buffer.from([0, 128, 255]));
|
||||
await fs.mkdir(path.join(first.executionRoot, "node_modules"));
|
||||
await fs.writeFile(path.join(first.executionRoot, "node_modules", "personal.txt"), "retain this too");
|
||||
await fs.writeFile(path.join(remoteCwd, "task-only.txt"), "task");
|
||||
expect((await execFile("git", ["-C", remoteCwd, "status", "--porcelain", "--untracked-files=all"])).stdout).toBe("?? task-only.txt\n");
|
||||
expect((await copies.collectStopped({ companyId, runId: first.runId, target: executionTarget }))?.state).toBe("saved");
|
||||
await copies.release(companyId, first.runId);
|
||||
expect((await copies.get(companyId, first.runId))?.receipt?.baseline).toBeUndefined();
|
||||
await expect(fs.stat(first.localRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await fs.rm(remoteCwd, { recursive: true });
|
||||
await fs.mkdir(remoteCwd);
|
||||
copies = agentInstructionWorkingCopyService(db);
|
||||
const second = await prepare();
|
||||
expect(await fs.readFile(path.join(second.executionRoot, "notes", "bytes.bin"))).toEqual(Buffer.from([0,128,255]));
|
||||
expect(await fs.readFile(path.join(second.executionRoot, "node_modules", "personal.txt"), "utf8")).toBe("retain this too");
|
||||
await expect(fs.stat(path.join(second.executionRoot, "task-only.txt"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await copies.collectStopped({ companyId, runId: second.runId, target: executionTarget });
|
||||
await copies.release(companyId, second.runId);
|
||||
});
|
||||
|
||||
it("stages SSH agent files at the registered root without a nested task workspace", async () => {
|
||||
const remoteCwd = path.join(home, "ssh-task");
|
||||
const exclude = vi.spyOn(executionTargetTools, "runAdapterExecutionTargetShellCommand").mockResolvedValue({ exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" });
|
||||
const stage = vi.spyOn(ssh, "syncDirectoryToSsh").mockImplementation(async input => {
|
||||
await fs.mkdir(path.dirname(input.remoteDir), { recursive: true });
|
||||
await fs.cp(input.localDir, input.remoteDir, { recursive: true });
|
||||
});
|
||||
const restore = vi.spyOn(ssh, "restoreWorkspaceFromSshExecution").mockImplementation(async input => {
|
||||
await fs.cp(input.remoteDir!, input.localDir, { recursive: true });
|
||||
});
|
||||
try {
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand", responsibleUserId: userId });
|
||||
const target = { kind: "remote" as const, transport: "ssh" as const, environmentId: randomUUID(), remoteCwd,
|
||||
spec: { host: "unused.invalid", port: 22, username: "test", remoteCwd } };
|
||||
const copy = (await copies.prepare({ companyId, agentId, runId, cwd: home, target }))!;
|
||||
expect(stage).toHaveBeenCalledWith(expect.objectContaining({ remoteDir: copy.executionRoot }));
|
||||
expect(await fs.readFile(path.join(copy.executionRoot, entryFile), "utf8")).toBe(initial);
|
||||
await fs.writeFile(path.join(copy.executionRoot, "ssh-note.txt"), "persistent SSH file");
|
||||
expect((await copies.collectStopped({ companyId, runId, target }))?.state).toBe("saved");
|
||||
expect(restore).toHaveBeenCalledWith(expect.objectContaining({ remoteDir: copy.executionRoot, restoreGitHistory: false }));
|
||||
expect(await fs.readFile(path.join(root, "ssh-note.txt"), "utf8")).toBe("persistent SSH file");
|
||||
} finally { stage.mockRestore(); restore.mockRestore(); exclude.mockRestore(); }
|
||||
});
|
||||
|
||||
it("deduplicates concurrent stopped callbacks and discards completed operational snapshots", async () => {
|
||||
const copy = await run();
|
||||
await fs.writeFile(path.join(copy.localRoot, "note.txt"), "one write");
|
||||
const results = await Promise.all([copies.collectStopped({ companyId, runId: copy.runId }), copies.collectStopped({ companyId, runId: copy.runId })]);
|
||||
expect(results.every(result => result?.state === "saved")).toBe(true);
|
||||
// Simulate a crash after save but before release. The restart sweeper must
|
||||
// reclaim operational baselines without touching canonical files.
|
||||
copies = agentInstructionWorkingCopyService(db);
|
||||
await copies.recoverCaptured();
|
||||
const row = await copies.get(companyId, copy.runId);
|
||||
expect(row?.receipt?.schema).toBe("paperclip.agent-files.v1");
|
||||
expect(row?.receipt?.baseline).toBeUndefined();
|
||||
expect(await fs.readFile(path.join(root, "note.txt"), "utf8")).toBe("one write");
|
||||
});
|
||||
|
||||
});
|
||||
@@ -0,0 +1,87 @@
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { execFile as execFileCallback } from "node:child_process";
|
||||
import { promisify } from "node:util";
|
||||
import { afterEach, beforeEach, expect, it } from "vitest";
|
||||
import { instructionGitExcludeProgram } from "../services/agent-instruction-files.js";
|
||||
|
||||
const execFile = promisify(execFileCallback);
|
||||
let root: string;
|
||||
beforeEach(async () => { root = await fs.mkdtemp(path.join(os.tmpdir(), "agent-git-exclusion-")); });
|
||||
afterEach(async () => { await fs.rm(root, { recursive: true, force: true }); });
|
||||
const git = (cwd: string, ...args: string[]) => execFile("git", ["-C", cwd, ...args]);
|
||||
const exclude = (cwd: string) => execFile(process.execPath, ["-e", instructionGitExcludeProgram, cwd]);
|
||||
|
||||
async function assertPrivateFilesIgnored(cwd: string) {
|
||||
const runtime = path.join(cwd, ".paperclip-runtime");
|
||||
await fs.writeFile(path.join(runtime, "private.md"), "agent instructions");
|
||||
await git(cwd, "add", "-A");
|
||||
expect((await git(cwd, "diff", "--cached", "--name-only")).stdout).not.toContain(".paperclip-runtime");
|
||||
expect((await git(cwd, "check-ignore", "--", ".paperclip-runtime/private.md", ".paperclip-runtime/.gitignore")).stdout.trim().split("\n")).toHaveLength(2);
|
||||
}
|
||||
|
||||
it("excludes runtime files without changing tracked ignore rules or Git metadata", async () => {
|
||||
await git(root, "init");
|
||||
await fs.writeFile(path.join(root, ".gitignore"), "keep-me\n");
|
||||
const metadata = path.join(root, ".git", "info", "exclude");
|
||||
const before = await fs.readFile(metadata, "utf8");
|
||||
await exclude(root);
|
||||
await exclude(root);
|
||||
await assertPrivateFilesIgnored(root);
|
||||
expect(await fs.readFile(path.join(root, ".gitignore"), "utf8")).toBe("keep-me\n");
|
||||
expect(await fs.readFile(metadata, "utf8")).toBe(before);
|
||||
});
|
||||
|
||||
it("never writes to an external gitdir selected by task-controlled metadata", async () => {
|
||||
const workspace = path.join(root, "workspace"), outside = path.join(root, "outside.git");
|
||||
await fs.mkdir(workspace);
|
||||
await git(workspace, "init", `--separate-git-dir=${outside}`);
|
||||
const metadata = path.join(outside, "info", "exclude");
|
||||
const before = await fs.readFile(metadata, "utf8");
|
||||
await exclude(workspace);
|
||||
await assertPrivateFilesIgnored(workspace);
|
||||
expect(await fs.readFile(metadata, "utf8")).toBe(before);
|
||||
});
|
||||
|
||||
it("supports linked worktrees and nested task directories without changing the common gitdir", async () => {
|
||||
const repository = path.join(root, "repository"), worktree = path.join(root, "worktree");
|
||||
await fs.mkdir(repository);
|
||||
await git(repository, "init");
|
||||
await git(repository, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "--allow-empty", "-m", "initial");
|
||||
await git(repository, "worktree", "add", "--detach", worktree);
|
||||
const nested = path.join(worktree, "nested");
|
||||
await fs.mkdir(nested);
|
||||
const metadata = path.join(repository, ".git", "info", "exclude");
|
||||
const before = await fs.readFile(metadata, "utf8");
|
||||
await exclude(nested);
|
||||
await assertPrivateFilesIgnored(nested);
|
||||
expect(await fs.readFile(metadata, "utf8")).toBe(before);
|
||||
});
|
||||
|
||||
it.each(["directory", "file"])("rejects a symlink at the runtime exclusion %s", async (kind) => {
|
||||
const workspace = path.join(root, "workspace"), outside = path.join(root, "outside");
|
||||
await fs.mkdir(workspace);
|
||||
await fs.mkdir(outside);
|
||||
const externalFile = path.join(outside, ".gitignore");
|
||||
await fs.writeFile(externalFile, "untouched\n");
|
||||
const runtime = path.join(workspace, ".paperclip-runtime");
|
||||
if (kind === "directory") await fs.symlink(outside, runtime);
|
||||
else {
|
||||
await fs.mkdir(runtime);
|
||||
await fs.symlink(externalFile, path.join(runtime, ".gitignore"));
|
||||
}
|
||||
await expect(exclude(workspace)).rejects.toThrow("Unsafe runtime exclusion");
|
||||
expect(await fs.readFile(externalFile, "utf8")).toBe("untouched\n");
|
||||
});
|
||||
|
||||
it("replaces a linked ignore file without writing through its external inode", async () => {
|
||||
const externalFile = path.join(root, "outside-ignore"), workspace = path.join(root, "workspace");
|
||||
const runtime = path.join(workspace, ".paperclip-runtime");
|
||||
await fs.mkdir(runtime, { recursive: true });
|
||||
await fs.writeFile(externalFile, "untouched\n");
|
||||
await fs.link(externalFile, path.join(runtime, ".gitignore"));
|
||||
await exclude(workspace);
|
||||
expect(await fs.readFile(externalFile, "utf8")).toBe("untouched\n");
|
||||
expect(await fs.readFile(path.join(runtime, ".gitignore"), "utf8")).toBe("*\n");
|
||||
});
|
||||
@@ -0,0 +1,247 @@
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { beforeAll, afterAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { agents, agentApiKeys, companies, authUsers, companyMemberships, principalPermissionGrants, heartbeatRuns,
|
||||
agentInstructionRevisions, agentInstructionHeads, issueThreadInteractions, issues, createDb } from "@paperclipai/db";
|
||||
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { agentInstructionRevisionService } from "../services/agent-instruction-revisions.js";
|
||||
import { agentInstructionWorkingCopyService } from "../services/agent-instruction-working-copies.js";
|
||||
import { instructionBytes, instructionPath, materializeInstructionBytes, readInstructionBytes } from "../services/agent-instruction-files.js";
|
||||
import { agentInstructionsService, resolveManagedInstructionsRoot } from "../services/agent-instructions.js";
|
||||
import type { AuthorizationActor } from "../services/authorization.js";
|
||||
import { upsertAgentInstructionsFileSchema } from "@paperclipai/shared";
|
||||
|
||||
describe("canonical instruction revisions", () => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let service: ReturnType<typeof agentInstructionRevisionService>;
|
||||
const previousHome = process.env.PAPERCLIP_HOME;
|
||||
let home: string;
|
||||
let companyId: string, agentId: string, userId: string, actorId: string, runId: string, root: string;
|
||||
let actor: AuthorizationActor;
|
||||
const entryFile = "policy/INSTRUCTIONS.txt";
|
||||
const initial = "\uFEFF# Original\r\n☃\0\n";
|
||||
beforeAll(async () => {
|
||||
home = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "instruction-revisions-")));
|
||||
process.env.PAPERCLIP_HOME = home;
|
||||
database = await startEmbeddedPostgresTestDatabase("instruction-revisions-db-");
|
||||
db = createDb(database.connectionString);
|
||||
service = agentInstructionRevisionService(db);
|
||||
}, 90_000);
|
||||
afterAll(async () => {
|
||||
vi.restoreAllMocks();
|
||||
if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome;
|
||||
await database?.cleanup();
|
||||
if (home) await fs.rm(home, { recursive: true, force: true });
|
||||
});
|
||||
beforeEach(async () => {
|
||||
companyId = randomUUID(); agentId = randomUUID(); actorId = randomUUID(); userId = randomUUID(); runId = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name: "Instruction tests", issuePrefix: randomUUID().slice(0, 8) });
|
||||
await db.insert(authUsers).values({ id: userId, name: "Editor", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() });
|
||||
root = resolveManagedInstructionsRoot({ id: agentId, companyId, name: "Target", adapterConfig: {} });
|
||||
await db.insert(agents).values([
|
||||
{ id: agentId, companyId, name: "Target", adapterConfig: { instructionsBundleMode: "managed", instructionsRootPath: root, instructionsEntryFile: entryFile } },
|
||||
{ id: actorId, companyId, name: "Writer" },
|
||||
]);
|
||||
await db.insert(companyMemberships).values([
|
||||
{ companyId, principalType: "user", principalId: userId, membershipRole: "operator" },
|
||||
{ companyId, principalType: "agent", principalId: actorId, membershipRole: "member" },
|
||||
]);
|
||||
await db.insert(principalPermissionGrants).values([
|
||||
{ companyId, principalType: "user", principalId: userId, permissionKey: "agents:configure", scope: { agentIds: [agentId] } },
|
||||
{ companyId, principalType: "agent", principalId: actorId, permissionKey: "agents:configure", scope: { agentIds: [agentId] } },
|
||||
]);
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId: actorId, invocationSource: "on_demand", responsibleUserId: userId });
|
||||
actor = { type: "agent", companyId, agentId: actorId, runId, source: "agent_jwt" };
|
||||
await fs.mkdir(path.dirname(path.join(root, entryFile)), { recursive: true });
|
||||
await fs.writeFile(path.join(root, entryFile), initial);
|
||||
});
|
||||
const target = () => ({ companyId, agentId });
|
||||
async function base() { return (await service.readCurrent(target(), actor))!; }
|
||||
async function save(content: string, baseRevisionId: string | null) {
|
||||
return service.commit({ ...target(), entryFile, content, baseRevisionId, source: "cleanup" }, actor);
|
||||
}
|
||||
it.each(["current", "history", "revision", "diff", "candidates"])("denies peer %s reads without a configuration grant or responsible-user target access", async (operation) => {
|
||||
const first = await base();
|
||||
await db.delete(principalPermissionGrants).where(eq(principalPermissionGrants.principalId, userId));
|
||||
const read = () => {
|
||||
if (operation === "current") return service.readCurrent(target(), actor);
|
||||
if (operation === "history") return service.history({ ...target(), entryFile }, actor);
|
||||
if (operation === "revision") return service.readRevision({ ...target(), entryFile, revisionId: first.revision.id }, actor);
|
||||
if (operation === "diff") return service.diff({ ...target(), entryFile, fromRevisionId: first.revision.id, toRevisionId: first.revision.id }, actor);
|
||||
return agentInstructionWorkingCopyService(db).list(companyId, agentId, actor);
|
||||
};
|
||||
await expect(read()).rejects.toMatchObject({ status: 403 });
|
||||
await db.insert(principalPermissionGrants).values({ companyId, principalType: "user", principalId: userId, permissionKey: "agents:configure", scope: { agentIds: [actorId] } });
|
||||
await expect(read()).rejects.toMatchObject({ status: 403 });
|
||||
});
|
||||
it.each(["current", "history", "revision", "diff", "candidates", "write"])("requires the caller's own target-scoped grant for peer %s access", async (operation) => {
|
||||
const first = await base();
|
||||
await db.delete(principalPermissionGrants).where(eq(principalPermissionGrants.principalId, actorId));
|
||||
const access = () => {
|
||||
if (operation === "current") return service.readCurrent(target(), actor);
|
||||
if (operation === "history") return service.history({ ...target(), entryFile }, actor);
|
||||
if (operation === "revision") return service.readRevision({ ...target(), entryFile, revisionId: first.revision.id }, actor);
|
||||
if (operation === "diff") return service.diff({ ...target(), entryFile, fromRevisionId: first.revision.id, toRevisionId: first.revision.id }, actor);
|
||||
if (operation === "write") return save("authorized peer edit", first.revision.id);
|
||||
return agentInstructionWorkingCopyService(db).list(companyId, agentId, actor);
|
||||
};
|
||||
// The responsible user retains permission on the target throughout.
|
||||
await expect(access()).rejects.toMatchObject({ status: 403 });
|
||||
await db.insert(principalPermissionGrants).values({ companyId, principalType: "agent", principalId: actorId, permissionKey: "agents:configure", scope: { agentIds: [actorId] } });
|
||||
await expect(access()).rejects.toMatchObject({ status: 403 });
|
||||
await db.update(principalPermissionGrants).set({ scope: { agentIds: [agentId] } }).where(eq(principalPermissionGrants.principalId, actorId));
|
||||
// New saves have no historical revision row; authorized history lookups
|
||||
// reach the normal not-found response rather than an authorization denial.
|
||||
if (operation === "revision" || operation === "diff") await expect(access()).rejects.toMatchObject({ status: 404 });
|
||||
else await expect(access()).resolves.toBeDefined();
|
||||
});
|
||||
it("allows self and board reads without requiring instruction edit permission", async () => {
|
||||
await db.delete(principalPermissionGrants).where(eq(principalPermissionGrants.principalId, userId));
|
||||
const selfRoot = resolveManagedInstructionsRoot({ id: actorId, companyId, name: "Writer", adapterConfig: {} });
|
||||
await fs.mkdir(selfRoot, { recursive: true });
|
||||
await fs.writeFile(path.join(selfRoot, "AGENTS.md"), "Self instructions");
|
||||
expect((await service.readCurrent({ companyId, agentId: actorId }, actor))?.content).toBe("Self instructions");
|
||||
expect((await service.readCurrent(target(), { type: "board", userId, source: "session" }))?.content).toBe(initial);
|
||||
});
|
||||
it("reads and replaces current bytes using content tokens without storing revisions", async () => {
|
||||
const first = await base();
|
||||
expect(first.content).toBe(initial);
|
||||
const saved = await save("# Changed\r\n", first.revision.id);
|
||||
expect(saved).toMatchObject({ changed: true, materialization: "current" });
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(saved.content);
|
||||
expect((await save(saved.content, first.revision.id)).changed).toBe(false);
|
||||
expect((await service.history({ ...target(), entryFile }, actor)).revisions).toHaveLength(0);
|
||||
expect(await db.select().from(agentInstructionRevisions).where(eq(agentInstructionRevisions.agentId, agentId))).toHaveLength(0);
|
||||
});
|
||||
it("requires a matching initial content token and serializes competing saves", async () => {
|
||||
await expect(save("without reading", null)).rejects.toMatchObject({ status: 409 });
|
||||
const first = await base();
|
||||
const results = await Promise.allSettled([save("one", first.revision.id), save("two", first.revision.id)]);
|
||||
expect(results.filter(result => result.status === "fulfilled")).toHaveLength(1);
|
||||
expect(results.find(result => result.status === "rejected")).toMatchObject({ reason: { status: 409 } });
|
||||
});
|
||||
it("does not claim a save when atomic file replacement fails", async () => {
|
||||
const first = await base();
|
||||
const rename = vi.spyOn(fs, "rename").mockRejectedValueOnce(new Error("interrupted replacement"));
|
||||
await expect(save("new content", first.revision.id)).rejects.toThrow("interrupted replacement");
|
||||
rename.mockRestore();
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(initial);
|
||||
service = agentInstructionRevisionService(db);
|
||||
await service.materializeCurrent(target());
|
||||
expect((await base()).content).toBe(initial);
|
||||
expect((await save("new content", first.revision.id)).content).toBe("new content");
|
||||
});
|
||||
it("allows authorized peer edits but rejects missing, forged and revoked responsible identity", async () => {
|
||||
const first = await base();
|
||||
await save("peer save", first.revision.id);
|
||||
await expect(service.commit({ ...target(), entryFile, content: "forged", baseRevisionId: first.revision.id, source: "tool" }, { ...actor, onBehalfOfUserId: "forged" })).rejects.toMatchObject({ status: 403 });
|
||||
await db.update(heartbeatRuns).set({ responsibleUserId: null }).where(eq(heartbeatRuns.id, runId));
|
||||
await expect(save("missing", first.revision.id)).rejects.toMatchObject({ status: 403 });
|
||||
await db.update(heartbeatRuns).set({ responsibleUserId: userId }).where(eq(heartbeatRuns.id, runId));
|
||||
await db.update(companyMemberships).set({ status: "suspended" }).where(and(eq(companyMemberships.principalId, userId), eq(companyMemberships.companyId, companyId)));
|
||||
await expect(save("revoked", first.revision.id)).rejects.toMatchObject({ status: 403 });
|
||||
});
|
||||
it("rechecks user target scope and agent containment, even with shadow authorization enabled", async () => {
|
||||
const first = await base();
|
||||
const previousShadow = process.env.PAPERCLIP_RESPONSIBLE_USER_AUTHZ_SHADOW;
|
||||
process.env.PAPERCLIP_RESPONSIBLE_USER_AUTHZ_SHADOW = "true";
|
||||
await db.update(principalPermissionGrants).set({ scope: { agentIds: [actorId] } }).where(eq(principalPermissionGrants.principalId, userId));
|
||||
await expect(save("denied", first.revision.id)).rejects.toMatchObject({ status: 403 });
|
||||
await db.update(principalPermissionGrants).set({ scope: null }).where(eq(principalPermissionGrants.principalId, userId));
|
||||
await db.update(agents).set({ permissions: { trustPreset: "low_trust_review" } }).where(eq(agents.id, actorId));
|
||||
await expect(save("restricted", first.revision.id)).rejects.toMatchObject({ status: 403 });
|
||||
if (previousShadow === undefined) delete process.env.PAPERCLIP_RESPONSIBLE_USER_AUTHZ_SHADOW; else process.env.PAPERCLIP_RESPONSIBLE_USER_AUTHZ_SHADOW = previousShadow;
|
||||
});
|
||||
it("keeps suggest-only protected-change consent and explicit configure scope restrictions", async () => {
|
||||
const first = await base();
|
||||
await db.delete(principalPermissionGrants).where(eq(principalPermissionGrants.principalId, actorId));
|
||||
await db.insert(principalPermissionGrants).values({ companyId, principalType: "agent", principalId: actorId, permissionKey: "agents:suggest-changes" });
|
||||
await expect(save("needs approval", first.revision.id)).rejects.toMatchObject({ status: 403 });
|
||||
await db.delete(principalPermissionGrants).where(eq(principalPermissionGrants.principalId, actorId));
|
||||
await db.insert(principalPermissionGrants).values({ companyId, principalType: "agent", principalId: actorId, permissionKey: "agents:configure", scope: { agentIds: [actorId] } });
|
||||
await expect(save("out of scope", first.revision.id)).rejects.toMatchObject({ status: 403 });
|
||||
});
|
||||
it("consumes accepted protected-change approval only with a successful CAS commit", async () => {
|
||||
const first = await base();
|
||||
const sourceRunId = randomUUID(), issueId = randomUUID(), interactionId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: sourceRunId, companyId, agentId: actorId, status: "succeeded" });
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Review instructions" });
|
||||
await db.delete(principalPermissionGrants).where(eq(principalPermissionGrants.principalId, actorId));
|
||||
await db.insert(principalPermissionGrants).values({ companyId, principalType: "agent", principalId: actorId, permissionKey: "agents:suggest-changes" });
|
||||
await db.insert(issueThreadInteractions).values({ id: interactionId, companyId, issueId, kind: "request_confirmation", status: "accepted", sourceRunId, createdByAgentId: actorId,
|
||||
payload: { version: 1, prompt: "Apply the instruction change?", detailsMarkdown: "```diff\n+approved\n```", target: { type: "custom", key: `agent:${agentId}:instructions`, revisionId: "proposal" } },
|
||||
result: { version: 1, outcome: "accepted" }, resolvedByUserId: userId, resolvedAt: new Date(),
|
||||
});
|
||||
expect((await service.readCurrent(target(), actor))?.revision.id).toBe(first.revision.id);
|
||||
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interactionId)))[0].result).not.toHaveProperty("consumedAt");
|
||||
await expect(save("approved", randomUUID())).rejects.toMatchObject({ status: 409 });
|
||||
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interactionId)))[0].result).not.toHaveProperty("consumedAt");
|
||||
const receipt = await save("approved", first.revision.id);
|
||||
expect(receipt.changed).toBe(true);
|
||||
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interactionId)))[0].result).toMatchObject({ consumedByRunId: runId });
|
||||
await expect(save("another unapproved change", receipt.revision.id)).rejects.toMatchObject({ status: 403 });
|
||||
});
|
||||
it("shares head/CAS and attribution between board and agent writers", async () => {
|
||||
const first = await base();
|
||||
const board = { type: "board", userId, source: "session" } as const;
|
||||
const saved = await service.commit({ ...target(), entryFile, content: "board edit", baseRevisionId: first.revision.id, source: "board" }, board);
|
||||
expect(saved.revision).toMatchObject({ actorUserId: userId, responsibleUserId: userId, actorAgentId: null, sourceRunId: null, source: "api" });
|
||||
await expect(save("stale agent edit", first.revision.id)).rejects.toMatchObject({ status: 409 });
|
||||
});
|
||||
it("seeds a nested legacy managed entry and never imports a different AGENTS.md", async () => {
|
||||
await fs.writeFile(path.join(root, "AGENTS.md"), "unrelated");
|
||||
await db.update(agents).set({ adapterConfig: { instructionsFilePath: path.join(root, entryFile) } }).where(eq(agents.id, agentId));
|
||||
const first = await base();
|
||||
expect(first.content).toBe(initial);
|
||||
expect(first.revision.entryFile).toBe(entryFile);
|
||||
await db.update(agents).set({ adapterConfig: { instructionsBundleMode: "managed", instructionsRootPath: root, instructionsEntryFile: "missing.md" } }).where(eq(agents.id, agentId));
|
||||
expect(await service.readCurrent(target(), actor)).toBeNull();
|
||||
await expect(save("wrong entry", first.revision.id)).rejects.toMatchObject({ status: 409, details: { code: "INSTRUCTION_ENTRY_CHANGED" } });
|
||||
});
|
||||
it("rejects cross-company targets, foreign revisions, and revoked API keys", async () => {
|
||||
const first = await base();
|
||||
await expect(service.readRevision({ ...target(), entryFile, revisionId: randomUUID() }, actor)).rejects.toMatchObject({ status: 404 });
|
||||
await expect(service.readCurrent({ companyId: randomUUID(), agentId }, actor)).rejects.toMatchObject({ status: 404 });
|
||||
const keyId = randomUUID();
|
||||
await db.insert(agentApiKeys).values({ id: keyId, companyId, agentId: actorId, name: "test", keyHash: randomUUID(), responsibleUserId: userId, revokedAt: new Date() });
|
||||
await expect(service.commit({ ...target(), entryFile, content: "key", baseRevisionId: first.revision.id, source: "api" }, { ...actor, keyId })).rejects.toMatchObject({ status: 403 });
|
||||
await expect(db.insert(agentInstructionHeads).values({ companyId: randomUUID(), agentId, entryFile, revisionId: first.revision.id })).rejects.toThrow();
|
||||
});
|
||||
it("rejects invalid content, symlinks, traversal and external bundles without changing canonical bytes", async () => {
|
||||
const first = await base();
|
||||
expect(() => instructionBytes("\ud800")).toThrow("UTF-8");
|
||||
expect(() => instructionBytes(new Uint8Array([0xff]))).toThrow("UTF-8");
|
||||
expect(() => instructionBytes("a".repeat(1024 * 1024 + 1))).toThrow("1 MiB");
|
||||
for (const value of ["../AGENTS.md", "/etc/passwd", "a/../AGENTS.md", "a\\b", "a//b"]) expect(() => instructionPath(value)).toThrow();
|
||||
await fs.unlink(path.join(root, entryFile));
|
||||
await fs.symlink(path.join(home, "outside"), path.join(root, entryFile));
|
||||
await expect(save("symlink", first.revision.id)).rejects.toMatchObject({ status: 422 });
|
||||
await fs.unlink(path.join(root, entryFile));
|
||||
await fs.rm(path.dirname(path.join(root, entryFile)), { recursive: true });
|
||||
await fs.symlink(home, path.dirname(path.join(root, entryFile)));
|
||||
await expect(readInstructionBytes(root, entryFile)).rejects.toMatchObject({ status: 422 });
|
||||
await db.update(agents).set({ adapterConfig: { instructionsBundleMode: "external", instructionsRootPath: home } }).where(eq(agents.id, agentId));
|
||||
await expect(save("external", first.revision.id)).rejects.toMatchObject({ details: { code: "INSTRUCTION_MANAGED_BUNDLE_REQUIRED" } });
|
||||
expect((await db.select().from(agentInstructionRevisions).where(eq(agentInstructionRevisions.agentId, agentId)))).toHaveLength(0);
|
||||
});
|
||||
it.skipIf(process.platform !== "darwin")("accepts macOS temporary directory aliases while rejecting links within storage", async () => {
|
||||
const aliasRoot = home.replace(/^\/private\/(var|tmp)\//, "/$1/");
|
||||
expect(aliasRoot).not.toBe(home);
|
||||
await materializeInstructionBytes(aliasRoot, "alias/AGENTS.md", Buffer.from("persisted"));
|
||||
expect(await readInstructionBytes(aliasRoot, "alias/AGENTS.md")).toEqual(Buffer.from("persisted"));
|
||||
await fs.symlink(path.join(home, "alias"), path.join(home, "linked-agent"));
|
||||
await expect(readInstructionBytes(path.join(aliasRoot, "linked-agent"), "AGENTS.md"))
|
||||
.rejects.toMatchObject({ status: 422 });
|
||||
});
|
||||
it("requires the content API for the configured entry and rejects forged fields", async () => {
|
||||
await base();
|
||||
const [agent] = await db.select().from(agents).where(eq(agents.id, agentId));
|
||||
await expect(agentInstructionsService(db).writeFile(agent, entryFile, "bypass")).rejects.toMatchObject({ details: { code: "INSTRUCTION_REVISION_REQUIRED" } });
|
||||
await expect(agentInstructionsService(db).deleteFile(agent, entryFile)).rejects.toMatchObject({ status: 422 });
|
||||
expect(upsertAgentInstructionsFileSchema.safeParse({ path: entryFile, content: "bad", responsibleUserId: userId }).success).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,345 @@
|
||||
import fs from "node:fs/promises";
|
||||
import { execFile as execFileCallback } from "node:child_process";
|
||||
import { promisify } from "node:util";
|
||||
import * as gitWorkspaceSync from "@paperclipai/adapter-utils/git-workspace-sync";
|
||||
import { DatabaseSync } from "node:sqlite";
|
||||
import { captureDirectorySnapshot } from "@paperclipai/adapter-utils/workspace-restore-merge";
|
||||
const execFile = promisify(execFileCallback);
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { agents, companies, authUsers, companyMemberships, principalPermissionGrants, heartbeatRuns, agentInstructionWorkingCopies, createDb } from "@paperclipai/db";
|
||||
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { agentInstructionRevisionService } from "../services/agent-instruction-revisions.js";
|
||||
import { agentInstructionWorkingCopyService } from "../services/agent-instruction-working-copies.js";
|
||||
import { appendHeartbeatRunEvent } from "../services/heartbeat-run-events.js";
|
||||
import { resolveManagedInstructionsRoot } from "../services/agent-instructions.js";
|
||||
import { buildNativeRuntimeContext } from "../services/native-runtime/runtime-context.js";
|
||||
|
||||
describe("registered run instruction copies", () => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let copies: ReturnType<typeof agentInstructionWorkingCopyService>;
|
||||
let revisions: ReturnType<typeof agentInstructionRevisionService>;
|
||||
const previousHome = process.env.PAPERCLIP_HOME;
|
||||
let home: string;
|
||||
let companyId: string, agentId: string, userId: string, root: string;
|
||||
const entryFile = "policy/INSTRUCTIONS.txt";
|
||||
const initial = "\uFEFF# Original\r\n☃\n";
|
||||
const target = () => ({ companyId, agentId });
|
||||
const board = () => ({ type: "board" as const, userId, source: "session" as const });
|
||||
async function run() {
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand", responsibleUserId: userId });
|
||||
return (await copies.prepare({ legacy: true, ...target(), runId, cwd: home }))!;
|
||||
}
|
||||
beforeAll(async () => {
|
||||
home = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "instruction-working-copies-")));
|
||||
process.env.PAPERCLIP_HOME = home;
|
||||
database = await startEmbeddedPostgresTestDatabase("instruction-copies-db-");
|
||||
db = createDb(database.connectionString);
|
||||
copies = agentInstructionWorkingCopyService(db);
|
||||
revisions = agentInstructionRevisionService(db);
|
||||
}, 90_000);
|
||||
afterAll(async () => {
|
||||
if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome;
|
||||
await database?.cleanup();
|
||||
if (home) {
|
||||
const writable = async (dir: string) => {
|
||||
await fs.chmod(dir, 0o700);
|
||||
for (const entry of await fs.readdir(dir, { withFileTypes: true })) if (entry.isDirectory()) await writable(path.join(dir, entry.name));
|
||||
};
|
||||
await writable(home);
|
||||
await fs.rm(home, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
beforeEach(async () => {
|
||||
companyId = randomUUID(); agentId = randomUUID(); userId = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name: "Instruction tests", issuePrefix: randomUUID().slice(0, 8) });
|
||||
await db.insert(authUsers).values({ id: userId, name: "Editor", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() });
|
||||
root = resolveManagedInstructionsRoot({ ...target(), id: agentId, name: "Target", adapterConfig: {} });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Target", adapterConfig: { instructionsBundleMode: "managed", instructionsRootPath: root, instructionsEntryFile: entryFile } });
|
||||
await db.insert(companyMemberships).values([
|
||||
{ companyId, principalType: "user", principalId: userId, membershipRole: "operator" },
|
||||
{ companyId, principalType: "agent", principalId: agentId, membershipRole: "member" },
|
||||
]);
|
||||
await db.insert(principalPermissionGrants).values({ companyId, principalType: "user", principalId: userId, permissionKey: "agents:configure", scope: { agentIds: [agentId] } });
|
||||
await fs.mkdir(path.dirname(path.join(root, entryFile)), { recursive: true });
|
||||
await fs.writeFile(path.join(root, entryFile), initial);
|
||||
});
|
||||
|
||||
it("persists an atomic editor replacement, ignores repository instructions, and gives the next run exact bytes", async () => {
|
||||
const copy = await run();
|
||||
await fs.writeFile(path.join(home, "AGENTS.md"), "Repository text must not become agent instructions");
|
||||
const edited = "\uFEFF# Updated\r\nKeep this persistent. ☃\n";
|
||||
const replacement = path.join(copy.localRoot, "edited.tmp");
|
||||
await fs.writeFile(replacement, edited);
|
||||
await fs.rename(replacement, path.join(copy.localRoot, entryFile));
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(initial);
|
||||
const saved = await copies.collectStopped({ companyId, runId: copy.runId });
|
||||
expect(saved).toMatchObject({ state: "saved", candidateHash: expect.any(String), processStoppedAt: expect.any(Date) });
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(edited);
|
||||
copies = agentInstructionWorkingCopyService(db);
|
||||
expect((await copies.collectStopped({ companyId, runId: copy.runId }))?.receipt).toEqual(saved?.receipt);
|
||||
expect((await revisions.history({ ...target(), entryFile }, board())).revisions).toHaveLength(0);
|
||||
const next = await run();
|
||||
expect(await fs.readFile(path.join(next.localRoot, entryFile), "utf8")).toBe(edited);
|
||||
});
|
||||
|
||||
it.each(["failed", "cancelled", "timed_out"])("retains edits from stopped %s runs without restarting execution", async (status) => {
|
||||
const copy = await run();
|
||||
await fs.writeFile(path.join(copy.localRoot, entryFile), status);
|
||||
await db.update(heartbeatRuns).set({ status, finishedAt: new Date() }).where(eq(heartbeatRuns.id, copy.runId));
|
||||
expect((await copies.collectStopped({ companyId, runId: copy.runId }))?.state).toBe("saved");
|
||||
expect((await revisions.readCurrent(target(), board()))?.content).toBe(status);
|
||||
const [unchangedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, copy.runId));
|
||||
expect(unchangedRun.status).toBe(status);
|
||||
});
|
||||
|
||||
it.each(["saved", "unchanged"])("refreshes a stopped %s copy from the current canonical revision before retry", async (state) => {
|
||||
const copy = await run();
|
||||
if (state === "saved") await fs.writeFile(path.join(copy.localRoot, entryFile), "previous saved turn");
|
||||
expect((await copies.collectStopped({ companyId, runId: copy.runId }))?.state).toBe(state);
|
||||
const before = (await revisions.readCurrent(target(), board()))!;
|
||||
const current = await revisions.commit({ ...target(), entryFile, baseRevisionId: before.revision.id, content: "new board instructions", source: "api" }, board());
|
||||
const retried = (await copies.prepare({ legacy: true, ...target(), runId: copy.runId, cwd: home }))!;
|
||||
expect(retried).toMatchObject({ state: "prepared", baseRevisionId: current.revision.id, baseHash: current.revision.contentHash, processStoppedAt: null });
|
||||
expect(await fs.readFile(path.join(retried.localRoot, entryFile), "utf8")).toBe("new board instructions");
|
||||
await fs.writeFile(path.join(retried.localRoot, entryFile), "retry adds a change");
|
||||
expect((await copies.collectStopped({ companyId, runId: copy.runId }))?.state).toBe("saved");
|
||||
});
|
||||
|
||||
it("never rebases uncollected private bytes or a live warm copy onto a newer head", async () => {
|
||||
const stopped = await run();
|
||||
await copies.collectStopped({ companyId, runId: stopped.runId });
|
||||
await fs.writeFile(path.join(stopped.localRoot, entryFile), "uncollected private edit");
|
||||
const warm = await run();
|
||||
expect(await copies.hasChanges({ companyId, runId: warm.runId })).toBe(false);
|
||||
const prior = (await revisions.readCurrent(target(), board()))!;
|
||||
await revisions.commit({ ...target(), entryFile, baseRevisionId: prior.revision.id, content: "board change", source: "api" }, board());
|
||||
for (const copy of [stopped, warm]) {
|
||||
const retried = (await copies.prepare({ legacy: true, ...target(), runId: copy.runId, cwd: home }))!;
|
||||
expect(retried.baseRevisionId).toBe(copy.baseRevisionId);
|
||||
}
|
||||
expect(await fs.readFile(path.join(stopped.localRoot, entryFile), "utf8")).toBe("uncollected private edit");
|
||||
expect(await fs.readFile(path.join(warm.localRoot, entryFile), "utf8")).toBe(initial);
|
||||
expect((await copies.collectStopped({ companyId, runId: stopped.runId }))?.state).toBe("conflict");
|
||||
expect((await revisions.readCurrent(target(), board()))?.content).toBe("board change");
|
||||
});
|
||||
|
||||
it.each(["before replacement", "after replacement"])("preserves the completed receipt when retry staging fails %s", async (point) => {
|
||||
const copy = await run();
|
||||
await fs.writeFile(path.join(copy.localRoot, entryFile), "completed turn");
|
||||
const saved = (await copies.collectStopped({ companyId, runId: copy.runId }))!;
|
||||
expect(saved.state).toBe("saved");
|
||||
const prior = (await revisions.readCurrent(target(), board()))!;
|
||||
const current = await revisions.commit({ ...target(), entryFile, baseRevisionId: prior.revision.id, content: "new board instructions", source: "api" }, board());
|
||||
const originalMkdir = fs.mkdir.bind(fs);
|
||||
const originalChmod = fs.chmod.bind(fs);
|
||||
const mkdir = vi.spyOn(fs, "mkdir").mockImplementation(async (...args: Parameters<typeof fs.mkdir>) => {
|
||||
if (point === "before replacement" && args[0] === copy.localRoot) throw new Error("retry staging failed");
|
||||
return originalMkdir(...args);
|
||||
});
|
||||
const chmod = vi.spyOn(fs, "chmod").mockImplementation(async (...args) => {
|
||||
if (point === "after replacement" && args[0] === path.join(copy.localRoot, entryFile)) throw new Error("retry staging failed");
|
||||
return originalChmod(...args);
|
||||
});
|
||||
try {
|
||||
await expect(copies.prepare({ legacy: true, ...target(), runId: copy.runId, cwd: home })).rejects.toThrow("retry staging failed");
|
||||
} finally { mkdir.mockRestore(); chmod.mockRestore(); }
|
||||
expect(await copies.get(companyId, copy.runId)).toEqual(saved);
|
||||
copies = agentInstructionWorkingCopyService(db);
|
||||
await copies.recoverStopped();
|
||||
expect(await copies.get(companyId, copy.runId)).toEqual(saved);
|
||||
expect((await revisions.readCurrent(target(), board()))?.revision.id).toBe(current.revision.id);
|
||||
const retried = (await copies.prepare({ legacy: true, ...target(), runId: copy.runId, cwd: home }))!;
|
||||
expect(retried).toMatchObject({ state: "prepared", baseRevisionId: current.revision.id, processStoppedAt: null });
|
||||
expect(await fs.readFile(path.join(retried.localRoot, entryFile), "utf8")).toBe(current.content);
|
||||
});
|
||||
|
||||
it("reads an existing canonical runtime snapshot without seeding or changing instruction bytes", async () => {
|
||||
expect((await revisions.readCommittedForRuntime(target()))?.content).toBe(initial);
|
||||
const copy = await run();
|
||||
await fs.writeFile(path.join(copy.localRoot, entryFile), "saved runtime content");
|
||||
await copies.collectStopped({ companyId, runId: copy.runId });
|
||||
const current = (await revisions.readCurrent(target(), board()))!;
|
||||
await fs.writeFile(path.join(root, entryFile), "current filesystem contents");
|
||||
expect((await revisions.readCommittedForRuntime(target()))?.content).toBe("current filesystem contents");
|
||||
const [agent] = await db.select().from(agents).where(eq(agents.id, agentId));
|
||||
const context = await buildNativeRuntimeContext({ db, agent, runId: copy.runId, runtimeConfig: {}, runtimeSkillEntries: [] });
|
||||
try {
|
||||
expect(context.instructions.entryPath).toBe(entryFile);
|
||||
expect(await fs.readFile(path.join(context.instructions.bundle.rootPath, entryFile), "utf8"))
|
||||
.toBe("current filesystem contents");
|
||||
} finally {
|
||||
// Runtime assets are immutable, so make only this fixture's directories
|
||||
// removable before the ordinary temporary-home cleanup.
|
||||
const makeRemovable = async (directory: string): Promise<void> => {
|
||||
await fs.chmod(directory, 0o700);
|
||||
for (const child of await fs.readdir(directory, { withFileTypes: true })) {
|
||||
if (child.isDirectory()) await makeRemovable(path.join(directory, child.name));
|
||||
}
|
||||
};
|
||||
await makeRemovable(context.instructions.bundle.rootPath);
|
||||
}
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe("current filesystem contents");
|
||||
expect((await revisions.history({ ...target(), entryFile }, board())).revisions).toHaveLength(0);
|
||||
await expect(revisions.readCommittedForRuntime({ companyId: randomUUID(), agentId })).rejects.toThrow("Agent not found");
|
||||
});
|
||||
|
||||
it("preserves a competing stale candidate and requires explicit resolution against the new head", async () => {
|
||||
const first = await run();
|
||||
const second = await run();
|
||||
await fs.writeFile(path.join(first.localRoot, entryFile), "first run");
|
||||
await fs.writeFile(path.join(second.localRoot, entryFile), "second run");
|
||||
await copies.collectStopped({ companyId, runId: first.runId });
|
||||
const conflict = await copies.collectStopped({ companyId, runId: second.runId });
|
||||
expect(conflict).toMatchObject({ state: "conflict", errorCode: "INSTRUCTION_REVISION_CONFLICT" });
|
||||
expect(Buffer.from(conflict!.candidateBase64!, "base64").toString("utf8")).toBe("second run");
|
||||
expect((await revisions.readCurrent(target(), board()))?.content).toBe("first run");
|
||||
copies = agentInstructionWorkingCopyService(db);
|
||||
expect(await copies.list(companyId, agentId, board())).toHaveLength(1);
|
||||
const current = (await revisions.readCurrent(target(), board()))!;
|
||||
await copies.resolve({ ...target(), runId: second.runId, baseRevisionId: current.revision.id, content: "explicitly combined" }, board());
|
||||
expect((await revisions.readCurrent(target(), board()))?.content).toBe("explicitly combined");
|
||||
expect((await copies.collectStopped({ companyId, runId: second.runId }))?.state).toBe("resolved");
|
||||
const resumed = (await copies.prepare({ legacy: true, ...target(), runId: second.runId, cwd: home }))!;
|
||||
const resolved = (await revisions.readCurrent(target(), board()))!;
|
||||
expect(resumed.baseRevisionId).toBe(resolved.revision.id);
|
||||
expect(await fs.readFile(path.join(resumed.localRoot, entryFile), "utf8")).toBe("explicitly combined");
|
||||
});
|
||||
|
||||
it("rechecks responsible-user permission at collection and preserves denied edits", async () => {
|
||||
const copy = await run();
|
||||
await fs.writeFile(path.join(copy.localRoot, entryFile), "candidate after revocation");
|
||||
await db.update(companyMemberships).set({ status: "suspended" }).where(eq(companyMemberships.principalId, userId));
|
||||
const denied = await copies.collectStopped({ companyId, runId: copy.runId });
|
||||
expect(denied).toMatchObject({ state: "conflict", errorCode: "RESPONSIBLE_USER_UNAVAILABLE" });
|
||||
expect(denied?.candidateBase64).not.toBeNull();
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(initial);
|
||||
});
|
||||
|
||||
it("does not import a symlink or confuse a removed entry with an empty file", async () => {
|
||||
const linked = await run();
|
||||
await fs.unlink(path.join(linked.localRoot, entryFile));
|
||||
await fs.symlink(path.join(home, "AGENTS.md"), path.join(linked.localRoot, entryFile));
|
||||
expect((await copies.collectStopped({ companyId, runId: linked.runId }))?.state).toBe("pending_collection");
|
||||
const missing = await run();
|
||||
await fs.unlink(path.join(missing.localRoot, entryFile));
|
||||
expect((await copies.collectStopped({ companyId, runId: missing.runId }))?.state).toBe("unavailable");
|
||||
const empty = await run();
|
||||
await fs.writeFile(path.join(empty.localRoot, entryFile), "");
|
||||
expect((await copies.collectStopped({ companyId, runId: empty.runId }))?.state).toBe("saved");
|
||||
expect((await revisions.readCurrent(target(), board()))?.content).toBe("");
|
||||
});
|
||||
|
||||
it("keeps unchanged warm copies reusable without a content revision", async () => {
|
||||
const copy = await run();
|
||||
expect(await copies.hasChanges({ companyId, runId: copy.runId })).toBe(false);
|
||||
expect((await copies.get(companyId, copy.runId))?.processStoppedAt).toBeNull();
|
||||
expect((await revisions.history({ ...target(), entryFile }, board())).revisions).toHaveLength(0);
|
||||
await fs.writeFile(path.join(copy.localRoot, entryFile), "changed after turn");
|
||||
expect(await copies.hasChanges({ companyId, runId: copy.runId })).toBe(true);
|
||||
expect((await revisions.readCurrent(target(), board()))?.content).toBe(initial);
|
||||
});
|
||||
|
||||
it("advances an explicit save baseline only when the private copy matches that revision", async () => {
|
||||
const copy = await run();
|
||||
const local = path.join(copy.localRoot, entryFile);
|
||||
await fs.writeFile(local, "explicit self-save");
|
||||
const first = await revisions.commit({ ...target(), entryFile, baseRevisionId: copy.baseRevisionId, content: "explicit self-save", source: "tool" }, board());
|
||||
await copies.acknowledgeExplicitSave({ ...target(), runId: copy.runId, entryFile, revisionId: first.revision.id, contentHash: first.revision.contentHash });
|
||||
await fs.writeFile(local, "local edit after explicit save");
|
||||
expect((await copies.collectStopped({ companyId, runId: copy.runId }))?.state).toBe("saved");
|
||||
expect((await revisions.readCurrent(target(), board()))?.content).toBe("local edit after explicit save");
|
||||
|
||||
const competing = await run();
|
||||
await fs.writeFile(path.join(competing.localRoot, entryFile), "older local draft");
|
||||
const separate = await revisions.commit({ ...target(), entryFile, baseRevisionId: competing.baseRevisionId, content: "different explicit content", source: "tool" }, board());
|
||||
await copies.acknowledgeExplicitSave({ ...target(), runId: competing.runId, entryFile, revisionId: separate.revision.id, contentHash: separate.revision.contentHash });
|
||||
expect((await copies.collectStopped({ companyId, runId: competing.runId }))?.state).toBe("conflict");
|
||||
expect((await revisions.readCurrent(target(), board()))?.content).toBe("different explicit content");
|
||||
expect((await copies.list(companyId, agentId, board()))[0]).not.toHaveProperty("localRoot");
|
||||
});
|
||||
|
||||
it("recovers captured bytes after restart and deduplicates concurrent cleanup callbacks", async () => {
|
||||
const copy = await run();
|
||||
await db.update(agentInstructionWorkingCopies).set({ state: "pending_commit", candidateBase64: Buffer.from("captured before restart").toString("base64"), processStoppedAt: new Date() })
|
||||
.where(eq(agentInstructionWorkingCopies.runId, copy.runId));
|
||||
copies = agentInstructionWorkingCopyService(db);
|
||||
await copies.recoverCaptured();
|
||||
expect((await revisions.readCurrent(target(), board()))?.content).toBe("captured before restart");
|
||||
const duplicate = await run();
|
||||
await fs.writeFile(path.join(duplicate.localRoot, entryFile), "one final edit");
|
||||
await Promise.all([copies.collectStopped({ companyId, runId: duplicate.runId }), copies.collectStopped({ companyId, runId: duplicate.runId })]);
|
||||
expect((await copies.get(companyId, duplicate.runId))?.state).toBe("saved");
|
||||
expect((await revisions.history({ ...target(), entryFile }, board())).revisions).toHaveLength(0);
|
||||
});
|
||||
it("requires durable stop evidence before recovering an uncaptured copy", async () => {
|
||||
const copy = await run();
|
||||
await fs.writeFile(path.join(copy.localRoot, entryFile), "edit before controller restart");
|
||||
await db.update(heartbeatRuns).set({ runtimeMode: "native", status: "failed", finishedAt: new Date() }).where(eq(heartbeatRuns.id, copy.runId));
|
||||
copies = agentInstructionWorkingCopyService(db);
|
||||
await copies.recoverStopped();
|
||||
expect((await revisions.readCurrent(target(), board()))?.content).toBe(initial);
|
||||
expect((await copies.list(companyId, agentId, board()))[0]).toMatchObject({ state: "pending_collection", content: null });
|
||||
await appendHeartbeatRunEvent(db, { ...target(), runId: copy.runId, eventType: "native.local_process_stopped", stream: "system" });
|
||||
await copies.recoverStopped();
|
||||
expect((await revisions.readCurrent(target(), board()))?.content).toBe("edit before controller restart");
|
||||
});
|
||||
|
||||
it("never treats a superseded stop receipt or unavailable environment as a save", async () => {
|
||||
const copy = await run();
|
||||
await fs.writeFile(path.join(copy.localRoot, entryFile), "unverified live edit");
|
||||
await db.update(heartbeatRuns).set({ runtimeMode: "native", status: "failed", finishedAt: new Date() }).where(eq(heartbeatRuns.id, copy.runId));
|
||||
await appendHeartbeatRunEvent(db, { ...target(), runId: copy.runId, eventType: "native.local_process_stopped", stream: "system" });
|
||||
await appendHeartbeatRunEvent(db, { ...target(), runId: copy.runId, eventType: "native.process_start_requested", stream: "system" });
|
||||
await copies.recoverStopped();
|
||||
expect((await revisions.readCurrent(target(), board()))?.content).toBe(initial);
|
||||
const lost = await copies.reportUnavailable(companyId, copy.runId);
|
||||
expect(lost).toMatchObject({ state: "unavailable", candidateBase64: null, processStoppedAt: null });
|
||||
expect(lost?.errorMessage).not.toContain(copy.localRoot);
|
||||
});
|
||||
|
||||
it("keeps private run copies out of normal Git staging and workspace snapshots", async () => {
|
||||
const workspace = path.join(home, `workspace-${randomUUID()}`);
|
||||
await fs.mkdir(workspace);
|
||||
await execFile("git", ["init", workspace]);
|
||||
await execFile("git", ["-C", workspace, "-c", "user.name=Fixture", "-c", "user.email=fixture@example.test", "commit", "--allow-empty", "-m", "fixture"]);
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand", responsibleUserId: userId });
|
||||
const copy = (await copies.prepare({ legacy: true, ...target(), runId, cwd: workspace }))!;
|
||||
expect(copy.localRoot).toBe(path.join(workspace, ".paperclip-runtime", `instruction-edits-${runId}`, "instructions"));
|
||||
await fs.writeFile(path.join(workspace, "deliverable.txt"), "public work");
|
||||
await execFile("git", ["-C", workspace, "add", "."]);
|
||||
const staged = await execFile("git", ["-C", workspace, "diff", "--cached", "--name-only"]);
|
||||
expect(staged.stdout).toBe("deliverable.txt\n");
|
||||
const snapshot = await gitWorkspaceSync.readGitWorkspaceSnapshot(workspace);
|
||||
try {
|
||||
const paths: unknown = snapshot?.overlayPaths;
|
||||
if (Array.isArray(paths)) {
|
||||
expect(paths).toEqual(["deliverable.txt"]);
|
||||
} else {
|
||||
// The streaming Git snapshot stores the same path set in a private
|
||||
// manifest. Inspect its actual records, not only the reported count.
|
||||
expect(paths).toMatchObject({ kind: "path_manifest", version: 1, category: "overlay", count: 1 });
|
||||
const manifest = paths as { filePath: string; category: string };
|
||||
const manifestDb = new DatabaseSync(manifest.filePath, { readOnly: true, allowExtension: false });
|
||||
try {
|
||||
const rows = manifestDb.prepare("SELECT path FROM records WHERE category = ? ORDER BY path").all(manifest.category);
|
||||
expect(rows.map((row) => row.path)).toEqual(["deliverable.txt"]);
|
||||
} finally { manifestDb.close(); }
|
||||
}
|
||||
} finally {
|
||||
if ("disposeGitWorkspaceSnapshot" in gitWorkspaceSync && typeof gitWorkspaceSync.disposeGitWorkspaceSnapshot === "function") {
|
||||
await gitWorkspaceSync.disposeGitWorkspaceSnapshot(snapshot);
|
||||
}
|
||||
}
|
||||
const files = await captureDirectorySnapshot(workspace, { exclude: [".git", ".paperclip-runtime"] });
|
||||
expect([...files.entries].map(([relative]) => relative)).toEqual(["deliverable.txt"]);
|
||||
await expect(copies.prepare({ legacy: true, ...target(), runId, cwd: home })).rejects.toThrow("different run workspace");
|
||||
});
|
||||
|
||||
});
|
||||
@@ -1,5 +1,6 @@
|
||||
import express from "express";
|
||||
import request from "supertest";
|
||||
import { Readable } from "node:stream";
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
const mockAgentService = vi.hoisted(() => ({
|
||||
@@ -24,6 +25,21 @@ const mockAgentInstructionsService = vi.hoisted(() => ({
|
||||
materializeManagedBundle: vi.fn(),
|
||||
}));
|
||||
|
||||
const mockInstructionWorkingCopies = vi.hoisted(() => ({ list: vi.fn(), resolve: vi.fn(), acknowledgeExplicitSave: vi.fn() }));
|
||||
vi.mock("../services/agent-instruction-working-copies.js", () => ({ agentInstructionWorkingCopyService: () => mockInstructionWorkingCopies }));
|
||||
|
||||
const mockDownloadAgentFile = vi.hoisted(() => vi.fn());
|
||||
vi.mock("../services/agent-file-store.js", async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import("../services/agent-file-store.js")>();
|
||||
return { ...actual, agentFileStore: (...args: Parameters<typeof actual.agentFileStore>) => ({ ...actual.agentFileStore(...args), download: mockDownloadAgentFile }) };
|
||||
});
|
||||
|
||||
const mockInstructionRevisions = vi.hoisted(() => ({ readCurrent: vi.fn(), commit: vi.fn(), restore: vi.fn(), history: vi.fn(), readRevision: vi.fn(), diff: vi.fn(), materializeCurrent: vi.fn() }));
|
||||
vi.mock("../services/agent-instruction-revisions.js", () => ({ agentInstructionRevisionService: () => mockInstructionRevisions }));
|
||||
|
||||
const mockAuthorizeInstructionRead = vi.hoisted(() => vi.fn());
|
||||
vi.mock("../services/agent-instruction-authorization.js", () => ({ authorizeInstructionRead: mockAuthorizeInstructionRead }));
|
||||
|
||||
const mockAccessService = vi.hoisted(() => ({
|
||||
canUser: vi.fn(),
|
||||
decide: vi.fn(),
|
||||
@@ -202,6 +218,9 @@ describe("agent instructions bundle routes", () => {
|
||||
vi.doUnmock("../middleware/index.js");
|
||||
registerModuleMocks();
|
||||
vi.clearAllMocks();
|
||||
mockAuthorizeInstructionRead.mockImplementation(async (_db, actor) => actor);
|
||||
mockInstructionRevisions.readCurrent.mockResolvedValue(null);
|
||||
mockInstructionRevisions.commit.mockResolvedValue({ revision: { id: "33333333-3333-4333-8333-333333333333", entryFile: "AGENTS.md", byteLength: 18 }, content: "# Updated Agent\n", changed: true, materialization: "current" });
|
||||
mockBuiltInAgentService.ensureCompanyDefaultAgentGrants.mockResolvedValue(0);
|
||||
mockSyncInstructionsBundleConfigFromFilePath.mockImplementation((_agent, config) => config);
|
||||
mockFindServerAdapter.mockImplementation((_type: string) => ({ type: _type }));
|
||||
@@ -286,6 +305,8 @@ describe("agent instructions bundle routes", () => {
|
||||
entryFile: "AGENTS.md",
|
||||
});
|
||||
expect(mockAgentInstructionsService.getBundle).toHaveBeenCalled();
|
||||
expect(mockAuthorizeInstructionRead).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({ type: "board" }),
|
||||
expect.objectContaining({ companyId: "company-1", id: "11111111-1111-4111-8111-111111111111" }));
|
||||
});
|
||||
|
||||
it("requires instance-admin access for every external instruction entry point", async () => {
|
||||
@@ -331,6 +352,7 @@ describe("agent instructions bundle routes", () => {
|
||||
expect(mockAgentInstructionsService.updateBundle).not.toHaveBeenCalled();
|
||||
expect(mockAgentInstructionsService.writeFile).not.toHaveBeenCalled();
|
||||
expect(mockAgentInstructionsService.deleteFile).not.toHaveBeenCalled();
|
||||
expect(mockAuthorizeInstructionRead).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("treats a host root mislabeled as managed as external", async () => {
|
||||
@@ -383,6 +405,7 @@ describe("agent instructions bundle routes", () => {
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(200);
|
||||
expect(mockAgentInstructionsService.getBundle).toHaveBeenCalled();
|
||||
expect(mockAuthorizeInstructionRead).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("rejects a company admin that requests a new external instruction root", async () => {
|
||||
@@ -484,7 +507,8 @@ describe("agent instructions bundle routes", () => {
|
||||
expect(mockAgentService.create).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("denies non-privileged agents from reading peer instructions bundles", async () => {
|
||||
it.each(["", "/file?path=AGENTS.md", "/file?path=private-support.md"])(
|
||||
"denies peer instruction reads before any data dispatch: %s", async (suffix) => {
|
||||
mockAgentService.getById.mockImplementation(async (id: string) => {
|
||||
if (id === "agent-reader") {
|
||||
return {
|
||||
@@ -496,11 +520,10 @@ describe("agent instructions bundle routes", () => {
|
||||
}
|
||||
return makeAgent();
|
||||
});
|
||||
mockAccessService.decide.mockResolvedValue({
|
||||
allowed: false,
|
||||
reason: "deny_no_grant",
|
||||
explanation: "Missing permission: agents:configure or agents:suggest-changes.",
|
||||
});
|
||||
// General agent visibility is allowed. It must not authorize instruction
|
||||
// metadata, canonical entry bytes, or supporting files from a peer.
|
||||
const { forbidden } = await vi.importActual<typeof import("../errors.js")>("../errors.js");
|
||||
mockAuthorizeInstructionRead.mockRejectedValue(forbidden("Missing permission to read peer instructions"));
|
||||
|
||||
const res = await requestApp(
|
||||
await createApp({
|
||||
@@ -510,20 +533,18 @@ describe("agent instructions bundle routes", () => {
|
||||
source: "agent_key",
|
||||
}),
|
||||
(baseUrl) => request(baseUrl)
|
||||
.get("/api/agents/11111111-1111-4111-8111-111111111111/instructions-bundle"),
|
||||
.get(`/api/agents/11111111-1111-4111-8111-111111111111/instructions-bundle${suffix}`),
|
||||
);
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(403);
|
||||
expect(res.body.error).toContain("Missing permission");
|
||||
expect(mockAccessService.decide).toHaveBeenCalledWith(expect.objectContaining({
|
||||
action: "agent_config:read",
|
||||
resource: {
|
||||
type: "agent",
|
||||
companyId: "company-1",
|
||||
agentId: "11111111-1111-4111-8111-111111111111",
|
||||
},
|
||||
}));
|
||||
expect(mockAuthorizeInstructionRead).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({
|
||||
type: "agent", agentId: "agent-reader",
|
||||
}), { companyId: "company-1", id: "11111111-1111-4111-8111-111111111111" });
|
||||
expect(mockAgentInstructionsService.getBundle).not.toHaveBeenCalled();
|
||||
expect(mockAgentInstructionsService.readFile).not.toHaveBeenCalled();
|
||||
expect(mockInstructionRevisions.readCurrent).not.toHaveBeenCalled();
|
||||
expect(mockInstructionRevisions.materializeCurrent).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("allows agents to read their own instructions bundles", async () => {
|
||||
@@ -540,9 +561,12 @@ describe("agent instructions bundle routes", () => {
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(200);
|
||||
expect(mockAgentInstructionsService.getBundle).toHaveBeenCalled();
|
||||
expect(mockAuthorizeInstructionRead).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({
|
||||
type: "agent", agentId: "11111111-1111-4111-8111-111111111111",
|
||||
}), { companyId: "company-1", id: "11111111-1111-4111-8111-111111111111" });
|
||||
});
|
||||
|
||||
it("allows agents with suggest grants to read peer instructions bundles", async () => {
|
||||
it.each(["AGENTS.md", "private-support.md"])("allows an authorized peer to read instruction file %s", async (filePath) => {
|
||||
mockAccessService.decide.mockResolvedValue({
|
||||
allowed: true,
|
||||
reason: "allow_explicit_grant",
|
||||
@@ -570,52 +594,100 @@ describe("agent instructions bundle routes", () => {
|
||||
}),
|
||||
(baseUrl) => request(baseUrl)
|
||||
.get("/api/agents/11111111-1111-4111-8111-111111111111/instructions-bundle/file")
|
||||
.query({ path: "AGENTS.md" }),
|
||||
.query({ path: filePath }),
|
||||
);
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(200);
|
||||
expect(mockAccessService.decide).toHaveBeenCalledWith(expect.objectContaining({
|
||||
action: "agent_config:read",
|
||||
resource: {
|
||||
type: "agent",
|
||||
companyId: "company-1",
|
||||
agentId: "11111111-1111-4111-8111-111111111111",
|
||||
},
|
||||
}));
|
||||
expect(mockAuthorizeInstructionRead).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({
|
||||
type: "agent", agentId: "coach-agent",
|
||||
}), { companyId: "company-1", id: "11111111-1111-4111-8111-111111111111" });
|
||||
expect(mockAgentInstructionsService.readFile).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ id: "11111111-1111-4111-8111-111111111111" }),
|
||||
"AGENTS.md",
|
||||
filePath,
|
||||
);
|
||||
});
|
||||
|
||||
it("writes a bundle file and persists compatibility config", async () => {
|
||||
it("commits entry bytes through the canonical service with the authenticated actor", async () => {
|
||||
const res = await requestApp(await createApp(), (baseUrl) => request(baseUrl)
|
||||
.put("/api/agents/11111111-1111-4111-8111-111111111111/instructions-bundle/file?companyId=company-1")
|
||||
.send({
|
||||
path: "AGENTS.md",
|
||||
content: "# Updated Agent\n",
|
||||
clearLegacyPromptTemplate: true,
|
||||
}));
|
||||
|
||||
.send({ path: "AGENTS.md", content: "# Updated Agent\n", baseRevisionId: null }));
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(200);
|
||||
expect(mockAgentInstructionsService.writeFile).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ id: "11111111-1111-4111-8111-111111111111" }),
|
||||
"AGENTS.md",
|
||||
"# Updated Agent\n",
|
||||
{ clearLegacyPromptTemplate: true },
|
||||
);
|
||||
expect(mockAgentService.update).toHaveBeenCalledWith(
|
||||
"11111111-1111-4111-8111-111111111111",
|
||||
expect.objectContaining({
|
||||
adapterConfig: expect.objectContaining({
|
||||
instructionsBundleMode: "managed",
|
||||
instructionsRootPath: "/tmp/agent-1",
|
||||
instructionsEntryFile: "AGENTS.md",
|
||||
instructionsFilePath: "/tmp/agent-1/AGENTS.md",
|
||||
}),
|
||||
}),
|
||||
expect.any(Object),
|
||||
);
|
||||
expect(mockInstructionRevisions.commit).toHaveBeenCalledWith(expect.objectContaining({
|
||||
agentId: "11111111-1111-4111-8111-111111111111", entryFile: "AGENTS.md", content: "# Updated Agent\n", baseRevisionId: null, source: "board",
|
||||
}), expect.objectContaining({ type: "board" }));
|
||||
expect(mockAgentInstructionsService.writeFile).not.toHaveBeenCalled();
|
||||
expect(res.body.receipt.revision.id).toBe("33333333-3333-4333-8333-333333333333");
|
||||
});
|
||||
|
||||
it("returns committed entry content independently of its disk copy", async () => {
|
||||
mockInstructionRevisions.readCurrent.mockResolvedValue({ revision: { id: "33333333-3333-4333-8333-333333333333", entryFile: "AGENTS.md", byteLength: 9 }, content: "committed" });
|
||||
const res = await requestApp(await createApp(), (baseUrl) => request(baseUrl)
|
||||
.get("/api/agents/11111111-1111-4111-8111-111111111111/instructions-bundle/file?path=AGENTS.md"));
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.content).toBe("committed");
|
||||
expect(res.body.revision.id).toBe("33333333-3333-4333-8333-333333333333");
|
||||
expect(mockAgentInstructionsService.readFile).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each([Buffer.from([0, 255, 128, 17]), Buffer.alloc(0)])("streams an authorized binary download without text conversion (%j)", async bytes => {
|
||||
mockAgentService.getById.mockResolvedValue({ ...makeAgent(), adapterConfig: { instructionsBundleMode: "managed" } });
|
||||
mockDownloadAgentFile.mockResolvedValue({ size: bytes.length, stream: Readable.from(bytes.length ? [bytes] : []) });
|
||||
const res = await requestApp(await createApp(), url => request(url)
|
||||
.get("/api/agents/11111111-1111-4111-8111-111111111111/instructions-bundle/file")
|
||||
.query({ path: "notes/data.bin", download: "true" }));
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(200);
|
||||
expect(res.body).toEqual(bytes);
|
||||
expect(res.headers["content-length"]).toBe(String(bytes.length));
|
||||
expect(res.headers["content-disposition"]).toContain('filename="data.bin"');
|
||||
expect(mockDownloadAgentFile).toHaveBeenCalledWith("company-1", "11111111-1111-4111-8111-111111111111", "notes/data.bin", expect.objectContaining({ type: "board" }));
|
||||
expect(mockAgentInstructionsService.readFile).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("lists and resolves preserved edits with server scope, actor, and explicit revision", async () => {
|
||||
const app = await createApp();
|
||||
const runId = "55555555-5555-4555-8555-555555555555";
|
||||
const baseRevisionId = "44444444-4444-4444-8444-444444444444";
|
||||
const prefix = "/api/agents/11111111-1111-4111-8111-111111111111/instructions-bundle/candidates";
|
||||
mockInstructionWorkingCopies.list.mockResolvedValue([{ runId, entryFile: "AGENTS.md", state: "conflict", content: "preserved" }]);
|
||||
mockInstructionWorkingCopies.resolve.mockResolvedValue({ revision: { id: baseRevisionId, entryFile: "AGENTS.md", byteLength: 8 }, content: "resolved", changed: true, materialization: "current" });
|
||||
const listed = await requestApp(app, (url) => request(url).get(prefix));
|
||||
expect(listed.status).toBe(200);
|
||||
expect(listed.body).toEqual([{ runId, entryFile: "AGENTS.md", state: "conflict", content: "preserved" }]);
|
||||
expect(mockInstructionWorkingCopies.list).toHaveBeenCalledWith("company-1", "11111111-1111-4111-8111-111111111111", expect.objectContaining({ type: "board", userId: "local-board" }));
|
||||
const resolved = await requestApp(app, (url) => request(url).post(`${prefix}/${runId}/resolve`).send({ baseRevisionId, content: "resolved" }));
|
||||
expect(resolved.status).toBe(200);
|
||||
expect(resolved.body).toMatchObject({ content: "resolved", receipt: { changed: true } });
|
||||
expect(mockInstructionWorkingCopies.resolve).toHaveBeenCalledWith({ companyId: "company-1", agentId: "11111111-1111-4111-8111-111111111111", runId, baseRevisionId, content: "resolved" }, expect.objectContaining({ type: "board", userId: "local-board" }));
|
||||
for (const body of [{ content: "missing base" }, { baseRevisionId, content: "forged", responsibleUserId: "other" }, { baseRevisionId, content: "forged", entryFile: "other.md" }]) {
|
||||
expect((await requestApp(app, (url) => request(url).post(`${prefix}/${runId}/resolve`).send(body))).status).toBe(400);
|
||||
}
|
||||
expect(mockInstructionWorkingCopies.resolve).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it("exposes scoped history, diff and restore with the server actor", async () => {
|
||||
const app = await createApp();
|
||||
const revisionId = "33333333-3333-4333-8333-333333333333";
|
||||
const baseRevisionId = "44444444-4444-4444-8444-444444444444";
|
||||
const prefix = "/api/agents/11111111-1111-4111-8111-111111111111/instructions-bundle";
|
||||
mockInstructionRevisions.history.mockResolvedValue({ revisions: [], nextCursor: null });
|
||||
mockInstructionRevisions.diff.mockResolvedValue({ removed: "old", added: "new" });
|
||||
mockInstructionRevisions.restore.mockResolvedValue({ revision: { id: revisionId, entryFile: "AGENTS.md", byteLength: 3 }, content: "old", changed: true, materialization: "current" });
|
||||
expect((await requestApp(app, (url) => request(url).get(`${prefix}/history?path=AGENTS.md`))).status).toBe(200);
|
||||
expect((await requestApp(app, (url) => request(url).get(`${prefix}/diff?path=AGENTS.md&from=${revisionId}&to=${baseRevisionId}`))).status).toBe(200);
|
||||
const restored = await requestApp(app, (url) => request(url).post(`${prefix}/restore`).send({ path: "AGENTS.md", revisionId, baseRevisionId }));
|
||||
expect(restored.status).toBe(200);
|
||||
expect(restored.body.receipt.changed).toBe(true);
|
||||
expect(mockInstructionRevisions.restore).toHaveBeenCalledWith({ companyId: "company-1", agentId: "11111111-1111-4111-8111-111111111111", entryFile: "AGENTS.md", revisionId, baseRevisionId }, expect.objectContaining({ type: "board", userId: "local-board" }));
|
||||
});
|
||||
|
||||
it("requires a base revision and rejects client-supplied responsible identity", async () => {
|
||||
const app = await createApp();
|
||||
for (const body of [ { path: "AGENTS.md", content: "stale" }, { path: "AGENTS.md", content: "forged", baseRevisionId: null, responsibleUserId: "forged" } ]) {
|
||||
const res = await requestApp(app, (baseUrl) => request(baseUrl)
|
||||
.put("/api/agents/11111111-1111-4111-8111-111111111111/instructions-bundle/file").send(body));
|
||||
expect([400, 422]).toContain(res.status);
|
||||
}
|
||||
expect(mockInstructionRevisions.commit).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("preserves managed instructions config when switching adapters", async () => {
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
||||
import { agentInstructionsService } from "../services/agent-instructions.js";
|
||||
|
||||
type TestAgent = {
|
||||
@@ -12,7 +12,7 @@ type TestAgent = {
|
||||
};
|
||||
|
||||
async function makeTempDir(prefix: string) {
|
||||
return fs.mkdtemp(path.join(os.tmpdir(), prefix));
|
||||
return fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), prefix)));
|
||||
}
|
||||
|
||||
function makeAgent(adapterConfig: Record<string, unknown>): TestAgent {
|
||||
@@ -29,6 +29,13 @@ describe("agent instructions service", () => {
|
||||
const originalPaperclipInstanceId = process.env.PAPERCLIP_INSTANCE_ID;
|
||||
const cleanupDirs = new Set<string>();
|
||||
|
||||
beforeEach(async () => {
|
||||
const home = await makeTempDir("agent-instructions-test-home-");
|
||||
cleanupDirs.add(home);
|
||||
process.env.PAPERCLIP_HOME = home;
|
||||
process.env.PAPERCLIP_INSTANCE_ID = "instructions-service-test";
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
if (originalPaperclipHome === undefined) delete process.env.PAPERCLIP_HOME;
|
||||
else process.env.PAPERCLIP_HOME = originalPaperclipHome;
|
||||
@@ -41,6 +48,37 @@ describe("agent instructions service", () => {
|
||||
}));
|
||||
});
|
||||
|
||||
it.each(["./AGENTS.md", "notes/../AGENTS.md", "notes\\..\\AGENTS.md", "../invalid"])("reads legacy entry configuration %s without breaking the bundle", async (entry) => {
|
||||
const root = await makeTempDir("legacy-entry-"); cleanupDirs.add(root);
|
||||
await fs.writeFile(path.join(root, "AGENTS.md"), "legacy instructions");
|
||||
const agent = makeAgent({ instructionsBundleMode: "external", instructionsRootPath: root, instructionsEntryFile: entry });
|
||||
const svc = agentInstructionsService();
|
||||
const bundle = await svc.getBundle(agent);
|
||||
expect(bundle.entryFile).toBe("AGENTS.md");
|
||||
expect((await svc.readFile(agent, bundle.entryFile)).content).toBe("legacy instructions");
|
||||
if (entry === "../invalid") expect(bundle.warnings.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it("lists external bundles containing large binary assets", async () => {
|
||||
const root = await makeTempDir("external-large-assets-"); cleanupDirs.add(root);
|
||||
await fs.writeFile(path.join(root, "AGENTS.md"), "external instructions");
|
||||
const asset = await fs.open(path.join(root, "large.bin"), "w");
|
||||
try { await asset.truncate(17 * 1024 * 1024); } finally { await asset.close(); }
|
||||
const agent = makeAgent({ instructionsBundleMode: "external", instructionsRootPath: root });
|
||||
const svc = agentInstructionsService();
|
||||
const bundle = await svc.getBundle(agent);
|
||||
expect(bundle.files).toContainEqual(expect.objectContaining({ path: "large.bin", binary: true, editable: false }));
|
||||
expect((await svc.readFile(agent, "AGENTS.md")).content).toBe("external instructions");
|
||||
});
|
||||
|
||||
it("rejects reserved paths while initializing an unconfigured managed bundle", async () => {
|
||||
const root = await makeTempDir("unconfigured-reserved-path-"); cleanupDirs.add(root);
|
||||
process.env.PAPERCLIP_HOME = root;
|
||||
const svc = agentInstructionsService();
|
||||
await expect(svc.writeFile(makeAgent({}), ".paperclip-runtime/state", "invalid")).rejects.toMatchObject({ status: 422 });
|
||||
expect(await fs.readdir(root)).toEqual([]);
|
||||
});
|
||||
|
||||
it("copies the existing bundle into the managed root when switching to managed mode", async () => {
|
||||
const paperclipHome = await makeTempDir("paperclip-agent-instructions-home-");
|
||||
const externalRoot = await makeTempDir("paperclip-agent-instructions-external-");
|
||||
@@ -53,7 +91,7 @@ describe("agent instructions service", () => {
|
||||
await fs.mkdir(path.join(externalRoot, "docs"), { recursive: true });
|
||||
await fs.writeFile(path.join(externalRoot, "docs", "TOOLS.md"), "## Tools\n", "utf8");
|
||||
|
||||
const svc = agentInstructionsService();
|
||||
const svc = agentInstructionsService({ select: () => ({ from: () => ({ where: async () => [] }) }) } as never);
|
||||
const agent = makeAgent({
|
||||
instructionsBundleMode: "external",
|
||||
instructionsRootPath: externalRoot,
|
||||
@@ -250,11 +288,10 @@ describe("agent instructions service", () => {
|
||||
|
||||
expect(bundle.mode).toBe("managed");
|
||||
expect(bundle.rootPath).toBe(managedRoot);
|
||||
expect(bundle.entryFile).toBe("AGENTS.md");
|
||||
expect(bundle.entryFile).toBe("docs/MISSING.md");
|
||||
expect(bundle.files.map((file) => file.path)).toEqual(["AGENTS.md"]);
|
||||
expect(bundle.warnings).toEqual([
|
||||
`Recovered managed instructions from disk at ${managedRoot}; ignoring stale configured root ${staleRoot}.`,
|
||||
"Recovered managed instructions entry file from disk as AGENTS.md; previous entry docs/MISSING.md was missing.",
|
||||
]);
|
||||
expect(exported.files).toEqual({ "AGENTS.md": "# Managed Agent\n" });
|
||||
});
|
||||
@@ -293,8 +330,8 @@ describe("agent instructions service", () => {
|
||||
expect(result.adapterConfig).toMatchObject({
|
||||
instructionsBundleMode: "managed",
|
||||
instructionsRootPath: managedRoot,
|
||||
instructionsEntryFile: "AGENTS.md",
|
||||
instructionsFilePath: path.join(managedRoot, "AGENTS.md"),
|
||||
instructionsEntryFile: "docs/MISSING.md",
|
||||
instructionsFilePath: path.join(managedRoot, "docs/MISSING.md"),
|
||||
});
|
||||
await expect(fs.readFile(path.join(managedRoot, "docs", "TOOLS.md"), "utf8")).resolves.toBe("## Tools\n");
|
||||
});
|
||||
@@ -334,8 +371,8 @@ describe("agent instructions service", () => {
|
||||
expect(result.adapterConfig).toMatchObject({
|
||||
instructionsBundleMode: "managed",
|
||||
instructionsRootPath: managedRoot,
|
||||
instructionsEntryFile: "AGENTS.md",
|
||||
instructionsFilePath: path.join(managedRoot, "AGENTS.md"),
|
||||
instructionsEntryFile: "docs/MISSING.md",
|
||||
instructionsFilePath: path.join(managedRoot, "docs/MISSING.md"),
|
||||
});
|
||||
await expect(fs.stat(path.join(managedRoot, "docs", "TOOLS.md"))).rejects.toThrow();
|
||||
expect(result.bundle.files.map((file) => file.path)).toEqual(["AGENTS.md"]);
|
||||
@@ -373,11 +410,10 @@ describe("agent instructions service", () => {
|
||||
|
||||
expect(bundle.mode).toBe("managed");
|
||||
expect(bundle.rootPath).toBe(managedRoot);
|
||||
expect(bundle.entryFile).toBe("AGENTS.md");
|
||||
expect(bundle.entryFile).toBe("docs/MISSING.md");
|
||||
expect(bundle.files.map((file) => file.path)).toEqual(["AGENTS.md"]);
|
||||
expect(bundle.warnings).toEqual([
|
||||
`Recovered managed instructions from disk at ${managedRoot}; ignoring stale configured root ${staleRoot}.`,
|
||||
"Recovered managed instructions entry file from disk as AGENTS.md; previous entry docs/MISSING.md was missing.",
|
||||
]);
|
||||
expect(exported.files).toEqual({ "AGENTS.md": "# Managed Agent\n" });
|
||||
});
|
||||
|
||||
@@ -412,7 +412,7 @@ describe("built-in agent routes", () => {
|
||||
.send({});
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(200);
|
||||
expect(mockBuiltInAgentService.reset).toHaveBeenCalledWith(companyId, "briefs", {});
|
||||
expect(mockBuiltInAgentService.reset).toHaveBeenCalledWith(companyId, "briefs", {}, expect.objectContaining({ type: "agent", agentId: "manager-agent" }));
|
||||
expect(mockLogActivity).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({
|
||||
companyId,
|
||||
actorType: "agent",
|
||||
|
||||
@@ -1,3 +1,6 @@
|
||||
import fs from "node:fs/promises";
|
||||
import { fileHash } from "../services/agent-file-store.js";
|
||||
import { stockHash } from "../services/managed-resource-drift.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
@@ -27,6 +30,7 @@ import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
} from "./helpers/embedded-postgres.js";
|
||||
import { agentInstructionRevisionService } from "../services/agent-instruction-revisions.js";
|
||||
import { agentInstructionsService } from "../services/agent-instructions.ts";
|
||||
import { agentService } from "../services/agents.ts";
|
||||
import { approvalService } from "../services/approvals.ts";
|
||||
@@ -116,6 +120,16 @@ describe("built-in agent asset loading", () => {
|
||||
describeEmbeddedPostgres("built-in agents", () => {
|
||||
let db!: ReturnType<typeof createDb>;
|
||||
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||
const instructionOperator = { type: "board", userId: "local-board", source: "local_implicit" } as const;
|
||||
async function writeInstructionEntry(agent: { id: string; companyId: string }, entryFile: string, content: string) {
|
||||
const service = agentInstructionRevisionService(db);
|
||||
const target = { companyId: agent.companyId, agentId: agent.id };
|
||||
const baseline = await service.readCurrent(target, instructionOperator);
|
||||
await service.commit({ ...target, entryFile, content, baseRevisionId: baseline?.revision.id ?? null, source: "board" }, instructionOperator);
|
||||
const refreshed = await agentService(db).getById(agent.id);
|
||||
return { adapterConfig: refreshed!.adapterConfig };
|
||||
}
|
||||
|
||||
|
||||
beforeAll(async () => {
|
||||
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-built-in-agents-");
|
||||
@@ -553,7 +567,7 @@ describeEmbeddedPostgres("built-in agents", () => {
|
||||
capabilities: "Custom purpose",
|
||||
});
|
||||
|
||||
const reset = await builtIns.reset(companyId, "briefs");
|
||||
const reset = await builtIns.reset(companyId, "briefs", {}, instructionOperator);
|
||||
|
||||
expect(reset).toMatchObject({
|
||||
status: "ready",
|
||||
@@ -795,13 +809,111 @@ describeEmbeddedPostgres("built-in agents", () => {
|
||||
expect(approvalRows).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("automatically upgrades untouched stock instructions while preserving personal files", async () => {
|
||||
const companyId = await seedCompany({ requireApproval: false });
|
||||
const svc = builtInAgentService(db);
|
||||
const created = await svc.ensure(companyId, "reflection-coach");
|
||||
const agent = created.agent!;
|
||||
const oldFiles = { "AGENTS.md": "# Previous stock instructions\n", "obsolete.txt": "old stock support" };
|
||||
await writeInstructionEntry(agent, "AGENTS.md", oldFiles["AGENTS.md"]);
|
||||
await agentInstructionsService(db).writeFile(agent, "obsolete.txt", oldFiles["obsolete.txt"]);
|
||||
await agentInstructionsService(db).writeFile(agent, "personal.txt", "keep my notes");
|
||||
await db.update(builtInManagedResources).set({ stockHash: stockHash(oldFiles), stockVersion: "previous",
|
||||
defaultsJson: { entryFile: "AGENTS.md", files: Object.keys(oldFiles) },
|
||||
}).where(and(eq(builtInManagedResources.companyId, companyId), eq(builtInManagedResources.resourceKind, "instructions")));
|
||||
const updated = await svc.ensure(companyId, "reflection-coach");
|
||||
expect(updated.resources.find(resource => resource.resourceKind === "instructions")).toMatchObject({ stockStatus: "stock_current" });
|
||||
expect((await agentInstructionsService().readFile(updated.agent!, "AGENTS.md")).content).toContain("You are Reflection Coach");
|
||||
expect((await agentInstructionsService().readFile(updated.agent!, "personal.txt")).content).toBe("keep my notes");
|
||||
await expect(agentInstructionsService().readFile(updated.agent!, "obsolete.txt")).rejects.toMatchObject({ status: 404 });
|
||||
const [event] = await db.select().from(activityLog).where(and(eq(activityLog.companyId, companyId), eq(activityLog.actorId, "built-in-reconcile")));
|
||||
expect(event).toMatchObject({ actorType: "system", action: "agent.files_updated" });
|
||||
});
|
||||
|
||||
it.each(["file", "database"])("retries a stock update after a %s failure without overwriting intervening edits", async (failure) => {
|
||||
const companyId = await seedCompany({ requireApproval: false });
|
||||
const svc = builtInAgentService(db);
|
||||
const created = await svc.ensure(companyId, "reflection-coach");
|
||||
const agent = created.agent!;
|
||||
const oldFiles = { "AGENTS.md": "# Old stock\n", "obsolete.txt": "old supporting file" };
|
||||
await writeInstructionEntry(agent, "AGENTS.md", oldFiles["AGENTS.md"]);
|
||||
await agentInstructionsService(db).writeFile(agent, "obsolete.txt", oldFiles["obsolete.txt"]);
|
||||
await db.update(builtInManagedResources).set({ stockHash: stockHash(oldFiles), stockVersion: "old",
|
||||
defaultsJson: { entryFile: "AGENTS.md", files: Object.keys(oldFiles) },
|
||||
}).where(and(eq(builtInManagedResources.companyId, companyId), eq(builtInManagedResources.resourceKind, "instructions")));
|
||||
const trigger = `stock_failure_${companyId.replaceAll("-", "")}`;
|
||||
const remove = fs.rm.bind(fs);
|
||||
const fault = failure === "file" ? vi.spyOn(fs, "rm").mockImplementation(async (file, options) => {
|
||||
if (String(file).endsWith("/obsolete.txt")) throw new Error("injected stock file failure");
|
||||
return remove(file, options);
|
||||
}) : null;
|
||||
if (failure === "database") {
|
||||
await db.execute(sql.raw(`CREATE FUNCTION ${trigger}() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN
|
||||
IF NEW.company_id = '${companyId}' AND NEW.resource_kind = 'instructions' AND NEW.stock_hash <> OLD.stock_hash THEN
|
||||
RAISE EXCEPTION 'injected stock binding failure'; END IF; RETURN NEW; END $$`));
|
||||
await db.execute(sql.raw(`CREATE TRIGGER ${trigger} BEFORE UPDATE ON built_in_managed_resources FOR EACH ROW EXECUTE FUNCTION ${trigger}()`));
|
||||
}
|
||||
try { await expect(svc.ensure(companyId, "reflection-coach")).rejects.toThrow(); }
|
||||
finally {
|
||||
fault?.mockRestore();
|
||||
if (failure === "database") {
|
||||
await db.execute(sql.raw(`DROP TRIGGER ${trigger} ON built_in_managed_resources`));
|
||||
await db.execute(sql.raw(`DROP FUNCTION ${trigger}()`));
|
||||
}
|
||||
}
|
||||
const [pending] = await db.select().from(builtInManagedResources).where(and(eq(builtInManagedResources.companyId, companyId), eq(builtInManagedResources.resourceKind, "instructions")));
|
||||
expect(pending.defaultsJson.pendingInstructionsUpdate).toBeTruthy();
|
||||
expect(pending.stockHash).toBe(stockHash(oldFiles));
|
||||
const partial = (await agentInstructionsService().readFile(agent, "AGENTS.md")).content;
|
||||
expect(partial).toContain("You are Reflection Coach");
|
||||
// A real operator edit after the failed attempt must stop the retry.
|
||||
await writeInstructionEntry(agent, "AGENTS.md", "operator edit after failure");
|
||||
await svc.ensure(companyId, "reflection-coach");
|
||||
expect((await agentInstructionsService().readFile(agent, "AGENTS.md")).content).toBe("operator edit after failure");
|
||||
await writeInstructionEntry(agent, "AGENTS.md", partial);
|
||||
const retried = await svc.ensure(companyId, "reflection-coach");
|
||||
expect(retried.resources.find(resource => resource.resourceKind === "instructions")).toMatchObject({ stockStatus: "stock_current" });
|
||||
await expect(agentInstructionsService().readFile(agent, "obsolete.txt")).rejects.toMatchObject({ status: 404 });
|
||||
const [complete] = await db.select().from(builtInManagedResources).where(eq(builtInManagedResources.id, pending.id));
|
||||
expect(complete.defaultsJson.pendingInstructionsUpdate).toBeUndefined();
|
||||
});
|
||||
|
||||
it.each([false, true])("supersedes an interrupted stock version, including rollback=%s", async (rollback) => {
|
||||
const companyId = await seedCompany({ requireApproval: false });
|
||||
const svc = builtInAgentService(db);
|
||||
const created = await svc.ensure(companyId, "reflection-coach");
|
||||
const agent = created.agent!;
|
||||
const currentStock = (await agentInstructionsService().readFile(agent, "AGENTS.md")).content;
|
||||
const oldFiles = { "AGENTS.md": rollback ? currentStock : "# Original stock" };
|
||||
const interruptedFiles = { "AGENTS.md": "# Intermediate stock", "intermediate-only.txt": "intermediate support" };
|
||||
await writeInstructionEntry(agent, "AGENTS.md", interruptedFiles["AGENTS.md"]);
|
||||
await agentInstructionsService(db).writeFile(agent, "intermediate-only.txt", interruptedFiles["intermediate-only.txt"]);
|
||||
await agentInstructionsService(db).writeFile(agent, "personal.txt", "personal notes");
|
||||
await db.update(builtInManagedResources).set({ stockHash: stockHash(oldFiles), stockVersion: "old",
|
||||
defaultsJson: { entryFile: "AGENTS.md", files: Object.keys(oldFiles), pendingInstructionsUpdate: {
|
||||
stockHash: stockHash(interruptedFiles),
|
||||
baseHashes: { "AGENTS.md": fileHash(Buffer.from(oldFiles["AGENTS.md"])), "intermediate-only.txt": null },
|
||||
nextHashes: Object.fromEntries(Object.entries(interruptedFiles).map(([file, text]) => [file, fileHash(Buffer.from(text))])),
|
||||
} },
|
||||
}).where(and(eq(builtInManagedResources.companyId, companyId), eq(builtInManagedResources.resourceKind, "instructions")));
|
||||
await writeInstructionEntry(agent, "AGENTS.md", "intervening operator edit");
|
||||
await svc.ensure(companyId, "reflection-coach");
|
||||
expect((await agentInstructionsService().readFile(agent, "AGENTS.md")).content).toBe("intervening operator edit");
|
||||
await writeInstructionEntry(agent, "AGENTS.md", interruptedFiles["AGENTS.md"]);
|
||||
const updated = await svc.ensure(companyId, "reflection-coach");
|
||||
expect(updated.resources.find(resource => resource.resourceKind === "instructions")).toMatchObject({ stockStatus: "stock_current" });
|
||||
expect((await agentInstructionsService().readFile(agent, "AGENTS.md")).content).toBe(currentStock);
|
||||
await expect(agentInstructionsService().readFile(agent, "intermediate-only.txt")).rejects.toMatchObject({ status: 404 });
|
||||
expect((await agentInstructionsService().readFile(agent, "personal.txt")).content).toBe("personal notes");
|
||||
});
|
||||
|
||||
it("preserves Reflection Coach instruction drift on reconcile and restores it on reset", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const builtIns = builtInAgentService(db);
|
||||
const created = await builtIns.ensure(companyId, "reflection-coach");
|
||||
const instructions = agentInstructionsService();
|
||||
|
||||
await instructions.writeFile(created.agent!, "AGENTS.md", "# Custom Reflection Coach\n\nOperator edit.\n");
|
||||
await writeInstructionEntry(created.agent!, "AGENTS.md", "# Custom Reflection Coach\n\nOperator edit.\n");
|
||||
|
||||
const reconciled = await builtIns.ensure(companyId, "reflection-coach");
|
||||
const drift = reconciled.resources.find((resource) => resource.resourceKind === "instructions");
|
||||
@@ -815,7 +927,7 @@ describeEmbeddedPostgres("built-in agents", () => {
|
||||
content: "# Custom Reflection Coach\n\nOperator edit.\n",
|
||||
});
|
||||
|
||||
const reset = await builtIns.reset(companyId, "reflection-coach");
|
||||
const reset = await builtIns.reset(companyId, "reflection-coach", {}, instructionOperator);
|
||||
expect(reset.resources.find((resource) => resource.resourceKind === "instructions")).toMatchObject({
|
||||
stockStatus: "stock_current",
|
||||
resetAvailable: false,
|
||||
@@ -823,6 +935,12 @@ describeEmbeddedPostgres("built-in agents", () => {
|
||||
const resetFile = await instructions.readFile(reset.agent!, "AGENTS.md");
|
||||
expect(resetFile.content).toContain("Reflection Coach");
|
||||
expect(resetFile.content).not.toContain("Operator edit.");
|
||||
const revisions = agentInstructionRevisionService(db);
|
||||
const target = { companyId, agentId: created.agent!.id, entryFile: "AGENTS.md" };
|
||||
const history = await revisions.history(target, instructionOperator);
|
||||
expect(history.revisions).toHaveLength(0);
|
||||
const current = await revisions.readCurrent(target, instructionOperator);
|
||||
expect(current?.content).toBe(resetFile.content);
|
||||
});
|
||||
|
||||
it("blocks deleting a built-in agent", async () => {
|
||||
@@ -1270,7 +1388,7 @@ describeEmbeddedPostgres("built-in agents", () => {
|
||||
const created = await builtIns.ensure(companyId, "summarizer");
|
||||
const instructions = agentInstructionsService();
|
||||
|
||||
await instructions.writeFile(created.agent!, "AGENTS.md", "# Custom Summarizer\n\nOperator edit.\n");
|
||||
await writeInstructionEntry(created.agent!, "AGENTS.md", "# Custom Summarizer\n\nOperator edit.\n");
|
||||
|
||||
const reconciled = await builtIns.ensure(companyId, "summarizer");
|
||||
expect(reconciled.resources.find((resource) => resource.resourceKind === "instructions")).toMatchObject({
|
||||
@@ -1279,7 +1397,7 @@ describeEmbeddedPostgres("built-in agents", () => {
|
||||
changedFiles: ["AGENTS.md"],
|
||||
});
|
||||
|
||||
const reset = await builtIns.reset(companyId, "summarizer");
|
||||
const reset = await builtIns.reset(companyId, "summarizer", {}, instructionOperator);
|
||||
expect(reset.resources.find((resource) => resource.resourceKind === "instructions")).toMatchObject({
|
||||
stockStatus: "stock_current",
|
||||
resetAvailable: false,
|
||||
@@ -1419,7 +1537,7 @@ describeEmbeddedPostgres("built-in agents", () => {
|
||||
const coach = created.agent!;
|
||||
const instructionsSvc = agentInstructionsService();
|
||||
const originalInstructions = "# Target Coder\n\nWork from the assigned issue.\n";
|
||||
const prepared = await instructionsSvc.writeFile(target, "AGENTS.md", originalInstructions);
|
||||
const prepared = await writeInstructionEntry(target, "AGENTS.md", originalInstructions);
|
||||
let persistedTarget = (await agentsSvc.update(target.id, { adapterConfig: prepared.adapterConfig }))!;
|
||||
|
||||
const interactionsSvc = issueThreadInteractionService(db);
|
||||
@@ -1431,7 +1549,7 @@ describeEmbeddedPostgres("built-in agents", () => {
|
||||
if (interaction?.kind !== "request_confirmation" || interaction.status !== "accepted") {
|
||||
return false;
|
||||
}
|
||||
const written = await instructionsSvc.writeFile(persistedTarget, "AGENTS.md", input.nextInstructions);
|
||||
const written = await writeInstructionEntry(persistedTarget, "AGENTS.md", input.nextInstructions);
|
||||
persistedTarget = (await agentsSvc.update(persistedTarget.id, { adapterConfig: written.adapterConfig }))!;
|
||||
return true;
|
||||
};
|
||||
@@ -1568,7 +1686,7 @@ describeEmbeddedPostgres("built-in agents", () => {
|
||||
const agent = created.agent!;
|
||||
|
||||
const instructionsSvc = agentInstructionsService();
|
||||
await instructionsSvc.writeFile(agent, "AGENTS.md", "# Custom Reflection Coach\n\nDo not overwrite me.\n");
|
||||
await writeInstructionEntry(agent, "AGENTS.md", "# Custom Reflection Coach\n\nDo not overwrite me.\n");
|
||||
await db
|
||||
.update(companySkills)
|
||||
.set({ markdown: "---\nname: reflection-coach\n---\n\n# Custom skill\n" })
|
||||
@@ -1597,7 +1715,7 @@ describeEmbeddedPostgres("built-in agents", () => {
|
||||
|
||||
const reset = await builtInAgentService(db).reset(companyId, "reflection-coach", {
|
||||
resources: ["instructions", "routine"],
|
||||
});
|
||||
}, instructionOperator);
|
||||
expect(reset.resources.find((resource) => resource.resourceKind === "instructions")).toMatchObject({
|
||||
stockStatus: "stock_current",
|
||||
resetAvailable: false,
|
||||
|
||||
@@ -264,6 +264,12 @@ describeEmbeddedPostgres("heartbeat list", () => {
|
||||
summary: "completed",
|
||||
stdout: oversizedStdout,
|
||||
nestedHuge: { payload: oversizedNestedPayload },
|
||||
instructionSave: {
|
||||
state: "unavailable", contract: "agent_files", entryFile: "AGENTS.md",
|
||||
errorCode: "AGENT_FILES_LIMIT_EXCEEDED",
|
||||
storageWarning: "Agent storage is full. Runs can continue.".repeat(50),
|
||||
privateSyncMetadata: oversizedNestedPayload,
|
||||
},
|
||||
workspaceRestoreFailure: "restore_unsafe_archive",
|
||||
finalResponseRecorded: true,
|
||||
executionBeforeRestore: { errorCode: "model_error", exitCode: 2, timedOut: false },
|
||||
@@ -278,6 +284,11 @@ describeEmbeddedPostgres("heartbeat list", () => {
|
||||
truncated: true,
|
||||
truncationReason: "oversized_result_json",
|
||||
stdoutTruncated: true,
|
||||
instructionSave: {
|
||||
state: "unavailable", contract: "agent_files", entryFile: "AGENTS.md",
|
||||
errorCode: "AGENT_FILES_LIMIT_EXCEEDED",
|
||||
storageWarning: "Agent storage is full. Runs can continue.".repeat(50).slice(0, 1024),
|
||||
},
|
||||
workspaceRestoreFailure: "restore_unsafe_archive",
|
||||
finalResponseRecorded: true,
|
||||
executionBeforeRestore: { errorCode: "model_error", exitCode: 2, timedOut: false },
|
||||
@@ -285,6 +296,7 @@ describeEmbeddedPostgres("heartbeat list", () => {
|
||||
expect(typeof result?.stdout).toBe("string");
|
||||
expect((result?.stdout as string).length).toBeLessThan(oversizedStdout.length);
|
||||
expect(result).not.toHaveProperty("nestedHuge");
|
||||
expect(result?.instructionSave).not.toHaveProperty("privateSyncMetadata");
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -7,6 +7,8 @@ import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
|
||||
import {
|
||||
activityLog,
|
||||
agentConfigRevisions,
|
||||
agentInstructionRevisions,
|
||||
agentInstructionHeads,
|
||||
agents,
|
||||
approvals,
|
||||
companies,
|
||||
@@ -23,6 +25,9 @@ import {
|
||||
} from "./helpers/embedded-postgres.js";
|
||||
import { buildHostServices } from "../services/plugin-host-services.js";
|
||||
import { agentService } from "../services/agents.js";
|
||||
import { agentInstructionRevisionService } from "../services/agent-instruction-revisions.js";
|
||||
import { agentInstructionsService } from "../services/agent-instructions.js";
|
||||
import { pluginRegistryService } from "../services/plugin-registry.js";
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
@@ -312,7 +317,7 @@ describeEmbeddedPostgres("plugin-managed agents", () => {
|
||||
it("materializes declared managed agent instructions with local folder paths", async () => {
|
||||
const previousHome = process.env.PAPERCLIP_HOME;
|
||||
const previousInstance = process.env.PAPERCLIP_INSTANCE_ID;
|
||||
const tempHome = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-managed-agent-home-"));
|
||||
const tempHome = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-managed-agent-home-")));
|
||||
const wikiRoot = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-managed-agent-wiki-")));
|
||||
process.env.PAPERCLIP_HOME = tempHome;
|
||||
process.env.PAPERCLIP_INSTANCE_ID = "test";
|
||||
@@ -369,6 +374,39 @@ describeEmbeddedPostgres("plugin-managed agents", () => {
|
||||
expect(content).toContain("You are the LLM Wiki Maintainer.");
|
||||
expect(content).toContain(`Wiki root: \`${wikiRoot}\``);
|
||||
expect(content).toContain(`Wiki schema: \`${path.join(wikiRoot, "AGENTS.md")}\``);
|
||||
|
||||
// A managed plugin reset must preserve the previous entry in canonical
|
||||
// history instead of deleting a shared bundle or bypassing its write guard.
|
||||
const edited = "# User-customized wiki instructions\n";
|
||||
await fs.writeFile(instructionsFilePath as string, edited);
|
||||
const reset = await services.agents.managedReset({ companyId, agentKey: "wiki-maintainer" });
|
||||
expect(reset.status).toBe("reset");
|
||||
expect(await fs.readFile(instructionsFilePath as string, "utf8")).toBe(content);
|
||||
const history = await db.select().from(agentInstructionRevisions)
|
||||
.where(eq(agentInstructionRevisions.agentId, created.agentId!));
|
||||
expect(history).toHaveLength(0);
|
||||
expect(await db.select().from(agentInstructionHeads)
|
||||
.where(eq(agentInstructionHeads.agentId, created.agentId!))).toHaveLength(0);
|
||||
const repeated = await services.agents.managedReset({ companyId, agentKey: "wiki-maintainer" });
|
||||
expect(repeated.status).toBe("reset");
|
||||
expect(await db.select().from(agentInstructionRevisions)
|
||||
.where(eq(agentInstructionRevisions.agentId, created.agentId!))).toHaveLength(0);
|
||||
const audit = await db.select().from(activityLog).where(eq(activityLog.action, "agent.files_updated"));
|
||||
expect(audit).toHaveLength(1);
|
||||
expect(audit.every((row) => row.actorType === "plugin" && row.actorId === pluginId)).toBe(true);
|
||||
const bundles = agentInstructionsService(db);
|
||||
await bundles.writeFile(repeated.agent!, "CUSTOM.md", "# Alternate configured entry\n");
|
||||
const switched = await bundles.updateBundle(repeated.agent!, { entryFile: "CUSTOM.md" });
|
||||
await agentService(db).update(created.agentId!, { adapterConfig: switched.adapterConfig });
|
||||
const resetEntry = await services.agents.managedReset({ companyId, agentKey: "wiki-maintainer" });
|
||||
expect(resetEntry.agent?.adapterConfig.instructionsEntryFile).toBe("AGENTS.md");
|
||||
expect(resetEntry.defaultDrift).toBeNull();
|
||||
expect(await fs.readFile(instructionsFilePath as string, "utf8")).toBe(content);
|
||||
const preserved = await db.select().from(agentInstructionRevisions).where(eq(agentInstructionRevisions.agentId, created.agentId!));
|
||||
expect(preserved).toHaveLength(0);
|
||||
expect(await fs.readFile(path.join(path.dirname(instructionsFilePath as string), "CUSTOM.md"), "utf8")).toBe("# Alternate configured entry\n");
|
||||
|
||||
|
||||
} finally {
|
||||
if (previousHome === undefined) delete process.env.PAPERCLIP_HOME;
|
||||
else process.env.PAPERCLIP_HOME = previousHome;
|
||||
@@ -379,6 +417,60 @@ describeEmbeddedPostgres("plugin-managed agents", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("fences plugin instruction reset by exact ownership, current capability and canonical CAS", async () => {
|
||||
const previousHome = process.env.PAPERCLIP_HOME;
|
||||
const tempHome = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "plugin-reset-cas-")));
|
||||
process.env.PAPERCLIP_HOME = tempHome;
|
||||
try {
|
||||
const pluginManifest = manifest();
|
||||
pluginManifest.agents![0]!.instructions = { content: "# Default\n" };
|
||||
const { companyId, pluginId, services } = await seedCompanyAndPlugin({ manifest: pluginManifest });
|
||||
const created = await services.agents.managedReconcile({ companyId, agentKey: "wiki-maintainer" });
|
||||
const target = { companyId, agentId: created.agentId! };
|
||||
const actor = { type: "plugin" as const, pluginId, pluginKey: pluginManifest.id, agentKey: "wiki-maintainer" };
|
||||
const revisions = agentInstructionRevisionService(db);
|
||||
const baselineRead = await revisions.readForPluginReset(target, actor);
|
||||
const baseline = baselineRead.snapshot!;
|
||||
const input = { ...target, entryFile: "AGENTS.md", baseRevisionId: baseline.revision.id,
|
||||
configuredEntryFile: baselineRead.configuredEntryFile, content: "# Reset\n" };
|
||||
await expect(revisions.commitPluginReset(input, { ...actor, pluginId: randomUUID() })).rejects.toMatchObject({ status: 403 });
|
||||
await expect(revisions.commitPluginReset({ ...input, companyId: randomUUID() }, actor)).rejects.toMatchObject({ status: 404 });
|
||||
await expect(revisions.commitPluginReset(input, { ...actor, agentKey: "foreign-agent" })).rejects.toMatchObject({ status: 403 });
|
||||
await db.update(plugins).set({ manifestJson: { ...pluginManifest, capabilities: [] } }).where(eq(plugins.id, pluginId));
|
||||
await expect(revisions.commitPluginReset(input, actor)).rejects.toMatchObject({ status: 403 });
|
||||
await db.update(plugins).set({ manifestJson: pluginManifest }).where(eq(plugins.id, pluginId));
|
||||
const originalMetadata = created.agent!.metadata;
|
||||
await db.update(agents).set({ metadata: {} }).where(eq(agents.id, target.agentId));
|
||||
await expect(revisions.commitPluginReset(input, actor)).rejects.toMatchObject({ status: 403 });
|
||||
await db.update(agents).set({ metadata: originalMetadata }).where(eq(agents.id, target.agentId));
|
||||
const [binding] = await db.select().from(pluginManagedResources).where(eq(pluginManagedResources.resourceId, target.agentId));
|
||||
await db.update(pluginManagedResources).set({ resourceId: randomUUID() }).where(eq(pluginManagedResources.id, binding.id));
|
||||
await expect(revisions.commitPluginReset(input, actor)).rejects.toMatchObject({ status: 403 });
|
||||
await db.update(pluginManagedResources).set({ resourceId: target.agentId }).where(eq(pluginManagedResources.id, binding.id));
|
||||
await expect(revisions.commitPluginReset({ ...input, entryFile: "OTHER.md" }, actor)).rejects.toMatchObject({ status: 403 });
|
||||
const bundles = agentInstructionsService(db);
|
||||
await bundles.writeFile(created.agent!, "OTHER.md", "# Other entry\n");
|
||||
const switched = await bundles.updateBundle(created.agent!, { entryFile: "OTHER.md" });
|
||||
await agentService(db).update(target.agentId, { adapterConfig: switched.adapterConfig });
|
||||
await expect(revisions.commitPluginReset(input, actor)).rejects.toMatchObject({ status: 409, details: { code: "INSTRUCTION_ENTRY_CHANGED" } });
|
||||
await agentService(db).update(target.agentId, { adapterConfig: created.agent!.adapterConfig });
|
||||
const results = await Promise.allSettled([
|
||||
revisions.commitPluginReset(input, actor),
|
||||
revisions.commitPluginReset({ ...input, content: "# Concurrent reset\n" }, actor),
|
||||
]);
|
||||
expect(results.filter((result) => result.status === "fulfilled")).toHaveLength(1);
|
||||
const loser = results.find((result) => result.status === "rejected") as PromiseRejectedResult;
|
||||
expect(loser.reason).toMatchObject({ status: 409, details: { code: "INSTRUCTION_REVISION_CONFLICT" } });
|
||||
const winner = results.find((result) => result.status === "fulfilled") as PromiseFulfilledResult<Awaited<ReturnType<typeof revisions.commitPluginReset>>>;
|
||||
expect(await fs.readFile(created.agent!.adapterConfig.instructionsFilePath as string, "utf8")).toBe(winner.value.content);
|
||||
const history = await db.select().from(agentInstructionRevisions).where(eq(agentInstructionRevisions.agentId, target.agentId));
|
||||
expect(history).toHaveLength(0);
|
||||
} finally {
|
||||
if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome;
|
||||
await fs.rm(tempHome, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("repairs a missing binding by relinking a same-company managed agent marker", async () => {
|
||||
const { companyId, pluginId, pluginManifest, services } = await seedCompanyAndPlugin();
|
||||
const agentId = randomUUID();
|
||||
@@ -410,6 +502,76 @@ describeEmbeddedPostgres("plugin-managed agents", () => {
|
||||
expect(binding?.data).toMatchObject({ agentId });
|
||||
});
|
||||
|
||||
it("preserves current-file authority when relinking after a hard uninstall and reinstall", async () => {
|
||||
const previousHome = process.env.PAPERCLIP_HOME;
|
||||
const tempHome = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "plugin-reinstall-reset-")));
|
||||
process.env.PAPERCLIP_HOME = tempHome;
|
||||
try {
|
||||
const pluginManifest = manifest();
|
||||
pluginManifest.agents![0]!.instructions = { content: "# Original stock\n" };
|
||||
const { companyId, pluginId, services } = await seedCompanyAndPlugin({ manifest: pluginManifest });
|
||||
const created = await services.agents.managedReconcile({ companyId, agentKey: "wiki-maintainer" });
|
||||
const target = { companyId, agentId: created.agentId! };
|
||||
const revisions = agentInstructionRevisionService(db);
|
||||
const original = await revisions.readForPluginReset(target, {
|
||||
type: "plugin", pluginId, pluginKey: pluginManifest.id, agentKey: "wiki-maintainer",
|
||||
});
|
||||
const registry = pluginRegistryService(db);
|
||||
await registry.uninstall(pluginId, true);
|
||||
expect(await db.select().from(pluginManagedResources)).toHaveLength(0);
|
||||
expect(await db.select().from(pluginEntities)).toHaveLength(0);
|
||||
const nextManifest = structuredClone(pluginManifest);
|
||||
nextManifest.agents![0]!.instructions = { content: "# Reinstalled stock\n" };
|
||||
const installed = await registry.install({ packageName: "@paperclipai/plugin-managed-agents-test" }, nextManifest);
|
||||
expect(installed!.id).not.toBe(pluginId);
|
||||
await registry.updateStatus(installed!.id, { status: "ready" });
|
||||
const reinstalledServices = buildHostServices(db, installed!.id, nextManifest.id, createEventBusStub(), undefined, {
|
||||
manifest: nextManifest,
|
||||
});
|
||||
const relinked = await reinstalledServices.agents.managedReconcile({ companyId, agentKey: "wiki-maintainer" });
|
||||
expect(relinked.status).toBe("relinked");
|
||||
expect(relinked.agentId).toBe(created.agentId);
|
||||
const reset = await reinstalledServices.agents.managedReset({ companyId, agentKey: "wiki-maintainer" });
|
||||
expect(reset.agentId).toBe(created.agentId);
|
||||
expect(reset.agent!.metadata).toMatchObject({ paperclipManagedResource: { pluginId: installed!.id }, pluginManagedAgent: { pluginId: installed!.id } });
|
||||
expect(await fs.readFile(reset.agent!.adapterConfig.instructionsFilePath as string, "utf8")).toBe("# Reinstalled stock\n");
|
||||
const history = await db.select().from(agentInstructionRevisions).where(eq(agentInstructionRevisions.agentId, created.agentId!));
|
||||
expect(history).toHaveLength(0);
|
||||
} finally {
|
||||
if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome;
|
||||
await fs.rm(tempHome, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["not-ready", "capability-removed", "declaration-removed", "other-plugin-owner"])(
|
||||
"does not relink a managed agent when its authority is %s", async (reason) => {
|
||||
const { companyId, pluginId, pluginManifest, services } = await seedCompanyAndPlugin();
|
||||
const created = await services.agents.managedReconcile({ companyId, agentKey: "wiki-maintainer" });
|
||||
await db.delete(pluginEntities);
|
||||
await db.delete(pluginManagedResources);
|
||||
if (reason === "not-ready") {
|
||||
await db.update(plugins).set({ status: "disabled" }).where(eq(plugins.id, pluginId));
|
||||
} else if (reason === "capability-removed") {
|
||||
await db.update(plugins).set({ manifestJson: { ...pluginManifest, capabilities: [] } }).where(eq(plugins.id, pluginId));
|
||||
} else if (reason === "declaration-removed") {
|
||||
await db.update(plugins).set({ manifestJson: { ...pluginManifest, agents: [] } }).where(eq(plugins.id, pluginId));
|
||||
} else {
|
||||
const otherId = randomUUID();
|
||||
await db.insert(plugins).values({ id: otherId, pluginKey: "paperclip.other-owner", packageName: "other",
|
||||
version: "0.1.0", apiVersion: 1, categories: [], manifestJson: { ...pluginManifest, id: "paperclip.other-owner" }, status: "ready", installOrder: 2 });
|
||||
await db.update(agents).set({ metadata: { ...created.agent!.metadata,
|
||||
paperclipManagedResource: { pluginId: otherId, pluginKey: pluginManifest.id, resourceKind: "agent", resourceKey: "wiki-maintainer" },
|
||||
} }).where(eq(agents.id, created.agentId!));
|
||||
}
|
||||
const [before] = await db.select().from(agents).where(eq(agents.id, created.agentId!));
|
||||
await expect(services.agents.managedReconcile({ companyId, agentKey: "wiki-maintainer" })).rejects.toMatchObject({ status: 403 });
|
||||
expect(await db.select().from(pluginEntities)).toHaveLength(0);
|
||||
expect(await db.select().from(pluginManagedResources)).toHaveLength(0);
|
||||
const [after] = await db.select().from(agents).where(eq(agents.id, created.agentId!));
|
||||
expect(after.metadata).toEqual(before.metadata);
|
||||
},
|
||||
);
|
||||
|
||||
it("respects board approval policy for new managed agents", async () => {
|
||||
const { companyId, services } = await seedCompanyAndPlugin({ requireApproval: true });
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ import {
|
||||
writeEnvFileAtomicallyIfChanged,
|
||||
} from "@paperclipai/shared/env-file";
|
||||
import { resourceStatus, stockHash } from "../services/managed-resource-drift.js";
|
||||
import { agentInstructionsService } from "../services/agent-instructions.js";
|
||||
import { agentInstructionsService, resolveManagedInstructionsRoot } from "../services/agent-instructions.js";
|
||||
|
||||
/**
|
||||
* Cross-cutting setup/sync rerun survival test.
|
||||
@@ -23,7 +23,7 @@ import { agentInstructionsService } from "../services/agent-instructions.js";
|
||||
* - .env -> updateEnvFileContents + writeEnvFileAtomicallyIfChanged
|
||||
* - managed sandbox env -> resourceStatus/stockHash drift gate (the `shouldWrite`
|
||||
* classifier the boot reconciler and built-in sync share)
|
||||
* - managed instructions -> agentInstructionsService materialize + the drift gate that
|
||||
* - managed instructions -> agentInstructionsService export + the drift gate that
|
||||
* makes built-in/plugin/portability rebuilds skip operator edits
|
||||
*
|
||||
* The DB-backed end-to-end variants of the sandbox-row, skills-assignment, and
|
||||
@@ -31,7 +31,7 @@ import { agentInstructionsService } from "../services/agent-instructions.js";
|
||||
* server/src/__tests__/company-skills-service.test.ts (skills add/remove/replace)
|
||||
* server/src/services/managed-environments.test.ts (sandbox row boot reconcile)
|
||||
* server/src/__tests__/environment-service.test.ts (sandbox skip + stock update)
|
||||
* server/src/__tests__/agent-instructions-service.test.ts (managed instruction drift)
|
||||
* server/src/__tests__/agent-instruction-revisions.test.ts (guarded bundle materialization)
|
||||
*/
|
||||
|
||||
const cleanupDirs = new Set<string>();
|
||||
@@ -147,7 +147,7 @@ describe("setup/sync rerun survival — cross-cutting", () => {
|
||||
).toBe("stock_current");
|
||||
});
|
||||
|
||||
it("keeps operator edits and additions in a managed instructions tree across a re-materialize", async () => {
|
||||
it("keeps operator edits and additions when managed instructions reconciliation detects drift", async () => {
|
||||
const home = await tmp("pap16587-instr-home-");
|
||||
process.env.PAPERCLIP_HOME = home;
|
||||
process.env.PAPERCLIP_INSTANCE_ID = "test-instance";
|
||||
@@ -156,18 +156,25 @@ describe("setup/sync rerun survival — cross-cutting", () => {
|
||||
const agent = { id: "agent-1", companyId: "company-1", name: "Agent 1", adapterConfig: {} };
|
||||
const stockFiles = { "AGENTS.md": "# stock\n", "docs/TOOLS.md": "## stock tools\n" };
|
||||
|
||||
const first = await svc.materializeManagedBundle(agent, stockFiles, { entryFile: "AGENTS.md" });
|
||||
const root = first.bundle.managedRootPath!;
|
||||
// Initial files are fixture setup. Bundle materialization now requires a
|
||||
// real database so that it cannot bypass a canonical instruction revision.
|
||||
const root = resolveManagedInstructionsRoot(agent);
|
||||
for (const [name, content] of Object.entries(stockFiles)) {
|
||||
await fs.mkdir(path.dirname(path.join(root, name)), { recursive: true });
|
||||
await fs.writeFile(path.join(root, name), content, "utf8");
|
||||
}
|
||||
const managedAgent = { ...agent, adapterConfig: {
|
||||
instructionsBundleMode: "managed", instructionsRootPath: root,
|
||||
instructionsEntryFile: "AGENTS.md", instructionsFilePath: path.join(root, "AGENTS.md"),
|
||||
} };
|
||||
|
||||
// Operator hand-edits a managed file and adds a brand-new operator-only file.
|
||||
await fs.writeFile(path.join(root, "AGENTS.md"), "# stock\n\noperator note\n", "utf8");
|
||||
await fs.writeFile(path.join(root, "OPERATOR.md"), "operator-added\n", "utf8");
|
||||
|
||||
// The drift gate the built-in/plugin/portability rebuilds share: classify the tree.
|
||||
const currentFiles = {
|
||||
"AGENTS.md": await fs.readFile(path.join(root, "AGENTS.md"), "utf8"),
|
||||
"docs/TOOLS.md": await fs.readFile(path.join(root, "docs", "TOOLS.md"), "utf8"),
|
||||
};
|
||||
const currentFiles = (await svc.exportFiles(managedAgent)).files;
|
||||
expect(currentFiles["OPERATOR.md"]).toBe("operator-added\n");
|
||||
const status = resourceStatus({
|
||||
resourceId: agent.id,
|
||||
currentHash: stockHash(currentFiles),
|
||||
|
||||
+154
-4
@@ -1,3 +1,5 @@
|
||||
import { agentFileStore, agentFileTokenFromHash } from "../services/agent-file-store.js";
|
||||
import { pipeline } from "node:stream/promises";
|
||||
import { resolveAgentAppearance, agentAvatarUrl } from "@paperclipai/shared";
|
||||
import { listOpenRouterModels } from "../services/openrouter-models.js";
|
||||
import { prepareManagedAiRuntime, assertManagedAiProjectAuth, stripAiAuthBindings } from "../services/ai-connection-runtime.js";
|
||||
@@ -38,6 +40,8 @@ import {
|
||||
type AgentSkillSnapshot,
|
||||
type InstanceSchedulerHeartbeatAgent,
|
||||
upsertAgentInstructionsFileSchema,
|
||||
restoreAgentInstructionSchema,
|
||||
resolveAgentInstructionCandidateSchema,
|
||||
updateAgentInstructionsBundleSchema,
|
||||
updateAgentPermissionsSchema,
|
||||
updateAgentInstructionsPathSchema,
|
||||
@@ -63,7 +67,11 @@ import {
|
||||
import { trackAgentCreated } from "@paperclipai/shared/telemetry";
|
||||
import { validate } from "../middleware/validate.js";
|
||||
import { inheritNativeRunnerAdapterConfig } from "../services/native-runtime/native-agent-runtime-inheritance.js";
|
||||
import { agentInstructionsBundleMode } from "../services/agent-instructions.js";
|
||||
import { agentInstructionRevisionService } from "../services/agent-instruction-revisions.js";
|
||||
import { agentInstructionWorkingCopyService } from "../services/agent-instruction-working-copies.js";
|
||||
import { authorizeInstructionRead } from "../services/agent-instruction-authorization.js";
|
||||
import { instructionPath } from "../services/agent-instruction-files.js";
|
||||
import { agentInstructionsBundleMode, deriveBundleState } from "../services/agent-instructions.js";
|
||||
import {
|
||||
agentService,
|
||||
agentInstructionsService,
|
||||
@@ -724,7 +732,17 @@ export function agentRoutes(
|
||||
const recovery = recoveryService(db, { enqueueWakeup: heartbeat.wakeup });
|
||||
const issueApprovalsSvc = issueApprovalService(db);
|
||||
const secretsSvc = secretService(db);
|
||||
const instructions = agentInstructionsService();
|
||||
const instructions = agentInstructionsService(db);
|
||||
const agentFiles = agentFileStore(db);
|
||||
const instructionRevisions = agentInstructionRevisionService(db);
|
||||
const instructionWorkingCopies = agentInstructionWorkingCopyService(db);
|
||||
function instructionFileDetail(snapshot: import("@paperclipai/shared").AgentInstructionSnapshot,
|
||||
receipt?: import("@paperclipai/shared").AgentInstructionCommitReceipt) {
|
||||
const path = snapshot.revision.entryFile;
|
||||
return { path, content: snapshot.content, contentHash: snapshot.revision.contentHash, size: snapshot.revision.byteLength, revision: snapshot.revision, receipt,
|
||||
language: path.toLowerCase().endsWith(".md") ? "markdown" : "text", markdown: path.toLowerCase().endsWith(".md"),
|
||||
isEntryFile: true, editable: true, deprecated: false, virtual: false };
|
||||
}
|
||||
const companySkills = companySkillService(db);
|
||||
const workspaceOperations = workspaceOperationService(db);
|
||||
const instanceSettings = instanceSettingsService(db);
|
||||
@@ -5067,8 +5085,24 @@ export function agentRoutes(
|
||||
const id = req.params.id as string;
|
||||
const existing = await getAccessibleResource(req, res, svc.getById(id), "Agent not found");
|
||||
if (!existing) return;
|
||||
await assertCanReadAgent(req, existing);
|
||||
assertExternalInstructionsAdmin(req, existing);
|
||||
if (agentInstructionsBundleMode(existing) === "external") {
|
||||
await assertCanReadAgent(req, existing);
|
||||
} else {
|
||||
await authorizeInstructionRead(db, req.actor, { companyId: existing.companyId, id: existing.id });
|
||||
}
|
||||
if (agentInstructionsBundleMode(existing) !== "external") {
|
||||
const target = { companyId: existing.companyId, agentId: existing.id };
|
||||
const current = await instructionRevisions.readCurrent(target, req.actor);
|
||||
if (current) {
|
||||
try { await instructionRevisions.materializeCurrent(target); }
|
||||
catch (error) {
|
||||
const bundle = await instructions.getBundle(existing);
|
||||
bundle.warnings.push(`Saved instruction revision needs materialization: ${error instanceof Error ? error.message : String(error)}`);
|
||||
res.json(bundle); return;
|
||||
}
|
||||
}
|
||||
}
|
||||
res.json(await instructions.getBundle(existing));
|
||||
});
|
||||
|
||||
@@ -5124,8 +5158,12 @@ export function agentRoutes(
|
||||
const id = req.params.id as string;
|
||||
const existing = await getAccessibleResource(req, res, svc.getById(id), "Agent not found");
|
||||
if (!existing) return;
|
||||
await assertCanReadAgent(req, existing);
|
||||
assertExternalInstructionsAdmin(req, existing);
|
||||
if (agentInstructionsBundleMode(existing) === "external") {
|
||||
await assertCanReadAgent(req, existing);
|
||||
} else {
|
||||
await authorizeInstructionRead(db, req.actor, { companyId: existing.companyId, id: existing.id });
|
||||
}
|
||||
|
||||
const relativePath = typeof req.query.path === "string" ? req.query.path : "";
|
||||
if (!relativePath.trim()) {
|
||||
@@ -5133,6 +5171,19 @@ export function agentRoutes(
|
||||
return;
|
||||
}
|
||||
|
||||
if (req.query.download === "true" && agentInstructionsBundleMode(existing) === "managed") {
|
||||
const download = await agentFiles.download(existing.companyId, existing.id, relativePath, req.actor);
|
||||
if (download === null) throw notFound("Agent file not found");
|
||||
res.setHeader("Content-Type", "application/octet-stream");
|
||||
res.setHeader("X-Content-Type-Options", "nosniff");
|
||||
res.attachment(relativePath.split("/").at(-1)!);
|
||||
res.setHeader("Content-Length", download.size);
|
||||
await pipeline(download.stream, res); return;
|
||||
}
|
||||
if (agentInstructionsBundleMode(existing) !== "external" && instructionPath(relativePath) === deriveBundleState(existing).entryFile) {
|
||||
const snapshot = await instructionRevisions.readCurrent({ companyId: existing.companyId, agentId: existing.id }, req.actor);
|
||||
if (snapshot) { res.json(instructionFileDetail(snapshot)); return; }
|
||||
}
|
||||
res.json(await instructions.readFile(existing, relativePath));
|
||||
});
|
||||
|
||||
@@ -5140,9 +5191,42 @@ export function agentRoutes(
|
||||
const id = req.params.id as string;
|
||||
const existing = await getAccessibleResource(req, res, svc.getById(id), "Agent not found");
|
||||
if (!existing) return;
|
||||
const entryFile = deriveBundleState(existing).entryFile;
|
||||
if (instructionPath(req.body.path) === entryFile) {
|
||||
assertExternalInstructionsAdmin(req, existing);
|
||||
if (req.body.baseHash !== undefined) req.body.baseRevisionId = req.body.baseHash === null ? null : agentFileTokenFromHash(req.body.baseHash);
|
||||
if (req.body.baseRevisionId === undefined) throw unprocessable("Read the entry and supply baseRevisionId (null for a new entry)", { code: "INSTRUCTION_BASE_REQUIRED" });
|
||||
// Clearing legacy prompt configuration remains a protected config change.
|
||||
if (req.body.clearLegacyPromptTemplate) await assertCanManageInstructionsPath(req, existing);
|
||||
const receipt = await instructionRevisions.commit({ companyId: existing.companyId, agentId: existing.id,
|
||||
entryFile, content: req.body.content, baseRevisionId: req.body.baseRevisionId,
|
||||
source: req.actor.type === "board" ? "board" : "api" }, req.actor);
|
||||
if (req.actor.type === "agent" && req.actor.runId) {
|
||||
await instructionWorkingCopies.acknowledgeExplicitSave({ companyId: existing.companyId, agentId: existing.id,
|
||||
runId: req.actor.runId, entryFile: receipt.revision.entryFile, revisionId: receipt.revision.id,
|
||||
contentHash: receipt.revision.contentHash }).catch(() => undefined);
|
||||
}
|
||||
if (req.body.clearLegacyPromptTemplate) {
|
||||
const fresh = await svc.getById(existing.id);
|
||||
if (fresh) {
|
||||
const adapterConfig = { ...asRecord(fresh.adapterConfig) };
|
||||
delete adapterConfig.promptTemplate;
|
||||
delete adapterConfig.bootstrapPromptTemplate;
|
||||
await svc.update(existing.id, { adapterConfig });
|
||||
}
|
||||
}
|
||||
res.json(instructionFileDetail(receipt, receipt));
|
||||
return;
|
||||
}
|
||||
await assertCanManageInstructionsPath(req, existing);
|
||||
assertExternalInstructionsAdmin(req, existing);
|
||||
|
||||
if (agentInstructionsBundleMode(existing) === "managed" && req.body.path !== "promptTemplate.legacy.md") {
|
||||
if (req.body.baseHash === undefined) throw unprocessable("Read the file and supply baseHash (null for a new file)");
|
||||
await agentFiles.write({ companyId: existing.companyId, agentId: existing.id, path: req.body.path,
|
||||
bytes: Buffer.from(req.body.content, "utf8"), baseHash: req.body.baseHash }, req.actor);
|
||||
res.json(await instructions.readFile(existing, req.body.path)); return;
|
||||
}
|
||||
const actor = getActorInfo(req);
|
||||
const result = await instructions.writeFile(existing, req.body.path, req.body.content, {
|
||||
clearLegacyPromptTemplate: req.body.clearLegacyPromptTemplate,
|
||||
@@ -5184,6 +5268,66 @@ export function agentRoutes(
|
||||
res.json(result.file);
|
||||
});
|
||||
|
||||
router.get("/agents/:id/instructions-bundle/candidates", async (req, res) => {
|
||||
const existing = await getAccessibleResource(req, res, svc.getById(req.params.id as string), "Agent not found");
|
||||
if (!existing) return;
|
||||
assertExternalInstructionsAdmin(req, existing);
|
||||
res.json(await instructionWorkingCopies.list(existing.companyId, existing.id, req.actor));
|
||||
});
|
||||
|
||||
router.post("/agents/:id/instructions-bundle/candidates/:runId/resolve", validate(resolveAgentInstructionCandidateSchema), async (req, res) => {
|
||||
const existing = await getAccessibleResource(req, res, svc.getById(req.params.id as string), "Agent not found");
|
||||
if (!existing) return;
|
||||
assertExternalInstructionsAdmin(req, existing);
|
||||
const runId = req.params.runId as string;
|
||||
if (!isUuidLike(runId)) throw unprocessable("Invalid instruction candidate run id");
|
||||
const receipt = await instructionWorkingCopies.resolve({ companyId: existing.companyId, agentId: existing.id,
|
||||
runId, baseRevisionId: req.body.baseRevisionId, content: req.body.content }, req.actor);
|
||||
res.json(instructionFileDetail(receipt, receipt));
|
||||
});
|
||||
|
||||
router.get("/agents/:id/instructions-bundle/history", async (req, res) => {
|
||||
const existing = await getAccessibleResource(req, res, svc.getById(req.params.id as string), "Agent not found");
|
||||
if (!existing) return;
|
||||
assertExternalInstructionsAdmin(req, existing);
|
||||
const entryFile = typeof req.query.path === "string" ? req.query.path : deriveBundleState(existing).entryFile;
|
||||
const cursor = typeof req.query.cursor === "string" ? req.query.cursor : undefined;
|
||||
if (cursor && !isUuidLike(cursor)) throw unprocessable("Invalid history cursor");
|
||||
res.json(await instructionRevisions.history({ companyId: existing.companyId, agentId: existing.id, entryFile, cursor }, req.actor));
|
||||
});
|
||||
router.get("/agents/:id/instructions-bundle/revision/:revisionId", async (req, res) => {
|
||||
const existing = await getAccessibleResource(req, res, svc.getById(req.params.id as string), "Agent not found");
|
||||
if (!existing) return;
|
||||
assertExternalInstructionsAdmin(req, existing);
|
||||
const revisionId = req.params.revisionId as string;
|
||||
if (!isUuidLike(revisionId)) throw unprocessable("Invalid instruction revision id");
|
||||
const entryFile = typeof req.query.path === "string" ? req.query.path : deriveBundleState(existing).entryFile;
|
||||
res.json(await instructionRevisions.readRevision({ companyId: existing.companyId, agentId: existing.id, entryFile, revisionId }, req.actor));
|
||||
});
|
||||
router.get("/agents/:id/instructions-bundle/diff", async (req, res) => {
|
||||
const existing = await getAccessibleResource(req, res, svc.getById(req.params.id as string), "Agent not found");
|
||||
if (!existing) return;
|
||||
assertExternalInstructionsAdmin(req, existing);
|
||||
const fromRevisionId = typeof req.query.from === "string" ? req.query.from : "";
|
||||
const toRevisionId = typeof req.query.to === "string" ? req.query.to : "";
|
||||
if (!isUuidLike(fromRevisionId) || !isUuidLike(toRevisionId)) throw unprocessable("Provide valid from and to revision ids");
|
||||
const entryFile = typeof req.query.path === "string" ? req.query.path : deriveBundleState(existing).entryFile;
|
||||
res.json(await instructionRevisions.diff({ companyId: existing.companyId, agentId: existing.id, entryFile, fromRevisionId, toRevisionId }, req.actor));
|
||||
});
|
||||
router.post("/agents/:id/instructions-bundle/restore", validate(restoreAgentInstructionSchema), async (req, res) => {
|
||||
const existing = await getAccessibleResource(req, res, svc.getById(req.params.id as string), "Agent not found");
|
||||
if (!existing) return;
|
||||
assertExternalInstructionsAdmin(req, existing);
|
||||
const receipt = await instructionRevisions.restore({ companyId: existing.companyId, agentId: existing.id,
|
||||
entryFile: req.body.path, baseRevisionId: req.body.baseRevisionId, revisionId: req.body.revisionId }, req.actor);
|
||||
if (req.actor.type === "agent" && req.actor.runId) {
|
||||
await instructionWorkingCopies.acknowledgeExplicitSave({ companyId: existing.companyId, agentId: existing.id,
|
||||
runId: req.actor.runId, entryFile: receipt.revision.entryFile, revisionId: receipt.revision.id,
|
||||
contentHash: receipt.revision.contentHash }).catch(() => undefined);
|
||||
}
|
||||
res.json(instructionFileDetail(receipt, receipt));
|
||||
});
|
||||
|
||||
router.delete("/agents/:id/instructions-bundle/file", async (req, res) => {
|
||||
const id = req.params.id as string;
|
||||
const existing = await getAccessibleResource(req, res, svc.getById(id), "Agent not found");
|
||||
@@ -5197,6 +5341,12 @@ export function agentRoutes(
|
||||
return;
|
||||
}
|
||||
|
||||
if (agentInstructionsBundleMode(existing) === "managed") {
|
||||
const baseHash = typeof req.query.baseHash === "string" && /^[a-f0-9]{64}$/.test(req.query.baseHash) ? req.query.baseHash : null;
|
||||
if (baseHash === null) throw unprocessable("Read the file and supply baseHash before deleting it");
|
||||
await agentFiles.write({ companyId: existing.companyId, agentId: existing.id, path: relativePath, bytes: null, baseHash }, req.actor);
|
||||
res.json(await instructions.getBundle(existing)); return;
|
||||
}
|
||||
const actor = getActorInfo(req);
|
||||
const result = await instructions.deleteFile(existing, relativePath);
|
||||
await logActivity(db, {
|
||||
|
||||
@@ -217,7 +217,7 @@ export function builtInAgentRoutes(db: Db) {
|
||||
const key = req.params.key as string;
|
||||
await assertBuiltInAgentsEnabled();
|
||||
await assertCanProvisionBuiltInAgents(req, companyId);
|
||||
const state = await svc.reset(companyId, key, req.body);
|
||||
const state = await svc.reset(companyId, key, req.body, req.actor);
|
||||
await logBuiltInAgentMutation(req, {
|
||||
companyId,
|
||||
action: "built_in_agent.reset",
|
||||
|
||||
@@ -27,6 +27,8 @@ import {
|
||||
updateAgentInstructionsPathSchema,
|
||||
updateAgentInstructionsBundleSchema,
|
||||
upsertAgentInstructionsFileSchema,
|
||||
restoreAgentInstructionSchema,
|
||||
resolveAgentInstructionCandidateSchema,
|
||||
createAgentKeySchema,
|
||||
builtInAgentEmptyMutationSchema,
|
||||
builtInAgentProvisionSchema,
|
||||
@@ -3455,8 +3457,8 @@ registry.registerPath({
|
||||
method: "get",
|
||||
path: "/api/agents/{id}/instructions-bundle/file",
|
||||
tags: ["agents"],
|
||||
summary: "Get agent instructions file",
|
||||
request: { params: z.object({ id: z.string() }) },
|
||||
summary: "Get agent file content or download its original bytes",
|
||||
request: { params: z.object({ id: z.string() }), query: z.object({ path: z.string(), download: z.enum(["true", "false"]).optional() }) },
|
||||
responses: { 200: r.ok(), 401: r.unauthorized, 404: r.notFound },
|
||||
});
|
||||
|
||||
@@ -3472,12 +3474,46 @@ registry.registerPath({
|
||||
responses: { 200: r.ok(), 400: r.badRequest, 401: r.unauthorized },
|
||||
});
|
||||
|
||||
for (const operation of [
|
||||
{ suffix: "history", summary: "List immutable instruction revisions", query: z.object({ path: z.string().optional(), cursor: z.string().uuid().optional() }) },
|
||||
{ suffix: "revision/{revisionId}", summary: "Read exact instruction content at a revision", query: z.object({ path: z.string().optional() }) },
|
||||
{ suffix: "diff", summary: "Compare instruction revisions (exact common prefix, removed, added, suffix)", query: z.object({ path: z.string().optional(), from: z.string().uuid(), to: z.string().uuid() }) },
|
||||
]) {
|
||||
registry.registerPath({ method: "get", path: `/api/agents/{id}/instructions-bundle/${operation.suffix}`, tags: ["agents"], summary: operation.summary,
|
||||
request: { params: operation.suffix.includes("revisionId") ? z.object({ id: z.string(), revisionId: z.string().uuid() }) : z.object({ id: z.string() }), query: operation.query },
|
||||
responses: { 200: r.ok(), 401: r.unauthorized, 403: r.forbidden, 404: r.notFound } });
|
||||
}
|
||||
registry.registerPath({ method: "post", path: "/api/agents/{id}/instructions-bundle/restore", tags: ["agents"], summary: "Restore a pre-upgrade instruction snapshot into current files with compare-and-swap",
|
||||
request: { params: z.object({ id: z.string() }), body: jsonBody(restoreAgentInstructionSchema) },
|
||||
responses: { 200: r.ok(), 401: r.unauthorized, 403: r.forbidden, 404: r.notFound, 409: r.conflict } });
|
||||
|
||||
registry.registerPath({
|
||||
method: "get",
|
||||
path: "/api/agents/{id}/instructions-bundle/candidates",
|
||||
tags: ["agents"],
|
||||
summary: "List preserved instruction edits and collection diagnostics",
|
||||
request: { params: z.object({ id: z.string() }) },
|
||||
responses: { 200: r.ok(), 401: r.unauthorized, 403: r.forbidden, 404: r.notFound },
|
||||
});
|
||||
|
||||
registry.registerPath({
|
||||
method: "post",
|
||||
path: "/api/agents/{id}/instructions-bundle/candidates/{runId}/resolve",
|
||||
tags: ["agents"],
|
||||
summary: "Explicitly save a preserved instruction edit with compare-and-swap",
|
||||
request: {
|
||||
params: z.object({ id: z.string(), runId: z.string().uuid() }),
|
||||
body: jsonBody(resolveAgentInstructionCandidateSchema),
|
||||
},
|
||||
responses: { 200: r.ok(), 401: r.unauthorized, 403: r.forbidden, 404: r.notFound, 409: r.conflict, 422: r.unprocessable },
|
||||
});
|
||||
|
||||
registry.registerPath({
|
||||
method: "delete",
|
||||
path: "/api/agents/{id}/instructions-bundle/file",
|
||||
tags: ["agents"],
|
||||
summary: "Delete agent instructions file",
|
||||
request: { params: z.object({ id: z.string() }) },
|
||||
summary: "Delete agent file with compare-and-swap for managed storage",
|
||||
request: { params: z.object({ id: z.string() }), query: z.object({ path: z.string(), baseHash: z.string().regex(/^[a-f0-9]{64}$/).optional() }) },
|
||||
responses: { 200: r.ok(), 401: r.unauthorized },
|
||||
});
|
||||
|
||||
|
||||
@@ -0,0 +1,258 @@
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { agents, environmentLeases, environments, agentInstructionWorkingCopies as copies, type Db } from "@paperclipai/db";
|
||||
import { syncDirectoryToSsh, restoreWorkspaceFromSshExecution } from "@paperclipai/adapter-utils/ssh";
|
||||
import { prepareAdapterExecutionTargetRuntime, runAdapterExecutionTargetShellCommand, type AdapterExecutionTarget, type PreparedAdapterExecutionTargetRuntime } from "@paperclipai/adapter-utils/execution-target";
|
||||
import { withDirectoryMergeLock, directorySnapshotSha256, parseDirectorySnapshot, serializeDirectorySnapshot } from "@paperclipai/adapter-utils/workspace-restore-merge";
|
||||
import { AGENT_FILES_CONTRACT, AgentFileLimitError, agentFileStore, agentStorageWarning, inspectAgentDirectory } from "./agent-file-store.js";
|
||||
import { agentInstructionsBundleMode, deriveBundleState, resolveManagedInstructionsRoot } from "./agent-instructions.js";
|
||||
import { instructionGitExcludeProgram } from "./agent-instruction-files.js";
|
||||
import { resolveInstructionActor } from "./agent-instruction-authorization.js";
|
||||
import { HttpError, conflict, notFound } from "../errors.js";
|
||||
import type { AuthorizationActor } from "./authorization.js";
|
||||
import type { EnvironmentRuntimeService } from "./environment-runtime.js";
|
||||
import type { Environment, EnvironmentLease } from "@paperclipai/shared";
|
||||
import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
|
||||
|
||||
type Copy = typeof copies.$inferSelect;
|
||||
const completed = new Set(["saved", "unchanged", "resolved", "unavailable"]);
|
||||
const transports = new Map<string, PreparedAdapterExecutionTargetRuntime>();
|
||||
const key = (row: Pick<Copy, "companyId" | "runId">) => `${row.companyId}:${row.runId}`;
|
||||
export function isAgentDirectoryCopy(row: Pick<Copy, "receipt"> | null): boolean {
|
||||
return row?.receipt?.schema === AGENT_FILES_CONTRACT;
|
||||
}
|
||||
function baseline(row: Copy) {
|
||||
const snapshot = parseDirectorySnapshot(row.receipt?.baseline);
|
||||
if (!snapshot || directorySnapshotSha256(snapshot) !== row.baseHash) throw new Error("Agent directory baseline is invalid");
|
||||
return snapshot;
|
||||
}
|
||||
function actor(row: Copy): AuthorizationActor {
|
||||
return { type: "agent", companyId: row.companyId, agentId: row.agentId, runId: row.runId, onBehalfOfUserId: row.responsibleUserId };
|
||||
}
|
||||
export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string, runId: string) => Promise<Copy | null>, patch: (row: Copy, values: Partial<typeof copies.$inferInsert>) => Promise<Copy>, environmentRuntime?: EnvironmentRuntimeService) {
|
||||
const store = agentFileStore(db);
|
||||
async function transport(row: Copy, target: AdapterExecutionTarget, recovering: boolean) {
|
||||
if (target.kind !== "remote" || row.location !== `remote:${target.environmentId ?? ""}`) throw new Error("Agent directory environment changed");
|
||||
if (recovering && target.transport === "ssh") throw new Error("The original SSH collector is unavailable");
|
||||
if (target.transport === "ssh") {
|
||||
// Reuse SSH's plain directory transfer without its task-workspace suffix
|
||||
// or Git-history discovery. The registered root is the exact writable root.
|
||||
await syncDirectoryToSsh({ spec: target.spec, localDir: row.localRoot, remoteDir: row.executionRoot, exclude: [".paperclip-runtime"] });
|
||||
return { target, workspaceRemoteDir: row.executionRoot, runtimeRootDir: null,
|
||||
assetDirs: {}, additionalSourceDirs: {}, additionalSourceFailures: [], workspaceSyncSnapshot: null,
|
||||
restoreWorkspace: () => restoreWorkspaceFromSshExecution({ spec: target.spec, localDir: row.localRoot,
|
||||
remoteDir: row.executionRoot, baselineSnapshot: { ...baseline(row), exclude: [".paperclip-runtime"] }, restoreGitHistory: false }) };
|
||||
}
|
||||
return prepareAdapterExecutionTargetRuntime({ target, runId: row.runId, adapterKey: "agent-files",
|
||||
workspaceLocalDir: row.localRoot, workspaceRemoteDir: row.executionRoot,
|
||||
syncWorkspace: true, workspaceInboundMode: recovering ? "adopt_remote" : undefined,
|
||||
workspaceBaseline: baseline(row), workspaceGitSnapshot: null, workspaceFileMode: "all",
|
||||
workspaceExclude: [".paperclip-runtime", ".paperclip-runtime/**"] });
|
||||
}
|
||||
async function prepare(input: { companyId: string; agentId: string; runId: string; target?: AdapterExecutionTarget | null; cwd: string }) {
|
||||
const [agent] = await db.select().from(agents).where(and(eq(agents.id, input.agentId), eq(agents.companyId, input.companyId)));
|
||||
if (!agent) throw notFound("Agent not found");
|
||||
if (agentInstructionsBundleMode(agent) !== "managed") return null;
|
||||
const bound = await resolveInstructionActor(db, { type: "agent", companyId: input.companyId, agentId: input.agentId, runId: input.runId });
|
||||
const root = resolveManagedInstructionsRoot(agent);
|
||||
const localRoot = path.join(path.dirname(root), "file-sync", "runs", input.runId, "live");
|
||||
// Local copies live outside the task cwd. Remote copies use the reserved,
|
||||
// excluded runtime area inside the provider's confined workspace. They are
|
||||
// synchronized independently; provider HOME is never reinterpreted.
|
||||
const executionRoot = input.target?.kind === "remote"
|
||||
? path.posix.join(input.target.remoteCwd, ".paperclip-runtime", "agent-files", input.agentId, input.runId)
|
||||
: localRoot;
|
||||
const location = input.target?.kind === "remote" ? `remote:${input.target.environmentId ?? ""}` : "local";
|
||||
let row = await get(input.companyId, input.runId);
|
||||
if (row && (row.localRoot !== localRoot || row.executionRoot !== executionRoot || row.agentId !== input.agentId || row.location !== location)) throw conflict("Agent directory belongs to a different execution environment");
|
||||
if (row && !completed.has(row.state) && row.state !== "preparing") {
|
||||
if (input.target?.kind === "remote" && !transports.has(key(row))) transports.set(key(row), await transport(row, input.target, true));
|
||||
return row;
|
||||
}
|
||||
// Register ownership before creating any bytes, so a crash during the copy
|
||||
// leaves a recoverable preparing receipt instead of an orphan directory.
|
||||
const preparing = { entryFile: deriveBundleState(agent).entryFile, baseRevisionId: null, baseHash: "preparing",
|
||||
localRoot, executionRoot, location, state: "preparing", candidateBase64: null, candidateHash: null,
|
||||
receipt: { schema: AGENT_FILES_CONTRACT, ...(input.target?.kind === "remote"
|
||||
? { cleanup: { leaseId: input.target.leaseId ?? null, remoteCwd: input.target.remoteCwd } } : {}) }, processStoppedAt: null, attempts: 0,
|
||||
nextAttemptAt: null, errorCode: null, errorMessage: null };
|
||||
if (row) row = await patch(row, preparing);
|
||||
else {
|
||||
await db.insert(copies).values({ runId: input.runId, companyId: input.companyId, agentId: input.agentId, responsibleUserId: bound.onBehalfOfUserId!, ...preparing });
|
||||
row = (await get(input.companyId, input.runId))!;
|
||||
}
|
||||
const { snapshot, storageWarning } = await store.locked(input.companyId, input.agentId, bound, false, async (_tx, _agent, canonical) => {
|
||||
// Storage quotas govern saves, never admission to a future run. Restore
|
||||
// existing bytes so the agent can work normally and remove excess files.
|
||||
// Path/link validation remains mandatory even for an over-quota folder.
|
||||
const inspected = await inspectAgentDirectory(canonical, false);
|
||||
// A stopped lifecycle has already accounted for all its bytes. No live
|
||||
// or pending candidate ever enters this replacement path.
|
||||
await fs.rm(localRoot, { recursive: true, force: true });
|
||||
await fs.mkdir(path.dirname(localRoot), { recursive: true, mode: 0o700 });
|
||||
try {
|
||||
await fs.cp(canonical, localRoot, { recursive: true, preserveTimestamps: true });
|
||||
return inspected;
|
||||
} catch (error) {
|
||||
await fs.rm(path.dirname(localRoot), { recursive: true, force: true });
|
||||
throw error;
|
||||
}
|
||||
}).catch(async error => {
|
||||
await fs.rm(path.dirname(localRoot), { recursive: true, force: true });
|
||||
throw error;
|
||||
});
|
||||
const values = { entryFile: deriveBundleState(agent).entryFile, baseRevisionId: null, baseHash: directorySnapshotSha256(snapshot),
|
||||
localRoot, executionRoot, location, state: "preparing", candidateBase64: null, candidateHash: null,
|
||||
receipt: { ...preparing.receipt, baseline: serializeDirectorySnapshot(snapshot), storageWarning },
|
||||
processStoppedAt: null, attempts: 0, nextAttemptAt: null, errorCode: null, errorMessage: null };
|
||||
try {
|
||||
row = await patch(row, values);
|
||||
} catch (error) {
|
||||
await fs.rm(path.dirname(localRoot), { recursive: true, force: true });
|
||||
throw error;
|
||||
}
|
||||
try {
|
||||
if (input.target?.kind === "remote") {
|
||||
const quote = (value: string) => `'${value.replaceAll("'", `'"'"'`)}'`;
|
||||
const excluded = await runAdapterExecutionTargetShellCommand(input.runId, input.target,
|
||||
`node -e ${quote(instructionGitExcludeProgram)} ${quote(input.target.remoteCwd)}`,
|
||||
{ cwd: input.target.remoteCwd, env: {}, timeoutSec: 15 });
|
||||
if (excluded.exitCode !== 0 || excluded.timedOut) throw new Error("Could not exclude agent files from task Git staging");
|
||||
transports.set(key(row), await transport(row, input.target, false));
|
||||
}
|
||||
return await patch(row, { state: "prepared" });
|
||||
} catch (error) {
|
||||
// Preparation failed before a provider could start using this copy.
|
||||
row = await patch(row, { state: "unavailable", processStoppedAt: new Date(), errorCode: "AGENT_FILES_PREPARE_FAILED",
|
||||
errorMessage: "Agent files could not be staged. The temporary copy was discarded.", nextAttemptAt: null });
|
||||
await release(row, input.target);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async function retrieve(row: Copy, target?: AdapterExecutionTarget | null) {
|
||||
if (row.location !== "local") {
|
||||
let runtime = transports.get(key(row));
|
||||
if (!runtime) {
|
||||
if (!target) throw new Error("Agent directory transport is unavailable");
|
||||
runtime = await transport(row, target, true);
|
||||
transports.set(key(row), runtime);
|
||||
}
|
||||
await runtime.restoreWorkspace();
|
||||
}
|
||||
return inspectAgentDirectory(row.localRoot);
|
||||
}
|
||||
async function hasChanges(_row: Copy, _target?: AdapterExecutionTarget | null) {
|
||||
// A whole-directory collector needs a quiescent provider, including its
|
||||
// child processes. Checkpoint and close at the turn boundary before reading
|
||||
// either local or remote files. The provider conversation remains resumable;
|
||||
// ordinary file edits do not change the session configuration digest.
|
||||
return true;
|
||||
}
|
||||
async function collectStopped(row: Copy, target?: AdapterExecutionTarget | null) {
|
||||
if (completed.has(row.state)) { await release(row); return (await get(row.companyId, row.runId))!; }
|
||||
row = await patch(row, { processStoppedAt: row.processStoppedAt ?? new Date() });
|
||||
// The stopped working copy is the only temporary tree. Retry transient I/O
|
||||
// at this boundary, then clean it up; there are no preserved run snapshots.
|
||||
let inspected: Awaited<ReturnType<typeof retrieve>> | undefined;
|
||||
for (;;) {
|
||||
row = await patch(row, { attempts: row.attempts + 1 });
|
||||
try {
|
||||
inspected ??= await retrieve(row, target);
|
||||
const { snapshot } = inspected;
|
||||
const candidateHash = directorySnapshotSha256(snapshot);
|
||||
let storageWarning = inspected.storageWarning;
|
||||
if (candidateHash === row.baseHash) {
|
||||
row = await patch(row, { state: "unchanged", errorCode: null, errorMessage: null, nextAttemptAt: null });
|
||||
} else {
|
||||
({ storageWarning } = await store.apply({ companyId: row.companyId, agentId: row.agentId, sourceDir: row.localRoot, baseline: baseline(row) }, actor(row)));
|
||||
row = await patch(row, { state: "saved", candidateHash, errorCode: null, errorMessage: null, nextAttemptAt: null });
|
||||
}
|
||||
row = await patch(row, { receipt: { ...row.receipt, storageWarning } });
|
||||
break;
|
||||
} catch (error) {
|
||||
const retryable = !(error instanceof HttpError) || error.status >= 500;
|
||||
if (retryable && row.attempts < 3) continue;
|
||||
row = await patch(row, { state: "unavailable", nextAttemptAt: null,
|
||||
receipt: { ...row.receipt, storageWarning: error instanceof AgentFileLimitError ? agentStorageWarning(error.message) : row.receipt?.storageWarning ?? null },
|
||||
errorCode: error instanceof AgentFileLimitError ? "AGENT_FILES_LIMIT_EXCEEDED" : "AGENT_FILES_SAVE_FAILED",
|
||||
errorMessage: error instanceof HttpError && error.status === 422
|
||||
? `${error.message}. This run's agent-folder changes were not saved; the temporary copy is discarded.`
|
||||
: "Agent-file synchronization failed. No successful save is claimed; the temporary copy is discarded.",
|
||||
});
|
||||
break;
|
||||
}
|
||||
}
|
||||
await release(row);
|
||||
return (await get(row.companyId, row.runId))!;
|
||||
}
|
||||
async function release(row: Copy, target?: AdapterExecutionTarget | null) {
|
||||
// Collection and environment teardown can both release the same copy.
|
||||
// A compact successful cleanup receipt is final, even after restart.
|
||||
if (completed.has(row.state) && row.processStoppedAt && row.receipt?.cleanupPending === false) return;
|
||||
const runtime = transports.get(key(row));
|
||||
transports.delete(key(row));
|
||||
let cleanupPending = false;
|
||||
await runtime?.cleanupWorkspaceSnapshot?.().catch(() => { cleanupPending = true; });
|
||||
const cleanupTarget = runtime?.target ?? target;
|
||||
if (row.processStoppedAt && completed.has(row.state) && cleanupTarget?.kind === "remote") {
|
||||
const expected = path.posix.join(cleanupTarget.remoteCwd, ".paperclip-runtime", "agent-files", row.agentId, row.runId);
|
||||
if (row.executionRoot !== expected) throw new Error("Agent directory cleanup path changed");
|
||||
const quoted = `'${expected.replaceAll("'", `'"'"'`)}'`;
|
||||
const remoteCleanupFailed = await runAdapterExecutionTargetShellCommand(row.runId, cleanupTarget, `rm -rf -- ${quoted}`,
|
||||
{ cwd: cleanupTarget.remoteCwd, env: {}, timeoutSec: 15 }).then(result => result.exitCode !== 0 || result.timedOut, () => true);
|
||||
cleanupPending ||= remoteCleanupFailed;
|
||||
} else if (row.processStoppedAt && completed.has(row.state) && row.location !== "local") {
|
||||
// Rebind only the original host-owned lease. Never acquire a replacement
|
||||
// sandbox or persist transport credentials in a working-copy receipt.
|
||||
cleanupPending ||= !await cleanupRemoteAfterRestart(row).catch(() => false);
|
||||
}
|
||||
if (completed.has(row.state) && row.processStoppedAt) {
|
||||
try { await fs.rm(path.dirname(row.localRoot), { recursive: true, force: true }); }
|
||||
catch {
|
||||
// Leave the baseline marker so restart recovery retries failed cleanup.
|
||||
await patch(row, { receipt: { ...row.receipt, cleanupPending: true } });
|
||||
return;
|
||||
}
|
||||
await patch(row, { receipt: { schema: AGENT_FILES_CONTRACT, state: row.state, appliedCandidateHash: row.candidateHash, storageWarning: row.receipt?.storageWarning ?? null, cleanupPending,
|
||||
...(cleanupPending ? { cleanup: row.receipt?.cleanup } : {}) },
|
||||
nextAttemptAt: cleanupPending ? new Date(Date.now() + 30_000) : null });
|
||||
}
|
||||
}
|
||||
async function cleanupRemoteAfterRestart(row: Copy): Promise<boolean> {
|
||||
const cleanup = row.receipt?.cleanup as { leaseId?: string; remoteCwd?: string } | undefined;
|
||||
// Older preview receipts did not record the lease ID. Accept only a unique
|
||||
// lease still bound to this run and environment in that compatibility case.
|
||||
const leases = await db.select().from(environmentLeases).where(and(
|
||||
eq(environmentLeases.companyId, row.companyId),
|
||||
cleanup?.leaseId ? eq(environmentLeases.id, cleanup.leaseId) : and(
|
||||
eq(environmentLeases.environmentId, row.location.slice("remote:".length)), eq(environmentLeases.heartbeatRunId, row.runId)),
|
||||
));
|
||||
if (leases.length !== 1) return false;
|
||||
const lease = leases[0]!;
|
||||
const termination = lease.metadata?.remoteExecutionTermination as { state?: string } | undefined;
|
||||
if (hasRemoteTerminationReceipt(lease) && termination?.state === "destroyed") return true;
|
||||
if (!environmentRuntime || !lease.environmentId || row.location !== `remote:${lease.environmentId}`) return false;
|
||||
const [environment] = await db.select().from(environments).where(eq(environments.id, lease.environmentId));
|
||||
if (!environment) return false;
|
||||
const remoteCwd = cleanup?.remoteCwd ?? lease.metadata?.remoteCwd;
|
||||
if (typeof remoteCwd !== "string" || row.executionRoot !== path.posix.join(remoteCwd, ".paperclip-runtime", "agent-files", row.agentId, row.runId)) return false;
|
||||
const result = await environmentRuntime.execute({ environment: environment as Environment, lease: lease as EnvironmentLease, command: "rm", args: ["-rf", "--", row.executionRoot],
|
||||
cwd: remoteCwd, env: {}, timeoutMs: 15_000, bypassSession: true });
|
||||
return result.exitCode === 0 && !result.timedOut;
|
||||
}
|
||||
async function serial<T>(row: Copy, fn: (current: Copy) => Promise<T>): Promise<T> {
|
||||
// Duplicate stop callbacks and restart recovery must not race while moving
|
||||
// the stopped working copy. This lock is outside the writable tree.
|
||||
return withDirectoryMergeLock(path.resolve(row.localRoot, "../../.."), async () => {
|
||||
const current = await get(row.companyId, row.runId);
|
||||
if (!current) throw notFound("Agent directory copy not found");
|
||||
return fn(current);
|
||||
});
|
||||
}
|
||||
return { prepare, hasChanges,
|
||||
collectStopped: (row: Copy, target?: AdapterExecutionTarget | null) => serial(row, current => collectStopped(current, target)),
|
||||
release: (row: Copy) => serial(row, release),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,282 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import fs from "node:fs/promises";
|
||||
import { constants } from "node:fs";
|
||||
import { Readable } from "node:stream";
|
||||
import path from "node:path";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { agents, activityLog, agentInstructionHeads, agentInstructionRevisions, type Db } from "@paperclipai/db";
|
||||
import { captureDirectorySnapshot, mergeDirectoryWithBaseline, type DirectorySnapshot } from "@paperclipai/adapter-utils/workspace-restore-merge";
|
||||
import { HttpError, conflict, notFound, unprocessable } from "../errors.js";
|
||||
import { authorizeInstructionCommit, authorizeInstructionRead } from "./agent-instruction-authorization.js";
|
||||
import { assertInstructionPathSafe, instructionPath, instructionBytes, materializeInstructionBytes, readInstructionBytes, MAX_INSTRUCTION_BYTES } from "./agent-instruction-files.js";
|
||||
import { agentInstructionsBundleMode, deriveBundleState, resolveManagedInstructionsRoot } from "./agent-instructions.js";
|
||||
import type { AuthorizationActor } from "./authorization.js";
|
||||
|
||||
type Tx = Parameters<Parameters<Db["transaction"]>[0]>[0];
|
||||
type Agent = typeof agents.$inferSelect;
|
||||
export const AGENT_FILES_CONTRACT = "paperclip.agent-files.v1";
|
||||
export const MAX_AGENT_FILE_BYTES = 256 * 1024 * 1024;
|
||||
export const MAX_AGENT_DIRECTORY_BYTES = 2 * 1024 * 1024 * 1024;
|
||||
export const MAX_AGENT_DIRECTORY_ENTRIES = 100_000;
|
||||
export const fileHash = (bytes: Uint8Array) => createHash("sha256").update(bytes).digest("hex");
|
||||
|
||||
export class AgentFileLimitError extends HttpError {
|
||||
constructor(message: string) {
|
||||
super(422, message, { code: "AGENT_FILES_LIMIT_EXCEEDED" });
|
||||
}
|
||||
}
|
||||
function assertFileSize(size: number, relative: string) {
|
||||
if (size > MAX_AGENT_FILE_BYTES) throw new AgentFileLimitError(`Agent file "${relative}" exceeds the 256 MiB per-file limit`);
|
||||
}
|
||||
function assertDirectorySize(size: number, count: number) {
|
||||
if (size > MAX_AGENT_DIRECTORY_BYTES) throw new AgentFileLimitError("Agent directory exceeds the 2 GiB total storage limit");
|
||||
if (count > MAX_AGENT_DIRECTORY_ENTRIES) throw new AgentFileLimitError("Agent directory exceeds the 100,000-entry limit (files and folders)");
|
||||
}
|
||||
export function agentStorageWarning(detail: string) {
|
||||
return `Agent storage is full. ${detail}. Runs can continue; remove or shrink files in AGENT_HOME to free space. Changes exceeding the storage limits will not be saved.`;
|
||||
}
|
||||
function storageWarning(size: number, count: number, fullFile?: string) {
|
||||
if (size >= MAX_AGENT_DIRECTORY_BYTES) return agentStorageWarning("The agent folder has reached its 2 GiB limit");
|
||||
if (count >= MAX_AGENT_DIRECTORY_ENTRIES) return agentStorageWarning("The agent folder has reached its 100,000-entry limit");
|
||||
if (fullFile) return agentStorageWarning(`Agent file "${fullFile}" has reached its 256 MiB limit`);
|
||||
return null;
|
||||
}
|
||||
|
||||
export function agentFilePath(value: string): string {
|
||||
const relative = instructionPath(value);
|
||||
if (relative.split("/").includes(".paperclip-runtime") || relative === "promptTemplate.legacy.md") {
|
||||
throw unprocessable(`${relative} is reserved and cannot be used for agent files`);
|
||||
}
|
||||
return relative;
|
||||
}
|
||||
|
||||
/** Compatibility ETag for clients whose old schema requires a UUID. This is a
|
||||
* content token, not a revision ID: no snapshot or history row is created. */
|
||||
export function agentFileToken(bytes: Uint8Array): string {
|
||||
return agentFileTokenFromHash(fileHash(bytes));
|
||||
}
|
||||
export function agentFileTokenFromHash(h: string): string {
|
||||
return `${h.slice(0, 8)}-${h.slice(8, 12)}-8${h.slice(13, 16)}-a${h.slice(17, 20)}-${h.slice(20, 32)}`;
|
||||
}
|
||||
|
||||
/** Call under the agent row lock. Import deployed revision heads once, then the
|
||||
* directory is authoritative. The adoption marker lives OUTSIDE agent files. */
|
||||
export async function adoptAgentFiles(tx: Tx, agent: Agent): Promise<string> {
|
||||
if (agentInstructionsBundleMode(agent) === "external") throw unprocessable("External instructions must be migrated to managed storage first");
|
||||
const root = resolveManagedInstructionsRoot(agent);
|
||||
const markerRoot = path.join(path.dirname(root), "file-sync");
|
||||
const marker = "adopted.json";
|
||||
await assertInstructionPathSafe(root, deriveBundleState(agent).entryFile);
|
||||
await fs.mkdir(root, { recursive: true, mode: 0o700 });
|
||||
const filename = await assertInstructionPathSafe(markerRoot, marker);
|
||||
const adopted = await fs.readFile(filename, "utf8").catch((error: NodeJS.ErrnoException) => {
|
||||
if (error.code === "ENOENT") return null;
|
||||
throw error;
|
||||
});
|
||||
if (adopted !== null) {
|
||||
if (JSON.parse(adopted).schema !== AGENT_FILES_CONTRACT) throw new Error("Unsupported agent file contract");
|
||||
return root;
|
||||
}
|
||||
const legacy = await tx.select({ entryFile: agentInstructionHeads.entryFile, contentBase64: agentInstructionRevisions.contentBase64 })
|
||||
.from(agentInstructionHeads).innerJoin(agentInstructionRevisions, eq(agentInstructionHeads.revisionId, agentInstructionRevisions.id))
|
||||
.where(and(eq(agentInstructionHeads.companyId, agent.companyId), eq(agentInstructionHeads.agentId, agent.id)));
|
||||
for (const entry of legacy) await materializeInstructionBytes(root, entry.entryFile, Buffer.from(entry.contentBase64, "base64"));
|
||||
await materializeInstructionBytes(markerRoot, marker, Buffer.from(JSON.stringify({ schema: AGENT_FILES_CONTRACT })));
|
||||
return root;
|
||||
}
|
||||
|
||||
async function openAgentFile(root: string, relative: string) {
|
||||
const filename = await assertInstructionPathSafe(root, relative);
|
||||
const handle = await fs.open(filename, constants.O_RDONLY | constants.O_NOFOLLOW).catch((error: NodeJS.ErrnoException) => {
|
||||
if (error.code === "ENOENT") return null;
|
||||
throw error;
|
||||
});
|
||||
if (!handle) return null;
|
||||
try {
|
||||
const stat = await handle.stat();
|
||||
if (!stat.isFile() || stat.nlink !== 1) throw unprocessable("Agent files must be regular files without links");
|
||||
assertFileSize(stat.size, relative);
|
||||
return { handle, size: stat.size };
|
||||
} catch (error) { await handle.close(); throw error; }
|
||||
}
|
||||
|
||||
/** Hash large files incrementally. Only editor-sized content is retained in RAM. */
|
||||
export async function inspectAgentFile(root: string, relative: string, bufferLimit = MAX_INSTRUCTION_BYTES) {
|
||||
const opened = await openAgentFile(root, relative);
|
||||
if (!opened) return null;
|
||||
const hash = createHash("sha256");
|
||||
let chunks: Buffer[] | null = opened.size <= bufferLimit ? [] : null;
|
||||
let size = 0;
|
||||
try {
|
||||
for await (const chunk of opened.handle.createReadStream()) {
|
||||
size += chunk.length;
|
||||
assertFileSize(size, relative);
|
||||
hash.update(chunk);
|
||||
if (size > bufferLimit) chunks = null;
|
||||
chunks?.push(chunk);
|
||||
}
|
||||
return { size, hash: hash.digest("hex"), bytes: chunks ? Buffer.concat(chunks) : null };
|
||||
} finally { await opened.handle.close(); }
|
||||
}
|
||||
|
||||
export async function readAgentFile(root: string, relative: string): Promise<Buffer | null> {
|
||||
return (await inspectAgentFile(root, relative, MAX_AGENT_FILE_BYTES))?.bytes ?? null;
|
||||
}
|
||||
|
||||
/** Validate before staging and again after provider stop; never follow links or
|
||||
* silently skip an unsupported file. Bounds apply to bytes, including binaries. */
|
||||
async function scanAgentFiles(root: string, enforceLimits = true) {
|
||||
await assertInstructionPathSafe(root, ".path-check");
|
||||
let size = 0, count = 0;
|
||||
const entries = new Set<string>();
|
||||
let fullFile: string | undefined;
|
||||
async function walk(dir: string) {
|
||||
for (const item of await fs.readdir(path.join(root, dir), { withFileTypes: true })) {
|
||||
const relative = agentFilePath(dir ? `${dir}/${item.name}` : item.name);
|
||||
const stat = await fs.lstat(path.join(root, relative));
|
||||
count++;
|
||||
if (enforceLimits) assertDirectorySize(size, count);
|
||||
entries.add(relative);
|
||||
if (stat.isDirectory()) await walk(relative);
|
||||
else if (stat.isFile() && stat.nlink === 1) {
|
||||
size += stat.size;
|
||||
if (stat.size >= MAX_AGENT_FILE_BYTES) fullFile ??= relative;
|
||||
if (enforceLimits) {
|
||||
assertFileSize(stat.size, relative);
|
||||
assertDirectorySize(size, count);
|
||||
}
|
||||
} else throw unprocessable("Agent directories support regular files and directories, without links or special files");
|
||||
}
|
||||
}
|
||||
await walk("");
|
||||
return { size, entries, storageWarning: storageWarning(size, count, fullFile) };
|
||||
}
|
||||
|
||||
export async function inspectAgentDirectory(root: string, enforceLimits = true) {
|
||||
const usage = await scanAgentFiles(root, enforceLimits);
|
||||
return { snapshot: await captureDirectorySnapshot(root), storageWarning: usage.storageWarning };
|
||||
}
|
||||
export async function snapshotAgentFiles(root: string, enforceLimits = true): Promise<DirectorySnapshot> {
|
||||
return (await inspectAgentDirectory(root, enforceLimits)).snapshot;
|
||||
}
|
||||
|
||||
export function agentFileStore(db: Db) {
|
||||
async function locked<T>(companyId: string, agentId: string, actor: AuthorizationActor, write: boolean,
|
||||
fn: (tx: Tx, agent: Agent, root: string, bound: AuthorizationActor) => Promise<T>) {
|
||||
return db.transaction(async tx => {
|
||||
const [agent] = await tx.select().from(agents).where(and(eq(agents.companyId, companyId), eq(agents.id, agentId))).for("update");
|
||||
if (!agent) throw notFound("Agent not found");
|
||||
const bound = write ? await authorizeInstructionCommit(tx, actor, agent) : await authorizeInstructionRead(tx, actor, agent);
|
||||
const root = await adoptAgentFiles(tx, agent);
|
||||
return fn(tx, agent, root, bound);
|
||||
});
|
||||
}
|
||||
async function audit(tx: Tx, agent: Agent, actor: AuthorizationActor, details: Record<string, unknown>) {
|
||||
await tx.insert(activityLog).values({ companyId: agent.companyId, actorType: actor.type === "board" ? "user" : "agent",
|
||||
actorId: (actor.type === "board" ? actor.userId : actor.agentId)!, agentId: actor.type === "agent" ? actor.agentId : null,
|
||||
runId: actor.runId, responsibleUserId: actor.type === "board" ? actor.userId : actor.onBehalfOfUserId,
|
||||
action: "agent.files_updated", entityType: "agent", entityId: agent.id, details });
|
||||
}
|
||||
return {
|
||||
locked,
|
||||
download: async (companyId: string, agentId: string, relative: string, actor: AuthorizationActor) => {
|
||||
const opened: { file: Awaited<ReturnType<typeof openAgentFile>> } = { file: null };
|
||||
try {
|
||||
const file = await locked(companyId, agentId, actor, false, async (_tx, _agent, root) => {
|
||||
opened.file = await openAgentFile(root, instructionPath(relative));
|
||||
return opened.file;
|
||||
});
|
||||
// Transfer outside the database lock. The open descriptor pins the file
|
||||
// across concurrent atomic replacements; end bounds concurrent growth.
|
||||
if (!file) return null;
|
||||
if (file.size === 0) { await file.handle.close(); return { size: 0, stream: Readable.from([]) }; }
|
||||
return { size: file.size, stream: file.handle.createReadStream({ end: file.size - 1 }) };
|
||||
} catch (error) { await opened.file?.handle.close(); throw error; }
|
||||
},
|
||||
read: (companyId: string, agentId: string, relative: string, actor: AuthorizationActor) =>
|
||||
locked(companyId, agentId, actor, false, (_tx, _agent, root) => readAgentFile(root, instructionPath(relative))),
|
||||
write: (input: { companyId: string; agentId: string; path: string; bytes: Buffer | null; baseHash: string | null }, actor: AuthorizationActor) =>
|
||||
locked(input.companyId, input.agentId, actor, true, async (tx, agent, root, bound) => {
|
||||
const relative = agentFilePath(input.path);
|
||||
if (input.bytes) assertFileSize(input.bytes.length, relative);
|
||||
const previous = await inspectAgentFile(root, relative);
|
||||
const currentHash = previous?.hash ?? null;
|
||||
const incomingHash = input.bytes === null ? null : fileHash(input.bytes);
|
||||
if (currentHash === incomingHash) return { contentHash: currentHash, changed: false };
|
||||
if (currentHash !== input.baseHash) throw conflict("This file changed since it was read. Reload before saving.", { code: "AGENT_FILE_CONFLICT", path: relative, currentHash });
|
||||
if (input.bytes === null && relative === deriveBundleState(agent).entryFile) throw unprocessable("The configured instruction entry cannot be deleted");
|
||||
if (input.bytes !== null) {
|
||||
if (relative === deriveBundleState(agent).entryFile) instructionBytes(input.bytes);
|
||||
// A quota check needs metadata only. Do not hash unrelated large
|
||||
// files while holding the editor's row lock for a one-file save.
|
||||
const snapshot = await scanAgentFiles(root);
|
||||
const total = snapshot.size + input.bytes.length - (previous?.size ?? 0);
|
||||
const newEntries = relative.split("/").map((_part, i, parts) => parts.slice(0, i + 1).join("/")).filter(name => !snapshot.entries.has(name)).length;
|
||||
assertDirectorySize(total, snapshot.entries.size + newEntries);
|
||||
}
|
||||
if (input.bytes === null) await fs.unlink(await assertInstructionPathSafe(root, relative));
|
||||
else await materializeInstructionBytes(root, relative, input.bytes);
|
||||
await audit(tx, agent, bound, { path: relative, contentHash: incomingHash });
|
||||
return { contentHash: incomingHash, changed: true };
|
||||
}),
|
||||
apply: (input: { companyId: string; agentId: string; sourceDir: string; baseline: DirectorySnapshot }, actor: AuthorizationActor) =>
|
||||
locked(input.companyId, input.agentId, actor, true, async (tx, agent, root, bound) => {
|
||||
const incoming = await snapshotAgentFiles(input.sourceDir);
|
||||
const entry = await readInstructionBytes(input.sourceDir, deriveBundleState(agent).entryFile);
|
||||
if (entry === null) throw unprocessable("The configured instruction entry cannot be deleted");
|
||||
instructionBytes(entry);
|
||||
// Previously saved/imported bytes must remain readable, including when
|
||||
// over quota. Enforce limits on the incoming and resulting tree so a
|
||||
// run can delete files to recover instead of being locked out forever.
|
||||
const current = await snapshotAgentFiles(root, false);
|
||||
// Rebase only the run's changed paths onto the current tree. This makes
|
||||
// same-file edits/deletions last-sync-wins while untouched files retain
|
||||
// changes from other runs. Ancestors may need recreating after a writer
|
||||
// replaced a directory with a file.
|
||||
const entries = new Map(input.baseline.entries);
|
||||
const changed = [...new Set([...input.baseline.entries, ...incoming.entries].map(([name]) => name))]
|
||||
.filter(name => JSON.stringify(input.baseline.entries.get(name)) !== JSON.stringify(incoming.entries.get(name)));
|
||||
for (const name of changed) {
|
||||
const present = current.entries.get(name);
|
||||
if (present) entries.set(name, present); else entries.delete(name);
|
||||
if (incoming.entries.has(name)) {
|
||||
for (let parent = path.posix.dirname(name); parent !== "."; parent = path.posix.dirname(parent)) {
|
||||
if (current.entries.get(parent)?.kind === "dir") continue;
|
||||
const before = current.entries.get(parent);
|
||||
if (before) entries.set(parent, before); else entries.delete(parent);
|
||||
}
|
||||
}
|
||||
}
|
||||
const applyBaseline = { ...input.baseline, entries };
|
||||
const finalEntries = new Map([...current.entries].map(([name, value]) => [name, { value, root }]));
|
||||
for (const name of changed) if (!incoming.entries.has(name)) finalEntries.delete(name);
|
||||
for (const [name, value] of incoming.entries) {
|
||||
if (JSON.stringify(input.baseline.entries.get(name)) === JSON.stringify(value)) continue;
|
||||
if (value.kind !== "dir") for (const child of finalEntries.keys()) if (child.startsWith(`${name}/`)) finalEntries.delete(child);
|
||||
finalEntries.set(name, { value, root: input.sourceDir });
|
||||
for (let parent = path.posix.dirname(name); parent !== "."; parent = path.posix.dirname(parent)) {
|
||||
if (finalEntries.get(parent)?.value.kind !== "dir") finalEntries.set(parent, { value: { kind: "dir" }, root: input.sourceDir });
|
||||
}
|
||||
}
|
||||
// An unchanged file created by another run can keep a removed folder
|
||||
// alive. Count those surviving parent directories in quota preflight.
|
||||
for (const name of finalEntries.keys()) {
|
||||
for (let parent = path.posix.dirname(name); parent !== "."; parent = path.posix.dirname(parent)) {
|
||||
if (!finalEntries.has(parent)) finalEntries.set(parent, { value: { kind: "dir" }, root });
|
||||
}
|
||||
}
|
||||
let total = 0;
|
||||
let fullFile: string | undefined;
|
||||
for (const [name, item] of finalEntries) if (item.value.kind === "file") {
|
||||
const size = (await fs.stat(path.join(item.root, name))).size;
|
||||
assertFileSize(size, name);
|
||||
if (size >= MAX_AGENT_FILE_BYTES) fullFile ??= name;
|
||||
total += size;
|
||||
}
|
||||
assertDirectorySize(total, finalEntries.size);
|
||||
await mergeDirectoryWithBaseline({ ...input, baseline: applyBaseline, targetDir: root });
|
||||
await audit(tx, agent, bound, { sourceRunId: actor.runId, contract: AGENT_FILES_CONTRACT });
|
||||
return { storageWarning: storageWarning(total, finalEntries.size, fullFile) };
|
||||
}),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,195 @@
|
||||
import { and, eq, isNull } from "drizzle-orm";
|
||||
import {
|
||||
agentApiKeys,
|
||||
authUsers,
|
||||
companyMemberships,
|
||||
heartbeatRuns,
|
||||
runIdentityContexts,
|
||||
type Db,
|
||||
} from "@paperclipai/db";
|
||||
import { forbidden } from "../errors.js";
|
||||
import {
|
||||
authorizationService,
|
||||
authorizationDeniedDetails,
|
||||
type AuthorizationActor,
|
||||
} from "./authorization.js";
|
||||
import {
|
||||
agentInstructionsChangeTargetKey,
|
||||
changeConsentGateService,
|
||||
} from "./change-consent-gate.js";
|
||||
|
||||
type Connection = Db | Parameters<Parameters<Db["transaction"]>[0]>[0];
|
||||
|
||||
/** Accept ONLY a server-authenticated actor, never request/tool JSON. Rebind run/key provenance on every call. */
|
||||
export async function resolveInstructionActor(
|
||||
db: Connection,
|
||||
actor: AuthorizationActor,
|
||||
): Promise<AuthorizationActor> {
|
||||
if (actor.type === "board") {
|
||||
if (!actor.userId)
|
||||
throw forbidden("Instruction edits require an authenticated user");
|
||||
// Discard caller membership caches so revoked access is observed.
|
||||
return {
|
||||
type: "board",
|
||||
userId: actor.userId,
|
||||
source: actor.source,
|
||||
ignoreInstanceAdmin: actor.ignoreInstanceAdmin,
|
||||
};
|
||||
}
|
||||
if (actor.type !== "agent" || !actor.agentId || !actor.companyId)
|
||||
throw forbidden("Instruction identity is missing");
|
||||
let responsibleUserId: string | null = null;
|
||||
let keyScope = actor.keyScope;
|
||||
if (actor.keyId) {
|
||||
const [key] = await db
|
||||
.select()
|
||||
.from(agentApiKeys)
|
||||
.where(
|
||||
and(
|
||||
eq(agentApiKeys.id, actor.keyId),
|
||||
eq(agentApiKeys.companyId, actor.companyId),
|
||||
eq(agentApiKeys.agentId, actor.agentId),
|
||||
isNull(agentApiKeys.revokedAt),
|
||||
),
|
||||
);
|
||||
if (!key)
|
||||
throw forbidden("Instruction API key is missing or revoked", {
|
||||
code: "INSTRUCTION_IDENTITY_INVALID",
|
||||
});
|
||||
responsibleUserId = key.responsibleUserId;
|
||||
keyScope = key.scopeConfig ?? undefined;
|
||||
}
|
||||
if (actor.runId) {
|
||||
const [run] = await db
|
||||
.select()
|
||||
.from(heartbeatRuns)
|
||||
.where(
|
||||
and(
|
||||
eq(heartbeatRuns.id, actor.runId),
|
||||
eq(heartbeatRuns.companyId, actor.companyId),
|
||||
eq(heartbeatRuns.agentId, actor.agentId),
|
||||
),
|
||||
);
|
||||
if (!run)
|
||||
throw forbidden("Instruction run identity does not match", {
|
||||
code: "INSTRUCTION_IDENTITY_INVALID",
|
||||
});
|
||||
if (actor.keyId && responsibleUserId !== run.responsibleUserId)
|
||||
throw forbidden("Key and run responsible identity do not match", {
|
||||
code: "INSTRUCTION_IDENTITY_INVALID",
|
||||
});
|
||||
responsibleUserId = run.responsibleUserId;
|
||||
if (run.activeIdentityContextId) {
|
||||
const [context] = await db
|
||||
.select()
|
||||
.from(runIdentityContexts)
|
||||
.where(
|
||||
and(
|
||||
eq(runIdentityContexts.id, run.activeIdentityContextId),
|
||||
eq(runIdentityContexts.runId, run.id),
|
||||
eq(runIdentityContexts.companyId, run.companyId),
|
||||
eq(runIdentityContexts.status, "accepted"),
|
||||
),
|
||||
);
|
||||
if (!context)
|
||||
throw forbidden("Instruction run identity is unavailable", {
|
||||
code: "INSTRUCTION_IDENTITY_INVALID",
|
||||
});
|
||||
responsibleUserId = context.responsibleUserId;
|
||||
}
|
||||
}
|
||||
if (
|
||||
!responsibleUserId ||
|
||||
(actor.onBehalfOfUserId && actor.onBehalfOfUserId !== responsibleUserId)
|
||||
) {
|
||||
throw forbidden("A current server-bound responsible user is required", {
|
||||
code: "INSTRUCTION_IDENTITY_INVALID",
|
||||
});
|
||||
}
|
||||
const [membership] = await db
|
||||
.select({ userId: authUsers.id })
|
||||
.from(companyMemberships)
|
||||
.innerJoin(authUsers, eq(authUsers.id, companyMemberships.principalId))
|
||||
.where(
|
||||
and(
|
||||
eq(companyMemberships.companyId, actor.companyId),
|
||||
eq(companyMemberships.principalType, "user"),
|
||||
eq(companyMemberships.principalId, responsibleUserId),
|
||||
eq(companyMemberships.status, "active"),
|
||||
),
|
||||
);
|
||||
if (!membership)
|
||||
throw forbidden("The responsible user is unavailable in this company", {
|
||||
code: "RESPONSIBLE_USER_UNAVAILABLE",
|
||||
});
|
||||
return {
|
||||
type: "agent",
|
||||
agentId: actor.agentId,
|
||||
companyId: actor.companyId,
|
||||
runId: actor.runId,
|
||||
keyId: actor.keyId,
|
||||
keyScope,
|
||||
source: actor.source,
|
||||
onBehalfOfUserId: responsibleUserId,
|
||||
};
|
||||
}
|
||||
|
||||
/** Instruction content is narrower than general same-company agent visibility. */
|
||||
export async function authorizeInstructionRead(
|
||||
db: Connection,
|
||||
actor: AuthorizationActor,
|
||||
target: { companyId: string; id: string },
|
||||
) {
|
||||
const bound = await resolveInstructionActor(db, actor);
|
||||
const access = authorizationService(db);
|
||||
const resource = { type: "agent" as const, companyId: target.companyId, agentId: target.id };
|
||||
const peer = bound.type === "agent" && bound.agentId !== target.id;
|
||||
const decision = await access.decide({
|
||||
actor: bound,
|
||||
action: peer ? "agent_config:read" : "agent:read",
|
||||
resource,
|
||||
scope: { targetAgentId: target.id },
|
||||
});
|
||||
if (!decision.allowed) throw forbidden(decision.explanation, authorizationDeniedDetails(decision));
|
||||
return bound;
|
||||
}
|
||||
|
||||
/** Current target edit access + agent containment + protected-change consent, shared by all writers. */
|
||||
export async function authorizeInstructionCommit(
|
||||
db: Connection,
|
||||
actor: AuthorizationActor,
|
||||
target: { companyId: string; id: string },
|
||||
) {
|
||||
const bound = await resolveInstructionActor(db, actor);
|
||||
const access = authorizationService(db);
|
||||
const input = {
|
||||
actor: bound,
|
||||
action: "agent_instructions:update" as const,
|
||||
resource: {
|
||||
type: "agent" as const,
|
||||
companyId: target.companyId,
|
||||
agentId: target.id,
|
||||
},
|
||||
scope: { targetAgentId: target.id, requiresChangeGrant: true },
|
||||
};
|
||||
let decision = await access.decide(input);
|
||||
if (
|
||||
!decision.allowed &&
|
||||
decision.reason === "deny_missing_consent" &&
|
||||
bound.type === "agent"
|
||||
) {
|
||||
await changeConsentGateService(db).assertConsented({
|
||||
companyId: target.companyId,
|
||||
actorAgentId: bound.agentId,
|
||||
actorRunId: bound.runId,
|
||||
targetKeys: [agentInstructionsChangeTargetKey(target.id)],
|
||||
});
|
||||
decision = await access.decide({
|
||||
...input,
|
||||
scope: { ...input.scope, consentedChange: true },
|
||||
});
|
||||
}
|
||||
if (!decision.allowed)
|
||||
throw forbidden(decision.explanation, authorizationDeniedDetails(decision));
|
||||
return bound;
|
||||
}
|
||||
@@ -0,0 +1,178 @@
|
||||
import fs from "node:fs/promises";
|
||||
import { constants } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { unprocessable } from "../errors.js";
|
||||
|
||||
export const MAX_INSTRUCTION_BYTES = 1024 * 1024;
|
||||
export function instructionPath(value: string): string {
|
||||
if (
|
||||
!value ||
|
||||
value.length > 512 ||
|
||||
value.includes("\0") ||
|
||||
value.includes("\\") ||
|
||||
path.posix.isAbsolute(value) ||
|
||||
/^[a-z]:/i.test(value) ||
|
||||
value.split("/").some((part) => !part || part === "." || part === "..")
|
||||
) {
|
||||
throw unprocessable("Use a relative instructions path without traversal", {
|
||||
code: "INSTRUCTION_PATH_INVALID",
|
||||
});
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
export function instructionBytes(content: string | Uint8Array): Buffer {
|
||||
const byteLength =
|
||||
typeof content === "string"
|
||||
? Buffer.byteLength(content, "utf8")
|
||||
: content.byteLength;
|
||||
if (byteLength > MAX_INSTRUCTION_BYTES) {
|
||||
throw unprocessable("Instructions exceed the 1 MiB limit", {
|
||||
code: "INSTRUCTION_CONTENT_INVALID",
|
||||
});
|
||||
}
|
||||
const bytes =
|
||||
typeof content === "string"
|
||||
? Buffer.from(content, "utf8")
|
||||
: Buffer.from(content);
|
||||
try {
|
||||
// Preserve BOM; reject invalid UTF-8 and unpaired JS surrogates instead of replacing bytes.
|
||||
const decoded = new TextDecoder("utf-8", {
|
||||
fatal: true,
|
||||
ignoreBOM: true,
|
||||
}).decode(bytes);
|
||||
if (typeof content === "string" && decoded !== content)
|
||||
throw new Error("invalid Unicode");
|
||||
} catch {
|
||||
throw unprocessable("Instructions must be valid UTF-8", {
|
||||
code: "INSTRUCTION_CONTENT_INVALID",
|
||||
});
|
||||
}
|
||||
return bytes;
|
||||
}
|
||||
|
||||
export async function assertInstructionPathSafe(
|
||||
root: string,
|
||||
entryFile: string,
|
||||
) {
|
||||
instructionPath(entryFile);
|
||||
let absolute = path.resolve(root, entryFile);
|
||||
// macOS exposes its system temporary directories through root-owned aliases.
|
||||
// Resolve only those fixed OS aliases, never a link in operator/agent storage.
|
||||
if (process.platform === "darwin") {
|
||||
const alias = absolute.split(path.sep)[1];
|
||||
if (alias === "var" || alias === "tmp" || alias === "etc") {
|
||||
const systemPath = `/${alias}`;
|
||||
const stat = await fs.lstat(systemPath);
|
||||
if (stat.isSymbolicLink() && stat.uid === 0 && await fs.realpath(systemPath) === `/private/${alias}`) {
|
||||
absolute = `/private${absolute}`;
|
||||
}
|
||||
}
|
||||
}
|
||||
let current = path.parse(absolute).root;
|
||||
const parts = absolute.slice(current.length).split(path.sep);
|
||||
for (const [i, part] of parts.entries()) {
|
||||
current = path.join(current, part);
|
||||
const stat = await fs
|
||||
.lstat(current)
|
||||
.catch((error: NodeJS.ErrnoException) => {
|
||||
if (error.code === "ENOENT") return null;
|
||||
throw error;
|
||||
});
|
||||
if (!stat) continue;
|
||||
if (
|
||||
stat.isSymbolicLink() ||
|
||||
(i < parts.length - 1 ? !stat.isDirectory() : !stat.isFile())
|
||||
) {
|
||||
throw unprocessable(
|
||||
"Instructions require regular files and directories without symlinks",
|
||||
{ code: "INSTRUCTION_PATH_INVALID" },
|
||||
);
|
||||
}
|
||||
}
|
||||
return absolute;
|
||||
}
|
||||
|
||||
export async function readInstructionBytes(
|
||||
root: string,
|
||||
entryFile: string,
|
||||
): Promise<Buffer | null> {
|
||||
const absolute = await assertInstructionPathSafe(root, entryFile);
|
||||
const file = await fs
|
||||
.open(absolute, constants.O_RDONLY | constants.O_NOFOLLOW)
|
||||
.catch((error: NodeJS.ErrnoException) => {
|
||||
if (error.code === "ENOENT") return null;
|
||||
throw error;
|
||||
});
|
||||
if (!file) return null;
|
||||
try {
|
||||
const stat = await file.stat();
|
||||
if (!stat.isFile() || stat.size > MAX_INSTRUCTION_BYTES) {
|
||||
throw unprocessable(
|
||||
"Instructions must be a regular UTF-8 file of at most 1 MiB",
|
||||
{ code: "INSTRUCTION_CONTENT_INVALID" },
|
||||
);
|
||||
}
|
||||
const buffer = Buffer.alloc(MAX_INSTRUCTION_BYTES + 1);
|
||||
let length = 0;
|
||||
while (length < buffer.length) {
|
||||
const result = await file.read(
|
||||
buffer,
|
||||
length,
|
||||
buffer.length - length,
|
||||
length,
|
||||
);
|
||||
if (result.bytesRead === 0) break;
|
||||
length += result.bytesRead;
|
||||
}
|
||||
return instructionBytes(buffer.subarray(0, length));
|
||||
} finally {
|
||||
await file.close();
|
||||
}
|
||||
}
|
||||
|
||||
/** Call while holding the canonical agent lock. Never writes through a symlink. */
|
||||
export async function materializeInstructionBytes(
|
||||
root: string,
|
||||
entryFile: string,
|
||||
bytes: Buffer,
|
||||
) {
|
||||
const absolute = await assertInstructionPathSafe(root, entryFile);
|
||||
await fs.mkdir(path.dirname(absolute), { recursive: true });
|
||||
await assertInstructionPathSafe(root, entryFile);
|
||||
const temporary = path.join(
|
||||
path.dirname(absolute),
|
||||
`.instruction-${randomUUID()}.tmp`,
|
||||
);
|
||||
try {
|
||||
const file = await fs.open(temporary, "wx", 0o600);
|
||||
try {
|
||||
await file.writeFile(bytes);
|
||||
await file.sync();
|
||||
} finally {
|
||||
await file.close();
|
||||
}
|
||||
await assertInstructionPathSafe(root, entryFile);
|
||||
await fs.rename(temporary, absolute);
|
||||
} finally {
|
||||
await fs.rm(temporary, { force: true });
|
||||
}
|
||||
}
|
||||
|
||||
// Keep the reserved runtime directory out of Git using a self-ignoring file
|
||||
// inside that directory. Never write to a gitdir or exclude path derived from
|
||||
// repository-controlled metadata; legitimate worktrees can keep external gitdirs.
|
||||
export const instructionGitExcludeProgram = String.raw`
|
||||
const fs=require('node:fs'),path=require('node:path'),crypto=require('node:crypto');
|
||||
const cwd=fs.realpathSync(process.argv[1]);
|
||||
const root=path.join(cwd,'.paperclip-runtime'),filename=path.join(root,'.gitignore');
|
||||
const stat=(name)=>{try{return fs.lstatSync(name)}catch(e){if(e.code==='ENOENT')return null;throw e}};
|
||||
const directory=stat(root);
|
||||
if(directory && (directory.isSymbolicLink() || !directory.isDirectory()))throw Error('Unsafe runtime exclusion directory');
|
||||
if(!directory)fs.mkdirSync(root,{mode:0o700});
|
||||
const existing=stat(filename);
|
||||
if(existing && (existing.isSymbolicLink() || !existing.isFile()))throw Error('Unsafe runtime exclusion file');
|
||||
const temporary=path.join(root,'.ignore-'+crypto.randomUUID()+'.tmp');
|
||||
try{fs.writeFileSync(temporary,'*\n',{flag:'wx',mode:0o600});fs.renameSync(temporary,filename)}finally{fs.rmSync(temporary,{force:true})}
|
||||
`;
|
||||
@@ -0,0 +1,429 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { adoptAgentFiles, agentFileToken } from "./agent-file-store.js";
|
||||
import { and, desc, eq, getTableColumns, lt, or, sql } from "drizzle-orm";
|
||||
import {
|
||||
activityLog,
|
||||
agents,
|
||||
plugins,
|
||||
pluginManagedResources,
|
||||
agentInstructionHeads as heads,
|
||||
agentInstructionRevisions as revisions,
|
||||
type Db,
|
||||
} from "@paperclipai/db";
|
||||
import type {
|
||||
AgentInstructionCommitReceipt,
|
||||
AgentInstructionDiff,
|
||||
AgentInstructionHistory,
|
||||
AgentInstructionSnapshot,
|
||||
AgentInstructionSource,
|
||||
} from "@paperclipai/shared";
|
||||
import { conflict, forbidden, notFound, unprocessable } from "../errors.js";
|
||||
import {
|
||||
agentInstructionsBundleMode,
|
||||
deriveBundleState,
|
||||
resolveManagedInstructionsRoot,
|
||||
} from "./agent-instructions.js";
|
||||
import {
|
||||
instructionBytes,
|
||||
instructionPath,
|
||||
assertInstructionPathSafe,
|
||||
readInstructionBytes,
|
||||
materializeInstructionBytes,
|
||||
} from "./agent-instruction-files.js";
|
||||
import {
|
||||
authorizeInstructionCommit,
|
||||
authorizeInstructionRead,
|
||||
} from "./agent-instruction-authorization.js";
|
||||
import type { AuthorizationActor } from "./authorization.js";
|
||||
|
||||
type Tx = Parameters<Parameters<Db["transaction"]>[0]>[0];
|
||||
type Revision = typeof revisions.$inferSelect;
|
||||
type PluginResetActor = { type: "plugin"; pluginId: string; pluginKey: string; agentKey: string };
|
||||
type RevisionActor = AuthorizationActor | PluginResetActor;
|
||||
const { contentBase64: _contentColumn, ...revisionMetadataColumns } =
|
||||
getTableColumns(revisions);
|
||||
export type InstructionTarget = { companyId: string; agentId: string };
|
||||
export type InstructionCommitInput = InstructionTarget & {
|
||||
entryFile: string;
|
||||
content: string | Uint8Array;
|
||||
baseRevisionId: string | null;
|
||||
source: Exclude<AgentInstructionSource, "seed" | "restore">;
|
||||
};
|
||||
function metadata(row: Omit<Revision, "contentBase64">) {
|
||||
const { createdAt, source, ...fields } = row;
|
||||
return {
|
||||
...fields,
|
||||
source: source as AgentInstructionSource,
|
||||
createdAt: createdAt.toISOString(),
|
||||
};
|
||||
}
|
||||
function snapshot(row: Revision): AgentInstructionSnapshot {
|
||||
const { contentBase64, ...fields } = row;
|
||||
return {
|
||||
revision: metadata(fields),
|
||||
content: Buffer.from(contentBase64, "base64").toString("utf8"),
|
||||
};
|
||||
}
|
||||
function owner(target: InstructionTarget, entryFile: string) {
|
||||
return and(
|
||||
eq(revisions.companyId, target.companyId),
|
||||
eq(revisions.agentId, target.agentId),
|
||||
eq(revisions.entryFile, entryFile),
|
||||
);
|
||||
}
|
||||
function headOwner(target: InstructionTarget, entryFile: string) {
|
||||
return and(
|
||||
eq(heads.companyId, target.companyId),
|
||||
eq(heads.agentId, target.agentId),
|
||||
eq(heads.entryFile, entryFile),
|
||||
);
|
||||
}
|
||||
|
||||
export function agentInstructionRevisionService(db: Db) {
|
||||
async function lockTarget(
|
||||
tx: Tx,
|
||||
target: InstructionTarget,
|
||||
actor?: AuthorizationActor,
|
||||
) {
|
||||
if (actor?.type === "agent" && actor.companyId !== target.companyId)
|
||||
throw notFound("Agent not found");
|
||||
const [agent] = await tx
|
||||
.select()
|
||||
.from(agents)
|
||||
.where(
|
||||
and(
|
||||
eq(agents.id, target.agentId),
|
||||
eq(agents.companyId, target.companyId),
|
||||
),
|
||||
)
|
||||
.for("update");
|
||||
if (!agent) throw notFound("Agent not found");
|
||||
if (agentInstructionsBundleMode(agent) === "external") {
|
||||
throw unprocessable(
|
||||
"External host instructions do not support revision write-back. Ask an instance administrator to migrate this bundle to managed storage.",
|
||||
{ code: "INSTRUCTION_MANAGED_BUNDLE_REQUIRED" },
|
||||
);
|
||||
}
|
||||
const state = deriveBundleState(agent);
|
||||
const entryFile = instructionPath(state.entryFile);
|
||||
return { agent, entryFile, root: resolveManagedInstructionsRoot(agent) };
|
||||
}
|
||||
async function authorizeRead(
|
||||
tx: Tx,
|
||||
actor: AuthorizationActor,
|
||||
target: InstructionTarget,
|
||||
) {
|
||||
return authorizeInstructionRead(tx, actor, { companyId: target.companyId, id: target.agentId });
|
||||
}
|
||||
// Trusted host-only reset authority. No HTTP or agent tool accepts this actor.
|
||||
async function authorizePluginReset(tx: Tx, actor: PluginResetActor, target: InstructionTarget, agent: typeof agents.$inferSelect) {
|
||||
const [plugin] = await tx.select().from(plugins).where(and(eq(plugins.id, actor.pluginId), eq(plugins.pluginKey, actor.pluginKey)));
|
||||
const marker = agent.metadata?.paperclipManagedResource as Record<string, unknown> | undefined;
|
||||
const [binding] = await tx.select().from(pluginManagedResources).where(and(
|
||||
eq(pluginManagedResources.pluginId, actor.pluginId), eq(pluginManagedResources.pluginKey, actor.pluginKey),
|
||||
eq(pluginManagedResources.companyId, target.companyId), eq(pluginManagedResources.resourceKind, "agent"),
|
||||
eq(pluginManagedResources.resourceKey, actor.agentKey), eq(pluginManagedResources.resourceId, target.agentId),
|
||||
));
|
||||
if (!plugin || plugin.status !== "ready" || !plugin.manifestJson.capabilities.includes("agents.managed")
|
||||
|| !plugin.manifestJson.agents?.some((declaration) => declaration.agentKey === actor.agentKey)
|
||||
|| !binding || marker?.pluginId !== actor.pluginId || marker.pluginKey !== actor.pluginKey
|
||||
|| marker.resourceKind !== "agent" || marker.resourceKey !== actor.agentKey) {
|
||||
throw forbidden("Plugin reset is limited to its declared, bound managed agent");
|
||||
}
|
||||
const declared = plugin.manifestJson.agents!.find((declaration) => declaration.agentKey === actor.agentKey)!;
|
||||
return { ...actor, declaredEntryFile: instructionPath(declared.instructions?.entryFile ?? "AGENTS.md") };
|
||||
}
|
||||
async function readForPluginReset(target: InstructionTarget, actor: PluginResetActor) {
|
||||
return db.transaction(async (tx) => {
|
||||
const state = await lockTarget(tx, target);
|
||||
const bound = await authorizePluginReset(tx, actor, target, state.agent);
|
||||
// Preserve the formerly configured entry before an explicit stock reset.
|
||||
await currentFile(tx, target, state, bound);
|
||||
const row = await currentFile(tx, target, { ...state, entryFile: bound.declaredEntryFile }, bound);
|
||||
return { snapshot: row ? snapshot(row) : null, configuredEntryFile: state.entryFile };
|
||||
});
|
||||
}
|
||||
async function head(tx: Tx, target: InstructionTarget, entryFile: string) {
|
||||
const [result] = await tx
|
||||
.select({ revision: revisions })
|
||||
.from(heads)
|
||||
.innerJoin(revisions, eq(revisions.id, heads.revisionId))
|
||||
.where(headOwner(target, entryFile));
|
||||
return result?.revision ?? null;
|
||||
}
|
||||
// The deployed revision tables are read-only upgrade input. Current bytes
|
||||
// live in the agent directory; old UUID clients receive a content ETag.
|
||||
async function currentFile(tx: Tx, target: InstructionTarget, state: Awaited<ReturnType<typeof lockTarget>>, actor?: RevisionActor) {
|
||||
await adoptAgentFiles(tx, state.agent);
|
||||
const bytes = await readInstructionBytes(state.root, state.entryFile);
|
||||
if (bytes === null) return null;
|
||||
return currentRow(target, state.entryFile, bytes, actor);
|
||||
}
|
||||
function currentRow(target: InstructionTarget, entryFile: string, bytes: Buffer, actor?: RevisionActor): Revision {
|
||||
return { ...target, id: agentFileToken(bytes), entryFile, contentBase64: bytes.toString("base64"),
|
||||
contentHash: createHash("sha256").update(bytes).digest("hex"), byteLength: bytes.length,
|
||||
parentRevisionId: null, baseRevisionId: null, restoredFromRevisionId: null, source: "api",
|
||||
actorAgentId: actor?.type === "agent" ? actor.agentId ?? null : null,
|
||||
actorUserId: actor?.type === "board" ? actor.userId ?? null : null,
|
||||
responsibleUserId: actor?.type === "board" ? actor.userId ?? null : actor?.type === "agent" ? actor.onBehalfOfUserId ?? null : null,
|
||||
sourceRunId: actor && actor.type !== "plugin" ? actor.runId ?? null : null, createdAt: new Date(0) };
|
||||
}
|
||||
async function getRevision(
|
||||
tx: Tx,
|
||||
target: InstructionTarget,
|
||||
entryFile: string,
|
||||
id: string,
|
||||
) {
|
||||
const [row] = await tx
|
||||
.select()
|
||||
.from(revisions)
|
||||
.where(and(owner(target, entryFile), eq(revisions.id, id)));
|
||||
if (!row) throw notFound("Instruction revision not found");
|
||||
return row;
|
||||
}
|
||||
async function readCurrent(
|
||||
target: InstructionTarget,
|
||||
actor: AuthorizationActor,
|
||||
): Promise<AgentInstructionSnapshot | null> {
|
||||
return db.transaction(async (tx) => {
|
||||
const state = await lockTarget(tx, target, actor);
|
||||
const bound = await authorizeRead(tx, actor, target);
|
||||
const row = await currentFile(tx, target, state, bound);
|
||||
return row ? snapshot(row) : null;
|
||||
});
|
||||
}
|
||||
/** Trusted orchestration read for the already authorized run target. This
|
||||
* never seeds or writes, and is not exposed through HTTP or model tools. */
|
||||
async function readCommittedForRuntime(target: InstructionTarget): Promise<AgentInstructionSnapshot | null> {
|
||||
return db.transaction(async (tx) => {
|
||||
const state = await lockTarget(tx, target);
|
||||
const current = await currentFile(tx, target, state);
|
||||
return current ? snapshot(current) : null;
|
||||
});
|
||||
}
|
||||
/** Rebuild disk from the current committed head under the same lock as commits. Safe after restart. */
|
||||
async function materializeCurrent(target: InstructionTarget): Promise<void> {
|
||||
await db.transaction(async (tx) => {
|
||||
const state = await lockTarget(tx, target);
|
||||
await adoptAgentFiles(tx, state.agent);
|
||||
});
|
||||
}
|
||||
async function commitInternal(
|
||||
input:
|
||||
| InstructionCommitInput
|
||||
| (Omit<InstructionCommitInput, "content" | "source"> & {
|
||||
restoreRevisionId: string;
|
||||
}),
|
||||
actor: RevisionActor,
|
||||
configuredEntryFile?: string,
|
||||
): Promise<AgentInstructionCommitReceipt> {
|
||||
instructionPath(input.entryFile);
|
||||
const bytes = "content" in input ? instructionBytes(input.content) : null;
|
||||
const result = await db.transaction(async (tx) => {
|
||||
const state = await lockTarget(tx, input, actor.type === "plugin" ? undefined : actor);
|
||||
const bound = actor.type === "plugin"
|
||||
? await authorizePluginReset(tx, actor, input, state.agent)
|
||||
: await authorizeInstructionCommit(tx, actor, state.agent);
|
||||
if (bound.type === "plugin") {
|
||||
if (input.entryFile !== bound.declaredEntryFile) throw forbidden("Plugin reset must restore its declared entry file");
|
||||
if (configuredEntryFile !== state.entryFile) throw conflict("Configured instruction entry changed; read the current entry and retry", { code: "INSTRUCTION_ENTRY_CHANGED", entryFile: state.entryFile });
|
||||
state.entryFile = input.entryFile;
|
||||
}
|
||||
if (state.entryFile !== input.entryFile)
|
||||
throw conflict(
|
||||
"Configured instruction entry changed; read the current entry and retry",
|
||||
{ code: "INSTRUCTION_ENTRY_CHANGED", entryFile: state.entryFile },
|
||||
);
|
||||
await assertInstructionPathSafe(state.root, state.entryFile);
|
||||
const current = await currentFile(tx, input, state, bound);
|
||||
const restored =
|
||||
"restoreRevisionId" in input
|
||||
? await getRevision(
|
||||
tx,
|
||||
input,
|
||||
input.entryFile,
|
||||
input.restoreRevisionId,
|
||||
)
|
||||
: null;
|
||||
const candidate = bytes ?? Buffer.from(restored!.contentBase64, "base64");
|
||||
const bindEntry = async () => {
|
||||
if (
|
||||
!state.agent.adapterConfig.instructionsRootPath ||
|
||||
!state.agent.adapterConfig.instructionsBundleMode ||
|
||||
(bound.type === "plugin" && configuredEntryFile !== input.entryFile)
|
||||
) {
|
||||
await tx
|
||||
.update(agents)
|
||||
.set({
|
||||
adapterConfig: {
|
||||
...state.agent.adapterConfig,
|
||||
instructionsBundleMode: "managed",
|
||||
instructionsRootPath: state.root,
|
||||
instructionsEntryFile: state.entryFile,
|
||||
instructionsFilePath: `${state.root}/${state.entryFile}`,
|
||||
},
|
||||
updatedAt: new Date(),
|
||||
})
|
||||
.where(eq(agents.id, state.agent.id));
|
||||
}
|
||||
};
|
||||
// An exact replay can safely return the durable receipt even after its base
|
||||
// advanced. Restore also deduplicates identical content, preserving history.
|
||||
if (current && current.contentBase64 === candidate.toString("base64")) {
|
||||
await bindEntry();
|
||||
return { row: current, changed: false };
|
||||
}
|
||||
// A pre-upgrade run can still pin a historical UUID. Compare its bytes,
|
||||
// never let the old head project over a newer directory.
|
||||
const legacyBase = input.baseRevisionId && current?.id !== input.baseRevisionId
|
||||
? await tx.select().from(revisions).where(and(owner(input, input.entryFile), eq(revisions.id, input.baseRevisionId))).then(rows => rows[0])
|
||||
: null;
|
||||
if ((current?.id ?? null) !== input.baseRevisionId && !(legacyBase && legacyBase.contentHash === current?.contentHash)) {
|
||||
throw conflict(
|
||||
"Instructions changed since the base revision. Read the current entry before saving.",
|
||||
{
|
||||
code: "INSTRUCTION_REVISION_CONFLICT",
|
||||
baseRevisionId: input.baseRevisionId,
|
||||
currentRevisionId: current?.id ?? null,
|
||||
},
|
||||
);
|
||||
}
|
||||
await materializeInstructionBytes(state.root, state.entryFile, candidate);
|
||||
const row = currentRow(input, input.entryFile, candidate, bound);
|
||||
await tx.insert(activityLog).values({ companyId: input.companyId,
|
||||
actorType: bound.type === "plugin" ? "plugin" : bound.type === "board" ? "user" : "agent",
|
||||
actorId: (bound.type === "plugin" ? bound.pluginId : bound.type === "board" ? bound.userId : bound.agentId)!,
|
||||
agentId: bound.type === "agent" ? bound.agentId : null, runId: row.sourceRunId,
|
||||
responsibleUserId: row.responsibleUserId, action: "agent.files_updated", entityType: "agent", entityId: input.agentId,
|
||||
details: { path: input.entryFile, contentHash: row.contentHash, source: restored ? "legacy_restore" : (input as InstructionCommitInput).source } });
|
||||
await bindEntry();
|
||||
return { row, changed: true };
|
||||
});
|
||||
return { ...snapshot(result.row), changed: result.changed, materialization: "current" };
|
||||
}
|
||||
|
||||
async function history(
|
||||
target: InstructionTarget & {
|
||||
entryFile: string;
|
||||
cursor?: string;
|
||||
limit?: number;
|
||||
},
|
||||
actor: AuthorizationActor,
|
||||
): Promise<AgentInstructionHistory> {
|
||||
const limit = Math.max(1, Math.min(100, target.limit ?? 50));
|
||||
return db.transaction(async (tx) => {
|
||||
const state = await lockTarget(tx, target, actor);
|
||||
const bound = await authorizeRead(tx, actor, target);
|
||||
await currentFile(tx, target, state, bound);
|
||||
instructionPath(target.entryFile);
|
||||
const cursor = target.cursor
|
||||
? await getRevision(tx, target, target.entryFile, target.cursor)
|
||||
: null;
|
||||
// Keep PostgreSQL timestamp precision; JS Date truncates microseconds and
|
||||
// would skip revisions in the cursor's fractional millisecond.
|
||||
const cursorTime = cursor
|
||||
? sql`(select created_at from agent_instruction_revisions where id = ${cursor.id})`
|
||||
: null;
|
||||
const rows = await tx
|
||||
.select(revisionMetadataColumns)
|
||||
.from(revisions)
|
||||
.where(
|
||||
and(
|
||||
owner(target, target.entryFile),
|
||||
cursor && cursorTime
|
||||
? or(
|
||||
lt(revisions.createdAt, cursorTime),
|
||||
and(
|
||||
eq(revisions.createdAt, cursorTime),
|
||||
lt(revisions.id, cursor.id),
|
||||
),
|
||||
)
|
||||
: undefined,
|
||||
),
|
||||
)
|
||||
.orderBy(desc(revisions.createdAt), desc(revisions.id))
|
||||
.limit(limit + 1);
|
||||
return {
|
||||
revisions: rows.slice(0, limit).map(metadata),
|
||||
nextCursor: rows.length > limit ? rows[limit - 1]!.id : null,
|
||||
};
|
||||
});
|
||||
}
|
||||
async function readRevision(
|
||||
target: InstructionTarget & { entryFile: string; revisionId: string },
|
||||
actor: AuthorizationActor,
|
||||
) {
|
||||
return db.transaction(async (tx) => {
|
||||
await lockTarget(tx, target, actor);
|
||||
await authorizeRead(tx, actor, target);
|
||||
return snapshot(
|
||||
await getRevision(
|
||||
tx,
|
||||
target,
|
||||
instructionPath(target.entryFile),
|
||||
target.revisionId,
|
||||
),
|
||||
);
|
||||
});
|
||||
}
|
||||
async function diff(
|
||||
target: InstructionTarget & {
|
||||
entryFile: string;
|
||||
fromRevisionId: string;
|
||||
toRevisionId: string;
|
||||
},
|
||||
actor: AuthorizationActor,
|
||||
): Promise<AgentInstructionDiff> {
|
||||
const from = await readRevision(
|
||||
{ ...target, revisionId: target.fromRevisionId },
|
||||
actor,
|
||||
);
|
||||
const to = await readRevision(
|
||||
{ ...target, revisionId: target.toRevisionId },
|
||||
actor,
|
||||
);
|
||||
// Linear, bounded exact replacement diff. Code points keep surrogate pairs intact.
|
||||
const a = Array.from(from.content),
|
||||
b = Array.from(to.content);
|
||||
let start = 0,
|
||||
end = 0;
|
||||
while (start < a.length && start < b.length && a[start] === b[start])
|
||||
start++;
|
||||
while (
|
||||
end < a.length - start &&
|
||||
end < b.length - start &&
|
||||
a[a.length - end - 1] === b[b.length - end - 1]
|
||||
)
|
||||
end++;
|
||||
return {
|
||||
from,
|
||||
to,
|
||||
prefix: a.slice(0, start).join(""),
|
||||
removed: a.slice(start, a.length - end).join(""),
|
||||
added: b.slice(start, b.length - end).join(""),
|
||||
suffix: end ? a.slice(-end).join("") : "",
|
||||
};
|
||||
}
|
||||
return {
|
||||
readCommittedForRuntime,
|
||||
readCurrent,
|
||||
readForPluginReset,
|
||||
commitPluginReset: (input: Omit<InstructionCommitInput, "source"> & { configuredEntryFile: string }, actor: PluginResetActor) =>
|
||||
commitInternal({ ...input, source: "api" }, actor, input.configuredEntryFile),
|
||||
readRevision,
|
||||
history,
|
||||
diff,
|
||||
materializeCurrent,
|
||||
commit: (input: InstructionCommitInput, actor: AuthorizationActor) =>
|
||||
commitInternal(input, actor),
|
||||
restore: (
|
||||
input: InstructionTarget & {
|
||||
entryFile: string;
|
||||
baseRevisionId: string;
|
||||
revisionId: string;
|
||||
},
|
||||
actor: AuthorizationActor,
|
||||
) =>
|
||||
commitInternal({ ...input, restoreRevisionId: input.revisionId }, actor),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,356 @@
|
||||
import { agentDirectoryWorkingCopyService, isAgentDirectoryCopy } from "./agent-directory-working-copies.js";
|
||||
import { createHash } from "node:crypto";
|
||||
import { execFile as execFileCallback } from "node:child_process";
|
||||
import { promisify } from "node:util";
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { and, asc, desc, eq, inArray, lte, or, isNull, isNotNull, sql } from "drizzle-orm";
|
||||
import { agents, heartbeatRuns, agentInstructionWorkingCopies as copies, type Db } from "@paperclipai/db";
|
||||
import {
|
||||
prepareAdapterExecutionTargetRuntime,
|
||||
runAdapterExecutionTargetShellCommand,
|
||||
type AdapterExecutionTarget,
|
||||
} from "@paperclipai/adapter-utils/execution-target";
|
||||
import { conflict, notFound } from "../errors.js";
|
||||
import { agentInstructionsService, agentInstructionsBundleMode } from "./agent-instructions.js";
|
||||
import { agentInstructionRevisionService } from "./agent-instruction-revisions.js";
|
||||
import { resolveInstructionActor } from "./agent-instruction-authorization.js";
|
||||
import { assertInstructionPathSafe, instructionBytes, instructionPath, materializeInstructionBytes, MAX_INSTRUCTION_BYTES, readInstructionBytes, instructionGitExcludeProgram } from "./agent-instruction-files.js";
|
||||
import { hasNativeLocalProcessStop } from "./native-local-process-stop.js";
|
||||
import { remoteExecutionHasStopped } from "./remote-execution-termination.js";
|
||||
import type { AuthorizationActor } from "./authorization.js";
|
||||
import type { EnvironmentRuntimeService } from "./environment-runtime.js";
|
||||
|
||||
const execFile = promisify(execFileCallback);
|
||||
type Copy = typeof copies.$inferSelect;
|
||||
const hash = (bytes: Uint8Array) => createHash("sha256").update(bytes).digest("hex");
|
||||
const quote = (value: string) => `'${value.replaceAll("'", `'"'"'`)}'`;
|
||||
const completed = new Set(["saved", "unchanged", "resolved"]);
|
||||
const MAX_COLLECTION_ATTEMPTS = 3;
|
||||
// Only orchestration can register a live target; tools and HTTP callers cannot
|
||||
// supply one. These handles are optional: durable captured bytes survive restart.
|
||||
const liveTargets = new Map<string, AdapterExecutionTarget>();
|
||||
const targetKey = (companyId: string, runId: string) => `${companyId}:${runId}`;
|
||||
|
||||
export function instructionWorkingCopyGuidance(copy: Pick<Copy, "executionRoot" | "entryFile" | "receipt">) {
|
||||
if (isAgentDirectoryCopy(copy)) return `Your persistent agent directory is ${copy.executionRoot} (AGENT_HOME). Your instruction entry is ${copy.executionRoot}/${copy.entryFile}. Read and write your own files and subfolders there. This directory belongs to this agent across tasks and sessions; task files belong in the task working directory. Paperclip restores this directory before execution and saves changes after the provider stops. Regular files, including binary files, persist; symlinks and special files are unsupported. Check the agent-files save receipt before claiming persistence; Only files you change or delete are synchronized. If another run changes the same file, the last completed synchronization wins. Temporary copies are removed after synchronization; there is no per-run file history. Storage allows 256 MiB per file, 2 GiB total, and 100,000 entries; the instruction entry must remain UTF-8 and at most 1 MiB. Reaching a storage limit never prevents this or future tasks from running. Remove or shrink files to free space; changes that exceed the limits will not be saved.${typeof copy.receipt?.storageWarning === "string" ? `\n\n${copy.receipt.storageWarning}` : ""}`;
|
||||
return `Your editable agent instruction file is ${copy.executionRoot}/${copy.entryFile}. Edit this registered private copy normally. After this run stops, Paperclip saves changed content as a persistent revision if your responsible user still has permission and the baseline has not changed. Check the run's instruction-save receipt before claiming persistence. Use read_agent_instructions, update_agent_instructions, get_agent_instruction_history, and restore_agent_instructions for immediate saves and history. Read first and pin the returned revision. Preserve conflicts; never silently retry against a newer head. Repository instructions, skills, and the loaded prompt are separate and are not collected.`;
|
||||
}
|
||||
|
||||
/** Remote reads use the registered entry only; never scan for files called AGENTS.md. */
|
||||
export function instructionCollectionScript(root: string, entry: string) {
|
||||
instructionPath(entry);
|
||||
return `node -e ${quote([
|
||||
"const fs=require('node:fs'),path=require('node:path')",
|
||||
"const root=process.argv[1],entry=process.argv[2],max=Number(process.argv[3])",
|
||||
"const absolute=path.resolve(root,entry),parts=absolute.split('/').filter(Boolean)",
|
||||
"let current='/'",
|
||||
"for(let i=0;i<parts.length;i++){current=path.join(current,parts[i]);let s;try{s=fs.lstatSync(current)}catch(e){if(e.code==='ENOENT'){process.stdout.write(JSON.stringify({missing:true}));process.exit(0)}throw e}if(s.isSymbolicLink()||(i<parts.length-1?!s.isDirectory():!s.isFile()))throw Error('INSTRUCTION_PATH_INVALID')}",
|
||||
"const fd=fs.openSync(absolute,fs.constants.O_RDONLY|fs.constants.O_NOFOLLOW)",
|
||||
"try{const s=fs.fstatSync(fd);if(!s.isFile()||s.size>max)throw Error('INSTRUCTION_CONTENT_INVALID');const bytes=Buffer.alloc(max+1);let n=0;while(n<bytes.length){const count=fs.readSync(fd,bytes,n,bytes.length-n,n);if(!count)break;n+=count}if(n>max)throw Error('INSTRUCTION_CONTENT_INVALID');process.stdout.write(JSON.stringify({contentBase64:bytes.subarray(0,n).toString('base64')}))}finally{fs.closeSync(fd)}",
|
||||
].join(";"))} ${quote(root)} ${quote(entry)} ${MAX_INSTRUCTION_BYTES}`;
|
||||
}
|
||||
|
||||
|
||||
export function agentInstructionWorkingCopyService(db: Db, options: { environmentRuntime?: EnvironmentRuntimeService } = {}) {
|
||||
const revisions = agentInstructionRevisionService(db);
|
||||
const scope = (companyId: string, runId: string) => and(eq(copies.companyId, companyId), eq(copies.runId, runId));
|
||||
async function get(companyId: string, runId: string) {
|
||||
const [row] = await db.select().from(copies).where(scope(companyId, runId));
|
||||
return row ?? null;
|
||||
}
|
||||
async function patch(row: Copy, values: Partial<typeof copies.$inferInsert>) {
|
||||
const [updated] = await db.update(copies).set({ ...values, updatedAt: new Date() }).where(and(
|
||||
scope(row.companyId, row.runId), eq(copies.state, row.state), eq(copies.attempts, row.attempts), eq(copies.baseHash, row.baseHash),
|
||||
)).returning();
|
||||
// A late cleanup must not overwrite an explicit resolution or a newer
|
||||
// baseline. Canonical content has its own independent head CAS.
|
||||
return updated ?? (await get(row.companyId, row.runId))!;
|
||||
}
|
||||
function actorFor(row: Copy): AuthorizationActor {
|
||||
return { type: "agent", companyId: row.companyId, agentId: row.agentId, runId: row.runId, onBehalfOfUserId: row.responsibleUserId };
|
||||
}
|
||||
const directories = agentDirectoryWorkingCopyService(db, get, patch, options.environmentRuntime);
|
||||
async function prepare(input: { companyId: string; agentId: string; runId: string; target?: AdapterExecutionTarget | null; cwd: string; legacy?: boolean }) {
|
||||
const existing = await get(input.companyId, input.runId);
|
||||
if (isAgentDirectoryCopy(existing) || (!existing && !input.legacy)) return directories.prepare(input);
|
||||
let refreshStoppedCopy = false;
|
||||
const workspace = await fs.realpath(input.cwd);
|
||||
const expectedLocalRoot = path.join(workspace, ".paperclip-runtime", `instruction-edits-${input.runId}`, "instructions");
|
||||
if (existing && existing.localRoot !== expectedLocalRoot) throw conflict("The registered instruction copy belongs to a different run workspace");
|
||||
const location = input.target?.kind === "remote" ? `remote:${input.target.environmentId ?? ""}` : "local";
|
||||
if (existing && (existing.agentId !== input.agentId || existing.location !== location)) {
|
||||
throw conflict("The registered instruction copy belongs to a different run environment");
|
||||
}
|
||||
// Preserve in-flight bytes and their CAS baseline. A completed, stopped copy
|
||||
// can start a fresh lifecycle only after its recorded bytes are accounted for.
|
||||
if (existing && existing.state !== "preparing") {
|
||||
if (["conflict", "pending_commit", "unavailable"].includes(existing.state)) {
|
||||
throw conflict("Resolve this run's preserved instruction candidate before editing its working copy again");
|
||||
}
|
||||
if (input.target) liveTargets.set(targetKey(input.companyId, input.runId), input.target);
|
||||
if (completed.has(existing.state) && existing.processStoppedAt) {
|
||||
const privateBytes = await readCandidate(existing, input.target);
|
||||
// Refresh only bytes whose previous lifecycle is complete. A changed
|
||||
// private file must keep its old CAS fence; never silently rebase edits.
|
||||
refreshStoppedCopy = privateBytes === null || hash(privateBytes) === (existing.candidateHash ?? existing.baseHash);
|
||||
if (!refreshStoppedCopy && privateBytes !== null) {
|
||||
// A failed staging attempt can already have replaced the file before
|
||||
// the completed row advances. Exact current canonical bytes are also
|
||||
// accounted for; any other private edit still keeps its old fence.
|
||||
const current = await revisions.readCommittedForRuntime({ companyId: input.companyId, agentId: input.agentId });
|
||||
refreshStoppedCopy = current?.revision.entryFile === existing.entryFile
|
||||
&& current.revision.contentHash === hash(privateBytes);
|
||||
}
|
||||
}
|
||||
if (!refreshStoppedCopy) {
|
||||
if (completed.has(existing.state) || existing.state === "unchanged_turn") {
|
||||
// unchanged_turn is a live warm-owner observation, not stop proof.
|
||||
return patch(existing, { state: "prepared", candidateBase64: null, candidateHash: null,
|
||||
receipt: null, processStoppedAt: null, attempts: 0, nextAttemptAt: null });
|
||||
}
|
||||
return existing;
|
||||
}
|
||||
}
|
||||
const [agent] = await db.select().from(agents).where(and(eq(agents.id, input.agentId), eq(agents.companyId, input.companyId)));
|
||||
if (!agent) throw notFound("Agent not found");
|
||||
if (agentInstructionsBundleMode(agent) !== "managed") return null;
|
||||
const bound = await resolveInstructionActor(db, { type: "agent", companyId: input.companyId, agentId: input.agentId, runId: input.runId });
|
||||
const baseline = existing?.baseRevisionId && !refreshStoppedCopy
|
||||
? await revisions.readRevision({ companyId: input.companyId, agentId: input.agentId, entryFile: existing.entryFile, revisionId: existing.baseRevisionId }, bound).catch(async (error) => {
|
||||
if ((error as { status?: number }).status !== 404) throw error;
|
||||
const current = await revisions.readCurrent({ companyId: input.companyId, agentId: input.agentId }, bound);
|
||||
if (current?.revision.id !== existing.baseRevisionId) throw conflict("The instruction baseline changed during preparation");
|
||||
return current;
|
||||
})
|
||||
: await revisions.readCurrent({ companyId: input.companyId, agentId: input.agentId }, bound);
|
||||
if (!baseline) return null;
|
||||
const exported = await agentInstructionsService().exportFiles(agent, { rejectSymlinks: true });
|
||||
const entryFile = instructionPath(baseline?.revision.entryFile ?? exported.entryFile);
|
||||
const content = instructionBytes(baseline?.content ?? exported.files[entryFile] ?? "");
|
||||
const localRoot = existing?.localRoot ?? expectedLocalRoot;
|
||||
const target = input.target?.kind === "remote" ? input.target : null;
|
||||
const executionRoot = target
|
||||
? path.posix.join(target.remoteCwd, ".paperclip-runtime", `instruction-edits-${input.runId}`, "instructions")
|
||||
: localRoot;
|
||||
if (!existing) {
|
||||
await db.insert(copies).values({
|
||||
runId: input.runId, companyId: input.companyId, agentId: input.agentId,
|
||||
responsibleUserId: bound.onBehalfOfUserId!, entryFile,
|
||||
baseRevisionId: baseline?.revision.id ?? null, baseHash: hash(content),
|
||||
localRoot, executionRoot, location, state: "preparing",
|
||||
});
|
||||
}
|
||||
const row = (await get(input.companyId, input.runId))!;
|
||||
await assertInstructionPathSafe(localRoot, entryFile);
|
||||
await execFile(process.execPath, ["-e", instructionGitExcludeProgram, workspace], { timeout: 15_000 });
|
||||
await fs.mkdir(localRoot, { recursive: true, mode: 0o700 });
|
||||
for (const [name, text] of Object.entries({ ...exported.files, [entryFile]: content.toString("utf8") })) {
|
||||
const relative = instructionPath(name);
|
||||
const filename = await assertInstructionPathSafe(localRoot, relative);
|
||||
await fs.mkdir(path.dirname(filename), { recursive: true, mode: 0o700 });
|
||||
// Atomic replacement also allows interrupted preparation to restage an
|
||||
// already-read-only sibling without writing through its existing inode.
|
||||
await materializeInstructionBytes(localRoot, relative, instructionBytes(text));
|
||||
await fs.chmod(filename, name === entryFile ? 0o600 : 0o400);
|
||||
}
|
||||
if (target) {
|
||||
const staged = await prepareAdapterExecutionTargetRuntime({
|
||||
target, runId: input.runId, adapterKey: `instruction-edits-${input.runId}`,
|
||||
workspaceLocalDir: input.cwd, syncWorkspace: false,
|
||||
assets: [{ key: "instructions", localDir: localRoot, followSymlinks: false }],
|
||||
});
|
||||
if (staged.assetDirs.instructions !== executionRoot) throw new Error("Instruction copy staging path changed");
|
||||
const excluded = await runAdapterExecutionTargetShellCommand(input.runId, target,
|
||||
`node -e ${quote(instructionGitExcludeProgram)} ${quote(target.remoteCwd)}`, { cwd: target.remoteCwd, env: {}, timeoutSec: 15 });
|
||||
if (excluded.exitCode !== 0 || excluded.timedOut) throw new Error("Could not exclude private instructions from Git staging");
|
||||
}
|
||||
// A refresh does not supersede the completed turn until every local and
|
||||
// remote staging step succeeds. Keep its receipt and stop evidence intact
|
||||
// if preparation throws or the controller stops midway through staging.
|
||||
const prepared = await patch(row, { state: "prepared", ...(refreshStoppedCopy ? {
|
||||
entryFile, baseRevisionId: baseline.revision.id, baseHash: hash(content),
|
||||
candidateBase64: null, candidateHash: null, receipt: null, processStoppedAt: null,
|
||||
attempts: 0, nextAttemptAt: null, errorCode: null, errorMessage: null,
|
||||
} : {}) });
|
||||
if (input.target) liveTargets.set(targetKey(input.companyId, input.runId), input.target);
|
||||
return prepared;
|
||||
}
|
||||
|
||||
async function readCandidate(row: Copy, target?: AdapterExecutionTarget | null) {
|
||||
if (row.location === "local") return readInstructionBytes(row.localRoot, row.entryFile);
|
||||
if (target?.kind !== "remote") throw new Error("The original execution environment is needed to retrieve this instruction copy");
|
||||
if (row.location !== `remote:${target.environmentId ?? ""}`) throw new Error("Instruction copy execution environment changed");
|
||||
const expected = path.posix.join(target.remoteCwd, ".paperclip-runtime", `instruction-edits-${row.runId}`, "instructions");
|
||||
if (expected !== row.executionRoot) throw new Error("Instruction copy execution environment changed");
|
||||
const result = await runAdapterExecutionTargetShellCommand(row.runId, target, instructionCollectionScript(row.executionRoot, row.entryFile), { cwd: target.remoteCwd, env: {}, timeoutSec: 30 });
|
||||
if (result.exitCode !== 0 || result.timedOut) throw new Error("Could not safely retrieve the stopped run's instruction file");
|
||||
const payload = JSON.parse(result.stdout) as { missing?: boolean; contentBase64?: string };
|
||||
if (payload.missing) return null;
|
||||
if (typeof payload.contentBase64 !== "string" || payload.contentBase64.length > Math.ceil(MAX_INSTRUCTION_BYTES / 3) * 4) throw new Error("Instruction response exceeds the collection bound");
|
||||
return instructionBytes(Buffer.from(payload.contentBase64, "base64"));
|
||||
}
|
||||
|
||||
/** A terminal-turn probe is not a save: changed/invalid copies require a
|
||||
* stopped provider before capture. Unchanged warm providers remain reusable. */
|
||||
async function hasChanges(input: { companyId: string; runId: string; target?: AdapterExecutionTarget | null }) {
|
||||
const row = await get(input.companyId, input.runId);
|
||||
if (!row || completed.has(row.state)) return false;
|
||||
if (isAgentDirectoryCopy(row)) return directories.hasChanges(row, input.target);
|
||||
try {
|
||||
const content = await readCandidate(row, input.target);
|
||||
if (content !== null && hash(content) === row.baseHash) {
|
||||
// This is an unchanged turn receipt, not stopped-process collection.
|
||||
// No candidate bytes or new revision are persisted from a live owner.
|
||||
await patch(row, { state: "unchanged_turn", nextAttemptAt: null });
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
} catch { return true; }
|
||||
}
|
||||
|
||||
async function acknowledgeExplicitSave(input: { companyId: string; agentId: string; runId: string; entryFile: string; revisionId: string; contentHash: string }) {
|
||||
const row = await get(input.companyId, input.runId);
|
||||
if (!row || isAgentDirectoryCopy(row) || row.agentId !== input.agentId || row.entryFile !== input.entryFile || row.state !== "prepared") return;
|
||||
const target = liveTargets.get(targetKey(row.companyId, row.runId));
|
||||
const content = await readCandidate(row, target).catch(() => null);
|
||||
// Never advance a stale local edit past an unrelated explicit update.
|
||||
if (content === null || hash(content) !== input.contentHash) return;
|
||||
await db.update(copies).set({ baseRevisionId: input.revisionId, baseHash: input.contentHash, updatedAt: new Date() })
|
||||
.where(and(scope(row.companyId, row.runId), eq(copies.state, "prepared"), eq(copies.baseHash, row.baseHash)));
|
||||
}
|
||||
|
||||
async function commitCandidate(row: Copy) {
|
||||
if (isAgentDirectoryCopy(row)) return directories.collectStopped(row);
|
||||
if (row.candidateBase64 === null || completed.has(row.state) || row.state === "conflict") return row;
|
||||
try {
|
||||
const receipt = await revisions.commit({
|
||||
companyId: row.companyId, agentId: row.agentId, entryFile: row.entryFile,
|
||||
baseRevisionId: row.baseRevisionId, content: Buffer.from(row.candidateBase64, "base64"), source: "cleanup",
|
||||
}, actorFor(row));
|
||||
return patch(row, { state: "saved", receipt: { ...receipt }, errorCode: null, errorMessage: null, nextAttemptAt: null });
|
||||
} catch (error) {
|
||||
const detail = error as { status?: number; details?: { code?: string }; message?: string };
|
||||
const code = detail.details?.code ?? "INSTRUCTION_SAVE_FAILED";
|
||||
const retryable = !detail.status || detail.status >= 500;
|
||||
const message = detail.status === 403
|
||||
? "Current permissions do not allow this instruction save. The candidate was preserved."
|
||||
: detail.status === 409
|
||||
? "Instructions changed after this run started. Review the preserved candidate against the current revision."
|
||||
: "Instruction edits were preserved but not saved. Review the candidate before retrying.";
|
||||
return patch(row, { state: retryable ? "pending_commit" : "conflict", errorCode: code, errorMessage: message, nextAttemptAt: retryable && row.attempts < MAX_COLLECTION_ATTEMPTS ? new Date(Date.now() + 30_000) : null });
|
||||
}
|
||||
}
|
||||
|
||||
/** Caller must have joined the owned provider process before granting this proof. */
|
||||
async function collectStopped(input: { companyId: string; runId: string; target?: AdapterExecutionTarget | null }) {
|
||||
let row = await get(input.companyId, input.runId);
|
||||
if (!row) return row;
|
||||
if (isAgentDirectoryCopy(row)) return directories.collectStopped(row, input.target);
|
||||
if (completed.has(row.state) || row.state === "conflict") return row;
|
||||
row = await patch(row, { processStoppedAt: row.processStoppedAt ?? new Date(), attempts: row.attempts + 1 });
|
||||
if (completed.has(row.state) || row.state === "conflict") return row;
|
||||
if (row.candidateBase64 !== null) return commitCandidate(row);
|
||||
try {
|
||||
const candidate = await readCandidate(row, input.target);
|
||||
if (candidate === null) {
|
||||
return patch(row, { state: "unavailable", errorCode: "INSTRUCTION_FILE_MISSING", errorMessage: "The registered instruction file was removed. Canonical instructions were kept; no edits were saved.", nextAttemptAt: null });
|
||||
}
|
||||
if (hash(candidate) === row.baseHash) return patch(row, { state: "unchanged", nextAttemptAt: null });
|
||||
// Save the candidate durably before attempting authorization/CAS or releasing the environment.
|
||||
row = await patch(row, { state: "pending_commit", candidateBase64: candidate.toString("base64"), candidateHash: hash(candidate), nextAttemptAt: new Date() });
|
||||
return commitCandidate(row);
|
||||
} catch (error) {
|
||||
return patch(row, { state: row.attempts < MAX_COLLECTION_ATTEMPTS ? "pending_collection" : "unavailable", errorCode: "INSTRUCTION_COLLECTION_FAILED", errorMessage: "The registered instruction copy could not be read safely. Canonical instructions were kept; no save receipt exists.", nextAttemptAt: row.attempts < MAX_COLLECTION_ATTEMPTS ? new Date(Date.now() + 30_000) : null });
|
||||
}
|
||||
}
|
||||
|
||||
/** Restarts can retry persisted bytes without touching a live filesystem or launching a model. */
|
||||
async function recoverCaptured() {
|
||||
const pending = await db.select().from(copies).where(and(
|
||||
eq(copies.state, "pending_commit"),
|
||||
or(isNull(copies.nextAttemptAt), lte(copies.nextAttemptAt, new Date())),
|
||||
lte(copies.attempts, MAX_COLLECTION_ATTEMPTS - 1),
|
||||
)).limit(20);
|
||||
for (const row of pending) {
|
||||
const result = await commitCandidate(await patch(row, { attempts: row.attempts + 1 }));
|
||||
if (isAgentDirectoryCopy(result) && completed.has(result.state)) await directories.release(result);
|
||||
}
|
||||
// A crash between a durable save and release must not retain run snapshots
|
||||
// forever. Only discard copies with recorded stop proof and a terminal receipt.
|
||||
const cleanup = await db.select().from(copies).where(and(
|
||||
inArray(copies.state, [...completed, "unavailable"]), isNotNull(copies.processStoppedAt),
|
||||
sql`${copies.receipt}->>'schema' = 'paperclip.agent-files.v1'`,
|
||||
or(sql`${copies.receipt} ? 'baseline'`, sql`${copies.receipt}->>'cleanupPending' = 'true'`),
|
||||
or(isNull(copies.nextAttemptAt), lte(copies.nextAttemptAt, new Date())),
|
||||
)).orderBy(asc(copies.updatedAt)).limit(20);
|
||||
for (const row of cleanup) await directories.release(row);
|
||||
return pending.length;
|
||||
}
|
||||
|
||||
/** Recover only the registered copy after independently recorded process-stop
|
||||
* proof. Terminal run status is never a substitute for that proof. Remote
|
||||
* receipts mean the environment is stopped; do not restart it just to read. */
|
||||
async function recoverStopped() {
|
||||
const pending = await db.select({ copy: copies, runtimeMode: heartbeatRuns.runtimeMode }).from(copies)
|
||||
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.companyId, copies.companyId), eq(heartbeatRuns.id, copies.runId)))
|
||||
.where(and(or(inArray(copies.state, ["prepared", "pending_collection"]),
|
||||
and(eq(copies.state, "preparing"), sql`${copies.receipt}->>'schema' = 'paperclip.agent-files.v1'`)),
|
||||
inArray(heartbeatRuns.status, ["succeeded", "failed", "cancelled", "timed_out", "interrupted"]),
|
||||
lte(copies.attempts, MAX_COLLECTION_ATTEMPTS - 1))).limit(20);
|
||||
for (const { copy: row, runtimeMode } of pending) {
|
||||
if (row.state === "preparing" && isAgentDirectoryCopy(row)) {
|
||||
// A provider cannot launch until preparation records "prepared". With
|
||||
// the owning run terminal, this is an interrupted staging copy only.
|
||||
await directories.release(await patch(row, { state: "unavailable", processStoppedAt: new Date(),
|
||||
errorCode: "AGENT_FILES_PREPARE_FAILED", errorMessage: "Agent-file preparation was interrupted. The temporary copy was discarded.", nextAttemptAt: null }));
|
||||
} else if (row.location === "local" && (row.processStoppedAt || runtimeMode === "native" &&
|
||||
await hasNativeLocalProcessStop(db, row.companyId, row.runId))) {
|
||||
const result = await collectStopped({ companyId: row.companyId, runId: row.runId });
|
||||
if (result && isAgentDirectoryCopy(result) && completed.has(result.state)) await directories.release(result);
|
||||
} else if (row.location !== "local" && await remoteExecutionHasStopped(db, row.companyId, row.runId)) {
|
||||
const unavailable = await reportUnavailable(row.companyId, row.runId);
|
||||
if (unavailable && isAgentDirectoryCopy(unavailable)) {
|
||||
await directories.release(await patch(unavailable, { processStoppedAt: unavailable.processStoppedAt ?? new Date() }));
|
||||
}
|
||||
} else if (row.state === "prepared") {
|
||||
await patch(row, { state: "pending_collection", errorCode: "INSTRUCTION_STOP_UNCONFIRMED",
|
||||
errorMessage: "The provider's stop has not been confirmed. Instruction collection is pending; no save is claimed.", nextAttemptAt: null });
|
||||
}
|
||||
}
|
||||
return pending.length;
|
||||
}
|
||||
|
||||
async function list(companyId: string, agentId: string, actor: AuthorizationActor) {
|
||||
await revisions.readCurrent({ companyId, agentId }, actor);
|
||||
const rows = await db.select().from(copies).where(and(eq(copies.companyId, companyId), eq(copies.agentId, agentId), inArray(copies.state, ["conflict", "pending_collection", "pending_commit", "unavailable"]))).orderBy(desc(copies.createdAt)).limit(50);
|
||||
return rows.map(row => ({ runId: row.runId, entryFile: row.entryFile, baseRevisionId: row.baseRevisionId,
|
||||
baseHash: row.baseHash, state: row.state as "conflict" | "pending_collection" | "pending_commit" | "unavailable",
|
||||
candidateHash: row.candidateHash, contract: isAgentDirectoryCopy(row) ? "agent_files" : "legacy",
|
||||
content: row.candidateBase64 === null ? null : Buffer.from(row.candidateBase64, "base64").toString("utf8"),
|
||||
errorCode: row.errorCode, errorMessage: row.errorMessage, createdAt: row.createdAt.toISOString(), updatedAt: row.updatedAt.toISOString() }));
|
||||
}
|
||||
|
||||
async function resolve(input: { companyId: string; agentId: string; runId: string; baseRevisionId: string | null; content: string }, actor: AuthorizationActor) {
|
||||
const row = await get(input.companyId, input.runId);
|
||||
if (!row || row.agentId !== input.agentId) throw notFound("Instruction candidate not found");
|
||||
if (completed.has(row.state)) throw conflict("This instruction candidate has already been resolved");
|
||||
if (isAgentDirectoryCopy(row)) throw conflict("Agent-file synchronization failures have no preserved candidate");
|
||||
const receipt = await revisions.commit({ companyId: input.companyId, agentId: input.agentId, entryFile: row.entryFile, content: input.content, baseRevisionId: input.baseRevisionId, source: "api" }, actor);
|
||||
await patch(row, { state: "resolved", receipt: { ...receipt }, nextAttemptAt: null });
|
||||
return receipt;
|
||||
}
|
||||
|
||||
/** A controller may report loss before disposal, never fabricate a save or
|
||||
* read a possibly live provider. Already captured candidates remain intact. */
|
||||
async function reportUnavailable(companyId: string, runId: string) {
|
||||
const row = await get(companyId, runId);
|
||||
if (!row || completed.has(row.state) || row.state === "unchanged_turn" || row.candidateBase64 !== null || (isAgentDirectoryCopy(row) && row.candidateHash !== null) || row.state === "conflict") return row;
|
||||
if (isAgentDirectoryCopy(row) && row.state === "unavailable") return row;
|
||||
return patch(row, { state: "unavailable", errorCode: "INSTRUCTION_COLLECTION_UNAVAILABLE",
|
||||
errorMessage: "The registered instruction copy could not be retrieved safely before environment release. No instruction save is claimed.", nextAttemptAt: null });
|
||||
}
|
||||
|
||||
async function release(companyId: string, runId: string) { liveTargets.delete(targetKey(companyId, runId)); const row = await get(companyId, runId); if (row && isAgentDirectoryCopy(row)) await directories.release(row); }
|
||||
return { prepare, get, hasChanges, acknowledgeExplicitSave, collectStopped, recoverCaptured, recoverStopped, list, resolve, reportUnavailable, release };
|
||||
}
|
||||
@@ -1,5 +1,9 @@
|
||||
import { inspectAgentFile, fileHash, agentFilePath, MAX_AGENT_FILE_BYTES } from "./agent-file-store.js";
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { agentInstructionHeads, agents, type Db } from "@paperclipai/db";
|
||||
import { instructionPath, assertInstructionPathSafe, instructionBytes, readInstructionBytes } from "./agent-instruction-files.js";
|
||||
import { notFound, unprocessable } from "../errors.js";
|
||||
import { resolveHomeAwarePath, resolvePaperclipInstanceRoot } from "../home-paths.js";
|
||||
|
||||
@@ -36,6 +40,8 @@ type AgentLike = {
|
||||
|
||||
type AgentInstructionsFileSummary = {
|
||||
path: string;
|
||||
contentHash?: string;
|
||||
binary?: boolean;
|
||||
size: number;
|
||||
language: string;
|
||||
markdown: boolean;
|
||||
@@ -53,6 +59,7 @@ type AgentInstructionsFileDetail = AgentInstructionsFileSummary & {
|
||||
type AgentInstructionsBundle = {
|
||||
agentId: string;
|
||||
companyId: string;
|
||||
persistence?: "agent_files";
|
||||
mode: BundleMode | null;
|
||||
rootPath: string | null;
|
||||
managedRootPath: string;
|
||||
@@ -112,11 +119,7 @@ function isMarkdown(relativePath: string) {
|
||||
}
|
||||
|
||||
function normalizeRelativeFilePath(candidatePath: string): string {
|
||||
const normalized = path.posix.normalize(candidatePath.replaceAll("\\", "/")).replace(/^\/+/, "");
|
||||
if (!normalized || normalized === "." || normalized === ".." || normalized.startsWith("../")) {
|
||||
throw unprocessable("Instructions file path must stay within the bundle root");
|
||||
}
|
||||
return normalized;
|
||||
return instructionPath(candidatePath);
|
||||
}
|
||||
|
||||
function resolvePathWithinRoot(rootPath: string, relativePath: string): string {
|
||||
@@ -130,7 +133,7 @@ function resolvePathWithinRoot(rootPath: string, relativePath: string): string {
|
||||
return absolutePath;
|
||||
}
|
||||
|
||||
function resolveManagedInstructionsRoot(agent: AgentLike): string {
|
||||
export function resolveManagedInstructionsRoot(agent: AgentLike): string {
|
||||
return path.resolve(
|
||||
resolvePaperclipInstanceRoot(),
|
||||
"companies",
|
||||
@@ -153,7 +156,7 @@ function resolveLegacyInstructionsPath(candidatePath: string, config: Record<str
|
||||
}
|
||||
|
||||
async function statIfExists(targetPath: string) {
|
||||
return fs.stat(targetPath).catch(() => null);
|
||||
return fs.lstat(targetPath).catch(() => null);
|
||||
}
|
||||
|
||||
function shouldIgnoreInstructionsEntry(entry: { name: string; isDirectory(): boolean; isFile(): boolean }) {
|
||||
@@ -172,14 +175,14 @@ function shouldIgnoreInstructionsEntry(entry: { name: string; isDirectory(): boo
|
||||
|
||||
async function listFilesRecursive(
|
||||
rootPath: string,
|
||||
options?: { rejectSymlinks?: boolean },
|
||||
options?: { rejectSymlinks?: boolean; legacyExcludes?: boolean },
|
||||
): Promise<string[]> {
|
||||
const output: string[] = [];
|
||||
|
||||
async function walk(currentPath: string, relativeDir: string) {
|
||||
const entries = await fs.readdir(currentPath, { withFileTypes: true }).catch(() => []);
|
||||
for (const entry of entries) {
|
||||
if (shouldIgnoreInstructionsEntry(entry)) continue;
|
||||
if (options?.legacyExcludes && shouldIgnoreInstructionsEntry(entry)) continue;
|
||||
const absolutePath = path.join(currentPath, entry.name);
|
||||
const relativePath = normalizeRelativeFilePath(
|
||||
relativeDir ? path.posix.join(relativeDir, entry.name) : entry.name,
|
||||
@@ -206,13 +209,21 @@ async function listFilesRecursive(
|
||||
async function readFileSummary(rootPath: string, relativePath: string, entryFile: string): Promise<AgentInstructionsFileSummary> {
|
||||
const absolutePath = resolvePathWithinRoot(rootPath, relativePath);
|
||||
const stat = await fs.stat(absolutePath);
|
||||
// External bundles may contain large assets; listing must not read them.
|
||||
const file = stat.size > MAX_AGENT_FILE_BYTES ? null : await inspectAgentFile(rootPath, relativePath);
|
||||
return summarizeFile(relativePath, entryFile, file?.size ?? stat.size, file?.bytes ?? null, file?.hash);
|
||||
}
|
||||
|
||||
function summarizeFile(relativePath: string, entryFile: string, size: number, bytes: Buffer | null, hash?: string): AgentInstructionsFileSummary {
|
||||
let binary = bytes === null;
|
||||
try { if (bytes?.includes(0)) binary = true; new TextDecoder("utf-8", { fatal: true }).decode(bytes ?? undefined); } catch { binary = true; }
|
||||
return {
|
||||
path: relativePath,
|
||||
size: stat.size,
|
||||
size,
|
||||
language: inferLanguage(relativePath),
|
||||
markdown: isMarkdown(relativePath),
|
||||
isEntryFile: relativePath === entryFile,
|
||||
editable: true,
|
||||
editable: !binary, binary, contentHash: hash,
|
||||
deprecated: false,
|
||||
virtual: false,
|
||||
};
|
||||
@@ -231,7 +242,7 @@ async function readLegacyInstructions(agent: AgentLike, config: Record<string, u
|
||||
return asString(config[PROMPT_KEY]) ?? "";
|
||||
}
|
||||
|
||||
function deriveBundleState(agent: AgentLike): BundleState {
|
||||
export function deriveBundleState(agent: AgentLike): BundleState {
|
||||
const config = asRecord(agent.adapterConfig);
|
||||
const warnings: string[] = [];
|
||||
const storedModeRaw = config[MODE_KEY];
|
||||
@@ -245,21 +256,23 @@ function deriveBundleState(agent: AgentLike): BundleState {
|
||||
const storedEntryRaw = asString(config[ENTRY_KEY]);
|
||||
if (storedEntryRaw) {
|
||||
try {
|
||||
entryFile = normalizeRelativeFilePath(storedEntryRaw);
|
||||
} catch {
|
||||
warnings.push(`Ignored invalid instructions entry file "${storedEntryRaw}".`);
|
||||
// Historical config accepted normalized relative paths. Keep that read
|
||||
// compatibility; new API writes still use strict instructionPath validation.
|
||||
entryFile = normalizeRelativeFilePath(path.posix.normalize(storedEntryRaw.replaceAll("\\", "/")).replace(/^\/+/, ""));
|
||||
} catch (error) {
|
||||
warnings.push(error instanceof Error ? error.message : String(error));
|
||||
}
|
||||
}
|
||||
|
||||
if (!rootPath && legacyInstructionsPath) {
|
||||
try {
|
||||
const resolvedLegacyPath = resolveLegacyInstructionsPath(legacyInstructionsPath, config);
|
||||
rootPath = path.dirname(resolvedLegacyPath);
|
||||
entryFile = path.basename(resolvedLegacyPath);
|
||||
mode = resolvedLegacyPath.startsWith(`${resolveManagedInstructionsRoot(agent)}${path.sep}`)
|
||||
|| resolvedLegacyPath === path.join(resolveManagedInstructionsRoot(agent), entryFile)
|
||||
? "managed"
|
||||
: "external";
|
||||
const managedRoot = resolveManagedInstructionsRoot(agent);
|
||||
const managedRelative = path.relative(managedRoot, resolvedLegacyPath);
|
||||
const inManagedRoot = managedRelative !== ".." && !managedRelative.startsWith(`..${path.sep}`) && !path.isAbsolute(managedRelative);
|
||||
rootPath = inManagedRoot ? managedRoot : path.dirname(resolvedLegacyPath);
|
||||
entryFile = inManagedRoot ? instructionPath(managedRelative.split(path.sep).join("/")) : path.basename(resolvedLegacyPath);
|
||||
mode = inManagedRoot ? "managed" : "external";
|
||||
if (!path.isAbsolute(legacyInstructionsPath)) {
|
||||
warnings.push("Using legacy relative instructionsFilePath; migrate this agent to a managed or absolute external bundle.");
|
||||
}
|
||||
@@ -303,7 +316,7 @@ async function recoverManagedBundleState(agent: AgentLike, state: BundleState):
|
||||
const files = await listFilesRecursive(managedRootPath);
|
||||
if (files.length === 0) return state;
|
||||
|
||||
const recoveredEntryFile = files.includes(state.entryFile)
|
||||
const recoveredEntryFile = asString(state.config[ENTRY_KEY]) ? state.entryFile : files.includes(state.entryFile)
|
||||
? state.entryFile
|
||||
: files.includes(ENTRY_FILE_DEFAULT)
|
||||
? ENTRY_FILE_DEFAULT
|
||||
@@ -370,6 +383,7 @@ function toBundle(agent: AgentLike, state: BundleState, files: AgentInstructions
|
||||
return {
|
||||
agentId: agent.id,
|
||||
companyId: agent.companyId,
|
||||
...(state.mode === "managed" ? { persistence: "agent_files" as const } : {}),
|
||||
mode: state.mode,
|
||||
rootPath: state.rootPath,
|
||||
managedRootPath: resolveManagedInstructionsRoot(agent),
|
||||
@@ -443,7 +457,8 @@ async function writeBundleFiles(
|
||||
) {
|
||||
for (const [relativePath, content] of Object.entries(files)) {
|
||||
const normalizedPath = normalizeRelativeFilePath(relativePath);
|
||||
const absolutePath = resolvePathWithinRoot(rootPath, normalizedPath);
|
||||
instructionBytes(content);
|
||||
const absolutePath = await assertInstructionPathSafe(rootPath, normalizedPath);
|
||||
const existingStat = await statIfExists(absolutePath);
|
||||
if (existingStat?.isFile() && !options?.overwriteExisting) continue;
|
||||
await fs.mkdir(path.dirname(absolutePath), { recursive: true });
|
||||
@@ -464,19 +479,26 @@ export function syncInstructionsBundleConfigFromFilePath(
|
||||
return next;
|
||||
}
|
||||
const resolvedPath = resolveLegacyInstructionsPath(instructionsFilePath, adapterConfig);
|
||||
const rootPath = path.dirname(resolvedPath);
|
||||
const entryFile = path.basename(resolvedPath);
|
||||
const mode: BundleMode = resolvedPath.startsWith(`${resolveManagedInstructionsRoot(agent)}${path.sep}`)
|
||||
|| resolvedPath === path.join(resolveManagedInstructionsRoot(agent), entryFile)
|
||||
? "managed"
|
||||
: "external";
|
||||
const managedRoot = resolveManagedInstructionsRoot(agent);
|
||||
const relative = path.relative(managedRoot, resolvedPath);
|
||||
const managed = relative !== ".." && !relative.startsWith(`..${path.sep}`) && !path.isAbsolute(relative);
|
||||
const rootPath = managed ? managedRoot : path.dirname(resolvedPath);
|
||||
const entryFile = managed ? instructionPath(relative.split(path.sep).join("/")) : path.basename(resolvedPath);
|
||||
const mode: BundleMode = managed ? "managed" : "external";
|
||||
return applyBundleConfig(next, { mode, rootPath, entryFile });
|
||||
}
|
||||
|
||||
export function agentInstructionsService() {
|
||||
export function agentInstructionsService(db?: Db) {
|
||||
async function assertUnversionedEntry(agent: AgentLike, entryFile: string, connection: Db | Parameters<Parameters<Db["transaction"]>[0]>[0] | undefined = db) {
|
||||
if (!connection) throw unprocessable("Bundle initialization requires the database-backed instructions service");
|
||||
const [head] = await connection.select().from(agentInstructionHeads).where(and(eq(agentInstructionHeads.companyId, agent.companyId),
|
||||
eq(agentInstructionHeads.agentId, agent.id), eq(agentInstructionHeads.entryFile, entryFile)));
|
||||
if (head) throw unprocessable("This entry has revision history. Use the instruction content API with its baseRevisionId to update it.", { code: "INSTRUCTION_REVISION_REQUIRED" });
|
||||
}
|
||||
async function getBundle(agent: AgentLike): Promise<AgentInstructionsBundle> {
|
||||
const state = await recoverManagedBundleState(agent, deriveBundleState(agent));
|
||||
if (!state.rootPath) return toBundle(agent, state, []);
|
||||
await assertInstructionPathSafe(state.rootPath, state.entryFile);
|
||||
const stat = await statIfExists(state.rootPath);
|
||||
if (!stat?.isDirectory()) {
|
||||
return toBundle(agent, {
|
||||
@@ -484,8 +506,12 @@ export function agentInstructionsService() {
|
||||
warnings: [...state.warnings, `Instructions root does not exist: ${state.rootPath}`],
|
||||
}, []);
|
||||
}
|
||||
const files = await listFilesRecursive(state.rootPath);
|
||||
const summaries = await Promise.all(files.map((relativePath) => readFileSummary(state.rootPath!, relativePath, state.entryFile)));
|
||||
const files = await listFilesRecursive(state.rootPath, { legacyExcludes: state.mode === "external" });
|
||||
const summaries: AgentInstructionsFileSummary[] = [];
|
||||
// Bound open descriptors and text buffers even for a large personal folder.
|
||||
for (let index = 0; index < files.length; index += 8) {
|
||||
summaries.push(...await Promise.all(files.slice(index, index + 8).map(relativePath => readFileSummary(state.rootPath!, relativePath, state.entryFile))));
|
||||
}
|
||||
return toBundle(agent, state, summaries);
|
||||
}
|
||||
|
||||
@@ -507,24 +533,11 @@ export function agentInstructionsService() {
|
||||
};
|
||||
}
|
||||
if (!state.rootPath) throw notFound("Agent instructions bundle is not configured");
|
||||
const absolutePath = resolvePathWithinRoot(state.rootPath, relativePath);
|
||||
const [content, stat] = await Promise.all([
|
||||
fs.readFile(absolutePath, "utf8").catch(() => null),
|
||||
fs.stat(absolutePath).catch(() => null),
|
||||
]);
|
||||
if (content === null || !stat?.isFile()) throw notFound("Instructions file not found");
|
||||
const normalizedPath = normalizeRelativeFilePath(relativePath);
|
||||
return {
|
||||
path: normalizedPath,
|
||||
size: stat.size,
|
||||
language: inferLanguage(normalizedPath),
|
||||
markdown: isMarkdown(normalizedPath),
|
||||
isEntryFile: normalizedPath === state.entryFile,
|
||||
editable: true,
|
||||
deprecated: false,
|
||||
virtual: false,
|
||||
content,
|
||||
};
|
||||
await assertInstructionPathSafe(state.rootPath, relativePath);
|
||||
const file = await inspectAgentFile(state.rootPath, relativePath);
|
||||
if (file === null) throw notFound("Instructions file not found");
|
||||
const summary = summarizeFile(relativePath, state.entryFile, file.size, file.bytes, file.hash);
|
||||
return { ...summary, content: summary.binary ? "" : file.bytes!.toString("utf8") };
|
||||
}
|
||||
|
||||
async function ensureWritableBundle(
|
||||
@@ -551,11 +564,12 @@ export function agentInstructionsService() {
|
||||
});
|
||||
await fs.mkdir(managedRoot, { recursive: true });
|
||||
|
||||
const entryPath = resolvePathWithinRoot(managedRoot, entryFile);
|
||||
const entryPath = await assertInstructionPathSafe(managedRoot, entryFile);
|
||||
const entryStat = await statIfExists(entryPath);
|
||||
if (!entryStat?.isFile()) {
|
||||
const legacyInstructions = await readLegacyInstructions(agent, current.config);
|
||||
if (legacyInstructions.trim().length > 0) {
|
||||
await assertUnversionedEntry(agent, entryFile);
|
||||
await fs.mkdir(path.dirname(entryPath), { recursive: true });
|
||||
await fs.writeFile(entryPath, legacyInstructions, "utf8");
|
||||
}
|
||||
@@ -595,9 +609,13 @@ export function agentInstructionsService() {
|
||||
nextRootPath = resolvedRoot;
|
||||
}
|
||||
|
||||
await assertInstructionPathSafe(nextRootPath, nextEntryFile);
|
||||
await fs.mkdir(nextRootPath, { recursive: true });
|
||||
|
||||
const existingFiles = await listFilesRecursive(nextRootPath);
|
||||
if (nextMode === "managed" && !existingFiles.includes(nextEntryFile)) {
|
||||
await assertUnversionedEntry(agent, nextEntryFile);
|
||||
}
|
||||
const exported = await exportFiles(agent);
|
||||
if (existingFiles.length === 0) {
|
||||
await writeBundleFiles(nextRootPath, exported.files);
|
||||
@@ -618,7 +636,7 @@ export function agentInstructionsService() {
|
||||
return { bundle: nextBundle, adapterConfig: nextConfig };
|
||||
}
|
||||
|
||||
async function writeFile(
|
||||
async function writeFileUnversioned(
|
||||
agent: AgentLike,
|
||||
relativePath: string,
|
||||
content: string,
|
||||
@@ -642,7 +660,14 @@ export function agentInstructionsService() {
|
||||
return { bundle, file, adapterConfig };
|
||||
}
|
||||
|
||||
const configured = await recoverManagedBundleState(agent, current);
|
||||
if (normalizeRelativeFilePath(relativePath) === configured.entryFile) {
|
||||
throw unprocessable("Entry edits require the canonical instruction commit service and baseRevisionId", { code: "INSTRUCTION_REVISION_REQUIRED" });
|
||||
}
|
||||
if (configured.mode !== "external") agentFilePath(relativePath);
|
||||
const prepared = await ensureWritableBundle(agent, options);
|
||||
instructionBytes(content);
|
||||
await assertInstructionPathSafe(prepared.state.rootPath!, relativePath);
|
||||
const absolutePath = resolvePathWithinRoot(prepared.state.rootPath!, relativePath);
|
||||
await fs.mkdir(path.dirname(absolutePath), { recursive: true });
|
||||
await fs.writeFile(absolutePath, content, "utf8");
|
||||
@@ -654,7 +679,7 @@ export function agentInstructionsService() {
|
||||
return { bundle, file, adapterConfig: prepared.adapterConfig };
|
||||
}
|
||||
|
||||
async function deleteFile(agent: AgentLike, relativePath: string): Promise<{
|
||||
async function deleteFileUnversioned(agent: AgentLike, relativePath: string): Promise<{
|
||||
bundle: AgentInstructionsBundle;
|
||||
adapterConfig: Record<string, unknown>;
|
||||
}> {
|
||||
@@ -668,13 +693,35 @@ export function agentInstructionsService() {
|
||||
if (normalizedPath === state.entryFile) {
|
||||
throw unprocessable("Cannot delete the bundle entry file");
|
||||
}
|
||||
const absolutePath = resolvePathWithinRoot(state.rootPath, normalizedPath);
|
||||
const absolutePath = await assertInstructionPathSafe(state.rootPath, normalizedPath);
|
||||
await fs.rm(absolutePath, { force: true });
|
||||
const adapterConfig = buildPersistedBundleConfig(derived, state);
|
||||
const bundle = await getBundle({ ...agent, adapterConfig });
|
||||
return { bundle, adapterConfig };
|
||||
}
|
||||
|
||||
// Reload configuration under the commit lock: a previously supporting file
|
||||
// may have become the configured entry since the route loaded its agent.
|
||||
async function withCurrentAgent<T>(agent: AgentLike, relativePath: string, operation: (current: AgentLike) => Promise<T>) {
|
||||
if (!db) return operation(agent);
|
||||
return db.transaction(async (tx) => {
|
||||
const [current] = await tx.select().from(agents)
|
||||
.where(and(eq(agents.id, agent.id), eq(agents.companyId, agent.companyId))).for("update");
|
||||
if (!current) throw notFound("Agent not found");
|
||||
if (relativePath !== LEGACY_PROMPT_TEMPLATE_PATH) {
|
||||
// Historical entries also remain projections if the configured entry changes.
|
||||
await assertUnversionedEntry(current, normalizeRelativeFilePath(relativePath), tx);
|
||||
}
|
||||
return operation(current);
|
||||
});
|
||||
}
|
||||
async function writeFile(...args: Parameters<typeof writeFileUnversioned>) {
|
||||
return withCurrentAgent(args[0], args[1], (current) => writeFileUnversioned(current, args[1], args[2], args[3]));
|
||||
}
|
||||
async function deleteFile(...args: Parameters<typeof deleteFileUnversioned>) {
|
||||
return withCurrentAgent(args[0], args[1], (current) => deleteFileUnversioned(current, args[1]));
|
||||
}
|
||||
|
||||
async function exportFiles(agent: AgentLike, options?: { rejectSymlinks?: boolean }): Promise<{
|
||||
files: Record<string, string>;
|
||||
entryFile: string;
|
||||
@@ -684,7 +731,7 @@ export function agentInstructionsService() {
|
||||
if (state.rootPath) {
|
||||
const stat = await statIfExists(state.rootPath);
|
||||
if (stat?.isDirectory()) {
|
||||
const relativePaths = await listFilesRecursive(state.rootPath, options);
|
||||
const relativePaths = await listFilesRecursive(state.rootPath, { ...options, legacyExcludes: true });
|
||||
const files = Object.fromEntries(await Promise.all(relativePaths.map(async (relativePath) => {
|
||||
const absolutePath = resolvePathWithinRoot(state.rootPath!, relativePath);
|
||||
const content = await fs.readFile(absolutePath, "utf8");
|
||||
@@ -704,7 +751,7 @@ export function agentInstructionsService() {
|
||||
};
|
||||
}
|
||||
|
||||
async function materializeManagedBundle(
|
||||
async function materializeManagedBundleUnversioned(
|
||||
agent: AgentLike,
|
||||
files: Record<string, string>,
|
||||
options?: {
|
||||
@@ -716,6 +763,14 @@ export function agentInstructionsService() {
|
||||
const rootPath = resolveManagedInstructionsRoot(agent);
|
||||
const entryFile = options?.entryFile ? normalizeRelativeFilePath(options.entryFile) : ENTRY_FILE_DEFAULT;
|
||||
|
||||
for (const [relativePath, content] of Object.entries(files)) {
|
||||
instructionBytes(content);
|
||||
await assertInstructionPathSafe(rootPath, agentFilePath(relativePath));
|
||||
}
|
||||
const previous = await readInstructionBytes(rootPath, entryFile);
|
||||
if (previous && !previous.equals(instructionBytes(files[entryFile] ?? ""))) {
|
||||
throw unprocessable("Existing entry content must be saved through the canonical revision API before replacing a bundle", { code: "INSTRUCTION_REVISION_REQUIRED" });
|
||||
}
|
||||
if (options?.replaceExisting) {
|
||||
await fs.rm(rootPath, { recursive: true, force: true });
|
||||
}
|
||||
@@ -744,6 +799,24 @@ export function agentInstructionsService() {
|
||||
return { bundle, adapterConfig };
|
||||
}
|
||||
|
||||
async function materializeManagedBundle(
|
||||
...args: Parameters<typeof materializeManagedBundleUnversioned>
|
||||
) {
|
||||
if (!db) throw unprocessable("Bundle initialization requires the database-backed instructions service");
|
||||
return db.transaction(async (tx) => {
|
||||
const agent = args[0];
|
||||
const [owner] = await tx.select({ id: agents.id }).from(agents).where(and(eq(agents.id, agent.id), eq(agents.companyId, agent.companyId))).for("update");
|
||||
if (!owner) throw notFound("Agent not found");
|
||||
await assertUnversionedEntry(agent, args[2]?.entryFile ? normalizeRelativeFilePath(args[2].entryFile) : ENTRY_FILE_DEFAULT, tx);
|
||||
if (args[2]?.replaceExisting) {
|
||||
const [existingHead] = await tx.select({ id: agentInstructionHeads.revisionId }).from(agentInstructionHeads)
|
||||
.where(and(eq(agentInstructionHeads.companyId, agent.companyId), eq(agentInstructionHeads.agentId, agent.id))).limit(1);
|
||||
if (existingHead) throw unprocessable("Replacing this bundle would remove a versioned entry. Use canonical content commits.", { code: "INSTRUCTION_REVISION_REQUIRED" });
|
||||
}
|
||||
return materializeManagedBundleUnversioned(...args);
|
||||
});
|
||||
}
|
||||
|
||||
return {
|
||||
getBundle,
|
||||
readFile,
|
||||
|
||||
@@ -62,6 +62,7 @@ export type AuthorizationAction =
|
||||
| PermissionKey
|
||||
| "agent_config:read"
|
||||
| "agent_config:update"
|
||||
| "agent_instructions:update"
|
||||
| "skill_config:update"
|
||||
| "agent:read"
|
||||
| "agent:wake"
|
||||
@@ -148,7 +149,7 @@ function companyIdForResource(resource: AuthorizationResource) {
|
||||
}
|
||||
|
||||
function permissionForAction(action: AuthorizationAction): PermissionKey | null {
|
||||
if (action === "agent_config:read" || action === "agent_config:update" || action === "skill_config:update") {
|
||||
if (action === "agent_config:read" || action === "agent_config:update" || action === "agent_instructions:update" || action === "skill_config:update") {
|
||||
return null;
|
||||
}
|
||||
if (
|
||||
@@ -998,6 +999,7 @@ export function authorizationService(db: Db | DbTransaction) {
|
||||
input.action === "decision_triage:manage" ||
|
||||
input.action === "agent_config:read" ||
|
||||
input.action === "agent_config:update" ||
|
||||
input.action === "agent_instructions:update" ||
|
||||
input.action === "skill_config:update" ||
|
||||
input.action === "inbox:manage" ||
|
||||
input.action === "runtime:manage" ||
|
||||
@@ -1734,7 +1736,7 @@ export function authorizationService(db: Db | DbTransaction) {
|
||||
if (input.action === "agent_config:read") {
|
||||
return decideWithAgentConfigReadGrant("user", input.actor.userId);
|
||||
}
|
||||
if (input.action === "agent_config:update") {
|
||||
if (input.action === "agent_config:update" || input.action === "agent_instructions:update") {
|
||||
return decideWithProtectedChangeGrants("user", input.actor.userId, {
|
||||
direct: "agents:configure",
|
||||
suggest: "agents:suggest-changes",
|
||||
@@ -2227,6 +2229,19 @@ export function authorizationService(db: Db | DbTransaction) {
|
||||
return decideWithAgentConfigReadGrant("agent", actorAgentId);
|
||||
}
|
||||
|
||||
if (input.action === "agent_instructions:update") {
|
||||
if (!isSimpleAssignableAgentStatus(actorAgent.status) || !input.actor.onBehalfOfUserId) {
|
||||
return deny({ action: input.action, reason: "deny_missing_membership", explanation: "Instruction edits require an active agent and a responsible user." });
|
||||
}
|
||||
// Explicit configure/suggest restrictions still govern content changes.
|
||||
// Only self edits may fall back to the responsible user's target access.
|
||||
const restricted = await decideWithProtectedChangeGrants("agent", actorAgentId, {
|
||||
direct: "agents:configure", suggest: "agents:suggest-changes",
|
||||
});
|
||||
if (restricted.reason !== "deny_no_grant" || input.resource.type !== "agent" || input.resource.agentId !== actorAgentId) return restricted;
|
||||
return allow({ action: input.action, reason: "allow_self", explanation: "Own instruction content edit, subject to the responsible user's target edit access." });
|
||||
}
|
||||
|
||||
if (input.action === "agent_config:update") {
|
||||
return decideWithProtectedChangeGrants("agent", actorAgentId, {
|
||||
direct: "agents:configure",
|
||||
@@ -2391,7 +2406,7 @@ export function authorizationService(db: Db | DbTransaction) {
|
||||
responsibleUserId,
|
||||
}, "responsible-user authorization intersection denied");
|
||||
|
||||
return responsibleUserAuthzShadowMode() ? agentDecision : denied;
|
||||
return input.action !== "agent_instructions:update" && responsibleUserAuthzShadowMode() ? agentDecision : denied;
|
||||
}
|
||||
|
||||
async function decide(input: {
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import fs from "node:fs/promises";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { createRequire } from "node:module";
|
||||
import path from "node:path";
|
||||
@@ -5,12 +6,16 @@ import { fileURLToPath } from "node:url";
|
||||
import { readPaperclipSkillSyncPreference, writePaperclipSkillSyncPreference } from "@paperclipai/adapter-utils/server-utils";
|
||||
import { and, desc, eq, ne } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { agents, builtInManagedResources, companies, issueThreadInteractions, issues, routines, routineTriggers } from "@paperclipai/db";
|
||||
import { activityLog, agents, builtInManagedResources, companies, issueThreadInteractions, issues, routines, routineTriggers } from "@paperclipai/db";
|
||||
import { syncRoutineVariablesWithTemplate } from "@paperclipai/shared";
|
||||
import type { Agent, Approval, CompanySkill, PermissionKey, Routine, RoutineTrigger, RoutineVariable } from "@paperclipai/shared";
|
||||
import { conflict, HttpError, notFound, unprocessable } from "../errors.js";
|
||||
import { logActivity } from "./activity-log.js";
|
||||
import { agentInstructionsService } from "./agent-instructions.js";
|
||||
import { adoptAgentFiles, agentFilePath, fileHash, readAgentFile, snapshotAgentFiles } from "./agent-file-store.js";
|
||||
import { assertInstructionPathSafe, instructionBytes, materializeInstructionBytes } from "./agent-instruction-files.js";
|
||||
import { agentInstructionRevisionService } from "./agent-instruction-revisions.js";
|
||||
import type { AuthorizationActor } from "./authorization.js";
|
||||
import { agentInstructionsBundleMode, agentInstructionsService } from "./agent-instructions.js";
|
||||
import { agentService } from "./agents.js";
|
||||
import { approvalService } from "./approvals.js";
|
||||
import {
|
||||
@@ -802,7 +807,7 @@ export function builtInAgentService(db: Db) {
|
||||
const agentSvc = agentService(db);
|
||||
const accessSvc = accessService(db);
|
||||
const approvalSvc = approvalService(db);
|
||||
const instructionsSvc = agentInstructionsService();
|
||||
const instructionsSvc = agentInstructionsService(db);
|
||||
const skillSvc = companySkillService(db);
|
||||
const routineSvc = routineService(db);
|
||||
|
||||
@@ -916,9 +921,9 @@ export function builtInAgentService(db: Db) {
|
||||
stockVersion: string;
|
||||
stockHash: string;
|
||||
defaultsJson: Record<string, unknown>;
|
||||
}) {
|
||||
}, connection: Db | Parameters<Parameters<Db["transaction"]>[0]>[0] = db) {
|
||||
const now = new Date();
|
||||
return db
|
||||
return connection
|
||||
.insert(builtInManagedResources)
|
||||
.values(input)
|
||||
.onConflictDoUpdate({
|
||||
@@ -952,10 +957,15 @@ export function builtInAgentService(db: Db) {
|
||||
return currentFiles;
|
||||
}
|
||||
|
||||
async function materializeInstructions(agent: Agent, definition: BuiltInAgentDefinition, mode: "reconcile" | "reset") {
|
||||
async function materializeInstructions(agent: Agent, definition: BuiltInAgentDefinition, mode: "reconcile" | "reset", actor?: AuthorizationActor) {
|
||||
const bundle = definition.bundle!;
|
||||
const stock = stockHash(bundle.instructions.files);
|
||||
const binding = await getManagedResourceBinding(agent.companyId, definition.key, "instructions", "AGENTS.md");
|
||||
// Stock operations also read the committed projection after interrupted writes.
|
||||
if (agentInstructionsBundleMode(agent) === "managed") {
|
||||
try { await agentInstructionRevisionService(db).materializeCurrent({ companyId: agent.companyId, agentId: agent.id }); }
|
||||
catch (error) { if (!(error instanceof HttpError && error.status === 404)) throw error; }
|
||||
}
|
||||
const currentFiles = await currentInstructionFiles(agent, bundle);
|
||||
const currentHash = Object.values(currentFiles).some((value) => value === null) ? null : stockHash(currentFiles);
|
||||
const currentState = stockState({
|
||||
@@ -969,6 +979,100 @@ export function builtInAgentService(db: Db) {
|
||||
changedFiles: changedFileList(currentFiles, bundle.instructions.files),
|
||||
});
|
||||
|
||||
if (!actor && mode === "reconcile" && agentInstructionsBundleMode(agent) === "managed"
|
||||
&& binding && (binding.stockHash !== stock || binding.defaultsJson.pendingInstructionsUpdate)) {
|
||||
const bindingWhere = and(eq(builtInManagedResources.companyId, agent.companyId),
|
||||
eq(builtInManagedResources.bundleKey, definition.key), eq(builtInManagedResources.resourceKind, "instructions"),
|
||||
eq(builtInManagedResources.resourceKey, "AGENTS.md"));
|
||||
const incoming = Object.fromEntries(Object.entries(bundle.instructions.files)
|
||||
.map(([file, content]) => [agentFilePath(file), instructionBytes(content)]));
|
||||
// Persist only the pending operation's baseline hashes before touching
|
||||
// files. If a write or DB commit fails, the next reconciliation can accept
|
||||
// already-applied bytes and retry. This is not instruction revision history.
|
||||
const prepared = await db.transaction(async (tx) => {
|
||||
const [current] = await tx.select().from(agents).where(and(eq(agents.companyId, agent.companyId), eq(agents.id, agent.id))).for("update");
|
||||
if (!current) throw notFound("Built-in agent not found");
|
||||
if (agentInstructionsBundleMode(current) !== "managed") return false;
|
||||
const root = await adoptAgentFiles(tx, current);
|
||||
const [latest] = await tx.select().from(builtInManagedResources).where(bindingWhere);
|
||||
if (!latest) return false;
|
||||
const pending = latest.defaultsJson.pendingInstructionsUpdate as {
|
||||
stockHash?: string; baseHashes?: Record<string, string | null>; nextHashes?: Record<string, string | null>;
|
||||
} | undefined;
|
||||
if (pending?.stockHash === stock) return true;
|
||||
if (!pending && latest.stockHash === stock) return false;
|
||||
const previousPaths = latest.defaultsJson.files;
|
||||
if (!Array.isArray(previousPaths) || !previousPaths.every((file): file is string => typeof file === "string")) return false;
|
||||
const ownedPaths = new Set(previousPaths);
|
||||
if (pending) {
|
||||
// A newer release (or rollback) can supersede an interrupted update.
|
||||
// Recognize only the previous operation's baseline or intended bytes;
|
||||
// never mistake an intervening operator edit for partially applied stock.
|
||||
if (!pending.baseHashes || !pending.nextHashes) return false;
|
||||
for (const [file, baseHash] of Object.entries(pending.baseHashes)) {
|
||||
if (!(file in pending.nextHashes)) return false;
|
||||
const bytes = await readAgentFile(root, agentFilePath(file));
|
||||
const hash = bytes === null ? null : fileHash(bytes);
|
||||
if (hash !== baseHash && hash !== pending.nextHashes[file]) return false;
|
||||
ownedPaths.add(file);
|
||||
}
|
||||
} else {
|
||||
const previousFiles: Record<string, string | null> = {};
|
||||
for (const file of previousPaths) {
|
||||
try { previousFiles[file] = (await instructionsSvc.readFile(current, file)).content; }
|
||||
catch { previousFiles[file] = null; }
|
||||
}
|
||||
if (stockHash(previousFiles) !== latest.stockHash) return false;
|
||||
}
|
||||
await snapshotAgentFiles(root);
|
||||
const baseHashes: Record<string, string | null> = {};
|
||||
const nextHashes: Record<string, string | null> = {};
|
||||
for (const file of new Set([...ownedPaths, ...Object.keys(incoming)])) {
|
||||
const bytes = await readAgentFile(root, agentFilePath(file));
|
||||
// A newly declared stock path must not overwrite a personal file.
|
||||
if (!ownedPaths.has(file) && bytes !== null && !bytes.equals(incoming[file]!)) return false;
|
||||
baseHashes[file] = bytes === null ? null : fileHash(bytes);
|
||||
nextHashes[file] = incoming[file] ? fileHash(incoming[file]) : null;
|
||||
}
|
||||
await tx.update(builtInManagedResources).set({ defaultsJson: { ...latest.defaultsJson,
|
||||
pendingInstructionsUpdate: { stockHash: stock, baseHashes, nextHashes } }, updatedAt: new Date() }).where(bindingWhere);
|
||||
return true;
|
||||
});
|
||||
if (!prepared) return currentState;
|
||||
return db.transaction(async (tx) => {
|
||||
const [current] = await tx.select().from(agents).where(and(eq(agents.companyId, agent.companyId), eq(agents.id, agent.id))).for("update");
|
||||
if (!current) throw notFound("Built-in agent not found");
|
||||
if (agentInstructionsBundleMode(current) !== "managed") return currentState;
|
||||
const root = await adoptAgentFiles(tx, current);
|
||||
const [latest] = await tx.select().from(builtInManagedResources).where(bindingWhere);
|
||||
const pending = latest?.defaultsJson.pendingInstructionsUpdate as { stockHash?: string; baseHashes?: Record<string, string | null> } | undefined;
|
||||
if (pending?.stockHash !== stock || !pending.baseHashes) return currentState;
|
||||
await snapshotAgentFiles(root);
|
||||
// Preflight every changed or removed path before mutation; preserve any
|
||||
// operator edit made between preparation, failure, and this retry.
|
||||
for (const [file, baseHash] of Object.entries(pending.baseHashes)) {
|
||||
const bytes = await readAgentFile(root, agentFilePath(file));
|
||||
const hash = bytes === null ? null : fileHash(bytes);
|
||||
const nextHash = incoming[file] ? fileHash(incoming[file]) : null;
|
||||
if (hash !== baseHash && hash !== nextHash) return currentState;
|
||||
}
|
||||
for (const [file, bytes] of Object.entries(incoming)) await materializeInstructionBytes(root, file, bytes);
|
||||
for (const file of Object.keys(pending.baseHashes)) {
|
||||
if (!(file in incoming)) await fs.rm(await assertInstructionPathSafe(root, file), { force: true });
|
||||
}
|
||||
await upsertManagedResourceBinding({ companyId: agent.companyId, bundleKey: definition.key,
|
||||
resourceKind: "instructions", resourceKey: "AGENTS.md", resourceId: agent.id,
|
||||
stockVersion: bundle.stockVersion, stockHash: stock,
|
||||
defaultsJson: { entryFile: bundle.instructions.entryFile, files: Object.keys(bundle.instructions.files) },
|
||||
}, tx);
|
||||
await tx.insert(activityLog).values({ companyId: agent.companyId, actorType: "system", actorId: "built-in-reconcile",
|
||||
action: "agent.files_updated", entityType: "agent", entityId: agent.id,
|
||||
details: { source: "built-in-stock-update", bundleKey: definition.key, stockHash: stock } });
|
||||
return stockState({ resourceKind: "instructions", resourceKey: "AGENTS.md", resourceId: agent.id,
|
||||
stockVersion: bundle.stockVersion, latestStockHash: stock, currentHash: stock, bindingStockHash: stock });
|
||||
});
|
||||
}
|
||||
|
||||
const shouldWrite =
|
||||
mode === "reset"
|
||||
|| currentState.stockStatus === "missing"
|
||||
@@ -992,13 +1096,36 @@ export function builtInAgentService(db: Db) {
|
||||
return currentState;
|
||||
}
|
||||
|
||||
const materialized = await instructionsSvc.materializeManagedBundle(agent, bundle.instructions.files, {
|
||||
entryFile: bundle.instructions.entryFile,
|
||||
replaceExisting: true,
|
||||
clearLegacyPromptTemplate: true,
|
||||
});
|
||||
let adapterConfig: Record<string, unknown>;
|
||||
if (currentFiles[bundle.instructions.entryFile] !== null && currentFiles[bundle.instructions.entryFile] !== undefined) {
|
||||
if (!actor && mode === "reconcile") return currentState;
|
||||
if (!actor) throw unprocessable("Resetting existing instructions requires an authenticated operator", { code: "INSTRUCTION_IDENTITY_INVALID" });
|
||||
const revisions = agentInstructionRevisionService(db);
|
||||
const target = { companyId: agent.companyId, agentId: agent.id };
|
||||
const baseline = await revisions.readCurrent(target, actor);
|
||||
const receipt = await revisions.commit({ ...target, entryFile: bundle.instructions.entryFile,
|
||||
baseRevisionId: baseline?.revision.id ?? null, content: bundle.instructions.files[bundle.instructions.entryFile] ?? "",
|
||||
source: actor.type === "board" ? "board" : "api" }, actor);
|
||||
if (receipt.materialization === "pending") throw conflict("Instruction revision saved; retry reset to repair its disk copy", { revisionId: receipt.revision.id });
|
||||
const refreshed = await agentSvc.getById(agent.id);
|
||||
if (!refreshed) throw notFound("Built-in agent not found");
|
||||
for (const [file, content] of Object.entries(bundle.instructions.files)) {
|
||||
if (file !== bundle.instructions.entryFile) await instructionsSvc.writeFile(refreshed, file, content);
|
||||
}
|
||||
// Preserve personal files outside this built-in bundle's declared paths.
|
||||
adapterConfig = { ...refreshed.adapterConfig };
|
||||
delete adapterConfig.promptTemplate;
|
||||
delete adapterConfig.bootstrapPromptTemplate;
|
||||
} else {
|
||||
const materialized = await instructionsSvc.materializeManagedBundle(agent, bundle.instructions.files, {
|
||||
entryFile: bundle.instructions.entryFile,
|
||||
replaceExisting: false,
|
||||
clearLegacyPromptTemplate: true,
|
||||
});
|
||||
adapterConfig = materialized.adapterConfig;
|
||||
}
|
||||
const updated = await agentSvc.update(agent.id, {
|
||||
adapterConfig: materialized.adapterConfig,
|
||||
adapterConfig,
|
||||
}, {
|
||||
allowBuiltInAgentMetadata: true,
|
||||
recordRevision: { source: `built-in-bundle:${mode}:instructions` },
|
||||
@@ -1480,13 +1607,14 @@ export function builtInAgentService(db: Db) {
|
||||
definition: BuiltInAgentDefinition,
|
||||
mode: "reconcile" | "reset",
|
||||
resources?: Array<"instructions" | "skill" | "routine">,
|
||||
actor?: AuthorizationActor,
|
||||
) {
|
||||
if (!definition.bundle) return [];
|
||||
const selected = new Set(resources ?? ["instructions", "skill", "routine"]);
|
||||
const existingStates = await bundleResourceStates(agent.companyId, definition, agent);
|
||||
const byKind = new Map(existingStates.map((state) => [state.resourceKind, state]));
|
||||
const instruction = selected.has("instructions")
|
||||
? await materializeInstructions(agent, definition, mode)
|
||||
? await materializeInstructions(agent, definition, mode, actor)
|
||||
: byKind.get("instructions")!;
|
||||
const refreshedAgent = await agentSvc.getById(agent.id) as Agent | null;
|
||||
if (!refreshedAgent) throw notFound("Built-in agent not found");
|
||||
@@ -1847,7 +1975,7 @@ export function builtInAgentService(db: Db) {
|
||||
return state(definition, updated as Agent);
|
||||
}
|
||||
|
||||
async function reset(companyId: string, key: string, input: { resources?: Array<"agent" | "instructions" | "skill" | "routine"> } = {}) {
|
||||
async function reset(companyId: string, key: string, input: { resources?: Array<"agent" | "instructions" | "skill" | "routine"> } = {}, actor?: AuthorizationActor) {
|
||||
const definition = requireBuiltInAgentDefinition(key);
|
||||
const resetAgentDefaults = !input.resources || input.resources.includes("agent");
|
||||
const current = resetAgentDefaults
|
||||
@@ -1862,6 +1990,7 @@ export function builtInAgentService(db: Db) {
|
||||
definition,
|
||||
"reset",
|
||||
input.resources ? selectedBundleResources ?? [] : undefined,
|
||||
actor,
|
||||
);
|
||||
return state(definition, await agentSvc.getById(current.agent.id) as Agent, resources);
|
||||
}
|
||||
|
||||
@@ -112,7 +112,7 @@ function expandTargetKeysForLegacyCompatibility(targetKeys: string[]) {
|
||||
return [...expanded];
|
||||
}
|
||||
|
||||
export function changeConsentGateService(db: Db) {
|
||||
export function changeConsentGateService(db: Db | Parameters<Parameters<Db["transaction"]>[0]>[0]) {
|
||||
return {
|
||||
assertConsented: async (input: {
|
||||
companyId: string;
|
||||
|
||||
@@ -3549,7 +3549,7 @@ export function companyPortabilityService(db: Db, storage?: StorageService) {
|
||||
const companies = companyService(db);
|
||||
const agents = agentService(db);
|
||||
const assetRecords = assetService(db);
|
||||
const instructions = agentInstructionsService();
|
||||
const instructions = agentInstructionsService(db);
|
||||
const access = accessService(db);
|
||||
const projects = projectService(db);
|
||||
const issues = issueService(db);
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import { isAgentDirectoryCopy } from "./agent-directory-working-copies.js";
|
||||
|
||||
import type { PaperclipTurnContext } from "@paperclipai/adapter-utils/server-utils";
|
||||
import { restoreNativeWorkspaceBestEffort } from "./native-runtime/native-workspace-best-effort.js";
|
||||
import {
|
||||
@@ -67,6 +69,7 @@ import {
|
||||
startAdapterExecutionTargetPaperclipBridge,
|
||||
} from "@paperclipai/adapter-utils/execution-target";
|
||||
import { agentService } from "./agents.js";
|
||||
import { agentInstructionWorkingCopyService, instructionWorkingCopyGuidance } from "./agent-instruction-working-copies.js";
|
||||
import { normalizeLegacyRunnerProvider } from "@paperclipai/adapter-utils";
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
@@ -3442,6 +3445,15 @@ const heartbeatRunSafeResultJsonColumn = sql<Record<string, unknown> | null>`
|
||||
'error', left(${heartbeatRuns.resultJson} ->> 'error', ${HEARTBEAT_RUN_RESULT_SUMMARY_MAX_CHARS}),
|
||||
'stdout', left(${heartbeatRuns.resultJson} ->> 'stdout', ${HEARTBEAT_RUN_RESULT_OUTPUT_MAX_CHARS}),
|
||||
'stderr', left(${heartbeatRuns.resultJson} ->> 'stderr', ${HEARTBEAT_RUN_RESULT_OUTPUT_MAX_CHARS}),
|
||||
'instructionSave', case when jsonb_typeof(${heartbeatRuns.resultJson} -> 'instructionSave') = 'object'
|
||||
then jsonb_strip_nulls(jsonb_build_object(
|
||||
'state', left(${heartbeatRuns.resultJson} #>> '{instructionSave,state}', 32),
|
||||
'contract', left(${heartbeatRuns.resultJson} #>> '{instructionSave,contract}', 32),
|
||||
'entryFile', left(${heartbeatRuns.resultJson} #>> '{instructionSave,entryFile}', 512),
|
||||
'errorCode', left(${heartbeatRuns.resultJson} #>> '{instructionSave,errorCode}', 128),
|
||||
'errorMessage', left(${heartbeatRuns.resultJson} #>> '{instructionSave,errorMessage}', 1024),
|
||||
'storageWarning', left(${heartbeatRuns.resultJson} #>> '{instructionSave,storageWarning}', 1024)
|
||||
)) end,
|
||||
'workspaceRestoreFailure', case when ${heartbeatRuns.resultJson} ->> 'workspaceRestoreFailure'
|
||||
in ('restore_permission_denied', 'restore_lock_timeout', 'restore_unsafe_archive', 'restore_failed')
|
||||
then ${heartbeatRuns.resultJson} -> 'workspaceRestoreFailure' end,
|
||||
@@ -9500,6 +9512,7 @@ export function heartbeatService(
|
||||
environmentRuntimeService(db, {
|
||||
pluginWorkerManager: options.pluginWorkerManager,
|
||||
});
|
||||
const instructionCopies = agentInstructionWorkingCopyService(db, { environmentRuntime });
|
||||
const envOrchestrator = environmentRunOrchestrator(db, {
|
||||
pluginWorkerManager: options.pluginWorkerManager,
|
||||
environmentRuntime,
|
||||
@@ -18866,6 +18879,14 @@ export function heartbeatService(
|
||||
async function reapOrphanedRuns(opts?: { staleThresholdMs?: number }) {
|
||||
const staleThresholdMs = opts?.staleThresholdMs ?? 0;
|
||||
const now = new Date();
|
||||
// Recovery never launches a provider or infers stopped ownership from
|
||||
// terminal status. Uncaptured local copies require durable stop evidence.
|
||||
await instructionCopies.recoverStopped().catch(error => {
|
||||
logger.warn({ err: error }, "failed to recover stopped instruction copies");
|
||||
});
|
||||
await instructionCopies.recoverCaptured().catch(error => {
|
||||
logger.warn({ err: error }, "failed to retry captured instruction revisions");
|
||||
});
|
||||
|
||||
// Complete persisted native results before generic orphan recovery. The
|
||||
// reconciler reads the durable workspace barrier and persisted runtime
|
||||
@@ -22348,6 +22369,29 @@ export function heartbeatService(
|
||||
await bindIssueToPersistedExecutionWorkspace(persistedExecutionWorkspace);
|
||||
const workspaceRealization = realizationResult.workspaceRealization;
|
||||
const executionTarget = realizationResult.executionTarget;
|
||||
let instructionCopy: Awaited<ReturnType<typeof instructionCopies.prepare>> = null;
|
||||
let instructionSave: Record<string, unknown> | null = null;
|
||||
const collectStoppedInstructions = async () => {
|
||||
if (!instructionCopy) return;
|
||||
let saved = await instructionCopies.collectStopped({ companyId: agent.companyId, runId: run.id, target: executionTarget });
|
||||
// Capture before disposal. Exhausted bounded collection leaves a durable
|
||||
// explicit loss report, never a claim that missing bytes were saved.
|
||||
while (saved?.state === "pending_collection" && saved.attempts < 3) {
|
||||
saved = await instructionCopies.collectStopped({ companyId: agent.companyId, runId: run.id, target: executionTarget });
|
||||
}
|
||||
if (!saved) return;
|
||||
const receipt = parseObject(saved.receipt);
|
||||
const storageWarning = readNonEmptyString(receipt.storageWarning);
|
||||
instructionSave = { state: saved.state, entryFile: saved.entryFile,
|
||||
...(isAgentDirectoryCopy(saved) ? { contract: "agent_files", appliedCandidateHash: saved.candidateHash }
|
||||
: { revisionId: parseObject(receipt.revision).id ?? null }), storageWarning, errorCode: saved.errorCode, errorMessage: saved.errorMessage };
|
||||
await appendRunEvent(run, { eventType: "instruction_save", stream: "system",
|
||||
level: !storageWarning && ["saved", "unchanged", "resolved"].includes(saved.state) ? "info" : "warn",
|
||||
message: storageWarning ?? (saved.state === "saved" ? "Agent files saved."
|
||||
: saved.state === "unchanged" ? "Instruction working copy is unchanged."
|
||||
: saved.errorMessage ?? "Instruction edits were not saved. Review the preserved candidate in the agent instruction editor."),
|
||||
payload: instructionSave });
|
||||
};
|
||||
if (managedAiRuntime && aiBinding) {
|
||||
try { await assertManagedAiProjectAuth({ ...resolvedConfig, cwd: executionWorkspace.cwd }, aiBinding.provider, executionTarget); }
|
||||
catch { throw new ConfigurationIncompleteFailure("Project authentication conflicts with this agent’s managed AI connection", { configurationIncomplete: { reason: "ai_connection_incompatible", actionUrl: `/agents/${agent.id}/runtime` } }); }
|
||||
@@ -23157,6 +23201,62 @@ export function heartbeatService(
|
||||
target: executionTarget,
|
||||
workspaceId: persistedExecutionWorkspace?.id ?? null,
|
||||
});
|
||||
const hasInstructionFilesystem = nativeRuntimeResolution.kind !== "native"
|
||||
? adapter.supportsInstructionsBundle === true
|
||||
: !["claude_managed_agents_api", "aws_agentcore_harness_api"].includes(nativeRuntimeResolution.profile.backend);
|
||||
if (hasInstructionFilesystem) {
|
||||
try {
|
||||
// Missing contract fields on a restored session mean the deployed
|
||||
// legacy format. New sessions opt into whole-directory persistence.
|
||||
const priorFileRun = taskSessionForRun?.lastRunId
|
||||
? await db.select({ profile: heartbeatRuns.runnerProfileJson }).from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.id, taskSessionForRun.lastRunId), eq(heartbeatRuns.companyId, agent.companyId), eq(heartbeatRuns.agentId, agent.id))).then(rows => rows[0])
|
||||
: null;
|
||||
const savedFileInput = parseObject(parseObject(run.runnerProfileJson).nativeExecutionInput);
|
||||
const priorFileInput = Object.keys(savedFileInput).length ? savedFileInput : parseObject(parseObject(priorFileRun?.profile).nativeExecutionInput);
|
||||
const priorWorkingCopy = parseObject(parseObject(parseObject(priorFileInput.runtimeContext).instructions).workingCopy);
|
||||
instructionCopy = await instructionCopies.prepare({
|
||||
companyId: agent.companyId, agentId: agent.id, runId: run.id,
|
||||
target: executionTarget, cwd: executionWorkspace.cwd,
|
||||
legacy: Object.keys(priorFileInput).length > 0 && priorWorkingCopy.kind !== "agent_files",
|
||||
});
|
||||
} catch (error) {
|
||||
if ((error as { status?: number }).status !== 403) throw error;
|
||||
// Missing write identity must not break a background run's read-only
|
||||
// prompt. It must also never imply that ordinary file edits will save.
|
||||
await appendRunEvent(run, { eventType: "instruction_save", stream: "system", level: "warn",
|
||||
message: "Persistent instruction editing is unavailable. Use an authenticated user with instruction edit access and a managed instruction bundle.",
|
||||
payload: { state: "unavailable", code: "INSTRUCTION_COPY_UNAVAILABLE" } });
|
||||
const guidance = "No editable agent instruction working copy is registered for this turn. Use authenticated agent file tools for persistent edits; do not edit a private copy named in an earlier turn or claim its changes will persist.";
|
||||
for (const key of ["paperclipTaskMarkdown", "paperclipTaskMarkdownCompact"]) {
|
||||
context[key] = [readNonEmptyString(context[key]), guidance].filter(Boolean).join("\n\n");
|
||||
}
|
||||
}
|
||||
if (instructionCopy) {
|
||||
const storageWarning = readNonEmptyString(instructionCopy.receipt?.storageWarning);
|
||||
if (storageWarning) {
|
||||
instructionSave = { state: "prepared", contract: "agent_files", storageWarning };
|
||||
// This is an advisory on the run, never an agent pause, execution
|
||||
// failure, or scheduling gate. Keep it visible while work runs.
|
||||
await db.update(heartbeatRuns).set({ resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify({ instructionSave })}::jsonb` }).where(eq(heartbeatRuns.id, run.id));
|
||||
await appendRunEvent(run, { eventType: "instruction_save", stream: "system", level: "warn",
|
||||
message: storageWarning, payload: instructionSave });
|
||||
}
|
||||
runtimeConfig = { ...runtimeConfig, instructionsFilePath: path.join(instructionCopy.localRoot, instructionCopy.entryFile) };
|
||||
if (isAgentDirectoryCopy(instructionCopy)) {
|
||||
const workspace = parseObject(context.paperclipWorkspace);
|
||||
context.paperclipWorkspace = { ...workspace, agentHome: instructionCopy.executionRoot,
|
||||
// Keep the pre-existing permission root stable for ACP session
|
||||
// identity. The per-run copy is already under the company root.
|
||||
agentHomeForPermissions: workspace.agentHome,
|
||||
};
|
||||
}
|
||||
const guidance = instructionWorkingCopyGuidance(instructionCopy);
|
||||
for (const key of ["paperclipTaskMarkdown", "paperclipTaskMarkdownCompact"]) {
|
||||
context[key] = [readNonEmptyString(context[key]), guidance].filter(Boolean).join("\n\n");
|
||||
}
|
||||
}
|
||||
}
|
||||
let nativeExecution: NativeExecutionInput | null = null;
|
||||
let nativeRunnerInstanceId: string | null = null;
|
||||
if (nativeRuntimeResolution.kind === "native") {
|
||||
@@ -23586,6 +23686,7 @@ export function heartbeatService(
|
||||
runId: run.id,
|
||||
runtimeConfig,
|
||||
runtimeSkillEntries,
|
||||
instructionWorkingCopy: instructionCopy ? { rootPath: instructionCopy.executionRoot, entryPath: instructionCopy.entryFile, ...(isAgentDirectoryCopy(instructionCopy) ? { kind: "agent_files" as const } : {}) } : undefined,
|
||||
});
|
||||
const nativeExecutionWithCheckpoint =
|
||||
buildNativeExecutionWithCheckpoint({
|
||||
@@ -24293,6 +24394,10 @@ export function heartbeatService(
|
||||
},
|
||||
onLog,
|
||||
onEvent: onAdapterEvent,
|
||||
instructionWorkingCopy: instructionCopy ? {
|
||||
hasChanges: () => instructionCopies.hasChanges({ companyId: agent.companyId, runId: run.id, target: executionTarget }),
|
||||
collectStopped: collectStoppedInstructions,
|
||||
} : undefined,
|
||||
preparationSpans: nativeRunnerPreparationSpans,
|
||||
// Bootstrap with executable/home discovery while keeping
|
||||
// configured provider values and the server-selected
|
||||
@@ -24306,6 +24411,7 @@ export function heartbeatService(
|
||||
process.env,
|
||||
executionWorkspace.cwd,
|
||||
),
|
||||
...(instructionCopy && isAgentDirectoryCopy(instructionCopy) ? { AGENT_HOME: instructionCopy.executionRoot } : {}),
|
||||
...(nativeMcpServer
|
||||
? {
|
||||
PAPERCLIP_NATIVE_MCP_NAME: nativeMcpServer.name,
|
||||
@@ -24486,6 +24592,7 @@ export function heartbeatService(
|
||||
issueId,
|
||||
);
|
||||
},
|
||||
onProviderStopped: collectStoppedInstructions,
|
||||
onDispatch: markDispatchStarted,
|
||||
signal: executionControl.controller.signal,
|
||||
...(executionTarget?.kind === "remote" && executionTarget.transport === "sandbox" ? {
|
||||
@@ -24534,6 +24641,7 @@ export function heartbeatService(
|
||||
if (!guardedDispatch.dispatched) return;
|
||||
adapterResult = await guardedDispatch.resultPromise;
|
||||
}
|
||||
if (instructionSave) adapterResult.resultJson = { ...adapterResult.resultJson, instructionSave };
|
||||
adapterResult = applyWorkspaceRestoreFailure(adapterResult);
|
||||
// A returned result can include a failed restore. Keep the workspace
|
||||
// barrier closed until required files have been restored.
|
||||
@@ -24769,6 +24877,7 @@ export function heartbeatService(
|
||||
"failed to revoke heartbeat-run MCP gateway tokens",
|
||||
);
|
||||
}
|
||||
await instructionCopies.release(agent.companyId, run.id);
|
||||
}
|
||||
// Reconcile the referenced-project set against the real remote staging outcome. A referenced
|
||||
// project can pass authorization and clone locally at run prep, then fail to stage into the
|
||||
@@ -26160,6 +26269,15 @@ export function heartbeatService(
|
||||
},
|
||||
);
|
||||
}
|
||||
// A retained or unverified process stays above this release boundary.
|
||||
// If no stopped-copy capture occurred, preserve an explicit loss report.
|
||||
const uncapturedInstructions = await instructionCopies.reportUnavailable(run.companyId, run.id);
|
||||
if (uncapturedInstructions?.state === "unavailable") {
|
||||
await appendRunEvent(run, { eventType: "instruction_save", stream: "system", level: "warn",
|
||||
message: "Instruction edits could not be recovered before environment release. No instruction save is claimed.",
|
||||
payload: { state: "unavailable", code: uncapturedInstructions.errorCode } });
|
||||
}
|
||||
await instructionCopies.release(run.companyId, run.id);
|
||||
await releaseEnvironmentLeasesForRun({
|
||||
runId: run.id,
|
||||
companyId: run.companyId,
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it } from "vitest";
|
||||
import { agents, authUsers, companies, companyMemberships, principalPermissionGrants, heartbeatRuns, issues, agentInstructionRevisions, createDb } from "@paperclipai/db";
|
||||
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
||||
import { resolveManagedInstructionsRoot } from "../agent-instructions.js";
|
||||
import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js";
|
||||
|
||||
describe("canonical instruction tools through native authority", () => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let home: string;
|
||||
const previousHome = process.env.PAPERCLIP_HOME;
|
||||
let companyId: string, agentId: string, targetAgentId: string, userId: string, runId: string, issueId: string, root: string;
|
||||
let authority: PaperclipRunnerToolAuthority;
|
||||
const entryFile = "policy/ENTRY.md";
|
||||
const original = "\uFEFF# Original\r\n\0☃\n";
|
||||
beforeAll(async () => {
|
||||
home = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "instruction-tool-home-")));
|
||||
process.env.PAPERCLIP_HOME = home;
|
||||
database = await startEmbeddedPostgresTestDatabase("instruction-tool-db-");
|
||||
db = createDb(database.connectionString);
|
||||
}, 90_000);
|
||||
afterAll(async () => {
|
||||
if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome;
|
||||
await database?.cleanup();
|
||||
if (home) await fs.rm(home, { recursive: true, force: true });
|
||||
});
|
||||
beforeEach(async () => {
|
||||
[companyId, agentId, targetAgentId, userId, runId, issueId] = Array.from({ length: 6 }, () => randomUUID());
|
||||
await db.insert(companies).values({ id: companyId, name: "Tool scope", issuePrefix: randomUUID().slice(0, 8) });
|
||||
await db.insert(authUsers).values({ id: userId, name: "Editor", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() });
|
||||
root = resolveManagedInstructionsRoot({ id: targetAgentId, companyId, name: "Target", adapterConfig: {} });
|
||||
await db.insert(agents).values([
|
||||
{ id: agentId, companyId, name: "Caller", status: "active" },
|
||||
{ id: targetAgentId, companyId, name: "Target", adapterConfig: { instructionsBundleMode: "managed", instructionsRootPath: root, instructionsEntryFile: entryFile } },
|
||||
]);
|
||||
await db.insert(companyMemberships).values([
|
||||
{ companyId, principalType: "user", principalId: userId, membershipRole: "operator" },
|
||||
{ companyId, principalType: "agent", principalId: agentId, membershipRole: "member" },
|
||||
]);
|
||||
await db.insert(principalPermissionGrants).values([
|
||||
{ companyId, principalType: "user", principalId: userId, permissionKey: "agents:configure", scope: { agentIds: [targetAgentId] } },
|
||||
{ companyId, principalType: "agent", principalId: agentId, permissionKey: "agents:configure", scope: { agentIds: [targetAgentId] } },
|
||||
]);
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Maintain instructions", status: "in_progress", workMode: "standard", assigneeAgentId: agentId });
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native", nativeIssueId: issueId, invocationSource: "on_demand", responsibleUserId: userId });
|
||||
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
|
||||
authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId });
|
||||
await fs.mkdir(path.dirname(path.join(root, entryFile)), { recursive: true });
|
||||
await fs.writeFile(path.join(root, entryFile), original);
|
||||
});
|
||||
const call = (tool: string, args: Record<string, unknown> = {}) => authority.execute({ tool, callId: randomUUID(), arguments: { targetAgentId, ...args } }) as Promise<any>;
|
||||
it("bridges old native instruction tools to current files without writing history", async () => {
|
||||
const first = await call("read_agent_instructions");
|
||||
expect(first).toMatchObject({ entryFile, content: original });
|
||||
const changed = await call("update_agent_instructions", { entryFile, content: "changed\r\n", baseRevisionId: first.revision.id });
|
||||
expect(changed.revision).toMatchObject({ responsibleUserId: userId, actorAgentId: agentId, sourceRunId: runId });
|
||||
expect((await call("get_agent_instruction_history", { entryFile })).revisions).toHaveLength(0);
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe("changed\r\n");
|
||||
await expect(call("update_agent_instructions", { entryFile, content: "stale", baseRevisionId: first.revision.id })).rejects.toMatchObject({ status: 409 });
|
||||
expect((await call("read_agent_instructions")).content).toBe("changed\r\n");
|
||||
});
|
||||
it("rejects supplied identity and missing CAS bases before changing content", async () => {
|
||||
const first = await call("read_agent_instructions");
|
||||
for (const identity of ["companyId", "agentId", "runId", "responsibleUserId", "onBehalfOfUserId", "actor"]) {
|
||||
await expect(call("update_agent_instructions", { entryFile, content: "forged", baseRevisionId: first.revision.id, [identity]: randomUUID() })).rejects.toMatchObject({ status: 400 });
|
||||
}
|
||||
await expect(call("update_agent_instructions", { entryFile, content: "missing base" })).rejects.toMatchObject({ status: 400 });
|
||||
expect((await call("read_agent_instructions")).content).toBe(original);
|
||||
expect(await db.select().from(agentInstructionRevisions).where(eq(agentInstructionRevisions.agentId, targetAgentId))).toHaveLength(0);
|
||||
});
|
||||
it("rechecks responsible-user access and rejects a cross-company target", async () => {
|
||||
const first = await call("read_agent_instructions");
|
||||
await db.delete(principalPermissionGrants).where(eq(principalPermissionGrants.principalId, userId));
|
||||
await expect(call("update_agent_instructions", { entryFile, content: "revoked", baseRevisionId: first.revision.id })).rejects.toMatchObject({ status: 403 });
|
||||
const foreignCompanyId = randomUUID(), foreignAgentId = randomUUID();
|
||||
await db.insert(companies).values({ id: foreignCompanyId, name: "Foreign", issuePrefix: randomUUID().slice(0, 8) });
|
||||
await db.insert(agents).values({ id: foreignAgentId, companyId: foreignCompanyId, name: "Foreign" });
|
||||
await expect(call("read_agent_instructions", { targetAgentId: foreignAgentId })).rejects.toMatchObject({ status: 404 });
|
||||
await expect(call("update_agent_instructions", { targetAgentId: foreignAgentId, entryFile, content: "foreign", baseRevisionId: null })).rejects.toMatchObject({ status: 404 });
|
||||
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, runId));
|
||||
await expect(call("read_agent_instructions")).rejects.toThrow("paperclip_runner_tool_binding_not_authorized");
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(original);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,20 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js";
|
||||
|
||||
const binding = { companyId: "00000000-0000-4000-8000-000000000001", agentId: "00000000-0000-4000-8000-000000000002", issueId: "00000000-0000-4000-8000-000000000003", runId: "00000000-0000-4000-8000-000000000004" };
|
||||
|
||||
describe("dedicated canonical instruction tools", () => {
|
||||
it("advertises read, CAS update, history, and restore on ordinary native runs", () => {
|
||||
const authority = new PaperclipRunnerToolAuthority({} as Db, binding);
|
||||
const tools = authority.definitions();
|
||||
expect(tools.map((tool) => tool.name)).toEqual(expect.arrayContaining([
|
||||
"read_agent_instructions", "update_agent_instructions", "get_agent_instruction_history", "restore_agent_instructions",
|
||||
]));
|
||||
for (const name of ["update_agent_instructions", "restore_agent_instructions"]) {
|
||||
const tool = tools.find((candidate) => candidate.name === name)!;
|
||||
expect(tool.inputSchema).toMatchObject({ required: expect.arrayContaining(["baseRevisionId"]), additionalProperties: false });
|
||||
expect(tool.inputSchema).not.toHaveProperty("properties.responsibleUserId");
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,76 @@
|
||||
import type { AgentInstructionCommitReceipt } from "@paperclipai/shared";
|
||||
import { z } from "zod";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { badRequest, notFound } from "../../errors.js";
|
||||
import { agentInstructionRevisionService } from "../agent-instruction-revisions.js";
|
||||
import { agentInstructionWorkingCopyService } from "../agent-instruction-working-copies.js";
|
||||
import { deriveBundleState } from "../agent-instructions.js";
|
||||
import { agentService } from "../agents.js";
|
||||
import type { AuthorizationActor } from "../authorization.js";
|
||||
|
||||
const target = { targetAgentId: z.string().uuid().optional() };
|
||||
const entryFile = z.string().min(1).max(4096);
|
||||
const baseRevisionId = z.string().uuid().nullable();
|
||||
const schemas = {
|
||||
read_agent_instructions: z.object({ ...target, entryFile: entryFile.optional(), revisionId: z.string().uuid().optional() }).strict()
|
||||
.refine((input) => Boolean(input.entryFile) === Boolean(input.revisionId), "Historical reads require both entryFile and revisionId"),
|
||||
update_agent_instructions: z.object({ ...target, entryFile, content: z.string().max(1024 * 1024), baseRevisionId }).strict(),
|
||||
get_agent_instruction_history: z.object({ ...target, entryFile, cursor: z.string().min(1).max(2048).optional(), limit: z.number().int().min(1).max(100).optional() }).strict(),
|
||||
restore_agent_instructions: z.object({ ...target, entryFile, revisionId: z.string().uuid(), baseRevisionId: z.string().uuid() }).strict(),
|
||||
};
|
||||
|
||||
export type AgentInstructionToolName = keyof typeof schemas;
|
||||
|
||||
/** Called only after the native authority validates its server-bound active run. */
|
||||
export async function executeAgentInstructionTool(input: {
|
||||
db: Db;
|
||||
binding: { companyId: string; agentId: string; runId: string };
|
||||
tool: AgentInstructionToolName;
|
||||
arguments: unknown;
|
||||
}) {
|
||||
// Never accept an actor, responsible user, company, or source run from tool JSON.
|
||||
const actor: AuthorizationActor = { type: "agent", source: "agent_jwt", companyId: input.binding.companyId, agentId: input.binding.agentId, runId: input.binding.runId };
|
||||
const service = agentInstructionRevisionService(input.db);
|
||||
const parse = <T>(schema: z.ZodType<T>): T => {
|
||||
const result = schema.safeParse(input.arguments);
|
||||
if (!result.success) throw badRequest("Invalid instruction tool arguments", result.error.flatten());
|
||||
return result.data;
|
||||
};
|
||||
const scope = (targetAgentId?: string) => ({ companyId: input.binding.companyId, agentId: targetAgentId ?? input.binding.agentId });
|
||||
const acknowledge = async (receipt: AgentInstructionCommitReceipt, targetAgentId?: string) => {
|
||||
// The commit is already durable. A failed baseline refresh keeps the old CAS
|
||||
// fence, so later cleanup preserves a conflict instead of overwriting it.
|
||||
await agentInstructionWorkingCopyService(input.db).acknowledgeExplicitSave({
|
||||
...scope(targetAgentId), runId: input.binding.runId, entryFile: receipt.revision.entryFile,
|
||||
revisionId: receipt.revision.id, contentHash: receipt.revision.contentHash,
|
||||
}).catch(() => undefined);
|
||||
return receipt;
|
||||
};
|
||||
switch (input.tool) {
|
||||
case "read_agent_instructions": {
|
||||
const args = parse(schemas.read_agent_instructions);
|
||||
const target = scope(args.targetAgentId);
|
||||
if (args.entryFile && args.revisionId) return service.readRevision({ ...target, entryFile: args.entryFile, revisionId: args.revisionId }, actor);
|
||||
const snapshot = await service.readCurrent(target, actor);
|
||||
if (snapshot) return { ...snapshot, entryFile: snapshot.revision.entryFile };
|
||||
// An unseeded empty entry still needs its configured filename for the first CAS.
|
||||
const agent = await agentService(input.db).getById(target.agentId);
|
||||
if (!agent || agent.companyId !== target.companyId) throw notFound("Agent not found");
|
||||
return { entryFile: deriveBundleState(agent).entryFile, revision: null, content: null };
|
||||
}
|
||||
case "update_agent_instructions": {
|
||||
const args = parse(schemas.update_agent_instructions);
|
||||
const receipt = await service.commit({ ...scope(args.targetAgentId), entryFile: args.entryFile, content: args.content, baseRevisionId: args.baseRevisionId, source: "tool" }, actor);
|
||||
return acknowledge(receipt, args.targetAgentId);
|
||||
}
|
||||
case "get_agent_instruction_history": {
|
||||
const args = parse(schemas.get_agent_instruction_history);
|
||||
return service.history({ ...scope(args.targetAgentId), entryFile: args.entryFile, cursor: args.cursor, limit: args.limit }, actor);
|
||||
}
|
||||
case "restore_agent_instructions": {
|
||||
const args = parse(schemas.restore_agent_instructions);
|
||||
const receipt = await service.restore({ ...scope(args.targetAgentId), entryFile: args.entryFile, revisionId: args.revisionId, baseRevisionId: args.baseRevisionId }, actor);
|
||||
return acknowledge(receipt, args.targetAgentId);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -5870,6 +5870,57 @@ describe("native session same-turn steering", () => {
|
||||
});
|
||||
|
||||
describe("native warm session supervision", () => {
|
||||
it.each([
|
||||
{ changed: false, closeFails: false },
|
||||
{ changed: true, closeFails: false },
|
||||
{ changed: true, closeFails: true },
|
||||
])("collects changed instructions only after the owned warm provider stops (changed=$changed, close fails=$closeFails)", async ({ changed, closeFails }) => {
|
||||
const identity = `instruction-close-${changed}-${closeFails}`;
|
||||
let releaseClose!: () => void;
|
||||
const closed = new Promise<void>((resolve) => { releaseClose = resolve; });
|
||||
const close = vi.fn(async () => { await closed; if (closeFails) throw new Error("instruction provider close failed"); });
|
||||
const collectStopped = vi.fn(async () => {});
|
||||
const hasChanges = vi.fn(async () => changed);
|
||||
const warmExecution = { ...execution,
|
||||
binding: { ...execution.binding, runId: identity, executionWorkspaceId: identity },
|
||||
session: { ...execution.session, normalizedSessionId: identity, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 } },
|
||||
} as NativeExecutionInputV1;
|
||||
state.execute.mockReset().mockImplementationOnce(async (options) => {
|
||||
expect(options.requireSessionCloseBeforeReturn).toBe(true);
|
||||
expect(options.onSessionClosed).toBe(collectStopped);
|
||||
await options.onSession?.({ close });
|
||||
return { result: { summary: "completed" }, terminal: { runTerminalState: "succeeded" },
|
||||
turnId: identity, normalizedSessionId: identity, providerSessionId: identity,
|
||||
driverKind: "test", driverVersion: "1", nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null };
|
||||
});
|
||||
const running = executePaperclipNativeSession({ db: leaseDb(warmExecution), execution: warmExecution,
|
||||
runnerInstanceId: identity, runnerExecutionTarget: { kind: "remote", transport: "sandbox", environmentId: identity, remoteCwd: `/tmp/${identity}` },
|
||||
instructionWorkingCopy: { hasChanges, collectStopped } });
|
||||
const observed = running.then(() => null, error => error);
|
||||
try {
|
||||
if (changed) {
|
||||
await vi.waitFor(() => expect(close).toHaveBeenCalledOnce());
|
||||
expect(collectStopped).not.toHaveBeenCalled();
|
||||
releaseClose();
|
||||
const error = await observed;
|
||||
if (closeFails) {
|
||||
expect(error?.message).toBe("instruction provider close failed");
|
||||
expect(collectStopped).not.toHaveBeenCalled();
|
||||
} else {
|
||||
expect(error).toBeNull();
|
||||
expect(collectStopped).toHaveBeenCalledOnce();
|
||||
}
|
||||
} else {
|
||||
expect(await observed).toBeNull();
|
||||
expect(close).not.toHaveBeenCalled();
|
||||
expect(collectStopped).not.toHaveBeenCalled();
|
||||
}
|
||||
} finally {
|
||||
releaseClose();
|
||||
await closeWarmNativeSessionsForEnvironment({ environmentId: identity, reason: "test cleanup" });
|
||||
}
|
||||
});
|
||||
|
||||
describe("warm session identity transitions", () => {
|
||||
let previousHome: string | undefined;
|
||||
let isolatedHome: string;
|
||||
|
||||
@@ -7167,6 +7167,12 @@ export async function executePaperclipNativeSession(input: {
|
||||
chatAttachmentReadScope?: NativeChatAttachmentReadScope;
|
||||
onLog?: (stream: "stdout" | "stderr", chunk: string) => Promise<void>;
|
||||
onEvent?: (event: AdapterRuntimeEvent) => Promise<void>;
|
||||
/** Only this run's registered private instruction entry.
|
||||
* Probe at terminal; persist only after owned shutdown. */
|
||||
instructionWorkingCopy?: {
|
||||
hasChanges: () => Promise<boolean>;
|
||||
collectStopped: () => Promise<void>;
|
||||
};
|
||||
/** Persist task-level continuity before a durable goal can outlive this run. */
|
||||
onGoalCheckpoint?: (snapshot: PersistedNativeSession) => Promise<void>;
|
||||
sessionGoalControl?: NativeSessionGoalControl | null;
|
||||
@@ -8305,7 +8311,8 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
resumeSessionGoalHeartbeat: input.resumeSessionGoalHeartbeat,
|
||||
// Every durable runner must finish its bounded suspension before
|
||||
// the next run verifies and rotates the saved authority.
|
||||
requireSessionCloseBeforeReturn: runnerdBackend !== null,
|
||||
requireSessionCloseBeforeReturn: runnerdBackend !== null || input.instructionWorkingCopy !== undefined,
|
||||
onSessionClosed: input.instructionWorkingCopy?.collectStopped,
|
||||
onCheckpoint: async (snapshot) => {
|
||||
if (warmSessionId !== null && warmConfigDigest !== null) {
|
||||
await persistWarmNativeCheckpoint(
|
||||
@@ -9065,12 +9072,22 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
// A following run cannot attach until the prior run's durable finalization
|
||||
// is committed. Provider completion alone is not an authority boundary.
|
||||
if (warmSessionId !== null && lifecyclePolicy.mode === "warm") {
|
||||
const instructionCopy = input.instructionWorkingCopy;
|
||||
const ownedSession = warmNativeSessions.get(warmSessionId);
|
||||
const collectInstructions = Boolean(instructionCopy && ownedSession?.ownerToken === warmSessionOwnerToken &&
|
||||
await instructionCopy.hasChanges());
|
||||
if (collectInstructions && ownedSession) {
|
||||
// Keep the unchanged warm path intact. A changed private instruction copy
|
||||
// requires the existing checkpoint-and-close boundary before collection.
|
||||
ownedSession.closeOnReleaseReason = "registered instruction edits require stopped-provider collection";
|
||||
}
|
||||
await releaseWarmNativeSession(
|
||||
warmSessionId,
|
||||
warmSessionOwnerToken,
|
||||
lifecyclePolicy.idleTimeoutMs,
|
||||
false,
|
||||
);
|
||||
if (collectInstructions) await instructionCopy!.collectStopped();
|
||||
}
|
||||
const adapterResult: AdapterExecutionResult = {
|
||||
exitCode: native.terminal.runTerminalState === "succeeded" ? 0 : 1,
|
||||
|
||||
@@ -99,7 +99,7 @@ describe("PaperclipRunnerToolAuthority", () => {
|
||||
issueId,
|
||||
runId,
|
||||
});
|
||||
expect(authority.definitions()).toHaveLength(30);
|
||||
expect(authority.definitions()).toHaveLength(34);
|
||||
const questions = authority.definitions().find(tool => tool.name === "request_human_input")!;
|
||||
expect(questions.description).toContain("ask only the next unanswered question");
|
||||
expect(questions.description).toContain("Never infer answers");
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { executeAgentInstructionTool } from "./agent-instruction-tools.js";
|
||||
import { createReadStream } from "node:fs";
|
||||
import { publicChatTaskUrl } from "../chat-task-url.js";
|
||||
import type { createAssignedMcpTools } from "./assigned-mcp-tools.js";
|
||||
@@ -83,6 +84,7 @@ import {
|
||||
} from "./chat-attachment-read.js";
|
||||
|
||||
const IMPLEMENTED_OPERATIONS = new Set([
|
||||
"read_agent_instructions", "update_agent_instructions", "get_agent_instruction_history", "restore_agent_instructions",
|
||||
"search_api", "call_api", "hire_agent",
|
||||
"get_task_context", "get_task_history", "search_tasks", "report_progress",
|
||||
"request_human_input",
|
||||
@@ -368,6 +370,13 @@ export class PaperclipRunnerToolAuthority {
|
||||
throw new Error("paperclip_runner_tool_mode_denied");
|
||||
}
|
||||
switch (call.tool) {
|
||||
case "read_agent_instructions":
|
||||
case "update_agent_instructions":
|
||||
case "get_agent_instruction_history":
|
||||
case "restore_agent_instructions":
|
||||
return executeAgentInstructionTool({ db: this.db, binding: {
|
||||
companyId: this.binding.companyId, agentId: this.binding.agentId, runId: this.binding.runId,
|
||||
}, tool: call.tool, arguments: call.arguments });
|
||||
case "create_skill": {
|
||||
const apiUrl = this.binding.apiUrl ?? process.env.PAPERCLIP_API_URL;
|
||||
const token = createLocalAgentJwt(this.binding.agentId, this.binding.companyId, context.actor.adapterType, this.binding.runId, context.run.responsibleUserId);
|
||||
|
||||
@@ -42,6 +42,10 @@ function words(text: string): string[] {
|
||||
}
|
||||
|
||||
function dedicatedTools(method: string, path: string): string[] {
|
||||
if (/\/agents\/\{[^}]+\}\/instructions-bundle\/file$/.test(path)) return method === "GET" ? ["read_agent_instructions"] : method === "PUT" ? ["update_agent_instructions"] : [];
|
||||
if (/\/agents\/\{[^}]+\}\/instructions-bundle\/history$/.test(path) && method === "GET") return ["get_agent_instruction_history"];
|
||||
if (/\/agents\/\{[^}]+\}\/instructions-bundle\/revision\/\{[^}]+\}$/.test(path) && method === "GET") return ["read_agent_instructions"];
|
||||
if (/\/agents\/\{[^}]+\}\/instructions-bundle\/restore$/.test(path) && method === "POST") return ["restore_agent_instructions"];
|
||||
if (/\/companies\/\{[^}]+\}\/skills$/.test(path) && method === "POST") return ["create_skill"];
|
||||
if (/\/companies\/\{[^}]+\}\/agent-hires$/.test(path) && method === "POST") return ["hire_agent"];
|
||||
if (/\/projects$/.test(path)) return method === "GET" ? ["list_projects"] : method === "POST" ? ["create_project"] : [];
|
||||
|
||||
@@ -72,6 +72,10 @@ describe("runner API catalog", () => {
|
||||
expect(runnerApiOperation(createProject).dedicatedTools).toEqual(["create_project"]);
|
||||
expect(runnerApiOperation(projects).dedicatedTools).toEqual(["list_projects"]);
|
||||
expect(runnerApiOperation("GET /api/companies/{companyId}/project-repositories").dedicatedTools).toEqual(["list_project_repositories"]);
|
||||
expect(runnerApiOperation("GET /api/agents/{id}/instructions-bundle/file").dedicatedTools).toEqual(["read_agent_instructions"]);
|
||||
expect(runnerApiOperation("PUT /api/agents/{id}/instructions-bundle/file").dedicatedTools).toEqual(["update_agent_instructions"]);
|
||||
expect(runnerApiOperation("GET /api/agents/{id}/instructions-bundle/history").dedicatedTools).toEqual(["get_agent_instruction_history"]);
|
||||
expect(runnerApiOperation("POST /api/agents/{id}/instructions-bundle/restore").dedicatedTools).toEqual(["restore_agent_instructions"]);
|
||||
expect(runnerApiOperation("POST /api/companies/{companyId}/agent-hires").dedicatedTools).toEqual(["hire_agent"]);
|
||||
expect(runnerApiOperation("POST /api/companies/{companyId}/agent-hires").dedicatedToolGuidance).toContain("inherits the caller's native runtime");
|
||||
});
|
||||
|
||||
@@ -10,6 +10,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
const serviceMocks = vi.hoisted(() => ({
|
||||
exportFiles: vi.fn(),
|
||||
readCommittedForRuntime: vi.fn(),
|
||||
getEffectiveProfilesForAgent: vi.fn(),
|
||||
githubBotConnectionIdsForRun: vi.fn(),
|
||||
}));
|
||||
@@ -20,6 +21,11 @@ vi.mock("../chat-github-tools.js", () => ({
|
||||
|
||||
vi.mock("../agent-instructions.js", () => ({
|
||||
agentInstructionsService: () => ({ exportFiles: serviceMocks.exportFiles }),
|
||||
agentInstructionsBundleMode: (agent: { adapterConfig?: { instructionsBundleMode?: string } }) => agent.adapterConfig?.instructionsBundleMode ?? null,
|
||||
}));
|
||||
|
||||
vi.mock("../agent-instruction-revisions.js", () => ({
|
||||
agentInstructionRevisionService: () => ({ readCommittedForRuntime: serviceMocks.readCommittedForRuntime }),
|
||||
}));
|
||||
|
||||
vi.mock("../tool-access.js", () => ({
|
||||
@@ -50,6 +56,7 @@ async function makeTreeWritable(target: string): Promise<void> {
|
||||
|
||||
beforeEach(async () => {
|
||||
vi.clearAllMocks();
|
||||
serviceMocks.readCommittedForRuntime.mockResolvedValue(null);
|
||||
serviceMocks.githubBotConnectionIdsForRun.mockResolvedValue(new Set());
|
||||
previousPaperclipHome = process.env.PAPERCLIP_HOME;
|
||||
previousInstanceId = process.env.PAPERCLIP_INSTANCE_ID;
|
||||
@@ -90,6 +97,60 @@ afterEach(async () => {
|
||||
});
|
||||
|
||||
describe("buildNativeRuntimeContext", () => {
|
||||
it("uses committed instructions when the disk projection is stale", async () => {
|
||||
serviceMocks.exportFiles.mockResolvedValue({
|
||||
entryFile: "AGENTS.md",
|
||||
files: { "AGENTS.md": "Old disk projection\n", "reference.md": "Sibling reference\n" },
|
||||
});
|
||||
serviceMocks.readCommittedForRuntime.mockResolvedValue({
|
||||
revision: { entryFile: "AGENTS.md" }, content: "Saved canonical instructions\n",
|
||||
});
|
||||
const context = await buildNativeRuntimeContext({
|
||||
db: {} as Db,
|
||||
agent: { id: "agent-1", companyId: "company-1", name: "Reviewer", adapterType: "paperclip_runner", adapterConfig: { instructionsBundleMode: "managed" } },
|
||||
runId: "run-1", runtimeConfig: {}, runtimeSkillEntries: [],
|
||||
});
|
||||
expect(await readFile(path.join(context.instructions.bundle.rootPath, context.instructions.entryPath), "utf8"))
|
||||
.toBe("Saved canonical instructions\n");
|
||||
expect(await readFile(path.join(context.instructions.bundle.rootPath, "reference.md"), "utf8"))
|
||||
.toBe("Sibling reference\n");
|
||||
expect(serviceMocks.readCommittedForRuntime).toHaveBeenCalledWith({ companyId: "company-1", agentId: "agent-1" });
|
||||
});
|
||||
|
||||
it("does not substitute canonical managed instructions into an external bundle", async () => {
|
||||
serviceMocks.exportFiles.mockResolvedValue({ entryFile: "AGENTS.md", files: { "AGENTS.md": "External instructions\n" } });
|
||||
const context = await buildNativeRuntimeContext({
|
||||
db: {} as Db,
|
||||
agent: { id: "agent-1", companyId: "company-1", name: "Reviewer", adapterConfig: { instructionsBundleMode: "external" } },
|
||||
runId: "run-1", runtimeConfig: {}, runtimeSkillEntries: [],
|
||||
});
|
||||
expect(await readFile(path.join(context.instructions.bundle.rootPath, context.instructions.entryPath), "utf8"))
|
||||
.toBe("External instructions\n");
|
||||
expect(serviceMocks.readCommittedForRuntime).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("keeps the pinned prompt immutable while exposing the registered editable instruction copy", async () => {
|
||||
serviceMocks.exportFiles.mockResolvedValue({
|
||||
entryFile: "instructions/CHARter.md",
|
||||
files: { "instructions/CHARter.md": "Original instructions.\n" },
|
||||
});
|
||||
const workingRoot = path.join(temporaryRoots.at(-1)!, "private-run-copy");
|
||||
await mkdir(path.join(workingRoot, "instructions"), { recursive: true });
|
||||
await writeFile(path.join(workingRoot, "instructions/CHARter.md"), "Original instructions.\n");
|
||||
const context = await buildNativeRuntimeContext({
|
||||
db: {} as Db,
|
||||
agent: { id: "agent-1", companyId: "company-1", name: "Reviewer", adapterType: "paperclip_runner", adapterConfig: {} },
|
||||
runId: "run-1",
|
||||
runtimeConfig: {},
|
||||
runtimeSkillEntries: [],
|
||||
instructionWorkingCopy: { rootPath: workingRoot, entryPath: "instructions/CHARter.md" },
|
||||
});
|
||||
expect(context.instructions.workingCopy).toEqual({ rootPath: workingRoot, entryPath: "instructions/CHARter.md" });
|
||||
await writeFile(path.join(workingRoot, "instructions/CHARter.md"), "Persist this next time.\n");
|
||||
expect(await readFile(path.join(context.instructions.bundle.rootPath, context.instructions.entryPath), "utf8"))
|
||||
.toBe("Original instructions.\n");
|
||||
});
|
||||
|
||||
it.each(["disabled", "degraded"] as const)(
|
||||
"omits an unavailable native MCP connection when it is %s without aborting runtime context creation",
|
||||
async (unavailableState) => {
|
||||
|
||||
@@ -21,7 +21,8 @@ import {
|
||||
type NativeRuntimeContextSnapshot,
|
||||
} from "../../vendor/paperclip-runner/index.js";
|
||||
import { resolvePaperclipInstanceRoot } from "../../home-paths.js";
|
||||
import { agentInstructionsService } from "../agent-instructions.js";
|
||||
import { agentInstructionsService, agentInstructionsBundleMode } from "../agent-instructions.js";
|
||||
import { agentInstructionRevisionService } from "../agent-instruction-revisions.js";
|
||||
import { toolAccessService } from "../tool-access.js";
|
||||
import { filterResolvedGitHubConnectionsForRun } from "../git-credentials.js";
|
||||
|
||||
@@ -145,8 +146,24 @@ export async function materializeAsset(files: AssetFile[]): Promise<NativeRuntim
|
||||
return { schema: NATIVE_RUNTIME_ASSET_SCHEMA, digest: assetDigest, manifestDigest, rootPath, fileCount: manifestFiles.length, totalBytes };
|
||||
}
|
||||
|
||||
async function materializeInstructionBundle(agent: RuntimeAgent) {
|
||||
async function materializeInstructionBundle(db: Db, agent: RuntimeAgent, agentFiles = false) {
|
||||
if (agentFiles) {
|
||||
const current = await agentInstructionRevisionService(db).readCommittedForRuntime({ companyId: agent.companyId, agentId: agent.id });
|
||||
if (!current) throw new Error("Configured instruction entry is missing");
|
||||
const entryPath = safeRelativePath(current.revision.entryFile, "instruction entry path");
|
||||
return { entryPath, bundle: await materializeAsset([{ path: entryPath, content: Buffer.from(current.content, "utf8"), mode: 0o444 }]) };
|
||||
}
|
||||
const exported = await agentInstructionsService().exportFiles(agent, { rejectSymlinks: true });
|
||||
if (agentInstructionsBundleMode(agent) === "managed") {
|
||||
// The database head remains authoritative when a saved revision could not
|
||||
// update its disk projection. Overlay the exact committed snapshot directly;
|
||||
// a repair followed by a separate export can race another pending commit.
|
||||
const current = await agentInstructionRevisionService(db).readCommittedForRuntime({ companyId: agent.companyId, agentId: agent.id });
|
||||
if (current) {
|
||||
exported.entryFile = current.revision.entryFile;
|
||||
exported.files[current.revision.entryFile] = current.content;
|
||||
}
|
||||
}
|
||||
const entryPath = safeRelativePath(exported.entryFile, "instruction entry path");
|
||||
if (!(entryPath in exported.files)) throw new Error(`configured instruction entry is missing: ${entryPath}`);
|
||||
const files = Object.entries(exported.files).map(([relativePath, content]) => ({ path: safeRelativePath(relativePath, "instruction path"), content: Buffer.from(content, "utf8"), mode: 0o444 }));
|
||||
@@ -222,12 +239,17 @@ export async function resolveNativeRuntimeMcpSnapshot(input: { db: Db; agent: Pi
|
||||
return { assignmentSetId: `sha256:${assignmentDigest}`, digest: assignmentDigest, bindingId: assignment.connections.length ? `native-mcp:${input.runId}` : null };
|
||||
}
|
||||
|
||||
export async function buildNativeRuntimeContext(input: { db: Db; agent: RuntimeAgent; runId: string; runtimeConfig: Record<string, unknown>; runtimeSkillEntries: PaperclipSkillEntry[] }): Promise<NativeRuntimeContextSnapshot> {
|
||||
export async function buildNativeRuntimeContext(input: { db: Db; agent: RuntimeAgent; runId: string; runtimeConfig: Record<string, unknown>; runtimeSkillEntries: PaperclipSkillEntry[]; instructionWorkingCopy?: { rootPath: string; entryPath: string; kind?: "agent_files" } }): Promise<NativeRuntimeContextSnapshot> {
|
||||
const [instructions, skills, mcp] = await Promise.all([
|
||||
materializeInstructionBundle(input.agent),
|
||||
materializeInstructionBundle(input.db, input.agent, input.instructionWorkingCopy?.kind === "agent_files"),
|
||||
materializeSelectedSkills(input.runtimeConfig, input.runtimeSkillEntries, input.agent.adapterType === "paperclip_runner"),
|
||||
resolveNativeRuntimeMcpSnapshot({ db: input.db, agent: input.agent, runId: input.runId }),
|
||||
]);
|
||||
const snapshot = { prompt: { revision: PAPERCLIP_EXECUTION_PROMPT_REVISION, text: PAPERCLIP_EXECUTION_PROMPT, digest: nativeRuntimePromptDigest() }, instructions, skills, mcp };
|
||||
const snapshot = {
|
||||
prompt: { revision: PAPERCLIP_EXECUTION_PROMPT_REVISION, text: PAPERCLIP_EXECUTION_PROMPT, digest: nativeRuntimePromptDigest() },
|
||||
instructions: { ...instructions, ...(input.instructionWorkingCopy ? { workingCopy: input.instructionWorkingCopy } : {}) },
|
||||
skills,
|
||||
mcp,
|
||||
};
|
||||
return parseNativeRuntimeContext({ ...snapshot, aggregateDigest: canonicalNativeRuntimeContextDigest(snapshot) });
|
||||
}
|
||||
@@ -5,6 +5,8 @@ import {
|
||||
companies,
|
||||
pluginEntities,
|
||||
pluginManagedResources,
|
||||
plugins,
|
||||
activityLog,
|
||||
} from "@paperclipai/db";
|
||||
import type {
|
||||
Agent,
|
||||
@@ -12,11 +14,13 @@ import type {
|
||||
PluginManagedAgentDeclaration,
|
||||
PluginManagedAgentResolution,
|
||||
} from "@paperclipai/shared";
|
||||
import { notFound } from "../errors.js";
|
||||
import { isUuidLike } from "@paperclipai/shared";
|
||||
import { conflict, forbidden, notFound } from "../errors.js";
|
||||
import { agentService } from "./agents.js";
|
||||
import { approvalService } from "./approvals.js";
|
||||
import { logActivity } from "./activity-log.js";
|
||||
import { agentInstructionsBundleMode, agentInstructionsService } from "./agent-instructions.js";
|
||||
import { agentInstructionRevisionService } from "./agent-instruction-revisions.js";
|
||||
|
||||
const MANAGED_AGENT_ENTITY_TYPE = "managed_agent";
|
||||
const DEFAULT_MANAGED_AGENT_ADAPTER_TYPE = "process";
|
||||
@@ -180,7 +184,7 @@ export function pluginManagedAgentService(
|
||||
) {
|
||||
const agentSvc = agentService(db);
|
||||
const approvalSvc = approvalService(db);
|
||||
const instructions = agentInstructionsService();
|
||||
const instructions = agentInstructionsService(db);
|
||||
|
||||
function declarationFor(agentKey: string) {
|
||||
const declaration = options.manifest?.agents?.find((agent) => agent.agentKey === agentKey);
|
||||
@@ -190,8 +194,8 @@ export function pluginManagedAgentService(
|
||||
return declaration;
|
||||
}
|
||||
|
||||
async function getBinding(companyId: string, agentKey: string) {
|
||||
return db
|
||||
async function getBinding(companyId: string, agentKey: string, client: Pick<Db, "select"> = db) {
|
||||
return client
|
||||
.select()
|
||||
.from(pluginEntities)
|
||||
.where(
|
||||
@@ -210,6 +214,7 @@ export function pluginManagedAgentService(
|
||||
agentId: string,
|
||||
extraData: Record<string, unknown> = {},
|
||||
effectiveAdapterType?: string,
|
||||
client: Pick<Db, "select" | "insert" | "update"> = db,
|
||||
) {
|
||||
const adapterType = effectiveAdapterType ?? (await resolveManagedAdapterType(companyId, declaration));
|
||||
const defaultsJson = {
|
||||
@@ -227,7 +232,7 @@ export function pluginManagedAgentService(
|
||||
budgetMonthlyCents: declaration.budgetMonthlyCents ?? 0,
|
||||
instructions: declaration.instructions ?? null,
|
||||
};
|
||||
const managedResource = await db
|
||||
const managedResource = await client
|
||||
.select({ id: pluginManagedResources.id })
|
||||
.from(pluginManagedResources)
|
||||
.where(and(
|
||||
@@ -238,12 +243,12 @@ export function pluginManagedAgentService(
|
||||
))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (managedResource) {
|
||||
await db
|
||||
await client
|
||||
.update(pluginManagedResources)
|
||||
.set({ resourceId: agentId, defaultsJson, updatedAt: new Date() })
|
||||
.where(eq(pluginManagedResources.id, managedResource.id));
|
||||
} else {
|
||||
await db.insert(pluginManagedResources).values({
|
||||
await client.insert(pluginManagedResources).values({
|
||||
companyId,
|
||||
pluginId: options.pluginId,
|
||||
pluginKey: options.pluginKey,
|
||||
@@ -265,9 +270,9 @@ export function pluginManagedAgentService(
|
||||
lastReconciledAt: new Date().toISOString(),
|
||||
...extraData,
|
||||
};
|
||||
const existing = await getBinding(companyId, declaration.agentKey);
|
||||
const existing = await getBinding(companyId, declaration.agentKey, client);
|
||||
if (existing) {
|
||||
return db
|
||||
return client
|
||||
.update(pluginEntities)
|
||||
.set({
|
||||
scopeKind: "company",
|
||||
@@ -281,7 +286,7 @@ export function pluginManagedAgentService(
|
||||
.returning()
|
||||
.then((rows) => rows[0]);
|
||||
}
|
||||
return db
|
||||
return client
|
||||
.insert(pluginEntities)
|
||||
.values({
|
||||
pluginId: options.pluginId,
|
||||
@@ -305,6 +310,46 @@ export function pluginManagedAgentService(
|
||||
return rows.find((row) => rowIsManagedAgent(row, options.pluginKey, declaration.agentKey)) ?? null;
|
||||
}
|
||||
|
||||
async function relinkManagedAgent(companyId: string, declaration: PluginManagedAgentDeclaration, agentId: string) {
|
||||
const adapterType = await resolveManagedAdapterType(companyId, declaration);
|
||||
return db.transaction(async (tx) => {
|
||||
const [agent] = await tx.select().from(agents).where(and(
|
||||
eq(agents.id, agentId), eq(agents.companyId, companyId), ne(agents.status, "terminated"),
|
||||
)).for("update");
|
||||
if (!agent || !rowIsManagedAgent(agent, options.pluginKey, declaration.agentKey)) {
|
||||
throw conflict("Managed agent ownership changed before relink");
|
||||
}
|
||||
const [plugin] = await tx.select().from(plugins).where(and(
|
||||
eq(plugins.id, options.pluginId), eq(plugins.pluginKey, options.pluginKey),
|
||||
)).for("share");
|
||||
if (!plugin || plugin.status !== "ready" || !plugin.manifestJson.capabilities.includes("agents.managed")
|
||||
|| !plugin.manifestJson.agents?.some((entry) => entry.agentKey === declaration.agentKey)) {
|
||||
throw forbidden("Plugin relink is limited to its registered managed agent declaration");
|
||||
}
|
||||
const marker = agent.metadata!.paperclipManagedResource as Record<string, unknown>;
|
||||
const previousPluginId = marker.pluginId;
|
||||
if (typeof previousPluginId !== "string" || !isUuidLike(previousPluginId)) throw forbidden("Managed agent has no plugin owner");
|
||||
if (previousPluginId !== options.pluginId) {
|
||||
const [previousPlugin] = await tx.select({ id: plugins.id }).from(plugins).where(eq(plugins.id, previousPluginId));
|
||||
if (previousPlugin) throw forbidden("Managed agent still belongs to another installed plugin");
|
||||
}
|
||||
// Hard uninstall cascades the bindings, but deliberately leaves the agent
|
||||
// and its canonical history. Transfer only its proven stable-key marker to
|
||||
// the replacement installation, atomically with the new scoped bindings.
|
||||
await upsertBinding(companyId, declaration, agentId, {}, adapterType, tx);
|
||||
const [relinked] = await tx.update(agents).set({
|
||||
metadata: managedMetadata(options.pluginId, options.pluginKey, declaration, agent.metadata),
|
||||
updatedAt: new Date(),
|
||||
}).where(and(eq(agents.id, agentId), eq(agents.companyId, companyId))).returning();
|
||||
await tx.insert(activityLog).values({
|
||||
companyId, actorType: "plugin", actorId: options.pluginId,
|
||||
action: "plugin.managed_agent.relinked", entityType: "agent", entityId: agentId,
|
||||
details: { sourcePluginKey: options.pluginKey, managedResourceKey: declaration.agentKey, previousPluginId },
|
||||
});
|
||||
return relinked as Agent;
|
||||
});
|
||||
}
|
||||
|
||||
async function companyAdapterUsage(companyId: string) {
|
||||
const rows = await db
|
||||
.select({ adapterType: agents.adapterType })
|
||||
@@ -336,23 +381,39 @@ export function pluginManagedAgentService(
|
||||
const declared = declaredInstructionFiles(declaration, variables);
|
||||
if (!declared) return agent;
|
||||
|
||||
const materialized = await instructions.materializeManagedBundle(
|
||||
agent,
|
||||
declared.files,
|
||||
{
|
||||
entryFile: declared.entryFile,
|
||||
replaceExisting: materializeOptions.replaceExisting,
|
||||
clearLegacyPromptTemplate: true,
|
||||
},
|
||||
);
|
||||
let adapterConfig: Record<string, unknown>;
|
||||
if (materializeOptions.replaceExisting) {
|
||||
const revisions = agentInstructionRevisionService(db);
|
||||
const target = { companyId, agentId: agent.id };
|
||||
const actor = { type: "plugin" as const, pluginId: options.pluginId, pluginKey: options.pluginKey, agentKey: declaration.agentKey };
|
||||
const baseline = await revisions.readForPluginReset(target, actor);
|
||||
const receipt = await revisions.commitPluginReset({ ...target, entryFile: declared.entryFile,
|
||||
baseRevisionId: baseline.snapshot?.revision.id ?? null, configuredEntryFile: baseline.configuredEntryFile, content: declared.files[declared.entryFile] ?? "" }, actor);
|
||||
if (receipt.materialization === "pending") throw conflict("Instruction revision saved; retry reset to repair its disk copy", { revisionId: receipt.revision.id });
|
||||
const refreshed = await agentSvc.getById(agent.id);
|
||||
if (!refreshed) throw notFound("Managed agent not found");
|
||||
for (const [file, content] of Object.entries(declared.files)) {
|
||||
if (file !== declared.entryFile) await instructions.writeFile(refreshed, file, content);
|
||||
}
|
||||
// A stock reset owns declared instruction paths, not the agent's other
|
||||
// persistent files (including a formerly configured entry).
|
||||
adapterConfig = { ...refreshed.adapterConfig };
|
||||
delete adapterConfig.promptTemplate;
|
||||
delete adapterConfig.bootstrapPromptTemplate;
|
||||
} else {
|
||||
const materialized = await instructions.materializeManagedBundle(agent, declared.files, {
|
||||
entryFile: declared.entryFile, replaceExisting: false, clearLegacyPromptTemplate: true,
|
||||
});
|
||||
adapterConfig = materialized.adapterConfig;
|
||||
}
|
||||
const updated = await agentSvc.update(agent.id, {
|
||||
adapterConfig: materialized.adapterConfig,
|
||||
adapterConfig,
|
||||
}, {
|
||||
recordRevision: {
|
||||
source: `plugin:${optionsForRevisionSource()}:managed-agent-instructions`,
|
||||
},
|
||||
});
|
||||
return (updated as Agent | null) ?? { ...agent, adapterConfig: materialized.adapterConfig };
|
||||
return (updated as Agent | null) ?? { ...agent, adapterConfig };
|
||||
}
|
||||
|
||||
async function managedInstructionDefaultDrift(
|
||||
@@ -375,8 +436,7 @@ export function pluginManagedAgentService(
|
||||
return { entryFile: declared.entryFile, changedFiles: [declared.entryFile] };
|
||||
}
|
||||
|
||||
const paths = new Set([...Object.keys(declared.files), ...Object.keys(exported.files)]);
|
||||
const changedFiles = [...paths]
|
||||
const changedFiles = Object.keys(declared.files)
|
||||
.filter((filePath) => (exported.files[filePath] ?? null) !== (declared.files[filePath] ?? null))
|
||||
.sort((left, right) => left.localeCompare(right));
|
||||
if (exported.entryFile !== declared.entryFile && !changedFiles.includes(declared.entryFile)) {
|
||||
@@ -433,7 +493,7 @@ export function pluginManagedAgentService(
|
||||
spentMonthlyCents: 0,
|
||||
lastHeartbeatAt: null,
|
||||
}) as Agent;
|
||||
created = await materializeDeclaredInstructions(companyId, created, declaration, { replaceExisting: true });
|
||||
created = await materializeDeclaredInstructions(companyId, created, declaration, { replaceExisting: false });
|
||||
|
||||
let approvalId: string | null = null;
|
||||
if (requiresApproval) {
|
||||
@@ -574,9 +634,7 @@ export function pluginManagedAgentService(
|
||||
|
||||
const relinkCandidate = await findRelinkCandidate(companyId, declaration);
|
||||
if (relinkCandidate) {
|
||||
await upsertBinding(companyId, declaration, relinkCandidate.id);
|
||||
const relinkedAgent = await agentSvc.getById(relinkCandidate.id) as Agent | null;
|
||||
if (!relinkedAgent) throw notFound("Managed agent not found");
|
||||
const relinkedAgent = await relinkManagedAgent(companyId, declaration, relinkCandidate.id);
|
||||
const agent = await backfillManagedPauseReason(
|
||||
companyId,
|
||||
declaration,
|
||||
@@ -596,8 +654,19 @@ export function pluginManagedAgentService(
|
||||
? reconciled.agent.metadata
|
||||
: {};
|
||||
const adapterType = await resolveManagedAdapterType(companyId, declaration);
|
||||
// Reset content through the canonical CAS path before changing defaults.
|
||||
// A conflict must leave the existing adapter configuration untouched.
|
||||
const withInstructions = await materializeDeclaredInstructions(companyId, reconciled.agent, declaration, { replaceExisting: true });
|
||||
const defaults = declarationPatch(declaration, { adapterType });
|
||||
const adapterConfig = { ...defaults.adapterConfig } as Record<string, unknown>;
|
||||
if (declaration.instructions) {
|
||||
for (const key of ["instructionsBundleMode", "instructionsRootPath", "instructionsEntryFile", "instructionsFilePath"]) {
|
||||
if (withInstructions.adapterConfig[key] !== undefined) adapterConfig[key] = withInstructions.adapterConfig[key];
|
||||
}
|
||||
}
|
||||
const updated = await agentSvc.update(reconciled.agent.id, {
|
||||
...declarationPatch(declaration, { adapterType }),
|
||||
...defaults,
|
||||
adapterConfig,
|
||||
metadata: managedMetadata(options.pluginId, options.pluginKey, declaration, currentMetadata),
|
||||
}, {
|
||||
recordRevision: {
|
||||
@@ -605,7 +674,7 @@ export function pluginManagedAgentService(
|
||||
},
|
||||
});
|
||||
if (!updated) throw notFound("Managed agent not found");
|
||||
const updatedAgent = await materializeDeclaredInstructions(companyId, updated as Agent, declaration, { replaceExisting: true });
|
||||
const updatedAgent = updated as Agent;
|
||||
await upsertBinding(companyId, declaration, updatedAgent.id, {}, adapterType);
|
||||
await logActivity(db, {
|
||||
companyId,
|
||||
|
||||
Reference in new issue
Block a user