mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 11:54:35 +01:00
fix(runner): preserve tool outcomes through shutdown and restart (#14734)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The runner sends authorized tool calls to the server and saves their results. > - A provider turn can stop while a server write is still running. > - The old shutdown path invented a failed result that could conflict with the real result. > - Truncated execution input and incomplete recovery records made the failure harder to diagnose. > - This pull request preserves exact inputs and actual outcomes through shutdown and restart. > - Tests force the race and crash boundaries so safe retries do not repeat writes. ## Linked Issues or Issue Description **What happened?** Stopping a turn during a server tool call could record a false failure, then reject the actual result as a conflict. The diagnostic input formatter could truncate instruction content before execution. A crash during saved-result delivery could leave that delivery permanently indeterminate. Cleanup could hide the first failure, and a retry could overwrite earlier run logs. **Expected behavior** Keep dispatched tools pending until their actual result is known. Preserve accepted input bytes. Accept identical result delivery without failing the task. Reject conflicting results with enough evidence to diagnose them. Recover saved-result delivery without repeating the business operation. **Steps to reproduce** 1. Hold an instruction update at the filesystem commit barrier. 2. Stop its provider turn before the server returns the result. 3. Release the write, deliver its result, and replay the same result. 4. Repeat with a restart before and after the delivery receipt is saved. 5. Check that there is one write and one audit row, and that the exact result survives. **Paperclip version or commit** The change was developed from `44736c9c7` and rebased onto `0e5830887`. **Deployment mode** Self-hosted server with the native runner. Tests use local runner processes, scripted providers, and PostgreSQL. Related work: #12353 added durable semantic tool receipts; #12384 added durable Codex tool recovery; #12404 bound semantic tools to ACPX sessions. #14633 covers separate native-provider cancellation and qualification work. This PR addresses server semantic-tool outcomes and their durable delivery. No duplicate fix was found. AgentMail discovery is outside this PR. ## What Changed - Close turn admission without inventing results for dispatched tools. Keep pending calls and accept late actual results. - Accept identical result replay with a diagnostic warning. Include call identity and both result hashes in real conflict errors. - Preserve exact execution arguments. Reject prohibited or oversized input before dispatch. Keep diagnostic previews redacted and bounded. - Commit instruction-attempt evidence before the filesystem write. Save completed mutation receipts so concurrent and restarted duplicates return the first result. Recheck authorization before replay. An attempt without a completed result stays unknown and cannot execute again. Definite pre-write failures save and replay their original error without another write. - Recover an interrupted saved-result delivery only for backends with durable result receipts. Never replay an ordinary business operation with an unknown outcome. - Preserve the initiating error when cleanup also fails. Record incomplete settlement evidence. Propagate typed unknown-outcome errors through the native tool wrapper without creating a false completed tool result. - Append run-log attempts and restore the durable log before appending after local file loss. Reject incomplete restores. Publish a restored prefix only if the destination is absent so concurrent attempts cannot overwrite new lines. - Add deterministic race, crash, replay, authorization, exact-content, and log-restoration tests. Document their assertions in `packages/paperclip-runner/docs/durable-recovery.md`. ## Verification - Current head: `7e088f4c7fba8ebabf98ae95485a5753b013d489`. All 55 applicable checks pass; four conditional/manual checks are skipped. This includes build, typecheck, Rust, both runner TypeScript shards, server and workspace tests, all eight browser shards, isolated runner compilation, and the clean-install release dry run. [CI run](https://github.com/paperclipai/paperclip/actions/runs/36746101110). - Greptile reviewed this exact head at 5/5 with zero new findings. All three earlier review threads are resolved. - Focused local verification includes 11 instruction integration tests, 23 surrounding authority/tool tests, 26 run-log tests, and 169 controller/driver tests. The post-rebase controller/transport/runtime selection passed 415 tests. The full Rust release suite passed 617 tests with two ignored. The real-process SIGKILL recovery test passed three consecutive runs. - The fault matrix in `packages/paperclip-runner/docs/durable-recovery.md` uses explicit barriers, real PostgreSQL rollback, durable journal reloads, and killed runner processes. It covers late results, identical and conflicting replay, exact long content, concurrent log restoration, lost commit acknowledgements, and definite failure replay after the original CAS base becomes valid again. No paid model calls are needed. - Full local recursive typecheck and build passed during implementation. Server typecheck and the runner TypeScript build passed after the review fixes. The broad local repository test run was stopped after repeated database startup timeouts. Four timing/launch failures in an earlier broad runner run passed focused reruns without changed assertions or timeouts. These are local verification limitations; the complete current-head CI suite is green. An earlier CI workspace job received an infrastructure shutdown signal; its current-head replacement passed. ## Risks - A stopped turn can remain blocked when a dispatched operation has no proven result. The system does not guess its outcome or rerun its effect. - Conflicting results still fail settlement. Existing failed or conflicting journals are not repaired automatically. - Accepted semantic input is limited to 480 KiB of encoded JSON to fit the encrypted transport. Larger input fails before execution. - Instruction filesystem writes and database receipts are not one atomic storage operation. A separately committed attempt and audit record survive rollback. An attempt without a completed success or definite pre-write failure receipt remains blocked as an unknown outcome. It is not replayed or reported as success. - Run-log restoration now reads the durable object before appending when the local log is missing. Failed or incomplete reads reject the append. - No schema migration, dependency change, workflow change, or AgentMail change is included. ## Model Used OpenAI Codex, GPT-6, with reasoning, tool use, code execution, and test analysis. The exact served model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused suites; the broad local run limitation is recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
af5c2d101c
commit
94e8dec56b
30 files changed
+1676
-565
No files matched your search
@@ -1,10 +1,10 @@
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it } from "vitest";
|
||||
import { agents, authUsers, companies, companyMemberships, principalPermissionGrants, heartbeatRuns, issues, agentInstructionRevisions, createDb } from "@paperclipai/db";
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { activityLog, agents, authUsers, companies, companyMemberships, principalPermissionGrants, heartbeatRuns, issues, agentInstructionRevisions, createDb } from "@paperclipai/db";
|
||||
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
||||
import { resolveManagedInstructionsRoot } from "../agent-instructions.js";
|
||||
import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js";
|
||||
@@ -64,6 +64,189 @@ describe("canonical instruction tools through native authority", () => {
|
||||
await expect(call("update_agent_instructions", { entryFile, content: "stale", baseRevisionId: first.revision.id })).rejects.toMatchObject({ status: 409 });
|
||||
expect((await call("read_agent_instructions")).content).toBe("changed\r\n");
|
||||
});
|
||||
it("serializes duplicate writes at the filesystem commit and replays the exact receipt after restart", async () => {
|
||||
const first = await call("read_agent_instructions");
|
||||
const content = "\uFEFF# Instructions — 日本語 🦀\r\n".repeat(600) + "\nFINAL TAIL\n";
|
||||
const request = { tool: "update_agent_instructions", callId: randomUUID(), arguments: {
|
||||
targetAgentId, entryFile, content, baseRevisionId: first.revision.id,
|
||||
} };
|
||||
const originalRename = fs.rename.bind(fs);
|
||||
let reached!: () => void, release!: () => void;
|
||||
const atCommit = new Promise<void>((resolve) => { reached = resolve; });
|
||||
const barrier = new Promise<void>((resolve) => { release = resolve; });
|
||||
let writes = 0;
|
||||
const spy = vi.spyOn(fs, "rename").mockImplementation(async (source, destination) => {
|
||||
if (destination === path.join(root, entryFile)) {
|
||||
writes++;
|
||||
reached();
|
||||
await barrier;
|
||||
}
|
||||
return originalRename(source, destination);
|
||||
});
|
||||
try {
|
||||
const pending = authority.execute(request);
|
||||
await atCommit;
|
||||
const duplicate = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId }).execute(request);
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(original);
|
||||
release();
|
||||
const receipt = await pending as any;
|
||||
expect(await duplicate).toEqual(receipt);
|
||||
authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId });
|
||||
expect(await authority.execute(request)).toEqual(receipt);
|
||||
const bytes = await fs.readFile(path.join(root, entryFile));
|
||||
expect(bytes).toEqual(Buffer.from(content));
|
||||
expect(receipt.revision.byteLength).toBe(bytes.length);
|
||||
expect(receipt.revision.contentHash).toBe(createHash("sha256").update(bytes).digest("hex"));
|
||||
expect(writes).toBe(1);
|
||||
expect((await db.select().from(activityLog).where(eq(activityLog.runId, runId))).filter((event) => event.action === "agent.files_updated")).toHaveLength(1);
|
||||
await expect(authority.execute({ ...request, arguments: { ...request.arguments, content: "conflicting replay" } })).rejects.toThrow("idempotency_conflict");
|
||||
expect(await fs.readFile(path.join(root, entryFile))).toEqual(bytes);
|
||||
await expect(call("update_agent_instructions", { entryFile, baseRevisionId: receipt.revision.id, content: "🦀".repeat(300_000) })).rejects.toMatchObject({ status: 422 });
|
||||
expect(writes).toBe(1);
|
||||
await db.delete(principalPermissionGrants).where(eq(principalPermissionGrants.principalId, userId));
|
||||
await expect(authority.execute(request)).rejects.toMatchObject({ status: 403 });
|
||||
} finally {
|
||||
release();
|
||||
spy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["after_file_rename", "receipt_persistence"])("retains durable attempt evidence and refuses replay after rollback at %s", async (fault) => {
|
||||
const first = await call("read_agent_instructions");
|
||||
const request = { tool: "update_agent_instructions", callId: randomUUID(), arguments: {
|
||||
targetAgentId, entryFile, content: "written before database rollback\r\n", baseRevisionId: first.revision.id,
|
||||
} };
|
||||
const key = `instruction:${request.callId}`;
|
||||
if (fault === "receipt_persistence") {
|
||||
// This is an actual PostgreSQL transaction rollback, after the real
|
||||
// filesystem rename and attempted audit insert, before receipt commit.
|
||||
await db.execute(sql.raw(`CREATE FUNCTION reject_instruction_test_receipt() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
IF NEW.id = '${runId}' AND NEW.result_json->'semanticToolReceipts' ? '${key}' THEN
|
||||
RAISE EXCEPTION 'injected receipt persistence failure';
|
||||
END IF;
|
||||
RETURN NEW;
|
||||
END $$`));
|
||||
await db.execute(sql.raw(`CREATE TRIGGER reject_instruction_test_receipt BEFORE UPDATE ON heartbeat_runs
|
||||
FOR EACH ROW EXECUTE FUNCTION reject_instruction_test_receipt()`));
|
||||
}
|
||||
const originalRename = fs.rename.bind(fs);
|
||||
let writes = 0;
|
||||
const spy = vi.spyOn(fs, "rename").mockImplementation(async (source, destination) => {
|
||||
await originalRename(source, destination);
|
||||
if (destination === path.join(root, entryFile)) {
|
||||
writes++;
|
||||
if (fault === "after_file_rename") throw new Error("injected failure after rename");
|
||||
}
|
||||
});
|
||||
try {
|
||||
await expect(authority.execute(request)).rejects.toMatchObject({ code: "semantic_tool_outcome_unknown" });
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(request.arguments.content);
|
||||
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
expect((run!.resultJson as any).instructionToolAttempts[key]).toMatchObject({
|
||||
operationId: request.tool, targetAgentId, inputDigest: expect.stringMatching(/^[a-f0-9]{64}$/),
|
||||
});
|
||||
expect((run!.resultJson as any).semanticToolReceipts?.[key]).toBeUndefined();
|
||||
const audit = await db.select().from(activityLog).where(eq(activityLog.runId, runId));
|
||||
expect(audit.filter((event) => event.action === "agent.instruction_write_attempted")).toHaveLength(1);
|
||||
expect(audit.filter((event) => event.action === "agent.files_updated")).toHaveLength(0);
|
||||
authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId });
|
||||
await expect(authority.execute(request)).rejects.toMatchObject({ code: "semantic_tool_outcome_unknown" });
|
||||
await expect(authority.execute({ ...request, arguments: { ...request.arguments, content: "different" } }))
|
||||
.rejects.toThrow("idempotency_conflict");
|
||||
expect(writes).toBe(1);
|
||||
await db.delete(principalPermissionGrants).where(eq(principalPermissionGrants.principalId, userId));
|
||||
await expect(authority.execute(request)).rejects.toMatchObject({ status: 403 });
|
||||
} finally {
|
||||
spy.mockRestore();
|
||||
if (fault === "receipt_persistence") {
|
||||
await db.execute(sql.raw("DROP TRIGGER reject_instruction_test_receipt ON heartbeat_runs"));
|
||||
await db.execute(sql.raw("DROP FUNCTION reject_instruction_test_receipt()"));
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
it("replays a committed receipt after its database commit acknowledgement is lost", async () => {
|
||||
const first = await call("read_agent_instructions");
|
||||
const request = { tool: "update_agent_instructions", callId: randomUUID(), arguments: {
|
||||
targetAgentId, entryFile, content: "committed but acknowledgement lost\n", baseRevisionId: first.revision.id,
|
||||
} };
|
||||
const transaction = db.transaction.bind(db);
|
||||
let commits = 0;
|
||||
const spy = vi.spyOn(db, "transaction").mockImplementation(async (...args) => {
|
||||
const result = await transaction(...args);
|
||||
if (++commits === 2) throw new Error("injected lost commit acknowledgement");
|
||||
return result;
|
||||
});
|
||||
try {
|
||||
await expect(authority.execute(request)).rejects.toThrow("injected lost commit acknowledgement");
|
||||
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
const receipt = (run!.resultJson as any).semanticToolReceipts[`instruction:${request.callId}`].result;
|
||||
expect(receipt).toMatchObject({ changed: true, content: request.arguments.content });
|
||||
authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId });
|
||||
expect(await authority.execute(request)).toEqual(receipt);
|
||||
const audit = await db.select().from(activityLog).where(eq(activityLog.runId, runId));
|
||||
expect(audit.filter((event) => event.action === "agent.instruction_write_attempted")).toHaveLength(1);
|
||||
expect(audit.filter((event) => event.action === "agent.files_updated")).toHaveLength(1);
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(request.arguments.content);
|
||||
} finally { spy.mockRestore(); }
|
||||
});
|
||||
|
||||
it.each(["validation", "stale_base", "oversized"])("replays a definite %s failure after restart without executing again", async (fault) => {
|
||||
const first = await call("read_agent_instructions");
|
||||
const request = { tool: "update_agent_instructions", callId: randomUUID(), arguments: {
|
||||
targetAgentId, entryFile, content: fault === "oversized" ? "🦀".repeat(300_000) : "requested update",
|
||||
baseRevisionId: first.revision.id, ...(fault === "validation" ? { unexpected: true } : {}),
|
||||
} };
|
||||
if (fault === "stale_base") await fs.writeFile(path.join(root, entryFile), "newer canonical bytes");
|
||||
const originalRename = fs.rename.bind(fs);
|
||||
let writes = 0;
|
||||
const spy = vi.spyOn(fs, "rename").mockImplementation(async (source, destination) => {
|
||||
if (destination === path.join(root, entryFile)) writes++;
|
||||
return originalRename(source, destination);
|
||||
});
|
||||
try {
|
||||
const failure = await authority.execute(request).then(() => null, (error: unknown) => error) as any;
|
||||
expect(failure).toMatchObject({ status: fault === "validation" ? 400 : fault === "stale_base" ? 409 : 422 });
|
||||
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
const saved = (run!.resultJson as any).instructionToolAttempts[`instruction:${request.callId}`].failure;
|
||||
expect(saved).toMatchObject({ status: failure.status, message: failure.message });
|
||||
// The stale request would now pass CAS if it were executed again.
|
||||
await fs.writeFile(path.join(root, entryFile), original);
|
||||
authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId });
|
||||
await expect(authority.execute(request)).rejects.toMatchObject(saved);
|
||||
await expect(authority.execute(request)).rejects.toMatchObject(saved);
|
||||
await expect(authority.execute({ ...request, arguments: { ...request.arguments, content: "different" } }))
|
||||
.rejects.toThrow("idempotency_conflict");
|
||||
expect(writes).toBe(0);
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(original);
|
||||
const audit = await db.select().from(activityLog).where(eq(activityLog.runId, runId));
|
||||
expect(audit.filter((event) => event.action === "agent.instruction_write_attempted")).toHaveLength(1);
|
||||
expect(audit.filter((event) => event.action === "agent.files_updated")).toHaveLength(0);
|
||||
} finally { spy.mockRestore(); }
|
||||
});
|
||||
|
||||
it("does not claim a final failure when its failure receipt cannot be persisted", async () => {
|
||||
const first = await call("read_agent_instructions");
|
||||
const request = { tool: "update_agent_instructions", callId: randomUUID(), arguments: {
|
||||
targetAgentId, entryFile, content: "invalid input", baseRevisionId: first.revision.id, unexpected: true,
|
||||
} };
|
||||
const transaction = db.transaction.bind(db);
|
||||
let attempts = 0;
|
||||
const spy = vi.spyOn(db, "transaction").mockImplementation(async (...args) => {
|
||||
if (++attempts === 3) throw new Error("injected failure receipt outage");
|
||||
return transaction(...args);
|
||||
});
|
||||
try {
|
||||
await expect(authority.execute(request)).rejects.toMatchObject({ code: "semantic_tool_outcome_unknown" });
|
||||
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
expect((run!.resultJson as any).instructionToolAttempts[`instruction:${request.callId}`].failure).toBeUndefined();
|
||||
authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId });
|
||||
await expect(authority.execute(request)).rejects.toMatchObject({ code: "semantic_tool_outcome_unknown" });
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe(original);
|
||||
} finally { spy.mockRestore(); }
|
||||
});
|
||||
|
||||
it("rejects supplied identity and missing CAS bases before changing content", async () => {
|
||||
const first = await call("read_agent_instructions");
|
||||
for (const identity of ["companyId", "agentId", "runId", "responsibleUserId", "onBehalfOfUserId", "actor"]) {
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { authorizeInstructionCommit } from "../agent-instruction-authorization.js";
|
||||
import { executeAgentInstructionTool } from "./agent-instruction-tools.js";
|
||||
import { createReadStream } from "node:fs";
|
||||
import { publicChatTaskUrl } from "../chat-task-url.js";
|
||||
@@ -26,7 +27,7 @@ import { getStorageService } from "../../storage/index.js";
|
||||
import type { StorageService } from "../../storage/types.js";
|
||||
import { assetService } from "../assets.js";
|
||||
import { workspaceFileResourceService } from "../workspace-file-resources.js";
|
||||
import { badRequest, forbidden } from "../../errors.js";
|
||||
import { badRequest, forbidden, notFound, HttpError } from "../../errors.js";
|
||||
import { searchRunnerApi } from "./runner-api-catalog.js";
|
||||
import { executeRunnerApi, validateRunnerApiCall, RUNNER_API_MAX_BYTES, type RunnerApiFile } from "./runner-api-client.js";
|
||||
import { acquireRunnerApiResponseSlot, runnerApiCompanyCaptureMaxBytes, RUNNER_API_RESPONSE_MAX_BYTES, RUNNER_API_RESPONSE_RUN_MAX_BYTES, RunnerApiResponseLimitError } from "./runner-api-response-limits.js";
|
||||
@@ -48,7 +49,7 @@ import {
|
||||
issues,
|
||||
issueThreadInteractions,
|
||||
} from "@paperclipai/db";
|
||||
import { CAPABILITY_SEMANTIC_TOOL_CATALOG, runnerCodexDynamicToolsFit } from "../../vendor/paperclip-runner/index.js";
|
||||
import { CAPABILITY_SEMANTIC_TOOL_CATALOG, runnerCodexDynamicToolsFit, SemanticToolOutcomeUnknownError } from "../../vendor/paperclip-runner/index.js";
|
||||
import { agentService } from "../agents.js";
|
||||
import { approvalService } from "../approvals.js";
|
||||
import { documentService } from "../documents.js";
|
||||
@@ -139,6 +140,25 @@ type ToolReceipt = {
|
||||
result: unknown;
|
||||
};
|
||||
|
||||
// Coalesce live callers, but never use this process-local lock as crash proof.
|
||||
// The database attempt record below is the authority after process loss.
|
||||
const instructionCallLocks = new WeakMap<Db, Map<string, Promise<void>>>();
|
||||
const instructionPreWriteFailureStatuses = new Set([400, 401, 403, 404, 409, 422]);
|
||||
async function withInstructionCallLock<T>(db: Db, key: string, work: () => Promise<T>): Promise<T> {
|
||||
let locks = instructionCallLocks.get(db);
|
||||
if (!locks) instructionCallLocks.set(db, locks = new Map());
|
||||
const previous = locks.get(key);
|
||||
let release!: () => void;
|
||||
const pending = new Promise<void>((resolve) => { release = resolve; });
|
||||
locks.set(key, pending);
|
||||
await previous;
|
||||
try { return await work(); }
|
||||
finally {
|
||||
release();
|
||||
if (locks.get(key) === pending) locks.delete(key);
|
||||
}
|
||||
}
|
||||
|
||||
function record(value: unknown): Record<string, unknown> {
|
||||
return typeof value === "object" && value !== null && !Array.isArray(value)
|
||||
? value as Record<string, unknown>
|
||||
@@ -371,12 +391,14 @@ export class PaperclipRunnerToolAuthority {
|
||||
}
|
||||
switch (call.tool) {
|
||||
case "read_agent_instructions":
|
||||
case "update_agent_instructions":
|
||||
case "get_agent_instruction_history":
|
||||
case "restore_agent_instructions":
|
||||
return executeAgentInstructionTool({ db: this.db, binding: {
|
||||
companyId: this.binding.companyId, agentId: this.binding.agentId, runId: this.binding.runId,
|
||||
}, tool: call.tool, arguments: call.arguments });
|
||||
case "update_agent_instructions":
|
||||
case "restore_agent_instructions": {
|
||||
return this.#withInstructionMutationReceipt(call.tool, call.callId, input);
|
||||
}
|
||||
case "create_skill": {
|
||||
const apiUrl = this.binding.apiUrl ?? process.env.PAPERCLIP_API_URL;
|
||||
const token = createLocalAgentJwt(this.binding.agentId, this.binding.companyId, context.actor.adapterType, this.binding.runId, context.run.responsibleUserId);
|
||||
@@ -1476,6 +1498,109 @@ export class PaperclipRunnerToolAuthority {
|
||||
return null;
|
||||
}
|
||||
|
||||
async #withInstructionMutationReceipt(
|
||||
tool: "update_agent_instructions" | "restore_agent_instructions",
|
||||
callId: string,
|
||||
input: Record<string, unknown>,
|
||||
): Promise<unknown> {
|
||||
if (!callId || callId.length > 500) throw badRequest("A bounded runner call id is required");
|
||||
const key = `instruction:${callId}`;
|
||||
const digest = createHash("sha256").update(canonicalJson(input)).digest("hex");
|
||||
const targetAgentId = typeof input.targetAgentId === "string" ? input.targetAgentId : this.binding.agentId;
|
||||
const authorize = async (tx: Db) => {
|
||||
const [target] = await tx.select({ id: agents.id, companyId: agents.companyId }).from(agents)
|
||||
.where(and(eq(agents.id, targetAgentId), eq(agents.companyId, this.binding.companyId)));
|
||||
if (!target) throw notFound("Agent not found");
|
||||
return authorizeInstructionCommit(tx, {
|
||||
type: "agent", source: "agent_jwt", companyId: this.binding.companyId,
|
||||
agentId: this.binding.agentId, runId: this.binding.runId,
|
||||
}, target);
|
||||
};
|
||||
return withInstructionCallLock(this.db, `${this.binding.companyId}:${this.binding.runId}:${key}`, async () => {
|
||||
// A filesystem rename cannot roll back with PostgreSQL. Commit proof of
|
||||
// the attempt *before* permitting that effect. If the effect transaction
|
||||
// or its acknowledgement is lost, a missing result is unknown, not a
|
||||
// license to run the instruction write a second time.
|
||||
const reserved = await this.db.transaction(async (tx) => {
|
||||
const context = await this.#lockAuthorizedMutationContext(tx as unknown as Db);
|
||||
const bound = await authorize(tx as unknown as Db);
|
||||
const resultJson = record(context.run.resultJson);
|
||||
if (record(resultJson.semanticToolReceipts)[key] !== undefined) return false;
|
||||
const attempts = record(resultJson.instructionToolAttempts);
|
||||
const prior = record(attempts[key]);
|
||||
if (attempts[key] !== undefined) {
|
||||
if (prior.operationId !== tool || prior.inputDigest !== digest) {
|
||||
throw new Error("paperclip_runner_tool_idempotency_conflict");
|
||||
}
|
||||
const failure = record(prior.failure);
|
||||
if (typeof failure.status === "number" && instructionPreWriteFailureStatuses.has(failure.status)
|
||||
&& typeof failure.message === "string") {
|
||||
throw new HttpError(failure.status, failure.message, failure.details);
|
||||
}
|
||||
throw new SemanticToolOutcomeUnknownError(`paperclip_runner_instruction_outcome_unknown call_id=${callId} operation_id=${tool} input_digest=${digest}`);
|
||||
}
|
||||
if (Object.keys(attempts).length >= 512) throw badRequest("Run instruction mutation limit reached");
|
||||
attempts[key] = { operationId: tool, inputDigest: digest, targetAgentId };
|
||||
await tx.update(heartbeatRuns).set({ resultJson: { ...resultJson, instructionToolAttempts: attempts } })
|
||||
.where(eq(heartbeatRuns.id, this.binding.runId));
|
||||
await tx.insert(activityLog).values({
|
||||
companyId: this.binding.companyId, actorType: "agent", actorId: this.binding.agentId,
|
||||
agentId: this.binding.agentId, runId: this.binding.runId,
|
||||
responsibleUserId: bound.type === "agent" ? bound.onBehalfOfUserId : null,
|
||||
action: "agent.instruction_write_attempted", entityType: "agent", entityId: targetAgentId,
|
||||
details: { callId, operationId: tool, inputDigest: digest },
|
||||
});
|
||||
return true;
|
||||
});
|
||||
try {
|
||||
return await this.#withMutationReceipt(tool, key, input, (tx) =>
|
||||
executeAgentInstructionTool({ db: tx, binding: {
|
||||
companyId: this.binding.companyId, agentId: this.binding.agentId, runId: this.binding.runId,
|
||||
}, tool, arguments: input }), { beforeReceiptReplay: async (tx) => { await authorize(tx); } });
|
||||
} catch (error) {
|
||||
// Instruction validation, authorization and CAS errors precede the
|
||||
// file write. Storage errors and lost commit acknowledgements do not
|
||||
// prove that a reserved filesystem effect rolled back.
|
||||
if (!reserved) throw error;
|
||||
if (error instanceof HttpError && instructionPreWriteFailureStatuses.has(error.status)) {
|
||||
const failure = { status: error.status, message: error.message, ...(error.details === undefined ? {} : { details: error.details }) };
|
||||
try {
|
||||
// Save the definite failure before exposing it as a final answer.
|
||||
// This is evidence of the admitted call, not another mutation, so
|
||||
// retain it even if the run stopped after the effect rolled back.
|
||||
await this.db.transaction(async (tx) => {
|
||||
const [run] = await tx.select().from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.id, this.binding.runId), eq(heartbeatRuns.companyId, this.binding.companyId),
|
||||
eq(heartbeatRuns.agentId, this.binding.agentId),
|
||||
)).for("update");
|
||||
if (!run) throw new Error("instruction_failure_run_missing");
|
||||
const resultJson = record(run.resultJson);
|
||||
const attempts = record(resultJson.instructionToolAttempts);
|
||||
const prior = record(attempts[key]);
|
||||
if (prior.operationId !== tool || prior.inputDigest !== digest
|
||||
|| record(resultJson.semanticToolReceipts)[key] !== undefined) {
|
||||
throw new Error("instruction_failure_receipt_conflict");
|
||||
}
|
||||
attempts[key] = { ...prior, failure };
|
||||
await tx.update(heartbeatRuns).set({ resultJson: { ...resultJson, instructionToolAttempts: attempts } })
|
||||
.where(eq(heartbeatRuns.id, this.binding.runId));
|
||||
});
|
||||
} catch (persistenceError) {
|
||||
throw new SemanticToolOutcomeUnknownError(
|
||||
`paperclip_runner_instruction_failure_receipt_unavailable call_id=${callId}`,
|
||||
{ cause: new AggregateError([error, persistenceError], "Definite instruction failure could not be saved") },
|
||||
);
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
throw new SemanticToolOutcomeUnknownError(
|
||||
`paperclip_runner_instruction_outcome_unknown call_id=${callId}: ${error instanceof Error ? error.message : String(error)}`,
|
||||
{ cause: error },
|
||||
);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
async #withMutationReceipt(
|
||||
operationId: string,
|
||||
idempotencyKey: string,
|
||||
|
||||
@@ -85,6 +85,96 @@ describe("createDurableRunLogStore", () => {
|
||||
expect(calls.get).toBe(0); // local file present -> no S3 read
|
||||
});
|
||||
|
||||
it.each([false, true])("retains every attempt across retry (pod restart: %s)", async (restart) => {
|
||||
const { provider, objects } = createMemoryProvider();
|
||||
let store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });
|
||||
const attempts: string[] = [];
|
||||
for (let attempt = 1; attempt <= 3; attempt++) {
|
||||
if (restart && attempt > 1) {
|
||||
await fs.rm(baseDir, { recursive: true, force: true });
|
||||
store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });
|
||||
}
|
||||
const handle = await store.begin(begin);
|
||||
attempts.push(handle.attemptId!);
|
||||
await store.append(handle, { stream: "stderr", chunk: `attempt-${attempt}`, ts: "same-time", seq: attempt });
|
||||
const summary = await store.finalize(handle);
|
||||
const saved = objects.get(handle.logRef)!;
|
||||
const lines = saved.toString().trim().split("\n").map((line) => JSON.parse(line));
|
||||
expect(lines.map((line) => line.chunk)).toEqual(Array.from({length: attempt}, (_, i) => `attempt-${i + 1}`));
|
||||
expect(lines.map((line) => line.attemptId)).toEqual(attempts);
|
||||
expect(new Set(attempts).size).toBe(attempt);
|
||||
expect(summary.bytes).toBe(saved.length);
|
||||
expect(summary.sha256).toBe(createHash("sha256").update(saved).digest("hex"));
|
||||
}
|
||||
});
|
||||
|
||||
it("never replaces concurrent appends when two stores restore the same missing log", async () => {
|
||||
const { provider, objects } = createMemoryProvider();
|
||||
const firstStore = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });
|
||||
const secondStore = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });
|
||||
const original = await firstStore.begin(begin);
|
||||
await firstStore.append(original, { stream: "stderr", chunk: "original", ts: "t0" });
|
||||
await firstStore.finalize(original);
|
||||
const prefix = Buffer.from(objects.get(original.logRef)!);
|
||||
await fs.rm(baseDir, { recursive: true, force: true });
|
||||
let reached!: () => void, release!: () => void;
|
||||
const restoring = new Promise<void>((resolve) => { reached = resolve; });
|
||||
const barrier = new Promise<void>((resolve) => { release = resolve; });
|
||||
const spy = vi.spyOn(provider, "getObject").mockImplementationOnce(async () => ({
|
||||
stream: Readable.from((async function* () {
|
||||
yield prefix;
|
||||
reached();
|
||||
await barrier;
|
||||
})()),
|
||||
contentLength: prefix.length,
|
||||
}));
|
||||
const slow = firstStore.begin(begin);
|
||||
try {
|
||||
await restoring;
|
||||
const fast = await secondStore.begin(begin);
|
||||
await secondStore.append(fast, { stream: "stderr", chunk: "fast attempt", ts: "t1" });
|
||||
release();
|
||||
const late = await slow;
|
||||
await firstStore.append(late, { stream: "stderr", chunk: "slow attempt", ts: "t2" });
|
||||
await firstStore.finalize(late);
|
||||
const saved = objects.get(original.logRef)!;
|
||||
expect(saved).toEqual(await fs.readFile(path.join(baseDir, original.logRef)));
|
||||
expect(saved.toString().trim().split("\n").map((line) => JSON.parse(line).chunk))
|
||||
.toEqual(["original", "fast attempt", "slow attempt"]);
|
||||
} finally {
|
||||
release();
|
||||
await slow.catch(() => undefined);
|
||||
spy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["stream_failure", "short_read"])("preserves the durable prefix when restore fails (%s)", async (fault) => {
|
||||
const { provider, objects, calls } = createMemoryProvider();
|
||||
const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });
|
||||
const handle = await store.begin(begin);
|
||||
await store.append(handle, { stream: "stderr", chunk: "original failure evidence", ts: "t1" });
|
||||
await store.finalize(handle);
|
||||
const prefix = Buffer.from(objects.get(handle.logRef)!);
|
||||
await fs.rm(baseDir, { recursive: true, force: true });
|
||||
const spy = vi.spyOn(provider, "getObject").mockResolvedValueOnce({
|
||||
stream: Readable.from((async function* () {
|
||||
yield prefix.subarray(0, 10);
|
||||
if (fault === "stream_failure") throw new Error("storage connection lost");
|
||||
})()),
|
||||
contentLength: prefix.length,
|
||||
});
|
||||
try {
|
||||
await expect(store.begin(begin)).rejects.toThrow(
|
||||
fault === "stream_failure" ? "storage connection lost" : "incomplete prefix",
|
||||
);
|
||||
await expect(fs.stat(path.join(baseDir, handle.logRef))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect(objects.get(handle.logRef)).toEqual(prefix);
|
||||
expect(calls.put).toBe(1);
|
||||
const next = await store.begin(begin);
|
||||
expect(await fs.readFile(path.join(baseDir, next.logRef))).toEqual(prefix);
|
||||
} finally { spy.mockRestore(); }
|
||||
});
|
||||
|
||||
it("uploads the complete log to S3 on finalize", async () => {
|
||||
const { provider, objects, calls } = createMemoryProvider();
|
||||
const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider, keyPrefix: "run-logs" } });
|
||||
@@ -133,7 +223,7 @@ describe("createDurableRunLogStore", () => {
|
||||
expect(summary.sha256).toBe(createHash("sha256").update(local).digest("hex"));
|
||||
expect(objects.get(handle.logRef)).toEqual(local);
|
||||
expect(local.toString()).toBe(outcome === "fails" ? "" : JSON.stringify({
|
||||
ts: "t1", stream: "stderr", chunk: "late diagnostic",
|
||||
ts: "t1", attemptId: handle.attemptId, stream: "stderr", chunk: "late diagnostic",
|
||||
}) + "\n");
|
||||
} finally {
|
||||
release();
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { createReadStream, promises as fs } from "node:fs";
|
||||
import { createReadStream, createWriteStream, promises as fs } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { createHash } from "node:crypto";
|
||||
import { pipeline } from "node:stream/promises";
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { addAbortSignal } from "node:stream";
|
||||
import { notFound } from "../errors.js";
|
||||
import { resolvePaperclipInstanceRoot } from "../home-paths.js";
|
||||
@@ -12,6 +13,7 @@ export type RunLogStoreType = "local_file";
|
||||
export interface RunLogHandle {
|
||||
store: RunLogStoreType;
|
||||
logRef: string;
|
||||
attemptId?: string;
|
||||
}
|
||||
|
||||
export interface RunLogReadOptions {
|
||||
@@ -295,9 +297,37 @@ export function createDurableRunLogStore(options: DurableRunLogStoreOptions): Ru
|
||||
const relPath = path.join(relDir, `${runId}.ndjson`);
|
||||
await ensureDir(relDir);
|
||||
const absPath = resolveWithin(basePath, relPath);
|
||||
await fs.writeFile(absPath, "", "utf8");
|
||||
await retireInflightMirror(relPath);
|
||||
return { store: "local_file", logRef: relPath };
|
||||
// Retries share a run ID. Never truncate their earlier evidence. On a
|
||||
// new pod, restore the durable prefix before opening another attempt.
|
||||
const exists = await fs.stat(absPath).then(() => true, (error) => {
|
||||
if (error.code === "ENOENT") return false;
|
||||
throw error;
|
||||
});
|
||||
if (!exists && s3) {
|
||||
const key = s3Key(relPath);
|
||||
const head = await s3.provider.headObject({ objectKey: key });
|
||||
if (head.exists) {
|
||||
const temporary = `${absPath}.${randomUUID()}.restore`;
|
||||
try {
|
||||
const object = await s3.provider.getObject({ objectKey: key });
|
||||
await pipeline(object.stream, createWriteStream(temporary, { flags: "wx", mode: 0o600 }));
|
||||
if (typeof head.contentLength === "number" && (await fs.stat(temporary)).size !== head.contentLength) {
|
||||
throw new Error("Durable run log restore returned an incomplete prefix");
|
||||
}
|
||||
// Publish the complete prefix without replacing a file another
|
||||
// attempt has already restored and started appending to.
|
||||
await fs.link(temporary, absPath).catch((error: NodeJS.ErrnoException) => {
|
||||
if (error.code !== "EEXIST") throw error;
|
||||
});
|
||||
} finally {
|
||||
await fs.rm(temporary, { force: true });
|
||||
}
|
||||
}
|
||||
}
|
||||
const file = await fs.open(absPath, "a", 0o600);
|
||||
await file.close();
|
||||
return { store: "local_file", logRef: relPath, attemptId: randomUUID() };
|
||||
},
|
||||
|
||||
async append(handle, event) {
|
||||
@@ -305,6 +335,7 @@ export function createDurableRunLogStore(options: DurableRunLogStoreOptions): Ru
|
||||
const absPath = resolveWithin(basePath, handle.logRef);
|
||||
const line = JSON.stringify({
|
||||
ts: event.ts,
|
||||
...(handle.attemptId ? { attemptId: handle.attemptId } : {}),
|
||||
stream: event.stream,
|
||||
chunk: event.chunk,
|
||||
// Monotonic per-run sequence so readers can dedupe and order records
|
||||
|
||||
+1
@@ -78,6 +78,7 @@ export const NativeSessionCleanupQuarantinedError =
|
||||
runner.NativeSessionCleanupQuarantinedError;
|
||||
export const NativeSessionProtocolIntegrityError =
|
||||
runner.NativeSessionProtocolIntegrityError;
|
||||
export const SemanticToolOutcomeUnknownError = runner.SemanticToolOutcomeUnknownError;
|
||||
export const PaperclipSemanticDispatcher = runner.PaperclipSemanticDispatcher;
|
||||
export const CAPABILITY_SEMANTIC_TOOL_CATALOG =
|
||||
runner.CAPABILITY_SEMANTIC_TOOL_CATALOG;
|
||||
|
||||
Reference in new issue
Block a user