mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-19 07:42:38 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents use the `download_artifact` tool to copy a saved API artifact into their workspace > - The server reads the artifact bytes from storage before it installs the file > - This storage read has no deadline. A stalled storage response or stream keeps the tool call open > - The provider MCP client stops the call after about five minutes with the error "The operation timed out." > - This pull request puts a 60-second deadline on the storage read and returns a clear, retryable error > - The benefit is that the agent gets a fast, clear result and can retry, and does not wait for five minutes ## Linked Issues or Issue Description **What happened?** Some `download_artifact` calls stopped for four to five minutes. Then the agent got only "The operation timed out." In one case, a retry of the same 1 MB artifact completed in about 20 seconds. **Expected behavior** A stalled storage read must fail quickly. The error must tell the agent that no file was written and that it can retry the same download. **Steps to reproduce** 1. Give `PaperclipRunnerToolAuthority` a storage service whose `getObject` never resolves, or whose stream sends one chunk and then stops. 2. Call `download_artifact` for an artifact in that storage. 3. The call does not settle. Nothing aborts the storage request. **Root cause** `#readApiFile` called `storage.getObject(...)` without a `signal` and read the stream with no deadline. Other storage reads in the runner, for example chat attachment reads, already have a deadline. ## What Changed - `server/src/services/native-runtime/paperclip-runner-tool-authority.ts`: the artifact branch of `#readApiFile` now uses a 60-second `AbortSignal.timeout`. It sends the signal to `storage.getObject`. It destroys the stream when the deadline expires. Every storage stream gets an error listener before it can be destroyed, so a late reply cannot raise an uncaught `AbortError`. It returns HTTP 504 with code `artifact_storage_timeout` and the message "Artifact storage read did not finish within 60 seconds. No file was written; retry the same download." - A new optional binding field, `storageReadTimeoutMs`, sets the deadline for tests. The default is 60 seconds. The production callers do not change. - `server/src/services/native-runtime/artifact-transfer.integration.test.ts`: a new regression test covers a storage response that never arrives and a stream that stops after one chunk. ## Verification - Failing before the first fix: `cd server && npx vitest run src/services/native-runtime/artifact-transfer.integration.test.ts -t "stalls"` failed with `Error: Test timed out in 15000ms.` The `download_artifact` call never settled. - Failing before the late-reply fix: the same test now also covers a storage reply that arrives after the deadline, with a real `createReadStream` that gets the aborted signal. It failed with an uncaught `AbortError` (`ABORT_ERR`). - Passing after both fixes: the same command passes. Each stalled call fails in less than 5 seconds with the retryable error. All storage signals are aborted. No uncaught error occurs. The workspace has no files. - `npx vitest run src/services/native-runtime/artifact-transfer.integration.test.ts src/services/native-runtime/paperclip-runner-tool-authority.test.ts`: 49 passed. - `npx vitest run src/services/native-runtime/runner-api-artifacts.integration.test.ts src/services/native-runtime/runner-api-artifacts.test.ts src/services/native-runtime/issue-document-batch.integration.test.ts`: 38 passed (first revision). - `pnpm run typecheck` (server): passed. - `pnpm verify` selects a very large set of files for this module. I did not run the full set locally. CI covers it. ## Risks - Low risk. The deadline applies only to the artifact read from storage, and artifacts are at most 10 MiB. A healthy read takes seconds. - A very slow but working storage provider that needs more than 60 seconds for 10 MiB now gets a retryable error. - The error occurs before any file write, so no partial file stays in the workspace. - This change does not fix a slow storage provider. It makes the failure fast and clear. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in a sandboxed workspace. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
177 lines
14 KiB
TypeScript
177 lines
14 KiB
TypeScript
import { randomUUID, createHash } from "node:crypto";
|
|
import { createReadStream } from "node:fs";
|
|
import { mkdtemp, readFile, readdir, rm, writeFile } from "node:fs/promises";
|
|
import { tmpdir } from "node:os";
|
|
import { dirname, join } from "node:path";
|
|
import { Readable } from "node:stream";
|
|
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import { eq } from "drizzle-orm";
|
|
import { agents, assets, companies, createDb, heartbeatRuns, issues, runnerApiResponseReservations } from "@paperclipai/db";
|
|
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
|
import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js";
|
|
import { SemanticToolOutcomeUnknownError } from "../../vendor/paperclip-runner/index.js";
|
|
import { createArtifactInstaller } from "./artifact-transfer.js";
|
|
import type { StorageService } from "../../storage/types.js";
|
|
|
|
describe.skipIf(process.platform !== "linux")("native authenticated artifact installation", () => {
|
|
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
|
let db: ReturnType<typeof createDb>;
|
|
beforeAll(async () => {
|
|
temporary = await startEmbeddedPostgresTestDatabase("artifact-transfer-");
|
|
db = createDb(temporary.connectionString);
|
|
});
|
|
afterAll(async () => { await temporary?.cleanup(); });
|
|
|
|
it("materializes all 388131 synthetic bytes without model transcription and verifies a retry", async () => {
|
|
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(), artifactId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Synthetic transfer", issuePrefix: `A${companyId.slice(0, 6)}` });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Transfer", role: "ceo", status: "active", adapterType: "paperclip_runner" });
|
|
await db.insert(issues).values({ id: issueId, companyId, title: "Transfer synthetic bytes", status: "in_progress", assigneeAgentId: agentId });
|
|
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native", nativeIssueId: issueId, invocationSource: "assignment", triggerDetail: "system", contextSnapshot: { issueId } });
|
|
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
|
|
const unit = Buffer.from('🧭é"\\\n\u0000 synthetic transfer\n');
|
|
const bytes = Buffer.concat([Buffer.concat(Array.from({ length: Math.floor(388131 / unit.length) }, () => unit)), Buffer.alloc(388131 % unit.length, 65)]);
|
|
const sha256 = createHash("sha256").update(bytes).digest("hex");
|
|
const objectKey = `${companyId}/synthetic/${artifactId}`;
|
|
await db.insert(assets).values({ id: artifactId, companyId, provider: "local_disk", objectKey, contentType: "application/octet-stream", byteSize: bytes.length, sha256, createdByAgentId: agentId });
|
|
let storedBytes = bytes;
|
|
let beforeRead: (() => Promise<void>) | undefined;
|
|
const getObject = vi.fn(async () => { await beforeRead?.(); return { stream: Readable.from([storedBytes.subarray(0, 24576), storedBytes.subarray(24576)]) }; });
|
|
const storage = { getObject } as unknown as StorageService;
|
|
const root = await mkdtemp(join(tmpdir(), "native-artifact-transfer-"));
|
|
try {
|
|
const authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId, workspaceRoot: root, storage, apiToolsEnabled: true });
|
|
expect(authority.definitions().some(tool => tool.name === "download_artifact")).toBe(true);
|
|
const call = { tool: "download_artifact", callId: "synthetic-install", arguments: { artifactId, destination: "synthetic.bin", byteSize: bytes.length, sha256 } };
|
|
const result = await authority.execute(call);
|
|
expect(result).toMatchObject({ outcome: "completed", byteSize: 388131, sha256, destination: "synthetic.bin" });
|
|
const restored = await readFile(join(root, "synthetic.bin"));
|
|
expect(restored.length).toBe(388131);
|
|
expect(createHash("sha256").update(restored).digest("hex")).toBe(sha256);
|
|
expect(restored.equals(bytes)).toBe(true);
|
|
expect(await authority.execute({ ...call, callId: "retry-install" })).toMatchObject({ outcome: "completed", sha256 });
|
|
expect(await authority.recover(call)).toMatchObject({ result: { outcome: "completed", sha256 } });
|
|
expect(await authority.recover({ ...call, arguments: { ...call.arguments, destination: "missing.bin" } })).toMatchObject({ isError: true, result: { error: "artifact_install_missing" } });
|
|
expect(await readdir(root)).toEqual(["synthetic.bin"]);
|
|
let loseReceipt = true;
|
|
const realInstaller = createArtifactInstaller(root);
|
|
const remoteInstaller = vi.fn<ReturnType<typeof createArtifactInstaller>>(async input => {
|
|
await realInstaller(input);
|
|
if (loseReceipt && !input.verifyOnly) { loseReceipt = false; throw new SemanticToolOutcomeUnknownError("lost commit receipt"); }
|
|
});
|
|
const remoteAuthority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId, workspaceRoot: root, storage,
|
|
executionTargetKind: "remote", installRemoteArtifact: remoteInstaller });
|
|
const remoteCall = { ...call, arguments: { ...call.arguments, destination: "remote-synthetic.bin" } };
|
|
await expect(remoteAuthority.execute(remoteCall)).rejects.toThrow("lost commit receipt");
|
|
expect(await remoteAuthority.recover(remoteCall)).toMatchObject({ result: { outcome: "completed", sha256 } });
|
|
expect(remoteInstaller).toHaveBeenCalledTimes(2);
|
|
expect(remoteInstaller.mock.calls[1][0].verifyOnly).toBe(true);
|
|
expect((await readFile(join(root, "remote-synthetic.bin"))).equals(bytes)).toBe(true);
|
|
await rm(join(root, "remote-synthetic.bin"));
|
|
// Inputs with a different identity or without an available asset fail before storage.
|
|
for (const arguments_ of [{ ...call.arguments, artifactId: randomUUID() }, { ...call.arguments, sha256: "0".repeat(64) }, { ...call.arguments, byteSize: 388130 }]) {
|
|
getObject.mockClear();
|
|
await expect(authority.execute({ ...call, arguments: arguments_ })).rejects.toThrow();
|
|
expect(getObject).not.toHaveBeenCalled();
|
|
}
|
|
const foreignCompany = randomUUID(), foreignAsset = randomUUID();
|
|
await db.insert(companies).values({ id: foreignCompany, name: "Foreign synthetic", issuePrefix: `F${foreignCompany.slice(0, 6)}` });
|
|
await db.insert(assets).values({ id: foreignAsset, companyId: foreignCompany, provider: "local_disk", objectKey: `${foreignCompany}/synthetic/file`, contentType: "application/octet-stream", byteSize: bytes.length, sha256 });
|
|
await expect(authority.execute({ ...call, arguments: { ...call.arguments, artifactId: foreignAsset } })).rejects.toThrow("company");
|
|
expect(getObject).not.toHaveBeenCalled();
|
|
|
|
// The content-read ACL retains capture ownership, independently of company membership.
|
|
const foreignRun = randomUUID(), foreignAgent = randomUUID(), privateAsset = randomUUID();
|
|
await db.insert(agents).values({ id: foreignAgent, companyId, name: "Other", status: "active", adapterType: "paperclip_runner" });
|
|
await db.insert(heartbeatRuns).values({ id: foreignRun, companyId, agentId: foreignAgent, status: "succeeded", invocationSource: "assignment", triggerDetail: "system" });
|
|
await db.insert(assets).values({ id: privateAsset, companyId, provider: "local_disk", objectKey: `${companyId}/assets/runner-api/private`, contentType: "application/json", byteSize: bytes.length, sha256 });
|
|
await db.insert(runnerApiResponseReservations).values({ companyId, runId: foreignRun, assetId: privateAsset, reservedBytes: bytes.length });
|
|
await expect(authority.execute({ ...call, arguments: { ...call.arguments, artifactId: privateAsset } })).rejects.toThrow("actor");
|
|
expect(getObject).not.toHaveBeenCalled();
|
|
await db.update(heartbeatRuns).set({ agentId, responsibleUserId: "other-synthetic-user" }).where(eq(heartbeatRuns.id, foreignRun));
|
|
await expect(authority.execute({ ...call, arguments: { ...call.arguments, artifactId: privateAsset } })).rejects.toThrow("actor");
|
|
expect(getObject).not.toHaveBeenCalled();
|
|
|
|
for (const binding of [{ workMode: "planning" as const }, { executionTargetKind: "remote" as const }, { firstPartyOnly: true }, { dotRuntime: true }, { museRuntime: true }, { apiToolsEnabled: false }]) {
|
|
const restricted = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId, workspaceRoot: root, storage, ...binding });
|
|
expect(restricted.definitions().some(tool => tool.name === "download_artifact")).toBe(false);
|
|
await expect(restricted.execute(call)).rejects.toThrow();
|
|
}
|
|
storedBytes = bytes.subarray(0, bytes.length - 1);
|
|
await expect(authority.execute({ ...call, arguments: { ...call.arguments, destination: "truncated.bin" } })).rejects.toThrow("identity");
|
|
storedBytes = bytes;
|
|
beforeRead = async () => { await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, runId)); };
|
|
await expect(authority.execute({ ...call, arguments: { ...call.arguments, destination: "revoked.bin" } })).rejects.toThrow("not_authorized");
|
|
expect(await readdir(root)).toEqual(["synthetic.bin"]);
|
|
|
|
} finally { await rm(root, { recursive: true, force: true }); }
|
|
});
|
|
|
|
it("fails fast with a retryable error when the artifact storage read stalls", async () => {
|
|
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(), artifactId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Stalled transfer", issuePrefix: `S${companyId.slice(0, 6)}` });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Transfer", role: "ceo", status: "active", adapterType: "paperclip_runner" });
|
|
await db.insert(issues).values({ id: issueId, companyId, title: "Transfer stalled bytes", status: "in_progress", assigneeAgentId: agentId });
|
|
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native", nativeIssueId: issueId, invocationSource: "assignment", triggerDetail: "system", contextSnapshot: { issueId } });
|
|
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
|
|
const bytes = Buffer.from("synthetic stalled transfer\n");
|
|
const sha256 = createHash("sha256").update(bytes).digest("hex");
|
|
await db.insert(assets).values({ id: artifactId, companyId, provider: "local_disk", objectKey: `${companyId}/synthetic/${artifactId}`, contentType: "application/octet-stream", byteSize: bytes.length, sha256, createdByAgentId: agentId });
|
|
const signals: AbortSignal[] = [];
|
|
// One provider never answers; the other sends a first chunk and then stops.
|
|
const stalledResponse = vi.fn(async (_companyId: string, _key: string, options?: { signal?: AbortSignal }) => {
|
|
if (options?.signal) signals.push(options.signal);
|
|
return new Promise<never>(() => undefined);
|
|
});
|
|
const stalledStream = vi.fn(async (_companyId: string, _key: string, options?: { signal?: AbortSignal }) => {
|
|
if (options?.signal) signals.push(options.signal);
|
|
const stream = new Readable({ read() {} });
|
|
stream.push(bytes.subarray(0, 4));
|
|
return { stream };
|
|
});
|
|
const root = await mkdtemp(join(tmpdir(), "native-artifact-stall-"));
|
|
try {
|
|
for (const getObject of [stalledResponse, stalledStream]) {
|
|
const authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId, workspaceRoot: root, apiToolsEnabled: true,
|
|
storage: { getObject } as unknown as StorageService, storageReadTimeoutMs: 50 });
|
|
const started = Date.now();
|
|
await expect(authority.execute({ tool: "download_artifact", callId: `stall-${getObject === stalledResponse ? "response" : "stream"}`,
|
|
arguments: { artifactId, destination: "stalled.bin", byteSize: bytes.length, sha256 } })).rejects.toThrow(/did not finish.*retry/i);
|
|
expect(Date.now() - started).toBeLessThan(5_000);
|
|
expect(getObject).toHaveBeenCalledOnce();
|
|
}
|
|
expect(signals).toHaveLength(2);
|
|
expect(signals.every(signal => signal.aborted)).toBe(true);
|
|
expect(await readdir(root)).toEqual([]);
|
|
|
|
// Local disk opens its stream with the request signal. A reply that
|
|
// arrives after the deadline gets an aborted stream, which emits AbortError.
|
|
const source = join(await mkdtemp(join(tmpdir(), "native-artifact-late-source-")), "late.bin");
|
|
await writeFile(source, bytes);
|
|
const lateReply = vi.fn(async (_companyId: string, _key: string, options?: { signal?: AbortSignal }) => {
|
|
await new Promise(resolve => options!.signal!.addEventListener("abort", resolve, { once: true }));
|
|
await new Promise(resolve => setTimeout(resolve, 10));
|
|
return { stream: createReadStream(source, { signal: options!.signal }) };
|
|
});
|
|
const uncaught: unknown[] = [];
|
|
const onUncaught = (error: unknown) => { uncaught.push(error); };
|
|
process.on("uncaughtException", onUncaught);
|
|
try {
|
|
const authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId, workspaceRoot: root, apiToolsEnabled: true,
|
|
storage: { getObject: lateReply } as unknown as StorageService, storageReadTimeoutMs: 50 });
|
|
await expect(authority.execute({ tool: "download_artifact", callId: "stall-late-reply",
|
|
arguments: { artifactId, destination: "stalled.bin", byteSize: bytes.length, sha256 } })).rejects.toThrow(/did not finish.*retry/i);
|
|
expect(lateReply).toHaveBeenCalledOnce();
|
|
await vi.waitFor(() => expect(lateReply.mock.results[0]?.type).toBe("return"));
|
|
await lateReply.mock.results[0]!.value;
|
|
await new Promise(resolve => setTimeout(resolve, 50));
|
|
} finally {
|
|
process.off("uncaughtException", onUncaught);
|
|
await rm(dirname(source), { recursive: true, force: true });
|
|
}
|
|
expect(uncaught).toEqual([]);
|
|
expect(await readdir(root)).toEqual([]);
|
|
} finally { await rm(root, { recursive: true, force: true }); }
|
|
}, 15_000);
|
|
});
|