From 8a295f944d280c5556dedba9a2fa35d2be942d21 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 18 Nov 2026 15:14:46 -0600 Subject: [PATCH] fix(runner): fail fast when an artifact storage read stalls (#22210) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents use the `download_artifact` tool to copy a saved API artifact into their workspace > - The server reads the artifact bytes from storage before it installs the file > - This storage read has no deadline. A stalled storage response or stream keeps the tool call open > - The provider MCP client stops the call after about five minutes with the error "The operation timed out." > - This pull request puts a 60-second deadline on the storage read and returns a clear, retryable error > - The benefit is that the agent gets a fast, clear result and can retry, and does not wait for five minutes ## Linked Issues or Issue Description **What happened?** Some `download_artifact` calls stopped for four to five minutes. Then the agent got only "The operation timed out." In one case, a retry of the same 1 MB artifact completed in about 20 seconds. **Expected behavior** A stalled storage read must fail quickly. The error must tell the agent that no file was written and that it can retry the same download. **Steps to reproduce** 1. Give `PaperclipRunnerToolAuthority` a storage service whose `getObject` never resolves, or whose stream sends one chunk and then stops. 2. Call `download_artifact` for an artifact in that storage. 3. The call does not settle. Nothing aborts the storage request. **Root cause** `#readApiFile` called `storage.getObject(...)` without a `signal` and read the stream with no deadline. Other storage reads in the runner, for example chat attachment reads, already have a deadline. ## What Changed - `server/src/services/native-runtime/paperclip-runner-tool-authority.ts`: the artifact branch of `#readApiFile` now uses a 60-second `AbortSignal.timeout`. It sends the signal to `storage.getObject`. It destroys the stream when the deadline expires. Every storage stream gets an error listener before it can be destroyed, so a late reply cannot raise an uncaught `AbortError`. It returns HTTP 504 with code `artifact_storage_timeout` and the message "Artifact storage read did not finish within 60 seconds. No file was written; retry the same download." - A new optional binding field, `storageReadTimeoutMs`, sets the deadline for tests. The default is 60 seconds. The production callers do not change. - `server/src/services/native-runtime/artifact-transfer.integration.test.ts`: a new regression test covers a storage response that never arrives and a stream that stops after one chunk. ## Verification - Failing before the first fix: `cd server && npx vitest run src/services/native-runtime/artifact-transfer.integration.test.ts -t "stalls"` failed with `Error: Test timed out in 15000ms.` The `download_artifact` call never settled. - Failing before the late-reply fix: the same test now also covers a storage reply that arrives after the deadline, with a real `createReadStream` that gets the aborted signal. It failed with an uncaught `AbortError` (`ABORT_ERR`). - Passing after both fixes: the same command passes. Each stalled call fails in less than 5 seconds with the retryable error. All storage signals are aborted. No uncaught error occurs. The workspace has no files. - `npx vitest run src/services/native-runtime/artifact-transfer.integration.test.ts src/services/native-runtime/paperclip-runner-tool-authority.test.ts`: 49 passed. - `npx vitest run src/services/native-runtime/runner-api-artifacts.integration.test.ts src/services/native-runtime/runner-api-artifacts.test.ts src/services/native-runtime/issue-document-batch.integration.test.ts`: 38 passed (first revision). - `pnpm run typecheck` (server): passed. - `pnpm verify` selects a very large set of files for this module. I did not run the full set locally. CI covers it. ## Risks - Low risk. The deadline applies only to the artifact read from storage, and artifacts are at most 10 MiB. A healthy read takes seconds. - A very slow but working storage provider that needs more than 60 seconds for 10 MiB now gets a retryable error. - The error occurs before any file write, so no partial file stays in the workspace. - This change does not fix a slow storage provider. It makes the failure fast and clear. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in a sandboxed workspace. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../artifact-transfer.integration.test.ts | 72 ++++++++++++++++++- .../paperclip-runner-tool-authority.ts | 47 +++++++++--- 2 files changed, 108 insertions(+), 11 deletions(-) diff --git a/server/src/services/native-runtime/artifact-transfer.integration.test.ts b/server/src/services/native-runtime/artifact-transfer.integration.test.ts index bfb275f8113..d3d5521098d 100644 --- a/server/src/services/native-runtime/artifact-transfer.integration.test.ts +++ b/server/src/services/native-runtime/artifact-transfer.integration.test.ts @@ -1,7 +1,8 @@ import { randomUUID, createHash } from "node:crypto"; -import { mkdtemp, readFile, readdir, rm } from "node:fs/promises"; +import { createReadStream } from "node:fs"; +import { mkdtemp, readFile, readdir, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; -import { join } from "node:path"; +import { dirname, join } from "node:path"; import { Readable } from "node:stream"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { eq } from "drizzle-orm"; @@ -105,4 +106,71 @@ describe.skipIf(process.platform !== "linux")("native authenticated artifact ins } finally { await rm(root, { recursive: true, force: true }); } }); + + it("fails fast with a retryable error when the artifact storage read stalls", async () => { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(), artifactId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Stalled transfer", issuePrefix: `S${companyId.slice(0, 6)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Transfer", role: "ceo", status: "active", adapterType: "paperclip_runner" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Transfer stalled bytes", status: "in_progress", assigneeAgentId: agentId }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native", nativeIssueId: issueId, invocationSource: "assignment", triggerDetail: "system", contextSnapshot: { issueId } }); + await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)); + const bytes = Buffer.from("synthetic stalled transfer\n"); + const sha256 = createHash("sha256").update(bytes).digest("hex"); + await db.insert(assets).values({ id: artifactId, companyId, provider: "local_disk", objectKey: `${companyId}/synthetic/${artifactId}`, contentType: "application/octet-stream", byteSize: bytes.length, sha256, createdByAgentId: agentId }); + const signals: AbortSignal[] = []; + // One provider never answers; the other sends a first chunk and then stops. + const stalledResponse = vi.fn(async (_companyId: string, _key: string, options?: { signal?: AbortSignal }) => { + if (options?.signal) signals.push(options.signal); + return new Promise(() => undefined); + }); + const stalledStream = vi.fn(async (_companyId: string, _key: string, options?: { signal?: AbortSignal }) => { + if (options?.signal) signals.push(options.signal); + const stream = new Readable({ read() {} }); + stream.push(bytes.subarray(0, 4)); + return { stream }; + }); + const root = await mkdtemp(join(tmpdir(), "native-artifact-stall-")); + try { + for (const getObject of [stalledResponse, stalledStream]) { + const authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId, workspaceRoot: root, apiToolsEnabled: true, + storage: { getObject } as unknown as StorageService, storageReadTimeoutMs: 50 }); + const started = Date.now(); + await expect(authority.execute({ tool: "download_artifact", callId: `stall-${getObject === stalledResponse ? "response" : "stream"}`, + arguments: { artifactId, destination: "stalled.bin", byteSize: bytes.length, sha256 } })).rejects.toThrow(/did not finish.*retry/i); + expect(Date.now() - started).toBeLessThan(5_000); + expect(getObject).toHaveBeenCalledOnce(); + } + expect(signals).toHaveLength(2); + expect(signals.every(signal => signal.aborted)).toBe(true); + expect(await readdir(root)).toEqual([]); + + // Local disk opens its stream with the request signal. A reply that + // arrives after the deadline gets an aborted stream, which emits AbortError. + const source = join(await mkdtemp(join(tmpdir(), "native-artifact-late-source-")), "late.bin"); + await writeFile(source, bytes); + const lateReply = vi.fn(async (_companyId: string, _key: string, options?: { signal?: AbortSignal }) => { + await new Promise(resolve => options!.signal!.addEventListener("abort", resolve, { once: true })); + await new Promise(resolve => setTimeout(resolve, 10)); + return { stream: createReadStream(source, { signal: options!.signal }) }; + }); + const uncaught: unknown[] = []; + const onUncaught = (error: unknown) => { uncaught.push(error); }; + process.on("uncaughtException", onUncaught); + try { + const authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId, workspaceRoot: root, apiToolsEnabled: true, + storage: { getObject: lateReply } as unknown as StorageService, storageReadTimeoutMs: 50 }); + await expect(authority.execute({ tool: "download_artifact", callId: "stall-late-reply", + arguments: { artifactId, destination: "stalled.bin", byteSize: bytes.length, sha256 } })).rejects.toThrow(/did not finish.*retry/i); + expect(lateReply).toHaveBeenCalledOnce(); + await vi.waitFor(() => expect(lateReply.mock.results[0]?.type).toBe("return")); + await lateReply.mock.results[0]!.value; + await new Promise(resolve => setTimeout(resolve, 50)); + } finally { + process.off("uncaughtException", onUncaught); + await rm(dirname(source), { recursive: true, force: true }); + } + expect(uncaught).toEqual([]); + expect(await readdir(root)).toEqual([]); + } finally { await rm(root, { recursive: true, force: true }); } + }, 15_000); }); diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts index 4df41025bf0..7469f505268 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts @@ -145,6 +145,9 @@ import { commentsArrivedDuringRun, recordTaskHistoryRead, taskHistoryHandlingGui import { recordChildTaskRead } from "./native-child-completion-delivery.js"; import { logger } from "../../middleware/logger.js"; +// Artifacts are at most RUNNER_API_MAX_BYTES; a healthy read takes seconds. +// Stay well inside the provider MCP client's five-minute request cutoff. +const ARTIFACT_STORAGE_READ_TIMEOUT_MS = 60_000; const EXTERNAL_WORKSPACE_OPERATIONS = new Set(["get_workspace", "list_workspaces", "select_workspace", "prepare_repository", "register_deliverable"]); const IMPLEMENTED_OPERATIONS = new Set([ "send_chat_message", @@ -227,6 +230,8 @@ type Binding = { /** Server-owned API origin and storage; never obtained from tool input. */ apiUrl?: string; storage?: StorageService; + /** Deadline for one artifact read from storage; defaults to ARTIFACT_STORAGE_READ_TIMEOUT_MS. */ + storageReadTimeoutMs?: number; /** Server-owned suppression for baseline evals; true never overrides operator opt-in. */ connectorAssignments?: Pick[]; assignedMcpTools?: Awaited>; @@ -1726,17 +1731,41 @@ export class PaperclipRunnerToolAuthority { }; await assertArtifactAccess(); if (asset.byteSize > RUNNER_API_MAX_BYTES) throw badRequest("Artifact exceeds API transfer limit"); - const object = await (this.binding.storage ?? getStorageService()).getObject(this.binding.companyId, asset.objectKey); + // A stalled storage read must fail well before the provider's MCP + // client gives up, so the call settles with a retryable error instead + // of an anonymous client timeout. Nothing is written before this read. + const timeoutMs = this.binding.storageReadTimeoutMs ?? ARTIFACT_STORAGE_READ_TIMEOUT_MS; + const signal = AbortSignal.timeout(timeoutMs); + let object: Awaited> | undefined; + let onAbort = () => {}; + const aborted = new Promise((_, reject) => { + onAbort = () => { + object?.stream.destroy(); + reject(new HttpError(504, `Artifact storage read did not finish within ${Math.ceil(timeoutMs / 1000)} seconds. No file was written; retry the same download.`, { code: "artifact_storage_timeout" })); + }; + signal.addEventListener("abort", onAbort, { once: true }); + }); const chunks: Buffer[] = []; - let length = 0; try { - for await (const chunk of object.stream) { - const bytes = Buffer.from(chunk); - length += bytes.length; - if (length > RUNNER_API_MAX_BYTES) throw badRequest("Artifact exceeds API transfer limit"); - chunks.push(bytes); - } - } finally { object.stream.destroy(); } + await Promise.race([aborted, (async () => { + const opened = await (this.binding.storage ?? getStorageService()).getObject(this.binding.companyId, asset.objectKey, { signal }); + // A reply after the deadline can carry a stream opened with the + // aborted signal. Its AbortError must never become uncaught. + opened.stream.on("error", () => undefined); + if (signal.aborted) { opened.stream.destroy(); return; } + object = opened; + let length = 0; + for await (const chunk of opened.stream) { + const bytes = Buffer.from(chunk); + length += bytes.length; + if (length > RUNNER_API_MAX_BYTES) throw badRequest("Artifact exceeds API transfer limit"); + chunks.push(bytes); + } + })()]); + } finally { + signal.removeEventListener("abort", onAbort); + object?.stream.destroy(); + } await assertArtifactAccess(); return { bytes: Buffer.concat(chunks), filename: asset.originalFilename ?? "file", contentType: asset.contentType }; }