fix(runner): fail fast when an artifact storage read stalls (#22210)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use the `download_artifact` tool to copy a saved API artifact
into their workspace
> - The server reads the artifact bytes from storage before it installs
the file
> - This storage read has no deadline. A stalled storage response or
stream keeps the tool call open
> - The provider MCP client stops the call after about five minutes with
the error "The operation timed out."
> - This pull request puts a 60-second deadline on the storage read and
returns a clear, retryable error
> - The benefit is that the agent gets a fast, clear result and can
retry, and does not wait for five minutes

## Linked Issues or Issue Description

**What happened?**
Some `download_artifact` calls stopped for four to five minutes. Then
the agent got only "The operation timed out." In one case, a retry of
the same 1 MB artifact completed in about 20 seconds.

**Expected behavior**
A stalled storage read must fail quickly. The error must tell the agent
that no file was written and that it can retry the same download.

**Steps to reproduce**
1. Give `PaperclipRunnerToolAuthority` a storage service whose
`getObject` never resolves, or whose stream sends one chunk and then
stops.
2. Call `download_artifact` for an artifact in that storage.
3. The call does not settle. Nothing aborts the storage request.

**Root cause**
`#readApiFile` called `storage.getObject(...)` without a `signal` and
read the stream with no deadline. Other storage reads in the runner, for
example chat attachment reads, already have a deadline.

## What Changed

-
`server/src/services/native-runtime/paperclip-runner-tool-authority.ts`:
the artifact branch of `#readApiFile` now uses a 60-second
`AbortSignal.timeout`. It sends the signal to `storage.getObject`. It
destroys the stream when the deadline expires. Every storage stream gets
an error listener before it can be destroyed, so a late reply cannot
raise an uncaught `AbortError`. It returns HTTP 504 with code
`artifact_storage_timeout` and the message "Artifact storage read did
not finish within 60 seconds. No file was written; retry the same
download."
- A new optional binding field, `storageReadTimeoutMs`, sets the
deadline for tests. The default is 60 seconds. The production callers do
not change.
-
`server/src/services/native-runtime/artifact-transfer.integration.test.ts`:
a new regression test covers a storage response that never arrives and a
stream that stops after one chunk.

## Verification

- Failing before the first fix: `cd server && npx vitest run
src/services/native-runtime/artifact-transfer.integration.test.ts -t
"stalls"` failed with `Error: Test timed out in 15000ms.` The
`download_artifact` call never settled.
- Failing before the late-reply fix: the same test now also covers a
storage reply that arrives after the deadline, with a real
`createReadStream` that gets the aborted signal. It failed with an
uncaught `AbortError` (`ABORT_ERR`).
- Passing after both fixes: the same command passes. Each stalled call
fails in less than 5 seconds with the retryable error. All storage
signals are aborted. No uncaught error occurs. The workspace has no
files.
- `npx vitest run
src/services/native-runtime/artifact-transfer.integration.test.ts
src/services/native-runtime/paperclip-runner-tool-authority.test.ts`: 49
passed.
- `npx vitest run
src/services/native-runtime/runner-api-artifacts.integration.test.ts
src/services/native-runtime/runner-api-artifacts.test.ts
src/services/native-runtime/issue-document-batch.integration.test.ts`:
38 passed (first revision).
- `pnpm run typecheck` (server): passed.
- `pnpm verify` selects a very large set of files for this module. I did
not run the full set locally. CI covers it.

## Risks

- Low risk. The deadline applies only to the artifact read from storage,
and artifacts are at most 10 MiB. A healthy read takes seconds.
- A very slow but working storage provider that needs more than 60
seconds for 10 MiB now gets a retryable error.
- The error occurs before any file write, so no partial file stays in
the workspace.
- This change does not fix a slow storage provider. It makes the failure
fast and clear.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in a sandboxed workspace.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-18 15:14:46 -06:00
1 parent ee68098584
commit 8a295f944d
2 files changed
+108 -11

No files matched your search

@@ -1,7 +1,8 @@
import { randomUUID, createHash } from "node:crypto";
import { mkdtemp, readFile, readdir, rm } from "node:fs/promises";
import { createReadStream } from "node:fs";
import { mkdtemp, readFile, readdir, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { dirname, join } from "node:path";
import { Readable } from "node:stream";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { eq } from "drizzle-orm";
@@ -105,4 +106,71 @@ describe.skipIf(process.platform !== "linux")("native authenticated artifact ins
} finally { await rm(root, { recursive: true, force: true }); }
});
it("fails fast with a retryable error when the artifact storage read stalls", async () => {
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(), artifactId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Stalled transfer", issuePrefix: `S${companyId.slice(0, 6)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Transfer", role: "ceo", status: "active", adapterType: "paperclip_runner" });
await db.insert(issues).values({ id: issueId, companyId, title: "Transfer stalled bytes", status: "in_progress", assigneeAgentId: agentId });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native", nativeIssueId: issueId, invocationSource: "assignment", triggerDetail: "system", contextSnapshot: { issueId } });
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
const bytes = Buffer.from("synthetic stalled transfer\n");
const sha256 = createHash("sha256").update(bytes).digest("hex");
await db.insert(assets).values({ id: artifactId, companyId, provider: "local_disk", objectKey: `${companyId}/synthetic/${artifactId}`, contentType: "application/octet-stream", byteSize: bytes.length, sha256, createdByAgentId: agentId });
const signals: AbortSignal[] = [];
// One provider never answers; the other sends a first chunk and then stops.
const stalledResponse = vi.fn(async (_companyId: string, _key: string, options?: { signal?: AbortSignal }) => {
if (options?.signal) signals.push(options.signal);
return new Promise<never>(() => undefined);
});
const stalledStream = vi.fn(async (_companyId: string, _key: string, options?: { signal?: AbortSignal }) => {
if (options?.signal) signals.push(options.signal);
const stream = new Readable({ read() {} });
stream.push(bytes.subarray(0, 4));
return { stream };
});
const root = await mkdtemp(join(tmpdir(), "native-artifact-stall-"));
try {
for (const getObject of [stalledResponse, stalledStream]) {
const authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId, workspaceRoot: root, apiToolsEnabled: true,
storage: { getObject } as unknown as StorageService, storageReadTimeoutMs: 50 });
const started = Date.now();
await expect(authority.execute({ tool: "download_artifact", callId: `stall-${getObject === stalledResponse ? "response" : "stream"}`,
arguments: { artifactId, destination: "stalled.bin", byteSize: bytes.length, sha256 } })).rejects.toThrow(/did not finish.*retry/i);
expect(Date.now() - started).toBeLessThan(5_000);
expect(getObject).toHaveBeenCalledOnce();
}
expect(signals).toHaveLength(2);
expect(signals.every(signal => signal.aborted)).toBe(true);
expect(await readdir(root)).toEqual([]);
// Local disk opens its stream with the request signal. A reply that
// arrives after the deadline gets an aborted stream, which emits AbortError.
const source = join(await mkdtemp(join(tmpdir(), "native-artifact-late-source-")), "late.bin");
await writeFile(source, bytes);
const lateReply = vi.fn(async (_companyId: string, _key: string, options?: { signal?: AbortSignal }) => {
await new Promise(resolve => options!.signal!.addEventListener("abort", resolve, { once: true }));
await new Promise(resolve => setTimeout(resolve, 10));
return { stream: createReadStream(source, { signal: options!.signal }) };
});
const uncaught: unknown[] = [];
const onUncaught = (error: unknown) => { uncaught.push(error); };
process.on("uncaughtException", onUncaught);
try {
const authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId, workspaceRoot: root, apiToolsEnabled: true,
storage: { getObject: lateReply } as unknown as StorageService, storageReadTimeoutMs: 50 });
await expect(authority.execute({ tool: "download_artifact", callId: "stall-late-reply",
arguments: { artifactId, destination: "stalled.bin", byteSize: bytes.length, sha256 } })).rejects.toThrow(/did not finish.*retry/i);
expect(lateReply).toHaveBeenCalledOnce();
await vi.waitFor(() => expect(lateReply.mock.results[0]?.type).toBe("return"));
await lateReply.mock.results[0]!.value;
await new Promise(resolve => setTimeout(resolve, 50));
} finally {
process.off("uncaughtException", onUncaught);
await rm(dirname(source), { recursive: true, force: true });
}
expect(uncaught).toEqual([]);
expect(await readdir(root)).toEqual([]);
} finally { await rm(root, { recursive: true, force: true }); }
}, 15_000);
});
@@ -145,6 +145,9 @@ import { commentsArrivedDuringRun, recordTaskHistoryRead, taskHistoryHandlingGui
import { recordChildTaskRead } from "./native-child-completion-delivery.js";
import { logger } from "../../middleware/logger.js";
// Artifacts are at most RUNNER_API_MAX_BYTES; a healthy read takes seconds.
// Stay well inside the provider MCP client's five-minute request cutoff.
const ARTIFACT_STORAGE_READ_TIMEOUT_MS = 60_000;
const EXTERNAL_WORKSPACE_OPERATIONS = new Set(["get_workspace", "list_workspaces", "select_workspace", "prepare_repository", "register_deliverable"]);
const IMPLEMENTED_OPERATIONS = new Set([
"send_chat_message",
@@ -227,6 +230,8 @@ type Binding = {
/** Server-owned API origin and storage; never obtained from tool input. */
apiUrl?: string;
storage?: StorageService;
/** Deadline for one artifact read from storage; defaults to ARTIFACT_STORAGE_READ_TIMEOUT_MS. */
storageReadTimeoutMs?: number;
/** Server-owned suppression for baseline evals; true never overrides operator opt-in. */
connectorAssignments?: Pick<ConnectorAssignment, "tools">[];
assignedMcpTools?: Awaited<ReturnType<typeof createAssignedMcpTools>>;
@@ -1726,17 +1731,41 @@ export class PaperclipRunnerToolAuthority {
};
await assertArtifactAccess();
if (asset.byteSize > RUNNER_API_MAX_BYTES) throw badRequest("Artifact exceeds API transfer limit");
const object = await (this.binding.storage ?? getStorageService()).getObject(this.binding.companyId, asset.objectKey);
// A stalled storage read must fail well before the provider's MCP
// client gives up, so the call settles with a retryable error instead
// of an anonymous client timeout. Nothing is written before this read.
const timeoutMs = this.binding.storageReadTimeoutMs ?? ARTIFACT_STORAGE_READ_TIMEOUT_MS;
const signal = AbortSignal.timeout(timeoutMs);
let object: Awaited<ReturnType<StorageService["getObject"]>> | undefined;
let onAbort = () => {};
const aborted = new Promise<never>((_, reject) => {
onAbort = () => {
object?.stream.destroy();
reject(new HttpError(504, `Artifact storage read did not finish within ${Math.ceil(timeoutMs / 1000)} seconds. No file was written; retry the same download.`, { code: "artifact_storage_timeout" }));
};
signal.addEventListener("abort", onAbort, { once: true });
});
const chunks: Buffer[] = [];
let length = 0;
try {
for await (const chunk of object.stream) {
const bytes = Buffer.from(chunk);
length += bytes.length;
if (length > RUNNER_API_MAX_BYTES) throw badRequest("Artifact exceeds API transfer limit");
chunks.push(bytes);
}
} finally { object.stream.destroy(); }
await Promise.race([aborted, (async () => {
const opened = await (this.binding.storage ?? getStorageService()).getObject(this.binding.companyId, asset.objectKey, { signal });
// A reply after the deadline can carry a stream opened with the
// aborted signal. Its AbortError must never become uncaught.
opened.stream.on("error", () => undefined);
if (signal.aborted) { opened.stream.destroy(); return; }
object = opened;
let length = 0;
for await (const chunk of opened.stream) {
const bytes = Buffer.from(chunk);
length += bytes.length;
if (length > RUNNER_API_MAX_BYTES) throw badRequest("Artifact exceeds API transfer limit");
chunks.push(bytes);
}
})()]);
} finally {
signal.removeEventListener("abort", onAbort);
object?.stream.destroy();
}
await assertArtifactAccess();
return { bytes: Buffer.concat(chunks), filename: asset.originalFilename ?? "file", contentType: asset.contentType };
}