fix(storage): wait out a store outage on a run's critical path instead of failing it (#20299)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A cloud instance keeps run logs and evidence in object storage, so a
run touches the store as soon as it starts
> - In storage chaos wave `w-20261109T145628-s9902` (ST1: 75–176 s
windows where every S3 call returns 503), only 1 of 8 tasks finished. A
run starting or resuming during an outage failed within a second as
`adapter_failed`, generic recovery burned its attempts seconds apart,
the task was escalated to the board, and the user saw the raw S3 error
> - This pull request treats a transient storage error as an
infrastructure wait: the run is retried on a minute-scale schedule,
rapid recovery is suppressed, the board shows it as working, and the run
error says storage was unavailable
> - The benefit is that a store outage of several minutes delays work
instead of failing and stranding it

## Linked Issues or Issue Description

Refs #19831. Storage chaos wave `w-20261109T145628-s9902` (master
`610b54f551`, `--storage s3`, ST1).

**Where the run fails.** At run start or resume, `run-output` `begin`
opens the run log: `getRunLogStore().begin` → object run log store →
object byte journal `claim` → S3 conditional PUT. The journal retries
only 3 times within about 300 ms, so a 503 escapes and fails the run as
`adapter_failed`. The async stack ends in the SDK middleware with no app
frame. `log_store` stays null on the failed run, which pins the claim.

**What followed.**
- `failRun` released the issue into generic immediate recovery. Recovery
retried 1–2 s apart inside the outage (t01: 14:58:33, :35, :59,
14:59:01), then stopped and opened a board-owned
`stranded_assigned_issue` action. These are the open recovery actions on
CHA-1, CHA-4, CHA-7 and CHA-8; a later Retry hit the outage again and
escalated the same way.
- The run error was the raw S3 text ("Injected storage outage").

**Slow store (ST2, `w-20261109T151946-s9903`).** With about 9.7 s of
added latency per call, the same claim hit the journal's 15 s
per-operation deadline (`AbortSignal.timeout(15_000)`). The run failed
after about 36 s with `AbortError: Request aborted` and burned 3 runs
the same way; 4 of 8 tasks needed a Retry.

t02/t03 (Retry parked with `process_identity_missing`) are a separate
native-recovery class and are handled in a follow-up.


**Daytona storage chaos ST1 (`w-20261109T161202-s9912`, 1/8):** with S3
returning 503 for 72 to 154 s at a time:
- Workspace restore failed at once (`remote_workspace_restore_failed:
step=snapshot_download; storage=provider_error`), three times in 40 s,
then `setup_failed`. It left an active recovery action behind.
- Runs reattached after a restart died within 2 s on the run-log claim,
then became reconciliation-required.
- Setup failed on other storage reads.
- 6 of 7 failures needed a UI Retry.

So the rule here is one class rule: a transient store error on a run's
critical path waits, it never fails the run.

## What Changed

- New `server/src/storage/storage-unavailable.ts`:
- `isStorageUnavailableError` reads the recorded storage failure
diagnostics first, then the SDK error shape. Unavailable means
429/500/502/503/504, the S3 throttling and timeout names, network codes
(`ECONNRESET`, `ECONNREFUSED`, `ETIMEDOUT`, `EPIPE`, `EAI_AGAIN`,
`ENOTFOUND`), or an `AbortError` whose cause is a `TimeoutError` (a
per-operation deadline). 403, 404 and 412 are not outages, and neither
is an abort caused by the run's own Stop.
- It defines the `storage_unavailable` code, the
`storage_unavailable_wait` retry reason, the retry policy (60 s × 10
attempts, about 10 minutes) and the user-facing message.
- `server/src/services/heartbeat/run-completion.ts`:
- `failRun` classifies such a failure as `storage_unavailable` with the
plain message.
- Unless native recovery owns the run, it schedules the storage wait
with `scheduleBoundedRetryForRun`, skips the interaction-continuation
retry, and releases the issue with `suppressImmediateRecovery`.
  - `failRunSetup` uses the same policy for this code.
- `server/src/services/heartbeat/retries.ts`: `storage_unavailable`
joins the transient infrastructure codes. It gets its own retry budget
and policy, like `runner_revision_mismatch`.
- `server/src/services/task-interruption-presentation.ts`:
`storage_unavailable_wait` is a resource wait, so the board shows the
task as working with no Retry while it waits.

- **One shared outage wait for the run's critical path**
(`server/src/storage/storage-unavailable.ts`):
- `retryThroughStorageOutage` retries a transient store failure with
backoff (2 s doubling to 30 s) until that storage operation's outage has
lasted 5 minutes, then gives the error back. The outermost wait of an
operation records its first failure, and nested and parallel waits
inside it share that start; no other request's success extends it.
Shared probes and shared reads run under their own scope, so one run's
Stop never cancels work another run waits on. A stop signal ends the
wait, and non-outage errors are never retried.
- Inside `waitOutStorageOutages`, every replayable S3 request waits
instead of failing: get, head, delete, buffered and file puts, and
multipart create, file parts and complete. A consumed upload stream is
never resent, and requests outside the scope keep failing fast.
- The scope covers run execution (`heartbeat.ts` `executeRun`: setup,
workspace restore, run-log claim and the run itself), plus
`loadWorkspaceSnapshotArchive` and `uploadWorkspaceSnapshot`, which
copy-back and recovery workers use too.
- The run-log claim retries as a whole, because the journal bounds each
operation with a 15 s deadline.
- Past the budget, the run takes the bounded `storage_unavailable` wait
below.
- The classifier follows the error's `cause` chain, so a restore failure
wrapping an outage counts. `failRunSetup` checks storage before restore
failures, so that case takes the storage wait instead of two quick
retries and an escalation (the R6 leak).
- `object-byte-journal.ts`: when its deadline expires, the journal
rethrows the recorded storage error instead of a bare `TimeoutError`.

## Verification

Red/green (each new test fails against master and passes with this PR):
- `run-completion.test.ts` "waits a minute and suppresses rapid recovery
when a run fails because storage is unavailable (storage chaos ST1)". An
S3-shaped 503 records `storage_unavailable` with the plain message (no
raw S3 text), schedules `STORAGE_UNAVAILABLE_RETRY`, releases with
`suppressImmediateRecovery: true`, and skips the continuation retry.
"keeps an ordinary adapter failure on the normal recovery path" guards
the default.
- `task-interruption-query.test.ts` "does not present an overdue
storage_unavailable_wait wait as a failed attempt": working, `canRetry:
false`.
- `storage-unavailable.test.ts`: which errors count as an outage and
which do not, including recorded diagnostics, a deadline-aborted call
(ST2), and a Stop-aborted call that must not count.

Static checks:
- `tsc --noEmit` passes for `server`.

Live verification: Phase 3 will rerun ST1 (seed 9902) with the MinIO
fault proxy after merge.

- `server/src/__tests__/storage-outage-wait.test.ts` runs against a
local S3 that answers 503 for 3 s:
- Inside the scope, a snapshot `getObject` and a run-log conditional PUT
succeed once the store is back.
  - Outside it, a request fails fast.
  - A stop ends the wait.
- Unit cases cover the budget, non-outage errors, and a
`RemoteWorkspaceRestoreError` wrapping an outage.
- `run-completion.test.ts`: a setup failure from a restore that wrapped
an outage takes the storage wait (red before).
- `object-byte-journal.test.ts`: a real `claim` whose head PUT hits its
deadline is classified as a storage outage (red before).

## Risks

- A storage outage now holds a task for up to about 10 minutes of
automatic waits instead of escalating within seconds. A longer outage
still ends in the existing board recovery.
- A run whose failure is misclassified as storage-unavailable waits
instead of using fast recovery. The classifier accepts only server-side
or network failures from the storage SDK, never 4xx answers.
- No schema change.

- A run can now wait until a storage outage has lasted 5 minutes before
it fails over to the bounded wait. It shows as working throughout, and
Stop ends the wait.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

server/src/storage/s3-failure.ts

@@ -25,12 +25,18 @@ function integerIn(value: unknown, min: number, max: number): number | null {
 }
 
 /** Record which S3 call failed and the provider's own correlation fields. */
-export function recordS3Failure(error: unknown, operation: StorageFailureOperation): void {
+function deadlineExpired(error: unknown, signal: AbortSignal | undefined) {
+  return ownValue(error, "name") === "AbortError" && signal?.aborted === true
+    && signal.reason instanceof DOMException && signal.reason.name === "TimeoutError";
+}
+
+export function recordS3Failure(error: unknown, operation: StorageFailureOperation, signal?: AbortSignal): void {
   const metadata = ownValue(error, "$metadata");
   recordStorageFailure(error, {
     provider: "s3",
     operation,
-    errorCode: matching(ownValue(error, "name"), ERROR_CODE) ?? matching(ownValue(error, "code"), ERROR_CODE) ?? "unknown",
+    errorCode: deadlineExpired(error, signal) ? "TimeoutError"
+      : matching(ownValue(error, "name"), ERROR_CODE) ?? matching(ownValue(error, "code"), ERROR_CODE) ?? "unknown",
     httpStatus: integerIn(ownValue(metadata, "httpStatusCode"), 100, 599),
     requestId: matching(ownValue(metadata, "requestId"), REQUEST_ID),
     attempts: integerIn(ownValue(metadata, "attempts"), 1, 100),
@@ -54,14 +60,15 @@ export function recordS3BodyFailure(error: unknown, response: unknown): void {
   });
 }
 
-export async function withS3FailureDiagnostics<T>(operation: StorageFailureOperation, send: () => Promise<T>): Promise<T> {
+export async function withS3FailureDiagnostics<T>(operation: StorageFailureOperation, send: () => Promise<T>,
+  signal?: AbortSignal): Promise<T> {
   try {
     const result = await send();
     recordStorageRequest(result, false);
     return result;
   } catch (error) {
     recordStorageRequest(error, true);
-    recordS3Failure(error, operation);
+    recordS3Failure(error, operation, signal);
     throw error;
   }
 }

server/src/storage/s3-multipart.ts

@@ -1,4 +1,5 @@
 import { AbortMultipartUploadCommand, CompleteMultipartUploadCommand, CreateMultipartUploadCommand, UploadPartCommand, type S3Client } from "@aws-sdk/client-s3";
+import { sendThroughStorageOutage } from "./storage-unavailable.js";
 import { recordStorageBackoff, recordStoragePart, recordStorageThrottle, withStorageRequestStats } from "./request-stats.js";
 import { createHash } from "node:crypto";
 import { once } from "node:events";
@@ -67,7 +68,7 @@ export async function putS3Multipart(client: S3Client, bucket: string, key: stri
   const partSize = Math.max(bufferedPartSize, Math.ceil(input.contentLength / 10_000));
   try {
     check();
-    uploadId = (await withS3FailureDiagnostics("create_multipart_upload", () => client.send(new CreateMultipartUploadCommand({ Bucket: bucket, Key: key, ContentType: input.contentType }), { abortSignal: controller.signal }))).UploadId;
+    uploadId = (await withS3FailureDiagnostics("create_multipart_upload", () => client.send(new CreateMultipartUploadCommand({ Bucket: bucket, Key: key, ContentType: input.contentType }), { abortSignal: controller.signal }), controller.signal)).UploadId;
     if (!uploadId) throw new Error("S3 did not return a multipart upload id");
     const parts: { ETag: string; PartNumber: number }[] = [];
     let partNumber = 0;
@@ -94,7 +95,7 @@ export async function putS3Multipart(client: S3Client, bucket: string, key: stri
         if (!body.write(chunk)) await once(body, "drain", { signal: controller.signal });
       };
       const sending = (async () => {
-        const result = await withS3FailureDiagnostics("upload_part", () => client.send(new UploadPartCommand({ Bucket: bucket, Key: key, UploadId: uploadId, PartNumber: number, Body: body, ContentLength: length }), { abortSignal: controller.signal }));
+        const result = await withS3FailureDiagnostics("upload_part", () => client.send(new UploadPartCommand({ Bucket: bucket, Key: key, UploadId: uploadId, PartNumber: number, Body: body, ContentLength: length }), { abortSignal: controller.signal }), controller.signal);
         if (remaining || !body.readableEnded || !result.ETag) throw new Error("S3 upload part is incomplete");
         parts[number - 1] = { ETag: result.ETag, PartNumber: number };
       })().catch(stop);
@@ -115,7 +116,7 @@ export async function putS3Multipart(client: S3Client, bucket: string, key: stri
     if (pending.length || !(await readNext()).done) throw new Error("S3 upload source exceeds its declared size");
     await Promise.all(active);
     check();
-    await withS3FailureDiagnostics("complete_multipart_upload", () => client.send(new CompleteMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId, MultipartUpload: { Parts: parts } }), { abortSignal: controller.signal }));
+    await withS3FailureDiagnostics("complete_multipart_upload", () => client.send(new CompleteMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId, MultipartUpload: { Parts: parts } }), { abortSignal: controller.signal }), controller.signal);
   } catch (error) {
     stop(error);
     await Promise.all(active);
@@ -218,7 +219,7 @@ export async function putS3MultipartFile(
     controller.signal.throwIfAborted();
     const partSize = Math.max(bufferedPartSize, Math.ceil(input.contentLength / 10_000));
     const count = Math.ceil(input.contentLength / partSize);
-    uploadId = (await withS3FailureDiagnostics("create_multipart_upload", () => client.send(new CreateMultipartUploadCommand({ Bucket: bucket, Key: key, ContentType: input.contentType }), { abortSignal: controller.signal }))).UploadId;
+    uploadId = (await sendThroughStorageOutage(() => withS3FailureDiagnostics("create_multipart_upload", () => client.send(new CreateMultipartUploadCommand({ Bucket: bucket, Key: key, ContentType: input.contentType }), { abortSignal: controller.signal }), controller.signal), controller.signal)).UploadId;
     if (!uploadId) throw new Error("S3 did not return a multipart upload id");
     const id = uploadId;
     const parts: { ETag: string; PartNumber: number }[] = new Array(count);
@@ -236,7 +237,7 @@ export async function putS3MultipartFile(
         try {
           const result = await withS3FailureDiagnostics("upload_part", () => client.send(new UploadPartCommand({
             Bucket: bucket, Key: key, UploadId: id, PartNumber: index + 1, Body: body, ContentLength: length, ContentMD5: contentMd5,
-          }), { abortSignal: controller.signal }));
+          }), { abortSignal: controller.signal }), controller.signal);
           if (!result.ETag) throw new Error("S3 upload part is incomplete");
           parts[index] = { ETag: result.ETag, PartNumber: index + 1 };
           recordStoragePart();
@@ -264,7 +265,7 @@ export async function putS3MultipartFile(
     const worker = async (slot: number) => {
       while (next < count && slot < concurrency) {
         const index = next++;
-        try { await sendPart(index); }
+        try { await sendThroughStorageOutage(() => sendPart(index), controller.signal); }
         catch (error) { if (!controller.signal.aborted) controller.abort(error); throw error; }
       }
     };
@@ -273,7 +274,7 @@ export async function putS3MultipartFile(
     const rejected = settled.find((entry): entry is PromiseRejectedResult => entry.status === "rejected");
     if (rejected) throw controller.signal.aborted && controller.signal.reason !== undefined ? controller.signal.reason : rejected.reason;
     controller.signal.throwIfAborted();
-    await withS3FailureDiagnostics("complete_multipart_upload", () => client.send(new CompleteMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: id, MultipartUpload: { Parts: parts } }), { abortSignal: controller.signal }));
+    await sendThroughStorageOutage(() => withS3FailureDiagnostics("complete_multipart_upload", () => client.send(new CompleteMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: id, MultipartUpload: { Parts: parts } }), { abortSignal: controller.signal }), controller.signal), controller.signal);
     uploadId = undefined;
     return { integrity: "server_verified_md5" };
   } catch (error) {

server/src/storage/s3-provider.ts

@@ -11,6 +11,7 @@ import { assertSourceFileSize, fileRangeMd5, putS3Multipart, putS3MultipartFile,
 import { createReadStream } from "node:fs";
 import { createHash } from "node:crypto";
 import { recordS3BodyFailure, withS3FailureDiagnostics } from "./s3-failure.js";
+import { sendThroughStorageOutage } from "./storage-unavailable.js";
 import { addAbortSignal, Readable } from "node:stream";
 import { retryS3RequestTimeouts } from "./s3-request-timeout-retry.js";
 import type { StorageProvider, GetObjectResult, HeadObjectResult } from "./types.js";
@@ -180,7 +181,7 @@ export function createS3StorageProvider(config: S3ProviderConfig): StorageProvid
     ])).digest("hex") : undefined,
 
     async putObjectConditional(input) {
-      const output = await client.send(new PutObjectCommand({
+      const send = () => withS3FailureDiagnostics("put_object", () => client.send(new PutObjectCommand({
         Bucket: bucket,
         Key: buildKey(prefix, input.objectKey),
         Body: input.body,
@@ -188,7 +189,8 @@ export function createS3StorageProvider(config: S3ProviderConfig): StorageProvid
         ContentLength: input.contentLength,
         IfMatch: input.ifMatch,
         IfNoneMatch: input.ifNoneMatch,
-      }), { abortSignal: input.signal });
+      }), { abortSignal: input.signal }), input.signal);
+      const output = Buffer.isBuffer(input.body) ? await sendThroughStorageOutage(send, input.signal) : await send();
       if (!output.ETag) throw new Error("Object storage did not return a publication ETag");
       return { etag: output.ETag };
     },
@@ -217,22 +219,22 @@ export function createS3StorageProvider(config: S3ProviderConfig): StorageProvid
         if (input.body instanceof Readable) input.body.destroy();
         await assertSourceFileSize(sourceFile, input.contentLength);
         const contentMd5 = await fileRangeMd5(sourceFile, 0, input.contentLength - 1, input.signal);
-        await sendFileRequestWithRetry(input.signal, async () => {
+        await sendThroughStorageOutage(() => sendFileRequestWithRetry(input.signal, async () => {
           const body = createReadStream(sourceFile);
           try {
             await withS3FailureDiagnostics("put_object", () => fileClient.send(new PutObjectCommand({
               Bucket: bucket, Key: key, Body: body, ContentType: input.contentType,
               ContentLength: input.contentLength, ContentMD5: contentMd5,
-            }), { abortSignal: input.signal }));
+            }), { abortSignal: input.signal }), input.signal);
           } finally { body.destroy(); }
-        });
+        }), input.signal);
         return { integrity: "server_verified_md5" } as const;
       }
       if (input.contentLength > 16 * 1024 * 1024) {
         await putS3Multipart(client, bucket, key, input);
         return;
       }
-      await withS3FailureDiagnostics("put_object", () => client.send(
+      const send = () => withS3FailureDiagnostics("put_object", () => client.send(
         new PutObjectCommand({
           Bucket: bucket,
           Key: key,
@@ -241,20 +243,21 @@ export function createS3StorageProvider(config: S3ProviderConfig): StorageProvid
           ContentLength: input.contentLength,
         }),
         { abortSignal: input.signal },
-      ));
+      ), input.signal);
+      await (Buffer.isBuffer(input.body) ? sendThroughStorageOutage(send, input.signal) : send());
     },
 
     async getObject(input): Promise<GetObjectResult> {
       const key = buildKey(prefix, input.objectKey);
       try {
-        const output = await withS3FailureDiagnostics("get_object", () => client.send(
+        const output = await sendThroughStorageOutage(() => withS3FailureDiagnostics("get_object", () => client.send(
           new GetObjectCommand({
             Bucket: bucket,
             Key: key,
             Range: input.range ? `bytes=${input.range.start}-${input.range.end}` : undefined,
           }),
           { abortSignal: input.signal },
-        ));
+        ), input.signal), input.signal);
 
         const first = observeBodyFailures(await toReadableStream(output.Body), output);
         const etag = output.ETag;
@@ -264,7 +267,7 @@ export function createS3StorageProvider(config: S3ProviderConfig): StorageProvid
           const resumed = await withS3FailureDiagnostics("get_object", () => client.send(new GetObjectCommand({
             Bucket: bucket, Key: key, IfMatch: etag,
             Range: `bytes=${start + offset}-${end ?? ""}`,
-          }), { abortSignal: resumeSignal }));
+          }), { abortSignal: resumeSignal }), resumeSignal);
           return observeBodyFailures(await toReadableStream(resumed.Body), resumed);
         };
         const stream = etag && end !== undefined
@@ -288,13 +291,13 @@ export function createS3StorageProvider(config: S3ProviderConfig): StorageProvid
     async headObject(input): Promise<HeadObjectResult> {
       const key = buildKey(prefix, input.objectKey);
       try {
-        const output = await withS3FailureDiagnostics("head_object", () => client.send(
+        const output = await sendThroughStorageOutage(() => withS3FailureDiagnostics("head_object", () => client.send(
           new HeadObjectCommand({
             Bucket: bucket,
             Key: key,
           }),
           { abortSignal: input.signal },
-        ));
+        ), input.signal), input.signal);
 
         return {
           exists: true,
@@ -313,7 +316,7 @@ export function createS3StorageProvider(config: S3ProviderConfig): StorageProvid
     async deleteObject(input): Promise<void> {
       const key = buildKey(prefix, input.objectKey);
       if (input.allVersions) {
-        await withS3FailureDiagnostics("delete_object", async () => {
+        await sendThroughStorageOutage(() => withS3FailureDiagnostics("delete_object", async () => {
           // Finish listing before deleting: removing a cursor version must not skip later pages.
           const versions = await listExactVersions(key, input.signal);
           for (const versionId of versions) {
@@ -322,16 +325,16 @@ export function createS3StorageProvider(config: S3ProviderConfig): StorageProvid
               { abortSignal: input.signal });
           }
           if ((await listExactVersions(key, input.signal)).size) throw new Error("s3_object_versions_still_present");
-        });
+        }, input.signal), input.signal);
         return;
       }
-      await withS3FailureDiagnostics("delete_object", () => client.send(
+      await sendThroughStorageOutage(() => withS3FailureDiagnostics("delete_object", () => client.send(
         new DeleteObjectCommand({
           Bucket: bucket,
           Key: key,
         }),
         { abortSignal: input.signal },
-      ));
+      ), input.signal), input.signal);
     },
   };
 }

server/src/storage/storage-unavailable.test.ts

@@ -0,0 +1,65 @@
+import { describe, expect, it } from "vitest";
+import { recordS3BodyFailure, withS3FailureDiagnostics } from "./s3-failure.js";
+import { isStorageUnavailableError } from "./storage-unavailable.js";
+
+function s3Error(name: string, status?: number, code?: string) {
+  return Object.assign(new Error(name), { name, ...(code ? { code } : {}),
+    $metadata: status === undefined ? {} : { httpStatusCode: status, attempts: 3 } });
+}
+
+async function recorded(error: unknown, signal?: AbortSignal) {
+  return withS3FailureDiagnostics("put_object", async () => { throw error; }, signal).catch(thrown => thrown);
+}
+
+function sdkAbort() {
+  const aborted = new Error("Request aborted");
+  aborted.name = "AbortError";
+  return aborted;
+}
+
+describe("storage unavailability", () => {
+  it.each([
+    ["503 ServiceUnavailable", s3Error("ServiceUnavailable", 503)],
+    ["503 SlowDown", s3Error("SlowDown", 503)],
+    ["429 throttling", s3Error("TooManyRequests", 429)],
+    ["500 InternalError", s3Error("InternalError", 500)],
+    ["408 RequestTimeout", s3Error("RequestTimeout", 408)],
+    ["a reset recorded under its error name", Object.assign(new Error("socket hang up"), { code: "ECONNRESET" })],
+    ["a refused connection without a status", s3Error("Error", undefined, "ECONNREFUSED")],
+  ])("treats %s from the store as unavailable", async (_label, error) => {
+    expect(isStorageUnavailableError(await recorded(error))).toBe(true);
+  });
+
+  it.each([
+    ["404 NoSuchKey", s3Error("NoSuchKey", 404)],
+    ["403 AccessDenied", s3Error("AccessDenied", 403)],
+    ["412 PreconditionFailed", s3Error("PreconditionFailed", 412)],
+    ["an ordinary error", new Error("adapter crashed")],
+  ])("does not treat %s as an outage", async (_label, error) => {
+    expect(isStorageUnavailableError(await recorded(error))).toBe(false);
+  });
+
+  it("treats a download body reset after a 200 response as unavailable", () => {
+    const reset = Object.assign(new Error("aborted"), { code: "ECONNRESET" });
+    recordS3BodyFailure(reset, { $metadata: { httpStatusCode: 200, requestId: "req-1" } });
+    expect(isStorageUnavailableError(reset)).toBe(true);
+  });
+
+  it("treats a storage call its deadline aborted as unavailable (storage chaos ST2)", async () => {
+    const deadline = AbortSignal.timeout(1);
+    await new Promise(resolve => setTimeout(resolve, 20));
+    expect(isStorageUnavailableError(await recorded(sdkAbort(), deadline))).toBe(true);
+  });
+
+  it("does not treat a run's own Stop abort as a storage outage", async () => {
+    const stop = new AbortController();
+    stop.abort(new Error("Cancelled by a board operator"));
+    expect(isStorageUnavailableError(await recorded(sdkAbort(), stop.signal))).toBe(false);
+    expect(isStorageUnavailableError(await recorded(sdkAbort()))).toBe(false);
+  });
+
+  it("requires the failure to come from a storage call", () => {
+    expect(isStorageUnavailableError(s3Error("ServiceUnavailable", 503))).toBe(false);
+    expect(isStorageUnavailableError(Object.assign(new Error("reset"), { code: "ECONNRESET" }))).toBe(false);
+  });
+});

server/src/storage/storage-unavailable.ts

@@ -0,0 +1,111 @@
+import { AsyncLocalStorage } from "node:async_hooks";
+import { setTimeout as delay } from "node:timers/promises";
+import { storageFailureDiagnosticsFor } from "./failure-diagnostics.js";
+import { recordStorageBackoff } from "./request-stats.js";
+
+export const STORAGE_UNAVAILABLE_CODE = "storage_unavailable";
+
+export const STORAGE_UNAVAILABLE_WAIT_RETRY_REASON = "storage_unavailable_wait";
+
+export const STORAGE_UNAVAILABLE_MESSAGE = "File storage was unavailable, so this attempt stopped before it could save its work. Paperclip will retry automatically when storage is back.";
+
+export const STORAGE_UNAVAILABLE_EXHAUSTED_MESSAGE = "File storage stayed unavailable through every automatic retry, so Paperclip stopped retrying this task. Retry once storage is back.";
+
+export const STORAGE_UNAVAILABLE_RETRY = {
+  retryReason: STORAGE_UNAVAILABLE_WAIT_RETRY_REASON, wakeReason: "storage_unavailable_retry", maxAttempts: 10, delayMs: 60_000,
+} as const;
+
+const UNAVAILABLE_STATUS = new Set([408, 429, 500, 502, 503, 504]);
+const UNAVAILABLE_NAMES = new Set(["ServiceUnavailable", "SlowDown", "InternalError", "RequestTimeout", "TimeoutError", "NetworkingError"]);
+const UNAVAILABLE_CODES = new Set(["ECONNRESET", "ECONNREFUSED", "ECONNABORTED", "ETIMEDOUT", "EPIPE", "EAI_AGAIN", "ENOTFOUND",
+  "ERR_STREAM_PREMATURE_CLOSE", "UND_ERR_SOCKET"]);
+
+function ownValue(value: unknown, key: string): unknown {
+  if (!value || typeof value !== "object") return undefined;
+  try {
+    const descriptor = Object.getOwnPropertyDescriptor(value, key);
+    return descriptor && "value" in descriptor ? descriptor.value : undefined;
+  } catch {
+    return undefined;
+  }
+}
+
+function networkFailure(error: unknown): boolean {
+  for (const candidate of [error, ownValue(error, "cause")]) {
+    const name = ownValue(candidate, "name"), code = ownValue(candidate, "code");
+    if ((typeof name === "string" && UNAVAILABLE_NAMES.has(name)) || (typeof code === "string" && UNAVAILABLE_CODES.has(code))) return true;
+  }
+  return false;
+}
+
+function recordedStorageFailure(error: unknown) {
+  for (let candidate = error, depth = 0; candidate && depth < 6; candidate = ownValue(candidate, "cause"), depth++) {
+    const diagnostics = storageFailureDiagnosticsFor(candidate);
+    if (diagnostics) return { error: candidate, diagnostics };
+  }
+  return null;
+}
+
+export function isStorageUnavailableError(error: unknown): boolean {
+  const recorded = recordedStorageFailure(error);
+  if (!recorded) return false;
+  const { diagnostics } = recorded;
+  const status = diagnostics.httpStatus;
+  if (status !== null && (status < 200 || status >= 300)) return UNAVAILABLE_STATUS.has(status);
+  return UNAVAILABLE_NAMES.has(diagnostics.errorCode) || UNAVAILABLE_CODES.has(diagnostics.errorCode) || networkFailure(recorded.error);
+}
+
+export const STORAGE_OUTAGE_WAIT = { budgetMs: 5 * 60_000, firstDelayMs: 2_000, maxDelayMs: 30_000 } as const;
+
+type OutageWaitScope = { stop?: AbortSignal };
+const outageWaitScope = new AsyncLocalStorage<OutageWaitScope>();
+type OutageWaitOperation = { firstFailureAt: number | null };
+const outageWaitOperation = new AsyncLocalStorage<OutageWaitOperation | null>();
+
+export function waitOutStorageOutages<T>(work: () => Promise<T>): Promise<T> {
+  return outageWaitScope.getStore() ? work() : outageWaitScope.run({}, work);
+}
+
+export function runInOwnStorageScope<T>(work: () => Promise<T>): Promise<T> {
+  return outageWaitScope.run({}, () => outageWaitOperation.run(null, work));
+}
+
+export function runSharedStorageWork<T>(work: () => Promise<T>): Promise<T> {
+  return outageWaitScope.getStore() ? outageWaitScope.run({}, () => outageWaitOperation.run(null, work)) : outageWaitOperation.run(null, work);
+}
+
+export function endStorageOutageWaitOn(stop: AbortSignal): void {
+  const current = outageWaitScope.getStore();
+  if (current) outageWaitScope.enterWith({ stop: current.stop ? AbortSignal.any([current.stop, stop]) : stop });
+}
+
+export function retryThroughStorageOutage<T>(send: () => Promise<T>, signal?: AbortSignal,
+  policy: { budgetMs: number; firstDelayMs: number; maxDelayMs: number } = STORAGE_OUTAGE_WAIT): Promise<T> {
+  const operation = outageWaitOperation.getStore();
+  return operation ? retryWithin(operation, send, signal, policy)
+    : outageWaitOperation.run({ firstFailureAt: null }, () => retryWithin(outageWaitOperation.getStore()!, send, signal, policy));
+}
+
+async function retryWithin<T>(operation: OutageWaitOperation, send: () => Promise<T>, signal: AbortSignal | undefined,
+  policy: { budgetMs: number; firstDelayMs: number; maxDelayMs: number }): Promise<T> {
+  const scopeStop = outageWaitScope.getStore()?.stop;
+  const stop = signal && scopeStop ? AbortSignal.any([signal, scopeStop]) : signal ?? scopeStop;
+  for (let wait = policy.firstDelayMs; ; wait = Math.min(wait * 2, policy.maxDelayMs)) {
+    try {
+      return await send();
+    } catch (error) {
+      if (!isStorageUnavailableError(error)) throw error;
+      const now = Date.now();
+      operation.firstFailureAt ??= now;
+      if (stop?.aborted || now - operation.firstFailureAt + wait > policy.budgetMs) throw error;
+      recordStorageBackoff(wait);
+      await delay(wait, undefined, { signal: stop }).catch((interrupted: unknown) => {
+        throw signal?.aborted && signal.reason instanceof DOMException && signal.reason.name === "TimeoutError" ? error : interrupted;
+      });
+    }
+  }
+}
+
+export function sendThroughStorageOutage<T>(send: () => Promise<T>, signal?: AbortSignal): Promise<T> {
+  return outageWaitScope.getStore() ? retryThroughStorageOutage(send, signal) : send();
+}