fix(storage): wait out a store outage on a run's critical path instead of failing it (#20299)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A cloud instance keeps run logs and evidence in object storage, so a
run touches the store as soon as it starts
> - In storage chaos wave `w-20261109T145628-s9902` (ST1: 75–176 s
windows where every S3 call returns 503), only 1 of 8 tasks finished. A
run starting or resuming during an outage failed within a second as
`adapter_failed`, generic recovery burned its attempts seconds apart,
the task was escalated to the board, and the user saw the raw S3 error
> - This pull request treats a transient storage error as an
infrastructure wait: the run is retried on a minute-scale schedule,
rapid recovery is suppressed, the board shows it as working, and the run
error says storage was unavailable
> - The benefit is that a store outage of several minutes delays work
instead of failing and stranding it

## Linked Issues or Issue Description

Refs #19831. Storage chaos wave `w-20261109T145628-s9902` (master
`610b54f551`, `--storage s3`, ST1).

**Where the run fails.** At run start or resume, `run-output` `begin`
opens the run log: `getRunLogStore().begin` → object run log store →
object byte journal `claim` → S3 conditional PUT. The journal retries
only 3 times within about 300 ms, so a 503 escapes and fails the run as
`adapter_failed`. The async stack ends in the SDK middleware with no app
frame. `log_store` stays null on the failed run, which pins the claim.

**What followed.**
- `failRun` released the issue into generic immediate recovery. Recovery
retried 1–2 s apart inside the outage (t01: 14:58:33, :35, :59,
14:59:01), then stopped and opened a board-owned
`stranded_assigned_issue` action. These are the open recovery actions on
CHA-1, CHA-4, CHA-7 and CHA-8; a later Retry hit the outage again and
escalated the same way.
- The run error was the raw S3 text ("Injected storage outage").

**Slow store (ST2, `w-20261109T151946-s9903`).** With about 9.7 s of
added latency per call, the same claim hit the journal's 15 s
per-operation deadline (`AbortSignal.timeout(15_000)`). The run failed
after about 36 s with `AbortError: Request aborted` and burned 3 runs
the same way; 4 of 8 tasks needed a Retry.

t02/t03 (Retry parked with `process_identity_missing`) are a separate
native-recovery class and are handled in a follow-up.


**Daytona storage chaos ST1 (`w-20261109T161202-s9912`, 1/8):** with S3
returning 503 for 72 to 154 s at a time:
- Workspace restore failed at once (`remote_workspace_restore_failed:
step=snapshot_download; storage=provider_error`), three times in 40 s,
then `setup_failed`. It left an active recovery action behind.
- Runs reattached after a restart died within 2 s on the run-log claim,
then became reconciliation-required.
- Setup failed on other storage reads.
- 6 of 7 failures needed a UI Retry.

So the rule here is one class rule: a transient store error on a run's
critical path waits, it never fails the run.

## What Changed

- New `server/src/storage/storage-unavailable.ts`:
- `isStorageUnavailableError` reads the recorded storage failure
diagnostics first, then the SDK error shape. Unavailable means
429/500/502/503/504, the S3 throttling and timeout names, network codes
(`ECONNRESET`, `ECONNREFUSED`, `ETIMEDOUT`, `EPIPE`, `EAI_AGAIN`,
`ENOTFOUND`), or an `AbortError` whose cause is a `TimeoutError` (a
per-operation deadline). 403, 404 and 412 are not outages, and neither
is an abort caused by the run's own Stop.
- It defines the `storage_unavailable` code, the
`storage_unavailable_wait` retry reason, the retry policy (60 s × 10
attempts, about 10 minutes) and the user-facing message.
- `server/src/services/heartbeat/run-completion.ts`:
- `failRun` classifies such a failure as `storage_unavailable` with the
plain message.
- Unless native recovery owns the run, it schedules the storage wait
with `scheduleBoundedRetryForRun`, skips the interaction-continuation
retry, and releases the issue with `suppressImmediateRecovery`.
  - `failRunSetup` uses the same policy for this code.
- `server/src/services/heartbeat/retries.ts`: `storage_unavailable`
joins the transient infrastructure codes. It gets its own retry budget
and policy, like `runner_revision_mismatch`.
- `server/src/services/task-interruption-presentation.ts`:
`storage_unavailable_wait` is a resource wait, so the board shows the
task as working with no Retry while it waits.

- **One shared outage wait for the run's critical path**
(`server/src/storage/storage-unavailable.ts`):
- `retryThroughStorageOutage` retries a transient store failure with
backoff (2 s doubling to 30 s) until that storage operation's outage has
lasted 5 minutes, then gives the error back. The outermost wait of an
operation records its first failure, and nested and parallel waits
inside it share that start; no other request's success extends it.
Shared probes and shared reads run under their own scope, so one run's
Stop never cancels work another run waits on. A stop signal ends the
wait, and non-outage errors are never retried.
- Inside `waitOutStorageOutages`, every replayable S3 request waits
instead of failing: get, head, delete, buffered and file puts, and
multipart create, file parts and complete. A consumed upload stream is
never resent, and requests outside the scope keep failing fast.
- The scope covers run execution (`heartbeat.ts` `executeRun`: setup,
workspace restore, run-log claim and the run itself), plus
`loadWorkspaceSnapshotArchive` and `uploadWorkspaceSnapshot`, which
copy-back and recovery workers use too.
- The run-log claim retries as a whole, because the journal bounds each
operation with a 15 s deadline.
- Past the budget, the run takes the bounded `storage_unavailable` wait
below.
- The classifier follows the error's `cause` chain, so a restore failure
wrapping an outage counts. `failRunSetup` checks storage before restore
failures, so that case takes the storage wait instead of two quick
retries and an escalation (the R6 leak).
- `object-byte-journal.ts`: when its deadline expires, the journal
rethrows the recorded storage error instead of a bare `TimeoutError`.

## Verification

Red/green (each new test fails against master and passes with this PR):
- `run-completion.test.ts` "waits a minute and suppresses rapid recovery
when a run fails because storage is unavailable (storage chaos ST1)". An
S3-shaped 503 records `storage_unavailable` with the plain message (no
raw S3 text), schedules `STORAGE_UNAVAILABLE_RETRY`, releases with
`suppressImmediateRecovery: true`, and skips the continuation retry.
"keeps an ordinary adapter failure on the normal recovery path" guards
the default.
- `task-interruption-query.test.ts` "does not present an overdue
storage_unavailable_wait wait as a failed attempt": working, `canRetry:
false`.
- `storage-unavailable.test.ts`: which errors count as an outage and
which do not, including recorded diagnostics, a deadline-aborted call
(ST2), and a Stop-aborted call that must not count.

Static checks:
- `tsc --noEmit` passes for `server`.

Live verification: Phase 3 will rerun ST1 (seed 9902) with the MinIO
fault proxy after merge.

- `server/src/__tests__/storage-outage-wait.test.ts` runs against a
local S3 that answers 503 for 3 s:
- Inside the scope, a snapshot `getObject` and a run-log conditional PUT
succeed once the store is back.
  - Outside it, a request fails fast.
  - A stop ends the wait.
- Unit cases cover the budget, non-outage errors, and a
`RemoteWorkspaceRestoreError` wrapping an outage.
- `run-completion.test.ts`: a setup failure from a restore that wrapped
an outage takes the storage wait (red before).
- `object-byte-journal.test.ts`: a real `claim` whose head PUT hits its
deadline is classified as a storage outage (red before).

## Risks

- A storage outage now holds a task for up to about 10 minutes of
automatic waits instead of escalating within seconds. A longer outage
still ends in the existing board recovery.
- A run whose failure is misclassified as storage-unavailable waits
instead of using fast recovery. The classifier accepts only server-side
or network failures from the storage SDK, never 4xx answers.
- No schema change.

- A run can now wait until a storage outage has lasted 5 minutes before
it fails over to the bounded wait. It shows as working throughout, and
Stop ends the wait.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-09 18:15:48 -06:00
1 parent 0116e45958
commit 4a2eb920cf
25 files changed
+821 -80

No files matched your search

@@ -5,6 +5,8 @@ import { agents, authUsers, companies, companyMemberships, createDb, heartbeatRu
import { registerServerAdapter, unregisterServerAdapter, type AdapterExecutionContext } from "../adapters/index.js";
import { heartbeatService } from "../services/heartbeat.js";
import { getRunLogStore } from "../services/run-log-store.js";
import { withS3FailureDiagnostics } from "../storage/s3-failure.js";
import { sendThroughStorageOutage } from "../storage/storage-unavailable.js";
import { subscribeCompanyLiveEvents } from "../services/live-events.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
@@ -101,6 +103,37 @@ describePostgres("heartbeat output through adapter dispatch and finalization", (
} finally { unsubscribe(); }
});
const storeOutage = () => withS3FailureDiagnostics("put_object", async () => {
throw Object.assign(new Error("Injected storage outage"), { name: "ServiceUnavailable", $metadata: { httpStatusCode: 503 } });
});
it("claims the run log once the store is back and finishes the run (storage chaos ST1)", async () => {
const f = await seed(async input => { await input.onLog("stdout", "after the outage\n"); });
const store = getRunLogStore();
const begin = store.begin.bind(store);
let outages = 0;
vi.spyOn(store, "begin").mockImplementation(async input => (++outages <= 1 ? storeOutage() : begin(input)));
const { finished } = await execute(f);
expect(outages).toBeGreaterThanOrEqual(2);
expect(finished.status, finished.error ?? undefined).toBe("succeeded");
expect(finished.stdoutExcerpt).toContain("after the outage");
}, 30_000);
it("ends a run stopped during a storage outage as cancelled, even in a storage call made without the run's signal", async () => {
const f = await seed(async () => undefined);
const store = getRunLogStore();
const begin = vi.spyOn(store, "begin").mockImplementation(() => sendThroughStorageOutage(storeOutage));
const run = await heartbeat.wakeup(f.agentId, { source: "on_demand", triggerDetail: "manual", reason: "manual",
payload: { issueId: f.issueId }, contextSnapshot: { issueId: f.issueId } });
await vi.waitFor(() => expect(begin).toHaveBeenCalled(), { timeout: 10_000 });
const stoppedAt = Date.now();
await heartbeat.cancelRun(run!.id, "Cancelled by a board operator");
await heartbeat.drainActiveRunExecutions();
expect(Date.now() - stoppedAt).toBeLessThan(10_000);
const [stopped] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id));
expect(stopped).toMatchObject({ status: "cancelled" });
}, 60_000);
it.each([false, true])("retains output arriving during finalization (adapter throws: %s)", async fail => {
let onLog: AdapterExecutionContext["onLog"];
const f = await seed(async input => {
@@ -0,0 +1,219 @@
import { createServer } from "node:http";
import type { AddressInfo } from "node:net";
import { afterAll, beforeAll, beforeEach, describe, expect, it } from "vitest";
import { createS3StorageProvider } from "../storage/s3-provider.js";
import { withS3FailureDiagnostics } from "../storage/s3-failure.js";
import { endStorageOutageWaitOn, isStorageUnavailableError, retryThroughStorageOutage, runInOwnStorageScope, runSharedStorageWork, waitOutStorageOutages } from "../storage/storage-unavailable.js";
import { RemoteWorkspaceRestoreError } from "../services/remote-workspace-transport.js";
const OUTAGE = `<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>ServiceUnavailable</Code><Message>Injected storage outage</Message><RequestId>req-503</RequestId></Error>`;
let outageUntil = 0;
let requests = 0;
const server = createServer((req, res) => {
req.resume();
req.on("end", () => {
requests++;
if (Date.now() < outageUntil) {
res.writeHead(503, { "content-type": "application/xml" });
res.end(OUTAGE);
return;
}
if (req.method === "GET") {
res.writeHead(200, { "content-length": "5", etag: '"v1"' });
res.end("saved");
return;
}
res.writeHead(200, { etag: '"v2"' });
res.end();
});
});
const savedEnv = { ...process.env };
let endpoint = "";
beforeAll(async () => {
await new Promise<void>(resolve => server.listen(0, "127.0.0.1", resolve));
endpoint = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
process.env.AWS_ACCESS_KEY_ID = "fixture-access-key";
process.env.AWS_SECRET_ACCESS_KEY = "fixture-secret-key";
delete process.env.AWS_SESSION_TOKEN;
delete process.env.AWS_PROFILE;
});
afterAll(async () => {
process.env = savedEnv;
await new Promise<void>(resolve => server.close(() => resolve()));
});
beforeEach(() => { requests = 0; outageUntil = 0; });
const provider = () => createS3StorageProvider({ bucket: "bucket", region: "us-east-1", endpoint, forcePathStyle: true });
async function read(stream: NodeJS.ReadableStream) {
const chunks: Buffer[] = [];
for await (const chunk of stream) chunks.push(Buffer.from(chunk as Buffer));
return Buffer.concat(chunks).toString("utf8");
}
describe("waiting out a storage outage on a run's critical path (Daytona storage chaos ST1)", () => {
it("restores a saved object once the store is back instead of failing the run", async () => {
outageUntil = Date.now() + 3_000;
const object = await waitOutStorageOutages(() => provider().getObject({ objectKey: "company/workspace.pcws" }));
expect(await read(object.stream)).toBe("saved");
expect(requests).toBeGreaterThan(1);
}, 30_000);
it("claims the run log once the store is back", async () => {
outageUntil = Date.now() + 3_000;
await expect(waitOutStorageOutages(() => provider().putObjectConditional!({ objectKey: "company/runs/head.json",
body: Buffer.from("{}"), contentType: "application/json", contentLength: 2, ifNoneMatch: "*" })))
.resolves.toEqual({ etag: '"v2"' });
}, 30_000);
it("keeps failing fast outside a run's critical path", async () => {
outageUntil = Date.now() + 60_000;
const startedAt = Date.now();
await expect(provider().headObject({ objectKey: "company/avatar.png" })).rejects.toMatchObject({ $metadata: { httpStatusCode: 503 } });
expect(Date.now() - startedAt).toBeLessThan(10_000);
}, 30_000);
it("stops waiting when the run is stopped", async () => {
outageUntil = Date.now() + 60_000;
const stop = new AbortController();
setTimeout(() => stop.abort(new Error("Stopped by a board operator")), 3_000);
const startedAt = Date.now();
await expect(waitOutStorageOutages(() => provider().getObject({ objectKey: "company/workspace.pcws", signal: stop.signal })))
.rejects.toBeTruthy();
expect(Date.now() - startedAt).toBeLessThan(15_000);
}, 30_000);
});
describe("the shared storage outage wait", () => {
const outage = () => withS3FailureDiagnostics("get_object", async () => {
throw Object.assign(new Error("Injected storage outage"), { name: "ServiceUnavailable", $metadata: { httpStatusCode: 503 } });
});
const policy = { budgetMs: 200, firstDelayMs: 10, maxDelayMs: 40 };
it("retries until the store answers, within its budget", async () => {
let calls = 0;
await expect(retryThroughStorageOutage(async () => (++calls < 4 ? outage() : "ok"), undefined, policy)).resolves.toBe("ok");
expect(calls).toBe(4);
});
it("gives the outage back to the run's storage wait once the budget is spent", async () => {
let calls = 0;
await expect(retryThroughStorageOutage(async () => { calls++; return outage(); }, undefined, policy))
.rejects.toMatchObject({ name: "ServiceUnavailable" });
expect(calls).toBeGreaterThan(1);
});
it("keeps the storage failure when a deadline ends the wait, but not when a Stop does", async () => {
const slow = { budgetMs: 60_000, firstDelayMs: 1_000, maxDelayMs: 1_000 };
await expect(retryThroughStorageOutage(outage, AbortSignal.timeout(50), slow))
.rejects.toSatisfy(isStorageUnavailableError);
const stop = new AbortController();
setTimeout(() => stop.abort(new Error("Stopped by a board operator")), 50);
const stopped = await retryThroughStorageOutage(outage, stop.signal, slow).catch((error: unknown) => error);
expect(stopped).toMatchObject({ name: "AbortError" });
expect(isStorageUnavailableError(stopped)).toBe(false);
});
it("lets nested and parallel waits ride out one outage shorter than the budget", async () => {
const wide = { budgetMs: 600, firstDelayMs: 20, maxDelayMs: 40 };
const endsAt = Date.now() + 300;
const send = async () => (Date.now() < endsAt ? outage() : "ok");
const nested = () => retryThroughStorageOutage(() => retryThroughStorageOutage(send, undefined, wide), undefined, wide);
await expect(waitOutStorageOutages(() => Promise.all(Array.from({ length: 8 }, nested))))
.resolves.toEqual(Array(8).fill("ok"));
});
it("gives up once one outage has lasted longer than the budget, however the waits overlap", async () => {
const startedAt = Date.now();
const failing = () => retryThroughStorageOutage(outage, undefined, policy);
const results = await waitOutStorageOutages(() => Promise.allSettled(Array.from({ length: 4 }, failing)));
expect(results.every(result => result.status === "rejected")).toBe(true);
expect(Date.now() - startedAt).toBeGreaterThanOrEqual(policy.budgetMs - policy.maxDelayMs);
expect(Date.now() - startedAt).toBeLessThan(policy.budgetMs * 3);
});
it("keeps the limit when only some keys fail and other storage calls succeed", async () => {
const startedAt = Date.now();
const result = await waitOutStorageOutages(async () => {
const reads = setInterval(() => void retryThroughStorageOutage(async () => "read", undefined, policy), 5);
try {
return await retryThroughStorageOutage(async () => {
await retryThroughStorageOutage(async () => "head read", undefined, policy);
return outage();
}, undefined, policy).catch((error: unknown) => error);
} finally { clearInterval(reads); }
});
expect(result).toMatchObject({ name: "ServiceUnavailable" });
expect(Date.now() - startedAt).toBeLessThan(policy.budgetMs * 3);
});
it("keeps one operation's limit when another operation in the same run recovers from its own outage", async () => {
const startedAt = Date.now();
const result = await waitOutStorageOutages(async () => {
let recovering = true;
void (async () => {
while (recovering) {
let failed = false;
await retryThroughStorageOutage(async () => (failed ? "recovered" : (failed = true, outage())), undefined, policy).catch(() => undefined);
}
})();
try {
return await retryThroughStorageOutage(outage, undefined, policy).catch((error: unknown) => error);
} finally { recovering = false; }
});
expect(result).toMatchObject({ name: "ServiceUnavailable" });
expect(Date.now() - startedAt).toBeLessThan(policy.budgetMs * 3);
});
it("gives a later outage its own limit once the waiting work recovered", async () => {
await waitOutStorageOutages(async () => {
const recoversAt = Date.now() + policy.budgetMs - 60;
await expect(retryThroughStorageOutage(async () => (Date.now() < recoversAt ? outage() : "ok"), undefined, policy)).resolves.toBe("ok");
const secondRecoversAt = Date.now() + policy.budgetMs - 60;
await expect(retryThroughStorageOutage(async () => (Date.now() < secondRecoversAt ? outage() : "again"), undefined, policy)).resolves.toBe("again");
});
});
it("gives a run started from a stopped run's cleanup its own scope", async () => {
const stopped = new AbortController();
stopped.abort(new Error("Stopped by a board operator"));
const wide = { budgetMs: 2_000, firstDelayMs: 20, maxDelayMs: 40 };
let calls = 0;
const next = await waitOutStorageOutages(async () => {
endStorageOutageWaitOn(stopped.signal);
return runInOwnStorageScope(() => retryThroughStorageOutage(async () => (++calls < 3 ? outage() : "next run"), undefined, wide));
});
expect(next).toBe("next run");
});
it("lets one run's Stop end only its own wait, not shared work another run waits on", async () => {
const endsAt = Date.now() + 150;
const wide = { budgetMs: 2_000, firstDelayMs: 20, maxDelayMs: 40 };
const stopA = new AbortController();
let shared!: Promise<string>;
const runA = waitOutStorageOutages(async () => {
endStorageOutageWaitOn(stopA.signal);
shared = runSharedStorageWork(() => retryThroughStorageOutage(async () => (Date.now() < endsAt ? outage() : "shared"), undefined, wide));
return retryThroughStorageOutage(outage, undefined, wide);
});
setTimeout(() => stopA.abort(new Error("Stopped by a board operator")), 30);
await expect(runA).rejects.toMatchObject({ name: "AbortError" });
await expect(shared).resolves.toBe("shared");
});
it("never retries a failure that is not an outage", async () => {
let calls = 0;
await expect(retryThroughStorageOutage(async () => { calls++; throw new Error("object_journal_conflict"); }, undefined, policy))
.rejects.toThrow("object_journal_conflict");
expect(calls).toBe(1);
});
it("recognizes an outage a workspace restore wrapped (snapshot_download)", async () => {
const cause = await outage().catch(error => error);
expect(isStorageUnavailableError(new RemoteWorkspaceRestoreError("snapshot_download", true, cause))).toBe(true);
expect(isStorageUnavailableError(new RemoteWorkspaceRestoreError("sandbox_restore", true, new Error("reason=occupied")))).toBe(false);
});
});
@@ -607,7 +607,7 @@ describe.skipIf(!support.supported)("task interruption latest run query", () =>
});
// N-03: ONB-1 on staging waited 16 minutes on an overdue workspace_busy
// retry. Nothing failed, yet the banner offered a Retry that cannot help.
it.each(["workspace_busy", "ai_connection_busy", "ai_connection_pool_wait", "runner_revision_wait"])(
it.each(["workspace_busy", "ai_connection_busy", "ai_connection_pool_wait", "runner_revision_wait", "storage_unavailable_wait"])(
"does not present an overdue %s wait as a failed attempt", async reason => {
const companyId = randomUUID(), agentId = randomUUID(), parentRunId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Waiting task", issuePrefix: `W${companyId.slice(0, 7)}` });