mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-15 18:56:05 +01:00
fix(storage): wait out a store outage on a run's critical path instead of failing it (#20299)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A cloud instance keeps run logs and evidence in object storage, so a run touches the store as soon as it starts > - In storage chaos wave `w-20261109T145628-s9902` (ST1: 75–176 s windows where every S3 call returns 503), only 1 of 8 tasks finished. A run starting or resuming during an outage failed within a second as `adapter_failed`, generic recovery burned its attempts seconds apart, the task was escalated to the board, and the user saw the raw S3 error > - This pull request treats a transient storage error as an infrastructure wait: the run is retried on a minute-scale schedule, rapid recovery is suppressed, the board shows it as working, and the run error says storage was unavailable > - The benefit is that a store outage of several minutes delays work instead of failing and stranding it ## Linked Issues or Issue Description Refs #19831. Storage chaos wave `w-20261109T145628-s9902` (master `610b54f551`, `--storage s3`, ST1). **Where the run fails.** At run start or resume, `run-output` `begin` opens the run log: `getRunLogStore().begin` → object run log store → object byte journal `claim` → S3 conditional PUT. The journal retries only 3 times within about 300 ms, so a 503 escapes and fails the run as `adapter_failed`. The async stack ends in the SDK middleware with no app frame. `log_store` stays null on the failed run, which pins the claim. **What followed.** - `failRun` released the issue into generic immediate recovery. Recovery retried 1–2 s apart inside the outage (t01: 14:58:33, :35, :59, 14:59:01), then stopped and opened a board-owned `stranded_assigned_issue` action. These are the open recovery actions on CHA-1, CHA-4, CHA-7 and CHA-8; a later Retry hit the outage again and escalated the same way. - The run error was the raw S3 text ("Injected storage outage"). **Slow store (ST2, `w-20261109T151946-s9903`).** With about 9.7 s of added latency per call, the same claim hit the journal's 15 s per-operation deadline (`AbortSignal.timeout(15_000)`). The run failed after about 36 s with `AbortError: Request aborted` and burned 3 runs the same way; 4 of 8 tasks needed a Retry. t02/t03 (Retry parked with `process_identity_missing`) are a separate native-recovery class and are handled in a follow-up. **Daytona storage chaos ST1 (`w-20261109T161202-s9912`, 1/8):** with S3 returning 503 for 72 to 154 s at a time: - Workspace restore failed at once (`remote_workspace_restore_failed: step=snapshot_download; storage=provider_error`), three times in 40 s, then `setup_failed`. It left an active recovery action behind. - Runs reattached after a restart died within 2 s on the run-log claim, then became reconciliation-required. - Setup failed on other storage reads. - 6 of 7 failures needed a UI Retry. So the rule here is one class rule: a transient store error on a run's critical path waits, it never fails the run. ## What Changed - New `server/src/storage/storage-unavailable.ts`: - `isStorageUnavailableError` reads the recorded storage failure diagnostics first, then the SDK error shape. Unavailable means 429/500/502/503/504, the S3 throttling and timeout names, network codes (`ECONNRESET`, `ECONNREFUSED`, `ETIMEDOUT`, `EPIPE`, `EAI_AGAIN`, `ENOTFOUND`), or an `AbortError` whose cause is a `TimeoutError` (a per-operation deadline). 403, 404 and 412 are not outages, and neither is an abort caused by the run's own Stop. - It defines the `storage_unavailable` code, the `storage_unavailable_wait` retry reason, the retry policy (60 s × 10 attempts, about 10 minutes) and the user-facing message. - `server/src/services/heartbeat/run-completion.ts`: - `failRun` classifies such a failure as `storage_unavailable` with the plain message. - Unless native recovery owns the run, it schedules the storage wait with `scheduleBoundedRetryForRun`, skips the interaction-continuation retry, and releases the issue with `suppressImmediateRecovery`. - `failRunSetup` uses the same policy for this code. - `server/src/services/heartbeat/retries.ts`: `storage_unavailable` joins the transient infrastructure codes. It gets its own retry budget and policy, like `runner_revision_mismatch`. - `server/src/services/task-interruption-presentation.ts`: `storage_unavailable_wait` is a resource wait, so the board shows the task as working with no Retry while it waits. - **One shared outage wait for the run's critical path** (`server/src/storage/storage-unavailable.ts`): - `retryThroughStorageOutage` retries a transient store failure with backoff (2 s doubling to 30 s) until that storage operation's outage has lasted 5 minutes, then gives the error back. The outermost wait of an operation records its first failure, and nested and parallel waits inside it share that start; no other request's success extends it. Shared probes and shared reads run under their own scope, so one run's Stop never cancels work another run waits on. A stop signal ends the wait, and non-outage errors are never retried. - Inside `waitOutStorageOutages`, every replayable S3 request waits instead of failing: get, head, delete, buffered and file puts, and multipart create, file parts and complete. A consumed upload stream is never resent, and requests outside the scope keep failing fast. - The scope covers run execution (`heartbeat.ts` `executeRun`: setup, workspace restore, run-log claim and the run itself), plus `loadWorkspaceSnapshotArchive` and `uploadWorkspaceSnapshot`, which copy-back and recovery workers use too. - The run-log claim retries as a whole, because the journal bounds each operation with a 15 s deadline. - Past the budget, the run takes the bounded `storage_unavailable` wait below. - The classifier follows the error's `cause` chain, so a restore failure wrapping an outage counts. `failRunSetup` checks storage before restore failures, so that case takes the storage wait instead of two quick retries and an escalation (the R6 leak). - `object-byte-journal.ts`: when its deadline expires, the journal rethrows the recorded storage error instead of a bare `TimeoutError`. ## Verification Red/green (each new test fails against master and passes with this PR): - `run-completion.test.ts` "waits a minute and suppresses rapid recovery when a run fails because storage is unavailable (storage chaos ST1)". An S3-shaped 503 records `storage_unavailable` with the plain message (no raw S3 text), schedules `STORAGE_UNAVAILABLE_RETRY`, releases with `suppressImmediateRecovery: true`, and skips the continuation retry. "keeps an ordinary adapter failure on the normal recovery path" guards the default. - `task-interruption-query.test.ts` "does not present an overdue storage_unavailable_wait wait as a failed attempt": working, `canRetry: false`. - `storage-unavailable.test.ts`: which errors count as an outage and which do not, including recorded diagnostics, a deadline-aborted call (ST2), and a Stop-aborted call that must not count. Static checks: - `tsc --noEmit` passes for `server`. Live verification: Phase 3 will rerun ST1 (seed 9902) with the MinIO fault proxy after merge. - `server/src/__tests__/storage-outage-wait.test.ts` runs against a local S3 that answers 503 for 3 s: - Inside the scope, a snapshot `getObject` and a run-log conditional PUT succeed once the store is back. - Outside it, a request fails fast. - A stop ends the wait. - Unit cases cover the budget, non-outage errors, and a `RemoteWorkspaceRestoreError` wrapping an outage. - `run-completion.test.ts`: a setup failure from a restore that wrapped an outage takes the storage wait (red before). - `object-byte-journal.test.ts`: a real `claim` whose head PUT hits its deadline is classified as a storage outage (red before). ## Risks - A storage outage now holds a task for up to about 10 minutes of automatic waits instead of escalating within seconds. A longer outage still ends in the existing board recovery. - A run whose failure is misclassified as storage-unavailable waits instead of using fast recovery. The classifier accepts only server-side or network failures from the storage SDK, never 4xx answers. - No schema change. - A run can now wait until a storage outage has lasted 5 minutes before it fails over to the bounded wait. It shows as working throughout, and Stop ends the wait. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
0116e45958
commit
4a2eb920cf
25 files changed
+821
-80
No files matched your search
@@ -5,6 +5,8 @@ import { agents, authUsers, companies, companyMemberships, createDb, heartbeatRu
|
||||
import { registerServerAdapter, unregisterServerAdapter, type AdapterExecutionContext } from "../adapters/index.js";
|
||||
import { heartbeatService } from "../services/heartbeat.js";
|
||||
import { getRunLogStore } from "../services/run-log-store.js";
|
||||
import { withS3FailureDiagnostics } from "../storage/s3-failure.js";
|
||||
import { sendThroughStorageOutage } from "../storage/storage-unavailable.js";
|
||||
import { subscribeCompanyLiveEvents } from "../services/live-events.js";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
|
||||
@@ -101,6 +103,37 @@ describePostgres("heartbeat output through adapter dispatch and finalization", (
|
||||
} finally { unsubscribe(); }
|
||||
});
|
||||
|
||||
const storeOutage = () => withS3FailureDiagnostics("put_object", async () => {
|
||||
throw Object.assign(new Error("Injected storage outage"), { name: "ServiceUnavailable", $metadata: { httpStatusCode: 503 } });
|
||||
});
|
||||
|
||||
it("claims the run log once the store is back and finishes the run (storage chaos ST1)", async () => {
|
||||
const f = await seed(async input => { await input.onLog("stdout", "after the outage\n"); });
|
||||
const store = getRunLogStore();
|
||||
const begin = store.begin.bind(store);
|
||||
let outages = 0;
|
||||
vi.spyOn(store, "begin").mockImplementation(async input => (++outages <= 1 ? storeOutage() : begin(input)));
|
||||
const { finished } = await execute(f);
|
||||
expect(outages).toBeGreaterThanOrEqual(2);
|
||||
expect(finished.status, finished.error ?? undefined).toBe("succeeded");
|
||||
expect(finished.stdoutExcerpt).toContain("after the outage");
|
||||
}, 30_000);
|
||||
|
||||
it("ends a run stopped during a storage outage as cancelled, even in a storage call made without the run's signal", async () => {
|
||||
const f = await seed(async () => undefined);
|
||||
const store = getRunLogStore();
|
||||
const begin = vi.spyOn(store, "begin").mockImplementation(() => sendThroughStorageOutage(storeOutage));
|
||||
const run = await heartbeat.wakeup(f.agentId, { source: "on_demand", triggerDetail: "manual", reason: "manual",
|
||||
payload: { issueId: f.issueId }, contextSnapshot: { issueId: f.issueId } });
|
||||
await vi.waitFor(() => expect(begin).toHaveBeenCalled(), { timeout: 10_000 });
|
||||
const stoppedAt = Date.now();
|
||||
await heartbeat.cancelRun(run!.id, "Cancelled by a board operator");
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
expect(Date.now() - stoppedAt).toBeLessThan(10_000);
|
||||
const [stopped] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id));
|
||||
expect(stopped).toMatchObject({ status: "cancelled" });
|
||||
}, 60_000);
|
||||
|
||||
it.each([false, true])("retains output arriving during finalization (adapter throws: %s)", async fail => {
|
||||
let onLog: AdapterExecutionContext["onLog"];
|
||||
const f = await seed(async input => {
|
||||
|
||||
@@ -0,0 +1,219 @@
|
||||
import { createServer } from "node:http";
|
||||
import type { AddressInfo } from "node:net";
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it } from "vitest";
|
||||
import { createS3StorageProvider } from "../storage/s3-provider.js";
|
||||
import { withS3FailureDiagnostics } from "../storage/s3-failure.js";
|
||||
import { endStorageOutageWaitOn, isStorageUnavailableError, retryThroughStorageOutage, runInOwnStorageScope, runSharedStorageWork, waitOutStorageOutages } from "../storage/storage-unavailable.js";
|
||||
import { RemoteWorkspaceRestoreError } from "../services/remote-workspace-transport.js";
|
||||
|
||||
const OUTAGE = `<?xml version="1.0" encoding="UTF-8"?>
|
||||
<Error><Code>ServiceUnavailable</Code><Message>Injected storage outage</Message><RequestId>req-503</RequestId></Error>`;
|
||||
|
||||
let outageUntil = 0;
|
||||
let requests = 0;
|
||||
const server = createServer((req, res) => {
|
||||
req.resume();
|
||||
req.on("end", () => {
|
||||
requests++;
|
||||
if (Date.now() < outageUntil) {
|
||||
res.writeHead(503, { "content-type": "application/xml" });
|
||||
res.end(OUTAGE);
|
||||
return;
|
||||
}
|
||||
if (req.method === "GET") {
|
||||
res.writeHead(200, { "content-length": "5", etag: '"v1"' });
|
||||
res.end("saved");
|
||||
return;
|
||||
}
|
||||
res.writeHead(200, { etag: '"v2"' });
|
||||
res.end();
|
||||
});
|
||||
});
|
||||
const savedEnv = { ...process.env };
|
||||
let endpoint = "";
|
||||
|
||||
beforeAll(async () => {
|
||||
await new Promise<void>(resolve => server.listen(0, "127.0.0.1", resolve));
|
||||
endpoint = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
|
||||
process.env.AWS_ACCESS_KEY_ID = "fixture-access-key";
|
||||
process.env.AWS_SECRET_ACCESS_KEY = "fixture-secret-key";
|
||||
delete process.env.AWS_SESSION_TOKEN;
|
||||
delete process.env.AWS_PROFILE;
|
||||
});
|
||||
afterAll(async () => {
|
||||
process.env = savedEnv;
|
||||
await new Promise<void>(resolve => server.close(() => resolve()));
|
||||
});
|
||||
beforeEach(() => { requests = 0; outageUntil = 0; });
|
||||
|
||||
const provider = () => createS3StorageProvider({ bucket: "bucket", region: "us-east-1", endpoint, forcePathStyle: true });
|
||||
|
||||
async function read(stream: NodeJS.ReadableStream) {
|
||||
const chunks: Buffer[] = [];
|
||||
for await (const chunk of stream) chunks.push(Buffer.from(chunk as Buffer));
|
||||
return Buffer.concat(chunks).toString("utf8");
|
||||
}
|
||||
|
||||
describe("waiting out a storage outage on a run's critical path (Daytona storage chaos ST1)", () => {
|
||||
it("restores a saved object once the store is back instead of failing the run", async () => {
|
||||
outageUntil = Date.now() + 3_000;
|
||||
const object = await waitOutStorageOutages(() => provider().getObject({ objectKey: "company/workspace.pcws" }));
|
||||
expect(await read(object.stream)).toBe("saved");
|
||||
expect(requests).toBeGreaterThan(1);
|
||||
}, 30_000);
|
||||
|
||||
it("claims the run log once the store is back", async () => {
|
||||
outageUntil = Date.now() + 3_000;
|
||||
await expect(waitOutStorageOutages(() => provider().putObjectConditional!({ objectKey: "company/runs/head.json",
|
||||
body: Buffer.from("{}"), contentType: "application/json", contentLength: 2, ifNoneMatch: "*" })))
|
||||
.resolves.toEqual({ etag: '"v2"' });
|
||||
}, 30_000);
|
||||
|
||||
it("keeps failing fast outside a run's critical path", async () => {
|
||||
outageUntil = Date.now() + 60_000;
|
||||
const startedAt = Date.now();
|
||||
await expect(provider().headObject({ objectKey: "company/avatar.png" })).rejects.toMatchObject({ $metadata: { httpStatusCode: 503 } });
|
||||
expect(Date.now() - startedAt).toBeLessThan(10_000);
|
||||
}, 30_000);
|
||||
|
||||
it("stops waiting when the run is stopped", async () => {
|
||||
outageUntil = Date.now() + 60_000;
|
||||
const stop = new AbortController();
|
||||
setTimeout(() => stop.abort(new Error("Stopped by a board operator")), 3_000);
|
||||
const startedAt = Date.now();
|
||||
await expect(waitOutStorageOutages(() => provider().getObject({ objectKey: "company/workspace.pcws", signal: stop.signal })))
|
||||
.rejects.toBeTruthy();
|
||||
expect(Date.now() - startedAt).toBeLessThan(15_000);
|
||||
}, 30_000);
|
||||
});
|
||||
|
||||
describe("the shared storage outage wait", () => {
|
||||
const outage = () => withS3FailureDiagnostics("get_object", async () => {
|
||||
throw Object.assign(new Error("Injected storage outage"), { name: "ServiceUnavailable", $metadata: { httpStatusCode: 503 } });
|
||||
});
|
||||
const policy = { budgetMs: 200, firstDelayMs: 10, maxDelayMs: 40 };
|
||||
|
||||
it("retries until the store answers, within its budget", async () => {
|
||||
let calls = 0;
|
||||
await expect(retryThroughStorageOutage(async () => (++calls < 4 ? outage() : "ok"), undefined, policy)).resolves.toBe("ok");
|
||||
expect(calls).toBe(4);
|
||||
});
|
||||
|
||||
it("gives the outage back to the run's storage wait once the budget is spent", async () => {
|
||||
let calls = 0;
|
||||
await expect(retryThroughStorageOutage(async () => { calls++; return outage(); }, undefined, policy))
|
||||
.rejects.toMatchObject({ name: "ServiceUnavailable" });
|
||||
expect(calls).toBeGreaterThan(1);
|
||||
});
|
||||
|
||||
it("keeps the storage failure when a deadline ends the wait, but not when a Stop does", async () => {
|
||||
const slow = { budgetMs: 60_000, firstDelayMs: 1_000, maxDelayMs: 1_000 };
|
||||
await expect(retryThroughStorageOutage(outage, AbortSignal.timeout(50), slow))
|
||||
.rejects.toSatisfy(isStorageUnavailableError);
|
||||
const stop = new AbortController();
|
||||
setTimeout(() => stop.abort(new Error("Stopped by a board operator")), 50);
|
||||
const stopped = await retryThroughStorageOutage(outage, stop.signal, slow).catch((error: unknown) => error);
|
||||
expect(stopped).toMatchObject({ name: "AbortError" });
|
||||
expect(isStorageUnavailableError(stopped)).toBe(false);
|
||||
});
|
||||
|
||||
it("lets nested and parallel waits ride out one outage shorter than the budget", async () => {
|
||||
const wide = { budgetMs: 600, firstDelayMs: 20, maxDelayMs: 40 };
|
||||
const endsAt = Date.now() + 300;
|
||||
const send = async () => (Date.now() < endsAt ? outage() : "ok");
|
||||
const nested = () => retryThroughStorageOutage(() => retryThroughStorageOutage(send, undefined, wide), undefined, wide);
|
||||
await expect(waitOutStorageOutages(() => Promise.all(Array.from({ length: 8 }, nested))))
|
||||
.resolves.toEqual(Array(8).fill("ok"));
|
||||
});
|
||||
|
||||
it("gives up once one outage has lasted longer than the budget, however the waits overlap", async () => {
|
||||
const startedAt = Date.now();
|
||||
const failing = () => retryThroughStorageOutage(outage, undefined, policy);
|
||||
const results = await waitOutStorageOutages(() => Promise.allSettled(Array.from({ length: 4 }, failing)));
|
||||
expect(results.every(result => result.status === "rejected")).toBe(true);
|
||||
expect(Date.now() - startedAt).toBeGreaterThanOrEqual(policy.budgetMs - policy.maxDelayMs);
|
||||
expect(Date.now() - startedAt).toBeLessThan(policy.budgetMs * 3);
|
||||
});
|
||||
|
||||
it("keeps the limit when only some keys fail and other storage calls succeed", async () => {
|
||||
const startedAt = Date.now();
|
||||
const result = await waitOutStorageOutages(async () => {
|
||||
const reads = setInterval(() => void retryThroughStorageOutage(async () => "read", undefined, policy), 5);
|
||||
try {
|
||||
return await retryThroughStorageOutage(async () => {
|
||||
await retryThroughStorageOutage(async () => "head read", undefined, policy);
|
||||
return outage();
|
||||
}, undefined, policy).catch((error: unknown) => error);
|
||||
} finally { clearInterval(reads); }
|
||||
});
|
||||
expect(result).toMatchObject({ name: "ServiceUnavailable" });
|
||||
expect(Date.now() - startedAt).toBeLessThan(policy.budgetMs * 3);
|
||||
});
|
||||
|
||||
it("keeps one operation's limit when another operation in the same run recovers from its own outage", async () => {
|
||||
const startedAt = Date.now();
|
||||
const result = await waitOutStorageOutages(async () => {
|
||||
let recovering = true;
|
||||
void (async () => {
|
||||
while (recovering) {
|
||||
let failed = false;
|
||||
await retryThroughStorageOutage(async () => (failed ? "recovered" : (failed = true, outage())), undefined, policy).catch(() => undefined);
|
||||
}
|
||||
})();
|
||||
try {
|
||||
return await retryThroughStorageOutage(outage, undefined, policy).catch((error: unknown) => error);
|
||||
} finally { recovering = false; }
|
||||
});
|
||||
expect(result).toMatchObject({ name: "ServiceUnavailable" });
|
||||
expect(Date.now() - startedAt).toBeLessThan(policy.budgetMs * 3);
|
||||
});
|
||||
|
||||
it("gives a later outage its own limit once the waiting work recovered", async () => {
|
||||
await waitOutStorageOutages(async () => {
|
||||
const recoversAt = Date.now() + policy.budgetMs - 60;
|
||||
await expect(retryThroughStorageOutage(async () => (Date.now() < recoversAt ? outage() : "ok"), undefined, policy)).resolves.toBe("ok");
|
||||
const secondRecoversAt = Date.now() + policy.budgetMs - 60;
|
||||
await expect(retryThroughStorageOutage(async () => (Date.now() < secondRecoversAt ? outage() : "again"), undefined, policy)).resolves.toBe("again");
|
||||
});
|
||||
});
|
||||
|
||||
it("gives a run started from a stopped run's cleanup its own scope", async () => {
|
||||
const stopped = new AbortController();
|
||||
stopped.abort(new Error("Stopped by a board operator"));
|
||||
const wide = { budgetMs: 2_000, firstDelayMs: 20, maxDelayMs: 40 };
|
||||
let calls = 0;
|
||||
const next = await waitOutStorageOutages(async () => {
|
||||
endStorageOutageWaitOn(stopped.signal);
|
||||
return runInOwnStorageScope(() => retryThroughStorageOutage(async () => (++calls < 3 ? outage() : "next run"), undefined, wide));
|
||||
});
|
||||
expect(next).toBe("next run");
|
||||
});
|
||||
|
||||
it("lets one run's Stop end only its own wait, not shared work another run waits on", async () => {
|
||||
const endsAt = Date.now() + 150;
|
||||
const wide = { budgetMs: 2_000, firstDelayMs: 20, maxDelayMs: 40 };
|
||||
const stopA = new AbortController();
|
||||
let shared!: Promise<string>;
|
||||
const runA = waitOutStorageOutages(async () => {
|
||||
endStorageOutageWaitOn(stopA.signal);
|
||||
shared = runSharedStorageWork(() => retryThroughStorageOutage(async () => (Date.now() < endsAt ? outage() : "shared"), undefined, wide));
|
||||
return retryThroughStorageOutage(outage, undefined, wide);
|
||||
});
|
||||
setTimeout(() => stopA.abort(new Error("Stopped by a board operator")), 30);
|
||||
await expect(runA).rejects.toMatchObject({ name: "AbortError" });
|
||||
await expect(shared).resolves.toBe("shared");
|
||||
});
|
||||
|
||||
it("never retries a failure that is not an outage", async () => {
|
||||
let calls = 0;
|
||||
await expect(retryThroughStorageOutage(async () => { calls++; throw new Error("object_journal_conflict"); }, undefined, policy))
|
||||
.rejects.toThrow("object_journal_conflict");
|
||||
expect(calls).toBe(1);
|
||||
});
|
||||
|
||||
it("recognizes an outage a workspace restore wrapped (snapshot_download)", async () => {
|
||||
const cause = await outage().catch(error => error);
|
||||
expect(isStorageUnavailableError(new RemoteWorkspaceRestoreError("snapshot_download", true, cause))).toBe(true);
|
||||
expect(isStorageUnavailableError(new RemoteWorkspaceRestoreError("sandbox_restore", true, new Error("reason=occupied")))).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -607,7 +607,7 @@ describe.skipIf(!support.supported)("task interruption latest run query", () =>
|
||||
});
|
||||
// N-03: ONB-1 on staging waited 16 minutes on an overdue workspace_busy
|
||||
// retry. Nothing failed, yet the banner offered a Retry that cannot help.
|
||||
it.each(["workspace_busy", "ai_connection_busy", "ai_connection_pool_wait", "runner_revision_wait"])(
|
||||
it.each(["workspace_busy", "ai_connection_busy", "ai_connection_pool_wait", "runner_revision_wait", "storage_unavailable_wait"])(
|
||||
"does not present an overdue %s wait as a failed attempt", async reason => {
|
||||
const companyId = randomUUID(), agentId = randomUUID(), parentRunId = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name: "Waiting task", issuePrefix: `W${companyId.slice(0, 7)}` });
|
||||
|
||||
Reference in new issue
Block a user