fix(ui-assets): retry a throttled asset upload and never fail startup on a storage outage (#21010)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A hosted Paperclip server can run volume-free. Before it serves, it
publishes its UI build assets to shared object storage, so an older
browser tab can still load its chunks after the old container is gone
> - A build has several hundred assets. The server uploads them one
after another at startup
> - Object storage throttles a burst with `SlowDown` (HTTP 503). The
upload body is a stream, so the storage client cannot replay the
request, and the error leaves `startServer`
> - The process exits, the host restarts it, and the next start uploads
the same assets again. A workspace stayed down while four starts in a
row failed this way
> - This pull request retries a throttled asset and lets the server
start when storage stays unavailable
> - The benefit is that a storage slowdown cannot keep a server from
starting

## Linked Issues or Issue Description

Refs #20990, #20952 (other startup failures found on the same hosted
workspace).

**What happened?**

A hosted volume-free workspace received a new release. Each start logged
many `An error was encountered in a non-retryable streaming request.`
lines and then `Paperclip server failed to start` with `SlowDown: Please
reduce your request rate.` for a key under `ui-assets/v1/`. The four
starts failed after 44, 42, 79 and 90 seconds, each on a later asset
name.

**Expected behavior**

The server waits and repeats a throttled upload. If shared storage stays
unavailable, the server starts and serves its build from local files.

**Steps to reproduce**

1. Run a server with `PAPERCLIP_SHARED_UI_ASSETS=true` and S3-compatible
storage.
2. Make the storage answer `SlowDown` (HTTP 503) to one `PutObject`
during startup.
3. The server exits with `Paperclip server failed to start`.

**Paperclip version or commit**

`52cdf76bf3e0cdf29037b63bf0f5e99aa464eab7`

## What Changed

- `retainedUiAssets.publish`: each asset is published in a retry loop. A
throttled or server-side storage failure waits (0.5 s to 8 s, seven
waits at most) and repeats the asset with a fresh stream.
- `isTransientStorageError`: recognizes `SlowDown`, throttling, request
timeout and HTTP 429/500/502/503/504 answers.
- `app.ts`: when publication still fails with a transient storage error,
the server logs an error and continues. Other failures, such as a name
collision with different bytes, are still fatal.
- Tests for the retry, for the give-up path, and for the error
classification.

## Verification

- `npx vitest run src/__tests__/retained-ui-assets.test.ts` in
`server/`: 4 tests passed. The two new tests fail on `master` because
`publish` rejects on the first `SlowDown` and takes no `sleep` option.
- Not run locally: the full suite. CI supplies that coverage.
- Not verified: a live start against throttling storage.

## Risks

- A start can take up to about 31 seconds longer for each throttled
asset before it gives up.
- If storage stays unavailable, the server starts without its assets in
shared storage. An older browser tab can then fail to load a chunk after
the old container is gone. The server logs this at error level.
- Collision detection is unchanged.

## Model Used

Claude Fable 5.2 (`claude-fable-5-2`), through Claude Code with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Fable 5.2 authored and GitHub committed 2026-11-11 21:21:27 -06:00
1 parent a9a08d8169
commit 82e0393598
3 files changed
+148 -16

No files matched your search

@@ -5,7 +5,7 @@ import os from "node:os";
import path from "node:path";
import { Readable } from "node:stream";
import { afterEach, expect, it } from "vitest";
import { retainedUiAssets } from "../retained-ui-assets.js";
import { isTransientStorageError, publishRetainedUiAssetsAtStartup, retainedUiAssets } from "../retained-ui-assets.js";
import type { StorageProvider } from "../storage/types.js";
import { staticUiFiles } from "../middleware/static-ui.js";
import { idleAdmissionMiddleware, trackIdleRequestHandlers } from "../middleware/idle-admission.js";
@@ -66,3 +66,79 @@ it("refuses a build which reuses a published name for different bytes", async ()
await expect(retained.publish(root)).rejects.toThrow("collides");
expect(objects.get("ui-assets/v1/chunk-hash.js")?.toString()).toBe("original");
});
// Staging, 2026-11-11: the bucket answered "SlowDown" (HTTP 503) while a new
// release published its assets. The stream body could not be replayed, the
// error escaped, and every start of the server exited within a minute.
it("retries an asset the storage throttled instead of failing the publication", async () => {
const root = await directory(), { provider, objects } = storage();
await fs.writeFile(path.join(root, "chunk-a.js"), "alpha");
await fs.writeFile(path.join(root, "chunk-b.js"), "beta");
const slowDown = () => Object.assign(new Error("Please reduce your request rate."), { name: "SlowDown", Code: "SlowDown", $metadata: { httpStatusCode: 503 } });
const put = provider.putObjectConditional!.bind(provider);
let refusals = 3;
provider.putObjectConditional = async input => {
if (refusals > 0) { refusals--; for await (const _chunk of input.body) { /* drain as a real client would */ } throw slowDown(); }
return put(input);
};
const waits: number[] = [];
await retainedUiAssets(provider, { sleep: async ms => { waits.push(ms); } }).publish(root);
expect(waits).toEqual([500, 1_000, 2_000]);
expect(objects.get("ui-assets/v1/chunk-a.js")?.toString()).toBe("alpha");
expect(objects.get("ui-assets/v1/chunk-b.js")?.toString()).toBe("beta");
});
it("gives up on storage that stays throttled, and reports it as a storage outage rather than a bad build", async () => {
const root = await directory(), { provider } = storage();
await fs.writeFile(path.join(root, "chunk-a.js"), "alpha");
provider.putObjectConditional = async () => { throw Object.assign(new Error("Please reduce your request rate."), { name: "SlowDown", $metadata: { httpStatusCode: 503 } }); };
const waits: number[] = [];
const failure = await retainedUiAssets(provider, { sleep: async ms => { waits.push(ms); } }).publish(root).catch(error => error);
expect(waits).toHaveLength(7);
expect(isTransientStorageError(failure)).toBe(true);
expect(isTransientStorageError(new Error("Static build asset name collides with different published bytes"))).toBe(false);
});
it("starts on local files when storage stays unavailable, and still refuses a build that collides", async () => {
const root = await directory(), { provider } = storage();
await fs.writeFile(path.join(root, "chunk-a.js"), "alpha");
const put = provider.putObjectConditional!.bind(provider);
provider.putObjectConditional = async () => { throw Object.assign(new Error("socket hang up"), { cause: Object.assign(new Error("connect ECONNREFUSED"), { code: "ECONNREFUSED" }) }); };
const retained = retainedUiAssets(provider, { sleep: async () => {} });
const reported: unknown[] = [];
await expect(publishRetainedUiAssetsAtStartup(retained, root, error => { reported.push(error); })).resolves.toBe(false);
expect(reported).toHaveLength(1);
// The server that started anyway serves the build from its own disk, and asks storage for nothing it lacks.
const app = express();
app.use("/assets", staticUiFiles(root), retained.serve);
await request(app).get("/assets/chunk-a.js").expect(200).expect("alpha");
await request(app).get("/assets/never-built.js").expect(404);
// Once storage answers, a published name with different bytes is a bad build, not an outage.
provider.putObjectConditional = put;
await expect(publishRetainedUiAssetsAtStartup(retained, root, () => {})).resolves.toBe(true);
await fs.writeFile(path.join(root, "chunk-a.js"), "collision");
await expect(publishRetainedUiAssetsAtStartup(retained, root, error => { reported.push(error); })).rejects.toThrow("collides");
expect(reported).toHaveLength(1);
});
it("retries an existing asset whose saved bytes storage could not serve, without accepting it unverified", async () => {
const root = await directory(), { provider } = storage();
await fs.writeFile(path.join(root, "chunk-a.js"), "alpha");
const waits: number[] = [];
const retained = retainedUiAssets(provider, { sleep: async ms => { waits.push(ms); } });
await retained.publish(root);
// A restart: the conditional write answers 412 and the verifying read is throttled twice.
const get = provider.getObject.bind(provider);
let refusals = 2, reads = 0;
provider.getObject = async input => {
reads++;
if (refusals > 0) { refusals--; throw Object.assign(new Error("Please reduce your request rate."), { name: "SlowDown", $metadata: { httpStatusCode: 503 } }); }
return get(input);
};
await retained.publish(root);
expect(waits).toEqual([500, 1_000]);
expect(reads).toBe(3);
// A deadline on the asset is an outage too; a read that says the object is gone is not.
expect(isTransientStorageError(Object.assign(new Error("The operation was aborted"), { name: "AbortError" }))).toBe(true);
expect(isTransientStorageError(Object.assign(new Error("Object missing"), { status: 404 }))).toBe(false);
});
+3 -2
View File
@@ -21,7 +21,7 @@ import { startAgentLifecycle } from "./services/agent-lifecycle.js";
import { idleAdmissionMiddleware, trackIdleRequestHandlers } from "./middleware/idle-admission.js";
import { databaseOperationMiddleware } from "./middleware/database-operation.js";
import { staticUiFiles } from "./middleware/static-ui.js";
import { retainedUiAssets } from "./retained-ui-assets.js";
import { publishRetainedUiAssetsAtStartup, retainedUiAssets } from "./retained-ui-assets.js";
import { createStorageProviderFromConfig } from "./storage/provider-registry.js";
import { loadStorageConfig } from "./storage/config.js";
import { isIdleTaskDrainActive, trackIdleWork } from "./services/task-admission.js";
@@ -1107,7 +1107,8 @@ export async function createApp(
);
if (process.env.PAPERCLIP_SHARED_UI_ASSETS === "true") {
const retained = retainedUiAssets(createStorageProviderFromConfig(loadStorageConfig()));
await retained.publish(path.join(uiDist, "assets"));
await publishRetainedUiAssetsAtStartup(retained, path.join(uiDist, "assets"), error =>
logger.error({ err: error }, "Shared UI assets could not be published because storage is unavailable; serving this build from local files"));
staticUi.use("/assets", retained.serve);
}
// Serve root/index through the same runtime HTML transform as SPA routes.
+68 -13
View File
@@ -19,11 +19,52 @@ async function digest(stream: Readable) {
return hash.digest("hex");
}
const transientStatuses = new Set([429, 500, 502, 503, 504]);
const transientNames = new Set(["SlowDown", "Throttling", "ThrottlingException", "RequestTimeout", "ServiceUnavailable", "InternalError",
// The 30-second deadline on one asset, and a connection that never completed.
"TimeoutError", "AbortError"]);
const transientNetworkCodes = new Set(["ECONNREFUSED", "ECONNRESET", "ECONNABORTED", "ETIMEDOUT", "EPIPE", "ENOTFOUND", "EAI_AGAIN", "ENETUNREACH", "EHOSTUNREACH",
"UND_ERR_CONNECT_TIMEOUT", "UND_ERR_SOCKET", "UND_ERR_HEADERS_TIMEOUT", "UND_ERR_BODY_TIMEOUT"]);
const publishRetryDelaysMs = [500, 1_000, 2_000, 4_000, 8_000, 8_000, 8_000];
/** Storage asked us to slow down or failed on its side; the same request can succeed later. */
export function isTransientStorageError(error: unknown): boolean {
// The storage client wraps a socket failure; the reason is on the cause.
for (let current: unknown = error, depth = 0; current && typeof current === "object" && depth < 4; depth++) {
const value = current as { name?: unknown; Code?: unknown; code?: unknown; status?: unknown; cause?: unknown; $metadata?: { httpStatusCode?: unknown } };
if ([value.name, value.Code].some(name => typeof name === "string" && transientNames.has(name))
|| (typeof value.code === "string" && transientNetworkCodes.has(value.code))
|| [value.status, value.$metadata?.httpStatusCode].some(status => typeof status === "number" && transientStatuses.has(status))) return true;
current = value.cause;
}
return false;
}
/** Publish at startup. Returns false, after saying so, when storage itself is
* unavailable: this container still serves its build from disk, so a storage
* outage costs an older tab its retained chunks, not everyone the server. A
* build that cannot be published for any other reason still rejects. */
export async function publishRetainedUiAssetsAtStartup(
retained: { publish(directory: string): Promise<void> },
directory: string,
report: (error: unknown) => void,
): Promise<boolean> {
try {
await retained.publish(directory);
return true;
} catch (error) {
if (!isTransientStorageError(error)) throw error;
report(error);
return false;
}
}
/** Public, immutable build assets only. Publishing before readiness lets an
* older tab fetch its chunks after the last old container has disappeared.
* Objects have no automatic expiry: retain them at least 30 days after the
* last serving deployment, including rollback, before operator reclamation. */
export function retainedUiAssets(provider: StorageProvider): { publish(directory: string): Promise<void>; serve: RequestHandler } {
export function retainedUiAssets(provider: StorageProvider, options: { sleep?: (ms: number) => Promise<void> } = {}): { publish(directory: string): Promise<void>; serve: RequestHandler } {
const sleep = options.sleep ?? ((ms: number) => new Promise<void>(resolve => setTimeout(resolve, ms)));
if (provider.id !== "s3" || !provider.putObjectConditional) throw new Error("Retained UI assets require conditional shared object storage");
async function publish(directory: string) {
async function visit(relativeDirectory: string) {
@@ -33,19 +74,33 @@ export function retainedUiAssets(provider: StorageProvider): { publish(directory
if (relative.endsWith(".map")) continue;
if (!entry.isFile() || !assetPath(relative)) throw new Error("Unsupported static build asset");
const filename = path.join(directory, relative);
const signal = AbortSignal.timeout(30_000);
const contentLength = (await fs.stat(filename)).size;
const expected = await digest(createReadStream(filename, { signal }));
const objectKey = `${prefix}${relative}`;
const body = createReadStream(filename, { signal });
try {
await provider.putObjectConditional!({ objectKey, body, contentLength, contentType: "application/octet-stream", ifNoneMatch: "*", signal });
} catch (error) {
// An existing key or uncertain upload may be accepted only if its
// complete bytes match. Never overwrite a colliding build asset.
const existing = await provider.getObject({ objectKey, signal }).catch(() => { throw error; });
if (await digest(existing.stream) !== expected) throw new Error("Static build asset name collides with different published bytes");
} finally { body.destroy(); }
// A build has hundreds of assets and storage throttles a burst. The
// upload body is a stream, so the client cannot replay it: retry the
// whole asset here, with a fresh stream, instead of failing startup.
for (let attempt = 0; ; attempt++) {
const signal = AbortSignal.timeout(30_000);
const contentLength = (await fs.stat(filename)).size;
const expected = await digest(createReadStream(filename, { signal }));
const body = createReadStream(filename, { signal });
try {
try {
await provider.putObjectConditional!({ objectKey, body, contentLength, contentType: "application/octet-stream", ifNoneMatch: "*", signal });
} catch (error) {
// An existing key or uncertain upload may be accepted only if its
// complete bytes match. Never overwrite a colliding build asset.
// A read that storage could not serve proves nothing either way:
// keep it, so the asset is tried again instead of being judged.
const existing = await provider.getObject({ objectKey, signal })
.catch(readError => { throw isTransientStorageError(readError) ? readError : error; });
if (await digest(existing.stream) !== expected) throw new Error("Static build asset name collides with different published bytes");
}
break;
} catch (error) {
if (!isTransientStorageError(error) || attempt >= publishRetryDelaysMs.length) throw error;
await sleep(publishRetryDelaysMs[attempt]!);
} finally { body.destroy(); }
}
}
}
await visit("");