mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(ui-assets): retry a throttled asset upload and never fail startup on a storage outage (#21010)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A hosted Paperclip server can run volume-free. Before it serves, it publishes its UI build assets to shared object storage, so an older browser tab can still load its chunks after the old container is gone > - A build has several hundred assets. The server uploads them one after another at startup > - Object storage throttles a burst with `SlowDown` (HTTP 503). The upload body is a stream, so the storage client cannot replay the request, and the error leaves `startServer` > - The process exits, the host restarts it, and the next start uploads the same assets again. A workspace stayed down while four starts in a row failed this way > - This pull request retries a throttled asset and lets the server start when storage stays unavailable > - The benefit is that a storage slowdown cannot keep a server from starting ## Linked Issues or Issue Description Refs #20990, #20952 (other startup failures found on the same hosted workspace). **What happened?** A hosted volume-free workspace received a new release. Each start logged many `An error was encountered in a non-retryable streaming request.` lines and then `Paperclip server failed to start` with `SlowDown: Please reduce your request rate.` for a key under `ui-assets/v1/`. The four starts failed after 44, 42, 79 and 90 seconds, each on a later asset name. **Expected behavior** The server waits and repeats a throttled upload. If shared storage stays unavailable, the server starts and serves its build from local files. **Steps to reproduce** 1. Run a server with `PAPERCLIP_SHARED_UI_ASSETS=true` and S3-compatible storage. 2. Make the storage answer `SlowDown` (HTTP 503) to one `PutObject` during startup. 3. The server exits with `Paperclip server failed to start`. **Paperclip version or commit** `52cdf76bf3e0cdf29037b63bf0f5e99aa464eab7` ## What Changed - `retainedUiAssets.publish`: each asset is published in a retry loop. A throttled or server-side storage failure waits (0.5 s to 8 s, seven waits at most) and repeats the asset with a fresh stream. - `isTransientStorageError`: recognizes `SlowDown`, throttling, request timeout and HTTP 429/500/502/503/504 answers. - `app.ts`: when publication still fails with a transient storage error, the server logs an error and continues. Other failures, such as a name collision with different bytes, are still fatal. - Tests for the retry, for the give-up path, and for the error classification. ## Verification - `npx vitest run src/__tests__/retained-ui-assets.test.ts` in `server/`: 4 tests passed. The two new tests fail on `master` because `publish` rejects on the first `SlowDown` and takes no `sleep` option. - Not run locally: the full suite. CI supplies that coverage. - Not verified: a live start against throttling storage. ## Risks - A start can take up to about 31 seconds longer for each throttled asset before it gives up. - If storage stays unavailable, the server starts without its assets in shared storage. An older browser tab can then fail to load a chunk after the old container is gone. The server logs this at error level. - Collision detection is unchanged. ## Model Used Claude Fable 5.2 (`claude-fable-5-2`), through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
1 parent
a9a08d8169
commit
82e0393598
3 files changed
+148
-16
No files matched your search
@@ -5,7 +5,7 @@ import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { Readable } from "node:stream";
|
||||
import { afterEach, expect, it } from "vitest";
|
||||
import { retainedUiAssets } from "../retained-ui-assets.js";
|
||||
import { isTransientStorageError, publishRetainedUiAssetsAtStartup, retainedUiAssets } from "../retained-ui-assets.js";
|
||||
import type { StorageProvider } from "../storage/types.js";
|
||||
import { staticUiFiles } from "../middleware/static-ui.js";
|
||||
import { idleAdmissionMiddleware, trackIdleRequestHandlers } from "../middleware/idle-admission.js";
|
||||
@@ -66,3 +66,79 @@ it("refuses a build which reuses a published name for different bytes", async ()
|
||||
await expect(retained.publish(root)).rejects.toThrow("collides");
|
||||
expect(objects.get("ui-assets/v1/chunk-hash.js")?.toString()).toBe("original");
|
||||
});
|
||||
|
||||
// Staging, 2026-11-11: the bucket answered "SlowDown" (HTTP 503) while a new
|
||||
// release published its assets. The stream body could not be replayed, the
|
||||
// error escaped, and every start of the server exited within a minute.
|
||||
it("retries an asset the storage throttled instead of failing the publication", async () => {
|
||||
const root = await directory(), { provider, objects } = storage();
|
||||
await fs.writeFile(path.join(root, "chunk-a.js"), "alpha");
|
||||
await fs.writeFile(path.join(root, "chunk-b.js"), "beta");
|
||||
const slowDown = () => Object.assign(new Error("Please reduce your request rate."), { name: "SlowDown", Code: "SlowDown", $metadata: { httpStatusCode: 503 } });
|
||||
const put = provider.putObjectConditional!.bind(provider);
|
||||
let refusals = 3;
|
||||
provider.putObjectConditional = async input => {
|
||||
if (refusals > 0) { refusals--; for await (const _chunk of input.body) { /* drain as a real client would */ } throw slowDown(); }
|
||||
return put(input);
|
||||
};
|
||||
const waits: number[] = [];
|
||||
await retainedUiAssets(provider, { sleep: async ms => { waits.push(ms); } }).publish(root);
|
||||
expect(waits).toEqual([500, 1_000, 2_000]);
|
||||
expect(objects.get("ui-assets/v1/chunk-a.js")?.toString()).toBe("alpha");
|
||||
expect(objects.get("ui-assets/v1/chunk-b.js")?.toString()).toBe("beta");
|
||||
});
|
||||
|
||||
it("gives up on storage that stays throttled, and reports it as a storage outage rather than a bad build", async () => {
|
||||
const root = await directory(), { provider } = storage();
|
||||
await fs.writeFile(path.join(root, "chunk-a.js"), "alpha");
|
||||
provider.putObjectConditional = async () => { throw Object.assign(new Error("Please reduce your request rate."), { name: "SlowDown", $metadata: { httpStatusCode: 503 } }); };
|
||||
const waits: number[] = [];
|
||||
const failure = await retainedUiAssets(provider, { sleep: async ms => { waits.push(ms); } }).publish(root).catch(error => error);
|
||||
expect(waits).toHaveLength(7);
|
||||
expect(isTransientStorageError(failure)).toBe(true);
|
||||
expect(isTransientStorageError(new Error("Static build asset name collides with different published bytes"))).toBe(false);
|
||||
});
|
||||
|
||||
it("starts on local files when storage stays unavailable, and still refuses a build that collides", async () => {
|
||||
const root = await directory(), { provider } = storage();
|
||||
await fs.writeFile(path.join(root, "chunk-a.js"), "alpha");
|
||||
const put = provider.putObjectConditional!.bind(provider);
|
||||
provider.putObjectConditional = async () => { throw Object.assign(new Error("socket hang up"), { cause: Object.assign(new Error("connect ECONNREFUSED"), { code: "ECONNREFUSED" }) }); };
|
||||
const retained = retainedUiAssets(provider, { sleep: async () => {} });
|
||||
const reported: unknown[] = [];
|
||||
await expect(publishRetainedUiAssetsAtStartup(retained, root, error => { reported.push(error); })).resolves.toBe(false);
|
||||
expect(reported).toHaveLength(1);
|
||||
// The server that started anyway serves the build from its own disk, and asks storage for nothing it lacks.
|
||||
const app = express();
|
||||
app.use("/assets", staticUiFiles(root), retained.serve);
|
||||
await request(app).get("/assets/chunk-a.js").expect(200).expect("alpha");
|
||||
await request(app).get("/assets/never-built.js").expect(404);
|
||||
// Once storage answers, a published name with different bytes is a bad build, not an outage.
|
||||
provider.putObjectConditional = put;
|
||||
await expect(publishRetainedUiAssetsAtStartup(retained, root, () => {})).resolves.toBe(true);
|
||||
await fs.writeFile(path.join(root, "chunk-a.js"), "collision");
|
||||
await expect(publishRetainedUiAssetsAtStartup(retained, root, error => { reported.push(error); })).rejects.toThrow("collides");
|
||||
expect(reported).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("retries an existing asset whose saved bytes storage could not serve, without accepting it unverified", async () => {
|
||||
const root = await directory(), { provider } = storage();
|
||||
await fs.writeFile(path.join(root, "chunk-a.js"), "alpha");
|
||||
const waits: number[] = [];
|
||||
const retained = retainedUiAssets(provider, { sleep: async ms => { waits.push(ms); } });
|
||||
await retained.publish(root);
|
||||
// A restart: the conditional write answers 412 and the verifying read is throttled twice.
|
||||
const get = provider.getObject.bind(provider);
|
||||
let refusals = 2, reads = 0;
|
||||
provider.getObject = async input => {
|
||||
reads++;
|
||||
if (refusals > 0) { refusals--; throw Object.assign(new Error("Please reduce your request rate."), { name: "SlowDown", $metadata: { httpStatusCode: 503 } }); }
|
||||
return get(input);
|
||||
};
|
||||
await retained.publish(root);
|
||||
expect(waits).toEqual([500, 1_000]);
|
||||
expect(reads).toBe(3);
|
||||
// A deadline on the asset is an outage too; a read that says the object is gone is not.
|
||||
expect(isTransientStorageError(Object.assign(new Error("The operation was aborted"), { name: "AbortError" }))).toBe(true);
|
||||
expect(isTransientStorageError(Object.assign(new Error("Object missing"), { status: 404 }))).toBe(false);
|
||||
});
|
||||
+3
-2
@@ -21,7 +21,7 @@ import { startAgentLifecycle } from "./services/agent-lifecycle.js";
|
||||
import { idleAdmissionMiddleware, trackIdleRequestHandlers } from "./middleware/idle-admission.js";
|
||||
import { databaseOperationMiddleware } from "./middleware/database-operation.js";
|
||||
import { staticUiFiles } from "./middleware/static-ui.js";
|
||||
import { retainedUiAssets } from "./retained-ui-assets.js";
|
||||
import { publishRetainedUiAssetsAtStartup, retainedUiAssets } from "./retained-ui-assets.js";
|
||||
import { createStorageProviderFromConfig } from "./storage/provider-registry.js";
|
||||
import { loadStorageConfig } from "./storage/config.js";
|
||||
import { isIdleTaskDrainActive, trackIdleWork } from "./services/task-admission.js";
|
||||
@@ -1107,7 +1107,8 @@ export async function createApp(
|
||||
);
|
||||
if (process.env.PAPERCLIP_SHARED_UI_ASSETS === "true") {
|
||||
const retained = retainedUiAssets(createStorageProviderFromConfig(loadStorageConfig()));
|
||||
await retained.publish(path.join(uiDist, "assets"));
|
||||
await publishRetainedUiAssetsAtStartup(retained, path.join(uiDist, "assets"), error =>
|
||||
logger.error({ err: error }, "Shared UI assets could not be published because storage is unavailable; serving this build from local files"));
|
||||
staticUi.use("/assets", retained.serve);
|
||||
}
|
||||
// Serve root/index through the same runtime HTML transform as SPA routes.
|
||||
|
||||
@@ -19,11 +19,52 @@ async function digest(stream: Readable) {
|
||||
return hash.digest("hex");
|
||||
}
|
||||
|
||||
const transientStatuses = new Set([429, 500, 502, 503, 504]);
|
||||
const transientNames = new Set(["SlowDown", "Throttling", "ThrottlingException", "RequestTimeout", "ServiceUnavailable", "InternalError",
|
||||
// The 30-second deadline on one asset, and a connection that never completed.
|
||||
"TimeoutError", "AbortError"]);
|
||||
const transientNetworkCodes = new Set(["ECONNREFUSED", "ECONNRESET", "ECONNABORTED", "ETIMEDOUT", "EPIPE", "ENOTFOUND", "EAI_AGAIN", "ENETUNREACH", "EHOSTUNREACH",
|
||||
"UND_ERR_CONNECT_TIMEOUT", "UND_ERR_SOCKET", "UND_ERR_HEADERS_TIMEOUT", "UND_ERR_BODY_TIMEOUT"]);
|
||||
const publishRetryDelaysMs = [500, 1_000, 2_000, 4_000, 8_000, 8_000, 8_000];
|
||||
|
||||
/** Storage asked us to slow down or failed on its side; the same request can succeed later. */
|
||||
export function isTransientStorageError(error: unknown): boolean {
|
||||
// The storage client wraps a socket failure; the reason is on the cause.
|
||||
for (let current: unknown = error, depth = 0; current && typeof current === "object" && depth < 4; depth++) {
|
||||
const value = current as { name?: unknown; Code?: unknown; code?: unknown; status?: unknown; cause?: unknown; $metadata?: { httpStatusCode?: unknown } };
|
||||
if ([value.name, value.Code].some(name => typeof name === "string" && transientNames.has(name))
|
||||
|| (typeof value.code === "string" && transientNetworkCodes.has(value.code))
|
||||
|| [value.status, value.$metadata?.httpStatusCode].some(status => typeof status === "number" && transientStatuses.has(status))) return true;
|
||||
current = value.cause;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/** Publish at startup. Returns false, after saying so, when storage itself is
|
||||
* unavailable: this container still serves its build from disk, so a storage
|
||||
* outage costs an older tab its retained chunks, not everyone the server. A
|
||||
* build that cannot be published for any other reason still rejects. */
|
||||
export async function publishRetainedUiAssetsAtStartup(
|
||||
retained: { publish(directory: string): Promise<void> },
|
||||
directory: string,
|
||||
report: (error: unknown) => void,
|
||||
): Promise<boolean> {
|
||||
try {
|
||||
await retained.publish(directory);
|
||||
return true;
|
||||
} catch (error) {
|
||||
if (!isTransientStorageError(error)) throw error;
|
||||
report(error);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/** Public, immutable build assets only. Publishing before readiness lets an
|
||||
* older tab fetch its chunks after the last old container has disappeared.
|
||||
* Objects have no automatic expiry: retain them at least 30 days after the
|
||||
* last serving deployment, including rollback, before operator reclamation. */
|
||||
export function retainedUiAssets(provider: StorageProvider): { publish(directory: string): Promise<void>; serve: RequestHandler } {
|
||||
export function retainedUiAssets(provider: StorageProvider, options: { sleep?: (ms: number) => Promise<void> } = {}): { publish(directory: string): Promise<void>; serve: RequestHandler } {
|
||||
const sleep = options.sleep ?? ((ms: number) => new Promise<void>(resolve => setTimeout(resolve, ms)));
|
||||
if (provider.id !== "s3" || !provider.putObjectConditional) throw new Error("Retained UI assets require conditional shared object storage");
|
||||
async function publish(directory: string) {
|
||||
async function visit(relativeDirectory: string) {
|
||||
@@ -33,19 +74,33 @@ export function retainedUiAssets(provider: StorageProvider): { publish(directory
|
||||
if (relative.endsWith(".map")) continue;
|
||||
if (!entry.isFile() || !assetPath(relative)) throw new Error("Unsupported static build asset");
|
||||
const filename = path.join(directory, relative);
|
||||
const signal = AbortSignal.timeout(30_000);
|
||||
const contentLength = (await fs.stat(filename)).size;
|
||||
const expected = await digest(createReadStream(filename, { signal }));
|
||||
const objectKey = `${prefix}${relative}`;
|
||||
const body = createReadStream(filename, { signal });
|
||||
try {
|
||||
await provider.putObjectConditional!({ objectKey, body, contentLength, contentType: "application/octet-stream", ifNoneMatch: "*", signal });
|
||||
} catch (error) {
|
||||
// An existing key or uncertain upload may be accepted only if its
|
||||
// complete bytes match. Never overwrite a colliding build asset.
|
||||
const existing = await provider.getObject({ objectKey, signal }).catch(() => { throw error; });
|
||||
if (await digest(existing.stream) !== expected) throw new Error("Static build asset name collides with different published bytes");
|
||||
} finally { body.destroy(); }
|
||||
// A build has hundreds of assets and storage throttles a burst. The
|
||||
// upload body is a stream, so the client cannot replay it: retry the
|
||||
// whole asset here, with a fresh stream, instead of failing startup.
|
||||
for (let attempt = 0; ; attempt++) {
|
||||
const signal = AbortSignal.timeout(30_000);
|
||||
const contentLength = (await fs.stat(filename)).size;
|
||||
const expected = await digest(createReadStream(filename, { signal }));
|
||||
const body = createReadStream(filename, { signal });
|
||||
try {
|
||||
try {
|
||||
await provider.putObjectConditional!({ objectKey, body, contentLength, contentType: "application/octet-stream", ifNoneMatch: "*", signal });
|
||||
} catch (error) {
|
||||
// An existing key or uncertain upload may be accepted only if its
|
||||
// complete bytes match. Never overwrite a colliding build asset.
|
||||
// A read that storage could not serve proves nothing either way:
|
||||
// keep it, so the asset is tried again instead of being judged.
|
||||
const existing = await provider.getObject({ objectKey, signal })
|
||||
.catch(readError => { throw isTransientStorageError(readError) ? readError : error; });
|
||||
if (await digest(existing.stream) !== expected) throw new Error("Static build asset name collides with different published bytes");
|
||||
}
|
||||
break;
|
||||
} catch (error) {
|
||||
if (!isTransientStorageError(error) || attempt >= publishRetryDelaysMs.length) throw error;
|
||||
await sleep(publishRetryDelaysMs[attempt]!);
|
||||
} finally { body.destroy(); }
|
||||
}
|
||||
}
|
||||
}
|
||||
await visit("");
|
||||
|
||||
Reference in new issue
Block a user