mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
perf(server): size run-log uploads and object journal IO from open run logs (#21685)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process controls every run of a workspace, and the target
is 1,000 concurrent agents on one server
> - Each native run writes its run log to object storage. The server
batches a run's log lines and uploads one batch per run each second
> - The server allowed only 4 log uploads and 8 object storage
operations at a time, for the whole process. These numbers were set when
50 agents was a lot
> - At 200 runs, the run logs fell about 8 seconds behind. At 1,000
runs, pending log bytes grow toward the 64 MiB process cap. Past that
cap, run events wait and log appends can fail
> - This pull request sizes the log upload slots and the object storage
operation slots from the number of open run logs, between the old floor
and a ceiling that comes from the storage client's socket budget
> - The benefit is that run logs stay about one flush interval behind at
200 runs, and fall 7.7 seconds behind instead of 44 seconds at 1,000
runs (p50), without more sockets than the storage client allows
## Linked Issues or Issue Description
Refs plan item 1g in
`doc/plans/2026-11-13-thousand-concurrent-agents.md` ("Size the shared
limiters ... from the active run count").
**What existing behavior does this improve?**
The shared run-log upload limiter and the shared object journal
operation limiter.
**Subsystem affected**
Server: object storage run logs (`object-run-log-store.ts`,
`object-byte-journal.ts`) and the S3 storage client.
**Current behavior**
The process allows 4 run-log uploads and 8 object journal operations at
a time, for any number of runs. Each upload is 3 to 4 sequential object
writes. At 200 runs and 50 ms per write, the run logs commit with a p50
delay of 8.0 s, and only 55% of the log lines offered in a 20 s window
commit inside it.
**Proposed behavior**
The upload slots are one per 10 open run logs, at least 4 and at most
21. The operation slots are the upload slots plus 4 for reads, at most
25. That is half of the 50 sockets that the S3 client may open, so
downloads and other storage calls keep the other half. The S3 client now
sets its 50-socket limit explicitly, from the same constant.
**Reason and benefit**
With the change, 200 runs commit their logs at the flush interval (p50
1.1 s), and 1,000 runs commit with a p50 delay of 7.7 s instead of 43.8
s. Fewer pending bytes stay in memory, so the 64 MiB pending cap and its
admission failure are farther away.
**Breaking changes**
None. With 40 or fewer open run logs the limits are the same as before
(4 uploads, 8 operations).
## What Changed
- `server/src/lib/active-run-slots.ts`: a small helper that gives a slot
count from an active run count, clamped between a floor and a ceiling.
- `server/src/storage/s3-client-limits.ts` and `s3-provider.ts`: the S3
client sets `maxSockets` to 50 (the SDK default) from a named constant,
so the budget is explicit.
- `object-byte-journal.ts`: the object operation limit is the upload
slots plus a read reserve of 4, from the number of open run-log writers.
Open writers are counted with `trackObjectJournalWriter()`. When the
limit grows, waiting operations start at once. When it shrinks, running
operations finish and no new ones start until the count is below the
limit.
- `object-byte-journal.ts` also owns the upload limiter
(`acquireObjectJournalUpload`), process-wide, next to the writer count.
When a run log opens, waiting uploads start at once. When run logs
close, running uploads finish and no new upload starts until the running
count is below the new limit.
- `object-run-log-store.ts`: uses the shared upload limiter. A run log
registers when it begins and releases when it is finalized. The waiting
queue stays first in, first out, and each run has at most one upload in
flight, so one run cannot starve another.
- Review fix: the first revision kept the upload limiter per store, so a
run log opened by another store did not wake uploads waiting in this one
(a lost wakeup on growth). The shared limiter fixes it; a new test
covers growth and shrinkage with queued work.
- Tests: limit growth with uploads waiting and shrinkage with uploads
running; helper bounds; the storage socket budget; 200 open runs upload
at least 20 logs at once (4 before); 1,000 open runs stay at the ceiling
and every upload finishes.
## Verification
- `cd server && npx vitest run src/lib/active-run-slots.test.ts
src/services/object-run-log-store.test.ts
src/services/object-byte-journal.test.ts`: all pass.
- Red/green: with the slot count forced back to the fixed floor of 4,
the two new scale tests fail (4 uploads in flight); with the change they
pass.
- Simulation (in-memory object store with 50 ms per write, each run
appends 1.5 KB at 0.7 records/s, the recorded native event rate, 20 s
window, flush every 1 s). Counts are stable on a loaded machine because
the latency is simulated:
| Runs | Build | Peak concurrent writes | Writes/s | Log lines committed
in window / offered | Commit delay p50 / p99 |
| --- | --- | --- | --- | --- | --- |
| 200 | before | 4 | 74 | 1,473 / 2,669 | 8.0 s / 13.5 s |
| 200 | after | 20 | 211 | 2,646 / 2,790 | 1.1 s / 1.3 s |
| 1,000 | before | 4 | 75 | 1,229 / 13,499 | 43.8 s / 62.3 s |
| 1,000 | after | 21 | 392 | 7,789 / 13,667 | 7.7 s / 12.6 s |
## Risks
- More object storage writes per second at high run counts (at 1,000
runs and 50 ms per write, about 390/s instead of about 75/s). This
raises object storage request cost in proportion. The ceiling bounds it.
- The S3 client now states its socket limit. The value is the SDK
default, so behavior does not change.
- The open-writer count is process-wide. A run log that is never
finalized keeps its count, as it already keeps its writer entry. The
ceiling bounds the effect.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
fad8f051ca
commit
3eabdd6d10
8 files changed
+277
-35
No files matched your search
@@ -1,7 +1,7 @@
|
||||
# Plan: 1,000 concurrent agents on one workspace
|
||||
|
||||
Date: 2026-11-13. Status: Phase 1 changes 1a to 1e and 1h are merged; 1f, 1g
|
||||
and Phase 2 are not started.
|
||||
Date: 2026-11-13. Status: Phase 1 changes 1a to 1e and 1h are merged; 1g is
|
||||
done for run-log uploads (see "Change 1g"); 1f and Phase 2 are not started.
|
||||
|
||||
## Finish line
|
||||
|
||||
@@ -211,6 +211,39 @@ What it shows:
|
||||
than its statement count suggests; the next changes should be judged by CPU
|
||||
per operation, not only by statements.
|
||||
|
||||
## Change 1g: shared limiters at 200 and 1,000 runs
|
||||
|
||||
Run-log uploads now have one slot per 10 open run logs (at least 4, at most
|
||||
21), and object journal operations have the upload slots plus 4 for reads (at
|
||||
most 25, half of the S3 client's 50 sockets). Simulation with 50 ms per
|
||||
object write, 0.7 log records per run per second, 20 s window:
|
||||
|
||||
| Runs | Build | Concurrent writes | Log lines committed in window / offered | Commit delay p50 / p99 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| 200 | before (4 slots) | 4 | 1,473 / 2,669 | 8.0 s / 13.5 s |
|
||||
| 200 | after | 20 | 2,646 / 2,790 | 1.1 s / 1.3 s |
|
||||
| 1,000 | before (4 slots) | 4 | 1,229 / 13,499 | 43.8 s / 62.3 s |
|
||||
| 1,000 | after | 21 | 7,789 / 13,667 | 7.7 s / 12.6 s |
|
||||
|
||||
Not changed, and why:
|
||||
|
||||
- Controller checkpoint pool (3 connections). Since #21553 a routine save is
|
||||
one pipelined statement, so the pool is limited by backend execution and
|
||||
commit time, not round trips. At 1 to 3 ms per save on a hosted database it
|
||||
allows about 1,000–3,000 saves/s. Demand at 1,000 runs is about 380/s
|
||||
(load model) to 1,400/s (recorded checkpoint rows). A larger pool needs a
|
||||
per-process connection budget for the hosted database, which is an owner
|
||||
decision.
|
||||
- `withDedicatedDbConnection` has no bound on connections in use (it keeps at
|
||||
most 4 idle). Workspace source and lifecycle locks nest two of them, so a
|
||||
bound must not let outer holders block inner ones. Steady state at 1,000
|
||||
runs is about 10–20 connections; a burst of run starts after a restart can
|
||||
open up to two per starting run.
|
||||
- Lease renewal (2 connections, 200 rows per statement), agent-file saves (2
|
||||
connections, a short read-only check when nothing changed), run dispatch
|
||||
(24 claims per agent pass, chained at once) and recovery per-pass caps do
|
||||
not limit a healthy run at 1,000 runs.
|
||||
|
||||
## Considered and not done
|
||||
|
||||
- **Stop writing `run.performance.span` rows.** These spans are the largest
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { assertActiveRunSlotBounds, slotsForActiveRuns } from "./active-run-slots.js";
|
||||
|
||||
describe("slotsForActiveRuns", () => {
|
||||
const bounds = assertActiveRunSlotBounds({ runsPerSlot: 10, floor: 4, ceiling: 21 });
|
||||
|
||||
it("keeps the floor while few runs are active", () => {
|
||||
expect(slotsForActiveRuns(0, bounds)).toBe(4);
|
||||
expect(slotsForActiveRuns(40, bounds)).toBe(4);
|
||||
});
|
||||
|
||||
it("adds a slot for each started group of active runs", () => {
|
||||
expect(slotsForActiveRuns(41, bounds)).toBe(5);
|
||||
expect(slotsForActiveRuns(200, bounds)).toBe(20);
|
||||
});
|
||||
|
||||
it("never passes the ceiling", () => {
|
||||
expect(slotsForActiveRuns(201, bounds)).toBe(21);
|
||||
expect(slotsForActiveRuns(1_000, bounds)).toBe(21);
|
||||
expect(slotsForActiveRuns(Number.MAX_SAFE_INTEGER, bounds)).toBe(21);
|
||||
});
|
||||
|
||||
it("treats an invalid run count as no active runs", () => {
|
||||
expect(slotsForActiveRuns(-5, bounds)).toBe(4);
|
||||
expect(slotsForActiveRuns(Number.NaN, bounds)).toBe(4);
|
||||
});
|
||||
|
||||
it("rejects bounds whose ceiling is below the floor", () => {
|
||||
expect(() => assertActiveRunSlotBounds({ runsPerSlot: 10, floor: 5, ceiling: 4 })).toThrow("invalid_active_run_slot_bounds");
|
||||
expect(() => assertActiveRunSlotBounds({ runsPerSlot: 0, floor: 1, ceiling: 4 })).toThrow("invalid_active_run_slot_bounds");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,20 @@
|
||||
export interface ActiveRunSlotBounds {
|
||||
runsPerSlot: number;
|
||||
floor: number;
|
||||
ceiling: number;
|
||||
}
|
||||
|
||||
export function assertActiveRunSlotBounds(bounds: ActiveRunSlotBounds): ActiveRunSlotBounds {
|
||||
const { runsPerSlot, floor, ceiling } = bounds;
|
||||
if (!Number.isSafeInteger(runsPerSlot) || runsPerSlot < 1
|
||||
|| !Number.isSafeInteger(floor) || floor < 1
|
||||
|| !Number.isSafeInteger(ceiling) || ceiling < floor) {
|
||||
throw new Error("invalid_active_run_slot_bounds");
|
||||
}
|
||||
return bounds;
|
||||
}
|
||||
|
||||
export function slotsForActiveRuns(activeRuns: number, bounds: ActiveRunSlotBounds): number {
|
||||
const runs = Number.isFinite(activeRuns) && activeRuns > 0 ? Math.floor(activeRuns) : 0;
|
||||
return Math.min(bounds.ceiling, Math.max(bounds.floor, Math.ceil(runs / bounds.runsPerSlot)));
|
||||
}
|
||||
@@ -3,6 +3,8 @@ import { runSharedStorageWork, sendThroughStorageOutage } from "../storage/stora
|
||||
import { z } from "zod";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import type { StorageProvider } from "../storage/types.js";
|
||||
import { S3_CLIENT_MAX_SOCKETS } from "../storage/s3-client-limits.js";
|
||||
import { assertActiveRunSlotBounds, slotsForActiveRuns } from "../lib/active-run-slots.js";
|
||||
|
||||
const MAX_INDEX_LEVEL = 52;
|
||||
const MAX_OBJECT_BYTES = 4 * 1024 * 1024;
|
||||
@@ -217,20 +219,90 @@ export function isObjectJournalBusyError(error: unknown): error is ObjectJournal
|
||||
}
|
||||
|
||||
export type ObjectJournalIOPriority = "background" | "interactive";
|
||||
const MAX_ACTIVE_IO = 8;
|
||||
export const OBJECT_JOURNAL_READ_IO_RESERVE = 4;
|
||||
export const OBJECT_JOURNAL_IO_CEILING = Math.floor(S3_CLIENT_MAX_SOCKETS / 2);
|
||||
export const OBJECT_JOURNAL_UPLOAD_SLOTS = assertActiveRunSlotBounds({
|
||||
runsPerSlot: 10,
|
||||
floor: 4,
|
||||
ceiling: OBJECT_JOURNAL_IO_CEILING - OBJECT_JOURNAL_READ_IO_RESERVE,
|
||||
});
|
||||
const MAX_WAITING_INTERACTIVE_IO = 32;
|
||||
let openWriters = 0;
|
||||
let activeIO = 0;
|
||||
const waitingIO: Record<ObjectJournalIOPriority, Array<() => void>> = {
|
||||
background: [],
|
||||
interactive: [],
|
||||
};
|
||||
export function objectJournalUploadSlots() {
|
||||
return slotsForActiveRuns(openWriters, OBJECT_JOURNAL_UPLOAD_SLOTS);
|
||||
}
|
||||
export function objectJournalIOSlots() {
|
||||
return objectJournalUploadSlots() + OBJECT_JOURNAL_READ_IO_RESERVE;
|
||||
}
|
||||
export function trackObjectJournalWriter(): () => void {
|
||||
openWriters++;
|
||||
grantWaitingUploads();
|
||||
grantWaitingIO();
|
||||
let released = false;
|
||||
return () => {
|
||||
if (released) return;
|
||||
released = true;
|
||||
openWriters--;
|
||||
};
|
||||
}
|
||||
let activeUploads = 0;
|
||||
const uploadWaiters: Array<() => void> = [];
|
||||
export function objectJournalUploadActivity() {
|
||||
return { active: activeUploads, waiting: uploadWaiters.length };
|
||||
}
|
||||
function grantWaitingUploads() {
|
||||
while (activeUploads < objectJournalUploadSlots()) {
|
||||
const next = uploadWaiters.shift();
|
||||
if (!next) return;
|
||||
activeUploads++;
|
||||
next();
|
||||
}
|
||||
}
|
||||
export async function acquireObjectJournalUpload(signal?: AbortSignal): Promise<() => void> {
|
||||
signal?.throwIfAborted();
|
||||
if (activeUploads >= objectJournalUploadSlots())
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
const enter = () => {
|
||||
signal?.removeEventListener("abort", abort);
|
||||
resolve();
|
||||
};
|
||||
const abort = () => {
|
||||
const index = uploadWaiters.indexOf(enter);
|
||||
if (index !== -1) uploadWaiters.splice(index, 1);
|
||||
reject(signal!.reason);
|
||||
};
|
||||
signal?.addEventListener("abort", abort, { once: true });
|
||||
uploadWaiters.push(enter);
|
||||
});
|
||||
else activeUploads++;
|
||||
let released = false;
|
||||
return () => {
|
||||
if (released) return;
|
||||
released = true;
|
||||
activeUploads--;
|
||||
grantWaitingUploads();
|
||||
};
|
||||
}
|
||||
function grantWaitingIO() {
|
||||
while (activeIO < objectJournalIOSlots()) {
|
||||
const next = waitingIO.background.shift() ?? waitingIO.interactive.shift();
|
||||
if (!next) return;
|
||||
activeIO++;
|
||||
next();
|
||||
}
|
||||
}
|
||||
async function objectIO<T>(
|
||||
operation: () => Promise<T>,
|
||||
priority: ObjectJournalIOPriority = "background",
|
||||
signal?: AbortSignal,
|
||||
): Promise<T> {
|
||||
signal?.throwIfAborted();
|
||||
if (activeIO >= MAX_ACTIVE_IO) {
|
||||
if (activeIO >= objectJournalIOSlots()) {
|
||||
const queue = waitingIO[priority];
|
||||
if (priority === "interactive" && queue.length >= MAX_WAITING_INTERACTIVE_IO)
|
||||
throw new ObjectJournalBusyError();
|
||||
@@ -251,9 +323,8 @@ async function objectIO<T>(
|
||||
try {
|
||||
return await operation();
|
||||
} finally {
|
||||
const next = waitingIO.background.shift() ?? waitingIO.interactive.shift();
|
||||
if (next) next();
|
||||
else activeIO--;
|
||||
activeIO--;
|
||||
grantWaitingIO();
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -2,7 +2,9 @@ import { createHash } from "node:crypto";
|
||||
import { Readable } from "node:stream";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import type { StorageProvider } from "../storage/types.js";
|
||||
import { ObjectJournalBusyError } from "./object-byte-journal.js";
|
||||
import { S3_CLIENT_MAX_SOCKETS } from "../storage/s3-client-limits.js";
|
||||
import { OBJECT_JOURNAL_IO_CEILING, OBJECT_JOURNAL_READ_IO_RESERVE, OBJECT_JOURNAL_UPLOAD_SLOTS, ObjectJournalBusyError,
|
||||
objectJournalIOSlots, objectJournalUploadActivity, objectJournalUploadSlots } from "./object-byte-journal.js";
|
||||
import { createObjectRunLogStore } from "./object-run-log-store.js";
|
||||
|
||||
async function readBody(body: AsyncIterable<Uint8Array>) {
|
||||
@@ -569,4 +571,103 @@ describe("S3 run log store", () => {
|
||||
await Promise.allSettled([...polls, excess, ...(internal ? [internal] : [])]);
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps the object storage sockets the journal may use within the storage client's budget", () => {
|
||||
expect(OBJECT_JOURNAL_UPLOAD_SLOTS.ceiling + OBJECT_JOURNAL_READ_IO_RESERVE).toBeLessThanOrEqual(OBJECT_JOURNAL_IO_CEILING);
|
||||
expect(OBJECT_JOURNAL_IO_CEILING).toBeLessThanOrEqual(S3_CLIENT_MAX_SOCKETS / 2);
|
||||
});
|
||||
|
||||
async function uploadsInFlightWithOpenRuns(runs: number) {
|
||||
const m = memory(),
|
||||
store = createObjectRunLogStore({ provider: m.provider, keyPrefix: "logs", flushMs: 30_000,
|
||||
assertWriterAuthority: async () => undefined });
|
||||
const slotsBefore = objectJournalUploadSlots();
|
||||
const handles = [];
|
||||
for (let run = 0; run < runs; run++) handles.push(await store.begin({ ...input, runId: `run-${run}` }));
|
||||
const put = m.provider.putObjectConditional!.bind(m.provider);
|
||||
let open!: () => void;
|
||||
const gate = new Promise<void>(resolve => { open = resolve; });
|
||||
let inFlight = 0, peak = 0;
|
||||
m.provider.putObjectConditional = async value => {
|
||||
inFlight++;
|
||||
peak = Math.max(peak, inFlight);
|
||||
try {
|
||||
await gate;
|
||||
return await put(value);
|
||||
} finally {
|
||||
inFlight--;
|
||||
}
|
||||
};
|
||||
const appended = await Promise.all(handles.map(handle => store.enqueue!(handle, event(1))));
|
||||
const flushes = handles.map(handle => store.flush!(handle));
|
||||
await vi.waitFor(() => expect(inFlight).toBe(Math.min(runs, objectJournalUploadSlots())));
|
||||
await new Promise<void>(resolve => setImmediate(resolve));
|
||||
const waiting = { slotsBefore, slots: objectJournalUploadSlots(), ioSlots: objectJournalIOSlots() };
|
||||
open();
|
||||
await Promise.all(flushes);
|
||||
await Promise.all(appended.map(entry => entry.committed));
|
||||
await Promise.all(handles.map(handle => store.finalize(handle, { computeDigest: false })));
|
||||
return { ...waiting, peak, slotsAfterFinalize: objectJournalUploadSlots() };
|
||||
}
|
||||
|
||||
it("gives a log upload slot to each ten open runs, so 200 runs upload at least twenty logs at once", async () => {
|
||||
const result = await uploadsInFlightWithOpenRuns(200);
|
||||
expect(result.slotsBefore).toBe(OBJECT_JOURNAL_UPLOAD_SLOTS.floor);
|
||||
expect(result.peak).toBeGreaterThanOrEqual(20);
|
||||
expect(result.peak).toBeLessThanOrEqual(result.ioSlots);
|
||||
});
|
||||
|
||||
it("caps log uploads at the storage budget with 1,000 open runs and finishes every run's upload", async () => {
|
||||
const result = await uploadsInFlightWithOpenRuns(1_000);
|
||||
expect(result.slots).toBe(OBJECT_JOURNAL_UPLOAD_SLOTS.ceiling);
|
||||
expect(result.peak).toBeLessThanOrEqual(result.ioSlots);
|
||||
expect(result.ioSlots).toBeLessThanOrEqual(S3_CLIENT_MAX_SOCKETS / 2);
|
||||
expect(result.slotsAfterFinalize).toBeLessThan(result.slots);
|
||||
});
|
||||
|
||||
it("starts waiting uploads when runs open and starts no new upload after runs close until the count is under the limit", async () => {
|
||||
const options = { keyPrefix: "logs", flushMs: 30_000, assertWriterAuthority: async () => undefined };
|
||||
const busy = memory(), idle = memory();
|
||||
const busyStore = createObjectRunLogStore({ ...options, provider: busy.provider });
|
||||
const idleStore = createObjectRunLogStore({ ...options, provider: idle.provider });
|
||||
const busyHandles = [];
|
||||
for (let run = 0; run < 40; run++) busyHandles.push(await busyStore.begin({ ...input, runId: `busy-${run}` }));
|
||||
const slotsBefore = objectJournalUploadSlots();
|
||||
const put = busy.provider.putObjectConditional!.bind(busy.provider);
|
||||
const held: Array<() => void> = [];
|
||||
busy.provider.putObjectConditional = async value => {
|
||||
await new Promise<void>(resolve => held.push(resolve));
|
||||
return put(value);
|
||||
};
|
||||
const appended = await Promise.all(busyHandles.map(handle => busyStore.enqueue!(handle, event(1))));
|
||||
const flushes = busyHandles.map(handle => busyStore.flush!(handle));
|
||||
await vi.waitFor(() => expect(objectJournalUploadActivity().active).toBe(slotsBefore));
|
||||
expect(objectJournalUploadActivity().waiting).toBe(40 - slotsBefore);
|
||||
|
||||
const idleHandles = [];
|
||||
for (let run = 0; run < 160; run++) idleHandles.push(await idleStore.begin({ ...input, runId: `idle-${run}` }));
|
||||
const grown = objectJournalUploadSlots();
|
||||
expect(grown).toBeGreaterThanOrEqual(slotsBefore + 16);
|
||||
expect(objectJournalUploadActivity().active).toBe(grown);
|
||||
await vi.waitFor(() => expect(held).toHaveLength(grown));
|
||||
|
||||
const closing = Promise.all(idleHandles.map(handle => idleStore.finalize(handle, { computeDigest: false })));
|
||||
let previous = grown, shrankWhileBusy = false;
|
||||
while (held.length || objectJournalUploadActivity().waiting || objectJournalUploadSlots() > slotsBefore) {
|
||||
held.shift()?.();
|
||||
await new Promise<void>(resolve => setImmediate(resolve));
|
||||
await new Promise<void>(resolve => setImmediate(resolve));
|
||||
const { active } = objectJournalUploadActivity(), limit = objectJournalUploadSlots();
|
||||
expect(active).toBeLessThanOrEqual(Math.max(previous, limit));
|
||||
if (active > limit && limit === slotsBefore) shrankWhileBusy = true;
|
||||
previous = active;
|
||||
}
|
||||
await closing;
|
||||
busy.provider.putObjectConditional = put;
|
||||
await Promise.all(flushes);
|
||||
await Promise.all(appended.map(entry => entry.committed));
|
||||
await Promise.all(busyHandles.map(handle => busyStore.finalize(handle, { computeDigest: false })));
|
||||
expect(shrankWhileBusy).toBe(true);
|
||||
expect(objectJournalUploadActivity()).toEqual({ active: 0, waiting: 0 });
|
||||
});
|
||||
});
|
||||
@@ -2,7 +2,8 @@ import { beginIdleTrackedWork } from "./task-admission.js";
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import type { StorageProvider } from "../storage/types.js";
|
||||
import type { RunLogStore, RunLogHandle, RunLogBeginInput } from "./run-log-store.js";
|
||||
import { createObjectByteJournal, ObjectJournalSourcePrefixProofUnavailableError, type ObjectJournalIOPriority, type ObjectJournalWriter } from "./object-byte-journal.js";
|
||||
import { createObjectByteJournal, ObjectJournalSourcePrefixProofUnavailableError, acquireObjectJournalUpload, trackObjectJournalWriter,
|
||||
type ObjectJournalIOPriority, type ObjectJournalWriter } from "./object-byte-journal.js";
|
||||
import { MAX_RUN_LOG_RECORD_BYTES } from "./run-log-record-limits.js";
|
||||
import { notFound } from "../errors.js";
|
||||
|
||||
@@ -39,6 +40,7 @@ interface WriterState {
|
||||
admitting: Set<Promise<unknown>>;
|
||||
failure?: unknown;
|
||||
signal?: AbortSignal;
|
||||
releaseUploadDemand: () => void;
|
||||
sourcePrefixVerified: boolean;
|
||||
coverageEligible: boolean;
|
||||
admittedSourceCursor: number;
|
||||
@@ -65,31 +67,6 @@ export function createObjectRunLogStore(options: ObjectRunLogStoreOptions): RunL
|
||||
for (const wake of capacityWaiters) wake();
|
||||
capacityWaiters.clear();
|
||||
};
|
||||
let activeUploads = 0;
|
||||
const uploadWaiters: Array<() => void> = [];
|
||||
async function acquireUpload(signal?: AbortSignal) {
|
||||
signal?.throwIfAborted();
|
||||
if (activeUploads >= 4)
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
const enter = () => {
|
||||
signal?.removeEventListener("abort", abort);
|
||||
resolve();
|
||||
};
|
||||
const abort = () => {
|
||||
const index = uploadWaiters.indexOf(enter);
|
||||
if (index !== -1) uploadWaiters.splice(index, 1);
|
||||
reject(signal!.reason);
|
||||
};
|
||||
signal?.addEventListener("abort", abort, { once: true });
|
||||
uploadWaiters.push(enter);
|
||||
});
|
||||
else activeUploads++;
|
||||
return () => {
|
||||
const next = uploadWaiters.shift();
|
||||
if (next) next();
|
||||
else activeUploads--;
|
||||
};
|
||||
}
|
||||
function journal(logRef: string, readPriority?: ObjectJournalIOPriority) {
|
||||
if (!/^[a-zA-Z0-9._-]+\/[a-zA-Z0-9._-]+\/[a-zA-Z0-9._-]+$/.test(logRef))
|
||||
throw new Error("Invalid object log reference");
|
||||
@@ -123,7 +100,7 @@ export function createObjectRunLogStore(options: ObjectRunLogStoreOptions): RunL
|
||||
state.flushing = (async () => {
|
||||
let release: (() => void) | undefined;
|
||||
try {
|
||||
release = await acquireUpload(state.signal);
|
||||
release = await acquireObjectJournalUpload(state.signal);
|
||||
state.writer = await state.journal.append(
|
||||
state.writer,
|
||||
bytes,
|
||||
@@ -278,6 +255,7 @@ export function createObjectRunLogStore(options: ObjectRunLogStoreOptions): RunL
|
||||
closed: false,
|
||||
admitting: new Set(),
|
||||
signal: input.signal,
|
||||
releaseUploadDemand: trackObjectJournalWriter(),
|
||||
sourcePrefixVerified,
|
||||
coverageEligible: true,
|
||||
admittedSourceCursor: writer.head.sourceCursor,
|
||||
@@ -352,6 +330,7 @@ export function createObjectRunLogStore(options: ObjectRunLogStoreOptions): RunL
|
||||
return { bytes: null, sha256: null, compressed: false };
|
||||
} finally {
|
||||
writers.delete(handle);
|
||||
state.releaseUploadDemand();
|
||||
}
|
||||
},
|
||||
async read(handle, opts = {}) {
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
export const S3_CLIENT_MAX_SOCKETS = 50;
|
||||
@@ -17,6 +17,7 @@ import { sendThroughStorageOutage } from "./storage-unavailable.js";
|
||||
import { addAbortSignal, Readable } from "node:stream";
|
||||
import { retryS3RequestTimeouts } from "./s3-request-timeout-retry.js";
|
||||
import { refreshS3CredentialsOnRejection } from "./s3-credential-refresh.js";
|
||||
import { S3_CLIENT_MAX_SOCKETS } from "./s3-client-limits.js";
|
||||
import type { StorageProvider, GetObjectResult, HeadObjectResult } from "./types.js";
|
||||
import { notFound, unprocessable } from "../errors.js";
|
||||
|
||||
@@ -138,6 +139,10 @@ export function createS3StorageProvider(config: S3ProviderConfig): StorageProvid
|
||||
region,
|
||||
endpoint: config.endpoint,
|
||||
forcePathStyle: Boolean(config.forcePathStyle),
|
||||
requestHandler: {
|
||||
httpAgent: { maxSockets: S3_CLIENT_MAX_SOCKETS },
|
||||
httpsAgent: { maxSockets: S3_CLIENT_MAX_SOCKETS },
|
||||
},
|
||||
});
|
||||
retryS3RequestTimeouts(client);
|
||||
refreshS3CredentialsOnRejection(client);
|
||||
|
||||
Reference in new issue
Block a user