mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process controls every run of a workspace, and the target
is 1,000 concurrent agents on one server
> - Each native run writes its run log to object storage. The server
batches a run's log lines and uploads one batch per run each second
> - The server allowed only 4 log uploads and 8 object storage
operations at a time, for the whole process. These numbers were set when
50 agents was a lot
> - At 200 runs, the run logs fell about 8 seconds behind. At 1,000
runs, pending log bytes grow toward the 64 MiB process cap. Past that
cap, run events wait and log appends can fail
> - This pull request sizes the log upload slots and the object storage
operation slots from the number of open run logs, between the old floor
and a ceiling that comes from the storage client's socket budget
> - The benefit is that run logs stay about one flush interval behind at
200 runs, and fall 7.7 seconds behind instead of 44 seconds at 1,000
runs (p50), without more sockets than the storage client allows
## Linked Issues or Issue Description
Refs plan item 1g in
`doc/plans/2026-11-13-thousand-concurrent-agents.md` ("Size the shared
limiters ... from the active run count").
**What existing behavior does this improve?**
The shared run-log upload limiter and the shared object journal
operation limiter.
**Subsystem affected**
Server: object storage run logs (`object-run-log-store.ts`,
`object-byte-journal.ts`) and the S3 storage client.
**Current behavior**
The process allows 4 run-log uploads and 8 object journal operations at
a time, for any number of runs. Each upload is 3 to 4 sequential object
writes. At 200 runs and 50 ms per write, the run logs commit with a p50
delay of 8.0 s, and only 55% of the log lines offered in a 20 s window
commit inside it.
**Proposed behavior**
The upload slots are one per 10 open run logs, at least 4 and at most
21. The operation slots are the upload slots plus 4 for reads, at most
25. That is half of the 50 sockets that the S3 client may open, so
downloads and other storage calls keep the other half. The S3 client now
sets its 50-socket limit explicitly, from the same constant.
**Reason and benefit**
With the change, 200 runs commit their logs at the flush interval (p50
1.1 s), and 1,000 runs commit with a p50 delay of 7.7 s instead of 43.8
s. Fewer pending bytes stay in memory, so the 64 MiB pending cap and its
admission failure are farther away.
**Breaking changes**
None. With 40 or fewer open run logs the limits are the same as before
(4 uploads, 8 operations).
## What Changed
- `server/src/lib/active-run-slots.ts`: a small helper that gives a slot
count from an active run count, clamped between a floor and a ceiling.
- `server/src/storage/s3-client-limits.ts` and `s3-provider.ts`: the S3
client sets `maxSockets` to 50 (the SDK default) from a named constant,
so the budget is explicit.
- `object-byte-journal.ts`: the object operation limit is the upload
slots plus a read reserve of 4, from the number of open run-log writers.
Open writers are counted with `trackObjectJournalWriter()`. When the
limit grows, waiting operations start at once. When it shrinks, running
operations finish and no new ones start until the count is below the
limit.
- `object-byte-journal.ts` also owns the upload limiter
(`acquireObjectJournalUpload`), process-wide, next to the writer count.
When a run log opens, waiting uploads start at once. When run logs
close, running uploads finish and no new upload starts until the running
count is below the new limit.
- `object-run-log-store.ts`: uses the shared upload limiter. A run log
registers when it begins and releases when it is finalized. The waiting
queue stays first in, first out, and each run has at most one upload in
flight, so one run cannot starve another.
- Review fix: the first revision kept the upload limiter per store, so a
run log opened by another store did not wake uploads waiting in this one
(a lost wakeup on growth). The shared limiter fixes it; a new test
covers growth and shrinkage with queued work.
- Tests: limit growth with uploads waiting and shrinkage with uploads
running; helper bounds; the storage socket budget; 200 open runs upload
at least 20 logs at once (4 before); 1,000 open runs stay at the ceiling
and every upload finishes.
## Verification
- `cd server && npx vitest run src/lib/active-run-slots.test.ts
src/services/object-run-log-store.test.ts
src/services/object-byte-journal.test.ts`: all pass.
- Red/green: with the slot count forced back to the fixed floor of 4,
the two new scale tests fail (4 uploads in flight); with the change they
pass.
- Simulation (in-memory object store with 50 ms per write, each run
appends 1.5 KB at 0.7 records/s, the recorded native event rate, 20 s
window, flush every 1 s). Counts are stable on a loaded machine because
the latency is simulated:
| Runs | Build | Peak concurrent writes | Writes/s | Log lines committed
in window / offered | Commit delay p50 / p99 |
| --- | --- | --- | --- | --- | --- |
| 200 | before | 4 | 74 | 1,473 / 2,669 | 8.0 s / 13.5 s |
| 200 | after | 20 | 211 | 2,646 / 2,790 | 1.1 s / 1.3 s |
| 1,000 | before | 4 | 75 | 1,229 / 13,499 | 43.8 s / 62.3 s |
| 1,000 | after | 21 | 392 | 7,789 / 13,667 | 7.7 s / 12.6 s |
## Risks
- More object storage writes per second at high run counts (at 1,000
runs and 50 ms per write, about 390/s instead of about 75/s). This
raises object storage request cost in proportion. The ceiling bounds it.
- The S3 client now states its socket limit. The value is the SDK
default, so behavior does not change.
- The open-writer count is process-wide. A run log that is never
finalized keeps its count, as it already keeps its writer entry. The
ceiling bounds the effect.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
407 lines
16 KiB
TypeScript
407 lines
16 KiB
TypeScript
import { beginIdleTrackedWork } from "./task-admission.js";
|
|
import { createHash, randomUUID } from "node:crypto";
|
|
import type { StorageProvider } from "../storage/types.js";
|
|
import type { RunLogStore, RunLogHandle, RunLogBeginInput } from "./run-log-store.js";
|
|
import { createObjectByteJournal, ObjectJournalSourcePrefixProofUnavailableError, acquireObjectJournalUpload, trackObjectJournalWriter,
|
|
type ObjectJournalIOPriority, type ObjectJournalWriter } from "./object-byte-journal.js";
|
|
import { MAX_RUN_LOG_RECORD_BYTES } from "./run-log-record-limits.js";
|
|
import { notFound } from "../errors.js";
|
|
|
|
const MAX_SEGMENT_BYTES = 1024 * 1024;
|
|
const MAX_PENDING_BYTES = MAX_RUN_LOG_RECORD_BYTES;
|
|
const MAX_READ_BYTES = 4 * MAX_SEGMENT_BYTES;
|
|
const ALIGN_WINDOW_BYTES = 32_000;
|
|
// Most records are short, so the first alignment window rides along with the
|
|
// body read. Longer records continue with ALIGN_WINDOW_BYTES windows.
|
|
const ALIGN_FIRST_LOOKBACK_BYTES = 4096;
|
|
export interface ObjectRunLogStoreOptions {
|
|
provider: StorageProvider;
|
|
keyPrefix: string;
|
|
flushMs?: number;
|
|
segmentBytes?: number;
|
|
|
|
assertWriterAuthority: (input: RunLogBeginInput) => Promise<void>;
|
|
}
|
|
interface PendingAppend {
|
|
bytes: Buffer;
|
|
sequence: number;
|
|
sourceCursor: number;
|
|
resolve: (bytes: number) => void;
|
|
reject: (error: unknown) => void;
|
|
}
|
|
interface WriterState {
|
|
journal: ReturnType<typeof createObjectByteJournal>;
|
|
writer: ObjectJournalWriter;
|
|
pending: PendingAppend[];
|
|
pendingBytes: number;
|
|
timer?: NodeJS.Timeout;
|
|
flushing?: Promise<void>;
|
|
closed: boolean;
|
|
admitting: Set<Promise<unknown>>;
|
|
failure?: unknown;
|
|
signal?: AbortSignal;
|
|
releaseUploadDemand: () => void;
|
|
sourcePrefixVerified: boolean;
|
|
coverageEligible: boolean;
|
|
admittedSourceCursor: number;
|
|
}
|
|
|
|
export function createObjectRunLogStore(options: ObjectRunLogStoreOptions): RunLogStore {
|
|
const flushMs = options.flushMs ?? 1000;
|
|
const segmentBytes = options.segmentBytes ?? MAX_SEGMENT_BYTES;
|
|
if (
|
|
!Number.isSafeInteger(flushMs) ||
|
|
flushMs < 1 ||
|
|
flushMs > 30_000 ||
|
|
!Number.isSafeInteger(segmentBytes) ||
|
|
segmentBytes < 1024 ||
|
|
segmentBytes > MAX_SEGMENT_BYTES
|
|
)
|
|
throw new Error("Invalid object log batching limits");
|
|
const writers = new Map<RunLogHandle, WriterState>();
|
|
let totalPendingBytes = 0;
|
|
let admissionBytes = 0;
|
|
let admissions = 0;
|
|
const capacityWaiters = new Set<() => void>();
|
|
const capacityChanged = () => {
|
|
for (const wake of capacityWaiters) wake();
|
|
capacityWaiters.clear();
|
|
};
|
|
function journal(logRef: string, readPriority?: ObjectJournalIOPriority) {
|
|
if (!/^[a-zA-Z0-9._-]+\/[a-zA-Z0-9._-]+\/[a-zA-Z0-9._-]+$/.test(logRef))
|
|
throw new Error("Invalid object log reference");
|
|
return createObjectByteJournal({
|
|
provider: options.provider,
|
|
prefix: `${options.keyPrefix}/segments-v1/${logRef}`,
|
|
readPriority,
|
|
});
|
|
}
|
|
function schedule(state: WriterState) {
|
|
if (state.closed || state.failure || state.timer || state.flushing || !state.pending.length)
|
|
return;
|
|
state.timer = setTimeout(() => {
|
|
state.timer = undefined;
|
|
void flush(state).catch(() => undefined);
|
|
}, flushMs);
|
|
}
|
|
async function flush(state: WriterState): Promise<void> {
|
|
if (state.timer) {
|
|
clearTimeout(state.timer);
|
|
state.timer = undefined;
|
|
}
|
|
if (state.flushing) {
|
|
await state.flushing;
|
|
return flush(state);
|
|
}
|
|
if (state.failure) throw state.failure;
|
|
if (!state.pending.length) return;
|
|
const batch = state.pending.splice(0);
|
|
const bytes = Buffer.concat(batch.map((p) => p.bytes));
|
|
state.flushing = (async () => {
|
|
let release: (() => void) | undefined;
|
|
try {
|
|
release = await acquireObjectJournalUpload(state.signal);
|
|
state.writer = await state.journal.append(
|
|
state.writer,
|
|
bytes,
|
|
state.signal,
|
|
batch.reduce(
|
|
(cursor, item) => Math.max(cursor, item.sequence),
|
|
state.writer.head.sequence,
|
|
),
|
|
batch.reduce(
|
|
(cursor, item) => Math.max(cursor, item.sourceCursor),
|
|
state.writer.head.sourceCursor,
|
|
),
|
|
);
|
|
if (state.sourcePrefixVerified) {
|
|
try {
|
|
state.writer = await state.journal.recordVerifiedSourcePrefix(
|
|
state.writer, state.writer.head.sourceCursor, state.signal,
|
|
);
|
|
} catch (error) {
|
|
if (!(error instanceof ObjectJournalSourcePrefixProofUnavailableError)) throw error;
|
|
// The byte/index/head append is already committed. Lose only the
|
|
// optional provenance optimization; later appends remain deliverable.
|
|
state.writer = error.writer;
|
|
state.sourcePrefixVerified = false;
|
|
state.coverageEligible = false;
|
|
}
|
|
}
|
|
for (const item of batch) item.resolve(item.bytes.length);
|
|
} catch (error) {
|
|
state.failure = error;
|
|
const waiting = state.pending.splice(0);
|
|
const waitingBytes = waiting.reduce((sum, item) => sum + item.bytes.length, 0);
|
|
state.pendingBytes -= waitingBytes;
|
|
totalPendingBytes -= waitingBytes;
|
|
for (const item of [...batch, ...waiting]) item.reject(error);
|
|
throw error;
|
|
} finally {
|
|
state.pendingBytes -= bytes.length;
|
|
totalPendingBytes -= bytes.length;
|
|
capacityChanged();
|
|
release?.();
|
|
state.flushing = undefined;
|
|
schedule(state);
|
|
}
|
|
})();
|
|
await state.flushing;
|
|
}
|
|
async function enqueue(handle: RunLogHandle, event: Parameters<RunLogStore["append"]>[1]) {
|
|
const state = writers.get(handle);
|
|
if (!state || state.closed) throw new Error("Object run log writer is closed");
|
|
const admission = admit(state, handle, event);
|
|
state.admitting.add(admission);
|
|
try { return await admission; }
|
|
finally { state.admitting.delete(admission); }
|
|
}
|
|
async function admit(state: WriterState, handle: RunLogHandle, event: Parameters<RunLogStore["append"]>[1]) {
|
|
state.signal?.throwIfAborted();
|
|
if (state.failure) throw state.failure;
|
|
const bytes = Buffer.from(
|
|
`${JSON.stringify({
|
|
ts: event.ts,
|
|
attemptId: handle.attemptId,
|
|
stream: event.stream,
|
|
chunk: event.chunk,
|
|
...(event.seq === undefined ? {} : { seq: event.seq }),
|
|
})}\n`,
|
|
);
|
|
if (bytes.length > MAX_PENDING_BYTES)
|
|
throw new Error("Object run log record exceeds pending-byte limit");
|
|
let waiting = false;
|
|
try {
|
|
while (
|
|
state.pendingBytes + bytes.length > MAX_PENDING_BYTES ||
|
|
totalPendingBytes + bytes.length > 64 * MAX_SEGMENT_BYTES
|
|
) {
|
|
if (state.failure) throw state.failure;
|
|
if (!waiting) {
|
|
if (admissions >= 32 || admissionBytes + bytes.length > MAX_PENDING_BYTES)
|
|
throw new Error("Object run log admission capacity exceeded");
|
|
waiting = true;
|
|
admissions++;
|
|
admissionBytes += bytes.length;
|
|
}
|
|
for (const writer of writers.values())
|
|
if (writer.pending.length) void flush(writer).catch(() => undefined);
|
|
await new Promise<void>((resolve, reject) => {
|
|
const wake = () => {
|
|
state.signal?.removeEventListener("abort", abort);
|
|
resolve();
|
|
};
|
|
const abort = () => {
|
|
capacityWaiters.delete(wake);
|
|
reject(state.signal!.reason);
|
|
};
|
|
state.signal?.addEventListener("abort", abort, { once: true });
|
|
capacityWaiters.add(wake);
|
|
});
|
|
}
|
|
state.signal?.throwIfAborted();
|
|
if (state.failure) throw state.failure;
|
|
if (event.preservesSourceCoverage !== true || event.sourceCursor !== undefined &&
|
|
(!Number.isSafeInteger(event.sourceCursor) || event.sourceCursor <= state.admittedSourceCursor)) {
|
|
state.sourcePrefixVerified = false;
|
|
state.coverageEligible = false;
|
|
}
|
|
state.admittedSourceCursor = Math.max(state.admittedSourceCursor, event.sourceCursor ?? 0);
|
|
state.pendingBytes += bytes.length;
|
|
totalPendingBytes += bytes.length;
|
|
const finishIdle = beginIdleTrackedWork();
|
|
const committed = new Promise<number>((resolve, reject) =>
|
|
state.pending.push({
|
|
bytes,
|
|
sourceCursor: event.sourceCursor ?? state.writer.head.sourceCursor,
|
|
sequence: event.seq ?? state.writer.head.sequence,
|
|
resolve,
|
|
reject,
|
|
}),
|
|
).finally(finishIdle);
|
|
if (state.pendingBytes >= segmentBytes) void flush(state).catch(() => undefined);
|
|
else schedule(state);
|
|
return { committed };
|
|
} finally {
|
|
if (waiting) {
|
|
admissions--;
|
|
admissionBytes -= bytes.length;
|
|
}
|
|
}
|
|
}
|
|
return {
|
|
async begin(input) {
|
|
const logRef = `${input.companyId}/${input.agentId}/${input.runId}`;
|
|
const store = journal(logRef);
|
|
const writer = await store.claim({
|
|
assertAuthority: () => options.assertWriterAuthority(input),
|
|
signal: input.signal,
|
|
});
|
|
const sourcePrefixVerified = writer.head.generation > 1 && await store.hasVerifiedSourcePrefix(writer.head, input.signal);
|
|
const handle: RunLogHandle = {
|
|
store: "s3_segments",
|
|
logRef,
|
|
attemptId: randomUUID(),
|
|
initialBytes: writer.head.end,
|
|
initialSequence: writer.head.sequence,
|
|
initialSourceCursor: writer.head.sourceCursor,
|
|
initialSourcePrefixVerified: sourcePrefixVerified,
|
|
};
|
|
writers.set(handle, {
|
|
journal: store,
|
|
writer,
|
|
pending: [],
|
|
pendingBytes: 0,
|
|
closed: false,
|
|
admitting: new Set(),
|
|
signal: input.signal,
|
|
releaseUploadDemand: trackObjectJournalWriter(),
|
|
sourcePrefixVerified,
|
|
coverageEligible: true,
|
|
admittedSourceCursor: writer.head.sourceCursor,
|
|
});
|
|
return handle;
|
|
},
|
|
enqueue,
|
|
async append(handle, event) {
|
|
return (await enqueue(handle, event)).committed;
|
|
},
|
|
async flush(handle) {
|
|
const state = writers.get(handle);
|
|
if (!state) throw new Error("Object run log writer not found");
|
|
await flush(state);
|
|
},
|
|
async recordVerifiedSourcePrefix(handle, sourceCursor) {
|
|
const state = writers.get(handle);
|
|
if (!state || state.closed) throw new Error("Object run log writer is closed");
|
|
if (!state.coverageEligible) throw new Error("native_log_source_prefix_changed");
|
|
await flush(state);
|
|
if (!state.coverageEligible) throw new Error("native_log_source_prefix_changed");
|
|
// Serialize metadata publication with byte flushes, including diagnostics
|
|
// admitted while the one-time source proof is being recorded.
|
|
state.flushing = (async () => {
|
|
try {
|
|
let proofUnavailable = false;
|
|
try {
|
|
state.writer = await state.journal.recordVerifiedSourcePrefix(state.writer, sourceCursor, state.signal);
|
|
} catch (error) {
|
|
if (!(error instanceof ObjectJournalSourcePrefixProofUnavailableError)) throw error;
|
|
state.writer = error.writer;
|
|
proofUnavailable = true;
|
|
}
|
|
if (!state.coverageEligible) throw new Error("native_log_source_prefix_changed");
|
|
state.sourcePrefixVerified = !proofUnavailable;
|
|
if (proofUnavailable) state.coverageEligible = false;
|
|
state.admittedSourceCursor = Math.max(state.admittedSourceCursor, sourceCursor);
|
|
handle.initialSourcePrefixVerified = state.sourcePrefixVerified;
|
|
handle.initialSourceCursor = sourceCursor;
|
|
handle.initialBytes = state.writer.head.end;
|
|
handle.initialSequence = state.writer.head.sequence;
|
|
} finally {
|
|
state.flushing = undefined;
|
|
schedule(state);
|
|
}
|
|
})();
|
|
await state.flushing;
|
|
},
|
|
async finalize(handle, finalizeOptions) {
|
|
const state = writers.get(handle);
|
|
if (!state) throw new Error("Object run log writer not found");
|
|
state.closed = true;
|
|
capacityChanged();
|
|
try {
|
|
await flush(state);
|
|
await Promise.allSettled([...state.admitting]);
|
|
await flush(state);
|
|
state.writer = await state.journal.close(state.writer, state.signal);
|
|
if (finalizeOptions?.computeDigest === false)
|
|
return { bytes: state.writer.head.end, sha256: null, compressed: false };
|
|
const hash = createHash("sha256");
|
|
for (let start = 0; start < state.writer.head.end; start += MAX_READ_BYTES) {
|
|
hash.update(
|
|
await state.journal.readRange(state.writer.head, start, MAX_READ_BYTES, state.signal),
|
|
);
|
|
}
|
|
return { bytes: state.writer.head.end, sha256: hash.digest("hex"), compressed: false };
|
|
} catch {
|
|
process.emitWarning(
|
|
"Run log persistence is incomplete; final size and digest are unknown.",
|
|
);
|
|
return { bytes: null, sha256: null, compressed: false };
|
|
} finally {
|
|
writers.delete(handle);
|
|
state.releaseUploadDemand();
|
|
}
|
|
},
|
|
async read(handle, opts = {}) {
|
|
if (handle.store !== "s3_segments" && handle.store !== "s3_archive") throw new Error("Invalid object run log backend");
|
|
const store = journal(handle.logRef, opts.priority ?? "interactive"),
|
|
snapshot = await store.head(opts.signal) ?? writers.get(handle)?.writer;
|
|
if (!snapshot || snapshot.head.deleted) throw notFound("Run log not found");
|
|
const offset = Math.max(0, opts.offset ?? 0),
|
|
limit = opts.limitBytes ?? 256_000;
|
|
if (!Number.isSafeInteger(limit) || limit < 1 || limit > MAX_READ_BYTES)
|
|
throw new Error("Invalid object log read limit");
|
|
const bodyLimit = Math.min(MAX_READ_BYTES, limit + 3);
|
|
// Read the alignment window and the body in one range, so a tail read
|
|
// walks the index once instead of once for each part.
|
|
const lookback = opts.alignStartToRecord && offset > 0
|
|
? Math.min(offset, ALIGN_FIRST_LOOKBACK_BYTES, MAX_READ_BYTES - bodyLimit)
|
|
: 0;
|
|
const window = await store.readRange(
|
|
snapshot.head,
|
|
offset - lookback,
|
|
lookback + bodyLimit,
|
|
opts.signal,
|
|
);
|
|
const lookbackBytes = window.subarray(0, lookback);
|
|
const fetched = window.subarray(lookback);
|
|
let end = Math.min(limit, fetched.length);
|
|
while (end < fetched.length && (fetched[end]! & 0xc0) === 0x80) end++;
|
|
if (
|
|
end === fetched.length &&
|
|
offset + end < snapshot.head.end &&
|
|
fetched.length === MAX_READ_BYTES
|
|
) {
|
|
let lead = end - 1;
|
|
while (lead >= 0 && (fetched[lead]! & 0xc0) === 0x80) lead--;
|
|
const first = fetched[lead] ?? 0;
|
|
const width = first >= 0xf0 ? 4 : first >= 0xe0 ? 3 : first >= 0xc0 ? 2 : 1;
|
|
if (end - lead < width) end = lead;
|
|
}
|
|
const bytes = fetched.subarray(0, end);
|
|
const nextOffset =
|
|
offset + bytes.length < snapshot.head.end ? offset + bytes.length : undefined;
|
|
if (!opts.alignStartToRecord || offset === 0 || bytes.length === 0)
|
|
return { content: bytes.toString("utf8"), nextOffset };
|
|
const parts = [bytes];
|
|
let before = offset;
|
|
if (lookback > 0) {
|
|
const boundary = lookbackBytes.lastIndexOf(10);
|
|
parts.unshift(lookbackBytes.subarray(boundary + 1));
|
|
before = boundary >= 0 ? 0 : offset - lookback;
|
|
if (before > 0 && parts.reduce((n, part) => n + part.length, 0) > MAX_PENDING_BYTES)
|
|
throw new Error("Object log record exceeds read alignment limit");
|
|
}
|
|
|
|
while (before > 0) {
|
|
const start = Math.max(0, before - ALIGN_WINDOW_BYTES);
|
|
const prefix = await store.readRange(snapshot.head, start, before - start, opts.signal);
|
|
const boundary = prefix.lastIndexOf(10);
|
|
parts.unshift(prefix.subarray(boundary + 1));
|
|
if (boundary >= 0) break;
|
|
before = start;
|
|
if (parts.reduce((n, part) => n + part.length, 0) > MAX_PENDING_BYTES)
|
|
throw new Error("Object log record exceeds read alignment limit");
|
|
}
|
|
return {
|
|
content: Buffer.concat(parts).toString("utf8"),
|
|
nextOffset: nextOffset ?? offset + bytes.length,
|
|
};
|
|
},
|
|
async flushInflightMirrors() {
|
|
await Promise.all([...writers.values()].map(flush));
|
|
},
|
|
};
|
|
}
|