mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 22:56:58 +01:00
DottaandClaude Opus 5.5
90404d2033
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Hosted instances keep run logs in object storage as a journal of
content-addressed segments and binary index nodes
> - The task chat reads the last 32 KB of a run's log, aligned to a
record, to show what the agent did
> - A run that flushes often has many small segments. The tail read
fetched each object one after another, and record alignment walked the
index a second time
> - On staging, one tail read took 11–12 s, and the task chat waited for
it (see #20277)
> - This pull request reads the tail in parallel, in one pass, and
caches the objects that cannot change
> - The benefit is that the first tail read is about 6.6x faster, and
repeated reads (live polling, reopening a task) use only the mutable
head
## Linked Issues or Issue Description
**What happened?**
`GET
/api/heartbeat-runs/:runId/log?offset=…&limitBytes=32000&alignStartToRecord=true`
took 11–12 s on staging for a 400 KB object-store log. Without alignment
it took 6.1 s. A read from offset 0 took 2.0 s. The task chat waited for
this read before it showed the run's output.
**Expected behavior**
A 32 KB tail read returns in well under a second, and a repeated read of
the same tail costs about one storage round trip.
**Steps to reproduce**
1. Use object-backed run logs (`s3_segments`).
2. Write a run log in many small flushes, as a live run does.
3. Read its last 32 KB with `alignStartToRecord=true`, and count the
object GETs and how many run at the same time.
**Paperclip version or commit**
`master` at 6459488984.
**Deployment mode**
Hosted (authenticated) deployment with S3-compatible run-log storage.
Refs #20276 (measurement and plan), #20277 (chat no longer waits on a
slow transcript).
## What Changed
- `object-byte-journal.ts`, `readRange`:
- Visits the index tree and fetches sibling objects concurrently, with
at most 4 fetches for each read.
- Reassembles the pieces by offset.
- Cancels the remaining fetches when one fails.
- The shared `objectIO` limiter still caps the process at 8 slots, and
background persistence still gets freed slots first (#19176).
- `object-byte-journal.ts`, `load`:
- Keeps segments and index nodes in a process-wide LRU, limited to 32 MB
in total and 1 MB per object. The budget counts each entry's key and a
fixed 256-byte overhead with its bytes.
- These objects cannot change: each key ends in the SHA-256 of its
bytes, and `load` verifies that digest before it caches the object.
- The mutable head is never cached.
- The cache and the shared fetches are scoped to the storage provider
instance. Production uses one provider in the run-log store singleton.
- Concurrent fetches of the same object are shared. A shared fetch has
its own controller. A reader that aborts only stops its own wait. The
fetch is aborted only when no reader still waits for it, so the other
readers never start a second GET for that object.
- `object-run-log-store.ts`, `read`:
- With `alignStartToRecord`, the first 4 KB alignment window is read in
the same range as the body.
- The existing 32 KB backward windows still handle longer records.
- The output is unchanged.
- New test `object-run-log-store-tail-read.test.ts`, written against a
fake provider that counts GETs:
- correct aligned and unaligned content
- concurrent fetches
- alignment costs at most 30 extra GETs, with no second walk of the
index
- a repeated read costs 1 GET (the head)
- appended output is visible after cached reads
- a reader that shares a fetch survives another reader's abort
- a reader that joined a shared fetch stops at once on its own abort
- when the reader that started a fetch aborts, no object is fetched
twice
- a shared fetch stops when its last reader aborts
- one store's cached objects are not served to a reader of another store
- a tail that starts inside a record longer than the 4 KB lookback (6 KB
and 90 KB records)
- the cache budget counts per-entry overhead
- the shared-fetch tests hold object GETs open with a gate instead of
wall-clock delays
## Verification
- `cd server && npx vitest run
src/services/object-run-log-store-tail-read.test.ts
src/services/object-run-log-store.test.ts
src/services/object-byte-journal.test.ts
src/services/run-log-store.test.ts`: 4 files, 74 tests pass. The
tail-read file passed 5 of 5 repeated runs.
- The 18 server test files that use the journal, the run-log store or
`readLog`: 381 tests pass. Two skill-materialization tests timed out
under load and pass when run alone (4/4).
- Red checks:
- On `master`, the aligned tail read made 357 GETs with at most 1
running at a time, and a repeated read made 356 GETs.
- When I removed the abort handling for shared fetches, the shared-abort
test failed 3 times out of 3.
- The tests for the review fixes (joiner abort, no duplicate GET after
the first reader aborts, last-reader abort, per-store cache scope, cache
entry overhead) fail without the fix.
- An off-by-one in the long-record lookback arithmetic makes both
long-record tests fail.
- Benchmark: same fixture, with 30 ms latency added to each GET, a 32 KB
tail of 1,200 single-record segments.
| Read | `master` | This PR |
|---|---|---|
| Aligned | 11,753 ms, 365 GETs, 1 at a time | 1,778 ms, 205 GETs, 4 at
a time |
| Unaligned | 5,979 ms, 184 GETs | 1,607 ms, 184 GETs |
| Repeat aligned | 11,686 ms, 365 GETs | 34 ms, 1 GET |
- `pnpm typecheck` in `server` passes.
## Risks
- **Memory:** the cache can hold up to 32 MB of decoded log bytes in
each server process. Authorization still happens in the routes before
any read. The cache key is the full object key, which includes the run's
journal prefix.
- **Storage concurrency:** one read now holds up to 4 I/O slots instead
of 1. Background persistence still gets freed slots first. Under many
simultaneous cold reads, interactive reads can reach the 32-waiter limit
sooner and receive the existing retryable 503 `object_journal_io_busy`.
The cache makes repeated reads cheap, which offsets this.
- **Remaining cost:** the first read of a cold tail still walks a binary
index over small segments, about 1.8 s at 30 ms per GET. A larger gain
needs writer-side changes, such as wider index nodes or compaction when
the log closes. Those change the persisted shape, so they are not in
this PR.
- No migration, configuration change, persisted-shape change or new
dependency.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, browser automation, file editing).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>