Commit Graph
10 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 3eabdd6d10 perf(server): size run-log uploads and object journal IO from open run logs (#21685)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process controls every run of a workspace, and the target
is 1,000 concurrent agents on one server
> - Each native run writes its run log to object storage. The server
batches a run's log lines and uploads one batch per run each second
> - The server allowed only 4 log uploads and 8 object storage
operations at a time, for the whole process. These numbers were set when
50 agents was a lot
> - At 200 runs, the run logs fell about 8 seconds behind. At 1,000
runs, pending log bytes grow toward the 64 MiB process cap. Past that
cap, run events wait and log appends can fail
> - This pull request sizes the log upload slots and the object storage
operation slots from the number of open run logs, between the old floor
and a ceiling that comes from the storage client's socket budget
> - The benefit is that run logs stay about one flush interval behind at
200 runs, and fall 7.7 seconds behind instead of 44 seconds at 1,000
runs (p50), without more sockets than the storage client allows

## Linked Issues or Issue Description

Refs plan item 1g in
`doc/plans/2026-11-13-thousand-concurrent-agents.md` ("Size the shared
limiters ... from the active run count").

**What existing behavior does this improve?**
The shared run-log upload limiter and the shared object journal
operation limiter.

**Subsystem affected**
Server: object storage run logs (`object-run-log-store.ts`,
`object-byte-journal.ts`) and the S3 storage client.

**Current behavior**
The process allows 4 run-log uploads and 8 object journal operations at
a time, for any number of runs. Each upload is 3 to 4 sequential object
writes. At 200 runs and 50 ms per write, the run logs commit with a p50
delay of 8.0 s, and only 55% of the log lines offered in a 20 s window
commit inside it.

**Proposed behavior**
The upload slots are one per 10 open run logs, at least 4 and at most
21. The operation slots are the upload slots plus 4 for reads, at most
25. That is half of the 50 sockets that the S3 client may open, so
downloads and other storage calls keep the other half. The S3 client now
sets its 50-socket limit explicitly, from the same constant.

**Reason and benefit**
With the change, 200 runs commit their logs at the flush interval (p50
1.1 s), and 1,000 runs commit with a p50 delay of 7.7 s instead of 43.8
s. Fewer pending bytes stay in memory, so the 64 MiB pending cap and its
admission failure are farther away.

**Breaking changes**
None. With 40 or fewer open run logs the limits are the same as before
(4 uploads, 8 operations).

## What Changed

- `server/src/lib/active-run-slots.ts`: a small helper that gives a slot
count from an active run count, clamped between a floor and a ceiling.
- `server/src/storage/s3-client-limits.ts` and `s3-provider.ts`: the S3
client sets `maxSockets` to 50 (the SDK default) from a named constant,
so the budget is explicit.
- `object-byte-journal.ts`: the object operation limit is the upload
slots plus a read reserve of 4, from the number of open run-log writers.
Open writers are counted with `trackObjectJournalWriter()`. When the
limit grows, waiting operations start at once. When it shrinks, running
operations finish and no new ones start until the count is below the
limit.
- `object-byte-journal.ts` also owns the upload limiter
(`acquireObjectJournalUpload`), process-wide, next to the writer count.
When a run log opens, waiting uploads start at once. When run logs
close, running uploads finish and no new upload starts until the running
count is below the new limit.
- `object-run-log-store.ts`: uses the shared upload limiter. A run log
registers when it begins and releases when it is finalized. The waiting
queue stays first in, first out, and each run has at most one upload in
flight, so one run cannot starve another.
- Review fix: the first revision kept the upload limiter per store, so a
run log opened by another store did not wake uploads waiting in this one
(a lost wakeup on growth). The shared limiter fixes it; a new test
covers growth and shrinkage with queued work.
- Tests: limit growth with uploads waiting and shrinkage with uploads
running; helper bounds; the storage socket budget; 200 open runs upload
at least 20 logs at once (4 before); 1,000 open runs stay at the ceiling
and every upload finishes.

## Verification

- `cd server && npx vitest run src/lib/active-run-slots.test.ts
src/services/object-run-log-store.test.ts
src/services/object-byte-journal.test.ts`: all pass.
- Red/green: with the slot count forced back to the fixed floor of 4,
the two new scale tests fail (4 uploads in flight); with the change they
pass.
- Simulation (in-memory object store with 50 ms per write, each run
appends 1.5 KB at 0.7 records/s, the recorded native event rate, 20 s
window, flush every 1 s). Counts are stable on a loaded machine because
the latency is simulated:

| Runs | Build | Peak concurrent writes | Writes/s | Log lines committed
in window / offered | Commit delay p50 / p99 |
| --- | --- | --- | --- | --- | --- |
| 200 | before | 4 | 74 | 1,473 / 2,669 | 8.0 s / 13.5 s |
| 200 | after | 20 | 211 | 2,646 / 2,790 | 1.1 s / 1.3 s |
| 1,000 | before | 4 | 75 | 1,229 / 13,499 | 43.8 s / 62.3 s |
| 1,000 | after | 21 | 392 | 7,789 / 13,667 | 7.7 s / 12.6 s |

## Risks

- More object storage writes per second at high run counts (at 1,000
runs and 50 ms per write, about 390/s instead of about 75/s). This
raises object storage request cost in proportion. The ceiling bounds it.
- The S3 client now states its socket limit. The value is the SDK
default, so behavior does not change.
- The open-writer count is process-wide. A run log that is never
finalized keeps its count, as it already keeps its writer entry. The
ceiling bounds the effect.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:52:23 -06:00
DottaandClaude Opus 5.5 bf514759f9 fix(run-log): answer 404 for an object run log that was never written (#20539)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each heartbeat run keeps a run log, and the board reads it through
`GET /api/heartbeat-runs/:id/log`
> - A run can store its log in object storage (`s3_segments`)
> - When a run fails before its log is written (for example in
`runOutput.begin`), the object store throws a plain `Error("Object run
log not found")`
> - The route turns that plain error into a 500, so the board shows a
server failure for a log that simply does not exist
> - The local file store already answers 404 `Run log not found` for the
same case, and the UI treats a 404 as an unavailable log
> - This pull request makes the object store throw the same 404
> - The benefit is that a missing object run log looks like a missing
log, and a real storage failure still answers 500

## Linked Issues or Issue Description

**What happened?**
`GET /api/heartbeat-runs/:id/log` answered 500 `Internal server error`
for an `s3_segments` run whose log was never written. The server error
was `Object run log not found` from the object run-log store.

**Expected behavior**
The route answers 404 `Run log not found`, the same as the local file
store. The transcript view already treats a 404 for a finished run as a
missing log, and the log viewer already treats it as an unavailable log.
A real storage failure still answers 500.

**Steps to reproduce**
1. Use S3 storage for run logs.
2. Make a run fail before its run log is written (for example in
`runOutput.begin`).
3. Read `GET /api/heartbeat-runs/<run id>/log`. It answers 500.

**Paperclip version or commit**
`master` at adb0e77a4c.

## What Changed

- `server/src/services/object-run-log-store.ts`: a read of a log with no
head (never written or deleted) throws `notFound("Run log not found")`
(HTTP 404) instead of a plain `Error`.
- `server/src/__tests__/agent-live-run-routes.test.ts`: route tests
drive the real object run-log store behind the route. A never-written
log answers 404. A store read failure still answers 500. A board user of
another company gets 404 and the store is not read.
- `server/src/services/object-run-log-store.test.ts`: the never-written
read test now expects the 404 error.

## Verification

- Red: with the old store, `answers an object run log that was never
written` fails with `expected 500 to be 404`.
- Green: `cd server && PAPERCLIP_IN_WORKTREE=false npx vitest run
src/__tests__/agent-live-run-routes.test.ts
src/services/object-run-log-store.test.ts` passes (120 tests).

## Risks

- Low risk. Other readers of the store (`feedback.ts`, issue comment
derivation, export) already catch any read error. They do not match the
old message text.
- The response for a missing object log changes from 500 to 404. The
company access checks run before the store read, so they do not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 07:24:42 -06:00
DottaandClaude Opus 5.5 adb0e77a4c fix(run-log): read a resumed run's own log while the store still hides its new head (#20523)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In cloud mode, run logs use the `s3_segments` store: an object byte
journal whose `head.json` names the current writer and the committed
bytes
> - When a run resumes after a restart, `runOutput.begin` claims the
journal (this publishes a new `head.json`) and then reads the existing
log prefix to find where native log export continues
(`nativeRunLogPrefixCursor`)
> - On an S3-compatible store with read-after-write lag, the `head.json`
that the claim just published can read as `NoSuchKey` for some seconds
> - The read then throws "Object run log not found", and the resumed run
settles as `adapter_failed` and needs a Retry
> - This pull request makes a read through a live writer's own handle
use that writer's head when the store does not show the head yet, and
makes the one-time source-prefix proof step treat a hidden head as
"proof unavailable" instead of "another writer"
> - The benefit is that a run that resumes right after a restart
survives a store that shows new objects late, with no added wait

## Linked Issues or Issue Description

Refs #20509 (the same lag on the journal claim).

**What happened?**
In restart-chaos wave w-20261110T095205-s701 (local, `--storage s3
--storage-faults ST7`, K1/K2, on master with #20504, #20509 and #20514),
two resumed runs (t03 run 4f4fc7bc, t04 run c92c53d8) failed in
`runOutput.begin`. The error was "Object run log not found" from
`object-run-log-store.ts` `read`, called by `nativeRunLogPrefixCursor`
(`native-run-log-export.ts:314`). Both runs settled `adapter_failed`,
and the grade was R3 `retry_needed`. ST7 makes a GET or HEAD of a
just-written key answer 404 `NoSuchKey` for 5–30 s.

**Expected behavior**
A resumed run reads the log prefix that it just claimed and continues. A
store that shows a new object late must not look like a missing log.

**Steps to reproduce**
Write a run log, begin a successor writer on a new store instance, then
make `head.json` read as `NoSuchKey`. `read(successor)` throws "Object
run log not found".

## What Changed

- `server/src/services/object-run-log-store.ts`: `read()` reads the head
from the store as before. When the store says the head does not exist
and this process has a live writer for that handle, it uses the writer's
head. The writer published that head, so it proves the log exists. A
handle without a live writer still fails at once with "Object run log
not found".
- `server/src/services/object-byte-journal.ts`:
`recordVerifiedSourcePrefix` reads the head again to confirm ownership
before it publishes the optional source-prefix proof. When that head
reads as missing, it now throws
`ObjectJournalSourcePrefixProofUnavailableError` (the existing "proof is
optional" result). The store then continues without the proof, and the
next resume scans the prefix again. A visible head with a different
etag, a closed head or a deleted head still fails as
`ObjectJournalConflictError`. Later appends still publish the head with
a conditional write against the writer's etag, so another writer is
still detected.
- `server/src/services/object-run-log-store.test.ts`: a test hides the
successor's just-published `head.json` for the full resume sequence that
`nativeRunLogPrefixCursor` runs against the store (`read`, then
`recordVerifiedSourcePrefix`). Before this change, `read` fails with
"Object run log not found"; with only the read fix, the proof step fails
with "The output journal is owned by another writer". After both, the
read returns records 1–3, the proof is marked unavailable, and a later
append and read return records 1–4. A second test checks that a read of
a log that was never written still fails at once.

## Verification

- `cd server && npx vitest run src/services/object-byte-journal.test.ts
src/services/object-run-log-store.test.ts
src/services/object-run-log-store-tail-read.test.ts
src/services/run-log-store.test.ts
src/services/run-log-store-cancellation.test.ts`: 87 pass. The new
resume test fails with "Object run log not found" without the read fix,
and with "The output journal is owned by another writer" without the
proof fix.
- `cd server && npx vitest run
src/services/native-runtime/native-run-log-recovery.test.ts
src/services/native-runtime/native-run-log-watermark.test.ts`: 27 pass.
- `cd server && npx tsc --noEmit -p tsconfig.json` passes.
- Live: a local ST7 rerun (seed 701) on a build with this fix follows in
a comment.

## Risks

- Low risk. The read change only affects a read whose store head is
missing and whose handle has a live writer in this process. All other
reads behave as before, including fetch counts and the cache.
- When the head is hidden during the proof step, the proof is skipped
for this attempt. The next resume of the same run scans the log prefix
once more. This is the existing degraded path for a proof outage.
- The fix does not wait for lagged segment objects. A segment that the
previous server wrote less than the lag window before a restart can
still read as `NoSuchKey` during the prefix scan. The wave evidence
shows only the head lag. A broad segment wait broke existing failure
tests, so it is not part of this change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 06:40:58 -06:00
Nicky LeachandClaude Opus 5.5 0e3fdebea8 perf: remove repeated reads and checks from run preparation (#20358)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before each run starts, Core prepares the run: it opens the run log,
resolves skills, and prepares the AI runtime
> - Every run pays this preparation time before the agent can begin, for
a new task or a follow-up, on a cold or a warm computer
> - A trace on a hosted test deployment showed some preparation steps
that repeat a read or a check that a later step does again
> - This pull request removes three of these repeated steps and keeps
the check that each one duplicated
> - The benefit is that each run starts sooner, with no new mechanism
and no change to what a run can see or write

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Run preparation (`task.prepare`). On a hosted test deployment it takes
about 2.8 to 3.6 s. Skills preparation has a P50 of 275 ms (1,161 ms on
a deployment with more skills and projects).
`heartbeat.prepare.run_output` takes 386 to 724 ms.

**Subsystem affected**
Run logs in object storage (`object-byte-journal`,
`object-run-log-store`) and runtime skill delivery (`company-skills`,
`skill-execution-content`).

**Current behavior**
- An object journal claim is preceded by a head read. Its only use is to
give the current owner to `claim()`. `claim()` then reads the head
again, and it rejects the claim only when the two reads differ. The run
log, provider trace capture, and run evidence migration all do this.
- Run log `begin` reads the source-prefix proof object for every claim.
For a new run log, no proof can exist, so the read always returns
"missing".
- Each delivered skill version calls `withVersion`, which authorizes the
skill again for the run's actor. The skill row was read with the same
actor's visibility condition in the same call. With an agent actor this
repeats the company source and project permission reads once per
version.

**Proposed behavior**
- `claim()` reads the head once. It still refuses a deleted journal. The
conditional head write still rejects any change after that read, and a
stale writer is still fenced.
- `begin` does not read a proof for a first claim (generation 1). A
resumed log still reads and validates its proof, so a changed or
malformed proof still fails closed.
- Runtime skill delivery authorizes each skill once, when it reads the
selected rows. Other callers of `withVersion` keep their own check.

**Reason and benefit**
Each removed step protects nothing that a later step does not protect
again:
- Head read: the claim builds on its own head read, and the conditional
write enforces it.
- Proof read: a proof is written only by a writer that holds this log's
head. The journal never removes a head (deletion leaves a tombstone that
`claim()` refuses). So a first claim cannot find a proof.
- Skill check: every caller of `materializeVersionSnapshot` passes a row
that it read through the same actor's visibility condition
(`readFullSkillRows`, `getById`, `listFull`).

Expected saving per run: two object-storage GETs on `run_output` (about
100 ms each on the trace), and the per-version visibility reads in
skills preparation (4 versions per run on the test deployment, about 388
ms per version on a heavier deployment).

**Breaking changes**
None. `claim()` no longer takes `expectedOwner`. It is an internal API
with three callers in this repository, all updated.

Refs #20284 (it made run-log tail reads parallel and cached; this change
covers the begin path).

## What Changed

- `object-byte-journal`: remove the `expectedOwner` input and comparison
from `claim()`. Remove the extra head read in its three callers (run log
`begin`, provider trace capture, run evidence migration).
- `object-run-log-store`: `begin` reads the source-prefix proof only
when the claimed head has `generation > 1`.
- `skill-execution-content`: split `withVersion` into the access check
and `withReadableVersion`. `withVersion` keeps the check for its other
callers.
- `company-skills`: `materializeVersionSnapshot` uses
`withReadableVersion`, because each of its callers already read the
skill row under the actor's visibility condition.
- Tests:
- A new run log `begin` reads `head.json` once and reads no proof. A
resumed log reads `head.json` once and reads its proof once.
- Runtime skill delivery for an agent actor does no per-version source
permission reads.

## Verification

- `cd server && npx vitest run src/services/object-byte-journal.test.ts
src/services/object-run-log-store.test.ts
src/services/object-run-log-store-tail-read.test.ts
src/services/run-log-store.test.ts
src/services/provider-trace-object-store.test.ts`
- `cd server && npx vitest run
src/__tests__/company-skills-runtime-selection.test.ts
src/__tests__/company-skills-runtime-concurrency.test.ts
src/__tests__/heartbeat-runtime-skills.test.ts
src/__tests__/company-skills-service.test.ts
src/__tests__/managed-skill-content.test.ts` (embedded Postgres)
- `cd server && pnpm run typecheck`
- Each new test fails on the base commit: the run log test sees 2 head
reads and 1 proof read; the skill test sees 1 source permission read per
delivered version.
- Existing tests still pass for: stale-writer fencing after a takeover,
a deleted journal that refuses a claim, a malformed proof that fails
closed, and a denied project-scoped skill that is not delivered.
- I did not run the full `pnpm test:run`. CI covers it.

## Risks

- Low. Two concurrent claims of the same journal: before, the second
claim failed when the first published between its two head reads. Now
the later claim takes over, and the earlier writer's next append fails
with a conflict. This is the same result as when the later claim starts
after the first publish. Writer authority (`assertAuthority`) is
unchanged.
- A skill whose visibility is revoked in the moment between its row read
and its materialization in the same call is still delivered for that
run. This was already true for a revocation just after the old
per-version check.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 13:06:12 -08:00
DottaandClaude Opus 5.5 90404d2033 perf(run-logs): read object-store log tails in parallel, once, and cached (#20284)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Hosted instances keep run logs in object storage as a journal of
content-addressed segments and binary index nodes
> - The task chat reads the last 32 KB of a run's log, aligned to a
record, to show what the agent did
> - A run that flushes often has many small segments. The tail read
fetched each object one after another, and record alignment walked the
index a second time
> - On staging, one tail read took 11–12 s, and the task chat waited for
it (see #20277)
> - This pull request reads the tail in parallel, in one pass, and
caches the objects that cannot change
> - The benefit is that the first tail read is about 6.6x faster, and
repeated reads (live polling, reopening a task) use only the mutable
head

## Linked Issues or Issue Description

**What happened?**
`GET
/api/heartbeat-runs/:runId/log?offset=…&limitBytes=32000&alignStartToRecord=true`
took 11–12 s on staging for a 400 KB object-store log. Without alignment
it took 6.1 s. A read from offset 0 took 2.0 s. The task chat waited for
this read before it showed the run's output.

**Expected behavior**
A 32 KB tail read returns in well under a second, and a repeated read of
the same tail costs about one storage round trip.

**Steps to reproduce**
1. Use object-backed run logs (`s3_segments`).
2. Write a run log in many small flushes, as a live run does.
3. Read its last 32 KB with `alignStartToRecord=true`, and count the
object GETs and how many run at the same time.

**Paperclip version or commit**
`master` at 6459488984.

**Deployment mode**
Hosted (authenticated) deployment with S3-compatible run-log storage.

Refs #20276 (measurement and plan), #20277 (chat no longer waits on a
slow transcript).

## What Changed

- `object-byte-journal.ts`, `readRange`:
- Visits the index tree and fetches sibling objects concurrently, with
at most 4 fetches for each read.
  - Reassembles the pieces by offset.
  - Cancels the remaining fetches when one fails.
- The shared `objectIO` limiter still caps the process at 8 slots, and
background persistence still gets freed slots first (#19176).
- `object-byte-journal.ts`, `load`:
- Keeps segments and index nodes in a process-wide LRU, limited to 32 MB
in total and 1 MB per object. The budget counts each entry's key and a
fixed 256-byte overhead with its bytes.
- These objects cannot change: each key ends in the SHA-256 of its
bytes, and `load` verifies that digest before it caches the object.
  - The mutable head is never cached.
- The cache and the shared fetches are scoped to the storage provider
instance. Production uses one provider in the run-log store singleton.
- Concurrent fetches of the same object are shared. A shared fetch has
its own controller. A reader that aborts only stops its own wait. The
fetch is aborted only when no reader still waits for it, so the other
readers never start a second GET for that object.
- `object-run-log-store.ts`, `read`:
- With `alignStartToRecord`, the first 4 KB alignment window is read in
the same range as the body.
  - The existing 32 KB backward windows still handle longer records.
  - The output is unchanged.
- New test `object-run-log-store-tail-read.test.ts`, written against a
fake provider that counts GETs:
  - correct aligned and unaligned content
  - concurrent fetches
- alignment costs at most 30 extra GETs, with no second walk of the
index
  - a repeated read costs 1 GET (the head)
  - appended output is visible after cached reads
  - a reader that shares a fetch survives another reader's abort
  - a reader that joined a shared fetch stops at once on its own abort
- when the reader that started a fetch aborts, no object is fetched
twice
  - a shared fetch stops when its last reader aborts
- one store's cached objects are not served to a reader of another store
- a tail that starts inside a record longer than the 4 KB lookback (6 KB
and 90 KB records)
  - the cache budget counts per-entry overhead
- the shared-fetch tests hold object GETs open with a gate instead of
wall-clock delays

## Verification

- `cd server && npx vitest run
src/services/object-run-log-store-tail-read.test.ts
src/services/object-run-log-store.test.ts
src/services/object-byte-journal.test.ts
src/services/run-log-store.test.ts`: 4 files, 74 tests pass. The
tail-read file passed 5 of 5 repeated runs.
- The 18 server test files that use the journal, the run-log store or
`readLog`: 381 tests pass. Two skill-materialization tests timed out
under load and pass when run alone (4/4).
- Red checks:
- On `master`, the aligned tail read made 357 GETs with at most 1
running at a time, and a repeated read made 356 GETs.
- When I removed the abort handling for shared fetches, the shared-abort
test failed 3 times out of 3.
- The tests for the review fixes (joiner abort, no duplicate GET after
the first reader aborts, last-reader abort, per-store cache scope, cache
entry overhead) fail without the fix.
- An off-by-one in the long-record lookback arithmetic makes both
long-record tests fail.
- Benchmark: same fixture, with 30 ms latency added to each GET, a 32 KB
tail of 1,200 single-record segments.

| Read | `master` | This PR |
|---|---|---|
| Aligned | 11,753 ms, 365 GETs, 1 at a time | 1,778 ms, 205 GETs, 4 at
a time |
| Unaligned | 5,979 ms, 184 GETs | 1,607 ms, 184 GETs |
| Repeat aligned | 11,686 ms, 365 GETs | 34 ms, 1 GET |

- `pnpm typecheck` in `server` passes.

## Risks

- **Memory:** the cache can hold up to 32 MB of decoded log bytes in
each server process. Authorization still happens in the routes before
any read. The cache key is the full object key, which includes the run's
journal prefix.
- **Storage concurrency:** one read now holds up to 4 I/O slots instead
of 1. Background persistence still gets freed slots first. Under many
simultaneous cold reads, interactive reads can reach the 32-waiter limit
sooner and receive the existing retryable 503 `object_journal_io_busy`.
The cache makes repeated reads cheap, which offsets this.
- **Remaining cost:** the first read of a cold tail still walks a binary
index over small segments, about 1.8 s at 30 ms per GET. A larger gain
needs writer-side changes, such as wider index nodes or compaction when
the log closes. Those change the persisted shape, so they are not in
this PR.
- No migration, configuration change, persisted-shape change or new
dependency.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, browser automation, file editing).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 10:26:36 -06:00
e482f0f632 fix(server): shed only retryable object-journal reads under load (#19176)
Client polling can exhaust the shared object journal queue and reject log persistence. Bound ordinary object-log and trace reads, keep persistence and internal verification queued, and return retryable 503 responses when reads are busy. Leave feedback exports unsent until every log page is available.

Validation: 72 latest-master affected tests and 10 workflow tests pass locally, with the workflow fixture passing strict TypeScript checks. The preceding broader run passed 280 server tests with two opt-in S3 skips. All applicable checks on d0a937768d are green (56 success, three skipped); exact-head Apex review is 5/5 with no unresolved findings. Background persistence still has no fixed queue bound during a prolonged storage outage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-06 13:47:25 -08:00
DottaandPaperclip 31266d7159 fix(logs): drain existing admissions before finalization (#19055)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Run logs retain execution output for operators.
> - Object storage uses bounded admission and asynchronous uploads.
> - A diagnostic can wait for capacity when its native execution ends.
> - Finalization closed the writer before that existing admission could
finish.
> - This pull request joins those admissions before it closes the log.
> - The benefit is complete accepted output with the existing storage
bounds.

## Linked Issues or Issue Description

Refs #18944 and #18327.

**What happened?**

A runner close diagnostic began object-log admission before its recorder
retired. The call waited at the 8 MiB pending-byte bound. Completion
then closed the writer. The waiting call failed with `Object run log
writer is closed`. A valid final log hash covered the stored prefix, but
the diagnostic was absent.

**Expected behavior**

Finalization must retain calls that started before closure. It must
reject new calls after closure. Cancellation and storage failures must
still settle without a success claim.

**Steps to reproduce**

1. Use the actual recorder and segmented object store with a controlled
conditional upload.
2. Fill the pending-byte bound. Start native warm-session retirement
while the diagnostic waits for capacity.
3. Complete the run through the real completion handler. Release the
upload.
4. Check the diagnostic bytes, final hash, source coverage, and warning
rows.

## What Changed

- Track active admission promises in each object-log writer.
- Reject new producers when finalization starts. Join existing bounded
admissions, then flush their tail before manifest closure.
- Add actual completion coverage and store controls for failure, Stop,
global capacity, and independent writers.
- Clarify the finalization contract in the storage documentation. Keep
feature maps unchanged for this routine fix.

## Verification

- Before the fix, the actual completion probe lost a pre-retirement
close diagnostic and emitted `object_log_persistence_failed`. Useful
canonical output, source coverage, and the final prefix hash remained
valid.
- After the fix, the exact probe retains the diagnostic and checks the
stored bytes and full hash. Ordinary completion and stalled local append
controls also pass.
- Focused store controls pass, including shared 64 MiB capacity,
new-producer rejection, storage failure, Stop, and independent writer
progress.
- Production and test source at
`bc03f8d53a721f01f80a12105a69d8583865e651`: 3 actual completion controls
and 53 store, recorder, journal, and diagnostic tests pass. Run with
`node node_modules/vitest/vitest.mjs run` from `server/`; select the
four safety files plus the completion file with `-t "retired recorder"`
in a separate invocation.
- Initial treatment repository typechecks and build pass. Current map
and module checks pass. A broader four-suite run had 99 passing
assertions but failed its existing 30-second completion teardown hook. A
later isolated completion run passed all 61 cases at the same limits.
The original teardown cause remains unresolved; no timeout was changed.
- Repository typechecks and build pass at that source commit. Current
head `6652adc7f3379ae250f7b726dad6c58c8482410f` removes only the
incidental map paragraph; production and test source are identical. All
current-head checks are terminal: 54 successful check runs, one
successful legacy status, and three skipped checks. Required `ci /
verify` and `ci / e2e` pass. Greptile reviewed exact current head at
5/5. Both review threads are resolved. GitHub reports the PR clean and
mergeable. Full repository tests stopped in chat shard 2: a GitHub
recovery fair-batch test expected no provider requests but observed `GET
/app/hook/config`. Its exact unchanged case passes in isolation. The
original cause remains unresolved; this full-suite run is red.
First-head remote server lane 2 failed a native-reassessment backlog
test: its quiet check observed two active work items instead of zero.
The exact unchanged case passes locally. That original CI cause remains
unresolved; no job retry or timeout change was made. Other first-head CI
lanes passed. The first native review rated the earlier source head 4/5
for the same removed map note. That is not a current-head assessment.
Current GitHub rules require zero approving reviews and only the two CI
checks above; a fresh native advisory re-review is optional and remains
unavailable while its earlier run has an unresolved workspace/accounting
obligation. That obligation is tracked separately and is not claimed
recovered.
- Removed the incidental map paragraph identified by Greptile and the
native review. The follow-up contains no runtime or test changes.
- This proves the local production-path mechanism. It does not attribute
any specific hosted warning to this cause. Hosted replay and served
revision checks remain separate acceptance work.

## Risks

Finalization now waits for existing bounded capacity admissions. Storage
cancellation and the existing operation deadlines still apply. Tests
check that failed uploads and Stop settle, and that a separate writer
can make progress. No capacity limit, timeout, or execution authority
changes.

## Model Used

OpenAI `gpt-6.1-sol`, through Codex, with reasoning and tool execution.
Context window size is not exposed by the host.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (56 focused checks; the
full-suite red remains documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 16:34:36 -06:00
Dotta 049c1eacf5 Preserve historical native logs during ephemeral enrollment (#18864)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Ephemeral web servers need retained run history in shared storage.
> - Some old native runs have surviving local logs but no completion
digest or verifiable native source coverage.
> - Enrollment must preserve those bytes without calling them a complete
native transcript.
> - This pull request adds a read-only archive path to offline
enrollment.
> - Users can still read old output after the original server filesystem
is removed.

## Linked Issues or Issue Description

**What happened?**

Offline enrollment blocks on interrupted runs and unfinalized historical
native logs. The current importer requires canonical native-event
coverage even when the original local file survives.

**Expected behavior**

Preserve the exact surviving file in shared storage. Keep the original
run status, completion metadata, and source events. Do not allow writers
or native repair to modify archived history.

**Steps to reproduce**

Use an S3-configured instance with a terminal native run whose local log
lacks a final size or digest. Stop all app deployments and run ephemeral
maintenance. Before this change, interrupted status or unproven native
coverage prevents enrollment.

Related preservation work: #18826, #18835, #18841. This is a bounded
migration fix, not a new live logging protocol.

## What Changed

- The run-evidence migration owner can archive unfinalized native local
files during offline enrollment.
- Require the offline migration environment, zero active or queued runs
across the instance, and expired controller/finalizer leases. Check
global quiescence at each page boundary. Recheck each run's ownership
and metadata before publication.
- Verify stable local file identity and metadata, any recorded
size/digest, uploaded prefixes, and remote SHA-256.
- Store raw history in a separate `<run>.archive` journal with source
cursor zero and backend `s3_archive`. Leave completion fields and native
events unchanged.
- The existing log store reads archives but rejects begin, append,
enqueue, flush, and finalize. Native source repair does not run for
archives.
- Keep the default background importer's coverage checks. Continue to
reject incomplete mirrors and corrupt files.
- Add real PostgreSQL/S3 and packaged maintenance coverage. Correct a
pre-existing large-chunk fixture: the current verifier can prove exact
compactor output, so the negative case now uses a different canonical
source.

## Verification

- Passed on final head b96e80dbb3: 13
migration/storage/worker tests against disposable local PostgreSQL and
S3.
- Passed: 76 log-store, cancellation, object-store, native watermark,
and maintenance-service tests.
- Passed: server typecheck and server build.
- The archive journey removes original log files, reads exact saved
bytes through the normal recovery-aware reader, checks unchanged
run/source metadata, and verifies read-only behavior.
- Negative cases cover ordinary-process use, another company's running
execution, live controller/finalizer leases, an invalid recorded digest,
an incomplete mirror, and a changed local source.
- Review follow-up: move the global run-table scan from every item to
page boundaries. A real PostgreSQL trigger admits new execution during
publication; the page then fails, retains verified copies, and resumes
after quiescence. All 5 migration cases passed. After rebasing onto
merged #18841, all 13 migration/storage/worker cases and all 7 packaged
maintenance cases passed, along with server typecheck, build, and
runtime bundle.
- Preserved failure: the old large-chunk expectation failed with both
this change and the unmodified base importer. The updated fixture checks
the verifier's current compaction contract and still blocks mismatched
source output.
- Passed: all 7 real PostgreSQL/S3 enrollment cases, including the
bundled maintenance process under plain Node. Its interrupted native log
is archived, the original log directory is removed, and the saved bytes
remain readable.
- Passed: packaged runtime build and full repository CI (54 successful
checks). Four CI jobs lost their runners; a separate voice-recovery
timing assertion failed. The unchanged voice case passed locally, and
all failed/interrupted CI jobs passed on retry. Greptile reviewed the
final head at 5/5 with no unresolved threads. The separate Paperclip
Review app remains queued/paused. No live tenant migration or volume
detachment is claimed by this PR.

## Risks

- Rollback after enrollment requires a reader that supports
`s3_archive`; old binaries cannot read the new backend.
- Offline mode relies on the deployment owner stopping all old app
processes. Database and file checks are additional fences, not a
substitute for that prerequisite.
- Archives may be incomplete transcripts. The implementation preserves
that uncertainty and does not fabricate a successful run or completion
digest.
- No schema change, new background polling, or telemetry change.

## Model Used

OpenAI Codex (GPT-6 family), with reasoning, repository tool use, and
code execution. The exact serving model ID and context window are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-03 16:52:30 -06:00
DottaandPaperclip 3910352c92 feat(runner): add experimental Cursor Cloud provider (#18068)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Runner gives each provider durable execution, tool, and completion
contracts.
> - Cursor Cloud currently has a direct adapter outside that native
contract.
> - Cloud work needs exact ownership across HTTP, SSE, Stop, and
recovery.
> - This PR adds a qualified Cloud API driver and verified proxy to
Runner.
> - Product setup and hosted placement remain separate dependent
changes.

## Linked Issues or Issue Description

Refs #5664, #11803, #17299, and #17471. Product integration is #18069.
Managed-only API placement is #18310. The related direct-adapter
model/transcript work is not replaced. No duplicate native Cursor Cloud
integration was found.

## What Changed

- Add native `cursor_cloud` / `cursor_cloud_api` with explicit
model/repository validation, bounded HTTP/SSE, durable per-turn
checkpoints and a qualified Node proxy launched through the Rust facade.
- Commit terminal custody before publication/release; coalesce
consumers, preserve ambiguous create/Stop/recovery fences, reject late
tools, and distinguish definitely-unsent oversized requests from
ambiguous responses.
- Support scoped cold attachment, current authority, reconnect and
verified artifact staging. Keep unsupported warm/external and Windows
paths closed.
- Preserve early source cursors and verify immutable legacy prefixes
before suffix replay, including supported compaction/masking. Optional
proof publication failure falls back to complete cold verification.
- Preserve canonical records at the durable native producer with
JSON-safe privacy and the existing 8MiB bound; drain admitted successor
flushes. Reconcile upstream semantic/pack/schema/lifecycle changes and
retain S3 and legacy CLI checks.

## Verification

Core head `37c442ec8dd4`: all 66 check contexts are terminal
successful/neutral/skipped, including required `ci / verify` and `ci /
e2e`; fresh exact-head Greptile is 5/5 with zero unresolved threads.
Dependent Product `81e828d20a77` and Top `dd2959689627` include the
later client handoff correction and are running their own fresh gates.

Core: 47/47 Rust, 4/4 compiled-native retirement, 19/19 settlement and
normal Runner build passed. Runtime Top `9af1a5de42a2`: build and
recursive typecheck passed. Earlier source-scoped evidence includes 398
focused tests, five relay tests and token/type gates. The 251-assertion,
7/8-suite run failed physical PostgreSQL teardown; the unchanged stable
WSP suite passed 128 through the documented wrapper.

The downstream fixture corrections passed 7/7 browser and 121/121
startup tests without timeout/oracle relaxation; a broken-unsubscribe
control fails. Hosted runtime `9af1a5de42a2` has a passing normal build
and recursive typecheck. Latest dependent heads additionally contain a
pure client fix for stale live/history overlap, with 48 focused tests,
UI typecheck and token gates passing. Its new client is not deployed in
the frozen hosted image; server/native/API/provider source is unchanged
by these four downstream files.

Fresh hosted Linux admission on `9af1a5de42a2` retired its owned
runner/proxy/socket with zero prompts. The saved configuration survived
reopening, and Test authenticated the bound key/catalog/selected model
without a prompt. Artifact with `claude-sonnet-4-5` completed,
downloaded the exact 27 bytes, matched 121 canonical identities against
complete logs and independently reached FINISHED. The active Stop retry
was clicked within its immutable boundary, produced one canonical
interrupted turn and acknowledged run-only cancellation, retained a
valid 35.076s quiet window (maximum gap 4.584s), and independently
remained CANCELLED/artifact-absent beyond the original 180-second
command deadline. All provider-observer credential callbacks, DB clients
and loaders closed; accounting is settled and explicitly unpriced. The
first fresh Stop retains its failed 10.373s sampling gap and every
sample.

On the upgraded9af server, the historical F226 artifact was reopened and
downloaded with its exact original hash after its provider agent had
been deleted/GET404 earlier today. Historical F226 full Artifact/Stop
cleanup, Question, corrected Plan and reconnect retain their original
identities. Deleting the three newly created disposable provider agents
and the new-case post-delete grade remain blocked on explicit user
approval after automatic review rejected expanding the two original
approved IDs. No new-case cleanup or complete fresh matrix pass is
claimed; this is separate from the directly observed task/Stop behavior
and retained exact cleanup proof.

Retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook
failures; Mac 17 SQL failures/skips; earlier startup-title, title-save,
custody/Stop and deletion failures with unknown causes and bounded
recoveries. `dea023` Artifact passed 17 checks but Stop failed its false
warning and 50 duplicate log identities. Pre-repair ACK failure and
original Core CI failure remain recorded; retry alone establishes no
cause or full local-suite pass. Provider dollar cost stays UNKNOWN.

## Risks

Experimental: Cursor independently acknowledges neither the effective
model nor instruction digest before the first prompt. The caller’s
explicit values are submitted and behavior checked live. Unpriced
provider dollar cost stays UNKNOWN.

Only qualified per-turn controllers are admitted. Uncertain work and
incomplete cleanup stay fenced. Existing damaged/duplicate archives
remain failed evidence. Prefix proof adds bounded object-store I/O; the
8MiB producer bound does not guarantee future providers’ output fits.

## Model Used

OpenAI Codex, GPT-6 family, assisted with reasoning, repository tools,
code execution, browser verification, and delegated code review. The
serving model’s exact API ID and context window are not exposed. No
value is inferred from the model catalog.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used and the limits of available model
metadata
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have linked public issues or described the issue in this PR
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch describes the change and contains no internal ticket ID
- [x] I have run the scoped local tests reported above; pending gates
are stated separately
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented the risks
- [x] All required CI gates pass on the current head
- [x] Fresh Greptile is 5/5 with no unresolved findings on the current
head
- [x] I will address all reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 08:38:57 -06:00
DottaandPaperclip dbff3747b8 feat(server): persist run logs and provider traces in S3 (#18327)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Operators use run logs and restricted provider traces to inspect
agent work.
> - Local log and trace files depend on the web server's persistent
volume.
> - That dependency prevents overlapping volume-backed Railway
deployments.
> - This PR reuses the instance S3 store for batched log and trace
payloads with conditional publication.
> - Existing Postgres rows retain their contracts; native export
recovery reuses events already committed before Runner acknowledgement.
> - S3-configured servers automatically copy and verify retained history
before a later deployment cutover.

## Linked Issues or Issue Description

**Subsystem affected**

Server run logs, restricted provider traces, storage providers, and
migration tooling.

**Problem or motivation**

Replacing the web server and its disk must preserve committed log output
and retained traces. The current S3 mirror can repeatedly upload a
growing file. Moving those payloads into Postgres would add database
writes, WAL, replication, and backup work.

**Proposed solution**

Reuse the existing instance storage configuration, S3-compatible
provider and credential chain. Store new run logs under run-logs/ and
opted-in raw traces under provider-traces/, below the existing prefix.
Cloud keeps its existing bucket and bucket-scoped credentials per stack.
Add no new configuration switches. Store immutable payload segments and
bounded binary range indexes directly in S3. Publish each batch with an
ETag-conditional head. Keep existing database ownership and lifecycle
records. Recover canonical native exports from their existing validated
control event source, checkpointing export progress in S3. Add no new
payload writes to Postgres. Preserve local defaults and migrate settled
history with verified byte counts and digests.

**Alternatives considered**

Postgres payload chunks add control-plane load. Repeated whole-file
mirrors amplify transfer work. The first index prototype left superseded
index nodes; the final binary index keeps all normally published nodes
reachable.

**Roadmap alignment**

This is a storage prerequisite for replacing the main web/API server. It
does not enable deployment overlap, detach a volume, or change native
Runner protocol acknowledgements. The maintainer approved this
implementation scope. ROADMAP.md has no duplicate implementation item.

Related PR: #16770 improves the existing in-flight mirror. This PR
preserves that legacy backend and uses the existing instance S3 profile
for new journals.

## What Changed

- Automatically select S3 segments for new logs and opted-in traces when
instance file storage uses S3. Reuse bucket, endpoint, region, prefix,
path style and credentials. Remove separate backend switches. Keep
local-disk instances local.
- Share the existing storage configuration resolver with product files.
Preserve config-file settings, environment precedence and the existing
root prefix. Keep pre-existing run-log mirror settings for legacy
history.
- Add bounded buffering, conditional ownership fencing, immutable range
reads, stable byte offsets, and completion hashing.
- Dispatch all existing log consumers through the recorded backend.
Coalesce output progress writes and publish live log messages after
storage receipts.
- Recover interrupted canonical native exports from their existing
durable source. Exclude separate credential, cleanup, and reprojection
records. Replay missing callback ranges and deduplicate late delivery.
Repair settled archives after controller lease expiry. Bound shared
repair work and stop it when all readers cancel. Serve the published
prefix when best-effort repair fails.
- Add encrypted optional trace channels, replacement reads, durable
deletion tombstones, and deadline-driven local/external expiry cleanup
independently of heartbeat scheduling. Stream historical source and
remote verification. Propagate worker cancellation through trace open,
conditional-write probing, upload, restore and reads. Fence deletion
even before the first object head exists; abort and drain owned
captures.
- Retry transient storage errors, distinguish 409 from failed
preconditions, and backpressure bounded log admission. Track interrupted
trace imports explicitly so ordinary local deletion stays independent of
S3.
- Automatically migrate retained history through the existing delivery
coordinator. Process one log and one trace per batch. Retry unresolved
history with backoff and stop scans after completion. Keep the
inventory/import command as optional diagnosis and repair. Verify source
and remote bytes before switching references. Verify finalized native
size and digest before setting its export cursor. Require ordered source
coverage when completeness metadata is missing. Preserve original trace
expiry. Block active, missing, corrupt, and unproven sources.
- Preserve identity and run-secret redaction in live and recovered
native logs. Register runtime log redaction secrets once through the
existing encrypted registry. Keep agent private keys in their existing
owner.
- Add storage, migration, retention, UTF-8, and route-privacy coverage,
plus operating documentation and qualification tooling.
- Add no schema, payload table, per-record SQL, production dependency,
or lockfile change.

## Verification

- Final fixes at `b3a44c898e` passed all 116 selected startup,
retention, attribution and cancellation controls. The disabled-scheduler
cleanup and concurrent-repair defects reproduced red before the fixes. A
timed-out repair still prevents later storage reads.
- The merged source passed 124 storage and real-Postgres Cloud
configuration tests. These include automatic history migration, existing
environment/file vault keys, source-directory removal, five real-SDK
stalled-request cancellation cases, shared capability-probe cancellation
and multi-chunk trace ordering. Saved Cloud S3 settings work after the
legacy config file is gone.
- Workspace typecheck and full build passed at the merged parent
`19f702d780`. Server typecheck and build passed again at `b3a44c898e`.
- Actual app acceptance: finish a process-agent run through the normal
API, stage its retained local-file input, then restart. Automatic
migration reports one migrated item and zero blockers. Remove the
original web home and restart at `b3a44c898e` with an empty home, the
existing S3 profile and the existing vault key file. The normal log API
preserves all five producer lines and exactly 788 bytes, with SHA-256
`74d9ae469da28bfe89b9b50a0e132aec24fe9e6d686496ed5e4ffa6167151036`. No
log-specific storage settings are present.
- Actual expiry acceptance: with heartbeat scheduling disabled, a trace
remains unexpired after startup and later reaches its fixture deadline.
The worker marks the row deleted and tombstones both S3 channel heads.
The retained run log stays byte-identical. The process fixture emits no
provider protocol frames, so nonempty encrypted payloads are qualified
separately through the real SDK round trips.
- Isolation tests use identical run identities in two stack buckets.
Cleanup in one existing prefix leaves another prefix, another company
and product files intact. Read-only Cloud source inspection confirms a
bucket and bucket-scoped credentials per stack. Four existing Cloud
storage environment-contract tests passed. No Cloud provisioning changed
and no new live vendor permission probe was performed.
- Earlier qualification passed 1,014 affected storage/native tests and
27 privacy-route tests with isolated Docker Postgres. An abrupt process
test commits 1,000 records, receives receipts, dies by SIGKILL and loses
its directory; another process reads every byte and continues. The real
Rust Runner experiment acknowledges 32 records while the export cursor
is zero, then continues the same Runner PID after controller loss. All
64 records export once with exactly 64 existing control rows. Repeated
snapshot/Stop commands completed in 162/81 ms.
- Emulator workload: 1,000 burst records used two batch PUTs. Eight
steady streams produced 240 records with 104 conditional PUTs and 72
GETs over 6.9 seconds. These are synthetic operation counts, not fleet
sizing or provider prices.
- Failure evidence is retained. Compact native events over 64 KiB,
missing file-key support, disabled-scheduler retention and repair
failure handling all reproduced before their fixes. Fixture repairs
addressed cold module setup, localhost/IPC permissions, the S3
bucket-name limit and a mode-0700 crash-test directory. Full local `pnpm
test:run` was stopped after host PostgreSQL semaphore exhaustion;
affected suites used isolated Docker databases instead. Native package
library symlinks were restored with the package's standard hydration
helper.
- The `19f702d780` browser session-recovery case timed out while saving
a task title. All eight browser shards passed on `b3a44c898e`, including
that case, with no UI behavior or assertion change. Fresh-head CI passed
at `b3a44c898e`: 55 successful checks, three skipped checks and one
neutral check. The branch is conflict-free.
- Map and module-boundary checks passed. Current-head Greptile is 5/5
with zero unresolved review threads.

Reproduce the opt-in storage suites with
`PAPERCLIP_LOG_TEST_S3_ENDPOINT` and a dedicated
`PAPERCLIP_RUN_EVIDENCE_MIGRATION_DATABASE_URL` ending in `_test`. See
`doc/run-evidence-storage.md` for operating limits, automatic migration,
optional diagnostics and rollout qualification.

## Risks

- Keep the existing per-stack bucket, prefix and vault key stable across
replacements. Relocating that storage is a separate whole-instance
operation.
- This release automatically changes new log storage for existing
S3-configured instances. Local-disk instances remain local. Canonical
native source replay is qualified with a deterministic real Runner;
actual remote execution and production-provider qualification remain
rollout gates. Generic server-only diagnostics and optional raw trace
tails without a surviving source can be incomplete after a crash. This
does not establish full zero-downtime deployment.
- Before deploying this release, qualify the actual S3 provider. It must
enforce conditional writes and pass read, version, deletion, workload,
and latency qualification. A capability probe rejects providers which
ignore preconditions. Moto is emulator evidence only.
- Dirty output is bounded and unconfirmed until S3 publication. Storage
failure leaves persistence incomplete while task results and usage can
settle.
- Trace expiry is not yet represented by Cloud's external wake contract.
Outstanding trace deadlines conservatively block sleep.
- Automatic history copying requires the original source volume. Keep it
until all retained resources pass verified import and clean-filesystem
reads. Stop old binaries from creating new local-file runs. Missing or
corrupt history remains blocked and visible in server logs. Rollback
requires a compatible reader against the same external state. Ordinary
object deletion does not erase retained historical object versions or
backups.

## Model Used

OpenAI Codex, `gpt-6.1-sol`, with reasoning, tool use, code execution,
and browser verification. The runtime did not expose a context-window
size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-30 15:17:35 -05:00