Commit Graph
7 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 bf514759f9 fix(run-log): answer 404 for an object run log that was never written (#20539)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each heartbeat run keeps a run log, and the board reads it through
`GET /api/heartbeat-runs/:id/log`
> - A run can store its log in object storage (`s3_segments`)
> - When a run fails before its log is written (for example in
`runOutput.begin`), the object store throws a plain `Error("Object run
log not found")`
> - The route turns that plain error into a 500, so the board shows a
server failure for a log that simply does not exist
> - The local file store already answers 404 `Run log not found` for the
same case, and the UI treats a 404 as an unavailable log
> - This pull request makes the object store throw the same 404
> - The benefit is that a missing object run log looks like a missing
log, and a real storage failure still answers 500

## Linked Issues or Issue Description

**What happened?**
`GET /api/heartbeat-runs/:id/log` answered 500 `Internal server error`
for an `s3_segments` run whose log was never written. The server error
was `Object run log not found` from the object run-log store.

**Expected behavior**
The route answers 404 `Run log not found`, the same as the local file
store. The transcript view already treats a 404 for a finished run as a
missing log, and the log viewer already treats it as an unavailable log.
A real storage failure still answers 500.

**Steps to reproduce**
1. Use S3 storage for run logs.
2. Make a run fail before its run log is written (for example in
`runOutput.begin`).
3. Read `GET /api/heartbeat-runs/<run id>/log`. It answers 500.

**Paperclip version or commit**
`master` at adb0e77a4c.

## What Changed

- `server/src/services/object-run-log-store.ts`: a read of a log with no
head (never written or deleted) throws `notFound("Run log not found")`
(HTTP 404) instead of a plain `Error`.
- `server/src/__tests__/agent-live-run-routes.test.ts`: route tests
drive the real object run-log store behind the route. A never-written
log answers 404. A store read failure still answers 500. A board user of
another company gets 404 and the store is not read.
- `server/src/services/object-run-log-store.test.ts`: the never-written
read test now expects the 404 error.

## Verification

- Red: with the old store, `answers an object run log that was never
written` fails with `expected 500 to be 404`.
- Green: `cd server && PAPERCLIP_IN_WORKTREE=false npx vitest run
src/__tests__/agent-live-run-routes.test.ts
src/services/object-run-log-store.test.ts` passes (120 tests).

## Risks

- Low risk. Other readers of the store (`feedback.ts`, issue comment
derivation, export) already catch any read error. They do not match the
old message text.
- The response for a missing object log changes from 500 to 404. The
company access checks run before the store read, so they do not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 07:24:42 -06:00
DottaandClaude Opus 5.5 adb0e77a4c fix(run-log): read a resumed run's own log while the store still hides its new head (#20523)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In cloud mode, run logs use the `s3_segments` store: an object byte
journal whose `head.json` names the current writer and the committed
bytes
> - When a run resumes after a restart, `runOutput.begin` claims the
journal (this publishes a new `head.json`) and then reads the existing
log prefix to find where native log export continues
(`nativeRunLogPrefixCursor`)
> - On an S3-compatible store with read-after-write lag, the `head.json`
that the claim just published can read as `NoSuchKey` for some seconds
> - The read then throws "Object run log not found", and the resumed run
settles as `adapter_failed` and needs a Retry
> - This pull request makes a read through a live writer's own handle
use that writer's head when the store does not show the head yet, and
makes the one-time source-prefix proof step treat a hidden head as
"proof unavailable" instead of "another writer"
> - The benefit is that a run that resumes right after a restart
survives a store that shows new objects late, with no added wait

## Linked Issues or Issue Description

Refs #20509 (the same lag on the journal claim).

**What happened?**
In restart-chaos wave w-20261110T095205-s701 (local, `--storage s3
--storage-faults ST7`, K1/K2, on master with #20504, #20509 and #20514),
two resumed runs (t03 run 4f4fc7bc, t04 run c92c53d8) failed in
`runOutput.begin`. The error was "Object run log not found" from
`object-run-log-store.ts` `read`, called by `nativeRunLogPrefixCursor`
(`native-run-log-export.ts:314`). Both runs settled `adapter_failed`,
and the grade was R3 `retry_needed`. ST7 makes a GET or HEAD of a
just-written key answer 404 `NoSuchKey` for 5–30 s.

**Expected behavior**
A resumed run reads the log prefix that it just claimed and continues. A
store that shows a new object late must not look like a missing log.

**Steps to reproduce**
Write a run log, begin a successor writer on a new store instance, then
make `head.json` read as `NoSuchKey`. `read(successor)` throws "Object
run log not found".

## What Changed

- `server/src/services/object-run-log-store.ts`: `read()` reads the head
from the store as before. When the store says the head does not exist
and this process has a live writer for that handle, it uses the writer's
head. The writer published that head, so it proves the log exists. A
handle without a live writer still fails at once with "Object run log
not found".
- `server/src/services/object-byte-journal.ts`:
`recordVerifiedSourcePrefix` reads the head again to confirm ownership
before it publishes the optional source-prefix proof. When that head
reads as missing, it now throws
`ObjectJournalSourcePrefixProofUnavailableError` (the existing "proof is
optional" result). The store then continues without the proof, and the
next resume scans the prefix again. A visible head with a different
etag, a closed head or a deleted head still fails as
`ObjectJournalConflictError`. Later appends still publish the head with
a conditional write against the writer's etag, so another writer is
still detected.
- `server/src/services/object-run-log-store.test.ts`: a test hides the
successor's just-published `head.json` for the full resume sequence that
`nativeRunLogPrefixCursor` runs against the store (`read`, then
`recordVerifiedSourcePrefix`). Before this change, `read` fails with
"Object run log not found"; with only the read fix, the proof step fails
with "The output journal is owned by another writer". After both, the
read returns records 1–3, the proof is marked unavailable, and a later
append and read return records 1–4. A second test checks that a read of
a log that was never written still fails at once.

## Verification

- `cd server && npx vitest run src/services/object-byte-journal.test.ts
src/services/object-run-log-store.test.ts
src/services/object-run-log-store-tail-read.test.ts
src/services/run-log-store.test.ts
src/services/run-log-store-cancellation.test.ts`: 87 pass. The new
resume test fails with "Object run log not found" without the read fix,
and with "The output journal is owned by another writer" without the
proof fix.
- `cd server && npx vitest run
src/services/native-runtime/native-run-log-recovery.test.ts
src/services/native-runtime/native-run-log-watermark.test.ts`: 27 pass.
- `cd server && npx tsc --noEmit -p tsconfig.json` passes.
- Live: a local ST7 rerun (seed 701) on a build with this fix follows in
a comment.

## Risks

- Low risk. The read change only affects a read whose store head is
missing and whose handle has a live writer in this process. All other
reads behave as before, including fetch counts and the cache.
- When the head is hidden during the proof step, the proof is skipped
for this attempt. The next resume of the same run scans the log prefix
once more. This is the existing degraded path for a proof outage.
- The fix does not wait for lagged segment objects. A segment that the
previous server wrote less than the lag window before a restart can
still read as `NoSuchKey` during the prefix scan. The wave evidence
shows only the head lag. A broad segment wait broke existing failure
tests, so it is not part of this change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 06:40:58 -06:00
Nicky LeachandClaude Opus 5.5 0e3fdebea8 perf: remove repeated reads and checks from run preparation (#20358)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before each run starts, Core prepares the run: it opens the run log,
resolves skills, and prepares the AI runtime
> - Every run pays this preparation time before the agent can begin, for
a new task or a follow-up, on a cold or a warm computer
> - A trace on a hosted test deployment showed some preparation steps
that repeat a read or a check that a later step does again
> - This pull request removes three of these repeated steps and keeps
the check that each one duplicated
> - The benefit is that each run starts sooner, with no new mechanism
and no change to what a run can see or write

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Run preparation (`task.prepare`). On a hosted test deployment it takes
about 2.8 to 3.6 s. Skills preparation has a P50 of 275 ms (1,161 ms on
a deployment with more skills and projects).
`heartbeat.prepare.run_output` takes 386 to 724 ms.

**Subsystem affected**
Run logs in object storage (`object-byte-journal`,
`object-run-log-store`) and runtime skill delivery (`company-skills`,
`skill-execution-content`).

**Current behavior**
- An object journal claim is preceded by a head read. Its only use is to
give the current owner to `claim()`. `claim()` then reads the head
again, and it rejects the claim only when the two reads differ. The run
log, provider trace capture, and run evidence migration all do this.
- Run log `begin` reads the source-prefix proof object for every claim.
For a new run log, no proof can exist, so the read always returns
"missing".
- Each delivered skill version calls `withVersion`, which authorizes the
skill again for the run's actor. The skill row was read with the same
actor's visibility condition in the same call. With an agent actor this
repeats the company source and project permission reads once per
version.

**Proposed behavior**
- `claim()` reads the head once. It still refuses a deleted journal. The
conditional head write still rejects any change after that read, and a
stale writer is still fenced.
- `begin` does not read a proof for a first claim (generation 1). A
resumed log still reads and validates its proof, so a changed or
malformed proof still fails closed.
- Runtime skill delivery authorizes each skill once, when it reads the
selected rows. Other callers of `withVersion` keep their own check.

**Reason and benefit**
Each removed step protects nothing that a later step does not protect
again:
- Head read: the claim builds on its own head read, and the conditional
write enforces it.
- Proof read: a proof is written only by a writer that holds this log's
head. The journal never removes a head (deletion leaves a tombstone that
`claim()` refuses). So a first claim cannot find a proof.
- Skill check: every caller of `materializeVersionSnapshot` passes a row
that it read through the same actor's visibility condition
(`readFullSkillRows`, `getById`, `listFull`).

Expected saving per run: two object-storage GETs on `run_output` (about
100 ms each on the trace), and the per-version visibility reads in
skills preparation (4 versions per run on the test deployment, about 388
ms per version on a heavier deployment).

**Breaking changes**
None. `claim()` no longer takes `expectedOwner`. It is an internal API
with three callers in this repository, all updated.

Refs #20284 (it made run-log tail reads parallel and cached; this change
covers the begin path).

## What Changed

- `object-byte-journal`: remove the `expectedOwner` input and comparison
from `claim()`. Remove the extra head read in its three callers (run log
`begin`, provider trace capture, run evidence migration).
- `object-run-log-store`: `begin` reads the source-prefix proof only
when the claimed head has `generation > 1`.
- `skill-execution-content`: split `withVersion` into the access check
and `withReadableVersion`. `withVersion` keeps the check for its other
callers.
- `company-skills`: `materializeVersionSnapshot` uses
`withReadableVersion`, because each of its callers already read the
skill row under the actor's visibility condition.
- Tests:
- A new run log `begin` reads `head.json` once and reads no proof. A
resumed log reads `head.json` once and reads its proof once.
- Runtime skill delivery for an agent actor does no per-version source
permission reads.

## Verification

- `cd server && npx vitest run src/services/object-byte-journal.test.ts
src/services/object-run-log-store.test.ts
src/services/object-run-log-store-tail-read.test.ts
src/services/run-log-store.test.ts
src/services/provider-trace-object-store.test.ts`
- `cd server && npx vitest run
src/__tests__/company-skills-runtime-selection.test.ts
src/__tests__/company-skills-runtime-concurrency.test.ts
src/__tests__/heartbeat-runtime-skills.test.ts
src/__tests__/company-skills-service.test.ts
src/__tests__/managed-skill-content.test.ts` (embedded Postgres)
- `cd server && pnpm run typecheck`
- Each new test fails on the base commit: the run log test sees 2 head
reads and 1 proof read; the skill test sees 1 source permission read per
delivered version.
- Existing tests still pass for: stale-writer fencing after a takeover,
a deleted journal that refuses a claim, a malformed proof that fails
closed, and a denied project-scoped skill that is not delivered.
- I did not run the full `pnpm test:run`. CI covers it.

## Risks

- Low. Two concurrent claims of the same journal: before, the second
claim failed when the first published between its two head reads. Now
the later claim takes over, and the earlier writer's next append fails
with a conflict. This is the same result as when the later claim starts
after the first publish. Writer authority (`assertAuthority`) is
unchanged.
- A skill whose visibility is revoked in the moment between its row read
and its materialization in the same call is still delivered for that
run. This was already true for a revocation just after the old
per-version check.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 13:06:12 -08:00
e482f0f632 fix(server): shed only retryable object-journal reads under load (#19176)
Client polling can exhaust the shared object journal queue and reject log persistence. Bound ordinary object-log and trace reads, keep persistence and internal verification queued, and return retryable 503 responses when reads are busy. Leave feedback exports unsent until every log page is available.

Validation: 72 latest-master affected tests and 10 workflow tests pass locally, with the workflow fixture passing strict TypeScript checks. The preceding broader run passed 280 server tests with two opt-in S3 skips. All applicable checks on d0a937768d are green (56 success, three skipped); exact-head Apex review is 5/5 with no unresolved findings. Background persistence still has no fixed queue bound during a prolonged storage outage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-06 13:47:25 -08:00
DottaandPaperclip 31266d7159 fix(logs): drain existing admissions before finalization (#19055)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Run logs retain execution output for operators.
> - Object storage uses bounded admission and asynchronous uploads.
> - A diagnostic can wait for capacity when its native execution ends.
> - Finalization closed the writer before that existing admission could
finish.
> - This pull request joins those admissions before it closes the log.
> - The benefit is complete accepted output with the existing storage
bounds.

## Linked Issues or Issue Description

Refs #18944 and #18327.

**What happened?**

A runner close diagnostic began object-log admission before its recorder
retired. The call waited at the 8 MiB pending-byte bound. Completion
then closed the writer. The waiting call failed with `Object run log
writer is closed`. A valid final log hash covered the stored prefix, but
the diagnostic was absent.

**Expected behavior**

Finalization must retain calls that started before closure. It must
reject new calls after closure. Cancellation and storage failures must
still settle without a success claim.

**Steps to reproduce**

1. Use the actual recorder and segmented object store with a controlled
conditional upload.
2. Fill the pending-byte bound. Start native warm-session retirement
while the diagnostic waits for capacity.
3. Complete the run through the real completion handler. Release the
upload.
4. Check the diagnostic bytes, final hash, source coverage, and warning
rows.

## What Changed

- Track active admission promises in each object-log writer.
- Reject new producers when finalization starts. Join existing bounded
admissions, then flush their tail before manifest closure.
- Add actual completion coverage and store controls for failure, Stop,
global capacity, and independent writers.
- Clarify the finalization contract in the storage documentation. Keep
feature maps unchanged for this routine fix.

## Verification

- Before the fix, the actual completion probe lost a pre-retirement
close diagnostic and emitted `object_log_persistence_failed`. Useful
canonical output, source coverage, and the final prefix hash remained
valid.
- After the fix, the exact probe retains the diagnostic and checks the
stored bytes and full hash. Ordinary completion and stalled local append
controls also pass.
- Focused store controls pass, including shared 64 MiB capacity,
new-producer rejection, storage failure, Stop, and independent writer
progress.
- Production and test source at
`bc03f8d53a721f01f80a12105a69d8583865e651`: 3 actual completion controls
and 53 store, recorder, journal, and diagnostic tests pass. Run with
`node node_modules/vitest/vitest.mjs run` from `server/`; select the
four safety files plus the completion file with `-t "retired recorder"`
in a separate invocation.
- Initial treatment repository typechecks and build pass. Current map
and module checks pass. A broader four-suite run had 99 passing
assertions but failed its existing 30-second completion teardown hook. A
later isolated completion run passed all 61 cases at the same limits.
The original teardown cause remains unresolved; no timeout was changed.
- Repository typechecks and build pass at that source commit. Current
head `6652adc7f3379ae250f7b726dad6c58c8482410f` removes only the
incidental map paragraph; production and test source are identical. All
current-head checks are terminal: 54 successful check runs, one
successful legacy status, and three skipped checks. Required `ci /
verify` and `ci / e2e` pass. Greptile reviewed exact current head at
5/5. Both review threads are resolved. GitHub reports the PR clean and
mergeable. Full repository tests stopped in chat shard 2: a GitHub
recovery fair-batch test expected no provider requests but observed `GET
/app/hook/config`. Its exact unchanged case passes in isolation. The
original cause remains unresolved; this full-suite run is red.
First-head remote server lane 2 failed a native-reassessment backlog
test: its quiet check observed two active work items instead of zero.
The exact unchanged case passes locally. That original CI cause remains
unresolved; no job retry or timeout change was made. Other first-head CI
lanes passed. The first native review rated the earlier source head 4/5
for the same removed map note. That is not a current-head assessment.
Current GitHub rules require zero approving reviews and only the two CI
checks above; a fresh native advisory re-review is optional and remains
unavailable while its earlier run has an unresolved workspace/accounting
obligation. That obligation is tracked separately and is not claimed
recovered.
- Removed the incidental map paragraph identified by Greptile and the
native review. The follow-up contains no runtime or test changes.
- This proves the local production-path mechanism. It does not attribute
any specific hosted warning to this cause. Hosted replay and served
revision checks remain separate acceptance work.

## Risks

Finalization now waits for existing bounded capacity admissions. Storage
cancellation and the existing operation deadlines still apply. Tests
check that failed uploads and Stop settle, and that a separate writer
can make progress. No capacity limit, timeout, or execution authority
changes.

## Model Used

OpenAI `gpt-6.1-sol`, through Codex, with reasoning and tool execution.
Context window size is not exposed by the host.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (56 focused checks; the
full-suite red remains documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 16:34:36 -06:00
DottaandPaperclip 3910352c92 feat(runner): add experimental Cursor Cloud provider (#18068)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Runner gives each provider durable execution, tool, and completion
contracts.
> - Cursor Cloud currently has a direct adapter outside that native
contract.
> - Cloud work needs exact ownership across HTTP, SSE, Stop, and
recovery.
> - This PR adds a qualified Cloud API driver and verified proxy to
Runner.
> - Product setup and hosted placement remain separate dependent
changes.

## Linked Issues or Issue Description

Refs #5664, #11803, #17299, and #17471. Product integration is #18069.
Managed-only API placement is #18310. The related direct-adapter
model/transcript work is not replaced. No duplicate native Cursor Cloud
integration was found.

## What Changed

- Add native `cursor_cloud` / `cursor_cloud_api` with explicit
model/repository validation, bounded HTTP/SSE, durable per-turn
checkpoints and a qualified Node proxy launched through the Rust facade.
- Commit terminal custody before publication/release; coalesce
consumers, preserve ambiguous create/Stop/recovery fences, reject late
tools, and distinguish definitely-unsent oversized requests from
ambiguous responses.
- Support scoped cold attachment, current authority, reconnect and
verified artifact staging. Keep unsupported warm/external and Windows
paths closed.
- Preserve early source cursors and verify immutable legacy prefixes
before suffix replay, including supported compaction/masking. Optional
proof publication failure falls back to complete cold verification.
- Preserve canonical records at the durable native producer with
JSON-safe privacy and the existing 8MiB bound; drain admitted successor
flushes. Reconcile upstream semantic/pack/schema/lifecycle changes and
retain S3 and legacy CLI checks.

## Verification

Core head `37c442ec8dd4`: all 66 check contexts are terminal
successful/neutral/skipped, including required `ci / verify` and `ci /
e2e`; fresh exact-head Greptile is 5/5 with zero unresolved threads.
Dependent Product `81e828d20a77` and Top `dd2959689627` include the
later client handoff correction and are running their own fresh gates.

Core: 47/47 Rust, 4/4 compiled-native retirement, 19/19 settlement and
normal Runner build passed. Runtime Top `9af1a5de42a2`: build and
recursive typecheck passed. Earlier source-scoped evidence includes 398
focused tests, five relay tests and token/type gates. The 251-assertion,
7/8-suite run failed physical PostgreSQL teardown; the unchanged stable
WSP suite passed 128 through the documented wrapper.

The downstream fixture corrections passed 7/7 browser and 121/121
startup tests without timeout/oracle relaxation; a broken-unsubscribe
control fails. Hosted runtime `9af1a5de42a2` has a passing normal build
and recursive typecheck. Latest dependent heads additionally contain a
pure client fix for stale live/history overlap, with 48 focused tests,
UI typecheck and token gates passing. Its new client is not deployed in
the frozen hosted image; server/native/API/provider source is unchanged
by these four downstream files.

Fresh hosted Linux admission on `9af1a5de42a2` retired its owned
runner/proxy/socket with zero prompts. The saved configuration survived
reopening, and Test authenticated the bound key/catalog/selected model
without a prompt. Artifact with `claude-sonnet-4-5` completed,
downloaded the exact 27 bytes, matched 121 canonical identities against
complete logs and independently reached FINISHED. The active Stop retry
was clicked within its immutable boundary, produced one canonical
interrupted turn and acknowledged run-only cancellation, retained a
valid 35.076s quiet window (maximum gap 4.584s), and independently
remained CANCELLED/artifact-absent beyond the original 180-second
command deadline. All provider-observer credential callbacks, DB clients
and loaders closed; accounting is settled and explicitly unpriced. The
first fresh Stop retains its failed 10.373s sampling gap and every
sample.

On the upgraded9af server, the historical F226 artifact was reopened and
downloaded with its exact original hash after its provider agent had
been deleted/GET404 earlier today. Historical F226 full Artifact/Stop
cleanup, Question, corrected Plan and reconnect retain their original
identities. Deleting the three newly created disposable provider agents
and the new-case post-delete grade remain blocked on explicit user
approval after automatic review rejected expanding the two original
approved IDs. No new-case cleanup or complete fresh matrix pass is
claimed; this is separate from the directly observed task/Stop behavior
and retained exact cleanup proof.

Retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook
failures; Mac 17 SQL failures/skips; earlier startup-title, title-save,
custody/Stop and deletion failures with unknown causes and bounded
recoveries. `dea023` Artifact passed 17 checks but Stop failed its false
warning and 50 duplicate log identities. Pre-repair ACK failure and
original Core CI failure remain recorded; retry alone establishes no
cause or full local-suite pass. Provider dollar cost stays UNKNOWN.

## Risks

Experimental: Cursor independently acknowledges neither the effective
model nor instruction digest before the first prompt. The caller’s
explicit values are submitted and behavior checked live. Unpriced
provider dollar cost stays UNKNOWN.

Only qualified per-turn controllers are admitted. Uncertain work and
incomplete cleanup stay fenced. Existing damaged/duplicate archives
remain failed evidence. Prefix proof adds bounded object-store I/O; the
8MiB producer bound does not guarantee future providers’ output fits.

## Model Used

OpenAI Codex, GPT-6 family, assisted with reasoning, repository tools,
code execution, browser verification, and delegated code review. The
serving model’s exact API ID and context window are not exposed. No
value is inferred from the model catalog.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used and the limits of available model
metadata
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have linked public issues or described the issue in this PR
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch describes the change and contains no internal ticket ID
- [x] I have run the scoped local tests reported above; pending gates
are stated separately
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented the risks
- [x] All required CI gates pass on the current head
- [x] Fresh Greptile is 5/5 with no unresolved findings on the current
head
- [x] I will address all reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 08:38:57 -06:00
DottaandPaperclip dbff3747b8 feat(server): persist run logs and provider traces in S3 (#18327)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Operators use run logs and restricted provider traces to inspect
agent work.
> - Local log and trace files depend on the web server's persistent
volume.
> - That dependency prevents overlapping volume-backed Railway
deployments.
> - This PR reuses the instance S3 store for batched log and trace
payloads with conditional publication.
> - Existing Postgres rows retain their contracts; native export
recovery reuses events already committed before Runner acknowledgement.
> - S3-configured servers automatically copy and verify retained history
before a later deployment cutover.

## Linked Issues or Issue Description

**Subsystem affected**

Server run logs, restricted provider traces, storage providers, and
migration tooling.

**Problem or motivation**

Replacing the web server and its disk must preserve committed log output
and retained traces. The current S3 mirror can repeatedly upload a
growing file. Moving those payloads into Postgres would add database
writes, WAL, replication, and backup work.

**Proposed solution**

Reuse the existing instance storage configuration, S3-compatible
provider and credential chain. Store new run logs under run-logs/ and
opted-in raw traces under provider-traces/, below the existing prefix.
Cloud keeps its existing bucket and bucket-scoped credentials per stack.
Add no new configuration switches. Store immutable payload segments and
bounded binary range indexes directly in S3. Publish each batch with an
ETag-conditional head. Keep existing database ownership and lifecycle
records. Recover canonical native exports from their existing validated
control event source, checkpointing export progress in S3. Add no new
payload writes to Postgres. Preserve local defaults and migrate settled
history with verified byte counts and digests.

**Alternatives considered**

Postgres payload chunks add control-plane load. Repeated whole-file
mirrors amplify transfer work. The first index prototype left superseded
index nodes; the final binary index keeps all normally published nodes
reachable.

**Roadmap alignment**

This is a storage prerequisite for replacing the main web/API server. It
does not enable deployment overlap, detach a volume, or change native
Runner protocol acknowledgements. The maintainer approved this
implementation scope. ROADMAP.md has no duplicate implementation item.

Related PR: #16770 improves the existing in-flight mirror. This PR
preserves that legacy backend and uses the existing instance S3 profile
for new journals.

## What Changed

- Automatically select S3 segments for new logs and opted-in traces when
instance file storage uses S3. Reuse bucket, endpoint, region, prefix,
path style and credentials. Remove separate backend switches. Keep
local-disk instances local.
- Share the existing storage configuration resolver with product files.
Preserve config-file settings, environment precedence and the existing
root prefix. Keep pre-existing run-log mirror settings for legacy
history.
- Add bounded buffering, conditional ownership fencing, immutable range
reads, stable byte offsets, and completion hashing.
- Dispatch all existing log consumers through the recorded backend.
Coalesce output progress writes and publish live log messages after
storage receipts.
- Recover interrupted canonical native exports from their existing
durable source. Exclude separate credential, cleanup, and reprojection
records. Replay missing callback ranges and deduplicate late delivery.
Repair settled archives after controller lease expiry. Bound shared
repair work and stop it when all readers cancel. Serve the published
prefix when best-effort repair fails.
- Add encrypted optional trace channels, replacement reads, durable
deletion tombstones, and deadline-driven local/external expiry cleanup
independently of heartbeat scheduling. Stream historical source and
remote verification. Propagate worker cancellation through trace open,
conditional-write probing, upload, restore and reads. Fence deletion
even before the first object head exists; abort and drain owned
captures.
- Retry transient storage errors, distinguish 409 from failed
preconditions, and backpressure bounded log admission. Track interrupted
trace imports explicitly so ordinary local deletion stays independent of
S3.
- Automatically migrate retained history through the existing delivery
coordinator. Process one log and one trace per batch. Retry unresolved
history with backoff and stop scans after completion. Keep the
inventory/import command as optional diagnosis and repair. Verify source
and remote bytes before switching references. Verify finalized native
size and digest before setting its export cursor. Require ordered source
coverage when completeness metadata is missing. Preserve original trace
expiry. Block active, missing, corrupt, and unproven sources.
- Preserve identity and run-secret redaction in live and recovered
native logs. Register runtime log redaction secrets once through the
existing encrypted registry. Keep agent private keys in their existing
owner.
- Add storage, migration, retention, UTF-8, and route-privacy coverage,
plus operating documentation and qualification tooling.
- Add no schema, payload table, per-record SQL, production dependency,
or lockfile change.

## Verification

- Final fixes at `b3a44c898e` passed all 116 selected startup,
retention, attribution and cancellation controls. The disabled-scheduler
cleanup and concurrent-repair defects reproduced red before the fixes. A
timed-out repair still prevents later storage reads.
- The merged source passed 124 storage and real-Postgres Cloud
configuration tests. These include automatic history migration, existing
environment/file vault keys, source-directory removal, five real-SDK
stalled-request cancellation cases, shared capability-probe cancellation
and multi-chunk trace ordering. Saved Cloud S3 settings work after the
legacy config file is gone.
- Workspace typecheck and full build passed at the merged parent
`19f702d780`. Server typecheck and build passed again at `b3a44c898e`.
- Actual app acceptance: finish a process-agent run through the normal
API, stage its retained local-file input, then restart. Automatic
migration reports one migrated item and zero blockers. Remove the
original web home and restart at `b3a44c898e` with an empty home, the
existing S3 profile and the existing vault key file. The normal log API
preserves all five producer lines and exactly 788 bytes, with SHA-256
`74d9ae469da28bfe89b9b50a0e132aec24fe9e6d686496ed5e4ffa6167151036`. No
log-specific storage settings are present.
- Actual expiry acceptance: with heartbeat scheduling disabled, a trace
remains unexpired after startup and later reaches its fixture deadline.
The worker marks the row deleted and tombstones both S3 channel heads.
The retained run log stays byte-identical. The process fixture emits no
provider protocol frames, so nonempty encrypted payloads are qualified
separately through the real SDK round trips.
- Isolation tests use identical run identities in two stack buckets.
Cleanup in one existing prefix leaves another prefix, another company
and product files intact. Read-only Cloud source inspection confirms a
bucket and bucket-scoped credentials per stack. Four existing Cloud
storage environment-contract tests passed. No Cloud provisioning changed
and no new live vendor permission probe was performed.
- Earlier qualification passed 1,014 affected storage/native tests and
27 privacy-route tests with isolated Docker Postgres. An abrupt process
test commits 1,000 records, receives receipts, dies by SIGKILL and loses
its directory; another process reads every byte and continues. The real
Rust Runner experiment acknowledges 32 records while the export cursor
is zero, then continues the same Runner PID after controller loss. All
64 records export once with exactly 64 existing control rows. Repeated
snapshot/Stop commands completed in 162/81 ms.
- Emulator workload: 1,000 burst records used two batch PUTs. Eight
steady streams produced 240 records with 104 conditional PUTs and 72
GETs over 6.9 seconds. These are synthetic operation counts, not fleet
sizing or provider prices.
- Failure evidence is retained. Compact native events over 64 KiB,
missing file-key support, disabled-scheduler retention and repair
failure handling all reproduced before their fixes. Fixture repairs
addressed cold module setup, localhost/IPC permissions, the S3
bucket-name limit and a mode-0700 crash-test directory. Full local `pnpm
test:run` was stopped after host PostgreSQL semaphore exhaustion;
affected suites used isolated Docker databases instead. Native package
library symlinks were restored with the package's standard hydration
helper.
- The `19f702d780` browser session-recovery case timed out while saving
a task title. All eight browser shards passed on `b3a44c898e`, including
that case, with no UI behavior or assertion change. Fresh-head CI passed
at `b3a44c898e`: 55 successful checks, three skipped checks and one
neutral check. The branch is conflict-free.
- Map and module-boundary checks passed. Current-head Greptile is 5/5
with zero unresolved review threads.

Reproduce the opt-in storage suites with
`PAPERCLIP_LOG_TEST_S3_ENDPOINT` and a dedicated
`PAPERCLIP_RUN_EVIDENCE_MIGRATION_DATABASE_URL` ending in `_test`. See
`doc/run-evidence-storage.md` for operating limits, automatic migration,
optional diagnostics and rollout qualification.

## Risks

- Keep the existing per-stack bucket, prefix and vault key stable across
replacements. Relocating that storage is a separate whole-instance
operation.
- This release automatically changes new log storage for existing
S3-configured instances. Local-disk instances remain local. Canonical
native source replay is qualified with a deterministic real Runner;
actual remote execution and production-provider qualification remain
rollout gates. Generic server-only diagnostics and optional raw trace
tails without a surviving source can be incomplete after a crash. This
does not establish full zero-downtime deployment.
- Before deploying this release, qualify the actual S3 provider. It must
enforce conditional writes and pass read, version, deletion, workload,
and latency qualification. A capability probe rejects providers which
ignore preconditions. Moto is emulator evidence only.
- Dirty output is bounded and unconfirmed until S3 publication. Storage
failure leaves persistence incomplete while task results and usage can
settle.
- Trace expiry is not yet represented by Cloud's external wake contract.
Outstanding trace deadlines conservatively block sleep.
- Automatic history copying requires the original source volume. Keep it
until all retained resources pass verified import and clean-filesystem
reads. Stop old binaries from creating new local-file runs. Missing or
corrupt history remains blocked and visible in server logs. Rollback
requires a compatible reader against the same external state. Ordinary
object deletion does not erase retained historical object versions or
backups.

## Model Used

OpenAI Codex, `gpt-6.1-sol`, with reasoning, tool use, code execution,
and browser verification. The runtime did not expose a context-window
size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-30 15:17:35 -05:00