feat(server): persist run logs and provider traces in S3 (#18327)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Operators use run logs and restricted provider traces to inspect
agent work.
> - Local log and trace files depend on the web server's persistent
volume.
> - That dependency prevents overlapping volume-backed Railway
deployments.
> - This PR reuses the instance S3 store for batched log and trace
payloads with conditional publication.
> - Existing Postgres rows retain their contracts; native export
recovery reuses events already committed before Runner acknowledgement.
> - S3-configured servers automatically copy and verify retained history
before a later deployment cutover.

## Linked Issues or Issue Description

**Subsystem affected**

Server run logs, restricted provider traces, storage providers, and
migration tooling.

**Problem or motivation**

Replacing the web server and its disk must preserve committed log output
and retained traces. The current S3 mirror can repeatedly upload a
growing file. Moving those payloads into Postgres would add database
writes, WAL, replication, and backup work.

**Proposed solution**

Reuse the existing instance storage configuration, S3-compatible
provider and credential chain. Store new run logs under run-logs/ and
opted-in raw traces under provider-traces/, below the existing prefix.
Cloud keeps its existing bucket and bucket-scoped credentials per stack.
Add no new configuration switches. Store immutable payload segments and
bounded binary range indexes directly in S3. Publish each batch with an
ETag-conditional head. Keep existing database ownership and lifecycle
records. Recover canonical native exports from their existing validated
control event source, checkpointing export progress in S3. Add no new
payload writes to Postgres. Preserve local defaults and migrate settled
history with verified byte counts and digests.

**Alternatives considered**

Postgres payload chunks add control-plane load. Repeated whole-file
mirrors amplify transfer work. The first index prototype left superseded
index nodes; the final binary index keeps all normally published nodes
reachable.

**Roadmap alignment**

This is a storage prerequisite for replacing the main web/API server. It
does not enable deployment overlap, detach a volume, or change native
Runner protocol acknowledgements. The maintainer approved this
implementation scope. ROADMAP.md has no duplicate implementation item.

Related PR: #16770 improves the existing in-flight mirror. This PR
preserves that legacy backend and uses the existing instance S3 profile
for new journals.

## What Changed

- Automatically select S3 segments for new logs and opted-in traces when
instance file storage uses S3. Reuse bucket, endpoint, region, prefix,
path style and credentials. Remove separate backend switches. Keep
local-disk instances local.
- Share the existing storage configuration resolver with product files.
Preserve config-file settings, environment precedence and the existing
root prefix. Keep pre-existing run-log mirror settings for legacy
history.
- Add bounded buffering, conditional ownership fencing, immutable range
reads, stable byte offsets, and completion hashing.
- Dispatch all existing log consumers through the recorded backend.
Coalesce output progress writes and publish live log messages after
storage receipts.
- Recover interrupted canonical native exports from their existing
durable source. Exclude separate credential, cleanup, and reprojection
records. Replay missing callback ranges and deduplicate late delivery.
Repair settled archives after controller lease expiry. Bound shared
repair work and stop it when all readers cancel. Serve the published
prefix when best-effort repair fails.
- Add encrypted optional trace channels, replacement reads, durable
deletion tombstones, and deadline-driven local/external expiry cleanup
independently of heartbeat scheduling. Stream historical source and
remote verification. Propagate worker cancellation through trace open,
conditional-write probing, upload, restore and reads. Fence deletion
even before the first object head exists; abort and drain owned
captures.
- Retry transient storage errors, distinguish 409 from failed
preconditions, and backpressure bounded log admission. Track interrupted
trace imports explicitly so ordinary local deletion stays independent of
S3.
- Automatically migrate retained history through the existing delivery
coordinator. Process one log and one trace per batch. Retry unresolved
history with backoff and stop scans after completion. Keep the
inventory/import command as optional diagnosis and repair. Verify source
and remote bytes before switching references. Verify finalized native
size and digest before setting its export cursor. Require ordered source
coverage when completeness metadata is missing. Preserve original trace
expiry. Block active, missing, corrupt, and unproven sources.
- Preserve identity and run-secret redaction in live and recovered
native logs. Register runtime log redaction secrets once through the
existing encrypted registry. Keep agent private keys in their existing
owner.
- Add storage, migration, retention, UTF-8, and route-privacy coverage,
plus operating documentation and qualification tooling.
- Add no schema, payload table, per-record SQL, production dependency,
or lockfile change.

## Verification

- Final fixes at `b3a44c898e` passed all 116 selected startup,
retention, attribution and cancellation controls. The disabled-scheduler
cleanup and concurrent-repair defects reproduced red before the fixes. A
timed-out repair still prevents later storage reads.
- The merged source passed 124 storage and real-Postgres Cloud
configuration tests. These include automatic history migration, existing
environment/file vault keys, source-directory removal, five real-SDK
stalled-request cancellation cases, shared capability-probe cancellation
and multi-chunk trace ordering. Saved Cloud S3 settings work after the
legacy config file is gone.
- Workspace typecheck and full build passed at the merged parent
`19f702d780`. Server typecheck and build passed again at `b3a44c898e`.
- Actual app acceptance: finish a process-agent run through the normal
API, stage its retained local-file input, then restart. Automatic
migration reports one migrated item and zero blockers. Remove the
original web home and restart at `b3a44c898e` with an empty home, the
existing S3 profile and the existing vault key file. The normal log API
preserves all five producer lines and exactly 788 bytes, with SHA-256
`74d9ae469da28bfe89b9b50a0e132aec24fe9e6d686496ed5e4ffa6167151036`. No
log-specific storage settings are present.
- Actual expiry acceptance: with heartbeat scheduling disabled, a trace
remains unexpired after startup and later reaches its fixture deadline.
The worker marks the row deleted and tombstones both S3 channel heads.
The retained run log stays byte-identical. The process fixture emits no
provider protocol frames, so nonempty encrypted payloads are qualified
separately through the real SDK round trips.
- Isolation tests use identical run identities in two stack buckets.
Cleanup in one existing prefix leaves another prefix, another company
and product files intact. Read-only Cloud source inspection confirms a
bucket and bucket-scoped credentials per stack. Four existing Cloud
storage environment-contract tests passed. No Cloud provisioning changed
and no new live vendor permission probe was performed.
- Earlier qualification passed 1,014 affected storage/native tests and
27 privacy-route tests with isolated Docker Postgres. An abrupt process
test commits 1,000 records, receives receipts, dies by SIGKILL and loses
its directory; another process reads every byte and continues. The real
Rust Runner experiment acknowledges 32 records while the export cursor
is zero, then continues the same Runner PID after controller loss. All
64 records export once with exactly 64 existing control rows. Repeated
snapshot/Stop commands completed in 162/81 ms.
- Emulator workload: 1,000 burst records used two batch PUTs. Eight
steady streams produced 240 records with 104 conditional PUTs and 72
GETs over 6.9 seconds. These are synthetic operation counts, not fleet
sizing or provider prices.
- Failure evidence is retained. Compact native events over 64 KiB,
missing file-key support, disabled-scheduler retention and repair
failure handling all reproduced before their fixes. Fixture repairs
addressed cold module setup, localhost/IPC permissions, the S3
bucket-name limit and a mode-0700 crash-test directory. Full local `pnpm
test:run` was stopped after host PostgreSQL semaphore exhaustion;
affected suites used isolated Docker databases instead. Native package
library symlinks were restored with the package's standard hydration
helper.
- The `19f702d780` browser session-recovery case timed out while saving
a task title. All eight browser shards passed on `b3a44c898e`, including
that case, with no UI behavior or assertion change. Fresh-head CI passed
at `b3a44c898e`: 55 successful checks, three skipped checks and one
neutral check. The branch is conflict-free.
- Map and module-boundary checks passed. Current-head Greptile is 5/5
with zero unresolved review threads.

Reproduce the opt-in storage suites with
`PAPERCLIP_LOG_TEST_S3_ENDPOINT` and a dedicated
`PAPERCLIP_RUN_EVIDENCE_MIGRATION_DATABASE_URL` ending in `_test`. See
`doc/run-evidence-storage.md` for operating limits, automatic migration,
optional diagnostics and rollout qualification.

## Risks

- Keep the existing per-stack bucket, prefix and vault key stable across
replacements. Relocating that storage is a separate whole-instance
operation.
- This release automatically changes new log storage for existing
S3-configured instances. Local-disk instances remain local. Canonical
native source replay is qualified with a deterministic real Runner;
actual remote execution and production-provider qualification remain
rollout gates. Generic server-only diagnostics and optional raw trace
tails without a surviving source can be incomplete after a crash. This
does not establish full zero-downtime deployment.
- Before deploying this release, qualify the actual S3 provider. It must
enforce conditional writes and pass read, version, deletion, workload,
and latency qualification. A capability probe rejects providers which
ignore preconditions. Moto is emulator evidence only.
- Dirty output is bounded and unconfirmed until S3 publication. Storage
failure leaves persistence incomplete while task results and usage can
settle.
- Trace expiry is not yet represented by Cloud's external wake contract.
Outstanding trace deadlines conservatively block sleep.
- Automatic history copying requires the original source volume. Keep it
until all retained resources pass verified import and clean-filesystem
reads. Stop old binaries from creating new local-file runs. Missing or
corrupt history remains blocked and visible in server logs. Rollback
requires a compatible reader against the same external state. Ordinary
object deletion does not erase retained historical object versions or
backups.

## Model Used

OpenAI Codex, `gpt-6.1-sol`, with reasoning, tool use, code execution,
and browser verification. The runtime did not expose a context-window
size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

doc/DATABASE.md

@@ -1545,6 +1545,15 @@ named `review_wait_benchmark...` can be supplied through
 `PAPERCLIP_TEST_DATABASE_URL` to compare another PostgreSQL major version.
 DataRow bytes exclude response metadata and transport overhead.
 
+## Object-backed run evidence
+
+Optional segmented run logs and raw provider traces store their payloads and
+range indexes directly in S3-compatible storage. Existing `heartbeat_runs` and
+`provider_trace_records` contain backend/reference and lifecycle metadata;
+there is no new payload table or per-segment SQL publication. Object manifests
+use conditional writes to fence publication. Existing structured events retain
+their database append contract. See [run evidence storage](run-evidence-storage.md).
+
 ## Cloud runtime configuration singleton
 
 The private `instance_settings` key `cloud-runtime-config:v1` holds a strictly

doc/DEVELOPING.md

@@ -3883,6 +3883,16 @@ in `_test` for `PAPERCLIP_MCP_SLEEP_TEST_DATABASE_URL`,
 also accepts `PAPERCLIP_INTERACTIONS_TEST_DATABASE_URL`. The suites migrate and write to those
 URLs; the sleep suite truncates company data between tests.
 
+## S3-backed run evidence
+
+New logs and opted-in provider traces automatically use the instance's existing
+S3 storage configuration, bucket, credentials and prefix, under `run-logs/` and
+`provider-traces/`. No extra storage switches are needed. Local-disk instances
+remain local. See [run evidence storage](run-evidence-storage.md) for required provider
+capabilities, stable keys, operating limits, automatic history migration and qualification.
+Do not remove a web volume until its complete retained evidence inventory and
+the independent disposable-server requirements are satisfied.
+
 ## Cloud startup configuration
 
 Cloud stores nonsecret startup choices in a private PostgreSQL singleton.

doc/plans/2026-10-30-external-run-logs-and-provider-traces.md

@@ -0,0 +1,567 @@
+# Durable run logs and provider traces for disposable web servers
+
+Status: implementation and PR qualification in progress. Runtime storage,
+history migration, and canonical native source replay are implemented using the
+instance's existing storage selection. Production-provider qualification remains
+a rollout gate.
+Owner: this Codex chat, through the reviewable PR. Date: 2026-10-30.
+Revision: object-storage-first, following Dotta's operational correction on 2026-10-30.
+Source revision inspected: `0c70cedce9a803bdc201324528abb35e423e7cd0`.
+Parent effort: near-zero-downtime Railway deployment of the main web/API server.
+
+## Goal and boundary
+
+After the main Paperclip web server and its local filesystem disappear, a
+replacement connected to the same Postgres, object storage, and secrets can
+read all acknowledged run-log output and retained provider traces, then continue
+recording the same remote execution without gaps or duplicate records.
+
+The ordinary journey is: open a task's run, inspect its output, replace the web
+server while its remote agent is working, and continue reading the run. An
+authorized administrator can still inspect, reveal, download, and delete a
+captured provider trace. Existing consumers of run output continue working.
+
+This is the **run logs and provider traces** slice of the parent plan. Railway
+still documents that volume-backed deployments cannot overlap, which makes
+removing durable filesystem dependencies necessary. This slice alone does not
+remove every volume dependency or prove zero-downtime HTTP deploys.
+[Railway volume limitations](https://docs.railway.com/volumes/reference#caveats).
+
+Firm requirements:
+
+- All retained historical logs and unexpired traces survive migration. Dotta
+  explicitly selected this on 2026-10-30. Do not shorten existing retention.
+- A successful durable append survives abrupt process and disk loss. An orderly
+  shutdown flush is useful but insufficient evidence.
+- Preserve company/task authorization, redaction, stable log byte offsets,
+  trace integrity checks, opt-in capture, existing limits, expiry, and deletion.
+- Store log and raw trace payloads directly in S3-compatible object storage.
+  Postgres holds small ownership/lifecycle/reference records only; no new
+  per-token or per-frame SQL, payload tables, or DB-to-S3 payload compaction.
+  Local files may be caches or unacknowledged buffers. Preserve local mode.
+- Keep task results, usage settlement, and optional diagnostics independent.
+- Preserve bounded work and existing sleep behavior.
+- Reuse existing instance storage settings and per-stack bucket credentials.
+  New logs and opted-in traces follow S3 file storage automatically, with their
+  own folders beneath the existing prefix. Add no required configuration.
+
+Dotta approved implementation on 2026-10-30. Delivery ends at a reviewable PR
+with relevant checks and review findings resolved. Deployment, volume removal,
+and production history migration require the parent effort's target and cutover.
+The acceptance matrix below remains the full storage-slice contract; an
+implemented primitive does not establish that every row has passed.
+
+Deferred to the parent effort: managed agent files, workspace recovery,
+accounting/credential journals, workspace-operation logs, readiness/draining,
+global live-event fanout, scheduler ownership, volume detach, and fleet rollout.
+These remain blockers to claiming the entire web server disposable. No changes
+to first-party Telemetry or operator OpenTelemetry export are proposed. Raw
+provider protocol traces here are a separate, restricted diagnostic store.
+
+## Current implementation and implications
+
+| Owner | Current behavior | Required change |
+| --- | --- | --- |
+| `server/src/services/run-log-store.ts` | Only `local_file`; local NDJSON append; best-effort final/in-flight S3 mirroring; local-first reads | Add an explicit external store with shared durable append and reads |
+| `server/src/services/heartbeat/run-output.ts` | Redacts/compacts before append; process-local output sequence; progress persisted on a 60-second cadence | Preserve stable source identities; batch output to S3; distinguish live delivery from durable receipts |
+| `server/src/services/heartbeat-run-events.ts` | Structured events already use atomic DB sequence allocation and native source deduplication | Preserve this authority; do not turn its event table into a duplicate blob store |
+| `server/src/services/provider-trace-store.ts` | DB metadata points to local native and `.rehydration` sidecars; 24-hour retention and 64 MiB capture limit | Remote payload storage, recoverable capture cursors, durable expiry/deletion obligations |
+| Runner trace sinks and `runnerd-codex-transport.ts` | Rust/TypeScript capture and interpretation producers use paths; TypeScript capture is bounded and fire-and-forget | Introduce a durable capture delivery boundary and preserve channel/frame lineage |
+| `doc/testing-native-remote-continuity.md` | Sparse checkpoints deliberately omit streaming output; acknowledged display output may be lost before app projection | Qualify replay/retention through web replacement without restoring per-token DB checkpoint writes |
+| Run readers | Heartbeat, issues, feedback, and customer-success inspection assume `local_file` | Resolve each persisted backend, including historical records, through one reader |
+
+Do not infer cloud durability from `RunLogStore.append()` succeeding today, or
+from a trace's local `acknowledgedDebugSequence`. Neither proves remote commit.
+Generic stdout/stderr and raw traces cannot be reconstructed from
+`heartbeat_run_events`. The implementation investigation found a narrower
+surviving source: native control ingestion already commits the exact validated
+PRP event before ACK. Its canonical `paperclip.prp.event` log export can replay
+that existing source with an S3 cursor, without adding payload writes. This
+replaces the tentative second Runner spool below for canonical native output.
+
+## Operational decision
+
+The original proposal used Postgres for payload chunks and then copied them to
+S3. That choice was premature. It adds payload writes, WAL/replication/backup
+bytes, index churn, deletion/vacuum work, and extra database connections to the
+same database serving the control plane. Batching would reduce transaction
+count but would not remove the byte load. No workload measurement justified it.
+
+Use **S3 as the durable payload store from the first persisted batch**. Local
+append files remain useful for buffering and cache speed; the web server's disk
+cannot be their only acknowledged durable home. Existing S3 mirroring is a useful
+starting point, but repeatedly uploading the entire growing file amplifies
+network and storage work. Upload each newly sealed segment once instead.
+
+| Location | Responsibility |
+| --- | --- |
+| Browser live stream / bounded server memory | Immediate progress display; not a durable receipt |
+| Existing native control event source / Runner outbox | Existing committed native records survive ACK; unacknowledged events remain replayable at the Runner |
+| S3-compatible storage | Immutable log/trace segments and bounded manifest/index pages |
+| Postgres | Existing company/run ownership, backend/reference, final status and expiry/deletion state; no per-batch writes |
+| Disposable web disk | Optional bounded buffer/read cache; never the only acknowledged copy |
+
+Existing structured `heartbeat_run_events`, task results, usage, and controller
+checkpoints retain their contracts. This is not a rewrite of those data paths.
+
+### Cost model and the first gate
+
+Let `A` be actively producing streams, `r` output records/second/stream, `b`
+bytes/second/stream, `T` maximum dirty-buffer age, and `S` target segment bytes.
+For a steady stream, approximate segment rate is
+`A × max(1/T, b/S)`; record rate is `A × r`. Empty streams emit no segments.
+Bursts, final flushes, retries, indexes, and trace channels add work and must be
+counted separately. A stream is not necessarily an agent: raw tracing can add
+multiple streams to one run.
+
+Illustrative arithmetic, **not a fleet measurement or chosen production setting**:
+100 continuously active streams at 10 records/second give 1,000 records/second.
+If five-second flushing dominates, they produce 20 segments/second, or 72,000
+segments/hour. With one small metadata publication transaction per segment,
+that is still 20 transactions/second. A 30-second interval gives about 3.3
+segments/second but retains more pending output. S3 index PUTs, GETs, retries,
+and transfer costs are additional. Neither option is free.
+
+The first gate measures real output sizes/rates and concurrency, then compares
+candidate size/age thresholds (initial experiment: 256 KiB–1 MiB and 5–30
+seconds, not defaults). Include quiet text, bursty tool output, raw tracing,
+long runs, slow S3, and the expected concurrent-run envelope. Record:
+
+- Incremental Postgres transactions, row updates, WAL bytes, connections, and
+  application p95 latency; no log/trace payload bytes inserted into Postgres.
+- S3 PUT/GET/HEAD/LIST counts, index overhead, object count, stored bytes,
+  compression CPU, transfer volume, and upload/read p95 latency.
+- Pending bytes, upload lag, read recovery time, and behavior at queue limits.
+- A daily/monthly cost projection using the **actual configured provider's**
+  rates and observed workload/duty cycle; do not substitute AWS prices for an
+  S3-compatible vendor or assume 24-hour production.
+- Zero added recurring SQL/network work after buffers and obligations settle.
+
+Record a concrete operating envelope, request-rate/backlog limits, and latency
+budget from this evidence before enabling the feature. If the batched metadata
+load is material, evaluate an object-store manifest head with conditional writes
+and lifecycle-only SQL. Choose one publication scheme for the release; do not
+ship two unqualified distributed state machines. Do not return to DB payload
+buffering as an automatic workaround for S3 latency or failure.
+
+## Design
+
+### 1. External backend and batched publication
+
+Keep local mode and add a versioned S3-backed store. Inject the existing object
+provider plus narrow metadata/ownership operations. Readers dispatch from the
+persisted backend/version, including historical records. Remove hardcoded
+`local_file` checks in heartbeat, issues, feedback, and customer-success
+inspection together. Preserve current `RUN_LOG_S3_*` mirror configuration for
+legacy mode, without redirecting unrelated product file storage.
+
+Reuse the existing instance storage configuration without additional switches.
+Cloud already provisions a bucket and bucket-scoped credentials per stack.
+Preserve any configured storage prefix and place journals in `run-logs/` and
+`provider-traces/`, with company/run namespaces underneath. New journals must not
+use a separately configured legacy mirror bucket. Configuration
+errors fail explicitly; never select local-only persistence silently. Add only
+small company/run-bound stream metadata, with unique identities and constraints.
+There are **no proposed run-log payload-chunk or raw-trace payload tables**.
+
+Selected publication protocol after the local operational experiment:
+
+1. Redact/compact log output through the existing owner and enqueue exact NDJSON
+   bytes with stable source/attempt identities in a bounded buffer. A buffered
+   enqueue and a durable receipt are different results.
+2. Seal on byte threshold, the first dirty buffer's actual deadline, explicit
+   flush, completion, or graceful handoff. Allocate sequence/byte offsets within
+   the current writer generation; do not write SQL per record. Seal channels
+   separately when their trace ordering requires it.
+3. Upload a new immutable segment with digest, length, source range, and logical
+   byte range. Then upload the bounded immutable index/manifest update. Keep
+   manifest pages bounded; do not rewrite a growing full-history JSON document
+   every few seconds. Retain exact logical bytes even when compressing storage.
+4. Publish one bounded head with S3 ETag `If-Match` (or `If-None-Match` for
+   creation). The head carries owner generation, committed cursor, and at most
+   53 immutable index roots. Check the admitted database execution authority
+   before claiming a new generation. No per-batch SQL or new stream table is
+   needed. Conditional head publication fences stale generations and tombstones.
+5. Release durable receipts for the published contiguous range. Reconcile a
+   lost object response from the exact head before retrying. Same-source replay
+   must not duplicate bytes; conflicting replay fails explicitly.
+
+Upload/index retries reuse immutable identities and verify ambiguous results.
+Do not hold a DB transaction or connection during network upload. New immutable
+index nodes combine completed binary subtrees; all normally published nodes
+remain referenced, avoiding an obsolete index object for every flush. Metadata must
+not grow with the number or size of individual text records: segment descriptors
+and source indexes live in S3. Keep lifecycle/audit metadata bounded.
+
+Use a company/agent/run prefix with a generation-fenced head. A stale owner may finish an upload,
+but cannot publish it after takeover. A successor adopts the last published
+manifest and replays its uncommitted tail. Per-stream prefix tracking plus scoped,
+paginated enumeration must recover unpublished-object cleanup after a crash;
+never expose bucket-wide listing through public APIs. Do not garbage-collect a
+pending publication merely because one DB query did not find its result.
+
+AWS supports conditional object creation and ETag-based updates, but the current
+storage interface does not expose them and compatible vendors need verification.
+If eliminating per-segment SQL becomes necessary, qualify conditional writes,
+ownership fencing, and manifest recovery explicitly before selecting that design.
+[Conditional writes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html).
+
+### 2. Live viewing, durability, and the pending tail
+
+Live output need not wait for an S3 PUT to appear in the browser. Preserve its
+stable source identity so reconnect/reload can reconcile it with durable history.
+An open viewer can see the pending tail; a different server can read the published
+S3 prefix. Do not promise zero-lag cross-server tail reads until a replay/relay
+path proves them. The flush threshold bounds normal durable-read lag, not a
+promise that storage is always available.
+
+The hard problem is **who retains output until publication**, not which database
+column it uses. A timer uploading from disposable web memory alone cannot
+preserve the pending tail after a crash. Distinguish these boundaries:
+
+- Store durable receipt: only after the segment/index and metadata publication.
+- Runner control-protocol ACK: currently may allow streaming output to leave
+  replay retention before log projection. Inspect and qualify this exact path.
+- Live browser delivery: presentation, not permission to discard the source.
+
+Selected after inspecting the real ACK path: reuse the native control port's
+already-durable canonical event rows. The S3 head advances their export cursor
+atomically with the log bytes. Takeover reads later native rows in bounded
+16-row pages, verifies source identity and hash, then uses the same formatter,
+redaction and compaction as live recording. The ordinary contiguous stream adds
+no SQL. Startup, missing callback ranges, and interrupted settled archives use
+bounded recovery reads; settled repair waits for the old native lease to expire.
+This avoids a second payload copy, new per-record DB writes, and a new Runner
+retention protocol. Other stdout and raw trace bytes are not inferred from this
+source. A retained native control row remains readable after Runner cleanup.
+The existing Runner outbox retains unacknowledged records. No second spool,
+broker, always-on logging service, or full transcript checkpoint is added.
+
+An alternative is holding the existing transport ACK until a batch is published,
+but use it only if a real protocol experiment proves bounded pipelining and
+prompt Stop/control handling. Serial awaited callbacks plus a five-second timer
+can become a five-second wait per record; that is unacceptable. Do not rebuild
+full transcript checkpoints in Postgres to sidestep this issue.
+
+For server-only diagnostics with no surviving source, successful durable receipts
+still require S3 publication. Local enqueue alone is never advertised as durable.
+Missing uncommitted optional diagnostics are explicitly incomplete. Preserve the
+original strict guarantee for every acknowledged durable record; any proposal
+to accept a bounded crash-loss window for ordinary output is a separate product
+decision, not an automatic relaxation in this revision. Host-local agent-process
+survival remains outside the web-server replacement guarantee.
+
+On S3 failure, retain bounded pending work at the source, back off, and surface
+capture/persistence state without falsely completing the log. At capacity,
+backpressure or stop new admission according to the existing execution owner;
+do not allocate unbounded memory/disk or switch payloads into Postgres. Optional
+trace overflow can become incomplete/truncated. Preserve essential control,
+Stop, task results, and usage settlement. Runner cleanup cannot destroy a pending
+required-output obligation; either finish export or leave a durable cleanup hold.
+
+### 3. Reading and storage amplification
+
+Read the published pointer, then bounded manifest/index pages and the segments
+covering the requested byte range. Cache immutable contents by digest. Preserve
+byte offsets, NDJSON/UTF-8 alignment, EOF, cancellation, and bounded memory.
+Compressed segments need logical offset metadata and bounded decompression;
+HTTP ranges over compressed bytes are not logical log offsets. Preserve exact
+full-stream digest semantics when finalizing.
+
+The first release can serve a finalized log by streaming its segments. Do not
+require a full re-upload or database compaction pass at every completion. Add
+optional segment coalescing only if measured read/request savings justify its
+extra I/O, publication/cleanup complexity, and temporary storage. Retention and
+read-path limits must account for small-object growth and indexes. Avoid one
+object per token/frame and repeated whole-log mirroring.
+
+S3 publication is not atomic with Postgres; object upload precedes pointer
+publication. An orphan object is recoverable cleanup, not proof of an accepted
+append. A missing/corrupt published object is a storage integrity error, not an
+empty log. Verify the actual provider's PUT/GET/range/checksum behavior rather
+than assuming AWS guarantees for every compatible backend.
+[S3 consistency model](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#ConsistencyModel).
+
+### 4. Provider trace boundaries
+
+Use the same batched object-payload mechanism with trace-specific ownership.
+Keep capture opt-in, existing secret/identity suppression, admin access,
+company/task-read checks, no-store responses, reveal/download audit, and trusted
+workspace-diff reprojection. Raw bytes never enter ordinary events, live logs,
+activity payloads, or general attachments. Provider parsing remains in drivers.
+
+Preserve native and rehydration channels, frame IDs, source/debug sequences,
+interpretation links, and raw digests. Capture transport must retain pending
+records outside disposable web storage until a storage receipt. Reconstruct
+frame lookup on takeover. A complete trace requires all required channel ranges
+and terminal records durably published and the integrity assessment passing.
+Capture failure yields incomplete/truncated without restarting the provider or
+blocking task/accounting settlement. Qualify all currently supported producer
+families; do not silently advertise a legacy path-only producer as durable.
+
+Encrypt raw trace objects with the existing stable-key facilities and bind them
+to company/trace/channel/segment identity. Keep only metadata and the appropriate
+key reference in Postgres. Preserve the existing 24-hour expiry from original
+capture and 64 MiB capture limit, with separate bounds for encoded/index overhead.
+Restart/import never extends expiry. Historical imports preserve exact bytes,
+including native-then-rehydration download ordering and digest.
+
+Expiry/deletion first creates a durable tombstone and fences publication, then
+cleans segments, index pages, abandoned uploads, and caches. Retain unresolved
+cleanup obligations through restart and concurrent uploads; no resurrection.
+Preserve `{ok:true}` only after required payload cleanup. Verify object-version
+behavior for the actual bucket, and document historical backup retention
+separately. Restore must honor expiry/deletion receipts before exposing traces.
+Audit contains content-free metadata only.
+
+### 5. Idle behavior and recovery
+
+A nonempty buffer schedules one actual flush deadline; successful settlement
+cancels it. Use the existing coordinator/scheduler for server-owned work and the
+Runner lifecycle for producer-owned work. Retries exist only for unresolved
+uploads/publications/deletes. Coalesce work and bound global as well as per-run
+upload concurrency; prevent many runs flushing at once from starving HTTP or
+connection pools. Do not create per-agent storage clients or idle keepalives.
+
+Start/resume/takeover reconciles active stream metadata, source retention, and
+unfinished cleanup. Existing DB work signals are process-local; do not rely on
+them to wake another server. Until the parent's cross-instance wake mechanism
+exists, retain explicit owner/recovery/sleep obligations. Completed history alone
+creates no recurring scan. Trace expiry uses its real deadline and the existing
+supported wake/blocker contract.
+
+## Delivery slices
+
+| Slice | Useful result and owned surfaces | Evidence/dependency |
+| --- | --- | --- |
+| 1. Operational and durability prototype | Workload measurement, S3 segments, publication/cost model, actual Runner pending-tail/ACK experiment | Select batching and one publication scheme; demonstrate acceptable cost and control responsiveness before schema commitment |
+| 2. S3-backed run logs | Store/readers, bounded buffering/indexes, small metadata schema, replay/ownership, cleanup and local compatibility | Slice 1; two-process replacement, ambiguous PUT/commit, stale-writer and backlog tests |
+| 3. S3-backed provider traces | Producer sinks/rehydration, trace store/routes, expiry/deletion | Slice 2 primitives; actual capture integrity, access and cleanup races |
+| 4. History migration and qualification | Resumable importer, compatibility/authority switch, documentation and staging evidence | All retained inventory accounted for; empty-disk reads and supported rollback after new writes |
+
+These are reviewable boundaries, not a fixed PR count. Keep each usable with the
+existing instance storage profile. Raw trace capture remains opt-in. Keep storage
+logic in its owners, not large heartbeat/executor additions.
+Generate necessary additive metadata migrations with `pnpm db:generate` and
+follow snapshot-retention rules. There is no DB payload migration or compactor.
+
+Update `doc/run-log-events.md`, `doc/DATABASE.md`, `doc/DEVELOPING.md`, and
+`doc/testing-native-remote-continuity.md` with actual guarantees and operating
+limits. Update affected run/inspection feature/access map entries and run
+`node scripts/check-maps.mjs`. Change shared types/UI only when their contract
+changes; keep incidental storage mechanics out of normal product controls.
+
+## Migration and rollback
+
+1. Ship a compatibility release that reads both stores, understands new formats,
+   and enforces the persisted minimum reader/writer capability on external runs.
+   Keep the volume and legacy writing enabled initially. Record this release as
+   the rollback floor before switching any authority; an environment flag alone
+   cannot stop an old binary from mishandling new rows. The deployment/rollback
+   preflight must reject binaries older than that floor: a pre-feature binary
+   cannot be expected to understand a newly introduced DB guard.
+2. Inventory every retained run log and unexpired trace by company/run. Classify
+   local complete files, remote mirrors, active files, missing payloads, conflicting
+   copies, and expired/deleted traces. A mirror may be a partial prefix: verify
+   against authoritative bytes/metadata, never assume it is complete.
+3. Backfill settled resources with bounded indexed keyset batches and resumable
+   receipts. Preserve original bytes, offsets, digests, attempt information,
+   source attribution, and trace expiry. Expired/deleted traces stay unavailable.
+4. Under the parent's permitted one-time maintenance window, stop admission and
+   settle/fence remaining legacy writers. Copy final active tails and both trace
+   channels. Recheck hashes and byte totals after fencing; no copy-while-writing
+   race can establish completion.
+5. Publish each resource's backend/reference transactionally. Record a migration
+   manifest of migrated, already-remote, expired/deleted, and unresolved resources.
+   Missing/conflicting retained bytes block the preservation gate. Report genuine
+   pre-existing loss; neither fabricate history nor quietly waive Dotta's choice.
+6. Start B with the log/trace directories absent and old storage inaccessible.
+   Read historical data and append new data through normal APIs. Confirm every
+   in-scope inventory item has a valid disposition. Keep the old volume retained
+   under the parent's recovery policy; this slice does not detach/delete it.
+7. Roll back to the compatibility release against the same external data and
+   prove it reads writes made after cutover. Do not switch authoritative reads
+   back to stale local files. Defer destructive schema cleanup until rollback
+   readers have retired.
+
+## Acceptance matrix
+
+| Case | Observable finish condition | Evidence |
+| --- | --- | --- |
+| Operational envelope | No new payload/per-token SQL; batch metadata and object traffic fit measured budgets | Real workload baseline, WAL/transactions, request/storage/transfer model and app/control latency |
+| Active/completed run | B retrieves every durable receipt exactly once after A and its disk disappear | Separate processes/directories; normal authorized reader; byte/digest comparison |
+| Pending tail | Web replacement retains/replays required output without duplicating provider work | Actual Runner spool/ACK experiment; kill between generation, upload and publication |
+| Existing history | All retained inventory preserved under original retention | Full checksums, resumable manifest; missing/conflicting data blocks cutover |
+| Ambiguous PUT/commit | Retry reconciles publication and never duplicates bytes | Fault after object PUT or DB commit but before response |
+| Ownership | Stale A cannot publish/finalize after B takes over | Concurrent server and manifest-generation tests |
+| Reads | Stable offsets/digests across segments, compression, caches and replacement | UTF-8/boundary/EOF/cancellation; bounded memory and GET counts |
+| Storage failure | No unbounded queues or SQL-payload fallback; Stop/API remain usable | Slow/failed S3, queue capacity, recovery and source cleanup hold |
+| Traces | Accepted frames survive; completeness and privacy remain truthful | All producer families, channel hashes, disabled capture and denied actors |
+| Downstream consumers | Issue previews, feedback, inspection and reprojection keep working | Normal routes and current authorization checks |
+| Expiry/delete | Deny after tombstone; clean payloads; no late-upload resurrection | Capture/upload/delete races, restart, actual bucket version behavior |
+| Rollback/local mode | Compatible release reads post-cutover writes; local defaults unchanged | Adjacent-version and local-store tests |
+| Idle | No recurring SQL/network work after settlement; real work wakes owners | Configured idle, completed history, retry/drain/resume observations |
+
+## First useful experiment and qualification order
+
+Use real Postgres, the existing S3 provider implementation against a disposable
+S3-compatible target, and two actual service processes with separate temporary
+directories. Produce representative output through the production recorder,
+including repeated chunks, multibyte text, and tool bursts. Measure the candidate
+batch settings and baseline API traffic. Test both published data and the pending
+tail: kill A at each boundary and read/replay through B. Never pre-populate the
+storage result being tested. Record revision, configuration, receipts, timing,
+SQL/WAL and object-operation evidence.
+
+Use the existing remote-continuity harness with a deterministic provider and
+actual Runner to qualify retention independently of control ACKs and responsive
+Stop. The decisive question is whether durable batching can coexist with ongoing
+provider execution and web replacement. Do not commit to a protocol mechanism
+based on a mocked callback or a successful single PUT.
+
+Before staging, record the chosen publication scheme, batching/concurrency
+limits, backlog capacity, supported recovery behavior, and measured cost/latency
+operating envelope. If those fail, revise the design before a broad schema or
+migration implementation. No production sizing/cost has yet been measured.
+
+Then verify the actual staging bucket and remote capture path with a bounded
+live provider run, normal UI/inspection/download, and post-replacement reads.
+Make the old web filesystem inaccessible. Preserve prior failures alongside
+successful retries. Full Railway HTTP handoff and volume removal remain parent
+plan gates.
+
+Run narrow storage/output/trace/continuity/auth/migration checks while iterating.
+At implementation PR handoff run `pnpm -r typecheck`, `pnpm test:run`, and
+`pnpm build`, plus affected Runner checks and user-journey verification. Do not
+claim live-provider or fleet evidence from deterministic fixtures.
+
+## Implementation state and remaining gates
+
+- Configuration follow-up: shared storage resolution now serves product files,
+  run logs and traces. S3 instances need no extra switches or connection fields.
+  Cloud's existing per-stack bucket and credentials remain the isolation
+  boundary; existing prefixes and company/run subfolders are preserved. The
+  configuration regression failed before the change, then 70 affected tests
+  passed, including real S3-compatible bucket/prefix isolation and cleanup tests.
+  Four existing Cloud environment-contract tests also passed without Cloud code
+  or deployment changes.
+- User follow-up: remove the added run/adapters map sections. Retained-history
+  conversion now belongs to an automatic, bounded delivery worker; the command
+  is optional diagnosis/repair. Existing local references remain durable work.
+  Startup reconstructs it, active or blocked history retries with backoff, and
+  completed history has no recurring SQL/network scan. Migration stays with its
+  owning service; index.ts only registers its lifecycle, and the shared delivery
+  registry adds the queue name. The source volume must survive until all retained
+  history is verified. Mixed versions must stop creating new local-file runs.
+- Trace encryption reuses either existing vault key source, including the
+  configured key file, and never creates a replacement key for external traces.
+  Both strict missing-key and file-key round-trip regressions failed before the fix.
+  The configuration/storage regression run passed 91 tests. Automatic migration
+  and retention passed 10 tests, including a real Postgres/S3 upgrade followed by
+  source-directory deletion. Harness failures remain in the local evidence: a
+  cold import exceeded a test deadline (moved module setup into the fixture), a
+  sandbox denied localhost access (rerun with network permission), and the first
+  automatic fixture bucket name exceeded S3's length limit (shortened the name).
+- Shutdown review: automatic trace import now propagates cancellation through
+  source reads, opening, conditional-write qualification, upload and verification.
+  Five real-SDK stalled-request regressions failed before the fix, then passed.
+  A shared capability probe survives one cancelled claimant and aborts when its
+  last claimant leaves. Source and remote trace hashing stream bounded chunks.
+  A multi-chunk native sidecar without a final newline retains exact download
+  ordering and digest. Existing file-key and empty-web-home app checks passed at
+  a142424fc9; subsequent changes affect copy cancellation and verification only.
+- Base integration: 879ba0d1fd reached Greptile 5/5 and all CI checks passed,
+  then master advanced and produced two append-only documentation conflicts.
+  Preserve both documentation sections and integrate the new Cloud snapshot
+  startup contract. The startup regression reproduced a redundant config read;
+  worker registration now accepts only the already-loaded storage-provider value.
+  All 103 startup controls plus storage/lifecycle controls passed (112 tests),
+  including saved Cloud profile reuse after the legacy config file is absent.
+  A further combined run passed all 124 storage and real-Postgres Cloud runtime
+  configuration controls on the merged source.
+- Further review: trace retention now registers independently of heartbeat
+  scheduling, and optional issue-comment attribution reads the published log
+  prefix after a failed best-effort native repair. Both defects reproduced red;
+  all 116 selected startup, retention, attribution and cancellation controls pass.
+  The 19f702d780 browser session-recovery check timed out; its job log is retained.
+  No UI behavior or browser assertion was changed to suppress that failure.
+- Branch: `codex/external-run-logs-traces`; this chat owns integration and PR review.
+- S3 holds payloads from the first committed batch. Conditional object heads
+  hold cursors and writer generations. Existing Postgres rows hold lifecycle and
+  references only. No schema change, per-record SQL, or DB payload compactor.
+- Implemented: stored-backend reader dispatch, bounded batching and I/O,
+  immutable binary range indexes, exact digest/range reads, ownership fencing,
+  encrypted optional trace channels, durable deletion/expiry obligations, and a
+  bounded resumable history importer. Local defaults remain unchanged.
+- Exercised locally with real Postgres and Moto 5.2.5 through the real AWS SDK:
+  encrypted trace replacement/deletion and history import with original source
+  directories removed. Missing, corrupt, and active history blocks migration.
+- Abrupt process qualification: a writer commits 1,000 records, receives storage
+  receipts, dies by `SIGKILL`, and loses its working directory. A separate
+  process reads every record exactly and continues the same journal.
+- App journey: create a deterministic process agent, click Run now, open its
+  run history, kill the web process, restart against the same external services,
+  and reload the viewer. All 31 NDJSON records remained byte-identical through
+  the normal authorized API (6,206 bytes; SHA-256
+  `3e8415c79379c583b1c4c37902187593fd69ebe85db78d6486d6077f205bacba`).
+- Local workload qualification reports burst and eight-stream steady request,
+  transfer, stored-object, and timing counts. It is a synthetic envelope,
+  not a fleet cost forecast. The first index prototype left obsolete COW
+  objects; the selected immutable binary index removes that normal growth.
+- Full workspace typecheck and build passed after native replay and the storage
+  review fixes. The pushed server revision was rebuilt for the final app check.
+  The native Postgres package initially lacked its postinstall-created library
+  symlinks; its standard hydration helper repaired that local test prerequisite.
+- Native pending-tail qualification passed with the real Rust recovery fixture,
+  `PaperclipControlPlanePort`, production recorder, real Postgres and Moto. A
+  received ACK released 32 records from the Runner while S3's source cursor was
+  still zero. After `SIGKILL` and deletion of the old controller directory, B
+  exported all 64 records once while the same Runner PID continued. Snapshot and
+  Stop commands completed in 453 ms and 122 ms. The first attempt failed because
+  its new controller directory had mode 0755; the fixture now creates mode 0700.
+  Both failed and successful evidence remain in this chat's temporary artifacts.
+  This is a deterministic protocol experiment, not a live model or Railway test.
+- Final app verification used a new web home on `c2cedcd116`. Retained history
+  remained byte-identical (31 records, 6,206 bytes), and a new UI-triggered process
+  run succeeded with every generated record present in its S3-backed transcript.
+  The browser initially reached the listener before startup completed; after
+  readiness, the normal page and API checks passed.
+- Durable log receipts and live log messages follow S3 publication. The native
+  export reuses existing control rows, adds no payload writes, and repairs missing
+  callback ranges. Interrupted settled archives can be repaired on normal reads
+  after the old native lease expires. Optional raw capture and server-only
+  diagnostics without a surviving source can remain incomplete after a crash.
+- Review fixes cover absent-head deletion fencing, shared capture ownership,
+  abort/drain on deletion, local expiry deadlines and multi-page cleanup, bounded
+  transient retries and admission backpressure, partial-startup cleanup, and
+  explicit trace-import cleanup markers. A source integrity failure leaves the
+  native archive's final size unknown instead of certifying a partial result.
+- Full local Vitest could not run on host embedded Postgres because concurrent
+  clusters exhausted macOS SysV semaphores. Affected suites are running against
+  isolated Docker databases. One local full typecheck attempt also lacked sandbox
+  permission for tsx IPC; the authorized rerun uses the normal workspace command.
+  The isolated Docker run then passed 1,014 affected storage/native tests and
+  all 27 privacy-route tests. Subsequent targeted review regressions passed for
+  native source classification, native import coverage, shared-reader repair
+  cancellation, serving the published prefix on repair conflicts, and a failed
+  local-file cleanup retaining its deadline. Live and recovered native output
+  redact existing agent identity material; runtime log secrets are registered
+  once through the existing encrypted redaction registry, without copying agent
+  private keys into run contexts.
+- Native history migration accepts finalized files whose recorded size and
+  digest match, preserving the existing formatter's truncated large display
+  chunks. Files missing either completeness field still require ordered source
+  coverage. Corrupt metadata blocks publication; the conditional switch checks
+  that the original completeness fields have not changed.
+- PR: https://github.com/paperclipai/paperclip/pull/18327. Initial CI passed 51
+  checks with one E2E teardown lock timeout. That E2E shard passed on the next
+  revision; its Runner shard then failed the Antigravity authentication fixture.
+  These failures are retained, and fresh-head CI remains required.
+- Production remains disabled. Railway Buckets are the parent fleet's current
+  S3 provider; their conditional-write, version/deletion, workload cost, remote
+  capture, and sleep/wake behavior need actual-target qualification. No fleet
+  environment or volume was changed.
+- Next gate: finish fresh-source checks and PR review; retain actual-provider,
+  live remote capture, and parent cutover gates before any volume-free rollout.
+- No Paperclip issue/run context was supplied. This repository plan is the state
+  record; no issue work product or production migration is claimed.

doc/run-evidence-storage.md

@@ -0,0 +1,276 @@
+# Run log and provider trace object storage
+
+Instances configured for S3-compatible file storage automatically use the
+`s3_segments` backend for new run logs and opted-in provider traces. Postgres
+retains the existing run/trace references and lifecycle rows.
+It adds no payload table, per-record SQL, or database-to-object compactor.
+Structured `heartbeat_run_events` retain their existing database path. Native
+events already committed there before a Runner ACK also provide the recovery
+source for an interrupted canonical native log export. No additional event copy
+or payload write is introduced.
+When native object logging uses runtime observability credentials, their log
+redaction values are registered once in the existing encrypted run-secret
+registry. This is bounded redaction metadata, not a transcript copy or a
+per-event write. Agent identity private material stays in its existing key store.
+
+## Configuration
+
+No additional storage configuration is required. The existing instance
+`storage.provider` setting, or `PAPERCLIP_STORAGE_PROVIDER`, selects storage.
+The same config-file S3 fields and `PAPERCLIP_STORAGE_S3_*` overrides select the
+bucket, endpoint, region, prefix, and path style used by product files. Credentials
+use the same AWS SDK chain. Local-disk instances keep `local_file` logs and traces.
+
+The instance's existing S3 prefix is preserved. The folders underneath it are:
+
+```text
+<existing-prefix>/run-logs/segments-v1/<company>/<agent>/<run>/
+<existing-prefix>/provider-traces/segments-v1/<company>/<run>/<trace>/<channel>/
+<existing-prefix>/<company>/...    existing product files
+```
+
+Cloud's existing provisioning supplies a separate bucket and bucket-scoped
+credentials for each stack, plus the stack's encryption key. The app reuses that
+boundary rather than selecting a global customer bucket. Within a stack, journal
+keys remain company/run scoped. Cleanup lists only the resource's own prefix;
+it cannot sweep other companies, other stacks' prefixes, or product files.
+Folders are organization, not a substitute for bucket permissions or API access
+checks. Shared-bucket self-hosted deployments must retain their existing distinct
+instance prefixes and appropriate storage credentials.
+
+Legacy run rows retain their recorded backend until verified migration. Their
+pre-existing `RUN_LOG_S3_*` mirror settings and key paths remain supported for
+legacy reads and writes. They do not override instance storage for new journals.
+Use stable bucket/prefix settings across replacements and releases.
+As with existing product files, changing the instance bucket or prefix is a
+separate storage move and requires copying the complete namespace and preserving
+its keys before changing configuration. Switching to local storage does not move
+existing S3 evidence. Readers reject an unavailable external store; they never
+substitute an empty local log. Restore the original stack's storage profile to
+read its history. This upgrade migrates legacy local evidence into the existing
+stack store; it does not introduce bucket relocation or reverse migration.
+
+Raw capture still requires its existing explicit capture request. Its encryption
+uses the existing vault key, from `PAPERCLIP_SECRETS_MASTER_KEY` or the existing
+configured key file. Cloud already supplies the environment key. No separate
+trace key is introduced, and a missing vault key fails rather than creating a
+replacement. Disposable deployments must preserve the existing vault key along
+with their other secrets.
+Raw payload and index objects
+use AES-GCM with a key derived for the particular company/run/trace/channel.
+The bounded head holds only ownership, range, digest, and publication metadata.
+Existing capture opt-in, redaction, instance-admin access, task/company access,
+reveal/download auditing, and the 24-hour capture expiry remain in effect.
+
+The provider must enforce `If-None-Match: *` and ETag `If-Match`, return ETags,
+and support prefix-scoped paginated listing. Claiming the first journal performs
+a conditional-write capability probe. A backend which ignores preconditions is
+rejected. Before deploying this release to an S3-configured production instance,
+qualify the provider's conditional writes, range/read consistency, encryption,
+object versions, and cleanup. AWS API compatibility alone is insufficient evidence.
+
+## Publication and operating limits
+
+Each batch uploads immutable payload segments and merges equal-height index
+roots into immutable binary nodes, then conditionally publishes one head object.
+The head carries at most 53 ordered roots. Every normally published payload and
+index node remains reachable; appends do not leave superseded index copies.
+Interrupted or fenced uploads can leave unpublished objects, which remain inside
+the resource prefix for scoped deletion. The head is the commit
+boundary. A stale writer can upload an orphan but cannot replace the current
+owner's published history. An uncertain response is reconciled against the exact
+head before returning a durability receipt. Committed reads use immutable roots,
+so publication cannot change the bytes behind an already selected snapshot.
+
+An individual store `append` resolves only after publication. Heartbeat's
+`onLog` diagnostic callback waits for bounded admission without serially
+waiting for each network batch. At capacity it waits for space; admission beyond
+the bounded waiting allowance is rejected without poisoning accepted batches.
+Transient network, 409, 429 and server errors receive bounded retries. A 412
+or a changed head fences the stale writer; a 409 alone does not prove ownership
+loss or conditional-write support. Live log messages and progress are published
+after storage confirmation. Finalization drains admitted writes before closing
+and hashing the stream. Logging failure reports incomplete persistence and
+leaves the final size/digest unknown rather than claiming a complete snapshot.
+The original progress projection remains coarsely throttled; concurrent batch
+completions share a single progress write.
+
+Defaults and fixed safeguards:
+
+| Limit | Value |
+| --- | --- |
+| Dirty batch deadline | 1 second, configurable from 1 ms to 30 seconds |
+| Segment target | 1 MiB |
+| Pending bytes per writer | 8 MiB |
+| Pending run-log bytes per process | 64 MiB |
+| Additional waiting admission | 8 MiB / 32 calls per process |
+| Concurrent run-log flushes | 4 |
+| Concurrent journal object operations | 8, with 32 queued operations |
+| Object operation/batch cancellation deadline | 15 seconds |
+| Current index fanout / maximum roots | 2 children / 53 roots |
+| Journal head read bound | 64 KiB |
+| Trace encoded payload bound per sidecar | 192 MiB, in addition to the existing raw capture limit |
+
+A configured but empty log store sends no periodic request. Dirty buffers,
+capture file-change events, and actual retention/retry obligations drive work.
+Use request/transfer counts, byte volume, steady versus bursty producers,
+concurrent runs, and control/API latency to size a deployment. The local S3
+qualification is a synthetic workload, not a production cost forecast.
+
+Finalization streams the committed bytes once to compute the existing complete
+file SHA-256. It does not re-upload the entire log. Range indexes avoid loading
+all segment descriptors to read a tail. Historical consumers, UTF-8/NDJSON record
+alignment, issue previews, feedback, and customer-success inspection dispatch
+through the persisted backend. Logical reads retain the existing authorization.
+On-demand native repair closes its verified committed range with an exact byte
+count and leaves an unavailable full-file SHA-256 null. It does not make a log
+page request re-download the entire archive to compute that optional digest.
+
+## Durability boundary
+
+Committed log receipts and captured object prefixes survive web disk loss.
+Output still buffered only by the current web process has no durable receipt.
+The live log path waits for publication before showing that output. Raw sidecar
+capture remains optional; a crash can leave an incomplete uncommitted tail, and
+only durably closed channels plus valid terminal markers qualify as complete.
+
+Native PRP events have a surviving source: the control port already commits
+validated, redacted events before acknowledging them. The S3 head checkpoints
+the highest exported database event cursor with the batch's bytes. A replacement
+replays later native source rows in bounded 16-row pages, checks their identity
+and canonical hash, and exports the same `paperclip.prp.event` representation
+through normal log redaction/compaction. Late callbacks and duplicate delivery
+cannot append the same source record again. This is a canonical native export;
+generic stdout, stderr, and raw provider traces are not reconstructed from event
+rows. The control ACK and sparse checkpoint formats remain unchanged.
+
+Recovery runs before native execution reconnects. If the run settled before its
+archive finished, the normal log API, issue preview, or feedback reader can repair
+it after the native controller lease expires. Concurrent requests share recovery;
+final metadata is saved only when no later native source remains. Read-only
+customer-success inspection reads the currently published prefix. A failed
+repair leaves final size/digest unknown and preserves the existing source.
+Retained native events are not deleted by Runner cleanup. Recovery adds bounded
+reads and one final metadata update, with no new per-record SQL on the normal
+contiguous output path. Raw optional capture can still be incomplete after a crash.
+Credential-update, retained-cleanup, and trace-reprojection records have separate
+source namespaces and do not enter this log export. Corrupt canonical records
+fail integrity validation. Shared on-demand repair has a 30-second lifetime;
+cancellation removes a waiting reader and aborts storage work when no readers
+remain. It preserves already published cursors so a later request can resume.
+Live export and recovery both redact the existing agent identity and known run
+secrets before publication. Recovery reads existing identity material without
+provisioning a key. If best-effort repair loses ownership or fails, ordinary log
+reads serve the published prefix and leave completion metadata unknown; request
+cancellation still propagates.
+
+The [remote continuity](testing-native-remote-continuity.md) protocol and the
+parent deployment's other filesystem and HTTP handoff requirements still apply.
+
+## Trace deletion, expiry, and sleep
+
+Deletion/expiry denies reads through a database tombstone before payload cleanup.
+Object journal tombstones prevent further publication. If an active writer may
+have an in-flight upload, cleanup waits at least 30 seconds and remains a durable
+retry obligation. A successful manual DELETE requires payload cleanup; an
+unsettled cleanup returns a retryable failure, and the shared retention worker
+retries it. Tombstones remain after object removal. Historical backups and
+object-version retention need a separate operator policy; an ordinary DELETE
+must not be mistaken for erasure of historical versions.
+
+The shared coordinator restores expiry/cleanup work on startup and registers
+actual deadlines for local and external captures. New local inserts also wake it;
+expired backlogs drain in bounded pages. No completed-history polling is added. Trace expiry
+is not yet part of Cloud's durable wake contract, so outstanding trace deadlines
+conservatively block instance sleep. Removing that blocker requires qualification
+of the corresponding external wake, not a longer polling interval.
+
+Trace expiry and failed-deletion recovery remain registered when
+`HEARTBEAT_SCHEDULER_ENABLED=false`. Disabling agent scheduling does not disable
+retained-data cleanup. The worker still respects warm standby, admission drain,
+actual deadlines, and shutdown.
+
+## History migration and rollback
+
+S3-configured servers automatically migrate retained local history in the background.
+No command or extra configuration is required. The existing delivery coordinator
+starts the upgrade outside request handling, independently of the heartbeat
+scheduler toggle. Each batch processes at most one log and one trace, then yields
+for one second. Existing references are the durable work inventory. Restarting
+reconstructs unfinished work from those references; already migrated rows are
+excluded. No new payload or job table is added.
+
+Active writers, missing files, corrupt data, and storage failures leave history
+unresolved. Other pages continue. The server logs each blocked resource and a
+pass summary. Unresolved passes retry after five minutes, with backoff up to
+30 minutes. Transport failures retry through the shared coordinator. After all
+retained history is migrated, the worker stops querying and contacting storage.
+Trace source hashing and remote verification stream bounded chunks instead of
+loading a complete capture into server memory. The worker cancellation signal
+reaches file reads, trace opening, conditional probing, uploads and verification.
+Local-disk instances do not register this worker. Outstanding migration conservatively
+blocks instance sleep, and shutdown aborts the batch and waits for owned work.
+
+Deploy compatible readers and stop old binaries from starting new local runs.
+The upgrade waits for existing writers to settle and their leases to expire.
+Keep the source volume attached until migration completes and clean-filesystem
+reads pass. Automatic copying cannot recover a file after its only copy is lost.
+
+`server/scripts/migrate-run-evidence.ts` remains an optional diagnostic and repair
+tool. Its default is a read-only inventory; `--apply` runs the same verified
+importer used by the automatic worker. `--company`, `--after-run`, `--after-trace`
+and `--limit` allow scoped investigation against an explicitly selected
+`DATABASE_URL`. Normal upgrades do not require this tool. It does not stop agents,
+select a production target, delete source files, or certify volume removal.
+
+For logs, migration verifies exact local bytes or a mirror with proven length
+and digest, compares any previously uploaded prefix, verifies remote SHA-256,
+then conditionally changes the run's backend/reference. Unknown-completeness
+mirrors, missing/corrupt payloads, and active writers block publication. Existing
+trace bytes, download ordering, integrity state and original expiry are retained.
+Neither path fabricates missing historical data or extends trace retention.
+For finalized native logs, matching the recorded size and SHA-256 proves the
+complete legacy file, including display chunks truncated by the existing 64 KiB
+formatter. Migration preserves those exact bytes and records the settled source
+cursor. If either completeness field is absent, a bounded NDJSON scan must prove
+ordered coverage of the existing canonical source records. Missing ranges or
+chunks whose identities cannot be verified block publication as
+`native_log_source_coverage_unproven`. A database's latest event cursor alone
+does not establish completeness. Publication checks that the original reference,
+completeness metadata, and source cursor have not changed. Keep the original
+volume and resolve blocked inventory before cutover.
+An interrupted trace import keeps an `s3-import-v1:` reference: reads still use
+the original local sidecars, while deletion and expiry retain responsibility for
+the staged object prefix. The automatic worker resumes the verified switch after restart or retry.
+Ordinary local traces have no S3 cleanup dependency.
+
+Keep the volume until every retained item has been verified from a clean web
+filesystem. Rollback must use a compatible reader against the same remote state.
+An older binary or stale local volume is not a valid rollback target after new
+external writes. Enforce that floor in the owning deployment procedure.
+
+## Qualification
+
+Run the storage, recorder, migration, privacy and retention tests. The opt-in
+S3/Postgres integration fixtures use `PAPERCLIP_LOG_TEST_S3_ENDPOINT` and a dedicated
+`PAPERCLIP_RUN_EVIDENCE_MIGRATION_DATABASE_URL` ending in `_test`. They create
+unique test buckets and schemas, and remove only those resources. The privacy
+route suite also accepts `PAPERCLIP_RUN_LOG_PRIVACY_TEST_DATABASE_URL` for a
+dedicated, fully migrated database ending in `_test`; it clears its fixtures.
+
+`server/scripts/qualify-object-run-logs.ts` exercises the actual AWS SDK against
+a loopback test backend, confirms exact replacement reads and stale-writer
+rejection, and reports requests/bytes/timing. It uses synthetic credentials and
+refuses a non-loopback target. This proves its tested transport and workload;
+production-provider qualification and actual Railway deployment remain separate.
+
+`server/scripts/qualify-native-log-recovery.ts` uses the real Rust
+`recovery-fixture` example, real Postgres, the control port, production recorder,
+and S3 backend. It kills the controller after Runner ACKs but before the S3
+cursor catches up, deletes that controller's directory, and restores sparse
+control state into a fresh controller. It checks exact source order, no duplicate
+records, the same surviving Runner PID, and control/Stop responsiveness. Build
+the example with `cargo build --locked --manifest-path packages/paperclip-runner/runner/Cargo.toml -p paperclip-runner-core --example recovery-fixture`.
+It accepts the same two explicit loopback test endpoints and creates a private
+schema and bucket. This deterministic protocol experiment does not qualify a
+live provider, remote execution environment, or production S3 vendor.

doc/run-log-events.md

@@ -694,3 +694,13 @@ telemetry-read authorization. Ordinary PRP reads/live delivery retain run-secret
 and current-user redaction. The task page uses one history read on mount/reconnect
 and the existing live transcript stream, then makes a bounded plugin readiness
 check. No panel discovery timer scans runs or provider state while connected.
+
+## Payload storage
+
+Structured events in this guide remain in `heartbeat_run_events`. When the
+instance's existing file storage uses S3, new stdout/stderr NDJSON logs use the
+same stack bucket and configured prefix under `run-logs/`, with existing run
+references in Postgres. Local-disk instances remain local. Raw provider
+traces have their own restricted, encrypted object path. See
+[run evidence storage](run-evidence-storage.md) for batching, durability,
+configuration, migration, retention, and qualification boundaries.

doc/testing-native-remote-continuity.md

@@ -172,10 +172,15 @@ disk outbox. No second remote journal or new polling loop is introduced.
 The live event window keeps the latest identity, turn, result and usage anchors,
 as well as unsettled semantic inputs, through long output streams. The 4,096-event
 and 500-command limits remain. This checkpoint is intentionally not a complete
-transcript: an abrupt crash can lose display output already acknowledged but not
-yet projected to the existing run log. The same provider turn continues; recovery
-never reruns work to reconstruct display text. Tool outcomes and terminal state
-retain their durable barriers.
+transcript. With local-only logs, an abrupt crash can lose a display projection
+already acknowledged by the controller. The opt-in `s3_segments` log backend
+checkpoints its canonical native export cursor in S3 and replays missing records
+from the existing validated `heartbeat_run_events` source on takeover. This adds
+no payload writes or per-record checkpoint transactions. Generic process output
+and optional raw provider sidecars have separate durability boundaries. See
+[run evidence storage](run-evidence-storage.md). The same provider turn continues;
+recovery never reruns provider work to reconstruct display text. Tool outcomes
+and terminal state retain their durable barriers.
 
 Rollback must use a controller that understands sparse checkpoints and the signed
 recovery cursor. A v2 decoder alone is insufficient. Runners admitted before this

docs/deploy/storage.md

@@ -4,6 +4,9 @@ summary: Local disk vs S3-compatible storage
 ---
 
 Paperclip stores uploaded files (issue attachments, images) using a configurable storage provider.
+New run logs and explicitly requested provider traces reuse the same settings.
+S3 instances store these under `run-logs/` and `provider-traces/` within the
+existing prefix. Local-disk instances keep them local.
 
 ## Local Disk (Default)
 

maps/feature-map/runs-adapters.md

@@ -348,7 +348,7 @@ revised-source live onboarding and follow-up proof.
   `/company/settings/connections` configures decision/fast-response models,
   not the general model-account library.
 
-- The run log is instance database data; it is not the opt-out first-party Telemetry system or operator-configured OpenTelemetry tracing.
+- Structured run events stay in the instance database. Optional stdout/stderr and raw trace payload storage can use S3; see [run evidence storage](../../doc/run-evidence-storage.md). Both are separate from first-party Telemetry and operator OpenTelemetry export.
 - Authentication/model discovery does not establish tool, image, steering, or resumption capability parity.
 - Use [recovery](./recovery.md) for interrupted runs and [execution environments](./execution-environments.md) for local/SSH/sandbox differences.
 

server/scripts/migrate-run-evidence.ts

@@ -0,0 +1,32 @@
+import { createDb } from "@paperclipai/db";
+import { migrateRunEvidence } from "../src/services/run-evidence-migration.js";
+
+if (!process.env.DATABASE_URL)
+  throw new Error("Set DATABASE_URL to the intended instance; run with legacy writers drained");
+const args = process.argv.slice(2);
+const flags = new Set(["--apply", "--company", "--after-run", "--after-trace", "--limit"]);
+for (let index = 0; index < args.length; index++) {
+  const flag = args[index]!;
+  if (!flags.has(flag)) throw new Error(`Unknown migration option: ${flag}`);
+  if (flag !== "--apply" && (!args[++index] || args[index]!.startsWith("--"))) {
+    throw new Error(`Missing value for ${flag}`);
+  }
+}
+const value = (flag: string) => {
+  const i = args.indexOf(flag);
+  return i < 0 ? undefined : args[i + 1];
+};
+const db = createDb(process.env.DATABASE_URL);
+try {
+  const result = await migrateRunEvidence(db, {
+    apply: args.includes("--apply"),
+    companyId: value("--company"),
+    afterRunId: value("--after-run"),
+    afterTraceId: value("--after-trace"),
+    limit: Number(value("--limit") || 50),
+  });
+  console.log(JSON.stringify(result, null, 2));
+  if (result.items.some((item) => item.state === "blocked")) process.exitCode = 1;
+} finally {
+  await db.$client.end();
+}

server/scripts/qualify-native-log-recovery.ts

@@ -0,0 +1,445 @@
+import assert from "node:assert/strict";
+import { randomUUID } from "node:crypto";
+import { spawn, type ChildProcess } from "node:child_process";
+import { once } from "node:events";
+import { promises as fs } from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+import { setTimeout as delay } from "node:timers/promises";
+import { eq, sql } from "drizzle-orm";
+import { createDb, heartbeatRuns } from "@paperclipai/db";
+import {
+  S3Client,
+  CreateBucketCommand,
+  DeleteBucketCommand,
+  ListObjectsV2Command,
+  DeleteObjectsCommand,
+} from "@aws-sdk/client-s3";
+import {
+  DurablePrpControlPlane,
+  decodeDurablePrpCheckpoint,
+} from "../src/vendor/paperclip-runner/index.js";
+import { createS3StorageProvider } from "../src/storage/s3-provider.js";
+import { createObjectRunLogStore } from "../src/services/object-run-log-store.js";
+import { createObjectByteJournal } from "../src/services/object-byte-journal.js";
+import { createHeartbeatRunOutput } from "../src/services/heartbeat/run-output.js";
+import { createAgentIdentityRedactor } from "../src/services/agent-identity-redaction.js";
+import { appendHeartbeatRunEvent } from "../src/services/heartbeat-run-events.js";
+import { PaperclipControlPlanePort } from "../src/services/native-runtime/paperclip-control-plane-port.js";
+import { formatNativeRunLogEvent } from "../src/services/native-runtime/native-run-log-export.js";
+import { createNativeRunLogTestTables } from "../src/__tests__/helpers/native-run-log-database.js";
+
+const endpoint = process.env.PAPERCLIP_LOG_TEST_S3_ENDPOINT;
+const databaseUrl = process.env.PAPERCLIP_RUN_EVIDENCE_MIGRATION_DATABASE_URL;
+if (
+  !endpoint ||
+  !databaseUrl ||
+  [endpoint, databaseUrl].some((url) => !["127.0.0.1", "localhost"].includes(new URL(url).hostname))
+)
+  throw new Error("Qualification requires explicit disposable loopback S3 and Postgres endpoints");
+const childMode = process.argv[2] === "--controller";
+type Config = {
+  schema: string;
+  bucket: string;
+  checkpoint: string;
+  companyId: string;
+  agentId: string;
+  runId: string;
+  sessionId: string;
+  runnerInstanceId: string;
+  environmentLeaseId: string;
+  turnId: string;
+  itemId: string;
+};
+const config: Config = childMode
+  ? JSON.parse(await fs.readFile(process.argv[3]!, "utf8"))
+  : {
+      schema: `native_log_proof_${randomUUID().replaceAll("-", "")}`,
+      bucket: `pc-native-proof-${randomUUID()}`,
+      checkpoint: "",
+      companyId: randomUUID(),
+      agentId: randomUUID(),
+      runId: randomUUID(),
+      sessionId: randomUUID(),
+      runnerInstanceId: randomUUID(),
+      environmentLeaseId: randomUUID(),
+      turnId: randomUUID(),
+      itemId: randomUUID(),
+    };
+const db = createDb(databaseUrl, { maxConnections: 1, idleTimeoutSeconds: 0 });
+await db.execute(sql`set client_min_messages=warning`);
+const provider = createS3StorageProvider({
+  endpoint,
+  bucket: config.bucket,
+  region: "us-east-1",
+  forcePathStyle: true,
+});
+const logStore = () =>
+  createObjectRunLogStore({
+    provider,
+    keyPrefix: "logs",
+    flushMs: 30_000,
+    assertWriterAuthority: async (input) => {
+      await input.assertWriterAuthority?.();
+    },
+  });
+const identity = {
+  runnerInstanceId: config.runnerInstanceId,
+  environmentLeaseId: config.environmentLeaseId,
+  runId: config.runId,
+  normalizedSessionId: config.sessionId,
+  turnId: config.turnId,
+  itemId: config.itemId,
+};
+const expectedRunnerVersion = "0.3.0",
+  expectedRunnerDigest = `sha256:${"c".repeat(64)}`;
+async function waitFor<T>(read: () => Promise<T>, done: (value: T) => boolean, timeout = 30_000) {
+  const until = Date.now() + timeout;
+  while (Date.now() < until) {
+    const value = await read();
+    if (done(value)) return value;
+    await delay(40);
+  }
+  throw new Error("Native log qualification deadline exceeded");
+}
+if (childMode) {
+  const home = process.argv[4]!,
+    portNumber = Number(process.argv[5] || 0);
+  await db.execute(sql.raw(`set search_path=${config.schema}`));
+  if (portNumber) {
+    const records = new Map<string, string>(
+      JSON.parse(await fs.readFile(config.checkpoint, "utf8")),
+    );
+    decodeDurablePrpCheckpoint(records).restoreSessionDirectory(home);
+  }
+  const store = logStore();
+  const [row] = await db
+    .select({
+      id: heartbeatRuns.id,
+      companyId: heartbeatRuns.companyId,
+      agentId: heartbeatRuns.agentId,
+      status: heartbeatRuns.status,
+      runtimeMode: heartbeatRuns.runtimeMode,
+      logStore: heartbeatRuns.logStore,
+      logRef: heartbeatRuns.logRef,
+      logBytes: heartbeatRuns.logBytes,
+      lastOutputSeq: heartbeatRuns.lastOutputSeq,
+      lastOutputAt: heartbeatRuns.lastOutputAt,
+    })
+    .from(heartbeatRuns)
+    .where(eq(heartbeatRuns.id, config.runId));
+  type Run = typeof heartbeatRuns.$inferSelect;
+  const output = createHeartbeatRunOutput(db, {
+    run: row as Run,
+    runId: config.runId,
+    issueId: null,
+    runLogStore: store,
+    identityRedactor: createAgentIdentityRedactor(),
+    secretKeys: new Set(),
+    getCurrentUserRedactionOptions: async () => ({ enabled: false }),
+    appendRunEvent: async (_run, event) => {
+      await appendHeartbeatRunEvent(db, {
+        companyId: config.companyId,
+        agentId: config.agentId,
+        runId: config.runId,
+        ...event,
+      });
+    },
+    assertWriterAuthority: async () => {
+      const [current] = await db
+        .select({ status: heartbeatRuns.status })
+        .from(heartbeatRuns)
+        .where(eq(heartbeatRuns.id, config.runId));
+      if (current?.status !== "running") throw new Error("fixture_writer_revoked");
+    },
+  });
+  const handlers = await output.begin(row as Run);
+  const record = async (event: Parameters<typeof formatNativeRunLogEvent>[0], cursor: number) => {
+    await handlers.onPersistedNativeLog!(cursor, formatNativeRunLogEvent(event));
+  };
+  const persistence = new PaperclipControlPlanePort(
+    db,
+    {
+      ...config,
+      issueId: randomUUID(),
+      sourceInstanceId: config.runnerInstanceId,
+      controlPlaneSourceInstanceId: `control-${config.runId}`,
+      completionContractId: randomUUID(),
+      completionContractSha256: "qualification",
+    },
+    { onCommittedEvent: record, onDuplicateEvent: record },
+  );
+  const core = new DurablePrpControlPlane({
+    stateDirectory: path.join(home, "control-plane"),
+    identity,
+    expectedRunnerVersion,
+    expectedRunnerDigest,
+    onCommittedEvent: async (event) => {
+      await persistence.appendEvent(event);
+    },
+  });
+  core.setCheckpointPersistence(async (checkpoint) => {
+    const temporary = `${config.checkpoint}.pending`;
+    await fs.writeFile(temporary, JSON.stringify([...checkpoint.records]));
+    await fs.rename(temporary, config.checkpoint);
+  });
+  await core.start(portNumber || undefined);
+  const ticket = core.issueBootstrapTicket();
+  await core.flushCheckpoint();
+  process.on("message", async (message: { id: string; type: string }) => {
+    try {
+      if (message.type === "checkpoint") await core.flushCheckpoint();
+      else if (message.type === "finish-log") {
+        const result = await store.finalize(output.handle!);
+        await output.flushOutputProgress({ force: true });
+        if (result.bytes === null) throw new Error("qualification_log_incomplete");
+      } else {
+        core.queueCommand(message.type, {}, message.id, true);
+        await waitFor(
+          async () => core.getCommand(message.id)?.status,
+          (status) => status === "completed",
+        );
+      }
+      process.send!({ id: message.id, ok: true });
+    } catch (error) {
+      process.send!({ id: message.id, error: String(error) });
+    }
+  });
+  process.send!({ ready: true, connectUrl: core.connectUrl, ticket });
+} else {
+  const root = await fs.mkdtemp(path.join(os.tmpdir(), "native-log-qualification-"));
+  config.checkpoint = path.join(root, "checkpoint.json");
+  const configFile = path.join(root, "config.json"),
+    receiptFile = path.join(root, "receipt.json"),
+    releaseFile = path.join(root, "release"),
+    scriptFile = path.join(root, "script.json");
+  const children: ChildProcess[] = [];
+  const client = new S3Client({ endpoint, region: "us-east-1", forcePathStyle: true });
+  let bucketCreated = false;
+  async function controller(home: string, portNumber = 0) {
+    await fs.mkdir(home, { mode: 0o700 });
+    const child = spawn(
+      process.execPath,
+      [
+        "--import",
+        path.resolve("server/node_modules/tsx/dist/loader.mjs"),
+        fileURLToPath(import.meta.url),
+        "--controller",
+        configFile,
+        home,
+        String(portNumber),
+      ],
+      {
+        cwd: process.cwd(),
+        env: { ...process.env, PAPERCLIP_HOME: home },
+        stdio: ["ignore", "ignore", "pipe", "ipc"],
+      },
+    );
+    children.push(child);
+    let diagnostics = "";
+    child.stderr!.on("data", (bytes) => {
+      diagnostics = (diagnostics + String(bytes)).slice(-8000);
+    });
+    const ready = await new Promise<{ connectUrl: string; ticket: string }>((resolve, reject) => {
+      const timeout = setTimeout(
+        () => reject(new Error(`Controller startup timeout: ${diagnostics}`)),
+        45_000,
+      );
+      child.on("message", (message: { ready?: boolean; connectUrl: string; ticket: string }) => {
+        if (message.ready) {
+          clearTimeout(timeout);
+          resolve(message);
+        }
+      });
+      child.once("exit", () => {
+        clearTimeout(timeout);
+        reject(new Error(`Controller startup failed: ${diagnostics}`));
+      });
+    });
+    async function command(type: string) {
+      const id = randomUUID();
+      await new Promise<void>((resolve, reject) => {
+        const timer = setTimeout(() => {
+          child.off("message", onMessage);
+          reject(new Error(`Command timeout: ${type}: ${diagnostics}`));
+        }, 35_000);
+        function onMessage(message: { id: string; error?: string }) {
+          if (message.id !== id) return;
+          clearTimeout(timer);
+          child.off("message", onMessage);
+          message.error ? reject(new Error(message.error)) : resolve();
+        }
+        child.on("message", onMessage);
+        child.send({ id, type });
+      });
+    }
+    return { child, ready, command };
+  }
+  try {
+    await db.execute(sql.raw(`create schema ${config.schema}`));
+    await db.execute(sql.raw(`set search_path=${config.schema}`));
+    await createNativeRunLogTestTables(db);
+    await db.execute(
+      sql`insert into heartbeat_runs(id,company_id,agent_id,status) values(${config.runId},${config.companyId},${config.agentId},'running')`,
+    );
+    await client.send(new CreateBucketCommand({ Bucket: config.bucket }));
+    bucketCreated = true;
+    await fs.writeFile(configFile, JSON.stringify(config));
+    await fs.writeFile(
+      scriptFile,
+      JSON.stringify({ steps: 64, waitAfter: 32, retentionLimit: 8, receiptFile, releaseFile }),
+    );
+    const homeA = path.join(root, "web-a"),
+      a = await controller(homeA);
+    const runner = spawn(
+      path.resolve("packages/paperclip-runner/runner/target/debug/examples/recovery-fixture"),
+      [
+        "--fixture-script",
+        scriptFile,
+        "--connect-url",
+        a.ready.connectUrl,
+        "--state-dir",
+        path.join(root, "runner"),
+        "--runner-id",
+        config.runnerInstanceId,
+        "--environment-lease-id",
+        config.environmentLeaseId,
+        "--run-id",
+        config.runId,
+        "--session-id",
+        config.sessionId,
+        "--turn-id",
+        config.turnId,
+        "--item-id",
+        config.itemId,
+        "--runner-version",
+        expectedRunnerVersion,
+        "--runner-digest",
+        expectedRunnerDigest,
+        "--max-outbox-bytes",
+        "65536",
+        "--p0-reserve-bytes",
+        "16384",
+        "--reconnect-grace-ms",
+        "60000",
+      ],
+      {
+        env: { ...process.env, PAPERCLIP_RUNNER_BOOTSTRAP_TICKET: a.ready.ticket },
+        stdio: ["ignore", "ignore", "pipe"],
+      },
+    );
+    children.push(runner);
+    let runnerDiagnostics = "";
+    runner.stderr!.on("data", (bytes) => {
+      runnerDiagnostics = (runnerDiagnostics + String(bytes)).slice(-4000);
+    });
+    const receipt = async () =>
+      JSON.parse(await fs.readFile(receiptFile, "utf8").catch(() => "{}")) as {
+        pid: number;
+        acknowledgedEvents: number;
+        completedSteps: number;
+        retainedEvents: number;
+        peakRetainedEvents: number;
+      };
+    await a.command("turn.start");
+    await fs.writeFile(releaseFile, "go");
+    const acknowledged = await waitFor(
+      receipt,
+      (value) => value.acknowledgedEvents === 32 && value.retainedEvents === 0,
+    );
+    const journal = createObjectByteJournal({
+      provider,
+      prefix: `logs/segments-v1/${config.companyId}/${config.agentId}/${config.runId}`,
+    });
+    const before = (await journal.head())!.head;
+    assert(
+      before.sourceCursor < 32,
+      "Experiment must kill a controller with acknowledged output still pending in S3",
+    );
+    const controlStarted = performance.now();
+    await a.command("session.snapshot");
+    const controlMs = performance.now() - controlStarted;
+    await a.command("checkpoint");
+    const exitA = once(a.child, "exit");
+    a.child.kill("SIGKILL");
+    await exitA;
+    await fs.rm(homeA, { recursive: true });
+    const b = await controller(path.join(root, "web-b"), Number(new URL(a.ready.connectUrl).port));
+    await b.command("turn.steer");
+    const completed = await waitFor(
+      receipt,
+      (value) => value.acknowledgedEvents === 64 && value.retainedEvents === 0,
+    );
+    assert.equal(completed.pid, acknowledged.pid);
+    assert.equal(runner.exitCode, null, runnerDiagnostics);
+    const stopStarted = performance.now();
+    await b.command("turn.stop");
+    const stopMs = performance.now() - stopStarted;
+    await b.command("finish-log");
+    const store = logStore(),
+      content = (
+        await store.read({
+          store: "s3_segments",
+          logRef: `${config.companyId}/${config.agentId}/${config.runId}`,
+        })
+      ).content;
+    const events = content
+      .trim()
+      .split("\n")
+      .map((line) => JSON.parse(JSON.parse(line).chunk).event);
+    assert.deepEqual(
+      events.map((event) => Number(event.payload.text.match(/^step=(\d+)/)[1])),
+      Array.from({ length: 64 }, (_, index) => index + 1),
+    );
+    assert.equal(new Set(events.map((event) => event.sourceEventId)).size, 64);
+    const [source] = await db.execute(
+      sql`select count(*)::int as count from heartbeat_run_events where run_id=${config.runId}`,
+    );
+    assert.equal(source!.count, 64);
+    const shutdown = once(runner, "exit");
+    await b.command("runner.shutdown");
+    assert.equal((await shutdown)[0], 0);
+    process.stdout.write(
+      `${JSON.stringify({
+        result: "passed",
+        source: "real Rust recovery fixture and PaperclipControlPlanePort",
+        acknowledgedBeforeKill: 32,
+        objectSourceCursorBeforeKill: before.sourceCursor,
+        controllerLoss: "SIGKILL",
+        oldDirectoryRemoved: true,
+        sameRunnerProcess: true,
+        recoveredRecords: events.length,
+        duplicates: 0,
+        existingControlRows: source!.count,
+        controlCommandMs: Math.round(controlMs),
+        stopMs: Math.round(stopMs),
+        peakRunnerRetained: completed.peakRetainedEvents,
+      })}\n`,
+    );
+  } finally {
+    for (const child of children)
+      if (child.exitCode === null && child.signalCode === null) {
+        const exit = once(child, "exit");
+        child.kill("SIGKILL");
+        await exit;
+      }
+    if (bucketCreated) {
+      for (;;) {
+        const page = await client.send(new ListObjectsV2Command({ Bucket: config.bucket }));
+        if (!page.Contents?.length) break;
+        await client.send(
+          new DeleteObjectsCommand({
+            Bucket: config.bucket,
+            Delete: { Objects: page.Contents.map((object) => ({ Key: object.Key! })) },
+          }),
+        );
+      }
+      await client.send(new DeleteBucketCommand({ Bucket: config.bucket }));
+    }
+    client.destroy();
+    await db.execute(sql.raw(`drop schema if exists ${config.schema} cascade`));
+    await db.$client.end();
+    await fs.rm(root, { recursive: true, force: true });
+  }
+}

server/scripts/qualify-object-run-logs.ts

@@ -0,0 +1,213 @@
+import { randomUUID } from "node:crypto";
+import { spawn } from "node:child_process";
+import { once } from "node:events";
+import { promises as fs } from "node:fs";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import { setTimeout as delay } from "node:timers/promises";
+import {
+  S3Client,
+  CreateBucketCommand,
+  DeleteBucketCommand,
+  ListObjectsV2Command,
+  DeleteObjectsCommand,
+} from "@aws-sdk/client-s3";
+import { createS3StorageProvider } from "../src/storage/s3-provider.js";
+import { createObjectRunLogStore } from "../src/services/object-run-log-store.js";
+
+const endpoint = process.env.PAPERCLIP_LOG_TEST_S3_ENDPOINT;
+if (!endpoint || !["127.0.0.1", "localhost"].includes(new URL(endpoint).hostname)) {
+  throw new Error(
+    "Qualification requires PAPERCLIP_LOG_TEST_S3_ENDPOINT on a disposable loopback backend",
+  );
+}
+const childMode = process.argv[2] === "--child-writer";
+const bucket = childMode ? process.argv[3]! : `paperclip-logs-test-${randomUUID()}`;
+const raw = createS3StorageProvider({
+  endpoint,
+  bucket,
+  region: "us-east-1",
+  forcePathStyle: true,
+});
+const counts = { get: 0, put: 0, uploadedBytes: 0 };
+const latencies: number[] = [];
+const provider = {
+  ...raw,
+  async getObject(...args: Parameters<typeof raw.getObject>) {
+    counts.get++;
+    return raw.getObject(...args);
+  },
+  async putObjectConditional(...args: Parameters<NonNullable<typeof raw.putObjectConditional>>) {
+    counts.put++;
+    counts.uploadedBytes += args[0].contentLength;
+    const start = performance.now();
+    try {
+      return await raw.putObjectConditional!(...args);
+    } finally {
+      latencies.push(performance.now() - start);
+    }
+  },
+};
+const input = { companyId: "company", agentId: "agent", runId: "crash" };
+const options = {
+  provider,
+  keyPrefix: "qualification",
+  flushMs: 1000,
+  assertWriterAuthority: async () => undefined,
+};
+const output = (seq: number) => ({
+  stream: "stdout" as const,
+  seq,
+  chunk: `${seq}: 再開😀 ${"x".repeat(900)}`,
+  ts: "2026-10-30T00:00:00Z",
+});
+if (childMode) {
+  const store = createObjectRunLogStore(options),
+    handle = await store.begin(input);
+  const before = counts.put;
+  await Promise.all(Array.from({ length: 1000 }, (_, i) => store.append(handle, output(i + 1))));
+  process.stdout.write(
+    JSON.stringify({ receipt: true, batchPuts: counts.put - before, counts }) + "\n",
+  );
+  setTimeout(() => process.exit(2), 30_000);
+} else {
+  const client = new S3Client({ endpoint, region: "us-east-1", forcePathStyle: true });
+  await client.send(new CreateBucketCommand({ Bucket: bucket }));
+  const directory = await fs.mkdtemp(path.join(tmpdir(), "run-log-crash-"));
+  let child: ReturnType<typeof spawn> | undefined;
+  try {
+    const started = performance.now();
+    child = spawn(
+      process.execPath,
+      [
+        "--import",
+        path.resolve("server/node_modules/tsx/dist/loader.mjs"),
+        import.meta.filename,
+        "--child-writer",
+        bucket,
+      ],
+      { cwd: directory, env: process.env, stdio: ["ignore", "pipe", "pipe"] },
+    );
+    const exited = once(child, "exit");
+    let childError = "";
+    child.stderr!.on("data", (bytes) => {
+      childError = (childError + bytes).slice(-8192);
+    });
+    const receipt = await new Promise<{ batchPuts: number; counts: typeof counts }>(
+      (resolve, reject) => {
+        let text = "";
+        const timeout = setTimeout(
+          () => reject(new Error(`Writer receipt timed out: ${childError}`)),
+          30_000,
+        );
+        child!.stdout!.on("data", (bytes) => {
+          text += bytes;
+          if (!text.includes("\n")) return;
+          clearTimeout(timeout);
+          try {
+            resolve(JSON.parse(text.split("\n")[0]!));
+          } catch (error) {
+            reject(error);
+          }
+        });
+        child!.on("error", (error) => {
+          clearTimeout(timeout);
+          reject(error);
+        });
+        void exited.then(() => {
+          clearTimeout(timeout);
+          if (!text) reject(new Error(`Writer exited before its receipt: ${childError}`));
+        });
+      },
+    );
+    child.kill("SIGKILL");
+    const [, signal] = await exited;
+    if (signal !== "SIGKILL") throw new Error("Writer did not die by SIGKILL");
+    await fs.rm(directory, { recursive: true, force: true });
+    const successorStore = createObjectRunLogStore(options);
+    const successor = await successorStore.begin(input);
+    let content = "",
+      offset = 0;
+    for (;;) {
+      const page = await successorStore.read(successor, { offset, limitBytes: 131072 });
+      content += page.content;
+      if (page.nextOffset === undefined) break;
+      offset = page.nextOffset;
+    }
+    const lines = content
+      .trim()
+      .split("\n")
+      .map((line) => JSON.parse(line));
+    if (
+      lines.length !== 1000 ||
+      !lines.every((line, i) => line.seq === i + 1 && line.chunk === output(i + 1).chunk)
+    )
+      throw new Error("Replacement bytes did not match");
+    await successorStore.append(successor, output(1001));
+    const summary = await successorStore.finalize(successor);
+
+    const steadyBefore = { ...counts },
+      steadyStarted = performance.now();
+    const steady = createObjectRunLogStore(options);
+    const handles = await Promise.all(
+      Array.from({ length: 8 }, (_, i) => steady.begin({ ...input, runId: `steady-${i}` })),
+    );
+    const pending: Promise<number>[] = [];
+    for (let tick = 0; tick < 30; tick++) {
+      for (const handle of handles) pending.push(steady.append(handle, output(tick + 1)));
+      await delay(100);
+    }
+    await Promise.all(pending);
+    await Promise.all(handles.map((handle) => steady.finalize(handle)));
+    const objects = await client.send(new ListObjectsV2Command({ Bucket: bucket }));
+    if (objects.IsTruncated) throw new Error("Qualification inventory exceeded one bounded page");
+    latencies.sort((a, b) => a - b);
+    console.log(
+      JSON.stringify(
+        {
+          backend: "loopback S3-compatible test service",
+          flushMs: options.flushMs,
+          burst: {
+            records: lines.length,
+            batchPuts: receipt.batchPuts,
+            writerCounts: receipt.counts,
+          },
+          steady: {
+            streams: handles.length,
+            records: pending.length,
+            elapsedMs: Math.round(performance.now() - steadyStarted),
+            putRequests: counts.put - steadyBefore.put,
+            getRequests: counts.get - steadyBefore.get,
+            uploadedBytes: counts.uploadedBytes - steadyBefore.uploadedBytes,
+          },
+          readerCounts: counts,
+          committedCrashBytes: summary.bytes,
+          objects: objects.KeyCount,
+          storedBytes: objects.Contents?.reduce((sum, object) => sum + (object.Size ?? 0), 0),
+          putP95Ms: Math.round(latencies[Math.floor(latencies.length * 0.95)] ?? 0),
+          elapsedMs: Math.round(performance.now() - started),
+          abruptProcessLoss: signal,
+          oldDirectoryRemoved: true,
+          replacement: "passed",
+        },
+        null,
+        2,
+      ),
+    );
+  } finally {
+    child?.kill("SIGKILL");
+    await fs.rm(directory, { recursive: true, force: true });
+    for (;;) {
+      const page = await client.send(new ListObjectsV2Command({ Bucket: bucket, MaxKeys: 1000 }));
+      if (!page.Contents?.length) break;
+      await client.send(
+        new DeleteObjectsCommand({
+          Bucket: bucket,
+          Delete: { Objects: page.Contents.map((object) => ({ Key: object.Key! })) },
+        }),
+      );
+    }
+    await client.send(new DeleteBucketCommand({ Bucket: bucket }));
+    client.destroy();
+  }
+}

server/src/__tests__/agent-identity.test.ts

@@ -10,6 +10,7 @@ import { eq } from "drizzle-orm";
 import { agentIdentityKeys, agents, companies, createDb, closeRegisteredClients, ensurePostgresDatabase, runDatabaseBackup, runDatabaseRestore, type Db } from "@paperclipai/db";
 import { resolveWorktreeSeedPlan } from "../../../cli/src/commands/worktree-lib.js";
 import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
+import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
 import { agentIdentityService, supportsManagedAgentIdentity } from "../services/agent-identity.js";
 import { agentService } from "../services/agents.js";
 import { localEncryptedProvider } from "../secrets/local-encrypted-provider.js";
@@ -21,9 +22,10 @@ import { actorMiddleware } from "../middleware/auth.js";
 import { errorHandler } from "../middleware/error-handler.js";
 import { drainHeartbeatRunsToQuiescence } from "./helpers/drain-heartbeat-runs.js";
 
+const externalIdentityDatabase = resolveExternalTestDatabaseUrl("PAPERCLIP_AGENT_IDENTITY_TEST_DATABASE_URL");
 describe("agent cryptographic identity", () => {
   let db: Db;
-  let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
+  let database: Pick<Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>, "connectionString" | "cleanup">;
   let directory: string;
   let companyId: string;
 
@@ -32,7 +34,9 @@ describe("agent cryptographic identity", () => {
     vi.stubEnv("PAPERCLIP_HOME", directory);
     vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "");
     vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY_FILE", join(directory, "master.key"));
-    database = await startEmbeddedPostgresTestDatabase("paperclip-identity-");
+    database = externalIdentityDatabase ? { connectionString: externalIdentityDatabase,
+      cleanup: () => closeRegisteredClients(externalIdentityDatabase) }
+      : await startEmbeddedPostgresTestDatabase("paperclip-identity-");
     db = createDb(database.connectionString);
     companyId = randomUUID();
     await db.insert(companies).values({ id: companyId, name: "Identity test", issuePrefix: "KEY", defaultResponsibleUserId: "identity-test-owner" });
@@ -51,6 +55,7 @@ describe("agent cryptographic identity", () => {
   it("migration and public reads leave existing agents unprovisioned", async () => {
     const agent = await oldAgent();
     expect(await agentIdentityService(db).getPublicIdentity(companyId, agent.id)).toBeNull();
+    expect((await agentIdentityService(db).createExistingLogRedactor(companyId, agent.id)).values).toEqual([]);
     expect(await db.select().from(agentIdentityKeys).where(eq(agentIdentityKeys.agentId, agent.id))).toHaveLength(0);
     for (const type of ["http", "openclaw_gateway", "hermes_gateway"]) expect(supportsManagedAgentIdentity(type)).toBe(false);
     for (const provider of ["claude_managed", "aws_agentcore"]) expect(supportsManagedAgentIdentity("paperclip_runner", { provider })).toBe(false);

server/src/__tests__/heartbeat-run-privacy-routes.test.ts

@@ -32,8 +32,10 @@ import { activityRoutes } from "../routes/activity.js";
 import { agentRoutes } from "../routes/agents.js";
 import { authorizationService } from "../services/authorization.js";
 import { canActorReadWorkspaceOperation, getHeartbeatRunReadDecisions } from "../services/heartbeat-run-privacy.js";
+import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
 
-const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
+const externalDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_RUN_LOG_PRIVACY_TEST_DATABASE_URL");
+const embeddedPostgresSupport = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
 const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
 
 describeEmbeddedPostgres("heartbeat run privacy routes", { concurrent: false }, () => {
@@ -42,8 +44,8 @@ describeEmbeddedPostgres("heartbeat run privacy routes", { concurrent: false },
   const previousPrivacyMode = process.env.PAPERCLIP_ISSUE_PRIVACY_MODE;
 
   beforeAll(async () => {
-    tempDb = await startEmbeddedPostgresTestDatabase("paperclip-run-privacy-routes-");
-    db = createDb(tempDb.connectionString);
+    if (!externalDatabaseUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-run-privacy-routes-");
+    db = createDb(externalDatabaseUrl ?? tempDb!.connectionString);
     process.env.PAPERCLIP_ISSUE_PRIVACY_MODE = "enforce";
   }, 120_000);
 
@@ -66,6 +68,7 @@ describeEmbeddedPostgres("heartbeat run privacy routes", { concurrent: false },
     if (previousPrivacyMode === undefined) delete process.env.PAPERCLIP_ISSUE_PRIVACY_MODE;
     else process.env.PAPERCLIP_ISSUE_PRIVACY_MODE = previousPrivacyMode;
     await tempDb?.cleanup();
+    if (externalDatabaseUrl) await db.$client.end({ timeout: 1 });
   });
 
   async function seedFixture() {
@@ -264,7 +267,7 @@ describeEmbeddedPostgres("heartbeat run privacy routes", { concurrent: false },
       runnerProfileJson: { instructions: payload }, stdoutExcerpt: payload, stderrExcerpt: payload,
     }).where(eq(heartbeatRuns.id, f.privateRunId));
     const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.privateRunId));
-    const observed = createDb(tempDb!.connectionString), events: Array<Record<string, unknown>> = [], now = Date.now();
+    const observed = createDb(externalDatabaseUrl ?? tempDb!.connectionString), events: Array<Record<string, unknown>> = [], now = Date.now();
     const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
     installDatabaseResponseCapture(observed, { heartbeatRuns }, capture);
     try {
@@ -281,8 +284,9 @@ describeEmbeddedPostgres("heartbeat run privacy routes", { concurrent: false },
     } finally { capture.flush(); await observed.$client.end(); }
   });
 
-  it("returns 404 for direct private-run content and metadata-only list stubs to a non-member", async () => {
+  it.each(["local_file", "s3_segments"])("returns 404 for private %s run content and metadata-only list stubs", async (logStore) => {
     const fixture = await seedFixture();
+    await db.update(heartbeatRuns).set({ logStore }).where(eq(heartbeatRuns.id, fixture.privateRunId));
     const app = createApp(fixture.companyId, fixture.otherAgentId);
 
     for (const path of [

server/src/__tests__/helpers/native-run-log-database.ts

@@ -0,0 +1,22 @@
+import { sql } from "drizzle-orm";
+import type { Db } from "@paperclipai/db";
+
+export async function createNativeRunLogTestTables(db: Db) {
+  await db.execute(sql`create table heartbeat_runs(id uuid primary key, company_id uuid not null, agent_id uuid not null,
+    status text not null, runtime_mode text not null default 'native', native_session_id text, runner_instance_id text,
+    native_issue_id uuid, issue_id uuid, context_snapshot jsonb,
+    log_store text, log_ref text, log_bytes bigint,
+    log_sha256 text, last_output_seq bigint not null default 0, last_output_at timestamptz, last_output_stream text,
+    last_output_bytes bigint, next_event_seq bigint not null default 1, controller_lease_expires_at timestamptz,
+    created_at timestamptz not null default now(), updated_at timestamptz not null default now())`);
+  await db.execute(sql`create table heartbeat_run_events(id bigserial primary key, company_id uuid not null,
+    run_id uuid not null, agent_id uuid not null, seq bigint not null, event_type text not null, stream text,
+    level text, color text, message text, payload jsonb, source_instance_id text, source_event_id text,
+    source_seq bigint, source_payload_sha256 text, protocol_schema_version int, created_at timestamptz not null default now(),
+    unique(run_id,seq), unique(run_id,source_event_id), unique(run_id,source_instance_id,source_seq))`);
+  await db.execute(
+    sql`create table native_run_finalizations(run_id uuid, company_id uuid, lease_expires_at timestamptz)`,
+  );
+  await db.execute(sql`create table agent_identity_keys(agent_id uuid primary key,company_id uuid,algorithm text,key_id text,
+    public_key_pem text,private_key_material jsonb,created_at timestamptz default now())`);
+}

server/src/__tests__/issues-service.test.ts

@@ -48,6 +48,7 @@ import {
   readIssueCommentRunLogText,
 } from "../services/issues.ts";
 import { getRunLogStore } from "../services/run-log-store.js";
+import * as nativeRunLogRecovery from "../services/native-runtime/native-run-log-recovery.js";
 import { terminalizeLegacyExecution } from "../services/legacy-execution-recovery.js";
 import { settleUnrecoverableExecutions } from "../services/execution-recovery-resolution.js";
 import {
@@ -192,6 +193,35 @@ describeEmbeddedPostgres("issueService run attachment artifacts", () => {
 });
 
 describe("readIssueCommentRunLogText", () => {
+  it("does not read after a timed-out native repair", async () => {
+    const repair = vi.spyOn(nativeRunLogRecovery, "recoverSettledNativeRunLog").mockImplementationOnce((_db, _store, _ref, _redaction, signal) => new Promise((_resolve, reject) => {
+      signal!.addEventListener("abort", () => reject(signal!.reason), { once: true });
+    }));
+    const read = vi.spyOn(getRunLogStore(), "read").mockResolvedValue({ content: "too late" });
+    vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] });
+    const pending = readIssueCommentRunLogText({ logStore: "s3_segments", logRef: "company/agent/run", logBytes: null }, {} as ReturnType<typeof createDb>);
+    try {
+      await vi.advanceTimersByTimeAsync(3_000);
+      await expect(pending).resolves.toBe("");
+      expect(read).not.toHaveBeenCalled();
+    } finally {
+      await pending;
+      vi.useRealTimers();
+      repair.mockRestore();
+      read.mockRestore();
+    }
+  });
+  it("keeps published evidence when a concurrent native repair wins ownership", async () => {
+    const repair = vi.spyOn(nativeRunLogRecovery, "recoverSettledNativeRunLog").mockRejectedValueOnce(new Error("The output journal is owned by another writer"));
+    const read = vi.spyOn(getRunLogStore(), "read").mockResolvedValueOnce({ content: "published evidence" });
+    try {
+      await expect(readIssueCommentRunLogText({ logStore: "s3_segments", logRef: "company/agent/run", logBytes: null }, {} as ReturnType<typeof createDb>)).resolves.toBe("published evidence");
+      expect(read).toHaveBeenCalledOnce();
+    } finally {
+      repair.mockRestore();
+      read.mockRestore();
+    }
+  });
   it("cancels timed-out storage reads so later listings can recover", async () => {
     let active = 0;
     const cleanups: Array<() => void> = [];

server/src/__tests__/local-encrypted-cipher.test.ts

@@ -1,9 +1,9 @@
 import { randomBytes } from "node:crypto";
-import { mkdtempSync, rmSync, unlinkSync, writeFileSync } from "node:fs";
+import { existsSync, mkdtempSync, rmSync, unlinkSync, writeFileSync } from "node:fs";
 import { tmpdir } from "node:os";
 import { join } from "node:path";
 import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
-import { createLocalEncryptedCipher, localEncryptedProvider } from "../secrets/local-encrypted-provider.js";
+import { createLocalEncryptedCipher, deriveLocalEncryptedInstanceSecret, localEncryptedProvider } from "../secrets/local-encrypted-provider.js";
 
 describe("operation-scoped local encryption", () => {
   let root: string;
@@ -60,4 +60,16 @@ describe("operation-scoped local encryption", () => {
       code: "provider_unavailable", operation: "resolveVersion",
     }));
   });
+  it("derives the same instance secret from an existing key file or environment without creating missing keys", () => {
+    expect(() => deriveLocalEncryptedInstanceSecret("trace-test", process.env, { allowCreate: false }))
+      .toThrow("Secrets master key file is missing");
+    expect(existsSync(keyFile)).toBe(false);
+    const key = randomBytes(32).toString("hex");
+    writeFileSync(keyFile, key, { mode: 0o600 });
+    const fromFile = deriveLocalEncryptedInstanceSecret("trace-test", process.env, { allowCreate: false });
+    unlinkSync(keyFile);
+    vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", key);
+    expect(deriveLocalEncryptedInstanceSecret("trace-test", process.env, { allowCreate: false })).toBe(fromFile);
+    expect(existsSync(keyFile)).toBe(false);
+  });
 });

server/src/__tests__/provider-trace-retention-work.test.ts

@@ -0,0 +1,66 @@
+import type { Db } from "@paperclipai/db";
+import { installDatabaseWorkSignals } from "../../../packages/db/src/work-signals.js";
+import { afterEach, expect, it, vi } from "vitest";
+import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
+import { registerProviderTraceRetentionWork } from "../services/provider-trace-retention-work.js";
+
+const state = vi.hoisted(() => ({
+  cleanup: vi.fn(async () => []),
+  next: vi.fn<() => Promise<number | null>>(),
+  probe: undefined as undefined | (() => string),
+  unregister: vi.fn(),
+}));
+vi.mock("../services/provider-trace-store.js", () => ({
+  providerTraceStore: () => ({ cleanupExpired: state.cleanup, nextRetentionDeadline: state.next }),
+}));
+vi.mock("../services/idle-local-work.js", () => ({
+  registerIdleLocalWorkProbe: (probe: () => string) => {
+    state.probe = probe;
+    return state.unregister;
+  },
+}));
+let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
+afterEach(async () => {
+  await coordinator?.stop();
+  vi.useRealTimers();
+  vi.clearAllMocks();
+});
+
+it("reconstructs trace deadlines, stays quiet between obligations, and discharges the sleep blocker", async () => {
+  vi.useFakeTimers();
+  const owner = installDatabaseWorkSignals({ execute: vi.fn(async () => []) });
+  const onError = vi.fn();
+  coordinator = createDeliveryWorkCoordinator({ owner, canRun: () => true, onError });
+  state.next.mockResolvedValue(null);
+  const worker = registerProviderTraceRetentionWork(
+    owner as unknown as Db,
+    coordinator,
+    () => true,
+  );
+  expect(state.probe!()).toBe("unknown");
+  await worker.ready;
+  expect(state.probe!()).toBe("none");
+  expect(coordinator.nextWakeAt()).toBeNull();
+  const initial = state.cleanup.mock.calls.length;
+  await vi.advanceTimersByTimeAsync(300_000);
+  expect(state.cleanup).toHaveBeenCalledTimes(initial);
+  expect(state.next).toHaveBeenCalledOnce();
+
+  const deadline = Date.now() + 60_000;
+  state.next.mockResolvedValue(deadline);
+  worker.wake();
+  await vi.advanceTimersByTimeAsync(0);
+  expect(state.probe!()).toBe("present");
+  expect(coordinator.nextWakeAt()).toBe(deadline);
+  const scheduled = state.cleanup.mock.calls.length;
+  await vi.advanceTimersByTimeAsync(59_999);
+  expect(state.cleanup).toHaveBeenCalledTimes(scheduled);
+  state.next.mockResolvedValue(null);
+  await vi.advanceTimersByTimeAsync(1);
+  expect(state.cleanup).toHaveBeenCalledTimes(scheduled + 1);
+  expect(state.probe!()).toBe("none");
+  expect(coordinator.nextWakeAt()).toBeNull();
+  await worker.stop();
+  expect(state.unregister).toHaveBeenCalledOnce();
+  expect(onError).not.toHaveBeenCalled();
+});

server/src/__tests__/run-evidence-migration-work.test.ts

@@ -0,0 +1,168 @@
+import type { Db } from "@paperclipai/db";
+import { installDatabaseWorkSignals } from "../../../packages/db/src/work-signals.js";
+import { afterEach, expect, it, vi } from "vitest";
+import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
+import { registerRunEvidenceMigrationWork } from "../services/run-evidence-migration-work.js";
+
+const state = vi.hoisted(() => ({
+  provider: "s3",
+  migrate: vi.fn(),
+  probe: undefined as undefined | (() => string),
+  unregister: vi.fn(),
+  info: vi.fn(),
+  warn: vi.fn(),
+}));
+vi.mock("../storage/config.js", () => ({
+  loadStorageConfig: () => ({ storageProvider: state.provider }),
+}));
+vi.mock("../services/run-evidence-migration.js", () => ({ migrateRunEvidence: state.migrate }));
+vi.mock("../middleware/logger.js", () => ({ logger: { info: state.info, warn: state.warn } }));
+vi.mock("../services/idle-local-work.js", () => ({
+  registerIdleLocalWorkProbe: (probe: () => string) => {
+    state.probe = probe;
+    return state.unregister;
+  },
+}));
+let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
+afterEach(async () => {
+  await coordinator?.stop();
+  coordinator = undefined;
+  state.provider = "s3";
+  state.probe = undefined;
+  vi.resetAllMocks();
+  vi.useRealTimers();
+});
+function setup(canRun = () => true) {
+  vi.useFakeTimers();
+  const owner = installDatabaseWorkSignals({ execute: vi.fn(async () => []) });
+  const onError = vi.fn();
+  coordinator = createDeliveryWorkCoordinator({ owner, canRun, canReconcile: () => true, onError });
+  const worker = registerRunEvidenceMigrationWork(owner as unknown as Db, coordinator, canRun);
+  return { worker, onError };
+}
+it("does no migration work for local storage", async () => {
+  state.provider = "local_disk";
+  const { worker } = setup();
+  expect(worker).toBeNull();
+  await vi.advanceTimersByTimeAsync(3_600_000);
+  expect(state.migrate).not.toHaveBeenCalled();
+  expect(state.probe).toBeUndefined();
+  expect(coordinator!.nextWakeAt()).toBeNull();
+});
+it("automatically pages both resources, then leaves completed history quiet", async () => {
+  state.migrate
+    .mockResolvedValueOnce({
+      items: [{ kind: "log", id: "one", state: "migrated" }],
+      nextRunId: "one",
+    })
+    .mockResolvedValueOnce({
+      items: [{ kind: "log", id: "two", state: "migrated" }],
+      nextRunId: "two",
+    })
+    .mockResolvedValue({ items: [] });
+  const { worker, onError } = setup();
+  expect(state.probe!()).toBe("unknown");
+  await worker!.ready;
+  expect(state.migrate).toHaveBeenNthCalledWith(
+    1,
+    expect.anything(),
+    expect.objectContaining({
+      apply: true,
+      limit: 1,
+      kinds: ["log", "trace"],
+    }),
+  );
+  expect(state.probe!()).toBe("present");
+  await vi.advanceTimersByTimeAsync(2_000);
+  expect(state.migrate).toHaveBeenNthCalledWith(
+    2,
+    expect.anything(),
+    expect.objectContaining({
+      afterRunId: "one",
+      kinds: ["log"],
+    }),
+  );
+  expect(state.migrate).toHaveBeenNthCalledWith(
+    3,
+    expect.anything(),
+    expect.objectContaining({
+      afterRunId: "two",
+      kinds: ["log"],
+    }),
+  );
+  expect(state.probe!()).toBe("none");
+  expect(coordinator!.nextWakeAt()).toBeNull();
+  await vi.advanceTimersByTimeAsync(3_600_000);
+  expect(state.migrate).toHaveBeenCalledTimes(3);
+  expect(onError).not.toHaveBeenCalled();
+  await worker!.stop();
+  expect(state.unregister).toHaveBeenCalledOnce();
+});
+it("keeps blocked history visible and retries it without stranding later pages", async () => {
+  state.migrate
+    .mockResolvedValueOnce({
+      items: [{ kind: "log", id: "active", state: "blocked", reason: "legacy_writer_not_settled" }],
+      nextRunId: "active",
+    })
+    .mockResolvedValueOnce({ items: [{ kind: "log", id: "later", state: "migrated" }] })
+    .mockResolvedValue({ items: [{ kind: "log", id: "active", state: "migrated" }] });
+  const { worker } = setup();
+  await worker!.ready;
+  await vi.advanceTimersByTimeAsync(1_000);
+  expect(state.warn).toHaveBeenCalledWith(
+    expect.objectContaining({ id: "active", reason: "legacy_writer_not_settled" }),
+    expect.any(String),
+  );
+  expect(state.probe!()).toBe("present");
+  await vi.advanceTimersByTimeAsync(299_999);
+  expect(state.migrate).toHaveBeenCalledTimes(2);
+  await vi.advanceTimersByTimeAsync(1);
+  expect(state.migrate).toHaveBeenNthCalledWith(
+    3,
+    expect.anything(),
+    expect.objectContaining({ afterRunId: undefined, kinds: ["log", "trace"] }),
+  );
+  expect(state.probe!()).toBe("none");
+  expect(coordinator!.nextWakeAt()).toBeNull();
+});
+it("retains startup work during drain and retries a failed sweep before becoming quiet", async () => {
+  let admitted = false;
+  state.migrate
+    .mockRejectedValueOnce(new Error("storage unavailable"))
+    .mockResolvedValue({ items: [] });
+  const { worker, onError } = setup(() => admitted);
+  await worker!.ready;
+  await vi.advanceTimersByTimeAsync(60_000);
+  expect(state.migrate).not.toHaveBeenCalled();
+  expect(state.probe!()).toBe("unknown");
+  admitted = true;
+  await vi.advanceTimersByTimeAsync(30_000);
+  expect(onError).toHaveBeenCalledOnce();
+  await vi.advanceTimersByTimeAsync(30_000);
+  expect(state.migrate).toHaveBeenCalledTimes(2);
+  expect(state.probe!()).toBe("none");
+  expect(coordinator!.nextWakeAt()).toBeNull();
+});
+it("aborts owned work on shutdown and reconstructs outstanding history on restart", async () => {
+  let started: AbortSignal | undefined;
+  state.migrate.mockImplementationOnce(
+    (_db, options) =>
+      new Promise((_resolve, reject) => {
+        started = options.signal;
+        started!.addEventListener("abort", () => reject(started!.reason), { once: true });
+      }),
+  );
+  const { worker } = setup();
+  await vi.advanceTimersByTimeAsync(0);
+  expect(started?.aborted).toBe(false);
+  await worker!.stop();
+  await worker!.ready;
+  expect(started?.aborted).toBe(true);
+  expect(coordinator!.nextWakeAt()).toBeNull();
+  await coordinator!.stop();
+  state.migrate.mockResolvedValue({ items: [] });
+  const restarted = setup();
+  await restarted.worker!.ready;
+  expect(state.migrate).toHaveBeenCalledTimes(2);
+  expect(state.probe!()).toBe("none");
+});

server/src/__tests__/server-startup-feedback-export.test.ts

@@ -697,6 +697,12 @@ describe("startServer feedback export wiring", () => {
     expect(deliveryWorkErrors.filter(error => error.queue === DELIVERY_QUEUES.nativeRunnerEvidenceRetention)).toEqual([]);
   });
 
+  it.each([false, true])("registers trace retention independently of heartbeat scheduling (%s)", async enabled => {
+    loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled }));
+    await startServer(); await settleStartupHousekeeping();
+    expect(deliveryWorkRegisterMock.mock.calls.filter(([queue]) => queue === DELIVERY_QUEUES.providerTraceRetention)).toHaveLength(1);
+  });
+
   it.each([false, true])("registers skill cache retention only when enabled (%s)", async enabled => {
     loadConfigMock.mockReturnValue(buildTestConfig({ skillVersionRetentionEnabled: enabled }));
     await startServer(); await settleStartupHousekeeping();

server/src/config.ts

@@ -16,13 +16,11 @@ import {
   BIND_MODES,
   DEPLOYMENT_EXPOSURES,
   SECRET_PROVIDERS,
-  STORAGE_PROVIDERS,
   type BindMode,
   type AuthBaseUrlMode,
   type DeploymentExposure,
   type DeploymentMode,
   type SecretProvider,
-  type StorageProvider,
   inferBindModeFromHost,
   resolveRuntimeBind,
   validateConfiguredBindMode,
@@ -31,9 +29,9 @@ import {
   resolveDefaultBackupDir,
   resolveDefaultEmbeddedPostgresDir,
   resolveDefaultSecretsKeyFilePath,
-  resolveDefaultStorageDir,
   resolveHomeAwarePath,
 } from "./home-paths.js";
+import { loadStorageConfig, type StorageConfig } from "./storage/config.js";
 
 if (!isCloudManagedInstance()) {
   const PAPERCLIP_ENV_FILE_PATH = resolvePaperclipEnvPath();
@@ -63,7 +61,7 @@ const TAILSCALE_DETECT_TIMEOUT_MS = 3000;
 
 type DatabaseMode = "embedded-postgres" | "postgres";
 
-export interface Config {
+export interface Config extends StorageConfig {
   deploymentMode: DeploymentMode;
   deploymentExposure: DeploymentExposure;
   bind: BindMode;
@@ -116,13 +114,6 @@ export interface Config {
   secretsProvider: SecretProvider;
   secretsStrictMode: boolean;
   secretsMasterKeyFilePath: string;
-  storageProvider: StorageProvider;
-  storageLocalDiskBaseDir: string;
-  storageS3Bucket: string;
-  storageS3Region: string;
-  storageS3Endpoint: string | undefined;
-  storageS3Prefix: string;
-  storageS3ForcePathStyle: boolean;
   feedbackExportBackendUrl: string | undefined;
   feedbackExportBackendToken: string | undefined;
   heartbeatSchedulerEnabled: boolean;
@@ -162,7 +153,6 @@ export function loadConfig(fileConfig: PaperclipConfig | null = readConfigFile()
       : undefined;
   const fileDatabaseBackup = fileConfig?.database.backup;
   const fileSecrets = fileConfig?.secrets;
-  const fileStorage = fileConfig?.storage;
 
   const providerFromEnvRaw = process.env.PAPERCLIP_SECRETS_PROVIDER;
   const providerFromEnv =
@@ -172,25 +162,7 @@ export function loadConfig(fileConfig: PaperclipConfig | null = readConfigFile()
   const providerFromFile = fileSecrets?.provider;
   const secretsProvider: SecretProvider = providerFromEnv ?? providerFromFile ?? "local_encrypted";
 
-  const storageProviderFromEnvRaw = process.env.PAPERCLIP_STORAGE_PROVIDER;
-  const storageProviderFromEnv =
-    storageProviderFromEnvRaw && STORAGE_PROVIDERS.includes(storageProviderFromEnvRaw as StorageProvider)
-      ? (storageProviderFromEnvRaw as StorageProvider)
-      : null;
-  const storageProvider: StorageProvider = storageProviderFromEnv ?? fileStorage?.provider ?? "local_disk";
-  const storageLocalDiskBaseDir = resolveHomeAwarePath(
-    process.env.PAPERCLIP_STORAGE_LOCAL_DIR ??
-      fileStorage?.localDisk?.baseDir ??
-      resolveDefaultStorageDir(),
-  );
-  const storageS3Bucket = process.env.PAPERCLIP_STORAGE_S3_BUCKET ?? fileStorage?.s3?.bucket ?? "paperclip";
-  const storageS3Region = process.env.PAPERCLIP_STORAGE_S3_REGION ?? fileStorage?.s3?.region ?? "us-east-1";
-  const storageS3Endpoint = process.env.PAPERCLIP_STORAGE_S3_ENDPOINT ?? fileStorage?.s3?.endpoint ?? undefined;
-  const storageS3Prefix = process.env.PAPERCLIP_STORAGE_S3_PREFIX ?? fileStorage?.s3?.prefix ?? "";
-  const storageS3ForcePathStyle =
-    process.env.PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE !== undefined
-      ? process.env.PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE === "true"
-      : (fileStorage?.s3?.forcePathStyle ?? false);
+  const storage = loadStorageConfig(fileConfig);
   const feedbackExportBackendUrl =
     process.env.PAPERCLIP_FEEDBACK_EXPORT_BACKEND_URL?.trim() ||
     process.env.PAPERCLIP_TELEMETRY_BACKEND_URL?.trim() ||
@@ -447,13 +419,7 @@ export function loadConfig(fileConfig: PaperclipConfig | null = readConfigFile()
           fileSecrets?.localEncrypted.keyFilePath ??
           resolveDefaultSecretsKeyFilePath(),
       ),
-    storageProvider,
-    storageLocalDiskBaseDir,
-    storageS3Bucket,
-    storageS3Region,
-    storageS3Endpoint,
-    storageS3Prefix,
-    storageS3ForcePathStyle,
+    ...storage,
     feedbackExportBackendUrl,
     feedbackExportBackendToken,
     heartbeatSchedulerEnabled: process.env.HEARTBEAT_SCHEDULER_ENABLED !== "false",

server/src/index.ts

@@ -1,3 +1,5 @@
+import { registerProviderTraceRetentionWork } from "./services/provider-trace-retention-work.js";
+import { registerRunEvidenceMigrationWork } from "./services/run-evidence-migration-work.js";
 import { registerSkillVersionRetentionWork } from "./services/skill-version-retention-work.js";
 import { registerDatabaseBackupWork } from "./services/database-backup-work.js";
 import { registerToolConnectionHealthWork } from "./services/tool-connection-health-work.js";
@@ -1365,6 +1367,13 @@ async function startServerWithDatabaseTeardown(
     }));
   });
   server.on("close", () => { void taskDrainExpiryWork.stop(); });
+  const runEvidenceMigrationWork = registerRunEvidenceMigrationWork(db, deliveryWork,
+    () => !heartbeatSchedulerStopped && !isIdleTaskDrainActive() && !isWarmStandby(), { storageProvider: config.storageProvider });
+  if (runEvidenceMigrationWork) trackHeartbeatSchedulerWork(runEvidenceMigrationWork.ready);
+  const providerTraceRetentionWork = registerProviderTraceRetentionWork(db, deliveryWork,
+    () => !heartbeatSchedulerStopped && !isIdleTaskDrainActive() && !isWarmStandby());
+  trackHeartbeatSchedulerWork(providerTraceRetentionWork.ready);
+  server.on("close", providerTraceRetentionWork.stop);
   const executionSchedule = executionControlSchedule(db);
   const workspaceRecoveryRuntime = environmentRuntimeService(db, { pluginWorkerManager });
   const nativeRunnerEvidenceRetentionWorker = config.nativeHarnessBackupExpirationEnabled
@@ -2236,6 +2245,8 @@ async function startServerWithDatabaseTeardown(
     unsubscribeNativeWorkspaceFinalization?.();
     unsubscribeRetainedNativeSessionCleanup?.();
     await deliveryWork.stop();
+    await runEvidenceMigrationWork?.stop();
+    await providerTraceRetentionWork.stop();
     await skillVersionRetentionWorker?.dispose();
     await workspaceRecoveryWork.stop();
     unregisterTerminalWorkspaceProbe?.();

server/src/secrets/local-encrypted-provider.ts

@@ -94,8 +94,8 @@ function loadOrCreateMasterKey(allowCreate = true, env: LocalEncryptedKeyEnv = p
 }
 
 /** Derive a purpose-specific instance secret without exposing or reusing the vault key. */
-export function deriveLocalEncryptedInstanceSecret(purpose: string, env: LocalEncryptedKeyEnv = process.env): string {
-  return Buffer.from(hkdfSync("sha256", loadOrCreateMasterKey(true, env), Buffer.alloc(0),
+export function deriveLocalEncryptedInstanceSecret(purpose: string, env: LocalEncryptedKeyEnv = process.env, options: { allowCreate?: boolean } = {}): string {
+  return Buffer.from(hkdfSync("sha256", loadOrCreateMasterKey(options.allowCreate ?? true, env), Buffer.alloc(0),
     `paperclip:instance-secret:v1\0${purpose}`, 32)).toString("base64url");
 }
 

server/src/services/agent-identity.ts

@@ -5,6 +5,7 @@ import type { AgentPublicIdentity } from "@paperclipai/shared";
 import { notFound } from "../errors.js";
 import { localEncryptedProvider } from "../secrets/local-encrypted-provider.js";
 import { persistActivity } from "./activity-log.js";
+import { createAgentIdentityRedactor } from "./agent-identity-redaction.js";
 
 const publicColumns = {
   algorithm: agentIdentityKeys.algorithm,
@@ -17,6 +18,16 @@ export function agentIdentityService(db: Db) {
   const ownership = (companyId: string, agentId: string) => and(
     eq(agentIdentityKeys.companyId, companyId), eq(agentIdentityKeys.agentId, agentId),
   );
+  async function resolveIdentity(row: typeof agentIdentityKeys.$inferSelect) {
+    const privateKeyPem = await localEncryptedProvider.resolveVersion({ material: row.privateKeyMaterial, externalRef: null });
+    const privateKey = createPrivateKey(privateKeyPem);
+    const derived = createPublicKey(privateKey).export({ type: "spki", format: "der" });
+    const stored = createPublicKey(row.publicKeyPem).export({ type: "spki", format: "der" });
+    if (row.algorithm !== "Ed25519" || privateKey.asymmetricKeyType !== "ed25519" ||
+      !derived.equals(stored) || fingerprint(stored) !== row.keyId)
+      throw new Error("Agent identity public and private keys do not match");
+    return { keyId: row.keyId, publicKeyPem: row.publicKeyPem, privateKeyPem };
+  }
 
   return {
     // Reading an identity must never provision it or decrypt private material.
@@ -24,6 +35,10 @@ export function agentIdentityService(db: Db) {
       const [row] = await db.select(publicColumns).from(agentIdentityKeys).where(ownership(companyId, agentId));
       return row ? { ...row, createdAt: row.createdAt.toISOString() } : null;
     },
+    async createExistingLogRedactor(companyId: string, agentId: string) {
+      const [row] = await db.select().from(agentIdentityKeys).where(ownership(companyId, agentId));
+      return createAgentIdentityRedactor(row ? (await resolveIdentity(row)).privateKeyPem : undefined);
+    },
 
     async ensureAgentIdentity(companyId: string, agentId: string) {
       return db.transaction(async (tx) => {
@@ -49,18 +64,8 @@ export function agentIdentityService(db: Db) {
             details: { algorithm: "Ed25519", keyId },
           });
         }
-        const privateKeyPem = await localEncryptedProvider.resolveVersion({
-          material: row.privateKeyMaterial, externalRef: null,
-        });
         // Fail closed on corruption or a wrong master key; never replace a key.
-        const privateKey = createPrivateKey(privateKeyPem);
-        const derived = createPublicKey(privateKey).export({ type: "spki", format: "der" });
-        const stored = createPublicKey(row.publicKeyPem).export({ type: "spki", format: "der" });
-        if (row.algorithm !== "Ed25519" || privateKey.asymmetricKeyType !== "ed25519" ||
-            !derived.equals(stored) || fingerprint(stored) !== row.keyId) {
-          throw new Error("Agent identity public and private keys do not match");
-        }
-        return { keyId: row.keyId, publicKeyPem: row.publicKeyPem, privateKeyPem };
+        return resolveIdentity(row);
       });
     },
   };

server/src/services/customer-success-inspection.ts

@@ -20,7 +20,7 @@ import {
 import { deriveBundleState } from "./agent-instructions.js";
 import { readInstructionBytes } from "./agent-instruction-files.js";
 import { skillPackageStore } from "./skill-package-store.js";
-import { getRunLogStore } from "./run-log-store.js";
+import { getRunLogStore, isRunLogStoreType } from "./run-log-store.js";
 
 // This is a reviewed catalog of readers, not a proxy for existing GET routes.
 // Select credentials out of queries; never load encrypted identity records.
@@ -311,9 +311,9 @@ export async function readCustomerSuccessResource(
       }
       if (q.operation === "runs.log") {
         const run = await ownRow(tx, schema.heartbeatRuns, q.companyId!, q.resourceId!);
-        if (!run.logRef || run.logStore !== "local_file") return unavailable("run_log_unavailable");
+        if (!run.logRef || !isRunLogStoreType(run.logStore)) return unavailable("run_log_unavailable");
         const result = await getRunLogStore().read(
-          { store: "local_file", logRef: String(run.logRef) },
+          { store: run.logStore, logRef: String(run.logRef) },
           {
             offset: q.range?.start ?? q.offset ?? 0,
             limitBytes: q.range ? q.range.end - q.range.start + 1 : 256000,

server/src/services/delivery-work-notifications.ts

@@ -2,6 +2,8 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db";
 
 /** Topics for existing durable queues; register intent before their writes. */
 export const DELIVERY_QUEUES = {
+  runEvidenceMigration: "run-evidence-migration",
+  providerTraceRetention: "provider-trace-retention",
   nativeRunnerEvidenceRetention: "native-runner-evidence-retention",
   skillVersionRetention: "skill-version-retention",
   databaseBackups: "database-backups",

server/src/services/feedback.ts

@@ -47,7 +47,8 @@ import {
   sanitizeFeedbackValue,
   sha256Digest,
 } from "./feedback-redaction.js";
-import { getRunLogStore } from "./run-log-store.js";
+import { getRunLogStore, isRunLogStoreType } from "./run-log-store.js";
+import { recoverSettledNativeRunLog } from "./native-runtime/native-run-log-recovery.js";
 import { getOperatorSettingDefaults } from "./setting-defaults.js";
 
 const FEEDBACK_SCHEMA_VERSION = "paperclip-feedback-envelope-v2";
@@ -318,17 +319,18 @@ async function findMatchingFile(
   return search(rootDir, 0);
 }
 
-async function readFullRunLog(run: {
+async function readFullRunLog(db: Db, run: {
   logStore: string | null;
   logRef: string | null;
 }) {
-  if (run.logStore !== "local_file" || !run.logRef) return null;
+  if (!isRunLogStoreType(run.logStore) || !run.logRef) return null;
   const store = getRunLogStore();
+  if (run.logStore === "s3_segments") await recoverSettledNativeRunLog(db, store, run.logRef).catch(() => undefined);
   let offset = 0;
   let combined = "";
 
   while (true) {
-    const result = await store.read({ store: "local_file", logRef: run.logRef }, {
+    const result = await store.read({ store: run.logStore, logRef: run.logRef }, {
       offset,
       limitBytes: 512_000,
     }).catch(() => null);
@@ -1526,7 +1528,7 @@ async function buildFeedbackTraceBundleFromRow(
         .from(heartbeatRunEvents)
         .where(eq(heartbeatRunEvents.runId, run.id))
         .orderBy(asc(heartbeatRunEvents.seq));
-      const logText = await readFullRunLog(run);
+      const logText = await readFullRunLog(db, run);
       const logEntries = parseRunLogEntries(logText);
       const stdoutText = logEntries
         .filter((entry) => entry.stream === "stdout")

server/src/services/heartbeat.ts

@@ -25,7 +25,10 @@ import { createHeartbeatPreparationTiming } from "./heartbeat/preparation-timing
 import { createHeartbeatInstructionCopyRelease, createHeartbeatInstructionWorkingCopy } from "./heartbeat/instruction-working-copy.js";
 import { prepareHeartbeatRuntimeServiceIntents, startHeartbeatRuntimeServices, recordHeartbeatAdapterRuntimeServices } from "./heartbeat/runtime-services.js";
 import { bindHeartbeatRunnerHomeMode, prepareHeartbeatExecutionConfiguration } from "./heartbeat/execution-configuration.js";
+import { createRunLogWriterAuthority } from "./heartbeat/run-log-authority.js";
 import { createHeartbeatRunOutput } from "./heartbeat/run-output.js";
+import { readRecoverableRunLog } from "./native-runtime/native-run-log-recovery.js";
+import { observabilitySecretValues } from "@paperclipai/adapter-utils/agent-observability";
 import { prepareHeartbeatSession, resolveHeartbeatAdapterSession } from "./heartbeat/session-preparation.js";
 import { prepareHeartbeatTaskContext } from "./heartbeat/task-context.js";
 import { selectHeartbeatEnvironment } from "./heartbeat/environment-selection.js";
@@ -370,7 +373,7 @@ import {
   type NativeSessionBackend,
 } from "../vendor/paperclip-runner/index.js";
 
-import { getRunLogStore } from "./run-log-store.js";
+import { getRunLogStore, isRunLogStoreType } from "./run-log-store.js";
 import {
   providerTraceStore,
 } from "./provider-trace-store.js";
@@ -2075,6 +2078,7 @@ export function heartbeatService(
         }
         try {
           providerTraceCapture = await traceStore.prepare({
+            assertWriterAuthority: createRunLogWriterAuthority(db, run, runOptions.nativeLeaseOwner),
             runId: run.id,
             companyId: run.companyId,
             provider:
@@ -3116,6 +3120,8 @@ export function heartbeatService(
       const runOutput = createHeartbeatRunOutput(db, {
         run, runId, issueId, runLogStore, identityRedactor, secretKeys,
         appendRunEvent, getCurrentUserRedactionOptions,
+        assertWriterAuthority: createRunLogWriterAuthority(db, run, runOptions.nativeLeaseOwner),
+        runtimeLogSecretValues: observabilitySecretValues(observability),
       });
       const goalCheckpointSession: {
         current: {
@@ -3201,7 +3207,7 @@ export function heartbeatService(
           message: "run started",
         });
 
-        const { currentUserRedactionOptions, appendIdentityRedactedLog, onLog, onAdapterMeta, onAdapterEvent } =
+        const { currentUserRedactionOptions, appendIdentityRedactedLog, onLog, onPersistedNativeLog, onAdapterMeta, onAdapterEvent } =
           await timePreparation("run_output", () => runOutput.begin(currentRun));
         if (runScopedMentionedSkillKeys.length > 0) {
           await onLog(
@@ -3435,6 +3441,7 @@ export function heartbeatService(
             identityRedactor,
             appendIdentityRedactedLog,
             onLog,
+            onPersistedNativeLog,
             onAdapterEvent,
             onAdapterMeta,
           },
@@ -4055,6 +4062,9 @@ export function heartbeatService(
             companyId: string;
             logStore: string | null;
             logRef: string | null;
+            runtimeMode?: string | null;
+            status?: string;
+            logBytes?: number | null;
           },
       opts?: { offset?: number; limitBytes?: number; alignStartToRecord?: boolean; signal?: AbortSignal },
     ) => {
@@ -4067,12 +4077,14 @@ export function heartbeatService(
       if (!run) throw notFound("Heartbeat run not found");
       if (!run.logStore || !run.logRef) throw notFound("Run log not found");
 
-      const result = await runLogStore.read(
+      if (!isRunLogStoreType(run.logStore)) throw notFound("Run log backend not supported");
+      const result = await readRecoverableRunLog(db, runLogStore,
         {
-          store: run.logStore as "local_file",
+          store: run.logStore,
           logRef: run.logRef,
         },
         opts,
+        run,
       );
 
       return {

server/src/services/heartbeat/run-log-authority.ts

@@ -0,0 +1,65 @@
+import { and, eq } from "drizzle-orm";
+import { heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db";
+import { legacyControllerBootId } from "../legacy-controller-lease.js";
+import { currentNativeControllerIdentity } from "../native-runtime/native-restart-recovery.js";
+
+export function createRunLogWriterAuthority(
+  db: Db,
+  admitted: typeof heartbeatRuns.$inferSelect,
+  nativeLeaseOwner?: string,
+) {
+  return async () => {
+    const [run] = await db
+      .select({
+        id: heartbeatRuns.id,
+        runtimeMode: heartbeatRuns.runtimeMode,
+        controllerBootId: heartbeatRuns.controllerBootId,
+        leaseExpiresAt: heartbeatRuns.controllerLeaseExpiresAt,
+        logStore: heartbeatRuns.logStore,
+        logRef: heartbeatRuns.logRef,
+      })
+      .from(heartbeatRuns)
+      .where(
+        and(
+          eq(heartbeatRuns.id, admitted.id),
+          eq(heartbeatRuns.companyId, admitted.companyId),
+          eq(heartbeatRuns.agentId, admitted.agentId),
+          eq(heartbeatRuns.status, "running"),
+        ),
+      );
+    if (!run) throw new Error("run_log_writer_authority_lost");
+    if (run.runtimeMode !== "native") {
+      if (
+        run.controllerBootId !== legacyControllerBootId ||
+        !run.leaseExpiresAt ||
+        run.leaseExpiresAt <= new Date()
+      )
+        throw new Error("run_log_writer_authority_lost");
+      return;
+    }
+    const [owner] = await db
+      .select({
+        leaseOwner: nativeRunFinalizations.leaseOwner,
+        bootId: nativeRunFinalizations.controllerBootId,
+        expiresAt: nativeRunFinalizations.leaseExpiresAt,
+      })
+      .from(nativeRunFinalizations)
+      .where(
+        and(
+          eq(nativeRunFinalizations.runId, admitted.id),
+          eq(nativeRunFinalizations.companyId, admitted.companyId),
+        ),
+      );
+
+    if (!owner && !run.logStore && !run.logRef && !admitted.logStore && !admitted.logRef) return;
+    const identity = await currentNativeControllerIdentity();
+    if (
+      !owner ||
+      owner.bootId !== identity.bootId ||
+      !owner.expiresAt ||
+      owner.expiresAt <= new Date() ||
+      (nativeLeaseOwner !== undefined && owner.leaseOwner !== nativeLeaseOwner)
+    )
+      throw new Error("run_log_writer_authority_lost");
+  };
+}

server/src/services/heartbeat/run-output.test.ts

@@ -7,6 +7,9 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
 import { createDurableRunLogStore } from "../run-log-store.js";
 import type { StorageProvider } from "../../storage/types.js";
 import { createHeartbeatRunOutput, isNewRunLog } from "./run-output.js";
+import { createAgentIdentityRedactor } from "../agent-identity-redaction.js";
+import { formatNativeRunLogEvent } from "../native-runtime/native-run-log-export.js";
+import { generateKeyPairSync } from "node:crypto";
 
 vi.mock("../live-events.js", () => ({ publishLiveEvent: vi.fn() }));
 
@@ -45,15 +48,24 @@ function createRunRowDb(row: Partial<Run>) {
   return {
     update: () => ({
       set: (values: Partial<Run>) => ({
-        where: async () => { Object.assign(row, values); },
+        where: async () => {
+          Object.assign(row, values);
+        },
       }),
     }),
   } as unknown as Db;
 }
 
 const newRunRow = {
-  id: "run-1", companyId: "company-1", agentId: "agent-1", status: "running",
-  logStore: null, logRef: null, logBytes: null, lastOutputSeq: 0, lastOutputAt: null,
+  id: "run-1",
+  companyId: "company-1",
+  agentId: "agent-1",
+  status: "running",
+  logStore: null,
+  logRef: null,
+  logBytes: null,
+  lastOutputSeq: 0,
+  lastOutputAt: null,
 } as Partial<Run>;
 
 async function beginAttempt(store: ReturnType<typeof createDurableRunLogStore>, row: Partial<Run>) {
@@ -81,11 +93,26 @@ afterEach(async () => {
 
 describe("heartbeat run output log recovery", () => {
   it("treats only a run row with no recorded log or output as new", () => {
-    expect(isNewRunLog({ logStore: null, logRef: null, logBytes: null, lastOutputSeq: 0 })).toBe(true);
-    expect(isNewRunLog({ logStore: "local_file", logRef: "c/a/r.ndjson", logBytes: null, lastOutputSeq: 0 })).toBe(false);
-    expect(isNewRunLog({ logStore: null, logRef: "c/a/r.ndjson", logBytes: null, lastOutputSeq: 0 })).toBe(false);
-    expect(isNewRunLog({ logStore: null, logRef: null, logBytes: null, lastOutputSeq: 3 })).toBe(false);
-    expect(isNewRunLog({ logStore: null, logRef: null, logBytes: 120, lastOutputSeq: 0 })).toBe(false);
+    expect(isNewRunLog({ logStore: null, logRef: null, logBytes: null, lastOutputSeq: 0 })).toBe(
+      true,
+    );
+    expect(
+      isNewRunLog({
+        logStore: "local_file",
+        logRef: "c/a/r.ndjson",
+        logBytes: null,
+        lastOutputSeq: 0,
+      }),
+    ).toBe(false);
+    expect(
+      isNewRunLog({ logStore: null, logRef: "c/a/r.ndjson", logBytes: null, lastOutputSeq: 0 }),
+    ).toBe(false);
+    expect(isNewRunLog({ logStore: null, logRef: null, logBytes: null, lastOutputSeq: 3 })).toBe(
+      false,
+    );
+    expect(isNewRunLog({ logStore: null, logRef: null, logBytes: 120, lastOutputSeq: 0 })).toBe(
+      false,
+    );
   });
 
   it("makes no remote log recovery request for a new run", async () => {
@@ -116,8 +143,121 @@ describe("heartbeat run output log recovery", () => {
     await second.onLog("stdout", "attempt two\n");
     await restartedStore.finalize(second.output.handle!);
 
-    const lines = objects.get(row.logRef!)!.toString().trim().split("\n").map((line) => JSON.parse(line));
+    const lines = objects
+      .get(row.logRef!)!
+      .toString()
+      .trim()
+      .split("\n")
+      .map((line) => JSON.parse(line));
     expect(lines.map((line) => line.chunk)).toEqual(["attempt one\n", "attempt two\n"]);
     expect(lines.map((line) => line.seq)).toEqual([1, 2]);
   });
 });
+
+describe("object-backed diagnostic recording", () => {
+  it("applies identity redaction to a native key-bearing event before S3 admission", async () => {
+    const privateKeyPem = generateKeyPairSync("ed25519")
+      .privateKey.export({ type: "pkcs8", format: "pem" })
+      .toString();
+    const row = { ...newRunRow, runtimeMode: "native" } as Run;
+    const db = createRunRowDb(row) as Db & { select: () => unknown };
+    db.select = (() => ({
+      from: () => ({ where: () => ({ orderBy: () => ({ limit: async () => [] }) }) }),
+    })) as typeof db.select;
+    const chunks: string[] = [];
+    const output = createHeartbeatRunOutput(db, {
+      run: row,
+      runId: row.id,
+      issueId: null,
+      secretKeys: new Set(),
+      identityRedactor: createAgentIdentityRedactor(privateKeyPem),
+      runLogStore: {
+        begin: async () => ({
+          store: "s3_segments",
+          logRef: "company/agent/run",
+          initialSourceCursor: 0,
+        }),
+        append: async () => 0,
+        enqueue: async (_handle, event) => {
+          chunks.push(event.chunk);
+          return { committed: Promise.resolve(100) };
+        },
+        flush: async () => undefined,
+      },
+      appendRunEvent: vi.fn(),
+      getCurrentUserRedactionOptions: async () => ({ enabled: false }),
+    });
+    const handlers = await output.begin(row);
+    await handlers.onPersistedNativeLog!(
+      1,
+      formatNativeRunLogEvent({
+        schema: "paperclip.prp.event.v1",
+        schemaVersion: 1,
+        runId: row.id,
+        sourceEventId: "source:1",
+        sourceInstanceId: "source",
+        sourceKind: "runner",
+        sourceSeq: 1,
+        normalizedSessionId: "session",
+        eventType: "harness.diagnostic",
+        priority: 1,
+        emittedAt: new Date().toISOString(),
+        payload: { text: privateKeyPem },
+      }),
+    );
+    expect(chunks).toHaveLength(1);
+    expect(chunks[0]).not.toContain(privateKeyPem.split("\n")[1]!);
+    expect(chunks[0]).toContain("REDACTED");
+  });
+  it("admits a burst without serial network waits, publishes after receipts, and coalesces progress", async () => {
+    const { publishLiveEvent } = await import("../live-events.js");
+    vi.mocked(publishLiveEvent).mockClear();
+    const row = { ...newRunRow };
+    const writes: Array<Partial<Run>> = [];
+    const db = {
+      update: () => ({
+        set: (values: Partial<Run>) => ({
+          where: async () => {
+            writes.push(values);
+            Object.assign(row, values);
+          },
+        }),
+      }),
+    } as unknown as Db;
+    const commit: Array<() => void> = [];
+    const recorder = createHeartbeatRunOutput(db, {
+      run: row as Run,
+      runId: row.id!,
+      issueId: null,
+      runLogStore: {
+        begin: async () => ({
+          store: "s3_segments",
+          logRef: "company/agent/run",
+          initialBytes: 0,
+          initialSequence: 10,
+        }),
+        append: () => new Promise<number>((resolve) => commit.push(() => resolve(100))),
+      },
+      identityRedactor: { chunk: (_stream, chunk) => chunk, redact: (value) => value },
+      secretKeys: new Set(),
+      appendRunEvent: vi.fn(),
+      getCurrentUserRedactionOptions: async () => ({ enabled: false }),
+    });
+    const { onLog } = await recorder.begin(row as Run);
+    for (let i = 0; i < 100; i++) await onLog("stdout", `line ${i}\n`);
+    expect(commit).toHaveLength(100);
+    expect(publishLiveEvent).not.toHaveBeenCalled();
+    for (const resolve of commit) resolve();
+    await vi.waitFor(() =>
+      expect(
+        vi
+          .mocked(publishLiveEvent)
+          .mock.calls.filter(([event]) => event.type === "heartbeat.run.log"),
+      ).toHaveLength(100),
+    );
+    await recorder.flushOutputProgress({ force: true });
+    expect(row.lastOutputSeq).toBe(110);
+    expect(row.lastOutputBytes).toBe(10000);
+    expect(writes.filter((write) => write.lastOutputSeq !== undefined)).toHaveLength(2);
+  });
+});

server/src/services/heartbeat/run-output.ts

@@ -5,9 +5,15 @@ import { redactCurrentUserText, type CurrentUserRedactionOptions } from "../../l
 import type { createAgentIdentityRedactor } from "../agent-identity-redaction.js";
 import { touchHeartbeatRunRuntimeStatus } from "../heartbeat-run-runtime-status.js";
 import { publishLiveEvent } from "../live-events.js";
-import type { RunLogHandle, RunLogStore } from "../run-log-store.js";
+import { isRunLogStoreType, type RunLogHandle, type RunLogStore } from "../run-log-store.js";
 import { appendExcerpt, compactRunLogChunk } from "./run-log.js";
-import { isHeartbeatRunRuntimeStatusActive, publishHeartbeatRunRuntimeProgress, type createHeartbeatLifecycle } from "./run-lifecycle.js";
+import { createNativeRunLogExporter } from "../native-runtime/native-run-log-export.js";
+import { createRunSecretRedactionRegistry } from "../run-secret-redaction.js";
+import {
+  isHeartbeatRunRuntimeStatusActive,
+  publishHeartbeatRunRuntimeProgress,
+  type createHeartbeatLifecycle,
+} from "./run-lifecycle.js";
 
 const MAX_LIVE_LOG_CHUNK_BYTES = 8 * 1024;
 export const ACTIVE_RUN_OUTPUT_PROGRESS_FLUSH_INTERVAL_MS = 60 * 1000;
@@ -19,11 +25,13 @@ export interface HeartbeatRunOutputInput {
   run: Run;
   runId: string;
   issueId: string | null;
-  runLogStore: Pick<RunLogStore, "begin" | "append">;
+  runLogStore: Pick<RunLogStore, "begin" | "append" | "enqueue" | "flush">;
   identityRedactor: Pick<ReturnType<typeof createAgentIdentityRedactor>, "chunk" | "redact">;
   secretKeys: Set<string>;
   appendRunEvent: ReturnType<typeof createHeartbeatLifecycle>["appendRunEvent"];
   getCurrentUserRedactionOptions: () => Promise<CurrentUserRedactionOptions>;
+  assertWriterAuthority?: () => Promise<void>;
+  runtimeLogSecretValues?: readonly string[];
 }
 
 /**
@@ -35,15 +43,26 @@ export interface HeartbeatRunOutputInput {
 export function isNewRunLog(
   run: Pick<Run, "logStore" | "logRef" | "logBytes" | "lastOutputSeq">,
 ): boolean {
-  return run.logStore == null
-    && run.logRef == null
-    && run.logBytes == null
-    && Number(run.lastOutputSeq ?? 0) === 0;
+  return (
+    run.logStore == null &&
+    run.logRef == null &&
+    run.logBytes == null &&
+    Number(run.lastOutputSeq ?? 0) === 0
+  );
 }
 
 /** One recorder per execution attempt; the log store itself stays service-scoped. */
 export function createHeartbeatRunOutput(db: Db, input: HeartbeatRunOutputInput) {
-  const { run, runId, issueId, runLogStore, identityRedactor, secretKeys, appendRunEvent, getCurrentUserRedactionOptions } = input;
+  const {
+    run,
+    runId,
+    issueId,
+    runLogStore,
+    identityRedactor,
+    secretKeys,
+    appendRunEvent,
+    getCurrentUserRedactionOptions,
+  } = input;
   let handle: RunLogHandle | null = null;
   let stdoutExcerpt = "";
   let stderrExcerpt = "";
@@ -59,7 +78,14 @@ export function createHeartbeatRunOutput(db: Db, input: HeartbeatRunOutputInput)
     } | null;
   } = { pending: null };
   let persistedLogBytes = Number(run.logBytes ?? 0);
+  let progressWrite: Promise<void> | null = null;
+  let recordingFailureReported = false;
   const flushOutputProgress = async (opts?: { force?: boolean }) => {
+    if (progressWrite) {
+      await progressWrite;
+      if (opts?.force) return flushOutputProgress(opts);
+      return;
+    }
     const pendingOutputProgress = outputProgressState.pending;
     if (!pendingOutputProgress) return;
     const shouldFlush =
@@ -68,7 +94,7 @@ export function createHeartbeatRunOutput(db: Db, input: HeartbeatRunOutputInput)
       pendingOutputProgress.at.getTime() - lastOutputFlushAt.getTime() >=
         ACTIVE_RUN_OUTPUT_PROGRESS_FLUSH_INTERVAL_MS;
     if (!shouldFlush) return;
-    await db
+    progressWrite = db
       .update(heartbeatRuns)
       .set({
         lastOutputAt: pendingOutputProgress.at,
@@ -77,111 +103,181 @@ export function createHeartbeatRunOutput(db: Db, input: HeartbeatRunOutputInput)
         lastOutputBytes: pendingOutputProgress.bytes,
         updatedAt: new Date(),
       })
-      .where(eq(heartbeatRuns.id, run.id));
-    lastOutputFlushAt = pendingOutputProgress.at;
-    outputProgressState.pending = null;
+      .where(eq(heartbeatRuns.id, run.id))
+      .then(() => {
+        lastOutputFlushAt = pendingOutputProgress.at;
+        if (outputProgressState.pending === pendingOutputProgress)
+          outputProgressState.pending = null;
+      })
+      .finally(() => {
+        progressWrite = null;
+      });
+    await progressWrite;
   };
   // Bind handlers to the run row persisted at execution start. Progress writes
   // above use only the immutable run ID from the initial row.
   async function begin(currentRun: Run) {
     const run = currentRun;
+    if (run.logStore != null && !isRunLogStoreType(run.logStore))
+      throw new Error("Run log backend is not supported");
     handle = await runLogStore.begin({
       companyId: run.companyId,
       agentId: run.agentId,
       runId,
       newRun: isNewRunLog(run),
+      assertWriterAuthority: input.assertWriterAuthority,
+      previousStore: run.logStore ?? undefined,
     });
 
+    outputSeq = handle.initialSequence ?? outputSeq;
+    persistedLogBytes = handle.initialBytes ?? persistedLogBytes;
     await db
       .update(heartbeatRuns)
       .set({
         logStore: handle.store,
         logRef: handle.logRef,
+        ...(handle.store === "s3_segments" ? { logBytes: null, logSha256: null } : {}),
         updatedAt: new Date(),
       })
       .where(eq(heartbeatRuns.id, runId));
 
-    const currentUserRedactionOptions =
-      await getCurrentUserRedactionOptions();
-    const appendIdentityRedactedLog = async (stream: "stdout" | "stderr", chunk: string) => {
+    const currentUserRedactionOptions = await getCurrentUserRedactionOptions();
+    if (
+      handle.store === "s3_segments" &&
+      run.runtimeMode === "native" &&
+      input.runtimeLogSecretValues?.length
+    ) {
+      const registry = createRunSecretRedactionRegistry(db);
+      for (const value of new Set(input.runtimeLogSecretValues))
+        await registry.register(run.companyId, runId, value);
+    }
+    const reportRecordingFailure = async () => {
+      if (recordingFailureReported) return;
+      recordingFailureReported = true;
+      await appendRunEvent(currentRun, {
+        eventType: "lifecycle",
+        stream: "system",
+        level: "warn",
+        message: "Run log persistence is incomplete",
+        payload: { reason: "object_log_persistence_failed" },
+      }).catch(() => undefined);
+    };
+    const appendIdentityRedactedLog = async (
+      stream: "stdout" | "stderr",
+      chunk: string,
+      waitForCommit = true,
+      sourceCursor?: number,
+    ) => {
       const sanitizedChunk = compactRunLogChunk(
         redactCurrentUserText(chunk, currentUserRedactionOptions),
       );
-      if (stream === "stdout")
-        stdoutExcerpt = appendExcerpt(stdoutExcerpt, sanitizedChunk);
-      if (stream === "stderr")
-        stderrExcerpt = appendExcerpt(stderrExcerpt, sanitizedChunk);
+      if (stream === "stdout") stdoutExcerpt = appendExcerpt(stdoutExcerpt, sanitizedChunk);
+      if (stream === "stderr") stderrExcerpt = appendExcerpt(stderrExcerpt, sanitizedChunk);
       const ts = new Date().toISOString();
 
       outputSeq += 1;
       const chunkSeq = outputSeq;
-      let appendedBytes = 0;
+      let committed: Promise<number> = Promise.resolve(0);
       if (handle) {
-        appendedBytes = await runLogStore.append(handle, {
+        const event = {
           stream,
           chunk: sanitizedChunk,
           ts,
           seq: chunkSeq,
-        });
+          sourceCursor,
+        };
+        committed = runLogStore.enqueue
+          ? (await runLogStore.enqueue(handle, event)).committed
+          : runLogStore.append(handle, event);
+      }
+      const recorded = committed.then(async (appendedBytes) => {
         persistedLogBytes += appendedBytes;
-      }
-      outputProgressState.pending = {
-        at: new Date(ts),
-        seq: chunkSeq,
-        stream,
-        bytes: persistedLogBytes,
-      };
-      await flushOutputProgress();
-
-      // Streamed CLI output is real run activity: keep the in-memory
-      // runtime status ("Working... / X ago") fresh between structured
-      // events so sandbox runs with mid-run log streaming never show a
-      // minutes-stale timestamp. Throttled to avoid churning the live
-      // event stream on every 250ms tail chunk.
-      const logActivityAt = new Date(ts);
-      if (
-        isHeartbeatRunRuntimeStatusActive(run.status) &&
-        logActivityAt.getTime() - lastLogRuntimeStatusTouchMs >=
-          ACTIVE_RUN_LOG_RUNTIME_STATUS_REFRESH_INTERVAL_MS
-      ) {
-        lastLogRuntimeStatusTouchMs = logActivityAt.getTime();
-        const touchedStatus = touchHeartbeatRunRuntimeStatus({
-          companyId: run.companyId,
-          issueId,
-          agentId: run.agentId,
-          runId: run.id,
-          at: logActivityAt,
-        });
-        if (touchedStatus)
-          publishHeartbeatRunRuntimeProgress(touchedStatus);
-      }
-
-      const payloadChunk =
-        sanitizedChunk.length > MAX_LIVE_LOG_CHUNK_BYTES
-          ? sanitizedChunk.slice(
-              sanitizedChunk.length - MAX_LIVE_LOG_CHUNK_BYTES,
-            )
-          : sanitizedChunk;
-
-      publishLiveEvent({
-        companyId: run.companyId,
-        type: "heartbeat.run.log",
-        payload: {
-          runId: run.id,
-          agentId: run.agentId,
-          issueId,
-          ts,
+        outputProgressState.pending = {
+          at: new Date(ts),
           seq: chunkSeq,
           stream,
-          chunk: payloadChunk,
-          logRef: handle?.logRef,
-          logStore: handle?.store,
-          truncated: payloadChunk.length !== sanitizedChunk.length,
-        },
+          bytes: persistedLogBytes,
+        };
+        await flushOutputProgress();
+
+        // Streamed CLI output is real run activity: keep the in-memory
+        // runtime status ("Working... / X ago") fresh between structured
+        // events so sandbox runs with mid-run log streaming never show a
+        // minutes-stale timestamp. Throttled to avoid churning the live
+        // event stream on every 250ms tail chunk.
+        const logActivityAt = new Date(ts);
+        if (
+          isHeartbeatRunRuntimeStatusActive(run.status) &&
+          logActivityAt.getTime() - lastLogRuntimeStatusTouchMs >=
+            ACTIVE_RUN_LOG_RUNTIME_STATUS_REFRESH_INTERVAL_MS
+        ) {
+          lastLogRuntimeStatusTouchMs = logActivityAt.getTime();
+          const touchedStatus = touchHeartbeatRunRuntimeStatus({
+            companyId: run.companyId,
+            issueId,
+            agentId: run.agentId,
+            runId: run.id,
+            at: logActivityAt,
+          });
+          if (touchedStatus) publishHeartbeatRunRuntimeProgress(touchedStatus);
+        }
+
+        const payloadChunk =
+          sanitizedChunk.length > MAX_LIVE_LOG_CHUNK_BYTES
+            ? sanitizedChunk.slice(sanitizedChunk.length - MAX_LIVE_LOG_CHUNK_BYTES)
+            : sanitizedChunk;
+
+        publishLiveEvent({
+          companyId: run.companyId,
+          type: "heartbeat.run.log",
+          payload: {
+            runId: run.id,
+            agentId: run.agentId,
+            issueId,
+            ts,
+            seq: chunkSeq,
+            stream,
+            chunk: payloadChunk,
+            logRef: handle?.logRef,
+            logStore: handle?.store,
+            truncated: payloadChunk.length !== sanitizedChunk.length,
+          },
+        });
       });
+      if (waitForCommit) await recorded;
+      else void recorded.catch(reportRecordingFailure);
     };
-    const onLog = (stream: "stdout" | "stderr", chunk: string) =>
-      appendIdentityRedactedLog(stream, identityRedactor.chunk(stream, chunk));
+    const onLog = (stream: "stdout" | "stderr", chunk: string) => {
+      const external = handle?.store === "s3_segments";
+      const recorded = appendIdentityRedactedLog(
+        stream,
+        identityRedactor.chunk(stream, chunk),
+        !external,
+      );
+      return external ? recorded.catch(reportRecordingFailure) : recorded;
+    };
+    const nativeExporter =
+      handle.store === "s3_segments" && run.runtimeMode === "native"
+        ? createNativeRunLogExporter({
+            db,
+            companyId: run.companyId,
+            runId,
+            afterCursor: handle.initialSourceCursor ?? 0,
+            enqueue: (cursor, chunk) =>
+              appendIdentityRedactedLog(
+                "stdout",
+                identityRedactor.chunk("stdout", chunk),
+                false,
+                cursor,
+              ),
+            flush: () => runLogStore.flush!(handle!),
+          })
+        : undefined;
+    if (nativeExporter) await nativeExporter.recover();
+    const onPersistedNativeLog = nativeExporter
+      ? (cursor: number, chunk: string) =>
+          nativeExporter.record(cursor, chunk).catch(reportRecordingFailure)
+      : undefined;
     const onAdapterMeta = async (meta: AdapterInvocationMeta) => {
       meta = identityRedactor.redact(meta);
       if (meta.env && secretKeys.size > 0) {
@@ -214,16 +310,29 @@ export function createHeartbeatRunOutput(db: Db, input: HeartbeatRunOutputInput)
       });
     };
 
-    return { currentUserRedactionOptions, appendIdentityRedactedLog, onLog, onAdapterMeta, onAdapterEvent };
+    return {
+      currentUserRedactionOptions,
+      appendIdentityRedactedLog,
+      onLog,
+      onPersistedNativeLog,
+      onAdapterMeta,
+      onAdapterEvent,
+    };
   }
 
   return {
     begin,
-    get handle() { return handle; },
+    get handle() {
+      return handle;
+    },
     outputProgressState,
     flushOutputProgress,
     // Finalization awaits I/O and can receive more output before persisting excerpts.
-    get stdoutExcerpt() { return stdoutExcerpt; },
-    get stderrExcerpt() { return stderrExcerpt; },
+    get stdoutExcerpt() {
+      return stdoutExcerpt;
+    },
+    get stderrExcerpt() {
+      return stderrExcerpt;
+    },
   };
 }

server/src/services/heartbeat/run-state.ts

@@ -355,6 +355,9 @@ const heartbeatRunLogAccessColumns = {
   ...heartbeatRunReadAccessColumns,
   logStore: heartbeatRuns.logStore,
   logRef: heartbeatRuns.logRef,
+  runtimeMode: heartbeatRuns.runtimeMode,
+  status: heartbeatRuns.status,
+  logBytes: heartbeatRuns.logBytes,
 } as const;
 
 const heartbeatRunIssueControlColumns = {

server/src/services/heartbeat/runtime-execution.ts

@@ -172,6 +172,7 @@ export interface HeartbeatRuntimeExecutionInput {
     >;
   };
   output: Pick<AdapterExecutionContext, "onLog"> & {
+    onPersistedNativeLog?: (cursor: number, chunk: string) => Promise<void>;
     onAdapterEvent: NonNullable<AdapterExecutionContext["onEvent"]>;
     onAdapterMeta: AdapterExecutionContext["onMeta"];
     identityRedactor: ReturnType<typeof createAgentIdentityRedactor>;
@@ -629,6 +630,7 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe
                 goalCheckpointSession.current = { params, displayId };
               },
               onLog,
+              onPersistedNativeLog: input.output.onPersistedNativeLog,
               onEvent: onAdapterEvent,
               instructionWorkingCopy: nativeInstructionWorkingCopy(),
 

server/src/services/issues.ts

@@ -189,7 +189,8 @@ import {
   resolveIssueGoalId,
   resolveNextIssueGoalId,
 } from "./issue-goal-fallback.js";
-import { getRunLogStore } from "./run-log-store.js";
+import { getRunLogStore, isRunLogStoreType } from "./run-log-store.js";
+import { recoverSettledNativeRunLog } from "./native-runtime/native-run-log-recovery.js";
 import { getDefaultCompanyGoal } from "./goals.js";
 import { assertAssignableAgent } from "./agent-assignability.js";
 import {
@@ -6712,13 +6713,14 @@ export async function readIssueCommentRunLogText(run: {
   logStore: string | null;
   logRef: string | null;
   logBytes: number | null;
-}) {
-  if (run.logStore !== "local_file" || !run.logRef) return "";
+}, db?: Db) {
+  if (!isRunLogStoreType(run.logStore) || !run.logRef) return "";
   // A timed-out finalization leaves size unknown even when earlier entries
   // exist. Read those logs within the same byte budget as a known-size log.
   if (run.logBytes !== null && (!Number.isFinite(run.logBytes) || run.logBytes <= 0)) return "";
 
   const logRef = run.logRef;
+  const storeType = run.logStore;
   const store = getRunLogStore();
   let offset = 0;
   let content = "";
@@ -6727,6 +6729,8 @@ export async function readIssueCommentRunLogText(run: {
   let readTimer: NodeJS.Timeout | undefined;
 
   const readChunks = async () => {
+    if (db && storeType === "s3_segments") await recoverSettledNativeRunLog(db, store, logRef, undefined, controller.signal)
+      .catch(() => { controller.signal.throwIfAborted(); });
     while (nextOffset !== undefined) {
       controller.signal.throwIfAborted();
       const remainingBytes =
@@ -6734,7 +6738,7 @@ export async function readIssueCommentRunLogText(run: {
         Buffer.byteLength(content, "utf8");
       if (remainingBytes <= 0) break;
       const chunk = await store.read(
-        { store: "local_file", logRef },
+        { store: storeType, logRef },
         {
           offset,
           limitBytes: Math.min(ISSUE_COMMENT_RUN_LOG_DERIVATION_CHUNK_BYTES, remainingBytes),
@@ -7259,7 +7263,7 @@ export function issueService(db: Db) {
           );
           await Promise.all(
             batch.map(async (run) => {
-              logByRunId.set(run.runId, await readIssueCommentRunLogText(run));
+              logByRunId.set(run.runId, await readIssueCommentRunLogText(run, db));
             }),
           );
         }

server/src/services/native-runtime/native-run-log-export.integration.test.ts

@@ -0,0 +1,474 @@
+import { randomUUID, createHash, generateKeyPairSync } from "node:crypto";
+import { eq, sql } from "drizzle-orm";
+import {
+  createDb,
+  heartbeatRunEvents,
+  heartbeatRuns,
+  agentIdentityKeys,
+  type Db,
+} from "@paperclipai/db";
+import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
+import {
+  S3Client,
+  CreateBucketCommand,
+  DeleteBucketCommand,
+  DeleteObjectsCommand,
+  ListObjectsV2Command,
+} from "@aws-sdk/client-s3";
+import { createS3StorageProvider } from "../../storage/s3-provider.js";
+import { createObjectRunLogStore } from "../object-run-log-store.js";
+import { createNativeRunLogExporter, formatNativeRunLogEvent } from "./native-run-log-export.js";
+import { recoverSettledNativeRunLog, readRecoverableRunLog } from "./native-run-log-recovery.js";
+import { PaperclipControlPlanePort } from "./paperclip-control-plane-port.js";
+import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
+import { createNativeRunLogTestTables } from "../../__tests__/helpers/native-run-log-database.js";
+import type { PrpEvent } from "../../vendor/paperclip-runner/index.js";
+import type { StorageProvider } from "../../storage/types.js";
+import { appendNativeCredentialUpdate } from "./native-credential-updates.js";
+import { appendRetainedNativeCleanupEvent } from "./native-session-executor.js";
+import { idleWorkSnapshot } from "../task-admission.js";
+import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js";
+import { localEncryptedProvider } from "../../secrets/local-encrypted-provider.js";
+import { createHeartbeatRunOutput } from "../heartbeat/run-output.js";
+import { createAgentIdentityRedactor } from "../agent-identity-redaction.js";
+import { ObjectJournalConflictError } from "../object-byte-journal.js";
+
+const url = resolveExternalTestDatabaseUrl("PAPERCLIP_RUN_EVIDENCE_MIGRATION_DATABASE_URL");
+const endpoint = process.env.PAPERCLIP_LOG_TEST_S3_ENDPOINT;
+describe.skipIf(!url || !endpoint)("native run log source recovery", () => {
+  const schema = `native_log_${randomUUID().replaceAll("-", "")}`,
+    bucket = `pc-native-log-${randomUUID()}`;
+  let db: Db,
+    client: S3Client,
+    bucketCreated = false;
+  beforeAll(async () => {
+    db = createDb(url!, { maxConnections: 1, idleTimeoutSeconds: 0 });
+    await db.execute(sql.raw(`create schema ${schema}`));
+    await db.execute(sql.raw(`set search_path=${schema}`));
+    await createNativeRunLogTestTables(db);
+    client = new S3Client({ endpoint, region: "us-east-1", forcePathStyle: true });
+    await client.send(new CreateBucketCommand({ Bucket: bucket }));
+    bucketCreated = true;
+  });
+  afterEach(() => vi.unstubAllEnvs());
+  afterAll(async () => {
+    if (client && bucketCreated) {
+      for (;;) {
+        const page = await client.send(new ListObjectsV2Command({ Bucket: bucket }));
+        if (!page.Contents?.length) break;
+        await client.send(
+          new DeleteObjectsCommand({
+            Bucket: bucket,
+            Delete: { Objects: page.Contents.map((item) => ({ Key: item.Key! })) },
+          }),
+        );
+      }
+      await client.send(new DeleteBucketCommand({ Bucket: bucket }));
+      client.destroy();
+    }
+    if (db) {
+      await db.execute(sql.raw(`drop schema ${schema} cascade`));
+      await db.$client.end();
+    }
+  });
+  async function fixture() {
+    const binding = {
+      companyId: randomUUID(),
+      runId: randomUUID(),
+      agentId: randomUUID(),
+      sessionId: randomUUID(),
+      issueId: randomUUID(),
+      sourceInstanceId: randomUUID(),
+      controlPlaneSourceInstanceId: randomUUID(),
+      completionContractId: randomUUID(),
+      completionContractSha256: "test",
+    };
+    await db.execute(
+      sql`insert into heartbeat_runs(id,company_id,agent_id,status) values(${binding.runId},${binding.companyId},${binding.agentId},'running')`,
+    );
+    const provider = createS3StorageProvider({
+      endpoint,
+      bucket,
+      region: "us-east-1",
+      forcePathStyle: true,
+    });
+    const store = (storage: StorageProvider = provider) =>
+      createObjectRunLogStore({
+        provider: storage,
+        keyPrefix: "logs",
+        flushMs: 30_000,
+        assertWriterAuthority: async (input) => {
+          await input.assertWriterAuthority?.();
+        },
+      });
+    const begin = { companyId: binding.companyId, agentId: binding.agentId, runId: binding.runId };
+    const port = new PaperclipControlPlanePort(db, binding);
+    const event = (seq: number): PrpEvent => ({
+      schema: "paperclip.prp.event.v1",
+      schemaVersion: 1,
+      sourceInstanceId: binding.sourceInstanceId,
+      sourceEventId: `${binding.sourceInstanceId}:${seq}`,
+      sourceSeq: seq,
+      sourceKind: "runner",
+      runId: binding.runId,
+      normalizedSessionId: binding.sessionId,
+      eventType: "item.delta",
+      priority: 2,
+      emittedAt: new Date().toISOString(),
+      payload: { text: `step ${seq} 再開` },
+    });
+    return { binding, store, begin, port, event, provider };
+  }
+  it("replays acknowledged native records into a new writer, repairs gaps, and ignores late duplicates", async () => {
+    const f = await fixture(),
+      a = f.store(),
+      h = await a.begin(f.begin);
+    await db.execute(
+      sql`update heartbeat_runs set log_store='s3_segments', log_ref=${h.logRef} where id=${f.binding.runId}`,
+    );
+    const events = [f.event(1), f.event(2), f.event(3)];
+    const cursors = await Promise.all(events.slice(0, 2).map((event) => f.port.appendEvent(event)));
+    const receipt = await a.enqueue!(h, {
+      stream: "stdout",
+      chunk: formatNativeRunLogEvent(events[0]!),
+      ts: "then",
+      seq: 1,
+      sourceCursor: cursors[0]!.cursor,
+    });
+    await a.flush!(h);
+    await receipt.committed;
+    const b = f.store(),
+      successor = await b.begin(f.begin);
+    let seq = successor.initialSequence!;
+    const exporter = createNativeRunLogExporter({
+      db,
+      companyId: f.binding.companyId,
+      runId: f.binding.runId,
+      afterCursor: successor.initialSourceCursor!,
+      enqueue: async (sourceCursor, chunk) => {
+        const queued = await b.enqueue!(successor, {
+          stream: "stdout",
+          chunk,
+          ts: "now",
+          seq: ++seq,
+          sourceCursor,
+        });
+        void queued.committed.catch(() => undefined);
+      },
+      flush: () => b.flush!(successor),
+    });
+    const third = await f.port.appendEvent(events[2]!);
+    await exporter.record(third.cursor, formatNativeRunLogEvent(events[2]!));
+    await exporter.record(cursors[0]!.cursor, formatNativeRunLogEvent(events[0]!));
+    await b.finalize(successor);
+    const entries = (await b.read(successor)).content
+      .trim()
+      .split("\n")
+      .map((line) => JSON.parse(JSON.parse(line).chunk).event);
+    expect(entries.map((event) => event.sourceSeq)).toEqual([1, 2, 3]);
+    expect(
+      await db
+        .select()
+        .from(heartbeatRunEvents)
+        .where(eq(heartbeatRunEvents.runId, f.binding.runId)),
+    ).toHaveLength(3);
+  });
+  it("repairs a settled run on demand after its controller lease expires, without extending source retention", async () => {
+    const f = await fixture(),
+      a = f.store(),
+      h = await a.begin(f.begin);
+    await f.port.appendEvent(f.event(1));
+    await db.execute(
+      sql`update heartbeat_runs set status='succeeded', log_store='s3_segments', log_ref=${h.logRef} where id=${f.binding.runId}`,
+    );
+    await db.execute(
+      sql`insert into native_run_finalizations values(${f.binding.runId},${f.binding.companyId},now()+interval '1 minute')`,
+    );
+    const b = f.store();
+    await recoverSettledNativeRunLog(db, b, h.logRef, async () => ({ enabled: false }));
+    expect((await b.read(h)).content).toBe("");
+    await db.execute(
+      sql`update native_run_finalizations set lease_expires_at=now()-interval '1 second' where run_id=${f.binding.runId}`,
+    );
+    await Promise.all(
+      [1, 2].map(() =>
+        recoverSettledNativeRunLog(db, b, h.logRef, async () => ({ enabled: false })),
+      ),
+    );
+    expect((await b.read(h)).content.trim().split("\n")).toHaveLength(1);
+    const [run] = await db
+      .select({ bytes: heartbeatRuns.logBytes })
+      .from(heartbeatRuns)
+      .where(eq(heartbeatRuns.id, f.binding.runId));
+    expect(run!.bytes).toBeGreaterThan(0);
+  });
+  it.each(["hash", "payload", "identity"])(
+    "rejects a corrupt canonical source %s without marking the partial archive complete",
+    async (kind) => {
+      const f = await fixture(),
+        a = f.store(),
+        h = await a.begin(f.begin);
+      await f.port.appendEvent(f.event(1));
+      if (kind === "hash")
+        await db.execute(
+          sql`update heartbeat_run_events set source_payload_sha256='wrong' where run_id=${f.binding.runId}`,
+        );
+      else if (kind === "payload")
+        await db.execute(
+          sql`update heartbeat_run_events set payload='{}'::jsonb where run_id=${f.binding.runId}`,
+        );
+      else
+        await db.execute(
+          sql`update heartbeat_run_events set source_instance_id='wrong' where run_id=${f.binding.runId}`,
+        );
+      await db.execute(
+        sql`update heartbeat_runs set status='succeeded', log_store='s3_segments', log_ref=${h.logRef} where id=${f.binding.runId}`,
+      );
+      await expect(
+        recoverSettledNativeRunLog(db, f.store(), h.logRef, async () => ({ enabled: false })),
+      ).rejects.toThrow("integrity");
+      const [run] = await db
+        .select({ bytes: heartbeatRuns.logBytes })
+        .from(heartbeatRuns)
+        .where(eq(heartbeatRuns.id, f.binding.runId));
+      expect(run!.bytes).toBeNull();
+    },
+  );
+  it("ignores credential and retained cleanup projections when replaying canonical log sources", async () => {
+    const f = await fixture(),
+      a = f.store(),
+      h = await a.begin(f.begin);
+    await f.port.appendEvent(f.event(1));
+    const credential = (sourceSeq: number): PrpEvent => ({
+      ...f.event(sourceSeq),
+      schema: "paperclip.prp.event.v3",
+      schemaVersion: 3,
+      priority: 0,
+      eventType: "auth.credentials.updated",
+      payload: {
+        schema: "paperclip.auth.credentials.updated.v1",
+        provider: "codex",
+        bindingId: "test-auth",
+        revision: 1,
+        detection: "credential_store",
+      },
+    });
+    await f.port.appendEvent(credential(2));
+    await db.execute(sql`update heartbeat_runs set status='succeeded', log_store='s3_segments', log_ref=${h.logRef},
+      native_session_id=${f.binding.sessionId}, runner_instance_id=${f.binding.sourceInstanceId} where id=${f.binding.runId}`);
+    await appendNativeCredentialUpdate(db, {
+      companyId: f.binding.companyId,
+      agentId: f.binding.agentId,
+      normalizedSessionId: f.binding.sessionId,
+      runnerInstanceId: f.binding.sourceInstanceId,
+      event: credential(20),
+    });
+    await appendRetainedNativeCleanupEvent(db, {
+      companyId: f.binding.companyId,
+      agentId: f.binding.agentId,
+      runId: f.binding.runId,
+      nativeSessionId: f.binding.sessionId,
+      runnerInstanceId: f.binding.sourceInstanceId,
+      requestId: "test-cleanup",
+      event: f.event(21),
+    });
+    await appendHeartbeatRunEvent(db, {
+      companyId: f.binding.companyId,
+      agentId: f.binding.agentId,
+      runId: f.binding.runId,
+      eventType: "workspace.diff.recorded",
+      payload: { complete: true },
+      nativeSource: {
+        sourceInstanceId: "provider-trace-reproject:test",
+        sourceEventId: "provider-trace-reproject:test:change",
+        sourceSeq: 1,
+        protocolSchemaVersion: 1,
+        canonicalPayload: { complete: true },
+      },
+    });
+    const b = f.store();
+    await recoverSettledNativeRunLog(db, b, h.logRef, async () => ({ enabled: false }));
+    const entries = (await b.read(h)).content.trim().split("\n");
+    expect(entries).toHaveLength(2);
+    expect(JSON.parse(JSON.parse(entries[0]!).chunk).event.sourceSeq).toBe(1);
+    expect(JSON.parse(JSON.parse(entries[1]!).chunk).event.eventType).toBe(
+      "auth.credentials.updated",
+    );
+    const [run] = await db
+      .select({ bytes: heartbeatRuns.logBytes })
+      .from(heartbeatRuns)
+      .where(eq(heartbeatRuns.id, f.binding.runId));
+    expect(run!.bytes).toBeGreaterThan(0);
+  });
+  it("aborts in-flight archive writes when its last reader cancels, then resumes without loss", async () => {
+    const f = await fixture(),
+      a = f.store(),
+      h = await a.begin(f.begin),
+      baseline = idleWorkSnapshot().active;
+    await f.port.appendEvent(f.event(1));
+    await db.execute(
+      sql`update heartbeat_runs set status='succeeded', log_store='s3_segments', log_ref=${h.logRef} where id=${f.binding.runId}`,
+    );
+    let entered = false,
+      aborted = false;
+    const blocked: StorageProvider = {
+      ...f.provider,
+      async putObjectConditional(input) {
+        if (input.objectKey.includes("/objects/")) {
+          entered = true;
+          await new Promise<void>((_resolve, reject) =>
+            input.signal!.addEventListener(
+              "abort",
+              () => {
+                aborted = true;
+                reject(input.signal!.reason);
+              },
+              { once: true },
+            ),
+          );
+        }
+        return f.provider.putObjectConditional!(input);
+      },
+    };
+    const controller = new AbortController();
+    const repairing = recoverSettledNativeRunLog(
+      db,
+      f.store(blocked),
+      h.logRef,
+      async () => ({ enabled: false }),
+      controller.signal,
+    );
+    const rejected = expect(repairing).rejects.toThrow("reader cancelled");
+    const lastReader = new AbortController();
+    const joined = recoverSettledNativeRunLog(
+      db,
+      f.store(blocked),
+      h.logRef,
+      async () => ({ enabled: false }),
+      lastReader.signal,
+    );
+    const joinedRejected = expect(joined).rejects.toThrow("last reader cancelled");
+    await vi.waitFor(() => expect(entered).toBe(true));
+    controller.abort(new Error("reader cancelled"));
+    await rejected;
+    expect(aborted).toBe(false);
+    lastReader.abort(new Error("last reader cancelled"));
+    await joinedRejected;
+    await vi.waitFor(() => {
+      expect(aborted).toBe(true);
+      expect(idleWorkSnapshot().active).toBe(baseline);
+    });
+    const b = f.store();
+    await vi.waitFor(async () => {
+      await recoverSettledNativeRunLog(db, b, h.logRef, async () => ({ enabled: false }));
+    });
+    expect((await b.read(h)).content.trim().split("\n")).toHaveLength(1);
+  });
+  it("redacts the existing agent identity and registered secrets before publishing recovered native output", async () => {
+    vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "e".repeat(64));
+    const f = await fixture();
+    const keys = generateKeyPairSync("ed25519"),
+      privateKeyPem = keys.privateKey.export({ type: "pkcs8", format: "pem" }).toString();
+    const publicKeyPem = keys.publicKey.export({ type: "spki", format: "pem" }).toString();
+    const keyId = `sha256:${createHash("sha256")
+      .update(keys.publicKey.export({ type: "spki", format: "der" }))
+      .digest("base64url")}`;
+    const encrypted = await localEncryptedProvider.createSecret({ value: privateKeyPem });
+    await db.insert(agentIdentityKeys).values({
+      agentId: f.binding.agentId,
+      companyId: f.binding.companyId,
+      algorithm: "Ed25519",
+      keyId,
+      publicKeyPem,
+      privateKeyMaterial: encrypted.material,
+    });
+    const secret = "registered-native-fixture-secret";
+    const redactor = createAgentIdentityRedactor(privateKeyPem);
+    redactor.values.push(secret);
+    const run = {
+      id: f.binding.runId,
+      companyId: f.binding.companyId,
+      agentId: f.binding.agentId,
+      status: "running",
+      runtimeMode: "native",
+      logStore: null,
+      logRef: null,
+      logBytes: null,
+      lastOutputSeq: 0,
+      lastOutputAt: null,
+    } as typeof heartbeatRuns.$inferSelect;
+    const output = createHeartbeatRunOutput(db, {
+      run,
+      runId: run.id,
+      issueId: null,
+      runLogStore: f.store(),
+      identityRedactor: redactor,
+      secretKeys: new Set(),
+      runtimeLogSecretValues: [secret],
+      appendRunEvent: async () => undefined,
+      getCurrentUserRedactionOptions: async () => ({ enabled: false }),
+    });
+    await output.begin(run);
+    const h = output.handle!;
+    await f.port.appendEvent({ ...f.event(1), payload: { text: `${privateKeyPem}\n${secret}` } });
+    await db.execute(
+      sql`update heartbeat_runs set status='succeeded', log_store='s3_segments',log_ref=${h.logRef} where id=${f.binding.runId}`,
+    );
+    const b = f.store();
+    await recoverSettledNativeRunLog(db, b, h.logRef, async () => ({ enabled: false }));
+    const content = (await b.read(h)).content;
+    expect(content).not.toContain(privateKeyPem.split("\n")[1]!);
+    expect(content).not.toContain(secret);
+    expect(content).toContain("REDACTED");
+    const [snapshot] = await db
+      .select({ context: heartbeatRuns.contextSnapshot })
+      .from(heartbeatRuns)
+      .where(eq(heartbeatRuns.id, f.binding.runId));
+    expect(JSON.stringify(snapshot!.context)).not.toContain(privateKeyPem.split("\n")[1]!);
+  });
+  it("serves the published snapshot when another writer wins a repair claim", async () => {
+    const f = await fixture(),
+      a = f.store(),
+      h = await a.begin(f.begin);
+    const event = f.event(1),
+      receipt = await f.port.appendEvent(event);
+    const queued = await a.enqueue!(h, {
+      stream: "stdout",
+      chunk: formatNativeRunLogEvent(event),
+      ts: "then",
+      seq: 1,
+      sourceCursor: receipt.cursor,
+    });
+    await a.flush!(h);
+    await queued.committed;
+    await f.port.appendEvent(f.event(2));
+    await db.execute(
+      sql`update heartbeat_runs set status='succeeded',log_store='s3_segments',log_ref=${h.logRef} where id=${f.binding.runId}`,
+    );
+    let enter!: () => void, release!: () => void;
+    const entered = new Promise<void>((resolve) => {
+        enter = resolve;
+      }),
+      blocked = new Promise<void>((resolve) => {
+        release = resolve;
+      });
+    const store = {
+      ...f.store(),
+      async begin() {
+        enter();
+        await blocked;
+        throw new ObjectJournalConflictError();
+      },
+    };
+    const repairing = recoverSettledNativeRunLog(db, store, h.logRef, async () => ({
+      enabled: false,
+    }));
+    const rejected = expect(repairing).rejects.toBeInstanceOf(ObjectJournalConflictError);
+    await entered;
+    const read = readRecoverableRunLog(db, store, h);
+    release();
+    await rejected;
+    expect((await read).content.trim().split("\n")).toHaveLength(1);
+    await a.finalize(h);
+  });
+});

server/src/services/native-runtime/native-run-log-export.ts

@@ -0,0 +1,210 @@
+import { and, asc, desc, eq, gt, isNotNull, lte, not, or, sql } from "drizzle-orm";
+import { StringDecoder } from "node:string_decoder";
+import { heartbeatRunEvents, type Db } from "@paperclipai/db";
+import { validatePrpEvent, type PrpEvent } from "../../vendor/paperclip-runner/index.js";
+import { nativeSha256 } from "./canonical.js";
+
+export function formatNativeRunLogEvent(event: PrpEvent) {
+  return `${JSON.stringify({ type: "paperclip.prp.event", event })}\n`;
+}
+export function nativeRunLogSourceWhere(companyId: string, runId: string) {
+  return and(
+    eq(heartbeatRunEvents.companyId, companyId),
+    eq(heartbeatRunEvents.runId, runId),
+    isNotNull(heartbeatRunEvents.sourceEventId),
+    not(
+      sql`coalesce((${or(
+        and(
+          eq(heartbeatRunEvents.eventType, "auth.credentials.updated"),
+          sql`${heartbeatRunEvents.sourceEventId} like 'credentials:%'`,
+          sql`${heartbeatRunEvents.sourceInstanceId} like '%:credentials'`,
+        ),
+        and(
+          eq(heartbeatRunEvents.eventType, "native.cleanup.event"),
+          sql`${heartbeatRunEvents.sourceEventId} like 'cleanup:%'`,
+          sql`${heartbeatRunEvents.sourceInstanceId} like '%:cleanup:%'`,
+        ),
+        and(
+          eq(heartbeatRunEvents.eventType, "workspace.diff.recorded"),
+          sql`${heartbeatRunEvents.sourceEventId} like 'provider-trace-reproject:%'`,
+          sql`${heartbeatRunEvents.sourceInstanceId} like 'provider-trace-reproject:%'`,
+        ),
+      )}), false)`,
+    ),
+  );
+}
+async function sourcePage(
+  db: Db,
+  companyId: string,
+  runId: string,
+  after: number,
+  through?: number,
+  signal?: AbortSignal,
+) {
+  signal?.throwIfAborted();
+  const rows = await db
+    .select({
+      seq: heartbeatRunEvents.seq,
+      payload: heartbeatRunEvents.payload,
+      sourceInstanceId: heartbeatRunEvents.sourceInstanceId,
+      sourceEventId: heartbeatRunEvents.sourceEventId,
+      sourceSeq: heartbeatRunEvents.sourceSeq,
+      sha256: heartbeatRunEvents.sourcePayloadSha256,
+    })
+    .from(heartbeatRunEvents)
+    .where(
+      and(
+        nativeRunLogSourceWhere(companyId, runId),
+        gt(heartbeatRunEvents.seq, after),
+        through === undefined ? undefined : lte(heartbeatRunEvents.seq, through),
+      ),
+    )
+    .orderBy(asc(heartbeatRunEvents.seq))
+    .limit(16);
+  signal?.throwIfAborted();
+  return rows.map((row) => {
+    const validated = validatePrpEvent(row.payload?.prpEvent);
+    if (
+      !validated.ok ||
+      validated.event.runId !== runId ||
+      validated.event.sourceInstanceId !== row.sourceInstanceId ||
+      validated.event.sourceEventId !== row.sourceEventId ||
+      validated.event.sourceSeq !== row.sourceSeq ||
+      nativeSha256(validated.event) !== row.sha256
+    )
+      throw new Error("native_run_log_source_integrity_error");
+    return { cursor: row.seq, event: validated.event };
+  });
+}
+
+export function createNativeRunLogCoverageVerifier(db: Db, companyId: string, runId: string) {
+  const decoder = new StringDecoder("utf8");
+  let text = "",
+    cursor = 0;
+  let page: Awaited<ReturnType<typeof sourcePage>> = [];
+  const unproven = () => new Error("native_log_source_coverage_unproven");
+  async function next() {
+    if (!page.length) page = await sourcePage(db, companyId, runId, cursor);
+    return page[0];
+  }
+  async function line(value: string) {
+    if (!value.trim()) return;
+    if (Buffer.byteLength(value) > 1024 * 1024) throw unproven();
+    let chunk: unknown;
+    try {
+      chunk = JSON.parse(value).chunk;
+    } catch {
+      throw unproven();
+    }
+    if (typeof chunk !== "string" || !chunk.startsWith('{"type":"paperclip.prp.event"')) return;
+    let event: PrpEvent;
+    try {
+      const parsed = validatePrpEvent(JSON.parse(chunk).event);
+      if (!parsed.ok) throw unproven();
+      event = parsed.event;
+    } catch {
+      throw unproven();
+    }
+    const expected = await next();
+    if (
+      !expected ||
+      event.runId !== runId ||
+      event.sourceInstanceId !== expected.event.sourceInstanceId ||
+      event.sourceEventId !== expected.event.sourceEventId ||
+      event.sourceSeq !== expected.event.sourceSeq ||
+      event.normalizedSessionId !== expected.event.normalizedSessionId ||
+      event.schemaVersion !== expected.event.schemaVersion ||
+      event.eventType !== expected.event.eventType
+    )
+      throw unproven();
+    cursor = expected.cursor;
+    page.shift();
+  }
+  return {
+    async write(bytes: Buffer) {
+      text += decoder.write(bytes);
+      let boundary: number;
+      while ((boundary = text.indexOf("\n")) !== -1) {
+        await line(text.slice(0, boundary));
+        text = text.slice(boundary + 1);
+      }
+      if (Buffer.byteLength(text) > 1024 * 1024) throw unproven();
+    },
+    async finish() {
+      text += decoder.end();
+      if (text) await line(text);
+      if (await next()) throw unproven();
+      return cursor;
+    },
+  };
+}
+
+export function createNativeRunLogExporter(input: {
+  db: Db;
+  companyId: string;
+  runId: string;
+  afterCursor: number;
+  enqueue: (cursor: number, chunk: string) => Promise<void>;
+  flush: () => Promise<void>;
+  signal?: AbortSignal;
+}) {
+  let cursor = input.afterCursor;
+  let queue = Promise.resolve();
+  let queuedBytes = 0,
+    queuedRecords = 0;
+  const sourceWhere = nativeRunLogSourceWhere(input.companyId, input.runId);
+  async function replay(through?: number) {
+    input.signal?.throwIfAborted();
+    if (through === undefined) {
+      const [last] = await input.db
+        .select({ seq: heartbeatRunEvents.seq })
+        .from(heartbeatRunEvents)
+        .where(sourceWhere)
+        .orderBy(desc(heartbeatRunEvents.seq))
+        .limit(1);
+      through = last?.seq ?? cursor;
+    }
+    while (cursor < through) {
+      const rows = await sourcePage(
+        input.db,
+        input.companyId,
+        input.runId,
+        cursor,
+        through,
+        input.signal,
+      );
+      if (!rows.length) break;
+      for (const row of rows) {
+        input.signal?.throwIfAborted();
+        await input.enqueue(row.cursor, formatNativeRunLogEvent(row.event));
+        cursor = row.cursor;
+      }
+      await input.flush();
+    }
+  }
+  return {
+    recover: () => replay(),
+    async record(sourceCursor: number, chunk: string) {
+      if (!Number.isSafeInteger(sourceCursor) || sourceCursor < 1)
+        throw new Error("invalid_native_run_log_cursor");
+      const bytes = Buffer.byteLength(chunk);
+      if (queuedRecords >= 32 || queuedBytes + bytes > 8 * 1024 * 1024)
+        throw new Error("native_run_log_admission_capacity_exceeded");
+      queuedRecords++;
+      queuedBytes += bytes;
+      const work = queue
+        .then(async () => {
+          if (sourceCursor <= cursor) return;
+          if (sourceCursor > cursor + 1) await replay(sourceCursor - 1);
+          await input.enqueue(sourceCursor, chunk);
+          cursor = sourceCursor;
+        })
+        .finally(() => {
+          queuedRecords--;
+          queuedBytes -= bytes;
+        });
+      queue = work.catch(() => undefined);
+      await work;
+    },
+  };
+}

server/src/services/native-runtime/native-run-log-recovery.ts

@@ -0,0 +1,223 @@
+import { and, eq, gt, inArray, isNull, sql } from "drizzle-orm";
+import {
+  heartbeatRuns,
+  heartbeatRunEvents,
+  nativeRunFinalizations,
+  type Db,
+} from "@paperclipai/db";
+import { redactCurrentUserText, type CurrentUserRedactionOptions } from "../../log-redaction.js";
+import type { RunLogHandle, RunLogReadOptions, RunLogStore } from "../run-log-store.js";
+import { instanceSettingsService } from "../instance-settings.js";
+import { compactRunLogChunk } from "../heartbeat/run-log.js";
+import { beginIdleTrackedWork } from "../task-admission.js";
+import { agentIdentityService } from "../agent-identity.js";
+import { createRunSecretRedactionRegistry } from "../run-secret-redaction.js";
+import { createAgentIdentityRedactor } from "../agent-identity-redaction.js";
+import { logger } from "../../middleware/logger.js";
+import { createNativeRunLogExporter, nativeRunLogSourceWhere } from "./native-run-log-export.js";
+
+interface Recovery {
+  work: Promise<void>;
+  controller: AbortController;
+  readers: number;
+}
+const recoveries = new WeakMap<Db, Map<string, Recovery>>();
+function awaitRecovery(recovery: Recovery, signal?: AbortSignal) {
+  signal?.throwIfAborted();
+  recovery.readers++;
+  return new Promise<void>((resolve, reject) => {
+    let finished = false;
+    const finish = () => {
+      if (finished) return false;
+      finished = true;
+      signal?.removeEventListener("abort", abort);
+      recovery.readers--;
+      return true;
+    };
+    const abort = () => {
+      if (!finish()) return;
+      if (!recovery.readers) recovery.controller.abort(signal!.reason);
+      reject(signal!.reason);
+    };
+    signal?.addEventListener("abort", abort, { once: true });
+    recovery.work.then(
+      () => {
+        if (finish()) resolve();
+      },
+      (error) => {
+        if (finish()) reject(error);
+      },
+    );
+  });
+}
+
+export async function readRecoverableRunLog(
+  db: Db,
+  store: RunLogStore,
+  handle: RunLogHandle,
+  options?: RunLogReadOptions,
+  known?: { runtimeMode?: string | null; status?: string; logBytes?: number | null },
+) {
+  if (
+    handle.store === "s3_segments" &&
+    known?.runtimeMode !== "legacy" &&
+    known?.status !== "running" &&
+    (known?.logBytes === undefined || known.logBytes === null)
+  )
+    await recoverSettledNativeRunLog(db, store, handle.logRef, undefined, options?.signal).catch(
+      () => {
+        options?.signal?.throwIfAborted();
+        logger.warn(
+          { runId: handle.logRef.split("/").at(-1) },
+          "Native run log repair is incomplete; serving the published prefix",
+        );
+      },
+    );
+  return store.read(handle, options);
+}
+
+export async function recoverSettledNativeRunLog(
+  db: Db,
+  store: RunLogStore,
+  logRef: string,
+  getRedactionOptions: () => Promise<CurrentUserRedactionOptions> = async () => ({
+    enabled: (await instanceSettingsService(db).getGeneral()).censorUsernameInLogs,
+  }),
+  requestSignal?: AbortSignal,
+) {
+  requestSignal?.throwIfAborted();
+  const binding = /^([a-f0-9-]+)\/([a-f0-9-]+)\/([a-f0-9-]+)$/.exec(logRef);
+  if (!binding) throw new Error("invalid_native_run_log_reference");
+  let pending = recoveries.get(db);
+  if (!pending) {
+    pending = new Map();
+    recoveries.set(db, pending);
+  }
+  const existing = pending.get(logRef);
+  if (existing) return awaitRecovery(existing, requestSignal);
+  const controller = new AbortController();
+  const signal = AbortSignal.any([controller.signal, AbortSignal.timeout(30_000)]);
+  const finishWork = beginIdleTrackedWork();
+  const runId = binding[3]!,
+    companyId = binding[1]!;
+  const eligible = () =>
+    and(
+      eq(heartbeatRuns.id, runId),
+      eq(heartbeatRuns.companyId, companyId),
+      eq(heartbeatRuns.agentId, binding[2]!),
+      eq(heartbeatRuns.logStore, "s3_segments"),
+      eq(heartbeatRuns.logRef, logRef),
+      eq(heartbeatRuns.runtimeMode, "native"),
+      inArray(heartbeatRuns.status, ["succeeded", "failed", "cancelled", "timed_out"]),
+      isNull(heartbeatRuns.logBytes),
+      sql`not exists (select 1 from ${nativeRunFinalizations} where ${and(
+        eq(nativeRunFinalizations.runId, runId),
+        eq(nativeRunFinalizations.companyId, companyId),
+        gt(nativeRunFinalizations.leaseExpiresAt, new Date()),
+      )})`,
+    );
+  const recover = (async () => {
+    signal.throwIfAborted();
+    const [run] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(eligible());
+    signal.throwIfAborted();
+    if (!run) return;
+    const [redaction, identityRedactor, secrets] = await Promise.all([
+      getRedactionOptions(),
+      agentIdentityService(db).createExistingLogRedactor(companyId, binding[2]!),
+      createRunSecretRedactionRegistry(db).redactionValuesForRun(companyId, runId),
+    ]);
+    for (const value of secrets)
+      identityRedactor.values.push(...createAgentIdentityRedactor(value).values);
+    identityRedactor.values.sort((left, right) => right.length - left.length);
+    const handle = await store.begin({
+      companyId,
+      agentId: binding[2]!,
+      runId,
+      previousStore: "s3_segments",
+      signal,
+      assertWriterAuthority: async () => {
+        signal.throwIfAborted();
+        const [current] = await db
+          .select({ id: heartbeatRuns.id })
+          .from(heartbeatRuns)
+          .where(eligible());
+        if (!current) throw new Error("native_run_log_recovery_authority_lost");
+      },
+    });
+    let sequence = handle.initialSequence ?? 0;
+    let lastSourceCursor = handle.initialSourceCursor ?? 0;
+    const receipts: Promise<number>[] = [];
+    let failure: unknown;
+    const exporter = createNativeRunLogExporter({
+      db,
+      companyId,
+      runId,
+      afterCursor: handle.initialSourceCursor ?? 0,
+      signal,
+      enqueue: async (sourceCursor, chunk) => {
+        const receipt = await store.enqueue!(handle, {
+          stream: "stdout",
+          ts: new Date().toISOString(),
+          seq: ++sequence,
+          sourceCursor,
+          chunk: compactRunLogChunk(
+            redactCurrentUserText(identityRedactor.chunk("stdout", chunk), redaction),
+          ),
+        });
+        receipts.push(
+          receipt.committed.catch((error) => {
+            failure ??= error;
+            return 0;
+          }),
+        );
+        lastSourceCursor = sourceCursor;
+      },
+      flush: async () => {
+        await store.flush!(handle);
+        await Promise.all(receipts.splice(0));
+        if (failure) throw failure;
+      },
+    });
+    try {
+      await exporter.recover();
+    } catch (error) {
+      failure ??= error;
+    } finally {
+      const summary = await store.finalize(handle, { computeDigest: false });
+      if (summary.bytes === null) failure ??= new Error("native_run_log_recovery_incomplete");
+      if (!failure && summary.bytes !== null) {
+        signal.throwIfAborted();
+        const updated = await db
+          .update(heartbeatRuns)
+          .set({
+            logBytes: summary.bytes,
+            logSha256: summary.sha256,
+            lastOutputSeq: sequence,
+            lastOutputBytes: summary.bytes,
+          })
+          .where(
+            and(
+              eligible(),
+              sql`not exists (select 1 from ${heartbeatRunEvents} where ${and(
+                nativeRunLogSourceWhere(companyId, runId),
+                gt(heartbeatRunEvents.seq, lastSourceCursor),
+              )})`,
+            ),
+          )
+          .returning({ id: heartbeatRuns.id });
+        if (!updated.length) throw new Error("native_run_log_recovery_source_changed");
+      }
+    }
+    if (failure) throw failure;
+  })();
+  const recovery: Recovery = {
+    controller,
+    readers: 0,
+    work: recover.finally(() => {
+      pending.delete(logRef);
+      finishWork();
+    }),
+  };
+  pending.set(logRef, recovery);
+  await awaitRecovery(recovery, requestSignal);
+}

server/src/services/native-runtime/native-session-executor.test.ts

@@ -249,8 +249,8 @@ vi.mock("./paperclip-control-plane-port.js", async importOriginal => {
   return { ...original, PaperclipControlPlanePort: class extends original.PaperclipControlPlanePort {
     constructor(...args: ConstructorParameters<typeof original.PaperclipControlPlanePort>) {
       super(...args);
-      accountingEvents.committed = args[2]?.onCommittedEvent;
-      accountingEvents.duplicate = args[2]?.onDuplicateEvent;
+      accountingEvents.committed = args[2]?.onCommittedEvent ? event => args[2]!.onCommittedEvent!(event, 1) : undefined;
+      accountingEvents.duplicate = args[2]?.onDuplicateEvent ? event => args[2]!.onDuplicateEvent!(event, 1) : undefined;
     }
   } };
 });

server/src/services/native-runtime/native-session-executor.ts

@@ -1,4 +1,5 @@
 import { measureRunnerOperation, recordRunnerTimingReceipt } from "./runner-operation-timing.js";
+import { formatNativeRunLogEvent } from "./native-run-log-export.js";
 import { notifyExternalAgentSettlement } from "../../modules/external-agents/index.js";
 import { notifyNativeResultCommit } from "../native-result-commit-notifications.js";
 import { releaseRetainedNativeSessionCleanup } from "../retained-native-session-cleanup-notifications.js";
@@ -7820,6 +7821,7 @@ export async function executePaperclipNativeSession(input: {
   /** Internal, run-owned file inspection lifetime; never supplied by tool arguments. */
   chatAttachmentReadScope?: NativeChatAttachmentReadScope;
   onLog?: (stream: "stdout" | "stderr", chunk: string) => Promise<void>;
+  onPersistedNativeLog?: (cursor: number, chunk: string) => Promise<void>;
   onEvent?: (event: AdapterRuntimeEvent) => Promise<void>;
   /** Only this run's registered private instruction entry.
    * Probe at terminal; persist only after owned shutdown. */
@@ -8653,7 +8655,7 @@ async function executePaperclipNativeSessionWithinScope(
             attributes:{eventKind:event.eventType}});
         }
       },
-      onCommittedEvent: async (event) => {
+      onCommittedEvent: async (event, cursor) => {
         if (event.eventType === "run.result.proposed") {
           const at = Date.now();
           await trace.record({name:"completion.tool_accepted",parentName:"native.session.execute",startedAtMs:at,endedAtMs:at});
@@ -8806,7 +8808,8 @@ async function executePaperclipNativeSessionWithinScope(
           });
           trace.activate(taskSettleScope);
         }
-        if (input.onLog)
+        if (input.onPersistedNativeLog) await input.onPersistedNativeLog(cursor, formatNativeRunLogEvent(event));
+        else if (input.onLog)
           await input.onLog(
             "stdout",
             `${JSON.stringify({ type: "paperclip.prp.event", event })}\n`,
@@ -8885,12 +8888,10 @@ async function executePaperclipNativeSessionWithinScope(
           },
         );
       },
-      onDuplicateEvent: async (event) => {
+      onDuplicateEvent: async (event, cursor) => {
+        await input.onPersistedNativeLog?.(cursor, formatNativeRunLogEvent(event));
         if (!firstTokenRecorded) observeDeliverableText(event);
         await observeAccountingEvent(event);
-        // A crash can happen after the event commit but before its callback
-        // finishes. Recover only idempotent durable projections here; activity,
-        // publication, logging, trace, and metric effects remain committed-only.
         await liveQuestions.observe(event);
         await projectSessionGoalEvent(event);
         providerUsageLimitObserved ||= nativeProviderUsageLimitFromEvent(event);

server/src/services/native-runtime/paperclip-control-plane-port.ts

@@ -88,8 +88,8 @@ export class PaperclipControlPlanePort implements ControlPlanePort {
   readonly #assertRunAdmissionActive?: () => void;
   readonly #onResultCommitted?: () => Promise<void>;
   readonly #onEventPersistence?: (event: PrpEvent, startedAtMs: number, endedAtMs: number) => Promise<void>;
-  readonly #onCommittedEvent?: (event: PrpEvent) => Promise<void>;
-  readonly #onDuplicateEvent?: (event: PrpEvent) => Promise<void>;
+  readonly #onCommittedEvent?: (event: PrpEvent, cursor: number) => Promise<void>;
+  readonly #onDuplicateEvent?: (event: PrpEvent, cursor: number) => Promise<void>;
 
   constructor(
     db: Db,
@@ -103,8 +103,8 @@ export class PaperclipControlPlanePort implements ControlPlanePort {
       assertRunAdmissionActive?: () => void;
       onResultCommitted?: () => Promise<void>;
       onEventPersistence?: (event: PrpEvent, startedAtMs: number, endedAtMs: number) => Promise<void>;
-      onCommittedEvent?: (event: PrpEvent) => Promise<void>;
-      onDuplicateEvent?: (event: PrpEvent) => Promise<void>;
+      onCommittedEvent?: (event: PrpEvent, cursor: number) => Promise<void>;
+      onDuplicateEvent?: (event: PrpEvent, cursor: number) => Promise<void>;
     } = {},
   ) {
     this.#db = db;
@@ -313,13 +313,13 @@ export class PaperclipControlPlanePort implements ControlPlanePort {
         });
       } catch {
       }
-      await this.#onCommittedEvent?.(event);
+      await this.#onCommittedEvent?.(event, persisted.row.seq);
     } else {
       // A recovered runner may replay the event whose durable side effects
       // parked the prior attempt. Do not repeat those effects, but let the
       // embedding runtime refresh observational state before appendEvent
       // returns to its synchronous governed-wait boundary.
-      await this.#onDuplicateEvent?.(event);
+      await this.#onDuplicateEvent?.(event, persisted.row.seq);
     }
     return {
       cursor: persisted.row.seq,

server/src/services/object-byte-journal.test.ts

@@ -0,0 +1,326 @@
+import { createHash } from "node:crypto";
+import { Readable } from "node:stream";
+import { afterEach, describe, expect, it, vi } from "vitest";
+import type { StorageProvider } from "../storage/types.js";
+import { createObjectByteJournal, ObjectJournalConflictError } from "./object-byte-journal.js";
+
+async function readBody(body: AsyncIterable<Uint8Array>) {
+  const chunks: Buffer[] = [];
+  for await (const bytes of body) chunks.push(Buffer.from(bytes));
+  return Buffer.concat(chunks);
+}
+
+function storage() {
+  const objects = new Map<string, Buffer>();
+  const calls = { get: 0, put: 0, maxPut: 0 };
+  let loseHeadResponse = false;
+  const etag = (bytes: Buffer) => createHash("sha256").update(bytes).digest("hex");
+  const missing = () => Object.assign(new Error("missing"), { name: "NoSuchKey" });
+  const conflict = () => Object.assign(new Error("conflict"), { name: "PreconditionFailed" });
+  const provider: StorageProvider = {
+    id: "s3",
+    async getObject({ objectKey, signal }) {
+      signal?.throwIfAborted();
+      calls.get++;
+      const bytes = objects.get(objectKey);
+      if (!bytes) throw missing();
+      return { stream: Readable.from([bytes]), etag: etag(bytes), contentLength: bytes.length };
+    },
+    async putObject() {
+      throw new Error("unconditional PUT forbidden");
+    },
+    async putObjectConditional(input) {
+      const old = objects.get(input.objectKey);
+      if (input.ifNoneMatch ? old : !old || etag(old) !== input.ifMatch) throw conflict();
+      const bytes = Buffer.isBuffer(input.body) ? input.body : await readBody(input.body);
+      objects.set(input.objectKey, Buffer.from(bytes));
+      calls.put++;
+      calls.maxPut = Math.max(calls.maxPut, bytes.length);
+      if (loseHeadResponse && input.objectKey.endsWith("/head.json")) {
+        loseHeadResponse = false;
+        throw new Error("response lost after PUT");
+      }
+      return { etag: etag(bytes) };
+    },
+    async headObject({ objectKey }) {
+      return { exists: objects.has(objectKey) };
+    },
+    async deleteObject({ objectKey }) {
+      objects.delete(objectKey);
+    },
+    async listObjects({ prefix }) {
+      return { keys: [...objects.keys()].filter((k) => k.startsWith(prefix)) };
+    },
+  };
+  return {
+    provider,
+    objects,
+    calls,
+    loseResponse: () => {
+      loseHeadResponse = true;
+    },
+  };
+}
+const authority = async () => undefined;
+afterEach(() => vi.useRealTimers());
+
+describe("object byte journal", () => {
+  it("keeps a shared capability probe alive for another writer when one claim cancels", async () => {
+    const s = storage();
+    const put = s.provider.putObjectConditional!;
+    let release!: () => void, started!: () => void, admitted!: () => void;
+    let probeSignal: AbortSignal | undefined;
+    const probing = new Promise<void>((resolve) => {
+      started = resolve;
+    });
+    const joined = new Promise<void>((resolve) => {
+      admitted = resolve;
+    });
+    s.provider.putObjectConditional = async (input) => {
+      if (input.objectKey.includes("conditional-probe-") && !probeSignal) {
+        probeSignal = input.signal;
+        await new Promise<void>((resolve, reject) => {
+          release = resolve;
+          input.signal!.addEventListener("abort", () => reject(input.signal!.reason), {
+            once: true,
+          });
+          started();
+        });
+      }
+      return put(input);
+    };
+    const controller = new AbortController();
+    const a = createObjectByteJournal({ provider: s.provider, prefix: "company/a/log" });
+    const first = a.claim({
+      expectedOwner: null,
+      assertAuthority: authority,
+      signal: controller.signal,
+    });
+    void first.catch(() => undefined);
+    await probing;
+    const b = createObjectByteJournal({ provider: s.provider, prefix: "company/b/log" });
+    const second = b.claim({
+      expectedOwner: null,
+      assertAuthority: async () => {
+        admitted();
+      },
+    });
+    await joined;
+    await Promise.resolve();
+    controller.abort();
+    await expect(first).rejects.toMatchObject({ name: "AbortError" });
+    expect(probeSignal!.aborted).toBe(false);
+    release();
+    const writer = await second;
+    expect((await b.head())!.head.owner).toBe(writer.head.owner);
+  });
+  it("recovers committed bytes and stable ranges with no local filesystem", async () => {
+    const s = storage(),
+      a = createObjectByteJournal({ provider: s.provider, prefix: "company/run/log" });
+    let writer = await a.claim({ expectedOwner: null, assertAuthority: authority });
+    const expected = Buffer.from("first\n再開\n".repeat(20));
+    writer = await a.append(writer, expected.subarray(0, 17));
+    writer = await a.append(writer, expected.subarray(17));
+    const b = createObjectByteJournal({ provider: s.provider, prefix: "company/run/log" });
+    const snapshot = (await b.head())!;
+    expect(await b.readRange(snapshot.head, 0, expected.length)).toEqual(expected);
+    expect(await b.readRange(snapshot.head, 7, 23)).toEqual(expected.subarray(7, 30));
+    expect(await b.readRange(snapshot.head, expected.length, 100)).toEqual(Buffer.alloc(0));
+  });
+  it("fences stale appends/finalization and preserves a pinned reader snapshot", async () => {
+    const s = storage(),
+      a = createObjectByteJournal({ provider: s.provider, prefix: "c/r/log" });
+    let old = await a.claim({ expectedOwner: null, assertAuthority: authority });
+    old = await a.append(old, Buffer.from("before"));
+    const snapshot = old.head;
+    let next = await a.claim({ expectedOwner: old.head.owner, assertAuthority: authority });
+    next = await a.append(next, Buffer.from("after"));
+    await expect(a.append(old, Buffer.from("lost"))).rejects.toBeInstanceOf(
+      ObjectJournalConflictError,
+    );
+    await expect(a.close(old)).rejects.toBeInstanceOf(ObjectJournalConflictError);
+    expect(await a.readRange(snapshot, 0, 100)).toEqual(Buffer.from("before"));
+    expect(await a.readRange(next.head, 0, 100)).toEqual(Buffer.from("beforeafter"));
+  });
+  it("reconciles an uncertain manifest commit without duplicating output", async () => {
+    const s = storage(),
+      a = createObjectByteJournal({ provider: s.provider, prefix: "c/r/log" });
+    let writer = await a.claim({ expectedOwner: null, assertAuthority: authority });
+    s.loseResponse();
+    writer = await a.append(writer, Buffer.from("once"));
+    expect(writer.head.end).toBe(4);
+    expect(await a.readRange((await a.head())!.head, 0, 100)).toEqual(Buffer.from("once"));
+  });
+  it("keeps index objects bounded over multiple tree levels", async () => {
+    const s = storage(),
+      a = createObjectByteJournal({ provider: s.provider, prefix: "c/r/log" });
+    let writer = await a.claim({ expectedOwner: null, assertAuthority: authority });
+    for (let i = 0; i < 180; i++)
+      writer = await a.append(writer, Buffer.from(String(i).padStart(4, "0")));
+    expect(await a.readRange(writer.head, 252, 16)).toEqual(Buffer.from("0063006400650066"));
+    expect(await a.readRange(writer.head, 700, 20)).toEqual(Buffer.from("01750176017701780179"));
+    expect(s.calls.maxPut).toBeLessThan(20_000);
+    expect(s.calls.put).toBeLessThan(180 * 5);
+    const reachable = new Set<string>();
+    function visit(ref: { key: string }, level: number) {
+      reachable.add(ref.key);
+      if (level === 0) return;
+      const node = JSON.parse(s.objects.get(ref.key)!.toString());
+      for (const child of node.children) visit(child, level - 1);
+    }
+    for (const root of writer.head.roots!) visit(root, root.level);
+    expect([...reachable].sort()).toEqual(
+      [...s.objects.keys()].filter((key) => key.includes("/objects/")).sort(),
+    );
+  });
+  it("retries a transient head conflict without fencing its current writer", async () => {
+    const s = storage(),
+      journal = createObjectByteJournal({ provider: s.provider, prefix: "c/r/log" });
+    let writer = await journal.claim({ expectedOwner: null, assertAuthority: authority });
+    const original = s.provider.putObjectConditional!;
+    let conflicts = 0;
+    s.provider.putObjectConditional = async (input) => {
+      if (input.objectKey.endsWith("/head.json") && conflicts++ === 0)
+        throw Object.assign(new Error("in-flight conflict"), {
+          name: "ConditionalRequestConflict",
+          $metadata: { httpStatusCode: 409 },
+        });
+      return original(input);
+    };
+    writer = await journal.append(writer, Buffer.from("once"));
+    expect(await journal.readRange(writer.head, 0, 100)).toEqual(Buffer.from("once"));
+    expect(conflicts).toBe(2);
+  });
+  it("does not accept a provider's 409 response as proof of conditional enforcement", async () => {
+    const s = storage(),
+      original = s.provider.putObjectConditional!;
+    s.provider.putObjectConditional = async (input) => {
+      if (s.objects.has(input.objectKey))
+        throw Object.assign(new Error("conflict"), { $metadata: { httpStatusCode: 409 } });
+      return original(input);
+    };
+    const journal = createObjectByteJournal({ provider: s.provider, prefix: "c/r/log" });
+    await expect(
+      journal.claim({ expectedOwner: null, assertAuthority: authority }),
+    ).rejects.toThrow("conflict");
+    expect(s.objects.size).toBe(0);
+  });
+  it("retries transient segment failures and acknowledges exactly one batch", async () => {
+    const s = storage(),
+      journal = createObjectByteJournal({ provider: s.provider, prefix: "c/r/log" });
+    let writer = await journal.claim({ expectedOwner: null, assertAuthority: authority });
+    const original = s.provider.putObjectConditional!;
+    let failures = 2;
+    s.provider.putObjectConditional = async (input) => {
+      if (input.objectKey.includes("/objects/") && failures-- > 0)
+        throw Object.assign(new Error("temporary outage"), { $metadata: { httpStatusCode: 503 } });
+      return original(input);
+    };
+    writer = await journal.append(writer, Buffer.from("one batch"));
+    expect(await journal.readRange(writer.head, 0, 100)).toEqual(Buffer.from("one batch"));
+  });
+  it("rejects corrupt published data and cancelled reads", async () => {
+    const s = storage(),
+      a = createObjectByteJournal({ provider: s.provider, prefix: "c/r/log" });
+    let writer = await a.claim({ expectedOwner: null, assertAuthority: authority });
+    writer = await a.append(writer, Buffer.from("data"));
+    const data = [...s.objects].find(([, bytes]) => bytes.toString() === "data")!;
+    s.objects.set(data[0], Buffer.from("oops"));
+    await expect(a.readRange(writer.head, 0, 4)).rejects.toThrow("integrity");
+    await expect(a.readRange(writer.head, 0, 4, AbortSignal.abort())).rejects.toThrow();
+  });
+  it("retains a deletion tombstone and denies a subsequent claim", async () => {
+    const s = storage(),
+      a = createObjectByteJournal({ provider: s.provider, prefix: "c/r/log" });
+    let writer = await a.claim({ expectedOwner: null, assertAuthority: authority });
+    writer = await a.append(writer, Buffer.from("private"));
+    await a.close(writer);
+    await a.remove();
+    expect([...s.objects.keys()]).toEqual(["c/r/log/head.json"]);
+    expect((await a.head())!.head.deleted).toBe(true);
+    await expect(
+      a.claim({ expectedOwner: writer.head.owner, assertAuthority: authority }),
+    ).rejects.toThrow();
+  });
+  it("does not claim ownership when execution authority rejects", async () => {
+    const s = storage(),
+      a = createObjectByteJournal({ provider: s.provider, prefix: "c/r/log" });
+    await expect(
+      a.claim({
+        expectedOwner: null,
+        assertAuthority: async () => {
+          throw new Error("revoked");
+        },
+      }),
+    ).rejects.toThrow("revoked");
+    expect(s.calls.put).toBe(0);
+  });
+  it("fences a pending first claim when deletion wins before journal creation", async () => {
+    const s = storage(),
+      journal = createObjectByteJournal({ provider: s.provider, prefix: "c/r/log" });
+    let enter!: () => void, release!: () => void;
+    const entered = new Promise<void>((resolve) => {
+      enter = resolve;
+    });
+    const blocked = new Promise<void>((resolve) => {
+      release = resolve;
+    });
+    const pending = journal.claim({
+      expectedOwner: null,
+      assertAuthority: async () => {
+        enter();
+        await blocked;
+      },
+    });
+    await entered;
+    await journal.remove();
+    release();
+    await expect(pending).rejects.toBeInstanceOf(ObjectJournalConflictError);
+    expect((await journal.head())!.head.deleted).toBe(true);
+    expect([...s.objects.keys()]).toEqual(["c/r/log/head.json"]);
+  });
+  it("rejects a compatible provider which silently ignores conditional writes", async () => {
+    const s = storage();
+    s.provider.putObjectConditional = async (input) => {
+      const bytes = Buffer.from(input.body as Buffer);
+      s.objects.set(input.objectKey, bytes);
+      return { etag: "ignored-condition" };
+    };
+    const journal = createObjectByteJournal({ provider: s.provider, prefix: "c/r/log" });
+    await expect(
+      journal.claim({ expectedOwner: null, assertAuthority: authority }),
+    ).rejects.toThrow("does not enforce conditional");
+    expect(s.objects.size).toBe(0);
+  });
+  it("fences an active writer before cleanup and preserves the retry obligation", async () => {
+    vi.useFakeTimers({ toFake: ["Date"] });
+    const s = storage(),
+      journal = createObjectByteJournal({ provider: s.provider, prefix: "c/r/log" });
+    let writer = await journal.claim({ expectedOwner: null, assertAuthority: authority });
+    writer = await journal.append(writer, Buffer.from("private"));
+    await expect(journal.remove()).rejects.toThrow("cleanup_pending");
+    expect((await journal.head())!.head.deleted).toBe(true);
+    await expect(journal.append(writer, Buffer.from("late"))).rejects.toBeInstanceOf(
+      ObjectJournalConflictError,
+    );
+    vi.setSystemTime(Date.now() + 30_001);
+    await journal.remove();
+    expect([...s.objects.keys()]).toEqual(["c/r/log/head.json"]);
+  });
+  it("does not publish a partial batch when a later segment fails", async () => {
+    const s = storage(),
+      journal = createObjectByteJournal({ provider: s.provider, prefix: "c/r/log" });
+    const writer = await journal.claim({ expectedOwner: null, assertAuthority: authority });
+    const original = s.provider.putObjectConditional!;
+    const failed = Buffer.alloc(1024 * 1024, 2);
+    const key = createHash("sha256").update(failed).digest("hex");
+    s.provider.putObjectConditional = (input) => {
+      if (input.objectKey.endsWith(key)) throw Error("second segment failed");
+      return original(input);
+    };
+    await expect(
+      journal.append(writer, Buffer.concat([Buffer.alloc(1024 * 1024, 1), failed])),
+    ).rejects.toThrow("second segment failed");
+    expect((await journal.head())!.head.end).toBe(0);
+  });
+});

server/src/services/object-byte-journal.ts

@@ -0,0 +1,574 @@
+import { createHash, randomUUID } from "node:crypto";
+import { z } from "zod";
+import { setTimeout as delay } from "node:timers/promises";
+import type { StorageProvider } from "../storage/types.js";
+
+const MAX_INDEX_LEVEL = 52;
+const MAX_OBJECT_BYTES = 4 * 1024 * 1024;
+const offset = z.number().int().nonnegative().max(Number.MAX_SAFE_INTEGER);
+const referenceSchema = z
+  .object({
+    key: z.string(),
+    start: offset,
+    end: offset,
+    sha256: z.string().regex(/^[a-f0-9]{64}$/),
+  })
+  .strict();
+type Reference = z.infer<typeof referenceSchema>;
+const nodeSchema = z
+  .object({
+    level: z.number().int().min(0).max(MAX_INDEX_LEVEL),
+    children: z.array(referenceSchema).min(1).max(64),
+  })
+  .strict();
+type Node = z.infer<typeof nodeSchema>;
+const rootSchema = referenceSchema.extend({ level: z.number().int().min(0).max(MAX_INDEX_LEVEL) });
+type Root = z.infer<typeof rootSchema>;
+const headSchema = z
+  .object({
+    v: z.literal(1),
+    owner: z.string().uuid(),
+    generation: offset,
+    end: offset,
+    sequence: offset.default(0),
+    sourceCursor: offset.default(0),
+    root: referenceSchema.nullable(),
+    roots: z
+      .array(rootSchema)
+      .max(MAX_INDEX_LEVEL + 1)
+      .optional(),
+    closed: z.boolean(),
+    deleted: z.boolean(),
+    cleanupAfter: offset.default(0),
+  })
+  .strict();
+export type ObjectJournalHead = z.infer<typeof headSchema>;
+export interface ObjectJournalWriter {
+  head: ObjectJournalHead;
+  etag: string;
+}
+export class ObjectJournalConflictError extends Error {
+  readonly code = "object_journal_writer_conflict";
+  constructor() {
+    super("The output journal is owned by another writer");
+  }
+}
+export function isObjectPreconditionFailure(error: unknown) {
+  const e = error as { name?: string; $metadata?: { httpStatusCode?: number } };
+  return e?.name === "PreconditionFailed" || e?.$metadata?.httpStatusCode === 412;
+}
+function isRetryable(error: unknown) {
+  const e = error as { name?: string; code?: string; $metadata?: { httpStatusCode?: number } };
+  const status = e?.$metadata?.httpStatusCode;
+  return (
+    e?.name === "ConditionalRequestConflict" ||
+    status === 409 ||
+    status === 429 ||
+    (status !== undefined && status >= 500) ||
+    ["TimeoutError", "RequestTimeout", "NetworkingError"].includes(e?.name ?? "") ||
+    ["ECONNRESET", "ECONNREFUSED", "ETIMEDOUT", "EPIPE"].includes(e?.code ?? "")
+  );
+}
+async function retryObjectOperation<T>(operation: () => Promise<T>, signal?: AbortSignal) {
+  for (let attempt = 0; ; attempt++) {
+    signal?.throwIfAborted();
+    try {
+      return await operation();
+    } catch (error) {
+      if (attempt >= 2 || !isRetryable(error)) throw error;
+      await delay(100 * 2 ** attempt, undefined, { signal });
+    }
+  }
+}
+function isMissing(error: unknown) {
+  const e = error as { name?: string; status?: number; $metadata?: { httpStatusCode?: number } };
+  return (
+    e?.name === "NoSuchKey" ||
+    e?.name === "NotFound" ||
+    e?.status === 404 ||
+    e?.$metadata?.httpStatusCode === 404
+  );
+}
+const sha256 = (bytes: Buffer) => createHash("sha256").update(bytes).digest("hex");
+interface ConditionalQualification {
+  work: Promise<void>;
+  controller: AbortController;
+  waiters: number;
+}
+const qualifiedProviders = new WeakSet<StorageProvider>();
+const qualifications = new WeakMap<StorageProvider, ConditionalQualification>();
+async function qualifyConditionalWrites(
+  provider: StorageProvider,
+  prefix: string,
+  signal?: AbortSignal,
+) {
+  signal?.throwIfAborted();
+  if (qualifiedProviders.has(provider)) return;
+  let qualification = qualifications.get(provider);
+  if (!qualification || qualification.controller.signal.aborted) {
+    const controller = new AbortController();
+    const probeSignal = AbortSignal.any([controller.signal, AbortSignal.timeout(15_000)]);
+    const current: ConditionalQualification = { controller, waiters: 0, work: Promise.resolve() };
+    qualification = current;
+    current.work = (async () => {
+      const objectKey = `${prefix}/conditional-probe-${randomUUID()}`;
+      const body = Buffer.from("paperclip-conditional-probe-v1");
+      const put = provider.putObjectConditional!.bind(provider);
+      try {
+        await retryObjectOperation(
+          () =>
+            put({
+              objectKey,
+              body,
+              contentLength: body.length,
+              contentType: "text/plain",
+              ifNoneMatch: "*",
+              signal: probeSignal,
+            }),
+          probeSignal,
+        );
+        for (const condition of [
+          { ifNoneMatch: "*" as const },
+          { ifMatch: '"paperclip-impossible-etag"' },
+        ]) {
+          let rejected = false;
+          try {
+            await retryObjectOperation(
+              () =>
+                put({
+                  objectKey,
+                  body,
+                  contentLength: body.length,
+                  contentType: "text/plain",
+                  ...condition,
+                  signal: probeSignal,
+                }),
+              probeSignal,
+            );
+          } catch (error) {
+            if (!isObjectPreconditionFailure(error)) throw error;
+            rejected = true;
+          }
+          if (!rejected) throw new Error("Object storage does not enforce conditional publication");
+        }
+      } finally {
+        await provider.deleteObject({ objectKey, signal: probeSignal });
+      }
+      probeSignal.throwIfAborted();
+      qualifiedProviders.add(provider);
+    })().finally(() => {
+      if (qualifications.get(provider) === current) qualifications.delete(provider);
+    });
+    qualifications.set(provider, current);
+    void current.work.catch(() => undefined);
+  }
+  const pending = qualification;
+  pending.waiters++;
+  await new Promise<void>((resolve, reject) => {
+    let finished = false;
+    const release = () => {
+      if (finished) return false;
+      finished = true;
+      signal?.removeEventListener("abort", abort);
+      pending.waiters--;
+      return true;
+    };
+    const abort = () => {
+      if (!release()) return;
+      if (!pending.waiters) {
+        pending.controller.abort(signal!.reason);
+        void pending.work.then(
+          () => reject(signal!.reason),
+          () => reject(signal!.reason),
+        );
+      } else reject(signal!.reason);
+    };
+    signal?.addEventListener("abort", abort, { once: true });
+    if (signal?.aborted) abort();
+    pending.work.then(
+      () => {
+        if (release()) resolve();
+      },
+      (error) => {
+        if (release()) reject(error);
+      },
+    );
+  });
+}
+
+let activeIO = 0;
+const waitingIO: Array<() => void> = [];
+async function objectIO<T>(operation: () => Promise<T>): Promise<T> {
+  if (activeIO >= 8) {
+    if (waitingIO.length >= 32) throw new Error("object_journal_io_capacity_exceeded");
+    await new Promise<void>((resolve) => waitingIO.push(resolve));
+  } else activeIO++;
+  try {
+    return await operation();
+  } finally {
+    const next = waitingIO.shift();
+    if (next) next();
+    else activeIO--;
+  }
+}
+
+export interface ObjectByteJournalOptions {
+  provider: StorageProvider;
+  prefix: string;
+
+  encode?: (bytes: Buffer) => Promise<Buffer>;
+  decode?: (bytes: Buffer) => Promise<Buffer>;
+}
+
+export function createObjectByteJournal(options: ObjectByteJournalOptions) {
+  const { provider } = options;
+  const prefix = options.prefix.replace(/\/+$/, "");
+  if (
+    !prefix ||
+    prefix.length > 512 ||
+    prefix.startsWith("/") ||
+    prefix.split("/").some((p) => !/^[a-zA-Z0-9._-]+$/.test(p))
+  ) {
+    throw new Error("invalid_object_journal_prefix");
+  }
+  if (!provider.putObjectConditional || !provider.listObjects)
+    throw new Error("Object storage requires conditional writes and scoped listing");
+  const headKey = `${prefix}/head.json`;
+  const objectPrefix = `${prefix}/objects/`;
+  const putConditional = (
+    input: Parameters<NonNullable<StorageProvider["putObjectConditional"]>>[0],
+  ) => objectIO(() => provider.putObjectConditional!(input));
+
+  async function getBytes(key: string, limit: number, signal?: AbortSignal) {
+    return objectIO(async () => {
+      signal?.throwIfAborted();
+      signal = signal
+        ? AbortSignal.any([signal, AbortSignal.timeout(15_000)])
+        : AbortSignal.timeout(15_000);
+      const result = await retryObjectOperation(
+        () => provider.getObject({ objectKey: key, signal }),
+        signal,
+      );
+      const chunks: Buffer[] = [];
+      let size = 0;
+      try {
+        for await (const chunk of result.stream) {
+          const bytes = Buffer.from(chunk);
+          size += bytes.length;
+          if (size > limit) throw new Error("object_journal_object_too_large");
+          chunks.push(bytes);
+        }
+      } finally {
+        result.stream.destroy();
+      }
+      signal?.throwIfAborted();
+      return { bytes: Buffer.concat(chunks), etag: result.etag };
+    });
+  }
+  async function head(signal?: AbortSignal): Promise<ObjectJournalWriter | null> {
+    let result;
+    try {
+      result = await getBytes(headKey, 64 * 1024, signal);
+    } catch (error) {
+      if (isMissing(error)) return null;
+      throw error;
+    }
+    if (!result.etag) throw new Error("object_journal_missing_etag");
+    const value = headSchema.parse(JSON.parse(result.bytes.toString("utf8")));
+    if (value.roots) {
+      let end = 0,
+        level = MAX_INDEX_LEVEL + 1;
+      if (value.root) throw new Error("object_journal_invalid_head");
+      for (const root of value.roots) {
+        validateReference(root);
+        if (root.start !== end || root.level >= level)
+          throw new Error("object_journal_invalid_head");
+        end = root.end;
+        level = root.level;
+      }
+      if (end !== value.end) throw new Error("object_journal_invalid_head");
+    } else if ((value.root?.end ?? 0) !== value.end || (value.root && value.root.start !== 0)) {
+      throw new Error("object_journal_invalid_head");
+    }
+    return { head: value, etag: result.etag };
+  }
+  async function publish(value: ObjectJournalHead, etag: string | null, signal?: AbortSignal) {
+    value = headSchema.parse(value);
+    signal = signal
+      ? AbortSignal.any([signal, AbortSignal.timeout(15_000)])
+      : AbortSignal.timeout(15_000);
+    const bytes = Buffer.from(JSON.stringify(value));
+    return retryObjectOperation(async () => {
+      try {
+        const result = await putConditional({
+          objectKey: headKey,
+          body: bytes,
+          contentLength: bytes.length,
+          contentType: "application/json",
+          signal,
+          ...(etag ? { ifMatch: etag } : { ifNoneMatch: "*" as const }),
+        });
+        return { head: value, etag: result.etag };
+      } catch (error) {
+        const observed = await head(signal);
+        if (observed && JSON.stringify(observed.head) === JSON.stringify(value)) return observed;
+        if (isObjectPreconditionFailure(error) || (observed?.etag ?? null) !== etag)
+          throw new ObjectJournalConflictError();
+        throw error;
+      }
+    }, signal);
+  }
+  async function claim(input: {
+    expectedOwner: string | null;
+    assertAuthority: () => Promise<void>;
+    signal?: AbortSignal;
+  }) {
+    const current = await head(input.signal);
+    if (current?.head.deleted || (current?.head.owner ?? null) !== input.expectedOwner)
+      throw new ObjectJournalConflictError();
+    await input.assertAuthority();
+    await qualifyConditionalWrites(provider, prefix, input.signal);
+    await input.assertAuthority();
+    const value: ObjectJournalHead = {
+      v: 1,
+      owner: randomUUID(),
+      generation: (current?.head.generation ?? 0) + 1,
+      end: current?.head.end ?? 0,
+      sequence: current?.head.sequence ?? 0,
+      sourceCursor: current?.head.sourceCursor ?? 0,
+      root: current?.head.root ?? null,
+      ...(current?.head.roots ? { roots: current.head.roots } : {}),
+      closed: false,
+      deleted: false,
+      cleanupAfter: 0,
+    };
+    return publish(value, current?.etag ?? null, input.signal);
+  }
+  function validateReference(ref: Reference) {
+    if (
+      !ref.key.startsWith(objectPrefix) ||
+      ref.key !== `${objectPrefix}${ref.sha256}` ||
+      ref.end <= ref.start
+    )
+      throw new Error("object_journal_invalid_reference");
+  }
+  async function load(ref: Reference, signal?: AbortSignal) {
+    validateReference(ref);
+    const stored = await getBytes(ref.key, MAX_OBJECT_BYTES * 2, signal);
+    const bytes = options.decode ? await options.decode(stored.bytes) : stored.bytes;
+    if (bytes.length > MAX_OBJECT_BYTES || sha256(bytes) !== ref.sha256)
+      throw new Error("object_journal_integrity_error");
+    return bytes;
+  }
+  async function save(
+    bytes: Buffer,
+    start: number,
+    end: number,
+    signal?: AbortSignal,
+  ): Promise<Reference> {
+    if (bytes.length > MAX_OBJECT_BYTES) throw new Error("object_journal_segment_too_large");
+    signal = signal
+      ? AbortSignal.any([signal, AbortSignal.timeout(15_000)])
+      : AbortSignal.timeout(15_000);
+    const digest = sha256(bytes),
+      key = `${objectPrefix}${digest}`;
+    const body = options.encode ? await options.encode(bytes) : bytes;
+    try {
+      await retryObjectOperation(
+        () =>
+          putConditional({
+            objectKey: key,
+            body,
+            contentLength: body.length,
+            contentType: "application/octet-stream",
+            ifNoneMatch: "*",
+            signal,
+          }),
+        signal,
+      );
+    } catch (error) {
+      const ref = { key, sha256: digest, start, end };
+      try {
+        await load(ref, signal);
+      } catch {
+        throw error;
+      }
+    }
+    return { key, start, end, sha256: digest };
+  }
+  async function loadNode(ref: Reference, signal?: AbortSignal): Promise<Node> {
+    const node = nodeSchema.parse(JSON.parse((await load(ref, signal)).toString("utf8")));
+    let next = ref.start;
+    for (const child of node.children) {
+      validateReference(child);
+      if (child.start !== next) throw new Error("object_journal_noncontiguous_index");
+      next = child.end;
+    }
+    if (next !== ref.end) throw new Error("object_journal_invalid_index_range");
+    return node;
+  }
+  const saveNode = (node: Node, signal?: AbortSignal) =>
+    save(
+      Buffer.from(JSON.stringify(node)),
+      node.children[0]!.start,
+      node.children.at(-1)!.end,
+      signal,
+    );
+  async function readRange(
+    snapshot: ObjectJournalHead,
+    start: number,
+    limit: number,
+    signal?: AbortSignal,
+  ): Promise<Buffer> {
+    if (snapshot.deleted) throw new Error("object_journal_deleted");
+    if (
+      !Number.isSafeInteger(start) ||
+      start < 0 ||
+      !Number.isSafeInteger(limit) ||
+      limit < 0 ||
+      limit > MAX_OBJECT_BYTES
+    )
+      throw new Error("invalid_object_journal_range");
+    if ((!snapshot.root && !snapshot.roots?.length) || limit === 0 || start >= snapshot.end)
+      return Buffer.alloc(0);
+    const end = Math.min(snapshot.end, start + limit),
+      chunks: Buffer[] = [];
+    async function readPayload(ref: Reference) {
+      const bytes = await load(ref, signal);
+      if (bytes.length !== ref.end - ref.start)
+        throw new Error("object_journal_invalid_segment_length");
+      chunks.push(
+        bytes.subarray(Math.max(0, start - ref.start), Math.min(bytes.length, end - ref.start)),
+      );
+    }
+    async function visit(ref: Reference, expectedLevel?: number) {
+      const node = await loadNode(ref, signal);
+      if (expectedLevel !== undefined && node.level !== expectedLevel)
+        throw new Error("object_journal_invalid_index_level");
+      for (const child of node.children) {
+        if (child.end <= start || child.start >= end) continue;
+        if (node.level > 0) await visit(child, node.level - 1);
+        else await readPayload(child);
+      }
+    }
+    if (snapshot.roots) {
+      for (const root of snapshot.roots) {
+        if (root.end <= start || root.start >= end) continue;
+        if (root.level === 0) await readPayload(root);
+        else await visit(root, root.level - 1);
+      }
+    } else if (snapshot.root) await visit(snapshot.root);
+    const bytes = Buffer.concat(chunks);
+    if (bytes.length !== end - start) throw new Error("object_journal_missing_range");
+    return bytes;
+  }
+  async function append(
+    writer: ObjectJournalWriter,
+    bytes: Buffer,
+    signal?: AbortSignal,
+    sequence = writer.head.sequence,
+    sourceCursor = writer.head.sourceCursor,
+  ) {
+    signal = signal
+      ? AbortSignal.any([signal, AbortSignal.timeout(15_000)])
+      : AbortSignal.timeout(15_000);
+    if (writer.head.closed || writer.head.deleted) throw new ObjectJournalConflictError();
+    if (bytes.length === 0) return writer;
+    if (
+      bytes.length > 8 * 1024 * 1024 ||
+      !Number.isSafeInteger(writer.head.end + bytes.length) ||
+      !Number.isSafeInteger(sequence) ||
+      sequence < writer.head.sequence ||
+      !Number.isSafeInteger(sourceCursor) ||
+      sourceCursor < writer.head.sourceCursor
+    )
+      throw new Error("object_journal_invalid_batch");
+    const roots: Root[] = writer.head.roots
+      ? [...writer.head.roots]
+      : writer.head.root
+        ? [{ ...writer.head.root, level: (await loadNode(writer.head.root, signal)).level + 1 }]
+        : [];
+    let end = writer.head.end;
+    for (let start = 0; start < bytes.length; start += 1024 * 1024) {
+      const part = bytes.subarray(start, start + 1024 * 1024);
+      let root: Root = { ...(await save(part, end, end + part.length, signal)), level: 0 };
+      end = root.end;
+      while (roots.at(-1)?.level === root.level) {
+        const left = roots.pop()!;
+        const { level: leftLevel, ...leftRef } = left;
+        const { level, ...rightRef } = root;
+        if (level >= MAX_INDEX_LEVEL || leftLevel !== level)
+          throw new Error("object_journal_index_capacity_exceeded");
+        root = {
+          ...(await saveNode({ level, children: [leftRef, rightRef] }, signal)),
+          level: level + 1,
+        };
+      }
+      roots.push(root);
+    }
+    return publish(
+      { ...writer.head, end, root: null, roots, sequence, sourceCursor },
+      writer.etag,
+      signal,
+    );
+  }
+
+  async function close(writer: ObjectJournalWriter, signal?: AbortSignal) {
+    return publish({ ...writer.head, closed: true }, writer.etag, signal);
+  }
+  async function remove(signal?: AbortSignal) {
+    let tombstone: ObjectJournalWriter | undefined;
+    for (let attempt = 0; attempt < 8; attempt++) {
+      const current = await head(signal);
+      if (current?.head.deleted) {
+        tombstone = current;
+        break;
+      }
+      const value: ObjectJournalHead = current
+        ? {
+            ...current.head,
+            deleted: true,
+            closed: true,
+            cleanupAfter: current.head.closed ? 0 : Date.now() + 30_000,
+          }
+        : {
+            v: 1,
+            sourceCursor: 0,
+            owner: randomUUID(),
+            generation: 0,
+            end: 0,
+            sequence: 0,
+            root: null,
+            deleted: true,
+            closed: true,
+            cleanupAfter: 0,
+          };
+      try {
+        tombstone = await publish(value, current?.etag ?? null, signal);
+        break;
+      } catch (error) {
+        if (!(error instanceof ObjectJournalConflictError)) throw error;
+      }
+    }
+    if (!tombstone) throw new ObjectJournalConflictError();
+    if (tombstone.head.cleanupAfter > Date.now()) throw new Error("object_journal_cleanup_pending");
+
+    let continuationToken: string | undefined;
+    do {
+      const deadline = () =>
+        signal
+          ? AbortSignal.any([signal, AbortSignal.timeout(15_000)])
+          : AbortSignal.timeout(15_000);
+      const page = await objectIO(() =>
+        provider.listObjects!({ prefix: objectPrefix, continuationToken, signal: deadline() }),
+      );
+      for (const key of page.keys) {
+        if (!key.startsWith(objectPrefix)) throw new Error("object_journal_invalid_listing");
+        await objectIO(() => provider.deleteObject({ objectKey: key, signal: deadline() }));
+      }
+      continuationToken = page.continuationToken;
+    } while (continuationToken);
+  }
+  return { head, claim, append, close, readRange, remove };
+}
+export type ObjectByteJournal = ReturnType<typeof createObjectByteJournal>;

server/src/services/object-run-log-store.test.ts

@@ -0,0 +1,188 @@
+import { createHash } from "node:crypto";
+import { Readable } from "node:stream";
+import { describe, expect, it, vi } from "vitest";
+import type { StorageProvider } from "../storage/types.js";
+import { createObjectRunLogStore } from "./object-run-log-store.js";
+
+async function readBody(body: AsyncIterable<Uint8Array>) {
+  const chunks: Buffer[] = [];
+  for await (const bytes of body) chunks.push(Buffer.from(bytes));
+  return Buffer.concat(chunks);
+}
+
+function memory() {
+  const objects = new Map<string, Buffer>();
+  let puts = 0,
+    unavailable = false;
+  const etag = (b: Buffer) => createHash("sha256").update(b).digest("hex");
+  const provider: StorageProvider = {
+    id: "s3",
+    async putObject() {
+      throw Error("unconditional");
+    },
+    async putObjectConditional(i) {
+      if (unavailable) throw Error("storage offline");
+      const old = objects.get(i.objectKey);
+      if (i.ifNoneMatch ? old : !old || etag(old) !== i.ifMatch)
+        throw Object.assign(Error("conflict"), { name: "PreconditionFailed" });
+      const b = Buffer.isBuffer(i.body) ? i.body : await readBody(i.body);
+      objects.set(i.objectKey, Buffer.from(b));
+      puts++;
+      return { etag: etag(b) };
+    },
+    async getObject(i) {
+      const b = objects.get(i.objectKey);
+      if (!b) throw Object.assign(Error("missing"), { name: "NoSuchKey" });
+      return { stream: Readable.from([b]), etag: etag(b) };
+    },
+    async headObject(i) {
+      return { exists: objects.has(i.objectKey) };
+    },
+    async deleteObject(i) {
+      objects.delete(i.objectKey);
+    },
+    async listObjects(i) {
+      return { keys: [...objects.keys()].filter((k) => k.startsWith(i.prefix)) };
+    },
+  };
+  return {
+    provider,
+    puts: () => puts,
+    offline: (value: boolean) => {
+      unavailable = value;
+    },
+  };
+}
+const input = { companyId: "company", agentId: "agent", runId: "run" };
+const event = (seq: number) => ({
+  stream: "stdout" as const,
+  chunk: "再開",
+  ts: "2026-10-30T00:00:00Z",
+  seq,
+});
+
+describe("S3 run log store", () => {
+  it("batches concurrent appends, confirms durability, and resumes offsets/sequence on a replacement", async () => {
+    const m = memory(),
+      options = {
+        provider: m.provider,
+        keyPrefix: "logs",
+        flushMs: 10,
+        assertWriterAuthority: async () => undefined,
+      };
+    const a = createObjectRunLogStore(options),
+      handle = await a.begin(input),
+      before = m.puts();
+    const sizes = await Promise.all(
+      Array.from({ length: 100 }, (_, i) => a.append(handle, event(i + 1))),
+    );
+    expect(m.puts() - before).toBe(2);
+    const b = createObjectRunLogStore(options);
+    const read = await b.read(handle);
+    expect(read.content.trim().split("\n")).toHaveLength(100);
+    const successor = await b.begin(input);
+    expect(successor.initialSequence).toBe(100);
+    expect(successor.initialBytes).toBe(sizes.reduce((sum, size) => sum + size, 0));
+    await b.append(successor, event(101));
+    const summary = await b.finalize(successor);
+    expect(summary.bytes).toBeGreaterThan(successor.initialBytes!);
+    expect(summary.sha256).toBe(
+      createHash("sha256")
+        .update((await b.read(successor)).content)
+        .digest("hex"),
+    );
+    await expect(a.append(handle, event(102))).rejects.toThrow("another writer");
+  });
+  it("finalization flushes already accepted writes and rejects later append admission", async () => {
+    const m = memory(),
+      store = createObjectRunLogStore({
+        provider: m.provider,
+        keyPrefix: "logs",
+        flushMs: 1000,
+        assertWriterAuthority: async () => undefined,
+      });
+    const h = await store.begin(input),
+      pending = store.append(h, event(1));
+    const finalizing = store.finalize(h);
+    await expect(store.append(h, event(2))).rejects.toThrow("closed");
+    const [size, result] = await Promise.all([pending, finalizing]);
+    expect(result.bytes).toBe(size);
+  });
+  it("does not acknowledge a batch when storage fails", async () => {
+    const m = memory(),
+      store = createObjectRunLogStore({
+        provider: m.provider,
+        keyPrefix: "logs",
+        flushMs: 1,
+        assertWriterAuthority: async () => undefined,
+      });
+    const h = await store.begin(input);
+    m.offline(true);
+    await expect(store.append(h, event(1))).rejects.toThrow("storage offline");
+    m.offline(false);
+    expect((await store.read(h)).content).toBe("");
+  });
+  it("preserves UTF-8 and byte offsets across tiny pages and aligned tails", async () => {
+    const m = memory(),
+      store = createObjectRunLogStore({
+        provider: m.provider,
+        keyPrefix: "logs",
+        flushMs: 1,
+        assertWriterAuthority: async () => undefined,
+      });
+    const h = await store.begin(input);
+    await store.append(h, { ...event(1), chunk: "再開😀" });
+    await store.append(h, event(2));
+    const full = Buffer.from((await store.read(h)).content);
+    let offset = 0,
+      text = "";
+    for (;;) {
+      const page = await store.read(h, { offset, limitBytes: 7 });
+      text += page.content;
+      if (page.nextOffset === undefined) break;
+      expect(page.nextOffset).toBeGreaterThan(offset);
+      offset = page.nextOffset;
+    }
+    expect(Buffer.from(text)).toEqual(full);
+    const start = full.indexOf(Buffer.from("再")) + 1;
+    const tail = await store.read(h, { offset: start, limitBytes: 1000, alignStartToRecord: true });
+    expect(Buffer.from(tail.content)).toEqual(full);
+  });
+  it("backpressures admission at capacity without poisoning the writer or losing accepted output", async () => {
+    const m = memory(),
+      original = m.provider.putObjectConditional!;
+    let release!: () => void,
+      entered = false;
+    const blocked = new Promise<void>((resolve) => {
+      release = resolve;
+    });
+    m.provider.putObjectConditional = async (input) => {
+      if (input.objectKey.includes("/objects/") && !entered) {
+        entered = true;
+        await blocked;
+      }
+      return original(input);
+    };
+    const store = createObjectRunLogStore({
+      provider: m.provider,
+      keyPrefix: "logs",
+      assertWriterAuthority: async () => undefined,
+    });
+    const h = await store.begin(input);
+    const first = await store.enqueue!(h, { ...event(1), chunk: "a".repeat(7 * 1024 * 1024) });
+    await vi.waitFor(() => expect(entered).toBe(true));
+    let admitted = false;
+    const second = store.enqueue!(h, { ...event(2), chunk: "b".repeat(2 * 1024 * 1024) }).then(
+      (receipt) => {
+        admitted = true;
+        return receipt.committed;
+      },
+    );
+    await Promise.resolve();
+    expect(admitted).toBe(false);
+    release();
+    await Promise.all([first.committed, second]);
+    await store.append(h, event(3));
+    expect((await store.finalize(h)).bytes).toBeGreaterThan(9 * 1024 * 1024);
+  });
+});

server/src/services/object-run-log-store.ts

@@ -0,0 +1,339 @@
+import { beginIdleTrackedWork } from "./task-admission.js";
+import { createHash, randomUUID } from "node:crypto";
+import type { StorageProvider } from "../storage/types.js";
+import type { RunLogStore, RunLogHandle, RunLogBeginInput } from "./run-log-store.js";
+import { createObjectByteJournal, type ObjectJournalWriter } from "./object-byte-journal.js";
+
+const MAX_SEGMENT_BYTES = 1024 * 1024;
+const MAX_PENDING_BYTES = 8 * MAX_SEGMENT_BYTES;
+const MAX_READ_BYTES = 4 * MAX_SEGMENT_BYTES;
+export interface ObjectRunLogStoreOptions {
+  provider: StorageProvider;
+  keyPrefix: string;
+  flushMs?: number;
+  segmentBytes?: number;
+
+  assertWriterAuthority: (input: RunLogBeginInput) => Promise<void>;
+}
+interface PendingAppend {
+  bytes: Buffer;
+  sequence: number;
+  sourceCursor: number;
+  resolve: (bytes: number) => void;
+  reject: (error: unknown) => void;
+}
+interface WriterState {
+  journal: ReturnType<typeof createObjectByteJournal>;
+  writer: ObjectJournalWriter;
+  pending: PendingAppend[];
+  pendingBytes: number;
+  timer?: NodeJS.Timeout;
+  flushing?: Promise<void>;
+  closed: boolean;
+  failure?: unknown;
+  signal?: AbortSignal;
+}
+
+export function createObjectRunLogStore(options: ObjectRunLogStoreOptions): RunLogStore {
+  const flushMs = options.flushMs ?? 1000;
+  const segmentBytes = options.segmentBytes ?? MAX_SEGMENT_BYTES;
+  if (
+    !Number.isSafeInteger(flushMs) ||
+    flushMs < 1 ||
+    flushMs > 30_000 ||
+    !Number.isSafeInteger(segmentBytes) ||
+    segmentBytes < 1024 ||
+    segmentBytes > MAX_SEGMENT_BYTES
+  )
+    throw new Error("Invalid object log batching limits");
+  const writers = new Map<RunLogHandle, WriterState>();
+  let totalPendingBytes = 0;
+  let admissionBytes = 0;
+  let admissions = 0;
+  const capacityWaiters = new Set<() => void>();
+  const capacityChanged = () => {
+    for (const wake of capacityWaiters) wake();
+    capacityWaiters.clear();
+  };
+  let activeUploads = 0;
+  const uploadWaiters: Array<() => void> = [];
+  async function acquireUpload(signal?: AbortSignal) {
+    signal?.throwIfAborted();
+    if (activeUploads >= 4)
+      await new Promise<void>((resolve, reject) => {
+        const enter = () => {
+          signal?.removeEventListener("abort", abort);
+          resolve();
+        };
+        const abort = () => {
+          const index = uploadWaiters.indexOf(enter);
+          if (index !== -1) uploadWaiters.splice(index, 1);
+          reject(signal!.reason);
+        };
+        signal?.addEventListener("abort", abort, { once: true });
+        uploadWaiters.push(enter);
+      });
+    else activeUploads++;
+    return () => {
+      const next = uploadWaiters.shift();
+      if (next) next();
+      else activeUploads--;
+    };
+  }
+  function journal(logRef: string) {
+    if (!/^[a-zA-Z0-9._-]+\/[a-zA-Z0-9._-]+\/[a-zA-Z0-9._-]+$/.test(logRef))
+      throw new Error("Invalid object log reference");
+    return createObjectByteJournal({
+      provider: options.provider,
+      prefix: `${options.keyPrefix}/segments-v1/${logRef}`,
+    });
+  }
+  function schedule(state: WriterState) {
+    if (state.closed || state.failure || state.timer || state.flushing || !state.pending.length)
+      return;
+    state.timer = setTimeout(() => {
+      state.timer = undefined;
+      void flush(state).catch(() => undefined);
+    }, flushMs);
+  }
+  async function flush(state: WriterState): Promise<void> {
+    if (state.timer) {
+      clearTimeout(state.timer);
+      state.timer = undefined;
+    }
+    if (state.flushing) {
+      await state.flushing;
+      if (state.pending.length) return flush(state);
+      return;
+    }
+    if (state.failure) throw state.failure;
+    if (!state.pending.length) return;
+    const batch = state.pending.splice(0);
+    const bytes = Buffer.concat(batch.map((p) => p.bytes));
+    state.flushing = (async () => {
+      let release: (() => void) | undefined;
+      try {
+        release = await acquireUpload(state.signal);
+        state.writer = await state.journal.append(
+          state.writer,
+          bytes,
+          state.signal,
+          batch.reduce(
+            (cursor, item) => Math.max(cursor, item.sequence),
+            state.writer.head.sequence,
+          ),
+          batch.reduce(
+            (cursor, item) => Math.max(cursor, item.sourceCursor),
+            state.writer.head.sourceCursor,
+          ),
+        );
+        for (const item of batch) item.resolve(item.bytes.length);
+      } catch (error) {
+        state.failure = error;
+        const waiting = state.pending.splice(0);
+        const waitingBytes = waiting.reduce((sum, item) => sum + item.bytes.length, 0);
+        state.pendingBytes -= waitingBytes;
+        totalPendingBytes -= waitingBytes;
+        for (const item of [...batch, ...waiting]) item.reject(error);
+        throw error;
+      } finally {
+        state.pendingBytes -= bytes.length;
+        totalPendingBytes -= bytes.length;
+        capacityChanged();
+        release?.();
+        state.flushing = undefined;
+        schedule(state);
+      }
+    })();
+    await state.flushing;
+  }
+  async function enqueue(handle: RunLogHandle, event: Parameters<RunLogStore["append"]>[1]) {
+    const state = writers.get(handle);
+    if (!state || state.closed) throw new Error("Object run log writer is closed");
+    state.signal?.throwIfAborted();
+    if (state.failure) throw state.failure;
+    const bytes = Buffer.from(
+      `${JSON.stringify({
+        ts: event.ts,
+        attemptId: handle.attemptId,
+        stream: event.stream,
+        chunk: event.chunk,
+        ...(event.seq === undefined ? {} : { seq: event.seq }),
+      })}\n`,
+    );
+    if (bytes.length > MAX_PENDING_BYTES)
+      throw new Error("Object run log record exceeds pending-byte limit");
+    let waiting = false;
+    try {
+      while (
+        state.pendingBytes + bytes.length > MAX_PENDING_BYTES ||
+        totalPendingBytes + bytes.length > 64 * MAX_SEGMENT_BYTES
+      ) {
+        if (state.closed) throw new Error("Object run log writer is closed");
+        if (state.failure) throw state.failure;
+        if (!waiting) {
+          if (admissions >= 32 || admissionBytes + bytes.length > MAX_PENDING_BYTES)
+            throw new Error("Object run log admission capacity exceeded");
+          waiting = true;
+          admissions++;
+          admissionBytes += bytes.length;
+        }
+        for (const writer of writers.values())
+          if (writer.pending.length) void flush(writer).catch(() => undefined);
+        await new Promise<void>((resolve, reject) => {
+          const wake = () => {
+            state.signal?.removeEventListener("abort", abort);
+            resolve();
+          };
+          const abort = () => {
+            capacityWaiters.delete(wake);
+            reject(state.signal!.reason);
+          };
+          state.signal?.addEventListener("abort", abort, { once: true });
+          capacityWaiters.add(wake);
+        });
+      }
+      if (state.closed) throw new Error("Object run log writer is closed");
+      state.signal?.throwIfAborted();
+      if (state.failure) throw state.failure;
+      state.pendingBytes += bytes.length;
+      totalPendingBytes += bytes.length;
+      const finishIdle = beginIdleTrackedWork();
+      const committed = new Promise<number>((resolve, reject) =>
+        state.pending.push({
+          bytes,
+          sourceCursor: event.sourceCursor ?? state.writer.head.sourceCursor,
+          sequence: event.seq ?? state.writer.head.sequence,
+          resolve,
+          reject,
+        }),
+      ).finally(finishIdle);
+      if (state.pendingBytes >= segmentBytes) void flush(state).catch(() => undefined);
+      else schedule(state);
+      return { committed };
+    } finally {
+      if (waiting) {
+        admissions--;
+        admissionBytes -= bytes.length;
+      }
+    }
+  }
+  return {
+    async begin(input) {
+      const logRef = `${input.companyId}/${input.agentId}/${input.runId}`;
+      const store = journal(logRef),
+        current = await store.head(input.signal);
+      const writer = await store.claim({
+        expectedOwner: current?.head.owner ?? null,
+        assertAuthority: () => options.assertWriterAuthority(input),
+        signal: input.signal,
+      });
+      const handle: RunLogHandle = {
+        store: "s3_segments",
+        logRef,
+        attemptId: randomUUID(),
+        initialBytes: writer.head.end,
+        initialSequence: writer.head.sequence,
+        initialSourceCursor: writer.head.sourceCursor,
+      };
+      writers.set(handle, {
+        journal: store,
+        writer,
+        pending: [],
+        pendingBytes: 0,
+        closed: false,
+        signal: input.signal,
+      });
+      return handle;
+    },
+    enqueue,
+    async append(handle, event) {
+      return (await enqueue(handle, event)).committed;
+    },
+    async flush(handle) {
+      const state = writers.get(handle);
+      if (!state) throw new Error("Object run log writer not found");
+      await flush(state);
+    },
+    async finalize(handle, finalizeOptions) {
+      const state = writers.get(handle);
+      if (!state) throw new Error("Object run log writer not found");
+      state.closed = true;
+      capacityChanged();
+      try {
+        await flush(state);
+        state.writer = await state.journal.close(state.writer, state.signal);
+        if (finalizeOptions?.computeDigest === false)
+          return { bytes: state.writer.head.end, sha256: null, compressed: false };
+        const hash = createHash("sha256");
+        for (let start = 0; start < state.writer.head.end; start += MAX_READ_BYTES) {
+          hash.update(
+            await state.journal.readRange(state.writer.head, start, MAX_READ_BYTES, state.signal),
+          );
+        }
+        return { bytes: state.writer.head.end, sha256: hash.digest("hex"), compressed: false };
+      } catch {
+        process.emitWarning(
+          "Run log persistence is incomplete; final size and digest are unknown.",
+        );
+        return { bytes: null, sha256: null, compressed: false };
+      } finally {
+        writers.delete(handle);
+      }
+    },
+    async read(handle, opts = {}) {
+      if (handle.store !== "s3_segments") throw new Error("Invalid object run log backend");
+      const store = journal(handle.logRef),
+        snapshot = await store.head(opts.signal);
+      if (!snapshot || snapshot.head.deleted) throw new Error("Object run log not found");
+      const offset = Math.max(0, opts.offset ?? 0),
+        limit = opts.limitBytes ?? 256_000;
+      if (!Number.isSafeInteger(limit) || limit < 1 || limit > MAX_READ_BYTES)
+        throw new Error("Invalid object log read limit");
+      const fetched = await store.readRange(
+        snapshot.head,
+        offset,
+        Math.min(MAX_READ_BYTES, limit + 3),
+        opts.signal,
+      );
+      let end = Math.min(limit, fetched.length);
+      while (end < fetched.length && (fetched[end]! & 0xc0) === 0x80) end++;
+      if (
+        end === fetched.length &&
+        offset + end < snapshot.head.end &&
+        fetched.length === MAX_READ_BYTES
+      ) {
+        let lead = end - 1;
+        while (lead >= 0 && (fetched[lead]! & 0xc0) === 0x80) lead--;
+        const first = fetched[lead] ?? 0;
+        const width = first >= 0xf0 ? 4 : first >= 0xe0 ? 3 : first >= 0xc0 ? 2 : 1;
+        if (end - lead < width) end = lead;
+      }
+      const bytes = fetched.subarray(0, end);
+      const nextOffset =
+        offset + bytes.length < snapshot.head.end ? offset + bytes.length : undefined;
+      if (!opts.alignStartToRecord || offset === 0 || bytes.length === 0)
+        return { content: bytes.toString("utf8"), nextOffset };
+      const parts = [bytes];
+      let before = offset;
+
+      while (before > 0) {
+        const start = Math.max(0, before - 32_000);
+        const prefix = await store.readRange(snapshot.head, start, before - start, opts.signal);
+        const boundary = prefix.lastIndexOf(10);
+        parts.unshift(prefix.subarray(boundary + 1));
+        if (boundary >= 0) break;
+        before = start;
+        if (parts.reduce((n, part) => n + part.length, 0) > MAX_PENDING_BYTES)
+          throw new Error("Object log record exceeds read alignment limit");
+      }
+      return {
+        content: Buffer.concat(parts).toString("utf8"),
+        nextOffset: nextOffset ?? offset + bytes.length,
+      };
+    },
+    async flushInflightMirrors() {
+      await Promise.all([...writers.values()].map(flush));
+    },
+  };
+}

server/src/services/provider-trace-cancellation.integration.test.ts

@@ -0,0 +1,143 @@
+import { randomUUID } from "node:crypto";
+import { promises as fs } from "node:fs";
+import { createServer, request as forward } from "node:http";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import { setTimeout as delay } from "node:timers/promises";
+import {
+  S3Client,
+  CreateBucketCommand,
+  ListObjectsV2Command,
+  DeleteObjectsCommand,
+  DeleteBucketCommand,
+} from "@aws-sdk/client-s3";
+import { afterEach, describe, expect, it, vi } from "vitest";
+import { createS3StorageProvider } from "../storage/s3-provider.js";
+import { createProviderTraceObjectStore } from "./provider-trace-object-store.js";
+import { idleWorkSnapshot } from "./task-admission.js";
+
+const endpoint = process.env.PAPERCLIP_LOG_TEST_S3_ENDPOINT;
+afterEach(() => vi.unstubAllEnvs());
+describe.skipIf(!endpoint)("trace migration cancellation through the real S3 SDK", () => {
+  it.each(["open", "probe", "upload", "restore", "read"])(
+    "cancels a stalled %s and releases its capture",
+    async (stage) => {
+      vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "a".repeat(64));
+      const bucket = `pc-trace-abort-${randomUUID()}`;
+      const directory = await fs.mkdtemp(path.join(tmpdir(), "trace-abort-"));
+      const client = new S3Client({ endpoint, region: "us-east-1", forcePathStyle: true });
+      await client.send(new CreateBucketCommand({ Bucket: bucket }));
+      const ref = `s3-segments-v1:company/run/${randomUUID()}.ndjson`;
+      const controller = new AbortController();
+      let enabled = true,
+        closed = false,
+        started!: () => void;
+      const stalled = new Promise<void>((resolve) => {
+        started = resolve;
+      });
+      const baseline = idleWorkSnapshot().active;
+      const proxy = createServer((request, response) => {
+        const url = request.url!;
+        const matches =
+          stage === "open"
+            ? request.method === "GET" && url.includes("head.json")
+            : stage === "probe"
+              ? request.method === "PUT" && url.includes("conditional-probe-")
+              : stage === "upload"
+                ? request.method === "PUT" && url.includes("/objects/")
+                : request.method === "GET" && url.includes("/objects/");
+        if (enabled && matches) {
+          response.on("close", () => {
+            closed = true;
+          });
+          started();
+          return;
+        }
+        const upstream = forward(
+          new URL(url, endpoint),
+          { method: request.method, headers: request.headers },
+          (result) => {
+            response.writeHead(result.statusCode!, result.headers);
+            result.pipe(response);
+          },
+        );
+        response.on("close", () => upstream.destroy());
+        upstream.on("error", (error) => response.destroy(error));
+        request.pipe(upstream);
+      });
+      await new Promise<void>((resolve) => proxy.listen(0, "127.0.0.1", resolve));
+      const address = proxy.address();
+      if (!address || typeof address === "string") throw new Error("proxy did not bind");
+      const store = createProviderTraceObjectStore(
+        createS3StorageProvider({
+          endpoint: `http://127.0.0.1:${address.port}`,
+          bucket,
+          region: "us-east-1",
+          forcePathStyle: true,
+        }),
+      );
+      let operation: Promise<unknown> | undefined;
+      try {
+        const file = path.join(directory, "import.ndjson");
+        const body = Buffer.from('{"kind":"frame","rawBase64":"e30="}\n');
+        if (stage === "read" || stage === "restore") {
+          const seed = createProviderTraceObjectStore(
+            createS3StorageProvider({
+              endpoint,
+              bucket,
+              region: "us-east-1",
+              forcePathStyle: true,
+            }),
+          );
+          const original = path.join(directory, "original.ndjson");
+          await fs.writeFile(original, body);
+          await seed.open(ref, original, async () => undefined, { restore: false });
+          await seed.finish(ref);
+        }
+        if (stage === "upload") {
+          await fs.writeFile(file, body);
+          await store.open(ref, file, async () => undefined, {
+            restore: false,
+            signal: controller.signal,
+          });
+          operation = store.finish(ref);
+        } else if (stage === "read") operation = store.read(ref, controller.signal);
+        else
+          operation = store.open(ref, file, async () => undefined, {
+            restore: stage === "restore",
+            signal: controller.signal,
+          });
+        const result = operation.then(
+          () => "resolved",
+          () => "aborted",
+        );
+        await stalled;
+        controller.abort();
+        expect(await Promise.race([result, delay(500).then(() => "still running")])).toBe(
+          "aborted",
+        );
+        await vi.waitFor(() => expect(closed).toBe(true));
+        expect(idleWorkSnapshot().active).toBe(baseline);
+        if (stage === "upload") expect(await fs.readFile(file)).toEqual(body);
+      } finally {
+        enabled = false;
+        controller.abort();
+        proxy.closeAllConnections();
+        await operation?.catch(() => undefined);
+        await store.stopRun("company", "run");
+        await new Promise<void>((resolve) => proxy.close(() => resolve()));
+        const objects = await client.send(new ListObjectsV2Command({ Bucket: bucket }));
+        if (objects.Contents?.length)
+          await client.send(
+            new DeleteObjectsCommand({
+              Bucket: bucket,
+              Delete: { Objects: objects.Contents.map((object) => ({ Key: object.Key! })) },
+            }),
+          );
+        await client.send(new DeleteBucketCommand({ Bucket: bucket }));
+        client.destroy();
+        await fs.rm(directory, { recursive: true, force: true });
+      }
+    },
+  );
+});

server/src/services/provider-trace-object-store.test.ts

@@ -0,0 +1,95 @@
+import { randomUUID } from "node:crypto";
+import { promises as fs } from "node:fs";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import { afterEach, describe, expect, it, vi } from "vitest";
+import { createS3StorageProvider } from "../storage/s3-provider.js";
+import { createProviderTraceObjectStore } from "./provider-trace-object-store.js";
+import {
+  S3Client,
+  CreateBucketCommand,
+  ListObjectsV2Command,
+  GetObjectCommand,
+  DeleteObjectsCommand,
+  DeleteBucketCommand,
+} from "@aws-sdk/client-s3";
+
+const endpoint = process.env.PAPERCLIP_LOG_TEST_S3_ENDPOINT;
+afterEach(() => vi.unstubAllEnvs());
+describe.skipIf(!endpoint)("encrypted object provider traces (S3 integration)", () => {
+  it.each(["environment", "existing key file"])(
+    "restores both sidecars on a new host, preserves exact downloads, and removes ciphertext (%s)",
+    async (source) => {
+      const bucket = `paperclip-trace-test-${randomUUID()}`;
+      const client = new S3Client({ endpoint, region: "us-east-1", forcePathStyle: true });
+      await client.send(new CreateBucketCommand({ Bucket: bucket }));
+      const directory = await fs.mkdtemp(path.join(tmpdir(), "object-trace-"));
+      const keyFile = path.join(directory, "existing-vault.key");
+      vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", source === "environment" ? "a".repeat(64) : "");
+      vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY_FILE", keyFile);
+      if (source === "existing key file")
+        await fs.writeFile(keyFile, "a".repeat(64), { mode: 0o600 });
+      const filename = `${randomUUID()}.ndjson`,
+        ref = `s3-segments-v1:company/run/${filename}`;
+      const bytes = Buffer.from('{"kind":"frame","rawBase64":"cHJpdmF0ZQ=="}\n');
+      const interpretation = Buffer.from('{"kind":"interpretation","frameId":1}\n');
+      const provider = createS3StorageProvider({
+        endpoint,
+        bucket,
+        region: "us-east-1",
+        forcePathStyle: true,
+      });
+      try {
+        const a = createProviderTraceObjectStore(provider),
+          fileA = path.join(directory, "a.ndjson");
+        await a.open(ref, fileA, async () => undefined);
+        await fs.appendFile(fileA, bytes);
+        await fs.appendFile(`${fileA}.rehydration`, interpretation);
+        await a.finish(ref);
+        const objects = await client.send(new ListObjectsV2Command({ Bucket: bucket }));
+        for (const object of objects.Contents ?? []) {
+          if (object.Key?.endsWith("head.json")) continue;
+          const stored = await client.send(
+            new GetObjectCommand({ Bucket: bucket, Key: object.Key }),
+          );
+          const payload = await stored.Body!.transformToByteArray();
+          expect(Buffer.from(payload).includes(Buffer.from("rawBase64"))).toBe(false);
+          expect(Buffer.from(payload).includes(Buffer.from("interpretation"))).toBe(false);
+        }
+        await fs.rm(fileA);
+        await fs.rm(`${fileA}.rehydration`);
+        const b = createProviderTraceObjectStore(provider),
+          fileB = path.join(directory, "b.ndjson");
+        expect(await b.read(ref)).toEqual(Buffer.concat([bytes, interpretation]));
+        await fs.mkdir(`${fileB}.rehydration`);
+        await expect(b.open(ref, fileB, async () => undefined)).rejects.toThrow();
+        expect(await b.settled(ref)).toBe(true);
+        await expect(fs.stat(fileB)).rejects.toMatchObject({ code: "ENOENT" });
+        expect(await b.read(ref)).toEqual(Buffer.concat([bytes, interpretation]));
+        await fs.rmdir(`${fileB}.rehydration`);
+        await b.open(ref, fileB, async () => undefined);
+        expect(await fs.readFile(fileB)).toEqual(bytes);
+        expect(await fs.readFile(`${fileB}.rehydration`)).toEqual(interpretation);
+        await fs.appendFile(fileB, Buffer.from('{"kind":"trace_status","status":"complete"}\n'));
+        await b.finish(ref);
+        await b.remove(ref);
+        await expect(b.read(ref)).rejects.toThrow("unavailable");
+        const left = await client.send(new ListObjectsV2Command({ Bucket: bucket }));
+        expect(left.Contents?.every((o) => o.Key!.endsWith("head.json"))).toBe(true);
+      } finally {
+        const objects = await client.send(new ListObjectsV2Command({ Bucket: bucket }));
+        if (objects.Contents?.length)
+          await client.send(
+            new DeleteObjectsCommand({
+              Bucket: bucket,
+              Delete: { Objects: objects.Contents.map((o) => ({ Key: o.Key! })) },
+            }),
+          );
+        await client.send(new DeleteBucketCommand({ Bucket: bucket }));
+        await fs.rm(directory, { recursive: true, force: true });
+        client.destroy();
+      }
+    },
+    30_000,
+  );
+});

server/src/services/provider-trace-object-store.ts

@@ -0,0 +1,293 @@
+import { createCipheriv, createDecipheriv, randomBytes } from "node:crypto";
+import { beginIdleTrackedWork } from "./task-admission.js";
+import { watch, type FSWatcher } from "node:fs";
+import { promises as fs } from "node:fs";
+import {
+  createObjectByteJournal,
+  type ObjectByteJournal,
+  type ObjectJournalWriter,
+} from "./object-byte-journal.js";
+import { deriveLocalEncryptedInstanceSecret } from "../secrets/local-encrypted-provider.js";
+import type { StorageProvider } from "../storage/types.js";
+
+const CHUNK_BYTES = 1024 * 1024;
+const MAX_CAPTURE_BYTES = 192 * CHUNK_BYTES;
+const REF = /^s3-segments-v1:([a-zA-Z0-9-]+)\/([a-zA-Z0-9-]+)\/([a-f0-9-]+\.ndjson)$/;
+export function objectProviderTraceRef(ref: string) {
+  return REF.exec(ref);
+}
+export function importingProviderTraceRef(ref: string) {
+  return ref.startsWith("s3-import-v1:")
+    ? objectProviderTraceRef(ref.replace("s3-import-v1:", "s3-segments-v1:"))
+    : null;
+}
+export function providerTraceCleanupRef(ref: string) {
+  return objectProviderTraceRef(ref)
+    ? ref
+    : importingProviderTraceRef(ref)
+      ? ref.replace("s3-import-v1:", "s3-segments-v1:")
+      : null;
+}
+interface Channel {
+  path: string;
+  journal: ObjectByteJournal;
+  writer: ObjectJournalWriter;
+  watcher?: FSWatcher;
+}
+interface Capture {
+  channels: Channel[];
+  controller: AbortController;
+  signal: AbortSignal;
+  flush?: Promise<void>;
+  timer?: NodeJS.Timeout;
+  stopped: boolean;
+  dirty: boolean;
+  error?: unknown;
+  finishIdle: () => void;
+}
+
+export function createProviderTraceObjectStore(
+  provider: StorageProvider,
+  prefix = "provider-traces",
+) {
+  deriveLocalEncryptedInstanceSecret("provider-trace:existing-key", process.env, {
+    allowCreate: false,
+  });
+  const captures = new Map<string, Capture>();
+  const openings = new Map<string, Promise<void>>();
+  function schedule(capture: Capture) {
+    if (capture.stopped || !capture.dirty || capture.timer || capture.flush || capture.error)
+      return;
+    capture.timer = setTimeout(() => {
+      capture.timer = undefined;
+      void sync(capture).catch(() => undefined);
+    }, 1000);
+    capture.timer.unref();
+  }
+  function channelsFor(ref: string, filePath = "") {
+    const match = objectProviderTraceRef(ref);
+    if (!match) throw new Error("invalid_object_trace_ref");
+    return ["native", "rehydration"].map((channel, i) => {
+      const root = `${prefix}/segments-v1/${match[1]}/${match[2]}/${match[3]}/${channel}`;
+      const key = Buffer.from(
+        deriveLocalEncryptedInstanceSecret(`provider-trace:${root}`, process.env, {
+          allowCreate: false,
+        }),
+        "base64url",
+      );
+      const aad = Buffer.from(root);
+      const journal = createObjectByteJournal({
+        provider,
+        prefix: root,
+        async encode(bytes) {
+          const iv = randomBytes(12),
+            cipher = createCipheriv("aes-256-gcm", key, iv);
+          cipher.setAAD(aad);
+          const body = Buffer.concat([cipher.update(bytes), cipher.final()]);
+          return Buffer.concat([iv, cipher.getAuthTag(), body]);
+        },
+        async decode(bytes) {
+          if (bytes.length < 28) throw new Error("invalid_encrypted_trace_object");
+          const cipher = createDecipheriv("aes-256-gcm", key, bytes.subarray(0, 12));
+          cipher.setAAD(aad);
+          cipher.setAuthTag(bytes.subarray(12, 28));
+          return Buffer.concat([cipher.update(bytes.subarray(28)), cipher.final()]);
+        },
+      });
+      return { journal, path: i === 0 ? filePath : `${filePath}.rehydration` };
+    });
+  }
+  async function sync(capture: Capture) {
+    if (capture.timer) {
+      clearTimeout(capture.timer);
+      capture.timer = undefined;
+    }
+    if (capture.flush) return capture.flush;
+    capture.dirty = false;
+    capture.flush = (async () => {
+      for (const channel of capture.channels) {
+        capture.signal.throwIfAborted();
+        const file = await fs.open(channel.path, "r");
+        try {
+          const stat = await file.stat();
+          if (stat.size > MAX_CAPTURE_BYTES || stat.size < channel.writer.head.end)
+            throw new Error("provider_trace_capture_size_invalid");
+          let offset = channel.writer.head.end;
+          while (offset < stat.size) {
+            const buffer = Buffer.alloc(Math.min(CHUNK_BYTES, stat.size - offset));
+            const { bytesRead } = await file.read(buffer, 0, buffer.length, offset);
+            if (bytesRead !== buffer.length) throw new Error("provider_trace_capture_changed");
+            channel.writer = await channel.journal.append(channel.writer, buffer, capture.signal);
+            offset += buffer.length;
+          }
+        } finally {
+          await file.close();
+        }
+      }
+    })()
+      .catch((error) => {
+        capture.error = error;
+        throw error;
+      })
+      .finally(() => {
+        capture.flush = undefined;
+        schedule(capture);
+      });
+    return capture.flush;
+  }
+  async function open(
+    ref: string,
+    filePath: string,
+    assertAuthority: () => Promise<void>,
+    options: { restore?: boolean; signal?: AbortSignal } = {},
+  ) {
+    const existing = openings.get(ref);
+    if (existing) return existing;
+    const opening = openCapture(ref, filePath, assertAuthority, options).finally(() =>
+      openings.delete(ref),
+    );
+    openings.set(ref, opening);
+    await opening;
+  }
+  async function openCapture(
+    ref: string,
+    filePath: string,
+    assertAuthority: () => Promise<void>,
+    options: { restore?: boolean; signal?: AbortSignal } = {},
+  ) {
+    if (captures.has(ref)) return;
+    const controller = new AbortController();
+    const capture: Capture = {
+      channels: [],
+      controller,
+      signal: options.signal
+        ? AbortSignal.any([controller.signal, options.signal])
+        : controller.signal,
+      stopped: false,
+      dirty: false,
+      finishIdle: () => undefined,
+    };
+    try {
+      for (const source of channelsFor(ref, filePath)) {
+        capture.signal.throwIfAborted();
+        const current = await source.journal.head(capture.signal);
+        const writer = await source.journal.claim({
+          expectedOwner: current?.head.owner ?? null,
+          assertAuthority,
+          signal: capture.signal,
+        });
+        capture.channels.push({ ...source, writer });
+        const file = await fs.open(source.path, options.restore === false ? "a+" : "w", 0o600);
+        try {
+          await file.chmod(0o600);
+          for (
+            let start = 0;
+            options.restore !== false && start < writer.head.end;
+            start += CHUNK_BYTES
+          ) {
+            await file.write(
+              await source.journal.readRange(writer.head, start, CHUNK_BYTES, capture.signal),
+            );
+          }
+        } finally {
+          await file.close();
+        }
+      }
+      capture.signal.throwIfAborted();
+      capture.finishIdle = beginIdleTrackedWork();
+      captures.set(ref, capture);
+      for (const channel of capture.channels) {
+        channel.watcher = watch(channel.path, () => {
+          capture.dirty = true;
+          schedule(capture);
+        });
+        channel.watcher.on("error", (error) => {
+          capture.error = error;
+        });
+        channel.watcher.unref();
+      }
+    } catch (error) {
+      capture.stopped = true;
+      for (const channel of capture.channels) channel.watcher?.close();
+      await Promise.allSettled(
+        capture.channels.map(async (channel) => {
+          await channel.journal.close(channel.writer, capture.signal);
+        }),
+      );
+      if (options.restore !== false)
+        await Promise.allSettled(capture.channels.map((channel) => fs.unlink(channel.path)));
+      captures.delete(ref);
+      capture.finishIdle();
+      throw error;
+    }
+  }
+  async function finish(ref: string) {
+    const capture = captures.get(ref);
+    if (!capture) return;
+    capture.stopped = true;
+    for (const channel of capture.channels) channel.watcher?.close();
+    try {
+      await capture.flush;
+      if (capture.error) throw capture.error;
+      await sync(capture);
+      for (const channel of capture.channels)
+        channel.writer = await channel.journal.close(channel.writer, capture.signal);
+    } finally {
+      if (capture.timer) clearTimeout(capture.timer);
+      captures.delete(ref);
+      capture.finishIdle();
+    }
+  }
+  async function* readChunks(ref: string, signal?: AbortSignal) {
+    let nativeTail: number | undefined;
+    for (const [index, { journal }] of channelsFor(ref).entries()) {
+      const snapshot = await journal.head(signal);
+      if (!snapshot || snapshot.head.deleted) throw new Error("provider_trace_payload_unavailable");
+      if (snapshot.head.end > MAX_CAPTURE_BYTES)
+        throw new Error("provider_trace_capture_size_invalid");
+      if (index === 1 && snapshot.head.end && nativeTail !== undefined && nativeTail !== 10)
+        yield Buffer.from("\n");
+      for (let offset = 0; offset < snapshot.head.end; offset += CHUNK_BYTES) {
+        const bytes = await journal.readRange(snapshot.head, offset, CHUNK_BYTES, signal);
+        if (index === 0 && bytes.length) nativeTail = bytes.at(-1);
+        yield bytes;
+      }
+    }
+  }
+  async function read(ref: string, signal?: AbortSignal) {
+    const chunks: Buffer[] = [];
+    for await (const bytes of readChunks(ref, signal)) chunks.push(bytes);
+    return Buffer.concat(chunks);
+  }
+  async function stop(ref: string) {
+    const capture = captures.get(ref);
+    if (!capture) return;
+    capture.stopped = true;
+    capture.controller.abort(new Error("provider_trace_capture_stopped"));
+    if (capture.timer) clearTimeout(capture.timer);
+    for (const channel of capture.channels) channel.watcher?.close();
+    await capture.flush?.catch(() => undefined);
+    captures.delete(ref);
+    capture.finishIdle();
+  }
+  async function stopRun(companyId: string, runId: string) {
+    const scope = `s3-segments-v1:${companyId}/${runId}/`;
+    await Promise.all([...captures.keys()].filter((ref) => ref.startsWith(scope)).map(stop));
+  }
+  async function remove(ref: string, signal?: AbortSignal) {
+    await stop(ref);
+    const results = await Promise.allSettled(
+      channelsFor(ref).map(({ journal }) => journal.remove(signal)),
+    );
+    for (const result of results) if (result.status === "rejected") throw result.reason;
+  }
+  async function settled(ref: string) {
+    for (const { journal } of channelsFor(ref)) {
+      const snapshot = await journal.head();
+      if (!snapshot || !snapshot.head.closed || snapshot.head.deleted) return false;
+    }
+    return true;
+  }
+  return { open, finish, read, readChunks, remove, settled, stopRun };
+}
+export type ProviderTraceObjectStore = ReturnType<typeof createProviderTraceObjectStore>;

server/src/services/provider-trace-retention-work.ts

@@ -0,0 +1,39 @@
+import type { Db } from "@paperclipai/db";
+import { providerTraceStore } from "./provider-trace-store.js";
+import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
+import { registerDeadlineWork } from "./deadline-work.js";
+import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
+import { registerIdleLocalWorkProbe } from "./idle-local-work.js";
+
+export function registerProviderTraceRetentionWork(
+  db: Db,
+  coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
+  canRun: () => boolean,
+) {
+  const traces = providerTraceStore(db);
+  let deadline: number | null | undefined;
+
+  const unregister = registerIdleLocalWorkProbe(() =>
+    deadline === undefined ? "unknown" : deadline === null ? "none" : "present",
+  );
+  const worker = registerDeadlineWork(coordinator, DELIVERY_QUEUES.providerTraceRetention, {
+    canRun,
+    retryMs: 30_000,
+    onSettled: () => {
+      deadline = undefined;
+    },
+    run: (signal) =>
+      signal.aborted ? Promise.resolve() : traces.cleanupExpired(new Date(), signal).then(() => undefined),
+    next: async () => {
+      deadline = await traces.nextRetentionDeadline();
+      return deadline;
+    },
+  });
+  return {
+    ...worker,
+    stop: async () => {
+      unregister();
+      await worker.stop();
+    },
+  };
+}

server/src/services/provider-trace-retention.integration.test.ts

@@ -0,0 +1,280 @@
+import { randomUUID } from "node:crypto";
+import { promises as fs } from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { sql } from "drizzle-orm";
+import { createDb, providerTraceRecords, type Db } from "@paperclipai/db";
+import {
+  S3Client,
+  CreateBucketCommand,
+  DeleteBucketCommand,
+  DeleteObjectsCommand,
+  ListObjectsV2Command,
+} from "@aws-sdk/client-s3";
+import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
+import { providerTraceStore } from "./provider-trace-store.js";
+import { idleWorkSnapshot } from "./task-admission.js";
+import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
+
+vi.mock("./activity-log.js", () => ({ logActivity: vi.fn(async () => undefined) }));
+const databaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_RUN_EVIDENCE_MIGRATION_DATABASE_URL");
+const endpoint = process.env.PAPERCLIP_LOG_TEST_S3_ENDPOINT;
+describe.skipIf(!databaseUrl)("provider trace retention and capture ownership", () => {
+  let db: Db, directory: string, client: S3Client | undefined;
+  const schema = `trace_retention_${randomUUID().replaceAll("-", "")}`;
+  const bucket = `pc-trace-retention-${randomUUID()}`;
+  let bucketCreated = false;
+  const owners: Array<{
+    store: ReturnType<typeof providerTraceStore>;
+    run: string;
+    company: string;
+  }> = [];
+  beforeAll(async () => {
+    db = createDb(databaseUrl!, { maxConnections: 1 });
+    await db.execute(sql.raw(`create schema ${schema}`));
+    await db.execute(sql.raw(`set search_path=${schema}`));
+    await db.execute(sql`create table provider_trace_records(id uuid primary key default gen_random_uuid(), company_id uuid not null, run_id uuid not null unique,
+      status text not null, provider text not null, trace_ref text not null, frame_count int not null default 0,
+      byte_count bigint not null default 0, digest text, reason text, requested_by text not null, expires_at timestamptz not null,
+      deleted_at timestamptz, created_at timestamptz not null default now(), updated_at timestamptz not null default now())`);
+    if (endpoint) {
+      client = new S3Client({ endpoint, region: "us-east-1", forcePathStyle: true });
+      await client.send(new CreateBucketCommand({ Bucket: bucket }));
+      bucketCreated = true;
+    }
+  });
+  beforeEach(async () => {
+    directory = await fs.mkdtemp(path.join(os.tmpdir(), "trace-retention-"));
+    vi.stubEnv("PAPERCLIP_CONFIG", path.join(directory, "config.json"));
+    vi.stubEnv("PROVIDER_TRACE_BASE_PATH", directory);
+    vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "local_disk");
+    vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "e".repeat(64));
+    await db.execute(sql`truncate provider_trace_records`);
+  });
+  afterEach(async () => {
+    for (const { store, run, company } of owners.splice(0))
+      await store.finalize(run, company).catch(() => undefined);
+    vi.useRealTimers();
+    vi.unstubAllEnvs();
+    await fs.rm(directory, { recursive: true, force: true });
+  });
+  afterAll(async () => {
+    if (client && bucketCreated) {
+      for (;;) {
+        const page = await client.send(new ListObjectsV2Command({ Bucket: bucket }));
+        if (!page.Contents?.length) break;
+        await client.send(
+          new DeleteObjectsCommand({
+            Bucket: bucket,
+            Delete: { Objects: page.Contents.map((object) => ({ Key: object.Key! })) },
+          }),
+        );
+      }
+      await client.send(new DeleteBucketCommand({ Bucket: bucket }));
+      client.destroy();
+    }
+    if (db) {
+      await db.execute(sql.raw(`drop schema ${schema} cascade`));
+      await db.$client.end();
+    }
+  });
+  function external() {
+    vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "s3");
+    vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", bucket);
+    vi.stubEnv("PAPERCLIP_STORAGE_S3_ENDPOINT", endpoint!);
+    vi.stubEnv("PAPERCLIP_STORAGE_S3_REGION", "us-east-1");
+    vi.stubEnv("PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE", "true");
+    vi.stubEnv("PAPERCLIP_STORAGE_S3_PREFIX", "");
+  }
+  it("schedules expiry when a new local trace is created after startup becomes quiet", async () => {
+    const store = providerTraceStore(db),
+      companyId = randomUUID(),
+      runId = randomUUID();
+    expect(await store.nextRetentionDeadline()).toBeNull();
+    const capture = await store.prepare({
+      companyId,
+      runId,
+      provider: "codex",
+      requestedBy: "test",
+    });
+    expect(await store.nextRetentionDeadline()).toBe(
+      new Date(capture.metadata.expiresAt).getTime(),
+    );
+    vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "s3");
+    vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", bucket);
+    vi.stubEnv("PAPERCLIP_STORAGE_S3_ENDPOINT", "http://127.0.0.1:1");
+    vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", undefined);
+    const replacement = providerTraceStore(db);
+    expect(await replacement.getByRun(runId, companyId)).not.toBeNull();
+    await replacement.remove(runId, companyId);
+    expect(await store.nextRetentionDeadline()).toBeNull();
+  });
+  it("drains more than one page of expired local files without requiring another producer", async () => {
+    const companyId = randomUUID(),
+      expiresAt = new Date(Date.now() - 60_000);
+    const rows = Array.from({ length: 201 }, () => ({
+      companyId,
+      runId: randomUUID(),
+      status: "complete",
+      provider: "codex",
+      traceRef: `${randomUUID()}.ndjson`,
+      requestedBy: "test",
+      expiresAt,
+    }));
+    for (const row of rows) await fs.writeFile(path.join(directory, row.traceRef), "{}\n");
+    await db.insert(providerTraceRecords).values(rows);
+    const store = providerTraceStore(db);
+    expect(await store.nextRetentionDeadline()).toBeLessThanOrEqual(Date.now());
+    expect(await store.cleanupExpired()).toHaveLength(100);
+    expect(await store.nextRetentionDeadline()).not.toBeNull();
+    expect(await store.cleanupExpired()).toHaveLength(100);
+    expect(await store.nextRetentionDeadline()).not.toBeNull();
+    expect(await store.cleanupExpired()).toHaveLength(1);
+    expect(await store.nextRetentionDeadline()).toBeNull();
+    expect(await fs.readdir(directory)).toEqual([]);
+  });
+  it("retains an expired local cleanup deadline after filesystem deletion fails", async () => {
+    const companyId = randomUUID(),
+      runId = randomUUID(),
+      traceRef = `${randomUUID()}.ndjson`;
+    const file = path.join(directory, traceRef);
+    await fs.mkdir(file);
+    await db
+      .insert(providerTraceRecords)
+      .values({
+        companyId,
+        runId,
+        traceRef,
+        status: "complete",
+        provider: "codex",
+        requestedBy: "test",
+        expiresAt: new Date(Date.now() - 60_000),
+      });
+    const store = providerTraceStore(db);
+    await expect(store.cleanupExpired()).rejects.toThrow();
+    expect(await store.nextRetentionDeadline()).toBeLessThanOrEqual(Date.now());
+    await fs.rmdir(file);
+    await fs.writeFile(file, "{}\n");
+    await store.cleanupExpired();
+    expect(await store.nextRetentionDeadline()).toBeNull();
+  });
+  it.skipIf(!endpoint)(
+    "shares capture ownership across routes and workers and releases deleted captures",
+    async () => {
+      external();
+      const company = randomUUID(),
+        run = randomUUID(),
+        baseline = idleWorkSnapshot().active;
+      const heartbeat = providerTraceStore(db),
+        route = providerTraceStore(db);
+      owners.push({ store: heartbeat, run, company });
+      await heartbeat.prepare({
+        companyId: company,
+        runId: run,
+        provider: "codex",
+        requestedBy: "test",
+        assertWriterAuthority: async () => undefined,
+      });
+      expect(idleWorkSnapshot().active).toBe(baseline + 1);
+      const other = randomUUID();
+      await db.insert(providerTraceRecords).values({
+        companyId: company,
+        runId: other,
+        status: "complete",
+        provider: "codex",
+        traceRef: `s3-segments-v1:${company}/${other}/${randomUUID()}.ndjson`,
+        requestedBy: "test",
+        expiresAt: new Date(Date.now() + 3600_000),
+      });
+      await expect(route.remove(run, company)).rejects.toThrow("cleanup_pending");
+      expect(await route.nextRetentionDeadline()).toBeLessThanOrEqual(Date.now());
+      await route.remove(other, company);
+      expect(idleWorkSnapshot().active).toBe(baseline);
+      expect(await heartbeat.finalize(run, company)).toBeNull();
+      vi.useFakeTimers({ toFake: ["Date"] });
+      vi.setSystemTime(Date.now() + 31_000);
+      await route.remove(run, company);
+      expect(await route.nextRetentionDeadline()).toBeNull();
+      expect(await fs.readdir(directory)).toEqual([]);
+    },
+  );
+  it.skipIf(!endpoint)(
+    "keeps an interrupted import readable locally and fences its staged remote prefix on deletion",
+    async () => {
+      external();
+      const companyId = randomUUID(),
+        runId = randomUUID(),
+        filename = `${randomUUID()}.ndjson`;
+      const traceRef = `s3-import-v1:${companyId}/${runId}/${filename}`;
+      const bytes = Buffer.from('{"kind":"frame","rawBase64":"e30="}\n');
+      await fs.writeFile(path.join(directory, filename), bytes);
+      await db.insert(providerTraceRecords).values({
+        companyId,
+        runId,
+        traceRef,
+        status: "complete",
+        provider: "codex",
+        requestedBy: "test",
+        expiresAt: new Date(Date.now() + 3600_000),
+      });
+      const store = providerTraceStore(db);
+      expect((await store.download(runId, companyId))!.bytes).toEqual(bytes);
+      await store.remove(runId, companyId);
+      expect(await store.download(runId, companyId)).toBeNull();
+      expect(await store.nextRetentionDeadline()).toBeNull();
+      expect(await fs.readdir(directory)).toEqual([]);
+      const objects = await client!.send(
+        new ListObjectsV2Command({
+          Bucket: bucket,
+          Prefix: `provider-traces/segments-v1/${companyId}/${runId}/${filename}/`,
+        }),
+      );
+      expect(objects.Contents).toHaveLength(2);
+      expect(objects.Contents!.every((object) => object.Key!.endsWith("head.json"))).toBe(true);
+    },
+  );
+  it.skipIf(!endpoint)(
+    "does not resurrect a trace deleted before its first journal claim",
+    async () => {
+      external();
+      const company = randomUUID(),
+        run = randomUUID(),
+        baseline = idleWorkSnapshot().active;
+      const heartbeat = providerTraceStore(db),
+        route = providerTraceStore(db);
+      owners.push({ store: heartbeat, run, company });
+      let enter!: () => void, release!: () => void;
+      const entered = new Promise<void>((resolve) => {
+        enter = resolve;
+      });
+      const blocked = new Promise<void>((resolve) => {
+        release = resolve;
+      });
+      const preparing = heartbeat.prepare({
+        companyId: company,
+        runId: run,
+        provider: "codex",
+        requestedBy: "test",
+        assertWriterAuthority: async () => {
+          enter();
+          await blocked;
+        },
+      });
+      await entered;
+      await route.remove(run, company);
+      release();
+      await expect(preparing).rejects.toThrow();
+      expect(idleWorkSnapshot().active).toBe(baseline);
+      expect(await heartbeat.finalize(run, company)).toBeNull();
+      expect(await route.download(run, company)).toBeNull();
+      const objects = await client!.send(
+        new ListObjectsV2Command({
+          Bucket: bucket,
+          Prefix: `provider-traces/segments-v1/${company}/${run}/`,
+        }),
+      );
+      expect(objects.Contents).toHaveLength(2);
+      for (const object of objects.Contents!) expect(object.Key).toMatch(/head\.json$/);
+    },
+  );
+});

server/src/services/provider-trace-store.ts

@@ -1,7 +1,7 @@
 import fs from "node:fs/promises";
 import path from "node:path";
 import { createHash, randomUUID } from "node:crypto";
-import { and, eq, gt, inArray, isNull, lte, ne, or } from "drizzle-orm";
+import { and, asc, eq, gt, inArray, isNull, lte, ne, or, sql } from "drizzle-orm";
 import type { Db } from "@paperclipai/db";
 import { providerTraceRecords } from "@paperclipai/db";
 import type {
@@ -9,6 +9,10 @@ import type {
   ProviderTraceMetadata,
 } from "@paperclipai/shared";
 import { resolvePaperclipInstanceRoot } from "../home-paths.js";
+import { createProviderTraceObjectStore, objectProviderTraceRef, importingProviderTraceRef, providerTraceCleanupRef, type ProviderTraceObjectStore } from "./provider-trace-object-store.js";
+import { resolveRunEvidenceStorage } from "./run-evidence-storage.js";
+import { loadStorageConfig } from "../storage/config.js";
+import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
 import { logActivity } from "./activity-log.js";
 
 export const PROVIDER_TRACE_MAX_BYTES = 64 * 1024 * 1024;
@@ -31,6 +35,8 @@ function traceRoot() {
 }
 
 function tracePath(traceRef: string) {
+  const external = objectProviderTraceRef(traceRef) ?? importingProviderTraceRef(traceRef);
+  if (external) traceRef = external[3]!;
   if (
     path.basename(traceRef) !== traceRef ||
     !/^[a-f0-9-]+\.ndjson$/.test(traceRef)
@@ -150,24 +156,6 @@ async function readTraceBytes(traceRef: string) {
   return Buffer.concat([nativeBytes, separator, rehydrationBytes]);
 }
 
-async function readEntries(traceRef: string) {
-  const content = (await readTraceBytes(traceRef)).toString("utf8");
-  return content.split("\n").flatMap((line) => {
-    if (!line.trim()) return [];
-    try {
-      return [record(JSON.parse(line))];
-    } catch {
-      return [
-        {
-          kind: "trace_status",
-          status: "incomplete",
-          reason: "invalid_sidecar_record",
-        },
-      ];
-    }
-  });
-}
-
 export function assessProviderTraceEntries(
   entries: Record<string, unknown>[],
   options: {
@@ -266,8 +254,34 @@ export function assessProviderTraceEntries(
   return { status: "complete", reason: null };
 }
 
-export function providerTraceStore(db: Db) {
-  async function removeTraceFiles(traceRef: string) {
+const traceObjectStores = new WeakMap<Db, ProviderTraceObjectStore>();
+
+export function providerTraceStore(db: Db, options: { objectStore?: ProviderTraceObjectStore } = {}) {
+  let external = options.objectStore;
+  function objectStore() {
+    if (external) return external;
+    const shared = traceObjectStores.get(db);
+    if (shared) return external = shared;
+    const config = resolveRunEvidenceStorage();
+    if (!config) throw new Error("External provider traces require instance S3 storage");
+    external = createProviderTraceObjectStore(config.provider, config.tracePrefix);
+    traceObjectStores.set(db, external);
+    return external;
+  }
+  async function readBytes(traceRef: string) {
+    return objectProviderTraceRef(traceRef) ? objectStore().read(traceRef) : readTraceBytes(traceRef);
+  }
+  async function entriesFor(traceRef: string) {
+    return (await readBytes(traceRef)).toString("utf8").split("\n").flatMap(line => {
+      if (!line.trim()) return [];
+      try { return [record(JSON.parse(line))]; }
+      catch { return [{ kind: "trace_status", status: "incomplete", reason: "invalid_sidecar_record" }]; }
+    });
+  }
+  async function removeTraceFiles(traceRef: string, signal?: AbortSignal) {
+    signal?.throwIfAborted();
+    const cleanupRef = providerTraceCleanupRef(traceRef);
+    if (cleanupRef) await objectStore().remove(cleanupRef, signal);
     for (const filePath of [tracePath(traceRef), rehydrationTracePath(traceRef)]) {
       try {
         await fs.unlink(filePath);
@@ -277,7 +291,7 @@ export function providerTraceStore(db: Db) {
     }
   }
 
-  async function expireRow(row: TraceRow, now: Date) {
+  async function expireRow(row: TraceRow, now: Date, signal?: AbortSignal) {
     let expired = row;
     let newlyExpired = false;
     if (!row.deletedAt) {
@@ -300,7 +314,11 @@ export function providerTraceStore(db: Db) {
       expired = updated;
       newlyExpired = true;
     } else if (row.expiresAt.getTime() > now.getTime()) {
-      return null;
+      if (row.status !== "deleted" || !providerTraceCleanupRef(row.traceRef)) return null;
+      await removeTraceFiles(row.traceRef, signal);
+      const [cleaned] = await db.update(providerTraceRecords).set({ traceRef: EXPIRED_PROVIDER_TRACE_REF, updatedAt: now })
+        .where(and(eq(providerTraceRecords.id, row.id), eq(providerTraceRecords.traceRef, row.traceRef))).returning();
+      return cleaned ?? row;
     }
 
     // The database tombstone is authoritative before filesystem cleanup, so
@@ -308,7 +326,7 @@ export function providerTraceStore(db: Db) {
     // marker only after both raw sidecars are absent; failures are retried by
     // later reads and cleanup passes.
     if (expired.traceRef !== EXPIRED_PROVIDER_TRACE_REF) {
-      await removeTraceFiles(expired.traceRef);
+      await removeTraceFiles(expired.traceRef, signal);
       await db
         .update(providerTraceRecords)
         .set({
@@ -342,24 +360,27 @@ export function providerTraceStore(db: Db) {
     return expired;
   }
 
-  async function cleanupExpired(now = new Date()) {
+  async function cleanupExpired(now = new Date(), signal?: AbortSignal) {
     const expired = await db
       .select()
       .from(providerTraceRecords)
       .where(
         and(
-          lte(providerTraceRecords.expiresAt, now),
-          or(
-            isNull(providerTraceRecords.deletedAt),
-            ne(providerTraceRecords.traceRef, EXPIRED_PROVIDER_TRACE_REF),
-          ),
+          or(lte(providerTraceRecords.expiresAt, now), and(eq(providerTraceRecords.status, "deleted"),
+            or(sql`${providerTraceRecords.traceRef} like 's3-segments-v1:%'`, sql`${providerTraceRecords.traceRef} like 's3-import-v1:%'`))),
+          or(isNull(providerTraceRecords.deletedAt), ne(providerTraceRecords.traceRef, EXPIRED_PROVIDER_TRACE_REF)),
         ),
-      );
+      ).orderBy(asc(providerTraceRecords.expiresAt)).limit(100);
     const removed: TraceRow[] = [];
+    let failure: unknown;
     for (const row of expired) {
-      const removedRow = await expireRow(row, now);
-      if (removedRow) removed.push(removedRow);
+      try {
+        signal?.throwIfAborted();
+        const removedRow = await expireRow(row, now, signal);
+        if (removedRow) removed.push(removedRow);
+      } catch (error) { failure ??= error; }
     }
+    if (failure) throw failure;
     return removed;
   }
 
@@ -368,8 +389,8 @@ export function providerTraceStore(db: Db) {
     companyId: string;
     provider: string;
     requestedBy: string;
+    assertWriterAuthority?: () => Promise<void>;
   }) {
-    await cleanupExpired().catch(() => []);
     const existing = await db
       .select()
       .from(providerTraceRecords)
@@ -377,39 +398,47 @@ export function providerTraceStore(db: Db) {
       .limit(1)
       .then((rows) => rows[0] ?? null);
     if (existing) {
+      if (importingProviderTraceRef(existing.traceRef)) throw new Error("provider_trace_import_in_progress");
       const now = new Date();
       const expired = existing.expiresAt.getTime() <= now.getTime();
       if (expired) await expireRow(existing, now);
       if (existing.deletedAt || expired) {
         throw new Error("provider_trace_unavailable");
       }
-      return {
-        metadata: asMetadata(existing),
-        path: tracePath(existing.traceRef),
-      };
+      if (objectProviderTraceRef(existing.traceRef)) {
+        if (!input.assertWriterAuthority) throw new Error("External trace capture requires admitted writer authority");
+        await fs.mkdir(traceRoot(), { recursive: true, mode: 0o700 });
+        await objectStore().open(existing.traceRef, tracePath(existing.traceRef), input.assertWriterAuthority);
+      }
+      return { metadata: asMetadata(existing), path: tracePath(existing.traceRef) };
     }
 
+    const selected = loadStorageConfig().storageProvider === "s3" ? "s3_segments" : "local_file";
+    if (selected === "s3_segments") {
+      if (!input.assertWriterAuthority) throw new Error("External trace capture requires admitted writer authority");
+      objectStore();
+    }
     await fs.mkdir(traceRoot(), { recursive: true, mode: 0o700 });
-    const traceRef = `${randomUUID()}.ndjson`;
+    const filename = `${randomUUID()}.ndjson`;
+    const traceRef = selected === "s3_segments" ? `s3-segments-v1:${input.companyId}/${input.runId}/${filename}` : filename;
     const filePath = tracePath(traceRef);
     const handle = await fs.open(filePath, "wx", 0o600);
     await handle.close();
     await fs.chmod(filePath, 0o600);
     const expiresAt = new Date(Date.now() + PROVIDER_TRACE_RETENTION_MS);
-    const row = await db
-      .insert(providerTraceRecords)
-      .values({
-        runId: input.runId,
-        companyId: input.companyId,
-        status: "capturing",
-        provider: input.provider,
-        traceRef,
-        requestedBy: input.requestedBy,
-        expiresAt,
-      })
-      .returning()
-      .then((rows) => rows[0]);
+    const values = {
+      runId: input.runId, companyId: input.companyId, status: "capturing", provider: input.provider,
+      traceRef, requestedBy: input.requestedBy, expiresAt,
+    };
+    const row = await db.transaction(async tx => {
+          await notifyDeliveryWork(tx, DELIVERY_QUEUES.providerTraceRetention);
+          return (await tx.insert(providerTraceRecords).values(values).returning())[0];
+        });
     if (!row) throw new Error("provider_trace_metadata_create_failed");
+    if (objectProviderTraceRef(traceRef)) {
+      if (!input.assertWriterAuthority) throw new Error("External trace capture requires admitted writer authority");
+      await objectStore().open(traceRef, filePath, input.assertWriterAuthority);
+    }
     return { metadata: asMetadata(row), path: filePath };
   }
 
@@ -462,7 +491,11 @@ export function providerTraceStore(db: Db) {
     return rows.map(asMetadata);
   }
 
-  async function finalize(
+  async function finalize(runId: string, companyId: string, failureReason?: string | null) {
+    try { return await finalizeCapture(runId, companyId, failureReason); }
+    finally { await external?.stopRun(companyId, runId); }
+  }
+  async function finalizeCapture(
     runId: string,
     companyId: string,
     failureReason?: string | null,
@@ -471,14 +504,18 @@ export function providerTraceStore(db: Db) {
     if (!row) return null;
     let bytes: Buffer;
     try {
-      bytes = await readTraceBytes(row.traceRef);
+      if (objectProviderTraceRef(row.traceRef)) {
+        await objectStore().finish(row.traceRef);
+        if (!await objectStore().settled(row.traceRef)) throw new Error("trace_capture_unsettled");
+      }
+      bytes = await readBytes(row.traceRef);
     } catch {
       const updatedAt = new Date();
       const finalized = await db
         .update(providerTraceRecords)
         .set({
           status: "incomplete",
-          reason: failureReason ?? "trace_sidecar_missing",
+          reason: failureReason ?? (objectProviderTraceRef(row.traceRef) ? "trace_storage_incomplete" : "trace_sidecar_missing"),
           updatedAt,
         })
         .where(and(
@@ -538,7 +575,7 @@ export function providerTraceStore(db: Db) {
   async function inspect(runId: string, companyId: string) {
     const row = await getReadableByRun(runId, companyId);
     if (!row) return { trace: null, entries: [] };
-    const entries = await readEntries(row.traceRef).catch(() => []);
+    const entries = await entriesFor(row.traceRef).catch(() => []);
     return {
       trace: asMetadata(row),
       entries: entries.map((entry) =>
@@ -550,7 +587,7 @@ export function providerTraceStore(db: Db) {
   async function readExactEntries(runId: string, companyId: string) {
     const row = await getReadableByRun(runId, companyId);
     if (!row) return null;
-    return readEntries(row.traceRef);
+    return entriesFor(row.traceRef);
   }
 
   async function revealFrame(
@@ -560,7 +597,7 @@ export function providerTraceStore(db: Db) {
   ) {
     const row = await getReadableByRun(runId, companyId);
     if (!row) return null;
-    const entry = (await readEntries(row.traceRef)).find(
+    const entry = (await entriesFor(row.traceRef)).find(
       (candidate) =>
         candidate.kind === "frame" && Number(candidate.frameId) === frameId,
     );
@@ -572,26 +609,38 @@ export function providerTraceStore(db: Db) {
   async function download(runId: string, companyId: string) {
     const row = await getReadableByRun(runId, companyId);
     if (!row) return null;
-    return { row, bytes: await readTraceBytes(row.traceRef) };
+    return { row, bytes: await readBytes(row.traceRef) };
   }
 
   async function remove(runId: string, companyId: string) {
     const row = await getByRun(runId, companyId);
-    if (!row || row.deletedAt) return null;
-    // Manual deletion is filesystem-first and strict: a reported success must
-    // never leave raw sidecars orphaned behind an inaccessible database row.
+    if (!row) return null;
+    if (providerTraceCleanupRef(row.traceRef)) {
+      const now = new Date();
+      await db.transaction(async tx => {
+        await notifyDeliveryWork(tx, DELIVERY_QUEUES.providerTraceRetention);
+        await tx.update(providerTraceRecords).set({ status: "deleted", deletedAt: row.deletedAt ?? now, updatedAt: now })
+          .where(and(eq(providerTraceRecords.id, row.id), eq(providerTraceRecords.traceRef, row.traceRef)));
+      });
+
+      await removeTraceFiles(row.traceRef);
+      return (await db.update(providerTraceRecords).set({ traceRef: EXPIRED_PROVIDER_TRACE_REF, updatedAt: new Date() })
+        .where(and(eq(providerTraceRecords.id, row.id), eq(providerTraceRecords.traceRef, row.traceRef))).returning())[0] ?? null;
+    }
+    if (row.deletedAt) return null;
     await removeTraceFiles(row.traceRef);
     const now = new Date();
-    return db
-      .update(providerTraceRecords)
-      .set({
-        status: "deleted",
-        deletedAt: now,
-        updatedAt: now,
-      })
-      .where(eq(providerTraceRecords.id, row.id))
-      .returning()
-      .then((rows) => rows[0] ?? null);
+    return db.update(providerTraceRecords).set({ status: "deleted", deletedAt: now, updatedAt: now })
+      .where(eq(providerTraceRecords.id, row.id)).returning().then(rows => rows[0] ?? null);
+  }
+
+  async function nextRetentionDeadline() {
+    const [row] = await db.select({ at: providerTraceRecords.expiresAt, deletedAt: providerTraceRecords.deletedAt })
+      .from(providerTraceRecords).where(and(ne(providerTraceRecords.traceRef, EXPIRED_PROVIDER_TRACE_REF),
+        or(isNull(providerTraceRecords.deletedAt), lte(providerTraceRecords.expiresAt, new Date()),
+          sql`${providerTraceRecords.traceRef} like 's3-segments-v1:%'`, sql`${providerTraceRecords.traceRef} like 's3-import-v1:%'`)))
+      .orderBy(asc(sql`case when ${providerTraceRecords.deletedAt} is not null then 0 else 1 end`), asc(providerTraceRecords.expiresAt)).limit(1);
+    return row ? row.deletedAt ? Date.now() : row.at.getTime() : null;
   }
 
   return {
@@ -605,5 +654,6 @@ export function providerTraceStore(db: Db) {
     getByRun,
     listMetadataForRuns,
     cleanupExpired,
+    nextRetentionDeadline,
   };
 }

server/src/services/run-evidence-migration-work.integration.test.ts

@@ -0,0 +1,147 @@
+import { randomUUID, createHash } from "node:crypto";
+import { promises as fs } from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { sql } from "drizzle-orm";
+import { createDb, type Db } from "@paperclipai/db";
+import {
+  S3Client,
+  CreateBucketCommand,
+  ListObjectsV2Command,
+  DeleteObjectsCommand,
+  DeleteBucketCommand,
+} from "@aws-sdk/client-s3";
+import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
+import { registerRunEvidenceMigrationWork } from "./run-evidence-migration-work.js";
+import { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
+import { createObjectRunLogStore } from "./object-run-log-store.js";
+import { createProviderTraceObjectStore } from "./provider-trace-object-store.js";
+import { createS3StorageProvider } from "../storage/s3-provider.js";
+import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
+
+const endpoint = process.env.PAPERCLIP_LOG_TEST_S3_ENDPOINT;
+const databaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_RUN_EVIDENCE_MIGRATION_DATABASE_URL");
+describe.skipIf(!endpoint || !databaseUrl)(
+  "automatic retained history upgrade (Postgres and S3)",
+  () => {
+    let db: Db, client: S3Client, bucket: string, directory: string;
+    const schema = `automatic_migration_${randomUUID().replaceAll("-", "")}`;
+    beforeAll(async () => {
+      db = createDb(databaseUrl!, { maxConnections: 1 });
+      await db.execute(sql.raw(`create schema ${schema}`));
+      await db.execute(sql.raw(`set search_path=${schema}`));
+      await db.execute(sql`create table heartbeat_runs (id uuid primary key, company_id uuid not null, agent_id uuid not null,
+      status text not null, runtime_mode text default 'legacy', log_store text, log_ref text, log_bytes bigint, log_sha256 text, last_output_seq int not null default 0, controller_lease_expires_at timestamptz)`);
+      await db.execute(
+        sql`create table native_run_finalizations(run_id uuid, company_id uuid, lease_expires_at timestamptz)`,
+      );
+      await db.execute(sql`create table heartbeat_run_events(company_id uuid,run_id uuid,seq bigint,payload jsonb,
+      source_instance_id text,source_event_id text,source_seq bigint,source_payload_sha256 text,event_type text)`);
+      await db.execute(sql`create table provider_trace_records (id uuid primary key, company_id uuid not null, run_id uuid not null,
+      status text not null, provider text not null, trace_ref text not null, frame_count int not null default 0,
+      byte_count bigint not null default 0, digest text, reason text, requested_by text not null, expires_at timestamptz not null,
+      deleted_at timestamptz, created_at timestamptz not null default now(), updated_at timestamptz not null default now())`);
+      directory = await fs.mkdtemp(path.join(os.tmpdir(), "evidence-migration-"));
+      vi.stubEnv("PAPERCLIP_CONFIG", path.join(directory, "config.json"));
+      vi.stubEnv("RUN_LOG_BASE_PATH", path.join(directory, "logs"));
+      vi.stubEnv("PROVIDER_TRACE_BASE_PATH", path.join(directory, "traces"));
+      vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "b".repeat(64));
+      vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "s3");
+      vi.stubEnv("PAPERCLIP_STORAGE_S3_ENDPOINT", endpoint!);
+      vi.stubEnv("PAPERCLIP_STORAGE_S3_REGION", "us-east-1");
+      vi.stubEnv("PAPERCLIP_STORAGE_S3_PREFIX", "");
+      vi.stubEnv("PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE", "true");
+      bucket = `pc-auto-migration-${randomUUID()}`;
+      vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", bucket);
+      client = new S3Client({
+        endpoint,
+        region: "us-east-1",
+        forcePathStyle: true,
+      });
+      await client.send(new CreateBucketCommand({ Bucket: bucket }));
+    }, 30_000);
+    afterAll(async () => {
+      if (client) {
+        for (;;) {
+          const page = await client.send(new ListObjectsV2Command({ Bucket: bucket }));
+          if (!page.Contents?.length) break;
+          await client.send(
+            new DeleteObjectsCommand({
+              Bucket: bucket,
+              Delete: { Objects: page.Contents.map((o) => ({ Key: o.Key! })) },
+            }),
+          );
+        }
+        await client.send(new DeleteBucketCommand({ Bucket: bucket }));
+        client.destroy();
+      }
+      if (db) {
+        await db.execute(sql.raw(`drop schema ${schema} cascade`));
+        await db.$client.end();
+      }
+      if (directory) await fs.rm(directory, { recursive: true, force: true });
+      vi.unstubAllEnvs();
+    });
+    it("copies retained logs and a trace automatically, then reads them without their source directories", async () => {
+      const companyId = randomUUID(),
+        agentId = randomUUID(),
+        runId = randomUUID(),
+        traceId = randomUUID();
+      const logRef = `${companyId}/${agentId}/${runId}.ndjson`;
+      const filename = `${randomUUID()}.ndjson`;
+      const log = Buffer.from('{"stream":"stdout","chunk":"retained 再開","seq":1}\n');
+      const trace = Buffer.from('{"kind":"frame","rawBase64":"e30="}\n');
+      await fs.mkdir(path.dirname(path.join(directory, "logs", logRef)), { recursive: true });
+      await fs.writeFile(path.join(directory, "logs", logRef), log);
+      await fs.mkdir(path.join(directory, "traces"), { recursive: true });
+      await fs.writeFile(path.join(directory, "traces", filename), trace);
+      const expiry = new Date(Date.now() + 3600_000).toISOString();
+      await db.execute(sql`insert into heartbeat_runs(id,company_id,agent_id,status,log_store,log_ref,log_bytes,log_sha256)
+      values(${runId},${companyId},${agentId},'succeeded','local_file',${logRef},${log.length},${createHash("sha256").update(log).digest("hex")})`);
+      await db.execute(sql`insert into provider_trace_records(id,company_id,run_id,status,provider,trace_ref,requested_by,expires_at,digest)
+      values(${traceId},${companyId},${runId},'complete','codex',${filename},'test',${expiry},${`sha256:${createHash("sha256").update(trace).digest("hex")}`})`);
+      const onError = vi.fn();
+      const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError });
+      const worker = registerRunEvidenceMigrationWork(db, coordinator, () => true)!;
+      try {
+        await worker.ready;
+        await vi.waitFor(() => expect(coordinator.nextWakeAt()).toBeNull(), { timeout: 5_000 });
+        expect(onError).not.toHaveBeenCalled();
+        await fs.rm(path.join(directory, "logs"), { recursive: true });
+        await fs.rm(path.join(directory, "traces"), { recursive: true });
+        const [run] = await db.$client.unsafe(
+          "select log_store,log_ref from heartbeat_runs where id=$1",
+          [runId],
+        );
+        const [saved] = await db.$client.unsafe(
+          "select trace_ref,expires_at from provider_trace_records where id=$1",
+          [traceId],
+        );
+        expect(run!.log_store).toBe("s3_segments");
+        expect(new Date(saved!.expires_at).toISOString()).toBe(expiry);
+        const provider = createS3StorageProvider({
+          endpoint,
+          bucket,
+          region: "us-east-1",
+          forcePathStyle: true,
+        });
+        const logs = createObjectRunLogStore({
+          provider,
+          keyPrefix: "run-logs",
+          assertWriterAuthority: async () => {
+            throw Error("read-only");
+          },
+        });
+        expect(
+          Buffer.from((await logs.read({ store: run!.log_store, logRef: run!.log_ref })).content),
+        ).toEqual(log);
+        expect(await createProviderTraceObjectStore(provider).read(saved!.trace_ref)).toEqual(
+          trace,
+        );
+      } finally {
+        await worker.stop();
+        await coordinator.stop();
+      }
+    });
+  },
+);

server/src/services/run-evidence-migration-work.ts

@@ -0,0 +1,84 @@
+import type { Db } from "@paperclipai/db";
+import { loadStorageConfig, type StorageConfig } from "../storage/config.js";
+import { logger } from "../middleware/logger.js";
+import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
+import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
+import { registerDeadlineWork } from "./deadline-work.js";
+import { registerIdleLocalWorkProbe, type IdleLocalWork } from "./idle-local-work.js";
+import { migrateRunEvidence } from "./run-evidence-migration.js";
+
+export function registerRunEvidenceMigrationWork(
+  db: Db,
+  coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
+  canRun: () => boolean,
+  storage: Pick<StorageConfig, "storageProvider"> = loadStorageConfig(),
+) {
+  if (storage.storageProvider !== "s3") return null;
+  let runCursor: string | undefined, traceCursor: string | undefined;
+  let logsDone = false,
+    tracesDone = false,
+    blocked = 0,
+    migrated = 0;
+  let retryMs = 5 * 60_000;
+  let next: number | null = Date.now();
+  let pending: IdleLocalWork = "unknown";
+  const unregister = registerIdleLocalWorkProbe(() => pending);
+  const worker = registerDeadlineWork(coordinator, DELIVERY_QUEUES.runEvidenceMigration, {
+    canRun,
+    retryMs: 30_000,
+    continuationMs: 1_000,
+    async run(signal) {
+      if (next === null) return;
+      const result = await migrateRunEvidence(db, {
+        apply: true,
+        limit: 1,
+        afterRunId: runCursor,
+        afterTraceId: traceCursor,
+        kinds: [...(logsDone ? [] : ["log" as const]), ...(tracesDone ? [] : ["trace" as const])],
+        signal,
+      });
+      signal.throwIfAborted();
+      for (const item of result.items) {
+        if (item.state === "blocked") {
+          blocked++;
+          logger.warn(
+            { kind: item.kind, id: item.id, reason: item.reason },
+            "Run evidence migration deferred; source storage is still required",
+          );
+        } else if (item.state === "migrated") migrated++;
+      }
+      if (!logsDone) {
+        runCursor = result.nextRunId;
+        logsDone = runCursor === undefined;
+      }
+      if (!tracesDone) {
+        traceCursor = result.nextTraceId;
+        tracesDone = traceCursor === undefined;
+      }
+      pending = "present";
+      next = Date.now() + 1_000;
+      if (logsDone && tracesDone) {
+        logger.info(
+          { migrated, blocked },
+          blocked
+            ? "Run evidence migration will retry unresolved history; keep source storage"
+            : "Run evidence migration complete",
+        );
+        next = blocked ? Date.now() + retryMs : null;
+        pending = blocked ? "present" : "none";
+        retryMs = Math.min(retryMs * 2, 30 * 60_000);
+        runCursor = traceCursor = undefined;
+        logsDone = tracesDone = false;
+        blocked = migrated = 0;
+      }
+    },
+    next: async () => next,
+  });
+  return {
+    ...worker,
+    stop: async () => {
+      await worker.stop();
+      unregister();
+    },
+  };
+}

server/src/services/run-evidence-migration.integration.test.ts

@@ -0,0 +1,349 @@
+import { randomUUID, createHash } from "node:crypto";
+import { promises as fs } from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { sql } from "drizzle-orm";
+import { createDb, type Db } from "@paperclipai/db";
+import {
+  S3Client,
+  CreateBucketCommand,
+  ListObjectsV2Command,
+  DeleteObjectsCommand,
+  DeleteBucketCommand,
+} from "@aws-sdk/client-s3";
+import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
+import { migrateRunEvidence } from "./run-evidence-migration.js";
+import { createS3StorageProvider } from "../storage/s3-provider.js";
+import { createObjectRunLogStore } from "./object-run-log-store.js";
+import { createProviderTraceObjectStore } from "./provider-trace-object-store.js";
+import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
+import { formatNativeRunLogEvent } from "./native-runtime/native-run-log-export.js";
+import { nativeSha256 } from "./native-runtime/canonical.js";
+import { createObjectByteJournal } from "./object-byte-journal.js";
+import type { PrpEvent } from "../vendor/paperclip-runner/index.js";
+import { compactRunLogChunk } from "./heartbeat/run-log.js";
+
+const endpoint = process.env.PAPERCLIP_LOG_TEST_S3_ENDPOINT;
+const databaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_RUN_EVIDENCE_MIGRATION_DATABASE_URL");
+describe.skipIf(!endpoint || !databaseUrl)("run evidence migration (Postgres and S3)", () => {
+  let db: Db, client: S3Client, bucket: string, directory: string;
+  const schema = `migration_${randomUUID().replaceAll("-", "")}`;
+  beforeAll(async () => {
+    db = createDb(databaseUrl!, { maxConnections: 1 });
+    await db.execute(sql.raw(`create schema ${schema}`));
+    await db.execute(sql.raw(`set search_path=${schema}`));
+    await db.execute(sql`create table heartbeat_runs (id uuid primary key, company_id uuid not null, agent_id uuid not null,
+      status text not null, runtime_mode text default 'legacy', log_store text, log_ref text, log_bytes bigint, log_sha256 text, last_output_seq int not null default 0, controller_lease_expires_at timestamptz)`);
+    await db.execute(
+      sql`create table native_run_finalizations(run_id uuid, company_id uuid, lease_expires_at timestamptz)`,
+    );
+    await db.execute(sql`create table heartbeat_run_events(company_id uuid,run_id uuid,seq bigint,payload jsonb,
+      source_instance_id text,source_event_id text,source_seq bigint,source_payload_sha256 text,event_type text)`);
+    await db.execute(sql`create table provider_trace_records (id uuid primary key, company_id uuid not null, run_id uuid not null,
+      status text not null, provider text not null, trace_ref text not null, frame_count int not null default 0,
+      byte_count bigint not null default 0, digest text, reason text, requested_by text not null, expires_at timestamptz not null,
+      deleted_at timestamptz, created_at timestamptz not null default now(), updated_at timestamptz not null default now())`);
+    directory = await fs.mkdtemp(path.join(os.tmpdir(), "evidence-migration-"));
+    vi.stubEnv("PAPERCLIP_CONFIG", path.join(directory, "config.json"));
+    vi.stubEnv("RUN_LOG_BASE_PATH", path.join(directory, "logs"));
+    vi.stubEnv("PROVIDER_TRACE_BASE_PATH", path.join(directory, "traces"));
+    vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "b".repeat(64));
+    vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "s3");
+    vi.stubEnv("PAPERCLIP_STORAGE_S3_ENDPOINT", endpoint!);
+    vi.stubEnv("PAPERCLIP_STORAGE_S3_REGION", "us-east-1");
+    vi.stubEnv("PAPERCLIP_STORAGE_S3_PREFIX", "");
+    vi.stubEnv("PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE", "true");
+    bucket = `paperclip-migration-test-${randomUUID()}`;
+    vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", bucket);
+    client = new S3Client({
+      endpoint,
+      region: "us-east-1",
+      forcePathStyle: true,
+    });
+    await client.send(new CreateBucketCommand({ Bucket: bucket }));
+  }, 30_000);
+  afterAll(async () => {
+    if (client) {
+      for (;;) {
+        const page = await client.send(new ListObjectsV2Command({ Bucket: bucket }));
+        if (!page.Contents?.length) break;
+        await client.send(
+          new DeleteObjectsCommand({
+            Bucket: bucket,
+            Delete: { Objects: page.Contents.map((o) => ({ Key: o.Key! })) },
+          }),
+        );
+      }
+      await client.send(new DeleteBucketCommand({ Bucket: bucket }));
+      client.destroy();
+    }
+    if (db) {
+      await db.execute(sql.raw(`drop schema ${schema} cascade`));
+      await db.$client.end();
+    }
+    if (directory) await fs.rm(directory, { recursive: true, force: true });
+    vi.unstubAllEnvs();
+  });
+  it("preserves exact historical bytes and expiry, while refusing active/missing/corrupt sources", async () => {
+    const company = randomUUID(),
+      agent = randomUUID(),
+      run = randomUUID(),
+      trace = randomUUID(),
+      filename = `${randomUUID()}.ndjson`;
+    const log = Buffer.from(
+      Array.from(
+        { length: 100 },
+        (_, i) =>
+          JSON.stringify({
+            ts: "then",
+            seq: i + 1,
+            stream: "stdout",
+            chunk: `再開 ${i}`,
+          }) + "\n",
+      ).join(""),
+    );
+    const logRef = `${company}/${agent}/${run}.ndjson`;
+    await fs.mkdir(path.dirname(path.join(directory, "logs", logRef)), {
+      recursive: true,
+    });
+    await fs.writeFile(path.join(directory, "logs", logRef), log);
+    await fs.mkdir(path.join(directory, "traces"), { recursive: true });
+    const native = Buffer.from('{"kind":"frame","rawBase64":"e30="}\n'.repeat(33000).trimEnd()),
+      rehydration = Buffer.from('{"kind":"interpretation","frameId":1}\n');
+    const expectedTrace = Buffer.concat([native, Buffer.from("\n"), rehydration]);
+    await fs.writeFile(path.join(directory, "traces", filename), native);
+    await fs.writeFile(path.join(directory, "traces", `${filename}.rehydration`), rehydration);
+    const expiry = new Date(Date.now() + 3600_000);
+    await db.execute(
+      sql`insert into heartbeat_runs(id, company_id, agent_id, status, log_store, log_ref, log_bytes, log_sha256, last_output_seq) values (${run}, ${company}, ${agent}, 'succeeded', 'local_file', ${logRef}, ${log.length}, ${createHash("sha256").update(log).digest("hex")}, 100)`,
+    );
+    await db.execute(sql`insert into provider_trace_records(id, company_id, run_id, status, provider, trace_ref, requested_by, expires_at, digest)
+      values (${trace}, ${company}, ${run}, 'complete', 'codex', ${filename}, 'test', ${expiry.toISOString()}, ${`sha256:${createHash(
+        "sha256",
+      )
+        .update(expectedTrace)
+        .digest("hex")}`})`);
+    const active = randomUUID(),
+      missing = randomUUID(),
+      corrupt = randomUUID();
+    await db.execute(sql`insert into heartbeat_runs(id, company_id, agent_id, status, log_store, log_ref, log_bytes, log_sha256)
+      values (${active}, ${company}, ${agent}, 'running', 'local_file', ${logRef}, ${log.length}, null),
+      (${missing}, ${company}, ${agent}, 'succeeded', 'local_file', 'missing.ndjson', 10, null),
+      (${corrupt}, ${company}, ${agent}, 'succeeded', 'local_file', ${logRef}, ${log.length + 1}, null)`);
+    const legacyLease = randomUUID(),
+      nativeLease = randomUUID(),
+      leaseExpiry = new Date(Date.now() + 60_000).toISOString();
+    await db.execute(sql`insert into heartbeat_runs(id, company_id, agent_id, status, log_store, log_ref, log_bytes, controller_lease_expires_at)
+      values (${legacyLease}, ${company}, ${agent}, 'succeeded', 'local_file', ${logRef}, ${log.length}, ${leaseExpiry}),
+      (${nativeLease}, ${company}, ${agent}, 'succeeded', 'local_file', ${logRef}, ${log.length}, null)`);
+    await db.execute(
+      sql`insert into native_run_finalizations values (${nativeLease}, ${company}, ${leaseExpiry})`,
+    );
+    const inventory = await migrateRunEvidence(db, {
+      apply: false,
+      companyId: company,
+    });
+    expect(inventory.items.find((i) => i.id === run)?.state).toBe("inventoried");
+    const result = await migrateRunEvidence(db, {
+      apply: true,
+      companyId: company,
+    });
+    expect(result.items.filter((i) => i.state === "migrated")).toHaveLength(2);
+    expect(result.items.filter((i) => i.state === "blocked")).toHaveLength(5);
+    const records = await db.$client.unsafe(
+      "select log_store, log_ref from heartbeat_runs where id=$1",
+      [run],
+    );
+    expect(records[0].log_store).toBe("s3_segments");
+    const provider = createS3StorageProvider({
+      endpoint,
+      bucket,
+      region: "us-east-1",
+      forcePathStyle: true,
+    });
+    const reader = createObjectRunLogStore({
+      provider,
+      keyPrefix: "run-logs",
+      assertWriterAuthority: async () => {
+        throw Error("read-only");
+      },
+    });
+    await fs.rm(path.join(directory, "logs"), { recursive: true });
+    await fs.rm(path.join(directory, "traces"), { recursive: true });
+    expect(
+      Buffer.from(
+        (
+          await reader.read({
+            store: "s3_segments",
+            logRef: records[0].log_ref,
+          })
+        ).content,
+      ),
+    ).toEqual(log);
+    const saved = await db.$client.unsafe(
+      "select trace_ref, expires_at, status from provider_trace_records where id=$1",
+      [trace],
+    );
+    expect(new Date(saved[0].expires_at).getTime()).toBe(expiry.getTime());
+    expect(saved[0].status).toBe("complete");
+    expect(await createProviderTraceObjectStore(provider).read(saved[0].trace_ref)).toEqual(
+      expectedTrace,
+    );
+  }, 30_000);
+  it("blocks a native archive with a missing committed tail and checkpoints only verified source coverage", async () => {
+    const companyId = randomUUID(),
+      agentId = randomUUID();
+    const runs: string[] = [];
+    for (const state of ["missing", "finalized", "covered"]) {
+      const complete = state !== "missing",
+        finalized = state === "finalized";
+      const runId = randomUUID(),
+        sourceInstanceId = randomUUID(),
+        normalizedSessionId = randomUUID();
+      runs.push(runId);
+      const events: PrpEvent[] = [1, 2].map((sourceSeq) => ({
+        schema: "paperclip.prp.event.v1",
+        schemaVersion: 1,
+        runId,
+        sourceInstanceId,
+        sourceSeq,
+        sourceEventId: `${sourceInstanceId}:${sourceSeq}`,
+        sourceKind: "runner",
+        normalizedSessionId,
+        priority: 2,
+        emittedAt: "2026-10-30T00:00:00Z",
+        eventType: "item.delta",
+        payload: { text: `part ${sourceSeq}` },
+      }));
+      const logRef = `${companyId}/${agentId}/${runId}.ndjson`;
+      const bytes = Buffer.from(
+        events
+          .slice(0, complete ? 2 : 1)
+          .map(
+            (event, index) =>
+              JSON.stringify({
+                ts: "then",
+                seq: index + 1,
+                stream: "stdout",
+                chunk: formatNativeRunLogEvent(event),
+              }) + "\n",
+          )
+          .join(""),
+      );
+      await fs.mkdir(path.dirname(path.join(directory, "logs", logRef)), { recursive: true });
+      await fs.writeFile(path.join(directory, "logs", logRef), bytes);
+      await db.execute(sql`insert into heartbeat_runs(id,company_id,agent_id,status,runtime_mode,log_store,log_ref,log_bytes,log_sha256)
+        values(${runId},${companyId},${agentId},'succeeded','native','local_file',${logRef},${finalized ? bytes.length : null},${finalized ? createHash("sha256").update(bytes).digest("hex") : null})`);
+      for (const [index, event] of events.entries())
+        await db.execute(sql`insert into heartbeat_run_events
+        values(${companyId},${runId},${index + 1},${JSON.stringify({ prpEvent: event })}::jsonb,${sourceInstanceId},${event.sourceEventId},${event.sourceSeq},${nativeSha256(event)},${event.eventType})`);
+    }
+    const migrated = await migrateRunEvidence(db, { apply: true, companyId });
+    expect(migrated.items.find((item) => item.id === runs[0])).toMatchObject({
+      state: "blocked",
+      reason: "native_log_source_coverage_unproven",
+    });
+    expect(migrated.items.find((item) => item.id === runs[1])?.state).toBe("migrated");
+    expect(migrated.items.find((item) => item.id === runs[2])?.state).toBe("migrated");
+    const provider = createS3StorageProvider({
+      endpoint,
+      bucket,
+      region: "us-east-1",
+      forcePathStyle: true,
+    });
+    for (const runId of runs.slice(1)) {
+      const journal = createObjectByteJournal({
+        provider,
+        prefix: `run-logs/segments-v1/${companyId}/${agentId}/${runId}`,
+      });
+      expect((await journal.head())!.head.sourceCursor).toBe(2);
+    }
+    const [blocked] = await db.$client.unsafe("select log_store from heartbeat_runs where id=$1", [
+      runs[0]!,
+    ]);
+    expect(blocked!.log_store).toBe("local_file");
+  });
+  it("migrates a digest-verified large native event while blocking unproven and corrupt files", async () => {
+    const companyId = randomUUID(),
+      agentId = randomUUID();
+    const cases = [
+      "finalized",
+      "missing-size",
+      "missing-digest",
+      "wrong-digest",
+      "wrong-size",
+    ] as const;
+    const runs = new Map<string, { runId: string; logRef: string; bytes: Buffer }>();
+    for (const state of cases) {
+      const runId = randomUUID(),
+        sourceInstanceId = randomUUID();
+      const event: PrpEvent = {
+        schema: "paperclip.prp.event.v1",
+        schemaVersion: 1,
+        runId,
+        sourceInstanceId,
+        sourceSeq: 1,
+        sourceEventId: `${sourceInstanceId}:1`,
+        sourceKind: "runner",
+        normalizedSessionId: randomUUID(),
+        priority: 2,
+        emittedAt: "2026-10-30T00:00:00Z",
+        eventType: "item.delta",
+        payload: { text: "large tool result ".repeat(12_000) },
+      };
+      const chunk = compactRunLogChunk(formatNativeRunLogEvent(event));
+      expect(chunk).toContain("paperclip truncated run log chunk");
+      expect(() => JSON.parse(chunk)).toThrow();
+      const bytes = Buffer.from(
+        `${JSON.stringify({ ts: "then", seq: 1, stream: "stdout", chunk })}\n`,
+      );
+      const logRef = `${companyId}/${agentId}/${runId}.ndjson`;
+      runs.set(state, { runId, logRef, bytes });
+      await fs.mkdir(path.dirname(path.join(directory, "logs", logRef)), { recursive: true });
+      await fs.writeFile(path.join(directory, "logs", logRef), bytes);
+      const digest = createHash("sha256").update(bytes).digest("hex");
+      await db.execute(sql`insert into heartbeat_runs(id,company_id,agent_id,status,runtime_mode,log_store,log_ref,log_bytes,log_sha256)
+        values(${runId},${companyId},${agentId},'succeeded','native','local_file',${logRef},
+        ${state === "missing-size" ? null : bytes.length + (state === "wrong-size" ? 1 : 0)},
+        ${state === "missing-digest" ? null : state === "wrong-digest" ? "0".repeat(64) : digest})`);
+      await db.execute(sql`insert into heartbeat_run_events values(${companyId},${runId},1,
+        ${JSON.stringify({ prpEvent: event })}::jsonb,${sourceInstanceId},${event.sourceEventId},1,${nativeSha256(event)},${event.eventType})`);
+    }
+    const result = await migrateRunEvidence(db, { apply: true, companyId });
+    const finalized = runs.get("finalized")!;
+    expect(result.items.find((item) => item.id === finalized.runId)?.state).toBe("migrated");
+    for (const state of cases.slice(1)) {
+      const run = runs.get(state)!;
+      expect(result.items.find((item) => item.id === run.runId)).toMatchObject({
+        state: "blocked",
+        reason: state.startsWith("wrong-")
+          ? "legacy_length_or_digest_mismatch"
+          : "native_log_source_coverage_unproven",
+      });
+      const [row] = await db.$client.unsafe("select log_store from heartbeat_runs where id=$1", [
+        run.runId,
+      ]);
+      expect(row!.log_store).toBe("local_file");
+    }
+    const provider = createS3StorageProvider({
+      endpoint,
+      bucket,
+      region: "us-east-1",
+      forcePathStyle: true,
+    });
+    const logRef = `${companyId}/${agentId}/${finalized.runId}`;
+    const journal = createObjectByteJournal({ provider, prefix: `run-logs/segments-v1/${logRef}` });
+    expect((await journal.head())!.head.sourceCursor).toBe(1);
+    await fs.unlink(path.join(directory, "logs", finalized.logRef));
+    const reader = createObjectRunLogStore({
+      provider,
+      keyPrefix: "run-logs",
+      assertWriterAuthority: async () => {
+        throw new Error("read-only");
+      },
+    });
+    expect(Buffer.from((await reader.read({ store: "s3_segments", logRef })).content)).toEqual(
+      finalized.bytes,
+    );
+  });
+});

server/src/services/run-evidence-migration.ts

@@ -0,0 +1,438 @@
+import { createHash } from "node:crypto";
+import { createReadStream, promises as fs } from "node:fs";
+import path from "node:path";
+import { and, asc, desc, eq, gt, inArray, isNull, lte, or, sql } from "drizzle-orm";
+import {
+  heartbeatRuns,
+  heartbeatRunEvents,
+  nativeRunFinalizations,
+  providerTraceRecords,
+  type Db,
+} from "@paperclipai/db";
+import { resolvePaperclipInstanceRoot } from "../home-paths.js";
+import { createObjectByteJournal } from "./object-byte-journal.js";
+import {
+  createProviderTraceObjectStore,
+  importingProviderTraceRef,
+} from "./provider-trace-object-store.js";
+import { openLegacyRunLogForMigration } from "./run-log-store.js";
+import { resolveRunEvidenceStorage } from "./run-evidence-storage.js";
+import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
+import {
+  createNativeRunLogCoverageVerifier,
+  nativeRunLogSourceWhere,
+} from "./native-runtime/native-run-log-export.js";
+
+const SETTLED = ["succeeded", "failed", "cancelled", "timed_out"];
+export interface RunEvidenceMigrationItem {
+  kind: "log" | "trace";
+  id: string;
+  state: "inventoried" | "migrated" | "blocked";
+  bytes?: number;
+  sha256?: string;
+  reason?: string;
+}
+
+export async function migrateRunEvidence(
+  db: Db,
+  options: {
+    apply: boolean;
+    companyId?: string;
+    afterRunId?: string;
+    afterTraceId?: string;
+    limit?: number;
+    kinds?: readonly ("log" | "trace")[];
+    signal?: AbortSignal;
+  },
+): Promise<{
+  items: RunEvidenceMigrationItem[];
+  nextRunId?: string;
+  nextTraceId?: string;
+}> {
+  const limit = options.limit ?? 50;
+  options.signal?.throwIfAborted();
+  if (!Number.isInteger(limit) || limit < 1 || limit > 100)
+    throw new Error("Migration limit must be between 1 and 100");
+  const config = resolveRunEvidenceStorage();
+  if (!config) throw new Error("Run evidence migration requires instance S3 storage");
+  function drainedWriter(now = new Date()) {
+    return and(
+      inArray(heartbeatRuns.status, SETTLED),
+      or(
+        isNull(heartbeatRuns.controllerLeaseExpiresAt),
+        lte(heartbeatRuns.controllerLeaseExpiresAt, now),
+      ),
+      sql`not exists (select 1 from ${nativeRunFinalizations} where ${and(
+        eq(nativeRunFinalizations.runId, heartbeatRuns.id),
+        eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId),
+        gt(nativeRunFinalizations.leaseExpiresAt, now),
+      )})`,
+    );
+  }
+  async function assertDrained(runId: string, companyId: string) {
+    const [run] = await db
+      .select({ id: heartbeatRuns.id })
+      .from(heartbeatRuns)
+      .where(
+        and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, companyId), drainedWriter()),
+      );
+    if (!run) throw new Error("legacy_writer_not_drained");
+  }
+  const items: RunEvidenceMigrationItem[] = [];
+  const logs =
+    options.kinds && !options.kinds.includes("log")
+      ? []
+      : await db
+          .select({
+            id: heartbeatRuns.id,
+            companyId: heartbeatRuns.companyId,
+            agentId: heartbeatRuns.agentId,
+            status: heartbeatRuns.status,
+            runtimeMode: heartbeatRuns.runtimeMode,
+            logRef: heartbeatRuns.logRef,
+            bytes: heartbeatRuns.logBytes,
+            sha256: heartbeatRuns.logSha256,
+            lastSequence: heartbeatRuns.lastOutputSeq,
+          })
+          .from(heartbeatRuns)
+          .where(
+            and(
+              eq(heartbeatRuns.logStore, "local_file"),
+              options.companyId ? eq(heartbeatRuns.companyId, options.companyId) : undefined,
+              options.afterRunId ? gt(heartbeatRuns.id, options.afterRunId) : undefined,
+            ),
+          )
+          .orderBy(asc(heartbeatRuns.id))
+          .limit(limit);
+  for (const run of logs) {
+    options.signal?.throwIfAborted();
+    const item: RunEvidenceMigrationItem = {
+      kind: "log",
+      id: run.id,
+      state: "inventoried",
+    };
+    items.push(item);
+    try {
+      if (!SETTLED.includes(run.status)) throw new Error("legacy_writer_not_settled");
+      if (!run.logRef) throw new Error("legacy_reference_missing");
+      await assertDrained(run.id, run.companyId);
+      const ref = `${run.companyId}/${run.agentId}/${run.id}`;
+      const journal = createObjectByteJournal({
+        provider: config.provider,
+        prefix: `${config.keyPrefix || "run-logs"}/segments-v1/${ref}`,
+      });
+      let writer = options.apply
+        ? await journal.claim({
+            expectedOwner: (await journal.head(options.signal))?.head.owner ?? null,
+            signal: options.signal,
+            assertAuthority: async () => {
+              const [current] = await db
+                .select({ id: heartbeatRuns.id })
+                .from(heartbeatRuns)
+                .where(
+                  and(
+                    eq(heartbeatRuns.id, run.id),
+                    eq(heartbeatRuns.companyId, run.companyId),
+                    eq(heartbeatRuns.logStore, "local_file"),
+                    eq(heartbeatRuns.logRef, run.logRef!),
+                    drainedWriter(),
+                  ),
+                );
+              if (!current) throw new Error("legacy_writer_changed");
+            },
+          })
+        : null;
+      const source = await openLegacyRunLogForMigration(run.logRef);
+      if (source.source === "mirror" && (run.bytes === null || !run.sha256)) {
+        source.stream.destroy();
+        throw new Error("mirror_completeness_unproven");
+      }
+      const hash = createHash("sha256");
+      const nativeCoverage =
+        run.runtimeMode === "native" && (run.bytes === null || run.sha256 === null)
+          ? createNativeRunLogCoverageVerifier(db, run.companyId, run.id)
+          : undefined;
+      let offset = 0,
+        tail = Buffer.alloc(0),
+        sequence = run.lastSequence;
+      try {
+        for await (const chunk of source.stream) {
+          options.signal?.throwIfAborted();
+          const bytes = Buffer.from(chunk);
+          hash.update(bytes);
+          await nativeCoverage?.write(bytes);
+          tail = Buffer.concat([tail, bytes]).subarray(-262144);
+          if (writer) {
+            const overlap = Math.min(bytes.length, Math.max(0, writer.head.end - offset));
+            if (
+              overlap &&
+              !(await journal.readRange(writer.head, offset, overlap, options.signal)).equals(
+                bytes.subarray(0, overlap),
+              )
+            )
+              throw new Error("migration_prefix_conflict");
+            if (overlap < bytes.length)
+              writer = await journal.append(writer, bytes.subarray(overlap), options.signal);
+          }
+          offset += bytes.length;
+        }
+      } finally {
+        source.stream.destroy();
+      }
+      const digest = hash.digest("hex");
+      if (
+        (source.bytes !== undefined && source.bytes !== offset) ||
+        (run.bytes !== null && run.bytes !== offset) ||
+        (run.sha256 && run.sha256 !== digest)
+      )
+        throw new Error("legacy_length_or_digest_mismatch");
+      for (const line of tail.toString("utf8").split("\n")) {
+        try {
+          const entry = JSON.parse(line);
+          if (Number.isSafeInteger(entry.seq)) sequence = Math.max(sequence, entry.seq);
+        } catch {}
+      }
+      item.bytes = offset;
+      item.sha256 = digest;
+      let sourceCursor = (await nativeCoverage?.finish()) ?? 0;
+      if (run.runtimeMode === "native" && !nativeCoverage) {
+        const [source] = await db
+          .select({ seq: heartbeatRunEvents.seq })
+          .from(heartbeatRunEvents)
+          .where(nativeRunLogSourceWhere(run.companyId, run.id))
+          .orderBy(desc(heartbeatRunEvents.seq))
+          .limit(1);
+        sourceCursor = source?.seq ?? 0;
+      }
+      if (writer) {
+        options.signal?.throwIfAborted();
+        if (writer.head.end !== offset) throw new Error("migration_prefix_exceeds_source");
+
+        const remoteHash = createHash("sha256");
+        for (let start = 0; start < offset; start += 1024 * 1024)
+          remoteHash.update(
+            await journal.readRange(writer.head, start, 1024 * 1024, options.signal),
+          );
+        if (remoteHash.digest("hex") !== digest)
+          throw new Error("migration_remote_digest_mismatch");
+        writer.head = { ...writer.head, sequence, sourceCursor };
+        await journal.close(writer, options.signal);
+        options.signal?.throwIfAborted();
+        const [updated] = await db
+          .update(heartbeatRuns)
+          .set({ logStore: "s3_segments", logRef: ref })
+          .where(
+            and(
+              eq(heartbeatRuns.id, run.id),
+              eq(heartbeatRuns.companyId, run.companyId),
+              eq(heartbeatRuns.logStore, "local_file"),
+              eq(heartbeatRuns.logRef, run.logRef),
+              run.bytes === null
+                ? isNull(heartbeatRuns.logBytes)
+                : eq(heartbeatRuns.logBytes, run.bytes),
+              run.sha256 === null
+                ? isNull(heartbeatRuns.logSha256)
+                : eq(heartbeatRuns.logSha256, run.sha256),
+              drainedWriter(),
+              run.runtimeMode === "native"
+                ? sql`not exists (select 1 from ${heartbeatRunEvents} where ${and(
+                    nativeRunLogSourceWhere(run.companyId, run.id),
+                    gt(heartbeatRunEvents.seq, sourceCursor),
+                  )})`
+                : undefined,
+            ),
+          )
+          .returning({ id: heartbeatRuns.id });
+        if (!updated) throw new Error("migration_publication_conflict");
+        item.state = "migrated";
+      }
+    } catch (error) {
+      options.signal?.throwIfAborted();
+      item.state = "blocked";
+      item.reason = error instanceof Error ? error.message : "migration_failed";
+    }
+  }
+  options.signal?.throwIfAborted();
+  const traces =
+    options.kinds && !options.kinds.includes("trace")
+      ? []
+      : await db
+          .select({ trace: providerTraceRecords, runStatus: heartbeatRuns.status })
+          .from(providerTraceRecords)
+          .innerJoin(
+            heartbeatRuns,
+            and(
+              eq(heartbeatRuns.id, providerTraceRecords.runId),
+              eq(heartbeatRuns.companyId, providerTraceRecords.companyId),
+            ),
+          )
+          .where(
+            and(
+              sql`${providerTraceRecords.traceRef} not like 's3-segments-v1:%'`,
+              isNull(providerTraceRecords.deletedAt),
+              gt(providerTraceRecords.expiresAt, new Date()),
+              options.companyId ? eq(providerTraceRecords.companyId, options.companyId) : undefined,
+              options.afterTraceId ? gt(providerTraceRecords.id, options.afterTraceId) : undefined,
+            ),
+          )
+          .orderBy(asc(providerTraceRecords.id))
+          .limit(limit);
+  const traceObjects =
+    traces.length && options.apply
+      ? createProviderTraceObjectStore(config.provider, config.tracePrefix)
+      : null;
+  for (const { trace, runStatus } of traces) {
+    options.signal?.throwIfAborted();
+    const item: RunEvidenceMigrationItem = {
+      kind: "trace",
+      id: trace.id,
+      state: "inventoried",
+    };
+    items.push(item);
+    try {
+      if (!SETTLED.includes(runStatus) || trace.status === "capturing")
+        throw new Error("legacy_trace_writer_not_settled");
+      await assertDrained(trace.runId, trace.companyId);
+      const importing = importingProviderTraceRef(trace.traceRef);
+      if (importing && (importing[1] !== trace.companyId || importing[2] !== trace.runId))
+        throw new Error("trace_import_binding_mismatch");
+      const filename = importing?.[3] ?? trace.traceRef;
+      if (!/^[a-f0-9-]+\.ndjson$/.test(filename)) throw new Error("invalid_legacy_trace_ref");
+      const file = path.join(
+        process.env.PROVIDER_TRACE_BASE_PATH ||
+          path.join(resolvePaperclipInstanceRoot(), "data", "provider-traces"),
+        filename,
+      );
+      for (const sidecar of [file, `${file}.rehydration`]) {
+        const stat = await fs.stat(sidecar).catch((error) => {
+          if (error.code === "ENOENT" && sidecar !== file) return null;
+          throw error;
+        });
+        if (stat && (!stat.isFile() || stat.size > 192 * 1024 * 1024))
+          throw new Error("legacy_trace_capture_size_invalid");
+      }
+      const sourceHash = createHash("sha256");
+      let byteCount = 0,
+        nativeTail: number | undefined;
+      for (const [index, sidecar] of [file, `${file}.rehydration`].entries()) {
+        const source = createReadStream(sidecar, {
+          highWaterMark: 1024 * 1024,
+          signal: options.signal,
+        });
+        try {
+          let first = true;
+          for await (const chunk of source) {
+            const bytes = Buffer.from(chunk);
+            if (
+              index === 1 &&
+              first &&
+              bytes.length &&
+              nativeTail !== undefined &&
+              nativeTail !== 10
+            ) {
+              sourceHash.update("\n");
+              byteCount++;
+            }
+            first = false;
+            sourceHash.update(bytes);
+            byteCount += bytes.length;
+            if (index === 0 && bytes.length) nativeTail = bytes.at(-1);
+          }
+        } catch (error) {
+          if (index !== 1 || (error as NodeJS.ErrnoException).code !== "ENOENT") throw error;
+        } finally {
+          source.destroy();
+        }
+      }
+      const digest = `sha256:${sourceHash.digest("hex")}`;
+      if (trace.digest && trace.digest !== digest) throw new Error("legacy_trace_digest_mismatch");
+      item.bytes = byteCount;
+      item.sha256 = digest;
+      if (traceObjects) {
+        options.signal?.throwIfAborted();
+        const ref = `s3-segments-v1:${trace.companyId}/${trace.runId}/${filename}`;
+        const pendingRef = ref.replace("s3-segments-v1:", "s3-import-v1:");
+        if (!importing) {
+          const staged = await db.transaction(async (tx) => {
+            await notifyDeliveryWork(tx, DELIVERY_QUEUES.providerTraceRetention);
+            return tx
+              .update(providerTraceRecords)
+              .set({ traceRef: pendingRef })
+              .where(
+                and(
+                  eq(providerTraceRecords.id, trace.id),
+                  eq(providerTraceRecords.traceRef, trace.traceRef),
+                  isNull(providerTraceRecords.deletedAt),
+                  gt(providerTraceRecords.expiresAt, new Date()),
+                ),
+              )
+              .returning({ id: providerTraceRecords.id });
+          });
+          if (!staged.length) throw new Error("trace_migration_staging_conflict");
+          trace.traceRef = pendingRef;
+        }
+        await traceObjects.open(
+          ref,
+          file,
+          async () => {
+            await assertDrained(trace.runId, trace.companyId);
+            const [current] = await db
+              .select({ id: providerTraceRecords.id })
+              .from(providerTraceRecords)
+              .where(
+                and(
+                  eq(providerTraceRecords.id, trace.id),
+                  eq(providerTraceRecords.traceRef, trace.traceRef),
+                  isNull(providerTraceRecords.deletedAt),
+                  gt(providerTraceRecords.expiresAt, new Date()),
+                ),
+              );
+            if (!current) throw new Error("trace_migration_authority_changed");
+          },
+          { restore: false, signal: options.signal },
+        );
+        await traceObjects.finish(ref);
+        const remoteHash = createHash("sha256");
+        let remoteBytes = 0;
+        for await (const chunk of traceObjects.readChunks(ref, options.signal)) {
+          remoteHash.update(chunk);
+          remoteBytes += chunk.length;
+        }
+        if (remoteBytes !== byteCount || `sha256:${remoteHash.digest("hex")}` !== digest)
+          throw new Error("trace_migration_remote_digest_mismatch");
+        options.signal?.throwIfAborted();
+        const updated = await db.transaction(async (tx) => {
+          await notifyDeliveryWork(tx, DELIVERY_QUEUES.providerTraceRetention);
+          return tx
+            .update(providerTraceRecords)
+            .set({ traceRef: ref })
+            .where(
+              and(
+                eq(providerTraceRecords.id, trace.id),
+                eq(providerTraceRecords.traceRef, trace.traceRef),
+                isNull(providerTraceRecords.deletedAt),
+                gt(providerTraceRecords.expiresAt, new Date()),
+                sql`exists (select 1 from ${heartbeatRuns} where ${and(
+                  eq(heartbeatRuns.id, trace.runId),
+                  eq(heartbeatRuns.companyId, trace.companyId),
+                  drainedWriter(),
+                )})`,
+              ),
+            )
+            .returning({ id: providerTraceRecords.id });
+        });
+        if (!updated.length) throw new Error("trace_migration_publication_conflict");
+        item.state = "migrated";
+      }
+    } catch (error) {
+      options.signal?.throwIfAborted();
+      item.state = "blocked";
+      item.reason = error instanceof Error ? error.message : "trace_migration_failed";
+    }
+  }
+  return {
+    items,
+    nextRunId: logs.length === limit ? logs.at(-1)!.id : undefined,
+    nextTraceId: traces.length === limit ? traces.at(-1)!.trace.id : undefined,
+  };
+}

server/src/services/run-evidence-storage.integration.test.ts

@@ -0,0 +1,225 @@
+import { randomUUID } from "node:crypto";
+import { promises as fs } from "node:fs";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
+import {
+  S3Client,
+  CreateBucketCommand,
+  ListObjectsV2Command,
+  DeleteObjectsCommand,
+  DeleteBucketCommand,
+} from "@aws-sdk/client-s3";
+import type { RunLogStore } from "./run-log-store.js";
+
+const endpoint = process.env.PAPERCLIP_LOG_TEST_S3_ENDPOINT;
+describe.skipIf(!endpoint)("instance storage for run evidence", () => {
+  const buckets = [0, 1].map(() => `pc-evidence-config-${randomUUID()}`);
+  const stores: RunLogStore[] = [];
+  let client: S3Client, directory: string;
+  beforeAll(async () => {
+    directory = await fs.mkdtemp(path.join(tmpdir(), "evidence-config-"));
+    vi.stubEnv("PAPERCLIP_HOME", directory);
+    vi.stubEnv("PAPERCLIP_CONFIG", path.join(directory, "config.json"));
+    vi.stubEnv("PAPERCLIP_DISABLE_CWD_ENV_FILE", "true");
+    vi.stubEnv("RUN_LOG_BASE_PATH", path.join(directory, "logs"));
+    for (const key of [
+      "RUN_LOG_STORE",
+      "PROVIDER_TRACE_STORE",
+      "RUN_LOG_S3_BUCKET",
+      "RUN_LOG_S3_PREFIX",
+      "PROVIDER_TRACE_S3_PREFIX",
+    ])
+      vi.stubEnv(key, undefined);
+    client = new S3Client({ endpoint, region: "us-east-1", forcePathStyle: true });
+    for (const bucket of buckets) await client.send(new CreateBucketCommand({ Bucket: bucket }));
+    for (const bucket of buckets) {
+      vi.resetModules();
+      selectStack(bucket);
+      const { getRunLogStore } = await import("./run-log-store.js");
+      stores.push(getRunLogStore());
+    }
+  });
+  afterAll(async () => {
+    if (client) {
+      for (const bucket of buckets) {
+        for (;;) {
+          const page = await client.send(new ListObjectsV2Command({ Bucket: bucket }));
+          if (!page.Contents?.length) break;
+          await client.send(
+            new DeleteObjectsCommand({
+              Bucket: bucket,
+              Delete: { Objects: page.Contents.map((object) => ({ Key: object.Key! })) },
+            }),
+          );
+        }
+        await client.send(new DeleteBucketCommand({ Bucket: bucket }));
+      }
+      client.destroy();
+    }
+    vi.unstubAllEnvs();
+    if (directory) await fs.rm(directory, { recursive: true, force: true });
+  });
+  function selectStack(bucket: string, prefix = "existing-root") {
+    vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "s3");
+    vi.stubEnv("PAPERCLIP_STORAGE_S3_BUCKET", bucket);
+    vi.stubEnv("PAPERCLIP_STORAGE_S3_REGION", "us-east-1");
+    vi.stubEnv("PAPERCLIP_STORAGE_S3_ENDPOINT", endpoint!);
+    vi.stubEnv("PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE", "true");
+    vi.stubEnv("PAPERCLIP_STORAGE_S3_PREFIX", prefix);
+  }
+  it("automatically uses each stack's existing bucket and prefix for identical run identities", async () => {
+    const handles: Awaited<ReturnType<RunLogStore["begin"]>>[] = [];
+    for (const [index, bucket] of buckets.entries()) {
+      const store = stores[index]!;
+      const handle = await store.begin({
+        companyId: "same-company",
+        agentId: "same-agent",
+        runId: "same-run",
+        newRun: true,
+        assertWriterAuthority: async () => undefined,
+      });
+      handles.push(handle);
+      expect(handle.store).toBe("s3_segments");
+      await store.append(handle, { stream: "stdout", ts: "then", chunk: `stack-${index}`, seq: 1 });
+      await store.finalize(handle);
+      const keys = (await client.send(new ListObjectsV2Command({ Bucket: bucket }))).Contents!.map(
+        (object) => object.Key!,
+      );
+      expect(
+        keys.every((key) =>
+          key.startsWith("existing-root/run-logs/segments-v1/same-company/same-agent/same-run/"),
+        ),
+      ).toBe(true);
+    }
+    expect(handles[0]!.logRef).toBe(handles[1]!.logRef);
+    for (const [index, store] of stores.entries()) {
+      const content = (await store.read(handles[1 - index]!)).content;
+      expect(content).toContain(`stack-${index}`);
+      expect(content).not.toContain(`stack-${1 - index}`);
+    }
+    await fs.rm(path.join(directory, "logs"), { recursive: true, force: true });
+    expect((await stores[0]!.read(handles[0]!)).content).toContain("stack-0");
+  });
+  it("honors config-file-only storage and keeps a locally configured instance local", async () => {
+    for (const key of [
+      "PAPERCLIP_STORAGE_PROVIDER",
+      "PAPERCLIP_STORAGE_S3_BUCKET",
+      "PAPERCLIP_STORAGE_S3_REGION",
+      "PAPERCLIP_STORAGE_S3_ENDPOINT",
+      "PAPERCLIP_STORAGE_S3_PREFIX",
+      "PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE",
+    ])
+      vi.stubEnv(key, undefined);
+    await fs.writeFile(
+      path.join(directory, "config.json"),
+      JSON.stringify({
+        $meta: { version: 1, updatedAt: "2026-10-30T00:00:00Z", source: "configure" },
+        database: { mode: "embedded-postgres" },
+        logging: { mode: "file" },
+        server: {},
+        storage: {
+          provider: "s3",
+          s3: {
+            bucket: buckets[0],
+            region: "us-east-1",
+            endpoint,
+            prefix: "file-config-root",
+            forcePathStyle: true,
+          },
+        },
+      }),
+    );
+    vi.resetModules();
+    const { getRunLogStore } = await import("./run-log-store.js");
+    const external = getRunLogStore();
+    const handle = await external.begin({
+      companyId: "file-company",
+      agentId: "agent",
+      runId: "run",
+      newRun: true,
+      assertWriterAuthority: async () => undefined,
+    });
+    expect(handle.store).toBe("s3_segments");
+    await external.append(handle, { stream: "stdout", ts: "then", chunk: "file-config" });
+    await external.finalize(handle);
+    expect(
+      (
+        await client.send(
+          new ListObjectsV2Command({ Bucket: buckets[0], Prefix: "file-config-root/run-logs/" }),
+        )
+      ).KeyCount,
+    ).toBeGreaterThan(0);
+    vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "local_disk");
+    vi.resetModules();
+    const local = (await import("./run-log-store.js")).getRunLogStore();
+    const localHandle = await local.begin({
+      companyId: "local-company",
+      agentId: "agent",
+      runId: "run",
+      newRun: true,
+    });
+    expect(localHandle.store).toBe("local_file");
+    expect(() => local.read(handle)).toThrow("Object run log storage is not configured");
+    await local.append(localHandle, { stream: "stdout", ts: "then", chunk: "local" });
+    await local.finalize(localHandle);
+    expect((await local.read(localHandle)).content).toContain("local");
+    vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "s3");
+    vi.resetModules();
+    const restored = (await import("./run-log-store.js")).getRunLogStore();
+    expect((await restored.read(handle)).content).toContain("file-config");
+    await fs.unlink(path.join(directory, "config.json"));
+  });
+  it("scopes trace publication and cleanup to the existing prefix, company and stack", async () => {
+    vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "a".repeat(64));
+    const { resolveRunEvidenceStorage } = await import("./run-evidence-storage.js");
+    const { createProviderTraceObjectStore } = await import("./provider-trace-object-store.js");
+    const { createStorageService } = await import("../storage/service.js");
+    const filename = `${randomUUID()}.ndjson`;
+    const records = [];
+    for (const [prefix, company] of [
+      ["stack-a", "same-company"],
+      ["stack-b", "same-company"],
+      ["stack-a", "other-company"],
+    ]) {
+      selectStack(buckets[0]!, prefix);
+      const config = resolveRunEvidenceStorage()!;
+      const store = createProviderTraceObjectStore(config.provider, config.tracePrefix);
+      const storage = createStorageService(config.provider);
+      const ref = `s3-segments-v1:${company}/same-run/${filename}`;
+      const file = path.join(directory, `${prefix}-${company}.ndjson`);
+      const body = Buffer.from(`${JSON.stringify({ fixture: `${prefix}:${company}` })}\n`);
+      await store.open(ref, file, async () => undefined);
+      await fs.appendFile(file, body);
+      await store.finish(ref);
+      const product = await storage.putFile({
+        companyId: company!,
+        namespace: "attachments",
+        originalFilename: "keep.txt",
+        contentType: "text/plain",
+        body: Buffer.from("keep"),
+      });
+      records.push({ store, storage, ref, body, product, company: company! });
+    }
+    expect(records[0]!.ref).toBe(records[1]!.ref);
+    await records[0]!.store.remove(records[0]!.ref);
+    await expect(records[0]!.store.read(records[0]!.ref)).rejects.toThrow("unavailable");
+    for (const record of records.slice(1))
+      expect(await record.store.read(record.ref)).toEqual(record.body);
+    for (const record of records) {
+      const object = await record.storage.getObject(record.company, record.product.objectKey);
+      const chunks = [];
+      for await (const chunk of object.stream) chunks.push(Buffer.from(chunk));
+      expect(Buffer.concat(chunks).toString()).toBe("keep");
+    }
+    const keys = (
+      await client.send(new ListObjectsV2Command({ Bucket: buckets[0], Prefix: "stack-b/" }))
+    ).Contents!.map((object) => object.Key!);
+    expect(
+      keys.some((key) =>
+        key.startsWith("stack-b/provider-traces/segments-v1/same-company/same-run/"),
+      ),
+    ).toBe(true);
+    expect(keys.some((key) => key.startsWith("stack-b/same-company/attachments/"))).toBe(true);
+  });
+});

server/src/services/run-evidence-storage.ts

@@ -0,0 +1,11 @@
+import { loadStorageConfig, type StorageConfig } from "../storage/config.js";
+import { createStorageProviderFromConfig } from "../storage/provider-registry.js";
+
+export function resolveRunEvidenceStorage(config: StorageConfig = loadStorageConfig()) {
+  if (config.storageProvider !== "s3") return undefined;
+  return {
+    provider: createStorageProviderFromConfig(config),
+    keyPrefix: "run-logs",
+    tracePrefix: "provider-traces",
+  };
+}

server/src/services/run-log-store.ts

@@ -7,13 +7,21 @@ import { notFound } from "../errors.js";
 import { resolvePaperclipInstanceRoot } from "../home-paths.js";
 import { createS3StorageProvider } from "../storage/s3-provider.js";
 import type { StorageProvider } from "../storage/types.js";
+import { createObjectRunLogStore } from "./object-run-log-store.js";
+import { resolveRunEvidenceStorage } from "./run-evidence-storage.js";
 
-export type RunLogStoreType = "local_file";
+export type RunLogStoreType = "local_file" | "s3_segments";
+export function isRunLogStoreType(value: unknown): value is RunLogStoreType {
+  return value === "local_file" || value === "s3_segments";
+}
 
 export interface RunLogHandle {
   store: RunLogStoreType;
   logRef: string;
   attemptId?: string;
+  initialBytes?: number;
+  initialSequence?: number;
+  initialSourceCursor?: number;
 }
 
 export interface RunLogReadOptions {
@@ -50,15 +58,24 @@ export interface RunLogBeginInput {
   // began this log, so no durable prefix can exist to restore. Omitted means
   // the run may be a retry or recovery and the durable mirror is checked.
   newRun?: boolean;
+
+  assertWriterAuthority?: () => Promise<void>;
+  previousStore?: RunLogStoreType;
+  signal?: AbortSignal;
 }
 
 export interface RunLogStore {
   begin(input: RunLogBeginInput): Promise<RunLogHandle>;
   append(
     handle: RunLogHandle,
-    event: { stream: "stdout" | "stderr" | "system"; chunk: string; ts: string; seq?: number },
+    event: { stream: "stdout" | "stderr" | "system"; chunk: string; ts: string; seq?: number; sourceCursor?: number },
   ): Promise<number>;
-  finalize(handle: RunLogHandle): Promise<RunLogFinalizeSummary>;
+  enqueue?(
+    handle: RunLogHandle,
+    event: Parameters<RunLogStore["append"]>[1],
+  ): Promise<{ committed: Promise<number> }>;
+  flush?(handle: RunLogHandle): Promise<void>;
+  finalize(handle: RunLogHandle, options?: { computeDigest?: boolean }): Promise<RunLogFinalizeSummary>;
   read(handle: RunLogHandle, opts?: RunLogReadOptions): Promise<RunLogReadResult>;
   // Optional so existing fakes/fixtures keep compiling: uploads every dirty
   // in-flight mirror immediately (graceful-shutdown path). No-op when the
@@ -509,7 +526,7 @@ export function createDurableRunLogStore(options: DurableRunLogStoreOptions): Ru
 // NOT redirect the product's workspace/file storage (smaller blast radius).
 // Unset RUN_LOG_S3_BUCKET -> no mirror -> local-only (safe degrade). Creds come
 // from the standard AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY chain.
-function resolveRunLogS3(): DurableRunLogStoreOptions["s3"] {
+export function resolveRunLogS3(): DurableRunLogStoreOptions["s3"] {
   const bucket = process.env.RUN_LOG_S3_BUCKET?.trim();
   if (!bucket) return undefined;
   const provider = createS3StorageProvider({
@@ -538,7 +555,40 @@ let cachedStore: RunLogStore | null = null;
 export function getRunLogStore() {
   if (cachedStore) return cachedStore;
   const basePath = process.env.RUN_LOG_BASE_PATH ?? path.resolve(resolvePaperclipInstanceRoot(), "data", "run-logs");
-  cachedStore = createDurableRunLogStore({ basePath, s3: resolveRunLogS3() });
+  const local = createDurableRunLogStore({ basePath, s3: resolveRunLogS3() });
+  const s3 = resolveRunEvidenceStorage();
+  const external = s3 ? createObjectRunLogStore({
+    provider: s3.provider, keyPrefix: s3.keyPrefix || "run-logs",
+    assertWriterAuthority: async input => {
+      if (!input.assertWriterAuthority) throw new Error("Object run logs require admitted writer authority");
+      await input.assertWriterAuthority();
+    },
+  }) : null;
+  const writer = external ?? local;
+  cachedStore = {
+    begin: input => {
+      if (input.previousStore === "local_file") return local.begin(input);
+      if (input.previousStore === "s3_segments") {
+        if (!external) throw new Error("Object run log storage is not configured");
+        return external.begin(input);
+      }
+      return writer.begin(input);
+    },
+    append: (handle, event) => handle.store === "s3_segments" ? external!.append(handle, event) : local.append(handle, event),
+    enqueue: (handle, event) => handle.store === "s3_segments"
+      ? external!.enqueue!(handle, event)
+      : Promise.resolve({ committed: local.append(handle, event) }),
+    flush: async handle => { if (handle.store === "s3_segments") await external!.flush!(handle); },
+    finalize: (handle, options) => handle.store === "s3_segments" ? external!.finalize(handle, options) : local.finalize(handle),
+    read: (handle, opts) => {
+      if (handle.store === "s3_segments") {
+        if (!external) throw new Error("Object run log storage is not configured");
+        return external.read(handle, opts);
+      }
+      return local.read(handle, opts);
+    },
+    flushInflightMirrors: async () => { await Promise.all([local.flushInflightMirrors?.(), external?.flushInflightMirrors?.()]); },
+  };
   return cachedStore;
 }
 
@@ -549,3 +599,21 @@ export function getRunLogStore() {
 export async function flushInFlightRunLogMirrors(): Promise<void> {
   await cachedStore?.flushInflightMirrors?.();
 }
+
+
+export async function openLegacyRunLogForMigration(logRef: string) {
+  const basePath = process.env.RUN_LOG_BASE_PATH ?? path.resolve(resolvePaperclipInstanceRoot(), "data", "run-logs");
+  const file = resolveWithin(basePath, logRef);
+  try {
+    const stat = await fs.stat(file);
+    if (!stat.isFile()) throw new Error("Legacy run log is not a regular file");
+    return { stream: createReadStream(file), bytes: stat.size, source: "local" as const };
+  } catch (error) {
+    if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;
+  }
+  const config = resolveRunLogS3();
+  if (!config) throw notFound("Legacy run log not found");
+  const key = `${config.keyPrefix?.replace(/\/+$/, "") || "run-logs"}/${logRef}`;
+  const object = await config.provider.getObject({ objectKey: key, signal: AbortSignal.timeout(30_000) });
+  return { stream: object.stream, bytes: object.contentLength, source: "mirror" as const };
+}

server/src/services/run-secret-redaction.ts

@@ -95,6 +95,9 @@ export function createRunSecretRedactionRegistry(db: Db) {
   }
 
   return {
+    redactionValuesForRun: async (companyId: string, runId: string) => valuesForRuns(await db
+      .select({ contextSnapshot: registrySnapshot }).from(heartbeatRuns)
+      .where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, runId)))),
     register: async (companyId: string, runId: string, value: string) => {
       const fingerprintSha256 = createHash("sha256").update(value).digest("hex");
       await db.transaction(async (tx) => {

server/src/storage/config.test.ts

@@ -0,0 +1,95 @@
+import { afterEach, describe, expect, it, vi } from "vitest";
+import { paperclipConfigSchema } from "@paperclipai/shared";
+import { loadStorageConfig } from "./config.js";
+import { resolveRunEvidenceStorage } from "../services/run-evidence-storage.js";
+import {
+  projectCloudRuntimeConfig,
+  setCloudRuntimeConfigSnapshot,
+} from "../services/cloud-runtime-config-snapshot.js";
+
+afterEach(() => {
+  setCloudRuntimeConfigSnapshot(null);
+  vi.unstubAllEnvs();
+});
+
+const file = paperclipConfigSchema.parse({
+  $meta: { version: 1, updatedAt: "2026-10-30T00:00:00Z", source: "configure" },
+  database: { mode: "embedded-postgres" },
+  logging: { mode: "file" },
+  server: {},
+  storage: {
+    provider: "s3",
+    s3: {
+      bucket: "stack-file",
+      region: "auto",
+      endpoint: "https://objects.example.test",
+      prefix: "existing-stack-root",
+      forcePathStyle: true,
+    },
+  },
+});
+describe("shared instance storage configuration", () => {
+  it("reuses the saved Cloud storage profile after its legacy config file is gone", () => {
+    vi.stubEnv("PAPERCLIP_MANAGED_CONFIG", "");
+    vi.stubEnv("PAPERCLIP_CONFIG", "/nonexistent/legacy-config.json");
+    for (const name of [
+      "PROVIDER",
+      "S3_BUCKET",
+      "S3_REGION",
+      "S3_ENDPOINT",
+      "S3_PREFIX",
+      "S3_FORCE_PATH_STYLE",
+    ])
+      vi.stubEnv(`PAPERCLIP_STORAGE_${name}`, undefined);
+    setCloudRuntimeConfigSnapshot({
+      version: 1,
+      revision: 4,
+      config: projectCloudRuntimeConfig(file),
+    });
+    expect(loadStorageConfig()).toMatchObject({
+      storageProvider: "s3",
+      storageS3Bucket: "stack-file",
+      storageS3Prefix: "existing-stack-root",
+    });
+    expect(resolveRunEvidenceStorage()).toMatchObject({
+      keyPrefix: "run-logs",
+      tracePrefix: "provider-traces",
+    });
+  });
+  it("keeps the existing config-file defaults and S3 connection fields", () => {
+    expect(loadStorageConfig(file, {})).toMatchObject({
+      storageProvider: "s3",
+      storageS3Bucket: "stack-file",
+      storageS3Region: "auto",
+      storageS3Endpoint: "https://objects.example.test",
+      storageS3Prefix: "existing-stack-root",
+      storageS3ForcePathStyle: true,
+    });
+  });
+  it("preserves environment precedence and an explicit false path-style setting", () => {
+    expect(
+      loadStorageConfig(file, {
+        PAPERCLIP_STORAGE_PROVIDER: "s3",
+        PAPERCLIP_STORAGE_S3_BUCKET: "stack-env",
+        PAPERCLIP_STORAGE_S3_REGION: "us-east-1",
+        PAPERCLIP_STORAGE_S3_ENDPOINT: "https://other.example.test",
+        PAPERCLIP_STORAGE_S3_PREFIX: "",
+        PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE: "false",
+      }),
+    ).toMatchObject({
+      storageProvider: "s3",
+      storageS3Bucket: "stack-env",
+      storageS3Region: "us-east-1",
+      storageS3Endpoint: "https://other.example.test",
+      storageS3Prefix: "",
+      storageS3ForcePathStyle: false,
+    });
+  });
+  it("keeps an unconfigured instance local when only a legacy mirror is configured", () => {
+    const config = loadStorageConfig(null, {
+      RUN_LOG_S3_BUCKET: "legacy-mirror",
+    });
+    expect(config.storageProvider).toBe("local_disk");
+    expect(resolveRunEvidenceStorage(config)).toBeUndefined();
+  });
+});

server/src/storage/config.ts

@@ -0,0 +1,39 @@
+import { STORAGE_PROVIDERS, type PaperclipConfig, type StorageProvider } from "@paperclipai/shared";
+import { readConfigFile } from "../config-file.js";
+import { resolveDefaultStorageDir, resolveHomeAwarePath } from "../home-paths.js";
+
+export interface StorageConfig {
+  storageProvider: StorageProvider;
+  storageLocalDiskBaseDir: string;
+  storageS3Bucket: string;
+  storageS3Region: string;
+  storageS3Endpoint: string | undefined;
+  storageS3Prefix: string;
+  storageS3ForcePathStyle: boolean;
+}
+
+export function loadStorageConfig(
+  fileConfig: PaperclipConfig | null = readConfigFile(),
+  env: NodeJS.ProcessEnv = process.env,
+): StorageConfig {
+  const storage = fileConfig?.storage;
+  const selected = env.PAPERCLIP_STORAGE_PROVIDER;
+  const provider =
+    selected && STORAGE_PROVIDERS.includes(selected as StorageProvider)
+      ? (selected as StorageProvider)
+      : null;
+  return {
+    storageProvider: provider ?? storage?.provider ?? "local_disk",
+    storageLocalDiskBaseDir: resolveHomeAwarePath(
+      env.PAPERCLIP_STORAGE_LOCAL_DIR ?? storage?.localDisk?.baseDir ?? resolveDefaultStorageDir(),
+    ),
+    storageS3Bucket: env.PAPERCLIP_STORAGE_S3_BUCKET ?? storage?.s3?.bucket ?? "paperclip",
+    storageS3Region: env.PAPERCLIP_STORAGE_S3_REGION ?? storage?.s3?.region ?? "us-east-1",
+    storageS3Endpoint: env.PAPERCLIP_STORAGE_S3_ENDPOINT ?? storage?.s3?.endpoint ?? undefined,
+    storageS3Prefix: env.PAPERCLIP_STORAGE_S3_PREFIX ?? storage?.s3?.prefix ?? "",
+    storageS3ForcePathStyle:
+      env.PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE !== undefined
+        ? env.PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE === "true"
+        : (storage?.s3?.forcePathStyle ?? false),
+  };
+}

server/src/storage/provider-registry.ts

@@ -1,9 +1,9 @@
-import type { Config } from "../config.js";
+import type { StorageConfig } from "./config.js";
 import type { StorageProvider } from "./types.js";
 import { createLocalDiskStorageProvider } from "./local-disk-provider.js";
 import { createS3StorageProvider } from "./s3-provider.js";
 
-export function createStorageProviderFromConfig(config: Config): StorageProvider {
+export function createStorageProviderFromConfig(config: StorageConfig): StorageProvider {
   if (config.storageProvider === "local_disk") {
     return createLocalDiskStorageProvider(config.storageLocalDiskBaseDir);
   }

server/src/storage/s3-provider.ts

@@ -4,6 +4,7 @@ import {
   GetObjectCommand,
   HeadObjectCommand,
   PutObjectCommand,
+  ListObjectsV2Command,
 } from "@aws-sdk/client-s3";
 import { putS3Multipart } from "./s3-multipart.js";
 import { withS3FailureDiagnostics } from "./s3-failure.js";
@@ -81,6 +82,33 @@ export function createS3StorageProvider(config: S3ProviderConfig): StorageProvid
   return {
     id: "s3",
 
+    async putObjectConditional(input) {
+      const output = await client.send(new PutObjectCommand({
+        Bucket: bucket,
+        Key: buildKey(prefix, input.objectKey),
+        Body: input.body,
+        ContentType: input.contentType,
+        ContentLength: input.contentLength,
+        IfMatch: input.ifMatch,
+        IfNoneMatch: input.ifNoneMatch,
+      }), { abortSignal: input.signal });
+      if (!output.ETag) throw new Error("Object storage did not return a publication ETag");
+      return { etag: output.ETag };
+    },
+
+    async listObjects(input) {
+      const keyPrefix = buildKey(prefix, input.prefix);
+      const output = await client.send(new ListObjectsV2Command({
+        Bucket: bucket, Prefix: keyPrefix,
+        ContinuationToken: input.continuationToken, MaxKeys: 1000,
+      }), { abortSignal: input.signal });
+      const strip = prefix ? prefix.length + 1 : 0;
+      return {
+        keys: (output.Contents ?? []).flatMap(object => object.Key ? [object.Key.slice(strip)] : []),
+        continuationToken: output.IsTruncated ? output.NextContinuationToken : undefined,
+      };
+    },
+
     async putObject(input) {
       const key = buildKey(prefix, input.objectKey);
       if (input.contentLength > 16 * 1024 * 1024) {

server/src/storage/types.ts

@@ -43,6 +43,16 @@ export interface StorageProvider {
   getObject(input: GetObjectInput): Promise<GetObjectResult>;
   headObject(input: GetObjectInput): Promise<HeadObjectResult>;
   deleteObject(input: GetObjectInput): Promise<void>;
+
+  putObjectConditional?(input: PutObjectInput & (
+    | { ifNoneMatch: "*"; ifMatch?: never }
+    | { ifMatch: string; ifNoneMatch?: never }
+  )): Promise<{ etag: string }>;
+
+  listObjects?(input: { prefix: string; continuationToken?: string; signal?: AbortSignal }): Promise<{
+    keys: string[];
+    continuationToken?: string;
+  }>;
 }
 
 export type PutFileInput = {