PaperClipAI/doc/run-evidence-storage.md

# Run log and provider trace object storage

Instances configured for S3-compatible file storage automatically use the
`s3_segments` backend for new run logs and opted-in provider traces. Postgres
retains the existing run/trace references and lifecycle rows.
It adds no payload table, per-record SQL, or database-to-object compactor.
Structured `heartbeat_run_events` retain their existing database path. Native
events already committed there before a Runner ACK also provide the recovery
source for an interrupted canonical native log export. No additional event copy
or payload write is introduced.
When native object logging uses runtime observability credentials, their log
redaction values are registered once in the existing encrypted run-secret
registry. This is bounded redaction metadata, not a transcript copy or a
per-event write. Agent identity private material stays in its existing key store.

## Configuration

No additional storage configuration is required. The existing instance
`storage.provider` setting, or `PAPERCLIP_STORAGE_PROVIDER`, selects storage.
The same config-file S3 fields and `PAPERCLIP_STORAGE_S3_*` overrides select the
bucket, endpoint, region, prefix, and path style used by product files. Credentials
use the same AWS SDK chain. Local-disk instances keep `local_file` logs and traces.

The instance's existing S3 prefix is preserved. The folders underneath it are:

```text
<existing-prefix>/run-logs/segments-v1/<company>/<agent>/<run>/
<existing-prefix>/provider-traces/segments-v1/<company>/<run>/<trace>/<channel>/
<existing-prefix>/<company>/...    existing product files
```

Cloud's existing provisioning supplies a separate bucket and bucket-scoped
credentials for each stack, plus the stack's encryption key. The app reuses that
boundary rather than selecting a global customer bucket. Within a stack, journal
keys remain company/run scoped. Cleanup lists only the resource's own prefix;
it cannot sweep other companies, other stacks' prefixes, or product files.
Folders are organization, not a substitute for bucket permissions or API access
checks. Shared-bucket self-hosted deployments must retain their existing distinct
instance prefixes and appropriate storage credentials.

Legacy run rows retain their recorded backend until verified migration. Their
pre-existing `RUN_LOG_S3_*` mirror settings and key paths remain supported for
legacy reads and writes. They do not override instance storage for new journals.
Use stable bucket/prefix settings across replacements and releases.
As with existing product files, changing the instance bucket or prefix is a
separate storage move and requires copying the complete namespace and preserving
its keys before changing configuration. Switching to local storage does not move
existing S3 evidence. Readers reject an unavailable external store; they never
substitute an empty local log. Restore the original stack's storage profile to
read its history. This upgrade migrates legacy local evidence into the existing
stack store; it does not introduce bucket relocation or reverse migration.

Raw capture still requires its existing explicit capture request. Its encryption
uses the existing vault key, from `PAPERCLIP_SECRETS_MASTER_KEY` or the existing
configured key file. Cloud already supplies the environment key. No separate
trace key is introduced, and a missing vault key fails rather than creating a
replacement. Disposable deployments must preserve the existing vault key along
with their other secrets.
Raw payload and index objects
use AES-GCM with a key derived for the particular company/run/trace/channel.
The bounded head holds only ownership, range, digest, and publication metadata.
Existing capture opt-in, redaction, instance-admin access, task/company access,
reveal/download auditing, and the 24-hour capture expiry remain in effect.

The provider must enforce `If-None-Match: *` and ETag `If-Match`, return ETags,
and support prefix-scoped paginated listing. Claiming the first journal performs
a conditional-write capability probe. A backend which ignores preconditions is
rejected. Before deploying this release to an S3-configured production instance,
qualify the provider's conditional writes, range/read consistency, encryption,
object versions, and cleanup. AWS API compatibility alone is insufficient evidence.

## Publication and operating limits

Each batch uploads immutable payload segments and merges equal-height index
roots into immutable binary nodes, then conditionally publishes one head object.
The head carries at most 53 ordered roots. Every normally published payload and
index node remains reachable; appends do not leave superseded index copies.
Interrupted or fenced uploads can leave unpublished objects, which remain inside
the resource prefix for scoped deletion. The head is the commit
boundary. A stale writer can upload an orphan but cannot replace the current
owner's published history. An uncertain response is reconciled against the exact
head before returning a durability receipt. Committed reads use immutable roots,
so publication cannot change the bytes behind an already selected snapshot.

An individual store `append` resolves only after publication. Heartbeat's
`onLog` diagnostic callback waits for bounded admission without serially
waiting for each network batch. At capacity it waits for space; admission beyond
the bounded waiting allowance is rejected without poisoning accepted batches.
Transient network, 409, 429 and server errors receive bounded retries. A 412
or a changed head fences the stale writer; a 409 alone does not prove ownership
loss or conditional-write support. Live log messages and progress are published
after storage confirmation. Finalization rejects new enqueue calls and drains
already-started bounded admissions, including calls waiting for capacity, before
closing and hashing the stream. Logging failure reports incomplete persistence and
leaves the final size/digest unknown rather than claiming a complete snapshot.
The original progress projection remains coarsely throttled; concurrent batch
completions share a single progress write.

Defaults and fixed safeguards:

| Limit | Value |
| --- | --- |
| Dirty batch deadline | 1 second, configurable from 1 ms to 30 seconds |
| Segment target | 1 MiB |
| Pending bytes per writer | 8 MiB |
| Pending run-log bytes per process | 64 MiB |
| Additional waiting admission | 8 MiB / 32 calls per process |
| Concurrent run-log flushes | 4 |
| Concurrent journal object operations | 8, with 32 queued operations |
| Object operation/batch cancellation deadline | 15 seconds |
| Current index fanout / maximum roots | 2 children / 53 roots |
| Journal head read bound | 64 KiB |
| Trace encoded payload bound per sidecar | 192 MiB, in addition to the existing raw capture limit |

A configured but empty log store sends no periodic request. Dirty buffers,
capture file-change events, and actual retention/retry obligations drive work.
Use request/transfer counts, byte volume, steady versus bursty producers,
concurrent runs, and control/API latency to size a deployment. The local S3
qualification is a synthetic workload, not a production cost forecast.

Finalization streams the committed bytes once to compute the existing complete
file SHA-256. It does not re-upload the entire log. Range indexes avoid loading
all segment descriptors to read a tail. Historical consumers, UTF-8/NDJSON record
alignment, issue previews, feedback, and customer-success inspection dispatch
through the persisted backend. Logical reads retain the existing authorization.
On-demand native repair closes its verified committed range with an exact byte
count and leaves an unavailable full-file SHA-256 null. It does not make a log
page request re-download the entire archive to compute that optional digest.

## Durability boundary

Committed log receipts and captured object prefixes survive web disk loss.
Output still buffered only by the current web process has no durable receipt.
The live log path waits for publication before showing that output. Raw sidecar
capture remains optional; a crash can leave an incomplete uncommitted tail, and
only durably closed channels plus valid terminal markers qualify as complete.

Native PRP events have a surviving source: the control port already commits
validated, redacted events before acknowledging them. The S3 head checkpoints
the highest exported database event cursor with the batch's bytes. A replacement
replays later native source rows in bounded 16-row pages, checks their identity
and canonical hash, and exports the same `paperclip.prp.event` representation
through normal log redaction/compaction. Late callbacks and duplicate delivery
cannot append the same source record again. This is a canonical native export;
generic stdout, stderr, and raw provider traces are not reconstructed from event
rows. The control ACK and sparse checkpoint formats remain unchanged.

Recovery runs before native execution reconnects. If the run settled before its
archive finished, the normal log API, issue preview, or feedback reader can repair
it after the native controller lease expires. Concurrent requests share recovery;
final metadata is saved only when no later native source remains. Read-only
customer-success inspection reads the currently published prefix. A failed
repair leaves final size/digest unknown and preserves the existing source.
Retained native events are not deleted by Runner cleanup. Recovery adds bounded
reads and one final metadata update, with no new per-record SQL on the normal
contiguous output path. Raw optional capture can still be incomplete after a crash.
Credential-update, retained-cleanup, and trace-reprojection records have separate
source namespaces and do not enter this log export. Corrupt canonical records
fail integrity validation. Shared on-demand repair has a 30-second lifetime;
cancellation removes a waiting reader and aborts storage work when no readers
remain. It preserves already published cursors so a later request can resume.
Live export and recovery both redact the existing agent identity and known run
secrets before publication. Recovery reads existing identity material without
provisioning a key. If best-effort repair loses ownership or fails, ordinary log
reads serve the published prefix and leave completion metadata unknown; request
cancellation still propagates.

The [remote continuity](testing-native-remote-continuity.md) protocol and the
parent deployment's other filesystem and HTTP handoff requirements still apply.

## Trace deletion, expiry, and sleep

Deletion/expiry denies reads through a database tombstone before payload cleanup.
Object journal tombstones prevent further publication. If an active writer may
have an in-flight upload, cleanup waits at least 30 seconds and remains a durable
retry obligation. A successful manual DELETE requires payload cleanup; an
unsettled cleanup returns a retryable failure, and the shared retention worker
retries it. Tombstones remain after object removal. Historical backups and
object-version retention need a separate operator policy; an ordinary DELETE
must not be mistaken for erasure of historical versions.

The shared coordinator restores expiry/cleanup work on startup and registers
actual deadlines for local and external captures. New local inserts also wake it;
expired backlogs drain in bounded pages. No completed-history polling is added. Trace expiry
is not yet part of Cloud's durable wake contract, so outstanding trace deadlines
conservatively block instance sleep. Removing that blocker requires qualification
of the corresponding external wake, not a longer polling interval.

Trace expiry and failed-deletion recovery remain registered when
`HEARTBEAT_SCHEDULER_ENABLED=false`. Disabling agent scheduling does not disable
retained-data cleanup. The worker still respects warm standby, admission drain,
actual deadlines, and shutdown.

## History migration and rollback

S3-configured servers automatically migrate retained local history in the background.
No command or extra configuration is required. The existing delivery coordinator
starts the upgrade outside request handling, independently of the heartbeat
scheduler toggle. Each batch processes at most one log and one trace, then yields
for one second. Existing references are the durable work inventory. Restarting
reconstructs unfinished work from those references; already migrated rows are
excluded. No new payload or job table is added.

Active writers, missing files, corrupt data, and storage failures leave history
unresolved. Other pages continue. The server logs each blocked resource and a
pass summary. Unresolved passes retry after five minutes, with backoff up to
30 minutes. Transport failures retry through the shared coordinator. After all
retained history is migrated, the worker stops querying and contacting storage.
Trace source hashing and remote verification stream bounded chunks instead of
loading a complete capture into server memory. The worker cancellation signal
reaches file reads, trace opening, conditional probing, uploads and verification.
Local-disk instances do not register this worker. Outstanding migration conservatively
blocks instance sleep, and shutdown aborts the batch and waits for owned work.

Deploy compatible readers and stop old binaries from starting new local runs.
The upgrade waits for existing writers to settle and their leases to expire.
Keep the source volume attached until migration completes and clean-filesystem
reads pass. Automatic copying cannot recover a file after its only copy is lost.

`server/scripts/migrate-run-evidence.ts` remains an optional diagnostic and repair
tool. Its default is a read-only inventory; `--apply` runs the same verified
importer used by the automatic worker. `--company`, `--after-run`, `--after-trace`
and `--limit` allow scoped investigation against an explicitly selected
`DATABASE_URL`. Normal upgrades do not require this tool. It does not stop agents,
select a production target, delete source files, or certify volume removal.

For logs, migration verifies exact local bytes or a mirror with proven length
and digest, compares any previously uploaded prefix, verifies remote SHA-256,
then conditionally changes the run's backend/reference. Unknown-completeness
mirrors, missing/corrupt payloads, and active writers block publication. Existing
trace bytes, download ordering, integrity state and original expiry are retained.
Neither path fabricates missing historical data or extends trace retention.
For finalized native logs, matching the recorded size and SHA-256 proves the
complete legacy file, including display chunks truncated by the existing 64 KiB
formatter. Migration preserves those exact bytes and records the settled source
cursor. If either completeness field is absent, a bounded NDJSON scan must prove
ordered coverage of the existing canonical source records. Missing ranges or
chunks whose identities cannot be verified block publication as
`native_log_source_coverage_unproven`. A database's latest event cursor alone
does not establish completeness. Publication checks that the original reference,
completeness metadata, and source cursor have not changed. Keep the original
volume and resolve blocked inventory before cutover.

The offline ephemeral-enrollment process can also preserve unfinalized native
local files as read-only `s3_archive` logs. This path requires
`PAPERCLIP_EPHEMERAL_MIGRATION=true`, all old deployments stopped, no queued or
running execution anywhere in the instance, and expired writer/finalizer leases.
The global execution check runs at page boundaries; per-run ownership and
metadata checks still fence every publication. New execution fails the page and
enrollment, while verified copies remain available for a safe retry.
It copies exact surviving bytes to a separate `<run>.archive` journal, checks
that the local source did not change, and verifies the remote SHA-256 before
switching the run's reference. It supports terminal `interrupted` runs as well.
It does not treat an unproven mirror as an original file or ignore a recorded
size/digest mismatch.

For online preservation of settled native files, use
`server/scripts/migrate-run-evidence.ts --preserve-settled-native-files` with
the existing company, cursor, limit and apply options. This separate mode does
not require global idle. It requires a native run with a finished status and
an explicitly released finalizer in `committed` or `terminal_failure`, no pending
attempt, no physical finalization owner, no retained allocation for that run,
and no task execution or checkout bound to it. A stale controller deadline or
historical PID does not establish this authority. Unknown recovery states stay
blocked; a terminal failure's existing blocked warning remains unchanged.

The online path locks the native finalization scope and rechecks the source
row, owner generation and reference before publication. The source must use its
canonical company/agent/run filename, with no aliases, symlinks or hard links.
It streams and verifies the exact local bytes, then publishes the read-only
archive and an activity receipt in one guarded transaction. Originals remain
on disk. A changed source or owner leaves the original reference unchanged.
This mode keeps raw history separate from canonical native-event coverage,
including when the original source has an integrity warning.

An archive preserves historical evidence; it does not certify a complete native
transcript. The run's status, canonical events, and original completion metadata
remain unchanged. Archive readers do not append missing native events, and
writers cannot reopen or modify the archive. Its source cursor remains zero.
The normal background importer retains its strict native coverage checks.
Rollbacks after enrollment require readers that support `s3_archive`.

An interrupted trace import keeps an `s3-import-v1:` reference: reads still use
the original local sidecars, while deletion and expiry retain responsibility for
the staged object prefix. The automatic worker resumes the verified switch after restart or retry.
Ordinary local traces have no S3 cleanup dependency.

Keep the volume until every retained item has been verified from a clean web
filesystem. Rollback must use a compatible reader against the same remote state.
An older binary or stale local volume is not a valid rollback target after new
external writes. Enforce that floor in the owning deployment procedure.

## Qualification

Run the storage, recorder, migration, privacy and retention tests. The opt-in
S3/Postgres integration fixtures use `PAPERCLIP_LOG_TEST_S3_ENDPOINT` and a dedicated
`PAPERCLIP_RUN_EVIDENCE_MIGRATION_DATABASE_URL` ending in `_test`. They create
unique test buckets and schemas, and remove only those resources. The privacy
route suite also accepts `PAPERCLIP_RUN_LOG_PRIVACY_TEST_DATABASE_URL` for a
dedicated, fully migrated database ending in `_test`; it clears its fixtures.

`server/scripts/qualify-object-run-logs.ts` exercises the actual AWS SDK against
a loopback test backend, confirms exact replacement reads and stale-writer
rejection, and reports requests/bytes/timing. It uses synthetic credentials and
refuses a non-loopback target. This proves its tested transport and workload;
production-provider qualification and actual Railway deployment remain separate.

`server/scripts/qualify-native-log-recovery.ts` uses the real Rust
`recovery-fixture` example, real Postgres, the control port, production recorder,
and S3 backend. It kills the controller after Runner ACKs but before the S3
cursor catches up, deletes that controller's directory, and restores sparse
control state into a fresh controller. It checks exact source order, no duplicate
records, the same surviving Runner PID, and control/Stop responsiveness. Build
the example with `cargo build --locked --manifest-path packages/paperclip-runner/runner/Cargo.toml -p paperclip-runner-core --example recovery-fixture`.
It accepts the same two explicit loopback test endpoints and creates a private
schema and bucket. This deterministic protocol experiment does not qualify a
live provider, remote execution environment, or production S3 vendor.