Commit Graph
5 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 47a2c82d35 fix(heartbeat): reach the previous server's live runs first after a release swap (#21450)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - During a release swap, the outgoing server stops and the incoming
server takes over the live runs on remote Runners
> - The incoming server reached those runs late: it waited up to 60 s
for the outgoing server's controller lease to lapse, re-verified each
run's whole log from object storage, and deferred a reattach when the
agent directory was busy. Queued starts competed with the reattaches for
the same resources.
> - On hosted staging, the later a live run was reached, the more likely
it was to fail. Runs reached in under 150 s never failed; runs reached
after 420 s almost always failed.
> - This pull request hands live runs over at the start of shutdown,
makes a reattach skip work that the same run already did, and holds
queued starts while handed-over runs are being reached, with a 180 s
cap.
> - The benefit is that live runs are reached within seconds of a swap
instead of minutes, so they continue instead of failing.

## Linked Issues or Issue Description

**What happened?**

Hosted staging, release swaps on 2026-11-13 (06:05Z to 09:45Z). For each
run that crossed a swap we measured the gap from the last Runner output
to the moment the new server recorded the Runner's process identity:

- gap under 150 s: 0 of 9 runs failed
- gap over 420 s: 14 of 15 runs failed
- median gap: 180 s for runs that resumed, 470 s for runs that failed

In one traced run the incoming server waited on
`remote_controller_lease_live` three times (09:14:21, 09:15:18,
09:16:17), claimed at 09:16:59, and spent 85 s in `task.prepare`. Of
that, `heartbeat.prepare.run_output` took 58.7 s, because it re-read the
run's whole log from object storage. A control run spent 23 s in
`task.prepare`. Queued starts used the same preparation path, the same 8
object-I/O slots and the same database pool, and nothing gave a reattach
priority. The first-attempt failures were `PRP command session.snapshot
timed out` and `codex_history_read_failed`. The retry then failed with
`runner_harness_state_mismatch: resumed_not_suspended`,
`runner_remote_process_identity_unavailable` or
`provider_turn_lost_on_restore`.

**Expected behavior**

Every live run is reached within about 30 s of the swap. Queued work
starts after that, or after a bounded wait.

**Steps to reproduce**

1. Run a two-server release swap with live remote runs and queued runs.
2. Observe `native.recovery.transition` "waiting" events with
`remote_controller_lease_live`, then the claim, then a long
`heartbeat.prepare.run_output` span.
3. The new tests in `reattach-phase.test.ts` and
`heartbeat-native-early-hand-off.test.ts` model this.

**Paperclip version or commit**

`master` at 426e8a35dc.

## What Changed

- Early hand-off at shutdown (`heartbeat/recovery.ts`,
`server/src/index.ts`): in deploy-overlap mode, the outgoing server
releases the remote controller lease of its live runs at SIGTERM
(`recoveryState` `awaiting_runner_reattach`). A run another server
already took over is left alone. The incoming server no longer waits for
the lease to lapse.
- Reattach fast path:
- `heartbeat/run-output.ts`, `native-runtime/native-run-log-export.ts`:
a reattach starts without waiting for the existing log to be
re-verified.
- `heartbeat/agent-directory-busy.ts`, `heartbeat.ts`: a reattach that
meets a busy agent directory waits in process (at most 30 s) instead of
being deferred to a full re-preparation. An ordinary start is still
deferred at once, with no added wait.
- Reattach phase lifecycle (`heartbeat/reattach-phase.ts`): the watch is
a worker on the shared delivery coordinator, queue `reattach-phase`.
- Trigger: one startup listing of a previous server's live runs; then
the hand-over notification (`notifyReattachHandOver`, sent in the
hand-over transaction); the earliest lease deadline it watches (a killed
owner); and a 2 s re-check only while handed-over runs are still
pending, which ends with the phase (cap 180 s). The bounded re-check
exists because a run can also end on another path that sends no
notification to this queue.
- Source of truth: `native_run_finalizations` and `heartbeat_runs` rows.
- Quiet condition: nothing pending and no watched lease, so no deadline
and no SQL. It stops with the coordinator at shutdown.
- Reached hook for every driver (`native-session-executor.ts`): a
reattached run counts as reached when the native session runtime hands
the executor its recovered or opened session (`onSession`). Every driver
reaches that point only after the Runner answered this server's attach
and recovery commands, including an adopted `RunnerdExternalDriver`
Runner, which records no new process identity. The identity hook stays
as an earlier signal.
- Claim wake: when the reattach-phase worker finds runs that just became
claimable (hand-over notification or a killed owner's lease deadline),
it starts this server's restart-recovery claim for them at once. A
validation cell showed the claim otherwise waited about 21.5 s after the
hand-over.
- `recovery/assigned-work-sweep.ts`: the `deferred_reattach_phase`
return now sits before every action path of `act()`, except starting an
overdue saved-work recovery (it resumes an existing run).
- The in-transaction admission re-check of a queued claim
(`confirmRunAdmissionStillOpen`) also honours the reattach phase. A
phase that begins during a claim returns the run to the queue through
the existing rollback.
- A reattach whose log re-verification runs in the background awaits it
before the log is finalized (`settleDeferredLogRecovery`, both
finalization sites in `run-completion.ts`).
- Reattach phase (`heartbeat/reattach-phase.ts`,
`heartbeat/run-admission.ts`): `isReattachPhaseActive()` is true while
runs that a previous controller handed over (lease released or observed
expired, or claimed by this server's restart recovery) are claimable but
not yet reached. A run is reached when its process identity is recorded.
Authentication is logged as a mark. The cap (180 s,
`PAPERCLIP_REATTACH_PHASE_CAP_MS`) counts from the moment the runs
became claimable. Later batches join or re-arm the phase. Runs reached
after 150 s are counted and logged.
- Gates on the phase:
- `claimQueuedRun`, start-next, resume-queued and dispatch in
`heartbeat/queue.ts`, `run-lifecycle.ts` and
`native-runner-execution-driver.ts` leave queued runs queued.
- `execution-recovery-resolution.ts`: one line, the first line of
`deliverReconciledExecutions`.
- `recovery/assigned-work-sweep.ts`: one line, outcome
`deferred_reattach_phase`.
- `heartbeat/recovery.ts`: a one-line claim hook
(`noteReattachRunsClaimed`) after restart recovery claims runs.
  - The owner of those three files agreed to these one-line gates.
- `tests/restart-chaos/daytona-image.ts`:
`PAPERCLIP_CHAOS_TEST_ACCEPT_IMAGE_CONTENT_ID` lets a local wave run a
controller-only branch against an existing Daytona image. It accepts a
content-ID mismatch only for the exact content ID of the named image,
and it prints a TEST OVERRIDE warning. This is harness-only.

## Verification

Red on master, green on this branch:

- `heartbeat/reattach-phase.test.ts` (27 tests). Five of the six
two-server scenarios were red on the earlier phase timing:
- `does not hold queued starts while the previous owner is alive and not
handing over`
- `closes the gate when the previous owner hands its runs over, and
counts reach latency from the hand-over`
- `runs the cap from the hand-over, so a wedged run ends the phase
without the overlap eating it`
- `begins when a killed owner's lease is observed expired, without any
hand-over`
- `re-arms for a later batch, including runs the previous owner started
after discovery`
- `closes the gate at once when this server claims a run between two
lease reads, dated from the hand-over`
- plus `claimQueuedRun leaves a queued run queued and unstarted, in the
admission gate's shape`, `is the same gate the assigned-work sweep and
reconciled continuation delivery read`, `ends at the cap, counted from
when the runs became claimable, so one wedged Runner cannot hold run
starts`
- `heartbeat-native-early-hand-off.test.ts`: `releases the remote
controller lease of %i live runs at once` (1, 20, 40 runs; 40 runs in 56
ms), `leaves a run another server already took over alone`, `holds the
incoming server's queued starts only from the hand-over, not while the
outgoing server still drives its runs`
- `heartbeat/run-output.test.ts`: `lets a reattach start before its
existing log is re-verified`
- `heartbeat/agent-directory-busy.test.ts`: `defers an ordinary start at
once, with no wait added to the run-start path`, `lets a reattach wait
its turn in process instead of a deferred full re-preparation`, `still
defers a reattach once its bounded wait runs out`
- `tests/restart-chaos/daytona-image.test.ts`: `accepts a mismatch only
for the exact image content ID named by the test override`
- Review fixes (red before the fix, green after):
`reattach-phase.test.ts` `starts the claim of handed-over runs within a
second of the hand-over notification`; `assigned-work-invariant.test.ts`
`takes no action while handed-over live runs are being reached, and acts
once the phase ends`; `heartbeat-native-early-hand-off.test.ts` `never
lets the outgoing server claim back a run it handed off, while the
incoming server may take it over`; `reattach-phase.test.ts` `retries a
failed first listing instead of going quiet, and still finds the
previous owner's runs`, `keeps a hand-over notification whose listing
failed until a listing succeeds`, `moves the cap earlier when a claimed
run turns out to have been handed over before its claim`;
`queue.test.ts` `returns the claim to the queue when a reattach phase
begins while the claim is being checked`; `run-completion.test.ts`
`finalizes the run log in completeRun|failRun only after a reattach's
deferred log recovery settles`. `native-session-executor.test.ts` `ends
the reattach phase once a reattached %s session is driven by this server
again, without the cap or manual settling` over codex_app_server,
opencode_server, acpx_runtime, muse_external, grok_bot_mcp and
openai_dot_mcp (6 of 6 red without the hook; the test drives the
executor with a mocked session runtime, not each real driver). Lifecycle
tests for the new worker: `does no database work and keeps no deadline
on an instance with nothing to reattach`, `waits for the hand-over
notification or the lease deadline instead of polling while the previous
owner drives its runs`, `begins at a killed owner's lease deadline,
without any hand-over or notification`, `stops with the delivery
coordinator at shutdown and ignores a later hand-over`.

Local runs on this branch: `tsc --noEmit` is clean for `server`. Passing
on this branch: `reattach-phase.test.ts`,
`heartbeat-native-early-hand-off.test.ts`, `run-output.test.ts` (53
tests together), `agent-directory-busy.test.ts`,
`heartbeat-native-restart-suspension.test.ts` (115 of 115, alone), the
new cases in `queue.test.ts` and `run-completion.test.ts`. Under load 30
to 96 on this shared machine, the full `queue.test.ts` and
`recovery-wake-herd.test.ts` runs stalled in their embedded-Postgres
setup; CI supplies them.

Daytona results that exist (local servers with Daytona sandboxes,
release-swap cells). They ran on a combined branch with this PR, #21462,
#21442 and #21362 (merged), so they cannot be credited to this PR alone:

- G5 baseline (master d1d71a7755): SIGTERM to PRP authentication 21.5 to
38 s at swap 1.
- G5c (combined branch 588c6ba267): SIGTERM to authentication 6.2 to 7.7
s at swap 1. All runs that crossed swap 2 were adopted on the same
Runner process. The staging chain (`provider recovery timed out after
900000ms`, then `resumed_not_suspended`) did not recur.
- Caveat: G5c ran before the phase timing fix in this PR. In G5c the
phase opened at process start, its 180 s cap ran out during the overlap,
and swap 2 ended `cap_reached` with 0 runs reached, 1 s before the lease
claims. The timing fix (the phase begins when the runs become claimable)
has not run on Daytona.
- G6c (12 live runs and 20 queued, two graceful swaps, no injection,
combined branch 588c6ba267): 0 failed runs at either swap; 11 of 11
survivors at both.
- Swap 1: hand-off of 11 runs in 291 ms; claims 0.3 to 2.9 s after
hand-off; authentication 7.6 to 12.5 s and identity 8.3 to 13.3 s after
SIGTERM. The 30 s target was met.
- Swap 2: hand-off took 8,562 ms and `hot_restart_prepare` timed out at
19 s, so the outgoing server drained gracefully. The incoming server did
not claim on the hand-over; it claimed only when the outgoing server
exited, 33 s after SIGTERM. Authentication was 46 to 54 s and identity
47 to 57 s after SIGTERM. The target was missed. Why the claims waited
for the exit is not explained.
- The host load was 49 to 70 from other work. 7 of 32 tasks timed out in
the queue or budget after the swaps; we suspect the overload.

## Risks

- Product trade-off: while handed-over live runs are being reached,
queued starts wait, up to the 180 s cap. On an instance with no
handed-over runs, nothing waits.
- Not proven: two-server behavior with the phase timing fix and the
notification-driven watch (no cell has run them); warm standby becoming
active; runs that the outgoing server starts after discovery.
- Early hand-off at SIGTERM orphans a server-side exec tool that is in
flight. With #21442 the next controller settles it as interrupted with
an unknown outcome; it is never run again.
- Swap 2 in G6c: the incoming server claimed only at the outgoing
server's exit. That is unexplained.
- Cross-session ownership: the gates in
`execution-recovery-resolution.ts` and
`recovery/assigned-work-sweep.ts`, and the claim hook in
`heartbeat/recovery.ts`, are in files that the restart-recovery work
owns. They are one-line gates agreed with that owner. That owner said
the gate did not need to be inside the claim transaction; that was
before the phase could begin during a claim. After review, the
in-transaction re-check in `heartbeat/queue.ts` also reads the phase,
through the existing rollback. #21427 (open) also edits
`execution-recovery-resolution.ts`, `heartbeat/queue.ts` and
`recovery/assigned-work-sweep.ts`; whichever merges second needs a
rebase. `native-runtime/native-restart-recovery.ts` (claim logic, owned
by the restart-recovery work): a 3-line guard in
`claimNativeRestartRecoveries` returns a run this process handed off at
shutdown as `blocked` / `handed_off_to_next_controller` before takeover
is evaluated, so the outgoing server never claims back a run it handed
off. #21478 (merged) added a sweep path that starts an overdue
saved-work recovery; it is deliberately not gated by the reattach phase,
because it resumes an existing run (reattach-like work), which the phase
makes room for rather than competes with.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:55:31 -06:00
DottaandClaude Opus 5.5 7a93cc6215 fix(server): redact invitation tokens from agent run logs (#20360)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People join a company through an invitation link. The link carries a
bearer token in its URL path.
> - An agent can create an invitation in chat and give the person the
join link.
> - The server writes each agent run to a run log file (`.ndjson`). This
file is a plain diagnostic copy. Local agents run as the same OS user,
so they can read it.
> - #20343 removed invitation tokens from HTTP logs. The run log still
kept the full join link and the raw token.
> - This pull request removes invitation tokens from run-log chunks when
the server writes them.
> - The benefit is that a live join link does not stay in a file that
other local agents can read. The person still gets the working link in
the chat reply.

## Linked Issues or Issue Description

Refs #20343 (same token policy for HTTP logs).

**What happened?**

An agent created an invitation in chat and replied with a
`/invite/<token>` join link. The agent run log file under
`data/run-logs/` kept the full token. The token was in the agent message
and in the tool result.

**Expected behavior**

The run log keeps the route and drops the token:
`/invite/***REDACTED***`. A raw `pcp_invite_...` token also becomes
`***REDACTED***`. The chat reply that the person reads keeps the working
join link.

**Steps to reproduce**

1. Check out master.
2. Run `server/src/services/heartbeat/run-log.test.ts` from this PR.
3. The "run-log invitation token privacy" tests fail. The token stays in
both the legacy text chunk and the canonical native envelope.

**Paperclip version or commit**

Master at the merge base of this branch.

**Deployment mode**

Source server, local run logs (`local_file`). The same functions also
format object-store run logs.

## What Changed

- Add `redactInvitationTokens()` in
`server/src/middleware/invitation-token-redaction.ts`, next to the HTTP
log redactor from #20343. It removes the token from `/invite/<token>`
segments and from raw `pcp_invite_` tokens inside free text and JSON
text. The character classes stop at JSON, Markdown and URL delimiters,
so the record structure stays valid. `redact-sensitive.ts` stays
byte-identical, because the chat eval pins its exact bytes before it
accepts a real invitation link.
- Apply it in `compactRunLogChunk()` and
`redactCanonicalNativeRunLogChunk()`. These are the two functions that
format every agent run-log chunk before the store writes it. The run
excerpt columns use the same chunk, so they are also redacted.
- `canonicalNativeRunLogRedactionForms()` also yields the earlier,
unredacted projection. A run log written by an earlier release still
verifies after an upgrade.
- `restoreCompactedEvent()` accepts a truncated record that the earlier
compactor wrote with the token. Cold recovery of such an old log still
works.
- Add a test that proves the chat reply comes from database run events,
not from the run log. The reply keeps the full join link, and the
presentation path never reads the run log.

## Verification

- From `server`: `node ../node_modules/vitest/vitest.mjs run --config
vitest.config.ts src/services/heartbeat/run-log.test.ts
src/__tests__/invitation-token-redaction.test.ts
src/__tests__/redact-sensitive.test.ts
src/__tests__/heartbeat-run-log.test.ts
src/__tests__/logger-invitation-redaction.test.ts
src/services/native-runtime/native-run-log-watermark.test.ts
src/services/native-runtime/native-run-log-recovery.test.ts
src/__tests__/heartbeat-run-output.test.ts
src/services/native-runtime/agent-chat-response-recovery.test.ts`
passes.
- Before the `run-log.ts` change, the two new redaction tests in
`run-log.test.ts` failed (red), then passed (green).
- `native-run-log-watermark.test.ts`: a legacy-compacted record with a
token in its retained head cold-recovers. It failed with
`native_log_source_coverage_unproven` before the restore fix.
- `agent-chat-response-recovery.test.ts` (embedded Postgres): a final
reply with a join link is saved as the exact comment with the token. No
run-log read happens.
- `tsc --noEmit -p server` shows no errors in the changed files.
- A live chat eval (`access.invite-teammate`) run with this change
confirms the run-log file has no token. The run is linked in a comment.

## Risks

- Low risk. Run-log readers (run transcript view, feedback export,
support inspection) now show `***REDACTED***` in place of invitation
tokens. This removes a live credential from a diagnostic copy. It does
not change the chat reply or the stored run events.
- A path segment after `/invite/` in free text is always redacted, even
if it is not a token. This loses only diagnostic detail.
- A token that a provider streams across two separate chunks is not
joined first. Each chunk is redacted on its own. A complete token in one
chunk, such as a final message or a tool result, is removed.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 17:52:19 -06:00
e482f0f632 fix(server): shed only retryable object-journal reads under load (#19176)
Client polling can exhaust the shared object journal queue and reject log persistence. Bound ordinary object-log and trace reads, keep persistence and internal verification queued, and return retryable 503 responses when reads are busy. Leave feedback exports unsent until every log page is available.

Validation: 72 latest-master affected tests and 10 workflow tests pass locally, with the workflow fixture passing strict TypeScript checks. The preceding broader run passed 280 server tests with two opt-in S3 skips. All applicable checks on d0a937768d are green (56 success, three skipped); exact-head Apex review is 5/5 with no unresolved findings. Background persistence still has no fixed queue bound during a prolonged storage outage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-06 13:47:25 -08:00
DottaandPaperclip 3910352c92 feat(runner): add experimental Cursor Cloud provider (#18068)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Runner gives each provider durable execution, tool, and completion
contracts.
> - Cursor Cloud currently has a direct adapter outside that native
contract.
> - Cloud work needs exact ownership across HTTP, SSE, Stop, and
recovery.
> - This PR adds a qualified Cloud API driver and verified proxy to
Runner.
> - Product setup and hosted placement remain separate dependent
changes.

## Linked Issues or Issue Description

Refs #5664, #11803, #17299, and #17471. Product integration is #18069.
Managed-only API placement is #18310. The related direct-adapter
model/transcript work is not replaced. No duplicate native Cursor Cloud
integration was found.

## What Changed

- Add native `cursor_cloud` / `cursor_cloud_api` with explicit
model/repository validation, bounded HTTP/SSE, durable per-turn
checkpoints and a qualified Node proxy launched through the Rust facade.
- Commit terminal custody before publication/release; coalesce
consumers, preserve ambiguous create/Stop/recovery fences, reject late
tools, and distinguish definitely-unsent oversized requests from
ambiguous responses.
- Support scoped cold attachment, current authority, reconnect and
verified artifact staging. Keep unsupported warm/external and Windows
paths closed.
- Preserve early source cursors and verify immutable legacy prefixes
before suffix replay, including supported compaction/masking. Optional
proof publication failure falls back to complete cold verification.
- Preserve canonical records at the durable native producer with
JSON-safe privacy and the existing 8MiB bound; drain admitted successor
flushes. Reconcile upstream semantic/pack/schema/lifecycle changes and
retain S3 and legacy CLI checks.

## Verification

Core head `37c442ec8dd4`: all 66 check contexts are terminal
successful/neutral/skipped, including required `ci / verify` and `ci /
e2e`; fresh exact-head Greptile is 5/5 with zero unresolved threads.
Dependent Product `81e828d20a77` and Top `dd2959689627` include the
later client handoff correction and are running their own fresh gates.

Core: 47/47 Rust, 4/4 compiled-native retirement, 19/19 settlement and
normal Runner build passed. Runtime Top `9af1a5de42a2`: build and
recursive typecheck passed. Earlier source-scoped evidence includes 398
focused tests, five relay tests and token/type gates. The 251-assertion,
7/8-suite run failed physical PostgreSQL teardown; the unchanged stable
WSP suite passed 128 through the documented wrapper.

The downstream fixture corrections passed 7/7 browser and 121/121
startup tests without timeout/oracle relaxation; a broken-unsubscribe
control fails. Hosted runtime `9af1a5de42a2` has a passing normal build
and recursive typecheck. Latest dependent heads additionally contain a
pure client fix for stale live/history overlap, with 48 focused tests,
UI typecheck and token gates passing. Its new client is not deployed in
the frozen hosted image; server/native/API/provider source is unchanged
by these four downstream files.

Fresh hosted Linux admission on `9af1a5de42a2` retired its owned
runner/proxy/socket with zero prompts. The saved configuration survived
reopening, and Test authenticated the bound key/catalog/selected model
without a prompt. Artifact with `claude-sonnet-4-5` completed,
downloaded the exact 27 bytes, matched 121 canonical identities against
complete logs and independently reached FINISHED. The active Stop retry
was clicked within its immutable boundary, produced one canonical
interrupted turn and acknowledged run-only cancellation, retained a
valid 35.076s quiet window (maximum gap 4.584s), and independently
remained CANCELLED/artifact-absent beyond the original 180-second
command deadline. All provider-observer credential callbacks, DB clients
and loaders closed; accounting is settled and explicitly unpriced. The
first fresh Stop retains its failed 10.373s sampling gap and every
sample.

On the upgraded9af server, the historical F226 artifact was reopened and
downloaded with its exact original hash after its provider agent had
been deleted/GET404 earlier today. Historical F226 full Artifact/Stop
cleanup, Question, corrected Plan and reconnect retain their original
identities. Deleting the three newly created disposable provider agents
and the new-case post-delete grade remain blocked on explicit user
approval after automatic review rejected expanding the two original
approved IDs. No new-case cleanup or complete fresh matrix pass is
claimed; this is separate from the directly observed task/Stop behavior
and retained exact cleanup proof.

Retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook
failures; Mac 17 SQL failures/skips; earlier startup-title, title-save,
custody/Stop and deletion failures with unknown causes and bounded
recoveries. `dea023` Artifact passed 17 checks but Stop failed its false
warning and 50 duplicate log identities. Pre-repair ACK failure and
original Core CI failure remain recorded; retry alone establishes no
cause or full local-suite pass. Provider dollar cost stays UNKNOWN.

## Risks

Experimental: Cursor independently acknowledges neither the effective
model nor instruction digest before the first prompt. The caller’s
explicit values are submitted and behavior checked live. Unpriced
provider dollar cost stays UNKNOWN.

Only qualified per-turn controllers are admitted. Uncertain work and
incomplete cleanup stay fenced. Existing damaged/duplicate archives
remain failed evidence. Prefix proof adds bounded object-store I/O; the
8MiB producer bound does not guarantee future providers’ output fits.

## Model Used

OpenAI Codex, GPT-6 family, assisted with reasoning, repository tools,
code execution, browser verification, and delegated code review. The
serving model’s exact API ID and context window are not exposed. No
value is inferred from the model catalog.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used and the limits of available model
metadata
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have linked public issues or described the issue in this PR
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch describes the change and contains no internal ticket ID
- [x] I have run the scoped local tests reported above; pending gates
are stated separately
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented the risks
- [x] All required CI gates pass on the current head
- [x] Fresh Greptile is 5/5 with no unresolved findings on the current
head
- [x] I will address all reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 08:38:57 -06:00
DottaandPaperclip dbff3747b8 feat(server): persist run logs and provider traces in S3 (#18327)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Operators use run logs and restricted provider traces to inspect
agent work.
> - Local log and trace files depend on the web server's persistent
volume.
> - That dependency prevents overlapping volume-backed Railway
deployments.
> - This PR reuses the instance S3 store for batched log and trace
payloads with conditional publication.
> - Existing Postgres rows retain their contracts; native export
recovery reuses events already committed before Runner acknowledgement.
> - S3-configured servers automatically copy and verify retained history
before a later deployment cutover.

## Linked Issues or Issue Description

**Subsystem affected**

Server run logs, restricted provider traces, storage providers, and
migration tooling.

**Problem or motivation**

Replacing the web server and its disk must preserve committed log output
and retained traces. The current S3 mirror can repeatedly upload a
growing file. Moving those payloads into Postgres would add database
writes, WAL, replication, and backup work.

**Proposed solution**

Reuse the existing instance storage configuration, S3-compatible
provider and credential chain. Store new run logs under run-logs/ and
opted-in raw traces under provider-traces/, below the existing prefix.
Cloud keeps its existing bucket and bucket-scoped credentials per stack.
Add no new configuration switches. Store immutable payload segments and
bounded binary range indexes directly in S3. Publish each batch with an
ETag-conditional head. Keep existing database ownership and lifecycle
records. Recover canonical native exports from their existing validated
control event source, checkpointing export progress in S3. Add no new
payload writes to Postgres. Preserve local defaults and migrate settled
history with verified byte counts and digests.

**Alternatives considered**

Postgres payload chunks add control-plane load. Repeated whole-file
mirrors amplify transfer work. The first index prototype left superseded
index nodes; the final binary index keeps all normally published nodes
reachable.

**Roadmap alignment**

This is a storage prerequisite for replacing the main web/API server. It
does not enable deployment overlap, detach a volume, or change native
Runner protocol acknowledgements. The maintainer approved this
implementation scope. ROADMAP.md has no duplicate implementation item.

Related PR: #16770 improves the existing in-flight mirror. This PR
preserves that legacy backend and uses the existing instance S3 profile
for new journals.

## What Changed

- Automatically select S3 segments for new logs and opted-in traces when
instance file storage uses S3. Reuse bucket, endpoint, region, prefix,
path style and credentials. Remove separate backend switches. Keep
local-disk instances local.
- Share the existing storage configuration resolver with product files.
Preserve config-file settings, environment precedence and the existing
root prefix. Keep pre-existing run-log mirror settings for legacy
history.
- Add bounded buffering, conditional ownership fencing, immutable range
reads, stable byte offsets, and completion hashing.
- Dispatch all existing log consumers through the recorded backend.
Coalesce output progress writes and publish live log messages after
storage receipts.
- Recover interrupted canonical native exports from their existing
durable source. Exclude separate credential, cleanup, and reprojection
records. Replay missing callback ranges and deduplicate late delivery.
Repair settled archives after controller lease expiry. Bound shared
repair work and stop it when all readers cancel. Serve the published
prefix when best-effort repair fails.
- Add encrypted optional trace channels, replacement reads, durable
deletion tombstones, and deadline-driven local/external expiry cleanup
independently of heartbeat scheduling. Stream historical source and
remote verification. Propagate worker cancellation through trace open,
conditional-write probing, upload, restore and reads. Fence deletion
even before the first object head exists; abort and drain owned
captures.
- Retry transient storage errors, distinguish 409 from failed
preconditions, and backpressure bounded log admission. Track interrupted
trace imports explicitly so ordinary local deletion stays independent of
S3.
- Automatically migrate retained history through the existing delivery
coordinator. Process one log and one trace per batch. Retry unresolved
history with backoff and stop scans after completion. Keep the
inventory/import command as optional diagnosis and repair. Verify source
and remote bytes before switching references. Verify finalized native
size and digest before setting its export cursor. Require ordered source
coverage when completeness metadata is missing. Preserve original trace
expiry. Block active, missing, corrupt, and unproven sources.
- Preserve identity and run-secret redaction in live and recovered
native logs. Register runtime log redaction secrets once through the
existing encrypted registry. Keep agent private keys in their existing
owner.
- Add storage, migration, retention, UTF-8, and route-privacy coverage,
plus operating documentation and qualification tooling.
- Add no schema, payload table, per-record SQL, production dependency,
or lockfile change.

## Verification

- Final fixes at `b3a44c898e` passed all 116 selected startup,
retention, attribution and cancellation controls. The disabled-scheduler
cleanup and concurrent-repair defects reproduced red before the fixes. A
timed-out repair still prevents later storage reads.
- The merged source passed 124 storage and real-Postgres Cloud
configuration tests. These include automatic history migration, existing
environment/file vault keys, source-directory removal, five real-SDK
stalled-request cancellation cases, shared capability-probe cancellation
and multi-chunk trace ordering. Saved Cloud S3 settings work after the
legacy config file is gone.
- Workspace typecheck and full build passed at the merged parent
`19f702d780`. Server typecheck and build passed again at `b3a44c898e`.
- Actual app acceptance: finish a process-agent run through the normal
API, stage its retained local-file input, then restart. Automatic
migration reports one migrated item and zero blockers. Remove the
original web home and restart at `b3a44c898e` with an empty home, the
existing S3 profile and the existing vault key file. The normal log API
preserves all five producer lines and exactly 788 bytes, with SHA-256
`74d9ae469da28bfe89b9b50a0e132aec24fe9e6d686496ed5e4ffa6167151036`. No
log-specific storage settings are present.
- Actual expiry acceptance: with heartbeat scheduling disabled, a trace
remains unexpired after startup and later reaches its fixture deadline.
The worker marks the row deleted and tombstones both S3 channel heads.
The retained run log stays byte-identical. The process fixture emits no
provider protocol frames, so nonempty encrypted payloads are qualified
separately through the real SDK round trips.
- Isolation tests use identical run identities in two stack buckets.
Cleanup in one existing prefix leaves another prefix, another company
and product files intact. Read-only Cloud source inspection confirms a
bucket and bucket-scoped credentials per stack. Four existing Cloud
storage environment-contract tests passed. No Cloud provisioning changed
and no new live vendor permission probe was performed.
- Earlier qualification passed 1,014 affected storage/native tests and
27 privacy-route tests with isolated Docker Postgres. An abrupt process
test commits 1,000 records, receives receipts, dies by SIGKILL and loses
its directory; another process reads every byte and continues. The real
Rust Runner experiment acknowledges 32 records while the export cursor
is zero, then continues the same Runner PID after controller loss. All
64 records export once with exactly 64 existing control rows. Repeated
snapshot/Stop commands completed in 162/81 ms.
- Emulator workload: 1,000 burst records used two batch PUTs. Eight
steady streams produced 240 records with 104 conditional PUTs and 72
GETs over 6.9 seconds. These are synthetic operation counts, not fleet
sizing or provider prices.
- Failure evidence is retained. Compact native events over 64 KiB,
missing file-key support, disabled-scheduler retention and repair
failure handling all reproduced before their fixes. Fixture repairs
addressed cold module setup, localhost/IPC permissions, the S3
bucket-name limit and a mode-0700 crash-test directory. Full local `pnpm
test:run` was stopped after host PostgreSQL semaphore exhaustion;
affected suites used isolated Docker databases instead. Native package
library symlinks were restored with the package's standard hydration
helper.
- The `19f702d780` browser session-recovery case timed out while saving
a task title. All eight browser shards passed on `b3a44c898e`, including
that case, with no UI behavior or assertion change. Fresh-head CI passed
at `b3a44c898e`: 55 successful checks, three skipped checks and one
neutral check. The branch is conflict-free.
- Map and module-boundary checks passed. Current-head Greptile is 5/5
with zero unresolved review threads.

Reproduce the opt-in storage suites with
`PAPERCLIP_LOG_TEST_S3_ENDPOINT` and a dedicated
`PAPERCLIP_RUN_EVIDENCE_MIGRATION_DATABASE_URL` ending in `_test`. See
`doc/run-evidence-storage.md` for operating limits, automatic migration,
optional diagnostics and rollout qualification.

## Risks

- Keep the existing per-stack bucket, prefix and vault key stable across
replacements. Relocating that storage is a separate whole-instance
operation.
- This release automatically changes new log storage for existing
S3-configured instances. Local-disk instances remain local. Canonical
native source replay is qualified with a deterministic real Runner;
actual remote execution and production-provider qualification remain
rollout gates. Generic server-only diagnostics and optional raw trace
tails without a surviving source can be incomplete after a crash. This
does not establish full zero-downtime deployment.
- Before deploying this release, qualify the actual S3 provider. It must
enforce conditional writes and pass read, version, deletion, workload,
and latency qualification. A capability probe rejects providers which
ignore preconditions. Moto is emulator evidence only.
- Dirty output is bounded and unconfirmed until S3 publication. Storage
failure leaves persistence incomplete while task results and usage can
settle.
- Trace expiry is not yet represented by Cloud's external wake contract.
Outstanding trace deadlines conservatively block sleep.
- Automatic history copying requires the original source volume. Keep it
until all retained resources pass verified import and clean-filesystem
reads. Stop old binaries from creating new local-file runs. Missing or
corrupt history remains blocked and visible in server logs. Rollback
requires a compatible reader against the same external state. Ordinary
object deletion does not erase retained historical object versions or
backups.

## Model Used

OpenAI Codex, `gpt-6.1-sol`, with reasoning, tool use, code execution,
and browser verification. The runtime did not expose a context-window
size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-30 15:17:35 -05:00