diff --git a/packages/paperclip-runner/README.md b/packages/paperclip-runner/README.md index 16d1a901f85..2eca4573895 100644 --- a/packages/paperclip-runner/README.md +++ b/packages/paperclip-runner/README.md @@ -8,10 +8,12 @@ Phase 0 runs both implementations against one language-neutral fixture. Phase 1 adds executable PRP schemas, a cross-language conformance corpus, deterministic static replay, and a standalone browser reference page. Phase 2 adds the Rust runner process, process-group supervision, a scripted fake harness over stdio -JSONL, a TypeScript mock core, and validated live browser replay. These phases do -not import or start Paperclip's server, UI, CLI, or production database. +JSONL, a TypeScript mock core, and validated live browser replay. Phase 3 adds +the Rust outbound WebSocket client, private durable state, replay and restart +recovery, and safe CLI/browser diagnostics. These phases do not import or start +Paperclip's server, UI, CLI, or production database. -## Phase 0–2 quick start +## Phase 0–3 quick start From the repository root: @@ -49,6 +51,7 @@ devtool: ```sh pnpm --filter @paperclipai/paperclip-runner replay:phase1 pnpm --filter @paperclipai/paperclip-runner trace:phase2 -- --scenario happy-path +pnpm --filter @paperclipai/paperclip-runner trace:phase3 -- --fault lost-ack pnpm --filter @paperclipai/paperclip-runner browser:dev --host 127.0.0.1 --port 4179 ``` @@ -68,12 +71,13 @@ pnpm --filter @paperclipai/paperclip-runner browser:dev --host 127.0.0.1 --port | `trace:phase0` | Run the Rust mock-core tracer, print the stable result, and exit. | | `trace:phase0:typescript` | Run the TypeScript reference tracer directly. | | `replay:phase1` | Validate and reduce a fixture to a final snapshot. | -| `browser:dev` | Open the editable fixture/static replay reference page. | | `trace:phase2` | Run one native local session through the Rust runner and fake harness. | | `record:phase2` | Capture a validated happy-path live trace as a replay fixture. | -| `browser:dev` | Start the standalone live/replay browser devtool. | +| `trace:phase3` | Run the Rust runner against the mock core with a selected recovery fault. | +| `record:phase3` | Regenerate the complete Phase 3 fault matrix and exact per-fault traces. | +| `browser:dev` | Start the standalone live/replay/recovery browser devtool. | | `test:browser` | Exercise static replay and live scenarios, then capture temporary screenshots under ignored test output. | -| `verify` | Run the complete Phase 0, Phase 1, and Phase 2 acceptance sequence. | +| `verify` | Run the complete Phase 0 through Phase 3 acceptance sequence. | | `verify:rootless` | Extract Debian/Ubuntu browser libraries without root, then run `verify`. | ## Navigate @@ -84,6 +88,8 @@ pnpm --filter @paperclipai/paperclip-runner browser:dev --host 127.0.0.1 --port - [Phase 1 hand-run tutorial](docs/tutorials/phase-01-static-replay.md) - [Phase 2 hand-run tutorial](docs/tutorials/phase-02-local-runner.md) - [Phase 2 local protocol reference](docs/phase-02-local-protocol.md) +- [Phase 3 break-it-on-purpose tutorial](docs/tutorials/phase-03-break-recovery.md) +- [Phase 3 durable transport reference](docs/phase-03-durable-transport.md) - [PRP compatibility/versioning policy](docs/protocol-compatibility.md) - [Cumulative end-to-end tutorial](docs/tutorials/end-to-end.md) - [Journal guide](docs/journal.md) @@ -91,6 +97,4 @@ pnpm --filter @paperclipai/paperclip-runner browser:dev --host 127.0.0.1 --port - [Implementation plan](spec/paperclip-native-runner-implementation-plan.md) - [Normative spike specification](spec/paperclip-native-runner-spike-spec.md) -Phase 2 is local and deterministic. It does not add durable network transport, -reconnect, production Paperclip integration, or a real model harness. Those -capabilities remain behind later checkpoints in the implementation plan. +Phase 3 adds package-local durable network recovery. Production Paperclip integration and a real model harness remain deferred. diff --git a/packages/paperclip-runner/docs/architecture.md b/packages/paperclip-runner/docs/architecture.md index fc98bb968d2..e273a3b6de5 100644 --- a/packages/paperclip-runner/docs/architecture.md +++ b/packages/paperclip-runner/docs/architecture.md @@ -169,6 +169,33 @@ Every browser event passes `validatePrpEvent` and `applyPrpEvent`. When the run ends, the browser reduces the complete event list again and compares the replay snapshot with the live snapshot. +## Phase 3 durable transport boundary + +```text +TypeScript mock core + | one-time ticket -> short-lived connection lease + | PRP v1 hello/welcome, commands, events, cumulative ACKs + v +paperclip-runnerd (Rust WebSocket client) + | atomic private JSON state + +-- durable outbox and processed-command cache + +-- stable runner/session/turn/item identities + +-- Phase 2 fake-harness process for restart proof +``` + +The runner initiates the loopback WebSocket. The bootstrap ticket is present +only in the runner process environment. The returned connection-lease token is +kept only in runner memory. The mock core stores SHA-256 digests of capabilities +and the runner state stores neither capability. The runner writes each event and +command result before network delivery, then removes outbox events only after a +valid cumulative ACK. + +The TypeScript peer is a package-local test controller, not production +Paperclip. It injects lost ACKs, socket loss, malformed input, process restarts, +lease expiry, storage pressure, drain, and revoke. The browser recovery page +calls that peer through guarded package-local Vite middleware and renders only +safe trace fields. + ## Future integration rule The [implementation plan](../spec/paperclip-native-runner-implementation-plan.md) diff --git a/packages/paperclip-runner/docs/index.md b/packages/paperclip-runner/docs/index.md index de11c505218..07074a0b531 100644 --- a/packages/paperclip-runner/docs/index.md +++ b/packages/paperclip-runner/docs/index.md @@ -8,6 +8,7 @@ Each tutorial starts with an explanation of what the phase is and what the phase - [Phase 0: standalone tracer](tutorials/phase-00-standalone-tracer.md) — install the workspace, validate its Rust/TypeScript boundary and knowledge bundle, then run the deterministic Rust mock-core path and cross-language parity check. - [Phase 1: static PRP replay](tutorials/phase-01-static-replay.md) — validate the shared schema/fixture corpus, reduce a fixture in the CLI, and inspect the same final snapshot in the standalone browser page. - [Phase 2: local runner and fake harness](tutorials/phase-02-local-runner.md) — run a supervised Rust process, exercise scripted live scenarios, resolve requests, interrupt a turn, and confirm live/replay parity. +- [Phase 3: break recovery on purpose](tutorials/phase-03-break-recovery.md) — lose an ACK, drop the socket, restart the runner side, replay durable events, and inspect recovery diagnostics. - [Cumulative end-to-end tutorial](tutorials/end-to-end.md) — the shortest complete workflow available at the current phase. ## Reference @@ -15,9 +16,9 @@ Each tutorial starts with an explanation of what the phase is and what the phase - [Architecture and dependency boundary](architecture.md) - [PRP compatibility and versioning policy](protocol-compatibility.md) - [Phase 2 local protocol and supervision](phase-02-local-protocol.md) +- [Phase 3 durable transport and recovery](phase-03-durable-transport.md) - [Engineering journal guide](journal.md) - [Dated shadcn/ui and AI Elements compatibility note](research/2026-08-07-ui-library-compatibility.md) - [Package README](../README.md) -Phase 2 is implemented and ready for its human checkpoint. Later tutorials are -added only after their phase is authorized and implemented. +Phase 3 is implemented and ready for its review and human checkpoint. Later tutorials remain deferred. diff --git a/packages/paperclip-runner/docs/phase-03-durable-transport.md b/packages/paperclip-runner/docs/phase-03-durable-transport.md new file mode 100644 index 00000000000..1c2434b0f1d --- /dev/null +++ b/packages/paperclip-runner/docs/phase-03-durable-transport.md @@ -0,0 +1,203 @@ +# Phase 3 Durable Transport and Recovery + +Phase 3 adds a real outbound WebSocket path to the standalone package. The +Rust `paperclip-runnerd` process is the client. The TypeScript mock core is the +remote peer. Neither side imports Paperclip server, UI, database, or shared +control-plane code. + +This is a local reliability proof. The mock uses `ws://127.0.0.1` so tests do +not need certificates. A production bridge must use `wss://` and remains a +later, separately reviewed phase. + +## Connection and authentication + +The connection starts in this order: + +1. The mock core creates a random bootstrap ticket with a five-second lifetime. +2. The ticket is passed to the runner through + `PAPERCLIP_RUNNER_BOOTSTRAP_TICKET`. It is not a command-line argument. +3. The runner opens an outbound WebSocket and sends the ticket in the HTTP + `Authorization` header. +4. The runner sends a PRP v1 `hello` envelope. It names the stable runner and + environment lease, approved runner version and digest, platform, + capabilities, and durable resume cursors. +5. The mock core consumes the ticket once and sends a `welcome` envelope. The + welcome selects PRP v1, returns a short-lived connection lease, reports the + cumulative committed event cursor, and carries at most one pending command. +6. Later connections authenticate with the connection lease. A real runner + process restart receives a new one-time bootstrap ticket and loads the same + durable state before it connects. + +The runner keeps the live connection lease token in memory only. Its state file +does not contain the bootstrap ticket or connection lease token. The mock core +persists only SHA-256 digests of those capabilities. A used or expired ticket is +rejected with `401`. The mock also rejects a changed runner identity, environment +lease, runner version, or runner digest. + +The daemon captures and removes the bootstrap environment variable before it +parses arguments or starts child work. Secret buffers are overwritten when they +are dropped. It resolves the destination once before sending a bearer value and +accepts only concrete loopback addresses. Userinfo, query strings, fragments, +wildcards, private-network addresses, public addresses, and mixed DNS answers +fail closed. + +## WebSocket limits + +The package client implements the RFC 6455 upgrade and masked client text +frames with Node-free Rust standard-library code. The mock core validates the +upgrade and parses bounded masked frames. + +- Maximum HTTP upgrade headers: 16 KiB. +- Maximum PRP frame: 1 MiB. +- Unknown frame opcodes and unmasked client frames close the connection. +- Malformed JSON is recorded as a bounded diagnostic. Durable state remains + available for reconnect. +- WebSocket ping/pong and PRP diagnostic pong state are supported. + +## Durable runner state + +The runner writes `runner-state.json` below its private state directory. The +directory uses mode `0700` and the file uses mode `0600` on Unix. Every update +is written to an exclusive unpredictable sibling file, synchronized, atomically +renamed, and followed by a parent-directory sync. Symlinked directories or state +files, wrong ownership, and wrong modes fail closed. + +The state contains: + +- stable runner, environment lease, run, session, turn, and item IDs; +- next source event sequence and cumulative acknowledged source sequence; +- unacknowledged event envelopes and their byte counts; +- a processed-command cache with a non-secret command digest and prior result; +- lifecycle, reconnect, backpressure, harness generation, and recovery facts; +- bounded, redacted diagnostics. + +Authentication capabilities and arbitrary command bodies are not stored. A +command is represented by an FNV-1a comparison digest, its stable ID and +controller sequence, the redacted result, and the logical-effect count. + +## Event delivery and ACKs + +The runner writes an event to the outbox before it sends the event. Event IDs +and source sequence numbers do not change after reconnect or restart. + +The mock core commits or deduplicates an event before it sends this cumulative +ACK: + +```json +{ + "kind": "ack", + "payload": { "ackedSourceSeq": 9 } +} +``` + +The ACK means that every source event through sequence 9 is committed or +deduplicated. The runner rejects an ACK that moves backward or beyond its +produced source cursor. It removes only events at or below a valid ACK. + +For the lost-ACK fault, the mock commits an event, drops the ACK and socket, and +reports the prior cursor once after reconnect. The runner sends the same bytes +again. The mock increments delivery count, keeps one logical event, and sends +the committed cumulative ACK. + +## Command delivery and effects + +Mock-core commands are durable before delivery. Only the lowest pending +controller sequence is sent. A command result advances the queue. + +The runner stores a command result before it sends the result. When a result or +socket is lost, the same command is delivered again. An equal command ID and +digest returns the stored result. It does not add events or repeat a process +effect. Reusing the ID with different bytes is rejected. + +The trace records `logicalEffectCount`. Every completed command has exactly one +logical effect. A policy rejection, such as a new turn during drain or storage +pressure, has zero effects. + +## Restart and reconciliation + +A socket drop keeps the same Rust process and in-memory lease. The process +reconnects and reloads the mock-core command and event cursor. + +A runner restart kills the real Rust process with unacknowledged work. A new +Rust process receives a fresh ticket, reads the same state file, emits a P0 +`runner.reconciled` event, and continues the same runner, session, turn, item, +command, and source-event identities. + +The harness-restart fault starts the real Phase 2 `fake-harness` child, waits +for its ready message, terminates its process group, and starts a second child. +The runner then emits `harness.exited`, `harness.ready`, and +`session.reconciled` events with the same normalized session, turn, and item +IDs. + +If a lease expires, reconnect fails closed. The runner records +`lease_expired_requires_bootstrap` and exits with its state intact. The mock +then gives the replacement runner a fresh ticket. Recovery continues from the +same cursor. + +If recovery cannot be truthful, state names the outcome. For example, failure +to reserve storage for a P0 event records `p0_storage_exhausted` and the +`unrecoverable` lifecycle. It never reports a fresh session as resumed. + +## Backpressure and bounded storage + +The runner has a byte limit and a reserved P0 region. + +- P2 item deltas coalesce before delivery. +- P1 events cannot consume the P0 reserve. +- New turns are rejected while backpressure is active. +- A P0 `runner.backpressure` event explains the state. +- P0 events are never dropped to make room. +- The state records peak outbox bytes so the final empty outbox does not hide a + limit violation. + +The storage-pressure trace emits 250 P2 updates. They become one coalesced +event, all P0 facts reach the mock core, peak bytes remain below the configured +limit, and the next turn is rejected without an effect. + +## Drain and revoke + +`runner.drain` persists the `draining` lifecycle and a P0 +`runner.draining` event. It rejects later `turn.start` commands but continues to +deliver the existing outbox. `runner.shutdown` stops only after the outbox is +acknowledged. + +A `revoke` envelope persists the `revoked` lifecycle. The runner flushes any +existing durable events and exits. It does not accept new work or delete +unacknowledged facts. + +## Diagnostics + +Run a fault and print operator-safe diagnostics: + +```sh +pnpm --filter @paperclipai/paperclip-runner trace:phase3 -- --fault lost-ack +``` + +Supported fault names are: + +```text +none +socket-drop +lost-ack +duplicate-command +runner-restart +harness-restart +malformed-input +lease-expiry +storage-pressure +drain +revoke +``` + +Use `--json` for the complete trace or `--output ` to write it. The CLI +and browser show connection counts, safe lease ID and expiry, stable identities, +source and ACK cursors, outbox current/peak bytes, command delivery/effect +counts, replay counts, restart counts, outcomes, and redaction assertions. + +They never show bootstrap tickets or connection lease tokens. + +Regenerate the checked fault matrix and every exact per-fault trace with: + +```sh +pnpm --filter @paperclipai/paperclip-runner record:phase3 +``` diff --git a/packages/paperclip-runner/docs/protocol-compatibility.md b/packages/paperclip-runner/docs/protocol-compatibility.md index 60f7bc7f3ce..c2df62eb30f 100644 --- a/packages/paperclip-runner/docs/protocol-compatibility.md +++ b/packages/paperclip-runner/docs/protocol-compatibility.md @@ -75,9 +75,30 @@ two surfaces. - The live browser rejects an event with an invalid schema, run ID, or session ID before it reaches the reducer. -These envelopes are local Phase 2 implementation contracts. Phase 3 may add a -durable outbound transport, but it must preserve the canonical PRP event and -command semantics. +These envelopes are local Phase 2 implementation contracts. + +## Phase 3 durable wire rules + +- The runner opens an outbound WebSocket and sends PRP v1 `hello` before any + command result or event. +- A one-use bootstrap bearer capability returns a short-lived connection lease + in `welcome`. Later connections use that lease. Neither raw capability is + durable state. +- `hello.resume` reports the last processed controller sequence, next source + sequence, cumulative ACK cursor, and current unacknowledged range. +- `welcome` selects the one overlapping protocol version, returns the core's + cumulative ACK cursor, and carries at most one durable pending command. +- An event is durable before send. Event IDs and source sequences stay stable + across replay and process restart. +- An ACK is cumulative. The runner rejects a cursor behind its durable ACK or + beyond its produced source cursor. +- An equal repeated command ID and canonical digest returns its stored result. + Reuse with different bytes fails closed and cannot repeat an effect. +- Frames are bounded at 1 MiB and upgrade headers at 16 KiB. Unknown or invalid + required protocol data fails closed; malformed JSON is a bounded diagnostic. + +These are package-local Phase 3 rules. Production TLS, control-plane admission, +and deployment policy remain separately reviewed work. ## Change policy diff --git a/packages/paperclip-runner/docs/tutorials/end-to-end.md b/packages/paperclip-runner/docs/tutorials/end-to-end.md index 71f31c090d5..11fb0d43126 100644 --- a/packages/paperclip-runner/docs/tutorials/end-to-end.md +++ b/packages/paperclip-runner/docs/tutorials/end-to-end.md @@ -2,16 +2,19 @@ ## What this tutorial is -This tutorial combines each implemented Native Runner phase into one procedure. It currently includes Phase 0, Phase 1, and Phase 2. +This tutorial combines each implemented Native Runner phase into one procedure. +It currently includes Phase 0 through Phase 3. ## What this tutorial proves This tutorial proves that the standalone package boundary, static replay path, -and local live-run path work together. It does not use Paperclip or a real model. +local live-run path, and durable recovery path work together. It does not use +Paperclip or a real model. The current system includes the Rust mock-core tracer, shared protocol fixtures, the Rust supervisor, a scripted fake harness, CLI live runs, and browser live and -replay modes. +replay modes. It also includes the Rust outbound WebSocket client, durable +outbox, package-local mock core, recovery CLI, and recovery browser view. ## Current end-to-end path @@ -32,6 +35,10 @@ replay modes. 12. Open the browser live mode and confirm the completed run says `Match` for live and replay output. 13. Open the [Phase 2 journal entry](../../knowledge/journal/2026-08-07-phase-02.md) and its linked verification evidence and screenshots. +14. Follow [Phase 3: Break Recovery on Purpose](phase-03-break-recovery.md). +15. Confirm that the recovered runner and session IDs stay the same. +16. Confirm that the outbox is empty after replay and cumulative acknowledgement. +17. Open the [Phase 3 journal entry](../../knowledge/journal/2026-08-07-phase-03.md). The one-command form after installation is: @@ -67,6 +74,14 @@ pnpm --filter @paperclipai/paperclip-runner verify:rootless - exactly one terminal event closes every completed local trace; - every live browser event passes the Phase 1 validator and reducer before display; - replaying the completed live event list produces the same final snapshot. +- a lost cumulative ACK replays the same durable event ID without a second + logical event; +- repeated commands return the stored result and cause one logical effect; +- runner and harness restarts preserve runner, session, turn, and item IDs; +- backpressure bounds local storage without dropping P0 events; +- lease expiry, drain, revoke, and unrecoverable storage outcomes are explicit; +- CLI and browser diagnostics do not expose bootstrap or connection-lease + tokens. -Phase 3 may extend this tutorial only after the Phase 2 human checkpoint is -accepted and the next phase is authorized. +Phase 4 remains uncreated until the Phase 3 review and human checkpoint are +complete. diff --git a/packages/paperclip-runner/docs/tutorials/phase-03-break-recovery.md b/packages/paperclip-runner/docs/tutorials/phase-03-break-recovery.md new file mode 100644 index 00000000000..6e062e33561 --- /dev/null +++ b/packages/paperclip-runner/docs/tutorials/phase-03-break-recovery.md @@ -0,0 +1,194 @@ +# Phase 3: Break It on Purpose + +## What this phase is + +Phase 3 adds a durable outbound WebSocket connection. + +The Rust runner connects to a package-local mock core. The runner stores events +before it sends them. It stores each command result before it replies. + +This phase does not use the Paperclip server, UI, database, or a real model. + +## What this phase proves + +This tutorial proves that a broken connection does not create a false new +session. + +You will drop a socket. You will lose an acknowledgement. You will restart the +real Rust runner process. You will also restart the real fake-harness child. + +Each test keeps the same runner, session, turn, item, command, and event IDs. +Repeated delivery causes one logical effect. P0 events are not lost. The +diagnostics do not show authentication secrets. + +## Before you start + +From the repository root, build the package binaries: + +```sh +pnpm --filter @paperclipai/paperclip-runner build:typescript +pnpm --filter @paperclipai/paperclip-runner build:phase2-binaries +``` + +The commands below use only loopback network access. They create temporary +state below the Paperclip run scratch directory when it is available. The CLI +removes that state after each successful proof. + +## Step 1: Drop the socket + +Run: + +```sh +pnpm --filter @paperclipai/paperclip-runner trace:phase3 -- --fault socket-drop +``` + +Confirm: + +- `Outcome: recovered` is shown. +- The connection count is at least 2. +- The runner ACK and core ACK are equal. +- Every assertion says `PASS`. + +## Step 2: Lose one ACK + +Run: + +```sh +pnpm --filter @paperclipai/paperclip-runner trace:phase3 -- --fault lost-ack +``` + +Confirm: + +- the output shows at least one replay delivery; +- the outbox ends with 0 pending events; +- `P0 lost 0` is shown; +- `PASS no duplicate logical events` is shown; +- the runner, session, turn, and item IDs are unchanged. + +The network may deliver one event more than once. The mock core keeps one +logical event and advances one cumulative cursor. + +## Step 3: Restart the Rust runner + +Run: + +```sh +pnpm --filter @paperclipai/paperclip-runner trace:phase3 -- --fault runner-restart +``` + +Confirm: + +- the output shows 1 runner restart; +- the output shows 2 fresh bootstraps; +- a replay delivery is present; +- every assertion says `PASS`. + +The test kills `paperclip-runnerd` while an event is unacknowledged. The next +process reads the same private state file. It emits `runner.reconciled` and +continues the same durable source cursor. + +## Step 4: Restart the fake harness + +Run: + +```sh +pnpm --filter @paperclipai/paperclip-runner trace:phase3 -- --fault harness-restart +``` + +Confirm: + +- the output shows 1 harness restart; +- `PASS stable identity` is shown; +- `PASS source cursor continuity` is shown. + +The test starts a real fake-harness child, stops its process group, and starts a +second child. The normalized session, turn, and item IDs do not change. + +## Step 5: Repeat a command + +Run: + +```sh +pnpm --filter @paperclipai/paperclip-runner trace:phase3 -- --fault duplicate-command +``` + +Confirm: + +- the output shows at least one duplicate command delivery; +- `PASS one logical effect per accepted command` is shown; +- `workspace.ready` exists one time in JSON output when you add `--json`. + +## Step 6: Apply storage pressure + +Run: + +```sh +pnpm --filter @paperclipai/paperclip-runner trace:phase3 -- --fault storage-pressure +``` + +Confirm: + +- backpressure is active in JSON output; +- peak storage is below the maximum; +- the next turn is rejected; +- `PASS P0 events preserved` is shown; +- `PASS bounded storage` is shown. + +## Step 7: Test lease expiry, drain, and revoke + +Run: + +```sh +pnpm --filter @paperclipai/paperclip-runner trace:phase3 -- --fault lease-expiry +pnpm --filter @paperclipai/paperclip-runner trace:phase3 -- --fault drain +pnpm --filter @paperclipai/paperclip-runner trace:phase3 -- --fault revoke +``` + +Confirm: + +- lease expiry uses a fresh bootstrap and resumes the same cursor; +- drain reports `Outcome: drained` and rejects new work; +- revoke reports `Outcome: revoked`; +- all three traces finish with an empty acknowledged outbox; +- every security assertion passes. + +## Step 8: Open the browser recovery view + +Run: + +```sh +pnpm --filter @paperclipai/paperclip-runner browser:dev --host 127.0.0.1 --port 4179 +``` + +Open `http://127.0.0.1:4179/`. Select **Recovery**. Select +**Run Phase 3 recovery**. + +Confirm: + +- recovery status becomes **complete**; +- connection, lease, outbox, ACK, replay, storage, and outcome fields appear; +- **Same runner and session** says **Preserved**; +- **Secrets redacted** says **Yes**; +- no bootstrap ticket or connection lease token appears. + +## Step 9: Run the package verification path + +Run: + +```sh +pnpm --filter @paperclipai/paperclip-runner verify +``` + +The verification path builds TypeScript, Rust, and the browser. It runs unit, +integration, browser, boundary, documentation, parity, and tracer checks for +Phases 0 through 3. + +## Evidence + +- [Phase 3 verification record](../../knowledge/evidence/2026-08-07-phase-03-verification.md) +- [Phase 3 recovery screenshot](../../knowledge/evidence/phase-03-recovery-diagnostics.png) +- [Phase 3 OKF journal entry](../../knowledge/journal/2026-08-07-phase-03.md) +- [Phase 3 transport reference](../phase-03-durable-transport.md) + +Phase 4 remains uncreated. A human must accept the Phase 3 checkpoint before +later work starts. diff --git a/packages/paperclip-runner/knowledge/evidence/2026-08-07-phase-03-verification.md b/packages/paperclip-runner/knowledge/evidence/2026-08-07-phase-03-verification.md new file mode 100644 index 00000000000..7c9638d7ec9 --- /dev/null +++ b/packages/paperclip-runner/knowledge/evidence/2026-08-07-phase-03-verification.md @@ -0,0 +1,78 @@ +--- +type: Verification Evidence +title: Phase 3 durable transport verification +description: Package-only evidence for lost-ACK replay, restart recovery, command deduplication, and redacted diagnostics. +tags: [native-runner, phase-3, websocket, recovery, evidence] +status: stable +generated: { by: openai/gpt-5, at: 2026-08-07T23:31:00Z } +--- + +# Scope + +This evidence covers the package-local mock core and runner transport. It does not use the Paperclip server or UI. + +# Required checks + +```sh +pnpm --filter @paperclipai/paperclip-runner typecheck +pnpm --filter @paperclipai/paperclip-runner test +pnpm --filter @paperclipai/paperclip-runner check:browser-tokens +pnpm --filter @paperclipai/paperclip-runner check:forbidden-imports +pnpm --filter @paperclipai/paperclip-runner docs:validate +pnpm --filter @paperclipai/paperclip-runner record:phase3 +pnpm --filter @paperclipai/paperclip-runner verify:rootless +``` + +# Acceptance mapping + +- The runner uses an outbound loopback WebSocket. +- A one-time bootstrap ticket creates a connection lease. +- A lost acknowledgement keeps one event in the durable outbox. +- A restarted runner loads the same identity and sends the event again. +- The mock core records one logical event and one duplicate delivery. +- The runner records one repeated command and one command effect. +- The terminal event uses the same runner and session IDs. +- Drain and revoke states are visible. +- CLI and browser diagnostics do not contain the ticket or lease token. + +# Recorded outputs + +- [CLI recovery trace](phase-03-recovery-trace.json) +- [Complete fault matrix](phase-03-fault-matrix.json) +- [Per-fault exact outputs](phase-03-faults/) +- [Browser recovery screenshot](phase-03-recovery-diagnostics.png), 1440 by + 1000 pixels, SHA-256 + `2c79ea7a9834943ee9dc3dfc6a5c4eb9359518cf773882dc75882031710f84d0`. + +# Local verification result + +- TypeScript tests: 61 passed. +- Rust tests: 34 passed, including fourteen Phase 3 transport, storage, + deduplication, secret-lifecycle, destination, and state-file tests. +- Focused Phase 3 and browser-middleware tests: 21 passed. +- Phase 3 fault tests: 14 passed, including one-time ticket reuse rejection, + cross-runner credential rejection, and symlink-safe mock-core state. +- Browser tests: 7 passed with the rootless Linux library path. +- TypeScript, Rust, and browser typechecks passed. +- Browser token, package boundary, documentation link, and OKF checks passed. +- All eleven recorded fault traces passed all seven assertions. +- The complete package `verify` path passed through `verify:rootless`. +- Destination tests reject public, private, wildcard, mixed-answer, userinfo, + query, fragment, and malformed WebSocket targets before a capability is sent. +- State tests reject symlinks, preserve an attacker-chosen sibling target, and + verify private file modes. A real child-process test proves the captured + bootstrap capability is absent from inherited environment data. + +The first browser attempt found missing host libraries. The rootless package +script supplied private copies without changing the host. + +The first documentation pass also found the linked Phase 3 screenshot was not +yet in the evidence directory. The passing browser run produced it, it was +visually inspected, and the final verification used the checked evidence file. + +# Review state + +- Security review: pending. +- UX review: pending. +- QA tutorial run: pending. +- Human checkpoint: pending. diff --git a/packages/paperclip-runner/knowledge/evidence/index.md b/packages/paperclip-runner/knowledge/evidence/index.md index a9be2dd18dc..f71610a37cb 100644 --- a/packages/paperclip-runner/knowledge/evidence/index.md +++ b/packages/paperclip-runner/knowledge/evidence/index.md @@ -6,3 +6,4 @@ * [Phase 0 Rust correction verification](2026-08-07-phase-00-rust-correction-verification.md) - Rust build/tests, cross-language parity, and corrected boundary evidence. * [Phase 1 verification](2026-08-07-phase-01-verification.md) - Schema, golden, reducer, Rust parity, CLI, browser, token, boundary, and screenshot evidence. * [Phase 2 verification](2026-08-07-phase-02-verification.md) - Rust supervisor, fake-driver scenarios, cleanup, live/replay parity, browser flows, screenshots, and complete package verification. +* [Phase 3 verification](2026-08-07-phase-03-verification.md) - Lost-ACK replay, restart recovery, command deduplication, storage, and redacted diagnostics. diff --git a/packages/paperclip-runner/knowledge/evidence/phase-03-fault-matrix.json b/packages/paperclip-runner/knowledge/evidence/phase-03-fault-matrix.json new file mode 100644 index 00000000000..7c9ce11f432 --- /dev/null +++ b/packages/paperclip-runner/knowledge/evidence/phase-03-fault-matrix.json @@ -0,0 +1,423 @@ +{ + "schema": "paperclip.runner.phase3.evidence.v1", + "traces": [ + { + "fault": "none", + "outcome": "recovered", + "connectionCount": 1, + "replayDeliveries": 8, + "runnerRestarts": 0, + "harnessRestarts": 0, + "commands": { + "issued": 4, + "completed": 4, + "rejected": 0, + "logicalEffects": 4, + "duplicateDeliveries": 0 + }, + "outbox": { + "events": 0, + "bytes": 0, + "peakBytes": 5315, + "maxBytes": 65536, + "backpressure": false, + "p0Committed": 5, + "p0Lost": 0 + }, + "security": { + "bootstrapTicketPersisted": false, + "connectionLeaseTokenPersisted": false, + "secretLeakCount": 0 + }, + "assertions": { + "stableIdentity": true, + "sourceCursorContinuous": true, + "oneLogicalEffectPerAcceptedCommand": true, + "noDuplicateLogicalEvents": true, + "p0Preserved": true, + "boundedStorage": true, + "secretsRedacted": true + } + }, + { + "fault": "socket-drop", + "outcome": "recovered", + "connectionCount": 2, + "replayDeliveries": 8, + "runnerRestarts": 0, + "harnessRestarts": 0, + "commands": { + "issued": 4, + "completed": 4, + "rejected": 0, + "logicalEffects": 4, + "duplicateDeliveries": 0 + }, + "outbox": { + "events": 0, + "bytes": 0, + "peakBytes": 5314, + "maxBytes": 65536, + "backpressure": false, + "p0Committed": 5, + "p0Lost": 0 + }, + "security": { + "bootstrapTicketPersisted": false, + "connectionLeaseTokenPersisted": false, + "secretLeakCount": 0 + }, + "assertions": { + "stableIdentity": true, + "sourceCursorContinuous": true, + "oneLogicalEffectPerAcceptedCommand": true, + "noDuplicateLogicalEvents": true, + "p0Preserved": true, + "boundedStorage": true, + "secretsRedacted": true + } + }, + { + "fault": "lost-ack", + "outcome": "recovered", + "connectionCount": 2, + "replayDeliveries": 9, + "runnerRestarts": 0, + "harnessRestarts": 0, + "commands": { + "issued": 4, + "completed": 4, + "rejected": 0, + "logicalEffects": 4, + "duplicateDeliveries": 1 + }, + "outbox": { + "events": 0, + "bytes": 0, + "peakBytes": 6023, + "maxBytes": 65536, + "backpressure": false, + "p0Committed": 5, + "p0Lost": 0 + }, + "security": { + "bootstrapTicketPersisted": false, + "connectionLeaseTokenPersisted": false, + "secretLeakCount": 0 + }, + "assertions": { + "stableIdentity": true, + "sourceCursorContinuous": true, + "oneLogicalEffectPerAcceptedCommand": true, + "noDuplicateLogicalEvents": true, + "p0Preserved": true, + "boundedStorage": true, + "secretsRedacted": true + } + }, + { + "fault": "duplicate-command", + "outcome": "recovered", + "connectionCount": 2, + "replayDeliveries": 8, + "runnerRestarts": 0, + "harnessRestarts": 0, + "commands": { + "issued": 4, + "completed": 4, + "rejected": 0, + "logicalEffects": 4, + "duplicateDeliveries": 1 + }, + "outbox": { + "events": 0, + "bytes": 0, + "peakBytes": 5314, + "maxBytes": 65536, + "backpressure": false, + "p0Committed": 5, + "p0Lost": 0 + }, + "security": { + "bootstrapTicketPersisted": false, + "connectionLeaseTokenPersisted": false, + "secretLeakCount": 0 + }, + "assertions": { + "stableIdentity": true, + "sourceCursorContinuous": true, + "oneLogicalEffectPerAcceptedCommand": true, + "noDuplicateLogicalEvents": true, + "p0Preserved": true, + "boundedStorage": true, + "secretsRedacted": true + } + }, + { + "fault": "runner-restart", + "outcome": "recovered", + "connectionCount": 2, + "replayDeliveries": 10, + "runnerRestarts": 1, + "harnessRestarts": 0, + "commands": { + "issued": 4, + "completed": 4, + "rejected": 0, + "logicalEffects": 4, + "duplicateDeliveries": 1 + }, + "outbox": { + "events": 0, + "bytes": 0, + "peakBytes": 6883, + "maxBytes": 65536, + "backpressure": false, + "p0Committed": 6, + "p0Lost": 0 + }, + "security": { + "bootstrapTicketPersisted": false, + "connectionLeaseTokenPersisted": false, + "secretLeakCount": 0 + }, + "assertions": { + "stableIdentity": true, + "sourceCursorContinuous": true, + "oneLogicalEffectPerAcceptedCommand": true, + "noDuplicateLogicalEvents": true, + "p0Preserved": true, + "boundedStorage": true, + "secretsRedacted": true + } + }, + { + "fault": "harness-restart", + "outcome": "recovered", + "connectionCount": 1, + "replayDeliveries": 11, + "runnerRestarts": 0, + "harnessRestarts": 1, + "commands": { + "issued": 5, + "completed": 5, + "rejected": 0, + "logicalEffects": 5, + "duplicateDeliveries": 0 + }, + "outbox": { + "events": 0, + "bytes": 0, + "peakBytes": 6827, + "maxBytes": 65536, + "backpressure": false, + "p0Committed": 8, + "p0Lost": 0 + }, + "security": { + "bootstrapTicketPersisted": false, + "connectionLeaseTokenPersisted": false, + "secretLeakCount": 0 + }, + "assertions": { + "stableIdentity": true, + "sourceCursorContinuous": true, + "oneLogicalEffectPerAcceptedCommand": true, + "noDuplicateLogicalEvents": true, + "p0Preserved": true, + "boundedStorage": true, + "secretsRedacted": true + } + }, + { + "fault": "malformed-input", + "outcome": "recovered", + "connectionCount": 2, + "replayDeliveries": 8, + "runnerRestarts": 0, + "harnessRestarts": 0, + "commands": { + "issued": 4, + "completed": 4, + "rejected": 0, + "logicalEffects": 4, + "duplicateDeliveries": 0 + }, + "outbox": { + "events": 0, + "bytes": 0, + "peakBytes": 5314, + "maxBytes": 65536, + "backpressure": false, + "p0Committed": 5, + "p0Lost": 0 + }, + "security": { + "bootstrapTicketPersisted": false, + "connectionLeaseTokenPersisted": false, + "secretLeakCount": 0 + }, + "assertions": { + "stableIdentity": true, + "sourceCursorContinuous": true, + "oneLogicalEffectPerAcceptedCommand": true, + "noDuplicateLogicalEvents": true, + "p0Preserved": true, + "boundedStorage": true, + "secretsRedacted": true + } + }, + { + "fault": "lease-expiry", + "outcome": "recovered", + "connectionCount": 2, + "replayDeliveries": 9, + "runnerRestarts": 1, + "harnessRestarts": 0, + "commands": { + "issued": 4, + "completed": 4, + "rejected": 0, + "logicalEffects": 4, + "duplicateDeliveries": 0 + }, + "outbox": { + "events": 0, + "bytes": 0, + "peakBytes": 5314, + "maxBytes": 65536, + "backpressure": false, + "p0Committed": 6, + "p0Lost": 0 + }, + "security": { + "bootstrapTicketPersisted": false, + "connectionLeaseTokenPersisted": false, + "secretLeakCount": 0 + }, + "assertions": { + "stableIdentity": true, + "sourceCursorContinuous": true, + "oneLogicalEffectPerAcceptedCommand": true, + "noDuplicateLogicalEvents": true, + "p0Preserved": true, + "boundedStorage": true, + "secretsRedacted": true + } + }, + { + "fault": "storage-pressure", + "outcome": "recovered", + "connectionCount": 1, + "replayDeliveries": 4, + "runnerRestarts": 0, + "harnessRestarts": 0, + "commands": { + "issued": 5, + "completed": 4, + "rejected": 1, + "logicalEffects": 4, + "duplicateDeliveries": 0 + }, + "outbox": { + "events": 0, + "bytes": 0, + "peakBytes": 2326, + "maxBytes": 16384, + "backpressure": true, + "p0Committed": 3, + "p0Lost": 0 + }, + "security": { + "bootstrapTicketPersisted": false, + "connectionLeaseTokenPersisted": false, + "secretLeakCount": 0 + }, + "assertions": { + "stableIdentity": true, + "sourceCursorContinuous": true, + "oneLogicalEffectPerAcceptedCommand": true, + "noDuplicateLogicalEvents": true, + "p0Preserved": true, + "boundedStorage": true, + "secretsRedacted": true + } + }, + { + "fault": "drain", + "outcome": "drained", + "connectionCount": 1, + "replayDeliveries": 3, + "runnerRestarts": 0, + "harnessRestarts": 0, + "commands": { + "issued": 5, + "completed": 4, + "rejected": 1, + "logicalEffects": 4, + "duplicateDeliveries": 0 + }, + "outbox": { + "events": 0, + "bytes": 0, + "peakBytes": 1478, + "maxBytes": 65536, + "backpressure": false, + "p0Committed": 3, + "p0Lost": 0 + }, + "security": { + "bootstrapTicketPersisted": false, + "connectionLeaseTokenPersisted": false, + "secretLeakCount": 0 + }, + "assertions": { + "stableIdentity": true, + "sourceCursorContinuous": true, + "oneLogicalEffectPerAcceptedCommand": true, + "noDuplicateLogicalEvents": true, + "p0Preserved": true, + "boundedStorage": true, + "secretsRedacted": true + } + }, + { + "fault": "revoke", + "outcome": "revoked", + "connectionCount": 1, + "replayDeliveries": 4, + "runnerRestarts": 0, + "harnessRestarts": 0, + "commands": { + "issued": 4, + "completed": 3, + "rejected": 1, + "logicalEffects": 3, + "duplicateDeliveries": 0 + }, + "outbox": { + "events": 0, + "bytes": 0, + "peakBytes": 5315, + "maxBytes": 65536, + "backpressure": false, + "p0Committed": 4, + "p0Lost": 0 + }, + "security": { + "bootstrapTicketPersisted": false, + "connectionLeaseTokenPersisted": false, + "secretLeakCount": 0 + }, + "assertions": { + "stableIdentity": true, + "sourceCursorContinuous": true, + "oneLogicalEffectPerAcceptedCommand": true, + "noDuplicateLogicalEvents": true, + "p0Preserved": true, + "boundedStorage": true, + "secretsRedacted": true + } + } + ] +} diff --git a/packages/paperclip-runner/knowledge/evidence/phase-03-faults/drain.json b/packages/paperclip-runner/knowledge/evidence/phase-03-faults/drain.json new file mode 100644 index 00000000000..f09e7f6bf0f --- /dev/null +++ b/packages/paperclip-runner/knowledge/evidence/phase-03-faults/drain.json @@ -0,0 +1,399 @@ +{ + "schema": "paperclip.runner.phase3.trace.v1", + "diagnostics": { + "schema": "paperclip.runner.phase3.diagnostics.v1", + "fault": "drain", + "connection": { + "state": "stopped", + "connectionCount": 1, + "reconnectCount": 0, + "leaseId": "connection_lease_0c922c88-0cfe-4a1f-b58d-15278a816adf", + "leaseExpiresAt": "2026-08-07T23:36:40.794Z" + }, + "identity": { + "runnerInstanceId": "runner_phase3_stable", + "environmentLeaseId": "environment_lease_phase3_stable", + "runId": "run_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "turnId": "turn_phase3_stable", + "itemId": "item_phase3_stable" + }, + "cursors": { + "runnerAckedSourceSeq": 4, + "runnerNextSourceSeq": 5, + "coreAckedSourceSeq": 4, + "highestCommittedSourceSeq": 4 + }, + "outbox": { + "events": 0, + "bytes": 0, + "peakBytes": 1478, + "maxBytes": 65536, + "backpressure": false, + "p0Committed": 3, + "p0Lost": 0 + }, + "commands": { + "issued": 5, + "completed": 4, + "rejected": 1, + "logicalEffects": 4, + "duplicateDeliveries": 0 + }, + "recovery": { + "replayDeliveries": 3, + "runnerRestarts": 0, + "harnessRestarts": 0, + "malformedFrames": 0, + "freshBootstraps": 1, + "outcome": "drained", + "reason": "drain_completed" + }, + "security": { + "bootstrapTicketPersisted": false, + "connectionLeaseTokenPersisted": false, + "secretLeakCount": 0 + }, + "committedEvents": [ + { + "sourceSeq": 1, + "sourceEventId": "event_runner_phase3_stable_000001", + "eventType": "workspace.ready", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000001", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.001Z", + "eventType": "workspace.ready", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "workspace": "phase-03-durable-fixture" + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000001", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 1, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.001Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 2, + "sourceEventId": "event_runner_phase3_stable_000002", + "eventType": "session.started", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000002", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.002Z", + "eventType": "session.started", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "driverSessionId": "driver_phase3_fake", + "normalizedSessionId": "session_phase3_stable", + "resumed": false + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000002", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 2, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.002Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 3, + "sourceEventId": "event_runner_phase3_stable_000003", + "eventType": "runner.draining", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000003", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.003Z", + "eventType": "runner.draining", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "newWorkAccepted": false + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000003", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 3, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.003Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 4, + "sourceEventId": "event_runner_phase3_stable_000004", + "eventType": "runner.stopped", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000004", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.004Z", + "eventType": "runner.stopped", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "afterDurableFlush": true + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000004", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 4, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.004Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 1, + "logicalEffectCount": 1 + } + ] + }, + "runnerState": { + "schema": "paperclip.runner.phase3.state.v1", + "runnerInstanceId": "runner_phase3_stable", + "environmentLeaseId": "environment_lease_phase3_stable", + "runId": "run_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "turnId": "turn_phase3_stable", + "itemId": "item_phase3_stable", + "lifecycle": "stopped", + "nextSourceSeq": 5, + "ackedSourceSeq": 4, + "lastControllerCommandSeq": 5, + "reconnectCount": 0, + "maxOutboxBytes": 65536, + "peakOutboxBytes": 1478, + "outbox": [], + "processedCommands": { + "command_phase3_001": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "commandDigest": "fnv1a64:c19dfbbf86a3dbb4", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "detail": "run prepared", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_002": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "commandDigest": "fnv1a64:b5198f3ce67ce88e", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "detail": "session bound", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_003": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "commandDigest": "fnv1a64:978d0ca44be61a36", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "detail": "runner draining", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_004": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "commandDigest": "fnv1a64:34d91505a30f60fb", + "status": "rejected", + "logicalEffectCount": 0, + "result": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "detail": "new turns are disabled while draining or backpressured", + "logicalEffectCount": 0, + "status": "rejected" + } + }, + "command_phase3_005": { + "commandId": "command_phase3_005", + "controllerSeq": 5, + "commandDigest": "fnv1a64:533318a30fba6d22", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_005", + "controllerSeq": 5, + "detail": "runner will stop after durable flush", + "logicalEffectCount": 1, + "status": "completed" + } + } + }, + "diagnostics": [], + "backpressure": false, + "recoverableFailure": null, + "unrecoverableOutcome": null, + "harnessGeneration": 1, + "stopAfterFlush": true + }, + "commands": [ + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_001", + "controllerSeq": 1, + "type": "run.prepare", + "issuedAt": "2026-08-07T23:30:01.000Z", + "payload": { + "workspace": "phase-03-durable-fixture" + }, + "status": "completed", + "result": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "detail": "run prepared", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_002", + "controllerSeq": 2, + "type": "session.open", + "issuedAt": "2026-08-07T23:30:02.000Z", + "payload": { + "reuse": "same_session" + }, + "status": "completed", + "result": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "detail": "session bound", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_003", + "controllerSeq": 3, + "type": "runner.drain", + "issuedAt": "2026-08-07T23:30:03.000Z", + "payload": {}, + "status": "completed", + "result": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "detail": "runner draining", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_004", + "controllerSeq": 4, + "type": "turn.start", + "issuedAt": "2026-08-07T23:30:04.000Z", + "payload": { + "turnId": "turn_phase3_stable", + "text": "Prove Phase 3 durable recovery." + }, + "status": "rejected", + "result": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "detail": "new turns are disabled while draining or backpressured", + "logicalEffectCount": 0, + "status": "rejected" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_005", + "controllerSeq": 5, + "type": "runner.shutdown", + "issuedAt": "2026-08-07T23:30:05.000Z", + "payload": {}, + "status": "completed", + "result": { + "commandId": "command_phase3_005", + "controllerSeq": 5, + "detail": "runner will stop after durable flush", + "logicalEffectCount": 1, + "status": "completed" + } + } + ], + "assertions": { + "stableIdentity": true, + "sourceCursorContinuous": true, + "oneLogicalEffectPerAcceptedCommand": true, + "noDuplicateLogicalEvents": true, + "p0Preserved": true, + "boundedStorage": true, + "secretsRedacted": true + } +} diff --git a/packages/paperclip-runner/knowledge/evidence/phase-03-faults/duplicate-command.json b/packages/paperclip-runner/knowledge/evidence/phase-03-faults/duplicate-command.json new file mode 100644 index 00000000000..e87daec3b97 --- /dev/null +++ b/packages/paperclip-runner/knowledge/evidence/phase-03-faults/duplicate-command.json @@ -0,0 +1,561 @@ +{ + "schema": "paperclip.runner.phase3.trace.v1", + "diagnostics": { + "schema": "paperclip.runner.phase3.diagnostics.v1", + "fault": "duplicate-command", + "connection": { + "state": "stopped", + "connectionCount": 2, + "reconnectCount": 1, + "leaseId": "connection_lease_2e6357a8-6a17-4450-a782-7aa4ca543a9c", + "leaseExpiresAt": "2026-08-07T23:36:39.046Z" + }, + "identity": { + "runnerInstanceId": "runner_phase3_stable", + "environmentLeaseId": "environment_lease_phase3_stable", + "runId": "run_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "turnId": "turn_phase3_stable", + "itemId": "item_phase3_stable" + }, + "cursors": { + "runnerAckedSourceSeq": 9, + "runnerNextSourceSeq": 10, + "coreAckedSourceSeq": 9, + "highestCommittedSourceSeq": 9 + }, + "outbox": { + "events": 0, + "bytes": 0, + "peakBytes": 5314, + "maxBytes": 65536, + "backpressure": false, + "p0Committed": 5, + "p0Lost": 0 + }, + "commands": { + "issued": 4, + "completed": 4, + "rejected": 0, + "logicalEffects": 4, + "duplicateDeliveries": 1 + }, + "recovery": { + "replayDeliveries": 8, + "runnerRestarts": 0, + "harnessRestarts": 0, + "malformedFrames": 0, + "freshBootstraps": 1, + "outcome": "recovered", + "reason": "duplicate_command_completed" + }, + "security": { + "bootstrapTicketPersisted": false, + "connectionLeaseTokenPersisted": false, + "secretLeakCount": 0 + }, + "committedEvents": [ + { + "sourceSeq": 1, + "sourceEventId": "event_runner_phase3_stable_000001", + "eventType": "workspace.ready", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000001", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.001Z", + "eventType": "workspace.ready", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "workspace": "phase-03-durable-fixture" + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000001", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 1, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.001Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 2, + "sourceEventId": "event_runner_phase3_stable_000002", + "eventType": "session.started", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000002", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.002Z", + "eventType": "session.started", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "driverSessionId": "driver_phase3_fake", + "normalizedSessionId": "session_phase3_stable", + "resumed": true + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000002", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 2, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.002Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 3, + "sourceEventId": "event_runner_phase3_stable_000003", + "eventType": "turn.accepted", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000003", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.003Z", + "eventType": "turn.accepted", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "sameSession": true, + "turnId": "turn_phase3_stable" + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000003", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 3, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.003Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 4, + "sourceEventId": "event_runner_phase3_stable_000004", + "eventType": "item.started", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000004", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.004Z", + "eventType": "item.started", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "kind": "assistant_message" + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000004", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 4, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.004Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 5, + "sourceEventId": "event_runner_phase3_stable_000005", + "eventType": "item.delta", + "priority": 2, + "envelope": { + "envelopeId": "envelope_event_000005", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.005Z", + "eventType": "item.delta", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "text": "Prove Phase 3 durable recovery." + }, + "priority": 2, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000005", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 5, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.005Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 6, + "sourceEventId": "event_runner_phase3_stable_000006", + "eventType": "item.completed", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000006", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.006Z", + "eventType": "item.completed", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "text": "Durable recovery completed." + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000006", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 6, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.006Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 7, + "sourceEventId": "event_runner_phase3_stable_000007", + "eventType": "run.result.proposed", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000007", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.007Z", + "eventType": "run.result.proposed", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "reportedWorkDisposition": "done", + "schema": "paperclip.prp.result.v1", + "status": "succeeded", + "summary": "Phase 3 durable transport recovered without duplicate effects." + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000007", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 7, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.007Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 8, + "sourceEventId": "event_runner_phase3_stable_000008", + "eventType": "run.terminal", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000008", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.008Z", + "eventType": "run.terminal", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "reportedWorkDisposition": "done", + "runTerminalState": "succeeded", + "schema": "paperclip.prp.terminal.v1", + "turnTerminalState": "completed" + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000008", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 8, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.008Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 9, + "sourceEventId": "event_runner_phase3_stable_000009", + "eventType": "runner.stopped", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000009", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.009Z", + "eventType": "runner.stopped", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "afterDurableFlush": true + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000009", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 9, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.009Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 1, + "logicalEffectCount": 1 + } + ] + }, + "runnerState": { + "schema": "paperclip.runner.phase3.state.v1", + "runnerInstanceId": "runner_phase3_stable", + "environmentLeaseId": "environment_lease_phase3_stable", + "runId": "run_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "turnId": "turn_phase3_stable", + "itemId": "item_phase3_stable", + "lifecycle": "stopped", + "nextSourceSeq": 10, + "ackedSourceSeq": 9, + "lastControllerCommandSeq": 4, + "reconnectCount": 1, + "maxOutboxBytes": 65536, + "peakOutboxBytes": 5314, + "outbox": [], + "processedCommands": { + "command_phase3_001": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "commandDigest": "fnv1a64:c19dfbbf86a3dbb4", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "detail": "run prepared", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_002": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "commandDigest": "fnv1a64:b5198f3ce67ce88e", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "detail": "session bound", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_003": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "commandDigest": "fnv1a64:fb3b206fe67e1dc8", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "detail": "turn completed", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_004": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "commandDigest": "fnv1a64:d11bf8308f863bad", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "detail": "runner will stop after durable flush", + "logicalEffectCount": 1, + "status": "completed" + } + } + }, + "diagnostics": [ + "WebSocket connection closed: failed to fill whole buffer" + ], + "backpressure": false, + "recoverableFailure": null, + "unrecoverableOutcome": null, + "harnessGeneration": 1, + "stopAfterFlush": true + }, + "commands": [ + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_001", + "controllerSeq": 1, + "type": "run.prepare", + "issuedAt": "2026-08-07T23:30:01.000Z", + "payload": { + "workspace": "phase-03-durable-fixture" + }, + "status": "completed", + "result": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "detail": "run prepared", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_002", + "controllerSeq": 2, + "type": "session.open", + "issuedAt": "2026-08-07T23:30:02.000Z", + "payload": { + "reuse": "same_session" + }, + "status": "completed", + "result": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "detail": "session bound", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_003", + "controllerSeq": 3, + "type": "turn.start", + "issuedAt": "2026-08-07T23:30:03.000Z", + "payload": { + "turnId": "turn_phase3_stable", + "text": "Prove Phase 3 durable recovery." + }, + "status": "completed", + "result": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "detail": "turn completed", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_004", + "controllerSeq": 4, + "type": "runner.shutdown", + "issuedAt": "2026-08-07T23:30:04.000Z", + "payload": {}, + "status": "completed", + "result": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "detail": "runner will stop after durable flush", + "logicalEffectCount": 1, + "status": "completed" + } + } + ], + "assertions": { + "stableIdentity": true, + "sourceCursorContinuous": true, + "oneLogicalEffectPerAcceptedCommand": true, + "noDuplicateLogicalEvents": true, + "p0Preserved": true, + "boundedStorage": true, + "secretsRedacted": true + } +} diff --git a/packages/paperclip-runner/knowledge/evidence/phase-03-faults/harness-restart.json b/packages/paperclip-runner/knowledge/evidence/phase-03-faults/harness-restart.json new file mode 100644 index 00000000000..9ef6cff0207 --- /dev/null +++ b/packages/paperclip-runner/knowledge/evidence/phase-03-faults/harness-restart.json @@ -0,0 +1,704 @@ +{ + "schema": "paperclip.runner.phase3.trace.v1", + "diagnostics": { + "schema": "paperclip.runner.phase3.diagnostics.v1", + "fault": "harness-restart", + "connection": { + "state": "stopped", + "connectionCount": 1, + "reconnectCount": 0, + "leaseId": "connection_lease_6de7f5ec-7f92-4c7c-806a-d262e02b2271", + "leaseExpiresAt": "2026-08-07T23:36:39.644Z" + }, + "identity": { + "runnerInstanceId": "runner_phase3_stable", + "environmentLeaseId": "environment_lease_phase3_stable", + "runId": "run_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "turnId": "turn_phase3_stable", + "itemId": "item_phase3_stable" + }, + "cursors": { + "runnerAckedSourceSeq": 12, + "runnerNextSourceSeq": 13, + "coreAckedSourceSeq": 12, + "highestCommittedSourceSeq": 12 + }, + "outbox": { + "events": 0, + "bytes": 0, + "peakBytes": 6827, + "maxBytes": 65536, + "backpressure": false, + "p0Committed": 8, + "p0Lost": 0 + }, + "commands": { + "issued": 5, + "completed": 5, + "rejected": 0, + "logicalEffects": 5, + "duplicateDeliveries": 0 + }, + "recovery": { + "replayDeliveries": 11, + "runnerRestarts": 0, + "harnessRestarts": 1, + "malformedFrames": 0, + "freshBootstraps": 1, + "outcome": "recovered", + "reason": "harness_restart_completed" + }, + "security": { + "bootstrapTicketPersisted": false, + "connectionLeaseTokenPersisted": false, + "secretLeakCount": 0 + }, + "committedEvents": [ + { + "sourceSeq": 1, + "sourceEventId": "event_runner_phase3_stable_000001", + "eventType": "workspace.ready", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000001", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.001Z", + "eventType": "workspace.ready", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "workspace": "phase-03-durable-fixture" + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000001", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 1, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.001Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 2, + "sourceEventId": "event_runner_phase3_stable_000002", + "eventType": "session.started", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000002", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.002Z", + "eventType": "session.started", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "driverSessionId": "driver_phase3_fake", + "normalizedSessionId": "session_phase3_stable", + "resumed": false + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000002", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 2, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.002Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 3, + "sourceEventId": "event_runner_phase3_stable_000003", + "eventType": "harness.exited", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000003", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.003Z", + "eventType": "harness.exited", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "generation": 1, + "reason": "fault_injection_restart", + "recoverable": true + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000003", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 3, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.003Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 4, + "sourceEventId": "event_runner_phase3_stable_000004", + "eventType": "harness.ready", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000004", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.004Z", + "eventType": "harness.ready", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "driverKind": "fake", + "generation": 2 + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000004", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 4, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.004Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 5, + "sourceEventId": "event_runner_phase3_stable_000005", + "eventType": "session.reconciled", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000005", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.005Z", + "eventType": "session.reconciled", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "outcome": "same_session_resumed", + "turnId": "turn_phase3_stable" + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000005", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 5, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.005Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 6, + "sourceEventId": "event_runner_phase3_stable_000006", + "eventType": "turn.accepted", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000006", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.006Z", + "eventType": "turn.accepted", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "sameSession": true, + "turnId": "turn_phase3_stable" + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000006", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 6, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.006Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 7, + "sourceEventId": "event_runner_phase3_stable_000007", + "eventType": "item.started", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000007", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.007Z", + "eventType": "item.started", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "kind": "assistant_message" + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000007", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 7, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.007Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 8, + "sourceEventId": "event_runner_phase3_stable_000008", + "eventType": "item.delta", + "priority": 2, + "envelope": { + "envelopeId": "envelope_event_000008", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.008Z", + "eventType": "item.delta", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "text": "Prove Phase 3 durable recovery." + }, + "priority": 2, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000008", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 8, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.008Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 9, + "sourceEventId": "event_runner_phase3_stable_000009", + "eventType": "item.completed", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000009", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.009Z", + "eventType": "item.completed", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "text": "Durable recovery completed." + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000009", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 9, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.009Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 10, + "sourceEventId": "event_runner_phase3_stable_000010", + "eventType": "run.result.proposed", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000010", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.010Z", + "eventType": "run.result.proposed", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "reportedWorkDisposition": "done", + "schema": "paperclip.prp.result.v1", + "status": "succeeded", + "summary": "Phase 3 durable transport recovered without duplicate effects." + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000010", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 10, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.010Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 11, + "sourceEventId": "event_runner_phase3_stable_000011", + "eventType": "run.terminal", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000011", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.011Z", + "eventType": "run.terminal", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "reportedWorkDisposition": "done", + "runTerminalState": "succeeded", + "schema": "paperclip.prp.terminal.v1", + "turnTerminalState": "completed" + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000011", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 11, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.011Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 12, + "sourceEventId": "event_runner_phase3_stable_000012", + "eventType": "runner.stopped", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000012", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.012Z", + "eventType": "runner.stopped", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "afterDurableFlush": true + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000012", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 12, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.012Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 1, + "logicalEffectCount": 1 + } + ] + }, + "runnerState": { + "schema": "paperclip.runner.phase3.state.v1", + "runnerInstanceId": "runner_phase3_stable", + "environmentLeaseId": "environment_lease_phase3_stable", + "runId": "run_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "turnId": "turn_phase3_stable", + "itemId": "item_phase3_stable", + "lifecycle": "stopped", + "nextSourceSeq": 13, + "ackedSourceSeq": 12, + "lastControllerCommandSeq": 5, + "reconnectCount": 0, + "maxOutboxBytes": 65536, + "peakOutboxBytes": 6827, + "outbox": [], + "processedCommands": { + "command_phase3_001": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "commandDigest": "fnv1a64:c19dfbbf86a3dbb4", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "detail": "run prepared", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_002": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "commandDigest": "fnv1a64:b5198f3ce67ce88e", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "detail": "session bound", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_003": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "commandDigest": "fnv1a64:284c1f5778b929c4", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "detail": "harness restarted and reconciled", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_004": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "commandDigest": "fnv1a64:34d91505a30f60fb", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "detail": "turn completed", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_005": { + "commandId": "command_phase3_005", + "controllerSeq": 5, + "commandDigest": "fnv1a64:533318a30fba6d22", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_005", + "controllerSeq": 5, + "detail": "runner will stop after durable flush", + "logicalEffectCount": 1, + "status": "completed" + } + } + }, + "diagnostics": [], + "backpressure": false, + "recoverableFailure": null, + "unrecoverableOutcome": null, + "harnessGeneration": 2, + "stopAfterFlush": true + }, + "commands": [ + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_001", + "controllerSeq": 1, + "type": "run.prepare", + "issuedAt": "2026-08-07T23:30:01.000Z", + "payload": { + "workspace": "phase-03-durable-fixture" + }, + "status": "completed", + "result": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "detail": "run prepared", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_002", + "controllerSeq": 2, + "type": "session.open", + "issuedAt": "2026-08-07T23:30:02.000Z", + "payload": { + "reuse": "same_session" + }, + "status": "completed", + "result": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "detail": "session bound", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_003", + "controllerSeq": 3, + "type": "fault.harness_restart", + "issuedAt": "2026-08-07T23:30:03.000Z", + "payload": {}, + "status": "completed", + "result": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "detail": "harness restarted and reconciled", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_004", + "controllerSeq": 4, + "type": "turn.start", + "issuedAt": "2026-08-07T23:30:04.000Z", + "payload": { + "turnId": "turn_phase3_stable", + "text": "Prove Phase 3 durable recovery." + }, + "status": "completed", + "result": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "detail": "turn completed", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_005", + "controllerSeq": 5, + "type": "runner.shutdown", + "issuedAt": "2026-08-07T23:30:05.000Z", + "payload": {}, + "status": "completed", + "result": { + "commandId": "command_phase3_005", + "controllerSeq": 5, + "detail": "runner will stop after durable flush", + "logicalEffectCount": 1, + "status": "completed" + } + } + ], + "assertions": { + "stableIdentity": true, + "sourceCursorContinuous": true, + "oneLogicalEffectPerAcceptedCommand": true, + "noDuplicateLogicalEvents": true, + "p0Preserved": true, + "boundedStorage": true, + "secretsRedacted": true + } +} diff --git a/packages/paperclip-runner/knowledge/evidence/phase-03-faults/lease-expiry.json b/packages/paperclip-runner/knowledge/evidence/phase-03-faults/lease-expiry.json new file mode 100644 index 00000000000..9e3e834c77d --- /dev/null +++ b/packages/paperclip-runner/knowledge/evidence/phase-03-faults/lease-expiry.json @@ -0,0 +1,603 @@ +{ + "schema": "paperclip.runner.phase3.trace.v1", + "diagnostics": { + "schema": "paperclip.runner.phase3.diagnostics.v1", + "fault": "lease-expiry", + "connection": { + "state": "stopped", + "connectionCount": 2, + "reconnectCount": 2, + "leaseId": "connection_lease_42900de0-4a64-47c1-b20c-d14890a9be42", + "leaseExpiresAt": "2026-08-07T23:36:40.312Z" + }, + "identity": { + "runnerInstanceId": "runner_phase3_stable", + "environmentLeaseId": "environment_lease_phase3_stable", + "runId": "run_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "turnId": "turn_phase3_stable", + "itemId": "item_phase3_stable" + }, + "cursors": { + "runnerAckedSourceSeq": 10, + "runnerNextSourceSeq": 11, + "coreAckedSourceSeq": 10, + "highestCommittedSourceSeq": 10 + }, + "outbox": { + "events": 0, + "bytes": 0, + "peakBytes": 5314, + "maxBytes": 65536, + "backpressure": false, + "p0Committed": 6, + "p0Lost": 0 + }, + "commands": { + "issued": 4, + "completed": 4, + "rejected": 0, + "logicalEffects": 4, + "duplicateDeliveries": 0 + }, + "recovery": { + "replayDeliveries": 9, + "runnerRestarts": 1, + "harnessRestarts": 0, + "malformedFrames": 0, + "freshBootstraps": 2, + "outcome": "recovered", + "reason": "lease_expiry_completed" + }, + "security": { + "bootstrapTicketPersisted": false, + "connectionLeaseTokenPersisted": false, + "secretLeakCount": 0 + }, + "committedEvents": [ + { + "sourceSeq": 1, + "sourceEventId": "event_runner_phase3_stable_000001", + "eventType": "runner.reconciled", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000001", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.001Z", + "eventType": "runner.reconciled", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "outcome": "same_durable_session_resumed", + "runnerInstanceId": "runner_phase3_stable", + "turnId": "turn_phase3_stable" + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000001", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 1, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.001Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 2, + "sourceEventId": "event_runner_phase3_stable_000002", + "eventType": "workspace.ready", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000002", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.002Z", + "eventType": "workspace.ready", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "workspace": "phase-03-durable-fixture" + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000002", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 2, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.002Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 3, + "sourceEventId": "event_runner_phase3_stable_000003", + "eventType": "session.started", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000003", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.003Z", + "eventType": "session.started", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "driverSessionId": "driver_phase3_fake", + "normalizedSessionId": "session_phase3_stable", + "resumed": true + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000003", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 3, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.003Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 4, + "sourceEventId": "event_runner_phase3_stable_000004", + "eventType": "turn.accepted", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000004", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.004Z", + "eventType": "turn.accepted", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "sameSession": true, + "turnId": "turn_phase3_stable" + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000004", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 4, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.004Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 5, + "sourceEventId": "event_runner_phase3_stable_000005", + "eventType": "item.started", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000005", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.005Z", + "eventType": "item.started", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "kind": "assistant_message" + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000005", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 5, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.005Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 6, + "sourceEventId": "event_runner_phase3_stable_000006", + "eventType": "item.delta", + "priority": 2, + "envelope": { + "envelopeId": "envelope_event_000006", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.006Z", + "eventType": "item.delta", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "text": "Prove Phase 3 durable recovery." + }, + "priority": 2, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000006", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 6, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.006Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 7, + "sourceEventId": "event_runner_phase3_stable_000007", + "eventType": "item.completed", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000007", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.007Z", + "eventType": "item.completed", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "text": "Durable recovery completed." + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000007", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 7, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.007Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 8, + "sourceEventId": "event_runner_phase3_stable_000008", + "eventType": "run.result.proposed", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000008", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.008Z", + "eventType": "run.result.proposed", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "reportedWorkDisposition": "done", + "schema": "paperclip.prp.result.v1", + "status": "succeeded", + "summary": "Phase 3 durable transport recovered without duplicate effects." + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000008", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 8, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.008Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 9, + "sourceEventId": "event_runner_phase3_stable_000009", + "eventType": "run.terminal", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000009", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.009Z", + "eventType": "run.terminal", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "reportedWorkDisposition": "done", + "runTerminalState": "succeeded", + "schema": "paperclip.prp.terminal.v1", + "turnTerminalState": "completed" + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000009", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 9, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.009Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 10, + "sourceEventId": "event_runner_phase3_stable_000010", + "eventType": "runner.stopped", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000010", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.010Z", + "eventType": "runner.stopped", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "afterDurableFlush": true + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000010", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 10, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.010Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 1, + "logicalEffectCount": 1 + } + ] + }, + "runnerState": { + "schema": "paperclip.runner.phase3.state.v1", + "runnerInstanceId": "runner_phase3_stable", + "environmentLeaseId": "environment_lease_phase3_stable", + "runId": "run_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "turnId": "turn_phase3_stable", + "itemId": "item_phase3_stable", + "lifecycle": "stopped", + "nextSourceSeq": 11, + "ackedSourceSeq": 10, + "lastControllerCommandSeq": 4, + "reconnectCount": 2, + "maxOutboxBytes": 65536, + "peakOutboxBytes": 5314, + "outbox": [], + "processedCommands": { + "command_phase3_001": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "commandDigest": "fnv1a64:c19dfbbf86a3dbb4", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "detail": "run prepared", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_002": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "commandDigest": "fnv1a64:b5198f3ce67ce88e", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "detail": "session bound", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_003": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "commandDigest": "fnv1a64:fb3b206fe67e1dc8", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "detail": "turn completed", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_004": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "commandDigest": "fnv1a64:d11bf8308f863bad", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "detail": "runner will stop after durable flush", + "logicalEffectCount": 1, + "status": "completed" + } + } + }, + "diagnostics": [ + "WebSocket connection closed: failed to fill whole buffer", + "connection lease expired; a fresh bootstrap may resume this state", + "runner process restored the same durable identity" + ], + "backpressure": false, + "recoverableFailure": null, + "unrecoverableOutcome": null, + "harnessGeneration": 1, + "stopAfterFlush": true + }, + "commands": [ + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_001", + "controllerSeq": 1, + "type": "run.prepare", + "issuedAt": "2026-08-07T23:30:01.000Z", + "payload": { + "workspace": "phase-03-durable-fixture" + }, + "status": "completed", + "result": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "detail": "run prepared", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_002", + "controllerSeq": 2, + "type": "session.open", + "issuedAt": "2026-08-07T23:30:02.000Z", + "payload": { + "reuse": "same_session" + }, + "status": "completed", + "result": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "detail": "session bound", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_003", + "controllerSeq": 3, + "type": "turn.start", + "issuedAt": "2026-08-07T23:30:03.000Z", + "payload": { + "turnId": "turn_phase3_stable", + "text": "Prove Phase 3 durable recovery." + }, + "status": "completed", + "result": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "detail": "turn completed", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_004", + "controllerSeq": 4, + "type": "runner.shutdown", + "issuedAt": "2026-08-07T23:30:04.000Z", + "payload": {}, + "status": "completed", + "result": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "detail": "runner will stop after durable flush", + "logicalEffectCount": 1, + "status": "completed" + } + } + ], + "assertions": { + "stableIdentity": true, + "sourceCursorContinuous": true, + "oneLogicalEffectPerAcceptedCommand": true, + "noDuplicateLogicalEvents": true, + "p0Preserved": true, + "boundedStorage": true, + "secretsRedacted": true + } +} diff --git a/packages/paperclip-runner/knowledge/evidence/phase-03-faults/lost-ack.json b/packages/paperclip-runner/knowledge/evidence/phase-03-faults/lost-ack.json new file mode 100644 index 00000000000..12f74f6211e --- /dev/null +++ b/packages/paperclip-runner/knowledge/evidence/phase-03-faults/lost-ack.json @@ -0,0 +1,561 @@ +{ + "schema": "paperclip.runner.phase3.trace.v1", + "diagnostics": { + "schema": "paperclip.runner.phase3.diagnostics.v1", + "fault": "lost-ack", + "connection": { + "state": "stopped", + "connectionCount": 2, + "reconnectCount": 1, + "leaseId": "connection_lease_db423713-98d6-4a86-940b-ec2f0f6cd6e3", + "leaseExpiresAt": "2026-08-07T23:36:38.759Z" + }, + "identity": { + "runnerInstanceId": "runner_phase3_stable", + "environmentLeaseId": "environment_lease_phase3_stable", + "runId": "run_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "turnId": "turn_phase3_stable", + "itemId": "item_phase3_stable" + }, + "cursors": { + "runnerAckedSourceSeq": 9, + "runnerNextSourceSeq": 10, + "coreAckedSourceSeq": 9, + "highestCommittedSourceSeq": 9 + }, + "outbox": { + "events": 0, + "bytes": 0, + "peakBytes": 6023, + "maxBytes": 65536, + "backpressure": false, + "p0Committed": 5, + "p0Lost": 0 + }, + "commands": { + "issued": 4, + "completed": 4, + "rejected": 0, + "logicalEffects": 4, + "duplicateDeliveries": 1 + }, + "recovery": { + "replayDeliveries": 9, + "runnerRestarts": 0, + "harnessRestarts": 0, + "malformedFrames": 0, + "freshBootstraps": 1, + "outcome": "recovered", + "reason": "lost_ack_completed" + }, + "security": { + "bootstrapTicketPersisted": false, + "connectionLeaseTokenPersisted": false, + "secretLeakCount": 0 + }, + "committedEvents": [ + { + "sourceSeq": 1, + "sourceEventId": "event_runner_phase3_stable_000001", + "eventType": "workspace.ready", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000001", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.001Z", + "eventType": "workspace.ready", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "workspace": "phase-03-durable-fixture" + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000001", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 1, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.001Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 3, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 2, + "sourceEventId": "event_runner_phase3_stable_000002", + "eventType": "session.started", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000002", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.002Z", + "eventType": "session.started", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "driverSessionId": "driver_phase3_fake", + "normalizedSessionId": "session_phase3_stable", + "resumed": false + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000002", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 2, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.002Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 3, + "sourceEventId": "event_runner_phase3_stable_000003", + "eventType": "turn.accepted", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000003", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.003Z", + "eventType": "turn.accepted", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "sameSession": true, + "turnId": "turn_phase3_stable" + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000003", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 3, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.003Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 4, + "sourceEventId": "event_runner_phase3_stable_000004", + "eventType": "item.started", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000004", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.004Z", + "eventType": "item.started", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "kind": "assistant_message" + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000004", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 4, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.004Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 5, + "sourceEventId": "event_runner_phase3_stable_000005", + "eventType": "item.delta", + "priority": 2, + "envelope": { + "envelopeId": "envelope_event_000005", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.005Z", + "eventType": "item.delta", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "text": "Prove Phase 3 durable recovery." + }, + "priority": 2, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000005", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 5, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.005Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 6, + "sourceEventId": "event_runner_phase3_stable_000006", + "eventType": "item.completed", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000006", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.006Z", + "eventType": "item.completed", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "text": "Durable recovery completed." + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000006", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 6, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.006Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 7, + "sourceEventId": "event_runner_phase3_stable_000007", + "eventType": "run.result.proposed", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000007", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.007Z", + "eventType": "run.result.proposed", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "reportedWorkDisposition": "done", + "schema": "paperclip.prp.result.v1", + "status": "succeeded", + "summary": "Phase 3 durable transport recovered without duplicate effects." + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000007", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 7, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.007Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 8, + "sourceEventId": "event_runner_phase3_stable_000008", + "eventType": "run.terminal", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000008", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.008Z", + "eventType": "run.terminal", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "reportedWorkDisposition": "done", + "runTerminalState": "succeeded", + "schema": "paperclip.prp.terminal.v1", + "turnTerminalState": "completed" + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000008", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 8, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.008Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 9, + "sourceEventId": "event_runner_phase3_stable_000009", + "eventType": "runner.stopped", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000009", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.009Z", + "eventType": "runner.stopped", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "afterDurableFlush": true + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000009", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 9, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.009Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 1, + "logicalEffectCount": 1 + } + ] + }, + "runnerState": { + "schema": "paperclip.runner.phase3.state.v1", + "runnerInstanceId": "runner_phase3_stable", + "environmentLeaseId": "environment_lease_phase3_stable", + "runId": "run_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "turnId": "turn_phase3_stable", + "itemId": "item_phase3_stable", + "lifecycle": "stopped", + "nextSourceSeq": 10, + "ackedSourceSeq": 9, + "lastControllerCommandSeq": 4, + "reconnectCount": 1, + "maxOutboxBytes": 65536, + "peakOutboxBytes": 6023, + "outbox": [], + "processedCommands": { + "command_phase3_001": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "commandDigest": "fnv1a64:c19dfbbf86a3dbb4", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "detail": "run prepared", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_002": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "commandDigest": "fnv1a64:b5198f3ce67ce88e", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "detail": "session bound", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_003": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "commandDigest": "fnv1a64:fb3b206fe67e1dc8", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "detail": "turn completed", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_004": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "commandDigest": "fnv1a64:d11bf8308f863bad", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "detail": "runner will stop after durable flush", + "logicalEffectCount": 1, + "status": "completed" + } + } + }, + "diagnostics": [ + "WebSocket frame write failed: Broken pipe (os error 32)" + ], + "backpressure": false, + "recoverableFailure": null, + "unrecoverableOutcome": null, + "harnessGeneration": 1, + "stopAfterFlush": true + }, + "commands": [ + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_001", + "controllerSeq": 1, + "type": "run.prepare", + "issuedAt": "2026-08-07T23:30:01.000Z", + "payload": { + "workspace": "phase-03-durable-fixture" + }, + "status": "completed", + "result": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "detail": "run prepared", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_002", + "controllerSeq": 2, + "type": "session.open", + "issuedAt": "2026-08-07T23:30:02.000Z", + "payload": { + "reuse": "same_session" + }, + "status": "completed", + "result": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "detail": "session bound", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_003", + "controllerSeq": 3, + "type": "turn.start", + "issuedAt": "2026-08-07T23:30:03.000Z", + "payload": { + "turnId": "turn_phase3_stable", + "text": "Prove Phase 3 durable recovery." + }, + "status": "completed", + "result": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "detail": "turn completed", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_004", + "controllerSeq": 4, + "type": "runner.shutdown", + "issuedAt": "2026-08-07T23:30:04.000Z", + "payload": {}, + "status": "completed", + "result": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "detail": "runner will stop after durable flush", + "logicalEffectCount": 1, + "status": "completed" + } + } + ], + "assertions": { + "stableIdentity": true, + "sourceCursorContinuous": true, + "oneLogicalEffectPerAcceptedCommand": true, + "noDuplicateLogicalEvents": true, + "p0Preserved": true, + "boundedStorage": true, + "secretsRedacted": true + } +} diff --git a/packages/paperclip-runner/knowledge/evidence/phase-03-faults/malformed-input.json b/packages/paperclip-runner/knowledge/evidence/phase-03-faults/malformed-input.json new file mode 100644 index 00000000000..46a4feb6e87 --- /dev/null +++ b/packages/paperclip-runner/knowledge/evidence/phase-03-faults/malformed-input.json @@ -0,0 +1,561 @@ +{ + "schema": "paperclip.runner.phase3.trace.v1", + "diagnostics": { + "schema": "paperclip.runner.phase3.diagnostics.v1", + "fault": "malformed-input", + "connection": { + "state": "stopped", + "connectionCount": 2, + "reconnectCount": 1, + "leaseId": "connection_lease_51b9ac50-5fdc-4f68-a289-6e6e76711bbf", + "leaseExpiresAt": "2026-08-07T23:36:39.957Z" + }, + "identity": { + "runnerInstanceId": "runner_phase3_stable", + "environmentLeaseId": "environment_lease_phase3_stable", + "runId": "run_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "turnId": "turn_phase3_stable", + "itemId": "item_phase3_stable" + }, + "cursors": { + "runnerAckedSourceSeq": 9, + "runnerNextSourceSeq": 10, + "coreAckedSourceSeq": 9, + "highestCommittedSourceSeq": 9 + }, + "outbox": { + "events": 0, + "bytes": 0, + "peakBytes": 5314, + "maxBytes": 65536, + "backpressure": false, + "p0Committed": 5, + "p0Lost": 0 + }, + "commands": { + "issued": 4, + "completed": 4, + "rejected": 0, + "logicalEffects": 4, + "duplicateDeliveries": 0 + }, + "recovery": { + "replayDeliveries": 8, + "runnerRestarts": 0, + "harnessRestarts": 0, + "malformedFrames": 1, + "freshBootstraps": 1, + "outcome": "recovered", + "reason": "malformed_input_completed" + }, + "security": { + "bootstrapTicketPersisted": false, + "connectionLeaseTokenPersisted": false, + "secretLeakCount": 0 + }, + "committedEvents": [ + { + "sourceSeq": 1, + "sourceEventId": "event_runner_phase3_stable_000001", + "eventType": "workspace.ready", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000001", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.001Z", + "eventType": "workspace.ready", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "workspace": "phase-03-durable-fixture" + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000001", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 1, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.001Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 2, + "sourceEventId": "event_runner_phase3_stable_000002", + "eventType": "session.started", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000002", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.002Z", + "eventType": "session.started", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "driverSessionId": "driver_phase3_fake", + "normalizedSessionId": "session_phase3_stable", + "resumed": true + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000002", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 2, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.002Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 3, + "sourceEventId": "event_runner_phase3_stable_000003", + "eventType": "turn.accepted", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000003", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.003Z", + "eventType": "turn.accepted", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "sameSession": true, + "turnId": "turn_phase3_stable" + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000003", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 3, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.003Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 4, + "sourceEventId": "event_runner_phase3_stable_000004", + "eventType": "item.started", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000004", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.004Z", + "eventType": "item.started", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "kind": "assistant_message" + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000004", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 4, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.004Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 5, + "sourceEventId": "event_runner_phase3_stable_000005", + "eventType": "item.delta", + "priority": 2, + "envelope": { + "envelopeId": "envelope_event_000005", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.005Z", + "eventType": "item.delta", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "text": "Prove Phase 3 durable recovery." + }, + "priority": 2, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000005", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 5, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.005Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 6, + "sourceEventId": "event_runner_phase3_stable_000006", + "eventType": "item.completed", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000006", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.006Z", + "eventType": "item.completed", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "text": "Durable recovery completed." + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000006", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 6, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.006Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 7, + "sourceEventId": "event_runner_phase3_stable_000007", + "eventType": "run.result.proposed", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000007", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.007Z", + "eventType": "run.result.proposed", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "reportedWorkDisposition": "done", + "schema": "paperclip.prp.result.v1", + "status": "succeeded", + "summary": "Phase 3 durable transport recovered without duplicate effects." + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000007", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 7, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.007Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 8, + "sourceEventId": "event_runner_phase3_stable_000008", + "eventType": "run.terminal", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000008", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.008Z", + "eventType": "run.terminal", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "reportedWorkDisposition": "done", + "runTerminalState": "succeeded", + "schema": "paperclip.prp.terminal.v1", + "turnTerminalState": "completed" + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000008", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 8, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.008Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 9, + "sourceEventId": "event_runner_phase3_stable_000009", + "eventType": "runner.stopped", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000009", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.009Z", + "eventType": "runner.stopped", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "afterDurableFlush": true + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000009", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 9, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.009Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 1, + "logicalEffectCount": 1 + } + ] + }, + "runnerState": { + "schema": "paperclip.runner.phase3.state.v1", + "runnerInstanceId": "runner_phase3_stable", + "environmentLeaseId": "environment_lease_phase3_stable", + "runId": "run_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "turnId": "turn_phase3_stable", + "itemId": "item_phase3_stable", + "lifecycle": "stopped", + "nextSourceSeq": 10, + "ackedSourceSeq": 9, + "lastControllerCommandSeq": 4, + "reconnectCount": 1, + "maxOutboxBytes": 65536, + "peakOutboxBytes": 5314, + "outbox": [], + "processedCommands": { + "command_phase3_001": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "commandDigest": "fnv1a64:c19dfbbf86a3dbb4", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "detail": "run prepared", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_002": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "commandDigest": "fnv1a64:b5198f3ce67ce88e", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "detail": "session bound", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_003": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "commandDigest": "fnv1a64:fb3b206fe67e1dc8", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "detail": "turn completed", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_004": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "commandDigest": "fnv1a64:d11bf8308f863bad", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "detail": "runner will stop after durable flush", + "logicalEffectCount": 1, + "status": "completed" + } + } + }, + "diagnostics": [ + "malformed WebSocket JSON: EOF while parsing a value at line 1 column 8" + ], + "backpressure": false, + "recoverableFailure": null, + "unrecoverableOutcome": null, + "harnessGeneration": 1, + "stopAfterFlush": true + }, + "commands": [ + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_001", + "controllerSeq": 1, + "type": "run.prepare", + "issuedAt": "2026-08-07T23:30:01.000Z", + "payload": { + "workspace": "phase-03-durable-fixture" + }, + "status": "completed", + "result": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "detail": "run prepared", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_002", + "controllerSeq": 2, + "type": "session.open", + "issuedAt": "2026-08-07T23:30:02.000Z", + "payload": { + "reuse": "same_session" + }, + "status": "completed", + "result": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "detail": "session bound", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_003", + "controllerSeq": 3, + "type": "turn.start", + "issuedAt": "2026-08-07T23:30:03.000Z", + "payload": { + "turnId": "turn_phase3_stable", + "text": "Prove Phase 3 durable recovery." + }, + "status": "completed", + "result": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "detail": "turn completed", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_004", + "controllerSeq": 4, + "type": "runner.shutdown", + "issuedAt": "2026-08-07T23:30:04.000Z", + "payload": {}, + "status": "completed", + "result": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "detail": "runner will stop after durable flush", + "logicalEffectCount": 1, + "status": "completed" + } + } + ], + "assertions": { + "stableIdentity": true, + "sourceCursorContinuous": true, + "oneLogicalEffectPerAcceptedCommand": true, + "noDuplicateLogicalEvents": true, + "p0Preserved": true, + "boundedStorage": true, + "secretsRedacted": true + } +} diff --git a/packages/paperclip-runner/knowledge/evidence/phase-03-faults/none.json b/packages/paperclip-runner/knowledge/evidence/phase-03-faults/none.json new file mode 100644 index 00000000000..ef42c9b62cc --- /dev/null +++ b/packages/paperclip-runner/knowledge/evidence/phase-03-faults/none.json @@ -0,0 +1,559 @@ +{ + "schema": "paperclip.runner.phase3.trace.v1", + "diagnostics": { + "schema": "paperclip.runner.phase3.diagnostics.v1", + "fault": "none", + "connection": { + "state": "stopped", + "connectionCount": 1, + "reconnectCount": 0, + "leaseId": "connection_lease_469ad2f3-1655-434c-a12b-85718229ea67", + "leaseExpiresAt": "2026-08-07T23:36:38.211Z" + }, + "identity": { + "runnerInstanceId": "runner_phase3_stable", + "environmentLeaseId": "environment_lease_phase3_stable", + "runId": "run_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "turnId": "turn_phase3_stable", + "itemId": "item_phase3_stable" + }, + "cursors": { + "runnerAckedSourceSeq": 9, + "runnerNextSourceSeq": 10, + "coreAckedSourceSeq": 9, + "highestCommittedSourceSeq": 9 + }, + "outbox": { + "events": 0, + "bytes": 0, + "peakBytes": 5315, + "maxBytes": 65536, + "backpressure": false, + "p0Committed": 5, + "p0Lost": 0 + }, + "commands": { + "issued": 4, + "completed": 4, + "rejected": 0, + "logicalEffects": 4, + "duplicateDeliveries": 0 + }, + "recovery": { + "replayDeliveries": 8, + "runnerRestarts": 0, + "harnessRestarts": 0, + "malformedFrames": 0, + "freshBootstraps": 1, + "outcome": "recovered", + "reason": "none_completed" + }, + "security": { + "bootstrapTicketPersisted": false, + "connectionLeaseTokenPersisted": false, + "secretLeakCount": 0 + }, + "committedEvents": [ + { + "sourceSeq": 1, + "sourceEventId": "event_runner_phase3_stable_000001", + "eventType": "workspace.ready", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000001", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.001Z", + "eventType": "workspace.ready", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "workspace": "phase-03-durable-fixture" + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000001", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 1, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.001Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 2, + "sourceEventId": "event_runner_phase3_stable_000002", + "eventType": "session.started", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000002", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.002Z", + "eventType": "session.started", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "driverSessionId": "driver_phase3_fake", + "normalizedSessionId": "session_phase3_stable", + "resumed": false + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000002", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 2, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.002Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 3, + "sourceEventId": "event_runner_phase3_stable_000003", + "eventType": "turn.accepted", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000003", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.003Z", + "eventType": "turn.accepted", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "sameSession": true, + "turnId": "turn_phase3_stable" + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000003", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 3, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.003Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 4, + "sourceEventId": "event_runner_phase3_stable_000004", + "eventType": "item.started", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000004", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.004Z", + "eventType": "item.started", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "kind": "assistant_message" + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000004", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 4, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.004Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 5, + "sourceEventId": "event_runner_phase3_stable_000005", + "eventType": "item.delta", + "priority": 2, + "envelope": { + "envelopeId": "envelope_event_000005", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.005Z", + "eventType": "item.delta", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "text": "Prove Phase 3 durable recovery." + }, + "priority": 2, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000005", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 5, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.005Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 6, + "sourceEventId": "event_runner_phase3_stable_000006", + "eventType": "item.completed", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000006", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.006Z", + "eventType": "item.completed", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "text": "Durable recovery completed." + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000006", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 6, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.006Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 7, + "sourceEventId": "event_runner_phase3_stable_000007", + "eventType": "run.result.proposed", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000007", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.007Z", + "eventType": "run.result.proposed", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "reportedWorkDisposition": "done", + "schema": "paperclip.prp.result.v1", + "status": "succeeded", + "summary": "Phase 3 durable transport recovered without duplicate effects." + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000007", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 7, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.007Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 8, + "sourceEventId": "event_runner_phase3_stable_000008", + "eventType": "run.terminal", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000008", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.008Z", + "eventType": "run.terminal", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "reportedWorkDisposition": "done", + "runTerminalState": "succeeded", + "schema": "paperclip.prp.terminal.v1", + "turnTerminalState": "completed" + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000008", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 8, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.008Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 9, + "sourceEventId": "event_runner_phase3_stable_000009", + "eventType": "runner.stopped", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000009", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.009Z", + "eventType": "runner.stopped", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "afterDurableFlush": true + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000009", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 9, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.009Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 1, + "logicalEffectCount": 1 + } + ] + }, + "runnerState": { + "schema": "paperclip.runner.phase3.state.v1", + "runnerInstanceId": "runner_phase3_stable", + "environmentLeaseId": "environment_lease_phase3_stable", + "runId": "run_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "turnId": "turn_phase3_stable", + "itemId": "item_phase3_stable", + "lifecycle": "stopped", + "nextSourceSeq": 10, + "ackedSourceSeq": 9, + "lastControllerCommandSeq": 4, + "reconnectCount": 0, + "maxOutboxBytes": 65536, + "peakOutboxBytes": 5315, + "outbox": [], + "processedCommands": { + "command_phase3_001": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "commandDigest": "fnv1a64:c19dfbbf86a3dbb4", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "detail": "run prepared", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_002": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "commandDigest": "fnv1a64:b5198f3ce67ce88e", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "detail": "session bound", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_003": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "commandDigest": "fnv1a64:fb3b206fe67e1dc8", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "detail": "turn completed", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_004": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "commandDigest": "fnv1a64:d11bf8308f863bad", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "detail": "runner will stop after durable flush", + "logicalEffectCount": 1, + "status": "completed" + } + } + }, + "diagnostics": [], + "backpressure": false, + "recoverableFailure": null, + "unrecoverableOutcome": null, + "harnessGeneration": 1, + "stopAfterFlush": true + }, + "commands": [ + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_001", + "controllerSeq": 1, + "type": "run.prepare", + "issuedAt": "2026-08-07T23:30:01.000Z", + "payload": { + "workspace": "phase-03-durable-fixture" + }, + "status": "completed", + "result": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "detail": "run prepared", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_002", + "controllerSeq": 2, + "type": "session.open", + "issuedAt": "2026-08-07T23:30:02.000Z", + "payload": { + "reuse": "same_session" + }, + "status": "completed", + "result": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "detail": "session bound", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_003", + "controllerSeq": 3, + "type": "turn.start", + "issuedAt": "2026-08-07T23:30:03.000Z", + "payload": { + "turnId": "turn_phase3_stable", + "text": "Prove Phase 3 durable recovery." + }, + "status": "completed", + "result": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "detail": "turn completed", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_004", + "controllerSeq": 4, + "type": "runner.shutdown", + "issuedAt": "2026-08-07T23:30:04.000Z", + "payload": {}, + "status": "completed", + "result": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "detail": "runner will stop after durable flush", + "logicalEffectCount": 1, + "status": "completed" + } + } + ], + "assertions": { + "stableIdentity": true, + "sourceCursorContinuous": true, + "oneLogicalEffectPerAcceptedCommand": true, + "noDuplicateLogicalEvents": true, + "p0Preserved": true, + "boundedStorage": true, + "secretsRedacted": true + } +} diff --git a/packages/paperclip-runner/knowledge/evidence/phase-03-faults/revoke.json b/packages/paperclip-runner/knowledge/evidence/phase-03-faults/revoke.json new file mode 100644 index 00000000000..444ac734a27 --- /dev/null +++ b/packages/paperclip-runner/knowledge/evidence/phase-03-faults/revoke.json @@ -0,0 +1,528 @@ +{ + "schema": "paperclip.runner.phase3.trace.v1", + "diagnostics": { + "schema": "paperclip.runner.phase3.diagnostics.v1", + "fault": "revoke", + "connection": { + "state": "revoked", + "connectionCount": 1, + "reconnectCount": 0, + "leaseId": "connection_lease_8fe123ed-6a53-42e1-bdc7-315a93568cbd", + "leaseExpiresAt": "2026-08-07T23:36:41.005Z" + }, + "identity": { + "runnerInstanceId": "runner_phase3_stable", + "environmentLeaseId": "environment_lease_phase3_stable", + "runId": "run_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "turnId": "turn_phase3_stable", + "itemId": "item_phase3_stable" + }, + "cursors": { + "runnerAckedSourceSeq": 8, + "runnerNextSourceSeq": 9, + "coreAckedSourceSeq": 8, + "highestCommittedSourceSeq": 8 + }, + "outbox": { + "events": 0, + "bytes": 0, + "peakBytes": 5315, + "maxBytes": 65536, + "backpressure": false, + "p0Committed": 4, + "p0Lost": 0 + }, + "commands": { + "issued": 4, + "completed": 3, + "rejected": 1, + "logicalEffects": 3, + "duplicateDeliveries": 0 + }, + "recovery": { + "replayDeliveries": 4, + "runnerRestarts": 0, + "harnessRestarts": 0, + "malformedFrames": 0, + "freshBootstraps": 1, + "outcome": "revoked", + "reason": "revoke_completed" + }, + "security": { + "bootstrapTicketPersisted": false, + "connectionLeaseTokenPersisted": false, + "secretLeakCount": 0 + }, + "committedEvents": [ + { + "sourceSeq": 1, + "sourceEventId": "event_runner_phase3_stable_000001", + "eventType": "workspace.ready", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000001", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.001Z", + "eventType": "workspace.ready", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "workspace": "phase-03-durable-fixture" + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000001", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 1, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.001Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 2, + "sourceEventId": "event_runner_phase3_stable_000002", + "eventType": "session.started", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000002", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.002Z", + "eventType": "session.started", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "driverSessionId": "driver_phase3_fake", + "normalizedSessionId": "session_phase3_stable", + "resumed": false + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000002", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 2, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.002Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 3, + "sourceEventId": "event_runner_phase3_stable_000003", + "eventType": "turn.accepted", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000003", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.003Z", + "eventType": "turn.accepted", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "sameSession": true, + "turnId": "turn_phase3_stable" + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000003", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 3, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.003Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 1, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 4, + "sourceEventId": "event_runner_phase3_stable_000004", + "eventType": "item.started", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000004", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.004Z", + "eventType": "item.started", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "kind": "assistant_message" + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000004", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 4, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.004Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 1, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 5, + "sourceEventId": "event_runner_phase3_stable_000005", + "eventType": "item.delta", + "priority": 2, + "envelope": { + "envelopeId": "envelope_event_000005", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.005Z", + "eventType": "item.delta", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "text": "Prove Phase 3 durable recovery." + }, + "priority": 2, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000005", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 5, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.005Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 1, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 6, + "sourceEventId": "event_runner_phase3_stable_000006", + "eventType": "item.completed", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000006", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.006Z", + "eventType": "item.completed", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "text": "Durable recovery completed." + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000006", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 6, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.006Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 1, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 7, + "sourceEventId": "event_runner_phase3_stable_000007", + "eventType": "run.result.proposed", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000007", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.007Z", + "eventType": "run.result.proposed", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "reportedWorkDisposition": "done", + "schema": "paperclip.prp.result.v1", + "status": "succeeded", + "summary": "Phase 3 durable transport recovered without duplicate effects." + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000007", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 7, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.007Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 1, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 8, + "sourceEventId": "event_runner_phase3_stable_000008", + "eventType": "run.terminal", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000008", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.008Z", + "eventType": "run.terminal", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "reportedWorkDisposition": "done", + "runTerminalState": "succeeded", + "schema": "paperclip.prp.terminal.v1", + "turnTerminalState": "completed" + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000008", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 8, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.008Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 3, + "logicalEffectCount": 1 + } + ] + }, + "runnerState": { + "schema": "paperclip.runner.phase3.state.v1", + "runnerInstanceId": "runner_phase3_stable", + "environmentLeaseId": "environment_lease_phase3_stable", + "runId": "run_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "turnId": "turn_phase3_stable", + "itemId": "item_phase3_stable", + "lifecycle": "revoked", + "nextSourceSeq": 9, + "ackedSourceSeq": 8, + "lastControllerCommandSeq": 4, + "reconnectCount": 0, + "maxOutboxBytes": 65536, + "peakOutboxBytes": 5315, + "outbox": [], + "processedCommands": { + "command_after_revoke": { + "commandId": "command_after_revoke", + "controllerSeq": 4, + "commandDigest": "fnv1a64:fd10e64f32894219", + "status": "rejected", + "logicalEffectCount": 0, + "result": { + "commandId": "command_after_revoke", + "controllerSeq": 4, + "detail": "connection capability is revoked; commands have no logical effect", + "logicalEffectCount": 0, + "status": "rejected" + } + }, + "command_phase3_001": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "commandDigest": "fnv1a64:c19dfbbf86a3dbb4", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "detail": "run prepared", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_002": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "commandDigest": "fnv1a64:b5198f3ce67ce88e", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "detail": "session bound", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_003": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "commandDigest": "fnv1a64:fb3b206fe67e1dc8", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "detail": "turn completed", + "logicalEffectCount": 1, + "status": "completed" + } + } + }, + "diagnostics": [ + "connection lease revoked; flushing durable events" + ], + "backpressure": false, + "recoverableFailure": null, + "unrecoverableOutcome": null, + "harnessGeneration": 1, + "stopAfterFlush": true + }, + "commands": [ + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_001", + "controllerSeq": 1, + "type": "run.prepare", + "issuedAt": "2026-08-07T23:30:01.000Z", + "payload": { + "workspace": "phase-03-durable-fixture" + }, + "status": "completed", + "result": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "detail": "run prepared", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_002", + "controllerSeq": 2, + "type": "session.open", + "issuedAt": "2026-08-07T23:30:02.000Z", + "payload": { + "reuse": "same_session" + }, + "status": "completed", + "result": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "detail": "session bound", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_003", + "controllerSeq": 3, + "type": "turn.start", + "issuedAt": "2026-08-07T23:30:03.000Z", + "payload": { + "turnId": "turn_phase3_stable", + "text": "Prove Phase 3 durable recovery." + }, + "status": "completed", + "result": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "detail": "turn completed", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_after_revoke", + "controllerSeq": 4, + "type": "turn.start", + "issuedAt": "2026-08-07T23:30:04.000Z", + "payload": { + "turnId": "turn_phase3_stable", + "text": "must be rejected after revoke" + }, + "status": "rejected", + "result": { + "commandId": "command_after_revoke", + "controllerSeq": 4, + "detail": "connection capability is revoked; commands have no logical effect", + "logicalEffectCount": 0, + "status": "rejected" + } + } + ], + "assertions": { + "stableIdentity": true, + "sourceCursorContinuous": true, + "oneLogicalEffectPerAcceptedCommand": true, + "noDuplicateLogicalEvents": true, + "p0Preserved": true, + "boundedStorage": true, + "secretsRedacted": true + } +} diff --git a/packages/paperclip-runner/knowledge/evidence/phase-03-faults/runner-restart.json b/packages/paperclip-runner/knowledge/evidence/phase-03-faults/runner-restart.json new file mode 100644 index 00000000000..3e751401863 --- /dev/null +++ b/packages/paperclip-runner/knowledge/evidence/phase-03-faults/runner-restart.json @@ -0,0 +1,601 @@ +{ + "schema": "paperclip.runner.phase3.trace.v1", + "diagnostics": { + "schema": "paperclip.runner.phase3.diagnostics.v1", + "fault": "runner-restart", + "connection": { + "state": "stopped", + "connectionCount": 2, + "reconnectCount": 1, + "leaseId": "connection_lease_dcd257e4-0a30-4bb8-8674-0446c617e29f", + "leaseExpiresAt": "2026-08-07T23:36:39.391Z" + }, + "identity": { + "runnerInstanceId": "runner_phase3_stable", + "environmentLeaseId": "environment_lease_phase3_stable", + "runId": "run_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "turnId": "turn_phase3_stable", + "itemId": "item_phase3_stable" + }, + "cursors": { + "runnerAckedSourceSeq": 10, + "runnerNextSourceSeq": 11, + "coreAckedSourceSeq": 10, + "highestCommittedSourceSeq": 10 + }, + "outbox": { + "events": 0, + "bytes": 0, + "peakBytes": 6883, + "maxBytes": 65536, + "backpressure": false, + "p0Committed": 6, + "p0Lost": 0 + }, + "commands": { + "issued": 4, + "completed": 4, + "rejected": 0, + "logicalEffects": 4, + "duplicateDeliveries": 1 + }, + "recovery": { + "replayDeliveries": 10, + "runnerRestarts": 1, + "harnessRestarts": 0, + "malformedFrames": 0, + "freshBootstraps": 2, + "outcome": "recovered", + "reason": "runner_restart_completed" + }, + "security": { + "bootstrapTicketPersisted": false, + "connectionLeaseTokenPersisted": false, + "secretLeakCount": 0 + }, + "committedEvents": [ + { + "sourceSeq": 1, + "sourceEventId": "event_runner_phase3_stable_000001", + "eventType": "workspace.ready", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000001", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.001Z", + "eventType": "workspace.ready", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "workspace": "phase-03-durable-fixture" + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000001", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 1, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.001Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 3, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 2, + "sourceEventId": "event_runner_phase3_stable_000002", + "eventType": "session.started", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000002", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.002Z", + "eventType": "session.started", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "driverSessionId": "driver_phase3_fake", + "normalizedSessionId": "session_phase3_stable", + "resumed": false + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000002", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 2, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.002Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 3, + "sourceEventId": "event_runner_phase3_stable_000003", + "eventType": "runner.reconciled", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000003", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.003Z", + "eventType": "runner.reconciled", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "outcome": "same_durable_session_resumed", + "runnerInstanceId": "runner_phase3_stable", + "turnId": "turn_phase3_stable" + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000003", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 3, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.003Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 4, + "sourceEventId": "event_runner_phase3_stable_000004", + "eventType": "turn.accepted", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000004", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.004Z", + "eventType": "turn.accepted", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "sameSession": true, + "turnId": "turn_phase3_stable" + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000004", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 4, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.004Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 5, + "sourceEventId": "event_runner_phase3_stable_000005", + "eventType": "item.started", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000005", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.005Z", + "eventType": "item.started", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "kind": "assistant_message" + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000005", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 5, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.005Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 6, + "sourceEventId": "event_runner_phase3_stable_000006", + "eventType": "item.delta", + "priority": 2, + "envelope": { + "envelopeId": "envelope_event_000006", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.006Z", + "eventType": "item.delta", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "text": "Prove Phase 3 durable recovery." + }, + "priority": 2, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000006", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 6, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.006Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 7, + "sourceEventId": "event_runner_phase3_stable_000007", + "eventType": "item.completed", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000007", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.007Z", + "eventType": "item.completed", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "text": "Durable recovery completed." + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000007", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 7, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.007Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 8, + "sourceEventId": "event_runner_phase3_stable_000008", + "eventType": "run.result.proposed", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000008", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.008Z", + "eventType": "run.result.proposed", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "reportedWorkDisposition": "done", + "schema": "paperclip.prp.result.v1", + "status": "succeeded", + "summary": "Phase 3 durable transport recovered without duplicate effects." + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000008", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 8, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.008Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 9, + "sourceEventId": "event_runner_phase3_stable_000009", + "eventType": "run.terminal", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000009", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.009Z", + "eventType": "run.terminal", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "reportedWorkDisposition": "done", + "runTerminalState": "succeeded", + "schema": "paperclip.prp.terminal.v1", + "turnTerminalState": "completed" + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000009", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 9, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.009Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 10, + "sourceEventId": "event_runner_phase3_stable_000010", + "eventType": "runner.stopped", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000010", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.010Z", + "eventType": "runner.stopped", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "afterDurableFlush": true + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000010", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 10, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.010Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 1, + "logicalEffectCount": 1 + } + ] + }, + "runnerState": { + "schema": "paperclip.runner.phase3.state.v1", + "runnerInstanceId": "runner_phase3_stable", + "environmentLeaseId": "environment_lease_phase3_stable", + "runId": "run_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "turnId": "turn_phase3_stable", + "itemId": "item_phase3_stable", + "lifecycle": "stopped", + "nextSourceSeq": 11, + "ackedSourceSeq": 10, + "lastControllerCommandSeq": 4, + "reconnectCount": 1, + "maxOutboxBytes": 65536, + "peakOutboxBytes": 6883, + "outbox": [], + "processedCommands": { + "command_phase3_001": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "commandDigest": "fnv1a64:c19dfbbf86a3dbb4", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "detail": "run prepared", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_002": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "commandDigest": "fnv1a64:b5198f3ce67ce88e", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "detail": "session bound", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_003": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "commandDigest": "fnv1a64:fb3b206fe67e1dc8", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "detail": "turn completed", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_004": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "commandDigest": "fnv1a64:d11bf8308f863bad", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "detail": "runner will stop after durable flush", + "logicalEffectCount": 1, + "status": "completed" + } + } + }, + "diagnostics": [ + "runner process restored the same durable identity" + ], + "backpressure": false, + "recoverableFailure": null, + "unrecoverableOutcome": null, + "harnessGeneration": 1, + "stopAfterFlush": true + }, + "commands": [ + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_001", + "controllerSeq": 1, + "type": "run.prepare", + "issuedAt": "2026-08-07T23:30:01.000Z", + "payload": { + "workspace": "phase-03-durable-fixture" + }, + "status": "completed", + "result": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "detail": "run prepared", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_002", + "controllerSeq": 2, + "type": "session.open", + "issuedAt": "2026-08-07T23:30:02.000Z", + "payload": { + "reuse": "same_session" + }, + "status": "completed", + "result": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "detail": "session bound", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_003", + "controllerSeq": 3, + "type": "turn.start", + "issuedAt": "2026-08-07T23:30:03.000Z", + "payload": { + "turnId": "turn_phase3_stable", + "text": "Prove Phase 3 durable recovery." + }, + "status": "completed", + "result": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "detail": "turn completed", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_004", + "controllerSeq": 4, + "type": "runner.shutdown", + "issuedAt": "2026-08-07T23:30:04.000Z", + "payload": {}, + "status": "completed", + "result": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "detail": "runner will stop after durable flush", + "logicalEffectCount": 1, + "status": "completed" + } + } + ], + "assertions": { + "stableIdentity": true, + "sourceCursorContinuous": true, + "oneLogicalEffectPerAcceptedCommand": true, + "noDuplicateLogicalEvents": true, + "p0Preserved": true, + "boundedStorage": true, + "secretsRedacted": true + } +} diff --git a/packages/paperclip-runner/knowledge/evidence/phase-03-faults/socket-drop.json b/packages/paperclip-runner/knowledge/evidence/phase-03-faults/socket-drop.json new file mode 100644 index 00000000000..7133c0466f9 --- /dev/null +++ b/packages/paperclip-runner/knowledge/evidence/phase-03-faults/socket-drop.json @@ -0,0 +1,561 @@ +{ + "schema": "paperclip.runner.phase3.trace.v1", + "diagnostics": { + "schema": "paperclip.runner.phase3.diagnostics.v1", + "fault": "socket-drop", + "connection": { + "state": "stopped", + "connectionCount": 2, + "reconnectCount": 1, + "leaseId": "connection_lease_efd9f53a-cc91-4232-8ec8-1a5d5963dad4", + "leaseExpiresAt": "2026-08-07T23:36:38.467Z" + }, + "identity": { + "runnerInstanceId": "runner_phase3_stable", + "environmentLeaseId": "environment_lease_phase3_stable", + "runId": "run_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "turnId": "turn_phase3_stable", + "itemId": "item_phase3_stable" + }, + "cursors": { + "runnerAckedSourceSeq": 9, + "runnerNextSourceSeq": 10, + "coreAckedSourceSeq": 9, + "highestCommittedSourceSeq": 9 + }, + "outbox": { + "events": 0, + "bytes": 0, + "peakBytes": 5314, + "maxBytes": 65536, + "backpressure": false, + "p0Committed": 5, + "p0Lost": 0 + }, + "commands": { + "issued": 4, + "completed": 4, + "rejected": 0, + "logicalEffects": 4, + "duplicateDeliveries": 0 + }, + "recovery": { + "replayDeliveries": 8, + "runnerRestarts": 0, + "harnessRestarts": 0, + "malformedFrames": 0, + "freshBootstraps": 1, + "outcome": "recovered", + "reason": "socket_drop_completed" + }, + "security": { + "bootstrapTicketPersisted": false, + "connectionLeaseTokenPersisted": false, + "secretLeakCount": 0 + }, + "committedEvents": [ + { + "sourceSeq": 1, + "sourceEventId": "event_runner_phase3_stable_000001", + "eventType": "workspace.ready", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000001", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.001Z", + "eventType": "workspace.ready", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "workspace": "phase-03-durable-fixture" + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000001", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 1, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.001Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 2, + "sourceEventId": "event_runner_phase3_stable_000002", + "eventType": "session.started", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000002", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.002Z", + "eventType": "session.started", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "driverSessionId": "driver_phase3_fake", + "normalizedSessionId": "session_phase3_stable", + "resumed": true + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000002", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 2, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.002Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 3, + "sourceEventId": "event_runner_phase3_stable_000003", + "eventType": "turn.accepted", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000003", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.003Z", + "eventType": "turn.accepted", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "sameSession": true, + "turnId": "turn_phase3_stable" + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000003", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 3, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.003Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 4, + "sourceEventId": "event_runner_phase3_stable_000004", + "eventType": "item.started", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000004", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.004Z", + "eventType": "item.started", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "kind": "assistant_message" + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000004", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 4, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.004Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 5, + "sourceEventId": "event_runner_phase3_stable_000005", + "eventType": "item.delta", + "priority": 2, + "envelope": { + "envelopeId": "envelope_event_000005", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.005Z", + "eventType": "item.delta", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "text": "Prove Phase 3 durable recovery." + }, + "priority": 2, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000005", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 5, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.005Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 6, + "sourceEventId": "event_runner_phase3_stable_000006", + "eventType": "item.completed", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000006", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.006Z", + "eventType": "item.completed", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "text": "Durable recovery completed." + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000006", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 6, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.006Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 7, + "sourceEventId": "event_runner_phase3_stable_000007", + "eventType": "run.result.proposed", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000007", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.007Z", + "eventType": "run.result.proposed", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "reportedWorkDisposition": "done", + "schema": "paperclip.prp.result.v1", + "status": "succeeded", + "summary": "Phase 3 durable transport recovered without duplicate effects." + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000007", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 7, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.007Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 8, + "sourceEventId": "event_runner_phase3_stable_000008", + "eventType": "run.terminal", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000008", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.008Z", + "eventType": "run.terminal", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "reportedWorkDisposition": "done", + "runTerminalState": "succeeded", + "schema": "paperclip.prp.terminal.v1", + "turnTerminalState": "completed" + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000008", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 8, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.008Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 9, + "sourceEventId": "event_runner_phase3_stable_000009", + "eventType": "runner.stopped", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000009", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.009Z", + "eventType": "runner.stopped", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "afterDurableFlush": true + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000009", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 9, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.009Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 1, + "logicalEffectCount": 1 + } + ] + }, + "runnerState": { + "schema": "paperclip.runner.phase3.state.v1", + "runnerInstanceId": "runner_phase3_stable", + "environmentLeaseId": "environment_lease_phase3_stable", + "runId": "run_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "turnId": "turn_phase3_stable", + "itemId": "item_phase3_stable", + "lifecycle": "stopped", + "nextSourceSeq": 10, + "ackedSourceSeq": 9, + "lastControllerCommandSeq": 4, + "reconnectCount": 1, + "maxOutboxBytes": 65536, + "peakOutboxBytes": 5314, + "outbox": [], + "processedCommands": { + "command_phase3_001": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "commandDigest": "fnv1a64:c19dfbbf86a3dbb4", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "detail": "run prepared", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_002": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "commandDigest": "fnv1a64:b5198f3ce67ce88e", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "detail": "session bound", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_003": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "commandDigest": "fnv1a64:fb3b206fe67e1dc8", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "detail": "turn completed", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_004": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "commandDigest": "fnv1a64:d11bf8308f863bad", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "detail": "runner will stop after durable flush", + "logicalEffectCount": 1, + "status": "completed" + } + } + }, + "diagnostics": [ + "WebSocket connection closed: failed to fill whole buffer" + ], + "backpressure": false, + "recoverableFailure": null, + "unrecoverableOutcome": null, + "harnessGeneration": 1, + "stopAfterFlush": true + }, + "commands": [ + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_001", + "controllerSeq": 1, + "type": "run.prepare", + "issuedAt": "2026-08-07T23:30:01.000Z", + "payload": { + "workspace": "phase-03-durable-fixture" + }, + "status": "completed", + "result": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "detail": "run prepared", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_002", + "controllerSeq": 2, + "type": "session.open", + "issuedAt": "2026-08-07T23:30:02.000Z", + "payload": { + "reuse": "same_session" + }, + "status": "completed", + "result": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "detail": "session bound", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_003", + "controllerSeq": 3, + "type": "turn.start", + "issuedAt": "2026-08-07T23:30:03.000Z", + "payload": { + "turnId": "turn_phase3_stable", + "text": "Prove Phase 3 durable recovery." + }, + "status": "completed", + "result": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "detail": "turn completed", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_004", + "controllerSeq": 4, + "type": "runner.shutdown", + "issuedAt": "2026-08-07T23:30:04.000Z", + "payload": {}, + "status": "completed", + "result": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "detail": "runner will stop after durable flush", + "logicalEffectCount": 1, + "status": "completed" + } + } + ], + "assertions": { + "stableIdentity": true, + "sourceCursorContinuous": true, + "oneLogicalEffectPerAcceptedCommand": true, + "noDuplicateLogicalEvents": true, + "p0Preserved": true, + "boundedStorage": true, + "secretsRedacted": true + } +} diff --git a/packages/paperclip-runner/knowledge/evidence/phase-03-faults/storage-pressure.json b/packages/paperclip-runner/knowledge/evidence/phase-03-faults/storage-pressure.json new file mode 100644 index 00000000000..253628c02cf --- /dev/null +++ b/packages/paperclip-runner/knowledge/evidence/phase-03-faults/storage-pressure.json @@ -0,0 +1,443 @@ +{ + "schema": "paperclip.runner.phase3.trace.v1", + "diagnostics": { + "schema": "paperclip.runner.phase3.diagnostics.v1", + "fault": "storage-pressure", + "connection": { + "state": "stopped", + "connectionCount": 1, + "reconnectCount": 0, + "leaseId": "connection_lease_d78971e6-232f-4e78-955f-9de455589f8c", + "leaseExpiresAt": "2026-08-07T23:36:40.583Z" + }, + "identity": { + "runnerInstanceId": "runner_phase3_stable", + "environmentLeaseId": "environment_lease_phase3_stable", + "runId": "run_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "turnId": "turn_phase3_stable", + "itemId": "item_phase3_stable" + }, + "cursors": { + "runnerAckedSourceSeq": 5, + "runnerNextSourceSeq": 6, + "coreAckedSourceSeq": 5, + "highestCommittedSourceSeq": 5 + }, + "outbox": { + "events": 0, + "bytes": 0, + "peakBytes": 2326, + "maxBytes": 16384, + "backpressure": true, + "p0Committed": 3, + "p0Lost": 0 + }, + "commands": { + "issued": 5, + "completed": 4, + "rejected": 1, + "logicalEffects": 4, + "duplicateDeliveries": 0 + }, + "recovery": { + "replayDeliveries": 4, + "runnerRestarts": 0, + "harnessRestarts": 0, + "malformedFrames": 0, + "freshBootstraps": 1, + "outcome": "recovered", + "reason": "storage_pressure_completed" + }, + "security": { + "bootstrapTicketPersisted": false, + "connectionLeaseTokenPersisted": false, + "secretLeakCount": 0 + }, + "committedEvents": [ + { + "sourceSeq": 1, + "sourceEventId": "event_runner_phase3_stable_000001", + "eventType": "workspace.ready", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000001", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.001Z", + "eventType": "workspace.ready", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "workspace": "phase-03-durable-fixture" + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000001", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 1, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.001Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 2, + "sourceEventId": "event_runner_phase3_stable_000002", + "eventType": "session.started", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000002", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.002Z", + "eventType": "session.started", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "driverSessionId": "driver_phase3_fake", + "normalizedSessionId": "session_phase3_stable", + "resumed": false + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000002", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 2, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.002Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 3, + "sourceEventId": "event_runner_phase3_stable_000003", + "eventType": "item.delta", + "priority": 2, + "envelope": { + "envelopeId": "envelope_event_000003", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.003Z", + "eventType": "item.delta", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "coalescedCount": 250, + "latest": { + "chunk": "bounded-progress-249", + "secretToken": "[REDACTED]" + } + }, + "priority": 2, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000003", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 3, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.003Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 4, + "sourceEventId": "event_runner_phase3_stable_000004", + "eventType": "runner.backpressure", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000004", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.004Z", + "eventType": "runner.backpressure", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "maxOutboxBytes": 16384, + "newTurnsAccepted": false, + "p0ReserveBytes": 8192, + "reason": "outbox_soft_limit" + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000004", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 4, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.004Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 5, + "sourceEventId": "event_runner_phase3_stable_000005", + "eventType": "runner.stopped", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000005", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.005Z", + "eventType": "runner.stopped", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "afterDurableFlush": true + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000005", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 5, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.005Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 1, + "logicalEffectCount": 1 + } + ] + }, + "runnerState": { + "schema": "paperclip.runner.phase3.state.v1", + "runnerInstanceId": "runner_phase3_stable", + "environmentLeaseId": "environment_lease_phase3_stable", + "runId": "run_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "turnId": "turn_phase3_stable", + "itemId": "item_phase3_stable", + "lifecycle": "stopped", + "nextSourceSeq": 6, + "ackedSourceSeq": 5, + "lastControllerCommandSeq": 5, + "reconnectCount": 0, + "maxOutboxBytes": 16384, + "peakOutboxBytes": 2326, + "outbox": [], + "processedCommands": { + "command_phase3_001": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "commandDigest": "fnv1a64:c19dfbbf86a3dbb4", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "detail": "run prepared", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_002": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "commandDigest": "fnv1a64:b5198f3ce67ce88e", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "detail": "session bound", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_003": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "commandDigest": "fnv1a64:778efd982aadc99f", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "detail": "storage pressure applied with P2 coalescing", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_004": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "commandDigest": "fnv1a64:34d91505a30f60fb", + "status": "rejected", + "logicalEffectCount": 0, + "result": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "detail": "new turns are disabled while draining or backpressured", + "logicalEffectCount": 0, + "status": "rejected" + } + }, + "command_phase3_005": { + "commandId": "command_phase3_005", + "controllerSeq": 5, + "commandDigest": "fnv1a64:533318a30fba6d22", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_005", + "controllerSeq": 5, + "detail": "runner will stop after durable flush", + "logicalEffectCount": 1, + "status": "completed" + } + } + }, + "diagnostics": [], + "backpressure": true, + "recoverableFailure": null, + "unrecoverableOutcome": null, + "harnessGeneration": 1, + "stopAfterFlush": true + }, + "commands": [ + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_001", + "controllerSeq": 1, + "type": "run.prepare", + "issuedAt": "2026-08-07T23:30:01.000Z", + "payload": { + "workspace": "phase-03-durable-fixture" + }, + "status": "completed", + "result": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "detail": "run prepared", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_002", + "controllerSeq": 2, + "type": "session.open", + "issuedAt": "2026-08-07T23:30:02.000Z", + "payload": { + "reuse": "same_session" + }, + "status": "completed", + "result": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "detail": "session bound", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_003", + "controllerSeq": 3, + "type": "fault.storage_pressure", + "issuedAt": "2026-08-07T23:30:03.000Z", + "payload": {}, + "status": "completed", + "result": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "detail": "storage pressure applied with P2 coalescing", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_004", + "controllerSeq": 4, + "type": "turn.start", + "issuedAt": "2026-08-07T23:30:04.000Z", + "payload": { + "turnId": "turn_phase3_stable", + "text": "Prove Phase 3 durable recovery." + }, + "status": "rejected", + "result": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "detail": "new turns are disabled while draining or backpressured", + "logicalEffectCount": 0, + "status": "rejected" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_005", + "controllerSeq": 5, + "type": "runner.shutdown", + "issuedAt": "2026-08-07T23:30:05.000Z", + "payload": {}, + "status": "completed", + "result": { + "commandId": "command_phase3_005", + "controllerSeq": 5, + "detail": "runner will stop after durable flush", + "logicalEffectCount": 1, + "status": "completed" + } + } + ], + "assertions": { + "stableIdentity": true, + "sourceCursorContinuous": true, + "oneLogicalEffectPerAcceptedCommand": true, + "noDuplicateLogicalEvents": true, + "p0Preserved": true, + "boundedStorage": true, + "secretsRedacted": true + } +} diff --git a/packages/paperclip-runner/knowledge/evidence/phase-03-recovery-diagnostics.png b/packages/paperclip-runner/knowledge/evidence/phase-03-recovery-diagnostics.png new file mode 100644 index 00000000000..00b6a3d4937 Binary files /dev/null and b/packages/paperclip-runner/knowledge/evidence/phase-03-recovery-diagnostics.png differ diff --git a/packages/paperclip-runner/knowledge/evidence/phase-03-recovery-trace.json b/packages/paperclip-runner/knowledge/evidence/phase-03-recovery-trace.json new file mode 100644 index 00000000000..12f74f6211e --- /dev/null +++ b/packages/paperclip-runner/knowledge/evidence/phase-03-recovery-trace.json @@ -0,0 +1,561 @@ +{ + "schema": "paperclip.runner.phase3.trace.v1", + "diagnostics": { + "schema": "paperclip.runner.phase3.diagnostics.v1", + "fault": "lost-ack", + "connection": { + "state": "stopped", + "connectionCount": 2, + "reconnectCount": 1, + "leaseId": "connection_lease_db423713-98d6-4a86-940b-ec2f0f6cd6e3", + "leaseExpiresAt": "2026-08-07T23:36:38.759Z" + }, + "identity": { + "runnerInstanceId": "runner_phase3_stable", + "environmentLeaseId": "environment_lease_phase3_stable", + "runId": "run_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "turnId": "turn_phase3_stable", + "itemId": "item_phase3_stable" + }, + "cursors": { + "runnerAckedSourceSeq": 9, + "runnerNextSourceSeq": 10, + "coreAckedSourceSeq": 9, + "highestCommittedSourceSeq": 9 + }, + "outbox": { + "events": 0, + "bytes": 0, + "peakBytes": 6023, + "maxBytes": 65536, + "backpressure": false, + "p0Committed": 5, + "p0Lost": 0 + }, + "commands": { + "issued": 4, + "completed": 4, + "rejected": 0, + "logicalEffects": 4, + "duplicateDeliveries": 1 + }, + "recovery": { + "replayDeliveries": 9, + "runnerRestarts": 0, + "harnessRestarts": 0, + "malformedFrames": 0, + "freshBootstraps": 1, + "outcome": "recovered", + "reason": "lost_ack_completed" + }, + "security": { + "bootstrapTicketPersisted": false, + "connectionLeaseTokenPersisted": false, + "secretLeakCount": 0 + }, + "committedEvents": [ + { + "sourceSeq": 1, + "sourceEventId": "event_runner_phase3_stable_000001", + "eventType": "workspace.ready", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000001", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.001Z", + "eventType": "workspace.ready", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "workspace": "phase-03-durable-fixture" + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000001", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 1, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.001Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 3, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 2, + "sourceEventId": "event_runner_phase3_stable_000002", + "eventType": "session.started", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000002", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.002Z", + "eventType": "session.started", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "driverSessionId": "driver_phase3_fake", + "normalizedSessionId": "session_phase3_stable", + "resumed": false + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000002", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 2, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.002Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 3, + "sourceEventId": "event_runner_phase3_stable_000003", + "eventType": "turn.accepted", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000003", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.003Z", + "eventType": "turn.accepted", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "sameSession": true, + "turnId": "turn_phase3_stable" + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000003", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 3, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.003Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 4, + "sourceEventId": "event_runner_phase3_stable_000004", + "eventType": "item.started", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000004", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.004Z", + "eventType": "item.started", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "kind": "assistant_message" + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000004", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 4, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.004Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 5, + "sourceEventId": "event_runner_phase3_stable_000005", + "eventType": "item.delta", + "priority": 2, + "envelope": { + "envelopeId": "envelope_event_000005", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.005Z", + "eventType": "item.delta", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "text": "Prove Phase 3 durable recovery." + }, + "priority": 2, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000005", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 5, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.005Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 6, + "sourceEventId": "event_runner_phase3_stable_000006", + "eventType": "item.completed", + "priority": 1, + "envelope": { + "envelopeId": "envelope_event_000006", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.006Z", + "eventType": "item.completed", + "itemId": "item_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "text": "Durable recovery completed." + }, + "priority": 1, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000006", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 6, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.006Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 7, + "sourceEventId": "event_runner_phase3_stable_000007", + "eventType": "run.result.proposed", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000007", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.007Z", + "eventType": "run.result.proposed", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "reportedWorkDisposition": "done", + "schema": "paperclip.prp.result.v1", + "status": "succeeded", + "summary": "Phase 3 durable transport recovered without duplicate effects." + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000007", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 7, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.007Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 8, + "sourceEventId": "event_runner_phase3_stable_000008", + "eventType": "run.terminal", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000008", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.008Z", + "eventType": "run.terminal", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "reportedWorkDisposition": "done", + "runTerminalState": "succeeded", + "schema": "paperclip.prp.terminal.v1", + "turnTerminalState": "completed" + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000008", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 8, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.008Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 2, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 9, + "sourceEventId": "event_runner_phase3_stable_000009", + "eventType": "runner.stopped", + "priority": 0, + "envelope": { + "envelopeId": "envelope_event_000009", + "kind": "event", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "emittedAt": "2026-08-07T23:00:00.009Z", + "eventType": "runner.stopped", + "normalizedSessionId": "session_phase3_stable", + "payload": { + "afterDurableFlush": true + }, + "priority": 0, + "runId": "run_phase3_stable", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "sourceEventId": "event_runner_phase3_stable_000009", + "sourceInstanceId": "runner_phase3_stable", + "sourceKind": "runner", + "sourceSeq": 9, + "turnId": "turn_phase3_stable" + }, + "protocol": "paperclip.runner", + "runId": "run_phase3_stable", + "runnerInstanceId": "runner_phase3_stable", + "sentAt": "2026-08-07T23:00:00.009Z", + "turnId": "turn_phase3_stable", + "version": 1 + }, + "deliveryCount": 1, + "logicalEffectCount": 1 + } + ] + }, + "runnerState": { + "schema": "paperclip.runner.phase3.state.v1", + "runnerInstanceId": "runner_phase3_stable", + "environmentLeaseId": "environment_lease_phase3_stable", + "runId": "run_phase3_stable", + "normalizedSessionId": "session_phase3_stable", + "turnId": "turn_phase3_stable", + "itemId": "item_phase3_stable", + "lifecycle": "stopped", + "nextSourceSeq": 10, + "ackedSourceSeq": 9, + "lastControllerCommandSeq": 4, + "reconnectCount": 1, + "maxOutboxBytes": 65536, + "peakOutboxBytes": 6023, + "outbox": [], + "processedCommands": { + "command_phase3_001": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "commandDigest": "fnv1a64:c19dfbbf86a3dbb4", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "detail": "run prepared", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_002": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "commandDigest": "fnv1a64:b5198f3ce67ce88e", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "detail": "session bound", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_003": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "commandDigest": "fnv1a64:fb3b206fe67e1dc8", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "detail": "turn completed", + "logicalEffectCount": 1, + "status": "completed" + } + }, + "command_phase3_004": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "commandDigest": "fnv1a64:d11bf8308f863bad", + "status": "completed", + "logicalEffectCount": 1, + "result": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "detail": "runner will stop after durable flush", + "logicalEffectCount": 1, + "status": "completed" + } + } + }, + "diagnostics": [ + "WebSocket frame write failed: Broken pipe (os error 32)" + ], + "backpressure": false, + "recoverableFailure": null, + "unrecoverableOutcome": null, + "harnessGeneration": 1, + "stopAfterFlush": true + }, + "commands": [ + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_001", + "controllerSeq": 1, + "type": "run.prepare", + "issuedAt": "2026-08-07T23:30:01.000Z", + "payload": { + "workspace": "phase-03-durable-fixture" + }, + "status": "completed", + "result": { + "commandId": "command_phase3_001", + "controllerSeq": 1, + "detail": "run prepared", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_002", + "controllerSeq": 2, + "type": "session.open", + "issuedAt": "2026-08-07T23:30:02.000Z", + "payload": { + "reuse": "same_session" + }, + "status": "completed", + "result": { + "commandId": "command_phase3_002", + "controllerSeq": 2, + "detail": "session bound", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_003", + "controllerSeq": 3, + "type": "turn.start", + "issuedAt": "2026-08-07T23:30:03.000Z", + "payload": { + "turnId": "turn_phase3_stable", + "text": "Prove Phase 3 durable recovery." + }, + "status": "completed", + "result": { + "commandId": "command_phase3_003", + "controllerSeq": 3, + "detail": "turn completed", + "logicalEffectCount": 1, + "status": "completed" + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_phase3_004", + "controllerSeq": 4, + "type": "runner.shutdown", + "issuedAt": "2026-08-07T23:30:04.000Z", + "payload": {}, + "status": "completed", + "result": { + "commandId": "command_phase3_004", + "controllerSeq": 4, + "detail": "runner will stop after durable flush", + "logicalEffectCount": 1, + "status": "completed" + } + } + ], + "assertions": { + "stableIdentity": true, + "sourceCursorContinuous": true, + "oneLogicalEffectPerAcceptedCommand": true, + "noDuplicateLogicalEvents": true, + "p0Preserved": true, + "boundedStorage": true, + "secretsRedacted": true + } +} diff --git a/packages/paperclip-runner/knowledge/journal/2026-08-07-phase-03.md b/packages/paperclip-runner/knowledge/journal/2026-08-07-phase-03.md new file mode 100644 index 00000000000..d288f87d5fb --- /dev/null +++ b/packages/paperclip-runner/knowledge/journal/2026-08-07-phase-03.md @@ -0,0 +1,69 @@ +--- +type: Engineering Journal Entry +title: Phase 3 durable outbound transport and recovery +description: Decisions and evidence for WebSocket authentication, durable replay, restart recovery, and redacted diagnostics. +tags: [native-runner, phase-3, websocket, outbox, replay, recovery] +status: stable +generated: { by: openai/gpt-5, at: 2026-08-07T23:31:00Z } +entry_kind: phase +phase: "3" +--- + +# Context + +Phase 3 proves network recovery before a real model driver or production control-plane connection exists. + +# Decisions + +1. Use a real loopback WebSocket. Keep the mock core and all state inside the package. +2. Use a one-time bootstrap ticket. Exchange it for a connection lease on the first connection. +3. Write events before network delivery. Remove events only after a cumulative acknowledgement. +4. Keep event IDs and command IDs stable across reconnect and restart. +5. Store processed command IDs with the outbox. Repeated commands do not repeat effects. +6. Protect P0 events when the outbox reaches its limit. +7. Use a private atomic JSON state file for the tracer. Do not store the bootstrap ticket. +8. Show counters and states in diagnostics. Do not show authentication secrets. +9. Resolve and pin only loopback destinations before sending a bearer + capability. Reject ambiguous or mixed DNS answers. +10. Use exclusive random temporary state files, no-follow opens, ownership and + mode checks, atomic rename, file sync, and directory sync. +11. Capture and remove the bootstrap environment value at daemon entry. Keep + bootstrap and lease values in overwrite-on-drop buffers. + +# Evidence + +See the [Phase 3 verification evidence](../evidence/2026-08-07-phase-03-verification.md). + +Follow the [Phase 3 tutorial](../../docs/tutorials/phase-03-break-recovery.md). + +# Failures + +- The first lost-ACK path let a socket write error escape before reconnect. The + recovery loop now records the failure, preserves state, and reconnects. +- The security diagnostic first reported capability persistence with fixed + `false` values. It now derives both facts from the runner state file and the + tests assert the result. +- The normal browser run could not start Chromium because this minimal host did + not include `libatk`. The supported rootless verifier downloaded private + runtime libraries and all seven browser checks passed. +- Documentation validation found the recovery screenshot link before the file + was captured. The browser proof produced the screenshot, it was inspected, + and the final documentation check passed. + +# Known gaps + +- The runner transport and durable store are Rust. The mock core, fault driver, + evidence recorder, and browser peer are TypeScript. +- The WebSocket uses loopback `ws` for local proof. Production `wss` + certificate and proxy policy remain deferred. +- The mock core supports one runner session. +- The atomic JSON state is a tracer store. A production database format, + migration policy, and long-term processed-command retention remain deferred. +- Lease replacement after expiry is an external fresh-bootstrap action. This + phase does not implement a production lease-renewal service. + +# Follow-up questions + +- Which TLS implementation and trust policy will own production `wss`? +- Which encrypted store and migration policy will replace the tracer JSON file? +- Which replay window will the production core retain for diagnostics? diff --git a/packages/paperclip-runner/knowledge/journal/index.md b/packages/paperclip-runner/knowledge/journal/index.md index 4c646070d58..df8d4fb71d8 100644 --- a/packages/paperclip-runner/knowledge/journal/index.md +++ b/packages/paperclip-runner/knowledge/journal/index.md @@ -6,3 +6,4 @@ * [Phase 0 Rust boundary correction](2026-08-07-phase-00-rust-boundary-correction.md) - Root cause and corrected Rust/TypeScript ownership after the human checkpoint. * [Phase 1 protocol, fixtures, reducer, and static replay](2026-08-07-phase-01.md) - Executable PRP schemas, deterministic replay decisions, browser findings, and checkpoint gaps. * [Phase 2 local runner and fake harness](2026-08-07-phase-02.md) - Rust supervision, stdio protocol, scripted driver, live browser, cleanup, and parity decisions. +* [Phase 3 durable outbound transport and recovery](2026-08-07-phase-03.md) - WebSocket authentication, durable ACK replay, command deduplication, and recovery diagnostics. diff --git a/packages/paperclip-runner/knowledge/log.md b/packages/paperclip-runner/knowledge/log.md index 0348300593c..170159fda3a 100644 --- a/packages/paperclip-runner/knowledge/log.md +++ b/packages/paperclip-runner/knowledge/log.md @@ -6,3 +6,4 @@ * **Rust boundary correction**: Added the production-direction Cargo workspace, shared-fixture parity evidence, negative Cargo boundary coverage, and a superseding journal entry after the Phase 0 checkpoint exposed the TypeScript-only omission. * **Phase 1 static replay**: Added executable PRP v1 schemas, cross-language fixture/golden parity, deterministic reducer, CLI/browser replay, compatibility policy, tutorial, screenshot, and verification evidence. * **Phase 2 local runner**: Added the deterministic Rust supervisor, stdio fake-harness protocol, scripted scenarios, bounded logs, cleanup and terminal guards, TypeScript mock core, validated live browser mode, screenshots, tutorial, and verification evidence. +* **Phase 3 durable recovery**: Added package-local WebSocket authentication, a durable outbox, cumulative acknowledgements, restart replay, command deduplication, recovery diagnostics, tutorial, and evidence. diff --git a/packages/paperclip-runner/spec/paperclip-native-runner-implementation-plan.md b/packages/paperclip-runner/spec/paperclip-native-runner-implementation-plan.md index 993feab78a8..499821666a5 100644 --- a/packages/paperclip-runner/spec/paperclip-native-runner-implementation-plan.md +++ b/packages/paperclip-runner/spec/paperclip-native-runner-implementation-plan.md @@ -6,7 +6,7 @@ This document is the implementation-phase plan for [`paperclip-native-runner-spi All implementation starts inside `packages/paperclip-runner/` with protocol and mock-control-plane layers. These standalone phases do not modify Paperclip control-plane semantics. A real Paperclip bridge is a late, separately reviewed phase after the standalone contract passes conformance tests. -Phase 0, Phase 1, and Phase 2 are complete. Their runnable demos, tests, package-local documentation, tutorials, journal entries, screenshots, and evidence satisfy the first three task-creation gates. The human accepted Phase 2 and authorized Phase 3 on August 7, 2026. Create only the Phase 3 task graph now. Keep Phase 4 and later issues deferred until the Phase 3 human checkpoint is complete. +Phase 0, Phase 1, and Phase 2 are complete. The Phase 3 tracer is implemented and awaits Security, UX, QA, and human acceptance. Keep Phase 4 and later issues deferred until the Phase 3 human checkpoint is complete. ## Non-negotiable delivery rules diff --git a/packages/paperclip-runner/spec/paperclip-native-runner-spike-spec.md b/packages/paperclip-runner/spec/paperclip-native-runner-spike-spec.md index 84a5c10f12d..81ba677faa4 100644 --- a/packages/paperclip-runner/spec/paperclip-native-runner-spike-spec.md +++ b/packages/paperclip-runner/spec/paperclip-native-runner-spike-spec.md @@ -6575,7 +6575,7 @@ Never change an active run from native to legacy mid-session. The implementation phases, dependency policy, package-local documentation layout, human checkpoints, and retained future backlog now live in [`paperclip-native-runner-implementation-plan.md`](./paperclip-native-runner-implementation-plan.md). -The spike specification remains the normative contract. Keep it updated when implementation evidence changes architecture, protocol, security, persistence, API, UI, performance, or exit criteria. Keep sequencing and task-creation details in the implementation plan. Phase 0, Phase 1, and Phase 2 are complete. The human accepted Phase 2 and authorized the Phase 3 task graph on August 7, 2026. Phase 3 must prove durable outbound transport and same-session recovery entirely inside `packages/paperclip-runner/` against the mock core. Its normative recovery invariants, tutorial evidence, and human checkpoint are defined in the package-local implementation plan. Keep Phase 4 and later tasks deferred until the Phase 3 human checkpoint is complete. +The spike specification remains the normative contract. Keep it updated when implementation evidence changes architecture, protocol, security, persistence, API, UI, performance, or exit criteria. Keep sequencing and task-creation details in the implementation plan. Phase 0, Phase 1, and Phase 2 are complete. The Phase 3 tracer now implements package-local outbound WebSocket authentication, private durable state, cumulative acknowledgement replay, command deduplication, bounded P0-preserving storage, drain, revoke, and redacted CLI/browser diagnostics. Security, UX, QA, and human acceptance remain open. Keep Phase 4 and later tasks deferred until the Phase 3 human checkpoint is complete. ## 30. Dependency sequencing