From 03abe1f1a1d51aaae347e01a17bb28bce6bcf03d Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 5 Nov 2026 07:51:39 -0600 Subject: [PATCH] fix(native): keep the transport cause on a closed notification stream failure (#19240) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs read provider events from a shared session notification stream (Codex, OpenCode and ACPX drivers use it) > - When that stream closes, the driver fails the session with `notification_transport_failed` > - The run error was only "Provider notification transport failed closed." The real cause was only in a diagnostic event > - In a staging soak, many runs on OpenRouter and Grok agents showed this error. Operators read it as a provider fault. The diagnostic events showed a host-side cause: a stopped sandbox plugin worker ended remote runner supervision > - This pull request keeps the redacted, bounded cause on the terminal failure message, and does not name the provider > - The benefit is that a failed run shows why it failed, so operators do not chase the wrong provider ## Linked Issues or Issue Description **What happened?** Native runs failed with `native_provider_terminal_failed` and the error "Provider notification transport failed closed." The run did not show a cause. All failures in one cluster happened inside one minute, on several agents and models (OpenRouter DeepSeek, Grok). Each run's `harness.diagnostic` event had the real cause, for example `runner_process_termination_unconfirmed: runnerd process supervision failed: Cannot call "environmentExecute" — worker for "" is stopped`, or `native_adopted_runner_identity_unverifiable: runner liveness could not be revalidated`. The sandbox duplex transport reported a `provider_exit` loss about 30 seconds before (the read-only process probe budget). **Expected behavior** The terminal run error keeps the redacted cause. It does not blame the model provider when the cause is host-side runner supervision. **Steps to reproduce** 1. Open a native Codex-protocol session and start a turn. 2. Make the notification stream throw (for example, a supervision error). 3. Read the `session.failed` and `turn.failed` messages. Before this change they are the bare generic text. ## What Changed - `codex-session-notifications.ts`: the `notification_transport_failed` terminal message is now `Notification transport failed closed: `. The cause is redacted first, then whitespace is collapsed and it is bounded to 1,000 characters. The `Error: ` prefix is removed. The code and `recoverable: false` do not change. - New regression in `codex-protocol-integrity.test.ts`: one `session.failed`, the cause and code are present, the secret is redacted, the length is bounded, and `turn.failed` has the same message. - `doc/execution-semantics.md`: an additive note that this failure can come from host-side runner supervision, and that the message keeps the redacted cause. ## Verification - Red: before the fix, the new test failed: `expected 'Provider notification transport faile…' to contain 'runner_process_termination_unconfirmed'`. - Green: `cd packages/paperclip-runner && npx vitest run src/drivers/codex/ src/conformance/local-provider-fault-matrix.test.ts src/native-session-runtime.test.ts` (565 passed). - `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json`: no new errors. The existing `src/eval/workflow-harness.ts` errors come from an unbuilt workspace package. - Staging evidence was read with read-only GETs only (run, events). ## Risks - Low risk. Only the message text changes. The failure code, `recoverable: false`, and the safe-replacement rules do not change. The server matches on the code, not on the message. - This PR does not add a retry in the same turn. Runner termination is unconfirmed when supervision is lost. A retry or replacement could start a second executor. The remote process probe already retries read-only checks inside a 30-second budget. Recovery stays with the existing ownership and recovery checks. - The cause text can include a plugin id. It goes through the existing provider-diagnostic redaction. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code with tool use (shell, file edits, browser reads) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- doc/execution-semantics.md | 6 +++ .../codex/codex-protocol-integrity.test.ts | 37 +++++++++++++++++++ .../codex/codex-session-notifications.ts | 18 ++++++++- 3 files changed, 60 insertions(+), 1 deletion(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 1f1386445c8..d1ceb36cb3c 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1884,6 +1884,12 @@ does not prove that the provider stopped or that a retry is safe; it remains non-recoverable until the existing ownership and recovery checks establish the next permitted action. +The same applies when the shared session notification stream closes +(`notification_transport_failed`). The stream can close because of host-side +runner supervision, such as a stopped sandbox plugin worker, as well as because +of the provider. The terminal failure message therefore keeps the redacted, +bounded cause. It does not name the provider unless the cause does. + If a failed sidecar control closes an ACPX session before the event pump observes the failure, its terminal retains the original closed-code diagnostic. Later polling must not replace it with a generic closed-session message or infer a diff --git a/packages/paperclip-runner/src/drivers/codex/codex-protocol-integrity.test.ts b/packages/paperclip-runner/src/drivers/codex/codex-protocol-integrity.test.ts index b3932975dfd..20dbb749a2a 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-protocol-integrity.test.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-protocol-integrity.test.ts @@ -1111,6 +1111,43 @@ describe("Codex protocol integrity propagation", () => { await expect(session.startTurn({ message: { role: "user", text: "Work" } })).rejects.toMatchObject({ code: "native_provider_terminal_failed", providerCode: "thread_binding_mismatch", recoverable: false }); await session.close({ reason: "test complete" }); }); + it("keeps the redacted, bounded transport cause on the terminal failure", async () => { + // A host-side supervision loss (here, a stopped sandbox plugin worker) + // closed the notification stream. The run's terminal error must explain + // it instead of a bare "failed closed" that reads like a provider fault. + const transport = new FakeCodexTransport(); + const session = await makeDriver([transport]).openSession({ + runId: "run-cause", + normalizedSessionId: "session-cause", + workingDirectory: WORKSPACE, + }); + const events: PrpEvent[] = []; + const consumed = (async () => { + for await (const event of session.events()) events.push(event); + })(); + try { + await session.startTurn({ message: { role: "user", text: "Work." } }); + transport.queue.fail(new Error( + `runner_process_termination_unconfirmed: runnerd process supervision failed: Cannot call "environmentExecute" — worker for "plugin-1" is stopped token=sandbox-secret ${"x".repeat(4_000)}`, + )); + await consumed; + const failed = events.filter((event) => event.eventType === "session.failed"); + expect(failed).toHaveLength(1); + const message = String(failed[0]!.payload.message); + expect(failed[0]!.payload).toMatchObject({ code: "notification_transport_failed", recoverable: false }); + expect(message).toMatch(/^Notification transport failed closed: /); + expect(message).toContain("runner_process_termination_unconfirmed"); + expect(message).toContain('worker for "plugin-1" is stopped'); + expect(message).not.toContain("Error: "); + expect(message).not.toContain("sandbox-secret"); + expect(message.length).toBeLessThanOrEqual(1_200); + const turnFailed = events.find((event) => event.eventType === "turn.failed"); + expect((turnFailed?.payload.error as { message?: string } | undefined)?.message).toBe(message); + } finally { + await session.close({ reason: "test cleanup" }); + await consumed; + } + }); it("does not promote a message-and-field lookalike transport error", async () => { const transport = new FakeCodexTransport(); const session = await makeDriver([transport]).openSession({ diff --git a/packages/paperclip-runner/src/drivers/codex/codex-session-notifications.ts b/packages/paperclip-runner/src/drivers/codex/codex-session-notifications.ts index 326d249e929..03ed995c121 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-session-notifications.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-session-notifications.ts @@ -74,13 +74,29 @@ export async function pumpNotifications(state: CodexSessionState): Promise message: redactCodexDiagnostic(String(error)), }); state.expirePendingInputRequestsAfterProviderLoss(); + // The terminal error is what the run shows. Keep the redacted cause on it: + // the stream can close because of host-side runner supervision, not only + // the provider, and a bare "failed closed" hides which one it was. state.failProtocol( "notification_transport_failed", - "Provider notification transport failed closed.", + notificationTransportFailureMessage(error), ); } } +const NOTIFICATION_TRANSPORT_CAUSE_LIMIT = 1_000; + +function notificationTransportFailureMessage(error: unknown): string { + const raw = error instanceof Error ? error.message : String(error); + // Redact before bounding so truncation cannot split a credential pattern. + const cause = redactCodexDiagnostic(raw).replace(/\s+/g, " ").trim(); + if (!cause) return "Notification transport failed closed."; + const bounded = cause.length > NOTIFICATION_TRANSPORT_CAUSE_LIMIT + ? `${cause.slice(0, NOTIFICATION_TRANSPORT_CAUSE_LIMIT)}…` + : cause; + return `Notification transport failed closed: ${bounded}`; +} + async function mapNotification(state: CodexSessionState, notification: CodexRpcNotification): Promise { const sourceSequenceBefore = state.sourceSequence; let rejected = false;