fix(native): keep the transport cause on a closed notification stream failure (#19240)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs read provider events from a shared session notification
stream (Codex, OpenCode and ACPX drivers use it)
> - When that stream closes, the driver fails the session with
`notification_transport_failed`
> - The run error was only "Provider notification transport failed
closed." The real cause was only in a diagnostic event
> - In a staging soak, many runs on OpenRouter and Grok agents showed
this error. Operators read it as a provider fault. The diagnostic events
showed a host-side cause: a stopped sandbox plugin worker ended remote
runner supervision
> - This pull request keeps the redacted, bounded cause on the terminal
failure message, and does not name the provider
> - The benefit is that a failed run shows why it failed, so operators
do not chase the wrong provider

## Linked Issues or Issue Description

**What happened?**
Native runs failed with `native_provider_terminal_failed` and the error
"Provider notification transport failed closed." The run did not show a
cause. All failures in one cluster happened inside one minute, on
several agents and models (OpenRouter DeepSeek, Grok). Each run's
`harness.diagnostic` event had the real cause, for example
`runner_process_termination_unconfirmed: runnerd process supervision
failed: Cannot call "environmentExecute" — worker for "<plugin>" is
stopped`, or `native_adopted_runner_identity_unverifiable: runner
liveness could not be revalidated`. The sandbox duplex transport
reported a `provider_exit` loss about 30 seconds before (the read-only
process probe budget).

**Expected behavior**
The terminal run error keeps the redacted cause. It does not blame the
model provider when the cause is host-side runner supervision.

**Steps to reproduce**
1. Open a native Codex-protocol session and start a turn.
2. Make the notification stream throw (for example, a supervision
error).
3. Read the `session.failed` and `turn.failed` messages. Before this
change they are the bare generic text.

## What Changed

- `codex-session-notifications.ts`: the `notification_transport_failed`
terminal message is now `Notification transport failed closed: <cause>`.
The cause is redacted first, then whitespace is collapsed and it is
bounded to 1,000 characters. The `Error: ` prefix is removed. The code
and `recoverable: false` do not change.
- New regression in `codex-protocol-integrity.test.ts`: one
`session.failed`, the cause and code are present, the secret is
redacted, the length is bounded, and `turn.failed` has the same message.
- `doc/execution-semantics.md`: an additive note that this failure can
come from host-side runner supervision, and that the message keeps the
redacted cause.

## Verification

- Red: before the fix, the new test failed: `expected 'Provider
notification transport faile…' to contain
'runner_process_termination_unconfirmed'`.
- Green: `cd packages/paperclip-runner && npx vitest run
src/drivers/codex/ src/conformance/local-provider-fault-matrix.test.ts
src/native-session-runtime.test.ts` (565 passed).
- `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json`: no new
errors. The existing `src/eval/workflow-harness.ts` errors come from an
unbuilt workspace package.
- Staging evidence was read with read-only GETs only (run, events).

## Risks

- Low risk. Only the message text changes. The failure code,
`recoverable: false`, and the safe-replacement rules do not change. The
server matches on the code, not on the message.
- This PR does not add a retry in the same turn. Runner termination is
unconfirmed when supervision is lost. A retry or replacement could start
a second executor. The remote process probe already retries read-only
checks inside a 30-second budget. Recovery stays with the existing
ownership and recovery checks.
- The cause text can include a plugin id. It goes through the existing
provider-diagnostic redaction.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in
Claude Code with tool use (shell, file edits, browser reads) and
extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-05 07:51:39 -06:00
1 parent b3da1d886a
commit 03abe1f1a1
3 files changed
+60 -1

No files matched your search

+6
View File
@@ -1884,6 +1884,12 @@ does not prove that the provider stopped or that a retry is safe; it remains
non-recoverable until the existing ownership and recovery checks establish the
next permitted action.
The same applies when the shared session notification stream closes
(`notification_transport_failed`). The stream can close because of host-side
runner supervision, such as a stopped sandbox plugin worker, as well as because
of the provider. The terminal failure message therefore keeps the redacted,
bounded cause. It does not name the provider unless the cause does.
If a failed sidecar control closes an ACPX session before the event pump observes
the failure, its terminal retains the original closed-code diagnostic. Later
polling must not replace it with a generic closed-session message or infer a
@@ -1111,6 +1111,43 @@ describe("Codex protocol integrity propagation", () => {
await expect(session.startTurn({ message: { role: "user", text: "Work" } })).rejects.toMatchObject({ code: "native_provider_terminal_failed", providerCode: "thread_binding_mismatch", recoverable: false });
await session.close({ reason: "test complete" });
});
it("keeps the redacted, bounded transport cause on the terminal failure", async () => {
// A host-side supervision loss (here, a stopped sandbox plugin worker)
// closed the notification stream. The run's terminal error must explain
// it instead of a bare "failed closed" that reads like a provider fault.
const transport = new FakeCodexTransport();
const session = await makeDriver([transport]).openSession({
runId: "run-cause",
normalizedSessionId: "session-cause",
workingDirectory: WORKSPACE,
});
const events: PrpEvent[] = [];
const consumed = (async () => {
for await (const event of session.events()) events.push(event);
})();
try {
await session.startTurn({ message: { role: "user", text: "Work." } });
transport.queue.fail(new Error(
`runner_process_termination_unconfirmed: runnerd process supervision failed: Cannot call "environmentExecute" — worker for "plugin-1" is stopped token=sandbox-secret ${"x".repeat(4_000)}`,
));
await consumed;
const failed = events.filter((event) => event.eventType === "session.failed");
expect(failed).toHaveLength(1);
const message = String(failed[0]!.payload.message);
expect(failed[0]!.payload).toMatchObject({ code: "notification_transport_failed", recoverable: false });
expect(message).toMatch(/^Notification transport failed closed: /);
expect(message).toContain("runner_process_termination_unconfirmed");
expect(message).toContain('worker for "plugin-1" is stopped');
expect(message).not.toContain("Error: ");
expect(message).not.toContain("sandbox-secret");
expect(message.length).toBeLessThanOrEqual(1_200);
const turnFailed = events.find((event) => event.eventType === "turn.failed");
expect((turnFailed?.payload.error as { message?: string } | undefined)?.message).toBe(message);
} finally {
await session.close({ reason: "test cleanup" });
await consumed;
}
});
it("does not promote a message-and-field lookalike transport error", async () => {
const transport = new FakeCodexTransport();
const session = await makeDriver([transport]).openSession({
@@ -74,13 +74,29 @@ export async function pumpNotifications(state: CodexSessionState): Promise<void>
message: redactCodexDiagnostic(String(error)),
});
state.expirePendingInputRequestsAfterProviderLoss();
// The terminal error is what the run shows. Keep the redacted cause on it:
// the stream can close because of host-side runner supervision, not only
// the provider, and a bare "failed closed" hides which one it was.
state.failProtocol(
"notification_transport_failed",
"Provider notification transport failed closed.",
notificationTransportFailureMessage(error),
);
}
}
const NOTIFICATION_TRANSPORT_CAUSE_LIMIT = 1_000;
function notificationTransportFailureMessage(error: unknown): string {
const raw = error instanceof Error ? error.message : String(error);
// Redact before bounding so truncation cannot split a credential pattern.
const cause = redactCodexDiagnostic(raw).replace(/\s+/g, " ").trim();
if (!cause) return "Notification transport failed closed.";
const bounded = cause.length > NOTIFICATION_TRANSPORT_CAUSE_LIMIT
? `${cause.slice(0, NOTIFICATION_TRANSPORT_CAUSE_LIMIT)}…`
: cause;
return `Notification transport failed closed: ${bounded}`;
}
async function mapNotification(state: CodexSessionState, notification: CodexRpcNotification): Promise<void> {
const sourceSequenceBefore = state.sourceSequence;
let rejected = false;