mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 01:57:31 +01:00
fix(native): keep the transport cause on a closed notification stream failure (#19240)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs read provider events from a shared session notification stream (Codex, OpenCode and ACPX drivers use it) > - When that stream closes, the driver fails the session with `notification_transport_failed` > - The run error was only "Provider notification transport failed closed." The real cause was only in a diagnostic event > - In a staging soak, many runs on OpenRouter and Grok agents showed this error. Operators read it as a provider fault. The diagnostic events showed a host-side cause: a stopped sandbox plugin worker ended remote runner supervision > - This pull request keeps the redacted, bounded cause on the terminal failure message, and does not name the provider > - The benefit is that a failed run shows why it failed, so operators do not chase the wrong provider ## Linked Issues or Issue Description **What happened?** Native runs failed with `native_provider_terminal_failed` and the error "Provider notification transport failed closed." The run did not show a cause. All failures in one cluster happened inside one minute, on several agents and models (OpenRouter DeepSeek, Grok). Each run's `harness.diagnostic` event had the real cause, for example `runner_process_termination_unconfirmed: runnerd process supervision failed: Cannot call "environmentExecute" — worker for "<plugin>" is stopped`, or `native_adopted_runner_identity_unverifiable: runner liveness could not be revalidated`. The sandbox duplex transport reported a `provider_exit` loss about 30 seconds before (the read-only process probe budget). **Expected behavior** The terminal run error keeps the redacted cause. It does not blame the model provider when the cause is host-side runner supervision. **Steps to reproduce** 1. Open a native Codex-protocol session and start a turn. 2. Make the notification stream throw (for example, a supervision error). 3. Read the `session.failed` and `turn.failed` messages. Before this change they are the bare generic text. ## What Changed - `codex-session-notifications.ts`: the `notification_transport_failed` terminal message is now `Notification transport failed closed: <cause>`. The cause is redacted first, then whitespace is collapsed and it is bounded to 1,000 characters. The `Error: ` prefix is removed. The code and `recoverable: false` do not change. - New regression in `codex-protocol-integrity.test.ts`: one `session.failed`, the cause and code are present, the secret is redacted, the length is bounded, and `turn.failed` has the same message. - `doc/execution-semantics.md`: an additive note that this failure can come from host-side runner supervision, and that the message keeps the redacted cause. ## Verification - Red: before the fix, the new test failed: `expected 'Provider notification transport faile…' to contain 'runner_process_termination_unconfirmed'`. - Green: `cd packages/paperclip-runner && npx vitest run src/drivers/codex/ src/conformance/local-provider-fault-matrix.test.ts src/native-session-runtime.test.ts` (565 passed). - `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json`: no new errors. The existing `src/eval/workflow-harness.ts` errors come from an unbuilt workspace package. - Staging evidence was read with read-only GETs only (run, events). ## Risks - Low risk. Only the message text changes. The failure code, `recoverable: false`, and the safe-replacement rules do not change. The server matches on the code, not on the message. - This PR does not add a retry in the same turn. Runner termination is unconfirmed when supervision is lost. A retry or replacement could start a second executor. The remote process probe already retries read-only checks inside a 30-second budget. Recovery stays with the existing ownership and recovery checks. - The cause text can include a plugin id. It goes through the existing provider-diagnostic redaction. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code with tool use (shell, file edits, browser reads) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
b3da1d886a
commit
03abe1f1a1
3 files changed
+60
-1
No files matched your search
@@ -1884,6 +1884,12 @@ does not prove that the provider stopped or that a retry is safe; it remains
|
||||
non-recoverable until the existing ownership and recovery checks establish the
|
||||
next permitted action.
|
||||
|
||||
The same applies when the shared session notification stream closes
|
||||
(`notification_transport_failed`). The stream can close because of host-side
|
||||
runner supervision, such as a stopped sandbox plugin worker, as well as because
|
||||
of the provider. The terminal failure message therefore keeps the redacted,
|
||||
bounded cause. It does not name the provider unless the cause does.
|
||||
|
||||
If a failed sidecar control closes an ACPX session before the event pump observes
|
||||
the failure, its terminal retains the original closed-code diagnostic. Later
|
||||
polling must not replace it with a generic closed-session message or infer a
|
||||
|
||||
@@ -1111,6 +1111,43 @@ describe("Codex protocol integrity propagation", () => {
|
||||
await expect(session.startTurn({ message: { role: "user", text: "Work" } })).rejects.toMatchObject({ code: "native_provider_terminal_failed", providerCode: "thread_binding_mismatch", recoverable: false });
|
||||
await session.close({ reason: "test complete" });
|
||||
});
|
||||
it("keeps the redacted, bounded transport cause on the terminal failure", async () => {
|
||||
// A host-side supervision loss (here, a stopped sandbox plugin worker)
|
||||
// closed the notification stream. The run's terminal error must explain
|
||||
// it instead of a bare "failed closed" that reads like a provider fault.
|
||||
const transport = new FakeCodexTransport();
|
||||
const session = await makeDriver([transport]).openSession({
|
||||
runId: "run-cause",
|
||||
normalizedSessionId: "session-cause",
|
||||
workingDirectory: WORKSPACE,
|
||||
});
|
||||
const events: PrpEvent[] = [];
|
||||
const consumed = (async () => {
|
||||
for await (const event of session.events()) events.push(event);
|
||||
})();
|
||||
try {
|
||||
await session.startTurn({ message: { role: "user", text: "Work." } });
|
||||
transport.queue.fail(new Error(
|
||||
`runner_process_termination_unconfirmed: runnerd process supervision failed: Cannot call "environmentExecute" — worker for "plugin-1" is stopped token=sandbox-secret ${"x".repeat(4_000)}`,
|
||||
));
|
||||
await consumed;
|
||||
const failed = events.filter((event) => event.eventType === "session.failed");
|
||||
expect(failed).toHaveLength(1);
|
||||
const message = String(failed[0]!.payload.message);
|
||||
expect(failed[0]!.payload).toMatchObject({ code: "notification_transport_failed", recoverable: false });
|
||||
expect(message).toMatch(/^Notification transport failed closed: /);
|
||||
expect(message).toContain("runner_process_termination_unconfirmed");
|
||||
expect(message).toContain('worker for "plugin-1" is stopped');
|
||||
expect(message).not.toContain("Error: ");
|
||||
expect(message).not.toContain("sandbox-secret");
|
||||
expect(message.length).toBeLessThanOrEqual(1_200);
|
||||
const turnFailed = events.find((event) => event.eventType === "turn.failed");
|
||||
expect((turnFailed?.payload.error as { message?: string } | undefined)?.message).toBe(message);
|
||||
} finally {
|
||||
await session.close({ reason: "test cleanup" });
|
||||
await consumed;
|
||||
}
|
||||
});
|
||||
it("does not promote a message-and-field lookalike transport error", async () => {
|
||||
const transport = new FakeCodexTransport();
|
||||
const session = await makeDriver([transport]).openSession({
|
||||
|
||||
@@ -74,13 +74,29 @@ export async function pumpNotifications(state: CodexSessionState): Promise<void>
|
||||
message: redactCodexDiagnostic(String(error)),
|
||||
});
|
||||
state.expirePendingInputRequestsAfterProviderLoss();
|
||||
// The terminal error is what the run shows. Keep the redacted cause on it:
|
||||
// the stream can close because of host-side runner supervision, not only
|
||||
// the provider, and a bare "failed closed" hides which one it was.
|
||||
state.failProtocol(
|
||||
"notification_transport_failed",
|
||||
"Provider notification transport failed closed.",
|
||||
notificationTransportFailureMessage(error),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const NOTIFICATION_TRANSPORT_CAUSE_LIMIT = 1_000;
|
||||
|
||||
function notificationTransportFailureMessage(error: unknown): string {
|
||||
const raw = error instanceof Error ? error.message : String(error);
|
||||
// Redact before bounding so truncation cannot split a credential pattern.
|
||||
const cause = redactCodexDiagnostic(raw).replace(/\s+/g, " ").trim();
|
||||
if (!cause) return "Notification transport failed closed.";
|
||||
const bounded = cause.length > NOTIFICATION_TRANSPORT_CAUSE_LIMIT
|
||||
? `${cause.slice(0, NOTIFICATION_TRANSPORT_CAUSE_LIMIT)}…`
|
||||
: cause;
|
||||
return `Notification transport failed closed: ${bounded}`;
|
||||
}
|
||||
|
||||
async function mapNotification(state: CodexSessionState, notification: CodexRpcNotification): Promise<void> {
|
||||
const sourceSequenceBefore = state.sourceSequence;
|
||||
let rejected = false;
|
||||
|
||||
Reference in new issue
Block a user