From 04b50e60079862a815acf7866c69464f1dd8d9a2 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 12 Nov 2026 21:01:04 -0600 Subject: [PATCH] fix(native-runtime): a timed-out step records its real cause instead of leaving the run unrecoverable (#21262) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs start a Runner and a provider session, then send the first turn > - When a startup step fails, the native session executor redacts the error, classifies it and records the failure > - A timed-out step fails with a `DOMException` (`TimeoutError` or `AbortError`). Its `message` property has only a getter > - The redaction assigned to `error.message`. The assignment threw a `TypeError`, so the classifier did not run and the real cause was lost > - The run then fell into a restart-recovery chain that ended in "requires reconciliation" wording for an operator > - This pull request makes the redaction safe for errors with a read-only message > - The benefit is that a timed-out step records its real cause and settles through the normal failure path ## Linked Issues or Issue Description **What happened?** In a local chaos cell with 8 opencode tasks on Daytona sandboxes, four first runs failed before their first turn. A step before the first turn ran under a five-minute `AbortSignal.timeout`. The step took longer than five minutes. The run log shows `TypeError: Cannot set property message of which has only a getter` at the startup failure handler, then `native_finalization_missing`. The run never recorded the timeout. Recovery then restarted the bootstrap, saw "runnerd exited while waiting for run.prepare", and ended with `runner_state_identity_mismatch: bootstrap_not_proven_incomplete` and `native_continuation_requires_reconciliation`. **Expected behavior** A failed startup step records its own cause. The failure handler classifies it and settles the run. It does not throw a second error that hides the first one. **Steps to reproduce** Make any step before the first turn reject with `AbortSignal.timeout(n).reason`. Before this change, `executePaperclipNativeSession` rejects with the `TypeError` above and the coordinator row keeps no failure detail. The new integration test does this with the embedded database. ## What Changed - Add `redactNativeFailureError` in `server/src/services/native-runtime/native-failure-redaction.ts`. It redacts the message and stack in place when it can. When the message is read-only, it returns a new `Error` with the redacted message, the same `name` and the redacted stack. - Use it in the startup failure handler of `executePaperclipNativeSession` in place of the direct assignment. - Add unit tests for a real `TimeoutError`, a read-only `AbortError` that holds a secret, and an ordinary typed error. - Add an integration test: a step before the first turn rejects with a real `TimeoutError`. The run settles through the classifier. `failureDetail.message` is "The operation was aborted due to timeout". The run error code is not `native_continuation_requires_reconciliation`. ## Verification - Red on master: `native-failure-redaction.test.ts` failed 2 of 3 with `TypeError: Cannot set property message of which has only a getter`. The new integration test in `native-startup-retry.integration.test.ts` failed: the call rejected with that `TypeError` and not with the `TimeoutError`. - Green with this change: - `pnpm --filter @paperclipai/server exec vitest run --config vitest.config.ts src/services/native-runtime/native-failure-redaction.test.ts src/services/native-runtime/native-startup-retry.integration.test.ts` - `src/services/native-runtime/native-session-executor.test.ts`: 1431 passed, 7 skipped. - `pnpm verify` for the affected files. ## Risks - Low risk. The change only alters behavior when the error message cannot be assigned. Before, that case threw a `TypeError` and skipped the failure handler. - The replacement error keeps the `name` but is a plain `Error`, so `instanceof DOMException` is false after redaction. No code in the handler checks for `DOMException`. - After this PR, a capture timeout before the first turn still ends blocked. The run records the real cause, but it shows generic wording and waits for a person to press Retry. The reason: a generic timeout has no typed cause, so the classifier records it as `native_session_interrupted` and treats it as ambiguous state. Follow-up PRs give the capture step a typed cause and per-step bounds, and retry a failure that provably happened before any prompt automatically. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../native-failure-redaction.test.ts | 40 +++++++++++++++++++ .../native-failure-redaction.ts | 17 ++++++++ .../native-runtime/native-session-executor.ts | 6 +-- .../native-startup-retry.integration.test.ts | 27 +++++++++++++ 4 files changed, 86 insertions(+), 4 deletions(-) create mode 100644 server/src/services/native-runtime/native-failure-redaction.test.ts create mode 100644 server/src/services/native-runtime/native-failure-redaction.ts diff --git a/server/src/services/native-runtime/native-failure-redaction.test.ts b/server/src/services/native-runtime/native-failure-redaction.test.ts new file mode 100644 index 00000000000..a734c024cc3 --- /dev/null +++ b/server/src/services/native-runtime/native-failure-redaction.test.ts @@ -0,0 +1,40 @@ +import { describe, expect, it } from "vitest"; +import { redactNativeFailureError } from "./native-failure-redaction.js"; + +const redact = (text: string) => text.replaceAll("agent-secret", "[redacted]"); + +describe("redactNativeFailureError", () => { + it("keeps a timed-out transfer's own cause when its message cannot be reassigned", async () => { + const signal = AbortSignal.timeout(1); + await new Promise(resolve => setTimeout(resolve, 20)); + const timeout = signal.reason as Error; + + const redacted = redactNativeFailureError(timeout, redact); + + expect(redacted).toBeInstanceOf(Error); + expect(redacted.name).toBe("TimeoutError"); + expect(redacted.message).toBe("The operation was aborted due to timeout"); + }); + + it("redacts a read-only message and stack without throwing", () => { + const failure = new DOMException("upload for agent-secret stalled", "AbortError"); + + const redacted = redactNativeFailureError(failure, redact); + + expect(redacted.name).toBe("AbortError"); + expect(redacted.message).toBe("upload for [redacted] stalled"); + expect(redacted.stack ?? "").not.toContain("agent-secret"); + }); + + it("redacts an ordinary error in place and keeps its type", () => { + class TypedFailure extends Error {} + const failure = new TypedFailure("provider rejected agent-secret"); + + const redacted = redactNativeFailureError(failure, redact); + + expect(redacted).toBe(failure); + expect(redacted).toBeInstanceOf(TypedFailure); + expect(redacted.message).toBe("provider rejected [redacted]"); + expect(redacted.stack).not.toContain("agent-secret"); + }); +}); diff --git a/server/src/services/native-runtime/native-failure-redaction.ts b/server/src/services/native-runtime/native-failure-redaction.ts new file mode 100644 index 00000000000..cf1be0fce9b --- /dev/null +++ b/server/src/services/native-runtime/native-failure-redaction.ts @@ -0,0 +1,17 @@ +export function redactNativeFailureError(error: Error, redact: (text: string) => string): Error { + const originalMessage = error.message; + const message = redact(originalMessage); + const stack = error.stack + ? redact(originalMessage ? error.stack.split(originalMessage).join(message) : error.stack) + : undefined; + try { + error.message = message; + if (stack !== undefined) error.stack = stack; + return error; + } catch { + const replacement = new Error(message); + replacement.name = error.name; + replacement.stack = stack ?? `${error.name}: ${message}`; + return replacement; + } +} diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index db4c578406e..853df3f50b6 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -100,6 +100,7 @@ import { copyBackGrokAuth } from "@paperclipai/adapter-grok-local/server"; import { inferOpenAiCompatibleBiller, type AdapterUsageCheckpoint } from "@paperclipai/adapter-utils"; import { createAgentIdentityRedactor } from "../agent-identity-redaction.js"; +import { redactNativeFailureError } from "./native-failure-redaction.js"; import { parseCodexCliVersion } from "./codex-runtime-compatibility.js"; import { createNativeToolTrace, type NativeToolTrace } from "./native-tool-trace.js"; import { createNativeProviderFailureObservation } from "./native-provider-failure.js"; @@ -12729,10 +12730,7 @@ async function executePaperclipNativeSessionAttempt( // Preserve typed failure semantics while removing credentials before any // persistence, reporting, logging, or rethrow at this boundary. if (error instanceof Error) { - const originalMessage = error.message; - error.message = redactIdentityText(originalMessage); - if (error.stack) error.stack = redactIdentityText(originalMessage - ? error.stack.split(originalMessage).join(error.message) : error.stack); + error = redactNativeFailureError(error, redactIdentityText); } else { error = identityRedactor.redact(error); } diff --git a/server/src/services/native-runtime/native-startup-retry.integration.test.ts b/server/src/services/native-runtime/native-startup-retry.integration.test.ts index 4d75db6db41..f32a6bfd65f 100644 --- a/server/src/services/native-runtime/native-startup-retry.integration.test.ts +++ b/server/src/services/native-runtime/native-startup-retry.integration.test.ts @@ -432,6 +432,33 @@ describe("native provider startup retry", () => { } }); + it("records a timed-out step's own cause and settles the run through the failure classifier", async () => { + const seeded = await seedRun(await createAgent()); + const expired = AbortSignal.timeout(1); + await new Promise(resolve => setTimeout(resolve, 20)); + provider.execute.mockReset().mockImplementation(async () => { throw expired.reason; }); + await expect(executePaperclipNativeSession({ db, execution: seeded.execution, + runnerInstanceId: seeded.runnerInstanceId, + backend: { + descriptor: async () => { throw new Error("Unexpected provider descriptor call"); }, + openSession: async () => { throw new Error("Unexpected provider session call"); }, + } })).rejects.toMatchObject({ name: "TimeoutError", message: "The operation was aborted due to timeout" }); + + const state = await read(seeded); + expect(state.coordinator).toMatchObject({ + attempt: 1, + leaseOwner: null, + failureDetail: expect.objectContaining({ + message: "The operation was aborted due to timeout", + originalFailureCode: "native_session_interrupted", + providerSessionEstablished: false, + providerEventsExist: false, + }), + }); + expect(state.run).toMatchObject({ status: "failed", errorCode: state.coordinator.failureCode }); + expect(state.run.errorCode).not.toBe("native_continuation_requires_reconciliation"); + }); + it("retries a rejected session.open with growing backoff instead of blocking the task", async () => { const seeded = await seedRun(await createAgent());