fix(native-runtime): a timed-out step records its real cause instead of leaving the run unrecoverable (#21262)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs start a Runner and a provider session, then send the
first turn
> - When a startup step fails, the native session executor redacts the
error, classifies it and records the failure
> - A timed-out step fails with a `DOMException` (`TimeoutError` or
`AbortError`). Its `message` property has only a getter
> - The redaction assigned to `error.message`. The assignment threw a
`TypeError`, so the classifier did not run and the real cause was lost
> - The run then fell into a restart-recovery chain that ended in
"requires reconciliation" wording for an operator
> - This pull request makes the redaction safe for errors with a
read-only message
> - The benefit is that a timed-out step records its real cause and
settles through the normal failure path

## Linked Issues or Issue Description

**What happened?**

In a local chaos cell with 8 opencode tasks on Daytona sandboxes, four
first runs failed before their first turn. A step before the first turn
ran under a five-minute `AbortSignal.timeout`. The step took longer than
five minutes. The run log shows `TypeError: Cannot set property message
of which has only a getter` at the startup failure handler, then
`native_finalization_missing`. The run never recorded the timeout.
Recovery then restarted the bootstrap, saw "runnerd exited while waiting
for run.prepare", and ended with `runner_state_identity_mismatch:
bootstrap_not_proven_incomplete` and
`native_continuation_requires_reconciliation`.

**Expected behavior**

A failed startup step records its own cause. The failure handler
classifies it and settles the run. It does not throw a second error that
hides the first one.

**Steps to reproduce**

Make any step before the first turn reject with
`AbortSignal.timeout(n).reason`. Before this change,
`executePaperclipNativeSession` rejects with the `TypeError` above and
the coordinator row keeps no failure detail. The new integration test
does this with the embedded database.

## What Changed

- Add `redactNativeFailureError` in
`server/src/services/native-runtime/native-failure-redaction.ts`. It
redacts the message and stack in place when it can. When the message is
read-only, it returns a new `Error` with the redacted message, the same
`name` and the redacted stack.
- Use it in the startup failure handler of
`executePaperclipNativeSession` in place of the direct assignment.
- Add unit tests for a real `TimeoutError`, a read-only `AbortError`
that holds a secret, and an ordinary typed error.
- Add an integration test: a step before the first turn rejects with a
real `TimeoutError`. The run settles through the classifier.
`failureDetail.message` is "The operation was aborted due to timeout".
The run error code is not `native_continuation_requires_reconciliation`.

## Verification

- Red on master: `native-failure-redaction.test.ts` failed 2 of 3 with
`TypeError: Cannot set property message of which has only a getter`. The
new integration test in `native-startup-retry.integration.test.ts`
failed: the call rejected with that `TypeError` and not with the
`TimeoutError`.
- Green with this change:
- `pnpm --filter @paperclipai/server exec vitest run --config
vitest.config.ts
src/services/native-runtime/native-failure-redaction.test.ts
src/services/native-runtime/native-startup-retry.integration.test.ts`
- `src/services/native-runtime/native-session-executor.test.ts`: 1431
passed, 7 skipped.
  - `pnpm verify` for the affected files.

## Risks

- Low risk. The change only alters behavior when the error message
cannot be assigned. Before, that case threw a `TypeError` and skipped
the failure handler.
- The replacement error keeps the `name` but is a plain `Error`, so
`instanceof DOMException` is false after redaction. No code in the
handler checks for `DOMException`.
- After this PR, a capture timeout before the first turn still ends
blocked. The run records the real cause, but it shows generic wording
and waits for a person to press Retry. The reason: a generic timeout has
no typed cause, so the classifier records it as
`native_session_interrupted` and treats it as ambiguous state. Follow-up
PRs give the capture step a typed cause and per-step bounds, and retry a
failure that provably happened before any prompt automatically.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs). Extended reasoning
was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-12 21:01:04 -06:00
1 parent 53608bd1d6
commit 04b50e6007
4 files changed
+86 -4

No files matched your search

@@ -0,0 +1,40 @@
import { describe, expect, it } from "vitest";
import { redactNativeFailureError } from "./native-failure-redaction.js";
const redact = (text: string) => text.replaceAll("agent-secret", "[redacted]");
describe("redactNativeFailureError", () => {
it("keeps a timed-out transfer's own cause when its message cannot be reassigned", async () => {
const signal = AbortSignal.timeout(1);
await new Promise(resolve => setTimeout(resolve, 20));
const timeout = signal.reason as Error;
const redacted = redactNativeFailureError(timeout, redact);
expect(redacted).toBeInstanceOf(Error);
expect(redacted.name).toBe("TimeoutError");
expect(redacted.message).toBe("The operation was aborted due to timeout");
});
it("redacts a read-only message and stack without throwing", () => {
const failure = new DOMException("upload for agent-secret stalled", "AbortError");
const redacted = redactNativeFailureError(failure, redact);
expect(redacted.name).toBe("AbortError");
expect(redacted.message).toBe("upload for [redacted] stalled");
expect(redacted.stack ?? "").not.toContain("agent-secret");
});
it("redacts an ordinary error in place and keeps its type", () => {
class TypedFailure extends Error {}
const failure = new TypedFailure("provider rejected agent-secret");
const redacted = redactNativeFailureError(failure, redact);
expect(redacted).toBe(failure);
expect(redacted).toBeInstanceOf(TypedFailure);
expect(redacted.message).toBe("provider rejected [redacted]");
expect(redacted.stack).not.toContain("agent-secret");
});
});
@@ -0,0 +1,17 @@
export function redactNativeFailureError(error: Error, redact: (text: string) => string): Error {
const originalMessage = error.message;
const message = redact(originalMessage);
const stack = error.stack
? redact(originalMessage ? error.stack.split(originalMessage).join(message) : error.stack)
: undefined;
try {
error.message = message;
if (stack !== undefined) error.stack = stack;
return error;
} catch {
const replacement = new Error(message);
replacement.name = error.name;
replacement.stack = stack ?? `${error.name}: ${message}`;
return replacement;
}
}
@@ -100,6 +100,7 @@ import { copyBackGrokAuth } from "@paperclipai/adapter-grok-local/server";
import { inferOpenAiCompatibleBiller, type AdapterUsageCheckpoint } from "@paperclipai/adapter-utils";
import { createAgentIdentityRedactor } from "../agent-identity-redaction.js";
import { redactNativeFailureError } from "./native-failure-redaction.js";
import { parseCodexCliVersion } from "./codex-runtime-compatibility.js";
import { createNativeToolTrace, type NativeToolTrace } from "./native-tool-trace.js";
import { createNativeProviderFailureObservation } from "./native-provider-failure.js";
@@ -12729,10 +12730,7 @@ async function executePaperclipNativeSessionAttempt(
// Preserve typed failure semantics while removing credentials before any
// persistence, reporting, logging, or rethrow at this boundary.
if (error instanceof Error) {
const originalMessage = error.message;
error.message = redactIdentityText(originalMessage);
if (error.stack) error.stack = redactIdentityText(originalMessage
? error.stack.split(originalMessage).join(error.message) : error.stack);
error = redactNativeFailureError(error, redactIdentityText);
} else {
error = identityRedactor.redact(error);
}
@@ -432,6 +432,33 @@ describe("native provider startup retry", () => {
}
});
it("records a timed-out step's own cause and settles the run through the failure classifier", async () => {
const seeded = await seedRun(await createAgent());
const expired = AbortSignal.timeout(1);
await new Promise(resolve => setTimeout(resolve, 20));
provider.execute.mockReset().mockImplementation(async () => { throw expired.reason; });
await expect(executePaperclipNativeSession({ db, execution: seeded.execution,
runnerInstanceId: seeded.runnerInstanceId,
backend: {
descriptor: async () => { throw new Error("Unexpected provider descriptor call"); },
openSession: async () => { throw new Error("Unexpected provider session call"); },
} })).rejects.toMatchObject({ name: "TimeoutError", message: "The operation was aborted due to timeout" });
const state = await read(seeded);
expect(state.coordinator).toMatchObject({
attempt: 1,
leaseOwner: null,
failureDetail: expect.objectContaining({
message: "The operation was aborted due to timeout",
originalFailureCode: "native_session_interrupted",
providerSessionEstablished: false,
providerEventsExist: false,
}),
});
expect(state.run).toMatchObject({ status: "failed", errorCode: state.coordinator.failureCode });
expect(state.run.errorCode).not.toBe("native_continuation_requires_reconciliation");
});
it("retries a rejected session.open with growing backoff instead of blocking the task", async () => {
const seeded = await seedRun(await createAgent());