mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 02:43:25 +01:00
fix(native-runtime): a timed-out step records its real cause instead of leaving the run unrecoverable (#21262)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs start a Runner and a provider session, then send the first turn > - When a startup step fails, the native session executor redacts the error, classifies it and records the failure > - A timed-out step fails with a `DOMException` (`TimeoutError` or `AbortError`). Its `message` property has only a getter > - The redaction assigned to `error.message`. The assignment threw a `TypeError`, so the classifier did not run and the real cause was lost > - The run then fell into a restart-recovery chain that ended in "requires reconciliation" wording for an operator > - This pull request makes the redaction safe for errors with a read-only message > - The benefit is that a timed-out step records its real cause and settles through the normal failure path ## Linked Issues or Issue Description **What happened?** In a local chaos cell with 8 opencode tasks on Daytona sandboxes, four first runs failed before their first turn. A step before the first turn ran under a five-minute `AbortSignal.timeout`. The step took longer than five minutes. The run log shows `TypeError: Cannot set property message of which has only a getter` at the startup failure handler, then `native_finalization_missing`. The run never recorded the timeout. Recovery then restarted the bootstrap, saw "runnerd exited while waiting for run.prepare", and ended with `runner_state_identity_mismatch: bootstrap_not_proven_incomplete` and `native_continuation_requires_reconciliation`. **Expected behavior** A failed startup step records its own cause. The failure handler classifies it and settles the run. It does not throw a second error that hides the first one. **Steps to reproduce** Make any step before the first turn reject with `AbortSignal.timeout(n).reason`. Before this change, `executePaperclipNativeSession` rejects with the `TypeError` above and the coordinator row keeps no failure detail. The new integration test does this with the embedded database. ## What Changed - Add `redactNativeFailureError` in `server/src/services/native-runtime/native-failure-redaction.ts`. It redacts the message and stack in place when it can. When the message is read-only, it returns a new `Error` with the redacted message, the same `name` and the redacted stack. - Use it in the startup failure handler of `executePaperclipNativeSession` in place of the direct assignment. - Add unit tests for a real `TimeoutError`, a read-only `AbortError` that holds a secret, and an ordinary typed error. - Add an integration test: a step before the first turn rejects with a real `TimeoutError`. The run settles through the classifier. `failureDetail.message` is "The operation was aborted due to timeout". The run error code is not `native_continuation_requires_reconciliation`. ## Verification - Red on master: `native-failure-redaction.test.ts` failed 2 of 3 with `TypeError: Cannot set property message of which has only a getter`. The new integration test in `native-startup-retry.integration.test.ts` failed: the call rejected with that `TypeError` and not with the `TimeoutError`. - Green with this change: - `pnpm --filter @paperclipai/server exec vitest run --config vitest.config.ts src/services/native-runtime/native-failure-redaction.test.ts src/services/native-runtime/native-startup-retry.integration.test.ts` - `src/services/native-runtime/native-session-executor.test.ts`: 1431 passed, 7 skipped. - `pnpm verify` for the affected files. ## Risks - Low risk. The change only alters behavior when the error message cannot be assigned. Before, that case threw a `TypeError` and skipped the failure handler. - The replacement error keeps the `name` but is a plain `Error`, so `instanceof DOMException` is false after redaction. No code in the handler checks for `DOMException`. - After this PR, a capture timeout before the first turn still ends blocked. The run records the real cause, but it shows generic wording and waits for a person to press Retry. The reason: a generic timeout has no typed cause, so the classifier records it as `native_session_interrupted` and treats it as ambiguous state. Follow-up PRs give the capture step a typed cause and per-step bounds, and retry a failure that provably happened before any prompt automatically. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
53608bd1d6
commit
04b50e6007
4 files changed
+86
-4
No files matched your search
@@ -0,0 +1,40 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { redactNativeFailureError } from "./native-failure-redaction.js";
|
||||
|
||||
const redact = (text: string) => text.replaceAll("agent-secret", "[redacted]");
|
||||
|
||||
describe("redactNativeFailureError", () => {
|
||||
it("keeps a timed-out transfer's own cause when its message cannot be reassigned", async () => {
|
||||
const signal = AbortSignal.timeout(1);
|
||||
await new Promise(resolve => setTimeout(resolve, 20));
|
||||
const timeout = signal.reason as Error;
|
||||
|
||||
const redacted = redactNativeFailureError(timeout, redact);
|
||||
|
||||
expect(redacted).toBeInstanceOf(Error);
|
||||
expect(redacted.name).toBe("TimeoutError");
|
||||
expect(redacted.message).toBe("The operation was aborted due to timeout");
|
||||
});
|
||||
|
||||
it("redacts a read-only message and stack without throwing", () => {
|
||||
const failure = new DOMException("upload for agent-secret stalled", "AbortError");
|
||||
|
||||
const redacted = redactNativeFailureError(failure, redact);
|
||||
|
||||
expect(redacted.name).toBe("AbortError");
|
||||
expect(redacted.message).toBe("upload for [redacted] stalled");
|
||||
expect(redacted.stack ?? "").not.toContain("agent-secret");
|
||||
});
|
||||
|
||||
it("redacts an ordinary error in place and keeps its type", () => {
|
||||
class TypedFailure extends Error {}
|
||||
const failure = new TypedFailure("provider rejected agent-secret");
|
||||
|
||||
const redacted = redactNativeFailureError(failure, redact);
|
||||
|
||||
expect(redacted).toBe(failure);
|
||||
expect(redacted).toBeInstanceOf(TypedFailure);
|
||||
expect(redacted.message).toBe("provider rejected [redacted]");
|
||||
expect(redacted.stack).not.toContain("agent-secret");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,17 @@
|
||||
export function redactNativeFailureError(error: Error, redact: (text: string) => string): Error {
|
||||
const originalMessage = error.message;
|
||||
const message = redact(originalMessage);
|
||||
const stack = error.stack
|
||||
? redact(originalMessage ? error.stack.split(originalMessage).join(message) : error.stack)
|
||||
: undefined;
|
||||
try {
|
||||
error.message = message;
|
||||
if (stack !== undefined) error.stack = stack;
|
||||
return error;
|
||||
} catch {
|
||||
const replacement = new Error(message);
|
||||
replacement.name = error.name;
|
||||
replacement.stack = stack ?? `${error.name}: ${message}`;
|
||||
return replacement;
|
||||
}
|
||||
}
|
||||
@@ -100,6 +100,7 @@ import { copyBackGrokAuth } from "@paperclipai/adapter-grok-local/server";
|
||||
|
||||
import { inferOpenAiCompatibleBiller, type AdapterUsageCheckpoint } from "@paperclipai/adapter-utils";
|
||||
import { createAgentIdentityRedactor } from "../agent-identity-redaction.js";
|
||||
import { redactNativeFailureError } from "./native-failure-redaction.js";
|
||||
import { parseCodexCliVersion } from "./codex-runtime-compatibility.js";
|
||||
import { createNativeToolTrace, type NativeToolTrace } from "./native-tool-trace.js";
|
||||
import { createNativeProviderFailureObservation } from "./native-provider-failure.js";
|
||||
@@ -12729,10 +12730,7 @@ async function executePaperclipNativeSessionAttempt(
|
||||
// Preserve typed failure semantics while removing credentials before any
|
||||
// persistence, reporting, logging, or rethrow at this boundary.
|
||||
if (error instanceof Error) {
|
||||
const originalMessage = error.message;
|
||||
error.message = redactIdentityText(originalMessage);
|
||||
if (error.stack) error.stack = redactIdentityText(originalMessage
|
||||
? error.stack.split(originalMessage).join(error.message) : error.stack);
|
||||
error = redactNativeFailureError(error, redactIdentityText);
|
||||
} else {
|
||||
error = identityRedactor.redact(error);
|
||||
}
|
||||
|
||||
@@ -432,6 +432,33 @@ describe("native provider startup retry", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("records a timed-out step's own cause and settles the run through the failure classifier", async () => {
|
||||
const seeded = await seedRun(await createAgent());
|
||||
const expired = AbortSignal.timeout(1);
|
||||
await new Promise(resolve => setTimeout(resolve, 20));
|
||||
provider.execute.mockReset().mockImplementation(async () => { throw expired.reason; });
|
||||
await expect(executePaperclipNativeSession({ db, execution: seeded.execution,
|
||||
runnerInstanceId: seeded.runnerInstanceId,
|
||||
backend: {
|
||||
descriptor: async () => { throw new Error("Unexpected provider descriptor call"); },
|
||||
openSession: async () => { throw new Error("Unexpected provider session call"); },
|
||||
} })).rejects.toMatchObject({ name: "TimeoutError", message: "The operation was aborted due to timeout" });
|
||||
|
||||
const state = await read(seeded);
|
||||
expect(state.coordinator).toMatchObject({
|
||||
attempt: 1,
|
||||
leaseOwner: null,
|
||||
failureDetail: expect.objectContaining({
|
||||
message: "The operation was aborted due to timeout",
|
||||
originalFailureCode: "native_session_interrupted",
|
||||
providerSessionEstablished: false,
|
||||
providerEventsExist: false,
|
||||
}),
|
||||
});
|
||||
expect(state.run).toMatchObject({ status: "failed", errorCode: state.coordinator.failureCode });
|
||||
expect(state.run.errorCode).not.toBe("native_continuation_requires_reconciliation");
|
||||
});
|
||||
|
||||
it("retries a rejected session.open with growing backoff instead of blocking the task", async () => {
|
||||
const seeded = await seedRun(await createAgent());
|
||||
|
||||
|
||||
Reference in new issue
Block a user