Files
PaperClipAI/ui/src/api/errors.test.ts
T
DottaandClaude Opus 5.5 b033cc124d fix(evals): repair private human steps for webhook, device and key entry; name failed private steps (#21012)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat evaluation catalog checks agent replies and then the human
step that follows, such as a person who reveals a key or approves a
device
> - Some human steps run in a separate private browser process, because
they handle credentials. CI keeps the output of that process private on
purpose
> - Four cases failed only in that private step, so CI showed one fixed
message and no cause
> - I ran each private step locally against a dev server, where its
output is visible, and found the cause of three failures
> - This pull request fixes the harness and two product defects, and
adds a closed failure record so that the next CI failure names its step
> - The benefit is that these human steps work, and a private-step
failure is diagnosable without any credential in logs

## Linked Issues or Issue Description

Bug report (following `.github/ISSUE_TEMPLATE/bug_report.yml`).

**What happened?**
- `routines.webhook-trigger`: the private person opens the routine
overview. The trigger cards are only in the Triggers section, so the
step waits for a missing button. An unhandled response wait then stops
the process.
- `connection-setup.secure-api-key`: a key-protected MCP server answers
401 without a challenge header. Paperclip shows "Remote app returned
HTTP 401" and does not ask for a key. The person cannot enter the key in
the normal flow.
- `assistant-connections.device-login`: an expired device code page
shows "This item doesn't exist or was moved." The server sends "This
code is expired, already decided, or invalid." in the OAuth
`error_description` field, and the UI drops it.
- `onboarding.new-company`: the private step reads back all saved state
and then fails one proof check. CI does not say which check.

**Expected behavior**
- The person reveals the webhook key in the Triggers section and sends
authenticated events.
- A pasted MCP link that answers 401 shows the key field.
- An expired device code says that the code is expired.
- A private-step failure names its fixed step and error class, with no
value.

**Steps to reproduce**
1. Start a local dev server.
2. Create a routine with a bearer webhook, and run the private webhook
worker. It times out on the "Edit webhook" button.
3. Paste the address of an MCP server that answers a plain 401 into
Connectors > Add by link. No key field appears.
4. Open `/mcp-device?user_code=<expired code>`. The page shows generic
not-found text.

**Paperclip version or commit**
`master` (reproduced on origin/master before this change)

**Deployment mode**
Local dev server (embedded PGlite) and the CI chat-eval runner

**Installation method**
Source checkout (`pnpm dev`)

## What Changed

- `integrator-webhook-private-worker.ts`: open `<routine>/triggers`
(also for the setup-resume link), and attach a handler to the early
response wait so a failed click reports its real step.
- `server/.../catalog/discovery.ts`: a 401 with no `WWW-Authenticate`
header, to a connection that sends no credential, is handled like a
challenge without discoverable sign-in (`oauth_challenge`). The existing
UI guidance then asks for a key. A 401 to a connection that already
sends a credential keeps the old error.
- `ui/src/api/errors.ts`: read the OAuth `error_description` field as a
readable server message.
- New `private-child-failure.ts`: a private child records only a fixed
step name and an error class from a fixed list. The parent echoes a step
only when it is in the caller's closed list. Error text, stdout and
stderr stay private.
- The webhook, private browser and identity children use it. Each has a
closed step list file.
- `identity-new-company-native-proof.ts`: each proof check has a fixed
reason name. The private child reports `new-company-proof-<reason>` when
a check fails. The accepted conditions do not change.
- Tests: server red/green test for the bare 401, UI red/green test for
`error_description`, unit tests for the failure record (no text leaks,
closed list only, extra fields dropped).

## Verification

- Local dev server (embedded PostgreSQL), private workers run directly
with the real child process:
- Webhook, ready and setup-pending triggers: before, timeout on "Edit
webhook". After, receipt with invalid status 401, valid status 202 and a
linked task.
- Secure key entry with the synthetic notes provider: before, timeout on
the key field. After, an active connection with health `ok` and 2
authorized `tools/list` calls.
- Expired device page: before, `at step device-expired-page
(TimeoutError)`. After, `expiredPageVisible: true`.
- `server`: `vitest run src/__tests__/tool-access-service.test.ts -t
"pasted link answers"` (fails before the discovery change with 502,
passes after).
- `ui`: `vitest run src/api/errors.test.ts` (fails before, passes
after).
- `pnpm test:e2e:runner:typecheck`, `pnpm test:e2e:runner:unit`, server
`tsc --noEmit`.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD sha>
--check`: unchanged.
- One paid run of the four cases on this branch. Results are in a PR
comment.

## Risks

- A plain 401 from an MCP link now returns 422 `oauth_challenge` instead
of 502. The draft connection is removed, as for other challenges. Agents
that call the connect API see the "needs sign-in" answer instead of a
raw status. This only applies when the connection sends no credential.
- Some API errors that carry `error_description` now show that text
instead of generic copy. That text is written for people.
- The failure record is opt-in per child and contains only constants.
The parent ignores a record with any other field.
- `onboarding.new-company` is not fixed here. Its native hello probe
cannot start locally on macOS, so the failing check is not yet known.
This change makes the next run name it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edits, local browser automation through Playwright). Extended
reasoning was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:06:29 -06:00

203 lines
8.8 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import { CancelledError } from "@tanstack/react-query";
import { describe, expect, it } from "vitest";
import { ApiError } from "./api-error";
import {
apiErrorMessage,
classifyError,
describeError,
errorCodes,
isConnectivityError,
isRawErrorCode,
parseRetryAfter,
type ErrorKind,
} from "./errors";
import { ApiUnavailableError } from "./response";
function apiError(status: number, body: unknown = null) {
return new ApiError(apiErrorMessage(status, body), status, body);
}
describe("classifyError", () => {
it.each<[string, unknown, ErrorKind]>([
["a browser network failure", new TypeError("Failed to fetch"), "transient"],
["a Safari network failure", new TypeError("Load failed"), "transient"],
["a proxy HTML page", new ApiUnavailableError(502), "transient"],
["408", apiError(408), "transient"],
["425", apiError(425), "transient"],
["429", apiError(429, { error: "Too many requests" }), "transient"],
["502", apiError(502), "transient"],
["503", apiError(503, { error: "Service unavailable" }), "transient"],
["504", apiError(504), "transient"],
["401", apiError(401, { error: "Unauthorized" }), "auth"],
["403", apiError(403, { error: "Forbidden" }), "forbidden"],
["404", apiError(404, { error: "Issue not found" }), "not_found"],
["410", apiError(410), "not_found"],
["409", apiError(409, { error: "Document is locked" }), "conflict"],
["412", apiError(412), "conflict"],
["400", apiError(400, { error: "Title is required" }), "invalid"],
["422", apiError(422), "invalid"],
["500", apiError(500, { error: "boom" }), "unknown"],
["an abort", new DOMException("Aborted", "AbortError"), "aborted"],
["a React Query cancellation", new CancelledError({ revert: false }), "aborted"],
["a programming TypeError", new TypeError("Cannot read properties of undefined"), "unknown"],
["a plain Error", new Error("nope"), "unknown"],
["undefined", undefined, "unknown"],
])("classifies %s", (_label, error, kind) => {
expect(classifyError(error)).toBe(kind);
});
it.each([200, 400, 403, 404, 409, 500])(
"treats the gateway's tenant_app_unavailable as transient in any status (%s)",
(status) => {
expect(classifyError(apiError(status, { error: "tenant_app_unavailable" }))).toBe("transient");
expect(classifyError(apiError(status, { error: "tenant_app_starting" }))).toBe("transient");
},
);
it.each(["WORKER_UNAVAILABLE", "TIMEOUT"])("treats plugin bridge %s as transient", (code) => {
expect(classifyError(apiError(500, { code, message: "Plugin worker is restarting" }))).toBe("transient");
});
it("reads a code from details, as HttpErrors serialize it", () => {
const error = apiError(409, { error: "Locked", details: { code: "tenant_app_starting" } });
expect(classifyError(error)).toBe("transient");
});
it("classifies structurally, without the ApiError class", () => {
expect(classifyError({ status: 503, body: null })).toBe("transient");
expect(classifyError({ name: "AuthApiError", status: 401, code: null })).toBe("auth");
});
});
describe("isConnectivityError", () => {
it("is true for server-wide outages only", () => {
expect(isConnectivityError(new TypeError("Failed to fetch"))).toBe(true);
expect(isConnectivityError(new ApiUnavailableError(200))).toBe(true);
expect(isConnectivityError(apiError(503))).toBe(true);
expect(isConnectivityError(apiError(404, { error: "tenant_app_unavailable" }))).toBe(true);
// Transient, but one route: left to query retry, not the app banner.
expect(isConnectivityError(apiError(429))).toBe(false);
expect(isConnectivityError(apiError(503, { code: "WORKER_UNAVAILABLE", message: "x" }))).toBe(false);
expect(isConnectivityError(apiError(500))).toBe(false);
expect(isConnectivityError(apiError(403))).toBe(false);
});
});
describe("describeError", () => {
const rawCode = /\b[a-z]+(?:_[a-z]+)+\b|\b[A-Z]+_[A-Z]+\b/;
it("never returns a raw code", () => {
const errors = [
apiError(503, { error: "tenant_app_unavailable" }),
apiError(200, { error: "tenant_app_starting" }),
apiError(404, { error: "issue_not_found" }),
apiError(500, { error: "internal_error" }),
apiError(503, { code: "WORKER_UNAVAILABLE", message: "WORKER_UNAVAILABLE" }),
new ApiError("tenant_app_unavailable", 503, { error: "tenant_app_unavailable" }),
];
for (const error of errors) {
const { title, body } = describeError(error);
expect(title).not.toMatch(rawCode);
expect(body).not.toMatch(rawCode);
}
});
it("uses the shared transient copy for outages", () => {
expect(describeError(new ApiUnavailableError(502))).toEqual({
title: "Connection interrupted",
body: "Paperclip is temporarily unavailable. Please try again in a moment.",
retryable: true,
});
expect(describeError(apiError(503, { error: "tenant_app_unavailable" }))).toMatchObject({
title: "Reconnecting to Paperclip",
retryable: true,
});
});
it("keeps a readable server message for client errors", () => {
expect(describeError(apiError(400, { error: "Title is required" }))).toEqual({
title: "Check your input",
body: "Title is required",
retryable: false,
});
});
it("keeps an OAuth error description instead of generic not-found copy", () => {
const error = apiError(404, {
error: "invalid_request",
error_description: "This code is expired, already decided, or invalid. Start a new connection from your assistant.",
});
expect(describeError(error).body).toBe("This code is expired, already decided, or invalid. Start a new connection from your assistant.");
expect(error.message).toBe("This code is expired, already decided, or invalid. Start a new connection from your assistant.");
});
it("falls back to the kind's copy when the server sent a code or nothing", () => {
expect(describeError(apiError(404, { error: "issue_not_found" })).body).toBe("This item doesn’t exist or was moved.");
expect(describeError(apiError(403, null))).toMatchObject({ title: "Not allowed", retryable: false });
expect(describeError(new Error("Something odd"))).toMatchObject({ body: "Something odd", retryable: true });
expect(describeError(undefined)).toMatchObject({ title: "Something went wrong", retryable: true });
});
it("names the action in the title for real failures, not outages", () => {
expect(describeError(apiError(409, { error: "Document is locked" }), { action: "save the document" })).toEqual({
title: "Couldn't save the document",
body: "Document is locked",
retryable: false,
});
expect(describeError(apiError(503), { action: "save the document" }).title).toBe("Connection interrupted");
});
});
describe("ApiError", () => {
it("keeps the raw body.error as code and a readable message", () => {
const error = apiError(503, { error: "tenant_app_unavailable" });
expect(error.code).toBe("tenant_app_unavailable");
expect(error.message).toBe("Paperclip is restarting or updating. Please try again in a moment.");
});
it("passes a readable server message through unchanged", () => {
const error = apiError(409, { error: "Document is locked" });
expect(error.code).toBe("Document is locked");
expect(error.message).toBe("Document is locked");
});
it("replaces the generic status line with readable copy", () => {
expect(apiError(404).message).toBe("This item doesn’t exist or was moved.");
expect(apiError(404).code).toBeNull();
});
it("exposes every code for matching", () => {
expect(errorCodes(apiError(403, { error: "Denied", code: "interaction_human_only", details: { code: "x_y" } })))
.toEqual(["Denied", "interaction_human_only", "x_y"]);
});
});
describe("isRawErrorCode", () => {
it.each(["tenant_app_unavailable", "WORKER_UNAVAILABLE", "TIMEOUT", "issue.not_found", "forbidden", "unauthorized", "conflict"])("%s is a code", (value) => {
expect(isRawErrorCode(value)).toBe(true);
});
it.each(["Forbidden", "Unauthorized", "Document is locked", "Request failed: 503", ""])("%s is not a code", (value) => {
expect(isRawErrorCode(value)).toBe(false);
});
});
describe("parseRetryAfter", () => {
it("parses delta-seconds", () => {
expect(parseRetryAfter("5")).toBe(5_000);
expect(parseRetryAfter(" 1.5 ")).toBe(1_500);
});
it("parses an HTTP date relative to now", () => {
const now = Date.parse("2026-10-09T12:00:00Z");
expect(parseRetryAfter("Fri, 09 Oct 2026 12:00:30 GMT", now)).toBe(30_000);
expect(parseRetryAfter("Fri, 09 Oct 2026 11:00:00 GMT", now)).toBe(0);
});
it("caps absurd values and ignores garbage", () => {
expect(parseRetryAfter("86400")).toBe(5 * 60_000);
expect(parseRetryAfter("soon")).toBeNull();
expect(parseRetryAfter(null)).toBeNull();
});
});