mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat evaluation catalog checks agent replies and then the human step that follows, such as a person who reveals a key or approves a device > - Some human steps run in a separate private browser process, because they handle credentials. CI keeps the output of that process private on purpose > - Four cases failed only in that private step, so CI showed one fixed message and no cause > - I ran each private step locally against a dev server, where its output is visible, and found the cause of three failures > - This pull request fixes the harness and two product defects, and adds a closed failure record so that the next CI failure names its step > - The benefit is that these human steps work, and a private-step failure is diagnosable without any credential in logs ## Linked Issues or Issue Description Bug report (following `.github/ISSUE_TEMPLATE/bug_report.yml`). **What happened?** - `routines.webhook-trigger`: the private person opens the routine overview. The trigger cards are only in the Triggers section, so the step waits for a missing button. An unhandled response wait then stops the process. - `connection-setup.secure-api-key`: a key-protected MCP server answers 401 without a challenge header. Paperclip shows "Remote app returned HTTP 401" and does not ask for a key. The person cannot enter the key in the normal flow. - `assistant-connections.device-login`: an expired device code page shows "This item doesn't exist or was moved." The server sends "This code is expired, already decided, or invalid." in the OAuth `error_description` field, and the UI drops it. - `onboarding.new-company`: the private step reads back all saved state and then fails one proof check. CI does not say which check. **Expected behavior** - The person reveals the webhook key in the Triggers section and sends authenticated events. - A pasted MCP link that answers 401 shows the key field. - An expired device code says that the code is expired. - A private-step failure names its fixed step and error class, with no value. **Steps to reproduce** 1. Start a local dev server. 2. Create a routine with a bearer webhook, and run the private webhook worker. It times out on the "Edit webhook" button. 3. Paste the address of an MCP server that answers a plain 401 into Connectors > Add by link. No key field appears. 4. Open `/mcp-device?user_code=<expired code>`. The page shows generic not-found text. **Paperclip version or commit** `master` (reproduced on origin/master before this change) **Deployment mode** Local dev server (embedded PGlite) and the CI chat-eval runner **Installation method** Source checkout (`pnpm dev`) ## What Changed - `integrator-webhook-private-worker.ts`: open `<routine>/triggers` (also for the setup-resume link), and attach a handler to the early response wait so a failed click reports its real step. - `server/.../catalog/discovery.ts`: a 401 with no `WWW-Authenticate` header, to a connection that sends no credential, is handled like a challenge without discoverable sign-in (`oauth_challenge`). The existing UI guidance then asks for a key. A 401 to a connection that already sends a credential keeps the old error. - `ui/src/api/errors.ts`: read the OAuth `error_description` field as a readable server message. - New `private-child-failure.ts`: a private child records only a fixed step name and an error class from a fixed list. The parent echoes a step only when it is in the caller's closed list. Error text, stdout and stderr stay private. - The webhook, private browser and identity children use it. Each has a closed step list file. - `identity-new-company-native-proof.ts`: each proof check has a fixed reason name. The private child reports `new-company-proof-<reason>` when a check fails. The accepted conditions do not change. - Tests: server red/green test for the bare 401, UI red/green test for `error_description`, unit tests for the failure record (no text leaks, closed list only, extra fields dropped). ## Verification - Local dev server (embedded PostgreSQL), private workers run directly with the real child process: - Webhook, ready and setup-pending triggers: before, timeout on "Edit webhook". After, receipt with invalid status 401, valid status 202 and a linked task. - Secure key entry with the synthetic notes provider: before, timeout on the key field. After, an active connection with health `ok` and 2 authorized `tools/list` calls. - Expired device page: before, `at step device-expired-page (TimeoutError)`. After, `expiredPageVisible: true`. - `server`: `vitest run src/__tests__/tool-access-service.test.ts -t "pasted link answers"` (fails before the discovery change with 502, passes after). - `ui`: `vitest run src/api/errors.test.ts` (fails before, passes after). - `pnpm test:e2e:runner:typecheck`, `pnpm test:e2e:runner:unit`, server `tsc --noEmit`. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD sha> --check`: unchanged. - One paid run of the four cases on this branch. Results are in a PR comment. ## Risks - A plain 401 from an MCP link now returns 422 `oauth_challenge` instead of 502. The draft connection is removed, as for other challenges. Agents that call the connect API see the "needs sign-in" answer instead of a raw status. This only applies when the connection sends no credential. - Some API errors that carry `error_description` now show that text instead of generic copy. That text is written for people. - The failure record is opt-in per child and contains only constants. The parent ignores a record with any other field. - `onboarding.new-company` is not fixed here. Its native hello probe cannot start locally on macOS, so the failing check is not yet known. This change makes the next run name it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits, local browser automation through Playwright). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
203 lines
8.8 KiB
TypeScript
203 lines
8.8 KiB
TypeScript
import { CancelledError } from "@tanstack/react-query";
|
||
import { describe, expect, it } from "vitest";
|
||
import { ApiError } from "./api-error";
|
||
import {
|
||
apiErrorMessage,
|
||
classifyError,
|
||
describeError,
|
||
errorCodes,
|
||
isConnectivityError,
|
||
isRawErrorCode,
|
||
parseRetryAfter,
|
||
type ErrorKind,
|
||
} from "./errors";
|
||
import { ApiUnavailableError } from "./response";
|
||
|
||
function apiError(status: number, body: unknown = null) {
|
||
return new ApiError(apiErrorMessage(status, body), status, body);
|
||
}
|
||
|
||
describe("classifyError", () => {
|
||
it.each<[string, unknown, ErrorKind]>([
|
||
["a browser network failure", new TypeError("Failed to fetch"), "transient"],
|
||
["a Safari network failure", new TypeError("Load failed"), "transient"],
|
||
["a proxy HTML page", new ApiUnavailableError(502), "transient"],
|
||
["408", apiError(408), "transient"],
|
||
["425", apiError(425), "transient"],
|
||
["429", apiError(429, { error: "Too many requests" }), "transient"],
|
||
["502", apiError(502), "transient"],
|
||
["503", apiError(503, { error: "Service unavailable" }), "transient"],
|
||
["504", apiError(504), "transient"],
|
||
["401", apiError(401, { error: "Unauthorized" }), "auth"],
|
||
["403", apiError(403, { error: "Forbidden" }), "forbidden"],
|
||
["404", apiError(404, { error: "Issue not found" }), "not_found"],
|
||
["410", apiError(410), "not_found"],
|
||
["409", apiError(409, { error: "Document is locked" }), "conflict"],
|
||
["412", apiError(412), "conflict"],
|
||
["400", apiError(400, { error: "Title is required" }), "invalid"],
|
||
["422", apiError(422), "invalid"],
|
||
["500", apiError(500, { error: "boom" }), "unknown"],
|
||
["an abort", new DOMException("Aborted", "AbortError"), "aborted"],
|
||
["a React Query cancellation", new CancelledError({ revert: false }), "aborted"],
|
||
["a programming TypeError", new TypeError("Cannot read properties of undefined"), "unknown"],
|
||
["a plain Error", new Error("nope"), "unknown"],
|
||
["undefined", undefined, "unknown"],
|
||
])("classifies %s", (_label, error, kind) => {
|
||
expect(classifyError(error)).toBe(kind);
|
||
});
|
||
|
||
it.each([200, 400, 403, 404, 409, 500])(
|
||
"treats the gateway's tenant_app_unavailable as transient in any status (%s)",
|
||
(status) => {
|
||
expect(classifyError(apiError(status, { error: "tenant_app_unavailable" }))).toBe("transient");
|
||
expect(classifyError(apiError(status, { error: "tenant_app_starting" }))).toBe("transient");
|
||
},
|
||
);
|
||
|
||
it.each(["WORKER_UNAVAILABLE", "TIMEOUT"])("treats plugin bridge %s as transient", (code) => {
|
||
expect(classifyError(apiError(500, { code, message: "Plugin worker is restarting" }))).toBe("transient");
|
||
});
|
||
|
||
it("reads a code from details, as HttpErrors serialize it", () => {
|
||
const error = apiError(409, { error: "Locked", details: { code: "tenant_app_starting" } });
|
||
expect(classifyError(error)).toBe("transient");
|
||
});
|
||
|
||
it("classifies structurally, without the ApiError class", () => {
|
||
expect(classifyError({ status: 503, body: null })).toBe("transient");
|
||
expect(classifyError({ name: "AuthApiError", status: 401, code: null })).toBe("auth");
|
||
});
|
||
});
|
||
|
||
describe("isConnectivityError", () => {
|
||
it("is true for server-wide outages only", () => {
|
||
expect(isConnectivityError(new TypeError("Failed to fetch"))).toBe(true);
|
||
expect(isConnectivityError(new ApiUnavailableError(200))).toBe(true);
|
||
expect(isConnectivityError(apiError(503))).toBe(true);
|
||
expect(isConnectivityError(apiError(404, { error: "tenant_app_unavailable" }))).toBe(true);
|
||
// Transient, but one route: left to query retry, not the app banner.
|
||
expect(isConnectivityError(apiError(429))).toBe(false);
|
||
expect(isConnectivityError(apiError(503, { code: "WORKER_UNAVAILABLE", message: "x" }))).toBe(false);
|
||
expect(isConnectivityError(apiError(500))).toBe(false);
|
||
expect(isConnectivityError(apiError(403))).toBe(false);
|
||
});
|
||
});
|
||
|
||
describe("describeError", () => {
|
||
const rawCode = /\b[a-z]+(?:_[a-z]+)+\b|\b[A-Z]+_[A-Z]+\b/;
|
||
|
||
it("never returns a raw code", () => {
|
||
const errors = [
|
||
apiError(503, { error: "tenant_app_unavailable" }),
|
||
apiError(200, { error: "tenant_app_starting" }),
|
||
apiError(404, { error: "issue_not_found" }),
|
||
apiError(500, { error: "internal_error" }),
|
||
apiError(503, { code: "WORKER_UNAVAILABLE", message: "WORKER_UNAVAILABLE" }),
|
||
new ApiError("tenant_app_unavailable", 503, { error: "tenant_app_unavailable" }),
|
||
];
|
||
for (const error of errors) {
|
||
const { title, body } = describeError(error);
|
||
expect(title).not.toMatch(rawCode);
|
||
expect(body).not.toMatch(rawCode);
|
||
}
|
||
});
|
||
|
||
it("uses the shared transient copy for outages", () => {
|
||
expect(describeError(new ApiUnavailableError(502))).toEqual({
|
||
title: "Connection interrupted",
|
||
body: "Paperclip is temporarily unavailable. Please try again in a moment.",
|
||
retryable: true,
|
||
});
|
||
expect(describeError(apiError(503, { error: "tenant_app_unavailable" }))).toMatchObject({
|
||
title: "Reconnecting to Paperclip",
|
||
retryable: true,
|
||
});
|
||
});
|
||
|
||
it("keeps a readable server message for client errors", () => {
|
||
expect(describeError(apiError(400, { error: "Title is required" }))).toEqual({
|
||
title: "Check your input",
|
||
body: "Title is required",
|
||
retryable: false,
|
||
});
|
||
});
|
||
|
||
it("keeps an OAuth error description instead of generic not-found copy", () => {
|
||
const error = apiError(404, {
|
||
error: "invalid_request",
|
||
error_description: "This code is expired, already decided, or invalid. Start a new connection from your assistant.",
|
||
});
|
||
expect(describeError(error).body).toBe("This code is expired, already decided, or invalid. Start a new connection from your assistant.");
|
||
expect(error.message).toBe("This code is expired, already decided, or invalid. Start a new connection from your assistant.");
|
||
});
|
||
|
||
it("falls back to the kind's copy when the server sent a code or nothing", () => {
|
||
expect(describeError(apiError(404, { error: "issue_not_found" })).body).toBe("This item doesn’t exist or was moved.");
|
||
expect(describeError(apiError(403, null))).toMatchObject({ title: "Not allowed", retryable: false });
|
||
expect(describeError(new Error("Something odd"))).toMatchObject({ body: "Something odd", retryable: true });
|
||
expect(describeError(undefined)).toMatchObject({ title: "Something went wrong", retryable: true });
|
||
});
|
||
|
||
it("names the action in the title for real failures, not outages", () => {
|
||
expect(describeError(apiError(409, { error: "Document is locked" }), { action: "save the document" })).toEqual({
|
||
title: "Couldn't save the document",
|
||
body: "Document is locked",
|
||
retryable: false,
|
||
});
|
||
expect(describeError(apiError(503), { action: "save the document" }).title).toBe("Connection interrupted");
|
||
});
|
||
});
|
||
|
||
describe("ApiError", () => {
|
||
it("keeps the raw body.error as code and a readable message", () => {
|
||
const error = apiError(503, { error: "tenant_app_unavailable" });
|
||
expect(error.code).toBe("tenant_app_unavailable");
|
||
expect(error.message).toBe("Paperclip is restarting or updating. Please try again in a moment.");
|
||
});
|
||
|
||
it("passes a readable server message through unchanged", () => {
|
||
const error = apiError(409, { error: "Document is locked" });
|
||
expect(error.code).toBe("Document is locked");
|
||
expect(error.message).toBe("Document is locked");
|
||
});
|
||
|
||
it("replaces the generic status line with readable copy", () => {
|
||
expect(apiError(404).message).toBe("This item doesn’t exist or was moved.");
|
||
expect(apiError(404).code).toBeNull();
|
||
});
|
||
|
||
it("exposes every code for matching", () => {
|
||
expect(errorCodes(apiError(403, { error: "Denied", code: "interaction_human_only", details: { code: "x_y" } })))
|
||
.toEqual(["Denied", "interaction_human_only", "x_y"]);
|
||
});
|
||
});
|
||
|
||
describe("isRawErrorCode", () => {
|
||
it.each(["tenant_app_unavailable", "WORKER_UNAVAILABLE", "TIMEOUT", "issue.not_found", "forbidden", "unauthorized", "conflict"])("%s is a code", (value) => {
|
||
expect(isRawErrorCode(value)).toBe(true);
|
||
});
|
||
|
||
it.each(["Forbidden", "Unauthorized", "Document is locked", "Request failed: 503", ""])("%s is not a code", (value) => {
|
||
expect(isRawErrorCode(value)).toBe(false);
|
||
});
|
||
});
|
||
|
||
describe("parseRetryAfter", () => {
|
||
it("parses delta-seconds", () => {
|
||
expect(parseRetryAfter("5")).toBe(5_000);
|
||
expect(parseRetryAfter(" 1.5 ")).toBe(1_500);
|
||
});
|
||
|
||
it("parses an HTTP date relative to now", () => {
|
||
const now = Date.parse("2026-10-09T12:00:00Z");
|
||
expect(parseRetryAfter("Fri, 09 Oct 2026 12:00:30 GMT", now)).toBe(30_000);
|
||
expect(parseRetryAfter("Fri, 09 Oct 2026 11:00:00 GMT", now)).toBe(0);
|
||
});
|
||
|
||
it("caps absurd values and ignores garbage", () => {
|
||
expect(parseRetryAfter("86400")).toBe(5 * 60_000);
|
||
expect(parseRetryAfter("soon")).toBeNull();
|
||
expect(parseRetryAfter(null)).toBeNull();
|
||
});
|
||
});
|