mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-15 18:56:05 +01:00
fix(evals): repair private human steps for webhook, device and key entry; name failed private steps (#21012)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat evaluation catalog checks agent replies and then the human step that follows, such as a person who reveals a key or approves a device > - Some human steps run in a separate private browser process, because they handle credentials. CI keeps the output of that process private on purpose > - Four cases failed only in that private step, so CI showed one fixed message and no cause > - I ran each private step locally against a dev server, where its output is visible, and found the cause of three failures > - This pull request fixes the harness and two product defects, and adds a closed failure record so that the next CI failure names its step > - The benefit is that these human steps work, and a private-step failure is diagnosable without any credential in logs ## Linked Issues or Issue Description Bug report (following `.github/ISSUE_TEMPLATE/bug_report.yml`). **What happened?** - `routines.webhook-trigger`: the private person opens the routine overview. The trigger cards are only in the Triggers section, so the step waits for a missing button. An unhandled response wait then stops the process. - `connection-setup.secure-api-key`: a key-protected MCP server answers 401 without a challenge header. Paperclip shows "Remote app returned HTTP 401" and does not ask for a key. The person cannot enter the key in the normal flow. - `assistant-connections.device-login`: an expired device code page shows "This item doesn't exist or was moved." The server sends "This code is expired, already decided, or invalid." in the OAuth `error_description` field, and the UI drops it. - `onboarding.new-company`: the private step reads back all saved state and then fails one proof check. CI does not say which check. **Expected behavior** - The person reveals the webhook key in the Triggers section and sends authenticated events. - A pasted MCP link that answers 401 shows the key field. - An expired device code says that the code is expired. - A private-step failure names its fixed step and error class, with no value. **Steps to reproduce** 1. Start a local dev server. 2. Create a routine with a bearer webhook, and run the private webhook worker. It times out on the "Edit webhook" button. 3. Paste the address of an MCP server that answers a plain 401 into Connectors > Add by link. No key field appears. 4. Open `/mcp-device?user_code=<expired code>`. The page shows generic not-found text. **Paperclip version or commit** `master` (reproduced on origin/master before this change) **Deployment mode** Local dev server (embedded PGlite) and the CI chat-eval runner **Installation method** Source checkout (`pnpm dev`) ## What Changed - `integrator-webhook-private-worker.ts`: open `<routine>/triggers` (also for the setup-resume link), and attach a handler to the early response wait so a failed click reports its real step. - `server/.../catalog/discovery.ts`: a 401 with no `WWW-Authenticate` header, to a connection that sends no credential, is handled like a challenge without discoverable sign-in (`oauth_challenge`). The existing UI guidance then asks for a key. A 401 to a connection that already sends a credential keeps the old error. - `ui/src/api/errors.ts`: read the OAuth `error_description` field as a readable server message. - New `private-child-failure.ts`: a private child records only a fixed step name and an error class from a fixed list. The parent echoes a step only when it is in the caller's closed list. Error text, stdout and stderr stay private. - The webhook, private browser and identity children use it. Each has a closed step list file. - `identity-new-company-native-proof.ts`: each proof check has a fixed reason name. The private child reports `new-company-proof-<reason>` when a check fails. The accepted conditions do not change. - Tests: server red/green test for the bare 401, UI red/green test for `error_description`, unit tests for the failure record (no text leaks, closed list only, extra fields dropped). ## Verification - Local dev server (embedded PostgreSQL), private workers run directly with the real child process: - Webhook, ready and setup-pending triggers: before, timeout on "Edit webhook". After, receipt with invalid status 401, valid status 202 and a linked task. - Secure key entry with the synthetic notes provider: before, timeout on the key field. After, an active connection with health `ok` and 2 authorized `tools/list` calls. - Expired device page: before, `at step device-expired-page (TimeoutError)`. After, `expiredPageVisible: true`. - `server`: `vitest run src/__tests__/tool-access-service.test.ts -t "pasted link answers"` (fails before the discovery change with 502, passes after). - `ui`: `vitest run src/api/errors.test.ts` (fails before, passes after). - `pnpm test:e2e:runner:typecheck`, `pnpm test:e2e:runner:unit`, server `tsc --noEmit`. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD sha> --check`: unchanged. - One paid run of the four cases on this branch. Results are in a PR comment. ## Risks - A plain 401 from an MCP link now returns 422 `oauth_challenge` instead of 502. The draft connection is removed, as for other challenges. Agents that call the connect API see the "needs sign-in" answer instead of a raw status. This only applies when the connection sends no credential. - Some API errors that carry `error_description` now show that text instead of generic copy. That text is written for people. - The failure record is opt-in per child and contains only constants. The parent ignores a record with any other field. - `onboarding.new-company` is not fixed here. Its native hello probe cannot start locally on macOS, so the failing check is not yet known. This change makes the next run name it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits, local browser automation through Playwright). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
1bb4dcbe4b
commit
b033cc124d
18 files changed
+346
-70
No files matched your search
@@ -16625,6 +16625,76 @@ describeEmbeddedPostgres("tool access service", () => {
|
||||
await expect(db.select().from(toolConnections)).resolves.toHaveLength(0);
|
||||
});
|
||||
|
||||
it("asks for a key when a pasted link answers 401 without a challenge header", async () => {
|
||||
const company = await createCompany(db);
|
||||
const app = createRouteApp(db);
|
||||
vi.spyOn(globalThis, "fetch").mockResolvedValue({
|
||||
ok: false,
|
||||
status: 401,
|
||||
headers: { get: () => null },
|
||||
text: async () => "",
|
||||
json: async () => ({}),
|
||||
} as unknown as Response);
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/companies/${company.id}/tools/apps/connect`)
|
||||
.send({
|
||||
link: "https://key.example.test/mcp",
|
||||
name: "Key app",
|
||||
});
|
||||
|
||||
expect(res.status).toBe(422);
|
||||
expect(res.body).toMatchObject({
|
||||
details: expect.objectContaining({ code: "oauth_challenge" }),
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps the raw rejection when a pasted link's own URL token is refused", async () => {
|
||||
const company = await createCompany(db);
|
||||
const app = createRouteApp(db);
|
||||
vi.spyOn(globalThis, "fetch").mockResolvedValue({
|
||||
ok: false,
|
||||
status: 401,
|
||||
headers: { get: () => null },
|
||||
text: async () => "",
|
||||
json: async () => ({}),
|
||||
} as unknown as Response);
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/companies/${company.id}/tools/apps/connect`)
|
||||
.send({
|
||||
link: "https://key.example.test/mcp?token=expired-url-token",
|
||||
name: "URL token app",
|
||||
});
|
||||
|
||||
expect(res.body?.details?.code).not.toBe("oauth_challenge");
|
||||
expect(res.status).not.toBe(422);
|
||||
});
|
||||
|
||||
it("still asks for a key when a pasted link carries only plain settings", async () => {
|
||||
const company = await createCompany(db);
|
||||
const app = createRouteApp(db);
|
||||
vi.spyOn(globalThis, "fetch").mockResolvedValue({
|
||||
ok: false,
|
||||
status: 401,
|
||||
headers: { get: () => null },
|
||||
text: async () => "",
|
||||
json: async () => ({}),
|
||||
} as unknown as Response);
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/companies/${company.id}/tools/apps/connect`)
|
||||
.send({
|
||||
link: "https://key.example.test/mcp?region=us",
|
||||
name: "Regional key app",
|
||||
});
|
||||
|
||||
expect(res.status).toBe(422);
|
||||
expect(res.body).toMatchObject({
|
||||
details: expect.objectContaining({ code: "oauth_challenge" }),
|
||||
});
|
||||
});
|
||||
|
||||
it.each(["catalog", "catalog/refresh", "health-check"])(
|
||||
"explains disabled Slack MCP access on %s without reporting a server error",
|
||||
async (path) => {
|
||||
|
||||
@@ -35,6 +35,14 @@ import type { ConnectionSetupPolicy } from "../connections/setup-policy.js";
|
||||
import type { ToolRemoteHttp } from "../remote-http.js";
|
||||
import { normalizeToolDescriptor, type McpToolDescriptor } from "./descriptors.js";
|
||||
import { googleWorkspaceServicesUnavailable, unsupportedToolConnectionTransport } from "./failures.js";
|
||||
|
||||
const URL_CREDENTIAL_PARAMETER = /token|secret|passw|api[-_]?key|^key$|auth|signature|^sig$|session|credential/i;
|
||||
|
||||
/** Whether a sent URL carries a credential: user info, or a query parameter named like one. Plain settings such as `?region=us` do not. */
|
||||
export function urlCarriesCredential(url: URL): boolean {
|
||||
if (url.username || url.password) return true;
|
||||
return [...url.searchParams.keys()].some((key) => URL_CREDENTIAL_PARAMETER.test(key));
|
||||
}
|
||||
function asRecord(value: unknown): Record<string, unknown> {
|
||||
if (value && typeof value === "object" && !Array.isArray(value))
|
||||
return value as Record<string, unknown>;
|
||||
@@ -263,9 +271,23 @@ export function createToolCatalogDiscovery(db: Db, { now, audit, secrets, creden
|
||||
);
|
||||
}
|
||||
const authenticate = response.headers.get("www-authenticate") ?? "";
|
||||
// A bare 401 to a request that carried no credential is the server's
|
||||
// answer to "does it need a key?". Treat it like a challenge whose sign-in
|
||||
// Paperclip cannot discover, so the person is asked for a key instead of
|
||||
// seeing a raw HTTP status. A request that carried a credential (stored
|
||||
// secret, header, or a token or user info in the URL) keeps the raw
|
||||
// rejection: that credential needs replacing, not a new key.
|
||||
const sentUrl = new URL(endpoint);
|
||||
const bareCredentialChallenge =
|
||||
!authenticate.trim() &&
|
||||
connection.authKind === "none" &&
|
||||
connection.credentialSecretRefs.length === 0 &&
|
||||
Object.keys(headers).length === 0 &&
|
||||
!urlCarriesCredential(sentUrl);
|
||||
if (
|
||||
response.status === 401 &&
|
||||
/bearer|oauth|authorization/i.test(authenticate)
|
||||
(/bearer|oauth|authorization/i.test(authenticate) ||
|
||||
bareCredentialChallenge)
|
||||
) {
|
||||
const endpoints = await discoverOAuthEndpoints(
|
||||
connection,
|
||||
|
||||
Reference in new issue
Block a user