fix(evals): repair private human steps for webhook, device and key entry; name failed private steps (#21012)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat evaluation catalog checks agent replies and then the human
step that follows, such as a person who reveals a key or approves a
device
> - Some human steps run in a separate private browser process, because
they handle credentials. CI keeps the output of that process private on
purpose
> - Four cases failed only in that private step, so CI showed one fixed
message and no cause
> - I ran each private step locally against a dev server, where its
output is visible, and found the cause of three failures
> - This pull request fixes the harness and two product defects, and
adds a closed failure record so that the next CI failure names its step
> - The benefit is that these human steps work, and a private-step
failure is diagnosable without any credential in logs

## Linked Issues or Issue Description

Bug report (following `.github/ISSUE_TEMPLATE/bug_report.yml`).

**What happened?**
- `routines.webhook-trigger`: the private person opens the routine
overview. The trigger cards are only in the Triggers section, so the
step waits for a missing button. An unhandled response wait then stops
the process.
- `connection-setup.secure-api-key`: a key-protected MCP server answers
401 without a challenge header. Paperclip shows "Remote app returned
HTTP 401" and does not ask for a key. The person cannot enter the key in
the normal flow.
- `assistant-connections.device-login`: an expired device code page
shows "This item doesn't exist or was moved." The server sends "This
code is expired, already decided, or invalid." in the OAuth
`error_description` field, and the UI drops it.
- `onboarding.new-company`: the private step reads back all saved state
and then fails one proof check. CI does not say which check.

**Expected behavior**
- The person reveals the webhook key in the Triggers section and sends
authenticated events.
- A pasted MCP link that answers 401 shows the key field.
- An expired device code says that the code is expired.
- A private-step failure names its fixed step and error class, with no
value.

**Steps to reproduce**
1. Start a local dev server.
2. Create a routine with a bearer webhook, and run the private webhook
worker. It times out on the "Edit webhook" button.
3. Paste the address of an MCP server that answers a plain 401 into
Connectors > Add by link. No key field appears.
4. Open `/mcp-device?user_code=<expired code>`. The page shows generic
not-found text.

**Paperclip version or commit**
`master` (reproduced on origin/master before this change)

**Deployment mode**
Local dev server (embedded PGlite) and the CI chat-eval runner

**Installation method**
Source checkout (`pnpm dev`)

## What Changed

- `integrator-webhook-private-worker.ts`: open `<routine>/triggers`
(also for the setup-resume link), and attach a handler to the early
response wait so a failed click reports its real step.
- `server/.../catalog/discovery.ts`: a 401 with no `WWW-Authenticate`
header, to a connection that sends no credential, is handled like a
challenge without discoverable sign-in (`oauth_challenge`). The existing
UI guidance then asks for a key. A 401 to a connection that already
sends a credential keeps the old error.
- `ui/src/api/errors.ts`: read the OAuth `error_description` field as a
readable server message.
- New `private-child-failure.ts`: a private child records only a fixed
step name and an error class from a fixed list. The parent echoes a step
only when it is in the caller's closed list. Error text, stdout and
stderr stay private.
- The webhook, private browser and identity children use it. Each has a
closed step list file.
- `identity-new-company-native-proof.ts`: each proof check has a fixed
reason name. The private child reports `new-company-proof-<reason>` when
a check fails. The accepted conditions do not change.
- Tests: server red/green test for the bare 401, UI red/green test for
`error_description`, unit tests for the failure record (no text leaks,
closed list only, extra fields dropped).

## Verification

- Local dev server (embedded PostgreSQL), private workers run directly
with the real child process:
- Webhook, ready and setup-pending triggers: before, timeout on "Edit
webhook". After, receipt with invalid status 401, valid status 202 and a
linked task.
- Secure key entry with the synthetic notes provider: before, timeout on
the key field. After, an active connection with health `ok` and 2
authorized `tools/list` calls.
- Expired device page: before, `at step device-expired-page
(TimeoutError)`. After, `expiredPageVisible: true`.
- `server`: `vitest run src/__tests__/tool-access-service.test.ts -t
"pasted link answers"` (fails before the discovery change with 502,
passes after).
- `ui`: `vitest run src/api/errors.test.ts` (fails before, passes
after).
- `pnpm test:e2e:runner:typecheck`, `pnpm test:e2e:runner:unit`, server
`tsc --noEmit`.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD sha>
--check`: unchanged.
- One paid run of the four cases on this branch. Results are in a PR
comment.

## Risks

- A plain 401 from an MCP link now returns 422 `oauth_challenge` instead
of 502. The draft connection is removed, as for other challenges. Agents
that call the connect API see the "needs sign-in" answer instead of a
raw status. This only applies when the connection sends no credential.
- Some API errors that carry `error_description` now show that text
instead of generic copy. That text is written for people.
- The failure record is opt-in per child and contains only constants.
The parent ignores a record with any other field.
- `onboarding.new-company` is not fixed here. Its native hello probe
cannot start locally on macOS, so the failing check is not yet known.
This change makes the next run name it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edits, local browser automation through Playwright). Extended
reasoning was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-11 22:06:29 -06:00
1 parent 1bb4dcbe4b
commit b033cc124d
18 files changed
+346 -70

No files matched your search

+9
View File
@@ -122,6 +122,15 @@ describe("describeError", () => {
});
});
it("keeps an OAuth error description instead of generic not-found copy", () => {
const error = apiError(404, {
error: "invalid_request",
error_description: "This code is expired, already decided, or invalid. Start a new connection from your assistant.",
});
expect(describeError(error).body).toBe("This code is expired, already decided, or invalid. Start a new connection from your assistant.");
expect(error.message).toBe("This code is expired, already decided, or invalid. Start a new connection from your assistant.");
});
it("falls back to the kind's copy when the server sent a code or nothing", () => {
expect(describeError(apiError(404, { error: "issue_not_found" })).body).toBe("This item doesn’t exist or was moved.");
expect(describeError(apiError(403, null))).toMatchObject({ title: "Not allowed", retryable: false });
+4 -1
View File
@@ -185,7 +185,10 @@ export function isRawErrorCode(value: string): boolean {
/** A server message that is safe to show: not a raw code, not a generic status line. */
function readableServerMessage(err: unknown): string | null {
const error = record(err);
const candidates = [stringField(record(error?.body)?.message), stringField(record(error?.body)?.error)];
const body = record(error?.body);
// OAuth endpoints (RFC 6749) put the readable sentence in `error_description`
// and a machine code in `error`.
const candidates = [stringField(body?.message), stringField(body?.error), stringField(body?.error_description)];
if (err instanceof Error) candidates.push(stringField(err.message));
for (const candidate of candidates) {
if (!candidate) continue;