From a726dce4f06f9c0ada9e42650d356bcf83d8cc6d Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 20:14:22 -0600 Subject: [PATCH 01/10] feat(mcp): show live progress for long connected-app tool calls in the task chat (#21449) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents call tools from connected apps through the Paperclip MCP gateway > - Some tool calls take a long time: exports, builds, large searches > - The MCP progress utility lets an app report how far a call is, but Paperclip ignored it > - So the task chat showed only a spinner until the call finished, with no sign of life > - This pull request reads the app's progress and shows it on the tool's row while the call runs > - The benefit is that people can see that a long call is working, how far it is, and how long it has run ## Linked Issues or Issue Description **Subsystem affected** Connected apps (MCP gateway) and the task chat. **Problem or motivation** A long tool call from a connected MCP app shows only a spinner in the task chat. Paperclip does not send a `progressToken`, and it drops `notifications/progress` from the app. **Proposed solution** Send a `progressToken` on each `tools/call` that an agent run makes. Read the app's progress notifications from the call's response stream. Show a progress bar (or an indeterminate sweep), the latest message, and elapsed time on the running tool row. Keep the latest state so that a reload mid-call shows it. **Alternatives considered** A new transport for progress. The existing company live-events socket already carries per-run events with run read checks, so this change uses it. **Additional context** Spec: https://modelcontextprotocol.io/specification/2026-07-28/basic/utilities/progress ## What Changed - `mcp-http.ts`: `mcpHttpRequestInit` can add `_meta.progressToken`. `readMcpHttpResponse` gives each notification on the response stream to an observer before the result. The observer cannot fail or delay the call. New `mcpProgressNotificationOf` matches a notification to its token. - `tool-gateway.ts`: remote HTTP and local stdio `tools/call` from an agent run send a per-call token and forward progress to a reporter. Calls without a run send no token. - New `server/src/services/tool-call-progress.ts`: checks each update (finite, increasing, plain single-line text of at most 200 characters). It publishes a `heartbeat.run.tool_progress` live event at most every 250 ms (latest wins) and saves the latest update on the invocation at most every 1 s. Nothing is awaited by the call. A final `done` event goes out when the call ends. - DB: nullable `tool_invocations.progress` jsonb column (migration `0479`, `ADD COLUMN` only). A save never writes onto a finished call. - Live delivery: the new event follows its run like transcript output. A client gets it only for runs it follows, after the run read check, in the run's company only. - New route `GET /api/issues/:issueId/tool-call-progress` (task read check) returns the latest progress of the task's running calls, for a reload mid-call. Documented in `docs/api/issues.md`. - UI: `ToolCallProgressProvider` (one per chat) keeps the latest update per call. A running tool row subscribes and shows a bar or sweep, the message, the percent and elapsed time (design tokens, reduced motion honored). Finished rows drop it and never fetch or re-render for progress. `LiveUpdatesProvider` ignores the event for query invalidation. - `packages/kv-demo-mcp-server`: new `slow_with_progress` demo tool (`steps`, `stepMs`, `withTotal`) in its own file. - Storybook: `Tasks/Tool progress` stories (running with total, running without total, done). - Docs: new `doc/connections/MCP-PROGRESS.md`; `MCP-PROTOCOL-VERSIONS.md` no longer lists progress as unsupported. ## Verification - Red/green: on the original source, 5 of the new server tests fail (token in `_meta`, SSE with interleaved progress and result, notification matching, run-interest delivery, gateway end-to-end). With this change they pass. - `server/src/__tests__/mcp-http.test.ts` (13), `tool-call-progress.test.ts` (9: throttling, latest wins, bounded text, out-of-order drop, finish, failures never throw, company/run/tool scoping), `tool-call-progress-store.test.ts` (2, embedded Postgres: company and task scoping, no write onto a finished call), `realtime/live-event-delivery.test.ts` (run interest), `tool-gateway.test.ts` end-to-end against the real SDK kv demo server over Streamable HTTP: 6 provider updates over about 1.2 s give 3 to 7 throttled events, all for the right company, run, task and tool, the last one `done`; no event reaches another company; the invocation keeps the progress; the result is unchanged. - UI: `tool-call-progress.test.tsx` (10: name matching, row states, plain-text rendering, drop on complete, restore after reload, no fetch for finished rows). All 67 task-chat test files pass. - `packages/kv-demo-mcp-server`: 13 tests, including progress over HTTP with and without a total. - Storybook: checked the three stories in a browser. The bar fills to 40%, the sweep animates, the elapsed time ticks, and the done row shows no progress. - `pnpm verify`: see the CI checks for the final revision. ## Risks - Low. The migration adds one nullable column. - An SSE response is now bounded per message, not per stream, so handled notifications (such as progress) do not use up the result's size limit. Each message is still bounded. - Progress code runs only while a run's tool call is in flight. It adds a few bytes to the request and throttled, fire-and-forget work. Agent boot and time to first token do not change. - A row finds its call by run and tool name. If one run calls the same tool twice at the same time, both rows show the newer call's progress. - Progress from a call that became an MCP Task is not shown. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/connections/MCP-PROGRESS.md | 66 +++ doc/connections/MCP-PROTOCOL-VERSIONS.md | 4 +- docs/api/issues.md | 27 ++ .../db/src/migrations/0484_certain_zemo.sql | 1 + ...{0479_snapshot.json => 0484_snapshot.json} | 434 +++++++++++++++++- packages/db/src/migrations/meta/_journal.json | 7 + packages/db/src/schema/tool_access.ts | 3 + packages/kv-demo-mcp-server/README.md | 6 +- packages/kv-demo-mcp-server/src/http.test.ts | 22 + packages/kv-demo-mcp-server/src/index.ts | 3 + .../kv-demo-mcp-server/src/slow-progress.ts | 47 ++ packages/kv-demo-mcp-server/src/tools.test.ts | 4 +- packages/shared/src/constants.ts | 1 + packages/shared/src/index.ts | 1 + packages/shared/src/tool-call-progress.ts | 68 +++ server/src/__tests__/mcp-http.test.ts | 90 ++++ .../tool-call-progress-store.test.ts | 74 +++ .../src/__tests__/tool-call-progress.test.ts | 174 +++++++ .../__tests__/tool-gateway-service.test.ts | 2 +- server/src/__tests__/tool-gateway.test.ts | 59 +++ .../src/realtime/live-event-delivery.test.ts | 12 + server/src/realtime/live-event-delivery.ts | 5 +- server/src/realtime/live-events-ws.ts | 2 +- server/src/routes/mcp-apps.ts | 13 +- server/src/routes/openapi.ts | 1 + server/src/services/mcp-http.ts | 33 +- server/src/services/tool-call-progress.ts | 195 ++++++++ server/src/services/tool-gateway.ts | 41 +- .../runner-e2e/source-sdk-public-inputs.json | 5 +- ui/src/api/mcpApps.ts | 5 +- ui/src/components/TaskChatThread.tsx | 3 + .../task-chat/TaskChatRunnerTurn.tsx | 5 +- .../components/task-chat/TaskChatToolCard.tsx | 12 +- .../components/task-chat/task-chat-model.ts | 2 + .../task-chat/tool-call-progress.test.tsx | 224 +++++++++ .../task-chat/tool-call-progress.tsx | 221 +++++++++ .../task-chat/transcript-adapter.ts | 13 + ui/src/context/LiveUpdatesProvider.tsx | 3 + ui/src/index.css | 13 + ui/src/lib/queryKeys.ts | 1 + .../tasks/task-chat-tool-progress.stories.tsx | 66 +++ 41 files changed, 1946 insertions(+), 22 deletions(-) create mode 100644 doc/connections/MCP-PROGRESS.md create mode 100644 packages/db/src/migrations/0484_certain_zemo.sql rename packages/db/src/migrations/meta/{0479_snapshot.json => 0484_snapshot.json} (99%) create mode 100644 packages/kv-demo-mcp-server/src/slow-progress.ts create mode 100644 packages/shared/src/tool-call-progress.ts create mode 100644 server/src/__tests__/tool-call-progress-store.test.ts create mode 100644 server/src/__tests__/tool-call-progress.test.ts create mode 100644 server/src/services/tool-call-progress.ts create mode 100644 ui/src/components/task-chat/tool-call-progress.test.tsx create mode 100644 ui/src/components/task-chat/tool-call-progress.tsx create mode 100644 ui/storybook/stories/tasks/task-chat-tool-progress.stories.tsx diff --git a/doc/connections/MCP-PROGRESS.md b/doc/connections/MCP-PROGRESS.md new file mode 100644 index 00000000000..83f51b13b4f --- /dev/null +++ b/doc/connections/MCP-PROGRESS.md @@ -0,0 +1,66 @@ +# Progress for long MCP tool calls + +When an agent run calls a connected app's tool and the app reports progress, +the task chat shows it on the tool's row while the call runs: a bar that fills +when the app knows the total (a quiet sweep when it does not), the app's latest +status line, and elapsed time. The row drops it when the call finishes. + +Spec: [MCP 2026-07-28, Progress](https://modelcontextprotocol.io/specification/2026-07-28/basic/utilities/progress). + +## How it works + +1. **Request.** For a `tools/call` made by an agent run, the gateway adds a + per-call `_meta.progressToken` (`paperclip-progress-`). This applies to + remote (Streamable HTTP) and local stdio connections, under every protocol + version. Calls not made by a run (board test calls, MCP Apps views) send no + token. +2. **Response stream.** The app sends `notifications/progress` (`progress`, + optional `total`, optional `message`) on the call's own response stream: SSE + events on the POST response for HTTP, stdout lines for stdio. Under + 2026-07-28, request-scoped notifications always stay on that stream. + `readMcpHttpResponse` hands each notification to an observer before it + reaches the result, and the observer can never fail or delay the call. +3. **Reporter** (`server/src/services/tool-call-progress.ts`). Each update is + checked (finite, non-negative, increasing; `total` kept only when positive and + at least `progress`) and its message reduced to plain single-line text of at + most 200 characters. The reporter then: + - publishes a `heartbeat.run.tool_progress` live event at most every 250 ms + (the latest update wins); + - saves the latest update on the `tool_invocations.progress` column at most + every second, and never onto a finished call; + - publishes a final event with `done: true` when the call ends. + Nothing the reporter does is awaited by the call. +4. **Delivery.** The event goes to the run's company only. The live socket + treats it like transcript output: a client receives it only for runs it + asked to follow, after the same run read check. +5. **Task chat.** `ToolCallProgressProvider` (one per chat) keeps the latest + update per call. A running tool row subscribes to it and finds its call by + run and tool name. Each update carries both the gateway name and the name + native runners give the tool (`assignedToolName`: shortened, with a hash), + since native transcripts show the latter. When a running row first mounts, the chat reads + `GET /api/issues/:issueId/tool-call-progress` (again on mount and on tab + focus), so a reload mid-call shows the latest saved state. The read also + marks recently finished calls `done`; a call is never dropped just because + the read does not list it, since its first save may still be in flight. Finished rows never fetch or re-render for + progress. + +## Speed + +The token adds a few bytes to the request. Parsing happens on the stream that +was already being read. Publishing and saving are throttled, fire-and-forget, +and stop when the call ends. Agent boot and time to first token are untouched: +the code runs only while a tool call is in flight. + +## Trying it + +`packages/kv-demo-mcp-server` has a `slow_with_progress` tool +(`steps`, `stepMs`, `withTotal`). Connect the demo server, then ask an agent to +call it with, say, `{ "steps": 20, "stepMs": 1000 }`; the tool row fills over +20 seconds. `withTotal: false` shows the indeterminate state. + +## Not covered + +- Matching uses the run and the tool name. If one run calls the same tool + twice at once, both rows show the newer call's progress. +- Progress from a call that became an MCP Task (see `MCP-TASKS.md`) is not shown. +- Progress is not forwarded to the agent's own MCP client. diff --git a/doc/connections/MCP-PROTOCOL-VERSIONS.md b/doc/connections/MCP-PROTOCOL-VERSIONS.md index 806707aa70e..044a4e3b358 100644 --- a/doc/connections/MCP-PROTOCOL-VERSIONS.md +++ b/doc/connections/MCP-PROTOCOL-VERSIONS.md @@ -169,5 +169,5 @@ Tracked as follow-ups: agent to retry. - `subscriptions/listen` change notifications (cache invalidation relies on `ttlMs`). -- The Tasks extension, Skills over MCP, progress and cancellation, icons, and - enterprise-managed authorization. +- Cancellation, icons, and enterprise-managed authorization. (Progress is + supported: see [MCP-PROGRESS.md](MCP-PROGRESS.md).) diff --git a/docs/api/issues.md b/docs/api/issues.md index d7425507ec1..9be228b683f 100644 --- a/docs/api/issues.md +++ b/docs/api/issues.md @@ -374,6 +374,33 @@ GET /api/attachments/{attachmentId}/content DELETE /api/attachments/{attachmentId} ``` +## Tool Call Progress + +``` +GET /api/issues/{issueId}/tool-call-progress +``` + +Returns the latest progress of the task's connected-app tool calls that are still running, for a page that loads mid-call, and the calls that finished in the last hour, marked `done`. A client should drop a call's progress only on `done`: a running call may not have saved its first update yet. Live updates arrive as `heartbeat.run.tool_progress` events on the company live-events socket. + +- Board callers only. The caller needs company access and read access to the task; otherwise the response is `404`. +- Lists unfinished calls made by a run in the last 24 hours that reported progress (at most 50), then calls finished in the last hour (at most 50). + +Each item: + +| Field | Type | Meaning | +| --- | --- | --- | +| `invocationId` | string | The tool call. | +| `runId` | string | The run that made the call. | +| `toolName` | string | The tool name the agent called. | +| `startedAt` | string (ISO) | When the call started. | +| `updatedAt` | string (ISO) | When the app last reported progress. | +| `progress` | number | Progress so far. | +| `total` | number or null | Known end value; `null` when the app does not know it. | +| `message` | string or null | The app's latest status line: plain text, at most 200 characters. | +| `done` | boolean | `true` once the call has finished. | + +See `doc/connections/MCP-PROGRESS.md`. + ## Issue Lifecycle ``` diff --git a/packages/db/src/migrations/0484_certain_zemo.sql b/packages/db/src/migrations/0484_certain_zemo.sql new file mode 100644 index 00000000000..8197a5714d9 --- /dev/null +++ b/packages/db/src/migrations/0484_certain_zemo.sql @@ -0,0 +1 @@ +ALTER TABLE "tool_invocations" ADD COLUMN "progress" jsonb; \ No newline at end of file diff --git a/packages/db/src/migrations/meta/0479_snapshot.json b/packages/db/src/migrations/meta/0484_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0479_snapshot.json rename to packages/db/src/migrations/meta/0484_snapshot.json index 36323203dc1..1debf9ce54b 100644 --- a/packages/db/src/migrations/meta/0479_snapshot.json +++ b/packages/db/src/migrations/meta/0484_snapshot.json @@ -1,6 +1,6 @@ { - "id": "f5f852bd-0568-4b2d-bc35-0dac5d9c8c5b", - "prevId": "349cdd89-db04-4340-a5d3-3b877bf7227d", + "id": "9389124c-379a-4b64-9c20-c7b37538e7c8", + "prevId": "ceaa38c4-681b-430e-9578-0854650d85b3", "version": "7", "dialect": "postgresql", "tables": { @@ -18092,6 +18092,12 @@ "notNull": true, "default": "'company'" }, + "system_purpose": { + "name": "system_purpose", + "type": "text", + "primaryKey": false, + "notNull": false + }, "owner_user_id": { "name": "owner_user_id", "type": "text", @@ -55953,6 +55959,153 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.tailscale_connections": { + "name": "tailscale_connections", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "secret_id": { + "name": "secret_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "credential_version": { + "name": "credential_version", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "credential_hash": { + "name": "credential_hash", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_by_user_id": { + "name": "created_by_user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "oauth_state_hash": { + "name": "oauth_state_hash", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "oauth_expires_at": { + "name": "oauth_expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "tailscale_connections_company_idx": { + "name": "tailscale_connections_company_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "tailscale_connections_oauth_state_idx": { + "name": "tailscale_connections_oauth_state_idx", + "columns": [ + { + "expression": "oauth_state_hash", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "tailscale_connections_company_id_companies_id_fk": { + "name": "tailscale_connections_company_id_companies_id_fk", + "tableFrom": "tailscale_connections", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "tailscale_connections_secret_id_company_secrets_id_fk": { + "name": "tailscale_connections_secret_id_company_secrets_id_fk", + "tableFrom": "tailscale_connections", + "tableTo": "company_secrets", + "columnsFrom": [ + "secret_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.tool_access_audit_events": { "name": "tool_access_audit_events", "schema": "", @@ -59454,6 +59607,12 @@ "primaryKey": false, "notNull": false }, + "progress": { + "name": "progress", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, "created_at": { "name": "created_at", "type": "timestamp with time zone", @@ -59960,6 +60119,277 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.tool_mcp_elicitations": { + "name": "tool_mcp_elicitations", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "issue_id": { + "name": "issue_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "agent_id": { + "name": "agent_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "run_id": { + "name": "run_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "invocation_id": { + "name": "invocation_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "connection_id": { + "name": "connection_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "grant_id": { + "name": "grant_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "catalog_tool_name": { + "name": "catalog_tool_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "call_identity": { + "name": "call_identity", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "era": { + "name": "era", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "sealed_request_state": { + "name": "sealed_request_state", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "questions": { + "name": "questions", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "arguments_identity": { + "name": "arguments_identity", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "dedupe_key": { + "name": "dedupe_key", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "claimed_until": { + "name": "claimed_until", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "sent_at": { + "name": "sent_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "tool_mcp_elicitations_call_idx": { + "name": "tool_mcp_elicitations_call_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "issue_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "connection_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "call_identity", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "tool_mcp_elicitations_invocation_idx": { + "name": "tool_mcp_elicitations_invocation_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "invocation_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "tool_mcp_elicitations_waiting_uq": { + "name": "tool_mcp_elicitations_waiting_uq", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "issue_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "dedupe_key", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"tool_mcp_elicitations\".\"status\" = 'waiting'", + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "tool_mcp_elicitations_company_id_companies_id_fk": { + "name": "tool_mcp_elicitations_company_id_companies_id_fk", + "tableFrom": "tool_mcp_elicitations", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "tool_mcp_elicitations_invocation_id_tool_invocations_id_fk": { + "name": "tool_mcp_elicitations_invocation_id_tool_invocations_id_fk", + "tableFrom": "tool_mcp_elicitations", + "tableTo": "tool_invocations", + "columnsFrom": [ + "invocation_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "tool_mcp_elicitations_connection_id_tool_connections_id_fk": { + "name": "tool_mcp_elicitations_connection_id_tool_connections_id_fk", + "tableFrom": "tool_mcp_elicitations", + "tableTo": "tool_connections", + "columnsFrom": [ + "connection_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.tool_mcp_gateway_tokens": { "name": "tool_mcp_gateway_tokens", "schema": "", diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index 54d4380b27f..d9e6e48b3f5 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -3375,6 +3375,13 @@ "when": 1794611818460, "tag": "0483_busy_tombstone", "breakpoints": true + }, + { + "idx": 484, + "version": "7", + "when": 1794621009918, + "tag": "0484_certain_zemo", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/tool_access.ts b/packages/db/src/schema/tool_access.ts index 621597fd6d3..2c57cba3b1e 100644 --- a/packages/db/src/schema/tool_access.ts +++ b/packages/db/src/schema/tool_access.ts @@ -16,6 +16,7 @@ import { uuid, } from "drizzle-orm/pg-core"; import type { + ToolCallProgress, ConnectionAgentInstructions, ConnectionTokenIssuanceOutcome, ConnectionTokenIssuancePath, @@ -804,6 +805,8 @@ export const toolInvocations = pgTable( errorMessage: text("error_message"), startedAt: timestamp("started_at", { withTimezone: true }), completedAt: timestamp("completed_at", { withTimezone: true }), + /** The provider's latest progress while the call runs (MCP `notifications/progress`), bounded. */ + progress: jsonb("progress").$type(), createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(), updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(), }, diff --git a/packages/kv-demo-mcp-server/README.md b/packages/kv-demo-mcp-server/README.md index ea1daaa90e1..88c18d64e11 100644 --- a/packages/kv-demo-mcp-server/README.md +++ b/packages/kv-demo-mcp-server/README.md @@ -1,7 +1,7 @@ # KV Demo MCP Server A standalone, self-contained MCP server for demos. One process exposes four -key/value MCP tools **and** a tiny web UI that renders the values those tools +key/value MCP tools (plus a progress demo tool) **and** a tiny web UI that renders the values those tools mutate — so you can call a tool from a Paperclip agent and watch the value appear in a browser tab. @@ -24,6 +24,10 @@ recorded walkthrough in [doc/MCP-DEMO-SCRIPT.md](../../doc/MCP-DEMO-SCRIPT.md). - `kv_set` (write) — set one key to a string value. - `kv_delete` (destructive) — delete a key. Carries `destructiveHint: true` so Paperclip's catalog quarantines it on first sight. +- `slow_with_progress` (read) — waits `steps` × `stepMs` and sends MCP + `notifications/progress` after each step (`withTotal: false` leaves the total + out). For the task chat's tool progress; see + [doc/connections/MCP-PROGRESS.md](../../doc/connections/MCP-PROGRESS.md). - A Values UI at `/` — an auto-refreshing HTML table over the same store. - A JSON state route at `/api/state` — what the UI polls. diff --git a/packages/kv-demo-mcp-server/src/http.test.ts b/packages/kv-demo-mcp-server/src/http.test.ts index 1134ee6acae..1fcb68bf5a3 100644 --- a/packages/kv-demo-mcp-server/src/http.test.ts +++ b/packages/kv-demo-mcp-server/src/http.test.ts @@ -83,4 +83,26 @@ describe("kv demo HTTP server", () => { expect(state.count).toBe(1); await client.close(); }); + + it("streams progress for slow_with_progress on the call's response, before the result", async () => { + const { base } = await startServer(); + const client = await mcpClient(base); + const updates: Array<{ progress: number; total?: number; message?: string }> = []; + const result = await client.callTool( + { name: "slow_with_progress", arguments: { steps: 3, stepMs: 5 } }, + undefined, + { onprogress: (update) => { updates.push(update); } }, + ); + expect(updates).toEqual([ + { progress: 1, total: 3, message: "Finished step 1 of 3" }, + { progress: 2, total: 3, message: "Finished step 2 of 3" }, + { progress: 3, total: 3, message: "Finished step 3 of 3" }, + ]); + expect(JSON.stringify(result.content)).toContain("completed"); + const indeterminate: Array<{ total?: number }> = []; + await client.callTool({ name: "slow_with_progress", arguments: { steps: 1, stepMs: 0, withTotal: false } }, undefined, + { onprogress: (update) => { indeterminate.push(update); } }); + expect(indeterminate[0]?.total).toBeUndefined(); + await client.close(); + }); }); diff --git a/packages/kv-demo-mcp-server/src/index.ts b/packages/kv-demo-mcp-server/src/index.ts index a67a5fb8ed9..51f451ff0c5 100644 --- a/packages/kv-demo-mcp-server/src/index.ts +++ b/packages/kv-demo-mcp-server/src/index.ts @@ -1,6 +1,7 @@ import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; import { KvStore } from "./store.js"; import { createToolDefinitions } from "./tools.js"; +import { registerSlowWithProgressTool } from "./slow-progress.js"; export interface CreateKvDemoMcpServerResult { server: McpServer; @@ -29,6 +30,7 @@ export function createKvDemoMcpServer(store: KvStore = new KvStore()): CreateKvD tool.execute, ); } + registerSlowWithProgressTool(server); return { server, store }; } @@ -36,6 +38,7 @@ export function createKvDemoMcpServer(store: KvStore = new KvStore()): CreateKvD export { KvStore } from "./store.js"; export type { KvEntry, KvStateSnapshot } from "./store.js"; export { createToolDefinitions } from "./tools.js"; +export { SLOW_WITH_PROGRESS_TOOL, registerSlowWithProgressTool } from "./slow-progress.js"; export { createKvDemoHttpServer } from "./http.js"; export type { KvDemoHttpServer, KvDemoHttpOptions } from "./http.js"; export { readConfigFromEnv } from "./config.js"; diff --git a/packages/kv-demo-mcp-server/src/slow-progress.ts b/packages/kv-demo-mcp-server/src/slow-progress.ts new file mode 100644 index 00000000000..aaf08c7c4a7 --- /dev/null +++ b/packages/kv-demo-mcp-server/src/slow-progress.ts @@ -0,0 +1,47 @@ +import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; +import { z } from "zod"; + +/** + * A demo tool that takes a while and reports MCP progress as it goes, so a + * client can show it. Each step waits `stepMs`, then sends + * `notifications/progress` for the caller's `progressToken` (when it sent + * one). `withTotal: false` leaves `total` out, for an indeterminate display. + */ +export const SLOW_WITH_PROGRESS_TOOL = "slow_with_progress"; + +const schema = z.object({ + steps: z.number().int().min(1).max(100).default(5), + stepMs: z.number().int().min(0).max(10_000).default(1_000), + withTotal: z.boolean().default(true), +}); + +export function registerSlowWithProgressTool(server: McpServer): void { + server.registerTool( + SLOW_WITH_PROGRESS_TOOL, + { + description: "Wait for a few steps, reporting progress after each one. For progress demos.", + inputSchema: schema.shape, + annotations: { title: "Slow task with progress", readOnlyHint: true, openWorldHint: false }, + }, + async (input, extra) => { + const { steps, stepMs, withTotal } = schema.parse(input); + const progressToken = extra._meta?.progressToken; + for (let step = 1; step <= steps; step++) { + await new Promise((resolve) => setTimeout(resolve, stepMs)); + if (extra.signal.aborted) break; + if (progressToken !== undefined) { + await extra.sendNotification({ + method: "notifications/progress", + params: { + progressToken, + progress: step, + ...(withTotal ? { total: steps } : {}), + message: `Finished step ${step} of ${steps}`, + }, + }); + } + } + return { content: [{ type: "text", text: JSON.stringify({ steps, completed: true }) }] }; + }, + ); +} diff --git a/packages/kv-demo-mcp-server/src/tools.test.ts b/packages/kv-demo-mcp-server/src/tools.test.ts index df62a84e39f..8bca3567f1c 100644 --- a/packages/kv-demo-mcp-server/src/tools.test.ts +++ b/packages/kv-demo-mcp-server/src/tools.test.ts @@ -19,10 +19,10 @@ async function connectedClient() { } describe("kv demo MCP tools", () => { - it("exposes exactly kv_set, kv_get, kv_list, kv_delete", async () => { + it("exposes exactly kv_set, kv_get, kv_list, kv_delete and the progress demo tool", async () => { const { client } = await connectedClient(); const { tools } = await client.listTools(); - expect(tools.map((t) => t.name).sort()).toEqual(["kv_delete", "kv_get", "kv_list", "kv_set"]); + expect(tools.map((t) => t.name).sort()).toEqual(["kv_delete", "kv_get", "kv_list", "kv_set", "slow_with_progress"]); }); it("reflects kv_set writes in the shared store", async () => { diff --git a/packages/shared/src/constants.ts b/packages/shared/src/constants.ts index 3149ee1cc5f..47973de810c 100644 --- a/packages/shared/src/constants.ts +++ b/packages/shared/src/constants.ts @@ -973,6 +973,7 @@ export const LIVE_EVENT_TYPES = [ "heartbeat.run.event", "heartbeat.run.log", "heartbeat.run.activity", + "heartbeat.run.tool_progress", "agent.session.goal.changed", "agent.status", "agent.runtime_state.updated", diff --git a/packages/shared/src/index.ts b/packages/shared/src/index.ts index 2d2b5ba044c..7128882e69b 100644 --- a/packages/shared/src/index.ts +++ b/packages/shared/src/index.ts @@ -2966,5 +2966,6 @@ export { } from "./issue-activity-identity.js"; export { recognizeLinkPreview, type LinkPreviewTarget, type LinkPreviewState, type LinkPreview, type LinkPreviewResult } from "./link-preview.js"; export * from "./mcp-apps.js"; +export * from "./tool-call-progress.js"; export * from "./mcp-skills.js"; export type { TailscaleConnection, TailscaleDeviceLogin } from "./types/tailscale.js"; diff --git a/packages/shared/src/tool-call-progress.ts b/packages/shared/src/tool-call-progress.ts new file mode 100644 index 00000000000..89655767bfd --- /dev/null +++ b/packages/shared/src/tool-call-progress.ts @@ -0,0 +1,68 @@ +/** + * Live progress of one connected-app tool call, from the provider's MCP + * `notifications/progress`. Everything here is untrusted provider content: + * numbers are checked and the message is plain text, bounded in length. + */ +export const TOOL_CALL_PROGRESS_MESSAGE_MAX_LENGTH = 200; + +export interface ToolCallProgress { + /** Increases with each update. Never shown as a raw number without `total`. */ + progress: number; + /** Known end value; absent when the provider does not know the total. */ + total: number | null; + /** The provider's latest status line, plain text. */ + message: string | null; + /** When the provider last reported progress (ISO). */ + updatedAt: string; +} + +/** A running tool call's progress, as the task chat reads it. */ +export interface ToolCallProgressSnapshot extends ToolCallProgress { + invocationId: string; + runId: string; + /** The tool name the agent called through Paperclip. */ + toolName: string; + /** The name native runners give that tool (shortened, with a hash); their transcripts show it. */ + nativeToolName?: string | null; + /** When the call started (ISO), for elapsed time. */ + startedAt: string; + /** True once the call has finished; the row then drops its progress. */ + done: boolean; +} + +function finite(value: unknown): number | null { + return typeof value === "number" && Number.isFinite(value) ? value : null; +} + +/** Plain, single-line text: control characters removed, whitespace collapsed, bounded. */ +export function boundToolCallProgressMessage(value: unknown): string | null { + if (typeof value !== "string") return null; + // eslint-disable-next-line no-control-regex + const plain = value.replace(/[\u0000-\u001f\u007f-\u009f​-‏‪-‮⁦-⁩]+/g, " ").replace(/\s+/g, " ").trim(); + if (!plain) return null; + return plain.length > TOOL_CALL_PROGRESS_MESSAGE_MAX_LENGTH + ? `${plain.slice(0, TOOL_CALL_PROGRESS_MESSAGE_MAX_LENGTH - 1)}…` + : plain; +} + +/** + * The progress a provider reported, or null when it is not usable. `total` is + * kept only when it is positive and at least the progress so far. + */ +export function normalizeToolCallProgress(input: { progress?: unknown; total?: unknown; message?: unknown }, now = new Date()): ToolCallProgress | null { + const progress = finite(input.progress); + if (progress === null || progress < 0) return null; + const total = finite(input.total); + return { + progress, + total: total !== null && total > 0 && total >= progress ? total : null, + message: boundToolCallProgressMessage(input.message), + updatedAt: now.toISOString(), + }; +} + +/** A fraction in [0, 1] when the total is known, otherwise null (indeterminate). */ +export function toolCallProgressFraction(progress: Pick): number | null { + if (progress.total === null || progress.total <= 0) return null; + return Math.min(1, Math.max(0, progress.progress / progress.total)); +} diff --git a/server/src/__tests__/mcp-http.test.ts b/server/src/__tests__/mcp-http.test.ts index 5271f2df485..4ea44109087 100644 --- a/server/src/__tests__/mcp-http.test.ts +++ b/server/src/__tests__/mcp-http.test.ts @@ -119,3 +119,93 @@ describe("parseMcpHttpResponseBody", () => { expect(() => parseMcpHttpResponseBody("event: ping\n\n", "text/event-stream")).toThrow(); }); }); + +describe("MCP progress notifications", () => { + it("sends the progress token in the request's _meta and keeps other metadata", async () => { + const { mcpHttpRequestInit } = await import("../services/mcp-http.js"); + const init = mcpHttpRequestInit({ "MCP-Protocol-Version": "2026-07-28" }, { + id: "call-1", method: "tools/call", params: { name: "export", arguments: {}, _meta: { other: true } }, + }, { progressToken: "token-1" }); + const meta = JSON.parse(init.body).params._meta; + expect(meta.progressToken).toBe("token-1"); + expect(meta.other).toBe(true); + expect(meta["io.modelcontextprotocol/protocolVersion"]).toBe("2026-07-28"); + const without = mcpHttpRequestInit(undefined, { id: "call-2", method: "tools/call", params: { name: "export" } }); + expect(JSON.parse(without.body).params._meta?.progressToken).toBeUndefined(); + }); + + it("reads progress interleaved with the result on one SSE stream, before the result", async () => { + const { mcpProgressNotificationOf } = await import("../services/mcp-http.js"); + const frames = [ + { jsonrpc: "2.0", method: "notifications/progress", params: { progressToken: "token-1", progress: 1, total: 3, message: "Exporting rows" } }, + { jsonrpc: "2.0", method: "notifications/message", params: { level: "info", data: "unrelated" } }, + { jsonrpc: "2.0", method: "notifications/progress", params: { progressToken: "someone-else", progress: 9 } }, + { jsonrpc: "2.0", method: "notifications/progress", params: { progressToken: "token-1", progress: 2, total: 3 } }, + { jsonrpc: "2.0", id: "call-1", result: { content: [{ type: "text", text: "done" }] } }, + { jsonrpc: "2.0", method: "notifications/progress", params: { progressToken: "token-1", progress: 3, total: 3 } }, + ]; + // Split mid-event so frames cross chunk boundaries. + const text = frames.map((frame) => `event: message\r\ndata: ${JSON.stringify(frame)}\r\n\r\n`).join(""); + const encoded = new TextEncoder().encode(text); + const body = new ReadableStream({ + start(controller) { + for (let i = 0; i < encoded.length; i += 37) controller.enqueue(encoded.slice(i, i + 37)); + controller.close(); + }, + }); + const seen: unknown[] = []; + let resultSeenBeforeProgress = false; + const result = await readMcpHttpResponse(new Response(body, { headers: { "content-type": "text/event-stream" } }), "call-1", { + onNotification: (message) => { + const update = mcpProgressNotificationOf(message, "token-1"); + if (update) seen.push(update); + }, + onProgress: (progress) => { if (progress.matchedResponse && seen.length === 0) resultSeenBeforeProgress = true; }, + }); + expect(result).toMatchObject({ id: "call-1", result: { content: [{ text: "done" }] } }); + expect(resultSeenBeforeProgress).toBe(false); + expect(seen).toEqual([ + { progress: 1, total: 3, message: "Exporting rows" }, + { progress: 2, total: 3, message: undefined }, + ]); + }); + + it("never lets a failing progress observer fail the call", async () => { + const text = [ + { jsonrpc: "2.0", method: "notifications/progress", params: { progressToken: "t", progress: 1 } }, + { jsonrpc: "2.0", id: 7, result: { ok: true } }, + ].map((frame) => `data: ${JSON.stringify(frame)}\n\n`).join(""); + const result = await readMcpHttpResponse(new Response(text, { headers: { "content-type": "text/event-stream" } }), 7, { + onNotification: () => { throw new Error("observer broke"); }, + }); + expect(result).toMatchObject({ result: { ok: true } }); + }); + + it("ignores requests, responses and other tokens", async () => { + const { mcpProgressNotificationOf } = await import("../services/mcp-http.js"); + expect(mcpProgressNotificationOf({ method: "notifications/progress", params: { progressToken: "t", progress: 1 } }, undefined)).toBeNull(); + expect(mcpProgressNotificationOf({ id: 1, method: "notifications/progress", params: { progressToken: "t", progress: 1 } }, "t")).toBeNull(); + expect(mcpProgressNotificationOf({ method: "notifications/progress", params: { progressToken: "x", progress: 1 } }, "t")).toBeNull(); + expect(mcpProgressNotificationOf({ method: "notifications/progress", params: { progressToken: "t", progress: 1 } }, "t")).toEqual({ progress: 1, total: undefined, message: undefined }); + }); + + it("does not count handled progress against the result's size limit", async () => { + const note = `data: ${JSON.stringify({ jsonrpc: "2.0", method: "notifications/progress", params: { progressToken: "t", progress: 1, message: "x".repeat(200) } })}\n\n`; + const text = note.repeat(50) + `data: ${JSON.stringify({ jsonrpc: "2.0", id: 1, result: { ok: true } })}\n\n`; + expect(Buffer.byteLength(text)).toBeGreaterThan(5_000); + const result = await readMcpHttpResponse(new Response(text, { headers: { "content-type": "text/event-stream" } }), 1, { maxBytes: 1_000, limitPerMessage: true }); + expect(result).toMatchObject({ result: { ok: true } }); + // Callers without an overall deadline (catalog discovery) keep the total limit, + // so a stream of small notifications cannot keep them waiting forever. + await expect(readMcpHttpResponse(new Response(text, { headers: { "content-type": "text/event-stream" } }), 1, { maxBytes: 1_000 })) + .rejects.toMatchObject({ reason: "too_large" }); + }); + + it("still rejects one SSE message over the limit, complete or not", async () => { + const big = `data: ${JSON.stringify({ jsonrpc: "2.0", id: 1, result: { text: "x".repeat(2_000) } })}`; + for (const text of [`${big}\n\n`, big]) { + await expect(readMcpHttpResponse(new Response(text, { headers: { "content-type": "text/event-stream" } }), 1, { maxBytes: 1_000, limitPerMessage: true })) + .rejects.toMatchObject({ reason: "too_large" }); + } + }); +}); diff --git a/server/src/__tests__/tool-call-progress-store.test.ts b/server/src/__tests__/tool-call-progress-store.test.ts new file mode 100644 index 00000000000..8df91536dbf --- /dev/null +++ b/server/src/__tests__/tool-call-progress-store.test.ts @@ -0,0 +1,74 @@ +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { agents, companies, createDb, heartbeatRuns, issues, toolInvocations, type Db } from "@paperclipai/db"; +import { listToolCallProgress, saveToolCallProgress } from "../services/tool-call-progress.js"; +import { assignedToolName } from "../services/native-runtime/assigned-mcp-tools.js"; +import { startRecoveryPostgres } from "./helpers/recovery-postgres.js"; + +describe("saved tool call progress", () => { + let temporary: Awaited>; + let db: Db; + + beforeAll(async () => { + temporary = await startRecoveryPostgres("tool-call-progress-"); + db = createDb(temporary.connectionString); + }, 120_000); + afterAll(async () => { await temporary?.cleanup(); }); + + async function task(prefix: string) { + const companyId = randomUUID(); + const issueId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: prefix, issuePrefix: prefix }); + const [agent] = await db.insert(agents).values({ companyId, name: "Agent", adapterType: "process" }).returning(); + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent!.id, status: "running" }).returning(); + await db.insert(issues).values({ id: issueId, companyId, title: "Export" }); + const call = async (values: Partial = {}) => (await db.insert(toolInvocations).values({ + companyId, issueId, runId: run!.id, agentId: agent!.id, toolName: "kv_demo__slow_with_progress", + status: "executing", startedAt: new Date("2026-11-13T10:00:00Z"), ...values, + }).returning())[0]!; + return { companyId, issueId, runId: run!.id, call }; + } + const progress = (value: number) => ({ progress: value, total: 10, message: "Exporting", updatedAt: "2026-11-13T10:00:05.000Z" }); + + it("returns a task's running calls with their latest progress, and nothing from another company or task", async () => { + const mine = await task("PRGA"); + const theirs = await task("PRGB"); + const running = await mine.call(); + await saveToolCallProgress(db, mine.companyId, running.id)(progress(3)); + await saveToolCallProgress(db, mine.companyId, running.id)(progress(4)); + const other = await theirs.call(); + await saveToolCallProgress(db, theirs.companyId, other.id)(progress(9)); + // A write naming the wrong company never lands. + await saveToolCallProgress(db, mine.companyId, other.id)(progress(1)); + const now = new Date("2026-11-13T10:01:00Z"); + + expect(await listToolCallProgress(db, mine.companyId, mine.issueId, now)).toEqual([{ + invocationId: running.id, runId: mine.runId, toolName: "kv_demo__slow_with_progress", + nativeToolName: assignedToolName("kv_demo__slow_with_progress"), + startedAt: "2026-11-13T10:00:00.000Z", progress: 4, total: 10, message: "Exporting", + updatedAt: "2026-11-13T10:00:05.000Z", done: false, + }]); + expect(await listToolCallProgress(db, mine.companyId, theirs.issueId, now)).toEqual([]); + expect(await listToolCallProgress(db, theirs.companyId, mine.issueId, now)).toEqual([]); + expect((await db.select().from(toolInvocations).where(eq(toolInvocations.id, other.id)))[0]!.progress).toMatchObject({ progress: 9 }); + }); + + it("never writes progress onto a finished call, and reports it as done", async () => { + const mine = await task("PRGC"); + const running = await mine.call({ createdAt: new Date() }); + await saveToolCallProgress(db, mine.companyId, running.id)(progress(2)); + expect(await listToolCallProgress(db, mine.companyId, mine.issueId)).toHaveLength(1); + await db.update(toolInvocations).set({ status: "succeeded", completedAt: new Date() }).where(eq(toolInvocations.id, running.id)); + // A late, throttled save after the result must not overwrite it. + await saveToolCallProgress(db, mine.companyId, running.id)(progress(8)); + expect((await db.select().from(toolInvocations).where(eq(toolInvocations.id, running.id)))[0]!.progress).toMatchObject({ progress: 2 }); + // A chat that missed the end learns it; a call that never saved progress is reported too. + const notSaved = await mine.call({ createdAt: new Date() }); + await db.update(toolInvocations).set({ status: "succeeded", completedAt: new Date() }).where(eq(toolInvocations.id, notSaved.id)); + const listed = await listToolCallProgress(db, mine.companyId, mine.issueId); + expect(listed.map((entry) => [entry.invocationId, entry.done, entry.progress]).sort()).toEqual( + [[running.id, true, 2], [notSaved.id, true, 0]].sort(), + ); + }); +}); diff --git a/server/src/__tests__/tool-call-progress.test.ts b/server/src/__tests__/tool-call-progress.test.ts new file mode 100644 index 00000000000..edf8a108de1 --- /dev/null +++ b/server/src/__tests__/tool-call-progress.test.ts @@ -0,0 +1,174 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { TOOL_CALL_PROGRESS_MESSAGE_MAX_LENGTH } from "@paperclipai/shared"; +import { + createToolCallProgressReporter, + TOOL_CALL_PROGRESS_PUBLISH_INTERVAL_MS, + TOOL_CALL_PROGRESS_SAVE_INTERVAL_MS, +} from "../services/tool-call-progress.js"; +import { assignedToolName } from "../services/native-runtime/assigned-mcp-tools.js"; + +function reporter() { + const published: Array<{ companyId: string; type: string; payload: Record }> = []; + const saved: unknown[] = []; + const instance = createToolCallProgressReporter({ + companyId: "company-a", + runId: "run-a", + issueId: "issue-a", + agentId: "agent-a", + invocationId: "invocation-a", + toolName: "kv_demo__slow_with_progress", + startedAt: new Date("2026-11-13T10:00:00Z"), + publish: (event) => { published.push(event); }, + save: async (progress) => { saved.push(progress); }, + }); + return { instance, published, saved }; +} + +describe("tool call progress reporter", () => { + beforeEach(() => { vi.useFakeTimers(); vi.setSystemTime(new Date("2026-11-13T10:00:01Z")); }); + afterEach(() => { vi.useRealTimers(); }); + + it("publishes at most one update per interval and always the latest", () => { + const { instance, published } = reporter(); + for (let step = 1; step <= 50; step++) instance.report({ progress: step, total: 50, message: `Step ${step}` }); + // The first update goes out at once; the rest wait for the interval. + expect(published.map((event) => event.payload.progress)).toEqual([1]); + vi.advanceTimersByTime(TOOL_CALL_PROGRESS_PUBLISH_INTERVAL_MS); + expect(published.map((event) => event.payload.progress)).toEqual([1, 50]); + vi.advanceTimersByTime(TOOL_CALL_PROGRESS_PUBLISH_INTERVAL_MS * 4); + expect(published).toHaveLength(2); + // A steady stream for one second publishes about four times, never per update. + for (let tick = 0; tick < 100; tick++) { + instance.report({ progress: 51 + tick, total: 200 }); + vi.advanceTimersByTime(10); + } + expect(published.length - 2).toBeLessThanOrEqual(1_000 / TOOL_CALL_PROGRESS_PUBLISH_INTERVAL_MS + 1); + }); + + it("saves on the invocation less often than it publishes", async () => { + const { instance, saved } = reporter(); + for (let tick = 0; tick < 200; tick++) { + instance.report({ progress: tick, total: 200 }); + await vi.advanceTimersByTimeAsync(10); + } + expect(saved.length).toBeLessThanOrEqual(2_000 / TOOL_CALL_PROGRESS_SAVE_INTERVAL_MS + 1); + expect(saved.length).toBeGreaterThanOrEqual(2); + }); + + it("ties every event to its company, run and tool call", () => { + const { instance, published } = reporter(); + instance.report({ progress: 1, total: 4, message: "Reading" }); + expect(published[0]).toEqual({ + companyId: "company-a", + type: "heartbeat.run.tool_progress", + payload: { + invocationId: "invocation-a", + runId: "run-a", + issueId: "issue-a", + agentId: "agent-a", + toolName: "kv_demo__slow_with_progress", + nativeToolName: assignedToolName("kv_demo__slow_with_progress"), + startedAt: "2026-11-13T10:00:00.000Z", + updatedAt: "2026-11-13T10:00:01.000Z", + progress: 1, + total: 4, + message: "Reading", + done: false, + }, + }); + }); + + it("treats provider text as untrusted: plain, single-line and bounded", () => { + const { instance, published } = reporter(); + instance.report({ progress: 1, message: `bold\n\u0007line‮${"x".repeat(1_000)}` }); + const message = published[0]!.payload.message as string; + expect(message.startsWith("bold line ")).toBe(true); + expect(message).not.toMatch(/[\n\u0007‮]/); + expect(message.length).toBe(TOOL_CALL_PROGRESS_MESSAGE_MAX_LENGTH); + }); + + it("drops unusable and out-of-order updates", () => { + const { instance, published } = reporter(); + instance.report({ progress: "1", total: 2 }); + instance.report({ progress: Number.NaN }); + instance.report({ progress: -1 }); + expect(published).toEqual([]); + instance.report({ progress: 5, total: 2 }); + expect(published[0]!.payload).toMatchObject({ progress: 5, total: null }); + vi.advanceTimersByTime(TOOL_CALL_PROGRESS_PUBLISH_INTERVAL_MS); + instance.report({ progress: 3, total: 10 }); + vi.advanceTimersByTime(TOOL_CALL_PROGRESS_PUBLISH_INTERVAL_MS); + expect(published).toHaveLength(1); + }); + + it("on finish stops pending work and tells viewers to drop the progress", () => { + const { instance, published, saved } = reporter(); + instance.report({ progress: 1, total: 3 }); + instance.report({ progress: 2, total: 3 }); + instance.finish(); + vi.advanceTimersByTime(TOOL_CALL_PROGRESS_SAVE_INTERVAL_MS * 3); + expect(published.map((event) => [event.payload.progress, event.payload.done])).toEqual([[1, false], [2, true]]); + expect(saved).toHaveLength(1); + instance.report({ progress: 3, total: 3 }); + expect(published).toHaveLength(2); + }); + + it("a call that never reported progress finishes silently", () => { + const { instance, published } = reporter(); + instance.finish(); + expect(published).toEqual([]); + }); + + it("never throws into the call when publishing or saving fails", async () => { + const instance = createToolCallProgressReporter({ + companyId: "c", runId: "r", issueId: null, agentId: null, invocationId: "i", toolName: "t", + publish: () => { throw new Error("socket gone"); }, + save: async () => { throw new Error("db gone"); }, + }); + expect(() => instance.report({ progress: 1 })).not.toThrow(); + expect(() => instance.finish()).not.toThrow(); + await vi.runAllTimersAsync(); + }); + + it("keeps one save in flight, only the newest update waits, and finish drops it", async () => { + let release!: () => void; + const saves: number[] = []; + const instance = createToolCallProgressReporter({ + companyId: "c", runId: "r", issueId: null, agentId: null, invocationId: "i", toolName: "t", + publish: () => undefined, + save: (progress) => { saves.push(progress.progress); return new Promise((resolve) => { release = resolve; }); }, + }); + instance.report({ progress: 1 }); + // The database is slow: three more save intervals pass while the first write runs. + for (let step = 2; step <= 4; step++) { + vi.advanceTimersByTime(TOOL_CALL_PROGRESS_SAVE_INTERVAL_MS); + instance.report({ progress: step }); + } + vi.advanceTimersByTime(TOOL_CALL_PROGRESS_SAVE_INTERVAL_MS); + expect(saves).toEqual([1]); + release(); + await vi.advanceTimersByTimeAsync(0); + expect(saves).toEqual([1, 4]); + instance.report({ progress: 5 }); + vi.advanceTimersByTime(TOOL_CALL_PROGRESS_SAVE_INTERVAL_MS); + instance.finish(); + release(); + await vi.advanceTimersByTimeAsync(TOOL_CALL_PROGRESS_SAVE_INTERVAL_MS * 3); + expect(saves).toEqual([1, 4]); + }); + + it("names the tool the way native runners name it, so their rows can find the call", () => { + const instance = createToolCallProgressReporter({ + companyId: "c", runId: "r", issueId: null, agentId: null, invocationId: "i", + toolName: "mcp.kv-demo-1a2b3c4d:slow-with-progress", + publish: (event) => { names.push(event.payload.nativeToolName); }, + }); + const names: unknown[] = []; + instance.report({ progress: 1 }); + expect(names).toEqual(["app_mcp_kv_demo_1a2b3c4d_slow_with_progress_889fb9a8d98e"]); + }); + + it("gives each call its own progress token", () => { + expect(reporter().instance.token).not.toBe(reporter().instance.token); + }); +}); diff --git a/server/src/__tests__/tool-gateway-service.test.ts b/server/src/__tests__/tool-gateway-service.test.ts index bf5f679e858..9f7967cb5e9 100644 --- a/server/src/__tests__/tool-gateway-service.test.ts +++ b/server/src/__tests__/tool-gateway-service.test.ts @@ -3297,7 +3297,7 @@ describeEmbeddedPostgres("tool gateway service", () => { const calls = provider.mock.calls.map(([, init]) => JSON.parse(String(init.body))) .filter((payload) => payload.method === "tools/call"); expect(calls).toHaveLength(1); - expect(calls[0].params).toEqual({ name: "search_messages", arguments: parameters }); + expect(calls[0].params).toEqual({ name: "search_messages", arguments: parameters, _meta: { progressToken: expect.stringMatching(/^paperclip-progress-/) } }); }); it("explains Google Workspace preview enrollment when a tool call is denied", async () => { diff --git a/server/src/__tests__/tool-gateway.test.ts b/server/src/__tests__/tool-gateway.test.ts index 9d64161f75e..1fc5b2701c1 100644 --- a/server/src/__tests__/tool-gateway.test.ts +++ b/server/src/__tests__/tool-gateway.test.ts @@ -6,6 +6,9 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import express from "express"; import { and, eq, sql } from "drizzle-orm"; +import type { LiveEvent } from "@paperclipai/shared"; +import { subscribeCompanyLiveEvents } from "../services/live-events.js"; +import { assignedToolName as nativeAssignedToolName } from "../services/native-runtime/assigned-mcp-tools.js"; import request from "supertest"; import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; import { @@ -3584,6 +3587,7 @@ rl.on("line", (line) => { "kv_get", "kv_list", "kv_set", + "slow_with_progress", ]); await allowAllToolsForAgent(db, company.id, agent.id); @@ -3618,6 +3622,61 @@ rl.on("line", (line) => { } }); + it("shows a run's viewers the provider's progress for a long call, saves it on the call, and returns the same result", async () => { + const company = await createCompany(db); + const other = await createCompany(db); + const agent = await createAgent(db, company.id); + const { issue, run } = await createIssueAndRun(db, company.id, agent.id); + const kvDemo: KvDemoHttpServer = createKvDemoHttpServer(); + const port = await kvDemo.listen(0, "127.0.0.1"); + const events: LiveEvent[] = []; + const otherEvents: LiveEvent[] = []; + const stop = subscribeCompanyLiveEvents(company.id, (event) => { events.push(event); }); + const stopOther = subscribeCompanyLiveEvents(other.id, (event) => { otherEvents.push(event); }); + try { + const access = toolAccessService(db); + const connection = await access.createConnection(company.id, { + name: "KV demo progress fixture", connectionPurpose: "tool", transport: "mcp_remote", authKind: "none", + ownership: "customer", connectionKind: "managed", + config: { url: `http://127.0.0.1:${port}/mcp` }, transportConfig: { url: `http://127.0.0.1:${port}/mcp` }, + credentialSecretRefs: [], enabled: true, status: "active", + }); + await access.refreshCatalog(connection.id, { actorType: "user", actorId: "board" }); + await allowAllToolsForAgent(db, company.id, agent.id); + const gateway = createTestToolGatewayService(db); + const session = await gateway.createSession({ companyId: company.id, agentId: agent.id, runId: run.id, issueId: issue.id }); + const slowTool = (await gateway.listToolsForSession(session.token)) + .find((tool) => tool.upstreamToolName === "slow_with_progress"); + expect(slowTool).toBeTruthy(); + + const result = await gateway.executeTool({ + sessionToken: session.token, + tool: slowTool!.name, + parameters: { steps: 6, stepMs: 200 }, + }); + expect(result).toMatchObject({ status: "completed", result: { data: { isError: false, transport: "mcp_http" } } }); + + const progress = events.filter((event) => event.type === "heartbeat.run.tool_progress"); + // Throttled: six provider updates over ~1.2s, a handful of events, the last one marks the end. + expect(progress.length).toBeGreaterThanOrEqual(3); + expect(progress.length).toBeLessThanOrEqual(7); + expect(progress.every((event) => event.companyId === company.id && event.payload.runId === run.id + && event.payload.issueId === issue.id && event.payload.toolName === slowTool!.name + && event.payload.nativeToolName === nativeAssignedToolName(slowTool!.name))).toBe(true); + expect(progress[0]!.payload).toMatchObject({ progress: 1, total: 6, message: "Finished step 1 of 6", done: false }); + expect(progress.at(-1)!.payload).toMatchObject({ done: true }); + expect(otherEvents.filter((event) => event.type === "heartbeat.run.tool_progress")).toEqual([]); + const [invocation] = await db.select().from(toolInvocations) + .where(eq(toolInvocations.id, String(progress[0]!.payload.invocationId))); + expect(invocation).toMatchObject({ companyId: company.id, runId: run.id, status: "succeeded" }); + expect(invocation!.progress).toMatchObject({ total: 6 }); + } finally { + stop(); + stopOther(); + await kvDemo.close(); + } + }); + it("expires legacy managed-connector approvals before provider dispatch", async () => { const company = await createCompany(db); const agent = await createAgent(db, company.id); diff --git a/server/src/realtime/live-event-delivery.test.ts b/server/src/realtime/live-event-delivery.test.ts index 7d3dd8c40b3..44f27a90395 100644 --- a/server/src/realtime/live-event-delivery.test.ts +++ b/server/src/realtime/live-event-delivery.test.ts @@ -303,3 +303,15 @@ it("keeps a read set at 249 ms and replaces it at 250 ms", async () => { f.delivery.dispose(); } finally { clock.mockRestore(); } }); +it("sends tool call progress only to clients watching that run", async () => { + const f = fixture(); f.subscribe(["run-1"]); + const progress = (id: number, runId: string): LiveEvent => ({ id, companyId: "company", type: "heartbeat.run.tool_progress", + createdAt: "2026-11-13T00:00:00Z", payload: { runId, agentId: "agent", invocationId: `call-${id}`, progress: id, total: 10, message: "Exporting" } }); + f.delivery.enqueue(progress(1, "run-1")); f.delivery.enqueue(progress(2, "run-2")); await settle(); + const delivered = f.frames.filter(frame => frame.type === "heartbeat.run.tool_progress"); + expect(delivered.map(frame => frame.payload.runId)).toEqual(["run-1"]); + // Another run's progress carries no content to this client. + expect(JSON.stringify(f.frames)).not.toContain("call-2"); + expect(f.decorate).not.toHaveBeenCalled(); + f.delivery.dispose(); +}); diff --git a/server/src/realtime/live-event-delivery.ts b/server/src/realtime/live-event-delivery.ts index 3c49e992bfb..2641be7d70d 100644 --- a/server/src/realtime/live-event-delivery.ts +++ b/server/src/realtime/live-event-delivery.ts @@ -16,7 +16,10 @@ export function sharedDeliveryRead(reads: LiveDeliveryReads | undefined, key: pending.catch(() => { if (reads.get(key) === pending) reads.delete(key); }); return pending; } -const transcript = (event: LiveEvent) => event.type === "heartbeat.run.event" || event.type === "heartbeat.run.log"; +// Per-run streams: sent only to clients that asked for the run. Tool progress +// follows its run's transcript, since only an open transcript shows it. +const transcript = (event: LiveEvent) => event.type === "heartbeat.run.event" || event.type === "heartbeat.run.log" + || event.type === "heartbeat.run.tool_progress"; const runId = (event: LiveEvent) => typeof event.payload.runId === "string" ? event.payload.runId : null; interface Socket { readyState: number; diff --git a/server/src/realtime/live-events-ws.ts b/server/src/realtime/live-events-ws.ts index b09a85d65d1..e5d3d23031b 100644 --- a/server/src/realtime/live-events-ws.ts +++ b/server/src/realtime/live-events-ws.ts @@ -315,7 +315,7 @@ export function setupLiveEventsWebSocketServer( : payload.entityType === "issue" && typeof payload.entityId === "string" ? payload.entityId : null; const runId = typeof payload.runId === "string" ? payload.runId : payload.entityType === "heartbeat_run" && typeof payload.entityId === "string" ? payload.entityId : null; - if (event.type === "heartbeat.run.event" || event.type === "heartbeat.run.log") { + if (event.type === "heartbeat.run.event" || event.type === "heartbeat.run.log" || event.type === "heartbeat.run.tool_progress") { const allowed = await sharedDeliveryRead(reads, "transcript-scope", async () => (await access.decide({ actor: context!.actor, action: "company_scope:read", resource: { type: "company", companyId: event.companyId } })).allowed); diff --git a/server/src/routes/mcp-apps.ts b/server/src/routes/mcp-apps.ts index 1a253223f24..e8a1537eaf9 100644 --- a/server/src/routes/mcp-apps.ts +++ b/server/src/routes/mcp-apps.ts @@ -15,6 +15,7 @@ import { assertBoard, getAccessibleResource } from "./authz.js"; import { connectionRequestUserId } from "./connection-management-authz.js"; import { authorizationService } from "../services/authorization.js"; import { mcpAppViewsService, toMcpAppViewSummary } from "../services/mcp-app-views.js"; +import { listToolCallProgress } from "../services/tool-call-progress.js"; import { ToolGatewayHttpError, type ToolGatewayService } from "../services/tool-gateway.js"; const uuidSchema = z.string().uuid(); @@ -34,7 +35,7 @@ export function mcpAppsRoutes(db: Db, toolGateway: McpAppsGateway) { const router = Router(); const views = mcpAppViewsService(db); - router.use("/issues/:issueId/mcp-app-views", async (req, res, next) => { + const readableTask = async (req: Request, res: Response, next: () => void) => { assertBoard(req); if (!uuidSchema.safeParse(req.params.issueId).success) { res.status(404).json({ error: "Task not found" }); @@ -57,7 +58,9 @@ export function mcpAppsRoutes(db: Db, toolGateway: McpAppsGateway) { res.locals.mcpAppIssue = issue; res.setHeader("Cache-Control", "no-store"); next(); - }); + }; + router.use("/issues/:issueId/mcp-app-views", readableTask); + router.use("/issues/:issueId/tool-call-progress", readableTask); async function viewFor(req: Request, res: Response) { const issue = res.locals.mcpAppIssue as { id: string; companyId: string }; @@ -162,6 +165,12 @@ export function mcpAppsRoutes(db: Db, toolGateway: McpAppsGateway) { res.status(201).json({ commentId: comment.id }); }); + // The latest progress of the task's running app tool calls (and recently finished ones, as done), for a reload mid-call. + router.get("/issues/:issueId/tool-call-progress", async (_req, res) => { + const issue = res.locals.mcpAppIssue as { id: string; companyId: string }; + res.json(await listToolCallProgress(db, issue.companyId, issue.id)); + }); + router.get("/issues/:issueId/mcp-app-views", async (req, res) => { const issue = res.locals.mcpAppIssue as { id: string; companyId: string }; res.json(await views.listForIssue(issue.companyId, issue.id)); diff --git a/server/src/routes/openapi.ts b/server/src/routes/openapi.ts index 69c73db0b19..b54469d607c 100644 --- a/server/src/routes/openapi.ts +++ b/server/src/routes/openapi.ts @@ -1447,6 +1447,7 @@ const browserUseOperations = [ // MCP Apps views in the task chat. Board only; reads follow task read access. const mcpAppsOperations = [ ["get", "/api/issues/{issueId}/mcp-app-views", "List a task's MCP Apps views", undefined], + ["get", "/api/issues/{issueId}/tool-call-progress", "Read the latest progress of a task's running app tool calls", undefined], ["get", "/api/issues/{issueId}/mcp-app-views/{viewId}", "Read an MCP Apps view with its document", undefined], ["post", "/api/issues/{issueId}/mcp-app-views/{viewId}/tool-calls", "Call a tool from an MCP Apps view", mcpAppToolCallSchema], ["put", "/api/issues/{issueId}/mcp-app-views/{viewId}/model-context", "Store an MCP Apps view's context for the task's next run", mcpAppModelContextUpdateSchema], diff --git a/server/src/services/mcp-http.ts b/server/src/services/mcp-http.ts index 8fbdb7fab90..ef7f590d4ce 100644 --- a/server/src/services/mcp-http.ts +++ b/server/src/services/mcp-http.ts @@ -209,11 +209,13 @@ export type McpJsonRpcRequest = { id?: string | number; method: string; params?: export function mcpHttpRequestInit( headers: Record | undefined, message: McpJsonRpcRequest, - options: { toolInputSchema?: unknown; declareTasks?: boolean; elicitationForm?: boolean } = {}, + options: { toolInputSchema?: unknown; declareTasks?: boolean; elicitationForm?: boolean; progressToken?: string } = {}, ): { method: "POST"; headers: Record; body: string } { const requestHeaders = mcpHttpRequestHeaders(headers); const protocolVersion = mcpProtocolVersionOf(headers); let params = message.params ?? {}; + // Asks the server to stream `notifications/progress` for this request. + if (options.progressToken) params = { ...params, _meta: { ...record(params._meta), progressToken: options.progressToken } }; if (isModernMcpProtocolVersion(protocolVersion)) { params = { ...params, @@ -666,6 +668,15 @@ export function parseMcpHttpResponseBody(bodyText: string, contentType: string | throw new SyntaxError("MCP SSE response contained no data events"); } +/** A `notifications/progress` for this request's token, or null for any other message. */ +export function mcpProgressNotificationOf(message: unknown, progressToken: string | undefined): { progress: unknown; total: unknown; message: unknown } | null { + const payload = record(message); + if (!progressToken || !payload || payload.method !== "notifications/progress" || "id" in payload) return null; + const params = record(payload.params); + if (!params || params.progressToken !== progressToken) return null; + return { progress: params.progress, total: params.total, message: params.message }; +} + export type McpHttpResponseProgress = { bytesReceived: number; chunksReceived: number; @@ -683,6 +694,15 @@ export async function readMcpHttpResponse( maxBytes?: number; onRequest?: (message: Record) => Promise; onProgress?: (progress: McpHttpResponseProgress) => void; + /** + * Bound an SSE stream per message instead of in total, so handled + * notifications (such as progress) do not use up the result's limit. Only + * for callers with an overall deadline: a stream of small notifications + * could otherwise never end. + */ + limitPerMessage?: boolean; + /** Notifications sent on this request's stream before its result. Must not block. */ + onNotification?: (message: Record) => void; } = {}, ): Promise { const progress: McpHttpResponseProgress = { bytesReceived: 0, chunksReceived: 0, eventsReceived: 0, matchedResponse: false, ended: false }; @@ -714,6 +734,10 @@ export async function readMcpHttpResponse( return record; } if ("method" in record && "id" in record) await options.onRequest?.(record); + else if ("method" in record) { + // A notification observer never fails or delays the call. + try { options.onNotification?.(record); } catch { /* ignored */ } + } return undefined; }; const event = async (value: string) => { @@ -729,17 +753,22 @@ export async function readMcpHttpResponse( progress.bytesReceived = bytes; if (value) progress.chunksReceived += 1; progress.ended = done; - if (bytes > maxBytes) throw new McpHttpResponseError("too_large", "MCP response exceeded the size limit"); + // A JSON body is one message. An SSE stream is bounded in total unless the + // caller asks for a per-message bound (see limitPerMessage). + const perMessage = isStream && options.limitPerMessage === true; + if (!perMessage && bytes > maxBytes) throw new McpHttpResponseError("too_large", "MCP response exceeded the size limit"); buffer += decoder.decode(value, { stream: !done }); if (isStream) { // Normalize CRLF after concatenating chunks, including split CR/LF pairs. buffer = buffer.replace(/\r\n/g, "\n"); let boundary: number; while ((boundary = buffer.indexOf("\n\n")) >= 0) { + if (perMessage && Buffer.byteLength(buffer.slice(0, boundary), "utf8") > maxBytes) throw new McpHttpResponseError("too_large", "MCP response exceeded the size limit"); const result = await event(buffer.slice(0, boundary)); buffer = buffer.slice(boundary + 2); if (result !== undefined) return result; } + if (perMessage && Buffer.byteLength(buffer, "utf8") > maxBytes) throw new McpHttpResponseError("too_large", "MCP response exceeded the size limit"); } if (done) break; } diff --git a/server/src/services/tool-call-progress.ts b/server/src/services/tool-call-progress.ts new file mode 100644 index 00000000000..429142b781e --- /dev/null +++ b/server/src/services/tool-call-progress.ts @@ -0,0 +1,195 @@ +import { randomUUID } from "node:crypto"; +import { and, desc, eq, gte, isNotNull, isNull } from "drizzle-orm"; +import { toolInvocations, type Db } from "@paperclipai/db"; +import { + normalizeToolCallProgress, + type LiveEventType, + type ToolCallProgress, + type ToolCallProgressSnapshot, +} from "@paperclipai/shared"; +import { logger } from "../middleware/logger.js"; +import { publishLiveEvent } from "./live-events.js"; +import { assignedToolName } from "./native-runtime/assigned-mcp-tools.js"; + +/** + * Live progress for a connected-app tool call that an agent run made. + * + * The provider streams MCP `notifications/progress` on the call's response. + * Each update is published to the run's viewers at most every + * PUBLISH_INTERVAL_MS (the latest one wins) and saved on the invocation at + * most every SAVE_INTERVAL_MS, so a reload mid-call shows the latest state. + * Nothing here is awaited by the call: the result is never delayed. + */ +export const TOOL_CALL_PROGRESS_PUBLISH_INTERVAL_MS = 250; +export const TOOL_CALL_PROGRESS_SAVE_INTERVAL_MS = 1_000; +export const TOOL_CALL_PROGRESS_EVENT: LiveEventType = "heartbeat.run.tool_progress"; +/** Progress older than this is not shown after a reload (the call is long gone). */ +const SNAPSHOT_MAX_AGE_MS = 24 * 60 * 60_000; + +export interface ToolCallProgressReporter { + /** Sent as the request's `_meta.progressToken`. */ + readonly token: string; + /** Records a provider update. Never throws, never blocks. */ + report(update: { progress: unknown; total?: unknown; message?: unknown }): void; + /** The call ended: stop timers and tell viewers to drop the progress. */ + finish(): void; +} + +type Publish = (input: { companyId: string; type: LiveEventType; payload: Record }) => void; +type Save = (progress: ToolCallProgress) => Promise; + +/** Leading-edge throttle that keeps only the latest value for its trailing run. */ +function latestThrottle(intervalMs: number, run: (value: T) => void) { + let lastAt = -Infinity; + let pending: { value: T } | null = null; + let timer: ReturnType | null = null; + const flush = () => { + timer = null; + if (!pending) return; + const { value } = pending; + pending = null; + lastAt = Date.now(); + run(value); + }; + return { + push(value: T) { + pending = { value }; + if (timer) return; + const wait = lastAt + intervalMs - Date.now(); + if (wait <= 0) flush(); + else { timer = setTimeout(flush, wait); timer.unref?.(); } + }, + cancel() { + if (timer) clearTimeout(timer); + timer = null; + pending = null; + }, + }; +} + +export function createToolCallProgressReporter(input: { + companyId: string; + runId: string; + issueId: string | null; + agentId: string | null; + invocationId: string; + toolName: string; + startedAt?: Date; + publish?: Publish; + save?: Save; +}): ToolCallProgressReporter { + const publish: Publish = input.publish ?? ((event) => { publishLiveEvent(event); }); + const startedAt = (input.startedAt ?? new Date()).toISOString(); + const token = `paperclip-progress-${randomUUID()}`; + let finished = false; + let last: ToolCallProgress | null = null; + const snapshot = (progress: ToolCallProgress, done: boolean): ToolCallProgressSnapshot => ({ + ...progress, invocationId: input.invocationId, runId: input.runId, toolName: input.toolName, + nativeToolName: assignedToolName(input.toolName), startedAt, done, + }); + const send = (value: ToolCallProgressSnapshot) => { + try { + publish({ + companyId: input.companyId, + type: TOOL_CALL_PROGRESS_EVENT, + payload: { ...value, issueId: input.issueId, agentId: input.agentId }, + }); + } catch (err) { + logger.debug({ err, invocationId: input.invocationId }, "tool call progress publish failed"); + } + }; + const live = latestThrottle(TOOL_CALL_PROGRESS_PUBLISH_INTERVAL_MS, (progress) => send(snapshot(progress, false))); + // At most one save in flight; while it runs only the newest update waits. + let saving = false; + let queuedSave: ToolCallProgress | null = null; + const save = (progress: ToolCallProgress) => { + if (!input.save || finished) return; + if (saving) { queuedSave = progress; return; } + saving = true; + input.save(progress) + .catch((err) => logger.debug({ err, invocationId: input.invocationId }, "tool call progress save failed")) + .finally(() => { + saving = false; + const next = queuedSave; + queuedSave = null; + if (next) save(next); + }); + }; + const saved = latestThrottle(TOOL_CALL_PROGRESS_SAVE_INTERVAL_MS, save); + return { + token, + report(update) { + if (finished) return; + const progress = normalizeToolCallProgress(update); + // Progress must increase; a lower value is out of order and is dropped. + if (!progress || (last && progress.progress < last.progress)) return; + last = progress; + live.push(progress); + if (input.save) saved.push(progress); + }, + finish() { + if (finished) return; + finished = true; + live.cancel(); + saved.cancel(); + queuedSave = null; + if (last) send(snapshot(last, true)); + }, + }; +} + +/** Saves a running call's latest progress. A finished call is never touched. */ +export function saveToolCallProgress(db: Db, companyId: string, invocationId: string): Save { + return async (progress) => { + await db.update(toolInvocations) + .set({ progress }) + .where(and(eq(toolInvocations.id, invocationId), eq(toolInvocations.companyId, companyId), isNull(toolInvocations.completedAt))); + }; +} + +/** Finished calls are reported for this long, so a chat that missed their end learns it. */ +const FINISHED_REPORT_MS = 60 * 60_000; + +/** + * The latest progress of a task's running tool calls, for a reload mid-call, + * plus its recently finished calls marked `done`. A chat drops a call only on + * `done`: a running call may not have saved its first update yet. + */ +export async function listToolCallProgress(db: Db, companyId: string, issueId: string, now = new Date()): Promise { + const columns = { + id: toolInvocations.id, + runId: toolInvocations.runId, + toolName: toolInvocations.toolName, + progress: toolInvocations.progress, + startedAt: toolInvocations.startedAt, + createdAt: toolInvocations.createdAt, + completedAt: toolInvocations.completedAt, + }; + const task = and(eq(toolInvocations.companyId, companyId), eq(toolInvocations.issueId, issueId), isNotNull(toolInvocations.runId)); + const [running, finished] = await Promise.all([ + db.select(columns).from(toolInvocations).where(and(task, + isNull(toolInvocations.completedAt), + isNotNull(toolInvocations.progress), + gte(toolInvocations.createdAt, new Date(now.getTime() - SNAPSHOT_MAX_AGE_MS)), + )).orderBy(desc(toolInvocations.createdAt)).limit(50), + db.select(columns).from(toolInvocations).where(and(task, + gte(toolInvocations.completedAt, new Date(now.getTime() - FINISHED_REPORT_MS)), + )).orderBy(desc(toolInvocations.completedAt)).limit(50), + ]); + return [...running, ...finished].flatMap((row) => { + const done = row.completedAt !== null; + const savedAt = row.progress ? Date.parse(String(row.progress.updatedAt)) : NaN; + const at = done ? row.completedAt! : Number.isFinite(savedAt) ? new Date(savedAt) : now; + const progress = normalizeToolCallProgress(row.progress ?? (done ? { progress: 0 } : {}), at); + if (!progress || !row.runId) return []; + return [{ + ...progress, + invocationId: row.id, + runId: row.runId, + toolName: row.toolName, + nativeToolName: assignedToolName(row.toolName), + startedAt: (row.startedAt ?? row.createdAt).toISOString(), + done, + }]; + }); +} diff --git a/server/src/services/tool-gateway.ts b/server/src/services/tool-gateway.ts index 90ac0b99ff1..7bd6d594bcd 100644 --- a/server/src/services/tool-gateway.ts +++ b/server/src/services/tool-gateway.ts @@ -203,7 +203,9 @@ import { type McpInputRequired, mcpHttpRequestHeaders, parseMcpHttpResponseBody, + mcpProgressNotificationOf, } from "./mcp-http.js"; +import { createToolCallProgressReporter, saveToolCallProgress, type ToolCallProgressReporter } from "./tool-call-progress.js"; import { buildElicitationQuestionSet, elicitationQuestionHash, @@ -5505,6 +5507,8 @@ export function createToolGatewayService( protocolMethod?: string; protocolParams?: Record; timeoutMs: number; + /** Receives the call's `notifications/progress`; its token is sent with tools/call. */ + progressReporter?: ToolCallProgressReporter | null; }, /** One restart and one deadline are shared by every protocol recovery in a call. */ recovery: { restarted: boolean; deadline: number } = { restarted: false, deadline: Date.now() + input.timeoutMs }, @@ -5585,6 +5589,8 @@ export function createToolGatewayService( try { const message = JSON.parse(line) as Record; const id = typeof message.id === "number" ? message.id : null; + const progressUpdate = mcpProgressNotificationOf(message, input.progressReporter?.token); + if (progressUpdate) input.progressReporter!.report(progressUpdate); if (id !== null && pending.has(id)) { const waiter = pending.get(id)!; pending.delete(id); @@ -5760,6 +5766,7 @@ export function createToolGatewayService( withMeta(protocolVersion, input.protocolParams ?? { name: input.entry?.toolName, arguments: input.parameters ?? {}, + ...(input.progressReporter ? { _meta: { progressToken: input.progressReporter.token } } : {}), }), ); } catch (error) { @@ -7816,6 +7823,7 @@ export function createToolGatewayService( const controller = new AbortController(); const timer = setTimeout(() => controller.abort(), ms); timer.unref?.(); + const progressReporter = toolCallProgressReporterFor(session, tool, invocationId); // The cached MCP session this call used, so a failure resets only this // identity's session and leaves other agents on the connection alone. let mcpSession: { scope: string; headers: Record; sessionId: string } | undefined; @@ -8026,6 +8034,7 @@ export function createToolGatewayService( toolInputSchema: entry.inputSchema ?? undefined, // Paperclip can wait on a task here, durably (see waitForMcpTask). declareTasks: tasksOn, + progressToken: progressReporter?.token, // A person can answer the app's form questions in the task chat. elicitationForm: Boolean(elicitationCall), }), @@ -8245,7 +8254,13 @@ export function createToolGatewayService( const body = response.ok ? JSON.stringify(await readMcpHttpResponse(response, callRequestId, { maxBytes: MAX_REMOTE_MCP_RESPONSE_BYTES, + // The call has a deadline (controller), so progress may outlast the total limit. + limitPerMessage: true, onProgress: (progress) => { execution.stream = progress; }, + onNotification: (message) => { + const update = mcpProgressNotificationOf(message, progressReporter?.token); + if (update) progressReporter!.report(update); + }, onRequest: async (message) => { const pending = extractRemoteMcpPending(message); if (pending) { @@ -8508,15 +8523,31 @@ export function createToolGatewayService( ); } finally { clearTimeout(timer); + progressReporter?.finish(); } } + /** Progress is shown to the run's viewers, so only a call made by a run reports it. */ + function toolCallProgressReporterFor(session: ToolGatewaySession, tool: ToolGatewayDescriptor, invocationId: string | undefined) { + if (!session.runId || !invocationId) return null; + return createToolCallProgressReporter({ + companyId: session.companyId, + runId: session.runId, + issueId: session.issueId, + agentId: session.agentId, + invocationId, + toolName: tool.name, + save: saveToolCallProgress(db, session.companyId, invocationId), + }); + } + async function executeLocalStdioTool( session: ToolGatewaySession, tool: ToolGatewayDescriptor, parameters: unknown, ms: number, useProviderDefaultTimeout = false, + invocationId?: string, ): Promise { const { entry, connection } = await resolveConnectedLocalStdioTool( session, @@ -8530,8 +8561,10 @@ export function createToolGatewayService( template, grant, ); - const invoke = () => callLocalStdioMcp({ connection, entry, template, env, parameters, - timeoutMs: useProviderDefaultTimeout && template.templateId === "paperclip.cognee-cloud" ? 60_000 : ms }); + const progressReporter = template.templateId === "paperclip.cognee-cloud" ? null : toolCallProgressReporterFor(session, tool, invocationId); + const invoke = () => callLocalStdioMcp({ connection, entry, template, env, parameters, progressReporter, + timeoutMs: useProviderDefaultTimeout && template.templateId === "paperclip.cognee-cloud" ? 60_000 : ms }) + .finally(() => progressReporter?.finish()); // Cognee is a bundled HTTP client. Provider failures are tool failures, not // crashed local processes, and must never consume slots or restart budgets. const result = template.templateId === "paperclip.cognee-cloud" @@ -9579,6 +9612,7 @@ export function createToolGatewayService( args.parameters, executionTimeoutMs, args.timeoutMs === undefined, + args.invocationId, ) : null; if (!connectedMcpExecution) { @@ -10456,6 +10490,8 @@ export function createToolGatewayService( tool, parameters, executionTimeoutMs, + false, + invocation.id, ) : null; const result = connectedMcpExecution @@ -13432,6 +13468,7 @@ export function createToolGatewayService( effectiveParameters, executionTimeoutMs, input.timeoutMs === undefined, + invocationId, ) : null; const result = recoveredGitHubPublication ?? (connectedMcpExecution diff --git a/tests/runner-e2e/source-sdk-public-inputs.json b/tests/runner-e2e/source-sdk-public-inputs.json index ddb291fc18b..3be3def9c97 100644 --- a/tests/runner-e2e/source-sdk-public-inputs.json +++ b/tests/runner-e2e/source-sdk-public-inputs.json @@ -229,7 +229,7 @@ "packages/shared/src/connector-availability.ts": "82aec0dfb2e9b8cb04e21c4bcd9aab74b087c76a", "packages/shared/src/connector-identity.test.ts": "d58357707faefa7c98bb906cdaeb4db39d31081a", "packages/shared/src/connector-identity.ts": "5ede83c9c5225dbe8af9cf3a49c74e477ef5b955", - "packages/shared/src/constants.ts": "3149ee1cc5ffa34b50ecd493276b3a70fc46adc2", + "packages/shared/src/constants.ts": "47973de810c0dfac21625e2687af80b37e8da00c", "packages/shared/src/crash-point-registry.ts": "3cbf2d290bc111061376b43020848623e493737c", "packages/shared/src/crash-points.test.ts": "3564a1f62d22f54a3c2fb0a72b3a6cde17c957d2", "packages/shared/src/crash-points.ts": "c15259f39610e352eac5423a491548ec1bae94fe", @@ -268,7 +268,7 @@ "packages/shared/src/home-paths.ts": "fa4f1c440653b76b3dbb7a51026d557287deb45d", "packages/shared/src/humanize-connection.test.ts": "ba9e90a44c63452b0bf9f733adb90241921e8eca", "packages/shared/src/humanize-connection.ts": "fbd414476a0ef7106a7c80d8572208781b091699", - "packages/shared/src/index.ts": "2d2b5ba044c9c7a22ff99022c0fac1e0da5e4a94", + "packages/shared/src/index.ts": "7128882e69b34ebeeefbab5b1ef5f0252c13d4b4", "packages/shared/src/issue-activity-identity.test.ts": "3dc294640f54c2be67943e3e0149cf4eff2c95ca", "packages/shared/src/issue-activity-identity.ts": "1b0f07ec3fe0506aaaf80a4fa9893f3baf8f0075", "packages/shared/src/issue-attribution.test.ts": "84563685e7f3c48029243cf1cc24a3a09ee3cde9", @@ -366,6 +366,7 @@ "packages/shared/src/telemetry/server-started-event.test.ts": "3d0e0e2289badb41c92ba0d69e3668d00b3b5fdb", "packages/shared/src/telemetry/state.ts": "a060af26a8e9a82a8afb0e2fccbfe6a59e9d0a8a", "packages/shared/src/telemetry/types.ts": "083d4341a72ac47bc1a3bbff163965de8b06f90a", + "packages/shared/src/tool-call-progress.ts": "89655767bfd0180e99ac517754d72e25f753eb33", "packages/shared/src/trust-policy.ts": "88001a8d629a2af66dcd8d75b7b4d36298455ef6", "packages/shared/src/types/access.ts": "015d51792da07acaebe59e0e395698d207811d0d", "packages/shared/src/types/activity.ts": "6f5f27417a0bb26ec4daba6ec16a04ce31e484f2", diff --git a/ui/src/api/mcpApps.ts b/ui/src/api/mcpApps.ts index 45d0b835b1e..b5b7fa0f746 100644 --- a/ui/src/api/mcpApps.ts +++ b/ui/src/api/mcpApps.ts @@ -1,4 +1,4 @@ -import type { McpAppCallToolResult, McpAppModelContext, McpAppViewDetail, McpAppViewSummary } from "@paperclipai/shared"; +import type { McpAppCallToolResult, McpAppModelContext, McpAppViewDetail, McpAppViewSummary, ToolCallProgressSnapshot } from "@paperclipai/shared"; import { api } from "./client"; /** MCP Apps views recorded for a task's tool calls. */ @@ -17,4 +17,7 @@ export const mcpAppsApi = { /** Records that the viewer sent a view's proposed message (sent with this client request id). */ messageSent: (issueId: string, viewId: string, clientRequestId: string) => api.post<{ commentId: string }>(`/issues/${issueId}/mcp-app-views/${viewId}/messages`, { clientRequestId }), + /** The latest progress of the task's running app tool calls (for a reload mid-call). */ + toolCallProgress: (issueId: string) => + api.get(`/issues/${issueId}/tool-call-progress`, { cache: "no-store" }), }; diff --git a/ui/src/components/TaskChatThread.tsx b/ui/src/components/TaskChatThread.tsx index 7594e3307a4..4c5286945fe 100644 --- a/ui/src/components/TaskChatThread.tsx +++ b/ui/src/components/TaskChatThread.tsx @@ -47,6 +47,7 @@ import { TaskChatLiveTail } from "@/components/task-chat/TaskChatLiveTail"; import { McpAppIssueProvider } from "@/components/mcp-app/McpAppIssueContext"; import { ProviderToolIconsProvider } from "@/components/ProviderToolIcon"; import { McpAppTurnViews, mcpAppRunRefreshKey } from "@/components/mcp-app/McpAppTurnViews"; +import { ToolCallProgressProvider } from "@/components/task-chat/tool-call-progress"; import { TaskChatRunnerHeader, TaskChatRunnerTurn } from "@/components/task-chat/TaskChatRunnerTurn"; import { partitionRunHealthItems } from "@/components/task-chat/TaskChatRunHealthBanner"; import { @@ -3372,6 +3373,7 @@ export function TaskChatThread(props: TaskChatThreadProps) { + + diff --git a/ui/src/components/task-chat/TaskChatRunnerTurn.tsx b/ui/src/components/task-chat/TaskChatRunnerTurn.tsx index 2416c719ca5..a72c4862cb3 100644 --- a/ui/src/components/task-chat/TaskChatRunnerTurn.tsx +++ b/ui/src/components/task-chat/TaskChatRunnerTurn.tsx @@ -2,6 +2,7 @@ import { groupImageResources } from "./image-resources"; import { AgentQueueWait } from "@/components/AgentQueueWait"; import { agentQueueWaitOf } from "@/lib/agent-queue-wait"; import { McpAppTurnViews, mcpAppRunRefreshKey } from "@/components/mcp-app/McpAppTurnViews"; +import { ToolCallProgressRunProvider } from "./tool-call-progress"; import { useRef, type ReactNode } from "react"; import type { ExecutionProjection } from "@paperclipai/shared"; import { useSecondTick } from "@/hooks/useSecondTick"; @@ -258,7 +259,7 @@ export function TaskChatRunnerTurn({ )); return ( -
) : null} {!final && currentActivityItems.length === 0 ? : null} -
+
); } diff --git a/ui/src/components/task-chat/TaskChatToolCard.tsx b/ui/src/components/task-chat/TaskChatToolCard.tsx index a344f482a53..7e79f3b4cd5 100644 --- a/ui/src/components/task-chat/TaskChatToolCard.tsx +++ b/ui/src/components/task-chat/TaskChatToolCard.tsx @@ -14,6 +14,8 @@ import { useTaskChatShimmer } from "./TaskChatShimmer"; import { TaskChatDiffCounts, TaskChatDiffView } from "./TaskChatDiffView"; import { useTaskReferenceLabel, withTaskReference } from "./task-reference"; import { ProviderToolIcon, useProviderToolIcons } from "@/components/ProviderToolIcon"; +import type { ToolCallProgressSnapshot } from "@paperclipai/shared"; +import { TaskChatToolProgress, useToolCallProgress } from "./tool-call-progress"; const STATUS_ICON = { pending: { Icon: Loader2, spin: false, tone: "text-muted-foreground" }, @@ -48,8 +50,15 @@ function TaskReferenceToolTarget({ taskRefId, target }: { taskRefId: string; tar * inset). An edit row carries its `+N −M` counts and opens to a compact, * height-capped diff of that one edit. */ -export function TaskChatToolCard({ item }: { item: TaskChatToolItem }) { +export function TaskChatToolCard({ item, progress: givenProgress }: { + item: TaskChatToolItem; + /** A fixed progress state (stories); otherwise a running call's live progress is shown. */ + progress?: ToolCallProgressSnapshot | null; +}) { const { Icon, spin, tone } = STATUS_ICON[item.status]; + const liveProgress = useToolCallProgress(item); + // Progress belongs to the running call only; a finished row drops it. + const progress = item.status === "in_progress" ? (givenProgress !== undefined ? givenProgress : liveProgress) : null; const shimmer = useTaskChatShimmer(item.id); const RowIcon = toolTaxonomy(item.rawName ?? item.name).icon; const providerIcons = useProviderToolIcons(item.rawName ?? item.name); @@ -130,6 +139,7 @@ export function TaskChatToolCard({ item }: { item: TaskChatToolItem }) { ) : null} + {progress ? : null} {item.target && showDetail && !item.diff?.lines?.length ? (
({ list: vi.fn() })); +vi.mock("@/api/mcpApps", () => ({ mcpAppsApi: { toolCallProgress: savedProgress.list } })); + +(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true; + +const snapshot = (fields: Partial = {}): ToolCallProgressSnapshot => ({ + invocationId: "call-1", runId: "run-1", toolName: "kv_demo__slow_with_progress", + startedAt: "2026-11-13T10:00:00.000Z", updatedAt: "2026-11-13T10:00:05.000Z", + progress: 2, total: 5, message: "Exporting rows", done: false, ...fields, +}); +const tool = (fields: Partial = {}): TaskChatToolItem => ({ + id: "toolu_1", kind: "tool", name: "slow_with_progress", rawName: "mcp__paperclip__kv_demo__slow_with_progress", + status: "in_progress", ...fields, +}); + +describe("tool call progress matching", () => { + it("matches the gateway tool name inside the agent's tool name, and nothing else", () => { + expect(toolCallNameMatches("mcp__paperclip__kv_demo__slow_with_progress", "kv_demo__slow_with_progress")).toBe(true); + expect(toolCallNameMatches("kv_demo__slow_with_progress", "kv_demo__slow_with_progress")).toBe(true); + expect(toolCallNameMatches("mcp__paperclip__kv_demo__slow_with_progressive", "kv_demo__slow_with_progress")).toBe(false); + expect(toolCallNameMatches("mcp__paperclip__other_slow_with_progress", "kv_demo__slow_with_progress")).toBe(false); + expect(toolCallNameMatches(undefined, "x")).toBe(false); + // Gateway names carry "." and ":", which agents may rewrite. + expect(toolCallNameMatches("mcp__paperclip__mcp_kv-demo-1a2b3c4d_slow-with-progress", "mcp.kv-demo-1a2b3c4d:slow-with-progress")).toBe(true); + expect(toolCallNameMatches("mcp__paperclip__mcp.kv-demo-1a2b3c4d:slow-with-progress", "mcp.kv-demo-1a2b3c4d:slow-with-progress")).toBe(true); + expect(toolCallNameMatches("mcp__paperclip__mcp.kv-demo-99999999:slow-with-progress", "mcp.kv-demo-1a2b3c4d:slow-with-progress")).toBe(false); + }); + + it("finds a native runner's row by the shortened, hashed name it gives the tool", () => { + const gatewayName = "mcp.kv-demo-1a2b3c4d:slow-with-progress"; + const nativeName = "app_mcp_kv_demo_1a2b3c4d_slow_with_progress_889fb9a8d98e"; + const entry = snapshot({ toolName: gatewayName, nativeToolName: nativeName }); + expect(pickToolCallProgress([entry], tool({ rawName: nativeName, name: nativeName }), "run-1")).toBe(entry); + expect(pickToolCallProgress([entry], tool({ rawName: `mcp__paperclip__${nativeName}` }), "run-1")).toBe(entry); + // Another tool's hash never matches. + expect(pickToolCallProgress([entry], tool({ rawName: "app_mcp_kv_demo_1a2b3c4d_slow_with_progress_000000000000" }), "run-1")).toBeNull(); + expect(readToolCallProgressEvent({ ...entry })?.nativeToolName).toBe(nativeName); + }); + + it("finds a wrapper call's row (OpenCode's paperclip_call_assigned_tool) by the tool it runs", () => { + const gatewayName = "mcp.kv-demo-1a2b3c4d:slow-with-progress"; + const nativeName = "app_mcp_kv_demo_1a2b3c4d_slow_with_progress_889fb9a8d98e"; + const items = transcriptToTaskChatItems([{ + kind: "tool_call", ts: "2026-11-13T10:00:00.000Z", toolUseId: "call-1", + name: "paperclip_call_assigned_tool", input: { name: nativeName, arguments: { steps: 10 } }, + }], { runId: "run-1", running: true }); + const row = items.find((item): item is TaskChatToolItem => item.kind === "tool")!; + expect(row.appToolName).toBe(nativeName); + const entry = snapshot({ toolName: gatewayName, nativeToolName: nativeName }); + expect(pickToolCallProgress([entry], row, "run-1")).toBe(entry); + // Any other call, or a wrapper without a name, carries none. + expect(wrappedAppToolName("Bash", { name: nativeName })).toBeUndefined(); + expect(wrappedAppToolName("mcp__paperclip__paperclip_call_assigned_tool", { name: nativeName })).toBe(nativeName); + expect(wrappedAppToolName("paperclip_call_assigned_tool", { arguments: {} })).toBeUndefined(); + }); + + it("picks the newest unfinished call of the row's tool in the row's run", () => { + const entries = [ + snapshot({ invocationId: "old", startedAt: "2026-11-13T09:00:00.000Z" }), + snapshot({ invocationId: "new", startedAt: "2026-11-13T10:00:00.000Z" }), + snapshot({ invocationId: "finished", startedAt: "2026-11-13T11:00:00.000Z", done: true }), + snapshot({ invocationId: "other-run", runId: "run-2", startedAt: "2026-11-13T12:00:00.000Z" }), + snapshot({ invocationId: "other-tool", toolName: "kv_demo__get", startedAt: "2026-11-13T13:00:00.000Z" }), + ]; + expect(pickToolCallProgress(entries, tool(), "run-1")?.invocationId).toBe("new"); + expect(pickToolCallProgress(entries, tool(), null)?.invocationId).toBe("other-run"); + expect(pickToolCallProgress(entries, tool({ rawName: "Bash", name: "Bash" }), "run-1")).toBeNull(); + }); + + it("keeps the newest state of each call, a finished call stays finished, and memory is bounded", () => { + const store = new ToolCallProgressStore(); + store.put(snapshot({ progress: 3, updatedAt: "2026-11-13T10:00:06.000Z" })); + store.put(snapshot({ progress: 2, updatedAt: "2026-11-13T10:00:05.000Z" })); + expect([...store.entries()].map((entry) => entry.progress)).toEqual([3]); + store.put(snapshot({ progress: 3, done: true, updatedAt: "2026-11-13T10:00:06.000Z" })); + store.put(snapshot({ progress: 4, updatedAt: "2026-11-13T10:00:09.000Z" })); + expect([...store.entries()]).toEqual([]); + for (let i = 0; i < 80; i++) store.put(snapshot({ invocationId: `call-${i + 2}` })); + expect([...store.entries()]).toHaveLength(50); + }); + + it("a fresh read ends calls that finished unseen, and never drops a call it does not list", () => { + const store = new ToolCallProgressStore(); + store.put(snapshot({ invocationId: "finished-while-hidden", updatedAt: "2026-11-13T10:00:05.000Z" })); + store.put(snapshot({ invocationId: "not-saved-yet", updatedAt: "2026-11-13T10:00:06.000Z" })); + // The read reports the finished call as done and does not list the other one yet. + for (const entry of [snapshot({ invocationId: "finished-while-hidden", done: true, updatedAt: "2026-11-13T10:00:09.000Z" })]) store.put(entry); + expect(pickToolCallProgress(store.entries(), tool(), "run-1")?.invocationId).toBe("not-saved-yet"); + expect([...store.entries()].map((entry) => entry.invocationId)).toEqual(["not-saved-yet"]); + }); + + it("keeps running calls when a read reports many finished ones", () => { + const store = new ToolCallProgressStore(); + for (let i = 0; i < 50; i++) store.put(snapshot({ invocationId: `running-${i}` })); + for (let i = 0; i < 50; i++) store.put(snapshot({ invocationId: `history-${i}`, done: true })); + expect([...store.entries()]).toHaveLength(50); + expect([...store.entries()].every((entry) => entry.invocationId.startsWith("running-") && !entry.done)).toBe(true); + }); + + it("a call's live end before a delayed read listing it as running keeps it ended", () => { + const store = new ToolCallProgressStore(); + // Opened near the end: the live end arrives first, then the read made just before it. + store.put(snapshot({ invocationId: "ending", done: true, updatedAt: "2026-11-13T10:00:10.000Z" })); + store.put(snapshot({ invocationId: "ending", progress: 9, total: 10, updatedAt: "2026-11-13T10:00:09.000Z" })); + expect([...store.entries()]).toEqual([]); + expect(pickToolCallProgress(store.entries(), tool(), "run-1")).toBeNull(); + }); + + it("reads live events defensively", () => { + expect(readToolCallProgressEvent({ ...snapshot(), message: "x".repeat(500) })?.message).toHaveLength(200); + expect(readToolCallProgressEvent({ ...snapshot(), runId: undefined })).toBeNull(); + expect(readToolCallProgressEvent({ ...snapshot(), progress: "2" })).toBeNull(); + expect(readToolCallProgressEvent({ ...snapshot(), total: -1 })?.total).toBeNull(); + }); +}); + +describe("tool row progress states", () => { + let container: HTMLDivElement; + let root: Root; + let queryClient: QueryClient; + beforeEach(() => { + container = document.createElement("div"); + document.body.appendChild(container); + root = createRoot(container); + queryClient = new QueryClient({ defaultOptions: { queries: { retry: false } } }); + savedProgress.list.mockReset(); + savedProgress.list.mockResolvedValue([]); + }); + afterEach(() => { + act(() => root.unmount()); + container.remove(); + }); + const render = (node: React.ReactNode) => act(() => root.render( + + {node} + , + )); + const settle = () => act(async () => { for (let i = 0; i < 5; i++) await Promise.resolve(); }); + + it("running with a total: a filling bar, the message, percent and elapsed time", () => { + act(() => root.render()); + const bar = container.querySelector('[role="progressbar"]')!; + expect(bar.getAttribute("aria-valuenow")).toBe("40"); + expect(container.querySelector('[data-testid="task-chat-tool-progress-fill"]')!.style.width).toBe("40%"); + expect(container.textContent).toContain("Exporting rows"); + expect(container.textContent).toContain("40%"); + expect(container.querySelector('[data-testid="task-chat-tool-progress-elapsed"]')!.textContent).toBe("1m 5s"); + }); + + it("running without a total: an indeterminate sweep and no percent", () => { + act(() => root.render()); + expect(container.querySelector('[data-testid="task-chat-tool-progress-indeterminate"]')).not.toBeNull(); + expect(container.querySelector('[role="progressbar"]')!.hasAttribute("aria-valuenow")).toBe(false); + expect(container.textContent).not.toContain("%"); + expect(container.querySelector('[data-testid="task-chat-tool-progress-message"]')).toBeNull(); + expect(container.textContent).toContain("12s"); + }); + + it("renders provider text as plain text, never markup", () => { + act(() => root.render( done' })} nowMs={Date.parse("2026-11-13T10:00:01.000Z")} />)); + expect(container.querySelector("img")).toBeNull(); + expect(container.textContent).toContain(' done'); + }); + + it("drops the progress when the call completes", () => { + render(); + expect(container.querySelector('[data-testid="task-chat-tool-progress"]')).not.toBeNull(); + render(); + expect(container.querySelector('[data-testid="task-chat-tool-progress"]')).toBeNull(); + }); + + it("restores the saved progress of a running call after a reload", async () => { + savedProgress.list.mockResolvedValue([snapshot({ progress: 3, message: "Writing file" })]); + render(); + await vi.waitFor(async () => { + await settle(); + expect(container.querySelector('[data-testid="task-chat-tool-progress-message"]')?.textContent).toBe("Writing file"); + }); + expect(savedProgress.list).toHaveBeenCalledWith("issue-1"); + }); + + it("reads fresh saved progress when someone leaves the task and comes back mid-call", async () => { + savedProgress.list.mockResolvedValue([snapshot({ progress: 1, message: "Step 1" })]); + render(); + await vi.waitFor(async () => { + await settle(); + expect(container.querySelector('[data-testid="task-chat-tool-progress-message"]')?.textContent).toBe("Step 1"); + }); + act(() => root.render(
)); + savedProgress.list.mockResolvedValue([snapshot({ progress: 4, message: "Step 4", updatedAt: "2026-11-13T10:00:09.000Z" })]); + render(); + await vi.waitFor(async () => { + await settle(); + expect(container.querySelector('[data-testid="task-chat-tool-progress-message"]')?.textContent).toBe("Step 4"); + }); + expect(savedProgress.list).toHaveBeenCalledTimes(2); + }); + + it("never asks for progress for a finished row", async () => { + render(); + await settle(); + expect(savedProgress.list).not.toHaveBeenCalled(); + expect(container.querySelector('[data-testid="task-chat-tool-progress"]')).toBeNull(); + }); +}); diff --git a/ui/src/components/task-chat/tool-call-progress.tsx b/ui/src/components/task-chat/tool-call-progress.tsx new file mode 100644 index 00000000000..0d71a7d3112 --- /dev/null +++ b/ui/src/components/task-chat/tool-call-progress.tsx @@ -0,0 +1,221 @@ +import { createContext, useContext, useEffect, useMemo, useState, useSyncExternalStore, type ReactNode } from "react"; +import { useQuery } from "@tanstack/react-query"; +import { + boundToolCallProgressMessage, + normalizeToolCallProgress, + toolCallProgressFraction, + type ToolCallProgressSnapshot, +} from "@paperclipai/shared"; +import { mcpAppsApi } from "@/api/mcpApps"; +import { useSecondTick } from "@/hooks/useSecondTick"; +import { useCompanyLiveEvent } from "@/context/LiveUpdatesProvider"; +import { queryKeys } from "@/lib/queryKeys"; +import { cn, formatDurationMs } from "@/lib/utils"; +import type { TaskChatToolItem } from "./task-chat-model"; + +/** + * Live progress for a running connected-app tool call. The provider streams + * MCP progress to Paperclip, which publishes it on the run's live events and + * saves the latest on the call (read once, for a reload mid-call). + */ + +/** The run whose rows are showing, when known; progress from other runs is ignored. */ +const ToolCallProgressRunContext = createContext(null); + +export function ToolCallProgressRunProvider({ runId, children }: { runId: string | null | undefined; children: ReactNode }) { + return {children}; +} + +/** Lowercase words joined by "_", so "mcp.kv-demo-1a:slow" and "mcp_kv_demo_1a_slow" compare equal. */ +function comparableToolName(name: string) { + return name.toLowerCase().replace(/[^a-z0-9]+/g, "_").replace(/^_+|_+$/g, ""); +} + +/** + * Whether a transcript row's tool name ("mcp__paperclip__") is + * the gateway tool a call reported. Agents may rewrite characters their tool + * names cannot hold, so names are compared word by word, as a whole suffix. + */ +export function toolCallNameMatches(rawName: string | undefined, toolName: string): boolean { + if (!rawName || !toolName) return false; + const row = comparableToolName(rawName); + const tool = comparableToolName(toolName); + return Boolean(tool) && (row === tool || row.endsWith(`_${tool}`)); +} + +/** A live event payload as a snapshot, or null when it is not one. Values are checked again here. */ +export function readToolCallProgressEvent(payload: Record): ToolCallProgressSnapshot | null { + const invocationId = typeof payload.invocationId === "string" ? payload.invocationId : null; + const runId = typeof payload.runId === "string" ? payload.runId : null; + const toolName = typeof payload.toolName === "string" ? payload.toolName : null; + const nativeToolName = typeof payload.nativeToolName === "string" ? payload.nativeToolName : null; + const startedAt = typeof payload.startedAt === "string" && Number.isFinite(Date.parse(payload.startedAt)) ? payload.startedAt : null; + if (!invocationId || !runId || !toolName || !startedAt) return null; + const updatedAt = typeof payload.updatedAt === "string" && Number.isFinite(Date.parse(payload.updatedAt)) ? new Date(payload.updatedAt) : new Date(); + const progress = normalizeToolCallProgress(payload, updatedAt); + if (!progress) return null; + return { ...progress, invocationId, runId, toolName, nativeToolName, startedAt, done: payload.done === true }; +} + +/** Whether a row is a call of this tool, under its gateway name or the native runner's name for it. */ +export function rowShowsTool(item: Pick, entry: Pick): boolean { + // A wrapper call (OpenCode's paperclip_call_assigned_tool) is matched by the tool it runs. + return [item.appToolName, item.rawName ?? item.name].some((rowName) => Boolean(rowName) && ( + toolCallNameMatches(rowName, entry.toolName) + || (Boolean(entry.nativeToolName) && toolCallNameMatches(rowName, entry.nativeToolName!)))); +} + +/** The progress for a row: the latest unfinished call of the same tool, in the same run when known. */ +export function pickToolCallProgress( + entries: Iterable, + item: Pick, + runId: string | null, +): ToolCallProgressSnapshot | null { + let best: ToolCallProgressSnapshot | null = null; + for (const entry of entries) { + if (entry.done || (runId && entry.runId !== runId)) continue; + if (!rowShowsTool(item, entry)) continue; + if (!best || entry.startedAt > best.startedAt) best = entry; + } + return best; +} + +/** Running calls kept per chat; the oldest are forgotten first. */ +const MAX_TRACKED_CALLS = 50; +/** Finished call ids remembered per chat, so a delayed read cannot revive one. */ +const MAX_REMEMBERED_FINISHED = 500; + +/** + * The chat's known tool call progress. Rows subscribe only while running, so a + * burst of updates re-renders the running row, never the whole chat. + */ +export class ToolCallProgressStore { + #entries = new Map(); + #finished = new Set(); + #listeners = new Set<() => void>(); + #onWanted: () => void; + constructor(onWanted: () => void = () => undefined) { this.#onWanted = onWanted; } + readonly subscribe = (listener: () => void) => { + this.#listeners.add(listener); + return () => { this.#listeners.delete(listener); }; + }; + /** A running row asks once for the saved progress (a reload mid-call). */ + want() { this.#onWanted(); } + entries() { return this.#entries.values(); } + /** + * Keeps the newer update of each running call. A finished call is removed + * and remembered (bounded, apart from shown progress), so a later "running" + * snapshot of it, from a delayed read, never brings its progress back. + */ + put(snapshot: ToolCallProgressSnapshot) { + const id = snapshot.invocationId; + if (this.#finished.has(id)) return; + if (snapshot.done) { + this.#finished.add(id); + while (this.#finished.size > MAX_REMEMBERED_FINISHED) this.#finished.delete(this.#finished.values().next().value!); + if (!this.#entries.delete(id)) return; + } else { + const previous = this.#entries.get(id); + if (previous && previous.updatedAt > snapshot.updatedAt) return; + this.#entries.delete(id); + this.#entries.set(id, snapshot); + while (this.#entries.size > MAX_TRACKED_CALLS) this.#entries.delete(this.#entries.keys().next().value!); + } + for (const listener of this.#listeners) listener(); + } +} + +const ToolCallProgressStoreContext = createContext(null); + +/** + * One per task chat: listens for the task's tool call progress on the shared + * live socket, and reads the saved progress once, only when a call is running. + */ +export function ToolCallProgressProvider({ issueId, children }: { issueId: string | null | undefined; children: ReactNode }) { + const [wantedFor, setWantedFor] = useState(null); + // A new store (and a fresh read) per task and per mount: updates sent while the + // chat was closed are only in the saved snapshot. + const store = useMemo(() => new ToolCallProgressStore(() => setWantedFor(issueId ?? null)), [issueId]); + const saved = useQuery({ + queryKey: queryKeys.toolCallProgress(issueId ?? ""), + queryFn: () => mcpAppsApi.toolCallProgress(issueId!), + enabled: Boolean(issueId) && wantedFor === issueId, + // Read again whenever a chat with a running call mounts; live events cover the rest. + staleTime: 0, + refetchOnMount: "always", + // Live events are not handled while the tab is hidden; read again on return. + refetchOnWindowFocus: true, + }); + useEffect(() => { + // Finished calls arrive as done; a call missing from the read is never dropped, + // since a running call may not have saved its first update yet. + for (const entry of saved.data ?? []) store.put(entry); + }, [saved.data, store]); + useCompanyLiveEvent((event) => { + if (event.type !== "heartbeat.run.tool_progress" || !issueId || event.payload?.issueId !== issueId) return; + const snapshot = readToolCallProgressEvent(event.payload); + if (snapshot) store.put(snapshot); + }); + return {children}; +} + +const noSubscription = () => () => undefined; + +/** Progress for a running tool row; null for any other row, which never re-renders for progress. */ +export function useToolCallProgress(item: TaskChatToolItem): ToolCallProgressSnapshot | null { + const store = useContext(ToolCallProgressStoreContext); + const runId = useContext(ToolCallProgressRunContext); + const running = item.status === "in_progress" && store !== null; + useEffect(() => { if (running) store!.want(); }, [running, store]); + return useSyncExternalStore( + running ? store!.subscribe : noSubscription, + () => (running ? pickToolCallProgress(store!.entries(), item, runId) : null), + ); +} + +/** + * The running row's progress: a bar that fills when the total is known (a + * quiet sweep otherwise), the provider's latest message as plain text, and + * elapsed time. + */ +export function TaskChatToolProgress({ progress, nowMs }: { progress: ToolCallProgressSnapshot; nowMs?: number }) { + // The shared one-second ticker, only while no fixed time is given. + useSecondTick(nowMs === undefined); + const now = nowMs ?? Date.now(); + const fraction = toolCallProgressFraction(progress); + const percent = fraction === null ? null : Math.round(fraction * 100); + // Rendered as text only: React escapes it, and it is bounded again here. + const message = boundToolCallProgressMessage(progress.message); + const elapsed = formatDurationMs(Math.max(0, now - Date.parse(progress.startedAt))); + return ( +
+
+
+
+
+ {message ? ( + {message} + ) : } + {percent !== null ? {percent}% : null} + {elapsed} +
+
+ ); +} diff --git a/ui/src/components/task-chat/transcript-adapter.ts b/ui/src/components/task-chat/transcript-adapter.ts index 964bfd91b65..65f29ebfc4f 100644 --- a/ui/src/components/task-chat/transcript-adapter.ts +++ b/ui/src/components/task-chat/transcript-adapter.ts @@ -197,6 +197,17 @@ function isOpaqueId(value: unknown): boolean { * The task a tool call is about, when its input names one by UUID. The row * shows that task's identifier (resolved at render time) instead of the id. */ +/** Runners without direct app tools call them through this wrapper, naming the tool in its input. */ +const WRAPPED_APP_TOOL = "paperclip_call_assigned_tool"; + +/** The app tool a wrapper call runs (its input's `name`), or undefined for any other call. */ +export function wrappedAppToolName(rawName: string | null | undefined, input: unknown): string | undefined { + if (!rawName || !(rawName === WRAPPED_APP_TOOL || rawName.endsWith(`_${WRAPPED_APP_TOOL}`) || rawName.endsWith(`.${WRAPPED_APP_TOOL}`))) return undefined; + if (input == null || typeof input !== "object" || Array.isArray(input)) return undefined; + const name = (input as Record).name; + return typeof name === "string" && name.trim() ? name.trim() : undefined; +} + export function toolInputTaskRef(input: unknown): string | undefined { if (input == null || typeof input !== "object" || Array.isArray(input)) return undefined; const record = input as Record; @@ -908,6 +919,7 @@ export function transcriptToTaskChatItems( } } if (entry.detail) applyToolDetail(existing, entry.detail); + existing.appToolName ??= wrappedAppToolName(existing.rawName, entry.input); lastToolIndex = existingIndex!; } else { const tool: TaskChatToolItem = { @@ -917,6 +929,7 @@ export function transcriptToTaskChatItems( rawName: entry.name ?? undefined, target: summarizeToolInput(entry.input), ...(toolInputTaskRef(entry.input) ? { taskRefId: toolInputTaskRef(entry.input) } : {}), + ...(wrappedAppToolName(entry.name, entry.input) ? { appToolName: wrappedAppToolName(entry.name, entry.input) } : {}), status: "in_progress", ...(entry.parentToolUseId ? { parentToolUseId: entry.parentToolUseId } : {}), }; diff --git a/ui/src/context/LiveUpdatesProvider.tsx b/ui/src/context/LiveUpdatesProvider.tsx index da10b1800a3..da33c16bbe1 100644 --- a/ui/src/context/LiveUpdatesProvider.tsx +++ b/ui/src/context/LiveUpdatesProvider.tsx @@ -2155,6 +2155,9 @@ function handleLiveEvent( const nameOf = (id: string) => resolveAgentName(queryClient, expectedCompanyId, id); + // Tool call progress is read by the tool row itself (useToolCallProgress); it + // changes no cached run state and must not invalidate queries several times a second. + if (event.type === "heartbeat.run.tool_progress") return; const payload = event.payload ?? {}; // Resolve membership before terminal lifecycle patches remove live-run rows. const suppressRunToast = diff --git a/ui/src/index.css b/ui/src/index.css index 425c885169e..c9839ffea65 100644 --- a/ui/src/index.css +++ b/ui/src/index.css @@ -3773,6 +3773,19 @@ span.paperclip-mention-chip[data-mention-kind="external-object"] { .skill-import-progress::-webkit-progress-value { transition: none; } } +/* Connected-app tool call progress: a filling bar when the total is known, a quiet sweep otherwise. */ +.tool-call-progress-fill { + transition: width var(--motion-line-scroll) var(--motion-ease-standard); +} +.tool-call-progress-sweep { + width: 40%; + animation: skill-import-sweep var(--motion-approval-pulse) var(--motion-ease-standard) infinite; +} +@media (prefers-reduced-motion: reduce) { + .tool-call-progress-fill { transition: none; } + .tool-call-progress-sweep { animation: none; } +} + /* Agent setup handoff: the same logo stack travels from trigger to prompt. */ :root { --agent-setup-flight-duration: var(--motion-duration-deliberate); diff --git a/ui/src/lib/queryKeys.ts b/ui/src/lib/queryKeys.ts index 549457d5292..1bae37d0722 100644 --- a/ui/src/lib/queryKeys.ts +++ b/ui/src/lib/queryKeys.ts @@ -318,6 +318,7 @@ export const queryKeys = { views: (issueId: string) => ["mcp-app-views", issueId] as const, detail: (issueId: string, viewId: string) => ["mcp-app-views", issueId, "detail", viewId] as const, }, + toolCallProgress: (issueId: string) => ["tool-call-progress", issueId] as const, issues: { recoveryActions: (id: string, actionId: string | null, runId: string | null) => ["issues", id, "recovery-actions", actionId, runId] as const, diff --git a/ui/storybook/stories/tasks/task-chat-tool-progress.stories.tsx b/ui/storybook/stories/tasks/task-chat-tool-progress.stories.tsx new file mode 100644 index 00000000000..009cb8fa3ce --- /dev/null +++ b/ui/storybook/stories/tasks/task-chat-tool-progress.stories.tsx @@ -0,0 +1,66 @@ +import type { Meta, StoryObj } from "@storybook/react-vite"; +import type { ToolCallProgressSnapshot } from "@paperclipai/shared"; +import { TaskChatToolCard } from "@/components/task-chat/TaskChatToolCard"; +import type { TaskChatToolItem } from "@/components/task-chat/task-chat-model"; + +type Props = { state: "total" | "no-total" | "done" }; + +const TOOL: TaskChatToolItem = { + id: "toolu_progress", + kind: "tool", + name: "slow_with_progress", + rawName: "mcp__paperclip__kv_demo__slow_with_progress", + target: "steps: 10", + status: "in_progress", +}; + +function progressFor(state: Props["state"]): ToolCallProgressSnapshot { + const now = Date.now(); + return { + invocationId: "call-1", + runId: "run-1", + toolName: "kv_demo__slow_with_progress", + startedAt: new Date(now - 65_000).toISOString(), + updatedAt: new Date(now - 400).toISOString(), + progress: 4, + total: state === "total" ? 10 : null, + message: state === "total" ? "Exporting rows 4,000 of 10,000" : "Indexing documents", + done: false, + }; +} + +function ToolProgressPreview({ state }: Props) { + return ( +
+ +
+ ); +} + +const meta = { + id: "tasks-task-chat-tool-progress", + title: "Tasks and conversations/Task creation and lifecycle/Tasks/Tool progress", + component: ToolProgressPreview, + parameters: { + layout: "fullscreen", + docs: { + description: { + component: + "A long connected-app tool call shows the provider's progress while it runs: a filling bar when the total is known, a quiet sweep otherwise, the latest status line as plain text, and elapsed time. The row drops it when the call completes.", + }, + }, + paperclip: { + capabilities: ["tasks/discussion", "runs-adapters/runs"], + status: "current", + implementation: "production", + boundary: "fixtures", + notes: "Production tool row with fixed progress. Live progress arrives from MCP notifications/progress through the run's live events.", + }, + }, +} satisfies Meta; +export default meta; +type Story = StoryObj; + +export const RunningWithTotal: Story = { name: "01 · Running, total known", args: { state: "total" } }; +export const RunningWithoutTotal: Story = { name: "02 · Running, total unknown", args: { state: "no-total" } }; +export const Done: Story = { name: "03 · Done", args: { state: "done" } }; From a82e23582831859ac9f2088d65519a8fb7b8b664 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 20:21:35 -0600 Subject: [PATCH 02/10] fix(heartbeat): keep recovery behind people's work, guarantee a start to tasks people wait on, and admit recovery a few runs at a time (#21427) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat queue starts an agent's queued runs when a slot is free. #21234, #21277, #21349 and #21391 order them: person work first, a recovery share, aging, and a share for the longest-waiting run > - On hosted staging, one agent with 20 slots had 295 queued runs. 139 of them were system recovery wakes that entered the queue within one hour. Runs that people asked for waited 7 hours > - Two rules let this happen. Recovery took every fourth start even when a person's run waited, and an aged recovery run moved into the agent-work tier ahead of todo tasks. The continuation producer also admitted any continuation held past 5 minutes, however deep the queue was > - This pull request keeps recovery behind people's work, gives a person's long-waiting run a bounded start, re-checks a recovery run before it starts, and admits recovery to an agent's queue a few runs at a time > - The benefit is that a task a person asked for starts within a bounded number of dispatches, and a recovery burst cannot flood an agent's queue ## Linked Issues or Issue Description Refs #21234, #21277, #21349, #21391, #21405, #21433, #21502 **What happened?** On hosted staging (2026-11-13), one agent had `maxConcurrentRuns` 20 and all 20 slots held hour-long runs. 295 runs were queued. 139 had wake reason `issue_recovery_action_restored` (source `execution.reconciled`), enqueued between about 10:46Z and 11:47Z. Two tasks that people created were stalled by a restart at 07:25Z. Their next runs were queued at 07:29Z and were still queued 7 hours later. Both were system wakes: one `issue_monitor_due`, one `issue_continuation_needed` from the assigned-work sweep. Recent starts had waited from 1 to 290 minutes. **Why did it happen?** 1. The recovery share gave every fourth start to system recovery, also while a person's run waited. 2. A recovery run that waited one hour aged into the agent-work tier. Recovery tasks are usually in progress, so they sorted ahead of todo tasks in that tier. 3. The oldest-run share served the oldest aged run of any class. Older agent work and older recovery came before a person's run. 4. Only a wake that a person requested counted as person work. A system wake on a task that a person created and waits on (a monitor check, a continuation after a restart) competed as ordinary work or recovery. 5. `deliverReconciledExecutions` admitted a continuation held past `RECOVERY_CONTINUATION_MAX_HOLD_MS` (5 minutes) whatever the queue depth. At one per agent per 15-second pass, one agent got about 139 continuations in an hour. **Expected behavior** A run that a person created or asked for never waits behind system recovery. Waiting never fails or cancels it. A long wait moves it forward, so it starts within a bounded number of dispatches. Recovery enters the queue a few runs at a time. A recovery run that is no longer needed is dropped quietly. **Steps to reproduce** 1. Give an agent 20 slots, all busy. 2. Queue 60 agent assignments, then two system wakes on tasks people created (a monitor check and a continuation), then 30 later person runs on in-progress tasks, then a burst of 139 recovery wakes. 3. Free one slot at a time while people and agents keep adding work. 4. See that the monitor check starts only at dispatch 244. ## What Changed - `server/src/services/heartbeat/queued-run-dispatch-order.ts`: - System recovery has its own tiers below agent work (fresh, then backlog). Aging moves old backlog only into the fresh recovery tier, never ahead of agent work. - The recovery share applies only when no ready person run waits. It comes out of agent work, never out of a person's turn. Within the share, the longest-waiting aged recovery run goes first. - The oldest-run share serves the longest-waiting aged run that a person asked for or whose task a person created (`issues.created_by_user_id`), first. This includes a system continuation of such a task. Then it serves the longest-waiting aged agent-work run. It never serves other recovery. - New dispatch fact `personTask`, read with the queued tasks' status and priority (same query). - `queuedRunDispatchLine` (the O(n log n) place-in-line order from #21405) follows the same rules. `queued-run-line.test.ts` still proves it equals the dispatcher's order over 400 random queues, now with `personTask` in them. - `server/src/services/heartbeat/queued-run-line.ts`: the start-history counters use `dispatchHistoryStart` and `dispatchCountersFromStarts`, so dispatch, the place-in-line view and the tests share one rule. - New pure helpers `oldestShareCandidate`, `dispatchHistoryStart` and `dispatchCountersFromStarts`. The queue and the tests use the same counter rule (which includes the #21391 rule that a person's recovery restart does not use up the recovery share). - `server/src/services/heartbeat/queue.ts`: - At claim, a system recovery run (`issue_recovery_action_restored`, `issue_continuation_needed`, not requested by a person) is re-checked. If the task is reassigned, or another run continues it (started after it was queued, running, a queued non-recovery run, or a newer queued recovery run), the run is cancelled before it starts with `recovery_no_longer_needed` and its wake is skipped. This is the same quiet path as the existing closed-task check: no failure, no provider work. A person's undelivered message on the run still keeps it. - `server/src/services/recovery-queue-admission.ts` (new): `QUEUED_RECOVERY_RUNS_PER_AGENT` = 3 and the queue-depth check. - `server/src/services/execution-recovery-resolution.ts`: a continuation is admitted only while its agent has fewer than 3 queued recovery runs, fresh or backlog. The 5-minute hold bound still orders held continuations, but no longer overrides the cap. The rest stays pending in its recovery action. The existing continuation pass (every 15 s while any continuation is pending) admits the next one after a queued recovery run starts or is dropped. No new polling. - `server/src/services/recovery/legacy-workspace-rehome.ts`: the pending-wake delivery for rehomed and released-writer tasks uses the same cap. A held wake waits 60 s for its next turn and does not count as a refusal. - The oldest-run share counts as served only by a start that people wait on while its candidate is one, so older agent work never uses up a person's turn (`servesOldestShare`, used by the dispatcher, the place-in-line order and the start history, which now records `personWaiting`). - `server/src/services/recovery/assigned-work-sweep.ts`: a rehome or released-writer wake held by the cap (`HELD_RECOVERY_WAKE_KEYS`) counts as the task's pending path while its assignee has queued recovery runs. - `server/src/services/recovery/assigned-work-sweep.ts` (with #21502's stalled-queued-run replacement): a start of work people wait on does not count as passing over a queued run that people do not wait on. Waiting behind people's work is intended, so the sweep does not cancel such a run and replace it with a recovery continuation. Test: `assigned-work-stalled-queued-run.test.ts`, "leaves an automatic run that waits behind work on tasks people created". It fails without the rule. - `server/src/services/recovery/assigned-work-sweep.ts`: a pending continuation still counts as the task's wake path past `PENDING_CONTINUATION_BOUND_MS` while its agent has queued recovery runs, so the sweep does not treat a held task as stranded. **Bounds** - A run a person asked for, or any run on a task a person created: once it has waited past `QUEUED_RUN_AGING_MS` (1 hour), it starts within `OLDEST_DISPATCH_SHARE` × (such aged runs queued before it + 1) dispatches. Before that, runs a person requested already go first. There is no time limit, and waiting never cancels it. - Recovery run, proposed bounded behavior: at most 3 queued per agent. When no person run waits, an aged recovery run starts within `RECOVERY_DISPATCH_SHARE` × (aged recovery runs before it + 1) dispatches. At start it is re-validated against the current task, and it is dropped quietly if stale. While people keep the agent busy, recovery waits. It is held durably in its recovery action, not lost and not failed. ## Verification Regression test that reproduces today's shape (`server/src/services/heartbeat/queued-run-dispatch-order.test.ts`, "deep per-agent queue: a recovery burst behind 20 busy slots"). It uses 20 busy slots, one dispatch per freed slot, 60 older agent assignments, the two 07:29 system wakes on tasks people created (monitor check and continuation), 30 later person runs, a 139-run recovery burst, and new person and assignment arrivals on every step. The same model, run against master's dispatcher and this one over 600 dispatches: | | Before (master) | After | | --- | --- | --- | | Dispatch at which the 07:29 monitor check starts | 244 (≈12 h at ~20 starts/h) | 4 | | Dispatch at which the 07:29 continuation starts | 5 | 8 | | Recovery starts while a person-requested run waited | 89 | 0 | With person-requested runs in the 07:29 slots instead (the first version of the test), master starts them at dispatch 244 and 248, and this change at 4 and 8. - Red on master: the new dispatch tests fail (`expected -1 to be greater than or equal to 0`, and recovery started ahead of agent assignments). Green with this change. - Admission cap, embedded Postgres (`recovery-wake-herd.test.ts`, "queues at most a few recovery runs per agent..."): with the cap removed, 12 of 12 continuations enter one agent's queue in 12 passes. With the cap, 3 enter, the rest stay pending, and all 12 are delivered as queued runs leave. The queue never holds more than 3 recovery runs. - Older agent work (`queued-run-dispatch-order.test.ts`): with 60 older, higher-priority assignments and no person arrivals, an aged `issue_monitor_due` run on a person-created task starts within `OLDEST_DISPATCH_SHARE` in dispatch and in the place-in-line order. With the old reset rule it does not start in 8 dispatches. - Held restored-task wake (`legacy-workspace-rehome.test.ts`): with the agent's recovery queue full, the wake stays pending with no failure counted, the sweep reads the task as `pending_intent`, and the wake is delivered after a queued recovery run leaves. - Start-time re-check, embedded Postgres, real heartbeat queue (`heartbeat-stale-queue-invalidation.test.ts`): a recovery run whose task was reassigned, or that a newer run already continued, is cancelled with `recovery_no_longer_needed`, its wake is `skipped`, and the adapter is never called. A recovery run that the task still needs starts. - Updated tests that encoded the old rules: recovery's share and aging now come out of agent work, not out of a person's turn. Sweep liveness while held is covered. - `pnpm --filter @paperclipai/server exec tsc --noEmit`: passes. ## Risks - Behavior change: a system run on a task a person created now takes the oldest-run share ahead of older agent work. On an agent whose queue is mostly such tasks, agent-made subtasks get that share less often. They still start in normal order. - Behavior change: while people keep an agent fully busy, system recovery for that agent waits. This is intended. Recovery is held durably and starts once no person run waits. - Behavior change: a queued system recovery run can be cancelled at start as no longer needed. It is cancelled before any provider work, with the same quiet outcome as a continuation whose task closed. - A continuation held behind the cap stays pending longer than 7 minutes. The sweep now treats it as live while its agent has queued recovery runs. If such queued runs can never start (for example dependency-blocked), the next continuations wait behind them. - One extra indexed read per recovery claim, on the existing `(company_id, context_snapshot->>'issueId', created_at)` index. No schema change. Mixed versions are safe: an old server keeps its old order until it is replaced. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edit). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../assigned-work-stalled-queued-run.test.ts | 36 +++- ...heartbeat-stale-queue-invalidation.test.ts | 44 ++++ .../__tests__/legacy-workspace-rehome.test.ts | 44 ++++ .../src/__tests__/recovery-wake-herd.test.ts | 91 ++++++++- .../services/execution-recovery-resolution.ts | 4 + server/src/services/heartbeat/queue.ts | 68 ++++++- .../queued-run-dispatch-order.test.ts | 162 +++++++++++---- .../heartbeat/queued-run-dispatch-order.ts | 191 +++++++++++++----- .../heartbeat/queued-run-line.test.ts | 1 + .../src/services/heartbeat/queued-run-line.ts | 92 +++++---- .../src/services/recovery-queue-admission.ts | 42 ++++ .../services/recovery/assigned-work-sweep.ts | 39 ++-- .../recovery/legacy-workspace-rehome.ts | 11 + .../recovery/stalled-queued-run-sql.ts | 46 +++++ 14 files changed, 714 insertions(+), 157 deletions(-) create mode 100644 server/src/services/recovery-queue-admission.ts create mode 100644 server/src/services/recovery/stalled-queued-run-sql.ts diff --git a/server/src/__tests__/assigned-work-stalled-queued-run.test.ts b/server/src/__tests__/assigned-work-stalled-queued-run.test.ts index 5b006a9c68b..794809a50a9 100644 --- a/server/src/__tests__/assigned-work-stalled-queued-run.test.ts +++ b/server/src/__tests__/assigned-work-stalled-queued-run.test.ts @@ -104,7 +104,7 @@ describe.skipIf(!support.supported)("a queued run the dispatcher keeps passing o const readRun = async (id: string) => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)))[0]!; const continuations = async () => db.select().from(activityLog).where(and(eq(activityLog.entityId, issue.id), eq(activityLog.action, ASSIGNED_WORK_CONTINUED_ACTION))); - return { companyId, agentId, issue, queuedRun, otherStarts, olderRun, queue, sweep, runsOfTask, readRun, continuations }; + return { companyId, agentId, issue, other, queuedRun, otherStarts, olderRun, queue, sweep, runsOfTask, readRun, continuations }; } async function replacementOf(f: Awaited>, stalledId: string) { @@ -191,6 +191,40 @@ describe.skipIf(!support.supported)("a queued run the dispatcher keeps passing o expect(await f.continuations()).toHaveLength(0); }, 60_000); + it("leaves an automatic run that waits behind work on tasks people created", async () => { + // Work people wait on goes first by design; a system run behind it is waiting its turn, not stalled. + const f = await seed(); + await db.update(issues).set({ createdByUserId: "person-1" }).where(eq(issues.id, f.other.id)); + const waiting = await f.queuedRun({ refused: false }); + await f.otherStarts(OLDEST_DISPATCH_SHARE + 3, 10 * MINUTE); + const sweep = f.sweep(); + expect(await sweep.checkIssue({ companyId: f.companyId, issueId: f.issue.id })) + .toMatchObject({ verdict: { kind: "ok", clause: "live_run", runId: waiting.id } }); + expect((await f.readRun(waiting.id)).status).toBe("queued"); + expect(await f.continuations()).toHaveLength(0); + }, 60_000); + + it("leaves an automatic run that waits behind continuations of a person's Retry", async () => { + // The dispatcher ranks a recovery run as person work when its source chain leads to a person's request. + const f = await seed(); + const waiting = await f.queuedRun({ refused: false }); + const personSourceAt = new Date(Date.now() - 30 * MINUTE); + const [personSource] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, invocationSource: "on_demand", + triggerDetail: "manual", status: "failed", createdAt: personSourceAt, startedAt: personSourceAt, finishedAt: personSourceAt, + contextSnapshot: { issueId: f.other.id, taskId: f.other.id, wakeReason: "task_retry" } }).returning(); + for (let index = 0; index < OLDEST_DISPATCH_SHARE + 3; index++) { + const startedAt = new Date(Date.now() - 10 * MINUTE + index * 1_000); + await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, invocationSource: "automation", triggerDetail: "system", + status: "succeeded", createdAt: new Date(startedAt.getTime() - 1_000), startedAt, finishedAt: new Date(startedAt.getTime() + 500), + contextSnapshot: { issueId: f.other.id, taskId: f.other.id, wakeReason: "issue_recovery_action_restored", previousRunId: personSource!.id } }); + } + const sweep = f.sweep(); + expect(await sweep.checkIssue({ companyId: f.companyId, issueId: f.issue.id })) + .toMatchObject({ verdict: { kind: "ok", clause: "live_run", runId: waiting.id } }); + expect((await f.readRun(waiting.id)).status).toBe("queued"); + expect(await f.continuations()).toHaveLength(0); + }, 60_000); + it("leaves a run that waits behind a full agent", async () => { const f = await seed(); const waiting = await f.queuedRun(); diff --git a/server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts b/server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts index e77682e455d..9cad1909412 100644 --- a/server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts +++ b/server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts @@ -1755,6 +1755,50 @@ describeEmbeddedPostgres("heartbeat stale queued-run invalidation", () => { }, { timeout: 10_000 }); }); + describe("a queued system recovery run re-checks that its task still needs it when its turn comes", () => { + async function queueRecovery(task: Partial) { + const { companyId, agentId } = await seedCompanyAndAgent(); + const [issue] = await db.insert(issues).values({ companyId, title: "Waited behind people's work", + status: "in_progress", assigneeAgentId: agentId, ...task }).returning(); + const queued = await seedQueuedRun({ companyId, agentId, issueId: issue.id, + wakeReason: "issue_recovery_action_restored", invocationSource: "automation" }); + return { companyId, agentId, issueId: issue.id, ...queued }; + } + async function expectDroppedQuietly(runId: string, wakeupRequestId: string) { + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + expect(run).toMatchObject({ status: "cancelled", errorCode: "recovery_no_longer_needed", + resultJson: expect.objectContaining({ executionRecovery: { kind: "bootstrap", providerWorkStarted: false } }) }); + expect(countExecuteCallsForRun(runId)).toBe(0); + expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wakeupRequestId)))[0]?.status) + .toBe("skipped"); + } + + it("drops it when the task was reassigned while it waited", async () => { + const { issueId, runId, wakeupRequestId } = await queueRecovery({}); + await db.update(issues).set({ assigneeAgentId: null, assigneeUserId: "responsible-user" }).where(eq(issues.id, issueId)); + await heartbeat.resumeQueuedRuns(); + await heartbeat.drainActiveRunExecutions(); + await expectDroppedQuietly(runId, wakeupRequestId); + }); + + it("drops it when a newer run already continued the task", async () => { + const { companyId, agentId, issueId, runId, wakeupRequestId } = await queueRecovery({}); + await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "assignment", triggerDetail: "system", + status: "succeeded", createdAt: new Date(Date.now() + 1_000), startedAt: new Date(Date.now() + 1_000), + finishedAt: new Date(Date.now() + 2_000), contextSnapshot: { issueId, wakeReason: "issue_commented" } }); + await heartbeat.resumeQueuedRuns(); + await heartbeat.drainActiveRunExecutions(); + await expectDroppedQuietly(runId, wakeupRequestId); + }); + + it("starts it when the task still needs it", async () => { + const { runId } = await queueRecovery({ status: "todo" }); + await heartbeat.resumeQueuedRuns(); + await heartbeat.drainActiveRunExecutions(); + expect(countExecuteCallsForRun(runId)).toBe(1); + }, 60_000); + }); + it("cancels queued max-turn continuations when another continuation owns the issue lock", async () => { const { companyId, agentId } = await seedCompanyAndAgent(); const issueId = randomUUID(); diff --git a/server/src/__tests__/legacy-workspace-rehome.test.ts b/server/src/__tests__/legacy-workspace-rehome.test.ts index d8025a9dbdc..10efdea679f 100644 --- a/server/src/__tests__/legacy-workspace-rehome.test.ts +++ b/server/src/__tests__/legacy-workspace-rehome.test.ts @@ -12,6 +12,9 @@ import { REMOTE_WORKSPACE_WRITER_PENDING_MESSAGE } from "../services/remote-work import { LEGACY_WORKSPACE_REHOME_OUTCOME, rehomeStrandedLegacyWorkspaceTasks } from "../services/recovery/legacy-workspace-rehome.js"; import { recoveryActionSweepCondition } from "../services/recovery/blocker-stall-wakeups.js"; import { recoveryService } from "../services/recovery/service.js"; +import { assignedWorkSweep } from "../services/recovery/assigned-work-sweep.js"; +import { evaluateAssignedWork } from "../services/recovery/assigned-work-invariant.js"; +import { QUEUED_RECOVERY_RUNS_PER_AGENT } from "../services/recovery-queue-admission.js"; import { recoveryPostgresSupport, startRecoveryPostgres } from "./helpers/recovery-postgres.js"; const describePostgres = recoveryPostgresSupport.supported ? describe : describe.skip; @@ -225,6 +228,47 @@ describePostgres("stranded subtasks bound to a parent's held workspace", () => { expect(last.nextWakeRetryAt).toBeNull(); }, 60_000); + it("holds a move's wake while its agent's recovery queue is full, and the assigned-work sweep sees it as pending", async () => { + const f = await seedSharedFamily(); + const hero = await f.strand(f.hero.id); + expect((await rehomeStrandedLegacyWorkspaceTasks(db, null, { companyId: f.companyId })).rehomed).toEqual([f.hero.id]); + const queuedIds: string[] = []; + for (let index = 0; index < QUEUED_RECOVERY_RUNS_PER_AGENT; index += 1) { + const [busy] = await db.insert(issues).values({ companyId: f.companyId, title: `Busy ${index}`, status: "in_progress", + assigneeAgentId: f.agentId }).returning(); + const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, issueId: busy!.id, status: "queued", + invocationSource: "automation", triggerDetail: "system", + contextSnapshot: { issueId: busy!.id, wakeReason: "issue_recovery_action_restored" } }).returning(); + queuedIds.push(run!.id); + } + const woken: string[] = []; + const wake = async (_agentId: string, options: { payload: Record }) => { + woken.push(String(options.payload.issueId)); return { id: randomUUID() }; + }; + const held = await rehomeStrandedLegacyWorkspaceTasks(db, wake as never, { companyId: f.companyId }); + expect(woken).toEqual([]); + const pending = (await action(hero.actionId)).evidence.legacyWorkspaceRehome as Record; + // Waiting its turn is not a refusal: no failure is counted. + expect(pending).toMatchObject({ wake: "pending" }); + expect(pending.wakeFailures).toBeUndefined(); + expect(held.nextWakeRetryAt).toBe(Number(pending.wakeRetryAt)); + + // The sweep reads the held wake as the task's pending path, so it queues no uncapped continuation. + const sweep = assignedWorkSweep(db, { enqueueWakeup: async () => null }); + const facts = (await sweep.loadFacts(f.companyId, [await issue(f.hero.id)])).get(f.hero.id)!; + expect(facts.pendingWakes).toBeGreaterThan(0); + expect(evaluateAssignedWork(facts)).toMatchObject({ kind: "ok", clause: "pending_intent" }); + + // A queued recovery run leaves the queue; the held wake is delivered on its next turn. + await db.update(heartbeatRuns).set({ status: "succeeded", startedAt: new Date(), finishedAt: new Date() }) + .where(eq(heartbeatRuns.id, queuedIds[0]!)); + await db.update(issueRecoveryActions).set({ evidence: { ...(await action(hero.actionId)).evidence, + legacyWorkspaceRehome: { ...pending, wakeRetryAt: Date.now() - 1 } } }).where(eq(issueRecoveryActions.id, hero.actionId)); + await rehomeStrandedLegacyWorkspaceTasks(db, wake as never, { companyId: f.companyId }); + expect(woken).toEqual([f.hero.id]); + expect((await action(hero.actionId)).evidence).toMatchObject({ legacyWorkspaceRehome: { wake: "delivered" } }); + }, 60_000); + it("pages past escalations the guards reject", async () => { const f = await seedSharedFamily(); // Thirty escalations that match the query but fail a guard (another setup error). diff --git a/server/src/__tests__/recovery-wake-herd.test.ts b/server/src/__tests__/recovery-wake-herd.test.ts index 2555d75ac59..a0f3f21f052 100644 --- a/server/src/__tests__/recovery-wake-herd.test.ts +++ b/server/src/__tests__/recovery-wake-herd.test.ts @@ -10,6 +10,7 @@ import { RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS, RECOVERY_CONTINUATION_DELIVERY_BOUND_MS, RECOVERY_CONTINUATION_MAX_HOLD_MS, deliverReconciledExecutions, } from "../services/execution-recovery-resolution.js"; +import { QUEUED_RECOVERY_RUNS_PER_AGENT } from "../services/recovery-queue-admission.js"; import { PENDING_CONTINUATION_BOUND_MS, assignedWorkSweep } from "../services/recovery/assigned-work-sweep.js"; import { evaluateAssignedWork } from "../services/recovery/assigned-work-invariant.js"; import { taskInterruptionPresentation } from "../services/task-interruption-presentation.js"; @@ -239,9 +240,12 @@ suite("recovery wake herd", () => { it("gives the longest-waiting queued run a free slot after three fresher starts passed it over", async () => { const agentId = await seedCompany(1); - const old = await queuedRecoveryRun(agentId, 150, 152); - const people: Array = []; - for (let index = 0; index < 3; index += 1) people.push(await queuedPersonRetry(agentId)); + // A person's run on a todo task, waiting 150 minutes behind fresher person runs on in-progress tasks. + const old = await queuedPersonRetry(agentId); + await db.update(issues).set({ status: "todo" }).where(eq(issues.id, old.issueId!)); + await db.update(heartbeatRuns).set({ createdAt: new Date(Date.now() - 150 * 60_000) }).where(eq(heartbeatRuns.id, old.id)); + const fresher: Array = []; + for (let index = 0; index < 3; index += 1) fresher.push(await queuedPersonRetry(agentId)); const issue = await task(agentId); for (let index = 0; index < 3; index += 1) { const startedAt = new Date(Date.now() - (3 - index) * 60_000); @@ -253,7 +257,29 @@ suite("recovery wake herd", () => { try { await heartbeat.resumeQueuedRuns(); expect((await readRun(old.id)).startedAt).not.toBeNull(); - for (const run of people) expect(await readRun(run.id)).toMatchObject({ status: "queued", startedAt: null }); + for (const run of fresher) expect(await readRun(run.id)).toMatchObject({ status: "queued", startedAt: null }); + } finally { + await heartbeat.drainActiveRunExecutions(); + } + }, 120_000); + + it("keeps a long-waiting recovery run behind people's queued work, even when the oldest-run share is due", async () => { + const agentId = await seedCompany(1); + const old = await queuedRecoveryRun(agentId, 150, 152); + const people: Array = []; + for (let index = 0; index < 3; index += 1) people.push(await queuedPersonRetry(agentId)); + const issue = await task(agentId); + for (let index = 0; index < 3; index += 1) { + const startedAt = new Date(Date.now() - (3 - index) * 60_000); + await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: issue.id, invocationSource: "on_demand", + triggerDetail: "manual", status: "succeeded", createdAt: new Date(startedAt.getTime() - 1_000), startedAt, + finishedAt: new Date(startedAt.getTime() + 1_000), contextSnapshot: { issueId: issue.id, wakeReason: "issue_commented" } }); + } + const heartbeat = heartbeatService(db); + try { + await heartbeat.resumeQueuedRuns(); + expect(await readRun(old.id)).toMatchObject({ status: "queued", startedAt: null }); + expect((await Promise.all(people.map(run => readRun(run.id)))).filter(run => run.startedAt)).toHaveLength(1); } finally { await heartbeat.drainActiveRunExecutions(); } @@ -285,6 +311,11 @@ suite("recovery wake herd", () => { } const deliveryOf = async (id: string) => (await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, id)))[0]!.evidence.continuationDelivery; + // The agent starts its queued recovery runs, which frees their turns in the queue. + const startQueuedRecovery = async (agentId: string) => db.update(heartbeatRuns) + .set({ status: "succeeded", startedAt: new Date(), finishedAt: new Date() }) + .where(and(eq(heartbeatRuns.agentId, agentId), eq(heartbeatRuns.status, "queued"), + sql`${heartbeatRuns.contextSnapshot}->>'wakeReason' in ('issue_recovery_action_restored', 'issue_continuation_needed')`)); const finishQueued = async (agentId: string) => db.update(heartbeatRuns).set({ status: "succeeded", startedAt: new Date(), finishedAt: new Date() }) .where(and(eq(heartbeatRuns.agentId, agentId), eq(heartbeatRuns.status, "queued"))); @@ -379,6 +410,7 @@ suite("recovery wake herd", () => { await pendingContinuation(agentId, 1); await deliverReconciledExecutions(db, wake as never); if (await deliveryOf(old.action.id) === "delivered") admittedAt = pass; + await startQueuedRecovery(agentId); } expect(admittedAt).toBeGreaterThan(0); expect(admittedAt).toBeLessThanOrEqual(bound); @@ -576,6 +608,10 @@ suite("recovery wake herd", () => { const recordedAt = new Date(Date.now() - PENDING_CONTINUATION_BOUND_MS - 60_000).toISOString(); await db.update(issueRecoveryActions).set({ resolvedAt: new Date(recordedAt), evidence: sql`jsonb_set(${issueRecoveryActions.evidence}, '{freshContinuation,recordedAt}', to_jsonb(${recordedAt}::text))` }).where(eq(issueRecoveryActions.id, held.action.id)); + // Past the bound it is still waiting its turn behind the agent's queued recovery run. + const waiting = (await sweep.loadFacts(companyId, [issueRow!])).get(held.issue.id)!; + expect(waiting.pendingWakes).toBe(facts.pendingWakes); + await startQueuedRecovery(agentId); const stale = (await sweep.loadFacts(companyId, [issueRow!])).get(held.issue.id)!; expect(stale.pendingWakes).toBe(facts.pendingWakes - 1); }, 60_000); @@ -627,7 +663,10 @@ suite("recovery wake herd", () => { const wake = fakeWake(); await deliverReconciledExecutions(db, wake as never); expect(deliveredIds(wake)).toEqual([fresh.action.id, backlog[0]!.action.id]); - for (let pass = 0; pass < backlog.length; pass += 1) await deliverReconciledExecutions(db, wake as never); + for (let pass = 0; pass < backlog.length; pass += 1) { + await startQueuedRecovery(agentId); + await deliverReconciledExecutions(db, wake as never); + } expect(await Promise.all(backlog.map(record => deliveryOf(record.action.id)))).toEqual(backlog.map(() => "delivered")); expect(wake).toHaveBeenCalledTimes(backlog.length + 1); }, 60_000); @@ -641,8 +680,38 @@ suite("recovery wake herd", () => { for (let pass = 0; pass < records.length; pass += 1) { await deliverReconciledExecutions(db, wake as never); expect(wake).toHaveBeenCalledTimes(pass + 1); + await startQueuedRecovery(agentId); } expect(deliveredIds(wake)).toEqual(records.map(record => record.action.id)); + }, 60_000); + + it("queues at most a few recovery runs per agent, holds the rest, and admits the next when one leaves the queue", async () => { + // paperclip.staging, 2026-11-13: 139 continuations for one agent entered its queue within an hour, + // because a continuation held past the hold bound was admitted however deep the queue was. + const agentId = await seedCompany(20); + await queueOrdinaryWork(agentId, 25); + const records: Array>> = []; + for (let index = 0; index < 12; index += 1) { + const record = await pendingContinuation(agentId, index % 2 ? 2 : 200); + await holdPastBound(record, 60 * 60_000); + records.push(record); + } + const wake = fakeWake(); + for (let pass = 0; pass < 12; pass += 1) await deliverReconciledExecutions(db, wake as never); + expect(wake).toHaveBeenCalledTimes(QUEUED_RECOVERY_RUNS_PER_AGENT); + expect((await Promise.all(records.map(record => deliveryOf(record.action.id)))).filter(state => state === "pending")) + .toHaveLength(records.length - QUEUED_RECOVERY_RUNS_PER_AGENT); + expect(await executionControlSchedule(db).continuation()).not.toBeNull(); + + for (let pass = 0; pass < 2 * records.length && wake.mock.calls.length < records.length; pass += 1) { + await startQueuedRecovery(agentId); + await deliverReconciledExecutions(db, wake as never); + const [queued] = await db.select({ count: sql`count(*)::int` }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.agentId, agentId), eq(heartbeatRuns.status, "queued"), + sql`${heartbeatRuns.contextSnapshot}->>'wakeReason' = 'issue_recovery_action_restored'`)); + expect(Number(queued!.count)).toBeLessThanOrEqual(QUEUED_RECOVERY_RUNS_PER_AGENT); + } + expect(await Promise.all(records.map(record => deliveryOf(record.action.id)))).toEqual(records.map(() => "delivered")); await deliverReconciledExecutions(db, wake as never); expect(wake).toHaveBeenCalledTimes(records.length); }, 60_000); @@ -882,12 +951,20 @@ suite("recovery wake herd", () => { contextSnapshot: { issueId: issue.id, taskId: issue.id, wakeReason: "issue_recovery_action_restored" } }).returning(); await db.update(agentWakeupRequests).set({ runId: run!.id }).where(eq(agentWakeupRequests.id, request!.id)); } - const people = [await queuedPersonRetry(agentId), await queuedPersonRetry(agentId)]; + // Recovery's share comes out of agent work; with a person's run waiting it would wait too. + const work: Array = []; + for (let index = 0; index < 2; index += 1) { + const busy = await task(agentId); + const [row] = await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: busy.id, invocationSource: "assignment", + triggerDetail: "system", status: "queued", contextSnapshot: { issueId: busy.id, taskId: busy.id, wakeReason: "issue_assigned" } }) + .returning(); + work.push(row!); + } const heartbeat = heartbeatService(db); try { await heartbeat.resumeQueuedRuns(); expect((await readRun(system.id)).startedAt).not.toBeNull(); - for (const run of people) expect(await readRun(run.id)).toMatchObject({ status: "queued", startedAt: null }); + for (const run of work) expect(await readRun(run.id)).toMatchObject({ status: "queued", startedAt: null }); } finally { await heartbeat.drainActiveRunExecutions(); } diff --git a/server/src/services/execution-recovery-resolution.ts b/server/src/services/execution-recovery-resolution.ts index acc9a9132b7..d87006be042 100644 --- a/server/src/services/execution-recovery-resolution.ts +++ b/server/src/services/execution-recovery-resolution.ts @@ -43,6 +43,7 @@ import { continueFreshInTransaction, continueHeldExecutions, freshContinuationCh settleCoalescedFreshContinuationHolds } from "./fresh-continuation-fallback.js"; import { alias } from "drizzle-orm/pg-core"; import { RECOVERY_BACKLOG_AGE_MS, RECOVERY_WAKE_REASONS } from "./heartbeat/queued-run-dispatch-order.js"; +import { agentRecoveryQueueIsFull } from "./recovery-queue-admission.js"; import { parseHeartbeatPolicy } from "./heartbeat/timer-policy.js"; import { RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS, RECOVERY_CONTINUATION_BACKLOG_SHARE_PER_PASS, RECOVERY_CONTINUATION_MAX_HOLD_MS, @@ -461,6 +462,9 @@ export async function deliverReconciledExecutions( )); const agentHasRoom = async (action: typeof issueRecoveryActions.$inferSelect, agentId: string, kind: ContinuationClass) => { if ((admittedByCompany.get(action.companyId) ?? 0) >= companyLimit) return false; + // A few at a time per agent, fresh or not: the rest waits here, durably, + // until a queued recovery run starts or is dropped. + if (await agentRecoveryQueueIsFull(db, action.companyId, agentId)) return false; if (kind === "fresh") return true; const [agent] = await db.select().from(agents).where(and(eq(agents.companyId, action.companyId), eq(agents.id, agentId))); if (!agent) return true; diff --git a/server/src/services/heartbeat/queue.ts b/server/src/services/heartbeat/queue.ts index ed49232fc19..f382ea499ba 100644 --- a/server/src/services/heartbeat/queue.ts +++ b/server/src/services/heartbeat/queue.ts @@ -12,7 +12,7 @@ import { hasAutomaticInteractionDeliveryHold, INTERACTION_DELIVERY_DENIED, reche import { deferredWorkIntentWait } from "../work-intent-outbox.js"; import { notifyIssueRecovery } from "../issue-recovery-notifications.js"; import { createHeartbeatTimerPolicy } from "./timer-policy.js"; -import { QUEUED_RUN_AGING_MS, createQueuedRunDispatcher, hasQueuedRunAged, queuedRunDispatchClass } from "./queued-run-dispatch-order.js"; +import { QUEUED_RUN_AGING_MS, RECOVERY_WAKE_REASONS, createQueuedRunDispatcher, hasQueuedRunAged, queuedRunDispatchClass } from "./queued-run-dispatch-order.js"; import { queuedRunDispatchFacts } from "./queued-run-line.js"; import { notifyExecutionRunWork } from "../execution-control-notifications.js"; import { PersistedExecutionWaitRefusal, writeDeferredExecutionWait } from "../deferred-execution-wait.js"; @@ -908,6 +908,47 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende if (claimedRun && isRunAdmissionClosed()) throw new RunAdmissionClosedDuringClaim(); } + const CLOSED_TASK_CONTINUATION = { error: "The task was closed before this continuation started.", + errorCode: "issue_closed_before_start" }; + const RECOVERY_NO_LONGER_NEEDED = { error: "The task no longer needed this recovery run when its turn came.", + errorCode: "recovery_no_longer_needed" }; + + /** + * A system recovery run waits behind people's work, so by its turn the task + * may have moved on. It is still needed only while its agent still owns the + * task and no other run continues it: none started after it was queued, + * none running, and no queued non-recovery run (a person's or an + * assignment's). Of two queued recovery runs for one task, the newer one stays. + */ + async function recoveryRunNoLongerNeeded(run: typeof heartbeatRuns.$inferSelect, issueId: string, + task: { assigneeAgentId: string | null }) { + if (run.wakeupRequestId) { + const [request] = await db.select({ actor: agentWakeupRequests.requestedByActorType }).from(agentWakeupRequests) + .where(and(eq(agentWakeupRequests.companyId, run.companyId), eq(agentWakeupRequests.id, run.wakeupRequestId))); + // A person asked for this one (a board restore): it is theirs to keep. + if (request?.actor === "user") return false; + } + if (task.assigneeAgentId !== run.agentId) return true; + const recoveryReasons = sql.join([...RECOVERY_WAKE_REASONS].map(reason => sql`${reason}`), sql`, `); + // The run it recovers is never newer work, whatever the two rows' timestamps say. + const notOwnRuns = [run.id, run.retryOfRunId, readNonEmptyString(parseObject(run.contextSnapshot).previousRunId)] + .filter((id): id is string => Boolean(id && isUuidLike(id))); + const [other] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, run.companyId), + sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`, + notInArray(heartbeatRuns.id, notOwnRuns), + or( + // Ran after this one was queued: the task already continued. + gt(heartbeatRuns.startedAt, run.createdAt), + eq(heartbeatRuns.status, "running"), + and(eq(heartbeatRuns.status, "queued"), or( + sql`coalesce(${heartbeatRuns.contextSnapshot}->>'wakeReason', '') not in (${recoveryReasons})`, + gt(heartbeatRuns.createdAt, run.createdAt))), + ), + )).limit(1); + return Boolean(other); + } + /** System continuations that only make sense for an open task. */ const SYSTEM_CONTINUATION_WAKE_REASONS = new Set(["issue_recovery_action_restored", "issue_continuation_needed", "issue_assignment_recovery", ISSUE_BLOCKER_STALLED_WAKE_REASON, "issue_children_completed"]); @@ -925,13 +966,13 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende * attached after the check changes the run, the cancel matches nothing, and * the check runs again. */ - async function cancelClosedTaskContinuation(run: typeof heartbeatRuns.$inferSelect, issueId: string) { + async function cancelClosedTaskContinuation(run: typeof heartbeatRuns.$inferSelect, issueId: string, + outcome: { error: string; errorCode: string } = CLOSED_TASK_CONTINUATION) { for (let attempt = 0; attempt < 3; attempt += 1) { const decision = await checkClosedTaskContinuation(run, issueId); if (decision.kind !== "cancel") return decision; const write = await setRunStatusFromLive(run.id, "cancelled", ["queued"], { - finishedAt: new Date(), error: "The task was closed before this continuation started.", - errorCode: "issue_closed_before_start", + finishedAt: new Date(), ...outcome, }, undefined, sql`${heartbeatRuns}.xmin::text = ${decision.rowVersion}`); if (write.updated && write.run) return { kind: "cancelled" as const, run: write.run }; if (write.run?.status !== "queued") return { kind: "not_queued" as const }; @@ -992,13 +1033,19 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende const continuedIssueId = readNonEmptyString(runContext.issueId); if (systemContinuationReason && SYSTEM_CONTINUATION_WAKE_REASONS.has(systemContinuationReason) && continuedIssueId && runContext.onboardingCompletion !== true) { - const [target] = await db.select({ status: issues.status }).from(issues) + const [target] = await db.select({ status: issues.status, assigneeAgentId: issues.assigneeAgentId }).from(issues) .where(and(eq(issues.companyId, run.companyId), eq(issues.id, continuedIssueId))); - if (target && ["done", "cancelled"].includes(target.status)) { - const closed = await cancelClosedTaskContinuation(run, continuedIssueId); + const closedTask = Boolean(target && ["done", "cancelled"].includes(target.status)); + // Dropped quietly: the run is cancelled before it started and its wake is + // skipped, the same as a continuation whose task closed. Nothing failed. + const outcome = closedTask ? CLOSED_TASK_CONTINUATION + : target && RECOVERY_WAKE_REASONS.has(systemContinuationReason) + && await recoveryRunNoLongerNeeded(run, continuedIssueId, target) ? RECOVERY_NO_LONGER_NEEDED : null; + if (outcome) { + const closed = await cancelClosedTaskContinuation(run, continuedIssueId, outcome); if (closed.kind === "cancelled") { const cancelled = closed.run; - await setWakeupStatus(run.wakeupRequestId, "skipped", { finishedAt: cancelled.finishedAt ?? new Date(), error: "issue_closed_before_start" }); + await setWakeupStatus(run.wakeupRequestId, "skipped", { finishedAt: cancelled.finishedAt ?? new Date(), error: outcome.errorCode }); await releaseIssueExecutionAndPromote(cancelled, { suppressImmediateRecovery: true, deferredPostCommitEffects }); } // A person's undelivered message rides on this run: it runs as their @@ -2425,7 +2472,8 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende agent: typeof agents.$inferSelect, queuedRuns: QueuedRunCandidate[], dependencyReadiness: Map, - issueById: Map, + issueById: Map, ) { const now = new Date(); pruneDeclinedAgedClaims(now); @@ -2531,7 +2579,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende }))]; const issueRows = queuedIssueIds.length > 0 ? await db.select({ id: issues.id, status: issues.status, priority: issues.priority, updatedAt: issues.updatedAt, - executionRunId: issues.executionRunId }).from(issues) + executionRunId: issues.executionRunId, createdByUserId: issues.createdByUserId }).from(issues) .where(and(eq(issues.companyId, agent.companyId), inArray(issues.id, queuedIssueIds))) : []; const issueById = new Map(issueRows.map((row) => [row.id, row])); diff --git a/server/src/services/heartbeat/queued-run-dispatch-order.test.ts b/server/src/services/heartbeat/queued-run-dispatch-order.test.ts index 1abcb6b1818..3ae58093c2d 100644 --- a/server/src/services/heartbeat/queued-run-dispatch-order.test.ts +++ b/server/src/services/heartbeat/queued-run-dispatch-order.test.ts @@ -4,9 +4,12 @@ import { QUEUED_RUN_AGING_MS, RECOVERY_BACKLOG_AGE_MS, createQueuedRunDispatcher, + dispatchCountersFromStarts, + queuedRunStartOf, RECOVERY_DISPATCH_SHARE, isRecoveryWakeReason, orderQueuedRunsForDispatch, + queuedRunDispatchLine, queuedRunDispatchClass, type QueuedRunDispatchFacts, } from "./queued-run-dispatch-order.js"; @@ -58,10 +61,17 @@ describe("queued run dispatch order", () => { expect(ordered.slice(0, 2).map(entry => entry.id).sort()).toEqual([chat.id, assignment.id].sort()); }); - it("keeps draining recovery wakes while people keep sending work", () => { + it("holds recovery wakes while people keep sending work", () => { + const burst = Array.from({ length: 10 }, () => backlogRecovery()); + const { started, pending } = dispatch(burst, { steps: 40, arrivals: () => [personRetry()] }); + expect(started.every(entry => entry.wakeReason === "task_retry")).toBe(true); + expect(pending.filter(entry => entry.wakeReason === "issue_recovery_action_restored")).toHaveLength(burst.length); + }); + + it("keeps draining recovery wakes at their share while agents keep sending work", () => { const burst = Array.from({ length: 10 }, () => backlogRecovery()); const steps = burst.length * RECOVERY_DISPATCH_SHARE; - const { started } = dispatch(burst, { steps, arrivals: () => [personRetry()] }); + const { started } = dispatch(burst, { steps, arrivals: () => [run({ wakeReason: "issue_assigned", createdAt: minutesAgo(1) })] }); const recoveryStarts = started.filter(entry => entry.wakeReason === "issue_recovery_action_restored").length; expect(recoveryStarts).toBe(burst.length); for (let window = 0; window + RECOVERY_DISPATCH_SHARE <= started.length; window += 1) { @@ -69,18 +79,22 @@ describe("queued run dispatch order", () => { if (window + RECOVERY_DISPATCH_SHARE <= steps) expect(slice.some(entry => entry.wakeReason === "issue_recovery_action_restored")).toBe(true); } - expect(started.filter(entry => entry.wakeReason === "task_retry").length).toBe(steps - burst.length); + expect(started.filter(entry => entry.wakeReason === "issue_assigned").length).toBe(steps - burst.length); }); - it("keeps recovery's share when one dispatch fills many slots", () => { - const people = Array.from({ length: 12 }, () => personRetry()); + it("keeps recovery's share of agent work when one dispatch fills many slots, and none of a person's", () => { + const work = Array.from({ length: 12 }, () => run({ wakeReason: "issue_assigned", createdAt: minutesAgo(1) })); const recovery = Array.from({ length: 6 }, () => backlogRecovery()); - const batch = orderQueuedRunsForDispatch([...recovery, ...people], { now, startsSinceRecovery: 0 }).slice(0, 8); - expect(batch.map(entry => entry.wakeReason === "task_retry" ? "P" : "R").join("")).toBe("PPPRPPPR"); + const batch = orderQueuedRunsForDispatch([...recovery, ...work], { now, startsSinceRecovery: 0 }).slice(0, 8); + expect(batch.map(entry => entry.wakeReason === "issue_assigned" ? "A" : "R").join("")).toBe("AAARAAAR"); + const people = Array.from({ length: 4 }, () => personRetry()); + const mixed = orderQueuedRunsForDispatch([...recovery, ...people, ...work], { now, startsSinceRecovery: 0 }).slice(0, 8); + expect(mixed.map(entry => entry.wakeReason === "task_retry" ? "P" : entry.wakeReason === "issue_assigned" ? "A" : "R").join("")) + .toBe("PPPPRAAA"); }); it("offers the next recovery run when the reserved recovery run cannot be claimed", () => { - const people = Array.from({ length: 4 }, () => personRetry()); + const people = Array.from({ length: 4 }, () => run({ wakeReason: "issue_assigned", createdAt: minutesAgo(1) })); const stuck = backlogRecovery(59); const claimable = backlogRecovery(30); const dispatcher = createQueuedRunDispatcher([...people, stuck, claimable], { now, startsSinceRecovery: RECOVERY_DISPATCH_SHARE - 1 }); @@ -89,7 +103,7 @@ describe("queued run dispatch order", () => { const second = dispatcher.next(); expect(second!.id).toBe(claimable.id); dispatcher.claimed(second!); - expect(dispatcher.next()!.wakeReason).toBe("task_retry"); + expect(dispatcher.next()!.wakeReason).toBe("issue_assigned"); }); it("does not put a fresh interruption's continuation behind old-backlog recovery wakes", () => { @@ -125,29 +139,22 @@ describe("queued run dispatch order", () => { expect(ordered.map(entry => entry.id)).toEqual([urgent.id, earlier.id, later.id, inProgress.id, todo.id]); }); - it("starts an old queued continuation within a bounded number of starts while fresh person and system work keeps arriving", () => { + it("starts an old queued continuation within a bounded number of starts while fresh agent and recovery work keeps arriving", () => { const continuation = run({ wakeReason: "issue_continuation_needed", createdAt: minutesAgo(150), sourceEndedAt: minutesAgo(152) }); const started: QueuedRunDispatchFacts[] = []; let pending: QueuedRunDispatchFacts[] = [continuation]; const steps = 40; for (let step = 0; step < steps; step += 1) { - pending.push(personRetry(), run({ wakeReason: "issue_assigned", createdAt: minutesAgo(1) }), + pending.push(run({ wakeReason: "issue_assigned", createdAt: minutesAgo(1) }), run({ wakeReason: "issue_continuation_needed", createdAt: minutesAgo(1), sourceEndedAt: minutesAgo(2) })); - const recent = started.slice(-RECOVERY_DISPATCH_SHARE).reverse(); - const lastRecovery = recent.findIndex(entry => isRecoveryWakeReason(entry.wakeReason)); - const oldest = pending.reduce((left, right) => left.createdAt <= right.createdAt ? left : right); - const recentForOldest = started.slice(-OLDEST_DISPATCH_SHARE).reverse(); - const lastOldest = recentForOldest.findIndex(entry => entry.createdAt <= oldest.createdAt); - const [next] = orderQueuedRunsForDispatch(pending, { now, - startsSinceRecovery: lastRecovery < 0 ? recent.length : lastRecovery, - startsSinceOldest: lastOldest < 0 ? recentForOldest.length : lastOldest }); + const [next] = orderQueuedRunsForDispatch(pending, { now, ...dispatchCountersFromStarts(pending, [...started].reverse().map(run => queuedRunStartOf(run, now)), now) }); started.push(next!); pending = pending.filter(entry => entry.id !== next!.id); } const position = started.findIndex(entry => entry.id === continuation.id); expect(position).toBeGreaterThanOrEqual(0); - expect(position).toBeLessThan(OLDEST_DISPATCH_SHARE); - expect(started.slice(0, position).every(entry => entry.wakeReason === "task_retry" || entry.wakeReason === "issue_continuation_needed")).toBe(true); + expect(position).toBeLessThan(RECOVERY_DISPATCH_SHARE); + expect(started.slice(0, position).every(entry => entry.wakeReason === "issue_assigned")).toBe(true); }); it("serves the longest-waiting run once every share even when fresher work outranks it", () => { @@ -166,12 +173,11 @@ describe("queued run dispatch order", () => { expect(ordered[0]!.wakeReason).toBe("task_retry"); }); - it("lets an old-backlog recovery run age into the work tier", () => { - const aged = run({ wakeReason: "issue_continuation_needed", createdAt: new Date(now.getTime() - QUEUED_RUN_AGING_MS), - sourceEndedAt: minutesAgo(200) }); + it("never lets a long-waiting recovery run age ahead of agent work or into the oldest-run share", () => { + const aged = run({ wakeReason: "issue_continuation_needed", createdAt: minutesAgo(400), sourceEndedAt: minutesAgo(402) }); const fresh = run({ wakeReason: "issue_assigned", createdAt: minutesAgo(1) }); - const ordered = orderQueuedRunsForDispatch([fresh, aged], { now, startsSinceRecovery: 0 }); - expect(ordered.map(entry => entry.id)).toEqual([aged.id, fresh.id]); + const ordered = orderQueuedRunsForDispatch([fresh, aged], { now, startsSinceRecovery: 0, startsSinceOldest: OLDEST_DISPATCH_SHARE - 1 }); + expect(ordered.map(entry => entry.id)).toEqual([fresh.id, aged.id]); }); it("passes the oldest-run share to the next oldest run while the oldest one keeps failing to claim", () => { @@ -194,17 +200,105 @@ describe("queued run dispatch order", () => { for (let step = 0; step < 3 * OLDEST_DISPATCH_SHARE; step += 1) { pending.push(run({ wakeReason: "issue_recovery_action_restored", createdAt: minutesAgo(1), sourceEndedAt: minutesAgo(2) }), run({ wakeReason: "issue_assigned", readinessRank: 1, createdAt: minutesAgo(1) })); - const recent = started.slice(-RECOVERY_DISPATCH_SHARE).reverse(); - const lastRecovery = recent.findIndex(entry => isRecoveryWakeReason(entry.wakeReason)); - const oldest = pending.reduce((left, right) => left.createdAt <= right.createdAt ? left : right); - const recentForOldest = started.slice(-OLDEST_DISPATCH_SHARE).reverse(); - const lastOldest = recentForOldest.findIndex(entry => entry.createdAt <= oldest.createdAt); - const [next] = orderQueuedRunsForDispatch(pending, { now, - startsSinceRecovery: lastRecovery < 0 ? recent.length : lastRecovery, - startsSinceOldest: lastOldest < 0 ? recentForOldest.length : lastOldest }); + const [next] = orderQueuedRunsForDispatch(pending, { now, ...dispatchCountersFromStarts(pending, [...started].reverse().map(run => queuedRunStartOf(run, now)), now) }); started.push(next!); pending = pending.filter(entry => entry.id !== next!.id); } for (const continuation of continuations) expect(started.map(entry => entry.id)).toContain(continuation.id); }); }); + +describe("deep per-agent queue: a recovery burst behind 20 busy slots (paperclip.staging, 2026-11-13)", () => { + // Codie, maxConcurrentRuns 20, every slot busy: each freed slot is one dispatch. + const at = new Date("2026-11-13T14:30:00.000Z"); + const clock = (iso: string) => new Date(`2026-11-13T${iso}:00.000Z`); + const addMinutes = (date: Date, minutes: number) => new Date(date.getTime() + minutes * 60_000); + let id = 0; + const make = (input: Partial & { wakeReason: string; createdAt: Date }): QueuedRunDispatchFacts => + ({ id: `codie-${++id}`, requestedByUser: false, sourceEndedAt: null, readinessRank: 1, priorityRank: 2, ...input }); + const isPerson = (run: QueuedRunDispatchFacts) => run.requestedByUser; + + function todaysQueue() { + // Agent-made assignments queued before the people's tasks. + const assignments = Array.from({ length: 60 }, (_, index) => make({ wakeReason: "issue_assigned", + createdAt: addMinutes(clock("05:00"), index * 2) })); + // Two tasks people created, stalled by a restart at 07:25 and waiting since 07:29 on system wakes: + // a monitor check and the assigned-work sweep's continuation. Neither wake was a person's. + const waitingPeople = [ + make({ wakeReason: "issue_monitor_due", personTask: true, readinessRank: 0, createdAt: clock("07:29") }), + make({ wakeReason: "issue_continuation_needed", personTask: true, readinessRank: 0, sourceEndedAt: clock("07:27"), + createdAt: addMinutes(clock("07:29"), 1) }), + ]; + // Later person-requested work on tasks already in progress, at higher priority. + const laterPeople = Array.from({ length: 30 }, (_, index) => make({ wakeReason: "issue_commented", requestedByUser: true, + readinessRank: 0, priorityRank: 1, createdAt: addMinutes(clock("08:00"), index * 12) })); + // The burst: 139 system recovery wakes between 10:46 and 11:47 for in-progress tasks. + const burst = Array.from({ length: 139 }, (_, index) => make({ wakeReason: "issue_recovery_action_restored", + readinessRank: 0, sourceEndedAt: clock("09:00"), createdAt: addMinutes(clock("10:46"), Math.floor(index * 61 / 139)) })); + return { assignments, waitingPeople, laterPeople, burst }; + } + + function dispatchOneAtATime(queue: QueuedRunDispatchFacts[], steps: number) { + const started: Array<{ run: QueuedRunDispatchFacts; personWaiting: boolean }> = []; + let pending = [...queue]; + for (let step = 0; step < steps; step += 1) { + // People and agents keep adding work while the queue drains. + if (step % 2 === 0) pending.push(make({ wakeReason: "issue_commented", requestedByUser: true, readinessRank: 0, + priorityRank: 1, createdAt: at })); + pending.push(make({ wakeReason: "issue_assigned", createdAt: at })); + const history = started.map(entry => entry.run); + const [next] = orderQueuedRunsForDispatch(pending, { now: at, ...dispatchCountersFromStarts(pending, [...history].reverse().map(run => queuedRunStartOf(run, at)), at) }); + started.push({ run: next!, personWaiting: pending.some(entry => entry !== next && isPerson(entry) && entry.readinessRank < 3) }); + pending = pending.filter(entry => entry.id !== next!.id); + } + return started; + } + + it("starts the oldest run on a task a person is waiting on within the oldest-run share, and never starts recovery while a person waits", () => { + const { assignments, waitingPeople, laterPeople, burst } = todaysQueue(); + const started = dispatchOneAtATime([...burst, ...assignments, ...laterPeople, ...waitingPeople], 40); + const position = started.findIndex(entry => entry.run.id === waitingPeople[0]!.id); + expect(position).toBeGreaterThanOrEqual(0); + expect(position).toBeLessThan(OLDEST_DISPATCH_SHARE); + // The next one starts within the following share. + const second = started.findIndex(entry => entry.run.id === waitingPeople[1]!.id); + expect(second).toBeGreaterThanOrEqual(0); + expect(second).toBeLessThan(2 * OLDEST_DISPATCH_SHARE); + const recoveryAheadOfPeople = started.filter(entry => entry.run.wakeReason === "issue_recovery_action_restored" && entry.personWaiting); + expect(recoveryAheadOfPeople).toHaveLength(0); + }); + + it("does not let older agent work use up the turn of a run on a task a person created", () => { + // No person-requested arrivals: normal order serves the older, higher-priority assignments. + const assignments = Array.from({ length: 60 }, (_, index) => make({ wakeReason: "issue_assigned", priorityRank: 1, + createdAt: addMinutes(clock("05:00"), index * 2) })); + const monitor = make({ wakeReason: "issue_monitor_due", personTask: true, priorityRank: 3, createdAt: clock("07:29") }); + const started: QueuedRunDispatchFacts[] = []; + let pending = [...assignments, monitor]; + for (let step = 0; step < 2 * OLDEST_DISPATCH_SHARE && pending.length; step += 1) { + const [next] = orderQueuedRunsForDispatch(pending, { now: at, + ...dispatchCountersFromStarts(pending, [...started].reverse().map(run => queuedRunStartOf(run, at)), at) }); + started.push(next!); + pending = pending.filter(entry => entry.id !== next!.id); + } + const position = started.findIndex(entry => entry.id === monitor.id); + expect(position).toBeGreaterThanOrEqual(0); + expect(position).toBeLessThan(OLDEST_DISPATCH_SHARE); + // The place-in-line order agrees. + const line = queuedRunDispatchLine([...assignments, monitor], { now: at, startsSinceRecovery: 0, startsSinceOldest: 0 }); + expect(line.findIndex(entry => entry.id === monitor.id)).toBeLessThan(OLDEST_DISPATCH_SHARE); + }); + + it("keeps recovery behind agent assignments once people are served, and still drains it at its share", () => { + const { assignments, burst } = todaysQueue(); + const started: QueuedRunDispatchFacts[] = []; + let pending = [...burst.slice(0, 3), ...assignments]; + for (let step = 0; step < 4 * RECOVERY_DISPATCH_SHARE && pending.length; step += 1) { + const [next] = orderQueuedRunsForDispatch(pending, { now: at, ...dispatchCountersFromStarts(pending, [...started].reverse().map(run => queuedRunStartOf(run, at)), at) }); + started.push(next!); + pending = pending.filter(entry => entry.id !== next!.id); + } + expect(started.filter(entry => entry.wakeReason === "issue_recovery_action_restored")).toHaveLength(3); + expect(started[0]!.wakeReason).toBe("issue_assigned"); + }); +}); diff --git a/server/src/services/heartbeat/queued-run-dispatch-order.ts b/server/src/services/heartbeat/queued-run-dispatch-order.ts index 149d4be515b..0aef1f561c5 100644 --- a/server/src/services/heartbeat/queued-run-dispatch-order.ts +++ b/server/src/services/heartbeat/queued-run-dispatch-order.ts @@ -27,6 +27,8 @@ export type QueuedRunDispatchFacts = { readinessRank: number; priorityRank: number; shareDeferred?: boolean; + /** The task was created by a person, who waits on its outcome whichever system wake moves it. */ + personTask?: boolean; }; export type QueuedRunDispatchClass = "person" | "work" | "recovery" | "recovery_backlog"; @@ -48,8 +50,18 @@ export function queuedRunDispatchClass(run: QueuedRunDispatchFacts, now: Date): return "work"; } -const CLASS_TIER: Record = { person: 0, work: 1, recovery: 1, recovery_backlog: 2 }; +// System recovery sits below all other work and never ages out of it: a +// person's run always goes first, and recovery takes only its share of the +// starts that would otherwise go to agent work. +const CLASS_TIER: Record = { person: 0, work: 1, recovery: 2, recovery_backlog: 3 }; const NOT_READY_RANK = 3; +const NOT_READY_TIER = 4; + +const isReady = (run: QueuedRunDispatchFacts) => run.readinessRank < NOT_READY_RANK; +const isPersonRun = (run: QueuedRunDispatchFacts, now: Date) => queuedRunDispatchClass(run, now) === "person"; +/** A run a person asked for, or any run on a task a person created. */ +const isPersonWaitingRun = (run: QueuedRunDispatchFacts, now: Date) => Boolean(run.personTask) || isPersonRun(run, now); +const isOldestShareEligible = (run: QueuedRunDispatchFacts) => isReady(run) && !run.shareDeferred; export type QueuedRunDispatchInput = { now: Date; startsSinceRecovery: number; startsSinceOldest?: number }; @@ -57,20 +69,102 @@ export function hasQueuedRunAged(run: Pick, return now.getTime() - run.createdAt.getTime() >= QUEUED_RUN_AGING_MS; } -export function oldestReadyQueuedRun(runs: readonly T[]): T | null { +export function oldestReadyQueuedRun(runs: readonly T[], + include: (run: T) => boolean = () => true): T | null { let oldest: T | null = null; for (const run of runs) { - if (run.readinessRank >= NOT_READY_RANK || run.shareDeferred) continue; + if (!isOldestShareEligible(run) || !include(run)) continue; if (!oldest || run.createdAt.getTime() < oldest.createdAt.getTime()) oldest = run; } return oldest; } +/** + * The run the oldest-run share serves next: the longest-waiting run past the + * aging bound that a person asked for or whose task a person created (a + * system continuation of a person's stalled task included), else the + * longest-waiting agent-work run past it. Other system recovery is never + * served by this share; it has its own. + * + * So such a run that has waited past the bound starts within + * OLDEST_DISPATCH_SHARE x (aged person-task runs queued before it + 1) starts, + * however much newer or higher-priority work arrives. Waiting never fails or + * cancels it; it only moves it forward. + */ +export function oldestShareCandidate(runs: readonly T[], now: Date): T | null { + const aged = (run: T) => hasQueuedRunAged(run, now); + return oldestReadyQueuedRun(runs, run => aged(run) && isPersonWaitingRun(run, now)) + ?? oldestReadyQueuedRun(runs, run => aged(run) && !isSystemRecoveryRun(run)); +} + +/** + * One of the agent's recent starts. `personOrigin`: a person asked for it (its + * wake or source chain). `personWaiting`: that, or its task was created by a + * person. + */ +export type QueuedRunStart = { wakeReason: string | null; createdAt: Date; startedAt?: Date | null; personOrigin?: boolean; + personWaiting?: boolean }; + +/** A dispatched run as the start history records it. */ +export function queuedRunStartOf(run: QueuedRunDispatchFacts, now: Date): QueuedRunStart { + return { wakeReason: run.wakeReason, createdAt: run.createdAt, personOrigin: isPersonRun(run, now), + personWaiting: isPersonWaitingRun(run, now) }; +} + +/** + * Whether a start used up the oldest-run share's turn: it was at least as old + * as the candidate, and, while the candidate is a run people wait on, it was + * one too. Older agent work never uses up a person's turn. + */ +function servesOldestShare(started: { createdAt: Date; personWaiting: boolean }, candidate: QueuedRunDispatchFacts, now: Date) { + return started.createdAt.getTime() <= candidate.createdAt.getTime() + && (started.personWaiting || !isPersonWaitingRun(candidate, now)); +} + +/** Where the start history must reach back to for `dispatchCountersFromStarts`, or null when no counter needs it. */ +export function dispatchHistoryStart(runs: readonly T[], now: Date): Date | null { + const recovery = runs.filter(run => isSystemRecoveryRun(run)); + const countsRecovery = recovery.length > 0 && recovery.length < runs.length; + const candidate = runs.length > 1 ? oldestShareCandidate(runs, now) : null; + const at = Math.min(countsRecovery ? Math.min(...recovery.map(run => run.createdAt.getTime())) : Infinity, + candidate?.createdAt.getTime() ?? Infinity); + return Number.isFinite(at) ? new Date(at) : null; +} + +/** + * Derives the share counters from this agent's recent starts, newest first. + * The dispatcher is rebuilt for every pass, so its shares carry over through + * the agent's own start history rather than process memory. A person's + * recovery restart (`personOrigin`) is person work, so it does not use up + * system recovery's share. + */ +export function dispatchCountersFromStarts(runs: readonly T[], + recentStarts: readonly QueuedRunStart[], now: Date): Pick { + const startsSince = (at: number) => recentStarts.filter(start => !start.startedAt || start.startedAt.getTime() >= at); + const recovery = runs.filter(run => isSystemRecoveryRun(run)); + let startsSinceRecovery = 0; + if (recovery.length > 0 && recovery.length < runs.length) { + const starts = startsSince(Math.min(...recovery.map(run => run.createdAt.getTime()))).slice(0, RECOVERY_DISPATCH_SHARE); + const last = starts.findIndex(start => isRecoveryWakeReason(start.wakeReason) && !start.personOrigin); + startsSinceRecovery = last < 0 ? starts.length : last; + } + let startsSinceOldest = 0; + const candidate = runs.length > 1 ? oldestShareCandidate(runs, now) : null; + if (candidate) { + const starts = startsSince(candidate.createdAt.getTime()).slice(0, OLDEST_DISPATCH_SHARE); + const last = starts.findIndex(start => servesOldestShare({ createdAt: start.createdAt, + personWaiting: Boolean(start.personWaiting) }, candidate, now)); + startsSinceOldest = last < 0 ? starts.length : last; + } + return { startsSinceRecovery, startsSinceOldest }; +} + function sortForDispatch(runs: readonly T[], now: Date): T[] { const tier = (run: T) => { - if (run.readinessRank >= NOT_READY_RANK) return 3; - const classTier = CLASS_TIER[queuedRunDispatchClass(run, now)]; - return hasQueuedRunAged(run, now) ? Math.min(classTier, CLASS_TIER.work) : classTier; + if (!isReady(run)) return NOT_READY_TIER; + const runClass = queuedRunDispatchClass(run, now); + // Old backlog that has waited past the aging bound rejoins the other recovery, never more. + return runClass === "recovery_backlog" && hasQueuedRunAged(run, now) ? CLASS_TIER.recovery : CLASS_TIER[runClass]; }; return [...runs].sort((left, right) => tier(left) - tier(right) @@ -89,17 +183,24 @@ export function createQueuedRunDispatcher( return { next(): T | null { if (!remaining.length) return null; - const oldest = sinceOldest >= OLDEST_DISPATCH_SHARE - 1 ? oldestReadyQueuedRun(remaining) : null; - if (oldest && hasQueuedRunAged(oldest, input.now)) return remaining.splice(remaining.indexOf(oldest), 1)[0]!; - const recoveryIndex = sinceRecovery >= RECOVERY_DISPATCH_SHARE - 1 - ? remaining.findIndex(run => run.readinessRank < NOT_READY_RANK && isSystemRecoveryRun(run)) - : -1; + const oldest = sinceOldest >= OLDEST_DISPATCH_SHARE - 1 ? oldestShareCandidate(remaining, input.now) : null; + if (oldest) return remaining.splice(remaining.indexOf(oldest), 1)[0]!; + // Recovery's share comes out of agent work, never out of a person's turn. + const personWaiting = remaining.some(run => isReady(run) && isPersonRun(run, input.now)); + // The share goes to the longest-waiting recovery run once one has waited + // past the aging bound, so fresh recovery cannot keep an old one waiting. + const agedRecovery = oldestReadyQueuedRun(remaining, run => isSystemRecoveryRun(run) && hasQueuedRunAged(run, input.now)); + const recoveryIndex = personWaiting || sinceRecovery < RECOVERY_DISPATCH_SHARE - 1 ? -1 + : agedRecovery ? remaining.indexOf(agedRecovery) + : remaining.findIndex(run => isReady(run) && isSystemRecoveryRun(run)); return remaining.splice(recoveryIndex < 0 ? 0 : recoveryIndex, 1)[0]!; }, claimed(run: T) { sinceRecovery = isSystemRecoveryRun(run) ? 0 : sinceRecovery + 1; - const oldest = oldestReadyQueuedRun(remaining); - sinceOldest = !oldest || oldest.createdAt.getTime() >= run.createdAt.getTime() ? 0 : sinceOldest + 1; + // The same rule dispatchCountersFromStarts reads from the start history. + const oldest = oldestShareCandidate([...remaining, run], input.now); + sinceOldest = !oldest || servesOldestShare({ createdAt: run.createdAt, personWaiting: isPersonWaitingRun(run, input.now) }, + oldest, input.now) ? 0 : sinceOldest + 1; }, }; } @@ -128,50 +229,50 @@ export function queuedRunDispatchLine( runs: readonly T[], input: QueuedRunDispatchInput, ): T[] { - const order = sortForDispatch(runs, input.now); + const now = input.now; + const order = sortForDispatch(runs, now); const removed = new Uint8Array(order.length); - // Eligible for the oldest share, oldest first; ties keep dispatch order, - // as oldestReadyQueuedRun keeps the first of equal ages. - const byAge = order.map((_, index) => index) - .filter(index => order[index]!.readinessRank < NOT_READY_RANK && !order[index]!.shareDeferred) - .sort((left, right) => order[left]!.createdAt.getTime() - order[right]!.createdAt.getTime() || left - right); - const recovery = order.map((_, index) => index) - .filter(index => order[index]!.readinessRank < NOT_READY_RANK && isSystemRecoveryRun(order[index]!)); - let head = 0; - let agePointer = 0; - let recoveryPointer = 0; - const firstOf = (indexes: number[] | null, pointer: number) => { - if (!indexes) { - while (head < order.length && removed[head]) head += 1; - return head < order.length ? head : -1; - } - while (pointer < indexes.length && removed[indexes[pointer]!]) pointer += 1; - return pointer; + const indexes = (include: (run: T) => boolean) => order.map((_, index) => index).filter(index => include(order[index]!)); + // Oldest first; ties keep dispatch order, as oldestReadyQueuedRun keeps the first of equal ages. + const byAge = (list: number[]) => + list.sort((left, right) => order[left]!.createdAt.getTime() - order[right]!.createdAt.getTime() || left - right); + // A list's first entry not yet started; lists stay in their order, so each pointer only moves forward. + const cursor = (list: number[]) => { + let pointer = 0; + return () => { + while (pointer < list.length && removed[list[pointer]!]) pointer += 1; + return pointer < list.length ? list[pointer]! : -1; + }; }; - const oldestIndex = () => { - agePointer = firstOf(byAge, agePointer); - return agePointer < byAge.length ? byAge[agePointer]! : -1; + const agedFirst = (next: () => number) => () => { + const index = next(); + return index >= 0 && hasQueuedRunAged(order[index]!, now) ? index : -1; }; + const oldestPersonWaiting = agedFirst(cursor(byAge(indexes(run => isOldestShareEligible(run) && isPersonWaitingRun(run, now))))); + const oldestWork = agedFirst(cursor(byAge(indexes(run => isOldestShareEligible(run) && !isSystemRecoveryRun(run))))); + const oldestShare = () => { const person = oldestPersonWaiting(); return person >= 0 ? person : oldestWork(); }; + const oldestRecovery = agedFirst(cursor(byAge(indexes(run => isOldestShareEligible(run) && isSystemRecoveryRun(run))))); + const firstRecovery = cursor(indexes(run => isReady(run) && isSystemRecoveryRun(run))); + const head = cursor(order.map((_, index) => index)); + let readyPeople = order.filter(run => isReady(run) && isPersonRun(run, now)).length; let sinceRecovery = input.startsSinceRecovery; let sinceOldest = input.startsSinceOldest ?? 0; const result: T[] = []; while (result.length < order.length) { - let pick = -1; - if (sinceOldest >= OLDEST_DISPATCH_SHARE - 1) { - const oldest = oldestIndex(); - if (oldest >= 0 && hasQueuedRunAged(order[oldest]!, input.now)) pick = oldest; + const candidate = oldestShare(); + let pick = sinceOldest >= OLDEST_DISPATCH_SHARE - 1 ? candidate : -1; + if (pick < 0 && readyPeople === 0 && sinceRecovery >= RECOVERY_DISPATCH_SHARE - 1) { + pick = oldestRecovery(); + if (pick < 0) pick = firstRecovery(); } - if (pick < 0 && sinceRecovery >= RECOVERY_DISPATCH_SHARE - 1) { - recoveryPointer = firstOf(recovery, recoveryPointer); - if (recoveryPointer < recovery.length) pick = recovery[recoveryPointer]!; - } - if (pick < 0) pick = firstOf(null, 0); + if (pick < 0) pick = head(); removed[pick] = 1; const run = order[pick]!; + if (isReady(run) && isPersonRun(run, now)) readyPeople -= 1; result.push(run); sinceRecovery = isSystemRecoveryRun(run) ? 0 : sinceRecovery + 1; - const oldest = oldestIndex(); - sinceOldest = oldest < 0 || order[oldest]!.createdAt.getTime() >= run.createdAt.getTime() ? 0 : sinceOldest + 1; + sinceOldest = candidate < 0 || servesOldestShare({ createdAt: run.createdAt, personWaiting: isPersonWaitingRun(run, now) }, + order[candidate]!, now) ? 0 : sinceOldest + 1; } return result; } diff --git a/server/src/services/heartbeat/queued-run-line.test.ts b/server/src/services/heartbeat/queued-run-line.test.ts index ded2a0c0d8f..c9b47af90b2 100644 --- a/server/src/services/heartbeat/queued-run-line.test.ts +++ b/server/src/services/heartbeat/queued-run-line.test.ts @@ -35,6 +35,7 @@ function queue(seed: number, size: number): QueuedRunDispatchFacts[] { readinessRank: pick([0, 0, 1, 2, 3]), priorityRank: pick([0, 1, 2, 3, 4]), shareDeferred: next() < 0.1, + personTask: next() < 0.3, }; }); } diff --git a/server/src/services/heartbeat/queued-run-line.ts b/server/src/services/heartbeat/queued-run-line.ts index 29d3c6bcd28..9fb3047faba 100644 --- a/server/src/services/heartbeat/queued-run-line.ts +++ b/server/src/services/heartbeat/queued-run-line.ts @@ -1,4 +1,4 @@ -import { and, asc, desc, eq, gte, inArray, isNotNull, sql } from "drizzle-orm"; +import { and, asc, desc, eq, gte, inArray, isNotNull, sql, type SQL } from "drizzle-orm"; import { agentWakeupRequests, agents, heartbeatRuns, issues, type Db } from "@paperclipai/db"; import type { ExecutionAgentQueue } from "@paperclipai/shared"; import { parseObject } from "../../adapters/utils.js"; @@ -6,10 +6,8 @@ import { OLDEST_DISPATCH_SHARE, PERSON_WAKE_REASONS, RECOVERY_DISPATCH_SHARE, - hasQueuedRunAged, - isRecoveryWakeReason, - isSystemRecoveryRun, - oldestReadyQueuedRun, + dispatchCountersFromStarts, + dispatchHistoryStart, queuedRunDispatchLine, type QueuedRunDispatchFacts, type QueuedRunDispatchInput, @@ -40,6 +38,33 @@ export function issueRunPriorityRank(priority: string | null | undefined) { } } +/** + * SQL: whether the run `runId` was asked for by a person, itself or through + * its source chain (`previousRunId`, else `retryOfRunId`), the same rule as + * {@link listPersonOriginSourceRunIds}. For use inside another query. + */ +export function personOriginRunSql(companyId: SQL, runId: SQL) { + const uuidPattern = "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$"; + return sql`exists ( + with recursive origin_chain(run_id, depth) as ( + select ${runId}, 0 + union all + select parent.id, origin_chain.depth + 1 + from origin_chain + join heartbeat_runs origin_child on origin_child.id = origin_chain.run_id + join heartbeat_runs parent on parent.company_id = ${companyId} and parent.id = coalesce( + case when origin_child.context_snapshot->>'previousRunId' ~* ${uuidPattern} + then (origin_child.context_snapshot->>'previousRunId')::uuid end, + origin_child.retry_of_run_id) + where origin_chain.depth < ${PERSON_ORIGIN_CHAIN_DEPTH} + ) + select 1 from origin_chain join heartbeat_runs origin_run on origin_run.id = origin_chain.run_id + where origin_run.context_snapshot->>'wakeReason' in (${sql.join([...PERSON_WAKE_REASONS].map(reason => sql`${reason}`), sql`, `)}) + or origin_run.context_snapshot->>'taskRetryRequestId' is not null + or exists (select 1 from agent_wakeup_requests origin_wake where origin_wake.company_id = ${companyId} + and origin_wake.requested_by_actor_type = 'user' and (origin_wake.run_id = origin_run.id or origin_wake.id = origin_run.wakeup_request_id)))`; +} + export async function listPersonOriginSourceRunIds(db: Db, companyId: string, sourceRunIds: string[]) { const uuidPattern = "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$"; const result = await db.execute<{ root_id: string }>(sql` @@ -77,7 +102,7 @@ export async function queuedRunDispatchFacts( agent: Pick; queuedRuns: R[]; dependencyReadiness: Map; - issueById: Map; + issueById: Map; shareDeferred?: (runId: string) => boolean; now: Date; }, @@ -135,39 +160,30 @@ export async function queuedRunDispatchFacts( readinessRank: issueId ? (ready ? (issue?.status === "in_progress" ? 0 : 1) : 3) : 2, priorityRank: issueRunPriorityRank(issue?.priority), shareDeferred: input.shareDeferred?.(run.id) ?? false, + personTask: Boolean(issue?.createdByUserId), }; }); - const recoveryRuns = facts.filter(entry => isSystemRecoveryRun(entry)); - const oldestReady = oldestReadyQueuedRun(facts); - const agedOldest = oldestReady && hasQueuedRunAged(oldestReady, now) ? oldestReady : null; - const countsRecovery = recoveryRuns.length > 0 && recoveryRuns.length < facts.length; - let startsSinceRecovery = 0; - let startsSinceOldest = 0; - if (countsRecovery || (agedOldest && facts.length > 1)) { - const oldestRecoveryAt = countsRecovery ? Math.min(...recoveryRuns.map(entry => entry.createdAt.getTime())) : Infinity; - const since = new Date(Math.min(oldestRecoveryAt, agedOldest?.createdAt.getTime() ?? Infinity)); - const recentStarts = await db.select({ id: heartbeatRuns.id, wakeReason: sql`${heartbeatRuns.contextSnapshot}->>'wakeReason'`, - startedAt: heartbeatRuns.startedAt, createdAt: heartbeatRuns.createdAt }) - .from(heartbeatRuns) - .where(and(eq(heartbeatRuns.companyId, agent.companyId), eq(heartbeatRuns.agentId, agent.id), - isNotNull(heartbeatRuns.startedAt), gte(heartbeatRuns.startedAt, since))) - .orderBy(desc(heartbeatRuns.startedAt)) - .limit(Math.max(RECOVERY_DISPATCH_SHARE, OLDEST_DISPATCH_SHARE)); - const startsSince = (at: number) => recentStarts.filter(row => row.startedAt && row.startedAt.getTime() >= at); - if (countsRecovery) { - const starts = startsSince(oldestRecoveryAt).slice(0, RECOVERY_DISPATCH_SHARE); - const recoveryStartIds = starts.flatMap(row => isRecoveryWakeReason(row.wakeReason) ? [row.id] : []); - const personStarts = new Set(recoveryStartIds.length - ? await listPersonOriginSourceRunIds(db, agent.companyId, recoveryStartIds) : []); - const lastRecovery = starts.findIndex(row => isRecoveryWakeReason(row.wakeReason) && !personStarts.has(row.id)); - startsSinceRecovery = lastRecovery < 0 ? starts.length : lastRecovery; - } - if (agedOldest && facts.length > 1) { - const starts = startsSince(agedOldest.createdAt.getTime()).slice(0, OLDEST_DISPATCH_SHARE); - const lastOldest = starts.findIndex(row => row.createdAt.getTime() <= agedOldest.createdAt.getTime()); - startsSinceOldest = lastOldest < 0 ? starts.length : lastOldest; - } - } + const since = dispatchHistoryStart(facts, now); + const startRows = since ? await db.select({ id: heartbeatRuns.id, wakeReason: sql`${heartbeatRuns.contextSnapshot}->>'wakeReason'`, + startedAt: heartbeatRuns.startedAt, createdAt: heartbeatRuns.createdAt, + issueId: sql`${heartbeatRuns.contextSnapshot}->>'issueId'` }) + .from(heartbeatRuns) + .where(and(eq(heartbeatRuns.companyId, agent.companyId), eq(heartbeatRuns.agentId, agent.id), + isNotNull(heartbeatRuns.startedAt), gte(heartbeatRuns.startedAt, since))) + .orderBy(desc(heartbeatRuns.startedAt)) + .limit(Math.max(RECOVERY_DISPATCH_SHARE, OLDEST_DISPATCH_SHARE)) : []; + // A person's recovery restart is person work, so it does not use up system recovery's share; + // only a start people wait on uses up their oldest-run turn. + const personStarts = new Set(startRows.length + ? await listPersonOriginSourceRunIds(db, agent.companyId, startRows.map(row => row.id)) : []); + const startedTaskIds = [...new Set(startRows.flatMap(row => row.issueId && RUN_ID_PATTERN.test(row.issueId) ? [row.issueId] : []))]; + const personTaskIds = new Set(startedTaskIds.length + ? (await db.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, agent.companyId), + inArray(issues.id, startedTaskIds), isNotNull(issues.createdByUserId)))).map(row => row.id) + : []); + const recentStarts = startRows.map(row => ({ ...row, personOrigin: personStarts.has(row.id), + personWaiting: personStarts.has(row.id) || Boolean(row.issueId && personTaskIds.has(row.issueId)) })); + const { startsSinceRecovery, startsSinceOldest } = dispatchCountersFromStarts(facts, recentStarts, now); return { facts, dispatchInput: { now, startsSinceRecovery, startsSinceOldest } }; } @@ -255,7 +271,7 @@ export async function agentQueuedRunLine( const [dependencyReadiness, issueRows] = issueIds.length ? await Promise.all([ issuesSvc.listDependencyReadiness(agent.companyId, issueIds), - db.select({ id: issues.id, status: issues.status, priority: issues.priority }).from(issues) + db.select({ id: issues.id, status: issues.status, priority: issues.priority, createdByUserId: issues.createdByUserId }).from(issues) .where(and(eq(issues.companyId, agent.companyId), inArray(issues.id, issueIds))), ]) : [new Map(), []]; diff --git a/server/src/services/recovery-queue-admission.ts b/server/src/services/recovery-queue-admission.ts new file mode 100644 index 00000000000..148bc4a54d0 --- /dev/null +++ b/server/src/services/recovery-queue-admission.ts @@ -0,0 +1,42 @@ +import { and, eq, sql, type SQL } from "drizzle-orm"; +import { heartbeatRuns, type Db } from "@paperclipai/db"; +import { RECOVERY_WAKE_REASONS } from "./heartbeat/queued-run-dispatch-order.js"; + +/** + * System recovery joins an agent's queue a few runs at a time. The rest stays + * pending with its producer (a resolved recovery action), which delivers the + * next one when a queued recovery run starts or is dropped. Recovery waits + * behind people's work in the dispatcher, so a deep recovery queue would only + * hold stale runs; pending recovery is re-read when it is admitted instead. + */ +export const QUEUED_RECOVERY_RUNS_PER_AGENT = 3; + +/** + * Evidence keys under which a restored task records the wake it is owed + * (legacy-workspace rehome, released workspace writer). While its assignee's + * recovery queue is full, that wake waits pending here. + */ +export const HELD_RECOVERY_WAKE_KEYS = ["legacyWorkspaceRehome", "releasedWorkspaceWriter"] as const; + +/** Queued recovery runs of `agentId`, as SQL on a `heartbeat_runs` alias. */ +export function queuedRecoveryRunsOf(companyId: SQL | string, agentId: SQL | string) { + const reasons = sql.join([...RECOVERY_WAKE_REASONS].map(reason => sql`${reason}`), sql`, `); + return sql`select 1 from ${heartbeatRuns} queued_recovery + where queued_recovery.company_id = ${companyId} and queued_recovery.agent_id = ${agentId} + and queued_recovery.status = 'queued' and queued_recovery.context_snapshot->>'wakeReason' in (${reasons})`; +} + +/** + * A soft cap, read before the producer's wake: producers that run at the same + * moment can each see room, so the queue can briefly hold one more run per + * concurrent producer (two today: continuation delivery and restored-task + * wakes). That stays "a few at a time"; the dispatcher keeps every recovery + * run behind people's work and re-checks it before it starts. + */ +export async function agentRecoveryQueueIsFull(db: Db, companyId: string, agentId: string) { + const reasons = [...RECOVERY_WAKE_REASONS]; + const [row] = await db.select({ count: sql`count(*)::int` }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.agentId, agentId), eq(heartbeatRuns.status, "queued"), + sql`${heartbeatRuns.contextSnapshot}->>'wakeReason' in (${sql.join(reasons.map(reason => sql`${reason}`), sql`, `)})`)); + return Number(row?.count ?? 0) >= QUEUED_RECOVERY_RUNS_PER_AGENT; +} diff --git a/server/src/services/recovery/assigned-work-sweep.ts b/server/src/services/recovery/assigned-work-sweep.ts index 880445c9921..6e69d610eb7 100644 --- a/server/src/services/recovery/assigned-work-sweep.ts +++ b/server/src/services/recovery/assigned-work-sweep.ts @@ -31,8 +31,9 @@ import { CHAT_CONTROL_RECOVERY_STOP_CODE, CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE import { onboardingFirstTaskAwaitingUserSql } from "./onboarding-first-task-wait.js"; import { workspaceRecoverySourceEligible } from "../workspace-recovery-source.js"; import { PENDING_CONTINUATION_BOUND_MS } from "../recovery-continuation-pacing.js"; -import { OLDEST_DISPATCH_SHARE, QUEUED_RUN_AGING_MS } from "../heartbeat/queued-run-dispatch-order.js"; +import { HELD_RECOVERY_WAKE_KEYS, queuedRecoveryRunsOf } from "../recovery-queue-admission.js"; import { listPersonOriginSourceRunIds } from "../heartbeat/queued-run-line.js"; +import { aliasedRun, queuedRunStalledSql } from "./stalled-queued-run-sql.js"; import { attributeBlock, BLOCK_ATTRIBUTION_WINDOW_MS, blockedStatusDetailsSql } from "../interruption-block-release.js"; import { ASSIGNED_WORK_CONTINUED_ACTION, taskAutomaticAttempts } from "./task-automatic-attempts.js"; import { isReattachPhaseActive, reattachPhaseRecheckAt } from "../heartbeat/run-admission.js"; @@ -343,11 +344,23 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) { eq(issueRecoveryActions.companyId, companyId), inArray(issueRecoveryActions.sourceIssueId, ids), eq(issueRecoveryActions.status, "resolved"), sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`, - sql`coalesce(${issueRecoveryActions.evidence}#>>'{freshContinuation,recordedAt}', ${issueRecoveryActions.resolvedAt}::text, + // Recent, or waiting its turn behind the agent's queued recovery runs. + sql`(coalesce(${issueRecoveryActions.evidence}#>>'{freshContinuation,recordedAt}', ${issueRecoveryActions.resolvedAt}::text, ${issueRecoveryActions.updatedAt}::text)::timestamptz - > ${at.toISOString()}::timestamptz - ${`${PENDING_CONTINUATION_BOUND_MS} milliseconds`}::interval`)); + > ${at.toISOString()}::timestamptz - ${`${PENDING_CONTINUATION_BOUND_MS} milliseconds`}::interval + or exists (${queuedRecoveryRunsOf(sql`${issueRecoveryActions.companyId}`, sql`${issueRecoveryActions.returnOwnerAgentId}`)}))`)); + // A restored task's wake that is waiting its turn behind its assignee's + // queued recovery runs (see HELD_RECOVERY_WAKE_KEYS) is its pending path too. + const heldWakes = await database.select({ issueId: issueRecoveryActions.sourceIssueId, agentId: issues.assigneeAgentId }) + .from(issueRecoveryActions) + .innerJoin(issues, and(eq(issues.companyId, issueRecoveryActions.companyId), eq(issues.id, issueRecoveryActions.sourceIssueId))) + .where(and( + eq(issueRecoveryActions.companyId, companyId), inArray(issueRecoveryActions.sourceIssueId, ids), + eq(issueRecoveryActions.status, "resolved"), + or(...HELD_RECOVERY_WAKE_KEYS.map(key => sql`${issueRecoveryActions.evidence} #>> array[${key}::text, 'wake'] = 'pending'`)), + sql`exists (${queuedRecoveryRunsOf(sql`${issues.companyId}`, sql`${issues.assigneeAgentId}`)})`)); const wakeCounts = counts(wakes); - for (const continuation of pendingContinuations) { + for (const continuation of [...pendingContinuations, ...heldWakes]) { const issue = issueRows.find(row => row.id === continuation.issueId); if (!issue || !continuation.agentId || issue.assigneeAgentId !== continuation.agentId) continue; wakeCounts.set(continuation.issueId, (wakeCounts.get(continuation.issueId) ?? 0) + 1); @@ -944,24 +957,6 @@ function interactionNotSupersededSql(companyId: SQLWrapper, issueIdText: SQLWrap and retry.created_at > ${createdAt})`; } -type StalledRunColumns = { id: SQLWrapper; companyId: SQLWrapper; agentId: SQLWrapper; status: SQLWrapper; createdAt: SQLWrapper }; - -function queuedRunStalledSql(run: StalledRunColumns, at: Date) { - const agedBefore = new Date(at.getTime() - QUEUED_RUN_AGING_MS).toISOString(); - return sql`(${run.status} = 'queued' and ${run.createdAt} <= ${agedBefore}::timestamptz - and not exists (select 1 from heartbeat_runs older where older.company_id = ${run.companyId} - and older.agent_id = ${run.agentId} and older.status = 'queued' and older.created_at < ${run.createdAt}) - and (select count(*) from (select 1 from heartbeat_runs started where started.company_id = ${run.companyId} - and started.agent_id = ${run.agentId} and started.id <> ${run.id} - and started.started_at > greatest(${run.createdAt} + ${`${QUEUED_RUN_AGING_MS} milliseconds`}::interval, - (select max(coalesce(prior.started_at, prior.finished_at)) from heartbeat_runs prior - where prior.company_id = ${run.companyId} and prior.agent_id = ${run.agentId} and prior.created_at < ${run.createdAt} - and (prior.started_at >= ${run.createdAt} or (prior.started_at is null and prior.finished_at >= ${run.createdAt})))) - limit ${OLDEST_DISPATCH_SHARE + 1}) passed_over) > ${OLDEST_DISPATCH_SHARE})`; -} - -const aliasedRun = (alias: string): StalledRunColumns => ({ id: sql.raw(`${alias}.id`), companyId: sql.raw(`${alias}.company_id`), - agentId: sql.raw(`${alias}.agent_id`), status: sql.raw(`${alias}.status`), createdAt: sql.raw(`${alias}.created_at`) }); /** The status a block replaced, from the activity entry that blocked the task: * the normal update route records it as `_previous.status`, recovery as diff --git a/server/src/services/recovery/legacy-workspace-rehome.ts b/server/src/services/recovery/legacy-workspace-rehome.ts index 733e425e791..12248901393 100644 --- a/server/src/services/recovery/legacy-workspace-rehome.ts +++ b/server/src/services/recovery/legacy-workspace-rehome.ts @@ -6,6 +6,7 @@ import { releaseAncestorWorkspaceBinding } from "../execution-workspaces.js"; import { issueService } from "../issues.js"; import { REMOTE_WORKSPACE_WRITER_PENDING_MESSAGE } from "../remote-workspace-writer-hold.js"; import { notifyStatusCardIssueChange } from "../status-card-notifications.js"; +import { agentRecoveryQueueIsFull } from "../recovery-queue-admission.js"; export const LEGACY_WORKSPACE_REHOME_SCHEMA = "paperclip.legacy-workspace-rehome.v1"; export const LEGACY_WORKSPACE_REHOME_OUTCOME = "legacy_shared_workspace_rehomed"; @@ -14,6 +15,7 @@ const BOARD_ESCALATION_POLICY = "board_escalation_no_takeover_v1"; const PAGE_SIZE = 25; const REHOME_WAKE_RETRY_BASE_MS = 5 * 60_000; const REHOME_WAKE_RETRY_MAX_MS = 60 * 60_000; +const RECOVERY_QUEUE_TURN_MS = 60_000; // A block set by the stranded escalation lands right after its failed run. const BLOCK_OWNERSHIP_SLACK_MS = 60_000; @@ -123,6 +125,15 @@ export async function deliverPendingRecoveryWakes(db: Db, wake: RehomeWakeup, sc eq(agentWakeupRequests.companyId, row.companyId), sql`${agentWakeupRequests.payload}->>'issueId' = ${row.issueId}`, sql`${agentWakeupRequests.payload}->>'recoveryActionId' = ${row.actionId}`, isNotNull(agentWakeupRequests.runId))).limit(1) : []; let accepted = !owed || Boolean(delivered); + // A few recovery runs per agent at a time: this one waits its turn, without + // counting as a refusal, and is offered again on the next pass. + if (!accepted && await agentRecoveryQueueIsFull(db, row.companyId, row.agentId!)) { + await db.update(issueRecoveryActions).set({ evidence: sql`jsonb_set(${issueRecoveryActions.evidence}, + array[${kind.key}::text, 'wakeRetryAt'], ${JSON.stringify(Date.now() + RECOVERY_QUEUE_TURN_MS)}::jsonb)` }) + .where(and(eq(issueRecoveryActions.companyId, row.companyId), eq(issueRecoveryActions.id, row.actionId), + sql`${wakeField(kind.key, "wake")} = 'pending'`)); + continue; + } try { // Only a created run settles the wake; a refusal (for example a paused // company or disabled on-demand wakes) returns null and is retried. diff --git a/server/src/services/recovery/stalled-queued-run-sql.ts b/server/src/services/recovery/stalled-queued-run-sql.ts new file mode 100644 index 00000000000..13e5552cfba --- /dev/null +++ b/server/src/services/recovery/stalled-queued-run-sql.ts @@ -0,0 +1,46 @@ +import { sql, type SQLWrapper } from "drizzle-orm"; +import { OLDEST_DISPATCH_SHARE, QUEUED_RUN_AGING_MS } from "../heartbeat/queued-run-dispatch-order.js"; +import { personOriginRunSql } from "../heartbeat/queued-run-line.js"; + +/** + * The assigned-work sweep's test for a queued run the dispatcher keeps passing + * over: its agent's oldest queued run, past the aging bound, with more than + * OLDEST_DISPATCH_SHARE later starts that the dispatcher ranked no higher. + */ +export type StalledRunColumns = { id: SQLWrapper; companyId: SQLWrapper; agentId: SQLWrapper; status: SQLWrapper; createdAt: SQLWrapper }; + +const UUID_TEXT = "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$"; + +/** + * A run people wait on, as the dispatcher ranks it: a person asked for it + * (itself or through its source chain, the dispatcher's own rule) or its task + * was created by a person. `alias` names a heartbeat_runs row in the enclosing query. + */ +function personWaitingRunSql(alias: string) { + const run = (column: string) => sql.raw(`${alias}.${column}`); + return sql`(${personOriginRunSql(sql`${run("company_id")}`, sql`${run("id")}`)} + or exists (select 1 from issues person_task where person_task.company_id = ${run("company_id")} + and person_task.id = case when ${run("context_snapshot")}->>'issueId' ~* ${UUID_TEXT} + then (${run("context_snapshot")}->>'issueId')::uuid end + and person_task.created_by_user_id is not null))`; +} + +export function queuedRunStalledSql(run: StalledRunColumns, at: Date) { + const agedBefore = new Date(at.getTime() - QUEUED_RUN_AGING_MS).toISOString(); + return sql`(${run.status} = 'queued' and ${run.createdAt} <= ${agedBefore}::timestamptz + and not exists (select 1 from heartbeat_runs older where older.company_id = ${run.companyId} + and older.agent_id = ${run.agentId} and older.status = 'queued' and older.created_at < ${run.createdAt}) + and (select count(*) from (select 1 from heartbeat_runs started where started.company_id = ${run.companyId} + and started.agent_id = ${run.agentId} and started.id <> ${run.id} + -- Work people wait on goes ahead of a run they do not by design; it does not pass that run over. + and not (${personWaitingRunSql("started")} and not exists (select 1 from heartbeat_runs waiting + where waiting.id = ${run.id} and ${personWaitingRunSql("waiting")})) + and started.started_at > greatest(${run.createdAt} + ${`${QUEUED_RUN_AGING_MS} milliseconds`}::interval, + (select max(coalesce(prior.started_at, prior.finished_at)) from heartbeat_runs prior + where prior.company_id = ${run.companyId} and prior.agent_id = ${run.agentId} and prior.created_at < ${run.createdAt} + and (prior.started_at >= ${run.createdAt} or (prior.started_at is null and prior.finished_at >= ${run.createdAt})))) + limit ${OLDEST_DISPATCH_SHARE + 1}) passed_over) > ${OLDEST_DISPATCH_SHARE})`; +} + +export const aliasedRun = (alias: string): StalledRunColumns => ({ id: sql.raw(`${alias}.id`), companyId: sql.raw(`${alias}.company_id`), + agentId: sql.raw(`${alias}.agent_id`), status: sql.raw(`${alias}.status`), createdAt: sql.raw(`${alias}.created_at`) }); From 833689a3fcd84c8968c15297231824c3c3de570d Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 20:22:07 -0600 Subject: [PATCH 03/10] fix(queue): say a long-queued turn is waiting to start, and name the routine execution that holds it (#21447) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #21405 made a task that waits for its agent's concurrency limit show "Waiting for · N slots max · P of Q in line · Configure" > - On hosted staging, after that deploy, a routine task stayed queued for 8 minutes while its agent had 19 free slots. Its chat said "Working for 7m · Thinking", and the server label was a plain "Queued" > - The real cause was the routine's one-execution-at-a-time rule: the previous execution of the same routine was still open with its run. That is correct scheduling, but the UI did not say it > - A queued turn also kept the startup feedback ("Working") for as long as it stayed queued, with no limit > - This pull request names the routine hold and limits the startup feedback to 30 seconds > - The benefit is that a queued task always says that it waits, and why when the server knows ## Linked Issues or Issue Description Refs #21405 **What happened?** On staging, one routine task's continuation run was queued at 17:39:03Z and claimed at 17:47:10Z. The agent ran 1 of 20 runs. The chat showed "Working for 7m · Thinking". The run's projection label was "Queued" with no cause. **Why did it happen?** 1. The routine allows one open execution with a run at a time (unique index `issues_open_routine_execution_uq`). The previous execution of the same routine was still open and held its run until 17:47:10.172Z. The waiting run was claimed 345 ms after that task closed. The claim was refused with "Another execution for this routine is already in progress" until then. This is the routine's intended policy, not a scheduling bug. 2. The execution projection did not name that hold, so the server label was "Queued". 3. The task chat keeps the startup feedback ("Working" and "Thinking") for a queued turn with no named hold (a DESIGN.md rule). That rule had no time limit. **Expected behavior** A routine task that waits for the routine's open execution says "Waiting for to finish". A queued turn that has no named hold says "Waiting to start" after a short grace period. ## What Changed - Server: `executionProjectionsForRuns` reads, in one query per batch, whether another open task of the same routine holds the routine's execution. For a never-started queued run that waits on it, the projection sets the cause `routine_execution_in_progress`, the label "Waiting for to finish", and a next action. This check comes before the agent-limit check, because a free slot would not start the run. - UI: a new hook, `useWaitingToStart`, keeps the startup feedback for 30 seconds. After that, a queued turn with no named hold shows "Waiting to start": an hourglass, no spinner, no shimmer. The task chat's runner header, its current-activity tail and the live run pill use it. The hook uses one timer at the threshold and no per-second ticks. - DESIGN.md: the startup-feedback rule now has the 30-second limit and the routine hold. - Storybook: `WaitingToStart` and `WaitingForRoutine` stories, with play checks. **Why 30 seconds:** DESIGN.md keeps "Working" from message submission through admission, so that a normal start does not flicker. On staging, runs that were not held started 3 to 8 seconds after they were queued (from the company's run list between 17:30 and 17:48Z). 30 seconds is several times that, and shorter than any real wait the user should be told about. ## Verification - Red/green server test (embedded Postgres), `execution-projection-agent-queue.test.ts`: a routine task waits while another open task of the same routine holds a run. The test checks the label "Waiting for Q-3093 to finish" and the cause. It checks that the holder is not marked as waiting on itself, and that the label returns to plain "Queued" when the holder closes. It fails without the server change. - Red/green UI tests: `TaskChatRunnerTurn.test.tsx` (startup feedback until the threshold, then "Waiting to start" in the header and the tail; a running turn is never called waiting) and `TaskChatLiveRunPill.test.tsx` (the label changes, with no spinner or shimmer). Both fail without the UI change. - `vitest run` on the four `execution-projection-*` server suites (36 tests) and all `task-chat` and `TaskChatThread` UI suites (1,414 tests). UI and server typecheck. - The new stories render in headless Chromium. - On staging, before this change, the #21405 waiting line was confirmed with a temporary 1-slot agent: "Waiting for Queue Check (temp) · 1 slot max · 2 of 2 in line · Configure". The temporary tasks and agent were cancelled and terminated. - After deploy: confirm the new labels on staging. ## Risks - Low risk. There is no schema change. - One extra query per projection batch that includes a never-started queued run on a routine task. It is a join on the company's routine-execution tasks through the existing `issues_company_origin_idx` index. - A queued turn that is admitted after 30 seconds, for a reason the server does not name, now says "Waiting to start" instead of "Working". This is the intended change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, Chrome for staging checks). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- DESIGN.md | 5 +- .../execution-projection-agent-queue.test.ts | 38 +++++++++++++++ server/src/services/execution-projection.ts | 36 ++++++++++++++ .../task-chat/TaskChatLiveRunPill.test.tsx | 14 ++++++ .../task-chat/TaskChatLiveRunPill.tsx | 15 ++++-- .../task-chat/TaskChatRunnerTurn.test.tsx | 48 +++++++++++++++++++ .../task-chat/TaskChatRunnerTurn.tsx | 21 ++++++-- .../components/task-chat/useWaitingToStart.ts | 29 +++++++++++ .../agent-queue-wait.stories.tsx | 20 ++++++++ 9 files changed, 216 insertions(+), 10 deletions(-) create mode 100644 ui/src/components/task-chat/useWaitingToStart.ts diff --git a/DESIGN.md b/DESIGN.md index 707a877b75e..aa2eba46b29 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -149,7 +149,10 @@ Runtime discovery must not replace the turn with a generic spinner or a A queued turn shows the same `Working` header and shimmering `Thinking` activity as a running turn, starting at message submission. Keep specific server holds, -such as `Waiting for maintenance` or agent setup, visible during admission. +such as `Waiting for maintenance`, agent setup, or another open task of the same +routine, visible during admission. Admission normally starts a run within seconds, +so this startup feedback lasts at most 30 seconds: a turn still queued after that +with no named hold says `Waiting to start`, without spinner or shimmer. A run that waits for one of its agent's concurrency slots is a hold too: show `Waiting for · N slots max · P of Q in line · Configure` (the shared `AgentQueueWait` component) with no spinner or elapsed timer, link Configure to the diff --git a/server/src/services/execution-projection-agent-queue.test.ts b/server/src/services/execution-projection-agent-queue.test.ts index b1e31cce202..ea86124347b 100644 --- a/server/src/services/execution-projection-agent-queue.test.ts +++ b/server/src/services/execution-projection-agent-queue.test.ts @@ -80,6 +80,44 @@ const minutesAgo = (minutes: number) => new Date(now.getTime() - minutes * 60_00 }); }); + it("names the routine's open execution that holds a queued routine task, before the agent's limit", async () => { + const f = await fixture({ maxConcurrentRuns: 20, running: 0 }); + const routineId = randomUUID(); + const holderIssueId = randomUUID(), holderRunId = randomUUID(); + const waitingIssueId = randomUUID(), waitingRunId = randomUUID(); + await db.insert(heartbeatRuns).values({ id: holderRunId, companyId: f.companyId, agentId: f.agentId, status: "running", + startedAt: minutesAgo(18), createdAt: minutesAgo(30), issueId: holderIssueId, contextSnapshot: { issueId: holderIssueId } }); + await db.insert(issues).values([ + { id: holderIssueId, companyId: f.companyId, identifier: "Q-3093", title: "Nightly test gardening", status: "in_progress", + assigneeAgentId: f.agentId, originKind: "routine_execution", originId: routineId, executionRunId: holderRunId }, + { id: waitingIssueId, companyId: f.companyId, identifier: "Q-3636", title: "Nightly test gardening", status: "in_progress", + assigneeAgentId: f.agentId, originKind: "routine_execution", originId: routineId }, + ]); + await db.insert(heartbeatRuns).values({ id: waitingRunId, companyId: f.companyId, agentId: f.agentId, status: "queued", + createdAt: minutesAgo(8), issueId: waitingIssueId, contextSnapshot: { issueId: waitingIssueId } }); + + expect((await executionProjectionsForRuns(db, f.companyId, [waitingRunId], now)).get(waitingRunId)).toMatchObject({ + phase: "queued", + label: "Waiting for Q-3093 to finish", + cause: "routine_execution_in_progress", + }); + // The holder's own run is not waiting on itself. + expect((await executionProjectionsForRuns(db, f.companyId, [holderRunId], now)).get(holderRunId)?.cause).not.toBe("routine_execution_in_progress"); + + // A private holder is not named to readers of the waiting task. + await db.update(issues).set({ visibility: "private" }).where(eq(issues.id, holderIssueId)); + const privateHold = (await executionProjectionsForRuns(db, f.companyId, [waitingRunId], now)).get(waitingRunId); + expect(privateHold).toMatchObject({ label: "Waiting for another task in this routine to finish", cause: "routine_execution_in_progress" }); + expect(JSON.stringify(privateHold)).not.toContain("Q-3093"); + await db.update(issues).set({ visibility: "open" }).where(eq(issues.id, holderIssueId)); + + // Once the holder closes, the routine slot is free and the run is plainly queued. + await db.update(issues).set({ status: "done" }).where(eq(issues.id, holderIssueId)); + expect((await executionProjectionsForRuns(db, f.companyId, [waitingRunId], now)).get(waitingRunId)).toMatchObject({ + label: "Queued", cause: null, + }); + }); + it("leaves out queued runs that a preview worktree's cutoff keeps from starting", async () => { const f = await fixture({ maxConcurrentRuns: 1, running: 1 }); const copiedOld = await f.queue({ priority: "critical", createdMinutesAgo: 120 }); diff --git a/server/src/services/execution-projection.ts b/server/src/services/execution-projection.ts index 714cc9546d8..0954d4dcf44 100644 --- a/server/src/services/execution-projection.ts +++ b/server/src/services/execution-projection.ts @@ -24,6 +24,7 @@ import { isAgentAwaitingSetup } from "../modules/agent-lifecycle/index.js"; import { NATIVE_RESPONSE_WAKE_WAITING_REASON } from "./native-runtime/status-arbiter.js"; import { agentQueueLinesForRuns } from "./heartbeat/queued-run-line.js"; const MAX_PREDECESSOR_CLEANUP_CHECKS = 16; +const ANOTHER_ROUTINE_TASK = "another task in this routine"; const text = (v: unknown) => (typeof v === "string" ? v : null); const executionRunColumns = { id: heartbeatRuns.id, @@ -211,6 +212,32 @@ export async function executionProjectionsForRuns( : new Map>(); const issueIdOf = (run: Run) => run.nativeIssueId ?? text(run.contextSnapshot?.issueId); const queuedIssueIds = [...new Set(queuedRuns.map(issueIdOf).filter((id): id is string => !!id))]; + // A routine runs one execution at a time (issues_open_routine_execution_uq): + // a never-started run of a routine task waits while another open task of + // the same routine holds an execution. One read for the batch. + const routineHolderByIssue = new Map(); + const unstartedIssueIds = [...new Set(queuedRuns.filter(run => !run.startedAt).map(issueIdOf).filter((id): id is string => !!id))]; + if (unstartedIssueIds.length) { + const holders = await read(async () => db.execute<{ issueId: string; holderIdentifier: string | null }>(sql` + -- A private holder is never named: this projection has no reader to check access for. + select waiting.id as "issueId", case when holder.visibility = 'open' then holder.identifier end as "holderIdentifier" + from ${issues} waiting + join ${issues} holder on holder.company_id = waiting.company_id + and holder.origin_kind = 'routine_execution' + and holder.origin_id = waiting.origin_id + and holder.origin_fingerprint is not distinct from waiting.origin_fingerprint + and holder.id <> waiting.id + and holder.hidden_at is null + and holder.execution_run_id is not null + and holder.status in ('backlog', 'todo', 'in_progress', 'in_review', 'blocked') + where waiting.company_id = ${companyId} + and waiting.origin_kind = 'routine_execution' + and waiting.origin_id is not null + and waiting.id in (${sql.join(unstartedIssueIds.map(id => sql`${id}::uuid`), sql`, `)})`)); + for (const row of [...holders]) { + if (!routineHolderByIssue.has(row.issueId)) routineHolderByIssue.set(row.issueId, row.holderIdentifier ?? ANOTHER_ROUTINE_TASK); + } + } if (queuedIssueIds.length) { const issueKey = sql`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId')`; // Two set-based reads for the whole batch, then one bounded, parallel @@ -285,6 +312,7 @@ export async function executionProjectionsForRuns( setupByAgent.get(run.agentId), currentResponseWait(run), agentQueueByRun.get(run.id), + run.status === "queued" && !run.startedAt ? routineHolderByIssue.get(issueIdOf(run) ?? "") : undefined, ), ); } @@ -313,6 +341,8 @@ export function projectExecution( setupAgent?: { lifecycleState: string; lifecycleHolds: readonly string[]; retryRequired?: boolean }, responseWaitCurrent = false, agentQueue?: ExecutionProjection["agentQueue"], + /** The open task of the same routine that holds the routine's one execution. */ + routineHolder?: string, ): ExecutionProjection { const detail = coordinator?.failureDetail ?? {}; const successorRunId = text(detail.successorRunId); @@ -561,6 +591,12 @@ export function projectExecution( projection.nextAction = "The previous run on this task is still releasing its runner. This turn starts automatically when it settles."; return set("queued", "Waiting for previous run to stop"); } + // Checked before the agent's limit: a free slot would not start this run. + if (run.status === "queued" && !run.startedAt && routineHolder) { + projection.cause = "routine_execution_in_progress"; + projection.nextAction = `This routine runs one task at a time and ${routineHolder === ANOTHER_ROUTINE_TASK ? "another of its tasks" : routineHolder} is still open with its run. This task starts automatically when that one finishes.`; + return set("queued", `Waiting for ${routineHolder} to finish`); + } if (run.status === "queued" && !run.startedAt && agentQueue) { projection.cause = "agent_at_capacity"; projection.agentQueue = agentQueue; diff --git a/ui/src/components/task-chat/TaskChatLiveRunPill.test.tsx b/ui/src/components/task-chat/TaskChatLiveRunPill.test.tsx index 70559fab815..869f33a7f7f 100644 --- a/ui/src/components/task-chat/TaskChatLiveRunPill.test.tsx +++ b/ui/src/components/task-chat/TaskChatLiveRunPill.test.tsx @@ -120,6 +120,20 @@ describe("TaskChatLiveRunPill", () => { expect(container.querySelector("a")?.getAttribute("href")).toBe("/agents/codie/advanced"); }); + it("says a queued run with no named hold is waiting to start once the startup grace passes", () => { + vi.useFakeTimers(); + vi.setSystemTime(new Date("2026-11-13T17:46:13Z")); + const queuedAt = Date.now() - 5_000; + const queued = { phase: "queued", label: "Queued", cause: null } as ExecutionProjection; + act(() => root.render()); + expect(container.textContent).toContain("Working"); + act(() => vi.advanceTimersByTime(26_000)); + expect(container.textContent).toContain("Waiting to start"); + expect(container.textContent).not.toContain("Working"); + expect(container.querySelector(".animate-spin")).toBeNull(); + expect(container.querySelector(".shimmer-text")).toBeNull(); + }); + it("names another server hold on a queued run instead of Working", () => { const execution = { phase: "queued", label: "Waiting for maintenance", cause: "task_drain" } as ExecutionProjection; act(() => root.render()); diff --git a/ui/src/components/task-chat/TaskChatLiveRunPill.tsx b/ui/src/components/task-chat/TaskChatLiveRunPill.tsx index e0e4194a44d..df953d50a7b 100644 --- a/ui/src/components/task-chat/TaskChatLiveRunPill.tsx +++ b/ui/src/components/task-chat/TaskChatLiveRunPill.tsx @@ -1,5 +1,5 @@ import type { ExecutionProjection } from "@paperclipai/shared"; -import { Loader2 } from "lucide-react"; +import { Hourglass, Loader2 } from "lucide-react"; import type { TranscriptEntry } from "../../adapters"; import { cn } from "@/lib/utils"; import { useSecondTick } from "@/hooks/useSecondTick"; @@ -9,6 +9,7 @@ import { isTerminalRunStatus } from "@/components/task-chat/transcript-adapter"; import { useTaskChatShimmer } from "./TaskChatShimmer"; import { AgentQueueWait } from "@/components/AgentQueueWait"; import { agentQueueWaitOf } from "@/lib/agent-queue-wait"; +import { WAITING_TO_START_LABEL, useWaitingToStart } from "./useWaitingToStart"; /** * "ran N commands, called M tools" for the live tail's status pill, counted off @@ -66,6 +67,8 @@ export function TaskChatLiveRunPill({ const agentQueue = status === "queued" ? agentQueueWaitOf(execution) : null; // A specific server hold (maintenance, agent setup) names the wait instead of "Working". const holdLabel = status === "queued" && execution?.cause ? execution.label : null; + // Past the startup grace period with no named hold, the turn says it waits. + const waitingToStart = useWaitingToStart(status === "queued" && !holdLabel && !agentQueue, startedAtMs); // One shared page-wide ticker drives the live elapsed readout, matching the // default view's `useLiveElapsed`. useSecondTick(active && startedAtMs != null && !agentQueue); @@ -76,7 +79,7 @@ export function TaskChatLiveRunPill({ ? formatDurationWords(elapsedMs) : null; const failed = ["failed", "timed_out", "cancelled", "interrupted"].includes(status); - const verb = active ? holdLabel ?? "Working" : failed ? "Stopped" : "Worked"; + const verb = active ? holdLabel ?? (waitingToStart ? WAITING_TO_START_LABEL : "Working") : failed ? "Stopped" : "Worked"; const suffix = elapsed ? `for ${elapsed}` : null; if (agentQueue) { @@ -93,7 +96,11 @@ export function TaskChatLiveRunPill({ data-testid="task-chat-live-run-pill" > - {active ? ( + {active && waitingToStart ? ( + + + ) : active ? ( @@ -102,7 +109,7 @@ export function TaskChatLiveRunPill({ )} - {active ? {verb} : verb} + {active && !waitingToStart ? {verb} : verb} {suffix ? {suffix} : null} {toolSummary ? · {toolSummary} : null} diff --git a/ui/src/components/task-chat/TaskChatRunnerTurn.test.tsx b/ui/src/components/task-chat/TaskChatRunnerTurn.test.tsx index 1c050084bbb..0864dfd42a7 100644 --- a/ui/src/components/task-chat/TaskChatRunnerTurn.test.tsx +++ b/ui/src/components/task-chat/TaskChatRunnerTurn.test.tsx @@ -111,6 +111,54 @@ describe("TaskChatRunnerTurn", () => { expect(container.textContent).not.toContain("Thinking"); }); + it("keeps startup feedback briefly, then says a queued turn with no named hold is waiting to start", () => { + vi.useFakeTimers(); + try { + vi.setSystemTime(new Date("2026-11-13T17:39:05Z")); + render([], "queued"); + expect(container.querySelector('[data-testid="task-chat-turn-status-header"]')?.textContent).toContain("Working"); + expect(container.querySelector('[data-testid="task-chat-current-activity-label"]')?.textContent).toBe("Thinking"); + // The fixture's run was queued 2 s before render: the grace ends at +28 s. + act(() => { vi.advanceTimersByTime(27_000); }); + expect(container.textContent).not.toContain("Waiting to start"); + act(() => { vi.advanceTimersByTime(2_000); }); + expect(container.querySelector('[data-testid="task-chat-turn-status-header"]')?.textContent).toContain("Waiting to start"); + expect(container.querySelector('[data-testid="task-chat-current-activity-label"]')?.textContent).toBe("Waiting to start"); + expect(container.textContent).not.toContain("Working"); + } finally { + vi.useRealTimers(); + } + }); + + it("does not repeat the agent's waiting line in the turn body", () => { + // The shared header (rendered by the thread) shows the waiting line; the body adds no second line. + act(() => root.render( + + + , + )); + expect(container.textContent).not.toContain("Waiting for Codie"); + expect(container.querySelector('[data-testid="task-chat-current-activity"]')).toBeNull(); + }); + + it("does not call a running turn waiting, however long it runs", () => { + vi.useFakeTimers(); + try { + render([], "running"); + act(() => { vi.advanceTimersByTime(10 * 60_000); }); + expect(container.textContent).not.toContain("Waiting to start"); + expect(container.querySelector('[data-testid="task-chat-turn-status-header"]')?.textContent).toContain("Working"); + } finally { + vi.useRealTimers(); + } + }); + const resolvedQuestion = ( id: string, prompt: string, diff --git a/ui/src/components/task-chat/TaskChatRunnerTurn.tsx b/ui/src/components/task-chat/TaskChatRunnerTurn.tsx index a72c4862cb3..3b1b6780087 100644 --- a/ui/src/components/task-chat/TaskChatRunnerTurn.tsx +++ b/ui/src/components/task-chat/TaskChatRunnerTurn.tsx @@ -1,6 +1,7 @@ import { groupImageResources } from "./image-resources"; import { AgentQueueWait } from "@/components/AgentQueueWait"; import { agentQueueWaitOf } from "@/lib/agent-queue-wait"; +import { WAITING_TO_START_LABEL, useWaitingToStart } from "./useWaitingToStart"; import { McpAppTurnViews, mcpAppRunRefreshKey } from "@/components/mcp-app/McpAppTurnViews"; import { ToolCallProgressRunProvider } from "./tool-call-progress"; import { useRef, type ReactNode } from "react"; @@ -135,11 +136,17 @@ interface RunnerHeaderProps { continuedAfterSteering?: boolean; } -function runnerWaitingLabel({ runId, status, execution, preparationLabel }: RunnerHeaderProps) { +function runnerWaitingLabel({ runId, status, execution, preparationLabel }: RunnerHeaderProps, waitingToStart = false) { if (isTerminalRunStatus(status)) return null; if (execution?.phase === "waiting_for_provider") return execution.label; if (!runId && preparationLabel) return preparationLabel; - return status === "queued" && execution?.cause ? execution.label : null; + if (status === "queued" && execution?.cause) return execution.label; + return waitingToStart ? WAITING_TO_START_LABEL : null; +} + +/** A run's queued turn with no named hold, past the startup grace period. */ +function useRunnerWaitingToStart({ runId, status, execution, startedAtMs }: Pick) { + return useWaitingToStart(status === "queued" && !!runId && !execution?.cause, startedAtMs); } function agentQueueWaitFor({ status, execution }: Pick) { @@ -148,10 +155,11 @@ function agentQueueWaitFor({ status, execution }: Pick {props.agentName ? : null} {statusContent ?? (agentQueueWaitFor(props) ? : null) ?? } + continuedAfterSteering={props.continuedAfterSteering} waitingLabel={runnerWaitingLabel(props, waitingToStart)} />}
; } @@ -207,7 +215,8 @@ export function TaskChatRunnerTurn({ }) { const { failures: mcpStartupFailures, rest: turnItems } = partitionRunHealthItems(items); const terminal = isTerminalRunStatus(status); - const waitingLabel = runnerWaitingLabel({ runId, status, execution, preparationLabel, startedAtMs }); + const waitingToStart = useRunnerWaitingToStart({ runId, status, execution, startedAtMs }); + const waitingLabel = runnerWaitingLabel({ runId, status, execution, preparationLabel, startedAtMs }, waitingToStart); const yielded = turnItems.some( (item) => item.kind === "protocol" && @@ -329,7 +338,9 @@ export function TaskChatRunnerTurn({ />
) : null} - {!final && currentActivityItems.length === 0 ? : null} + {/* The header's waiting line already names the agent's queue; no second line. */} + {!final && currentActivityItems.length === 0 && !agentQueueWaitFor({ status, execution }) + ? : null}
); } diff --git a/ui/src/components/task-chat/useWaitingToStart.ts b/ui/src/components/task-chat/useWaitingToStart.ts new file mode 100644 index 00000000000..68ce4ba4640 --- /dev/null +++ b/ui/src/components/task-chat/useWaitingToStart.ts @@ -0,0 +1,29 @@ +import { useEffect, useReducer } from "react"; + +/** + * How long a queued turn keeps the startup feedback ("Working" / "Thinking") + * before it says it is waiting. DESIGN.md keeps that feedback from message + * submission through admission so a normal start does not flicker. Admission + * normally claims a run within a few seconds; a run still queued after 30 s + * with no named hold is not starting, and saying "Working" would be untrue. + */ +export const QUEUED_START_GRACE_MS = 30_000; +export const WAITING_TO_START_LABEL = "Waiting to start"; + +/** + * True once an eligible queued turn has waited past the grace period since + * `sinceMs`. One timer at the threshold; no per-second ticking. + */ +export function useWaitingToStart(eligible: boolean, sinceMs: number | null): boolean { + const [, rerender] = useReducer((count: number) => count + 1, 0); + const remainingMs = eligible && sinceMs != null ? QUEUED_START_GRACE_MS - (Date.now() - sinceMs) : null; + const pending = remainingMs != null && remainingMs > 0; + useEffect(() => { + if (!eligible || sinceMs == null) return; + const delay = QUEUED_START_GRACE_MS - (Date.now() - sinceMs); + if (delay <= 0) return; + const timer = setTimeout(rerender, delay + 50); + return () => clearTimeout(timer); + }, [eligible, sinceMs, pending]); + return remainingMs != null && remainingMs <= 0; +} diff --git a/ui/storybook/stories/runs-adapters/agent-queue-wait.stories.tsx b/ui/storybook/stories/runs-adapters/agent-queue-wait.stories.tsx index 0c4aaaa2e1a..1abe417eb4e 100644 --- a/ui/storybook/stories/runs-adapters/agent-queue-wait.stories.tsx +++ b/ui/storybook/stories/runs-adapters/agent-queue-wait.stories.tsx @@ -103,3 +103,23 @@ export const SingleSlot: Story = { }; export const Mobile: Story = { globals: { viewport: { value: "mobile", isRotated: false } } }; + +/** A queued turn with no named hold, past the 30-second startup feedback. */ +export const WaitingToStart: Story = { + render: () => , + play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + await expect(canvas.getByText("Waiting to start")).toBeVisible(); + await expect(canvas.queryByText("Working")).not.toBeInTheDocument(); + }, +}; + +/** A routine task that waits for the routine's open execution. */ +export const WaitingForRoutine: Story = { + render: () => , + play: async ({ canvasElement }) => { + await expect(within(canvasElement).getByText(/Waiting for PAP-3093 to finish/)).toBeVisible(); + }, +}; From dc018f9bc0a4ae526351722fb1782577d0fd0bb3 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Fri, 13 Nov 2026 18:45:20 -0800 Subject: [PATCH 04/10] Keep agent-owned connections healthy during catalog refresh (#21539) Use dedicated agent grants for catalog refresh, GitHub health checks, and reconnect status. Preserve shared connection health when a different agent lacks authorization. Add PostgreSQL and HTTP regressions for credential selection, stale health recovery, caller isolation, and actual provider rejection. Correct the MCP demo approval fixture to submit a valid saved-card option through normal answer validation. Validation: 629 focused tests, server typecheck, all CI checks, and Greptile 5/5 with no unresolved review comments. Co-Authored-By: Paperclip --- doc/connections/CONNECTOR-PLAYBOOK.md | 10 + ...ent-connection-catalog.integration.test.ts | 214 ++++++++++++++++++ .../__tests__/github-health-recovery.test.ts | 1 + .../__tests__/mcp-tasks-demo-server.test.ts | 13 +- server/src/services/tool-access.ts | 8 +- .../services/tool-access/catalog/failures.ts | 10 +- .../services/tool-access/catalog/health.ts | 10 +- .../tool-access/catalog/publication.ts | 3 +- .../services/tool-access/credential-access.ts | 27 +++ 9 files changed, 284 insertions(+), 12 deletions(-) create mode 100644 server/src/__tests__/agent-connection-catalog.integration.test.ts diff --git a/doc/connections/CONNECTOR-PLAYBOOK.md b/doc/connections/CONNECTOR-PLAYBOOK.md index a5df04b1c6f..26eaaadc723 100644 --- a/doc/connections/CONNECTOR-PLAYBOOK.md +++ b/doc/connections/CONNECTOR-PLAYBOOK.md @@ -181,6 +181,16 @@ refs and the same ownership checks. A personal resolver must not fall back to a company credential. Ownership mismatch is `grant_credential_invalid` and tells the owner to reconnect. +Dedicated agent connections keep credentials on their agent grant. Background +catalog and health work may use a single active grant for that connection and +company. An agent caller must use its own grant. Missing, revoked, or ambiguous +agent authorizations fail before provider requests; they never fall back to a +personal, organization, or connection-owned credential. Renaming or pausing the +connection must preserve the grant and its secret bindings. A later successful +catalog refresh clears an earlier connection-health error without new consent. +An agent caller denied for lacking its own grant must not change shared health +or the catalog, including when that agent has permission to manage the connection. + Existing personal connections with company-scoped invocation credentials require their owner to reconnect and enter a fresh key or secret URL. Reconnect creates a user-owned credential and updates the grant and declarations while preserving diff --git a/server/src/__tests__/agent-connection-catalog.integration.test.ts b/server/src/__tests__/agent-connection-catalog.integration.test.ts new file mode 100644 index 00000000000..e92d11bafaa --- /dev/null +++ b/server/src/__tests__/agent-connection-catalog.integration.test.ts @@ -0,0 +1,214 @@ +import { randomUUID } from "node:crypto"; +import { createServer, type Server } from "node:http"; +import type { AddressInfo } from "node:net"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { agents, authUsers, companies, companyMemberships, companySecretBindings, connectionGrants, createDb, principalPermissionGrants, toolApplications, toolCatalogEntries, toolConnections } from "@paperclipai/db"; +import { and, eq } from "drizzle-orm"; +import { syncConnectionCredentialBindings } from "../services/connection-credential-bindings.js"; +import { connectionManagementService } from "../services/connection-management.js"; +import { secretService } from "../services/secrets.js"; +import { toolAccessService } from "../services/tool-access.js"; +import type { PaperclipCloudConnector } from "../services/paperclip-cloud-connector.js"; +import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; + +describe("dedicated agent connection discovery", () => { + let database: Awaited>; + let db: ReturnType; + let provider: Server; + let origin: string; + let requests: Array; + let rejectCredential: boolean; + + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("agent-connection-catalog-"); + db = createDb(database.connectionString); + provider = createServer(async (req, res) => { + if (req.method !== "POST") { res.writeHead(404); res.end(); return; } + const chunks: Buffer[] = []; + for await (const chunk of req) chunks.push(Buffer.from(chunk)); + const message = JSON.parse(Buffer.concat(chunks).toString()); + requests.push(req.headers.authorization); + if (rejectCredential || req.headers.authorization !== "Bearer fixture-agent-access") { + res.writeHead(401, { "www-authenticate": "Bearer", "content-type": "application/json" }); + res.end(JSON.stringify({ error: "unauthorized" })); + return; + } + if (message.method === "notifications/initialized") { + res.writeHead(202); res.end(); return; + } + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ jsonrpc: "2.0", id: message.id, result: message.method === "initialize" + ? { protocolVersion: "2025-03-26", capabilities: { tools: {} }, serverInfo: { name: "fixture", version: "1" } } + : { tools: [{ name: "get_fixture", description: "Read a fixture", inputSchema: { type: "object" }, annotations: { readOnlyHint: true } }] } })); + }); + await new Promise(resolve => provider.listen(0, "127.0.0.1", resolve)); + origin = `http://127.0.0.1:${(provider.address() as AddressInfo).port}`; + }); + + beforeEach(() => { + requests = []; + rejectCredential = false; + const localFetch = globalThis.fetch; + vi.spyOn(globalThis, "fetch").mockImplementation(async (url, init) => { + const href = String(url); + if (href.startsWith(`${origin}/`)) return localFetch(url, init); + if (href === "https://api.github.com/user") return Response.json({ id: 42, login: "fixture-bot" }); + if (href.startsWith("https://api.github.com/user/installations?")) return Response.json({ installations: [{ id: 101, repository_selection: "selected", account: { login: "example" } }] }); + if (href.startsWith("https://api.github.com/user/installations/101/repositories?")) return Response.json({ total_count: 1, repositories: [{ id: 1, full_name: "example/fixture" }] }); + throw new Error("Unexpected provider request"); + }); + }); + afterEach(() => vi.restoreAllMocks()); + afterAll(async () => { + if (provider) await new Promise(resolve => provider.close(() => resolve())); + await db?.$client.end({ timeout: 1 }); + await database?.cleanup(); + }); + + async function fixture() { + const [company] = await db.insert(companies).values({ name: "Agent catalog fixture", issuePrefix: `AC${randomUUID().slice(0, 6)}` }).returning(); + const [agent, otherAgent] = await db.insert(agents).values(["Owner", "Other"].map(name => ({ companyId: company.id, name, role: "engineer", adapterType: "process" }))).returning(); + const [application] = await db.insert(toolApplications).values({ companyId: company.id, name: "GitHub", type: "mcp_http" }).returning(); + const [connection] = await db.insert(toolConnections).values({ + companyId: company.id, applicationId: application.id, name: "GitHub", uid: randomUUID(), + transport: "mcp_remote", authKind: "oauth", credentialPolicy: "per_agent", status: "active", enabled: true, + healthStatus: "failed", healthMessage: "OAuth credentials have expired and need to be reconnected.", + config: { sourceTemplateKey: "github", url: `${origin}/mcp`, oauth: { + strategy: "paperclip_cloud_connector", provider: "github", connectorProfile: "github.code", + tokenUrl: "https://github.com/login/oauth/access_token", expiresAt: null, + } }, + }).returning(); + const secret = await secretService(db).create(company.id, { name: "Agent access", key: randomUUID(), provider: "local_encrypted", value: "fixture-agent-access" }); + const [grant] = await db.insert(connectionGrants).values({ + companyId: company.id, connectionId: connection.id, kind: "agent", subjectAgentId: agent.id, status: "active", + credentialSecretRefs: [{ secretId: secret.id, configPath: "oauth.access_token", versionSelector: "latest", required: true }], + providerTenant: { oauth: { strategy: "paperclip_cloud_connector", accessTokenExpiresAt: null }, + github: { userId: "42", login: "fixture-bot", installationCount: 1, repositoryCount: 1, repositorySelection: "selected", installationIds: ["101"], installationOwnerLogins: ["example"] } }, + }).returning(); + await syncConnectionCredentialBindings(db, connection); + const connector = { setWebhookBinding: vi.fn(async () => undefined) } as unknown as PaperclipCloudConnector; + const service = toolAccessService(db, { deploymentMode: "local_trusted", deploymentExposure: "private", paperclipCloudConnector: connector }); + return { company, agent, otherAgent, connection, grant, secret, service }; + } + + it.each(["background", "system", "operator", "agent"] as const)("renames and refreshes an agent-owned GitHub connection without reconnect (%s)", async mode => { + const f = await fixture(); + const actor = mode === "background" ? undefined : mode === "system" + ? { actorType: "system" as const } : mode === "operator" + ? { actorType: "user" as const, actorId: "fixture-operator" } + : { actorType: "agent" as const, actorId: f.agent.id }; + await f.service.updateConnection(f.connection.id, { name: "Renamed bot" }); + const refreshed = await f.service.refreshCatalog(f.connection.id, actor); + expect(refreshed.connection).toMatchObject({ name: "Renamed bot", healthStatus: "ok", enabled: true }); + expect((await f.service.getConnection(f.connection.id, f.company.id)).requiresReauthorization).toBe(false); + expect(refreshed.catalog.map(tool => tool.toolName)).toEqual(["get_fixture"]); + expect(requests.length).toBeGreaterThan(0); + expect(requests.every(header => header === "Bearer fixture-agent-access")).toBe(true); + await expect(f.service.checkHealth(f.connection.id, actor)).resolves.toMatchObject({ connection: { healthStatus: "ok" } }); + const [grant] = await db.select().from(connectionGrants).where(eq(connectionGrants.id, f.grant.id)); + expect(grant.status).toBe("active"); + expect(grant.credentialSecretRefs).toEqual(f.grant.credentialSecretRefs); + const bindings = await db.select().from(companySecretBindings).where(and(eq(companySecretBindings.targetId, f.connection.id), eq(companySecretBindings.secretId, f.secret.id))); + expect(bindings).toHaveLength(1); + }); + + describe.each(["refreshCatalog", "checkHealth"] as const)("%s caller isolation", operation => { + it.each(["another agent", "another company", "missing agent identity"])("denies %s without changing the owner's connection", async scope => { + const f = await fixture(); + const actorId = scope === "another company" ? (await fixture()).agent.id : scope === "missing agent identity" ? null : f.otherAgent.id; + await f.service.refreshCatalog(f.connection.id, { actorType: "agent", actorId: f.agent.id }); + const before = await db.select().from(toolConnections).where(eq(toolConnections.id, f.connection.id)); + const catalog = await db.select().from(toolCatalogEntries).where(eq(toolCatalogEntries.connectionId, f.connection.id)); + requests = []; + vi.mocked(fetch).mockClear(); + await expect(f.service[operation](f.connection.id, { actorType: "agent", actorId })) + .rejects.toMatchObject({ status: 422, details: { code: "agent_authorization_required" } }); + expect(requests).toEqual([]); + expect(fetch).not.toHaveBeenCalled(); + expect(await db.select().from(toolConnections).where(eq(toolConnections.id, f.connection.id))).toEqual(before); + expect(await db.select().from(toolCatalogEntries).where(eq(toolCatalogEntries.connectionId, f.connection.id))).toEqual(catalog); + await expect(f.service.checkHealth(f.connection.id, { actorType: "agent", actorId: f.agent.id })) + .resolves.toMatchObject({ connection: { healthStatus: "ok" } }); + }); + + it("keeps the owner's tools healthy when a permitted managing agent lacks the credential", async () => { + const f = await fixture(); + const userId = randomUUID(); + await db.insert(authUsers).values({ id: userId, name: "Manager", email: `${userId}@fixture.invalid`, createdAt: new Date(), updatedAt: new Date() }); + await db.insert(companyMemberships).values([ + { companyId: f.company.id, principalType: "user", principalId: userId, membershipRole: "owner", status: "active" }, + { companyId: f.company.id, principalType: "agent", principalId: f.otherAgent.id, status: "active" }, + ]); + await db.insert(principalPermissionGrants).values([ + { companyId: f.company.id, principalType: "agent", principalId: f.otherAgent.id, permissionKey: "tools:manage_connections" }, + { companyId: f.company.id, principalType: "user", principalId: userId, permissionKey: "tools:manage_connections" }, + ]); + const managementContext = await connectionManagementService(db).authorize({ + type: "agent", agentId: f.otherAgent.id, companyId: f.company.id, onBehalfOfUserId: userId, + }, "connections:configure", { type: "connection_management", companyId: f.company.id, connectionId: f.connection.id, operation }); + expect(managementContext.decision.allowed).toBe(true); + await f.service.refreshCatalog(f.connection.id, { actorType: "agent", actorId: f.agent.id }); + const before = await db.select().from(toolConnections).where(eq(toolConnections.id, f.connection.id)); + requests = []; + vi.mocked(fetch).mockClear(); + await expect(f.service[operation](f.connection.id, { actorType: "agent", actorId: f.otherAgent.id, managementContext })) + .rejects.toMatchObject({ status: 422, details: { code: "agent_authorization_required" } }); + expect(requests).toEqual([]); + expect(fetch).not.toHaveBeenCalled(); + expect(await db.select().from(toolConnections).where(eq(toolConnections.id, f.connection.id))).toEqual(before); + expect((await f.service.listCatalog(f.connection.id, f.company.id)).map(tool => tool.toolName)).toEqual(["get_fixture"]); + }); + }); + + it("does not fall back to personal or organization grants when the agent grant is revoked", async () => { + const f = await fixture(); + await db.update(connectionGrants).set({ status: "revoked" }).where(eq(connectionGrants.id, f.grant.id)); + await db.insert(connectionGrants).values([ + { companyId: f.company.id, connectionId: f.connection.id, kind: "organization", status: "active", isDefault: true, credentialSecretRefs: f.grant.credentialSecretRefs }, + { companyId: f.company.id, connectionId: f.connection.id, kind: "user", subjectUserId: "fixture-operator", status: "active", credentialSecretRefs: f.grant.credentialSecretRefs }, + ]); + await expect(f.service.refreshCatalog(f.connection.id)) + .rejects.toMatchObject({ status: 422, details: { code: "agent_authorization_required" } }); + await expect(f.service.refreshCatalog(f.connection.id, { actorType: "user", actorId: "fixture-operator" })) + .rejects.toMatchObject({ status: 422, details: { code: "agent_authorization_required" } }); + expect((await f.service.getConnection(f.connection.id, f.company.id)).requiresReauthorization).toBe(true); + await expect(f.service.checkHealth(f.connection.id)) + .rejects.toMatchObject({ status: 422, details: { code: "agent_authorization_required" } }); + expect(requests).toEqual([]); + expect(fetch).not.toHaveBeenCalled(); + }); + + it("uses the same agent grant for generic OAuth MCP health and discovery", async () => { + const f = await fixture(); + await f.service.updateConnection(f.connection.id, { config: { url: `${origin}/mcp`, oauth: { + provider: origin, tokenUrl: `${origin}/token`, expiresAt: null, + } } }); + await expect(f.service.refreshCatalog(f.connection.id)).resolves.toMatchObject({ connection: { healthStatus: "ok" } }); + await expect(f.service.checkHealth(f.connection.id)).resolves.toMatchObject({ connection: { healthStatus: "ok" } }); + expect(requests.length).toBeGreaterThan(0); + expect(requests.every(header => header === "Bearer fixture-agent-access")).toBe(true); + }); + + it("does not choose between multiple active agent authorizations in background work", async () => { + const f = await fixture(); + await db.insert(connectionGrants).values({ companyId: f.company.id, connectionId: f.connection.id, + kind: "agent", subjectAgentId: f.otherAgent.id, status: "active", credentialSecretRefs: f.grant.credentialSecretRefs }); + await expect(f.service.refreshCatalog(f.connection.id)) + .rejects.toMatchObject({ status: 422, details: { code: "agent_authorization_ambiguous" } }); + await expect(f.service.checkHealth(f.connection.id)) + .rejects.toMatchObject({ status: 422, details: { code: "agent_authorization_ambiguous" } }); + expect(requests).toEqual([]); + expect(fetch).not.toHaveBeenCalled(); + }); + + it("still requires reconnect when the provider rejects the agent's credential", async () => { + const f = await fixture(); + rejectCredential = true; + await expect(f.service.refreshCatalog(f.connection.id)).rejects.toThrow(); + expect(requests.length).toBeGreaterThan(0); + expect(requests.every(header => header === "Bearer fixture-agent-access")).toBe(true); + const [grant] = await db.select().from(connectionGrants).where(eq(connectionGrants.id, f.grant.id)); + expect(grant.status).toBe("needs_reauthorization"); + expect((await f.service.getConnection(f.connection.id, f.company.id)).requiresReauthorization).toBe(true); + }); +}); diff --git a/server/src/__tests__/github-health-recovery.test.ts b/server/src/__tests__/github-health-recovery.test.ts index 410d6ce7c6c..5cf14b3b99d 100644 --- a/server/src/__tests__/github-health-recovery.test.ts +++ b/server/src/__tests__/github-health-recovery.test.ts @@ -65,6 +65,7 @@ function githubConnectionHealth() { discovery: {} as never, setupPolicy: { assertConnectorConnectionSetupEnabled: async () => undefined } as never, resolveCredentialHeaders: vi.fn() as never, + vaultGrantForConnection: vi.fn() as never, refreshManagedGitHubGrantAccess: (async () => { await loadGitHubGrantMetadata("ghu_secret", suspendedGitHub); }) as never, diff --git a/server/src/__tests__/mcp-tasks-demo-server.test.ts b/server/src/__tests__/mcp-tasks-demo-server.test.ts index 48cdc91519e..ffe1cb95911 100644 --- a/server/src/__tests__/mcp-tasks-demo-server.test.ts +++ b/server/src/__tests__/mcp-tasks-demo-server.test.ts @@ -15,10 +15,12 @@ import { toolMcpTasks, } from "@paperclipai/db"; import { and, eq, lte, ne } from "drizzle-orm"; +import type { AskUserQuestionsPayload } from "@paperclipai/shared"; import { forgetMcpHttpSessions } from "../services/mcp-http.js"; import { createToolGatewayService } from "../services/tool-gateway.js"; import { toolAccessService } from "../services/tool-access.js"; import { instanceSettingsService } from "../services/instance-settings.js"; +import { issueThreadInteractionService } from "../services/issue-thread-interactions.js"; import { adapterExecutionControls, createAdapterExecutionControl } from "../services/adapter-execution-control.js"; import { createTasksDemoHttpServer, @@ -144,11 +146,14 @@ const support = await getEmbeddedPostgresTestSupport(); expect(first).toMatchObject({ reasonCode: "elicitation_required" }); const [card] = await db.select().from(issueThreadInteractions).where(and(eq(issueThreadInteractions.companyId, org.id), eq(issueThreadInteractions.issueId, issue.id))); expect(card).toMatchObject({ kind: "ask_user_questions", status: "pending" }); - await db.update(issueThreadInteractions).set({ status: "answered", result: { version: 1, answers: [ - { questionId: "decision", optionIds: ["approve"] }, + const decision = (card!.payload as unknown as AskUserQuestionsPayload).questions.find((question) => question.id === "decision")!; + const approve = decision.options.find((option) => option.label === "approve"); + expect(approve).toBeDefined(); + await issueThreadInteractionService(db).answerQuestions({ id: issue.id, companyId: org.id }, card!.id, { answers: [ + { questionId: decision.id, optionIds: [approve!.id] }, { questionId: "note", optionIds: [], otherText: "go ahead" }, - ] } }).where(eq(issueThreadInteractions.id, card!.id)); - const pending = call("needs_approval", { action: "rotate the demo key" }); + ] }, { userId: actor.actorId }); + const pending = call("needs_approval", { action: "rotate the demo key" }).catch((error) => error); await vi.waitFor(() => expect(demoTask("needs_approval")?.status).toBe("working"), { timeout: 5_000 }); clock += 3_000; expect(await pending).toMatchObject({ status: "completed", result: { content: "Approved: rotate the demo key was carried out. Note: go ahead" } }); diff --git a/server/src/services/tool-access.ts b/server/src/services/tool-access.ts index cd3f5dd5b1e..7bccdf12de7 100644 --- a/server/src/services/tool-access.ts +++ b/server/src/services/tool-access.ts @@ -803,7 +803,7 @@ export function toolAccessService( const defaultCatalogAccess = createCatalogDefaultAccess(db); const { enableCatalogEntriesByDefault, upsertAskFirstPolicies } = defaultCatalogAccess; const catalogDiscovery = createToolCatalogDiscovery(db, { now, audit, secrets, credentials: credentialAccess, vercel: vercelCredentials, vercelConnect, setupPolicy, requestRemoteHttpEndpoint, resolveOAuthGrantSecret, discoverOAuthEndpoints, localTools }); - const connectionHealth = createConnectionHealth(db, { now, audit, getConnectionRow, discovery: catalogDiscovery, setupPolicy, resolveCredentialHeaders, refreshManagedGitHubGrantAccess, ensureRuntimeSlot, stdioTemplateId }); + const connectionHealth = createConnectionHealth(db, { now, audit, getConnectionRow, discovery: catalogDiscovery, setupPolicy, resolveCredentialHeaders, vaultGrantForConnection, refreshManagedGitHubGrantAccess, ensureRuntimeSlot, stdioTemplateId }); const { updateConnectionHealth, checkConnectionHealth, sweepConnectionHealth } = connectionHealth; const refreshCatalog = createToolCatalogPublication(db, { now, audit, getConnectionRow, discoverTools: catalogDiscovery.discoverTools, discoverSkills: catalogDiscovery.discoverSkills, assertConnectorConnectionSetupEnabled, updateConnectionHealth, vaultGrantForConnection, ensureRuntimeSlot, enableCatalogEntriesByDefault }); const catalogReader = createToolCatalogReader(db, options, now, getConnectionRow, refreshCatalog, toCatalogEntryForConnection); @@ -1065,8 +1065,10 @@ export function toolAccessService( const eligible = grants.filter( (grant) => grant.connectionId === connection.id && - (connection.credentialPolicy !== "per_user" || - (grant.kind === "user" && grant.subjectUserId === userId)), + (connection.credentialPolicy === "per_agent" + ? grant.kind === "agent" + : connection.credentialPolicy !== "per_user" || + (grant.kind === "user" && grant.subjectUserId === userId)), ); connection.requiresReauthorization = !eligible.some( (grant) => grant.status === "active", diff --git a/server/src/services/tool-access/catalog/failures.ts b/server/src/services/tool-access/catalog/failures.ts index 106a3323755..ee7d72bbb5a 100644 --- a/server/src/services/tool-access/catalog/failures.ts +++ b/server/src/services/tool-access/catalog/failures.ts @@ -20,6 +20,12 @@ export function isOAuthEndpointRejection(error: unknown): boolean { return typeof code === "string" && code.endsWith("_endpoint_rejected"); } +/** A caller without an agent grant has not tested the connection's credential. */ +export function isCallerAuthorizationFailure(error: unknown, actor?: ActorInfo): boolean { + return actor?.actorType === "agent" && error instanceof HttpError + && asRecord(error.details).code === "agent_authorization_required"; +} + export function healthFailureHttpStatus(failure: { status: ToolConnectionHealthStatus; code: string; @@ -31,6 +37,7 @@ export function healthFailureHttpStatus(failure: { if (failure.code === "oauth_insufficient_scope") return 422; if (failure.code === "slack_mcp_access_disabled") return 422; if (failure.code === "user_authorization_required") return 422; + if (failure.code === "agent_authorization_required" || failure.code === "agent_authorization_ambiguous") return 422; if (failure.code === "github_installation_required") return 422; if (failure.code === "composio_broker_retired") return 422; if (failure.code === "tool_connection_transport_unsupported") return 422; @@ -62,7 +69,8 @@ export function sanitizeHttpFailure(error: unknown): { if (code === "slack_mcp_access_disabled" || code === "google_workspace_services_unavailable") { return { status: "error", message: error.message, code }; } - if (code === "user_authorization_required" || code === "github_installation_required") { + if (code === "user_authorization_required" || code === "agent_authorization_required" + || code === "agent_authorization_ambiguous" || code === "github_installation_required") { return { status: "error", message: error.message, code }; } if (code === "tool_connection_transport_unsupported") { diff --git a/server/src/services/tool-access/catalog/health.ts b/server/src/services/tool-access/catalog/health.ts index c343d2daf00..2716415199e 100644 --- a/server/src/services/tool-access/catalog/health.ts +++ b/server/src/services/tool-access/catalog/health.ts @@ -21,7 +21,7 @@ import type { ConnectionCredentialAccess } from "../credential-access.js"; import type { ConnectionSetupPolicy } from "../connections/setup-policy.js"; import { toConnection } from "../connections/projection.js"; import { isAgentMailConnection, type ToolCatalogDiscovery } from "./discovery.js"; -import { healthFailureHttpStatus, sanitizeHttpFailure, unreachableServerRemediation, unsupportedToolConnectionTransport } from "./failures.js"; +import { healthFailureHttpStatus, isCallerAuthorizationFailure, sanitizeHttpFailure, unreachableServerRemediation, unsupportedToolConnectionTransport } from "./failures.js"; function asRecord(value: unknown): Record { if (value && typeof value === "object" && !Array.isArray(value)) return value as Record; @@ -35,12 +35,13 @@ type HealthDependencies = { discovery: ToolCatalogDiscovery; setupPolicy: ConnectionSetupPolicy; resolveCredentialHeaders: ConnectionCredentialAccess["resolveCredentialHeaders"]; + vaultGrantForConnection: ConnectionCredentialAccess["vaultGrantForConnection"]; refreshManagedGitHubGrantAccess: ReturnType["refreshManagedGitHubGrantAccess"]; ensureRuntimeSlot: ReturnType["ensureRuntimeSlot"]; stdioTemplateId(companyId: string, config: Record): Promise; }; -export function createConnectionHealth(db: Db, { now, audit, getConnectionRow, discovery, setupPolicy, resolveCredentialHeaders, refreshManagedGitHubGrantAccess, ensureRuntimeSlot, stdioTemplateId }: HealthDependencies) { +export function createConnectionHealth(db: Db, { now, audit, getConnectionRow, discovery, setupPolicy, resolveCredentialHeaders, vaultGrantForConnection, refreshManagedGitHubGrantAccess, ensureRuntimeSlot, stdioTemplateId }: HealthDependencies) { const { discoverTools, remoteTools, validateAgentMailConnection, validateCogneeConnection } = discovery; const { assertConnectorConnectionSetupEnabled } = setupPolicy; async function updateConnectionHealth( @@ -94,7 +95,9 @@ export function createConnectionHealth(db: Db, { now, audit, getConnectionRow, d config.sourceTemplateKey === "github" && oauth.connectorProfile === "github.code" ) { - const activeGrants = await db + const activeGrants = connection.credentialPolicy === "per_agent" + ? [await vaultGrantForConnection(connection, actor)].filter(grant => grant !== null) + : await db .select() .from(connectionGrants) .where( @@ -187,6 +190,7 @@ export function createConnectionHealth(db: Db, { now, audit, getConnectionRow, d }); return { connection: toConnection(updated), runtimeSlot }; } catch (error) { + if (isCallerAuthorizationFailure(error, actor)) throw error; if ( error instanceof HttpError && asRecord(error.details).code === "github_access_changed" diff --git a/server/src/services/tool-access/catalog/publication.ts b/server/src/services/tool-access/catalog/publication.ts index e5e0ab241d5..fb5a2b0911d 100644 --- a/server/src/services/tool-access/catalog/publication.ts +++ b/server/src/services/tool-access/catalog/publication.ts @@ -29,7 +29,7 @@ import { isAgentMailConnection, type ToolCatalogDiscovery } from "./discovery.js import { classifyRisk, shouldQuarantineNewEntries, isGoogleWorkspaceToolAllowed, type McpToolDescriptor } from "./descriptors.js"; import { toConnection } from "../connections/projection.js"; import { toCatalogEntry } from "./projection.js"; -import { healthFailureHttpStatus, sanitizeHttpFailure, unreachableServerRemediation } from "./failures.js"; +import { healthFailureHttpStatus, isCallerAuthorizationFailure, sanitizeHttpFailure, unreachableServerRemediation } from "./failures.js"; function asRecord(value: unknown): Record { if (value && typeof value === "object" && !Array.isArray(value)) return value as Record; @@ -94,6 +94,7 @@ export function createToolCatalogPublication(db: Db, { now, audit, getConnection unavailableServices, ); } catch (error) { + if (isCallerAuthorizationFailure(error, actor)) throw error; if ( error instanceof HttpError && asRecord(error.details).code === "github_access_changed" diff --git a/server/src/services/tool-access/credential-access.ts b/server/src/services/tool-access/credential-access.ts index 65551c7aa30..5fdf4f70901 100644 --- a/server/src/services/tool-access/credential-access.ts +++ b/server/src/services/tool-access/credential-access.ts @@ -39,6 +39,33 @@ export function createConnectionCredentialAccess(db: Db, { secrets, audit, verce connection: typeof toolConnections.$inferSelect, actor?: ActorInfo, ): Promise { + if (connection.credentialPolicy === "per_agent") { + const agentId = actor?.actorType === "agent" ? actor.actorId : null; + if (actor?.actorType === "agent" && !agentId) { + throw unprocessable("This connection needs the agent's authorization", { + code: "agent_authorization_required", + }); + } + // Discovery has no run identity in background/operator work. Use only + // an unambiguous active agent grant; an agent caller must use its own. + const grants = await db.select().from(connectionGrants).where(and( + eq(connectionGrants.companyId, connection.companyId), + eq(connectionGrants.connectionId, connection.id), + eq(connectionGrants.kind, "agent"), + eq(connectionGrants.status, "active"), + ...(agentId ? [eq(connectionGrants.subjectAgentId, agentId)] : []), + )).limit(2); + if (grants.length !== 1) { + throw unprocessable(grants.length > 1 + ? "This connection has multiple agent accounts. Use a separate connection for each agent." + : "This connection needs the agent's authorization", { + code: grants.length > 1 ? "agent_authorization_ambiguous" : "agent_authorization_required", + setupUrl: connectionSetupUrl(connection), + reconnectUrl: connectionReconnectUrl(connection), + }); + } + return grants[0]!; + } const actorUserId = connectionActorUserId(actor); if (actorUserId) { From 1dee32184459bbfb0e8800249268b8c17b54d727 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 20:48:17 -0600 Subject: [PATCH 05/10] perf(run-events): read the wake's transaction id and the replay probe in the lock read (#21528) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every native event that announces work to another part of the server (chat progress, a final response, an evidence outcome, recovery) registers a wake in its transaction before it writes the event > - Such an event takes six statements: BEGIN, the run lock read, a replay probe, a read of the transaction id for the wake, the append, and COMMIT. A controller checkpoint save takes five: BEGIN, the owner lock, the transaction id, the write, COMMIT > - In the agent scaling load model these two paths are now the largest per-agent cost after #21511 and #21520, and other events fall behind first at 600 agents > - This pull request returns the transaction id from the lock read that each path already sends, and moves the replay probe into the run lock read when it is safe > - The benefit is four statements for such an event and four for a checkpoint save, with the same locks, the same wake order and the same replay rule ## Linked Issues or Issue Description Refs #21511 and #21520 (the delta batch path). Refs #21418 (agent scaling plan and load model). **What existing behavior does this improve?** The statement count of native events that register a wake (`appendNativeSourceEvent`) and of controller checkpoint saves (`nativeRunnerCheckpointWriter`). **Subsystem affected** `packages/db/src/work-signals.ts` (one new export), `server/src/services/heartbeat-run-events.ts`, `server/src/services/native-runtime/native-runner-checkpoint.ts`. **Current behavior** `signalDatabaseWork` sends `select pg_current_xact_id()` for the first wake of each transaction. A native event with a wake reads the run lock, then probes for its replay in a separate statement, because a statement that waits for the lock keeps its earlier snapshot. **Proposed behavior** - The run lock read and the checkpoint owner lock read also return `pg_current_xact_id()::text`. Both already lock a row, so the transaction has an id at that point. `primeDatabaseWorkTransactionId` gives it to the transaction's wake state, and the wake sends no statement for it. - For an event that may register a progress, response, reassessment or retention wake, the run lock read also reads `next_event_seq` from its own snapshot and checks for a replay in that snapshot. Every event append advances `next_event_seq` in its transaction. So when the locked row has the same `next_event_seq` as the snapshot, no append committed in between, and the probe result is current. Otherwise, or when the probe finds a replay, the separate probe runs as before. **Reason and benefit** Fewer round trips while a pool connection is held in a transaction. In the load model these paths hold the main pool and the three-connection checkpoint pool longest per event. **Breaking changes** None. No schema change. ## What Changed - `primeDatabaseWorkTransactionId(connection, xid)` (`work-signals.ts`, exported from `@paperclipai/db`): inside a transaction whose id is not yet known, it records the id. Outside a transaction it does nothing. An uncertain commit still resolves with `pg_xact_status` on the same id. - `appendNativeSourceEvent`: the lock read returns the transaction id when the event has any wake. For a probing wake it is one statement: a CTE that reads `next_event_seq` from the snapshot, a CTE that locks the run row (`for no key update`, unchanged), and `exists (...)` for the replay. The wake is still registered after the replay check and before the append. - `lockNativeRunnerCheckpointOwner` returns the transaction id with its owner lock and primes the wake state. Its return value and every caller are unchanged. ## Verification - Red/green on embedded PostgreSQL; each new assertion failed against `origin/master` first: - `heartbeat-run-event-atomic-append.test.ts` "registers a native event's wakes before its write and only when the event is new": four statements (before: six), the wake's intent and settlement are published, and a replay publishes no wake. - New: "finds a replay that another writer committed while the lock read waited, and registers no wake for it". Another connection holds the run row (the test waits until PostgreSQL reports the append waiting for that lock), stores the same event and moves `next_event_seq`; the waiting append returns `duplicate`, runs the separate probe, and publishes no wake. With the `next_event_seq` comparison removed, this test fails. - New `native-runner-checkpoint-save.test.ts`: a save is BEGIN, the owner lock with the transaction id, the write, COMMIT (before: five statements), and its recovery wake is published; a save after the lease expired writes nothing and fails with `native_runner_checkpoint_owner_lost`. - `work-signals.test.ts`: a primed transaction sends no id statement for its wakes; a primed transaction whose connection is killed after its write stays pending and then resolves through `reconcileDatabaseWork`, with no row stored. - Added after merging `origin/master` (with #21511, #21520 and #21436) into this branch: - "gives the wake the transaction id of the transaction that took the run lock": inside a caller's transaction, an event with a progress wake sends no separate `pg_current_xact_id()` statement; the transaction's connection is then killed, and `reconcileDatabaseWork` checks `pg_xact_status` with exactly the id that `pg_current_xact_id()` returns on that same transaction. Priming a wrong id makes it fail. - "primes nothing when the lock read fails, so the transaction is not left uncertain": a lock read that fails on `lock_timeout` leaves no pending work; after the same failure inside a savepoint, the next signal on that transaction still sends its own `pg_current_xact_id()`. Priming before the lock read makes it fail. - Both call sites pass only the value returned by the statement they just ran on the same `tx`: `appendNativeSourceEvent` (`tx.execute(...)` then `primeDatabaseWorkTransactionId(tx, run.xid)`) and `lockNativeRunnerCheckpointOwner` (`tx.with(locked).select(...)` then `primeDatabaseWorkTransactionId(tx, xid)`). - Retest on the combined tree, one file at a time: atomic-append 28 (including the equivalence test and both race tests), the port suite with conformance 105, checkpoint-save 2, `live-recovery-work` 51, `work-signals` 25, runner `native-session-runtime` 297; server typecheck passes. No code change was needed for the combination. - Other focused suites pass, one file at a time: the port suite with its conformance suite (100), `chat-native-progress-work` (35), `native-observer-replay` (1), `heartbeat-run-event-sequencing` (5), `native-event-boundary-golden` (1), `native-recovered-owner-followup` (97), `native-restart-recovery-isolation` (1), `heartbeat-native-restart-suspension` (115), `live-recovery-work` (51), `work-signals` (25). - `pnpm --filter @paperclipai/server run typecheck` and `pnpm check:module-boundaries` pass. - Load model (`scripts/benchmark-agent-scaling-load.ts` as merged in #21418). Stacked: this pull request merged locally with #21511, and the model's local `DELTA_COALESCE=1` mode that mirrors #21520 (the model applies the port's batching rules itself; that mode is not in any pull request). Default pool of 10 connections; embedded PostgreSQL on the same, shared machine; runs one after the other with a pause between them. No run had a failed operation, a failed checkpoint save or a lost lease. "Before" rows are `origin/master` from the same session. | Agents | Code | Window | Load avg | Event-loop utilization | Statements/s | Delta lateness p50 / p99 (ms) | Other events done / offered | Other event lateness p50 / p99 (ms) | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 200 | before | 60 s | 9.6 | 0.56 | 7,043 | 283 / 3,368 | 12,000 / 12,000 | 278 / 3,895 | | 200 | stacked | 60 s | 7.9 | 0.31 | 2,725 | 102 / 253 | 12,000 / 12,000 | 1 / 70 | | 400 | before | 60 s | 11.8 | 0.59 | 7,158 | 9,927 / 18,779 | 6,068 / 24,000 | 1,904 / 60,154 | | 400 | stacked | 60 s | 8.6 | 0.60 | 5,609 | 117 / 955 | 24,000 / 24,000 | 2 / 1,274 | | 600 | before | 60 s | 15.0 | 0.85 | 12,419 | 7,108 / 11,017 | 7,836 / 36,000 | 10,822 / 52,171 | | 600 | #21511 + #21520 only | 120 s | 11.4 | 0.72 | 6,730 | 1,147 / 6,335 | 61,158 / 72,000 | 11,678 / 21,212 | | 600 | stacked | 120 s | 9.6 | 0.93 | 10,290 | 656 / 4,233 | 71,895 / 72,000 | 1,136 / 4,980 | | 600 | stacked (repeat) | 120 s | 9.8 | 0.90 | 9,579 | 648 / 4,509 | 71,958 / 72,000 | 1,319 / 5,712 | | 800 | stacked | 60 s | 9.7 | 0.98 | 10,971 | 920 / 3,305 | 46,244 / 48,000 | 1,974 / 3,835 | With the three changes, one process finishes 99.9% of the offered work at 600 agents, but its event-loop utilization is 0.90–0.93, above the 0.75 target, and p99 lateness is about 5 s. It saturates between 600 and 800 agents. - Statements: an event with a wake 6 → 4; a checkpoint save 5 → 4 (6 → 5 where commit-time notifications are on). ## Risks - The transaction id is read in a statement that already assigns one (a row lock), so no transaction gets an id it did not have before. - The probe in the lock read is used only when `next_event_seq` did not change between the snapshot and the lock. Every writer of `heartbeat_run_events` moves `next_event_seq` in the same transaction (native and ordinary appends, the run coordinator store, the retry queue). A future writer that inserts events without it would make this probe miss a concurrent replay; the append statement then still finds the replay and stores nothing, but the wake would already be registered (a spurious wake, not a lost or duplicate event). - This pull request and #21511 both edit `heartbeat-run-events.ts`, in different functions. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- packages/db/src/index.ts | 2 +- packages/db/src/work-signals.test.ts | 29 +++- packages/db/src/work-signals.ts | 5 + .../heartbeat-run-event-atomic-append.test.ts | 163 ++++++++++++++++-- server/src/services/heartbeat-run-events.ts | 34 ++-- .../native-runner-checkpoint-save.test.ts | 82 +++++++++ .../native-runner-checkpoint.ts | 11 +- 7 files changed, 292 insertions(+), 34 deletions(-) create mode 100644 server/src/services/native-runtime/native-runner-checkpoint-save.test.ts diff --git a/packages/db/src/index.ts b/packages/db/src/index.ts index 0c4c0fcd9e8..7e41782d377 100644 --- a/packages/db/src/index.ts +++ b/packages/db/src/index.ts @@ -54,5 +54,5 @@ export { issueRelations } from "./schema/issue_relations.js"; export { issueReferenceMentions } from "./schema/issue_reference_mentions.js"; export * from "./schema/index.js"; -export { databaseWorkScopeKey, signalDatabaseWork, subscribeDatabaseWork, databaseWorkPending, reconcileDatabaseWork } from "./work-signals.js"; +export { databaseWorkScopeKey, primeDatabaseWorkTransactionId, signalDatabaseWork, subscribeDatabaseWork, databaseWorkPending, reconcileDatabaseWork } from "./work-signals.js"; export { withDatabaseOperation } from "./response-capture.js"; diff --git a/packages/db/src/work-signals.test.ts b/packages/db/src/work-signals.test.ts index b98a868af96..704afac48a7 100644 --- a/packages/db/src/work-signals.test.ts +++ b/packages/db/src/work-signals.test.ts @@ -3,7 +3,7 @@ import { drizzle } from "drizzle-orm/postgres-js"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { createDb, withDedicatedDbConnection } from "./client.js"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./test-embedded-postgres.js"; -import { DATABASE_WORK_CHANNEL, databaseWorkScopeKey, databaseWorkPending, enableDatabaseWorkNotifications, installDatabaseWorkSignals, notifyDatabaseWorkOnCommit, reconcileDatabaseWork, signalDatabaseWork, subscribeDatabaseWork } from "./work-signals.js"; +import { DATABASE_WORK_CHANNEL, databaseWorkScopeKey, databaseWorkPending, enableDatabaseWorkNotifications, installDatabaseWorkSignals, notifyDatabaseWorkOnCommit, primeDatabaseWorkTransactionId, reconcileDatabaseWork, signalDatabaseWork, subscribeDatabaseWork } from "./work-signals.js"; const support = await getEmbeddedPostgresTestSupport(); (support.supported ? describe : describe.skip)("database work signals", () => { @@ -145,6 +145,22 @@ const support = await getEmbeddedPostgresTestSupport(); } finally { release(); await serverDone?.catch(() => {}); unsubscribe(); } }); + it("resolves an uncertain transaction with a transaction id that a statement returned", async () => { + await expect(db.transaction(async tx => { + const [{ xid }] = await tx.execute(sql`select pg_current_xact_id()::text as xid`); + primeDatabaseWorkTransactionId(tx, xid as string); + await signalDatabaseWork(tx, "primed-disconnect"); + await tx.execute(sql`insert into work_signal_test values ('primed-disconnect')`); + const [{ pid }] = await tx.execute(sql`select pg_backend_pid() as pid`); + await db.execute(sql`select pg_terminate_backend(${pid as number})`); + await tx.execute(sql`select 1`); + })).rejects.toThrow(); + expect(databaseWorkPending(db, "primed-disconnect")).toBe(true); + await reconcileDatabaseWork(db, "primed-disconnect"); + expect(databaseWorkPending(db, "primed-disconnect")).toBe(false); + expect(await db.execute(sql`select * from work_signal_test where id = 'primed-disconnect'`)).toHaveLength(0); + }); + it("recovers after a real backend disconnect without replaying the queue write", async () => { await expect(db.transaction(async tx => { await signalDatabaseWork(tx, "disconnect"); @@ -196,6 +212,17 @@ describe("work signal query boundaries", () => { expect(write).not.toHaveBeenCalled(); expect(databaseWorkPending(connection, "failed")).toBe(false); }); + it("reads no transaction id when a statement of the transaction already returned it", async () => { + const execute = vi.fn(); + const connection = installDatabaseWorkSignals({ execute, transaction: async (callback: any) => callback({ execute, transaction: async () => {} }) }); + primeDatabaseWorkTransactionId(connection, "outside"); + await connection.transaction(async (tx: any) => { + primeDatabaseWorkTransactionId(tx, "42"); + await signalDatabaseWork(tx, "primed"); + await signalDatabaseWork(tx, "primed-again"); + }); + expect(execute).not.toHaveBeenCalled(); + }); it("rejects registration on an autocommit root", async () => { const connection = installDatabaseWorkSignals({ execute: vi.fn(), transaction: vi.fn() }); await expect(signalDatabaseWork(connection, "bad")).rejects.toThrow("inside a transaction"); diff --git a/packages/db/src/work-signals.ts b/packages/db/src/work-signals.ts index 05b83a417cf..aa312072cd7 100644 --- a/packages/db/src/work-signals.ts +++ b/packages/db/src/work-signals.ts @@ -88,6 +88,11 @@ export async function signalDatabaseWork(connection: object, topic: string, hint if (context.scope.notify) notifyAtCommit(context, notification(context.scope, topic)); } +export function primeDatabaseWorkTransactionId(connection: object, xid: string): void { + const state = contexts.get(connection)?.transaction; + if (state && !state.xid) state.xid = Promise.resolve(xid); +} + function notifyAtCommit(context: Context, payload: string): void { if (context.ended) throw new Error("Database work was signalled after its transaction or savepoint ended"); const key = `${DATABASE_WORK_CHANNEL}\u0000${payload}`; diff --git a/server/src/__tests__/heartbeat-run-event-atomic-append.test.ts b/server/src/__tests__/heartbeat-run-event-atomic-append.test.ts index b805d99e9a6..9ff9edb5694 100644 --- a/server/src/__tests__/heartbeat-run-event-atomic-append.test.ts +++ b/server/src/__tests__/heartbeat-run-event-atomic-append.test.ts @@ -1,9 +1,11 @@ import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import { randomBytes } from "node:crypto"; -import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { eq, inArray, sql } from "drizzle-orm"; import { drizzle } from "drizzle-orm/postgres-js"; -import { agents, companies, createDb, heartbeatRunEvents, heartbeatRuns, issues, nativeRunnerCheckpoints } from "@paperclipai/db"; +import { agents, companies, createDb, heartbeatRunEvents, heartbeatRuns, issues, nativeRunnerCheckpoints, databaseWorkPending, + reconcileDatabaseWork, signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db"; +import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js"; import { appendHeartbeatRunEvent, appendHeartbeatRunEventBatch } from "../services/heartbeat-run-events.js"; import { nativeSha256 } from "../services/native-runtime/canonical.js"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; @@ -293,6 +295,7 @@ describeEmbeddedPostgres("atomic ordinary run events", () => { locked(); await released; }); + void holder.catch(() => undefined); await lockedPromise; return { release: () => { release(); return holder; } }; }; @@ -485,17 +488,143 @@ describeEmbeddedPostgres("atomic ordinary run events", () => { const runId = await makeRun(); const [issue] = await db.insert(issues).values({ companyId, title: "Progress task" }).returning(); await db.update(heartbeatRuns).set({ nativeIssueId: issue!.id, runtimeMode: "native" }).where(eq(heartbeatRuns.id, runId)); - const first = await sentStatements(db.$client, () => appendHeartbeatRunEvent(db, nativeEvent(runId, 1, "item.completed"))); - // BEGIN, lock, duplicate probe, the wake's transaction id, the append, COMMIT. - expect(first.statements).toHaveLength(6); - const wake = first.statements.findIndex(text => text.includes("pg_current_xact_id")); - expect(wake).toBeGreaterThan(0); - expect(wake).toBeLessThan(first.statements.findIndex(text => text.includes("insert into heartbeat_run_events"))); - expect(first.result).toMatchObject({ disposition: "committed", highestContiguousSourceSeq: 1 }); - const replay = await sentStatements(db.$client, () => appendHeartbeatRunEvent(db, nativeEvent(runId, 1, "item.completed"))); - expect(replay.statements.some(text => text.includes("pg_current_xact_id"))).toBe(false); - expect(replay.result).toEqual({ row: first.result.row, disposition: "duplicate", highestContiguousSourceSeq: 1 }); - expect(await readEvents(runId)).toEqual([first.result.row]); + const wakes: string[] = []; + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatNativeProgress, event => { wakes.push(event); }); + try { + const first = await sentStatements(db.$client, () => appendHeartbeatRunEvent(db, nativeEvent(runId, 1, "item.completed"))); + expect(first.statements).toHaveLength(4); + expect(first.statements[0]).toMatch(/^begin/i); + expect(first.statements[1]).toMatch(/^with "snapshot_run" as .*pg_current_xact_id\(\).* for no key update.*"replay_found"/); + expect(first.statements[2]).toMatch(/insert into heartbeat_run_events/); + expect(first.statements[3]).toMatch(/^commit/i); + expect(wakes).toEqual(["intent", "settled"]); + expect(first.result).toMatchObject({ disposition: "committed", highestContiguousSourceSeq: 1 }); + wakes.length = 0; + const replay = await sentStatements(db.$client, () => appendHeartbeatRunEvent(db, nativeEvent(runId, 1, "item.completed"))); + expect(wakes).toEqual([]); + expect(replay.statements).toHaveLength(4); + expect(replay.result).toEqual({ row: first.result.row, disposition: "duplicate", highestContiguousSourceSeq: 1 }); + expect(await readEvents(runId)).toEqual([first.result.row]); + } finally { unsubscribe(); } + }); + + it("gives the wake the transaction id of the transaction that took the run lock", async () => { + const runId = await makeRun(); + const [issue] = await db.insert(issues).values({ companyId, title: "Primed wake" }).returning(); + await db.update(heartbeatRuns).set({ nativeIssueId: issue!.id, runtimeMode: "native" }).where(eq(heartbeatRuns.id, runId)); + const topic = DELIVERY_QUEUES.chatNativeProgress; + const options = db.$client.options as { debug?: unknown }; + const sent: Array<{ text: string; parameters: unknown[] }> = []; + let ownXid: string | undefined; + options.debug = (_connection: number, text: string, parameters: unknown[]) => { + sent.push({ text: text.trim().replace(/\s+/g, " "), parameters }); + }; + try { + await expect(db.transaction(async tx => { + await appendHeartbeatRunEvent(tx as unknown as typeof db, nativeEvent(runId, 1, "item.completed")); + expect(sent.filter(statement => /^select pg_current_xact_id\(\)/.test(statement.text))).toEqual([]); + expect(sent.filter(statement => statement.text.includes("pg_current_xact_id()"))).toHaveLength(1); + const [own] = await tx.execute<{ xid: string; pid: number }>(sql`select pg_current_xact_id()::text as xid, pg_backend_pid() as pid`); + ownXid = own!.xid; + await otherDb.execute(sql`select pg_terminate_backend(${own!.pid})`); + await tx.execute(sql`select 1`); + })).rejects.toThrow(); + expect(databaseWorkPending(db, topic)).toBe(true); + sent.length = 0; + await reconcileDatabaseWork(db, topic); + } finally { options.debug = undefined; } + const statusRead = sent.find(statement => statement.text.includes("pg_xact_status")); + expect(ownXid).toMatch(/^\d+$/); + expect(statusRead?.parameters).toEqual([ownXid]); + expect(databaseWorkPending(db, topic)).toBe(false); + expect(await readEvents(runId)).toEqual([]); + }); + + it("primes nothing when the lock read fails, so the transaction is not left uncertain", async () => { + const runId = await makeRun(); + const [issue] = await db.insert(issues).values({ companyId, title: "Failed lock read" }).returning(); + await db.update(heartbeatRuns).set({ nativeIssueId: issue!.id, runtimeMode: "native" }).where(eq(heartbeatRuns.id, runId)); + let release!: () => void, locked!: () => void; + const released = new Promise(resolve => { release = resolve; }); + const lockedRun = new Promise(resolve => { locked = resolve; }); + const holder = otherDb.transaction(async tx => { + await tx.execute(sql`select id from heartbeat_runs where id = ${runId} for update`); + locked(); + await released; + }); + void holder.catch(() => undefined); + await lockedRun; + const wakes: string[] = []; + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatNativeProgress, wake => { wakes.push(wake); }); + const options = db.$client.options as { debug?: unknown }; + const sent: string[] = []; + try { + await expect(db.transaction(async tx => { + await tx.execute(sql`set local lock_timeout = '100ms'`); + await appendHeartbeatRunEvent(tx as unknown as typeof db, nativeEvent(runId, 1, "item.completed")); + })).rejects.toThrow(); + expect(databaseWorkPending(db, DELIVERY_QUEUES.chatNativeProgress)).toBe(false); + await expect(db.transaction(async tx => { + await tx.execute(sql`set local lock_timeout = '100ms'`); + await expect(tx.transaction(savepoint => + appendHeartbeatRunEvent(savepoint as unknown as typeof db, nativeEvent(runId, 1, "item.completed")))).rejects.toThrow(); + options.debug = (_connection: number, text: string) => { sent.push(text.trim().replace(/\s+/g, " ")); }; + await signalDatabaseWork(tx, "lock-read-failed"); + options.debug = undefined; + throw new Error("roll back"); + })).rejects.toThrow("roll back"); + } finally { + options.debug = undefined; + unsubscribe(); + release(); + await holder; + } + expect(sent.filter(text => /^select pg_current_xact_id\(\)/.test(text))).toHaveLength(1); + await reconcileDatabaseWork(db, "lock-read-failed"); + expect(databaseWorkPending(db, "lock-read-failed")).toBe(false); + expect(wakes).toEqual([]); + expect(databaseWorkPending(db, DELIVERY_QUEUES.chatNativeProgress)).toBe(false); + expect(await readEvents(runId)).toEqual([]); + }); + + it("finds a replay that another writer committed while the lock read waited, and registers no wake for it", async () => { + const runId = await makeRun(); + const [issue] = await db.insert(issues).values({ companyId, title: "Progress race" }).returning(); + await db.update(heartbeatRuns).set({ nativeIssueId: issue!.id, runtimeMode: "native" }).where(eq(heartbeatRuns.id, runId)); + const event = nativeEvent(runId, 1, "item.completed"); + let release!: () => void, locked!: () => void; + const released = new Promise(resolve => { release = resolve; }); + const lockedRun = new Promise(resolve => { locked = resolve; }); + const holder = otherDb.transaction(async tx => { + await tx.execute(sql`select id from heartbeat_runs where id = ${runId} for update`); + await tx.insert(heartbeatRunEvents).values({ companyId, agentId, runId, seq: 1, eventType: event.eventType, stream: "system", + level: "info", payload: event.payload, sourceInstanceId: "native-writer", sourceEventId: event.nativeSource.sourceEventId, + sourceSeq: 1, sourcePayloadSha256: nativeSha256(event.nativeSource.canonicalPayload), protocolSchemaVersion: 1 }); + await tx.update(heartbeatRuns).set({ nextEventSeq: 2 }).where(eq(heartbeatRuns.id, runId)); + locked(); + await released; + }); + void holder.catch(() => undefined); + await lockedRun; + const wakes: string[] = []; + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatNativeProgress, wake => { wakes.push(wake); }); + try { + const appended = sentStatements(db.$client, () => appendHeartbeatRunEvent(db, event)); + void appended.catch(() => undefined); + await vi.waitFor(async () => { + const [waiting] = await otherDb.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity + where datname = current_database() and wait_event_type = 'Lock' and query like '%"locked_run"%'`); + expect(waiting?.count).toBe(1); + }, { timeout: 10_000, interval: 20 }); + release(); + await holder; + const { result, statements } = await appended; + expect(result).toMatchObject({ disposition: "duplicate", row: { seq: 1, sourceSeq: 1 } }); + expect(statements.some(text => /^select \* from heartbeat_run_events/.test(text))).toBe(true); + expect(wakes).toEqual([]); + } finally { unsubscribe(); } + expect((await readEvents(runId)).map(row => row.seq)).toEqual([1]); + expect((await readRun(runId)).nextEventSeq).toBe(2); }); it("keeps one ordered row per source event when two servers append the same stream", async () => { @@ -601,11 +730,11 @@ describeEmbeddedPostgres("atomic ordinary run events", () => { expect(replay.highestContiguousSourceSeq).toBe(kind === "native" ? 1 : 0); expect((await readRun(runId)).nextEventSeq).toBe(2); expect(await readEvents(runId)).toEqual([committed.row]); - const lockReads = statements.filter(query => query.startsWith("select ") && - query.includes('from "heartbeat_runs"') && query.endsWith("for no key update")); + const lockReads = statements.filter(query => /^(select|with) /.test(query) && + query.includes('from "heartbeat_runs"') && query.includes("for no key update")); expect(lockReads).toHaveLength(2); for (const query of lockReads) { - expect(query).toMatch(/^select "company_id", "agent_id"(, \("native_source_cursors" ->> \$1\)::bigint as "source_cursor")? from "heartbeat_runs" /); + expect(query).toMatch(/select "company_id", "agent_id"(, \("native_source_cursors" ->> \$\d+\)::bigint as "source_cursor")?(, pg_current_xact_id\(\)::text as "xid")?(, "next_event_seq")? from "heartbeat_runs" /); expect(query).not.toMatch(/context_snapshot|runner_profile_json|result_json|stdout_excerpt/); } }); @@ -631,9 +760,11 @@ describeEmbeddedPostgres("atomic ordinary run events", () => { holderLocked(); await held; }); + void holder.catch(() => undefined); await locked; let secondSettled = false; const second = appendHeartbeatRunEvent(db, native(2)).finally(() => { secondSettled = true; }); + void second.catch(() => undefined); try { // A controller checkpoint row references the run row by foreign key. // Its insert must not wait for the open append. diff --git a/server/src/services/heartbeat-run-events.ts b/server/src/services/heartbeat-run-events.ts index 4aa41b431ff..30f41f5c78a 100644 --- a/server/src/services/heartbeat-run-events.ts +++ b/server/src/services/heartbeat-run-events.ts @@ -9,7 +9,7 @@ import { boundRunIssueIdSql } from "./run-task-binding.js"; import { and, asc, eq, getTableColumns, is, sql } from "drizzle-orm"; import { PgTransaction } from "drizzle-orm/pg-core"; import type { Db } from "@paperclipai/db"; -import { heartbeatRunEvents, heartbeatRuns } from "@paperclipai/db"; +import { heartbeatRunEvents, heartbeatRuns, primeDatabaseWorkTransactionId } from "@paperclipai/db"; import { nativeSha256 } from "./native-runtime/canonical.js"; import { isUniqueViolation } from "../db-errors.js"; @@ -345,9 +345,9 @@ function decodeRunEventRow(raw: Record): typeof heartbeatRunEve /** * A native source event: one lock read, then one statement that finds a * replayed event or reserves the sequence and inserts, and reads how much of - * the source is retained. The lock must stay a statement of its own. A - * statement that waits for the run lock keeps the snapshot it took before - * waiting, so it would not see an event the lock holder had just committed. + * the source is retained. A statement that waits for the run lock keeps the + * snapshot it took before waiting, so it would not see an event the lock + * holder had just committed. */ async function appendNativeSourceEvent( tx: Db, @@ -361,15 +361,26 @@ async function appendNativeSourceEvent( // checkpoints, that are written for the same run while events stream. // Only progress needs the task binding. Other native/recovery events // must not read a potentially large context snapshot to append a log. + const replayed = sql`run_id = ${input.runId} and (source_event_id = ${source.sourceEventId} + or (source_instance_id = ${source.sourceInstanceId} and source_seq = ${source.sourceSeq}))`; + const probes = wakes.reassessment || wakes.progress || wakes.response || wakes.retention; + const columns = sql`"company_id", "agent_id", ("native_source_cursors" ->> ${source.sourceInstanceId})::bigint as "source_cursor"${ + wakes.progress || wakes.response || wakes.reassessment ? sql`, "runtime_mode"` : sql.empty()}${ + wakes.progress ? sql`, ${boundRunIssueIdSql()} as "issue_id"` : sql.empty()}${ + Object.values(wakes).some(Boolean) ? sql`, pg_current_xact_id()::text as "xid"` : sql.empty()}`; const [run] = await tx.execute<{ company_id: string; agent_id: string; source_cursor: string | number | null; - runtime_mode?: string | null; issue_id?: string | null }>( - sql`select "company_id", "agent_id", ("native_source_cursors" ->> ${source.sourceInstanceId})::bigint as "source_cursor"${ - wakes.progress || wakes.response || wakes.reassessment ? sql`, "runtime_mode"` : sql.empty()}${ - wakes.progress ? sql`, ${boundRunIssueIdSql()} as "issue_id"` : sql.empty() - } from "heartbeat_runs" where "id" = ${input.runId} for no key update`); + runtime_mode?: string | null; issue_id?: string | null; xid?: string; probe_current?: boolean; replay_found?: boolean }>(probes + ? sql`with "snapshot_run" as (select "next_event_seq" from "heartbeat_runs" where "id" = ${input.runId}), + "locked_run" as (select ${columns}, "next_event_seq" from "heartbeat_runs" where "id" = ${input.runId} for no key update) + select "locked_run".*, "locked_run"."next_event_seq" = (select "next_event_seq" from "snapshot_run") as "probe_current", + exists (select 1 from heartbeat_run_events where ${replayed}) as "replay_found" + from "locked_run"` + : sql`select ${columns} from "heartbeat_runs" where "id" = ${input.runId} for no key update`); if (!run || run.company_id !== input.companyId || run.agent_id !== input.agentId) { throw new Error("heartbeat_run_event_binding_mismatch"); } + if (run.xid) primeDatabaseWorkTransactionId(tx, run.xid); + const knownNew = run.probe_current === true && run.replay_found === false; // The stored cursor says the run holds every event of this source up to it. // An event at or just past it moves the cursor without counting the run's // events; any other event, or a run without a cursor, recounts them. @@ -380,15 +391,14 @@ async function appendNativeSourceEvent( if (wakes.recovery) await notifyLiveRunRecovery(tx, input); const sourceHash = nativeSha256(source.canonicalPayload); - const replayed = sql`run_id = ${input.runId} and (source_event_id = ${source.sourceEventId} - or (source_instance_id = ${source.sourceInstanceId} and source_seq = ${source.sourceSeq}))`; const native = run.runtime_mode === "native"; const reassessment = wakes.reassessment && native, progress = wakes.progress && native && run.issue_id; const response = wakes.response && native; if (reassessment || progress || response || wakes.retention) { // A wake is registered before the write it announces, and a replayed // event registers none. So these events look for their replay first. - const [existing] = await tx.execute>(sql`select * from heartbeat_run_events where ${replayed} limit 1`); + const [existing] = knownNew ? [] + : await tx.execute>(sql`select * from heartbeat_run_events where ${replayed} limit 1`); if (existing) { const row = decodeRunEventRow(existing); if (row.sourcePayloadSha256 !== sourceHash) throw new HeartbeatRunEventConflictError(); diff --git a/server/src/services/native-runtime/native-runner-checkpoint-save.test.ts b/server/src/services/native-runtime/native-runner-checkpoint-save.test.ts new file mode 100644 index 00000000000..bd179a05724 --- /dev/null +++ b/server/src/services/native-runtime/native-runner-checkpoint-save.test.ts @@ -0,0 +1,82 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { eq } from "drizzle-orm"; +import { agents, checkpointDb, closeCheckpointDb, companies, createDb, heartbeatRuns, issues, nativeRunFinalizations, nativeRunnerCheckpoints, + subscribeDatabaseWork } from "@paperclipai/db"; +import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; +import { nativeRunnerCheckpointWriter } from "./native-runner-checkpoint.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; + +const support = await getEmbeddedPostgresTestSupport(); + +(support.supported ? describe : describe.skip)("native runner checkpoint save", () => { + let temporary: Awaited>; + let db: ReturnType; + beforeAll(async () => { + process.env.PAPERCLIP_SECRETS_MASTER_KEY ??= randomBytes(32).toString("base64"); + temporary = await startEmbeddedPostgresTestDatabase("paperclip-checkpoint-save-"); + db = createDb(temporary.connectionString); + }, 60_000); + afterAll(async () => { + if (db) await closeCheckpointDb(db, 1); + await db?.$client.end({ timeout: 1 }); + await temporary?.cleanup(); + }); + + const fixture = async (leaseExpiresAt: Date) => { + const [company] = await db.insert(companies).values({ name: "Checkpoint save", issuePrefix: `CS${randomUUID().slice(0, 4)}` }).returning(); + const [agent] = await db.insert(agents).values({ companyId: company!.id, name: "Checkpoint agent", adapterType: "process" }).returning(); + const [issue] = await db.insert(issues).values({ companyId: company!.id, title: "Checkpoint task" }).returning(); + const runId = randomUUID(), leaseOwner = randomUUID(), bootId = randomUUID(); + await db.insert(heartbeatRuns).values({ id: runId, companyId: company!.id, agentId: agent!.id, status: "running", + runtimeMode: "native", nativeIssueId: issue!.id, issueId: issue!.id }); + await db.insert(nativeRunFinalizations).values({ runId, companyId: company!.id, issueId: issue!.id, phase: "observed", attempt: 0, + controllerBootId: bootId, controllerPid: process.pid, controllerProcessStartedAt: new Date(), controllerGeneration: 1, + leaseOwner, leaseExpiresAt }); + const write = nativeRunnerCheckpointWriter(db, { companyId: company!.id, issueId: issue!.id, runId, authorityRunId: runId, + bootId, generation: 1, leaseOwner }); + const checkpoint = (changed: Map, reset: boolean) => ({ + runId, records: new Map(changed), changedRecords: changed, removedRecords: [], reset, workCommandIds: [], predecessor: null, + runnerVersion: "test", runnerDigest: `sha256:${"a".repeat(64)}`, + recoveryEvidence: { runnerInstanceId: "test", normalizedSessionId: "test", authenticatedConnectionCount: 1, highestRecordedSourceSeq: 0 }, + restoreSessionDirectory() {}, + }) as unknown as Parameters[0]; + return { runId, write, checkpoint }; + }; + + const sentStatements = async (action: () => Promise) => { + const statements: string[] = []; + const options = checkpointDb(db).$client.options as { debug?: unknown }; + options.debug = (_connection: number, text: string) => { statements.push(text.trim().replace(/\s+/g, " ")); }; + try { await action(); return statements; } finally { options.debug = undefined; } + }; + + it("saves a checkpoint with the owner lock, the write and the commit, and registers its wake before the write", async () => { + const f = await fixture(new Date(Date.now() + 3_600_000)); + await f.write(f.checkpoint(new Map([["header", "{}"], ["cursor", "{\"ackedSourceSeq\":0}"]]), true)); + const events: string[] = []; + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.nativeProcessSessions, event => { events.push(event); }); + let statements: string[]; + try { + statements = await sentStatements(() => f.write(f.checkpoint(new Map([["event:1", "{}"], ["cursor", "{\"ackedSourceSeq\":1}"]]), false))); + } finally { unsubscribe(); } + expect(statements).toHaveLength(4); + expect(statements[0]).toMatch(/^begin/i); + expect(statements[1]).toMatch(/pg_current_xact_id\(\)/); + expect(statements[2]).toMatch(/^insert into "native_runner_checkpoints"/); + expect(statements[3]).toMatch(/^commit/i); + expect(events).toEqual(["intent", "settled"]); + expect((await db.select({ part: nativeRunnerCheckpoints.part }).from(nativeRunnerCheckpoints) + .where(eq(nativeRunnerCheckpoints.runId, f.runId))).map(row => row.part).sort()).toEqual(["cursor", "event:1", "header"]); + }); + + it("refuses a save after the owner's lease expired and writes nothing", async () => { + const f = await fixture(new Date(Date.now() + 3_600_000)); + await f.write(f.checkpoint(new Map([["header", "{}"]]), true)); + await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() - 1_000) }) + .where(eq(nativeRunFinalizations.runId, f.runId)); + await expect(f.write(f.checkpoint(new Map([["event:1", "{}"]]), false))).rejects.toThrow("native_runner_checkpoint_owner_lost"); + expect((await db.select({ part: nativeRunnerCheckpoints.part }).from(nativeRunnerCheckpoints) + .where(eq(nativeRunnerCheckpoints.runId, f.runId))).map(row => row.part)).toEqual(["header"]); + }); +}); diff --git a/server/src/services/native-runtime/native-runner-checkpoint.ts b/server/src/services/native-runtime/native-runner-checkpoint.ts index d4a9b2be91e..8582d339ed7 100644 --- a/server/src/services/native-runtime/native-runner-checkpoint.ts +++ b/server/src/services/native-runtime/native-runner-checkpoint.ts @@ -3,7 +3,7 @@ import { NATIVE_PREPARATION_FENCING_VERSION } from "./native-active-runs.js"; import { notifyLiveRunRecovery } from "../live-run-recovery-notifications.js"; import { createHash } from "node:crypto"; import { and, eq, exists, gt, isNull, lte, inArray, notExists, or, sql, type SQL } from "drizzle-orm"; -import { checkpointDb, heartbeatRuns, issues, nativeRunFinalizations, nativeRunnerCheckpoints, type Db } from "@paperclipai/db"; +import { checkpointDb, heartbeatRuns, issues, nativeRunFinalizations, nativeRunnerCheckpoints, primeDatabaseWorkTransactionId, type Db } from "@paperclipai/db"; import { inspectDurablePrpCheckpointHeader, inspectUnauthenticatedBootstrapCheckpoint, inspectUnclaimedBootstrapCheckpoint, decodeDurablePrpCheckpoint, type DurablePrpCheckpoint, type DurablePrpControlPlane, type DurableRecoveryIdentity } from "../../vendor/paperclip-runner/index.js"; import { createLocalEncryptedCipher, deriveLocalEncryptedInstanceSecret } from "../../secrets/local-encrypted-provider.js"; import { legacyControllerBootId } from "../legacy-controller-lease.js"; @@ -76,9 +76,12 @@ export async function lockNativeRunnerCheckpointOwner( // wall time. Sampling in the locking SELECT can precede its lock wait. // Keep both steps in one statement without another database round trip. const locked = nativeRunnerCheckpointOwnerLock(tx, owner); - const [owned] = await tx.with(locked).select({ leaseOwner: locked.leaseOwner, - leaseExpiresAt: locked.leaseExpiresAt, resultId: locked.resultId, phase: locked.phase, now: sql`clock_timestamp()` }).from(locked); - if (!owned) throw new Error("native_runner_checkpoint_owner_lost"); + const [row] = await tx.with(locked).select({ leaseOwner: locked.leaseOwner, + leaseExpiresAt: locked.leaseExpiresAt, resultId: locked.resultId, phase: locked.phase, now: sql`clock_timestamp()`, + xid: sql`pg_current_xact_id()::text` }).from(locked); + if (!row) throw new Error("native_runner_checkpoint_owner_lost"); + const { xid, ...owned } = row; + primeDatabaseWorkTransactionId(tx, xid); return owned; } From 6b631dd4773a3c6ac29e4a4ca0fd85bdfeca8d4e Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 20:48:21 -0600 Subject: [PATCH 06/10] fix(recovery): check held retained orphans only for candidate leases (#21542) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The ownership reconciliation pass lists in-progress records of each kind every 30 seconds, and settles the records whose owner is gone > - One kind, the retained orphan sandbox, lists leases with a large SQL filter that has correlated subqueries on `heartbeat_runs`, `issues`, `issue_recovery_actions` and `environment_leases` > - The planner ran those subqueries for every released, failed and expired lease in history, before the cheap orphan predicates. The cost grew with lease history, not with the number of orphans > - On a busy hosted database one page of this listing ran for 17 to 138 seconds as one statement. It held an AccessShareLock on `heartbeat_runs` for all of that time, so an online `ALTER TABLE heartbeat_runs ADD COLUMN` with a 5 second lock timeout failed on every try > - This pull request runs a cheap candidate filter first and evaluates the held checks only for candidate rows > - The benefit is that the listing cost now follows the number of candidates. Schema changes on `heartbeat_runs` no longer wait behind it, and the pass reads far fewer pages ## Linked Issues or Issue Description No public issue exists. This describes the bug in the format of the bug report template. **What happened?** A hosted instance could not apply a pending migration that adds a column to `heartbeat_runs`. The deployer sets `lock_timeout = 5s` and tries three times. All three tries timed out. The database showed one session that ran `select "id", "company_id", "heartbeat_run_id", "issue_id", "metadata", "updated_at" from "environment_leases" ...` in state `active` for up to 138 seconds. Its transaction start and query start differed by less than 0.1 seconds, so it was one statement. It held an AccessShareLock on `heartbeat_runs`. That instance had about 18,600 lease rows: 13,240 expired, 4,695 released, 610 failed, 69 retained, 17 active and 9 pending cleanup. Each failed try also stalled all statements on `heartbeat_runs` for up to 5 seconds. **Expected behavior** The ownership listing must finish in a short time that does not grow with lease history. It must not keep a lock on `heartbeat_runs` long enough to block an online schema change. **Steps to reproduce** 1. Use a database with many terminal environment leases (expired, released or failed) and few or no retained orphan sandboxes. 2. Run the `environment_retained_orphan_sandbox` listing (`OWNABLE_KINDS`, `listInProgress`), or wait for the ownership reconciliation pass. 3. Run `EXPLAIN (ANALYZE, BUFFERS)` on the listing. The `heartbeat_runs` primary key subplans run once for each terminal lease, not only for candidates. 4. While the listing runs, run `ALTER TABLE heartbeat_runs ADD COLUMN probe jsonb` with `lock_timeout = '5s'`. The ALTER waits until the listing ends. **Paperclip version or commit** `master` at 0a96f972e1. **Database mode** Hosted PostgreSQL. Reproduced on embedded PostgreSQL 18. **Additional context** The 138 second session was not a transaction held open across remote work. The call path is `registerOwnershipReconciliationWork` → `runOwnershipReconciliationPass(db)` → `kind.listInProgress(db, 200, offset)` on the pool, with no transaction. Only the `leaseKind` and `operationKind` listings send this exact unqualified column list. Of these, only this kind's filter reads `heartbeat_runs`. ## What Changed - `releasableRetainedOrphanCondition` and `nextRetainedOrphanDueAt` first apply `retainedOrphanCandidateCondition`. This is a filter with no subqueries and no NULL results: cleanup status, run, provider, lease status, lease policy, and the metadata keys of the three retained orphan shapes. Expired, active and pending-cleanup leases fail it at once. - The held checks (always held and softly held) now run inside `CASE WHEN THEN ... ELSE false END`. PostgreSQL evaluates a CASE in order, so the correlated subqueries run only for candidates. The planner can no longer split the `NOT (a OR b OR ...)` into separate top-level subplans and run them before the candidate predicates. - The two `heartbeat_runs` checks of the always-held part are now one scalar subquery with a primary key lookup. It reads `runner_profile_json->'nativeWorkspaceSync'` once through a fenced lateral, instead of about 12 `#>>` path reads. A scalar subquery cannot become a hashed subplan, so the planner cannot replace it with a full scan of `heartbeat_runs`. - The listed rows, the order and the paging do not change. `softlyHeldSql` is now exported so that the test can rebuild the previous condition. - The paging stays OFFSET-based. The `listInProgress(db, limit, offset)` contract is shared by all ownership kinds and by the I2 rule, which compares consecutive passes. Keyset paging is a change to that shared contract, so this pull request does not make it. ## Verification - `cd server && npx vitest run src/__tests__/environment-retained-orphan-listing-cost.test.ts src/__tests__/environment-retained-orphan-sandbox.test.ts src/__tests__/abandoned-workspace-files-hold.test.ts`: 3 files, 55 tests pass. - New test `environment-retained-orphan-listing-cost.test.ts`: - Equivalence: four seeded random fixtures of 600 leases each, across two companies. The fixtures cover every lease status and policy, runs of each status and runtime mode, valid and invalid native workspace descriptors, every always-held metadata key, sibling leases on the same sandbox, open and resolved recovery actions, open runs on the issue, and past, future and absent expiry. Every page (limit 7 and 200) equals the previous SQL, which is kept only in the test. `nextRetainedOrphanDueAt` also equals the previous query. - Red/green: over 2,000 history leases with the hosted status mix (71% expired, 25% released, 4% failed), the test counts the `heartbeat_runs` rows that the listing reads, from `EXPLAIN (ANALYZE, FORMAT JSON)`. On `master` it reads 1,760 rows for 1 candidate, so the test fails. With this change the count is within the bound of 10 per candidate. - Incident regression: while listings run in a loop over 5,000 history leases, `ALTER TABLE heartbeat_runs ADD COLUMN ...` and `DROP COLUMN` with `lock_timeout = '5s'` complete in less than 5 seconds. - Local measurement (embedded PostgreSQL 18, Apple M-series, warm cache), not in CI: - 18,640 leases with the hosted status mix and a 19 KB runner profile per run. `master`: 422 ms, 296,771 buffers (12,357 read). This change: 26 ms, 36,004 buffers. The `heartbeat_runs` subplans run 13 times (once for each candidate), not once for each lease. - 300,000 history leases. `master`: one listing page takes 2,822 ms, and the ALTER under load waits 2,448 ms. This change: 212 ms, and the ALTER takes 96 ms. - `pnpm check:module-boundaries` passes. ## Risks - Low risk. The listing and the due-time query return the same rows, and the equivalence test compares them. The settle is unchanged and still re-checks every hold under the row lock. - The candidate filter must stay a superset of `retainedOrphanSandboxCondition`. If a later change adds a retained orphan shape and does not update the candidate filter, the listing skips that shape. The equivalence test catches this only for the shapes in the fixture. - The listing still reads each lease row once with cheap predicates. A partial index on the candidate filter could make it scale only with candidates. This pull request does not add a migration. - The ownership pass still lists all kinds every 30 seconds when its fingerprint changes. This pull request does not change that cadence. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, embedded PostgreSQL runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- ...nment-retained-orphan-listing-cost.test.ts | 299 ++++++++++++++++++ .../environment-retained-orphan-settlement.ts | 61 ++-- 2 files changed, 338 insertions(+), 22 deletions(-) create mode 100644 server/src/__tests__/environment-retained-orphan-listing-cost.test.ts diff --git a/server/src/__tests__/environment-retained-orphan-listing-cost.test.ts b/server/src/__tests__/environment-retained-orphan-listing-cost.test.ts new file mode 100644 index 00000000000..767c09a0ab8 --- /dev/null +++ b/server/src/__tests__/environment-retained-orphan-listing-cost.test.ts @@ -0,0 +1,299 @@ +import { randomUUID } from "node:crypto"; +import { and, asc, sql, type SQL } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest"; +import { + agents, + companies, + createDb, + environmentLeases, + environments, + executionWorkspaces, + heartbeatRuns, + issueRecoveryActions, + issues, +} from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { + nextRetainedOrphanDueAt, + releasableRetainedOrphanCondition, + retainedOrphanSandboxCondition, + softlyHeldSql, +} from "../services/environment-retained-orphan-settlement.js"; +import { abandonedWorkspaceFilesHeldOnLeaseRowSql } from "../services/remote-workspace-abandoned-files-hold.js"; +import { OWNABLE_KINDS } from "../services/recovery/ownership-registry.js"; + +const support = await getEmbeddedPostgresTestSupport(); +const suite = support.supported ? describe : describe.skip; +const KIND = OWNABLE_KINDS.find(kind => kind.id === "environment_retained_orphan_sandbox")!; +const TERMINAL = ["succeeded", "failed", "cancelled", "timed_out", "interrupted"]; + +function previousAlwaysHeldSql(): SQL { + return sql`(${environmentLeases.metadata} ?| array['nativeWorkspaceExportResume', 'sandboxStopAndRetain', + 'workspaceRestoreRecovery', 'workspaceRecoveryCleanup', 'runnerTaskSourceConsumer', 'reusableSandboxKeptAfterReplacement'] + or ${abandonedWorkspaceFilesHeldOnLeaseRowSql()} + or coalesce(${environmentLeases.metadata}->>'pendingCleanupInFlight', 'false') = 'true' + or coalesce(${environmentLeases.metadata}#>>'{legacyShutdownWorkspaceSettlement,state}' = 'pending', false) + or exists (select 1 from heartbeat_runs run where run.id = ${environmentLeases.heartbeatRunId} + and run.status not in ('succeeded', 'failed', 'cancelled', 'timed_out', 'interrupted')) + or exists (select 1 from heartbeat_runs run where run.id = ${environmentLeases.heartbeatRunId} + and run.runtime_mode = 'native' + and run.runner_profile_json#>>'{nativeWorkspaceSync,schema}' in + ('paperclip.native-workspace-sync/v1', 'paperclip.native-workspace-sync/v2', 'paperclip.native-workspace-sync/v3') + and run.runner_profile_json#>>'{nativeWorkspaceSync,state}' = 'prepared' + and coalesce(run.runner_profile_json#>>'{nativeWorkspaceSync,descriptorSha256}' ~ '^[0-9a-f]{64}$', false) + and coalesce(run.runner_profile_json#>>'{nativeWorkspaceSync,baselineSha256}' ~ '^[0-9a-f]{64}$', false) + and coalesce(jsonb_typeof(run.runner_profile_json#>'{nativeWorkspaceSync,finalHostSha256}'), 'null') = 'null' + and jsonb_typeof(run.runner_profile_json#>'{nativeWorkspaceSync,workspaceId}') = 'string' + and jsonb_typeof(run.runner_profile_json#>'{nativeWorkspaceSync,remoteCwd}') = 'string' + and coalesce(run.runner_profile_json#>>'{nativeWorkspaceSync,resourceDisposition}', 'destroy') + in ('keep_running', 'stop_and_retain', 'destroy') + and run.runner_profile_json#>>'{nativeWorkspaceSync,leaseId}' = ${environmentLeases.id}::text + and run.runner_profile_json#>>'{nativeWorkspaceSync,providerLeaseId}' = ${environmentLeases.providerLeaseId} + and coalesce(run.runner_profile_json#>>'{nativeExecutionInput,workspace,access}', '') <> 'none') + or exists (select 1 from environment_leases other where other.id <> ${environmentLeases.id} + and other.provider = ${environmentLeases.provider} and other.provider_lease_id = ${environmentLeases.providerLeaseId} + and other.status in ('active', 'retained', 'pending_cleanup')))`; +} + +function previousReleasableCondition(now?: Date) { + return and(retainedOrphanSandboxCondition(), sql`not ${previousAlwaysHeldSql()}`, sql`not (${softlyHeldSql(now)})`); +} + +function random(seed: number) { + let state = seed >>> 0; + return () => { + state = (state + 0x6d2b79f5) >>> 0; + let value = state; + value = Math.imul(value ^ (value >>> 15), value | 1); + value ^= value + Math.imul(value ^ (value >>> 7), value | 61); + return ((value ^ (value >>> 14)) >>> 0) / 4294967296; + }; +} + +type PlanNode = { "Relation Name"?: string; "Actual Loops"?: number; "Actual Rows"?: number; + "Rows Removed by Filter"?: number; Plans?: PlanNode[] }; + +function heartbeatRunRowsRead(node: PlanNode): number { + const own = node["Relation Name"] === "heartbeat_runs" + ? (node["Actual Loops"] ?? 0) * ((node["Actual Rows"] ?? 0) + (node["Rows Removed by Filter"] ?? 0)) : 0; + return own + (node.Plans ?? []).reduce((total, child) => total + heartbeatRunRowsRead(child), 0); +} + +suite("retained orphan sandbox listing cost", () => { + let db!: ReturnType; + let temporary: Awaited> | null = null; + const tenants: Array<{ companyId: string; agentId: string; issueIds: string[]; workspaceIds: string[] }> = []; + let environmentId = ""; + + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("paperclip-retained-orphan-listing-"); + db = createDb(temporary.connectionString); + environmentId = randomUUID(); + await db.insert(environments).values({ id: environmentId, name: "Fake Sandbox", driver: "sandbox", + status: "active", config: { provider: "fake", image: "ubuntu:24.04" } }); + for (const prefix of ["ROA", "ROB"]) { + const companyId = randomUUID(); + const agentId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: prefix, issuePrefix: prefix, + requireBoardApprovalForNewAgents: false }); + await db.insert(agents).values({ id: agentId, companyId, name: "Coder", role: "engineer", status: "idle", + adapterType: "codex_local", adapterConfig: {}, runtimeConfig: {}, permissions: {} }); + const workspaceIds: string[] = []; + for (let index = 0; index < 5; index += 1) { + const [workspace] = await db.insert(executionWorkspaces).values({ companyId, mode: "isolated_workspace", + strategyType: "git_worktree", name: `${prefix} workspace ${index}` }).returning({ id: executionWorkspaces.id }); + workspaceIds.push(workspace!.id); + } + const issueIds: string[] = []; + for (let index = 0; index < 6; index += 1) { + const issueId = randomUUID(); + await db.insert(issues).values({ id: issueId, companyId, title: `${prefix} issue ${index}`, + executionWorkspaceId: index < 3 ? workspaceIds[index] : null }); + issueIds.push(issueId); + } + tenants.push({ companyId, agentId, issueIds, workspaceIds }); + } + }, 60_000); + + afterEach(async () => { + await db.delete(issueRecoveryActions); + await db.delete(environmentLeases); + await db.delete(heartbeatRuns); + }); + + afterAll(async () => { + await temporary?.cleanup(); + }); + + async function seedVariedFixture(seed: number, count: number) { + const next = random(seed); + const pick = (values: readonly T[]): T => values[Math.floor(next() * values.length)]!; + const chance = (probability: number) => next() < probability; + const runs: Array = []; + const leases: Array = []; + const base = Date.now() - 24 * 60 * 60_000; + for (let index = 0; index < count; index += 1) { + const tenant = pick(tenants); + const leaseId = randomUUID(); + const provider = pick(["fake", "fake", "fake", "local", null]); + const providerLeaseId = chance(0.9) ? `sbx-${Math.floor(next() * 300)}` : null; + let runId: string | null = null; + if (chance(0.9)) { + runId = randomUUID(); + const sync = pick(["none", "prepared", "prepared", "access_none", "other_sandbox", "bad_descriptor", "final"]); + const runnerProfileJson = sync === "none" ? {} : { + ...(sync === "access_none" ? { nativeExecutionInput: { workspace: { access: "none" } } } : {}), + nativeWorkspaceSync: { + schema: pick(["paperclip.native-workspace-sync/v3", "paperclip.native-workspace-sync/v1", "unknown"]), + state: "prepared", descriptorSha256: sync === "bad_descriptor" ? "zz" : "a".repeat(64), + baselineSha256: "b".repeat(64), finalHostSha256: sync === "final" ? "c".repeat(64) : null, + workspaceId: randomUUID(), remoteCwd: "/workspace", resourceDisposition: pick(["destroy", "keep_running", "odd"]), + leaseId, providerLeaseId: sync === "other_sandbox" ? "sbx-elsewhere" : providerLeaseId, + }, + }; + runs.push({ id: runId, companyId: tenant.companyId, agentId: tenant.agentId, issueId: pick(tenant.issueIds), + nativeIssueId: chance(0.2) ? pick(tenant.issueIds) : null, invocationSource: "on_demand", + status: chance(0.06) ? pick(["running", "queued", "scheduled_retry"]) : pick(TERMINAL), nextEventSeq: 1, + runtimeMode: pick(["native", "native", "legacy"]), runnerProfileJson }); + } + const terminationRunId = chance(0.85) ? runId : randomUUID(); + const metadata: Record = {}; + if (chance(0.75)) metadata.orphanedRunLeaseRecovery = { recoveredAt: "x" }; + if (chance(0.85)) metadata.sandboxStopAndRetainReceipt = { stoppedAt: "x" }; + if (chance(0.75)) metadata.remoteExecutionTermination = { state: pick(["stopped", "stopped", "running"]), + leaseId: chance(0.85) ? leaseId : randomUUID(), runId: terminationRunId }; + for (const key of ["nativeWorkspaceExportResume", "sandboxStopAndRetain", "workspaceRestoreRecovery", + "workspaceRecoveryCleanup", "runnerTaskSourceConsumer", "reusableSandboxKeptAfterReplacement"]) { + if (chance(0.04)) metadata[key] = { at: "x" }; + } + if (chance(0.1)) metadata.pendingCleanupInFlight = pick([true, false, "true", "maybe"]); + if (chance(0.06)) metadata.legacyShutdownWorkspaceSettlement = { state: pick(["pending", "done"]) }; + const expiresRoll = next(); + leases.push({ id: leaseId, companyId: tenant.companyId, environmentId, + executionWorkspaceId: chance(0.7) ? pick(tenant.workspaceIds) : null, + issueId: chance(0.9) ? pick(tenant.issueIds) : null, heartbeatRunId: runId, + status: pick(["released", "released", "released", "failed", "failed", "retained", "retained", "expired", "active", "pending_cleanup"]), + leasePolicy: pick(["ephemeral", "ephemeral", "reuse_by_environment", "reuse_by_environment", "retain_on_failure"]), + provider, providerLeaseId, cleanupStatus: pick(["success", "success", "success", "success", "failed", null]), + metadata: chance(0.03) ? null : metadata, + expiresAt: expiresRoll < 0.3 ? null : new Date(Date.now() + (expiresRoll < 0.6 ? -1 : 1) * 60 * 60_000), + createdAt: new Date(base + Math.floor(next() * 5) * 1000) }); + } + for (let index = 0; index < runs.length; index += 200) await db.insert(heartbeatRuns).values(runs.slice(index, index + 200)); + for (let index = 0; index < leases.length; index += 200) { + await db.insert(environmentLeases).values(leases.slice(index, index + 200)); + } + for (const tenant of tenants) { + for (const issueId of tenant.issueIds) { + if (!chance(0.25)) continue; + await db.insert(issueRecoveryActions).values({ companyId: tenant.companyId, sourceIssueId: issueId, + kind: "active_run_watchdog", status: pick(["active", "resolved", "cancelled", "escalated"]), + cause: "fixture", fingerprint: randomUUID(), nextAction: "Recover the workspace." }); + } + } + } + + async function previousPage(limit: number, offset: number, now?: Date) { + const rows = await db.select({ id: environmentLeases.id }).from(environmentLeases) + .where(previousReleasableCondition(now)).orderBy(asc(environmentLeases.id)).limit(limit + 1).offset(offset); + return { truncated: rows.length > limit, ids: rows.slice(0, limit).map(row => row.id) }; + } + + it("lists exactly the rows the previous listing listed, page by page, over a varied fixture", async () => { + for (const seed of [11, 29, 47, 83]) { + await seedVariedFixture(seed, 600); + const expected = await previousPage(10_000, 0); + expect(expected.ids.length).toBeGreaterThan(0); + expect(expected.ids.length).toBeLessThan(600); + for (const limit of [7, 200]) { + for (let offset = 0; ; offset += limit) { + const [listed, previous] = await Promise.all([KIND.listInProgress!(db, limit, offset), previousPage(limit, offset)]); + expect(listed.records.map(record => record.recordId)).toEqual(previous.ids); + expect(listed.truncated).toBe(previous.truncated); + if (!previous.truncated) break; + } + } + const now = new Date(); + const [current] = await db.select({ count: sql`count(*)::int` }).from(environmentLeases) + .where(releasableRetainedOrphanCondition(now)); + expect(current!.count).toBe(expected.ids.length); + const [previousDue] = await db.select({ due: sql`min(${environmentLeases.expiresAt})` }) + .from(environmentLeases).where(and(retainedOrphanSandboxCondition(), sql`not ${previousAlwaysHeldSql()}`, + sql`${softlyHeldSql(now)}`, sql`${environmentLeases.expiresAt} is not null`)); + expect((await nextRetainedOrphanDueAt(db, now))?.getTime() ?? null) + .toBe(previousDue?.due ? new Date(previousDue.due).getTime() : null); + await db.delete(issueRecoveryActions); + await db.delete(environmentLeases); + await db.delete(heartbeatRuns); + } + }, 120_000); + + async function seedHistory(count: number) { + const tenant = tenants[0]!; + await db.execute(sql` + with history as ( + select gen_random_uuid() as run_id, gen_random_uuid() as lease_id, g from generate_series(1, ${count}) g + ), inserted as ( + insert into heartbeat_runs (id, company_id, agent_id, issue_id, invocation_source, status, next_event_seq, + runtime_mode, runner_profile_json) + select run_id, ${tenant.companyId}::uuid, ${tenant.agentId}::uuid, ${tenant.issueIds[0]}::uuid, 'on_demand', + 'succeeded', 1, 'native', jsonb_build_object('nativeWorkspaceSync', jsonb_build_object( + 'schema', 'paperclip.native-workspace-sync/v3', 'state', 'finalized', 'leaseId', lease_id::text)) + from history returning id + ) + insert into environment_leases (id, company_id, environment_id, execution_workspace_id, issue_id, heartbeat_run_id, + status, lease_policy, provider, provider_lease_id, cleanup_status, metadata, expires_at) + select lease_id, ${tenant.companyId}::uuid, ${environmentId}::uuid, ${tenant.workspaceIds[0]}::uuid, + ${tenant.issueIds[0]}::uuid, run_id, + case when g % 100 < 71 then 'expired' when g % 100 < 96 then 'released' else 'failed' end, + case when g % 3 = 0 then 'reuse_by_environment' else 'ephemeral' end, + 'fake', 'sbx-history-' || (g % 50), 'success', jsonb_build_object('sandboxStopAndRetainReceipt', '{}'::jsonb), + now() + interval '1 day' + from history`); + await db.execute(sql`analyze heartbeat_runs; analyze environment_leases`); + } + + async function listingPlan() { + const query = db.select({ id: environmentLeases.id }).from(environmentLeases) + .where(releasableRetainedOrphanCondition()).orderBy(asc(environmentLeases.id)).limit(201).offset(0).toSQL(); + const rows = await db.$client.unsafe(`explain (analyze, format json) ${query.sql}`, query.params as never[]); + return (rows[0] as unknown as { "QUERY PLAN": Array<{ Plan: PlanNode }> })["QUERY PLAN"][0]!.Plan; + } + + it("reads heartbeat runs only for retained orphan candidates, not for the whole lease history", async () => { + await seedHistory(2_000); + await seedVariedFixture(5, 20); + const [{ candidates }] = await db.select({ candidates: sql`count(*)::int` }).from(environmentLeases) + .where(retainedOrphanSandboxCondition()) as Array<{ candidates: number }>; + const plan = await listingPlan(); + expect(heartbeatRunRowsRead(plan)).toBeLessThanOrEqual(10 * candidates + 50); + }, 120_000); + + it("lets a schema change take the heartbeat_runs lock within a 5 second lock timeout while listings run", async () => { + await seedHistory(5_000); + let stop = false; + let listings = 0; + const load = (async () => { + while (!stop) { + await KIND.listInProgress!(db, 200, 0); + listings += 1; + } + })(); + const ddl = createDb(temporary!.connectionString); + try { + await new Promise(resolve => setTimeout(resolve, 200)); + const started = Date.now(); + await ddl.transaction(async tx => { + await tx.execute(sql`set local lock_timeout = '5s'`); + await tx.execute(sql`alter table heartbeat_runs add column retained_orphan_listing_probe jsonb`); + await tx.execute(sql`alter table heartbeat_runs drop column retained_orphan_listing_probe`); + }); + expect(Date.now() - started).toBeLessThan(5_000); + } finally { + stop = true; + await load; + await ddl.$client.end({ timeout: 1 }); + } + expect(listings).toBeGreaterThan(0); + }, 120_000); +}); diff --git a/server/src/services/environment-retained-orphan-settlement.ts b/server/src/services/environment-retained-orphan-settlement.ts index 53fa0be16bc..911a295299c 100644 --- a/server/src/services/environment-retained-orphan-settlement.ts +++ b/server/src/services/environment-retained-orphan-settlement.ts @@ -1,4 +1,4 @@ -import { and, eq, inArray, ne, notInArray, or, sql } from "drizzle-orm"; +import { and, eq, inArray, ne, notInArray, or, sql, type SQL } from "drizzle-orm"; import { environmentLeases, heartbeatRuns, issueRecoveryActions, type Db } from "@paperclipai/db"; import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js"; import { nativeWorkspaceExportPendingForLease } from "./native-runtime/native-workspace-finalization-state.js"; @@ -97,7 +97,7 @@ function orphanedRunSandboxCondition() { * `heldBy`, so a held orphan is not listed (and not polled) until its last * reference clears or its retention window ends. The settle re-checks all of * them, plus the pending native export, under the row lock. */ -function softlyHeldSql(now?: Date) { +export function softlyHeldSql(now?: Date) { const at = now ? sql`${now.toISOString()}::timestamptz` : sql`now()`; const issue = sql`coalesce(run.native_issue_id, run.issue_id, ${environmentLeases.issueId})`; return sql`(${environmentLeases.expiresAt} is null or ${environmentLeases.expiresAt} > ${at}) and ( @@ -116,8 +116,7 @@ function alwaysHeldSql() { or ${abandonedWorkspaceFilesHeldOnLeaseRowSql()} or coalesce(${environmentLeases.metadata}->>'pendingCleanupInFlight', 'false') = 'true' or coalesce(${environmentLeases.metadata}#>>'{legacyShutdownWorkspaceSettlement,state}' = 'pending', false) - or exists (select 1 from heartbeat_runs run where run.id = ${environmentLeases.heartbeatRunId} - and run.status not in ('succeeded', 'failed', 'cancelled', 'timed_out', 'interrupted')) + or coalesce((select run.status not in ('succeeded', 'failed', 'cancelled', 'timed_out', 'interrupted') -- A prepared native workspace bound to this exact sandbox, for a run with -- workspace access. This is deliberately no wider than the settle's own -- export check: an orphan the settle would release must never be hidden @@ -125,21 +124,23 @@ function alwaysHeldSql() { -- are listed and held by the settle until their export settles. -- The descriptor must also be one the settle's reader accepts -- (readNativeWorkspaceSyncReference); an invalid one holds nothing. - or exists (select 1 from heartbeat_runs run where run.id = ${environmentLeases.heartbeatRunId} - and run.runtime_mode = 'native' - and run.runner_profile_json#>>'{nativeWorkspaceSync,schema}' in - ('paperclip.native-workspace-sync/v1', 'paperclip.native-workspace-sync/v2', 'paperclip.native-workspace-sync/v3') - and run.runner_profile_json#>>'{nativeWorkspaceSync,state}' = 'prepared' - and coalesce(run.runner_profile_json#>>'{nativeWorkspaceSync,descriptorSha256}' ~ '^[0-9a-f]{64}$', false) - and coalesce(run.runner_profile_json#>>'{nativeWorkspaceSync,baselineSha256}' ~ '^[0-9a-f]{64}$', false) - and coalesce(jsonb_typeof(run.runner_profile_json#>'{nativeWorkspaceSync,finalHostSha256}'), 'null') = 'null' - and jsonb_typeof(run.runner_profile_json#>'{nativeWorkspaceSync,workspaceId}') = 'string' - and jsonb_typeof(run.runner_profile_json#>'{nativeWorkspaceSync,remoteCwd}') = 'string' - and coalesce(run.runner_profile_json#>>'{nativeWorkspaceSync,resourceDisposition}', 'destroy') - in ('keep_running', 'stop_and_retain', 'destroy') - and run.runner_profile_json#>>'{nativeWorkspaceSync,leaseId}' = ${environmentLeases.id}::text - and run.runner_profile_json#>>'{nativeWorkspaceSync,providerLeaseId}' = ${environmentLeases.providerLeaseId} - and coalesce(run.runner_profile_json#>>'{nativeExecutionInput,workspace,access}', '') <> 'none') + or (run.runtime_mode = 'native' + and profile.sync->>'schema' in + ('paperclip.native-workspace-sync/v1', 'paperclip.native-workspace-sync/v2', 'paperclip.native-workspace-sync/v3') + and profile.sync->>'state' = 'prepared' + and coalesce(profile.sync->>'descriptorSha256' ~ '^[0-9a-f]{64}$', false) + and coalesce(profile.sync->>'baselineSha256' ~ '^[0-9a-f]{64}$', false) + and coalesce(jsonb_typeof(profile.sync->'finalHostSha256'), 'null') = 'null' + and jsonb_typeof(profile.sync->'workspaceId') = 'string' + and jsonb_typeof(profile.sync->'remoteCwd') = 'string' + and coalesce(profile.sync->>'resourceDisposition', 'destroy') in ('keep_running', 'stop_and_retain', 'destroy') + and profile.sync->>'leaseId' = ${environmentLeases.id}::text + and profile.sync->>'providerLeaseId' = ${environmentLeases.providerLeaseId} + and coalesce(profile.access, '') <> 'none') + from heartbeat_runs run + cross join lateral (select run.runner_profile_json->'nativeWorkspaceSync' as sync, + run.runner_profile_json#>>'{nativeExecutionInput,workspace,access}' as access offset 0) profile + where run.id = ${environmentLeases.heartbeatRunId}), false) or exists (select 1 from environment_leases other where other.id <> ${environmentLeases.id} and other.provider = ${environmentLeases.provider} and other.provider_lease_id = ${environmentLeases.providerLeaseId} and other.status in ('active', 'retained', 'pending_cleanup')))`; @@ -148,15 +149,31 @@ function alwaysHeldSql() { /** Retained orphans that nothing appears to reference now. */ // The listing is built once, so it reads the database clock. export function releasableRetainedOrphanCondition(now?: Date) { - return and(retainedOrphanSandboxCondition(), sql`not ${alwaysHeldSql()}`, sql`not (${softlyHeldSql(now)})`); + return onlyForRetainedOrphans(sql`not ${alwaysHeldSql()} and not (${softlyHeldSql(now)})`); +} + +function retainedOrphanCandidateCondition() { + return and(eq(environmentLeases.cleanupStatus, "success"), sql`${environmentLeases.heartbeatRunId} is not null`, + sql`${environmentLeases.provider} is not null`, sql`${environmentLeases.providerLeaseId} is not null`, + or(and(eq(environmentLeases.status, "released"), eq(environmentLeases.leasePolicy, "ephemeral"), + sql`${environmentLeases.metadata} ? ${ORPHANED_SANDBOX_RECOVERY_KEY}`, + sql`${environmentLeases.metadata} ? 'sandboxStopAndRetainReceipt'`), + and(eq(environmentLeases.leasePolicy, "reuse_by_environment"), inArray(environmentLeases.status, ["released", "failed"]), + sql`coalesce(${environmentLeases.metadata}#>>'{remoteExecutionTermination,state}' = 'stopped', false)`), + and(eq(environmentLeases.status, "retained"), eq(environmentLeases.leasePolicy, "reuse_by_environment"), + sql`${environmentLeases.executionWorkspaceId} is not null`))); +} + +function onlyForRetainedOrphans(condition: SQL, candidate: SQL | undefined = retainedOrphanSandboxCondition()) { + return and(retainedOrphanCandidateCondition(), sql`(case when ${candidate} then ${condition} else false end)`); } /** When the next retained orphan's retention window ends while only soft * references hold it; the reconciler wakes then to release it. */ export async function nextRetainedOrphanDueAt(db: Db, now: Date): Promise { const [row] = await db.select({ due: sql`min(${environmentLeases.expiresAt})` }).from(environmentLeases) - .where(and(retainedOrphanSandboxCondition(), sql`not ${alwaysHeldSql()}`, sql`${softlyHeldSql(now)}`, - sql`${environmentLeases.expiresAt} is not null`)); + .where(onlyForRetainedOrphans(sql`not ${alwaysHeldSql()} and ${softlyHeldSql(now)}`, + and(retainedOrphanSandboxCondition(), sql`${environmentLeases.expiresAt} is not null`))); return row?.due ? new Date(row.due) : null; } From f6c5986b6112951369fd9e3d688bac4b50abf2f7 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 21:10:32 -0600 Subject: [PATCH 07/10] feat(connections): machine sign-in (MCP OAuth client credentials) for organization connections (#21473) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents use apps through connections to remote MCP servers > - Today an organization connection must sign in as a person (browser sign-in) or use a static key > - Service accounts and CI-style integrations have no person who can sign in > - The MCP OAuth Client Credentials extension (2026-07-28) defines machine-to-machine sign-in for this case > - This pull request adds "Machine sign-in" for organization connections, with a Paperclip-held signing key (recommended) or a client secret > - The benefit is that organizations can connect automation-owned MCP servers safely, with credentials only in the vault and bound to one sign-in service ## Linked Issues or Issue Description **Subsystem affected** Connections: remote MCP OAuth (server `tool-access/oauth`, tool gateway token refresh) and the custom MCP server setup flow in the UI. **Problem or motivation** Organization-owned automation (service accounts, CI integrations) cannot connect an MCP server that requires OAuth, because no person can complete a browser sign-in. The MCP OAuth Client Credentials extension (`io.modelcontextprotocol/oauth-client-credentials`) solves this with `grant_type=client_credentials` and RFC 7523 signed client assertions. **Proposed solution** Add machine sign-in for organization connections only. Paperclip creates a keypair per connection, keeps the private key in the company vault, and shows the public key (PEM and JWK) for the admin to register. A client secret is also supported. The sign-in service (issuer and token endpoint) is pinned at setup. Tokens use the existing organization grant refresh path, so caching, concurrent dedupe and 401 re-request reuse the existing lease and single flight. **Alternatives considered** Reuse the deployment-wide `PAPERCLIP_TOOL_OAUTH_*` client env path that already sent `client_credentials`. Rejected: it is not per connection, it keeps secrets in env, and it has no issuer pin. Route machine sign-in through `connectGalleryApp`. Rejected: its rollback deletes the draft on a failed test, which would lose a key the admin already registered. **Roadmap alignment** Phase 1 of the enterprise MCP plan (no enterprise single sign-on needed). OIDC login and enterprise-managed authorization (ID-JAG) are later phases. ## What Changed - `oauth/client-credentials.ts`: keypair generation (RS256, or ES256 when the service requires it), RFC 7638 key IDs, and RFC 7523 assertions (`iss` = `sub` = client ID, `aud` = token endpoint, 60 s `exp`, unique `jti`) with Node `crypto` only. - `oauth/client.ts`: the token exchange accepts a signed assertion and then sends only `client_assertion_type` and `client_assertion` (no `client_id` or secret), as the extension specifies. - `oauth/client-credentials-service.ts`: discovery of the sign-in service (protected-resource metadata, then RFC 8414/OIDC metadata with issuer check, or a token URL entered directly), issuer pinning, vault-only key and secret handling, draft creation, test-and-activate, and no-downtime rotation (test the new secret first; stage a new key and switch only after the provider accepts it). - `oauth/grant-refresh.ts` and `oauth/connection-refresh.ts`: machine sign-in requests a new access token with the connection's own credentials instead of needing a refresh token. It never falls back to deployment-wide OAuth clients. - `catalog/discovery.ts`: a rejected token on a machine sign-in connection never re-runs discovery or rewrites the pinned endpoints. - `mcp-http.ts` and the tool gateway: declare `io.modelcontextprotocol/oauth-client-credentials` in client capabilities only for connections that use it. - Routes (with OpenAPI and activity logging): `POST /companies/{companyId}/tools/machine-sign-in`, `GET|POST /tool-connections/{id}/machine-sign-in[/test|/rotate|/rotate/complete]`. Removal activity now records `machineSignIn: true`. The runner API treats these routes as sensitive. - Shared validators and types for machine sign-in. - UI: "Machine sign-in" under Advanced authentication in the custom MCP server flow (organization connections only), `MachineSignInSetup` (choose Paperclip key or client secret, register the public key, test), and Storybook stories for each setup state. - Docs: `doc/connections/MCP-CLIENT-CREDENTIALS.md`, linked from `doc/connections/README.md` and `GENERIC-REMOTE-MCP.md`. ## Verification - `server/src/__tests__/tool-access-client-credentials.test.ts`: 16/16 pass. It uses a fake sign-in service that verifies real signatures against registered keys, and a fake MCP server. It covers RS256 and ES256 assertion claims and signatures, assertion-only client authentication, the secret flow (`client_secret_basic`), discovery and direct token URL, issuer pinning (refuses a different sign-in service and sends it nothing), the pinned endpoint surviving a hostile challenge, the token cache (0 token requests while live), concurrent dedupe (5 callers on 2 service instances produce 1 token request), 401 re-request, key and secret rotation, a refused personal connection, secret redaction in responses, rows and activity, and vault cleanup on removal. - Red/green: with the grant-refresh machine sign-in branch and the discovery pin guard disabled, 5 runtime tests fail (pin, cache/dedupe, 401 re-request, both rotations). Restored: 16/16 pass. The other tests cover new routes and services that do not exist on `master`. - `ui/src/features/connections/MachineSignInSetup.test.tsx`: 3/3. `ConnectionSetupFlow.architecture` and `.oauth` tests: 19/19. - `openapi-routes.test.ts`: 51/51. `runner-api.test.ts`: 2153/2153. `mcp-http.test.ts`: 9/9. Existing `tool-access-service.test.ts` refresh and legacy client-credentials tests: 54/54 (filtered). - Server package `tsc --noEmit`: clean. UI `tsc`: clean except the pre-existing generated Storybook catalog imports. - Not run locally (machine under heavy load): full suites, repo-wide typecheck, browser check of the live app. CI covers breadth. ## Risks - No migration. New config lives in `config.oauth` (`grantType: "client_credentials"` plus `clientAuth`). Older env-configured client-credentials connections have no `clientAuth` and keep their original path. - The capability declaration adds one extension key only for machine sign-in connections. Other requests are unchanged. - Rotation has no settings-page control yet; it is available through the API. This is a follow-up. - Low risk to agent boot and time to first token: a live token is read exactly like a browser sign-in token. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/connections/GENERIC-REMOTE-MCP.md | 6 +- doc/connections/MCP-CLIENT-CREDENTIALS.md | 129 +++ doc/connections/README.md | 9 +- packages/shared/src/index.ts | 12 + packages/shared/src/types/index.ts | 3 + packages/shared/src/types/tool-access.ts | 33 + packages/shared/src/validators/index.ts | 9 + packages/shared/src/validators/tool-access.ts | 39 + .../tool-access-client-credentials.test.ts | 682 ++++++++++++++++ server/src/routes/openapi.ts | 65 ++ server/src/routes/tool-access.ts | 106 ++- server/src/services/mcp-http.ts | 10 +- .../runner-api-sensitive-response.ts | 2 +- server/src/services/tool-access.ts | 24 +- .../services/tool-access/catalog/discovery.ts | 19 +- .../oauth/client-credentials-service.ts | 753 ++++++++++++++++++ .../tool-access/oauth/client-credentials.ts | 188 +++++ .../src/services/tool-access/oauth/client.ts | 19 +- .../tool-access/oauth/connection-refresh.ts | 17 +- .../tool-access/oauth/grant-refresh.ts | 42 +- server/src/services/tool-gateway.ts | 2 + .../runner-e2e/source-sdk-public-inputs.json | 10 +- ui/src/api/tools.ts | 10 + .../connections/ConnectionSetupFlow.tsx | 24 +- .../GenericMcpConnectStep.machine.test.tsx | 69 ++ .../connections/GenericMcpConnectStep.tsx | 83 +- .../connections/MachineSignInSetup.test.tsx | 137 ++++ .../connections/MachineSignInSetup.tsx | 324 ++++++++ .../machine-sign-in.stories.tsx | 102 +++ 29 files changed, 2861 insertions(+), 67 deletions(-) create mode 100644 doc/connections/MCP-CLIENT-CREDENTIALS.md create mode 100644 server/src/__tests__/tool-access-client-credentials.test.ts create mode 100644 server/src/services/tool-access/oauth/client-credentials-service.ts create mode 100644 server/src/services/tool-access/oauth/client-credentials.ts create mode 100644 ui/src/features/connections/GenericMcpConnectStep.machine.test.tsx create mode 100644 ui/src/features/connections/MachineSignInSetup.test.tsx create mode 100644 ui/src/features/connections/MachineSignInSetup.tsx create mode 100644 ui/storybook/stories/connection-setup/machine-sign-in.stories.tsx diff --git a/doc/connections/GENERIC-REMOTE-MCP.md b/doc/connections/GENERIC-REMOTE-MCP.md index 026eab63090..91a4ee3fd90 100644 --- a/doc/connections/GENERIC-REMOTE-MCP.md +++ b/doc/connections/GENERIC-REMOTE-MCP.md @@ -91,7 +91,7 @@ this as a setup error and keeps the connection available for retry. See The connect screen states the sign-in choice in one line, **Paperclip asks the server how to sign in.**, with a **Change** link, the same shape as the access -line below it. **Change** lists five options, each with its fields directly +line below it. **Change** lists five options (six for organization connections), each with its fields directly beneath it when selected: - **Let Paperclip check** — the default; Paperclip probes and walks you through it. @@ -101,6 +101,10 @@ beneath it when selected: Paperclip first. - **Custom headers** — for servers that name their own headers. - **No sign-in needed** — the server is open to anyone with the address. +- **Machine sign-in** — organization connections only. Paperclip signs in as a + client you registered with the provider, using a Paperclip key or a client + secret, with no person involved. It runs its own setup and test in place of + **Check link**. See [Machine sign-in](./MCP-CLIENT-CREDENTIALS.md). **Done** collapses the list back to a one-line summary of the choice. It stays open while the chosen option still needs a value. diff --git a/doc/connections/MCP-CLIENT-CREDENTIALS.md b/doc/connections/MCP-CLIENT-CREDENTIALS.md new file mode 100644 index 00000000000..8fcfada8748 --- /dev/null +++ b/doc/connections/MCP-CLIENT-CREDENTIALS.md @@ -0,0 +1,129 @@ +# Machine sign-in (MCP OAuth client credentials) + +Machine sign-in connects an **organization-owned** remote MCP server where no +person signs in: service accounts, CI-style integrations, internal automation. +Paperclip signs in as a client the provider registered for it, using the +[MCP OAuth Client Credentials extension](https://modelcontextprotocol.io/extensions/auth/oauth-client-credentials) +(`io.modelcontextprotocol/oauth-client-credentials`, revision 2026-07-28), +which builds on [RFC 6749 §4.4](https://datatracker.ietf.org/doc/html/rfc6749#section-4.4) +and [RFC 7523](https://datatracker.ietf.org/doc/html/rfc7523). + +It works the same on Paperclip Cloud and self-hosted instances. + +## Setting it up + +**Apps → Connect an app → Connect your own MCP server**, paste the address, +press **Change** beside the sign-in line and choose **Machine sign-in**. The option +appears only when the connection is shared with the organization; personal +connections sign in as the person. + +1. **Sign-in service.** Leave it blank and Paperclip asks the server where it + signs in (its protected-resource metadata, then that service's metadata). + Otherwise enter the sign-in service's address, or its token URL when the + provider publishes no metadata. +2. **How Paperclip proves who it is.** + - **Paperclip key (recommended).** Press **Create key**. Paperclip creates a + signing key, keeps the private half in the company vault, and shows the + public key as PEM and as JSON. Register it with the provider for the + Paperclip client. The public key isn't secret. + - **Client secret.** For providers that only issue secrets. The secret goes + straight into the vault and is never shown again. +3. **Client ID and scopes.** Enter the client ID the provider shows for the + Paperclip client, and any scopes its docs list (space separated). +4. **Test connection.** Paperclip asks the sign-in service for a token, then + lists the server's tools with it. Success turns the connection on and goes to + the normal review step. A failed test keeps the draft and its key, so you + can fix the provider side and test again. + +Errors say what to change: an unaccepted key or secret, refused scopes, a +provider that doesn't accept keys (choose Client secret) or secrets (choose +Paperclip key), or a sign-in service Paperclip couldn't find. + +## How tokens are requested + +- `grant_type=client_credentials`, with `scope` and the RFC 8707 `resource` + (the MCP server address) on every request. +- **Paperclip key:** `private_key_jwt`. Each request carries a fresh assertion + signed with RS256 (or ES256 when the service accepts only that): + `iss` = `sub` = client ID, `aud` = the token endpoint, `exp` 60 seconds after + `iat`, a unique `jti`, and `kid` = the key's RFC 7638 thumbprint. Per the + extension, no `client_id` or secret is sent alongside it. +- **Client secret:** `client_secret_basic`, or `client_secret_post` when that + is all the service advertises. +- Requests that carry a 2026-07-28 client capability object declare + `io.modelcontextprotocol/oauth-client-credentials` for these connections only. + +Tokens live in the vault on the connection's organization grant, like any +other OAuth token, and are reused until a minute before they expire. There is +no refresh token: Paperclip asks for a new access token. The existing grant +refresh lease and in-process single flight mean concurrent agents, across +server processes, share one token request. A `401` from the server requests a +new token once; a second `401` reports the problem without touching the +pinned settings. Agent boot and time to first token do not change: a live +token is read the same way as for browser sign-in connections. + +## The sign-in service is pinned + +The connection records the sign-in service's issuer and token endpoint at +setup. Credentials and assertions go only to that token endpoint, never to a +deployment-wide OAuth client and never to a different service the server later +advertises: + +- Test and rotation check the server's protected-resource metadata; if it now + lists other sign-in services and not the pinned one, Paperclip refuses and + asks you to set up machine sign-in again. +- A rejected token at runtime never re-runs discovery or rewrites the + connection's endpoints. + +When you enter a token URL directly there is no issuer to compare; the token +URL itself is the pin. + +## Rotation without downtime + +- **New client secret:** enter it; Paperclip tests it against the sign-in + service first and only then replaces the stored secret (a new version of the + same vault entry). A wrong secret changes nothing. +- **New key:** start a key rotation; Paperclip creates a replacement key and + shows its public key while the current key keeps working. Register the new + public key with the provider, then complete the rotation: Paperclip tests the + new key and only then switches to it and deletes the old one. + +API: `POST /api/tool-connections/{id}/machine-sign-in/rotate` with +`{ "method": "client_secret", "clientSecret": "…" }` or +`{ "method": "paperclip_key" }`, then +`POST /api/tool-connections/{id}/machine-sign-in/rotate/complete` for a key. + +## Security + +- Private keys and client secrets exist only in the company vault + (`oauth.client_private_key`, `oauth.client_private_key_next`, + `oauth.client_secret`). Config, API responses, activity and logs carry only + the public key, key ID, issuer, method and scope count. +- Company scoped. Only connection managers can set it up, and only for + organization connections; the service refuses personal connections. +- Activity: `tool_connection.machine_sign_in_created`, `…_verified`, + `…_rotation_started`, `…_rotated`, and removal as `tool_connection.archived` + with `machineSignIn: true`. Removal destroys the vault entries the connection + owned. + +## API + +| Route | Use | +| --- | --- | +| `POST /api/companies/{companyId}/tools/machine-sign-in` | Create the draft (finds and pins the sign-in service, creates the key or stores the secret). | +| `GET /api/tool-connections/{id}/machine-sign-in` | Settings and public keys. | +| `POST /api/tool-connections/{id}/machine-sign-in/test` | Test with an optional client ID, scopes or secret; turns the connection on. | +| `POST /api/tool-connections/{id}/machine-sign-in/rotate` | Rotate the secret, or stage a new key. | +| `POST /api/tool-connections/{id}/machine-sign-in/rotate/complete` | Switch to the staged key. | + +Code: `server/src/services/tool-access/oauth/client-credentials.ts` (keys and +assertions), `client-credentials-service.ts` (setup, pinning, token requests), +the machine sign-in branch in `grant-refresh.ts`, and +`ui/src/features/connections/MachineSignInSetup.tsx`. Tests: +`server/src/__tests__/tool-access-client-credentials.test.ts`. + +## Not yet covered + +- A settings-page control for rotation (rotation is API-only today). +- Enterprise single sign-on and enterprise-managed authorization (ID-JAG) are + later phases. diff --git a/doc/connections/README.md b/doc/connections/README.md index 31fb2959d61..d66cb81371d 100644 --- a/doc/connections/README.md +++ b/doc/connections/README.md @@ -20,9 +20,14 @@ MCP protocol versions (2026-07-28 stateless MCP and older handshake servers): MCP extensions: [MCP Apps](./MCP-APPS.md) (interactive views in the task chat), [MCP Skills](./MCP-SKILLS.md) (skills that connected apps publish, reviewed by the board and delivered to agents), [MCP Tasks](./MCP-TASKS.md) (long -provider calls that survive timeouts, restarts, and deploys), and +provider calls that survive timeouts, restarts, and deploys), [MCP elicitation](./MCP-ELICITATION.md) (an app's question to the person, -answered in the task chat). +answered in the task chat), and [Machine sign-in](./MCP-CLIENT-CREDENTIALS.md) +(OAuth client credentials for organization connections where no person signs +in). + +[Machine sign-in](./MCP-CLIENT-CREDENTIALS.md) (OAuth client credentials for +organization connections where no person signs in). Long-term memory: [Experimental memory connectors](./MEMORY.md). diff --git a/packages/shared/src/index.ts b/packages/shared/src/index.ts index 7128882e69b..efab7ad18d8 100644 --- a/packages/shared/src/index.ts +++ b/packages/shared/src/index.ts @@ -1458,6 +1458,9 @@ export type { RejectSecretProposalInput, ConnectToolAppAuthChallenge, ConnectToolAppResult, + MachineSignInPublicKey, + MachineSignInStatus, + MachineSignInSetupResult, ToolAppMetadataPreflightAttempt, ToolAppMetadataPreflightResult, ToolOAuthClientRegistrationSource, @@ -2302,6 +2305,15 @@ export { genericMcpOAuthClientSchema, type GenericMcpAuthMode, type GenericMcpOAuthClient, + MACHINE_SIGN_IN_METHODS, + machineSignInMethodSchema, + createMachineSignInSchema, + testMachineSignInSchema, + rotateMachineSignInSchema, + type MachineSignInMethod, + type CreateMachineSignIn, + type TestMachineSignIn, + type RotateMachineSignIn, reconnectToolAppSchema, createToolApplicationSchema, finishToolAppSchema, diff --git a/packages/shared/src/types/index.ts b/packages/shared/src/types/index.ts index f63f67250d6..8abf1f4ed91 100644 --- a/packages/shared/src/types/index.ts +++ b/packages/shared/src/types/index.ts @@ -525,6 +525,9 @@ export type { ToolAccessSelector, ConnectToolAppAuthChallenge, ConnectToolAppResult, + MachineSignInPublicKey, + MachineSignInStatus, + MachineSignInSetupResult, ToolAppMetadataPreflightAttempt, ToolAppMetadataPreflightResult, FinishToolAppResult, diff --git a/packages/shared/src/types/tool-access.ts b/packages/shared/src/types/tool-access.ts index 4c05d176f9b..1d2c29fb7c0 100644 --- a/packages/shared/src/types/tool-access.ts +++ b/packages/shared/src/types/tool-access.ts @@ -1221,6 +1221,39 @@ export interface ConnectToolAppAuthChallenge { manualClientRequired?: boolean; } +/** Public half of a key Paperclip holds for machine sign-in. Safe to show and share. */ +export interface MachineSignInPublicKey { + keyId: string; + algorithm: "RS256" | "ES256"; + /** PEM (SPKI) for providers that take a certificate-style key. */ + pem: string; + /** JWK for providers that take a JSON key. */ + jwk: Record; + createdAt: string; +} + +/** Machine sign-in settings for one organization connection. Never includes secret or private key values. */ +export interface MachineSignInStatus { + connectionId: string; + method: "paperclip_key" | "client_secret"; + clientId: string | null; + /** The sign-in service this connection's credentials are bound to. */ + issuer: string | null; + tokenUrl: string; + scopes: string[]; + publicKey: MachineSignInPublicKey | null; + /** A replacement key waiting to be registered with the provider. */ + pendingPublicKey: MachineSignInPublicKey | null; + hasClientSecret: boolean; + lastTokenAt: string | null; +} + +export interface MachineSignInSetupResult { + connectionId: string; + connection: ToolConnection; + machineSignIn: MachineSignInStatus; +} + export interface ConnectToolAppResult { connectionId: string; application: ToolApplication; diff --git a/packages/shared/src/validators/index.ts b/packages/shared/src/validators/index.ts index a62024f564b..b0bc5449eb5 100644 --- a/packages/shared/src/validators/index.ts +++ b/packages/shared/src/validators/index.ts @@ -919,6 +919,15 @@ export { genericMcpOAuthClientSchema, type GenericMcpAuthMode, type GenericMcpOAuthClient, + MACHINE_SIGN_IN_METHODS, + machineSignInMethodSchema, + createMachineSignInSchema, + testMachineSignInSchema, + rotateMachineSignInSchema, + type MachineSignInMethod, + type CreateMachineSignIn, + type TestMachineSignIn, + type RotateMachineSignIn, reconnectToolAppSchema, finishToolAppSchema, finalizeOAuthAccessSchema, diff --git a/packages/shared/src/validators/tool-access.ts b/packages/shared/src/validators/tool-access.ts index cd28549a930..f6ada4ebdc2 100644 --- a/packages/shared/src/validators/tool-access.ts +++ b/packages/shared/src/validators/tool-access.ts @@ -372,6 +372,45 @@ export const genericMcpOAuthClientSchema = z.object({ export type GenericMcpOAuthClient = z.infer; +/** + * Machine sign-in (MCP OAuth client credentials) for organization-owned + * connections. "paperclip_key" means Paperclip holds a signing key and the admin + * registers its public half with the provider; "client_secret" stores a secret + * the provider issued. Secret values are write-only. + */ +export const MACHINE_SIGN_IN_METHODS = ["paperclip_key", "client_secret"] as const; +export const machineSignInMethodSchema = z.enum(MACHINE_SIGN_IN_METHODS); +export type MachineSignInMethod = z.infer; + +const machineSignInScopesSchema = z.string().trim().max(2048) + .refine((value) => !/[\u0000-\u001f"\\]/.test(value), "Scopes can only contain letters, numbers and punctuation, separated by spaces."); + +export const createMachineSignInSchema = z.object({ + link: z.string().trim().url("Enter the server's full address, starting with https://").max(4096), + name: z.string().trim().min(1).max(160).optional(), + /** The sign-in service's address, or its token URL. Omitted means "find it from the server". */ + authorizationServer: z.string().trim().url("Enter the sign-in service's full address, starting with https://").max(4096).optional(), + method: machineSignInMethodSchema, + clientId: z.string().trim().min(1).max(4096).optional(), + clientSecret: z.string().min(1).max(16384).optional(), + scopes: machineSignInScopesSchema.optional(), + applicationId: z.string().uuid().optional(), +}).strict(); +export type CreateMachineSignIn = z.infer; + +export const testMachineSignInSchema = z.object({ + clientId: z.string().trim().min(1).max(4096).optional(), + clientSecret: z.string().min(1).max(16384).optional(), + scopes: machineSignInScopesSchema.optional(), +}).strict(); +export type TestMachineSignIn = z.infer; + +export const rotateMachineSignInSchema = z.discriminatedUnion("method", [ + z.object({ method: z.literal("paperclip_key") }).strict(), + z.object({ method: z.literal("client_secret"), clientSecret: z.string().min(1).max(16384) }).strict(), +]); +export type RotateMachineSignIn = z.infer; + /** * Reject `headers.*` credential paths whose header name Paperclip refuses to * send, and any value that could split the outbound request. This runs at the diff --git a/server/src/__tests__/tool-access-client-credentials.test.ts b/server/src/__tests__/tool-access-client-credentials.test.ts new file mode 100644 index 00000000000..c39fb3d0b42 --- /dev/null +++ b/server/src/__tests__/tool-access-client-credentials.test.ts @@ -0,0 +1,682 @@ +import { createPublicKey, randomUUID, verify, type KeyObject } from "node:crypto"; +import express from "express"; +import request from "supertest"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { + activityLog, + applyPendingMigrations, + authUsers, + companies, + companySecretBindings, + companySecrets, + connectionGrants, + createDb, + secretAccessEvents, + toolAccessAuditEvents, + toolApplications, + toolCatalogEntries, + toolConnections, +} from "@paperclipai/db"; +import { and, eq } from "drizzle-orm"; +import { toolAccessService } from "../services/tool-access.js"; +import { secretService } from "../services/secrets.js"; +import { toolAccessRoutes } from "../routes/tool-access.js"; +import { errorHandler } from "../middleware/index.js"; +import { createOAuthClientProtocol } from "../services/tool-access/oauth/client.js"; +import { + CLIENT_ASSERTION_LIFETIME_SECONDS, + generateClientKeyPair, + signClientAssertion, +} from "../services/tool-access/oauth/client-credentials.js"; +import { MCP_OAUTH_CLIENT_CREDENTIALS_EXTENSION_ID, mcpClientCapabilities } from "../services/mcp-http.js"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; + +const ISSUER = "https://auth.example.test"; +const TOKEN_URL = `${ISSUER}/oauth/token`; +const OTHER_ISSUER = "https://other-auth.example.test"; +const MCP_URL = "https://mcp.example.test/mcp"; +const CLIENT_ID = "paperclip-automation"; +const CLIENT_SECRET = "s3cret-value-never-shown"; + +function decodeJwt(token: string) { + const [header, claims, signature] = token.split("."); + return { + header: JSON.parse(Buffer.from(header!, "base64url").toString()) as Record, + claims: JSON.parse(Buffer.from(claims!, "base64url").toString()) as Record, + signingInput: `${header}.${claims}`, + signature: Buffer.from(signature!, "base64url"), + }; +} + +function verifyJwt(token: string, key: KeyObject): boolean { + const { header, signingInput, signature } = decodeJwt(token); + return verify( + "sha256", + Buffer.from(signingInput), + header.alg === "ES256" ? { key, dsaEncoding: "ieee-p1363" } : key, + signature, + ); +} + +function json(payload: unknown, status = 200, headers: Record = {}) { + return new Response(JSON.stringify(payload), { status, headers: { "content-type": "application/json", ...headers } }); +} + +/** + * A fake sign-in service and MCP server behind the outbound HTTP transport. + * The sign-in service verifies real signatures against keys an "admin" + * registered, exactly as a provider would. + */ +function fakeProvider() { + const state = { + advertisedIssuers: [ISSUER], + registeredKeys: new Map(), + clientSecret: CLIENT_SECRET, + tokenRequests: [] as Array<{ url: string; body: URLSearchParams; authorization: string | null }>, + otherIssuerRequests: 0, + mcpRequests: [] as Array<{ authorization: string | null; body: Record }>, + acceptedTokens: new Set(), + issued: 0, + tokenDelayMs: 0, + rejectAllTokens: false, + /** Other clients the sign-in service knows (id → secret). */ + extraClients: new Map(), + /** Clients whose tokens the MCP server refuses. */ + mcpDeniedClients: new Set(), + tokenClients: new Map(), + /** Serve protected-resource metadata only at the address the challenge names. */ + metadataOnlyAtHint: false, + /** Runs once when the server first receives a signed-in request (to stage races). */ + onAuthorizedMcpRequest: null as null | (() => Promise), + }; + const metadata = (issuer: string) => ({ + issuer, + token_endpoint: `${issuer}/oauth/token`, + grant_types_supported: ["client_credentials"], + token_endpoint_auth_methods_supported: ["private_key_jwt", "client_secret_basic", "client_secret_post"], + token_endpoint_auth_signing_alg_values_supported: ["RS256", "ES256"], + }); + const handle = async (input: string | URL | Request, init?: RequestInit): Promise => { + const url = String(input instanceof Request ? input.url : input); + const headers = new Headers(init?.headers); + if (url.startsWith(OTHER_ISSUER)) { + state.otherIssuerRequests += 1; + if (url.includes("/.well-known/")) return json(metadata(OTHER_ISSUER)); + return json({ access_token: "stolen", expires_in: 3600 }); + } + if (url === `${new URL(MCP_URL).origin}/meta/custom-resource` && state.metadataOnlyAtHint) { + return json({ resource: MCP_URL, authorization_servers: state.advertisedIssuers }); + } + if (url === `${new URL(MCP_URL).origin}/.well-known/oauth-protected-resource/mcp` + || url === `${new URL(MCP_URL).origin}/.well-known/oauth-protected-resource`) { + if (state.metadataOnlyAtHint) return new Response("not found", { status: 404 }); + return json({ resource: MCP_URL, authorization_servers: state.advertisedIssuers }); + } + if (url === `${ISSUER}/.well-known/oauth-authorization-server`) return json(metadata(ISSUER)); + if (url.startsWith(ISSUER) && url.includes("/.well-known/")) return new Response("not found", { status: 404 }); + if (url === TOKEN_URL) { + const body = new URLSearchParams(String(init?.body)); + state.tokenRequests.push({ url, body, authorization: headers.get("authorization") }); + if (state.tokenDelayMs) await new Promise((resolve) => setTimeout(resolve, state.tokenDelayMs)); + if (body.get("grant_type") !== "client_credentials") return json({ error: "unsupported_grant_type" }, 400); + const assertion = body.get("client_assertion"); + let authenticated = false; + let clientId = CLIENT_ID; + if (assertion) { + const { header, claims } = decodeJwt(assertion); + const key = state.registeredKeys.get(String(header.kid)); + authenticated = Boolean(key && verifyJwt(assertion, key) + && claims.iss === CLIENT_ID && claims.sub === CLIENT_ID && claims.aud === TOKEN_URL + && typeof claims.exp === "number" && claims.exp * 1000 > Date.now()); + } else { + const basic = headers.get("authorization"); + const expected = `Basic ${Buffer.from(`${CLIENT_ID}:${state.clientSecret}`).toString("base64")}`; + authenticated = basic === expected; + for (const [id, secret] of state.extraClients) { + if (basic === `Basic ${Buffer.from(`${id}:${secret}`).toString("base64")}`) { + authenticated = true; + clientId = id; + } + } + } + if (!authenticated) return json({ error: "invalid_client" }, 401); + state.issued += 1; + const token = `machine-token-${state.issued}`; + state.acceptedTokens.add(token); + state.tokenClients.set(token, clientId); + return json({ access_token: token, token_type: "Bearer", expires_in: 3600, scope: body.get("scope") ?? undefined }); + } + if (url === MCP_URL) { + const body = JSON.parse(String(init?.body ?? "{}")) as Record; + const authorization = headers.get("authorization"); + state.mcpRequests.push({ authorization, body }); + const bearer = authorization?.replace(/^Bearer /, "") ?? ""; + if (state.rejectAllTokens || !state.acceptedTokens.has(bearer) || state.mcpDeniedClients.has(state.tokenClients.get(bearer) ?? "")) { + return new Response("", { + status: 401, + headers: { "www-authenticate": `Bearer resource_metadata="${new URL(MCP_URL).origin}${state.metadataOnlyAtHint ? "/meta/custom-resource" : "/.well-known/oauth-protected-resource/mcp"}"` }, + }); + } + if (state.onAuthorizedMcpRequest) { + const hook = state.onAuthorizedMcpRequest; + state.onAuthorizedMcpRequest = null; + await hook(); + } + if (body.method === "server/discover") { + return json({ jsonrpc: "2.0", id: body.id, result: { supportedVersions: ["2026-07-28"], capabilities: {} } }); + } + if (body.method === "tools/list") { + return json({ jsonrpc: "2.0", id: body.id, result: { tools: [ + { name: "list_reports", description: "List reports", inputSchema: { type: "object", properties: {} }, annotations: { readOnlyHint: true } }, + ] } }); + } + return json({ jsonrpc: "2.0", id: body.id, result: {} }); + } + throw new Error(`unexpected request ${url}`); + }; + return { state, handle }; +} + +describe("client credentials protocol", () => { + it("creates keys without blocking other work on the server", async () => { + let otherWorkRan = false; + const generating = generateClientKeyPair("RS256"); + setImmediate(() => { otherWorkRan = true; }); + await generating; + expect(otherWorkRan).toBe(true); + }); + + it("signs an RS256 assertion with iss = sub = client ID, the token endpoint as audience, a short expiry and a unique jti", async () => { + const { privateKeyPem, publicKey } = await generateClientKeyPair("RS256"); + const now = new Date("2026-11-13T12:00:00Z"); + const first = signClientAssertion({ privateKeyPem, algorithm: "RS256", keyId: publicKey.keyId, clientId: CLIENT_ID, audience: TOKEN_URL, now }); + const second = signClientAssertion({ privateKeyPem, algorithm: "RS256", keyId: publicKey.keyId, clientId: CLIENT_ID, audience: TOKEN_URL, now }); + const decoded = decodeJwt(first); + expect(decoded.header).toEqual({ alg: "RS256", typ: "JWT", kid: publicKey.keyId }); + expect(decoded.claims).toMatchObject({ iss: CLIENT_ID, sub: CLIENT_ID, aud: TOKEN_URL, iat: now.getTime() / 1000 }); + expect(Number(decoded.claims.exp) - Number(decoded.claims.iat)).toBe(CLIENT_ASSERTION_LIFETIME_SECONDS); + expect(CLIENT_ASSERTION_LIFETIME_SECONDS).toBeLessThanOrEqual(60); + expect(decoded.claims.jti).not.toEqual(decodeJwt(second).claims.jti); + expect(verifyJwt(first, createPublicKey(publicKey.publicKeyPem))).toBe(true); + // The JWK the admin registers verifies the same signature. + expect(verifyJwt(first, createPublicKey({ key: publicKey.publicJwk, format: "jwk" }))).toBe(true); + expect(publicKey.publicJwk).toMatchObject({ kid: publicKey.keyId, alg: "RS256", use: "sig", kty: "RSA" }); + expect(JSON.stringify(publicKey)).not.toContain("PRIVATE KEY"); + expect(publicKey.publicJwk).not.toHaveProperty("d"); + }); + + it("signs ES256 assertions in JOSE (raw r||s) form", async () => { + const { privateKeyPem, publicKey } = await generateClientKeyPair("ES256"); + const token = signClientAssertion({ privateKeyPem, algorithm: "ES256", keyId: publicKey.keyId, clientId: CLIENT_ID, audience: TOKEN_URL }); + expect(decodeJwt(token).signature).toHaveLength(64); + expect(verifyJwt(token, createPublicKey(publicKey.publicKeyPem))).toBe(true); + }); + + it("sends only the signed assertion as client authentication", async () => { + const sent: Array<{ body: URLSearchParams; headers: Headers }> = []; + const { exchangeOAuthToken } = createOAuthClientProtocol({ + assertOAuthEndpointUrl: (_kind, value) => String(value), + fetchRemoteHttpUrl: async (_url, init) => { + sent.push({ body: new URLSearchParams(String(init?.body)), headers: new Headers(init?.headers) }); + return json({ access_token: "t", expires_in: 60 }); + }, + }); + await exchangeOAuthToken({ + tokenUrl: TOKEN_URL, clientId: CLIENT_ID, grantType: "client_credentials", scopes: ["reports.read"], + clientAssertion: "a.b.c", resource: MCP_URL, + }); + const { body, headers } = sent[0]!; + expect(body.get("grant_type")).toBe("client_credentials"); + expect(body.get("client_assertion_type")).toBe("urn:ietf:params:oauth:client-assertion-type:jwt-bearer"); + expect(body.get("client_assertion")).toBe("a.b.c"); + expect(body.get("scope")).toBe("reports.read"); + expect(body.get("resource")).toBe(MCP_URL); + expect(body.has("client_id")).toBe(false); + expect(body.has("client_secret")).toBe(false); + expect(headers.get("authorization")).toBeNull(); + }); + + it("declares the extension only when a connection uses machine sign-in", () => { + expect(JSON.stringify(mcpClientCapabilities("2026-07-28"))).not.toContain(MCP_OAUTH_CLIENT_CREDENTIALS_EXTENSION_ID); + expect(mcpClientCapabilities("2026-07-28", { declareClientCredentials: true })).toMatchObject({ + extensions: { [MCP_OAUTH_CLIENT_CREDENTIALS_EXTENSION_ID]: {} }, + }); + }); +}); + +const externalDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_TOOL_ACCESS_TEST_DATABASE_URL"); +const embeddedPostgresSupport = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); +const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; + +describeEmbeddedPostgres("machine sign-in connections", () => { + let db!: ReturnType; + let tempDb: Awaited> | null = null; + const transport = { + remoteHttpEndpointLookup: async () => [{ address: "8.8.8.8", family: 4 as const }], + remoteHttpRequest: async (url: string, init: RequestInit) => fetch(url, init), + }; + const service = () => toolAccessService(db, transport); + const actor = { actorType: "user" as const, actorId: "board-user" }; + + beforeAll(async () => { + if (externalDatabaseUrl) { + await applyPendingMigrations(externalDatabaseUrl); + db = createDb(externalDatabaseUrl, { maxConnections: 6 }); + return; + } + tempDb = await startEmbeddedPostgresTestDatabase("paperclip-client-credentials-"); + db = createDb(tempDb.connectionString, { maxConnections: 6 }); + }, 120_000); + + afterEach(async () => { + vi.restoreAllMocks(); + await db.delete(secretAccessEvents); + await db.delete(companySecretBindings); + await db.delete(activityLog); + await db.delete(toolAccessAuditEvents); + await db.delete(toolCatalogEntries); + await db.delete(connectionGrants); + await db.delete(toolConnections); + await db.delete(companySecrets); + await db.delete(toolApplications); + await db.delete(companies); + await db.delete(authUsers); + }); + + afterAll(async () => { + await db?.$client.end({ timeout: 1 }); + await tempDb?.cleanup(); + }); + + async function createCompany() { + const [company] = await db.insert(companies).values({ + name: `Machine sign-in ${randomUUID()}`, + issuePrefix: `MS${randomUUID().slice(0, 6).toUpperCase()}`, + }).returning(); + return company!; + } + + function useProvider() { + const provider = fakeProvider(); + vi.spyOn(globalThis, "fetch").mockImplementation(provider.handle); + return provider; + } + + function registerKey(provider: ReturnType, key: { keyId: string; pem: string }) { + provider.state.registeredKeys.set(key.keyId, createPublicKey(key.pem)); + } + + async function organizationGrant(connectionId: string) { + const [grant] = await db.select().from(connectionGrants) + .where(and(eq(connectionGrants.connectionId, connectionId), eq(connectionGrants.kind, "organization"))); + return grant!; + } + + async function setupKeyConnection(companyId: string, provider: ReturnType) { + const svc = service(); + const created = await svc.createMachineSignIn(companyId, { link: MCP_URL, method: "paperclip_key", scopes: "reports.read" }, actor); + registerKey(provider, created.machineSignIn.publicKey!); + const result = await svc.testMachineSignIn(companyId, created.connectionId, { clientId: CLIENT_ID }, actor); + return { svc, created, result }; + } + + it("sets up a Paperclip key connection: finds the sign-in service, shows the public key, signs in and lists tools", async () => { + const company = await createCompany(); + const provider = useProvider(); + const svc = service(); + const created = await svc.createMachineSignIn(company.id, { link: MCP_URL, method: "paperclip_key", scopes: "reports.read" }, actor); + expect(created.machineSignIn).toMatchObject({ method: "paperclip_key", issuer: ISSUER, tokenUrl: TOKEN_URL, clientId: null, scopes: ["reports.read"] }); + expect(created.machineSignIn.publicKey?.pem).toContain("BEGIN PUBLIC KEY"); + expect(created.connection.status).toBe("draft"); + // The private key is in the vault, nowhere else. + const serialized = JSON.stringify(created); + expect(serialized).not.toContain("PRIVATE KEY"); + const [row] = await db.select().from(toolConnections).where(eq(toolConnections.id, created.connectionId)); + expect(JSON.stringify(row!.config)).not.toContain("PRIVATE KEY"); + expect(row!.credentialSecretRefs.map((ref) => ref.configPath)).toEqual(["oauth.client_private_key"]); + + // Testing before the key is registered fails clearly and keeps the draft and its key. + await expect(svc.testMachineSignIn(company.id, created.connectionId, { clientId: CLIENT_ID }, actor)) + .rejects.toMatchObject({ status: 422, details: expect.objectContaining({ code: "client_credentials_rejected" }) }); + expect((await svc.getMachineSignIn(company.id, created.connectionId)).publicKey?.keyId).toBe(created.machineSignIn.publicKey!.keyId); + + registerKey(provider, created.machineSignIn.publicKey!); + const result = await svc.testMachineSignIn(company.id, created.connectionId, { clientId: CLIENT_ID }, actor); + expect(result.connection.status).toBe("active"); + expect(result.catalog.map((entry) => entry.name)).toContain("list_reports"); + + const accepted = provider.state.tokenRequests.at(-1)!; + expect(accepted.body.get("scope")).toBe("reports.read"); + expect(accepted.body.get("resource")).toBe(MCP_URL); + const { header, claims } = decodeJwt(accepted.body.get("client_assertion")!); + expect(header.kid).toBe(created.machineSignIn.publicKey!.keyId); + expect(claims).toMatchObject({ iss: CLIENT_ID, sub: CLIENT_ID, aud: TOKEN_URL }); + const authorized = provider.state.mcpRequests.filter((call) => call.authorization === "Bearer machine-token-1"); + expect(authorized.length).toBeGreaterThan(0); + const listCall = provider.state.mcpRequests.find((call) => call.body.method === "tools/list" && call.authorization); + expect(JSON.stringify(listCall?.body)).toContain(MCP_OAUTH_CLIENT_CREDENTIALS_EXTENSION_ID); + const grant = await organizationGrant(created.connectionId); + expect(grant.status).toBe("active"); + expect(grant.credentialSecretRefs.map((ref) => ref.configPath)).toContain("oauth.access_token"); + }); + + it("uses a client secret with client_secret_basic and never returns the secret", async () => { + const company = await createCompany(); + const provider = useProvider(); + const svc = service(); + const created = await svc.createMachineSignIn(company.id, { + link: MCP_URL, method: "client_secret", authorizationServer: ISSUER, clientId: CLIENT_ID, clientSecret: CLIENT_SECRET, + }, actor); + expect(created.machineSignIn).toMatchObject({ method: "client_secret", hasClientSecret: true, publicKey: null }); + const result = await svc.testMachineSignIn(company.id, created.connectionId, {}, actor); + expect(result.connection.status).toBe("active"); + const tokenRequest = provider.state.tokenRequests.at(-1)!; + expect(tokenRequest.authorization).toBe(`Basic ${Buffer.from(`${CLIENT_ID}:${CLIENT_SECRET}`).toString("base64")}`); + expect(tokenRequest.body.has("client_secret")).toBe(false); + for (const value of [created, result, await svc.getMachineSignIn(company.id, created.connectionId)]) { + expect(JSON.stringify(value)).not.toContain(CLIENT_SECRET); + } + const [row] = await db.select().from(toolConnections).where(eq(toolConnections.id, created.connectionId)); + expect(JSON.stringify(row)).not.toContain(CLIENT_SECRET); + }); + + it("follows the server's challenge to metadata published outside the well-known addresses", async () => { + const company = await createCompany(); + const provider = useProvider(); + provider.state.metadataOnlyAtHint = true; + const created = await service().createMachineSignIn(company.id, { link: MCP_URL, method: "paperclip_key" }, actor); + expect(created.machineSignIn).toMatchObject({ issuer: ISSUER, tokenUrl: TOKEN_URL }); + }); + + it("accepts a token URL directly when the provider publishes no sign-in metadata", async () => { + const company = await createCompany(); + useProvider(); + const created = await service().createMachineSignIn(company.id, { + link: MCP_URL, method: "client_secret", authorizationServer: TOKEN_URL, clientId: CLIENT_ID, clientSecret: CLIENT_SECRET, + }, actor); + expect(created.machineSignIn).toMatchObject({ issuer: null, tokenUrl: TOKEN_URL }); + }); + + it("refuses to follow the server to a different sign-in service and never sends credentials there", async () => { + const company = await createCompany(); + const provider = useProvider(); + const svc = service(); + const created = await svc.createMachineSignIn(company.id, { link: MCP_URL, method: "paperclip_key" }, actor); + registerKey(provider, created.machineSignIn.publicKey!); + provider.state.advertisedIssuers = [OTHER_ISSUER]; + const tokenRequestsBefore = provider.state.tokenRequests.length; + await expect(svc.testMachineSignIn(company.id, created.connectionId, { clientId: CLIENT_ID }, actor)) + .rejects.toMatchObject({ status: 422, details: expect.objectContaining({ code: "client_credentials_issuer_changed" }) }); + expect(provider.state.tokenRequests.length).toBe(tokenRequestsBefore); + expect(provider.state.otherIssuerRequests).toBe(0); + expect((await svc.getMachineSignIn(company.id, created.connectionId)).tokenUrl).toBe(TOKEN_URL); + }); + + it("keeps the pinned token endpoint when an active server's challenge points elsewhere", async () => { + const company = await createCompany(); + const provider = useProvider(); + const { svc, created } = await setupKeyConnection(company.id, provider); + provider.state.advertisedIssuers = [OTHER_ISSUER]; + provider.state.rejectAllTokens = true; + await expect(svc.checkHealth(created.connectionId, actor)).rejects.toMatchObject({ + message: expect.stringContaining("didn't accept this connection's machine sign-in"), + }); + const [row] = await db.select().from(toolConnections).where(eq(toolConnections.id, created.connectionId)); + expect((row!.config.oauth as Record).tokenUrl).toBe(TOKEN_URL); + expect((row!.config.oauth as Record).issuer).toBe(ISSUER); + expect(provider.state.otherIssuerRequests).toBe(0); + }); + + it("reuses a cached token and shares one token request across concurrent callers", async () => { + const company = await createCompany(); + const provider = useProvider(); + const { created } = await setupKeyConnection(company.id, provider); + const grant = await organizationGrant(created.connectionId); + const issuedAfterSetup = provider.state.issued; + + // A live token is reused: no token request at all. + const first = service(); + await first.refreshOAuthGrantCredentials({ companyId: company.id, connectionId: created.connectionId, grantId: grant.id, actor }); + expect(provider.state.issued).toBe(issuedAfterSetup); + + // Shortly before expiry, many agents on two server processes ask at once. + await db.update(connectionGrants).set({ + providerTenant: { ...(grant.providerTenant ?? {}), oauth: { ...((grant.providerTenant ?? {}).oauth as object ?? {}), accessTokenExpiresAt: new Date(Date.now() + 10_000).toISOString() } }, + }).where(eq(connectionGrants.id, grant.id)); + provider.state.tokenDelayMs = 150; + const second = service(); + const input = { companyId: company.id, connectionId: created.connectionId, grantId: grant.id, actor }; + await Promise.all([ + first.refreshOAuthGrantCredentials(input), first.refreshOAuthGrantCredentials(input), first.refreshOAuthGrantCredentials(input), + second.refreshOAuthGrantCredentials(input), second.refreshOAuthGrantCredentials(input), + ]); + expect(provider.state.issued).toBe(issuedAfterSetup + 1); + const refreshed = await organizationGrant(created.connectionId); + const expiresAt = Date.parse(String((refreshed.providerTenant?.oauth as Record).accessTokenExpiresAt)); + expect(expiresAt).toBeGreaterThan(Date.now() + 3_000_000); + }); + + it("requests a new token when the server rejects the current one", async () => { + const company = await createCompany(); + const provider = useProvider(); + const { svc, created } = await setupKeyConnection(company.id, provider); + const issuedAfterSetup = provider.state.issued; + provider.state.acceptedTokens.clear(); + const health = await svc.checkHealth(created.connectionId, actor); + expect(health.connection.healthStatus).toBe("ok"); + expect(provider.state.issued).toBe(issuedAfterSetup + 1); + expect(provider.state.mcpRequests.at(-1)?.authorization).toBe(`Bearer machine-token-${issuedAfterSetup + 1}`); + }); + + it("rotates the key without downtime: the old key works until the new one is registered and confirmed", async () => { + const company = await createCompany(); + const provider = useProvider(); + const { svc, created } = await setupKeyConnection(company.id, provider); + const oldKeyId = created.machineSignIn.publicKey!.keyId; + const started = await svc.rotateMachineSignIn(company.id, created.connectionId, { method: "paperclip_key" }, actor); + expect(started.completed).toBe(false); + const pending = started.machineSignIn.pendingPublicKey!; + expect(pending.keyId).not.toBe(oldKeyId); + expect(started.machineSignIn.publicKey?.keyId).toBe(oldKeyId); + + // Tokens keep coming from the current key while the new one is pending. + const grant = await organizationGrant(created.connectionId); + await svc.refreshOAuthGrantCredentials({ companyId: company.id, connectionId: created.connectionId, grantId: grant.id, forceRefresh: true, actor }); + expect(decodeJwt(provider.state.tokenRequests.at(-1)!.body.get("client_assertion")!).header.kid).toBe(oldKeyId); + + // Completing before the provider knows the new key fails and changes nothing. + await expect(svc.completeMachineSignInRotation(company.id, created.connectionId, actor)) + .rejects.toMatchObject({ details: expect.objectContaining({ code: "client_credentials_rejected" }) }); + expect((await svc.getMachineSignIn(company.id, created.connectionId)).publicKey?.keyId).toBe(oldKeyId); + + registerKey(provider, pending); + const completed = await svc.completeMachineSignInRotation(company.id, created.connectionId, actor); + expect(completed.machineSignIn.publicKey?.keyId).toBe(pending.keyId); + expect(completed.machineSignIn.pendingPublicKey).toBeNull(); + provider.state.registeredKeys.delete(oldKeyId); + await svc.refreshOAuthGrantCredentials({ companyId: company.id, connectionId: created.connectionId, grantId: grant.id, forceRefresh: true, actor }); + expect(decodeJwt(provider.state.tokenRequests.at(-1)!.body.get("client_assertion")!).header.kid).toBe(pending.keyId); + const [row] = await db.select().from(toolConnections).where(eq(toolConnections.id, created.connectionId)); + expect(row!.credentialSecretRefs.map((ref) => ref.configPath)).not.toContain("oauth.client_private_key_next"); + }); + + it("signs with a key ID that matches the key a refresh actually reads", async () => { + const company = await createCompany(); + const provider = useProvider(); + const { svc, created } = await setupKeyConnection(company.id, provider); + // A rotation lands between a refresh loading the settings and reading the + // vault: the vault holds the new key while config still names the old one. + const [row] = await db.select().from(toolConnections).where(eq(toolConnections.id, created.connectionId)); + const keyRef = row!.credentialSecretRefs.find((ref) => ref.configPath === "oauth.client_private_key")!; + const next = await generateClientKeyPair("RS256"); + registerKey(provider, { keyId: next.publicKey.keyId, pem: next.publicKey.publicKeyPem }); + await secretService(db).rotate(keyRef.secretId, { value: next.privateKeyPem }); + const grant = await organizationGrant(created.connectionId); + const before = provider.state.issued; + await svc.refreshOAuthGrantCredentials({ companyId: company.id, connectionId: created.connectionId, grantId: grant.id, forceRefresh: true, actor }); + expect(provider.state.issued).toBe(before + 1); + expect(decodeJwt(provider.state.tokenRequests.at(-1)!.body.get("client_assertion")!).header.kid).toBe(next.publicKey.keyId); + }); + + it("never promotes a staged key that changed after it was registered", async () => { + const company = await createCompany(); + const provider = useProvider(); + const { svc, created } = await setupKeyConnection(company.id, provider); + const started = await svc.rotateMachineSignIn(company.id, created.connectionId, { method: "paperclip_key" }, actor); + registerKey(provider, started.machineSignIn.pendingPublicKey!); + // Another rotation overwrites the staged private key in the vault. + const [row] = await db.select().from(toolConnections).where(eq(toolConnections.id, created.connectionId)); + const staged = row!.credentialSecretRefs.find((ref) => ref.configPath === "oauth.client_private_key_next")!; + const other = await generateClientKeyPair("RS256"); + registerKey(provider, { keyId: other.publicKey.keyId, pem: other.publicKey.publicKeyPem }); + await secretService(db).rotate(staged.secretId, { value: other.privateKeyPem }); + await expect(svc.completeMachineSignInRotation(company.id, created.connectionId, actor)) + .rejects.toMatchObject({ status: 409, details: expect.objectContaining({ code: "client_credentials_rotation_changed" }) }); + const status = await svc.getMachineSignIn(company.id, created.connectionId); + expect(status.publicKey?.keyId).toBe(created.machineSignIn.publicKey!.keyId); + // The current key still signs in. + const grant = await organizationGrant(created.connectionId); + await svc.refreshOAuthGrantCredentials({ companyId: company.id, connectionId: created.connectionId, grantId: grant.id, forceRefresh: true, actor }); + expect(decodeJwt(provider.state.tokenRequests.at(-1)!.body.get("client_assertion")!).header.kid).toBe(created.machineSignIn.publicKey!.keyId); + }); + + it("keeps working credentials when a retest's client can't use the server", async () => { + const company = await createCompany(); + const provider = useProvider(); + const svc = service(); + const created = await svc.createMachineSignIn(company.id, { + link: MCP_URL, method: "client_secret", clientId: CLIENT_ID, clientSecret: CLIENT_SECRET, + }, actor); + await svc.testMachineSignIn(company.id, created.connectionId, {}, actor); + provider.state.extraClients.set("other-client", "other-secret"); + provider.state.mcpDeniedClients.add("other-client"); + await expect(svc.testMachineSignIn(company.id, created.connectionId, { clientId: "other-client", clientSecret: "other-secret" }, actor)) + .rejects.toMatchObject({ details: expect.objectContaining({ code: "client_credentials_rejected" }) }); + const status = await svc.getMachineSignIn(company.id, created.connectionId); + expect(status.clientId).toBe(CLIENT_ID); + // The working connection still serves tools with the original client. + const health = await svc.checkHealth(created.connectionId, actor); + expect(health.connection.healthStatus).toBe("ok"); + expect(provider.state.tokenClients.get(provider.state.mcpRequests.at(-1)!.authorization!.replace("Bearer ", ""))).toBe(CLIENT_ID); + + // A retest that passes gives the organization grant the new token's expiry. + const grant = await organizationGrant(created.connectionId); + await db.update(connectionGrants).set({ + providerTenant: { ...(grant.providerTenant ?? {}), oauth: { ...((grant.providerTenant ?? {}).oauth as object ?? {}), accessTokenExpiresAt: "2000-01-01T00:00:00.000Z" } }, + }).where(eq(connectionGrants.id, grant.id)); + await svc.testMachineSignIn(company.id, created.connectionId, {}, actor); + const retested = await organizationGrant(created.connectionId); + expect(Date.parse(String((retested.providerTenant?.oauth as Record).accessTokenExpiresAt))).toBeGreaterThan(Date.now() + 3_000_000); + }); + + it("never reactivates a removed connection", async () => { + const company = await createCompany(); + const provider = useProvider(); + const { svc, created } = await setupKeyConnection(company.id, provider); + await svc.archiveConnection(created.connectionId, company.id, actor); + await expect(svc.testMachineSignIn(company.id, created.connectionId, { clientSecret: CLIENT_SECRET }, actor)) + .rejects.toMatchObject({ details: expect.objectContaining({ code: "client_credentials_connection_removed" }) }); + const [row] = await db.select().from(toolConnections).where(eq(toolConnections.id, created.connectionId)); + expect(row!.status).toBe("archived"); + }); + + it("does not undo a rotation that lands while a test is running", async () => { + const company = await createCompany(); + const provider = useProvider(); + const { svc, created } = await setupKeyConnection(company.id, provider); + provider.state.clientSecret = "switched-secret"; + // While the test talks to the server, an admin switches to a client secret. + provider.state.onAuthorizedMcpRequest = async () => { + await svc.rotateMachineSignIn(company.id, created.connectionId, { method: "client_secret", clientSecret: "switched-secret" }, actor); + }; + await expect(svc.testMachineSignIn(company.id, created.connectionId, {}, actor)) + .rejects.toMatchObject({ status: 409, details: expect.objectContaining({ code: "client_credentials_changed" }) }); + const status = await svc.getMachineSignIn(company.id, created.connectionId); + expect(status.method).toBe("client_secret"); + const [row] = await db.select().from(toolConnections).where(eq(toolConnections.id, created.connectionId)); + expect(row!.credentialSecretRefs.map((ref) => ref.configPath)).not.toContain("oauth.client_private_key"); + // The next token request uses the secret, not the deleted key. + const grant = await organizationGrant(created.connectionId); + await svc.refreshOAuthGrantCredentials({ companyId: company.id, connectionId: created.connectionId, grantId: grant.id, forceRefresh: true, actor }); + expect(provider.state.tokenRequests.at(-1)!.authorization).toBe(`Basic ${Buffer.from(`${CLIENT_ID}:switched-secret`).toString("base64")}`); + }); + + it("tries a new client secret before replacing the old one", async () => { + const company = await createCompany(); + const provider = useProvider(); + const svc = service(); + const created = await svc.createMachineSignIn(company.id, { + link: MCP_URL, method: "client_secret", clientId: CLIENT_ID, clientSecret: CLIENT_SECRET, + }, actor); + await svc.testMachineSignIn(company.id, created.connectionId, {}, actor); + await expect(svc.rotateMachineSignIn(company.id, created.connectionId, { method: "client_secret", clientSecret: "wrong-new-secret" }, actor)) + .rejects.toMatchObject({ details: expect.objectContaining({ code: "client_credentials_rejected" }) }); + const grant = await organizationGrant(created.connectionId); + // The old secret still works. + await svc.refreshOAuthGrantCredentials({ companyId: company.id, connectionId: created.connectionId, grantId: grant.id, forceRefresh: true, actor }); + provider.state.clientSecret = "rotated-secret"; + const rotated = await svc.rotateMachineSignIn(company.id, created.connectionId, { method: "client_secret", clientSecret: "rotated-secret" }, actor); + expect(rotated.completed).toBe(true); + await svc.refreshOAuthGrantCredentials({ companyId: company.id, connectionId: created.connectionId, grantId: grant.id, forceRefresh: true, actor }); + expect(provider.state.tokenRequests.at(-1)!.authorization).toBe(`Basic ${Buffer.from(`${CLIENT_ID}:rotated-secret`).toString("base64")}`); + }); + + it("refuses machine sign-in on a personal connection", async () => { + const company = await createCompany(); + useProvider(); + const svc = service(); + const created = await svc.createMachineSignIn(company.id, { link: MCP_URL, method: "client_secret", clientId: CLIENT_ID, clientSecret: CLIENT_SECRET }, actor); + await db.update(toolConnections).set({ credentialPolicy: "per_user" }).where(eq(toolConnections.id, created.connectionId)); + await expect(svc.testMachineSignIn(company.id, created.connectionId, {}, actor)) + .rejects.toMatchObject({ status: 422, details: expect.objectContaining({ code: "client_credentials_personal_refused" }) }); + }); + + it("logs setup, verification, rotation and removal without any secret material", async () => { + const company = await createCompany(); + const provider = useProvider(); + const app = express(); + app.use(express.json()); + app.use((req, _res, next) => { + req.actor = { type: "board", userId: "board-user", userName: "Board User", userEmail: null, isInstanceAdmin: true, source: "local_implicit" }; + next(); + }); + app.use("/api", toolAccessRoutes(db, transport)); + app.use(errorHandler); + + const created = await request(app).post(`/api/companies/${company.id}/tools/machine-sign-in`) + .send({ link: MCP_URL, method: "paperclip_key", clientId: CLIENT_ID }); + expect(created.status).toBe(201); + registerKey(provider, created.body.machineSignIn.publicKey); + const connectionId = created.body.connectionId as string; + expect((await request(app).post(`/api/tool-connections/${connectionId}/machine-sign-in/test`).send({})).status).toBe(200); + expect((await request(app).post(`/api/tool-connections/${connectionId}/machine-sign-in/rotate`).send({ method: "client_secret", clientSecret: CLIENT_SECRET })).status).toBe(200); + expect((await request(app).get(`/api/tool-connections/${connectionId}/machine-sign-in`)).body).toMatchObject({ method: "client_secret", hasClientSecret: true }); + expect((await request(app).delete(`/api/tool-connections/${connectionId}`)).status).toBe(200); + + const rows = await db.select().from(activityLog).where(eq(activityLog.entityId, connectionId)); + expect(rows.map((row) => row.action)).toEqual(expect.arrayContaining([ + "tool_connection.machine_sign_in_created", + "tool_connection.machine_sign_in_verified", + "tool_connection.machine_sign_in_rotated", + "tool_connection.archived", + ])); + expect(rows.find((row) => row.action === "tool_connection.archived")?.details).toMatchObject({ machineSignIn: true }); + const logged = JSON.stringify(rows); + expect(logged).not.toContain(CLIENT_SECRET); + expect(logged).not.toContain("PRIVATE KEY"); + expect(logged).not.toContain("machine-token-"); + // Removal destroys the vault entries the connection owned. + const remaining = await db.select({ id: companySecrets.id, status: companySecrets.status }).from(companySecrets) + .where(eq(companySecrets.companyId, company.id)); + expect(remaining.filter((secret) => secret.status === "active")).toEqual([]); + }); + + it("refuses a validation error in plain words", async () => { + const company = await createCompany(); + useProvider(); + await expect(service().createMachineSignIn(company.id, { + link: "https://mcp.example.test/mcp?api_key=abc", method: "client_secret", + }, actor)).rejects.toMatchObject({ status: 400, message: expect.stringContaining("Remove the key from the server address") }); + }); +}); diff --git a/server/src/routes/openapi.ts b/server/src/routes/openapi.ts index b54469d607c..754c7ab8649 100644 --- a/server/src/routes/openapi.ts +++ b/server/src/routes/openapi.ts @@ -291,6 +291,9 @@ import { workspaceFileResourceQuerySchema, // Tool access connectToolAppSchema, + createMachineSignInSchema, + testMachineSignInSchema, + rotateMachineSignInSchema, configureRailwaySshSchema, createToolApplicationSchema, updateToolApplicationSchema, @@ -12545,6 +12548,68 @@ registerCurrentRoute({ summary: "Archive a tool connection", }); +const MACHINE_SIGN_IN_ERRORS = { + 400: r.badRequest, + 401: r.unauthorized, + 403: r.forbidden, + 404: r.notFound, + 422: r.unprocessable, +}; + +registerCurrentRoute({ + "x-paperclip-transport": "rest", + "x-paperclip-connection-management": "connections:configure", + method: "post", + path: "/api/companies/{companyId}/tools/machine-sign-in", + tags: ["tool-access"], + summary: "Start machine sign-in (OAuth client credentials) for an organization connection", + description: "Creates a draft organization connection to a remote MCP server that signs in as a registered client, with no person involved. With method paperclip_key, Paperclip creates a signing key, keeps the private half in the company vault, and returns the public key to register with the provider. Secrets are write-only.", + body: createMachineSignInSchema, + responses: { 201: r.ok(), ...MACHINE_SIGN_IN_ERRORS }, +}); + +registerCurrentRoute({ + "x-paperclip-transport": "rest", + "x-paperclip-connection-management": "connections:configure", + method: "get", + path: "/api/tool-connections/{connectionId}/machine-sign-in", + tags: ["tool-access"], + summary: "Read a connection's machine sign-in settings (no secret values)", + responses: { 200: r.ok(), ...MACHINE_SIGN_IN_ERRORS }, +}); + +registerCurrentRoute({ + "x-paperclip-transport": "rest", + "x-paperclip-connection-management": "connections:configure", + method: "post", + path: "/api/tool-connections/{connectionId}/machine-sign-in/test", + tags: ["tool-access"], + summary: "Test machine sign-in against the provider and server, then turn the connection on", + body: testMachineSignInSchema, + responses: { 200: r.ok(), ...MACHINE_SIGN_IN_ERRORS }, +}); + +registerCurrentRoute({ + "x-paperclip-transport": "rest", + "x-paperclip-connection-management": "connections:configure", + method: "post", + path: "/api/tool-connections/{connectionId}/machine-sign-in/rotate", + tags: ["tool-access"], + summary: "Replace the machine sign-in key or client secret without downtime", + body: rotateMachineSignInSchema, + responses: { 200: r.ok(), ...MACHINE_SIGN_IN_ERRORS }, +}); + +registerCurrentRoute({ + "x-paperclip-transport": "rest", + "x-paperclip-connection-management": "connections:configure", + method: "post", + path: "/api/tool-connections/{connectionId}/machine-sign-in/rotate/complete", + tags: ["tool-access"], + summary: "Switch to the new machine sign-in key once the provider accepts it", + responses: { 200: r.ok(), ...MACHINE_SIGN_IN_ERRORS }, +}); + registerCurrentRoute({ "x-paperclip-transport": "rest", "x-paperclip-connection-management": "connections:configure", diff --git a/server/src/routes/tool-access.ts b/server/src/routes/tool-access.ts index 8faa88e2409..146d1f61b4a 100644 --- a/server/src/routes/tool-access.ts +++ b/server/src/routes/tool-access.ts @@ -23,6 +23,10 @@ import { type ToolConnection, type ToolConnectionCreateCapabilities, connectToolAppSchema, + createMachineSignInSchema, + testMachineSignInSchema, + rotateMachineSignInSchema, + type MachineSignInStatus, configureRailwaySshSchema, createConnectionGrantDelegationSchema, createToolStdioCommandTemplateSchema, @@ -105,6 +109,21 @@ const ORGANIZATION_GRANT_DENIAL_REASON = type Heartbeat = ReturnType; /** Allowlist (e.g. Google Sheets allowed spreadsheet ids) lives in connection config. */ +function asRecord(value: unknown): Record { + return value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; +} + +/** Activity for machine sign-in carries identifiers only: never a secret, key or token. */ +function machineSignInActivityDetails(status: MachineSignInStatus) { + return { + method: status.method, + issuer: status.issuer, + keyId: status.publicKey?.keyId ?? null, + pendingKeyId: status.pendingPublicKey?.keyId ?? null, + scopeCount: status.scopes.length, + }; +} + function allowlistIds(config: Record | null | undefined): string[] { const raw = config?.allowedSpreadsheetIds; if (!Array.isArray(raw)) return []; @@ -1005,6 +1024,87 @@ function connectorEnrollmentPrincipal(req: Request): string { } }); + // Machine sign-in (MCP OAuth client credentials): organization connections + // that authenticate as a registered client, with no person signing in. + router.post("/companies/:companyId/tools/machine-sign-in", managementGuard("connections:configure", "POST /companies/:companyId/tools/machine-sign-in"), validate(createMachineSignInSchema), async (req, res) => { + const companyId = req.params.companyId as string; + assertToolAppMutationAccess(req, companyId); + if (!await isToolConnectionManagerQuiet(req, companyId)) throw forbidden(ORGANIZATION_GRANT_DENIAL_REASON); + try { + const result = await svc.createMachineSignIn(companyId, req.body, connectionRequestActor(req)); + await logActivity(db, { + companyId, + ...connectionRequestActor(req), + action: "tool_connection.machine_sign_in_created", + entityType: "tool_connection", + entityId: result.connectionId, + details: machineSignInActivityDetails(result.machineSignIn), + }); + res.status(201).json(result); + } catch (error) { + svc.ensureNoDuplicateNameError(error); + } + }); + + router.get("/tool-connections/:connectionId/machine-sign-in", managementGuard("connections:configure", "GET /tool-connections/:connectionId/machine-sign-in"), async (req, res) => { + const existing = await getAccessibleResource(req, res, svc.getConnection(req.params.connectionId as string), "Tool connection not found"); + if (!existing) return; + await assertToolConnectionConfigureAccess(req, existing); + res.json(await svc.getMachineSignIn(existing.companyId, existing.id)); + }); + + router.post("/tool-connections/:connectionId/machine-sign-in/test", managementGuard("connections:configure", "POST /tool-connections/:connectionId/machine-sign-in/test"), validate(testMachineSignInSchema), async (req, res) => { + const existing = await getAccessibleResource(req, res, svc.getConnection(req.params.connectionId as string), "Tool connection not found"); + if (!existing) return; + await assertToolConnectionConfigureAccess(req, existing); + const result = await svc.testMachineSignIn(existing.companyId, existing.id, req.body, connectionRequestActor(req)); + await logActivity(db, { + companyId: existing.companyId, + ...connectionRequestActor(req), + action: "tool_connection.machine_sign_in_verified", + entityType: "tool_connection", + entityId: existing.id, + details: { + ...machineSignInActivityDetails(await svc.getMachineSignIn(existing.companyId, existing.id)), + clientSecretReplaced: Boolean(req.body.clientSecret), + catalogEntryCount: result.catalog.length, + }, + }); + res.json(result); + }); + + router.post("/tool-connections/:connectionId/machine-sign-in/rotate", managementGuard("connections:configure", "POST /tool-connections/:connectionId/machine-sign-in/rotate"), validate(rotateMachineSignInSchema), async (req, res) => { + const existing = await getAccessibleResource(req, res, svc.getConnection(req.params.connectionId as string), "Tool connection not found"); + if (!existing) return; + await assertToolConnectionConfigureAccess(req, existing); + const result = await svc.rotateMachineSignIn(existing.companyId, existing.id, req.body, connectionRequestActor(req)); + await logActivity(db, { + companyId: existing.companyId, + ...connectionRequestActor(req), + action: result.completed ? "tool_connection.machine_sign_in_rotated" : "tool_connection.machine_sign_in_rotation_started", + entityType: "tool_connection", + entityId: existing.id, + details: machineSignInActivityDetails(result.machineSignIn), + }); + res.json(result); + }); + + router.post("/tool-connections/:connectionId/machine-sign-in/rotate/complete", managementGuard("connections:configure", "POST /tool-connections/:connectionId/machine-sign-in/rotate/complete"), async (req, res) => { + const existing = await getAccessibleResource(req, res, svc.getConnection(req.params.connectionId as string), "Tool connection not found"); + if (!existing) return; + await assertToolConnectionConfigureAccess(req, existing); + const result = await svc.completeMachineSignInRotation(existing.companyId, existing.id, connectionRequestActor(req)); + await logActivity(db, { + companyId: existing.companyId, + ...connectionRequestActor(req), + action: "tool_connection.machine_sign_in_rotated", + entityType: "tool_connection", + entityId: existing.id, + details: machineSignInActivityDetails(result.machineSignIn), + }); + res.json(result); + }); + router.post( "/companies/:companyId/tools/connections/:connectionId/start-authorization", managementGuard("connections:configure", "POST /companies/:companyId/tools/connections/:connectionId/start-authorization", { personal: true }), validate(startConnectionAuthorizationSchema), @@ -2286,7 +2386,11 @@ function connectorEnrollmentPrincipal(req: Request): string { action: "tool_connection.archived", entityType: "tool_connection", entityId: connection.id, - details: { transport: connection.transport, ...removal }, + details: { + transport: connection.transport, + ...(asRecord(asRecord(existing.config).oauth).grantType === "client_credentials" ? { machineSignIn: true } : {}), + ...removal, + }, }); if (applicationBefore.status !== "archived" && applicationAfter.status === "archived") { await logActivity(db, { diff --git a/server/src/services/mcp-http.ts b/server/src/services/mcp-http.ts index ef7f590d4ce..0a0b1dadbf2 100644 --- a/server/src/services/mcp-http.ts +++ b/server/src/services/mcp-http.ts @@ -36,6 +36,9 @@ const MODERN_ERROR_CODES = new Set([MCP_ERROR_HEADER_MISMATCH, MCP_ERROR_MISSING export const MCP_TASKS_EXTENSION_ID = "io.modelcontextprotocol/tasks"; const MCP_TASK_METHODS = new Set(["tasks/get", "tasks/update", "tasks/cancel"]); +/** MCP OAuth Client Credentials extension; declared only by connections that use machine sign-in. */ +export const MCP_OAUTH_CLIENT_CREDENTIALS_EXTENSION_ID = "io.modelcontextprotocol/oauth-client-credentials"; + const META_PROTOCOL_VERSION = "io.modelcontextprotocol/protocolVersion"; const META_CLIENT_CAPABILITIES = "io.modelcontextprotocol/clientCapabilities"; const META_CLIENT_INFO = "io.modelcontextprotocol/clientInfo"; @@ -54,7 +57,7 @@ export function isModernMcpProtocolVersion(version: unknown): version is string * they may then return URL input requests, which Paperclip shows as a link. * Legacy sessions keep the handshake capabilities they have always received. */ -export function mcpClientCapabilities(protocolVersion: string, options: { declareTasks?: boolean; elicitationForm?: boolean } = {}): Record { +export function mcpClientCapabilities(protocolVersion: string, options: { declareTasks?: boolean; elicitationForm?: boolean; declareClientCredentials?: boolean } = {}): Record { const version: string = protocolVersion; // The skills and tasks extensions are defined against stateless MCP only. // Tasks are declared only where Paperclip can wait on one: a provider tool @@ -66,6 +69,7 @@ export function mcpClientCapabilities(protocolVersion: string, options: { declar return { elicitation: options.elicitationForm ? { form: {}, url: {} } : { url: {} }, extensions: { ...MCP_APPS_CLIENT_CAPABILITY, ...MCP_SKILLS_CLIENT_CAPABILITY, ...(options.declareTasks ? { [MCP_TASKS_EXTENSION_ID]: {} } : {}), + ...(options.declareClientCredentials ? { [MCP_OAUTH_CLIENT_CREDENTIALS_EXTENSION_ID]: {} } : {}), } }; } // 2025-06-18 knows only form questions (`{}`); 2025-11-25 added URL mode. @@ -209,7 +213,7 @@ export type McpJsonRpcRequest = { id?: string | number; method: string; params?: export function mcpHttpRequestInit( headers: Record | undefined, message: McpJsonRpcRequest, - options: { toolInputSchema?: unknown; declareTasks?: boolean; elicitationForm?: boolean; progressToken?: string } = {}, + options: { toolInputSchema?: unknown; declareTasks?: boolean; elicitationForm?: boolean; declareClientCredentials?: boolean; progressToken?: string } = {}, ): { method: "POST"; headers: Record; body: string } { const requestHeaders = mcpHttpRequestHeaders(headers); const protocolVersion = mcpProtocolVersionOf(headers); @@ -222,7 +226,7 @@ export function mcpHttpRequestInit( _meta: { ...record(params._meta), [META_PROTOCOL_VERSION]: protocolVersion, - [META_CLIENT_CAPABILITIES]: mcpClientCapabilities(protocolVersion, { declareTasks: options.declareTasks, elicitationForm: options.elicitationForm }), + [META_CLIENT_CAPABILITIES]: mcpClientCapabilities(protocolVersion, { declareTasks: options.declareTasks, elicitationForm: options.elicitationForm, declareClientCredentials: options.declareClientCredentials }), [META_CLIENT_INFO]: MCP_CLIENT_INFO, }, }; diff --git a/server/src/services/native-runtime/runner-api-sensitive-response.ts b/server/src/services/native-runtime/runner-api-sensitive-response.ts index 97410b665cf..c84d81421c9 100644 --- a/server/src/services/native-runtime/runner-api-sensitive-response.ts +++ b/server/src/services/native-runtime/runner-api-sensitive-response.ts @@ -1,6 +1,6 @@ export function runnerApiSensitiveResponse(method: string, path: string): boolean { if (method === "POST" && /^\/api\/chat-endpoints\/\{[^}]+\}\/setup$/.test(path)) return true; - if (/(\/apps\/connect$|\/email\/connections$|setup-secret|\/github\/(setup|identity\/start)|\/slack\/resume|start-authorization|finalize-oauth-access|\/oauth\/|\/registration|\/managed\/authorize|\/install$|\/login|\/setup-token-login-sessions|\/local\/(attempts|check)(\/|$)|\/test-calls(\/|$))/.test(path)) return true; + if (/(\/apps\/connect$|\/email\/connections$|setup-secret|\/github\/(setup|identity\/start)|\/slack\/resume|start-authorization|finalize-oauth-access|\/machine-sign-in|\/oauth\/|\/registration|\/managed\/authorize|\/install$|\/login|\/setup-token-login-sessions|\/local\/(attempts|check)(\/|$)|\/test-calls(\/|$))/.test(path)) return true; if (/\/secrets\/\{[^}]+\}\/value$/.test(path)) return true; return /^\/api\/tool-gateway\/sessions(?:\/|$)/.test(path) || /^\/api\/companies\/\{[^}]+\}\/exports?(?:\/|$)/.test(path) diff --git a/server/src/services/tool-access.ts b/server/src/services/tool-access.ts index 7bccdf12de7..f11b63d9c2b 100644 --- a/server/src/services/tool-access.ts +++ b/server/src/services/tool-access.ts @@ -45,6 +45,7 @@ import { managedConnectorProfile } from "./tool-access/oauth/client-settings.js" import { createOAuthClientRegistration } from "./tool-access/oauth/client-registration.js"; import { createOAuthRefreshLeases } from "./tool-access/oauth/refresh-leases.js"; import { createLegacyOAuthRefresh } from "./tool-access/oauth/connection-refresh.js"; +import { createClientCredentialsService } from "./tool-access/oauth/client-credentials-service.js"; import { createOAuthGrantRefresh } from "./tool-access/oauth/grant-refresh.js"; import { createGitHubGrantContinuity } from "./tool-access/github-grant-continuity.js"; import { createToolRuntimeHealth } from "./tool-access/runtime/health.js"; @@ -67,7 +68,7 @@ import { createOAuthClientProtocol } from "./tool-access/oauth/client.js"; import type { OAuthClientRegistrationSource } from "./tool-access/oauth/types.js"; export { OAUTH_CLIENT_ID_METADATA_DOCUMENT_PATH, resolveOAuthClientIdMetadataDocumentUrl, oauthClientIdMetadataDocument } from "./tool-access/oauth/client-metadata.js"; -import { connectionMethodFor, connectionMethodForConnection, credentialFieldsFor, normalizeConnectionMethodConfig, projectedConnectionToolInputSchema } from "./tool-access/connection-methods.js"; +import { connectionMethodFor, connectionMethodForConnection, credentialFieldsFor, normalizeConnectionMethodConfig, projectedConnectionHeaders, projectedConnectionToolInputSchema } from "./tool-access/connection-methods.js"; export { normalizeConnectionMethodConfig, projectedConnectionHeaders, @@ -788,8 +789,19 @@ export function toolAccessService( const { syncCredentialBindings, annotateSavedOAuthClientSecret, oauthClientForConnection, createOrRotateOAuthSecret, getOAuthGrantRow, resolveOAuthGrantSecret } = credentialStore; const ensureOAuthClient = createOAuthClientRegistration(db, { now, options, store: credentialStore, getConnectionRow, requestOAuthClientRegistration }); const refreshLeases = createOAuthRefreshLeases(db, now, getConnectionRow, getOAuthGrantRow); - const maybeRefreshOAuthCredentials = createLegacyOAuthRefresh(db, { now, secrets, store: credentialStore, leases: refreshLeases, getConnectionRow, exchangeOAuthToken }); - const refreshOAuthGrantCredentials = createOAuthGrantRefresh(db, { now, store: credentialStore, leases: refreshLeases, currentCloudConnector, getConnectionRow, exchangeOAuthToken }); + // Later dependencies are reached through closures: they are only called at request time. + const clientCredentials = createClientCredentialsService(db, { + now, secrets, store: credentialStore, exchangeOAuthToken, getConnectionRow, fetchRemoteHttpUrl, assertOAuthEndpointUrl, + assertRemoteConnectionEndpointsAllowed, ensureDefaultOrganizationGrant, nextAvailableConnectionName, + checkConnectionHealth: (connectionId, actor) => checkConnectionHealth(connectionId, actor), + refreshCatalog: (connectionId, actor, refreshOptions) => refreshCatalog(connectionId, actor, refreshOptions), + probeServerWithToken: (connection, authorization, actor) => catalogDiscovery.remoteTools( + connection, { ...projectedConnectionHeaders(connection), Authorization: authorization }, actor, undefined, [], { live: true }, + ), + }); + const { requestClientCredentialsToken } = clientCredentials; + const maybeRefreshOAuthCredentials = createLegacyOAuthRefresh(db, { now, secrets, store: credentialStore, leases: refreshLeases, getConnectionRow, exchangeOAuthToken, requestClientCredentialsToken }); + const refreshOAuthGrantCredentials = createOAuthGrantRefresh(db, { now, store: credentialStore, leases: refreshLeases, currentCloudConnector, getConnectionRow, exchangeOAuthToken, requestClientCredentialsToken }); const { refreshManagedGitHubGrantAccess, syncGitHubGrantBindings, sweepGitHubConnectionContinuity, nextGitHubConnectionContinuityAt, githubConnectionContinuitySchedule } = createGitHubGrantContinuity(db, { now, currentCloudConnector, refreshOAuthGrantCredentials, resolveOAuthGrantSecret }); const vercelCredentials = createVercelCredentialService(db, now, vercelConnect); const { resolveVercelCredentialHeaders, vercelGrantForConnection } = vercelCredentials; @@ -3895,6 +3907,12 @@ export function toolAccessService( finishGalleryAppConnection, + createMachineSignIn: clientCredentials.createMachineSignIn, + testMachineSignIn: clientCredentials.testMachineSignIn, + rotateMachineSignIn: clientCredentials.rotateMachineSignIn, + completeMachineSignInRotation: clientCredentials.completeMachineSignInRotation, + getMachineSignIn: clientCredentials.getMachineSignIn, + reconnectGalleryApp, storeConnectorOAuthSecret: createOrRotateOAuthSecret, diff --git a/server/src/services/tool-access/catalog/discovery.ts b/server/src/services/tool-access/catalog/discovery.ts index b7e21d06eb0..70c5145e0e7 100644 --- a/server/src/services/tool-access/catalog/discovery.ts +++ b/server/src/services/tool-access/catalog/discovery.ts @@ -1,4 +1,5 @@ import { assertSupportedConnection } from "../connections/setup-policy.js"; +import { usesClientCredentials } from "../oauth/client-credentials.js"; import { vercelCredentialFor, vercelConnectResourcesFor } from "../vercel-credentials.js"; import { connectionSetupUrl, connectionReconnectUrl } from "../identifiers.js"; import type { ActorInfo } from "../types.js"; @@ -168,7 +169,9 @@ export function createToolCatalogDiscovery(db: Db, { now, audit, secrets, creden const sendRemote = (init: RequestInit) => withMcpConnectionFailure(() => requestRemoteHttpEndpoint(new URL(endpoint), init)); const sendToolsList = (requestHeaders: Record, cursor?: string) => { sessionHeaders = requestHeaders; - return sendRemote(mcpHttpRequestInit(requestHeaders, { id: listRequestId, method: "tools/list", params: cursor ? { cursor } : {} })); + return sendRemote(mcpHttpRequestInit(requestHeaders, { id: listRequestId, method: "tools/list", params: cursor ? { cursor } : {} }, { + declareClientCredentials: usesClientCredentials(connection), + })); }; const connectionConfig = asRecord(connection.config); const sourceTemplateKey = typeof connectionConfig?.sourceTemplateKey === "string" @@ -333,7 +336,10 @@ export function createToolCatalogDiscovery(db: Db, { now, audit, secrets, creden if ( response.status === 401 && connection.authKind === "oauth" && - connection.credentialSource === "paperclip_vault" + connection.credentialSource === "paperclip_vault" && + // A machine sign-in probe with candidate credentials must fail on its own + // answer, not succeed by falling back to the stored ones. + !(credentialHeaders && usesClientCredentials(connection)) ) { headers = { ...projectedConnectionHeaders(connection), @@ -386,6 +392,15 @@ export function createToolCatalogDiscovery(db: Db, { now, audit, secrets, creden connection.credentialSecretRefs.length === 0 && Object.keys(headers).length === 0 && !urlCarriesCredential(sentUrl); + if (response.status === 401 && usesClientCredentials(connection)) { + // Machine sign-in is pinned to the sign-in service recorded at setup. + // Never adopt whatever the server's challenge now advertises. + await response.body?.cancel().catch(() => undefined); + throw unprocessable( + "The server didn't accept this connection's machine sign-in. Test machine sign-in again from the connection's settings.", + { code: "client_credentials_rejected", status: response.status, setupUrl: connectionSetupUrl(connection) }, + ); + } if ( response.status === 401 && (/bearer|oauth|authorization/i.test(authenticate) || diff --git a/server/src/services/tool-access/oauth/client-credentials-service.ts b/server/src/services/tool-access/oauth/client-credentials-service.ts new file mode 100644 index 00000000000..3ef09066467 --- /dev/null +++ b/server/src/services/tool-access/oauth/client-credentials-service.ts @@ -0,0 +1,753 @@ +import { randomUUID } from "node:crypto"; +import { and, eq, ne, sql } from "drizzle-orm"; +import type { Db } from "@paperclipai/db"; +import { connectionGrants, toolApplications, toolConnections } from "@paperclipai/db"; +import type { + ConnectToolAppResult, + ToolCatalogRefreshResult, + ToolConnectionHealthCheckResult, + CreateMachineSignIn, + MachineSignInPublicKey, + MachineSignInSetupResult, + MachineSignInStatus, + RotateMachineSignIn, + TestMachineSignIn, + ToolCredentialSecretRef, +} from "@paperclipai/shared"; +import { badRequest, conflict, HttpError, unprocessable } from "../../../errors.js"; +import { writeConnectionCredential } from "../../connection-credentials.js"; +import { emitConnectionCreated, emitConnectionUpdated } from "../../connector-telemetry.js"; +import { notifyToolConnectionHealth } from "../../tool-connection-health-notifications.js"; +import type { secretService } from "../../secrets.js"; +import { splitRemoteUrlCredential } from "../../remote-url-credentials.js"; +import { connectionActorUserId } from "../actor.js"; +import { toApplication } from "../applications/projection.js"; +import { groupedActions } from "../catalog/action-summary.js"; +import type { createOrganizationGrantStore } from "../connections/organization-grant.js"; +import { toConnection } from "../connections/projection.js"; +import { connectionUid } from "../identifiers.js"; +import type { ActorInfo } from "../types.js"; +import type { createOAuthClientProtocol } from "./client.js"; +import { + CLIENT_PRIVATE_KEY_NEXT_PATH, + CLIENT_PRIVATE_KEY_PATH, + CLIENT_SECRET_PATH, + chooseAssertionAlgorithm, + chooseSecretAuthMethod, + clientCredentialsSettings, + generateClientKeyPair, + publicKeyForPrivateKey, + signClientAssertion, + type ClientCredentialsAuthMethod, + type ClientCredentialsSettings, + type ClientPublicKey, +} from "./client-credentials.js"; +import { actorForSecret, type OAuthCredentialStore } from "./credential-store.js"; +import { + asRecord, + canonicalResourceIndicator, + challengeOAuthHints, + normalizeOauthScopes, + oauthConfig, + protectedResourceMetadataUrls, + sameOAuthIssuer, + wellKnownMetadataUrls, +} from "./metadata.js"; +import { normalizeOAuthProviderError } from "./provider-errors.js"; +import type { OAuthTransport } from "./types.js"; + +type Dependencies = OAuthTransport & { + now: () => Date; + secrets: ReturnType; + store: Pick; + exchangeOAuthToken: ReturnType["exchangeOAuthToken"]; + getConnectionRow(idOrUid: string, companyId?: string): Promise; + assertRemoteConnectionEndpointsAllowed(config: Record): Promise; + ensureDefaultOrganizationGrant: ReturnType; + checkConnectionHealth(connectionId: string, actor?: ActorInfo): Promise; + refreshCatalog(connectionId: string, actor?: ActorInfo, options?: { enableAllByDefault?: boolean }): Promise; + /** List the server's tools with exactly these credentials, never the stored ones. */ + probeServerWithToken(connection: typeof toolConnections.$inferSelect, authorization: string, actor?: ActorInfo): Promise; + nextAvailableConnectionName(requestedName: string, existingNames: readonly string[]): string; +}; + +type DiscoveredSignInService = { + issuer: string | null; + tokenUrl: string; + authMethods: string[]; + signingAlgorithms: string[]; +}; + +type TokenOverrides = { + clientId?: string | null; + clientSecret?: string | null; + scopes?: string[]; + /** Sign with the staged replacement key instead of the current one. */ + usePendingKey?: boolean; + /** Sign with exactly this key (already read from the vault), so the key tested is the key promoted. */ + signingKey?: { privateKeyPem: string; publicKey: ClientPublicKey }; + authMethod?: ClientCredentialsAuthMethod; +}; + +const PERSONAL_REFUSAL = + "Machine sign-in is only for connections your organization shares. Personal connections sign in as you."; + +function issuerChangedError(advertised: string[]) { + const hosts = advertised.map((value) => { + try { return new URL(value).host; } catch { return value; } + }); + return unprocessable( + `This server now sends sign-in to a different service${hosts.length ? ` (${hosts.join(", ")})` : ""}. Paperclip won't send this connection's credentials there. Set up machine sign-in again to trust the new service.`, + { code: "client_credentials_issuer_changed" }, + ); +} + +function machineSignInChangedError() { + return conflict("This connection's machine sign-in changed or was removed while this was running. Reload it and try again.", { + code: "client_credentials_changed", + }); +} + +function rotationChangedError() { + return conflict("The new key changed while the rotation was finishing. Register the key shown now, then try again.", { + code: "client_credentials_rotation_changed", + }); +} + +function toPublicKey(key: ClientPublicKey | null): MachineSignInPublicKey | null { + if (!key) return null; + return { + keyId: key.keyId, + algorithm: key.algorithm, + pem: key.publicKeyPem, + jwk: key.publicJwk as Record, + createdAt: key.createdAt, + }; +} + +export function machineSignInStatus(connection: typeof toolConnections.$inferSelect): MachineSignInStatus { + const settings = clientCredentialsSettings(connection); + if (!settings) throw unprocessable("This connection doesn't use machine sign-in", { code: "client_credentials_not_configured" }); + const oauth = oauthConfig(connection); + return { + connectionId: connection.id, + method: settings.authMethod === "private_key_jwt" ? "paperclip_key" : "client_secret", + clientId: settings.clientId, + issuer: settings.issuer, + tokenUrl: settings.tokenUrl, + scopes: settings.scopes, + publicKey: toPublicKey(settings.key), + pendingPublicKey: toPublicKey(settings.pendingKey), + hasClientSecret: connection.credentialSecretRefs.some((ref) => ref.configPath === CLIENT_SECRET_PATH), + lastTokenAt: typeof oauth.refreshedAt === "string" ? oauth.refreshedAt + : typeof oauth.connectedAt === "string" ? oauth.connectedAt : null, + }; +} + +function scopesFrom(value: string | undefined): string[] | undefined { + return value === undefined ? undefined : normalizeOauthScopes(value.split(/\s+/)); +} + +export function requireClientCredentials( + request: ClientCredentialsService["requestClientCredentialsToken"] | undefined, +): ClientCredentialsService["requestClientCredentialsToken"] { + if (!request) throw new Error("Machine sign-in token requests are not configured for this service"); + return request; +} + +export function createClientCredentialsService(db: Db, deps: Dependencies) { + const { now, secrets, store, exchangeOAuthToken, getConnectionRow, fetchRemoteHttpUrl, assertOAuthEndpointUrl } = deps; + + async function fetchJson(url: string): Promise | null> { + try { + const response = await fetchRemoteHttpUrl(assertOAuthEndpointUrl("metadata", url), { headers: { accept: "application/json" } }); + if (!response.ok) { + await response.body?.cancel().catch(() => undefined); + return null; + } + const parsed = asRecord(await response.json().catch(() => null)); + return Object.keys(parsed).length ? parsed : null; + } catch { + return null; + } + } + + /** RFC 8414 / OIDC metadata for one issuer, accepted only when it names that same issuer. */ + async function signInServiceMetadata(issuer: string): Promise { + for (const url of wellKnownMetadataUrls(issuer)) { + const metadata = await fetchJson(url); + if (!metadata || typeof metadata.token_endpoint !== "string") continue; + if (typeof metadata.issuer !== "string" || !sameOAuthIssuer(metadata.issuer, issuer)) continue; + const grants = normalizeOauthScopes(metadata.grant_types_supported); + if (grants.length > 0 && !grants.includes("client_credentials")) { + throw unprocessable("This sign-in service doesn't offer machine sign-in. Check the provider's docs for a service account option.", { + code: "client_credentials_unsupported", + }); + } + return { + issuer: metadata.issuer.trim(), + tokenUrl: assertOAuthEndpointUrl("token", metadata.token_endpoint), + authMethods: normalizeOauthScopes(metadata.token_endpoint_auth_methods_supported), + signingAlgorithms: normalizeOauthScopes(metadata.token_endpoint_auth_signing_alg_values_supported), + }; + } + return null; + } + + /** The `resource_metadata` address from the server's answer to an unauthenticated request, if any. */ + async function challengedMetadataUrl(serverUrl: string): Promise { + try { + const response = await fetchRemoteHttpUrl(serverUrl, { + method: "POST", + headers: { "content-type": "application/json", accept: "application/json, text/event-stream" }, + body: JSON.stringify({ jsonrpc: "2.0", id: "paperclip-sign-in-probe", method: "server/discover", params: {} }), + }); + await response.body?.cancel().catch(() => undefined); + if (response.status !== 401) return null; + return challengeOAuthHints(response.headers.get("www-authenticate") ?? "").metadataUrl; + } catch { + return null; + } + } + + /** The sign-in services an MCP server's protected-resource metadata lists, or null when it publishes none. */ + async function advertisedSignInServices(serverUrl: string): Promise { + let endpoint: URL; + try { endpoint = new URL(serverUrl); } catch { return null; } + // The server's own challenge may name where its metadata lives; follow + // that (through the same URL checks) before the well-known locations. + const hinted = await challengedMetadataUrl(serverUrl); + for (const url of [...new Set([...(hinted ? [hinted] : []), ...protectedResourceMetadataUrls(endpoint)])]) { + const metadata = await fetchJson(url); + if (!metadata) continue; + const servers = Array.isArray(metadata.authorization_servers) + ? metadata.authorization_servers.filter((value): value is string => typeof value === "string" && value.trim().length > 0) + : []; + if (servers.length > 0) return servers.map((value) => value.trim()); + } + return null; + } + + async function discoverSignInService(serverUrl: string, authorizationServer?: string): Promise { + if (authorizationServer) { + const asIssuer = await signInServiceMetadata(authorizationServer); + if (asIssuer) return asIssuer; + // No metadata at that address: the admin gave the token URL itself. + return { issuer: null, tokenUrl: assertOAuthEndpointUrl("token", authorizationServer), authMethods: [], signingAlgorithms: [] }; + } + for (const issuer of (await advertisedSignInServices(serverUrl)) ?? []) { + const found = await signInServiceMetadata(issuer); + if (found) return found; + } + // A server may be its own sign-in service. + const self = await signInServiceMetadata(new URL(serverUrl).origin); + if (self) return self; + throw unprocessable( + "Paperclip couldn't find this server's sign-in service. Enter the sign-in service address or token URL from the provider's docs.", + { code: "client_credentials_server_not_found" }, + ); + } + + /** + * The issuer pin. A connection's credentials are bound to the sign-in service + * recorded at setup. If the MCP server now advertises other services and not + * that one, refuse rather than follow it. + */ + async function assertPinnedSignInService(connection: typeof toolConnections.$inferSelect, settings: ClientCredentialsSettings) { + if (!settings.issuer) return; + const serverUrl = typeof connection.config.url === "string" ? connection.config.url : null; + if (!serverUrl) return; + const advertised = await advertisedSignInServices(serverUrl); + if (advertised && !advertised.some((issuer) => sameOAuthIssuer(issuer, settings.issuer))) { + throw issuerChangedError(advertised); + } + } + + async function vaultValue(connection: typeof toolConnections.$inferSelect, configPath: string, actor?: ActorInfo) { + const ref = connection.credentialSecretRefs.find((candidate) => candidate.configPath === configPath); + if (!ref) return null; + return secrets.resolveSecretValue(connection.companyId, ref.secretId, ref.versionSelector ?? "latest", { + consumerType: "tool_connection", + consumerId: connection.id, + configPath, + actorType: actor?.actorType ?? "system", + actorId: actor?.actorId ?? null, + }); + } + + /** + * One token request with `grant_type=client_credentials`, authenticated the + * way this connection was set up. Credentials go only to the pinned token + * endpoint; there is no fallback to deployment-wide OAuth clients. + */ + async function requestClientCredentialsToken( + connection: typeof toolConnections.$inferSelect, + actor?: ActorInfo, + overrides: TokenOverrides = {}, + ) { + const settings = clientCredentialsSettings(connection); + if (!settings) throw unprocessable("This connection doesn't use machine sign-in", { code: "client_credentials_not_configured" }); + const clientId = overrides.clientId ?? settings.clientId; + if (!clientId) { + throw unprocessable("Enter the client ID the provider gave you for Paperclip.", { code: "client_credentials_client_id_missing" }); + } + const authMethod = overrides.authMethod ?? settings.authMethod; + const scopes = overrides.scopes ?? settings.scopes; + let clientAssertion: string | null = null; + let clientSecret: string | null = null; + if (authMethod === "private_key_jwt") { + const key = overrides.signingKey?.publicKey ?? (overrides.usePendingKey ? settings.pendingKey : settings.key); + const privateKeyPem = overrides.signingKey?.privateKeyPem + ?? await vaultValue(connection, overrides.usePendingKey ? CLIENT_PRIVATE_KEY_NEXT_PATH : CLIENT_PRIVATE_KEY_PATH, actor); + if (!key || !privateKeyPem) { + throw unprocessable("This connection's Paperclip key is missing. Set up machine sign-in again.", { code: "client_credentials_key_missing" }); + } + // The key ID comes from the private key actually read, never from config + // alone: a refresh that overlaps a rotation may read the new key while + // holding the old settings, and its signature and kid must still agree. + const signing = publicKeyForPrivateKey(privateKeyPem, key.algorithm, new Date(key.createdAt)); + clientAssertion = signClientAssertion({ + privateKeyPem, + algorithm: key.algorithm, + keyId: signing.keyId, + clientId, + audience: settings.tokenUrl, + now: now(), + }); + } else { + clientSecret = overrides.clientSecret ?? await vaultValue(connection, CLIENT_SECRET_PATH, actor); + if (!clientSecret) { + throw unprocessable("Enter the client secret the provider gave you for Paperclip.", { code: "client_credentials_secret_missing" }); + } + } + try { + return await exchangeOAuthToken({ + tokenUrl: settings.tokenUrl, + clientId, + clientSecret, + tokenEndpointAuthMethod: authMethod === "private_key_jwt" ? "none" : authMethod, + clientAssertion, + grantType: "client_credentials", + scopes, + resource: settings.resource, + }); + } catch (error) { + if (!(error instanceof HttpError)) throw error; + const providerError = normalizeOAuthProviderError(asRecord(error.details).providerError); + if (providerError === "invalid_client" || providerError === "unauthorized_client") { + throw unprocessable( + authMethod === "private_key_jwt" + ? "The provider didn't accept Paperclip's key. Check the client ID and that the public key shown here is registered for it." + : "The provider didn't accept the client ID and secret. Check both and try again.", + { code: "client_credentials_rejected", providerError }, + ); + } + if (providerError === "invalid_scope") { + throw unprocessable("The provider refused the requested scopes. Check the scopes in the provider's docs.", { + code: "client_credentials_scope_rejected", + providerError, + }); + } + throw error; + } + } + + function assertOrganizationConnection(connection: typeof toolConnections.$inferSelect) { + if (connection.credentialPolicy !== "shared") throw unprocessable(PERSONAL_REFUSAL, { code: "client_credentials_personal_refused" }); + } + + async function machineSignInConnection(companyId: string, connectionId: string) { + const connection = await getConnectionRow(connectionId, companyId); + if (connection.status === "archived") { + throw unprocessable("This connection was removed. Set up machine sign-in again to reconnect.", { + code: "client_credentials_connection_removed", + }); + } + assertOrganizationConnection(connection); + if (!clientCredentialsSettings(connection)) { + throw unprocessable("This connection doesn't use machine sign-in", { code: "client_credentials_not_configured" }); + } + return connection; + } + + async function writeVaultCredential( + connection: { companyId: string; name: string }, + configPath: string, + label: string, + value: string, + actor: ActorInfo | undefined, + existingRef?: ToolCredentialSecretRef, + ): Promise { + const { secret } = await db.transaction((tx) => writeConnectionCredential(tx, { + companyId: connection.companyId, + connectionName: connection.name, + configPath, + label, + value, + existingRef, + actor: actorForSecret(actor), + })); + return { secretId: secret.id, versionSelector: "latest", configPath, required: false, label }; + } + + async function saveConnection( + previous: typeof toolConnections.$inferSelect, + patch: Partial, + ) { + // Commit only if nothing that decides how this connection signs in changed + // since it was read (a rotation or another test), and it was not removed. + const clientAuth = asRecord(oauthConfig(previous).clientAuth); + const [updated] = await db.update(toolConnections) + .set({ ...patch, ...(patch.config ? { transportConfig: patch.config } : {}), updatedAt: now() }) + .where(and( + eq(toolConnections.id, previous.id), + eq(toolConnections.companyId, previous.companyId), + ne(toolConnections.status, "archived"), + sql`coalesce(${toolConnections.config} -> 'oauth' -> 'clientAuth', '{}'::jsonb) = ${JSON.stringify(clientAuth)}::jsonb`, + sql`${toolConnections.credentialSecretRefs} = ${JSON.stringify(previous.credentialSecretRefs)}::jsonb`, + )) + .returning(); + if (!updated) throw machineSignInChangedError(); + await store.syncCredentialBindings(updated); + if (updated.status === "active") await deps.ensureDefaultOrganizationGrant(updated); + emitConnectionUpdated(updated, previous, "gallery"); + return updated; + } + + async function createMachineSignIn(companyId: string, input: CreateMachineSignIn, actor?: ActorInfo): Promise { + await actor?.managementContext?.reauthorize(); + const split = splitRemoteUrlCredential(input.link); + if (split?.secretUrl) { + throw badRequest("Remove the key from the server address. Machine sign-in sends credentials only to the sign-in service."); + } + const serverUrl = split?.publicUrl ?? input.link; + const config: Record = { url: serverUrl, quarantineNewEntries: false, unverifiedServer: true }; + await deps.assertRemoteConnectionEndpointsAllowed(config); + const service = await discoverSignInService(serverUrl, input.authorizationServer); + const createdAt = now(); + let authMethod: ClientCredentialsAuthMethod; + let generated: Awaited> | null = null; + if (input.method === "paperclip_key") { + if (service.authMethods.length > 0 && !service.authMethods.includes("private_key_jwt")) { + throw unprocessable("This provider doesn't accept Paperclip keys. Choose Client secret instead.", { + code: "client_credentials_key_unsupported", + }); + } + authMethod = "private_key_jwt"; + generated = await generateClientKeyPair(chooseAssertionAlgorithm(service.signingAlgorithms), createdAt); + } else { + if (service.authMethods.length > 0 && !service.authMethods.some((method) => method === "client_secret_basic" || method === "client_secret_post")) { + throw unprocessable("This provider doesn't accept client secrets. Choose Paperclip key instead.", { + code: "client_credentials_secret_unsupported", + }); + } + authMethod = chooseSecretAuthMethod(service.authMethods); + } + config.oauth = { + provider: "client_credentials", + grantType: "client_credentials", + tokenUrl: service.tokenUrl, + issuer: service.issuer, + resource: canonicalResourceIndicator(serverUrl), + scopes: scopesFrom(input.scopes) ?? [], + clientId: input.clientId ?? null, + tokenEndpointAuthMethodsSupported: service.authMethods, + clientAuth: { method: authMethod, ...(generated ? { key: generated.publicKey } : {}), configuredAt: createdAt.toISOString() }, + }; + + const [applicationNames, connectionNames] = await Promise.all([ + db.select({ name: toolApplications.name }).from(toolApplications).where(eq(toolApplications.companyId, companyId)), + db.select({ name: toolConnections.name }).from(toolConnections).where(eq(toolConnections.companyId, companyId)), + ]); + const requestedName = input.name ?? new URL(serverUrl).host.replace(/^www\./, ""); + const name = deps.nextAvailableConnectionName(requestedName, [...applicationNames, ...connectionNames].map((row) => row.name)); + + const credentialSecretRefs: ToolCredentialSecretRef[] = []; + try { + if (generated) { + credentialSecretRefs.push(await writeVaultCredential({ companyId, name }, CLIENT_PRIVATE_KEY_PATH, "Machine sign-in key", generated.privateKeyPem, actor)); + } else if (input.clientSecret) { + credentialSecretRefs.push(await writeVaultCredential({ companyId, name }, CLIENT_SECRET_PATH, "Machine sign-in client secret", input.clientSecret, actor)); + } + const connection = await db.transaction(async (tx) => { + let application: typeof toolApplications.$inferSelect | undefined; + if (input.applicationId) { + [application] = await tx.select().from(toolApplications) + .where(and(eq(toolApplications.id, input.applicationId), eq(toolApplications.companyId, companyId))); + if (!application) throw badRequest("The app for this connection was not found"); + } else { + [application] = await tx.insert(toolApplications).values({ + companyId, + applicationKey: `app-gallery:link:${randomUUID()}`, + name, + description: `Connected app at ${serverUrl}`, + type: "mcp_http", + status: "draft", + metadata: { source: "link" }, + }).returning(); + } + const connectionId = randomUUID(); + const [row] = await tx.insert(toolConnections).values({ + id: connectionId, + companyId, + applicationId: application!.id, + name, + uid: connectionUid(application!.applicationKey ?? application!.name, name, connectionId), + connectionKind: "managed", + ownership: "customer", + authKind: "oauth", + credentialSource: "paperclip_vault", + transport: "mcp_remote", + status: "draft", + enabled: false, + config, + transportConfig: config, + credentialRefs: [], + credentialSecretRefs, + credentialPolicy: "shared", + createdByAgentId: actor?.actorType === "agent" ? (actor.actorId ?? null) : null, + createdByUserId: connectionActorUserId(actor), + }).returning(); + return row!; + }); + await store.syncCredentialBindings(connection); + emitConnectionCreated(connection, "gallery"); + return { connectionId: connection.id, connection: toConnection(connection), machineSignIn: machineSignInStatus(connection) }; + } catch (error) { + // Nothing references these yet; never leave an orphaned key or secret behind. + for (const ref of credentialSecretRefs) await secrets.remove(ref.secretId).catch(() => undefined); + throw error; + } + } + + /** + * Test the settings against the real sign-in service and server, then turn + * the connection on. A failed test leaves the draft (and its key) in place so + * the admin can fix the provider side and try again. + */ + async function testMachineSignIn( + companyId: string, + connectionId: string, + input: TestMachineSignIn, + actor?: ActorInfo, + ): Promise { + await actor?.managementContext?.reauthorize(); + let connection = await machineSignInConnection(companyId, connectionId); + const settings = clientCredentialsSettings(connection)!; + await assertPinnedSignInService(connection, settings); + const scopes = scopesFrom(input.scopes); + const token = await requestClientCredentialsToken(connection, actor, { + clientId: input.clientId, + clientSecret: input.clientSecret, + scopes, + }); + // Prove the server accepts the new client before anything is saved: a client + // the sign-in service accepts may still lack access to this server, and a + // failed retest must leave a working connection's credentials untouched. + await deps.probeServerWithToken(connection, `Bearer ${token.accessToken}`, actor); + // The settings worked: keep them. + const credentialSecretRefs = [...connection.credentialSecretRefs]; + if (input.clientSecret) { + const existing = credentialSecretRefs.find((ref) => ref.configPath === CLIENT_SECRET_PATH); + const ref = await writeVaultCredential(connection, CLIENT_SECRET_PATH, "Machine sign-in client secret", input.clientSecret, actor, existing); + if (!existing) credentialSecretRefs.push(ref); + } + const accessRef = await store.createOrRotateOAuthSecret({ + companyId, + connection, + configPath: "oauth.access_token", + label: "OAuth access token", + value: token.accessToken, + actor, + }); + const at = now(); + const expiresAt = token.expiresIn ? new Date(at.getTime() + token.expiresIn * 1000).toISOString() : null; + const oauth = oauthConfig(connection); + const nextConfig = { + ...connection.config, + oauth: { + ...oauth, + clientId: input.clientId ?? settings.clientId, + scopes: scopes ?? settings.scopes, + expiresAt, + scope: token.scope, + tokenType: token.tokenType, + connectedAt: oauth.connectedAt ?? at.toISOString(), + refreshedAt: at.toISOString(), + }, + }; + connection = await saveConnection(connection, { + config: nextConfig, + credentialSecretRefs: [...credentialSecretRefs.filter((ref) => ref.configPath !== "oauth.access_token"), accessRef], + credentialRefs: [ + ...connection.credentialRefs.filter((ref) => ref.name !== "oauth.access_token"), + { name: "oauth.access_token", secretId: accessRef.secretId, version: "latest", placement: "header", key: "Authorization", prefix: "Bearer " }, + ], + }); + // The organization grant is what tool calls run on; give it the token now, + // with this token's expiry (the grant's own expiry wins over the connection's). + await deps.ensureDefaultOrganizationGrant(connection); + await db.update(connectionGrants).set({ + providerTenant: sql`jsonb_set(coalesce(${connectionGrants.providerTenant}, '{}'::jsonb), '{oauth}', (coalesce(${connectionGrants.providerTenant}->'oauth', '{}'::jsonb) || ${JSON.stringify({ accessTokenExpiresAt: expiresAt, tokenType: token.tokenType, refreshedAt: at.toISOString() })}::jsonb) - 'refreshLease')`, + updatedAt: now(), + }).where(and( + eq(connectionGrants.companyId, companyId), + eq(connectionGrants.connectionId, connection.id), + eq(connectionGrants.kind, "organization"), + eq(connectionGrants.isDefault, true), + )); + await deps.checkConnectionHealth(connection.id, actor); + const before = connection; + connection = await db.transaction(async (tx) => { + await notifyToolConnectionHealth(tx, before); + const [active] = await tx.update(toolConnections) + .set({ status: "active", enabled: true, updatedAt: now() }) + .where(and(eq(toolConnections.id, before.id), eq(toolConnections.companyId, companyId), ne(toolConnections.status, "archived"))) + .returning(); + // Removed while the test ran: never bring it back. + if (!active) throw machineSignInChangedError(); + await tx.update(toolApplications) + .set({ status: "active", updatedAt: now() }) + .where(and(eq(toolApplications.id, before.applicationId), eq(toolApplications.companyId, companyId))); + return active!; + }); + emitConnectionUpdated(connection, before, "gallery"); + const refresh = await deps.refreshCatalog(connection.id, actor, { enableAllByDefault: false }); + const [application] = await db.select().from(toolApplications).where(eq(toolApplications.id, connection.applicationId)); + return { + connectionId: refresh.connection.id, + application: toApplication(application!), + connection: refresh.connection, + catalog: refresh.catalog, + actions: groupedActions(refresh.catalog), + suggestedDefaults: { access: "all_agents", askFirstRiskLevels: [] }, + auth: null, + }; + } + + /** + * Replace the key or secret without downtime. A new secret is tried before it + * replaces the old one. A new key is staged: the current key keeps working + * until the admin registers the new public key and completes the rotation. + */ + async function rotateMachineSignIn( + companyId: string, + connectionId: string, + input: RotateMachineSignIn, + actor?: ActorInfo, + ): Promise { + await actor?.managementContext?.reauthorize(); + const connection = await machineSignInConnection(companyId, connectionId); + const settings = clientCredentialsSettings(connection)!; + const oauth = oauthConfig(connection); + const clientAuth = asRecord(oauth.clientAuth); + if (input.method === "client_secret") { + const authMethod = settings.authMethod === "private_key_jwt" + ? chooseSecretAuthMethod(normalizeOauthScopes(oauth.tokenEndpointAuthMethodsSupported)) + : settings.authMethod; + await assertPinnedSignInService(connection, settings); + await requestClientCredentialsToken(connection, actor, { clientSecret: input.clientSecret, authMethod }); + const existing = connection.credentialSecretRefs.find((ref) => ref.configPath === CLIENT_SECRET_PATH); + const ref = await writeVaultCredential(connection, CLIENT_SECRET_PATH, "Machine sign-in client secret", input.clientSecret, actor, existing); + const retired = connection.credentialSecretRefs.filter((candidate) => + candidate.configPath === CLIENT_PRIVATE_KEY_PATH || candidate.configPath === CLIENT_PRIVATE_KEY_NEXT_PATH); + const updated = await saveConnection(connection, { + config: { ...connection.config, oauth: { ...oauth, clientAuth: { method: authMethod, configuredAt: now().toISOString() } } }, + credentialSecretRefs: [ + ...connection.credentialSecretRefs.filter((candidate) => candidate.configPath !== CLIENT_SECRET_PATH && !retired.includes(candidate)), + ref, + ], + }); + for (const old of retired) await secrets.remove(old.secretId).catch(() => undefined); + return { connectionId, connection: toConnection(updated), machineSignIn: machineSignInStatus(updated), completed: true }; + } + const generated = await generateClientKeyPair(settings.key?.algorithm ?? "RS256", now()); + const existing = connection.credentialSecretRefs.find((ref) => ref.configPath === CLIENT_PRIVATE_KEY_NEXT_PATH); + const ref = await writeVaultCredential(connection, CLIENT_PRIVATE_KEY_NEXT_PATH, "Machine sign-in replacement key", generated.privateKeyPem, actor, existing); + const updated = await saveConnection(connection, { + config: { ...connection.config, oauth: { ...oauth, clientAuth: { ...clientAuth, pendingKey: generated.publicKey } } }, + credentialSecretRefs: [...connection.credentialSecretRefs.filter((candidate) => candidate.configPath !== CLIENT_PRIVATE_KEY_NEXT_PATH), ref], + }); + return { connectionId, connection: toConnection(updated), machineSignIn: machineSignInStatus(updated), completed: false }; + } + + /** Switch to the staged key once the provider accepts it. */ + async function completeMachineSignInRotation( + companyId: string, + connectionId: string, + actor?: ActorInfo, + ): Promise { + await actor?.managementContext?.reauthorize(); + const connection = await machineSignInConnection(companyId, connectionId); + const settings = clientCredentialsSettings(connection)!; + if (!settings.pendingKey) throw unprocessable("There's no new key waiting. Start a key rotation first.", { code: "client_credentials_no_pending_key" }); + await assertPinnedSignInService(connection, settings); + // Read the staged key once and derive its public half from that exact value: + // the key that passes the test is the key that gets promoted, even if + // another rotation replaces the staged key meanwhile. + const pendingValue = await vaultValue(connection, CLIENT_PRIVATE_KEY_NEXT_PATH, actor); + if (!pendingValue) throw unprocessable("The new key is missing. Start the rotation again.", { code: "client_credentials_key_missing" }); + const promoted = publicKeyForPrivateKey(pendingValue, settings.pendingKey.algorithm, new Date(settings.pendingKey.createdAt)); + if (promoted.keyId !== settings.pendingKey.keyId) throw rotationChangedError(); + await requestClientCredentialsToken(connection, actor, { + authMethod: "private_key_jwt", + signingKey: { privateKeyPem: pendingValue, publicKey: promoted }, + }); + const pendingRef = connection.credentialSecretRefs.find((ref) => ref.configPath === CLIENT_PRIVATE_KEY_NEXT_PATH); + // One locked step: the row still stages this key, then the vault and config change together. + const updated = await db.transaction(async (tx) => { + const [locked] = await tx.select().from(toolConnections) + .where(and(eq(toolConnections.id, connection.id), eq(toolConnections.companyId, companyId))) + .for("update"); + const lockedSettings = locked ? clientCredentialsSettings(locked) : null; + if (!locked || lockedSettings?.pendingKey?.keyId !== promoted.keyId) throw rotationChangedError(); + const current = locked.credentialSecretRefs.find((ref) => ref.configPath === CLIENT_PRIVATE_KEY_PATH); + const { secret } = await writeConnectionCredential(tx, { + companyId, connectionName: locked.name, configPath: CLIENT_PRIVATE_KEY_PATH, label: "Machine sign-in key", + value: pendingValue, existingRef: current, actor: actorForSecret(actor), + }); + const keyRef: ToolCredentialSecretRef = { + secretId: secret.id, versionSelector: "latest", configPath: CLIENT_PRIVATE_KEY_PATH, required: false, label: "Machine sign-in key", + }; + const lockedOauth = oauthConfig(locked); + const nextConfig = { + ...locked.config, + oauth: { ...lockedOauth, clientAuth: { method: "private_key_jwt", key: promoted, configuredAt: now().toISOString() } }, + }; + const [row] = await tx.update(toolConnections).set({ + config: nextConfig, + transportConfig: nextConfig, + credentialSecretRefs: [ + ...locked.credentialSecretRefs.filter((ref) => + ref.configPath !== CLIENT_PRIVATE_KEY_PATH && ref.configPath !== CLIENT_PRIVATE_KEY_NEXT_PATH && ref.configPath !== CLIENT_SECRET_PATH), + keyRef, + ], + updatedAt: now(), + }).where(eq(toolConnections.id, locked.id)).returning(); + return row!; + }); + await store.syncCredentialBindings(updated); + if (updated.status === "active") await deps.ensureDefaultOrganizationGrant(updated); + emitConnectionUpdated(updated, connection, "gallery"); + const retired = connection.credentialSecretRefs.filter((ref) => + (ref === pendingRef) || (ref.configPath === CLIENT_SECRET_PATH)); + for (const old of retired) await secrets.remove(old.secretId).catch(() => undefined); + return { connectionId, connection: toConnection(updated), machineSignIn: machineSignInStatus(updated) }; + } + + async function getMachineSignIn(companyId: string, connectionId: string): Promise { + return machineSignInStatus(await machineSignInConnection(companyId, connectionId)); + } + + return { + requestClientCredentialsToken, + assertPinnedSignInService, + createMachineSignIn, + testMachineSignIn, + rotateMachineSignIn, + completeMachineSignInRotation, + getMachineSignIn, + }; +} + +export type ClientCredentialsService = ReturnType; diff --git a/server/src/services/tool-access/oauth/client-credentials.ts b/server/src/services/tool-access/oauth/client-credentials.ts new file mode 100644 index 00000000000..802238d5bcd --- /dev/null +++ b/server/src/services/tool-access/oauth/client-credentials.ts @@ -0,0 +1,188 @@ +import { createHash, createPublicKey, generateKeyPair, randomUUID, sign, type JsonWebKey, type KeyObject } from "node:crypto"; +import type { toolConnections } from "@paperclipai/db"; +import { asRecord, normalizeOauthScopes, oauthConfig } from "./metadata.js"; + +/** + * MCP OAuth Client Credentials extension (2026-07-28): machine-to-machine sign-in + * for organization-owned connections. No person signs in; Paperclip presents a + * client it was registered as, either with a signed assertion (RFC 7523, + * `private_key_jwt`, recommended) or a client secret. + * + * This module is the protocol half: key material, assertions, and how the + * connection records its pinned authorization server. Secret values never live + * here; they are read from the company vault by the caller just before a token + * request and never stored in config. + */ +export const CLIENT_ASSERTION_TYPE = "urn:ietf:params:oauth:client-assertion-type:jwt-bearer"; + +/** Vault paths. The private key and client secret are only ever sent to the pinned token endpoint. */ +export const CLIENT_PRIVATE_KEY_PATH = "oauth.client_private_key"; +export const CLIENT_PRIVATE_KEY_NEXT_PATH = "oauth.client_private_key_next"; +export const CLIENT_SECRET_PATH = "oauth.client_secret"; + +export type ClientCredentialsAuthMethod = "private_key_jwt" | "client_secret_basic" | "client_secret_post"; +export type ClientAssertionAlgorithm = "RS256" | "ES256"; + +/** Public half of a Paperclip-held key, safe to show and to store in config. */ +export type ClientPublicKey = { + keyId: string; + algorithm: ClientAssertionAlgorithm; + publicKeyPem: string; + publicJwk: JsonWebKey; + createdAt: string; +}; + +export type ClientCredentialsSettings = { + clientId: string | null; + tokenUrl: string; + /** Pinned authorization-server issuer. Null when the admin entered a token URL directly. */ + issuer: string | null; + scopes: string[]; + resource: string | null; + authMethod: ClientCredentialsAuthMethod; + key: ClientPublicKey | null; + pendingKey: ClientPublicKey | null; +}; + +function isAuthMethod(value: unknown): value is ClientCredentialsAuthMethod { + return value === "private_key_jwt" || value === "client_secret_basic" || value === "client_secret_post"; +} + +function publicKeyFrom(value: unknown): ClientPublicKey | null { + const record = asRecord(value); + if (typeof record.keyId !== "string" || typeof record.publicKeyPem !== "string") return null; + if (record.algorithm !== "RS256" && record.algorithm !== "ES256") return null; + return { + keyId: record.keyId, + algorithm: record.algorithm, + publicKeyPem: record.publicKeyPem, + publicJwk: asRecord(record.publicJwk) as JsonWebKey, + createdAt: typeof record.createdAt === "string" ? record.createdAt : new Date(0).toISOString(), + }; +} + +/** + * The machine sign-in settings a connection was set up with, or null when the + * connection does not use them. Older env-configured client-credentials rows + * (no `clientAuth`) keep their original path. + */ +export function clientCredentialsSettings( + connection: Pick, +): ClientCredentialsSettings | null { + const oauth = oauthConfig(connection); + if (oauth.grantType !== "client_credentials") return null; + const clientAuth = asRecord(oauth.clientAuth); + if (!isAuthMethod(clientAuth.method) || typeof oauth.tokenUrl !== "string") return null; + return { + clientId: typeof oauth.clientId === "string" && oauth.clientId.trim() ? oauth.clientId.trim() : null, + tokenUrl: oauth.tokenUrl, + issuer: typeof oauth.issuer === "string" && oauth.issuer.trim() ? oauth.issuer.trim() : null, + scopes: normalizeOauthScopes(oauth.scopes), + resource: typeof oauth.resource === "string" && oauth.resource ? oauth.resource : null, + authMethod: clientAuth.method, + key: publicKeyFrom(clientAuth.key), + pendingKey: publicKeyFrom(clientAuth.pendingKey), + }; +} + +export function usesClientCredentials(connection: Pick): boolean { + return clientCredentialsSettings(connection) !== null; +} + +/** Prefer RS256 (the most widely accepted); use ES256 only when the server asks for it. */ +export function chooseAssertionAlgorithm(supported: string[] | undefined): ClientAssertionAlgorithm { + if (!supported?.length || supported.includes("RS256")) return "RS256"; + if (supported.includes("ES256")) return "ES256"; + return "RS256"; +} + +function base64Url(value: Buffer | string): string { + return Buffer.from(value).toString("base64url"); +} + +/** RFC 7638 JWK thumbprint, used as the key ID so it is stable and verifiable. */ +export function jwkThumbprint(jwk: JsonWebKey): string { + const canonical = jwk.kty === "EC" + ? { crv: jwk.crv, kty: jwk.kty, x: jwk.x, y: jwk.y } + : { e: jwk.e, kty: jwk.kty, n: jwk.n }; + return base64Url(createHash("sha256").update(JSON.stringify(canonical)).digest()); +} + +export function publicKeyForPrivateKey(privateKeyPem: string, algorithm: ClientAssertionAlgorithm, createdAt: Date): ClientPublicKey { + const publicKey = createPublicKey(privateKeyPem); + const exported = publicKey.export({ format: "jwk" }); + const keyId = jwkThumbprint(exported); + return { + keyId, + algorithm, + publicKeyPem: publicKey.export({ format: "pem", type: "spki" }).toString(), + publicJwk: { ...exported, kid: keyId, alg: algorithm, use: "sig" }, + createdAt: createdAt.toISOString(), + }; +} + +function generatePrivateKey(algorithm: ClientAssertionAlgorithm): Promise { + return new Promise((resolve, reject) => { + const done = (error: Error | null, _publicKey: KeyObject, privateKey: KeyObject) => + error ? reject(error) : resolve(privateKey); + if (algorithm === "ES256") generateKeyPair("ec", { namedCurve: "P-256" }, done); + else generateKeyPair("rsa", { modulusLength: 2048 }, done); + }); +} + +/** + * A fresh signing key. The private half goes straight to the vault; only the + * public half is kept in config. Generated off the event loop: RSA key + * generation takes long enough to stall unrelated requests if done inline. + */ +export async function generateClientKeyPair(algorithm: ClientAssertionAlgorithm, now = new Date()): Promise<{ + privateKeyPem: string; + publicKey: ClientPublicKey; +}> { + const privateKey = await generatePrivateKey(algorithm); + const privateKeyPem = privateKey.export({ format: "pem", type: "pkcs8" }).toString(); + return { privateKeyPem, publicKey: publicKeyForPrivateKey(privateKeyPem, algorithm, now) }; +} + +/** Assertions are single-use and short-lived: a captured one is useless a minute later. */ +export const CLIENT_ASSERTION_LIFETIME_SECONDS = 60; + +/** + * RFC 7523 §3 client assertion: `iss` = `sub` = client ID, `aud` = the token + * endpoint it is presented to, a short `exp`, and a unique `jti`. + */ +export function signClientAssertion(input: { + privateKeyPem: string; + algorithm: ClientAssertionAlgorithm; + keyId: string; + clientId: string; + audience: string; + now?: Date; +}): string { + const issuedAt = Math.floor((input.now ?? new Date()).getTime() / 1000); + const header = { alg: input.algorithm, typ: "JWT", kid: input.keyId }; + const claims = { + iss: input.clientId, + sub: input.clientId, + aud: input.audience, + iat: issuedAt, + exp: issuedAt + CLIENT_ASSERTION_LIFETIME_SECONDS, + jti: randomUUID(), + }; + const signingInput = `${base64Url(JSON.stringify(header))}.${base64Url(JSON.stringify(claims))}`; + const signature = sign( + "sha256", + Buffer.from(signingInput), + input.algorithm === "ES256" + ? { key: input.privateKeyPem, dsaEncoding: "ieee-p1363" } + : input.privateKeyPem, + ); + return `${signingInput}.${base64Url(signature)}`; +} + +/** Secret authentication method the server accepts, preferring Basic as the spec requires servers to support it. */ +export function chooseSecretAuthMethod(supported: string[] | undefined): "client_secret_basic" | "client_secret_post" { + if (!supported?.length || supported.includes("client_secret_basic")) return "client_secret_basic"; + if (supported.includes("client_secret_post")) return "client_secret_post"; + return "client_secret_basic"; +} diff --git a/server/src/services/tool-access/oauth/client.ts b/server/src/services/tool-access/oauth/client.ts index 71d1eadea52..d5a1d898e9b 100644 --- a/server/src/services/tool-access/oauth/client.ts +++ b/server/src/services/tool-access/oauth/client.ts @@ -3,6 +3,7 @@ import { asRecord, originOf } from "./metadata.js"; import { normalizeOAuthProviderError, oauthProviderErrorMessage } from "./provider-errors.js"; import type { OAuthProviderEndpoints, OAuthTokenEndpointAuthMethod, OAuthTransport } from "./types.js"; import { oauthApplicationType } from "./client-metadata.js"; +import { CLIENT_ASSERTION_TYPE } from "./client-credentials.js"; const MAX_OAUTH_DCR_CLIENT_ID_LENGTH = 4_096; @@ -309,6 +310,12 @@ export function createOAuthClientProtocol({ fetchRemoteHttpUrl, assertOAuthEndpo refreshToken?: string | null; /** RFC 8707 resource indicator: the MCP server this token is for. */ resource?: string | null; + /** + * RFC 7523 signed client assertion (`private_key_jwt`). When present it is + * the only client authentication sent: the client is identified by the + * assertion's `sub`, so no client_id or secret goes in the request. + */ + clientAssertion?: string | null; }) { const body = new URLSearchParams(); if (input.grantType === "client_credentials") { @@ -324,10 +331,14 @@ export function createOAuthClientProtocol({ fetchRemoteHttpUrl, assertOAuthEndpo body.set("redirect_uri", input.redirectUri ?? ""); body.set("code_verifier", input.codeVerifier ?? ""); } - const tokenEndpointAuthMethod = - input.tokenEndpointAuthMethod ?? - (input.clientSecret ? "client_secret_post" : "none"); - if (tokenEndpointAuthMethod !== "client_secret_basic") + const tokenEndpointAuthMethod = input.clientAssertion + ? "private_key_jwt" + : input.tokenEndpointAuthMethod ?? + (input.clientSecret ? "client_secret_post" : "none"); + if (input.clientAssertion) { + body.set("client_assertion_type", CLIENT_ASSERTION_TYPE); + body.set("client_assertion", input.clientAssertion); + } else if (tokenEndpointAuthMethod !== "client_secret_basic") body.set("client_id", input.clientId); if (tokenEndpointAuthMethod === "client_secret_post") { if (!input.clientSecret) { diff --git a/server/src/services/tool-access/oauth/connection-refresh.ts b/server/src/services/tool-access/oauth/connection-refresh.ts index 4cdf1e8a8e5..c67de010179 100644 --- a/server/src/services/tool-access/oauth/connection-refresh.ts +++ b/server/src/services/tool-access/oauth/connection-refresh.ts @@ -14,6 +14,8 @@ import { withoutOAuthRefreshLease, type OAuthRefreshLeases } from "./refresh-lea import { oauthExpiresAtMs, oauthSecretRef } from "./client-settings.js"; import { connectionSetupUrl, connectionReconnectUrl } from "../identifiers.js"; import { singleFlight } from "../single-flight.js"; +import { usesClientCredentials } from "./client-credentials.js"; +import { requireClientCredentials, type ClientCredentialsService } from "./client-credentials-service.js"; function asRecord(value: unknown): Record { if (value && typeof value === "object" && !Array.isArray(value)) return value as Record; @@ -27,9 +29,11 @@ type LegacyRefreshDependencies = { leases: Pick; getConnectionRow(idOrUid: string, companyId?: string): Promise; exchangeOAuthToken: ReturnType["exchangeOAuthToken"]; + /** Absent only in narrow test harnesses that never use machine sign-in. */ + requestClientCredentialsToken?: ClientCredentialsService["requestClientCredentialsToken"]; }; -export function createLegacyOAuthRefresh(db: Db, { now, secrets, store, leases, getConnectionRow, exchangeOAuthToken }: LegacyRefreshDependencies) { +export function createLegacyOAuthRefresh(db: Db, { now, secrets, store, leases, getConnectionRow, exchangeOAuthToken, requestClientCredentialsToken }: LegacyRefreshDependencies) { // These maps remove duplicate work inside one service instance. OAuth also // uses the database refresh lease below as its cross-process boundary. const oauthRefreshFlights = new Map>(); @@ -138,12 +142,15 @@ const oauthRefreshFlights = new Map>(); }, ); } - const client = await oauthClientForConnection( + // Machine sign-in uses only the connection's own vault credentials and its + // pinned token endpoint, never a deployment-wide client. + const machineSignIn = usesClientCredentials(connection); + const client = machineSignIn ? null : await oauthClientForConnection( connection, oauth.provider, actor, ); - if (!client.clientId) + if (client && !client.clientId) throw unprocessable( `OAuth client id is not configured for ${oauth.provider}`, ); @@ -180,9 +187,9 @@ const oauthRefreshFlights = new Map>(); : null; let token: Awaited>; try { - token = await exchangeOAuthToken({ + token = machineSignIn || !client ? await requireClientCredentials(requestClientCredentialsToken)(connection, actor) : await exchangeOAuthToken({ tokenUrl: oauth.tokenUrl, - clientId: client.clientId, + clientId: client.clientId!, clientSecret: client.clientSecret, tokenEndpointAuthMethod: storedOAuthTokenEndpointAuthMethod( oauth, diff --git a/server/src/services/tool-access/oauth/grant-refresh.ts b/server/src/services/tool-access/oauth/grant-refresh.ts index 87b2dfc5618..e3ee3c79d95 100644 --- a/server/src/services/tool-access/oauth/grant-refresh.ts +++ b/server/src/services/tool-access/oauth/grant-refresh.ts @@ -18,6 +18,8 @@ import { withoutOAuthRefreshLease, type OAuthRefreshLeases } from "./refresh-lea import { managedConnectorProfile, oauthGrantConfig, oauthGrantExpiresAtMs } from "./client-settings.js"; import { connectionSetupUrl, connectionReconnectUrl } from "../identifiers.js"; import { singleFlight } from "../single-flight.js"; +import { usesClientCredentials } from "./client-credentials.js"; +import { requireClientCredentials, type ClientCredentialsService } from "./client-credentials-service.js"; function asRecord(value: unknown): Record { if (value && typeof value === "object" && !Array.isArray(value)) return value as Record; @@ -31,9 +33,11 @@ type GrantRefreshDependencies = { currentCloudConnector: () => PaperclipCloudConnector | null; getConnectionRow(idOrUid: string, companyId?: string): Promise; exchangeOAuthToken: ReturnType["exchangeOAuthToken"]; + /** Absent only in narrow test harnesses that never use machine sign-in. */ + requestClientCredentialsToken?: ClientCredentialsService["requestClientCredentialsToken"]; }; -export function createOAuthGrantRefresh(db: Db, { now, store, leases, currentCloudConnector, getConnectionRow, exchangeOAuthToken }: GrantRefreshDependencies) { +export function createOAuthGrantRefresh(db: Db, { now, store, leases, currentCloudConnector, getConnectionRow, exchangeOAuthToken, requestClientCredentialsToken }: GrantRefreshDependencies) { const oauthGrantRefreshFlights = new Map>(); const { getOAuthGrantRow, resolveOAuthGrantSecret, createOrRotateOAuthSecret, oauthClientForConnection, syncCredentialBindings } = store; const { acquireOAuthGrantRefreshLease, clearOAuthGrantRefreshLease } = leases; @@ -272,10 +276,13 @@ export function createOAuthGrantRefresh(db: Db, { now, store, leases, currentClo if (!lease.leaseId) return lease.grant; try { const grant = lease.grant; - const refreshRef = grant.credentialSecretRefs.find( + // Machine sign-in has no refresh token: it simply asks for a new access + // token with the connection's own client credentials. + const machineSignIn = usesClientCredentials(connection); + const refreshRef = machineSignIn ? undefined : grant.credentialSecretRefs.find( (ref) => ref.configPath === "oauth.refresh_token", ); - if (!refreshRef) { + if (!machineSignIn && !refreshRef) { throw unprocessable( "OAuth credentials have expired and no refresh token is available", { @@ -285,29 +292,32 @@ export function createOAuthGrantRefresh(db: Db, { now, store, leases, currentClo }, ); } - const refreshSecret = await resolveOAuthGrantSecret( + const refreshSecret = refreshRef ? await resolveOAuthGrantSecret( connection, grant, refreshRef, input.actor, input, - ); + ) : null; const credentialActor: ActorInfo | undefined = grant.kind === "user" && grant.subjectUserId ? { actorType: "user", actorId: grant.subjectUserId } : input.actor; - const client = await oauthClientForConnection( - connection, - oauthProvider, - credentialActor, - ); - if (!client.clientId) - throw unprocessable( - `OAuth client id is not configured for ${oauthProvider}`, - ); const grantOauth = oauthGrantConfig(grant); let token: Awaited>; try { + if (machineSignIn) { + token = await requireClientCredentials(requestClientCredentialsToken)(connection, input.actor); + } else { + const client = await oauthClientForConnection( + connection, + oauthProvider, + credentialActor, + ); + if (!client.clientId) + throw unprocessable( + `OAuth client id is not configured for ${oauthProvider}`, + ); token = await exchangeOAuthToken({ tokenUrl: oauthTokenUrl, clientId: client.clientId, @@ -323,14 +333,16 @@ export function createOAuthGrantRefresh(db: Db, { now, store, leases, currentClo : normalizeOauthScopes(oauth.scopes).length > 0 ? normalizeOauthScopes(oauth.scopes) : normalizeOauthScopes(oauth.scope), - refreshToken: refreshSecret.value, + refreshToken: refreshSecret!.value, resource: typeof oauth.resource === "string" && oauth.resource ? oauth.resource : null, }); + } } catch (error) { if ( + refreshRef && refreshSecret && error instanceof HttpError && asRecord(error.details).code === "oauth_reauthorization_required" ) { diff --git a/server/src/services/tool-gateway.ts b/server/src/services/tool-gateway.ts index 7bd6d594bcd..9a2adc2011b 100644 --- a/server/src/services/tool-gateway.ts +++ b/server/src/services/tool-gateway.ts @@ -30,6 +30,7 @@ import { isInsufficientConnectionScope, INSUFFICIENT_CONNECTION_SCOPE_MESSAGE } import { githubAccessDeniedInstructions, githubAccessError } from "./tool-access/github-access-errors.js"; import { githubDefaultOwnerApplies, githubDefaultOwnerArguments } from "./tool-access/github-default-owner.js"; import { toolSourceTemplateKeySql } from "./tool-access/source-template-key.js"; +import { usesClientCredentials } from "./tool-access/oauth/client-credentials.js"; import { boundedMcpToolName } from "./mcp-tool-names.js"; import { browserUseService } from "./browser-use.js"; import { isBrowserUseConnection } from "./browser-use-client.js"; @@ -8037,6 +8038,7 @@ export function createToolGatewayService( progressToken: progressReporter?.token, // A person can answer the app's form questions in the task chat. elicitationForm: Boolean(elicitationCall), + declareClientCredentials: usesClientCredentials(connection), }), redirect: "manual", signal: controller.signal, diff --git a/tests/runner-e2e/source-sdk-public-inputs.json b/tests/runner-e2e/source-sdk-public-inputs.json index 3be3def9c97..46250ab43be 100644 --- a/tests/runner-e2e/source-sdk-public-inputs.json +++ b/tests/runner-e2e/source-sdk-public-inputs.json @@ -268,7 +268,7 @@ "packages/shared/src/home-paths.ts": "fa4f1c440653b76b3dbb7a51026d557287deb45d", "packages/shared/src/humanize-connection.test.ts": "ba9e90a44c63452b0bf9f733adb90241921e8eca", "packages/shared/src/humanize-connection.ts": "fbd414476a0ef7106a7c80d8572208781b091699", - "packages/shared/src/index.ts": "7128882e69b34ebeeefbab5b1ef5f0252c13d4b4", + "packages/shared/src/index.ts": "efab7ad18d89a60ddd3f961e35060b68424ef40b", "packages/shared/src/issue-activity-identity.test.ts": "3dc294640f54c2be67943e3e0149cf4eff2c95ca", "packages/shared/src/issue-activity-identity.ts": "1b0f07ec3fe0506aaaf80a4fa9893f3baf8f0075", "packages/shared/src/issue-attribution.test.ts": "84563685e7f3c48029243cf1cc24a3a09ee3cde9", @@ -408,7 +408,7 @@ "packages/shared/src/types/inbox-agent-policy.ts": "cb9d41d9681e80b834889268ad96015a39df92f6", "packages/shared/src/types/inbox-dismissal.ts": "58fb7013847396ac9971eb5799dd0bf0486e0371", "packages/shared/src/types/inbox-mine.ts": "1d302b531d34842189704abd2f75f8743d09490d", - "packages/shared/src/types/index.ts": "f63f67250d6d5a0b737478c634244ae3aba0328a", + "packages/shared/src/types/index.ts": "8abf1f4ed91e67f8e620a1a0b127799c793554ff", "packages/shared/src/types/instance.ts": "ddabd7cc0d885b5fdf45b33a7937b7b704f8fb7d", "packages/shared/src/types/issue-tree-control.ts": "75489fffd491b8350b05caf57d7796419797d9df", "packages/shared/src/types/issue.ts": "1a59939658d0b28807b8018bacf83a3ded4fef74", @@ -431,7 +431,7 @@ "packages/shared/src/types/tailscale.ts": "d02a917945d7779009a2dea4c4b535e9478e0680", "packages/shared/src/types/task-workspace.ts": "a23afa13d2cb58dc2eb72692ed28c16ed6fb84fb", "packages/shared/src/types/teams-catalog.ts": "632fe75f8e5a4cfc455843c2a1a2daff28a022be", - "packages/shared/src/types/tool-access.ts": "4c05d176f9bb1d30593fd904c7c4fc3e91b431ca", + "packages/shared/src/types/tool-access.ts": "1d2c29fb7c0c22ae30b53ff18fe44545c18b4835", "packages/shared/src/types/user-profile.ts": "1ac26e033a79fab21dc267fd983959814c8b680f", "packages/shared/src/types/voice-sessions.ts": "95e3b62cdd51f62290e68536728ddd7297bc0459", "packages/shared/src/types/work-product.ts": "26f9129f606a207ce7a218ccef40f96747283086", @@ -485,7 +485,7 @@ "packages/shared/src/validators/folder.ts": "b486444ab09ec3d5a9afec7dbd2e71c1bc2beb67", "packages/shared/src/validators/goal.ts": "602724134915e1e83c50115abba36b288c0abd3d", "packages/shared/src/validators/inbox-agent-policy.ts": "29cef157f4d9b564fd813868038aa8c813df4a7b", - "packages/shared/src/validators/index.ts": "a62024f564b768d87e9b8a2869c0b8b9ce92dc09", + "packages/shared/src/validators/index.ts": "b0bc5449eb5c21fc89064d9d5243223ab33cf5e0", "packages/shared/src/validators/instance.test.ts": "3faa9593a9a05f74429d9b1d2251906a03e9a903", "packages/shared/src/validators/instance.ts": "d801db219659707f0c9a8c1fbad06e862be4892d", "packages/shared/src/validators/issue-chat-bootstrap.test.ts": "e1ebb9aa94da22c825bc3ddd32ce2cfc98bfe645", @@ -520,7 +520,7 @@ "packages/shared/src/validators/teams-catalog.ts": "7bddf40c091a52aaa54e79de784bea5ae496f9aa", "packages/shared/src/validators/text.ts": "74c430fe17cdcfd4b56dd08a4c57cd3d567670a8", "packages/shared/src/validators/tool-access.test.ts": "b5838320115946a936b256919a5bd3a2294c6070", - "packages/shared/src/validators/tool-access.ts": "cd28549a930e0b222904bb1121db17659bf12b05", + "packages/shared/src/validators/tool-access.ts": "f6ada4ebdc2d318c0bd50a82dac0692082e03468", "packages/shared/src/validators/trust-policy.test.ts": "1fc4818984eb0e14f034d56ad147600224496d4e", "packages/shared/src/validators/trust-policy.ts": "f16b002db940b06bd882d51df8b5712c576e1d7b", "packages/shared/src/validators/voice-sessions.ts": "018a71f9ac544645041ac3cc9919e59c95fbe998", diff --git a/ui/src/api/tools.ts b/ui/src/api/tools.ts index 9334522137c..2903ec92f1a 100644 --- a/ui/src/api/tools.ts +++ b/ui/src/api/tools.ts @@ -306,6 +306,16 @@ export const toolsApi = { }, connectApp: (companyId: string, input: ConnectToolApp) => api.post(`/companies/${companyId}/tools/apps/connect`, input), + createMachineSignIn: (companyId: string, input: import("@paperclipai/shared").CreateMachineSignIn) => + api.post(`/companies/${companyId}/tools/machine-sign-in`, input), + getMachineSignIn: (connectionId: string) => + api.get(`/tool-connections/${connectionId}/machine-sign-in`), + testMachineSignIn: (connectionId: string, input: import("@paperclipai/shared").TestMachineSignIn) => + api.post(`/tool-connections/${connectionId}/machine-sign-in/test`, input), + rotateMachineSignIn: (connectionId: string, input: import("@paperclipai/shared").RotateMachineSignIn) => + api.post(`/tool-connections/${connectionId}/machine-sign-in/rotate`, input), + completeMachineSignInRotation: (connectionId: string) => + api.post(`/tool-connections/${connectionId}/machine-sign-in/rotate/complete`, {}), startOAuth: ( connectionId: string, input: { diff --git a/ui/src/features/connections/ConnectionSetupFlow.tsx b/ui/src/features/connections/ConnectionSetupFlow.tsx index e3ba1e0123d..aa311babc4b 100644 --- a/ui/src/features/connections/ConnectionSetupFlow.tsx +++ b/ui/src/features/connections/ConnectionSetupFlow.tsx @@ -1465,6 +1465,19 @@ function StandardConnectionSetupFlow({ }; const effectiveGrantKind = fixedGrantKind ?? grantKind; + // A connected result (from the link probe or machine sign-in) goes straight to + // review with every discovered action enabled by default. + function acceptConnectedResult(result: ConnectToolAppResult) { + setLinkGuidance(null); + setConnectResult(result); + const defaults: Record = {}; + for (const a of result.actions.readOnly) defaults[a.catalogEntryId] = true; + for (const a of result.actions.canMakeChanges) defaults[a.catalogEntryId] = true; + setEnabled(defaults); + finishMutation.mutate({ result, enabled: defaults }); + } + const acceptMachineSignInResult = acceptConnectedResult; + const connectMutation = useMutation({ mutationFn: async (entryOverride?: AppDefinition) => { if (requiredAgentId && !resumeConnectionId && !reconnectConnectionId @@ -1590,13 +1603,7 @@ function StandardConnectionSetupFlow({ } oauthPopupRef.current?.close(); oauthPopupRef.current = null; - setLinkGuidance(null); - setConnectResult(result); - const defaults: Record = {}; - for (const a of result.actions.readOnly) defaults[a.catalogEntryId] = true; - for (const a of result.actions.canMakeChanges) defaults[a.catalogEntryId] = true; - setEnabled(defaults); - finishMutation.mutate({ result, enabled: defaults }); + acceptConnectedResult(result); }, onError: (error) => { oauthPopupRef.current?.close(); @@ -2826,6 +2833,9 @@ function StandardConnectionSetupFlow({ setLinkGuidance(null); connectApp(); }} + machineSignIn={effectiveGrantKind === "organization" && selectedCompanyId && !reconnectConnectionId + ? { companyId: selectedCompanyId, onConnected: acceptMachineSignInResult } + : null} /> )} diff --git a/ui/src/features/connections/GenericMcpConnectStep.machine.test.tsx b/ui/src/features/connections/GenericMcpConnectStep.machine.test.tsx new file mode 100644 index 00000000000..3304705a9fd --- /dev/null +++ b/ui/src/features/connections/GenericMcpConnectStep.machine.test.tsx @@ -0,0 +1,69 @@ +// @vitest-environment jsdom +import { act } from "react"; +import { createRoot, type Root } from "react-dom/client"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { newCustomHeaderRow } from "@/pages/apps/generic-mcp-connect"; +import { GenericMcpConnectStep } from "./GenericMcpConnectStep"; + +vi.mock("@/api/tools", () => ({ toolsApi: {} })); +vi.mock("@/pages/apps/AppLogo", () => ({ AppLogo: () => null })); + +describe("Machine sign-in in the custom server step", () => { + let root: Root; + let container: HTMLDivElement; + + beforeEach(() => { + container = document.createElement("div"); + document.body.append(container); + root = createRoot(container); + }); + afterEach(() => { + act(() => root.unmount()); + container.remove(); + }); + + async function render(machineSignIn: { companyId: string; onConnected: () => void } | null) { + await act(async () => { + root.render( + undefined} + authMode="auto" onAuthModeChange={() => undefined} headers={[newCustomHeaderRow()]} onHeadersChange={() => undefined} + oauthClientId="" onOAuthClientIdChange={() => undefined} oauthClientSecret="" onOAuthClientSecretChange={() => undefined} + advancedOpen onAdvancedOpenChange={() => undefined} guidance={null} submitting={false} + onBack={() => undefined} onConnect={() => undefined} machineSignIn={machineSignIn} + defaults={

Connects for every agent. Change

} + />, + ); + }); + } + const buttons = () => [...container.querySelectorAll("button")].map((button) => button.textContent ?? ""); + + it("is offered for organization connections and replaces Check link with its own setup", async () => { + await render({ companyId: "company-1", onConnected: () => undefined }); + const option = [...container.querySelectorAll('[role="radio"]')].find((item) => item.textContent?.includes("Machine sign-in")); + expect(option).toBeDefined(); + expect(buttons()).toContain("Check link"); + await act(async () => { (option as HTMLElement).click(); }); + expect(container.querySelector('[data-testid="machine-sign-in-setup"]')).not.toBeNull(); + expect(buttons()).not.toContain("Check link"); + expect(buttons()).toContain("Create key"); + // Who gets access stays visible and changeable while machine setup is open. + expect(container.querySelector('[data-testid="access-defaults"]')).not.toBeNull(); + }); + + it("keeps machine sign-in open when arrow keys move its own method choice", async () => { + await render({ companyId: "company-1", onConnected: () => undefined }); + const option = [...container.querySelectorAll('[role="radio"]')].find((item) => item.textContent?.includes("Machine sign-in")); + await act(async () => { (option as HTMLElement).click(); }); + const inner = [...container.querySelectorAll('[role="radio"]')].find((item) => item.textContent === "Paperclip key (recommended)") as HTMLElement; + await act(async () => { inner.dispatchEvent(new KeyboardEvent("keydown", { key: "ArrowRight", bubbles: true })); }); + expect(container.querySelector('[data-testid="machine-sign-in-setup"]')).not.toBeNull(); + const secret = [...container.querySelectorAll('[role="radio"]')].find((item) => item.textContent === "Client secret"); + expect(secret?.getAttribute("aria-checked")).toBe("true"); + }); + + it("is not offered for personal connections", async () => { + await render(null); + expect(container.textContent).not.toContain("Machine sign-in"); + }); +}); diff --git a/ui/src/features/connections/GenericMcpConnectStep.tsx b/ui/src/features/connections/GenericMcpConnectStep.tsx index 40e4b14759a..34dc87c8c9d 100644 --- a/ui/src/features/connections/GenericMcpConnectStep.tsx +++ b/ui/src/features/connections/GenericMcpConnectStep.tsx @@ -1,6 +1,6 @@ import { useState, type KeyboardEvent, type ReactNode } from "react"; import { Loader2, Plus, X } from "lucide-react"; -import type { AppDefinition } from "@paperclipai/shared"; +import type { AppDefinition, ConnectToolAppResult } from "@paperclipai/shared"; import { InlineBanner } from "@/components/InlineBanner"; import { Button } from "@/components/ui/button"; import { Collapsible, CollapsibleContent, CollapsibleTrigger } from "@/components/ui/collapsible"; @@ -8,6 +8,7 @@ import { Input } from "@/components/ui/input"; import { RadioCard } from "@/components/ui/radio-card"; import { redactUrlSecrets } from "@/lib/redact-url-secrets"; import { AppLogo } from "@/pages/apps/AppLogo"; +import { MachineSignInSetup } from "./MachineSignInSetup"; import { canSubmitGenericConnect, customHeaderError, @@ -112,6 +113,7 @@ export function GenericMcpConnectStep({ defaults, onBack, onConnect, + machineSignIn, }: { link: string; name: string; @@ -138,8 +140,13 @@ export function GenericMcpConnectStep({ defaults?: ReactNode; onBack: () => void; onConnect: () => void; + /** Offered only for organization connections: personal connections sign in as the person. */ + machineSignIn?: { companyId: string; onConnected: (result: ConnectToolAppResult) => void } | null; }) { const [clientFieldsOpen, setClientFieldsOpen] = useState(false); + // Machine sign-in runs its own setup and test, so it is a choice beside the modes, not one of them. + const [machineSelected, setMachineSelected] = useState(false); + const machine = Boolean(machineSelected && machineSignIn); const headerError = authMode === "custom_headers" ? customHeaderError(headers) : null; const canSubmit = canSubmitGenericConnect({ link, @@ -156,7 +163,7 @@ export function GenericMcpConnectStep({ // Stay open while the chosen option still needs something typed into it, so // a disabled primary action always has its reason on screen. const choiceNeedsInput = (authMode === "bearer" || authMode === "custom_headers") && !canSubmit; - const expanded = advancedOpen || choiceNeedsInput; + const expanded = advancedOpen || choiceNeedsInput || machine; const showClientFields = clientFieldsOpen || oauthClientId.trim().length > 0 || oauthClientSecret.trim().length > 0 @@ -169,6 +176,8 @@ export function GenericMcpConnectStep({ const moveSelection = (event: KeyboardEvent) => { // Arrow keys inside an option's own fields belong to that field. if ((event.target as HTMLElement).getAttribute("role") !== "radio") return; + // Only this group's own options: a nested choice (machine sign-in's) handles its own keys. + if ((event.target as HTMLElement).closest('[role="radiogroup"]') !== event.currentTarget) return; const step = event.key === "ArrowDown" || event.key === "ArrowRight" ? 1 : event.key === "ArrowUp" || event.key === "ArrowLeft" @@ -176,9 +185,14 @@ export function GenericMcpConnectStep({ : 0; if (step === 0) return; event.preventDefault(); - const index = AUTH_OPTIONS.findIndex((option) => option.mode === authMode); - const next = (index + step + AUTH_OPTIONS.length) % AUTH_OPTIONS.length; - onAuthModeChange(AUTH_OPTIONS[next]!.mode); + const count = AUTH_OPTIONS.length + (machineSignIn ? 1 : 0); + const index = machine ? AUTH_OPTIONS.length : AUTH_OPTIONS.findIndex((option) => option.mode === authMode); + const next = (index + step + count) % count; + if (next === AUTH_OPTIONS.length) setMachineSelected(true); + else { + setMachineSelected(false); + onAuthModeChange(AUTH_OPTIONS[next]!.mode); + } event.currentTarget.querySelectorAll('[role="radio"]')[next]?.focus(); }; @@ -353,7 +367,7 @@ export function GenericMcpConnectStep({ type="button" variant="link" className="h-auto p-0 text-xs font-semibold underline underline-offset-2" - disabled={submitting || choiceNeedsInput} + disabled={submitting || choiceNeedsInput || machine} data-testid="generic-mcp-sign-in-toggle" > {expanded ? "Done" : "Change"} @@ -368,7 +382,7 @@ export function GenericMcpConnectStep({ onKeyDown={moveSelection} > {AUTH_OPTIONS.map((option) => { - const selected = option.mode === authMode; + const selected = !machine && option.mode === authMode; const fields = selected ? optionFields(option.mode) : null; return (
@@ -379,27 +393,60 @@ export function GenericMcpConnectStep({ tabIndex={selected ? 0 : -1} disabled={submitting} className="bg-background" - onClick={() => onAuthModeChange(option.mode)} + onClick={() => { + setMachineSelected(false); + onAuthModeChange(option.mode); + }} /> {fields ?
{fields}
: null}
); })} + {machineSignIn ? ( +
+ setMachineSelected(true)} + /> + {machine ? ( +
+ +
+ ) : null} +
+ ) : null} - {defaults} + {/* Machine sign-in owns its footer (Test connection, not Check link) and shows the access line above it. */} + {machine ? null : ( + <> + {defaults} -
- - -
+
+ + +
+ + )} ); } diff --git a/ui/src/features/connections/MachineSignInSetup.test.tsx b/ui/src/features/connections/MachineSignInSetup.test.tsx new file mode 100644 index 00000000000..3dd8b219370 --- /dev/null +++ b/ui/src/features/connections/MachineSignInSetup.test.tsx @@ -0,0 +1,137 @@ +// @vitest-environment jsdom +import { act } from "react"; +import { createRoot, type Root } from "react-dom/client"; +import type { ConnectToolAppResult, MachineSignInSetupResult } from "@paperclipai/shared"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { MachineSignInSetup, type MachineSignInClient } from "./MachineSignInSetup"; + +vi.mock("@/api/tools", () => ({ toolsApi: {} })); + +const draft = { + connectionId: "connection-1", + connection: {}, + machineSignIn: { + connectionId: "connection-1", method: "paperclip_key", clientId: null, issuer: "https://auth.example", tokenUrl: "https://auth.example/token", + scopes: [], hasClientSecret: false, lastTokenAt: null, pendingPublicKey: null, + publicKey: { keyId: "kid-1", algorithm: "RS256", pem: "-----BEGIN PUBLIC KEY-----\nabc\n-----END PUBLIC KEY-----\n", jwk: { kty: "RSA", kid: "kid-1" }, createdAt: "2026-11-13T00:00:00Z" }, + }, +} as unknown as MachineSignInSetupResult; + +describe("Machine sign-in setup", () => { + let root: Root; + let container: HTMLDivElement; + const flush = async () => { await act(async () => { await new Promise((resolve) => setTimeout(resolve, 0)); }); }; + const button = (name: string) => [...container.querySelectorAll("button")].find((item) => item.textContent === name); + const click = async (name: string) => { await act(async () => { button(name)!.click(); }); await flush(); }; + const type = async (id: string, value: string) => { + const input = container.querySelector(`#${id}`)!; + await act(async () => { + Object.getOwnPropertyDescriptor(HTMLInputElement.prototype, "value")!.set!.call(input, value); + input.dispatchEvent(new Event("input", { bubbles: true })); + }); + }; + + beforeEach(() => { + container = document.createElement("div"); + document.body.append(container); + root = createRoot(container); + }); + afterEach(() => { + act(() => root.unmount()); + container.remove(); + }); + + it("creates a key, shows the public key to register, then tests with the client ID", async () => { + const client: MachineSignInClient = { + create: vi.fn(async () => draft), + test: vi.fn(async () => ({ connectionId: "connection-1" }) as ConnectToolAppResult), + status: vi.fn(), + }; + const onConnected = vi.fn(); + await act(async () => { + root.render( undefined} onConnected={onConnected} />); + }); + expect(container.textContent).toContain("Paperclip key (recommended)"); + await click("Create key"); + expect(client.create).toHaveBeenCalledWith("company-1", { link: "https://mcp.example/mcp", method: "paperclip_key" }); + expect(container.textContent).toContain("Register this public key"); + expect(container.querySelector("textarea")?.value).toContain("BEGIN PUBLIC KEY"); + expect(button("Test connection")?.disabled).toBe(true); + await type("machine-sign-in-client-id-confirm", "paperclip-automation"); + await click("Test connection"); + expect(client.test).toHaveBeenCalledWith("connection-1", { clientId: "paperclip-automation", scopes: "" }); + expect(onConnected).toHaveBeenCalled(); + }); + + it("keeps the draft and shows the provider's answer when the test fails", async () => { + const client: MachineSignInClient = { + create: vi.fn(async () => draft), + test: vi.fn(async () => { throw new Error("The provider didn't accept Paperclip's key."); }), + status: vi.fn(), + }; + await act(async () => { + root.render( undefined} onConnected={() => undefined} />); + }); + await type("machine-sign-in-client-id-confirm", "wrong"); + await click("Test connection"); + expect(container.textContent).toContain("Machine sign-in didn’t work"); + expect(container.textContent).toContain("The provider didn't accept Paperclip's key."); + expect(container.textContent).toContain("Register this public key"); + }); + + it("asks for the client ID and secret together for a client secret", async () => { + const client: MachineSignInClient = { + create: vi.fn(async () => ({ ...draft, machineSignIn: { ...draft.machineSignIn, method: "client_secret" as const, publicKey: null } })), + test: vi.fn(async () => ({}) as ConnectToolAppResult), + status: vi.fn(), + }; + await act(async () => { + root.render( undefined} onConnected={() => undefined} />); + }); + expect(button("Test connection")?.disabled).toBe(true); + await type("machine-sign-in-client-id", "id-1"); + await type("machine-sign-in-client-secret", "secret-1"); + await click("Test connection"); + expect(client.create).toHaveBeenCalledWith("company-1", expect.objectContaining({ method: "client_secret", clientId: "id-1", clientSecret: "secret-1" })); + expect(client.test).toHaveBeenCalledWith("connection-1", { clientId: "id-1", scopes: "" }); + }); + + it("lets a wrong secret be replaced after a failed test", async () => { + const secretDraft = { ...draft, machineSignIn: { ...draft.machineSignIn, method: "client_secret" as const, publicKey: null, clientId: "id-1", hasClientSecret: true, scopes: ["a.read"] } }; + const client: MachineSignInClient = { + create: vi.fn(), + test: vi.fn(async () => { throw new Error("The provider didn't accept the client ID and secret."); }), + status: vi.fn(), + }; + await act(async () => { + root.render( undefined} onConnected={() => undefined} />); + }); + await click("Test connection"); + expect(container.querySelector("#machine-sign-in-client-secret-replace")).not.toBeNull(); + await type("machine-sign-in-client-secret-replace", "fixed-secret"); + await type("machine-sign-in-scopes-confirm", ""); + await click("Test connection"); + expect(client.test).toHaveBeenLastCalledWith("connection-1", { clientId: "id-1", scopes: "", clientSecret: "fixed-secret" }); + }); + + it("resumes the key draft this tab started instead of creating another", async () => { + sessionStorage.setItem("paperclip.machine-sign-in-draft:company-1:https://mcp.example/mcp", "connection-1"); + let finish!: (value: typeof draft.machineSignIn) => void; + const client: MachineSignInClient = { + create: vi.fn(), + test: vi.fn(), + status: vi.fn(() => new Promise((resolve) => { finish = resolve; })), + }; + await act(async () => { + root.render( undefined} onConnected={() => undefined} />); + }); + // No second draft can be started while the saved one loads. + expect(button("Create key")?.disabled).toBe(true); + await act(async () => { finish(draft.machineSignIn); }); + await flush(); + expect(client.status).toHaveBeenCalledWith("connection-1"); + expect(container.textContent).toContain("Register this public key"); + expect(client.create).not.toHaveBeenCalled(); + sessionStorage.clear(); + }); +}); diff --git a/ui/src/features/connections/MachineSignInSetup.tsx b/ui/src/features/connections/MachineSignInSetup.tsx new file mode 100644 index 00000000000..98f29980b62 --- /dev/null +++ b/ui/src/features/connections/MachineSignInSetup.tsx @@ -0,0 +1,324 @@ +import { useEffect, useState } from "react"; +import { Loader2 } from "lucide-react"; +import type { + ConnectToolAppResult, + CreateMachineSignIn, + MachineSignInMethod, + MachineSignInSetupResult, + MachineSignInStatus, + TestMachineSignIn, +} from "@paperclipai/shared"; +import { toolsApi } from "@/api/tools"; +import { describeError } from "@/api/errors"; +import { InlineBanner } from "@/components/InlineBanner"; +import { Button } from "@/components/ui/button"; +import { Input } from "@/components/ui/input"; +import { Textarea } from "@/components/ui/textarea"; +import { cn } from "@/lib/utils"; +import { copyTextToClipboard } from "@/lib/clipboard"; + +/** The calls this setup makes. Stories pass in-memory fakes. */ +export interface MachineSignInClient { + create(companyId: string, input: CreateMachineSignIn): Promise; + test(connectionId: string, input: TestMachineSignIn): Promise; + status(connectionId: string): Promise; +} + +export const machineSignInApiClient: MachineSignInClient = { + create: (companyId, input) => toolsApi.createMachineSignIn(companyId, input), + test: (connectionId, input) => toolsApi.testMachineSignIn(connectionId, input), + status: (connectionId) => toolsApi.getMachineSignIn(connectionId), +}; + +type MachineSignInDraft = Pick; + +/** + * The draft this browser tab started for a server, so switching options or + * reloading resumes it (and its key) instead of creating another. + */ +function draftStorageKey(companyId: string, link: string) { + return `paperclip.machine-sign-in-draft:${companyId}:${link}`; +} +function readDraftId(key: string): string | null { + try { return sessionStorage.getItem(key); } catch { return null; } +} +function writeDraftId(key: string, connectionId: string | null) { + try { + if (connectionId) sessionStorage.setItem(key, connectionId); + else sessionStorage.removeItem(key); + } catch { /* storage unavailable: the draft is still listed in Apps */ } +} + +const METHOD_OPTIONS: Array<{ method: MachineSignInMethod; label: string; hint: string }> = [ + { + method: "paperclip_key", + label: "Paperclip key (recommended)", + hint: "Paperclip creates a key and keeps its private half. You register the public key with the provider; nothing secret is copied.", + }, + { + method: "client_secret", + label: "Client secret", + hint: "For providers that only issue a secret. Paperclip stores it securely and never shows it again.", + }, +]; + +export interface MachineSignInSetupProps { + companyId: string; + link: string; + name?: string; + client?: MachineSignInClient; + /** Resume at "register the key" with a key Paperclip already created. */ + initialSetup?: MachineSignInSetupResult | null; + initialMethod?: MachineSignInMethod; + initialError?: string | null; + onBack: () => void; + onConnected: (result: ConnectToolAppResult) => void; + /** Shown above the footer, such as who gets access to the connection. */ + beforeFooter?: React.ReactNode; +} + +/** + * Machine sign-in for an organization connection: the server's provider issues + * Paperclip a client of its own, so no person signs in. With a Paperclip key the + * admin registers the public key we show; with a client secret they paste it. + * Either way the last step is a real test against the provider and the server. + */ +export function MachineSignInSetup({ + companyId, + link, + name, + client = machineSignInApiClient, + initialSetup = null, + initialMethod = "paperclip_key", + initialError = null, + onBack, + onConnected, + beforeFooter, +}: MachineSignInSetupProps) { + const [method, setMethod] = useState(initialSetup?.machineSignIn.method ?? initialMethod); + const [signInService, setSignInService] = useState(""); + const [clientId, setClientId] = useState(initialSetup?.machineSignIn.clientId ?? ""); + const [clientSecret, setClientSecret] = useState(""); + const [scopes, setScopes] = useState(initialSetup?.machineSignIn.scopes.join(" ") ?? ""); + const [setup, setSetup] = useState(initialSetup); + const [error, setError] = useState(initialError); + const [busy, setBusy] = useState<"creating" | "testing" | null>(null); + const storageKey = draftStorageKey(companyId, link); + // While a saved draft loads, creating another would race it; wait for it. + const [restoring, setRestoring] = useState(() => !initialSetup && Boolean(readDraftId(storageKey))); + + useEffect(() => { + if (initialSetup) return; + const draftId = readDraftId(storageKey); + if (!draftId) return; + let cancelled = false; + setRestoring(true); + client.status(draftId).then((status) => { + if (cancelled) return; + setSetup({ connectionId: draftId, machineSignIn: status }); + setMethod(status.method); + setClientId((current) => current || (status.clientId ?? "")); + setScopes((current) => current || status.scopes.join(" ")); + }, () => writeDraftId(storageKey, null)).finally(() => { + if (!cancelled) setRestoring(false); + }); + return () => { cancelled = true; }; + }, [client, initialSetup, storageKey]); + + const trimmedScopes = scopes.trim(); + const canCreate = method === "paperclip_key" || (clientId.trim().length > 0 && clientSecret.length > 0); + const canTest = clientId.trim().length > 0; + + async function createDraft(): Promise { + setBusy("creating"); + setError(null); + try { + const created = await client.create(companyId, { + link, + ...(name?.trim() ? { name: name.trim() } : {}), + ...(signInService.trim() ? { authorizationServer: signInService.trim() } : {}), + method, + ...(clientId.trim() ? { clientId: clientId.trim() } : {}), + ...(method === "client_secret" && clientSecret ? { clientSecret } : {}), + ...(trimmedScopes ? { scopes: trimmedScopes } : {}), + }); + setSetup(created); + writeDraftId(storageKey, created.connectionId); + // The secret is saved with the draft; a later test only sends a replacement. + setClientSecret(""); + return created; + } catch (caught) { + setError(describeError(caught).body); + return null; + } finally { + setBusy(null); + } + } + + /** `replacementSecret` is sent only when the admin typed a new one after the draft was saved. */ + async function testConnection(target: MachineSignInDraft, replacementSecret: string) { + setBusy("testing"); + setError(null); + try { + const result = await client.test(target.connectionId, { + clientId: clientId.trim(), + // Always sent, so clearing the field clears the saved scopes. + scopes: trimmedScopes, + ...(target.machineSignIn.method === "client_secret" && replacementSecret ? { clientSecret: replacementSecret } : {}), + }); + writeDraftId(storageKey, null); + onConnected(result); + } catch (caught) { + setError(describeError(caught).body); + } finally { + setBusy(null); + } + } + + const host = (() => { + try { return new URL(link).host; } catch { return link; } + })(); + const publicKey = setup?.machineSignIn.publicKey ?? null; + + return ( +
+

+ For automation your organization owns. Agents use this connection as the provider client you set up for + Paperclip, not as any person. Only shared connections can use machine sign-in. +

+ + {error ? {error} : null} + + {!setup ? ( +
+
{ + const step = event.key === "ArrowRight" || event.key === "ArrowDown" ? 1 + : event.key === "ArrowLeft" || event.key === "ArrowUp" ? -1 : 0; + if (step === 0) return; + event.preventDefault(); + event.stopPropagation(); + const index = METHOD_OPTIONS.findIndex((option) => option.method === method); + const next = (index + step + METHOD_OPTIONS.length) % METHOD_OPTIONS.length; + setMethod(METHOD_OPTIONS[next]!.method); + event.currentTarget.querySelectorAll('[role="radio"]')[next]?.focus(); + }} + > + {METHOD_OPTIONS.map((option) => ( + + ))} +
+

{METHOD_OPTIONS.find((option) => option.method === method)?.hint}

+ + setSignInService(event.target.value)} placeholder="Found automatically" autoComplete="off" className="h-11 font-mono" /> + + + setClientId(event.target.value)} autoComplete="off" className="h-11 font-mono" /> + + {method === "client_secret" ? ( + + setClientSecret(event.target.value)} autoComplete="off" className="h-11 font-mono" /> + + ) : null} + + setScopes(event.target.value)} placeholder="Optional" autoComplete="off" className="h-11 font-mono" /> + +
+ ) : ( +
+ {publicKey ? ( +
+

+ Register this public key with the provider for the Paperclip client. It isn’t secret, so you can paste it + into the provider’s console or send it to whoever manages it. +

+ + +
+ ) : null} + + setClientId(event.target.value)} autoComplete="off" className="h-11 font-mono" /> + + {setup.machineSignIn.method === "client_secret" ? ( + + setClientSecret(event.target.value)} placeholder={setup.machineSignIn.hasClientSecret ? "••••••••" : ""} autoComplete="off" className="h-11 font-mono" /> + + ) : null} + + setScopes(event.target.value)} autoComplete="off" className="h-11 font-mono" /> + +
+ )} + + {beforeFooter} + +
+ + {!setup && method === "paperclip_key" ? ( + + ) : ( + + )} +
+
+ ); +} + +function Field({ id, label, hint, children }: { id: string; label: string; hint?: string; children: React.ReactNode }) { + return ( +
+ +
{children}
+ {hint ?

{hint}

: null} +
+ ); +} + +function PublicKeyBlock({ label, value }: { label: string; value: string }) { + const [copied, setCopied] = useState(false); + return ( +
+
+ {label} + +
+