From 0a96f972e1a7018e4f7989bda47bc4983363a0c5 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Fri, 13 Nov 2026 17:38:55 -0800 Subject: [PATCH 01/11] Connect Tailscale networks through device sign-in (#21526) ## Thinking Path > - Paperclip manages AI agents and the environments where they run. > - SSH environments can reach machines through a company Tailscale connection. > - The initial connector requires a managed OAuth app that Tailscale limits to its own tailnet. > - Customers need to connect their own tailnets without registering an app or supplying a key. > - This pull request uses Tailscale's normal device sign-in for new connections. > - Customers approve the Paperclip device and then select their machines in SSH environments. ## Linked Issues or Issue Description **Problem or motivation** The current Tailscale enrollment path cannot serve independent customer tailnets through the documented self-service OAuth app model. It also requires Cloud broker configuration and pilot admission before a customer can start. **Proposed solution** Start a private Tailscale device in the tenant and open its normal sign-in link. Save its encrypted device state before reporting a successful connection. Retain the existing SSH machine selection and agent account behavior. **Alternatives considered** Auth-key entry would require customers to copy a credential. A public OAuth integration still needs a confirmed registration path. Device sign-in uses a documented Tailscale flow without either requirement. See [device authentication](https://tailscale.com/docs/features/tsnet#device-creation-and-authentication) and [OAuth app limitations](https://tailscale.com/docs/features/oauth-apps). **Related PR search** Searched GitHub PRs for `tailscale` and `device sign-in`, and issues for `tailscale`. This follows the merged network connector in [#20617](https://github.com/paperclipai/paperclip/pull/20617) and preserves the SSH agent-account support in [#21248](https://github.com/paperclipai/paperclip/pull/21248). The open [#15648](https://github.com/paperclipai/paperclip/pull/15648) proposes storing an OAuth client and checking its health; this change instead enrolls a device through interactive sign-in so customers do not need an OAuth app. ## What Changed - New connections use device sign-in. Existing connected devices and pending legacy OAuth callbacks remain supported. - Pending sign-ins are bound to their company and initiating user. They have a fixed ten-minute deadline. Device state uses server-owned vault secrets with trusted version and digest checks. - The active Tailscale CLI waits for consent and device approval. Cancellation, expiry, secret deletion, and stale completion cannot restore a cancelled connection. - The connector stays open during sign-in. It provides a link when popups are blocked and shows device approval separately. Progress checks stop on completion, failure, expiry, or unmount. - Shared response types, OpenAPI contracts, SDK source pins, and the API/setup documentation describe the new flow. No database migration or Cloud provider configuration is required. ## Verification - 109 focused tests cover network processes, the real database and encrypted vault, routes, UI, and legacy connection behavior. - Server and UI typechecks pass. - Follow-up `29cf296a597`: reproduced the missing OpenAPI route failure locally. All 70 OpenAPI and Tailscale route tests and server typecheck pass after synchronizing the API contract. - Follow-up `79de05f38b9`: the transient-recovery regression failed before the fix; all 20 connector UI tests and UI typecheck pass after it. The query-error policy check reports no findings. - The public SDK input registry check passes. - [CI attempt 2](https://github.com/paperclipai/paperclip/actions/runs/43535594257/attempts/2) passed on the final head, `29cf296a597e470d432b84767050bc539caa280e`: 57 successful checks, no failures or pending checks. The interrupted canary, E2E shard, and serialized-suite jobs passed on rerun. Greptile is 5/5 on this head with no unresolved review threads. - `pnpm verify --explain` exits 2 because the documentation change selects broad CI coverage. Focused local checks passed; broad CI passed on `29cf296a597e470d432b84767050bc539caa280e`. - Live local qualification on `29cf296a597e470d432b84767050bc539caa280e`: native device sign-in completed with interactive user approval. An isolated Ubuntu 24.04 ARM64 EC2 machine appeared in discovery and passed SSH pairing and workspace probes. The assigned Claude Code agent used a Paperclip-managed subscription, created `hello_world.txt` containing exactly `Hello, world!` plus a newline, posted a completion comment, and marked the task done. An independent read through the private connector verified the bytes, remote user, and hostname. The remote machine had no stored Claude login and no leftover temporary credential directories. - Restarted the local Paperclip server and verified the same Tailscale device identity, online peer discovery, successful SSH probe, and unchanged file without new consent. Temporary public SSH ingress was removed. - Local qualification covers one user tailnet and the final Core revision. Browser UI acceptance, two unrelated tailnets, a hosted restart, and an agent task on the final staging revision are not yet verified. ## Risks - The enrollment method changes. It must pass the live qualification above before broad rollout claims. - Existing encrypted device state and the vault master key must survive a restart or redeploy. Device expiry or removal still requires reconnecting. - The existing one-serving-process-per-instance constraint remains. Concurrent replicas need durable network ownership. - Pending setup uses bounded progress requests and an event-driven CLI with a deadline. It adds no idle database scan. ## Model Used OpenAI GPT-6 Astra (`gpt-6-astra`) through Codex in T3 Code, with xhigh reasoning. Context window size was not exposed by the runtime. No subagents were used. ## Checklist - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] My branch name describes the change and contains no internal ticket or instance details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/tailscale-environments.md | 108 +++++------- docs/api/tailscale.md | 37 ++-- packages/shared/src/index.ts | 1 + packages/shared/src/types/tailscale.ts | 7 + server/src/__tests__/openapi-routes.test.ts | 19 ++- .../tailscale-connection-routes.test.ts | 43 +++-- server/src/routes/openapi.ts | 32 +++- server/src/routes/tailscale-connections.ts | 36 ++-- .../tailscale/device-login-network.test.ts | 108 ++++++++++++ .../tailscale/device-login-status.ts | 14 ++ .../connectors/tailscale/device-login.test.ts | 158 ++++++++++++++++++ .../connectors/tailscale/device-login.ts | 121 ++++++++++++++ .../services/connectors/tailscale/network.ts | 101 +++++++++-- .../runner-e2e/source-sdk-public-inputs.json | 3 +- ui/src/api/tailscale.ts | 7 +- ui/src/pages/apps/TailscaleConnector.test.tsx | 98 ++++++++--- ui/src/pages/apps/TailscaleConnector.tsx | 43 +++-- ui/src/pages/apps/TailscaleDeviceSignIn.tsx | 43 +++++ 18 files changed, 796 insertions(+), 183 deletions(-) create mode 100644 packages/shared/src/types/tailscale.ts create mode 100644 server/src/services/connectors/tailscale/device-login-network.test.ts create mode 100644 server/src/services/connectors/tailscale/device-login-status.ts create mode 100644 server/src/services/connectors/tailscale/device-login.test.ts create mode 100644 server/src/services/connectors/tailscale/device-login.ts create mode 100644 ui/src/pages/apps/TailscaleDeviceSignIn.tsx diff --git a/doc/tailscale-environments.md b/doc/tailscale-environments.md index 733f4cab51d..3f40328ce48 100644 --- a/doc/tailscale-environments.md +++ b/doc/tailscale-environments.md @@ -16,11 +16,12 @@ with the normal Tailscale application, enable macOS Remote Login. Tailscale SSH server support on macOS requires the open-source tailscaled variant; ordinary OpenSSH over Tailscale works with the standard Mac application. -Cloud reauthentication saves the pending OAuth handoff in the current browser -tab before redirecting. If consent is canceled or completion fails after the -state and company checks, the browser returns to the saved connection with a -fixed explanation and a retry. The return URL contains only the connection ID -and a known outcome, without provider error text, claims, or OAuth state. +Device sign-in opens Tailscale in a separate tab and keeps the connector page +open. A normal sign-in link remains available if the browser blocks the tab. +Approve the Paperclip device in your own tailnet; no OAuth app, API key, Cloud +broker credential, or per-instance pilot admission is needed. If your tailnet +requires device approval, the connector shows that separate pending state. +Sign-in expires after ten minutes and can be cancelled from the connector. SSH setup hides the Network selector when Direct connection is the only available option. A sole connected tailnet is selected automatically and its @@ -66,30 +67,35 @@ resolution looks up its current address through that connection’s private socket. Offline, removed, disconnected, and wrong-company machines fail closed. The host’s existing Tailscale connection is never a fallback. -## OAuth and credential custody +## Device sign-in and credential custody -The connection uses the Cloud OAuth broker profile `tailscale.tailnet`, with -`auth_keys:create:once`. It is a network connector on the Connectors surface; -it does not publish agent tools or enter MCP discovery. Management requires an -instance administrator with access to the company. The catalog and setup page -check that access before requesting network data; other company members are -directed to an instance administrator. Existing SSH environment access governs -which agents can execute there. Self-hosted local -instances first complete the existing Paperclip Cloud enrollment and return to -their company's Tailscale connector page to continue network consent. Cloud -stacks use their existing broker identity and handoff. +It is a network connector on the Connectors surface; it does not publish agent +tools or enter MCP discovery. Management requires an instance administrator with +access to the company. Only the person who started a pending sign-in can retrieve +its sign-in link. Existing SSH environment access governs which agents execute +there. Cloud and self-hosted instances use the same Tailscale device sign-in. -Tailscale returns a single-use device enrollment token without a refresh token. -The broker seals it to the originating instance. The instance validates the -user, company, state expiry and single-use callback, stores the token through the -configured company secret provider, and immediately replaces its current value -with the enrolled node state before permitting execution. The connection records +The instance starts a private userspace Tailscale device and returns its generated +`https://login.tailscale.com/a/...` sign-in address. Pending device state is stored +in the configured company vault before the address is returned. The active +`tailscale up` command watches Tailscale state until consent and any required +device approval complete, or the ten-minute deadline expires. It saves the +registered device state before marking the connection connected. Pending sign-ins +can resume from the saved state after a server restart without extending their +deadline. There is no background database scan: browser progress requests exist +only during the bounded sign-in, and the CLI waits for Tailscale state events. + +The previous OAuth callback remains available for already-started pilot flows; +existing enrolled devices continue to use their saved state. New connections +use device sign-in and do not call the Cloud OAuth broker. + +The connection records the exact trusted vault version and a SHA-256 digest of the server-written value. Runtime reads that version and verifies its digest. A generic company secret rotation cannot change the authorized network. Trusted node-state saves advance both fields together, with a compare-and-set guard so an older save cannot replace a newer state. Reads keep using the previous trusted version until the -new version is committed. Tokens, private node state, digests and client secrets +new version is committed. Private node state, digests and enrollment credentials never appear in connection or environment responses. Enrollment tokens and node state are server-owned secrets with a durable @@ -150,7 +156,6 @@ all Tailscale environments use the connector's lifecycle. This implementation currently supports one serving process per instance. Concurrent replicas need a durable connection ownership lease before rollout. -The enrollment token is consumed once; it is not the runtime credential. Tailscale identity belongs to the persisted machine key in the saved node state, not the container ID or hostname. A redeploy must retain both the database's encrypted node state and the same vault master key. Cloud already materializes @@ -161,31 +166,10 @@ fresh consent. The app-owned device has the consenting user's tailnet rights. ## Local development -Run the Paperclip instance at `http://localhost:3100` and the local Cloud broker -at `http://localhost:3000`. Register the OAuth app callback as -`http://localhost:3000/v1/connector/oauth/tailscale/callback` with scope -`auth_keys:create:once`. The OAuth app must be registered in the authorizing -user’s tailnet under Tailscale’s currently documented restriction. - -Configure the local broker with: - -- `CLOUD_HARNESS_CONNECTOR_TAILSCALE_ENABLED=1` -- `CLOUD_HARNESS_CONNECTOR_TAILSCALE_CLIENT_ID` -- `CLOUD_HARNESS_CONNECTOR_TAILSCALE_CLIENT_SECRET_REF=secret://local/tailscale` -- `CLOUD_HARNESS_CONNECTOR_TAILSCALE_LOCAL_CLIENT_SECRET_FILE`, an owner-only - regular file outside the checkout containing the OAuth app client secret -- `CLOUD_HARNESS_CONNECTOR_TAILSCALE_PILOT_INSTANCE_IDS`, the exact enrolled local - instance ID - -The file credential path is accepted only for a loopback HTTP development broker -using the local fake provider. Hosted credentials continue to use the configured -provider secret store. Never regenerate an existing Cloud `.env` or identity -salt to configure this test. - -The instance needs `tailscale` and `tailscaled`. Its OAuth return path is -`/api/tools/oauth/tailscale/callback`, and completion returns to the Tailscale -connector page. For a separate local userspace -client, install the pinned programs outside the checkout: +Run the Paperclip instance at `http://localhost:3100`. No local Cloud broker or +Tailscale OAuth app is required. The instance needs `tailscale` and `tailscaled`. +For a separate local userspace client, install the pinned programs outside the +checkout: ```sh GOBIN="$HOME/.local/share/paperclip-tailscale/bin" go install tailscale.com/cmd/tailscale@v1.104.5 @@ -195,19 +179,18 @@ GOBIN="$HOME/.local/share/paperclip-tailscale/bin" go install tailscale.com/cmd/ Set `PAPERCLIP_TAILSCALE_CLI` and `PAPERCLIP_TAILSCALED` to their absolute paths. These settings can be set in the supported `.env` files. Each new client reads them at startup, after environment loading. -Set `PAPERCLIP_CLOUD_CONNECTOR_BASE_URL=http://localhost:3000` and -`PAPERCLIP_CLOUD_CONNECTOR_ENVIRONMENT=development` for local enrollment. ## Verification and rollout gates -Offline tests cover broker handoff and sealed claims, company and user binding, -expired and competing callbacks, disconnect during claim exchange, secret -creation, credential reload and daemon startup, superseding authorizations, -failed-enrollment cleanup, denied authorization, device discovery, driver -selection, private subprocess state, shutdown, and restart. These do not prove -live consent or remote execution. +Offline tests cover device sign-in without an auth key, saved state and restart, +separate company identities, initiator binding, cancellation and expiry, stale +completion, trusted credential versions, device approval, popup fallback, and +route authorization/CSRF. Existing tests retain legacy broker callback, SSH, +pairing, and network lifecycle coverage. These do not prove live consent or +remote execution. -Before rollout, qualify real consent, machine discovery, a Claude task that +Before rollout, qualify two unrelated customer tailnets through ordinary device +sign-in, machine discovery, and an agent task using its selected AI account that writes a file on the selected computer, result streaming, cancellation, server restart, machine sleep/reboot, and revocation. A local instance whose public URL is localhost also needs an API path reachable by the remote agent for tasks that @@ -218,14 +201,15 @@ The connector's bundled artwork comes from the vendor's `https://tailscale.com/favicon.png`, copied without modification (SHA-256 `680570c9090b604450792f68ceed10beeb33a69c9bccc2a92e398564c7075a5c`). -The GA announcement describes third-party integrations, but the published OAuth -setup guide still restricts each app to its own tailnet. This profile remains -behind explicit instance pilot admission. A local personal-tailnet test does not -establish that one Paperclip-owned OAuth app can authorize every customer’s -personal tailnet. Verify that provider registration path before general rollout. +The old managed OAuth profile remains disabled unless explicitly configured for +its existing pilot. Its same-tailnet registration restriction does not apply to +the new device sign-in path. Do not configure a customer-specific OAuth app or +admit a pilot instance to make device sign-in work. Provider references: +- [Device authentication](https://tailscale.com/docs/features/tsnet#device-creation-and-authentication) +- [Tailscale CLI](https://tailscale.com/docs/reference/tailscale-cli/up) - [OAuth apps](https://tailscale.com/docs/features/oauth-apps) - [Device provisioning](https://tailscale.com/docs/features/oauth-apps/device-provisioning) - [Tailscale SSH](https://tailscale.com/docs/features/tailscale-ssh) diff --git a/docs/api/tailscale.md b/docs/api/tailscale.md index 8bd2a4c79f2..867606c1690 100644 --- a/docs/api/tailscale.md +++ b/docs/api/tailscale.md @@ -10,22 +10,37 @@ cannot manage a Tailscale connection. Browser mutations require a trusted Origin | Method | Path | Result | |--------|------|--------| | GET | `/api/companies/{companyId}/tailscale-connections` | Connection metadata | -| POST | `/api/companies/{companyId}/tailscale-connections` | Authorization URL | +| POST | `/api/companies/{companyId}/tailscale-connections` | Start device sign-in | +| POST | `/api/companies/{companyId}/tailscale-connections/{connectionId}/device-login` | Resume device sign-in and read progress | | GET | `/api/companies/{companyId}/tailscale-connections/{connectionId}/devices` | Available machines | | POST | `/api/companies/{companyId}/tailscale-connections/{connectionId}/pairing` | Public SSH pairing command | | DELETE | `/api/companies/{companyId}/tailscale-connections/{connectionId}` | Disconnected connection metadata | A connection response contains only `id`, `companyId`, `name`, and `status`. -Status is `pending`, `connecting`, `connected`, or `disconnected`. Responses do -not include enrollment tokens, persisted node state, or OAuth state digests. +Status is `pending`, `authorizing`, `connecting`, `connected`, `disconnected`, +`expired`, or `error`. Responses do not include enrollment tokens, persisted +node state, or OAuth state digests. -To start consent, POST a JSON object with optional `name` and `connectionId`. -The name defaults to `My tailnet`. An existing connection ID starts a new -authorization for that connection. The `201` response contains `authorizationUrl` -and `expiresAt`. It also contains `connection` when consent can start immediately. -An instance without Cloud enrollment first returns the enrollment URL. Hosted -consent can include a `handoff` with `kind: "paperclip_cloud"` and an opaque -`session`. Use the normal browser connector flow for that signed handoff. +To start device sign-in, POST a JSON object with optional `name` and +`connectionId`. The name defaults to `My tailnet`. No OAuth app, Cloud broker +enrollment, or pasted auth key is required. An existing connection ID resumes an +unexpired sign-in or starts a new attempt. Disconnect a connected network before +replacing it. Pending sign-in is bound to the initiating user and expires after +ten minutes; progress requests do not extend that deadline. + +The `201` response contains `connection` and `expiresAt`, plus `authorizationUrl` +when the sign-in link is ready. Open that link so the user can approve the +Paperclip device in their tailnet. `needsApproval: true` means a tailnet +administrator must approve the device. `expiresAt` is null when no sign-in is +pending. No Cloud handoff is returned. + +POST to `/{connectionId}/device-login` under the connections path to resume the +private client and read progress. This endpoint takes no request body and +returns the same shape with status `200`. Only the user who started the sign-in +may continue it, with the same instance-admin and company access checks. Both +responses use `Cache-Control: no-store` and `Referrer-Policy: no-referrer`. +Stop progress requests when the connection leaves `authorizing` or the deadline +passes. DELETE the connection to cancel a pending attempt. Machine discovery returns `id`, `name`, `hostname`, `address`, `os`, and `online` for each peer. Use the stable `id` when configuring a Tailscale-backed SSH @@ -43,7 +58,7 @@ and `command`. Each preparation generates a fresh key pair. Run the command as the target SSH user and configure the environment with the returned secret ID. The private key never appears in this response. -The browser returns from consent to +Pending legacy OAuth flows can still return to `GET /api/tools/oauth/tailscale/callback` with required `state` and either `claim_id` or an authorization `error`. The callback rechecks the initiating operator's company access and consumes state once. On success it returns a diff --git a/packages/shared/src/index.ts b/packages/shared/src/index.ts index 17f32ce6e70..2d2b5ba044c 100644 --- a/packages/shared/src/index.ts +++ b/packages/shared/src/index.ts @@ -2967,3 +2967,4 @@ export { export { recognizeLinkPreview, type LinkPreviewTarget, type LinkPreviewState, type LinkPreview, type LinkPreviewResult } from "./link-preview.js"; export * from "./mcp-apps.js"; export * from "./mcp-skills.js"; +export type { TailscaleConnection, TailscaleDeviceLogin } from "./types/tailscale.js"; diff --git a/packages/shared/src/types/tailscale.ts b/packages/shared/src/types/tailscale.ts new file mode 100644 index 00000000000..d02a917945d --- /dev/null +++ b/packages/shared/src/types/tailscale.ts @@ -0,0 +1,7 @@ +export type TailscaleConnection = { id: string; companyId: string; name: string; status: string }; +export type TailscaleDeviceLogin = { + connection: TailscaleConnection; + expiresAt: string | null; + authorizationUrl?: string; + needsApproval?: boolean; +}; diff --git a/server/src/__tests__/openapi-routes.test.ts b/server/src/__tests__/openapi-routes.test.ts index 1241419c306..d4b6e5b8e77 100644 --- a/server/src/__tests__/openapi-routes.test.ts +++ b/server/src/__tests__/openapi-routes.test.ts @@ -1255,6 +1255,7 @@ describe("openapi routes", () => { const { spec } = loadSpecRoutes(); const base = "/api/companies/{companyId}/tailscale-connections"; for (const [path, method] of [[base, "get"], [base, "post"], [base + "/{connectionId}", "delete"], + [base + "/{connectionId}/device-login", "post"], [base + "/{connectionId}/devices", "get"], [base + "/{connectionId}/pairing", "post"], ["/api/tools/oauth/tailscale/callback", "get"]]) { const operation = spec.paths[path]?.[method]; expect(operation).toBeDefined(); @@ -1265,13 +1266,27 @@ describe("openapi routes", () => { const connection = spec.paths[base].get.responses["200"].content["application/json"].schema.items; expect(Object.keys(connection.properties).sort()).toEqual(["companyId", "id", "name", "status"]); expect(connection.additionalProperties).toBe(false); + expect(connection.properties.status.enum).toEqual(["pending", "authorizing", "connecting", "connected", "disconnected", "expired", "error"]); const start = spec.paths[base].post; expect(start.requestBody.content["application/json"].schema).toMatchObject({ additionalProperties: false, properties: { name: { minLength: 1, maxLength: 100 }, connectionId: { format: "uuid" } } }); expect(start.requestBody.content["application/json"].schema.required ?? []).toEqual([]); const session = start.responses["201"].content["application/json"].schema; - expect(session.required).toEqual(["authorizationUrl", "expiresAt"]); - expect(session.properties.handoff.properties.kind).toEqual({ type: "string", enum: ["paperclip_cloud"] }); + expect(session.required).toEqual(["connection", "expiresAt"]); + expect(Object.keys(session.properties).sort()).toEqual(["authorizationUrl", "connection", "expiresAt", "needsApproval"]); + expect(session.properties.expiresAt).toMatchObject({ type: "string", format: "date-time", nullable: true }); + expect(session.properties.authorizationUrl).toMatchObject({ type: "string", format: "uri" }); + expect(session.properties.needsApproval).toEqual({ type: "boolean" }); + expect(session.additionalProperties).toBe(false); + const progress = spec.paths[base + "/{connectionId}/device-login"].post; + expect(progress.responses["200"].content["application/json"].schema).toEqual(session); + expect(progress.requestBody).toBeUndefined(); + for (const response of [start.responses["201"], progress.responses["200"]]) { + expect(response.headers).toMatchObject({ + "Cache-Control": { schema: { enum: ["no-store"] } }, + "Referrer-Policy": { schema: { enum: ["no-referrer"] } }, + }); + } const pairing = spec.paths[base + "/{connectionId}/pairing"].post; expect(pairing.requestBody.content["application/json"].schema).toMatchObject({ additionalProperties: false, required: ["deviceId"], properties: { deviceId: { minLength: 1, maxLength: 256 }, port: { minimum: 1, maximum: 65535 } } }); diff --git a/server/src/__tests__/tailscale-connection-routes.test.ts b/server/src/__tests__/tailscale-connection-routes.test.ts index 9da255c2645..89cd3c90c07 100644 --- a/server/src/__tests__/tailscale-connection-routes.test.ts +++ b/server/src/__tests__/tailscale-connection-routes.test.ts @@ -7,10 +7,11 @@ import { errorHandler } from "../middleware/error-handler.js"; import { badRequest } from "../errors.js"; const fixture = vi.hoisted(() => ({ - service: { list: vi.fn(), start: vi.fn(), devices: vi.fn(), complete: vi.fn(), disconnect: vi.fn(), pairing: vi.fn() }, + service: { list: vi.fn(), start: vi.fn(), devices: vi.fn(), complete: vi.fn(), disconnect: vi.fn(), pairing: vi.fn(), resume: vi.fn() }, configured: vi.fn(), enrollment: vi.fn(), activity: vi.fn(), })); vi.mock("../services/connectors/tailscale/connections.js", () => ({ tailscaleConnectionService: () => fixture.service })); +vi.mock("../services/connectors/tailscale/device-login.js", () => ({ tailscaleDeviceLoginService: () => fixture.service })); vi.mock("../services/paperclip-cloud-connector.js", () => ({ paperclipCloudConnectorConfigFromEnv: fixture.configured })); vi.mock("../services/paperclip-cloud-connector-enrollment.js", () => ({ startPaperclipCloudConnectorEnrollment: fixture.enrollment })); vi.mock("../services/activity-log.js", () => ({ logActivity: fixture.activity })); @@ -41,10 +42,10 @@ describe("Tailscale connector routes", () => { fixture.service.disconnect.mockResolvedValue({ id: connectionId, companyId, status: "disconnected" }); fixture.service.pairing.mockResolvedValue({ privateKeySecretId: "fixture-key-ref", publicKey: "fixture-public-key", command: "fixture-public-pairing-command" }); }); - it("starts consent with the signed-in user, company, and fixed connector callback", async () => { + it("starts device sign-in with the signed-in user and company", async () => { const response = await request(app()).post(path).set("Origin", "http://localhost:3100").send({ name: "Home" }); expect(response.status).toBe(201); - expect(fixture.service.start).toHaveBeenCalledWith({ companyId, userId: "fixture-user", name: "Home", returnUri: "http://localhost:3100/api/tools/oauth/tailscale/callback" }); + expect(fixture.service.start).toHaveBeenCalledWith({ companyId, userId: "fixture-user", name: "Home" }); expect(fixture.activity).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({ action: "tailscale.authorization_started", companyId })); }); it("rejects foreign-origin browser mutations before starting consent", async () => { @@ -113,26 +114,15 @@ describe("Tailscale connector routes", () => { expect(response.headers.location).toBeUndefined(); expect(fixture.activity).not.toHaveBeenCalled(); }); - it("starts existing instance enrollment when no broker identity exists", async () => { + it("starts device sign-in without a Cloud broker identity or pilot admission", async () => { fixture.configured.mockReturnValue(null); - fixture.enrollment.mockResolvedValue({ instanceId: "fixture-instance", environment: "development", status: "pending", verificationUrl: "http://localhost:3000/connections/enroll?id=fixture", expiresAt: "2099-01-01T00:00:00Z" }); const response = await request(app()).post(path).set("Origin", "http://localhost:3100").send({ name: "Home" }); expect(response.status).toBe(201); - expect(response.body.authorizationUrl).toContain("/connections/enroll"); - expect(fixture.enrollment).toHaveBeenCalledWith(expect.objectContaining({ companyId, initiatedBy: "user:fixture-user", origin: "http://localhost:3100", returnTo: "/apps/tailscale" })); - expect(fixture.service.start).not.toHaveBeenCalled(); - expect(fixture.activity).toHaveBeenCalledExactlyOnceWith(expect.anything(), { - companyId, - actorType: "user", - actorId: "fixture-user", - action: "paperclip_cloud_connector.enrollment_started", - entityType: "connector_instance", - entityId: "fixture-instance", - details: { environment: "development", status: "pending" }, - }); + expect(fixture.enrollment).not.toHaveBeenCalled(); + expect(fixture.service.start).toHaveBeenCalledWith({ companyId, userId: "fixture-user", name: "Home" }); + expect(response.headers["cache-control"]).toBe("no-store"); }); - it("checks origin, admin status, and company access before self-hosted enrollment", async () => { - fixture.configured.mockReturnValue(null); + it("checks origin, admin status, and company access before device sign-in and status requests", async () => { for (const [actor, origin] of [ [board, "https://untrusted.example.test"], [{ ...board, isInstanceAdmin: false }, "http://localhost:3100"], @@ -140,18 +130,25 @@ describe("Tailscale connector routes", () => { [{ type: "agent", source: "agent_key", agentId: "fixture-agent", companyId }, "http://localhost:3100"], ] as const) { expect((await request(app(actor as any)).post(path).set("Origin", origin).send({ name: "Home" })).status).toBe(403); + expect((await request(app(actor as any)).post(path + `/${connectionId}/device-login`).set("Origin", origin).send({})).status).toBe(403); } - expect(fixture.enrollment).not.toHaveBeenCalled(); expect(fixture.service.start).not.toHaveBeenCalled(); + expect(fixture.service.resume).not.toHaveBeenCalled(); expect(fixture.activity).not.toHaveBeenCalled(); }); + it("returns device sign-in progress only to the authorized initiator", async () => { + fixture.service.resume.mockResolvedValue({ connection: { id: connectionId, status: "authorizing" }, needsApproval: true }); + const response = await request(app()).post(path + `/${connectionId}/device-login`).set("Origin", "http://localhost:3100").send({}); + expect(response.status).toBe(200); + expect(fixture.service.resume).toHaveBeenCalledWith(companyId, connectionId, "fixture-user"); + expect(response.body.needsApproval).toBe(true); + expect(response.headers["cache-control"]).toBe("no-store"); + }); it("does not record a successful enrollment start when enrollment fails", async () => { - fixture.configured.mockReturnValue(null); - fixture.enrollment.mockRejectedValue(new Error("Enrollment unavailable")); + fixture.service.start.mockRejectedValueOnce(new Error("Enrollment unavailable")); const response = await request(app()).post(path).set("Origin", "http://localhost:3100").send({ name: "Home" }); expect(response.status).toBe(500); expect(fixture.activity).not.toHaveBeenCalled(); - expect(fixture.service.start).not.toHaveBeenCalled(); }); it("prepares machine pairing for an authorized board user without returning a private key", async () => { const response = await request(app()).post(path + `/${connectionId}/pairing`).set("Origin", "http://localhost:3100").send({ deviceId: "stable-device" }); diff --git a/server/src/routes/openapi.ts b/server/src/routes/openapi.ts index 5164179d3fc..69c73db0b19 100644 --- a/server/src/routes/openapi.ts +++ b/server/src/routes/openapi.ts @@ -1630,6 +1630,7 @@ const BOARD_ONLY_OPERATIONS = new Set([ const INSTANCE_ADMIN_OPERATIONS = new Set([ "GET /api/companies/{companyId}/tailscale-connections", "POST /api/companies/{companyId}/tailscale-connections", + "POST /api/companies/{companyId}/tailscale-connections/{connectionId}/device-login", "GET /api/companies/{companyId}/tailscale-connections/{connectionId}/devices", "DELETE /api/companies/{companyId}/tailscale-connections/{connectionId}", "POST /api/companies/{companyId}/tailscale-connections/{connectionId}/pairing", @@ -9607,8 +9608,20 @@ const tailscaleCompanyParams = z.object({ companyId: z.string().uuid() }); const tailscaleConnectionParams = tailscaleCompanyParams.extend({ connectionId: z.string().uuid() }); const tailscaleConnectionResponse = z.object({ id: z.string().uuid(), companyId: z.string().uuid(), name: z.string(), - status: z.enum(["pending", "connecting", "connected", "disconnected"]), + status: z.enum(["pending", "authorizing", "connecting", "connected", "disconnected", "expired", "error"]), }).strict(); +const tailscaleDeviceLoginResponse = { + ...r.ok(z.object({ + connection: tailscaleConnectionResponse, + expiresAt: z.string().datetime().nullable(), + authorizationUrl: z.string().url().optional(), + needsApproval: z.boolean().optional(), + }).strict()), + headers: { + "Cache-Control": { schema: { type: "string", enum: ["no-store"] } }, + "Referrer-Policy": { schema: { type: "string", enum: ["no-referrer"] } }, + }, +}; const tailscaleErrors = { 400: r.badRequest, 401: r.unauthorized, 403: r.forbidden, 404: r.notFound, 422: r.unprocessable }; registry.registerPath({ @@ -9621,15 +9634,20 @@ registry.registerPath({ registry.registerPath({ method: "post", path: "/api/companies/{companyId}/tailscale-connections", tags: ["tailscale"], - summary: "Start Tailscale network authorization", - description: "Requires an instance administrator with company access, or local implicit board access. Starts consent for a new connection or replaces consent for connectionId. An instance without a Cloud broker identity first returns an enrollment URL without a connection. Hosted authorization can return a signed Cloud handoff.", + summary: "Start Tailscale device sign-in", + description: "Requires an instance administrator with company access, or local implicit board access. Starts device sign-in without a Cloud broker or OAuth app. Reuses an unexpired sign-in for connectionId; a connected network must be disconnected before replacement. Pending sign-in is bound to its initiating user and has a fixed ten-minute deadline. The authorization URL is present when sign-in is ready; needsApproval identifies a device waiting for tailnet administrator approval.", request: { params: tailscaleCompanyParams, body: jsonBody(z.object({ name: z.string().trim().min(1).max(100).default("My tailnet"), connectionId: z.string().uuid().optional(), }).strict()) }, - responses: { 201: r.ok(z.object({ - connection: tailscaleConnectionResponse.optional(), authorizationUrl: z.string().url(), expiresAt: z.string().datetime(), - handoff: z.object({ kind: z.literal("paperclip_cloud"), session: z.string().min(16).max(512).regex(/^[A-Za-z0-9_-]+$/) }).strict().optional(), - }).strict()), ...tailscaleErrors }, + responses: { 201: tailscaleDeviceLoginResponse, ...tailscaleErrors }, +}); + +registry.registerPath({ + method: "post", path: "/api/companies/{companyId}/tailscale-connections/{connectionId}/device-login", tags: ["tailscale"], + summary: "Resume Tailscale device sign-in and read its progress", + description: "Requires the initiating user to retain instance-admin and company access, or local implicit board access. Resumes the private client for an existing sign-in without extending its deadline. Returns connection metadata, an optional sign-in URL or device approval flag, and expiresAt (null when no sign-in is pending). Does not return credentials or node state.", + request: { params: tailscaleConnectionParams }, + responses: { 200: tailscaleDeviceLoginResponse, ...tailscaleErrors }, }); registry.registerPath({ diff --git a/server/src/routes/tailscale-connections.ts b/server/src/routes/tailscale-connections.ts index a50ba7a5c01..8b8c33ca178 100644 --- a/server/src/routes/tailscale-connections.ts +++ b/server/src/routes/tailscale-connections.ts @@ -2,13 +2,10 @@ import { Router, type Request } from "express"; import { eq } from "drizzle-orm"; import { companies, type Db } from "@paperclipai/db"; import { z } from "zod"; -import { loadConfig } from "../config.js"; -import { forbidden, unprocessable } from "../errors.js"; +import { forbidden } from "../errors.js"; import { validate } from "../middleware/validate.js"; import { tailscaleConnectionService } from "../services/connectors/tailscale/connections.js"; -import { paperclipCloudConnectorConfigFromEnv } from "../services/paperclip-cloud-connector.js"; -import { startPaperclipCloudConnectorEnrollment } from "../services/paperclip-cloud-connector-enrollment.js"; -import { paperclipCloudEnrollmentPrincipal } from "../services/paperclip-cloud-connector-principal.js"; +import { tailscaleDeviceLoginService } from "../services/connectors/tailscale/device-login.js"; import { logActivity } from "../services/activity-log.js"; import { assertBoard, assertCompanyAccess, getActorInfo } from "./authz.js"; @@ -21,6 +18,7 @@ function operator(req: Request) { export function tailscaleConnectionRoutes(db: Db) { const router = Router(); const service = tailscaleConnectionService(db); + const deviceLogin = tailscaleDeviceLoginService(db); router.get("/companies/:companyId/tailscale-connections", async (req, res) => { operator(req); assertCompanyAccess(req, z.string().uuid().parse(req.params.companyId)); @@ -30,25 +28,21 @@ export function tailscaleConnectionRoutes(db: Db) { const userId = operator(req); const companyId = z.string().uuid().parse(req.params.companyId); assertCompanyAccess(req, companyId); - const origin = loadConfig().authPublicBaseUrl; - if (!origin) throw unprocessable("Set this instance's public URL before connecting Tailscale."); - if (!paperclipCloudConnectorConfigFromEnv()) { - const enrollment = await startPaperclipCloudConnectorEnrollment({ origin, companyId, initiatedBy: paperclipCloudEnrollmentPrincipal(req.actor), - returnTo: "/apps/tailscale", - }); - await logActivity(db, { - companyId, actorType: "user", actorId: userId, - action: "paperclip_cloud_connector.enrollment_started", - entityType: "connector_instance", entityId: enrollment.instanceId ?? "pending", - details: { environment: enrollment.environment, status: enrollment.status }, - }); - res.status(201).json({ authorizationUrl: enrollment.verificationUrl, expiresAt: enrollment.expiresAt }); - return; - } - const result = await service.start({ companyId, userId, ...req.body, returnUri: new URL("/api/tools/oauth/tailscale/callback", origin).toString() }); + const result = await deviceLogin.start({ companyId, userId, ...req.body }); await logActivity(db, { companyId, actorType: "user", actorId: userId, action: "tailscale.authorization_started", entityType: "tailscale_connection", entityId: result.connection.id }); + res.setHeader("Cache-Control", "no-store"); + res.setHeader("Referrer-Policy", "no-referrer"); res.status(201).json(result); }); + router.post("/companies/:companyId/tailscale-connections/:connectionId/device-login", async (req, res) => { + const userId = operator(req); + const companyId = z.string().uuid().parse(req.params.companyId); + assertCompanyAccess(req, companyId); + const result = await deviceLogin.resume(companyId, z.string().uuid().parse(req.params.connectionId), userId); + res.setHeader("Cache-Control", "no-store"); + res.setHeader("Referrer-Policy", "no-referrer"); + res.json(result); + }); router.get("/companies/:companyId/tailscale-connections/:connectionId/devices", async (req, res) => { operator(req); assertCompanyAccess(req, z.string().uuid().parse(req.params.companyId)); diff --git a/server/src/services/connectors/tailscale/device-login-network.test.ts b/server/src/services/connectors/tailscale/device-login-network.test.ts new file mode 100644 index 00000000000..59bc0739ee3 --- /dev/null +++ b/server/src/services/connectors/tailscale/device-login-network.test.ts @@ -0,0 +1,108 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { createTailscaleNetworkManager } from "./network.js"; +import { parseTailscaleLoginStatus } from "./device-login-status.js"; + +describe("Tailscale interactive device enrollment", () => { + let dir: string; + let manager: ReturnType; + beforeEach(async () => { + dir = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-ts-login-test-")); + await fs.writeFile(path.join(dir, "daemon"), `#!/usr/bin/env node +const fs = require('node:fs'); +const path = require('node:path'); +const dir = path.dirname(process.argv.find(arg => arg.startsWith('--socket=')).slice(9)); +fs.writeFileSync(path.join(dir, 'pid'), String(process.pid)); +process.on('SIGTERM', () => process.exit(0)); +setInterval(() => {}, 1000); +`, { mode: 0o700 }); + await fs.writeFile(path.join(dir, "cli"), `#!/usr/bin/env node +const fs = require('node:fs'); +const path = require('node:path'); +const args = process.argv.slice(2); +const dir = path.dirname(args[0].slice(9)); +const statePath = path.join(dir, 'state'); +if (!fs.existsSync(path.join(dir, 'pid'))) process.exit(4); +let state = JSON.parse(fs.readFileSync(statePath, 'utf8')); +if (args[1] === 'up') { + if (args.some(arg => arg.startsWith('--auth-key'))) process.exit(5); + if (state.BackendState === 'Running') process.exit(0); + state = { MachineKey: state.MachineKey || path.basename(dir), BackendState: 'NeedsLogin', AuthURL: 'https://login.tailscale.com/a/fixture' }; + fs.writeFileSync(statePath, JSON.stringify(state)); + fs.watch(dir, () => { + if (JSON.parse(fs.readFileSync(statePath, 'utf8')).BackendState === 'Running') process.exit(0); + }); +} else if (args[1] === 'status') console.log(JSON.stringify(state)); +`, { mode: 0o700 }); + manager = createTailscaleNetworkManager({ daemon: path.join(dir, "daemon"), cli: path.join(dir, "cli") }); + }); + afterEach(async () => { await manager.close(); await fs.rm(dir, { recursive: true, force: true }); }); + + const emptyState = "paperclip-tailscale-state-v1:" + Buffer.from("{}").toString("base64"); + it("returns a login URL without a key, persists consent, and restores the same device", async () => { + let credential = emptyState; + const authorized = vi.fn().mockResolvedValue(undefined); + const input = { identity: "company:secret", credential, persist: async (value: string) => { credential = value; }, + login: { expiresAt: Date.now() + 60_000, onAuthorized: authorized, onFailure: vi.fn() } }; + const socket = await manager.connect(input); + expect(await manager.loginStatus(input.identity)).toEqual({ status: "authorizing", authorizationUrl: "https://login.tailscale.com/a/fixture" }); + const statePath = path.join(path.dirname(socket), "state"); + const state = JSON.parse(await fs.readFile(statePath, "utf8")); + expect(credential).not.toBe(emptyState); + expect(authorized).not.toHaveBeenCalled(); + await fs.writeFile(statePath, JSON.stringify({ ...state, BackendState: "Running", AuthURL: "" })); + await vi.waitFor(() => expect(authorized).toHaveBeenCalledOnce()); + await manager.close(); + const restored = await manager.connect({ identity: input.identity, credential, persist: input.persist }); + expect(JSON.parse(await fs.readFile(path.join(path.dirname(restored), "state"), "utf8")).MachineKey).toBe(state.MachineKey); + }); + + it("reports device approval separately and cancels pending enrollment", async () => { + const authorized = vi.fn(); + const input = { identity: "awaiting-approval", credential: emptyState, persist: vi.fn(), + login: { expiresAt: Date.now() + 60_000, onAuthorized: authorized, onFailure: vi.fn() } }; + const socket = await manager.connect(input); + const statePath = path.join(path.dirname(socket), "state"); + await fs.writeFile(statePath, JSON.stringify({ BackendState: "NeedsMachineAuth" })); + expect(await manager.loginStatus(input.identity)).toEqual({ status: "needs_approval" }); + await manager.disconnect(input.identity); + expect(await manager.loginStatus(input.identity)).toBeNull(); + expect(authorized).not.toHaveBeenCalled(); + expect(input.login.onFailure).not.toHaveBeenCalled(); + await expect(fs.stat(path.dirname(socket))).rejects.toMatchObject({ code: "ENOENT" }); + }); + + it("stops an expired sign-in without accepting late consent", async () => { + const authorized = vi.fn(); + const failure = vi.fn().mockResolvedValue(undefined); + const socket = await manager.connect({ identity: "expires", credential: emptyState, persist: vi.fn(), + login: { expiresAt: Date.now() + 1200, onAuthorized: authorized, onFailure: failure } }); + await vi.waitFor(() => expect(failure).toHaveBeenCalledOnce(), { timeout: 4000 }); + await vi.waitFor(async () => expect(await manager.loginStatus("expires")).toBeNull()); + expect(authorized).not.toHaveBeenCalled(); + await expect(fs.stat(path.dirname(socket))).rejects.toMatchObject({ code: "ENOENT" }); + }); + + it("lets secret deletion cancel a blocked credential save", async () => { + let saveEntered!: () => void, releaseSave!: () => void; + const entered = new Promise(resolve => { saveEntered = resolve; }); + const blocked = new Promise(resolve => { releaseSave = resolve; }); + const connecting = manager.connect({ identity: "deleted-secret", credential: emptyState, + persist: async () => { saveEntered(); await blocked; }, + login: { expiresAt: Date.now() + 60_000, onAuthorized: vi.fn(), onFailure: vi.fn() } }); + const settled = connecting.catch(() => undefined); + try { + await entered; + await manager.disconnect("deleted-secret", { discardState: true }); + await expect(connecting).rejects.toThrow("Tailscale"); + expect(await manager.loginStatus("deleted-secret")).toBeNull(); + } finally { releaseSave(); await settled; } + }); + + it.each(["http://login.tailscale.com/a/fixture", "https://attacker.example/a/fixture", "https://login.tailscale.com/a/fixture?redirect=elsewhere", + "https://user:password@login.tailscale.com/a/fixture", "https://login.tailscale.com/a/fixture#token"])("rejects an unexpected sign-in address: %s", AuthURL => { + expect(() => parseTailscaleLoginStatus({ BackendState: "NeedsLogin", AuthURL })).toThrow("Invalid Tailscale"); + }); +}); diff --git a/server/src/services/connectors/tailscale/device-login-status.ts b/server/src/services/connectors/tailscale/device-login-status.ts new file mode 100644 index 00000000000..046296d80a5 --- /dev/null +++ b/server/src/services/connectors/tailscale/device-login-status.ts @@ -0,0 +1,14 @@ +export type TailscaleLoginStatus = { status: "authorizing" | "needs_approval" | "connected"; authorizationUrl?: string }; + +export function parseTailscaleLoginStatus(value: unknown): TailscaleLoginStatus { + const status = value as { BackendState?: unknown; AuthURL?: unknown } | null; + if (status?.BackendState === "Running") return { status: "connected" }; + if (status?.BackendState === "NeedsMachineAuth") return { status: "needs_approval" }; + if (typeof status?.AuthURL === "string" && status.AuthURL) { + const url = new URL(status.AuthURL); + if (url.origin !== "https://login.tailscale.com" || url.username || url.password || url.search || url.hash + || !/^\/a\/[a-zA-Z0-9_-]+$/.test(url.pathname)) throw new Error("Invalid Tailscale sign-in address"); + return { status: "authorizing", authorizationUrl: url.toString() }; + } + return { status: "authorizing" }; +} diff --git a/server/src/services/connectors/tailscale/device-login.test.ts b/server/src/services/connectors/tailscale/device-login.test.ts new file mode 100644 index 00000000000..53c6da4b7d9 --- /dev/null +++ b/server/src/services/connectors/tailscale/device-login.test.ts @@ -0,0 +1,158 @@ +import { randomUUID } from "node:crypto"; +import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { companies, createDb, tailscaleConnections, type Db } from "@paperclipai/db"; +import { eq } from "drizzle-orm"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../../__tests__/helpers/embedded-postgres.js"; +import { createIsolatedTestHome } from "../../../__tests__/helpers/isolated-test-home.js"; +import { secretService } from "../../secrets.js"; +import { tailscaleDeviceLoginService } from "./device-login.js"; +import { tailscaleConnectionService } from "./connections.js"; +import { tailscaleNetworks } from "./network.js"; + +const support = await getEmbeddedPostgresTestSupport(); +describe.skipIf(!support.supported)("durable device sign-in", () => { + let db: Db; + let cleanup: () => Promise; + let home: ReturnType; + let companyId: string; + let now: Date; + let vault: ReturnType; + let service: ReturnType; + const connect = vi.fn(); + const disconnect = vi.fn(); + const loginStatus = vi.fn(); + const state = (machine: string) => "paperclip-tailscale-state-v1:" + Buffer.from(JSON.stringify({ MachineKey: machine })).toString("base64"); + beforeAll(async () => { + home = createIsolatedTestHome(); + vi.stubEnv("PAPERCLIP_HOME", home.home); + vi.stubEnv("PAPERCLIP_INSTANCE_ID", "tailscale-device-login-fixture"); + vi.stubEnv("PAPERCLIP_SECRETS_PROVIDER", "local_encrypted"); + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "b".repeat(64)); + const started = await startEmbeddedPostgresTestDatabase("tailscale-device-login"); + cleanup = started.cleanup; + db = createDb(started.connectionString); + }); + afterAll(async () => { await cleanup?.(); vi.unstubAllEnvs(); home?.cleanup(); }); + beforeEach(async () => { + companyId = randomUUID(); + now = new Date(); + await db.insert(companies).values({ id: companyId, name: "Device login fixture", issuePrefix: randomUUID().slice(0, 8).toUpperCase() }); + vault = secretService(db); + connect.mockReset().mockImplementation(async input => { await input.reload?.(); await input.persist(state(input.identity)); return "/private/tailscale-fixture/socket"; }); + disconnect.mockReset().mockResolvedValue(undefined); + loginStatus.mockReset().mockResolvedValue({ status: "authorizing", authorizationUrl: "https://login.tailscale.com/a/fixture" }); + service = tailscaleDeviceLoginService(db, { secrets: vault, networks: { connect, disconnect, loginStatus }, now: () => now }); + }); + const start = () => service.start({ companyId, userId: "fixture-user", name: "Home" }); + async function row(id: string) { return (await db.select().from(tailscaleConnections).where(eq(tailscaleConnections.id, id)))[0]!; } + + it("stores device state without an OAuth app and completes only after the daemon confirms consent", async () => { + const result = await start(); + expect(result).toMatchObject({ connection: { companyId, status: "authorizing" }, authorizationUrl: "https://login.tailscale.com/a/fixture" }); + expect(JSON.stringify(result)).not.toContain("MachineKey"); + const saved = await row(result.connection.id); + const input = connect.mock.calls[0]![0]; + expect(await vault.resolveSystemSecretValue(companyId, saved.secretId!, saved.credentialVersion!, "tailscale_network", "fixture")).toBe(state(input.identity)); + await input.login!.onAuthorized(); + expect(await service.resume(companyId, saved.id, "fixture-user")).toMatchObject({ connection: { status: "connected" }, expiresAt: null }); + expect(await row(saved.id)).toMatchObject({ secretId: saved.secretId, oauthExpiresAt: null, status: "connected" }); + }); + + it("resumes an interrupted sign-in with the same encrypted state and deadline", async () => { + const first = await start(); + const saved = await row(first.connection.id); + const resumedService = tailscaleDeviceLoginService(db, { secrets: vault, networks: { connect, disconnect, loginStatus }, now: () => now }); + const resumed = await resumedService.resume(companyId, saved.id, "fixture-user"); + expect(resumed.expiresAt).toBe(first.expiresAt); + expect(connect.mock.calls[1]![0].credential).toBe(state(connect.mock.calls[0]![0].identity)); + expect((await row(saved.id)).secretId).toBe(saved.secretId); + }); + + it("keeps independent companies and sign-in initiators isolated", async () => { + const first = await start(); + await expect(service.resume(randomUUID(), first.connection.id, "fixture-user")).rejects.toMatchObject({ status: 404 }); + await expect(service.resume(companyId, first.connection.id, "other-user")).rejects.toMatchObject({ status: 403 }); + const secondCompany = randomUUID(); + await db.insert(companies).values({ id: secondCompany, name: "Separate tailnet", issuePrefix: randomUUID().slice(0, 8).toUpperCase() }); + const second = await service.start({ companyId: secondCompany, userId: "other-user", name: "Work" }); + expect(connect.mock.calls[0]![0].identity).not.toBe(connect.mock.calls[1]![0].identity); + expect((await row(first.connection.id)).secretId).not.toBe((await row(second.connection.id)).secretId); + }); + + it("does not revive a disconnected or replaced sign-in when an old daemon finishes", async () => { + const first = await start(); + const old = connect.mock.calls[0]![0]; + const connections = tailscaleConnectionService(db, { secrets: vault, networks: { connect, disconnect } }); + await connections.disconnect(companyId, first.connection.id); + await expect(old.login!.onAuthorized()).rejects.toMatchObject({ status: 422 }); + const before = await row(first.connection.id); + await old.persist(state("stale")); + expect((await row(first.connection.id)).credentialVersion).toBe(before.credentialVersion); + const replacement = await service.start({ companyId, userId: "fixture-user", name: "Home", connectionId: first.connection.id }); + const latest = connect.mock.calls.at(-1)![0]; + await latest.login!.onAuthorized(); + await old.login!.onFailure(); + expect((await row(replacement.connection.id)).status).toBe("connected"); + expect(latest.identity).not.toBe(old.identity); + }); + + it("expires pending consent and stops its network without extending the deadline", async () => { + const first = await start(); + const input = connect.mock.calls[0]![0]; + now = new Date(Date.parse(first.expiresAt!) + 1); + expect(await service.resume(companyId, first.connection.id, "fixture-user")).toMatchObject({ connection: { status: "expired" }, expiresAt: null }); + expect(disconnect).toHaveBeenCalledWith(input.identity); + await expect(input.login!.onAuthorized()).rejects.toMatchObject({ status: 422 }); + expect(connect).toHaveBeenCalledOnce(); + }); + + it("does not stop an approved connection when an older progress request fails", async () => { + const first = await start(); + const input = connect.mock.calls[0]![0]; + let entered!: () => void, reject!: (error: Error) => void; + const waiting = new Promise(resolve => { entered = resolve; }); + connect.mockImplementationOnce(() => { entered(); return new Promise((_resolve, fail) => { reject = fail; }); }); + const older = service.resume(companyId, first.connection.id, "fixture-user"); + const failed = expect(older).rejects.toMatchObject({ status: 422 }); + await waiting; + await input.login!.onAuthorized(); + reject(new Error("Old request failed")); + await failed; + expect((await row(first.connection.id)).status).toBe("connected"); + expect(disconnect).not.toHaveBeenCalled(); + }); + + it("removes the new secret if disconnect wins while sign-in is being prepared", async () => { + const first = await start(); + const connections = tailscaleConnectionService(db, { secrets: vault, networks: { connect, disconnect } }); + await connections.disconnect(companyId, first.connection.id); + let entered!: () => void, release!: () => void; + const waiting = new Promise(resolve => { entered = resolve; }); + const blocked = new Promise(resolve => { release = resolve; }); + const create = vault.createSystemSecret.bind(vault); + let createdId = ""; + vi.spyOn(vault, "createSystemSecret").mockImplementationOnce(async (...args) => { + const created = await create(...args); createdId = created.id; entered(); await blocked; return created; + }); + const removed = vi.spyOn(vault, "remove"); + const replacement = service.start({ companyId, userId: "fixture-user", name: "Home", connectionId: first.connection.id }); + const failed = expect(replacement).rejects.toMatchObject({ status: 422 }); + await waiting; + await connections.disconnect(companyId, first.connection.id); + release(); + await failed; + expect(removed).toHaveBeenCalledWith(createdId); + expect((await row(first.connection.id)).status).toBe("disconnected"); + }); + + it("rejects a rotated credential instead of joining another tailnet", async () => { + const first = await start(); + const saved = await row(first.connection.id); + await vault.rotate(saved.secretId!, { value: "tskey-auth-untrusted" }); + await service.resume(companyId, saved.id, "fixture-user"); + expect(connect.mock.calls.at(-1)![0].credential).toBe(state(connect.mock.calls[0]![0].identity)); + await db.update(tailscaleConnections).set({ credentialHash: "0".repeat(64) }).where(eq(tailscaleConnections.id, saved.id)); + await expect(service.resume(companyId, saved.id, "fixture-user")).rejects.toMatchObject({ status: 422 }); + expect(connect).toHaveBeenCalledTimes(2); + }); +}); diff --git a/server/src/services/connectors/tailscale/device-login.ts b/server/src/services/connectors/tailscale/device-login.ts new file mode 100644 index 00000000000..4488f38613a --- /dev/null +++ b/server/src/services/connectors/tailscale/device-login.ts @@ -0,0 +1,121 @@ +import { createHash, randomBytes } from "node:crypto"; +import { and, eq, gt, isNull } from "drizzle-orm"; +import { tailscaleConnections, type Db } from "@paperclipai/db"; +import type { TailscaleDeviceLogin } from "@paperclipai/shared"; +import { forbidden, notFound, unprocessable } from "../../../errors.js"; +import { logActivity } from "../../activity-log.js"; +import { secretService } from "../../secrets.js"; +import { getConfiguredSecretProvider } from "../../../secrets/configured-provider.js"; +import { tailscaleNetworks } from "./network.js"; + +const digest = (value: string) => createHash("sha256").update(value).digest("hex"); +const emptyState = "paperclip-tailscale-state-v1:" + Buffer.from("{}").toString("base64"); +const projection = (row: typeof tailscaleConnections.$inferSelect) => ({ id: row.id, companyId: row.companyId, name: row.name, status: row.status }); + +export function tailscaleDeviceLoginService(db: Db, dependencies: { + networks?: Pick; + secrets?: Pick, "createSystemSecret" | "resolveSystemSecretValue" | "rotate" | "remove">; + now?: () => Date; +} = {}) { + const networks = dependencies.networks ?? tailscaleNetworks; + const secrets = dependencies.secrets ?? secretService(db); + const now = dependencies.now ?? (() => new Date()); + const cancelled = () => unprocessable("Tailscale sign-in expired or was cancelled. Connect again.", { code: "tailscale_authorization_cancelled" }); + async function get(companyId: string, id: string) { + const [row] = await db.select().from(tailscaleConnections).where(and(eq(tailscaleConnections.companyId, companyId), eq(tailscaleConnections.id, id))); + if (!row) throw notFound("Tailscale connection not found"); + return row; + } + + async function resume(companyId: string, id: string, userId: string): Promise { + const row = await get(companyId, id); + if (row.createdByUserId !== userId) throw forbidden("Only the person who started this sign-in can continue it."); + if (row.status !== "authorizing") return { connection: projection(row), expiresAt: null }; + if (!row.secretId || !row.oauthStateHash?.startsWith("device:")) throw cancelled(); + const secretId = row.secretId; + const identity = `${companyId}:${secretId}`; + const owned = and(eq(tailscaleConnections.companyId, companyId), eq(tailscaleConnections.id, id), + eq(tailscaleConnections.secretId, secretId), eq(tailscaleConnections.oauthStateHash, row.oauthStateHash)); + if (!row.oauthExpiresAt || row.oauthExpiresAt <= now()) { + const expired = await db.update(tailscaleConnections).set({ status: "expired", oauthExpiresAt: null, updatedAt: now() }) + .where(and(owned, eq(tailscaleConnections.status, "authorizing"))).returning(); + if (expired.length) await networks.disconnect(identity); + return { connection: projection(await get(companyId, id)), expiresAt: null }; + } + async function current() { + const latest = await get(companyId, id); + if (latest.secretId !== secretId || latest.oauthStateHash !== row.oauthStateHash + || !["authorizing", "connected"].includes(latest.status)) throw cancelled(); + if (!latest.credentialVersion || !latest.credentialHash) throw cancelled(); + return latest; + } + const read = async () => { + const latest = await current(); + const value = await secrets.resolveSystemSecretValue(companyId, secretId, latest.credentialVersion!, "tailscale_network", `tailscale-connection:${id}`); + if (digest(value) !== latest.credentialHash) throw cancelled(); + return value; + }; + try { + await networks.connect({ identity, credential: await read(), reload: read, + persist: async value => { + const latest = await get(companyId, id); + if (latest.secretId !== secretId || latest.oauthStateHash !== row.oauthStateHash + || !["authorizing", "connected"].includes(latest.status)) return; + const saved = await secrets.rotate(secretId, { value }); + const [updated] = await db.update(tailscaleConnections).set({ credentialVersion: saved.latestVersion, credentialHash: digest(value), updatedAt: now() }) + .where(and(owned, eq(tailscaleConnections.credentialVersion, latest.credentialVersion!), eq(tailscaleConnections.credentialHash, latest.credentialHash!))).returning(); + if (!updated) throw cancelled(); + }, + login: { expiresAt: row.oauthExpiresAt.getTime(), + onAuthorized: async () => { + await db.transaction(async tx => { + const [ready] = await tx.update(tailscaleConnections).set({ status: "connected", oauthExpiresAt: null, updatedAt: now() }) + .where(and(owned, eq(tailscaleConnections.status, "authorizing"), gt(tailscaleConnections.oauthExpiresAt, now()))).returning(); + if (!ready) throw cancelled(); + await logActivity(tx as unknown as Db, { companyId, actorType: "user", actorId: userId, action: "tailscale.connected", entityType: "tailscale_connection", entityId: id }); + }); + }, + onFailure: async () => { + await db.update(tailscaleConnections).set({ status: row.oauthExpiresAt! <= now() ? "expired" : "error", oauthExpiresAt: null, updatedAt: now() }) + .where(and(owned, eq(tailscaleConnections.status, "authorizing"))); + }, + }, + }); + const status = await networks.loginStatus(identity); + const latest = await current(); + return { connection: projection(latest), expiresAt: latest.oauthExpiresAt?.toISOString() ?? null, + ...(latest.status === "authorizing" && status ? { authorizationUrl: status.authorizationUrl, needsApproval: status.status === "needs_approval" } : {}) }; + } catch { + const failed = await db.update(tailscaleConnections).set({ status: "error", oauthExpiresAt: null, updatedAt: now() }) + .where(and(owned, eq(tailscaleConnections.status, "authorizing"))).returning(); + if (failed.length) await networks.disconnect(identity); + throw unprocessable("Tailscale sign-in could not start. Check that Tailscale is installed, then try again.", { code: "tailscale_device_login_failed" }); + } + } + + return { + resume, + async start(input: { companyId: string; userId: string; name: string; connectionId?: string }): Promise { + const row = input.connectionId ? await get(input.companyId, input.connectionId) + : (await db.insert(tailscaleConnections).values({ companyId: input.companyId, name: input.name, createdByUserId: input.userId }).returning())[0]!; + if (row.status === "connected") throw unprocessable("Disconnect this network before replacing its connection."); + if (row.status === "authorizing" && row.oauthExpiresAt && row.oauthExpiresAt > now()) return resume(input.companyId, row.id, input.userId); + const state = `device:${randomBytes(32).toString("hex")}`; + const expiresAt = new Date(now().getTime() + 10 * 60_000); + const secret = await secrets.createSystemSecret(input.companyId, { name: `Tailscale connection ${row.id} ${randomBytes(4).toString("hex")}`, + provider: getConfiguredSecretProvider(), value: emptyState }, "tailscale_network", { userId: input.userId }); + try { + const [started] = await db.update(tailscaleConnections).set({ status: "authorizing", secretId: secret.id, + credentialVersion: secret.latestVersion, credentialHash: digest(emptyState), oauthStateHash: state, oauthExpiresAt: expiresAt, + createdByUserId: input.userId, updatedAt: now() }).where(and(eq(tailscaleConnections.id, row.id), eq(tailscaleConnections.companyId, input.companyId), eq(tailscaleConnections.status, row.status), + row.oauthStateHash === null ? isNull(tailscaleConnections.oauthStateHash) : eq(tailscaleConnections.oauthStateHash, row.oauthStateHash))).returning(); + if (!started) throw cancelled(); + } catch (error) { + await secrets.remove(secret.id); + throw error; + } + if (row.secretId) await networks.disconnect(`${input.companyId}:${row.secretId}`); + return resume(input.companyId, row.id, input.userId); + }, + }; +} diff --git a/server/src/services/connectors/tailscale/network.ts b/server/src/services/connectors/tailscale/network.ts index aa7055ca602..095ecf36c5a 100644 --- a/server/src/services/connectors/tailscale/network.ts +++ b/server/src/services/connectors/tailscale/network.ts @@ -3,6 +3,7 @@ import { promises as fs } from "node:fs"; import os from "node:os"; import path from "node:path"; import { promisify } from "node:util"; +import { parseTailscaleLoginStatus, type TailscaleLoginStatus } from "./device-login-status.js"; const exec = promisify(execFile); const STATE_PREFIX = "paperclip-tailscale-state-v1:"; @@ -12,7 +13,10 @@ export function isTailscaleNodeState(value: string): boolean { return value.startsWith(STATE_PREFIX); } -type Connection = { credential: string; savedCredential?: string; child?: ChildProcess; closed: boolean; discardState: AbortController; closing?: Promise; ready: Promise; close: () => Promise }; +type DeviceLogin = { expiresAt: number; onAuthorized: () => Promise; onFailure: () => Promise }; +type Connection = { credential: string; savedCredential?: string; child?: ChildProcess; loginChild?: ChildProcess; + loginStatus?: () => Promise; closed: boolean; discardState: AbortController; + closing?: Promise; ready: Promise; close: () => Promise }; export function createTailscaleNetworkManager(commands: { cli?: string; daemon?: string } = {}) { const connections = new Map(); @@ -23,6 +27,7 @@ export function createTailscaleNetworkManager(commands: { cli?: string; daemon?: credential: string; persist: (credential: string) => Promise; reload?: () => Promise; + login?: DeviceLogin; }, expectedGeneration = generation): Promise { if (expectedGeneration !== generation) throw new Error("Tailscale stopped"); const existing = connections.get(input.identity); @@ -64,7 +69,7 @@ export function createTailscaleNetworkManager(commands: { cli?: string; daemon?: return connection.ready; } - async function start(input: { identity: string; credential: string; persist: (credential: string) => Promise }, connection: Connection) { + async function start(input: { identity: string; credential: string; persist: (credential: string) => Promise; login?: DeviceLogin }, connection: Connection) { const restored = isTailscaleNodeState(input.credential); const state = restored ? Buffer.from(input.credential.slice(STATE_PREFIX.length), "base64") : null; if (state && (state.length > MAX_STATE_BYTES || !state.length)) throw new Error("Invalid state"); @@ -73,8 +78,34 @@ export function createTailscaleNetworkManager(commands: { cli?: string; daemon?: let dir: string | undefined; let statePath: string | undefined; + let saving = Promise.resolve(); + const persist = async (value: string) => { + const signal = connection.discardState.signal; + if (signal.aborted) return; + let discard!: () => void; + const discarded = new Promise(resolve => { discard = resolve; }); + signal.addEventListener("abort", discard, { once: true }); + try { + await Promise.race([input.persist(value), discarded]); + if (!signal.aborted) connection.savedCredential = value; + } finally { signal.removeEventListener("abort", discard); } + }; + const save = () => { + const pending = saving.then(async () => { + if (connection.discardState.signal.aborted || !statePath) return; + const bytes = await fs.readFile(statePath); + if (!bytes.length || bytes.length > MAX_STATE_BYTES) throw new Error("Invalid state"); + const next = STATE_PREFIX + bytes.toString("base64"); + if (next !== connection.savedCredential) { + await persist(next); + } + }); + saving = pending.catch(() => undefined); + return pending; + }; connection.close = () => connection.closing ??= (async () => { connection.closed = true; + connection.loginChild?.kill("SIGKILL"); const child = connection.child; if (child?.pid && child.exitCode === null && child.signalCode === null) { await new Promise((resolve) => { @@ -84,20 +115,13 @@ export function createTailscaleNetworkManager(commands: { cli?: string; daemon?: }); } try { + await saving; if (!connection.discardState.signal.aborted && connection.savedCredential && statePath) { const bytes = await fs.readFile(statePath); if (bytes.length && bytes.length <= MAX_STATE_BYTES) { const latest = STATE_PREFIX + bytes.toString("base64"); if (latest !== connection.savedCredential) { - const signal = connection.discardState.signal; - let discard!: () => void; - const discarded = new Promise(resolve => { discard = resolve; }); - signal.addEventListener("abort", discard, { once: true }); - try { - // Deletion can hold the vault lock needed by a pending state flush. - if (!signal.aborted) await Promise.race([input.persist(latest), discarded]); - if (!signal.aborted) connection.savedCredential = latest; - } finally { signal.removeEventListener("abort", discard); } + await persist(latest); } } } @@ -134,6 +158,48 @@ export function createTailscaleNetworkManager(commands: { cli?: string; daemon?: await new Promise((resolve) => setTimeout(resolve, 100)); } } + if (input.login) { + const login = input.login; + const remaining = login.expiresAt - Date.now(); + if (remaining <= 0 || remaining > 10 * 60_000) throw new Error("Tailscale sign-in expired"); + const up = exec(cliCommand, [`--socket=${socket}`, "up", "--accept-dns=false", "--accept-routes=false", "--shields-up", + `--timeout=${remaining}ms`, `--hostname=paperclip-${input.identity.replace(/[^a-z0-9]/gi, "").slice(-16)}`], + { timeout: remaining + 1000, maxBuffer: MAX_STATE_BYTES }); + connection.loginChild = up.child; + let loginFailed = false; + const completion = up.then(async () => { + if (connection.closed) return; + if (Date.now() >= login.expiresAt) throw new Error("Tailscale sign-in expired"); + const status = parseTailscaleLoginStatus(JSON.parse((await cli(["status", "--json"])).stdout)); + if (status.status !== "connected") throw new Error("Tailscale sign-in did not complete"); + await save(); + if (!connection.closed) await login.onAuthorized(); + }).catch(async () => { + loginFailed = true; + if (connection.closed) return; + try { await login.onFailure(); } finally { await connection.close(); } + }); + // The active CLI watches Tailscale events until consent, cancellation, or expiry. + void completion.catch(() => undefined); + connection.loginStatus = async () => { + if (loginFailed || connection.closed) throw new Error("Tailscale sign-in did not complete"); + const status = parseTailscaleLoginStatus(JSON.parse((await cli(["status", "--json"])).stdout)); + if (status.status === "connected") await completion; + if (loginFailed || connection.closed) throw new Error("Tailscale sign-in did not complete"); + return status; + }; + const loginDeadline = Math.min(login.expiresAt, Date.now() + 10_000); + for (;;) { + const status = await connection.loginStatus(); + if (status.status !== "authorizing" || status.authorizationUrl) { + await save(); + if (failed || connection.closed) throw new Error("Tailscale stopped"); + return socket; + } + if (Date.now() >= loginDeadline) throw new Error("Tailscale sign-in timed out"); + await new Promise(resolve => setTimeout(resolve, 100)); + } + } if (!restored) await fs.writeFile(keyPath, input.credential, { mode: 0o600 }); try { await cli(["up", "--accept-dns=false", "--accept-routes=false", "--shields-up", "--timeout=30s", @@ -144,17 +210,20 @@ export function createTailscaleNetworkManager(commands: { cli?: string; daemon?: } const status = JSON.parse((await cli(["status", "--json"])).stdout); if (status.BackendState !== "Running") throw new Error("Tailscale requires authorization"); - const nextState = await fs.readFile(statePath); - if (!nextState.length || nextState.length > MAX_STATE_BYTES) throw new Error("Invalid state"); - const saved = STATE_PREFIX + nextState.toString("base64"); - if (saved !== input.credential) await input.persist(saved); - connection.savedCredential = saved; + connection.savedCredential = input.credential; + await save(); if (failed || connection.closed) throw new Error("Tailscale stopped"); return socket; } return { connect: (input: Parameters[0]) => connect(input), + async loginStatus(identity: string) { + const connection = connections.get(identity); + if (!connection) return null; + await connection.ready; + return connection.loginStatus ? connection.loginStatus() : { status: "connected" as const }; + }, async disconnect(identity: string, options: { discardState?: boolean } = {}) { const connection = connections.get(identity); if (!connection) return; diff --git a/tests/runner-e2e/source-sdk-public-inputs.json b/tests/runner-e2e/source-sdk-public-inputs.json index 8c62ce51c2e..ddb291fc18b 100644 --- a/tests/runner-e2e/source-sdk-public-inputs.json +++ b/tests/runner-e2e/source-sdk-public-inputs.json @@ -268,7 +268,7 @@ "packages/shared/src/home-paths.ts": "fa4f1c440653b76b3dbb7a51026d557287deb45d", "packages/shared/src/humanize-connection.test.ts": "ba9e90a44c63452b0bf9f733adb90241921e8eca", "packages/shared/src/humanize-connection.ts": "fbd414476a0ef7106a7c80d8572208781b091699", - "packages/shared/src/index.ts": "17f32ce6e701bef6a98cdf685ab1616ab10e99b9", + "packages/shared/src/index.ts": "2d2b5ba044c9c7a22ff99022c0fac1e0da5e4a94", "packages/shared/src/issue-activity-identity.test.ts": "3dc294640f54c2be67943e3e0149cf4eff2c95ca", "packages/shared/src/issue-activity-identity.ts": "1b0f07ec3fe0506aaaf80a4fa9893f3baf8f0075", "packages/shared/src/issue-attribution.test.ts": "84563685e7f3c48029243cf1cc24a3a09ee3cde9", @@ -427,6 +427,7 @@ "packages/shared/src/types/skill-source.ts": "f34624008f2f852df198bb5f499f75f657c4f30e", "packages/shared/src/types/smoke-lab.ts": "534b1a1ef3f8fda6ae9a15e009c4f247a6f0b9e3", "packages/shared/src/types/summary-slot.ts": "9698bb7d628be3f2310d050037e30157d474ed48", + "packages/shared/src/types/tailscale.ts": "d02a917945d7779009a2dea4c4b535e9478e0680", "packages/shared/src/types/task-workspace.ts": "a23afa13d2cb58dc2eb72692ed28c16ed6fb84fb", "packages/shared/src/types/teams-catalog.ts": "632fe75f8e5a4cfc455843c2a1a2daff28a022be", "packages/shared/src/types/tool-access.ts": "4c05d176f9bb1d30593fd904c7c4fc3e91b431ca", diff --git a/ui/src/api/tailscale.ts b/ui/src/api/tailscale.ts index dbddea10380..9d4b5b6ddd1 100644 --- a/ui/src/api/tailscale.ts +++ b/ui/src/api/tailscale.ts @@ -1,11 +1,12 @@ -import type { ToolOAuthStartResult } from "@paperclipai/shared"; +import type { TailscaleConnection, TailscaleDeviceLogin } from "@paperclipai/shared"; import { api } from "./client"; -export type TailscaleConnection = { id: string; companyId: string; name: string; status: string }; +export type { TailscaleConnection } from "@paperclipai/shared"; export type TailscaleDevice = { id: string; name: string; hostname: string; address: string; os: string; online: boolean }; export const tailscaleApi = { list: (companyId: string) => api.get(`/companies/${companyId}/tailscale-connections`), - connect: (companyId: string, connectionId?: string) => api.post(`/companies/${companyId}/tailscale-connections`, { name: "My tailnet", ...(connectionId ? { connectionId } : {}) }), + connect: (companyId: string, connectionId?: string) => api.post(`/companies/${companyId}/tailscale-connections`, { name: "My tailnet", ...(connectionId ? { connectionId } : {}) }), + deviceLogin: (companyId: string, connectionId: string) => api.post(`/companies/${companyId}/tailscale-connections/${connectionId}/device-login`, {}), devices: (companyId: string, id: string) => api.get(`/companies/${companyId}/tailscale-connections/${id}/devices`), disconnect: (companyId: string, id: string) => api.delete(`/companies/${companyId}/tailscale-connections/${id}`), pairing: (companyId: string, id: string, deviceId: string, port: number) => api.post<{ privateKeySecretId: string; publicKey: string; command: string }>(`/companies/${companyId}/tailscale-connections/${id}/pairing`, { deviceId, port }), diff --git a/ui/src/pages/apps/TailscaleConnector.test.tsx b/ui/src/pages/apps/TailscaleConnector.test.tsx index 4e7f8ce22c3..e4929353b5f 100644 --- a/ui/src/pages/apps/TailscaleConnector.test.tsx +++ b/ui/src/pages/apps/TailscaleConnector.test.tsx @@ -5,12 +5,11 @@ import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { TailscaleConnector, TailscaleConnectorCard } from "./TailscaleConnector"; import { ApiError } from "@/api/client"; -import { readPendingCloudHandoff } from "@/lib/oauthHandoff"; -import { PaperclipCloudOAuthHandoffPage } from "./PaperclipCloudOAuthHandoff"; +import { shouldRetryRequest } from "@/lib/query-client"; -const fixture = vi.hoisted(() => ({ companyId: "company-1", search: "", navigate: vi.fn(), access: vi.fn(), list: vi.fn(), connect: vi.fn(), disconnect: vi.fn(), breadcrumbs: vi.fn() })); +const fixture = vi.hoisted(() => ({ companyId: "company-1", search: "", navigate: vi.fn(), access: vi.fn(), list: vi.fn(), connect: vi.fn(), deviceLogin: vi.fn(), disconnect: vi.fn(), breadcrumbs: vi.fn() })); vi.mock("@/api/access", () => ({ accessApi: { getCurrentBoardAccess: fixture.access } })); -vi.mock("@/api/tailscale", () => ({ tailscaleApi: { list: fixture.list, connect: fixture.connect, disconnect: fixture.disconnect } })); +vi.mock("@/api/tailscale", () => ({ tailscaleApi: { list: fixture.list, connect: fixture.connect, deviceLogin: fixture.deviceLogin, disconnect: fixture.disconnect } })); vi.mock("@/context/CompanyContext", () => ({ useCompany: () => ({ selectedCompanyId: fixture.companyId }) })); vi.mock("@/context/BreadcrumbContext", () => ({ useBreadcrumbs: () => ({ setBreadcrumbs: fixture.breadcrumbs }) })); vi.mock("./AppLogo", () => ({ AppLogo: () => null })); @@ -33,6 +32,8 @@ function button(label: string) { describe("Tailscale network connector", () => { beforeEach(() => { vi.clearAllMocks(); + vi.spyOn(window, "open").mockReturnValue(null); + fixture.deviceLogin.mockResolvedValue({ connection: { ...connected, status: "authorizing" }, authorizationUrl: "https://login.tailscale.com/a/fixture", expiresAt: new Date(Date.now() + 600_000).toISOString() }); fixture.companyId = "company-1"; fixture.search = ""; window.sessionStorage.clear(); @@ -83,7 +84,7 @@ describe("Tailscale network connector", () => { expect(fixture.connect).not.toHaveBeenCalled(); }); - it("starts OAuth from Connectors only when the user clicks Connect", async () => { + it("starts device sign-in from Connectors only when the user clicks Connect", async () => { render();await settle(); flushSync(() => button("Connect Tailscale Network").click());await settle(); expect(fixture.connect).toHaveBeenCalledWith("company-1", undefined); @@ -97,24 +98,77 @@ describe("Tailscale network connector", () => { expect(fixture.connect).toHaveBeenCalledWith("company-1", "tailnet-1"); }); - it("preserves the current tab's Cloud handoff through reauthentication and resumes consent", async () => { - const session = "tailscale_handoff_session_abcdefghijklmnop"; - const reauthenticationUrl = new URL(`/cloud/connections/reauth?session=${session}`, window.location.origin).toString(); - fixture.connect.mockResolvedValue({ authorizationUrl: "", handoff: { kind: "paperclip_cloud", session } }); - vi.stubGlobal("fetch", vi.fn() - .mockResolvedValueOnce(Response.json({ error: "RECENT_LOGIN_REQUIRED", reauthenticationUrl }, { status: 401 })) - .mockResolvedValueOnce(Response.json({ authorizationUrl: "https://login.tailscale.com/a/oauth_authorize" }))); - fixture.navigate.mockImplementationOnce(() => { - expect(readPendingCloudHandoff()).toEqual({ kind: "paperclip_cloud", session }); - }); + it("keeps the connector open and offers a sign-in link when popups are blocked", async () => { + fixture.connect.mockResolvedValue({ connection: { ...connected, status: "authorizing" }, authorizationUrl: "https://login.tailscale.com/a/fixture", expiresAt: new Date(Date.now() + 600_000).toISOString() }); render();await settle(); flushSync(() => button("Connect Tailscale Network").click());await settle(); - expect(readPendingCloudHandoff()).toEqual({ kind: "paperclip_cloud", session }); - expect(fixture.navigate).toHaveBeenCalledWith(reauthenticationUrl); - flushSync(() => root.render());await settle(); - expect(fixture.navigate).toHaveBeenLastCalledWith("https://login.tailscale.com/a/oauth_authorize"); - expect(readPendingCloudHandoff()).toBeNull(); - expect(container.textContent).not.toContain("expired"); + expect(window.open).toHaveBeenCalledWith("about:blank", "_blank"); + expect(fixture.navigate).not.toHaveBeenCalled(); + expect(fixture.deviceLogin).toHaveBeenCalledWith("company-1", "tailnet-1"); + expect(container.querySelector('a[href="https://login.tailscale.com/a/fixture"]')?.textContent).toBe("Continue to Tailscale"); + expect(button("Cancel sign-in")).toBeDefined(); + }); + + it("shows administrator approval separately from sign-in", async () => { + fixture.list.mockResolvedValue([{ ...connected, status: "authorizing" }]); + fixture.deviceLogin.mockResolvedValue({ connection: { ...connected, status: "authorizing" }, needsApproval: true, expiresAt: new Date(Date.now() + 600_000).toISOString() }); + render();await settle(); + expect(container.textContent).toContain("Tailscale administrator"); + expect(container.querySelector('a[href="https://login.tailscale.com/admin/machines"]')).not.toBeNull(); + }); + + it("recovers a transient sign-in status failure using the shared retry policy", async () => { + client.setDefaultOptions({ queries: { retry: shouldRetryRequest, retryDelay: 0 } }); + fixture.list.mockResolvedValue([{ ...connected, status: "authorizing" }]); + fixture.deviceLogin.mockRejectedValueOnce(new ApiError("Unavailable", 503, {})); + render();await settle(); + expect(fixture.deviceLogin).toHaveBeenCalledTimes(2); + expect(container.querySelector('a[href="https://login.tailscale.com/a/fixture"]')).not.toBeNull(); + expect(container.querySelector('[role="alert"]')).toBeNull(); + }); + + it("does not automatically retry a denied sign-in status request", async () => { + client.setDefaultOptions({ queries: { retry: shouldRetryRequest, retryDelay: 0 } }); + fixture.list.mockResolvedValue([{ ...connected, status: "authorizing" }]); + fixture.deviceLogin.mockRejectedValue(new ApiError("Forbidden", 403, {})); + render();await settle(); + expect(fixture.deviceLogin).toHaveBeenCalledOnce(); + expect(container.querySelector('[role="alert"]')).not.toBeNull(); + expect(button("Check again")).toBeDefined(); + }); + + it("shows the connected network when the server confirms enrollment", async () => { + fixture.list.mockResolvedValue([{ ...connected, status: "authorizing" }]); + fixture.deviceLogin.mockResolvedValue({ connection: connected, expiresAt: null }); + render();await settle(); + expect(container.textContent).toContain("Create SSH environment"); + expect(container.textContent).not.toContain("Cancel sign-in"); + }); + + it("does not open a late sign-in response after changing companies", async () => { + let complete!: (value: unknown) => void; + fixture.connect.mockImplementation(() => new Promise(resolve => { complete = resolve; })); + const assign = vi.fn(); + vi.spyOn(window, "open").mockReturnValue({ opener: null, closed: false, location: { assign }, focus: vi.fn(), close: vi.fn() } as unknown as Window); + fixture.list.mockImplementation(async id => id === "company-1" ? [connected] : []); + render();await settle(); + flushSync(() => button("Connect Tailscale Network").click());await settle(); + fixture.companyId = "company-2";render();await settle(); + complete({ connection: { ...connected, status: "authorizing" }, authorizationUrl: "https://login.tailscale.com/a/fixture", expiresAt: new Date(Date.now() + 600_000).toISOString() }); + await settle(); + expect(assign).not.toHaveBeenCalled(); + expect(container.textContent).not.toContain("Home tailnet"); + }); + + it("stops polling and gives a recovery action when the sign-in deadline passes", async () => { + fixture.list.mockResolvedValue([{ ...connected, status: "authorizing" }]); + fixture.deviceLogin.mockResolvedValue({ connection: { ...connected, status: "authorizing" }, authorizationUrl: "https://login.tailscale.com/a/fixture", expiresAt: new Date(Date.now() - 1).toISOString() }); + render();await settle(); + expect(container.textContent).toContain("Sign-in expired"); + expect(container.querySelector('a[href="https://login.tailscale.com/a/fixture"]')).toBeNull(); + await new Promise(resolve => setTimeout(resolve, 2100)); + expect(fixture.deviceLogin).toHaveBeenCalledOnce(); + expect(button("Cancel sign-in")).toBeDefined(); }); it.each(["cancelled", "failed"])("explains %s consent and retries the same saved connection", async outcome => { @@ -149,7 +203,7 @@ describe("Tailscale network connector", () => { expect(container.textContent).not.toContain("Create SSH environment"); }); - it("reports access failure without presenting a usable connection or starting OAuth", async () => { + it("reports access failure without presenting a usable connection or starting sign-in", async () => { fixture.list.mockRejectedValue(new ApiError("Instance admin access required", 403, { error: "forbidden" })); render();await settle(); expect(container.querySelector('[role="alert"]')).not.toBeNull(); diff --git a/ui/src/pages/apps/TailscaleConnector.tsx b/ui/src/pages/apps/TailscaleConnector.tsx index 43f7ae379ee..378456fe71b 100644 --- a/ui/src/pages/apps/TailscaleConnector.tsx +++ b/ui/src/pages/apps/TailscaleConnector.tsx @@ -1,15 +1,15 @@ -import { useEffect } from "react"; +import { useEffect, useRef } from "react"; import { useMutation, useQueryClient } from "@tanstack/react-query"; -import { tailscaleApi } from "@/api/tailscale"; +import { tailscaleApi, type TailscaleConnection } from "@/api/tailscale"; import { describeError } from "@/api/errors"; import { Button } from "@/components/ui/button"; import { useCompany } from "@/context/CompanyContext"; import { useBreadcrumbs } from "@/context/BreadcrumbContext"; import { tailscaleConnectionsKey, useTailscaleConnections } from "@/hooks/useTailscaleConnections"; import { Link, useSearchParams } from "@/lib/router"; -import { prepareOAuthNavigation, savePendingCloudHandoff } from "@/lib/oauthHandoff"; -import { navigateTopLevel } from "@/lib/browserNavigation"; +import { useAuthorizationWindow } from "@/hooks/useAuthorizationWindow"; import { AppLogo } from "./AppLogo"; +import { TailscaleDeviceSignIn } from "./TailscaleDeviceSignIn"; export const TAILSCALE_CONNECTOR_PATH = "/apps/tailscale"; @@ -38,14 +38,26 @@ export function TailscaleConnector() { const client = useQueryClient(); const [searchParams] = useSearchParams(); const connections = useTailscaleConnections(selectedCompanyId); - const connect = useMutation({ mutationFn: async (connectionId?: string) => { - const result = await tailscaleApi.connect(selectedCompanyId!, connectionId); - const target = await prepareOAuthNavigation(result); - if (target.kind === "reauthentication" && result.handoff) savePendingCloudHandoff(result.handoff.session); - navigateTopLevel(target.url); - } }); + const authorization = useAuthorizationWindow(); + const company = useRef(selectedCompanyId); + company.current = selectedCompanyId; + const connect = useMutation({ mutationFn: async (input: { companyId: string; connectionId?: string }) => { + const result = await tailscaleApi.connect(input.companyId, input.connectionId); + client.setQueryData(tailscaleConnectionsKey(input.companyId), rows => [ + ...(rows ?? []).filter(row => row.id !== result.connection.id), result.connection, + ]); + if (company.current !== input.companyId) return; + if (result.authorizationUrl) await authorization.open(result.authorizationUrl); + else authorization.cancel(); + }, onError: () => authorization.cancel() }); + const start = (connectionId?: string) => { + if (!selectedCompanyId) return; + authorization.reserve(); + connect.mutate({ companyId: selectedCompanyId, connectionId }); + }; const disconnect = useMutation({ mutationFn: (connectionId: string) => tailscaleApi.disconnect(selectedCompanyId!, connectionId), - onSuccess: () => client.invalidateQueries({ queryKey: tailscaleConnectionsKey(selectedCompanyId) }) }); + onSuccess: () => { authorization.cancel(); return client.invalidateQueries({ queryKey: tailscaleConnectionsKey(selectedCompanyId) }); } }); + useEffect(() => authorization.cancel, [selectedCompanyId, authorization.cancel]); useEffect(() => { setBreadcrumbs([{ label: "Connectors", href: "/apps" }, { label: "Tailscale" }]); return () => setBreadcrumbs([]); @@ -63,18 +75,19 @@ export function TailscaleConnector() {

Tailscale

Give Paperclip access to your tailnet, then choose one of its machines when creating an SSH environment. Agents can use any harness supported by that environment.

{connections.isPending &&

Loading tailnet connections…

} - {returnError &&

{returnError}

{retryConnection && }
} + {returnError &&

{returnError}

{retryConnection && }
} {error &&

{describeError(error, { action: "connect Tailscale" }).body}

}
{(connections.data ?? []).map(connection =>
-

{connection.name}

{connection.status === "connected" ? "Connected" : connection.status === "connecting" ? "Connecting" : connection.status === "disconnected" ? "Disconnected" : "Authorization not completed"}

+

{connection.name}

{connection.status === "connected" ? "Connected" : connection.status === "authorizing" ? "Waiting for sign-in" : connection.status === "connecting" ? "Connecting" : connection.status === "disconnected" ? "Disconnected" : connection.status === "expired" ? "Sign-in expired" : "Authorization not completed"}

{connection.status === "connected" ? <> - : } + : connection.status === "authorizing" ? disconnect.mutate(connection.id)} disabled={disconnect.isPending} /> + : }
)}
- +
; } diff --git a/ui/src/pages/apps/TailscaleDeviceSignIn.tsx b/ui/src/pages/apps/TailscaleDeviceSignIn.tsx new file mode 100644 index 00000000000..293b347a0d4 --- /dev/null +++ b/ui/src/pages/apps/TailscaleDeviceSignIn.tsx @@ -0,0 +1,43 @@ +import { useEffect, useState } from "react"; +import { useQuery, useQueryClient } from "@tanstack/react-query"; +import type { TailscaleConnection } from "@paperclipai/shared"; +import { tailscaleApi } from "@/api/tailscale"; +import { describeError } from "@/api/errors"; +import { Button } from "@/components/ui/button"; +import { tailscaleConnectionsKey } from "@/hooks/useTailscaleConnections"; + +export function TailscaleDeviceSignIn({ connection, cancel, disabled }: { + connection: TailscaleConnection; cancel: () => void; disabled: boolean; +}) { + const client = useQueryClient(); + const [expired, setExpired] = useState(false); + const status = useQuery({ + queryKey: ["tailscale-device-login", connection.companyId, connection.id], + queryFn: () => tailscaleApi.deviceLogin(connection.companyId, connection.id), + gcTime: 0, + refetchInterval: query => expired || query.state.error || query.state.data?.connection.status !== "authorizing" ? false : 2000, + refetchOnWindowFocus: !expired, + }); + useEffect(() => { + const next = status.data?.connection; + if (!next || next.status === "authorizing") return; + client.setQueryData(tailscaleConnectionsKey(connection.companyId), rows => rows?.map(row => row.id === next.id ? next : row)); + }, [status.data, client, connection.companyId]); + useEffect(() => { + const deadline = status.data?.expiresAt ? Date.parse(status.data.expiresAt) : Date.now() + 10 * 60_000; + const timer = window.setTimeout(() => setExpired(true), Math.max(0, deadline - Date.now())); + return () => window.clearTimeout(timer); + }, [status.data?.expiresAt]); + return
+

{expired ? "Sign-in expired. Cancel and connect again." + : status.data?.needsApproval ? "Waiting for a Tailscale administrator to approve this device." + : "Approve the connection in Tailscale. This page will update when you finish."}

+ {status.error &&

{describeError(status.error, { action: "check Tailscale sign-in" }).body}

} +
+ {!expired && status.data?.authorizationUrl && } + {status.data?.needsApproval && !expired && } + {status.error && !expired && } + +
+
; +} From f610e6a880a63c964da9be348263faa6e5449591 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 19:40:38 -0600 Subject: [PATCH 02/11] fix(recovery): replace a queued run the dispatcher keeps passing over instead of counting it as a pending path (#21502) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The assigned-work sweep is the backstop that makes sure every open agent task has a live run, a pending path, or a named wait on a person > - It counts a task with a queued run as healthy. A queued run that the dispatcher declines on every pass is never started, so the task waits forever and the sweep never looks again > - On a hosted instance one agent had more than 30 queued continuations that the claim declined on every pass. The sweep saw each task as healthy > - This pull request makes the sweep recognize a queued run that the dispatcher keeps passing over, report it, and replace it once with a new continuation when it is automatic work > - The benefit is that an automatic task cannot stay stuck behind a run that never starts, while a person's work is never cancelled ## Linked Issues or Issue Description Refs #21391, #21349, #21433, #21394, #21443, #21450 **What happened?** An agent had more than 30 queued automatic continuations (`issue_continuation_needed`) that never started. The claim declined each one on every dispatch pass. #21391 fixed that specific cause in the chat-control gate. But the sweep still trusted any queued run as a pending path. A run that the claim declines for any persistent reason keeps the task "healthy" forever. **Expected behavior** A queued run that the dispatcher keeps passing over is not a healthy pending path. The sweep reports it. For automatic work it cancels the run through the run lifecycle and queues one new continuation in its place. For a person's work it reports the run and leaves it. **Steps to reproduce** 1. Queue an automatic run for a task and make the claim decline it persistently (for example, a chat-control admission marker that does not verify). 2. Let the run wait for more than one hour while its agent starts other runs. 3. Run the assigned-work sweep for the task. Before this change, it reports the task as `live_run` and does nothing. ## What Changed - New violation `queued_run_stalled` in `recovery/assigned-work-invariant.ts`. A queued run is stalled when all of these are true (read from the database, on the database clock): - it has been queued for longer than `QUEUED_RUN_AGING_MS` (60 minutes, from `queued-run-dispatch-order.ts`); - it is the oldest queued run of its agent; - its agent started more than `OLDEST_DISPATCH_SHARE` other runs after the run became its oldest queued run: after the later of the run's aging time and the moment the last older run of the agent started or finished. - A stalled run is not a live run, and it does not let the task's interruption projection ("working") count as a pending path. Its own wake is not counted as a pending wake. A stalled run on a blocked task still counts as live, so the block decides first. - `listSuspects` lets a task through when its only active run is a stalled queued run. - `act()` handles the violation after all existing early returns: - person work (the #21349 source-chain rule through `listPersonOriginSourceRunIds`, which this change exports, or an explicit user continuation): reported, never cancelled; - a replacement that stalls again (`replacesStalledRunId` in its context): reported, not cancelled again; - both log one warning per run, with the run, agent and how long it waited, and look again after `CONTINUATION_CAP_WINDOW_MS` (one hour); - automatic work: the existing claim under the task row lock re-checks the facts and the shared hourly cap and spacing. Inside that claim, before the person check, it reads the run's row version (`xmin`) and the number of wakes attached to it. Then the run is cancelled through the run lifecycle (`setRunStatusFromLive` from `queued`, with that version and wake count as its condition; wake cancelled, run event, `releaseIssueExecutionAndPromote` with `suppressImmediateRecovery`). A person's wake that joins the run after the check, or a claim that starts it, changes one of them, so the cancel does nothing. Then one continuation wake is queued with `previousRunId` and `replacesStalledRunId` set to the cancelled run. If the cancel does nothing, the unpublished claim entry is removed and nothing is queued. - New `heartbeat/stalled-queued-run.ts` builds that cancel from the run lifecycle functions. `heartbeat.ts` and `recovery/service.ts` pass it to the sweep. - Plain activity text: "Continued automatically: this task had waited over an hour to start while its agent started other work, so Paperclip queued it again." ### Why this signal - #21433 keeps refused claims and claim back-off in memory only (`declinedAgedClaimAt`, `claimBackoffs` in `heartbeat/queue.ts`). Another server cannot read them. Run rows can be read from any server, on the database clock. - After #21349, an aged run ranks no lower than ordinary work, and the oldest-run share starts the oldest ready queued run at least once in every `OLDEST_DISPATCH_SHARE` starts. So when a run has been its agent's oldest queued run since after the aging bound, and its agent started more runs in that time than the share allows, the dispatcher offered the run a slot and the claim declined it. Starts that happened while older runs were still waiting do not count: those older runs were getting their turns. - A run behind a full agent sees no such starts and is left alone. Under a slow backlog only the oldest queued run can match. So the younger queued runs keep their place in line. - Starts are counted only after the run reached 60 minutes of age, not after the current server's boot. 60 minutes is longer than a release swap overlap. So a run that only waits for a new server to reattach to its Runners does not match. #21450 will add a reattach-phase gate before this path. ### Why this does not set `retryOfRunId` An earlier plan added `retryOfRunId` to the sweep's continuation so the chat-control gate could follow it. An audit of its readers found three changes in behavior: the admission-time chat-control check would start to refuse sweep wakes; the claim would wait on the source run's cleanup (a possible wait forever); and the agent prompt would call the run "a Retry" that "re-delivers the wake". #21391 already makes these runs claimable. So this change leaves `retryOfRunId` unset, as today. The replacement links its source with `previousRunId`, as the sweep already does. ### Questions from review, answered from the code - **How often is the invariant pass triggered, and can two passes overlap?** The `assigned-work-invariant` delivery queue (`recovery/work.ts`, `scoped-deadline-work.ts`) runs on database-work notifications, on each task's own deadline (`retryAt`), and on a 30 s reconstruction of the suspect list. Passes are at least 1 s apart (`minimumPassIntervalMs` default). In one process, passes of the queue never overlap (`running` guard in `delivery-work-coordinator.ts`). Two servers can sweep at the same time. Each repair is serialized by the claim under the task row lock (`FOR UPDATE` in `claimContinuation`), which re-reads the facts and the spacing. This change does not add a new loop. - **Can two runs or no run exist for one task?** Two runs: no. The claim's spacing makes a second server back off, and the cancel happens before the new wake (a queued run on the task would otherwise absorb the wake). No run: there is a short gap between the cancel and the new wake, because wake admission locks the task row in its own transaction. If the server stops in that gap, the task has nothing pending, and the next pass repairs it as `no_pending_path` under the same cap. ## Verification - `server/src/__tests__/assigned-work-stalled-queued-run.test.ts` (new, embedded Postgres, real `createHeartbeatQueue` admission and `claimQueuedRun`): - an automatic run that the real claim declines persistently, whose agent started runs after it aged: reported, cancelled, exactly one replacement, and the real claim starts the replacement; - a run behind a full agent: left alone, not a suspect; - a person's declined Retry: reported, not cancelled, one warning; - a replacement that stalls again: not cancelled on two checks, one warning, next look one hour later; - two sweep instances on the same task at the same time: one replacement; - three aged runs queued in order, the first two get their turns slowly: the third is not stalled when it becomes oldest, and is stalled only after more than four further starts pass it over; - a person's message joins the run (real wake admission) between the claim and the cancel: the run is not cancelled, there is no replacement, and the person's wake stays attached to the run; - the run starts between the claim and the cancel: no cancel, no replacement. - First round, red before the fix: 4 failed, 1 passed (the full-agent case passes before and after). Green: 5 passed. - Review round: the three new tests on the previous head: 2 failed (the backlog case and the person-join case), 1 passed (the started-run case, which the `queued` check already covered). Green: 8 passed. - Existing files run one at a time: `assigned-work-invariant.test.ts` 31 passed, `assigned-work-sweep-round-trips.test.ts` 2 passed, `failed-continuation-continues-again.test.ts` 28 passed, `recovery-wake-herd.test.ts` 35 passed, `saved-work-recovery-overdue.test.ts` 4 passed. - `pnpm --filter @paperclipai/server exec tsc --noEmit`: clean. - `pnpm verify`: not completed locally (the machine is heavily loaded and the run was stopped when the code changed). Required CI supplies the broad coverage. ## Risks - A task's oldest queued automatic run is cancelled and queued again at most once per run. Its replacement starts at the back of the line. Only a run that the dispatcher already passed over can match. - A replacement that stalls again is only reported, so a persistent claim refusal cannot cause repeated cancellations. - The new subqueries run only for queued runs older than one hour. They use the existing `(company_id, agent_id, started_at)` index. Healthy runs and the path before `turn.start` do not change. - #21443 (automatic-run brake) and #21450 (reattach first) touch `assigned-work-sweep.ts`. The new path comes after every early return that `act()` has today, so their gates can go before it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). A subagent on the same model audited the `retryOfRunId` readers. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../assigned-work-stalled-queued-run.test.ts | 252 ++++++++++++++++++ .../src/__tests__/helpers/heartbeat-queue.ts | 4 +- server/src/services/heartbeat.ts | 2 + .../src/services/heartbeat/queued-run-line.ts | 2 +- .../services/heartbeat/stalled-queued-run.ts | 32 +++ .../recovery/assigned-work-invariant.ts | 18 +- .../services/recovery/assigned-work-sweep.ts | 123 ++++++++- server/src/services/recovery/service.ts | 2 + 8 files changed, 416 insertions(+), 19 deletions(-) create mode 100644 server/src/__tests__/assigned-work-stalled-queued-run.test.ts create mode 100644 server/src/services/heartbeat/stalled-queued-run.ts diff --git a/server/src/__tests__/assigned-work-stalled-queued-run.test.ts b/server/src/__tests__/assigned-work-stalled-queued-run.test.ts new file mode 100644 index 00000000000..5b006a9c68b --- /dev/null +++ b/server/src/__tests__/assigned-work-stalled-queued-run.test.ts @@ -0,0 +1,252 @@ +import { randomUUID } from "node:crypto"; +import { and, eq, sql } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { activityLog, agents, agentWakeupRequests, companies, createDb, heartbeatRuns, issues, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { createHeartbeatQueueTestDependencies } from "./helpers/heartbeat-queue.js"; +import { createHeartbeatQueue, shouldDeferFollowupWakeForSameIssue, shouldQueueFollowupForRunningIssueWake } from "../services/heartbeat/queue.js"; +import { mergeCoalescedContextSnapshot } from "../services/heartbeat/run-preparation.js"; +import { createWakeQueue } from "../modules/wake-queue/index.js"; +import { OLDEST_DISPATCH_SHARE, QUEUED_RUN_AGING_MS } from "../services/heartbeat/queued-run-dispatch-order.js"; +import { createStalledQueuedRunCancel } from "../services/heartbeat/stalled-queued-run.js"; +import { CHAT_CONTROL_RECOVERY_ADMISSION_KEY } from "../services/chat-control-recovery-stop.js"; +import { ASSIGNED_WORK_CONTINUED_ACTION, assignedWorkSweep } from "../services/recovery/assigned-work-sweep.js"; +import { CONTINUATION_CAP_WINDOW_MS } from "../services/recovery/assigned-work-invariant.js"; +import { logger } from "../middleware/logger.js"; + +const support = await getEmbeddedPostgresTestSupport(); +const MINUTE = 60_000; + +describe.skipIf(!support.supported)("a queued run the dispatcher keeps passing over", () => { + let database: Awaited>; + let db: ReturnType; + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("stalled-queued-run-"); + db = createDb(database.connectionString); + }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); + afterAll(async () => { await db?.$client.end(); await database?.cleanup(); }); + afterEach(() => { vi.restoreAllMocks(); }); + + async function seed() { + const prefix = `S${randomUUID().slice(0, 5)}`; + const [company] = await db.insert(companies).values({ name: "Stalled", issuePrefix: prefix }).returning(); + const companyId = company!.id; + const [agent] = await db.insert(agents).values({ companyId, name: "Builder", adapterType: "process", status: "idle" }).returning(); + const agentId = agent!.id; + let number = 0; + const task = async () => { + number += 1; + const [row] = await db.insert(issues).values({ companyId, title: `Task ${number}`, status: "in_progress", assigneeAgentId: agentId, responsibleUserId: "person-1", + issueNumber: number, identifier: `${prefix}-${number}`, createdAt: new Date(Date.now() - 5 * 60 * MINUTE), + updatedAt: new Date(Date.now() - 3 * 60 * MINUTE) }).returning(); + return row!; + }; + const issue = await task(); + const other = await task(); + + async function queuedRun(options: { ageMs?: number; person?: boolean; refused?: boolean; context?: Record } = {}) { + const createdAt = new Date(Date.now() - (options.ageMs ?? 2 * 60 * MINUTE)); + const reason = options.person ? "task_retry" : "issue_continuation_needed"; + const [wake] = await db.insert(agentWakeupRequests).values({ companyId, agentId, source: options.person ? "on_demand" : "automation", + triggerDetail: options.person ? "manual" : "system", reason, payload: { issueId: issue.id }, status: "claimed", + requestedByActorType: options.person ? "user" : "system", requestedByActorId: options.person ? "person-1" : null, + claimedAt: createdAt, createdAt }).returning(); + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: options.person ? "on_demand" : "automation", + triggerDetail: options.person ? "manual" : "system", status: "queued", wakeupRequestId: wake!.id, createdAt, updatedAt: createdAt, + contextSnapshot: { issueId: issue.id, taskId: issue.id, wakeReason: reason, ...options.context } }).returning(); + if (options.refused !== false) { + await db.update(heartbeatRuns).set({ runnerProfileJson: { [CHAT_CONTROL_RECOVERY_ADMISSION_KEY]: { version: 1, phase: "lost" } } }) + .where(eq(heartbeatRuns.id, run!.id)); + } + await db.update(agentWakeupRequests).set({ runId: run!.id }).where(eq(agentWakeupRequests.id, wake!.id)); + return run!; + } + + async function otherStarts(count: number, startedAgoMs: number, status = "succeeded") { + for (let index = 0; index < count; index++) { + const startedAt = new Date(Date.now() - startedAgoMs + index * 1_000); + await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "assignment", triggerDetail: "system", status, + createdAt: new Date(startedAt.getTime() - 1_000), startedAt, finishedAt: status === "running" ? null : new Date(startedAt.getTime() + 500), + contextSnapshot: { issueId: other.id, taskId: other.id, wakeReason: "issue_assigned" } }); + } + } + + const callbacks = createHeartbeatQueueTestDependencies(db); + callbacks.getAgent.mockImplementation(async id => (await db.select().from(agents).where(eq(agents.id, id)))[0] ?? null); + callbacks.instanceSettings.getExperimental.mockResolvedValue({} as never); + callbacks.setWakeupStatus.mockImplementation(async (id, status, values) => { + if (id) await db.update(agentWakeupRequests).set({ ...values, status }).where(eq(agentWakeupRequests.id, id)); + }); + const admission = createWakeQueue(db, { + resolveResponsibleUserId: async () => "person-1", + getRoutineEnv: async () => ({ routineId: null, env: null, responsibleUserId: null }), + resolveSessionBeforeForWakeup: async () => null, + wakeAdmissionHelpers: { filterZombieCoalesceTarget: callbacks.filterZombieCoalesceTarget, + mergeCoalescedContextSnapshot, shouldDeferFollowupWakeForSameIssue, shouldQueueFollowupForRunningIssueWake }, + recovery: { escalateStrandedAssignedIssue: async () => undefined, escalateStrandedRecoveryIssueInPlace: async () => undefined }, + }); + callbacks.wakeQueue.createAdmissionTransactionScope.mockImplementation(admission.createAdmissionTransactionScope); + callbacks.wakeQueue.admitWakeBehindIssueExecution.mockImplementation(admission.admitWakeBehindIssueExecution); + const queue = createHeartbeatQueue(db, callbacks); + const cancel = createStalledQueuedRunCancel(callbacks); + const sweep = (beforeCancel?: () => Promise) => assignedWorkSweep(db, { + enqueueWakeup: (id, opts) => queue.enqueueWakeup(id, opts) as Promise<{ id: string } | null>, + cancelStalledQueuedRun: async scope => { await beforeCancel?.(); return cancel(scope); }, + }); + async function olderRun(createdAgoMs: number, startedAgoMs: number) { + const startedAt = new Date(Date.now() - startedAgoMs); + await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "automation", triggerDetail: "system", status: "succeeded", + createdAt: new Date(Date.now() - createdAgoMs), startedAt, finishedAt: new Date(startedAt.getTime() + 500), + contextSnapshot: { issueId: other.id, taskId: other.id, wakeReason: "issue_continuation_needed" } }); + } + const runsOfTask = async () => db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), + sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`)).orderBy(heartbeatRuns.createdAt); + const readRun = async (id: string) => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)))[0]!; + const continuations = async () => db.select().from(activityLog).where(and(eq(activityLog.entityId, issue.id), + eq(activityLog.action, ASSIGNED_WORK_CONTINUED_ACTION))); + return { companyId, agentId, issue, queuedRun, otherStarts, olderRun, queue, sweep, runsOfTask, readRun, continuations }; + } + + async function replacementOf(f: Awaited>, stalledId: string) { + const runs = await f.runsOfTask(); + return runs.filter(run => run.id !== stalledId); + } + + it("cancels an automatic run the claim keeps refusing and queues exactly one claimable replacement", async () => { + const f = await seed(); + const stalled = await f.queuedRun(); + await f.otherStarts(OLDEST_DISPATCH_SHARE + 1, QUEUED_RUN_AGING_MS / 2); + expect(await f.queue.claimQueuedRun(await f.readRun(stalled.id))).toBeNull(); + expect((await f.readRun(stalled.id)).status).toBe("queued"); + + const sweep = f.sweep(); + expect((await sweep.listSuspects({ companyId: f.companyId }, null, 50)).map(row => row.id)).toContain(f.issue.id); + const check = await sweep.checkIssue({ companyId: f.companyId, issueId: f.issue.id }); + expect(check).toMatchObject({ verdict: { kind: "violation", reason: "queued_run_stalled", runId: stalled.id }, outcome: "replaced_stalled_run" }); + + expect(await f.readRun(stalled.id)).toMatchObject({ status: "cancelled", errorCode: "queued_run_stalled" }); + const replacements = await replacementOf(f, stalled.id); + expect(replacements).toHaveLength(1); + const replacement = replacements[0]!; + expect(replacement.contextSnapshot).toMatchObject({ wakeReason: "issue_continuation_needed", previousRunId: stalled.id, + replacesStalledRunId: stalled.id }); + expect(replacement.retryOfRunId).toBeNull(); + const [entry] = await f.continuations(); + expect(entry?.details).toMatchObject({ outcome: "queued", violation: "queued_run_stalled" }); + expect(String((entry?.details as Record).message)).not.toMatch(/queued_run_stalled|[0-9a-f]{8}-/); + + const current = await f.readRun(replacement.id); + if (current.status === "queued") expect(await f.queue.claimQueuedRun(current)).not.toBeNull(); + expect((await f.readRun(replacement.id)).status).toBe("running"); + expect(await sweep.checkIssue({ companyId: f.companyId, issueId: f.issue.id })) + .toMatchObject({ verdict: { kind: "ok", clause: "live_run", runId: replacement.id } }); + }, 60_000); + + it("counts only the starts that passed the run over after it became its agent's oldest queued run", async () => { + const f = await seed(); + const waiting = await f.queuedRun(); + await f.olderRun(4 * 60 * MINUTE, 50 * MINUTE); + await f.otherStarts(OLDEST_DISPATCH_SHARE + 2, 55 * MINUTE); + await f.olderRun(3 * 60 * MINUTE, 20 * MINUTE); + const sweep = f.sweep(); + expect(await sweep.checkIssue({ companyId: f.companyId, issueId: f.issue.id })) + .toMatchObject({ verdict: { kind: "ok", clause: "live_run", runId: waiting.id } }); + await f.otherStarts(OLDEST_DISPATCH_SHARE, 10 * MINUTE); + expect(await sweep.checkIssue({ companyId: f.companyId, issueId: f.issue.id })) + .toMatchObject({ verdict: { kind: "ok", clause: "live_run", runId: waiting.id } }); + expect((await f.readRun(waiting.id)).status).toBe("queued"); + await f.otherStarts(1, 5 * MINUTE); + expect(await sweep.checkIssue({ companyId: f.companyId, issueId: f.issue.id })) + .toMatchObject({ verdict: { reason: "queued_run_stalled", runId: waiting.id }, outcome: "replaced_stalled_run" }); + }, 60_000); + + it("keeps the run when a person's message joins it after the check", async () => { + const f = await seed(); + const stalled = await f.queuedRun(); + await f.otherStarts(OLDEST_DISPATCH_SHARE + 1, QUEUED_RUN_AGING_MS / 2); + const joined = await f.sweep(async () => { + await f.queue.enqueueWakeup(f.agentId, { source: "on_demand", triggerDetail: "manual", reason: "issue_commented", + payload: { issueId: f.issue.id }, requestedByActorType: "user", requestedByActorId: "person-1", + contextSnapshot: { issueId: f.issue.id, taskId: f.issue.id, wakeReason: "issue_commented" } }); + }).checkIssue({ companyId: f.companyId, issueId: f.issue.id }); + expect(joined).toMatchObject({ verdict: { reason: "queued_run_stalled" }, outcome: "changed" }); + expect((await f.readRun(stalled.id)).status).toBe("queued"); + expect(await f.runsOfTask()).toHaveLength(1); + expect(await f.continuations()).toHaveLength(0); + const personWakes = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.runId, stalled.id), + eq(agentWakeupRequests.requestedByActorType, "user"))); + expect(personWakes).toHaveLength(1); + }, 60_000); + + it("does nothing when the run starts after the check", async () => { + const f = await seed(); + const stalled = await f.queuedRun(); + await f.otherStarts(OLDEST_DISPATCH_SHARE + 1, QUEUED_RUN_AGING_MS / 2); + const started = await f.sweep(async () => { + await db.update(heartbeatRuns).set({ status: "running", startedAt: new Date() }).where(eq(heartbeatRuns.id, stalled.id)); + }).checkIssue({ companyId: f.companyId, issueId: f.issue.id }); + expect(started).toMatchObject({ outcome: "changed" }); + expect((await f.readRun(stalled.id)).status).toBe("running"); + expect(await f.runsOfTask()).toHaveLength(1); + expect(await f.continuations()).toHaveLength(0); + }, 60_000); + + it("leaves a run that waits behind a full agent", async () => { + const f = await seed(); + const waiting = await f.queuedRun(); + await f.otherStarts(1, QUEUED_RUN_AGING_MS * 3, "running"); + const sweep = f.sweep(); + expect((await sweep.listSuspects({ companyId: f.companyId }, null, 50)).map(row => row.id)).not.toContain(f.issue.id); + expect(await sweep.checkIssue({ companyId: f.companyId, issueId: f.issue.id })) + .toMatchObject({ verdict: { kind: "ok", clause: "live_run", runId: waiting.id }, outcome: "ok" }); + expect((await f.readRun(waiting.id)).status).toBe("queued"); + expect(await f.runsOfTask()).toHaveLength(1); + expect(await f.continuations()).toHaveLength(0); + }, 60_000); + + it("reports a person's refused Retry and never cancels it", async () => { + const f = await seed(); + const retry = await f.queuedRun({ person: true }); + await f.otherStarts(OLDEST_DISPATCH_SHARE + 1, QUEUED_RUN_AGING_MS / 2); + const warn = vi.spyOn(logger, "warn"); + const check = await f.sweep().checkIssue({ companyId: f.companyId, issueId: f.issue.id }); + expect(check).toMatchObject({ verdict: { kind: "violation", reason: "queued_run_stalled", runId: retry.id }, outcome: "left_person_work" }); + expect((await f.readRun(retry.id)).status).toBe("queued"); + expect(await f.runsOfTask()).toHaveLength(1); + expect(await f.continuations()).toHaveLength(0); + expect(warn.mock.calls.filter(([payload]) => (payload as { event?: string })?.event === "assigned_work_invariant.queued_run_stalled")) + .toHaveLength(1); + }, 60_000); + + it("leaves a replacement that stalls again, warning once and looking again after the long backoff", async () => { + const f = await seed(); + const original = randomUUID(); + const replacement = await f.queuedRun({ context: { previousRunId: original, replacesStalledRunId: original } }); + await f.otherStarts(OLDEST_DISPATCH_SHARE + 1, QUEUED_RUN_AGING_MS / 2); + const warn = vi.spyOn(logger, "warn"); + const sweep = f.sweep(); + const before = Date.now(); + const first = await sweep.checkIssue({ companyId: f.companyId, issueId: f.issue.id }); + const second = await sweep.checkIssue({ companyId: f.companyId, issueId: f.issue.id }); + for (const check of [first, second]) { + expect(check).toMatchObject({ verdict: { reason: "queued_run_stalled", runId: replacement.id }, outcome: "left_stalled_replacement" }); + expect(check!.retryAt!.getTime()).toBeGreaterThanOrEqual(before + CONTINUATION_CAP_WINDOW_MS); + } + expect((await f.readRun(replacement.id)).status).toBe("queued"); + expect(await f.runsOfTask()).toHaveLength(1); + expect(await f.continuations()).toHaveLength(0); + expect(warn.mock.calls.filter(([payload]) => (payload as { event?: string })?.event === "assigned_work_invariant.queued_run_stalled")) + .toHaveLength(1); + }, 60_000); + + it("produces one replacement when two servers sweep the task at once", async () => { + const f = await seed(); + const stalled = await f.queuedRun(); + await f.otherStarts(OLDEST_DISPATCH_SHARE + 1, QUEUED_RUN_AGING_MS / 2); + const checks = await Promise.all([f.sweep(), f.sweep()].map(sweep => sweep.checkIssue({ companyId: f.companyId, issueId: f.issue.id }))); + expect(checks.filter(check => check?.outcome === "replaced_stalled_run")).toHaveLength(1); + expect((await f.readRun(stalled.id)).status).toBe("cancelled"); + expect(await replacementOf(f, stalled.id)).toHaveLength(1); + expect((await f.continuations()).filter(entry => (entry.details as Record).outcome === "queued")).toHaveLength(1); + }, 60_000); +}); diff --git a/server/src/__tests__/helpers/heartbeat-queue.ts b/server/src/__tests__/helpers/heartbeat-queue.ts index bea3762cb23..dc3496919c7 100644 --- a/server/src/__tests__/helpers/heartbeat-queue.ts +++ b/server/src/__tests__/helpers/heartbeat-queue.ts @@ -28,9 +28,9 @@ export function createHeartbeatQueueTestDependencies(db: Db): QueueTestDependenc getAgent: vi.fn(async () => { throw new Error("Unexpected agent lookup"); }), options: {}, // Same compare-and-set contract as the run lifecycle: write only from one of fromStatuses. - setRunStatusFromLive: vi.fn(async (id, status, fromStatuses, patch) => { + setRunStatusFromLive: vi.fn(async (id, status, fromStatuses, patch, _report, condition) => { const [updated] = await db.update(heartbeatRuns).set({ ...patch, status }) - .where(and(eq(heartbeatRuns.id, id), inArray(heartbeatRuns.status, fromStatuses))).returning(); + .where(and(eq(heartbeatRuns.id, id), inArray(heartbeatRuns.status, fromStatuses), ...(condition ? [condition] : []))).returning(); if (updated) return { run: updated, updated: true as const }; const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)); return { run: current ?? null, updated: false as const }; diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 85387808006..6dc109a8501 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -2,6 +2,7 @@ import { registerHeldPreparationResumer } from "./interrupted-preparation-hold-s import { registerCompletedTurnCleanupResumer } from "./native-runtime/completed-turn-cleanup-hold-settlement.js"; import { endStorageOutageWaitOn, reportStorageOutageWaitsTo, runInOwnStorageScope } from "../storage/storage-unavailable.js"; import { createStorageWaitNotice } from "./heartbeat/storage-wait-notice.js"; +import { createStalledQueuedRunCancel } from "./heartbeat/stalled-queued-run.js"; import { resolveProviderApiPlacement } from "./native-runtime/provider-api-placement.js"; import { isProcessRunAdmissionClosed } from "./heartbeat/run-admission.js"; import { isYieldingRunStarts, refreshRunStartYieldIfStale, type RunStartHandover } from "./run-start-handover.js"; @@ -1041,6 +1042,7 @@ export function heartbeatService( // Bound at call time: the heartbeat recovery module is created below. settleLostRunProcess: scope => recoverActiveExecutions(scope), startSavedWorkRecovery: source => startOverdueWorkspaceRecovery(db, environmentRuntime, resumeRemoteStopComments, source), + cancelStalledQueuedRun: createStalledQueuedRunCancel({ setRunStatusFromLive, setWakeupStatus, appendRunEvent, releaseIssueExecutionAndPromote }), settleExplicitContinuationRetry: releaseIssueExecutionAndPromote, scheduleRecoveryRetry: async (runId, outcome) => { const [run] = await db diff --git a/server/src/services/heartbeat/queued-run-line.ts b/server/src/services/heartbeat/queued-run-line.ts index 097cf00fbd9..29d3c6bcd28 100644 --- a/server/src/services/heartbeat/queued-run-line.ts +++ b/server/src/services/heartbeat/queued-run-line.ts @@ -40,7 +40,7 @@ export function issueRunPriorityRank(priority: string | null | undefined) { } } -async function listPersonOriginSourceRunIds(db: Db, companyId: string, sourceRunIds: string[]) { +export async function listPersonOriginSourceRunIds(db: Db, companyId: string, sourceRunIds: string[]) { const uuidPattern = "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$"; const result = await db.execute<{ root_id: string }>(sql` with recursive chain(root_id, run_id, depth) as ( diff --git a/server/src/services/heartbeat/stalled-queued-run.ts b/server/src/services/heartbeat/stalled-queued-run.ts new file mode 100644 index 00000000000..82b4865b01e --- /dev/null +++ b/server/src/services/heartbeat/stalled-queued-run.ts @@ -0,0 +1,32 @@ +import { sql } from "drizzle-orm"; +import type { HeartbeatQueueDependencies } from "./queue.js"; + +export const QUEUED_RUN_STALLED_CODE = "queued_run_stalled"; +export const QUEUED_RUN_STALLED_MESSAGE = + "Paperclip replaced this run with a new one: it had waited over an hour to start while its agent started other work."; + +type StalledRunLifecycle = Pick; + +export function stalledRunUnchangedCondition(fence: { version: string; attachedWakes: number }) { + return sql`"heartbeat_runs".xmin::text = ${fence.version} + and (select count(*)::int from agent_wakeup_requests attached where attached.company_id = "heartbeat_runs".company_id + and (attached.run_id = "heartbeat_runs".id or attached.id = "heartbeat_runs".wakeup_request_id)) = ${fence.attachedWakes}`; +} + +export function createStalledQueuedRunCancel(lifecycle: StalledRunLifecycle) { + return async function cancelStalledQueuedRun(scope: { companyId: string; runId: string; version: string; attachedWakes: number }): + Promise { + const finishedAt = new Date(); + const write = await lifecycle.setRunStatusFromLive(scope.runId, "cancelled", ["queued"], { + finishedAt, errorCode: QUEUED_RUN_STALLED_CODE, error: QUEUED_RUN_STALLED_MESSAGE, + }, undefined, stalledRunUnchangedCondition(scope)); + if (!write.updated || !write.run) return false; + const cancelled = write.run; + await lifecycle.setWakeupStatus(cancelled.wakeupRequestId, "cancelled", { finishedAt, error: QUEUED_RUN_STALLED_MESSAGE }); + await lifecycle.appendRunEvent(cancelled, { eventType: "lifecycle", stream: "system", level: "warn", + message: QUEUED_RUN_STALLED_MESSAGE, payload: { reason: QUEUED_RUN_STALLED_CODE } }); + await lifecycle.releaseIssueExecutionAndPromote(cancelled, { suppressImmediateRecovery: true }); + return true; + }; +} diff --git a/server/src/services/recovery/assigned-work-invariant.ts b/server/src/services/recovery/assigned-work-invariant.ts index 518287850bd..317f69c5516 100644 --- a/server/src/services/recovery/assigned-work-invariant.ts +++ b/server/src/services/recovery/assigned-work-invariant.ts @@ -69,6 +69,7 @@ export const VIOLATION_REASONS = [ * recovery queue has not started that recovery within its bound. Repaired * by starting the recovery, never by a wake its hold would refuse. */ "saved_work_recovery_overdue", + "queued_run_stalled", ] as const; export type ViolationReason = (typeof VIOLATION_REASONS)[number]; @@ -141,7 +142,8 @@ export type AssignedWorkFacts = { awaitingFirstMessage: boolean; }; /** Non-terminal runs (queued, running, scheduled_retry) for this task. */ - activeRuns: Array<{ id: string; status: string; scheduledRetryAt: Date | null; owner: RunOwnerVerdict }>; + activeRuns: Array<{ id: string; status: string; scheduledRetryAt: Date | null; owner: RunOwnerVerdict; + queuedStall?: { personWork: boolean; replacement: boolean; queuedSince: Date } | null }>; latestRun: { id: string; status: string; @@ -228,8 +230,10 @@ export function evaluateAssignedWork(facts: AssignedWorkFacts, now: Date = new D // 1. A live run with an owner that is not proven gone. An unprovable owner // fails closed: during a deploy overlap another server may still own it. + const stalledQueued = issue.status === "blocked" ? undefined + : facts.activeRuns.find(run => run.status === "queued" && run.queuedStall); const live = facts.activeRuns.find(run => (run.status === "queued" || run.status === "running") - && run.owner !== "gone" && run.owner !== "stalled"); + && run.owner !== "gone" && run.owner !== "stalled" && !(stalledQueued && run.queuedStall)); if (live) { if (issue.status === "blocked" && !waitingReason(facts, now)) { return { kind: "violation", reason: "blocked_with_live_run", runId: live.id, stale: false }; @@ -245,7 +249,7 @@ export function evaluateAssignedWork(facts: AssignedWorkFacts, now: Date = new D if (stalled) return { kind: "violation", reason: "native_run_stalled", runId: stalled.id, stale: false }; // 2. A durable intent that starts a run without anyone acting. - const intent = pendingIntent(facts, now); + const intent = pendingIntent(facts, now, Boolean(stalledQueued)); if (intent) return { kind: "ok", clause: "pending_intent", intent }; // A run that ended moments ago is still settling: its own failure path may @@ -261,6 +265,8 @@ export function evaluateAssignedWork(facts: AssignedWorkFacts, now: Date = new D // them in time. A continuation wake would only wait behind the same hold. if (facts.savedWorkRecovery) return { kind: "violation", reason: "saved_work_recovery_overdue", stale: false }; + if (stalledQueued) return { kind: "violation", reason: "queued_run_stalled", runId: stalledQueued.id, stale: false }; + if (issue.status === "blocked") { const blockedAt = issue.blockedTransitionAt ?? issue.updatedAt; const stale = now.getTime() - blockedAt.getTime() > STALE_BLOCK_MS; @@ -275,7 +281,7 @@ export function evaluateAssignedWork(facts: AssignedWorkFacts, now: Date = new D return { kind: "violation", reason: "no_pending_path", stale: false }; } -function pendingIntent(facts: AssignedWorkFacts, now: Date): PendingIntent | null { +function pendingIntent(facts: AssignedWorkFacts, now: Date, queuedRunStalled = false): PendingIntent | null { if (facts.pendingWakes > 0) return "queued_wake"; if (facts.activeRuns.some(run => run.status === "scheduled_retry")) return "scheduled_retry"; if (facts.nativeRecoveryPending) return "native_recovery"; @@ -285,7 +291,7 @@ function pendingIntent(facts: AssignedWorkFacts, now: Date): PendingIntent | nul if (facts.savedWorkRecovery && savedWorkRecoveryPending(facts.savedWorkRecovery, now)) return "saved_work_recovery"; const interruption = facts.interruption; if (interruption?.responsibility === "system") { - if (interruption.state === "working" && !(facts.latestRun?.finishedAt + if (interruption.state === "working" && !queuedRunStalled && !(facts.latestRun?.finishedAt && now.getTime() - facts.latestRun.finishedAt.getTime() > SYSTEM_RECOVERY_WORKING_BOUND_MS)) return "system_recovery_working"; if (interruption.state === "retry_scheduled" && interruption.nextRetryAt && Date.parse(interruption.nextRetryAt) > now.getTime()) return "scheduled_retry"; @@ -395,6 +401,8 @@ export function continuationActivityMessage(reason: ViolationReason): string { return "No longer blocked: this task is already queued to continue, so nothing needs you."; case "saved_work_recovery_overdue": return "Paperclip started recovering this task's saved files. The task continues once they are recovered."; + case "queued_run_stalled": + return "Continued automatically: this task had waited over an hour to start while its agent started other work, so Paperclip queued it again."; default: return "Continued automatically after an interruption. Nothing was running or scheduled, so Paperclip queued the task again."; } diff --git a/server/src/services/recovery/assigned-work-sweep.ts b/server/src/services/recovery/assigned-work-sweep.ts index c5c34a68819..880445c9921 100644 --- a/server/src/services/recovery/assigned-work-sweep.ts +++ b/server/src/services/recovery/assigned-work-sweep.ts @@ -31,6 +31,8 @@ import { CHAT_CONTROL_RECOVERY_STOP_CODE, CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE import { onboardingFirstTaskAwaitingUserSql } from "./onboarding-first-task-wait.js"; import { workspaceRecoverySourceEligible } from "../workspace-recovery-source.js"; import { PENDING_CONTINUATION_BOUND_MS } from "../recovery-continuation-pacing.js"; +import { OLDEST_DISPATCH_SHARE, QUEUED_RUN_AGING_MS } from "../heartbeat/queued-run-dispatch-order.js"; +import { listPersonOriginSourceRunIds } from "../heartbeat/queued-run-line.js"; import { attributeBlock, BLOCK_ATTRIBUTION_WINDOW_MS, blockedStatusDetailsSql } from "../interruption-block-release.js"; import { ASSIGNED_WORK_CONTINUED_ACTION, taskAutomaticAttempts } from "./task-automatic-attempts.js"; import { isReattachPhaseActive, reattachPhaseRecheckAt } from "../heartbeat/run-admission.js"; @@ -45,6 +47,7 @@ import { savedWorkRecoveryDueSince, CONTINUATION_REFUSED_MESSAGE, CONTINUATION_REVIEW_MESSAGE, + CONTINUATION_CAP_WINDOW_MS, evaluateAssignedWork, STALE_BLOCK_MS, NATIVE_STALL_MS, @@ -94,6 +97,7 @@ export type AssignedWorkDeps = { /** Starts a saved-work recovery the recovery queue has not reached in time. */ startSavedWorkRecovery?: (source: { companyId: string; issueId: string; runId: string; actionId: string }) => Promise<"started" | "in_flight" | "busy" | "not_eligible">; + cancelStalledQueuedRun?: (scope: { companyId: string; runId: string; version: string; attachedWakes: number }) => Promise; /** Shared owner proof, injectable for tests. */ proveOwner?: (stamp: LocalOwnerStamp) => Promise; now?: () => Date; @@ -102,10 +106,13 @@ export type AssignedWorkDeps = { export type AssignedWorkCheck = { verdict: AssignedWorkVerdict; outcome: "ok" | "settling" | "repaired" | "settled_lost_run" | "deferred_rate_limit" | "deferred_cap" | "deferred_reattach_phase" | "stale_reported" | "left_to_hold_owner" | "left_to_native_recovery" | "changed" | "enqueue_refused" - | "waiting_on_recovery" | "started_saved_work_recovery" | "saved_work_recovery_busy"; + | "waiting_on_recovery" | "started_saved_work_recovery" | "saved_work_recovery_busy" + | "replaced_stalled_run" | "left_person_work" | "left_stalled_replacement"; retryAt: Date | null; }; +const rowsOf = (value: unknown): T[] => Array.isArray(value) ? value as T[] : ((value as { rows?: T[] })?.rows ?? []); + const json = (value: unknown): Record => value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; @@ -133,7 +140,8 @@ function createPassLog() { } else if (verdict.kind === "violation") { pass.violations[verdict.reason] = (pass.violations[verdict.reason] ?? 0) + 1; } - if (check.outcome === "repaired" || check.outcome === "settled_lost_run" || check.outcome === "started_saved_work_recovery") pass.repaired += 1; + if (check.outcome === "repaired" || check.outcome === "settled_lost_run" || check.outcome === "started_saved_work_recovery" + || check.outcome === "replaced_stalled_run") pass.repaired += 1; if (check.outcome === "deferred_rate_limit" || check.outcome === "deferred_cap") pass.deferred += 1; if (check.outcome === "stale_reported") pass.stale += 1; if (!timer) { timer = setTimeout(flush, 5_000); timer.unref?.(); } @@ -147,6 +155,7 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) { const proveOwner = deps.proveOwner ?? ((stamp: LocalOwnerStamp) => proveOwnerGone(stamp, { ...currentProcessProbe(), readOnly: true })); const passLog = createPassLog(); const repairTimes: number[] = []; + const reportedStalledRuns = new Set(); const sweepTreeControl = issueTreeControlService(db); async function runOwner(run: RunRow, at: Date, proveOwners = true): Promise { @@ -194,8 +203,11 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) { const [activeRuns, latestRuns, wakes, chatWakes, nativePending, handoffs, goals, interactions, approvalRows, blockerRows, resolvedBlockerRows, childRows, actionMap, agentRows, companyRows, onboardingRows, blockActivity, interruptions, savedWorkRows] = await Promise.all([ - database.select(runColumns).from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), - inArray(heartbeatRuns.status, NON_TERMINAL_RUN_STATUSES), sql`${runIssue} = any(${idParam})`)), + database.select({ ...runColumns, createdAt: heartbeatRuns.createdAt, stalledQueued: queuedRunStalledSql(aliasedRun("heartbeat_runs"), at), + replacesStalledRunId: sql`${heartbeatRuns.contextSnapshot}->>'replacesStalledRunId'`, + explicitUserContinuation: sql`jsonb_typeof(${heartbeatRuns.contextSnapshot}->'explicitUserContinuation') = 'object'` }) + .from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), + inArray(heartbeatRuns.status, NON_TERMINAL_RUN_STATUSES), sql`${runIssue} = any(${idParam})`)), // The newest run per task, read through the (company, issueId, created) index. database.execute(sql`select requested.id as "issueId", latest.* from unnest(${idParam}) as requested(id) @@ -219,6 +231,8 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) { or (${agentWakeupRequests.status} = 'claimed' and (${agentWakeupRequests.runId} is not null or coalesce(${agentWakeupRequests.claimedAt}, ${agentWakeupRequests.createdAt}) > ${at.toISOString()}::timestamptz - ${`${DEFERRED_WAKE_PENDING_BOUND_MS} milliseconds`}::interval)))`, + sql`not exists (select 1 from heartbeat_runs wake_run where wake_run.id = ${agentWakeupRequests.runId} + and ${queuedRunStalledSql(aliasedRun("wake_run"), at)})`, sql`${agentWakeupRequests.payload}->>'issueId' = any(${idParam})`)) .groupBy(sql`${agentWakeupRequests.payload}->>'issueId'`), database.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, companyId), inArray(issues.id, ids), @@ -302,7 +316,6 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) { .orderBy(desc(issueRecoveryActions.createdAt)), ]); - const rowsOf = (value: unknown): T[] => Array.isArray(value) ? value as T[] : ((value as { rows?: T[] })?.rows ?? []); const strandedRunIds = [...actionMap.values()].filter(action => action.kind === STRANDED_KIND && action.ownerType === "board") .map(action => action.evidence?.latestRunId).filter((id): id is string => typeof id === "string" && /^[0-9a-f-]{36}$/i.test(id)); const strandedRuns = new Map((strandedRunIds.length ? await database.select({ id: heartbeatRuns.id, @@ -366,11 +379,16 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) { const blockActivityByIssue = new Map(rowsOf<{ issueId: string; actorType: string; actorId: string; agentId: string | null; details: unknown }>(blockActivity) .map(row => [row.issueId, row])); + const stalledQueuedIds = activeRuns.filter(row => row.stalledQueued).map(row => row.id); + const personQueuedIds = new Set(stalledQueuedIds.length ? await listPersonOriginSourceRunIds(database, companyId, stalledQueuedIds) : []); const ownedRunsByIssue = new Map(); for (const issue of issueRows) { const ownedRuns: AssignedWorkFacts["activeRuns"] = []; - for (const run of activeRuns.filter(row => row.issueId === issue.id) as RunRow[]) { - ownedRuns.push({ id: run.id, status: run.status, scheduledRetryAt: run.scheduledRetryAt, owner: stalledRunIds.has(run.id) && run.status === "running" ? "stalled" : await runOwner(run, at, proveOwners) }); + for (const run of activeRuns.filter(row => row.issueId === issue.id)) { + ownedRuns.push({ id: run.id, status: run.status, scheduledRetryAt: run.scheduledRetryAt, + owner: stalledRunIds.has(run.id) && run.status === "running" ? "stalled" : await runOwner(run as RunRow, at, proveOwners), + queuedStall: run.stalledQueued ? { personWork: personQueuedIds.has(run.id) || Boolean(run.explicitUserContinuation), + replacement: Boolean(run.replacesStalledRunId), queuedSince: new Date(run.createdAt) } : null }); } ownedRunsByIssue.set(issue.id, ownedRuns); } @@ -474,6 +492,12 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) { and r.status in ('queued', 'running', 'scheduled_retry') and not (r.status = 'running' and r.runtime_mode = 'legacy' and r.controller_lease_expires_at is not null and r.controller_lease_expires_at < ${at.toISOString()}::timestamptz))`; + const liveUnstalledRun = sql`exists (select 1 from heartbeat_runs r where r.company_id = ${issues.companyId} + and (r.context_snapshot->>'issueId') = ${issues.id}::text + and r.status in ('queued', 'running', 'scheduled_retry') + and not (r.status = 'running' and r.runtime_mode = 'legacy' and r.controller_lease_expires_at is not null + and r.controller_lease_expires_at < ${at.toISOString()}::timestamptz) + and not ${queuedRunStalledSql(aliasedRun("r"), at)})`; return db.select({ id: issues.id, companyId: issues.companyId, updatedAt: issues.updatedAt }).from(issues).where(and( scope.companyId ? eq(issues.companyId, scope.companyId) : undefined, scope.issueId ? eq(issues.id, scope.issueId) : undefined, @@ -489,13 +513,15 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) { sql`(${issues.monitorNextCheckAt} is null or ${issues.monitorNextCheckAt} <= ${at.toISOString()}::timestamptz)`, // A legacy run whose controller lease lapsed may have lost its owner; // the predicate judges it with the owner proof. Other active runs are live. - sql`(${issues.status} = 'blocked' or not ${liveRun})`, + sql`(${issues.status} = 'blocked' or not ${liveUnstalledRun})`, sql`((${issues.status} = 'blocked' and ${liveRun}) or not exists (select 1 from agent_wakeup_requests w where w.company_id = ${issues.companyId} and (w.payload->>'issueId') = ${issues.id}::text and (w.status = 'queued' or (w.status = 'deferred_issue_execution' and w.created_at > ${new Date(at.getTime() - DEFERRED_WAKE_PENDING_BOUND_MS).toISOString()}::timestamptz) or (w.status = 'claimed' and (w.run_id is not null - or coalesce(w.claimed_at, w.created_at) > ${new Date(at.getTime() - DEFERRED_WAKE_PENDING_BOUND_MS).toISOString()}::timestamptz)))))`, + or coalesce(w.claimed_at, w.created_at) > ${new Date(at.getTime() - DEFERRED_WAKE_PENDING_BOUND_MS).toISOString()}::timestamptz))) + and not exists (select 1 from heartbeat_runs wake_run where wake_run.id = w.run_id + and ${queuedRunStalledSql(aliasedRun("wake_run"), at)})))`, sql`not exists (select 1 from issue_thread_interactions t where t.company_id = ${issues.companyId} and t.issue_id = ${issues.id} and t.status = 'pending' and ${interactionNotSupersededSql(sql`t.company_id`, sql`t.issue_id::text`, sql`t.created_at`)})`, @@ -573,6 +599,14 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) { await deps.settleLostRunProcess?.({ companyId: issue.companyId, runId: verdict.runId! }); return { verdict, outcome: "settled_lost_run", retryAt: new Date(at.getTime() + 90_000) }; } + if (verdict.reason === "queued_run_stalled") { + const stall = facts.activeRuns.find(run => run.id === verdict.runId)?.queuedStall ?? null; + if (!stall || stall.personWork || stall.replacement || !deps.cancelStalledQueuedRun) { + reportStalledRun(issue, verdict.runId!, stall, at); + return { verdict, outcome: stall?.personWork ? "left_person_work" : "left_stalled_replacement", + retryAt: new Date(at.getTime() + CONTINUATION_CAP_WINDOW_MS) }; + } + } // A small process-wide budget keeps a post-deploy backlog from starting // hundreds of runs at once. Deferred tasks come back on their deadline. const minuteAgo = at.getTime() - 60_000; @@ -584,6 +618,7 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) { if (claimed.kind !== "claimed") return { verdict, outcome: claimed.kind === "capped" ? "deferred_cap" : claimed.kind === "changed" ? "changed" : "deferred_rate_limit", retryAt: claimed.kind === "changed" ? null : claimed.retryAt }; repairTimes.push(at.getTime()); + if (claimed.reason === "queued_run_stalled") return replaceStalledRun(issue, claimed, claimed.verdict.runId!); // A block that interrupted a review goes back to review; the review's own // recovery wakes its participant. No agent work starts from here. if (claimed.restoredTo === "in_review") { @@ -621,6 +656,40 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) { return { verdict: claimed.verdict, outcome: run ? "repaired" : "enqueue_refused", retryAt: claimed.nextCheckAt }; } + function reportStalledRun(issue: IssueRow, runId: string, stall: NonNullable | null, at: Date) { + if (reportedStalledRuns.has(runId)) return; + if (reportedStalledRuns.size >= 1_000) reportedStalledRuns.clear(); + reportedStalledRuns.add(runId); + logger.warn({ event: "assigned_work_invariant.queued_run_stalled", companyId: issue.companyId, issueId: issue.id, runId, + agentId: issue.assigneeAgentId, queuedForMs: stall ? at.getTime() - stall.queuedSince.getTime() : null, + personWork: stall?.personWork ?? null, replacement: stall?.replacement ?? null }, + "queued run its agent keeps passing over was left in place"); + } + + async function replaceStalledRun(issue: IssueRow, claimed: Claimed, runId: string): Promise { + const cancelled = !claimed.stalledFence ? false : await deps.cancelStalledQueuedRun!({ companyId: issue.companyId, runId, + ...claimed.stalledFence }).catch(error => { + logger.warn({ err: error, companyId: issue.companyId, issueId: issue.id, runId }, "assigned-work sweep could not cancel a stalled queued run"); + return false; + }); + if (!cancelled) { + await db.delete(activityLog).where(eq(activityLog.id, claimed.activityId)); + return { verdict: claimed.verdict, outcome: "changed", retryAt: null }; + } + const run = await deps.enqueueWakeup(issue.assigneeAgentId!, { + source: "automation", triggerDetail: "system", reason: "issue_continuation_needed", + payload: { issueId: issue.id, mutation: "assigned_work_invariant" }, + requestedByActorType: "system", requestedByActorId: null, + contextSnapshot: { issueId: issue.id, taskId: issue.id, wakeReason: "issue_continuation_needed", + source: ASSIGNED_WORK_CONTINUATION_SOURCE, assignedWorkViolation: claimed.reason, previousRunId: runId, replacesStalledRunId: runId }, + }).catch(error => { + logger.warn({ err: error, companyId: issue.companyId, issueId: issue.id }, "assigned-work replacement for a stalled run was not admitted"); + return null; + }); + await settleContinuationActivity(claimed, claimed.reason, run ? "queued" : "not_started"); + return { verdict: claimed.verdict, outcome: run ? "replaced_stalled_run" : "enqueue_refused", retryAt: claimed.nextCheckAt }; + } + async function continueFromSavedWork(issue: IssueRow, facts: AssignedWorkFacts, at: Date): Promise { const source = facts.latestRun; if (!source?.finishedAt || !["failed", "timed_out", "interrupted", "cancelled"].includes(source.status) @@ -660,7 +729,8 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) { type Claimed = { kind: "claimed"; nextCheckAt: Date; activityId: string; publications: ActivityPublication[]; verdict: Extract; reason: ViolationReason; restoredTo: "todo" | "in_progress" | "in_review" | null; restoredStatusVersion: number | null; - blockedTransitionAt: Date | null; unblockDescriptor: IssueRow["unblockDescriptor"]; blockedOwnerNotifiedAt: Date | null }; + blockedTransitionAt: Date | null; unblockDescriptor: IssueRow["unblockDescriptor"]; blockedOwnerNotifiedAt: Date | null; + stalledFence: { version: string; attachedWakes: number } | null }; /** * The durable, serialized claim. Under the task's row lock it loads the @@ -679,10 +749,22 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) { .where(and(eq(issues.companyId, issue.companyId), eq(issues.id, issue.id))).for("update"); if (!current || current.status !== issue.status || current.statusVersion !== issue.statusVersion || current.assigneeAgentId !== issue.assigneeAgentId || current.assigneeUserId || current.hiddenAt) return { kind: "changed" as const }; + const fences = rowsOf<{ id: string; version: string; attachedWakes: number }>(await tx.execute(sql` + select r.id, r.xmin::text as version, (select count(*)::int from agent_wakeup_requests w + where w.company_id = r.company_id and (w.run_id = r.id or w.id = r.wakeup_request_id)) as "attachedWakes" + from heartbeat_runs r where r.company_id = ${issue.companyId} and r.status = 'queued' + and (r.context_snapshot->>'issueId') = ${issue.id}`)); const fresh = (await loadFacts(issue.companyId, [current], { database: tx as unknown as Db, proveOwners: false })).get(current.id)!; const verdict = evaluateAssignedWork(fresh, at); + let stalledFence: Claimed["stalledFence"] = null; if (verdict.kind !== "violation" || verdict.stale || verdict.reason === "run_owner_gone" || verdict.reason === "native_run_stalled" || (current.status === "blocked" && fresh.activeRecoveryAction && !fresh.activeRecoveryAction.continuedBySweep)) return { kind: "changed" as const }; + if (verdict.reason === "queued_run_stalled") { + const stall = fresh.activeRuns.find(run => run.id === verdict.runId)?.queuedStall; + const fence = fences.find(row => row.id === verdict.runId); + if (!stall || stall.personWork || stall.replacement || !deps.cancelStalledQueuedRun || !fence) return { kind: "changed" as const }; + stalledFence = { version: fence.version, attachedWakes: Number(fence.attachedWakes) }; + } const attempts = await taskAutomaticAttempts(tx as unknown as Db, issue.companyId, issue.id, at); const recent = attempts.continuations; const allowance = continuationAllowance(attempts.started, at, attempts.all); @@ -717,7 +799,7 @@ export function assignedWorkSweep(db: Db, deps: AssignedWorkDeps) { attempt: recent.length + 1 }, }, publications); const next = continuationAllowance([at, ...attempts.started], at, [at, ...attempts.all]); - return { kind: "claimed" as const, activityId: activity.id, publications, verdict, reason: verdict.reason, + return { kind: "claimed" as const, activityId: activity.id, publications, verdict, reason: verdict.reason, stalledFence, restoredTo, restoredStatusVersion, blockedTransitionAt: current.blockedTransitionAt, unblockDescriptor: current.unblockDescriptor, blockedOwnerNotifiedAt: current.blockedOwnerNotifiedAt, nextCheckAt: next.allowed ? new Date(at.getTime() + 90_000) : next.retryAt }; @@ -862,6 +944,25 @@ function interactionNotSupersededSql(companyId: SQLWrapper, issueIdText: SQLWrap and retry.created_at > ${createdAt})`; } +type StalledRunColumns = { id: SQLWrapper; companyId: SQLWrapper; agentId: SQLWrapper; status: SQLWrapper; createdAt: SQLWrapper }; + +function queuedRunStalledSql(run: StalledRunColumns, at: Date) { + const agedBefore = new Date(at.getTime() - QUEUED_RUN_AGING_MS).toISOString(); + return sql`(${run.status} = 'queued' and ${run.createdAt} <= ${agedBefore}::timestamptz + and not exists (select 1 from heartbeat_runs older where older.company_id = ${run.companyId} + and older.agent_id = ${run.agentId} and older.status = 'queued' and older.created_at < ${run.createdAt}) + and (select count(*) from (select 1 from heartbeat_runs started where started.company_id = ${run.companyId} + and started.agent_id = ${run.agentId} and started.id <> ${run.id} + and started.started_at > greatest(${run.createdAt} + ${`${QUEUED_RUN_AGING_MS} milliseconds`}::interval, + (select max(coalesce(prior.started_at, prior.finished_at)) from heartbeat_runs prior + where prior.company_id = ${run.companyId} and prior.agent_id = ${run.agentId} and prior.created_at < ${run.createdAt} + and (prior.started_at >= ${run.createdAt} or (prior.started_at is null and prior.finished_at >= ${run.createdAt})))) + limit ${OLDEST_DISPATCH_SHARE + 1}) passed_over) > ${OLDEST_DISPATCH_SHARE})`; +} + +const aliasedRun = (alias: string): StalledRunColumns => ({ id: sql.raw(`${alias}.id`), companyId: sql.raw(`${alias}.company_id`), + agentId: sql.raw(`${alias}.agent_id`), status: sql.raw(`${alias}.status`), createdAt: sql.raw(`${alias}.created_at`) }); + /** The status a block replaced, from the activity entry that blocked the task: * the normal update route records it as `_previous.status`, recovery as * `previousStatus`, the native status committer as `fromStatus`. */ diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts index 7bc2608bb8f..8b9a3ad4a5f 100644 --- a/server/src/services/recovery/service.ts +++ b/server/src/services/recovery/service.ts @@ -1016,6 +1016,7 @@ export function recoveryService( settleLostRunProcess?: (scope: { companyId: string; runId: string; onDeferredDeadline?: (at: number) => void }) => Promise; /** Starts a saved-work recovery the recovery queue has not reached in time. */ startSavedWorkRecovery?: AssignedWorkDeps["startSavedWorkRecovery"]; + cancelStalledQueuedRun?: AssignedWorkDeps["cancelStalledQueuedRun"]; beforeOrphanedRunTerminalWrite?: (runId: string) => Promise; beforeStaleLockClear?: (issueId: string) => Promise; afterStrandedLatestRunRead?: (input: { issueId: string; runId: string | null }) => Promise; @@ -6902,6 +6903,7 @@ export function recoveryService( liveRunExecutions: deps.liveRunExecutions, settleLostRunProcess: deps.settleLostRunProcess, startSavedWorkRecovery: deps.startSavedWorkRecovery, + cancelStalledQueuedRun: deps.cancelStalledQueuedRun, }); return { From ce5f10d534e036a1195d48697930b5a7c1c757b8 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 19:40:42 -0600 Subject: [PATCH 03/11] fix(wake-queue): start a person's saved message after a legacy run is continued from saved work (#21518) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When an agent run fails and cannot resume exactly, Paperclip continues the task in a fresh run from its last saved work > - If a person sent a message while the task was blocked, the fresh-continuation fallback lets that saved message start the next run (`delivery: saved_input`) and does not queue its own run > - The release step that promotes the saved message skips the "needs reconciliation" hold for a native run that the fallback superseded, but not for a legacy run > - So for a legacy run the task moves to To do, the person's message stays saved, and no run starts until a watchdog finds the task idle > - This pull request applies the same superseded-run rule to legacy runs in the release step > - The benefit is that a person's message after a failed run starts the next run at once, for legacy runs as for native runs ## Linked Issues or Issue Description **What happened?** A person sent a message on a task whose last legacy run failed. Paperclip continued the task from saved work and moved it to To do. The message stayed saved. No run started until the assigned-work watchdog queued one later. **Steps to reproduce** 1. Let a legacy run fail with an active recovery hold, so the task is Blocked. 2. Post a comment as a person. The wake is saved as `deferred_issue_execution`. 3. Let the fresh-continuation fallback continue the task. It records `delivery: saved_input`. 4. Run the deferred receipt and stranded-queue passes. **Expected behavior** One run starts and carries the person's message. **Actual behavior** No run starts. The release step returns `released` because `legacyExecutionNeedsReconciliation` is true for the superseded legacy run. ## What Changed - `server/src/modules/wake-queue/adapters/postgres.ts`: the release pre-drain check does not ask for legacy reconciliation when the run is superseded by a fresh continuation. This matches the existing native rule in the same function. - `server/src/services/fresh-continuation-fallback.test.ts`: a red/green test for native and legacy runs. A person's comment goes through the real wake path while the task is blocked, the task continues from saved work, and exactly one run must start with the comment. ## Verification - Red: before the fix, the legacy case fails. The saved wake stays `deferred_issue_execution` and no run starts. The native case passes. - Green: after the fix, both cases pass. - `pnpm verify` for the affected selection. ## Risks - Low risk. The change applies only when the fresh-continuation fallback has already marked the run superseded. That decision already records that the old run is not replayed, and the successor runs as a fresh turn. Other callers of `legacyExecutionNeedsReconciliationWithEvidence` (retry scheduling, recovery) are not changed, so a superseded run still cannot be retried automatically. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../modules/wake-queue/adapters/postgres.ts | 2 +- .../fresh-continuation-fallback.test.ts | 27 +++++++++++++++++++ 2 files changed, 28 insertions(+), 1 deletion(-) diff --git a/server/src/modules/wake-queue/adapters/postgres.ts b/server/src/modules/wake-queue/adapters/postgres.ts index 7e92e667918..471e7a00088 100644 --- a/server/src/modules/wake-queue/adapters/postgres.ts +++ b/server/src/modules/wake-queue/adapters/postgres.ts @@ -1375,7 +1375,7 @@ export function createPostgresWakeQueueAdapter(db: Db, deps: WakeQueuePostgresAd issueStatus: issueRow?.status ?? "", hasAssigneeUser: Boolean(issueRow?.assigneeUserId), assigneeAgentMatchesRunAgent: issueRow?.assigneeAgentId === run.agentId, - legacyExecutionNeedsReconciliation: evidence.kind === "complete" && + legacyExecutionNeedsReconciliation: evidence.kind === "complete" && !agentRunSupersededByFreshContinuation(run) && await legacyExecutionNeedsReconciliationWithEvidence(tx as unknown as Db, evidence.run), // An operator stop never promotes old queued work by itself. The // next explicit wake adopts those messages atomically when it diff --git a/server/src/services/fresh-continuation-fallback.test.ts b/server/src/services/fresh-continuation-fallback.test.ts index d49d055867d..e4f65754f30 100644 --- a/server/src/services/fresh-continuation-fallback.test.ts +++ b/server/src/services/fresh-continuation-fallback.test.ts @@ -995,6 +995,33 @@ suite("fresh-continuation fallback", () => { .toMatchObject({ kind: "continued", delivery: taken ? "recorded" : "saved_input" }); }, 60_000); + it.each(["native", "legacy"] as const)("starts a run for the person's message once the task continues from saved work (%s)", async runtime => { + const f = await unfinishedRun({ runtime, errorCode: runtime === "native" ? "native_provider_terminal_failed" : "process_lost" }); + const [comment] = await db.insert(issueComments).values({ companyId, issueId: f.issueId, authorType: "user", + authorUserId: userId, body: "try again" } as typeof issueComments.$inferInsert).returning(); + const heartbeat = heartbeatService(db); + await heartbeat.wakeup(agentId, { source: "automation", triggerDetail: "system", reason: "issue_commented", + payload: { issueId: f.issueId, commentId: comment!.id, mutation: "comment" }, + requestedByActorType: "user", requestedByActorId: userId, + contextSnapshot: { issueId: f.issueId, taskId: f.issueId, commentId: comment!.id, wakeCommentId: comment!.id, + source: "issue.comment", wakeReason: "issue_commented" } }); + const wakes = () => db.select().from(agentWakeupRequests).where(sql`${agentWakeupRequests.payload}->>'issueId' = ${f.issueId}`); + expect((await wakes()).map(w => w.status)).toEqual(["deferred_issue_execution"]); + expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })) + .toMatchObject({ kind: "continued", delivery: "saved_input" }); + await occupySlot(); + await deliver(); + await heartbeat.resumeQueuedRuns(); + const runs = await db.select().from(heartbeatRuns).where(and( + sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${f.issueId}`, sql`${heartbeatRuns.id} <> ${f.runId}`)); + const saved = await wakes(); + expect(runs, JSON.stringify({ issue: await issue(f.issueId), + wakes: saved.map(w => ({ status: w.status, error: w.error, runId: w.runId, payload: w.payload })) })).toHaveLength(1); + expect(runs[0]!.contextSnapshot?.wakeCommentIds).toContain(comment!.id); + expect(saved).toHaveLength(1); + expect(saved[0]!.runId).toBe(runs[0]!.id); + }, 60_000); + it.each([ { title: "a pending approval", setup: "approval" as const, errorCode: "native_continuation_requires_reconciliation" }, { title: "a credential the person has not shared", setup: "none" as const, errorCode: "configuration_incomplete" }, From 01f764fe61bea76f5fdcbe70c9ef300747747cfb Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 19:41:04 -0600 Subject: [PATCH 04/11] test(evals): pin the case clock in the original-deadline rejection tests (#21501) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner support suite tests the chat catalog observers that grade agent work > - One observer test fails at random on unrelated pull requests > - The test simulates a late poll result with a fixed clock jump from the fixture time. The observer computes its deadline from a later clock read. If 2 ms or more pass between the two reads, the jump does not pass the deadline. > - This pull request pins the clock to the fixture time for the deadline rejection cases, as the sibling deadline tests do > - The benefit is a deterministic test that still fails when the late-deadline guard is removed ## Linked Issues or Issue Description **What happened?** `tests/runner-e2e/chat-catalog-domains/identity-plan-decision-observer.test.ts` > "rejects late await original deadline without resetting it" failed in the "ci / Runner support suite" job of PR #21418 (run 43503383519, 2026-11-13 20:38Z). The error was: `expected [Function] to throw error including 'CONTROL_SOURCE_NOT_SETTLED' but got 'CONTROL_AFTER_SOURCE_SETTLED'`. PR #21418 changes only a plan document and a benchmark script. **Expected behavior** The test passes every time. The result must not depend on how fast the test worker runs. **Steps to reproduce** 1. Add `await new Promise(r => setTimeout(r, 2));` before the `act` call in the "late await" case. 2. Run `node node_modules/vitest/vitest.mjs run --config tests/runner-e2e/vitest.config.ts chat-catalog-domains/identity-plan-decision-observer.test.ts`. 3. The test fails with the CI error. With a gap of 0 or 1 ms it passes. **Paperclip version or commit** `master` at 2a8b485140. **Relevant logs or output** The test sets the clock to `f.now + 180_001` after the poll loads. `f.now` is the fixture's `Date.now()`. The observer sets `deadlineAt = Date.now() + 180_000` later, in `act`. The accept check is `Date.now() < deadlineAt`. It is true when `act` reads the clock 2 ms or more after the fixture. A loaded CI worker can create that gap. ## What Changed - In the four "rejects … original deadline without resetting it" cases, `Date.now` now returns the fixture time until the test jumps it. The "late await" jump is then exactly 1 ms past the observer's deadline. The other three cases fail at the budget check as before. ## Verification - Red: with a 2 ms or 5 ms gap before `act`, the unchanged test fails with the CI error (2 of 2 runs). With a 0 or 1 ms gap, it passes. - Green: with the fix, the test passes with gaps of 0, 2, 5 and 50 ms (39/39 tests each). - Mutation: with the fix, I removed `Date.now() < deadlineAt &&` from the accept check in `identity-plan-review.ts`. The "late await" test fails. The assertion still detects the regression that it guards. - `pnpm verify` selected broad CI coverage (exit 2) because the file is a global CI input. Required CI supplies that coverage. ## Risks - Low risk. The change affects only one test file. It does not change product code or assertions. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use and shell execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- .../identity-plan-decision-observer.test.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/runner-e2e/chat-catalog-domains/identity-plan-decision-observer.test.ts b/tests/runner-e2e/chat-catalog-domains/identity-plan-decision-observer.test.ts index 180c7883ee0..4f54cb0b066 100644 --- a/tests/runner-e2e/chat-catalog-domains/identity-plan-decision-observer.test.ts +++ b/tests/runner-e2e/chat-catalog-domains/identity-plan-decision-observer.test.ts @@ -124,7 +124,7 @@ it("retains the original180-second phase when the case has more time remaining", expect(port.deadline).toBe(f.now + 180_000); }); it.each(["missing", "invalid", "expired", "late await"])("rejects %s original deadline without resetting it", async fault => { - const f = fixture("questions-and-approvals.review-plan", fault === "invalid" ? NaN : fault === "expired" ? -1 : 600_000); + const f = fixture("questions-and-approvals.review-plan", fault === "invalid" ? NaN : fault === "expired" ? -1 : 600_000); vi.spyOn(Date, "now").mockReturnValue(f.now); const ctx: ChatCatalogContext = fault === "missing" ? { ...f.ctx, observationDeadlineAt: undefined } : f.ctx; if (fault === "late await") port.afterLoadNow = f.now + 180_001; await expect(planReviewIdentityCases[0]!.act!(ctx, f.state)).rejects.toThrow(fault === "late await" ? "CONTROL_SOURCE_NOT_SETTLED" : "Original plan-review observation budget"); From e6df84a9928edabcad760e409c8f7f69ce58aa3b Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 19:41:10 -0600 Subject: [PATCH 05/11] fix(ci): count a step cancelled as "A task was canceled." as runner loss when the log shows the shutdown (#21514) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - PR CI runs on AWS spot runners, and `ci-runner-loss-rerun.yml` re-runs a PR's first attempt once when every failed job lost its runner (#21291) > - On 2026-11-13 a watcher was added to the runner image that stops the GitHub runner about 10 seconds after AWS announces a spot reclaim, so the lost job ends in about 28 seconds instead of up to 10 minutes > - When the runner is stopped this way, the cancelled step's check annotation is usually "The operation was canceled.", but a step that was still starting up gets "A task was canceled." instead > - The classifier only recognised the first text, so it counted such a job as a real failure and refused the re-run, even though the job log has the shutdown-signal line > - This pull request recognises both cancellation texts, still only together with shutdown evidence in the log > - The benefit is that lost jobs stopped early in a step are re-run automatically too ## Linked Issues or Issue Description Refs #21291 **What happened?** In the first hour after the spot-notice watcher went live, a `ci / Build` job was stopped 13 s after it started. Its job log has `##[error]The runner has received a shutdown signal...`, but its only failure annotation was `A task was canceled.`. `classifyJob` treated that annotation as a real failure, so the whole run was not re-run. **Expected behavior** A job whose log shows the shutdown signal is counted as runner loss, whichever cancellation text the step was given. **Actual behavior** It is counted as a real failure, and the run stays red until a person re-runs it. ## What Changed - `.github/scripts/runner-loss-rerun.mjs`: the cancellation annotation check accepts "The operation was canceled." and "A task was canceled.". The log check still decides. A cancellation with no shutdown or lost-communication line in the log is still a real failure, and its evidence now quotes the actual annotation text. - `.github/scripts/tests/runner-loss-rerun.test.mjs`: a test for "A task was canceled." with and without shutdown evidence. - `doc/DEVELOPING.md`: the runner-loss paragraph names both texts. ## Verification - Red: the new test fails against the unchanged script (25 pass, 1 fail). - Green: `node --test .github/scripts/tests/runner-loss-rerun.test.mjs`: 26 pass. - `node --test '.github/scripts/tests/*.test.mjs'` (the `policy` job's command): 808 pass. - The workflow runs from `master`, so this PR's own CI cannot exercise the new behaviour; the tests are the evidence. ## Risks - A job cancelled as "A task was canceled." for another reason is still treated as a real failure unless its log has the shutdown or lost-communication line, so a person cancelling a run still blocks the automatic re-run. - No change to triggers, permissions, the once-only rule or the stale-head checks. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- .github/scripts/runner-loss-rerun.mjs | 13 +++++++------ .github/scripts/tests/runner-loss-rerun.test.mjs | 12 ++++++++++++ doc/DEVELOPING.md | 4 ++-- 3 files changed, 21 insertions(+), 8 deletions(-) diff --git a/.github/scripts/runner-loss-rerun.mjs b/.github/scripts/runner-loss-rerun.mjs index f581cdbbca5..0384e1b0866 100644 --- a/.github/scripts/runner-loss-rerun.mjs +++ b/.github/scripts/runner-loss-rerun.mjs @@ -27,9 +27,10 @@ export const AGGREGATE_JOBS = new Set(["ci / verify", "ci / e2e"]); // would move the whole run to GitHub-hosted runners, so leave it to a human. export const GATE_JOB_PREFIX = "ci / gate / "; export const RERUN_CONCLUSIONS = new Set(["failure", "cancelled"]); -// A shutdown signal often reaches the check run only as this annotation; the -// signal itself is logged as an error line in the job log. -const CANCELLED_MESSAGE = "The operation was canceled."; +// A shutdown signal often reaches the check run only as one of these +// annotations; the signal itself is logged as an error line in the job log. +const CANCELLED_MESSAGES = new Set(["The operation was canceled.", "A task was canceled."]); +const isCancellation = (annotation) => CANCELLED_MESSAGES.has(annotation.message); export function createGithubClient({ token, fetcher = fetch, apiUrl = "https://api.github.com" }) { if (!token) throw new Error("GITHUB_TOKEN is required"); @@ -83,12 +84,12 @@ export async function classifyJob(job, { annotations, readLog }) { const isRunnerLoss = (annotation) => RUNNER_LOSS_PATTERN.test(annotation.message ?? ""); const match = failures.find(isRunnerLoss); if (match) return { lost: true, evidence: firstLine(match.message) }; - if (failures.some((annotation) => annotation.message === CANCELLED_MESSAGE) && logShowsRunnerLoss(await readLog(job))) { + if (failures.some(isCancellation) && logShowsRunnerLoss(await readLog(job))) { return { lost: true, evidence: "job log: runner received a shutdown signal or lost communication" }; } - const real = failures.find((annotation) => annotation.message !== CANCELLED_MESSAGE); + const real = failures.find((annotation) => !isCancellation(annotation)); if (real) return { lost: false, evidence: firstLine(real.message) }; - if (failures.length > 0) return { lost: false, evidence: `${CANCELLED_MESSAGE} (no runner-loss error in the job log)` }; + if (failures.length > 0) return { lost: false, evidence: `${firstLine(failures[0].message)} (no runner-loss error in the job log)` }; return { lost: false, evidence: "no failure annotation" }; } diff --git a/.github/scripts/tests/runner-loss-rerun.test.mjs b/.github/scripts/tests/runner-loss-rerun.test.mjs index 819f160881e..4873c508ad5 100644 --- a/.github/scripts/tests/runner-loss-rerun.test.mjs +++ b/.github/scripts/tests/runner-loss-rerun.test.mjs @@ -344,6 +344,18 @@ test("the client refuses to follow pagination to another host", async () => { await assert.rejects(client.paginate("/repos/paperclipai/paperclip/commits/x/pulls"), /Refusing to call https:\/\/example.com/); }); +test("a step cancelled as \"A task was canceled.\" counts as lost only with shutdown evidence", async () => { + const annotations = [{ annotation_level: "failure", message: "A task was canceled." }]; + const lost = await classifyJob({ id: 1 }, { + annotations, + readLog: async () => "##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n##[error]A task was canceled.\n", + }); + assert.equal(lost.lost, true); + const noSignal = await classifyJob({ id: 1 }, { annotations, readLog: async () => "##[error]A task was canceled.\n" }); + assert.equal(noSignal.lost, false); + assert.match(noSignal.evidence, /A task was canceled\./); +}); + test("classification is conservative about cancelled jobs", async () => { const cancelled = [{ annotation_level: "failure", message: "The operation was canceled." }]; const noSignal = await classifyJob({ id: 1 }, { annotations: cancelled, readLog: async () => "##[error]The operation was canceled.\n" }); diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 24a26f1d11f..e07b05ba2dc 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -195,8 +195,8 @@ failed jobs of a PR's first CI attempt once when every failed or cancelled job lost its runner. A spot reclaim often ends the job, and the run, as `cancelled`. A lost runner is recognized by the check-run annotation "lost communication with the server" or "received a shutdown signal". When a job has -the annotation "The operation was canceled.", the job log is checked for the -same messages. This evidence counts even if the job also has other error +the annotation "The operation was canceled." or "A task was canceled.", the job +log is checked for the same messages. This evidence counts even if the job also has other error annotations, so a test that really failed just before a reclaim gets one wasted re-run. A cancelled job without this evidence, such as a job a person cancelled, counts as a real failure. The `verify` and `e2e` aggregates are not From e67f3bd63a99e68f9fc56b427747b65b1d0dc865 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 19:42:08 -0600 Subject: [PATCH 06/11] test(environments): add an opt-in live Daytona proof for warm sandbox expiry (#21463) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run in remote sandboxes, such as Daytona, through reusable environment leases > - #21410 made Paperclip stop a warm sandbox when its warm window ends > - Its tests use a real database but a mocked provider stop, and staging had no warm reusable sandboxes to observe > - This pull request adds an opt-in live test that proves the behavior against real Daytona > - The benefit is a repeatable live proof of warm expiry, resume, and recovery after a restart ## Linked Issues or Issue Description Refs #21410 **What happened?** #21410 has no live provider proof. Staging creates no warm reusable sandboxes today, so the deployed change cannot be observed there. **Expected behavior** A repeatable live check shows that a warm sandbox stays started inside its window and stops after it. It also shows that the next turn resumes the same sandbox, and that a restarted cleanup worker still stops the sandbox on time. **Steps to reproduce** 1. Set `PAPERCLIP_LIVE_WARM_EXPIRY=1` and `DAYTONA_API_KEY`. 2. Run `pnpm exec vitest run src/__tests__/warm-sandbox-expiry.live.test.ts` in `server`. **Additional context** The test makes no model turn. It drives the production environment runtime and the cleanup worker against real Daytona sandboxes. ## What Changed - New `server/src/__tests__/warm-sandbox-expiry.live.test.ts`. The test is skipped unless `PAPERCLIP_LIVE_WARM_EXPIRY=1` is set. It uses: - the real Daytona plugin and real sandboxes; - `environmentRuntime.acquireRunLease` with a 2 minute warm window; - `releaseRunLeases(..., "keep_running")`; - the production `registerEnvironmentCleanupWork` on a real `createDeliveryWorkCoordinator`, which must wake itself at the deadline. - The test checks the following: - The sandbox is started 20 s before the window ends. - After the window, the worker stops the sandbox. The sandbox is stopped, not deleted, and the lease is released as stopped and kept. - The second turn resumes the same sandbox (`acquisition: resumed`). - A new database client, a freshly loaded plugin module, and a new cleanup worker start in the middle of the window. That worker stops the sandbox after the window ends, not before. - At the end, the test deletes only the sandboxes whose labels match its own company and environment. ## Verification - Live run on 2026-11-13 against the Daytona org, window 120 s, sandbox `170bb215…`: - 18:39:32.624Z turn 1 lease retained; window ends 18:41:32.624Z - 18:41:13Z sandbox `started` (20 s before the window ends) - 18:41:36Z sandbox `stopped`, auto-delete -1: the worker stopped it 4 s after the window ended - 18:41:39Z turn 2 resumed the same sandbox (`resumed`, `started`); window ends 18:43:39.549Z - 18:42:10Z restart: a new worker booted; sandbox still `started` - 18:43:45Z sandbox `stopped`, 5 s after the window ended - 18:43:45Z fixture sandbox deleted; Daytona returns 404 - Provider calls: one acquire, one resume, two stops, no destroy before cleanup - Without the env flag the test is skipped (1 skipped). ## Risks - Low risk. The change adds a test file only, and the test is skipped by default. It makes real provider calls only when someone opts in. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, Daytona API). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../warm-sandbox-expiry.live.test.ts | 229 ++++++++++++++++++ 1 file changed, 229 insertions(+) create mode 100644 server/src/__tests__/warm-sandbox-expiry.live.test.ts diff --git a/server/src/__tests__/warm-sandbox-expiry.live.test.ts b/server/src/__tests__/warm-sandbox-expiry.live.test.ts new file mode 100644 index 00000000000..d4a022959b1 --- /dev/null +++ b/server/src/__tests__/warm-sandbox-expiry.live.test.ts @@ -0,0 +1,229 @@ +// Opt-in live provider proof for warm sandbox expiry. Not a Product E2E +// workflow and no model turn: real Daytona sandboxes, the real Daytona plugin, +// the production environment runtime (acquire, warm retain, resume), and the +// production environment cleanup worker on a real delivery coordinator, which +// must wake itself at the warm-window deadline. +// +// Prerequisite: the Daytona provider is outside the root workspace, so install +// and build it on its own first (this test loads its dist/ and its own SDK): +// pnpm --dir packages/plugins/sandbox-providers/daytona install +// pnpm --dir packages/plugins/sandbox-providers/daytona build +// Then, from server/: +// PAPERCLIP_LIVE_WARM_EXPIRY=1 DAYTONA_API_KEY=... \ +// pnpm exec vitest run src/__tests__/warm-sandbox-expiry.live.test.ts +import { randomUUID } from "node:crypto"; +import { createRequire } from "node:module"; +import { access, writeFile } from "node:fs/promises"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { agents, companies, createDb, environmentLeases, environments, heartbeatRuns, issues, plugins } from "@paperclipai/db"; +import type { Environment } from "@paperclipai/shared"; +import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { nextEnvironmentCleanupAt } from "../services/environment-cleanup-state.js"; +import { registerEnvironmentCleanupWork } from "../services/environment-cleanup-work.js"; +import { environmentRuntimeService } from "../services/environment-runtime.js"; +import { heartbeatService } from "../services/heartbeat.js"; +import type { PluginWorkerManager } from "../services/plugin-worker-manager.js"; + +const enabled = process.env.PAPERCLIP_LIVE_WARM_EXPIRY === "1"; +const describeLive = enabled ? describe : describe.skip; +const WINDOW_MS = Number(process.env.PAPERCLIP_LIVE_WARM_EXPIRY_WINDOW_MS ?? 120_000); +const pluginUrl = new URL("../../../packages/plugins/sandbox-providers/daytona/dist/plugin.js", import.meta.url).href; +const manifestUrl = new URL("../../../packages/plugins/sandbox-providers/daytona/dist/manifest.js", import.meta.url).href; +const sleep = (ms: number) => new Promise(resolve => setTimeout(resolve, Math.max(0, ms))); +const iso = (ms: number) => new Date(ms).toISOString(); + +describeLive("live Daytona warm sandbox expiry", () => { + let temporary: Awaited>; + let db: ReturnType; + let plugin: any; + let manifest: any; + let daytona: any; + let environment: Environment; + const sandboxIds = new Set(); + const ids = { company: randomUUID(), agent: randomUUID(), environment: randomUUID(), issue: randomUUID(), plugin: randomUUID() }; + const config = { provider: "daytona", reuseLease: true, runnerLifecycleMode: "warm", runnerIdleTimeoutMs: WINDOW_MS, + timeoutMs: 180_000, autoStopInterval: 30, autoArchiveInterval: 60, autoDeleteInterval: 120 }; + const timeline: Array> = []; + const methods: string[] = []; + const supported = ["environmentValidateConfig", "environmentProbe", "environmentAcquireLease", "environmentResumeLease", + "environmentReleaseLease", "environmentStopLease", "environmentDestroyLease", "environmentRealizeWorkspace", + "environmentExecute", "environmentInspectLease"]; + const manager = () => ({ + isRunning: () => true, + getWorker: () => ({ supportedMethods: supported }), + call: async (_id: string, method: string, input: any) => { + methods.push(method); + const handler = `on${method[0]!.toUpperCase()}${method.slice(1)}`; + if (typeof plugin[handler] !== "function") throw new Error(`Unsupported provider operation ${method}`); + return await plugin[handler](input); + }, + }) as unknown as PluginWorkerManager; + + async function provider(id: string) { + const sandbox = await daytona.get(id); + await sandbox.refreshData(); + return { state: sandbox.state as string, autoDeleteInterval: sandbox.autoDeleteInterval as number, + labels: sandbox.labels as Record, updatedAt: sandbox.updatedAt as string }; + } + async function note(step: string, providerLeaseId: string, extra: Record = {}) { + const observed = await provider(providerLeaseId); + const entry = { at: new Date().toISOString(), step, sandboxState: observed.state, + sandboxAutoDelete: observed.autoDeleteInterval, ...extra }; + timeline.push(entry); + console.log(JSON.stringify(entry)); + return observed; + } + async function run(status: string) { + const id = randomUUID(); + await db.insert(heartbeatRuns).values({ id, companyId: ids.company, agentId: ids.agent, issueId: ids.issue, + invocationSource: "on_demand", status, nextEventSeq: 1 }); + return id; + } + async function leaseRows() { + return db.select().from(environmentLeases).where(eq(environmentLeases.companyId, ids.company)); + } + /** One warm turn: acquire the reusable lease through the production runtime, + * then end the run the way a succeeded warm run does (keep_running). */ + async function warmTurn(label: string) { + const runId = await run("running"); + const runtime = environmentRuntimeService(db, { pluginWorkerManager: manager() }); + const acquired = await runtime.acquireRunLease({ companyId: ids.company, environment, issueId: ids.issue, + agentId: ids.agent, heartbeatRunId: runId, persistedExecutionWorkspace: null, adapterType: "paperclip_runner", warmRunnerIdleTimeoutMs: WINDOW_MS }); + const providerLeaseId = acquired.lease.providerLeaseId!; + sandboxIds.add(providerLeaseId); + await note(`${label}: acquired`, providerLeaseId, { providerLeaseId, + acquisition: (acquired.lease.metadata?.sandboxLeaseAcquisition as { outcome?: string } | undefined)?.outcome, + recordedWindowMs: acquired.lease.metadata?.warmSandboxIdleTimeoutMs }); + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, runId)); + const released = await runtime.releaseRunLeases(runId, "released", undefined, "keep_running"); + expect(released).toHaveLength(1); + const [retained] = (await leaseRows()).filter(row => row.heartbeatRunId === runId); + expect(retained).toMatchObject({ status: "retained", leasePolicy: "reuse_by_environment", providerLeaseId }); + const windowEndsAt = retained!.lastUsedAt.getTime() + WINDOW_MS; + await note(`${label}: run ended, lease retained`, providerLeaseId, { releasedAt: iso(retained!.lastUsedAt.getTime()), + windowEndsAt: iso(windowEndsAt) }); + return { runId, providerLeaseId, windowEndsAt }; + } + /** The production cleanup worker on a real coordinator, as the server + * registers it at boot. Returns a stop function. */ + async function startCleanupWorker(workerDb: ReturnType) { + const heartbeat = heartbeatService(workerDb, { environmentRuntime: environmentRuntimeService(workerDb, { pluginWorkerManager: manager() }) }); + const coordinator = createDeliveryWorkCoordinator({ owner: workerDb, canRun: () => true, onError: error => { console.error("cleanup worker error", (error as Error).name); } }); + const sweeps: string[] = []; + const worker = registerEnvironmentCleanupWork(workerDb, coordinator, { + sweepPending: async backoffMs => { sweeps.push(new Date().toISOString()); await heartbeat.sweepPendingCleanupLeases({ backoffMs }); }, + canRun: () => true, + minimumSpacingMs: 1_000, + }); + await worker.ready; + return { sweeps, nextWakeAt: () => coordinator.nextWakeAt(), stop: async () => { worker.stop(); await coordinator.stop(); } }; + } + async function waitForState(providerLeaseId: string, state: string, deadlineMs: number) { + while (Date.now() < deadlineMs) { + const observed = await provider(providerLeaseId); + if (observed.state === state) return Date.now(); + await sleep(3_000); + } + return null; + } + + beforeAll(async () => { + if (!process.env.DAYTONA_API_KEY) throw new Error("DAYTONA_API_KEY required"); + for (const url of [pluginUrl, manifestUrl]) { + await access(new URL(url)).catch(() => { + throw new Error("Build the Daytona provider first: pnpm --dir packages/plugins/sandbox-providers/daytona install && pnpm --dir packages/plugins/sandbox-providers/daytona build"); + }); + } + temporary = await startEmbeddedPostgresTestDatabase("live-warm-expiry-"); + db = createDb(temporary.connectionString); + plugin = (await import(pluginUrl)).default.definition; + manifest = (await import(manifestUrl)).default; + const require = createRequire(new URL("../../../packages/plugins/sandbox-providers/daytona/package.json", import.meta.url)); + const { Daytona } = require("@daytonaio/sdk"); + daytona = new Daytona({ apiKey: process.env.DAYTONA_API_KEY }); + await db.insert(companies).values({ id: ids.company, name: "Disposable warm expiry", issuePrefix: "LWE" }); + await db.insert(agents).values({ id: ids.agent, companyId: ids.company, name: "No provider turn", adapterType: "paperclip_runner" }); + await db.insert(plugins).values({ id: ids.plugin, pluginKey: manifest.id, packageName: "@paperclipai/plugin-daytona", + version: manifest.version, apiVersion: 1, categories: ["automation"], manifestJson: manifest, status: "ready", installOrder: 1 }); + await db.insert(environments).values({ id: ids.environment, name: `Warm expiry ${ids.environment.slice(0, 8)}`, driver: "sandbox", config }); + environment = (await db.select().from(environments).where(eq(environments.id, ids.environment)))[0] as unknown as Environment; + await db.insert(issues).values({ id: ids.issue, companyId: ids.company, title: "Warm sandbox expiry", assigneeAgentId: ids.agent }); + }, 120_000); + + afterAll(async () => { + const leftBehind: string[] = []; + for (const id of sandboxIds) { + try { + const sandbox = await daytona.get(id); + if (sandbox.labels?.["paperclip-company-id"] !== ids.company || sandbox.labels?.["paperclip-environment-id"] !== ids.environment) { + throw new Error("Fixture cleanup ownership mismatch"); + } + await sandbox.delete(60); + timeline.push({ at: new Date().toISOString(), step: "cleanup: sandbox deleted", providerLeaseId: id }); + } catch (error) { + leftBehind.push(id); + timeline.push({ at: new Date().toISOString(), step: "cleanup failed", providerLeaseId: id, error: (error as Error).message }); + } + } + if (process.env.PAPERCLIP_LIVE_WARM_EXPIRY_EVIDENCE) { + await writeFile(process.env.PAPERCLIP_LIVE_WARM_EXPIRY_EVIDENCE, JSON.stringify({ windowMs: WINDOW_MS, timeline, methods }, null, 2)); + } + await temporary?.cleanup(); + if (leftBehind.length) { + // The stop path disables provider auto-delete, so a sandbox left here + // would never be reaped: fail loudly with the exact IDs to delete. + console.error(`Live fixture sandboxes left behind: ${leftBehind.join(", ")}`); + throw new Error(`Live fixture cleanup failed for ${leftBehind.length} sandbox(es): ${leftBehind.join(", ")}`); + } + }, 180_000); + + it("keeps a warm sandbox inside its window, stops it after, resumes it, and stops it again after a restart", async () => { + const worker = await startCleanupWorker(db); + try { + // Turn 1. + const first = await warmTurn("turn 1"); + expect(await nextEnvironmentCleanupAt(db)).toBe(first.windowEndsAt); + // Inside the window: the worker sleeps until the deadline; the sandbox stays started. + await sleep(first.windowEndsAt - Date.now() - 20_000); + expect((await note("turn 1: 20s before window end", first.providerLeaseId)).state).toBe("started"); + // Past the window: the worker wakes by itself and stops the exact sandbox. + const stoppedAt = await waitForState(first.providerLeaseId, "stopped", first.windowEndsAt + 90_000); + const afterStop = await note("turn 1: stopped by cleanup worker", first.providerLeaseId, + { secondsAfterWindowEnd: stoppedAt === null ? null : Math.round((stoppedAt - first.windowEndsAt) / 1000) }); + expect(stoppedAt).not.toBeNull(); + expect(stoppedAt!).toBeGreaterThanOrEqual(first.windowEndsAt); + expect(afterStop.autoDeleteInterval).toBe(-1); + const [kept] = (await leaseRows()).filter(row => row.heartbeatRunId === first.runId); + expect(kept).toMatchObject({ status: "released", cleanupStatus: "success", metadata: { remoteExecutionTermination: { state: "stopped" } } }); + + // Turn 2 on the same task resumes the same sandbox. + const second = await warmTurn("turn 2"); + expect(second.providerLeaseId).toBe(first.providerLeaseId); + expect((await provider(second.providerLeaseId)).state).toBe("started"); + } finally { + await worker.stop(); + } + + // Restart mid-window: a fresh database client, a freshly loaded plugin + // module, and a new cleanup worker, as at server boot. + const [retained] = (await leaseRows()).filter(row => row.status === "retained"); + const windowEndsAt = retained!.lastUsedAt.getTime() + WINDOW_MS; + await sleep(Math.min(30_000, windowEndsAt - Date.now() - 30_000)); + plugin = (await import(`${pluginUrl}?restart=${randomUUID()}`)).default.definition; + const restartedDb = createDb(temporary.connectionString); + timeline.push({ at: new Date().toISOString(), step: "restart: new process state, cleanup worker boots", windowEndsAt: iso(windowEndsAt) }); + const restarted = await startCleanupWorker(restartedDb); + try { + expect((await note("restart: boot pass inside window", retained!.providerLeaseId!)).state).toBe("started"); + const stoppedAt = await waitForState(retained!.providerLeaseId!, "stopped", windowEndsAt + 90_000); + await note("restart: stopped by rebooted cleanup worker", retained!.providerLeaseId!, + { secondsAfterWindowEnd: stoppedAt === null ? null : Math.round((stoppedAt - windowEndsAt) / 1000) }); + expect(stoppedAt).not.toBeNull(); + expect(stoppedAt!).toBeGreaterThanOrEqual(windowEndsAt); + } finally { + await restarted.stop(); + } + }, 15 * 60_000); +}); From 4879a146fb28d4cb5a4235bb52b52befcb5c16ab Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 19:46:50 -0600 Subject: [PATCH 07/11] fix(secrets): stop credential reads from aborting concurrent budget admissions (#21452) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paid model calls (company decisions, fast responses, browser use, prompt-injection checks) pass a budget admission before they start > - Admission runs in a REPEATABLE READ transaction and share-locks the credential's secret row, so a key rotation cannot slip between admission and the call > - Every credential read also wrote "Last used" (`last_resolved_at`, `updated_at`) to that same row > - When two calls ran at once, one call's credential read aborted the other's admission with a serialization failure (40001). The retry then saw the first call's new budget hold and denied the work as `budget_blocked` > - This is the flaky `decision-models.test.ts` failure that held Cloud readiness, and so releases, on 2026-11-13 > - This pull request stops credential reads from writing the secret row. "Last used" now comes from the access event that every read already records > - The benefit is that concurrent paid calls are no longer wrongly denied or retried, and the release gate no longer fails on this test ## Linked Issues or Issue Description No public issue exists. The bug is described here. **What happened?** Cloud readiness `verify / General tests (server (4/10))` failed on master commits 6917d0568d and 5ede64f247. The test "admits concurrent work from the same budget observation and records both obligations" reported `expected "vi.fn()" to be called 2 times, but got 1 times`. #21387 replaced the 1-second poll with an unbounded wait. That does not fix it: one decision really is denied, so the test now waits until the test timeout. **Root cause** I logged the accounting retry loop under load. Each batch of 50 runs had 6 to 10 retries with `could not serialize access due to concurrent update` on `select … from company_secrets … for share`. The other decision's credential read had updated the row. In 4 of 5 batches, the retried admission saw the other decision's committed reservation and returned `budget_blocked`. **Expected behavior** A credential read does not change admission. Two decisions admitted from the same budget observation both run and both record their obligation. **Steps to reproduce** Run the updated test on the old `secrets.ts`. The test makes the second credential read land after the first admission's snapshot. It then commits an unrelated hold after both observations. The old code fails every run with `first settled before its provider call: {"status":"unavailable","reason":"budget_blocked"}`. ## What Changed - `secretService` no longer writes `last_resolved_at` / `updated_at` when it resolves a value. It still records the `secret_access_events` row, as before. - `secretService.list` reports `lastResolvedAt` as the later of the stored column (legacy values) and the newest successful access event. It reads one row per secret, backwards on the existing `(secret_id, created_at)` index. - A read without a consumer context (for example a GitHub API credential read) now records a `system` access event, so it still counts as "Last used". - The removed `UPDATE` also made a credential read wait for a concurrent rotation to commit. `credential()` in `ai-connections.ts` depends on that wait to re-read the rotated version instead of handing a run a consumed single-use token. Its existing version re-read now takes `FOR SHARE`, so it keeps that wait. `prepareManagedAiRuntime` has its own before/after freshness read around `runtimeCredential()`; that read takes `FOR SHARE` too. A share lock creates no new row version and cannot abort an admission. It adds no statement. - The first version of this PR throttled the write to once a minute. Review found that a read crossing the minute could still abort an admission, so the write is removed instead. - The concurrent-admission test is now deterministic. It starts decision A and waits until A is blocked, then starts decision B and waits again. It commits an unrelated 1-cent hold after both observations, then checks that both decisions are admitted. A decision that settles before its provider call fails the test immediately with its result, instead of hanging. ## Verification - Red: the new test against the old `secrets.ts` fails on every run with the `budget_blocked` result above. - Green: the same test with `repeats: 49` (50 runs) passes 50/50 locally on a machine at load average 20 to 60. - Before the change, 3 parallel batches of 50 runs of the old test each produced a false denial. - `npx tsc --noEmit -p server`: no errors. - New `secrets-service.test.ts` test: a resolution leaves the secret row unchanged, and `list()` still shows "Last used". - `ai-connections.test.ts` quota test now checks that a read leaves `updated_at` unchanged and records a success access event. ## Risks - `updated_at` on a secret no longer moves when the secret is read. It still moves on every real change. - "Last used" on the Secrets page costs one indexed lookup per listed secret. - Only `list()` derives "Last used". Other endpoints that return a secret row show the stored column, which is now set only by older releases. - No migration. No API change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- server/src/__tests__/ai-connections.test.ts | 8 ++-- .../__tests__/antigravity-ai-runtime.test.ts | 2 +- .../antigravity-managed-subscription.test.ts | 5 +- server/src/__tests__/decision-models.test.ts | 48 +++++++++++++++++++ server/src/__tests__/secrets-service.test.ts | 31 ++++++++++++ server/src/services/ai-connection-runtime.ts | 5 +- server/src/services/ai-connections.ts | 7 +-- server/src/services/secrets.ts | 36 +++++++++++--- 8 files changed, 126 insertions(+), 16 deletions(-) diff --git a/server/src/__tests__/ai-connections.test.ts b/server/src/__tests__/ai-connections.test.ts index c72e1dff56b..ae747a42c69 100644 --- a/server/src/__tests__/ai-connections.test.ts +++ b/server/src/__tests__/ai-connections.test.ts @@ -17,7 +17,7 @@ import { mkdtemp, realpath, rm, access, readFile, writeFile, stat } from "node:f import os from "node:os"; import path from "node:path"; import { and, eq, sql } from "drizzle-orm"; -import { createDb, type Db, companies, agents, agentTaskSessions, heartbeatRuns, companyMemberships, connectionGrants, toolApplications, connectionGrantDelegations, connectionGrantMembers, toolConnections, toolConnectionInstalls, aiConnectionDefaults, aiProviderDefaults, adapterAuthSessions, environments, plugins, issues, issueThreadInteractions, issueRecoveryActions, connectionIntentDeliveries, agentWakeupRequests, companySecrets, companySecretVersions, userSecretDefinitions, principalPermissionGrants } from "@paperclipai/db"; +import { createDb, type Db, companies, agents, agentTaskSessions, heartbeatRuns, companyMemberships, connectionGrants, toolApplications, connectionGrantDelegations, connectionGrantMembers, toolConnections, toolConnectionInstalls, aiConnectionDefaults, aiProviderDefaults, adapterAuthSessions, environments, plugins, issues, issueThreadInteractions, issueRecoveryActions, connectionIntentDeliveries, agentWakeupRequests, companySecrets, companySecretVersions, secretAccessEvents, userSecretDefinitions, principalPermissionGrants } from "@paperclipai/db"; import { startEmbeddedPostgresTestDatabase } from "@paperclipai/db/test-embedded-postgres"; import { fetchCompanyQuotaWindows } from "../services/quota-windows.js"; import { quotaCredentialRecoveryPath, quotaCredentialRecovery, quotaCredentialHash } from "../services/quota-credential-recovery.js"; @@ -717,8 +717,10 @@ describe("managed AI connections", () => { try { const first = await fetchCompanyQuotaWindows(db, companyId, owner); const [after] = await db.select().from(companySecrets).where(eq(companySecrets.id, secretId)); - expect(after.lastResolvedAt).not.toBeNull(); - expect(after.updatedAt.getTime()).toBeGreaterThan(before.updatedAt.getTime()); + // A read is recorded as an access event, never written to the row that + // admissions share-lock to fence rotation. + expect(after.updatedAt.getTime()).toBe(before.updatedAt.getTime()); + expect(await db.select().from(secretAccessEvents).where(and(eq(secretAccessEvents.secretId, secretId), eq(secretAccessEvents.outcome, "success")))).not.toHaveLength(0); expect(first[0].ok).toBe(true); expect(await fetchCompanyQuotaWindows(db, companyId, owner)).toEqual(first); expect(fetchSpy).toHaveBeenCalledTimes(1); diff --git a/server/src/__tests__/antigravity-ai-runtime.test.ts b/server/src/__tests__/antigravity-ai-runtime.test.ts index 7c81ddc1926..3d766eaa66f 100644 --- a/server/src/__tests__/antigravity-ai-runtime.test.ts +++ b/server/src/__tests__/antigravity-ai-runtime.test.ts @@ -35,7 +35,7 @@ describe("Antigravity private managed AI home", () => { const version = [{ epoch: 1, version: 1 }]; const db = { select: (fields: Record) => ({ from: () => ({ where: () => Object.assign(Promise.resolve("refs" in fields ? [{ refs: account.refs, status: "active" }] : version), - { limit: async () => version }), + { limit: async () => version, for: async () => version }), }) }) } as unknown as Db; const input = { companyId: "44444444-4444-4444-8444-444444444444", agentId: "55555555-5555-4555-8555-555555555555", responsibleUserId: "alice", diff --git a/server/src/__tests__/antigravity-managed-subscription.test.ts b/server/src/__tests__/antigravity-managed-subscription.test.ts index 0a60321d15e..3da53671f3c 100644 --- a/server/src/__tests__/antigravity-managed-subscription.test.ts +++ b/server/src/__tests__/antigravity-managed-subscription.test.ts @@ -123,7 +123,8 @@ describe("managed Antigravity subscription projection and writeback", () => { await expect(access(join(provider, "auth.json"))).rejects.toMatchObject({ code: "ENOENT" }); await writeFile(token, credential("resolved-project")); await runtime.cleanup(); - expect(account.locks).toEqual(["grant:update", "secret:update"]); + // Preparation reads the secret version under a share lock before and after resolving it. + expect(account.locks).toEqual(["secret:share", "secret:share", "grant:update", "secret:update"]); expect(account.rotation).toHaveBeenCalledWith(account.refs[0].secretId, { value: credential("resolved-project"), preserveAiSessionEpoch: true }, { userId: "alice" }); await expect(access(runtime.home!)).rejects.toMatchObject({ code: "ENOENT" }); @@ -216,7 +217,7 @@ describe("managed Antigravity subscription projection and writeback", () => { } else { await expect(runtime.cleanup()).rejects.toThrow("The refreshed Google account does not match the selected connection."); expect(account.rotation).not.toHaveBeenCalled(); - expect(account.locks).toEqual([]); + expect(account.locks).toEqual(["secret:share", "secret:share"]); expect(account.journal).toMatchObject({ provider: "google", google: { verified: false }, value: credential("resolved-project", "fixture-rotated-refresh") }); expect(await readFile(token, "utf8")).toBe(credential("resolved-project", "fixture-rotated-refresh")); } diff --git a/server/src/__tests__/decision-models.test.ts b/server/src/__tests__/decision-models.test.ts index 92299eb18b6..a5890fdb8b8 100644 --- a/server/src/__tests__/decision-models.test.ts +++ b/server/src/__tests__/decision-models.test.ts @@ -405,6 +405,54 @@ describe("company decision service", () => { expect((await first).status).toBe("succeeded"); expect((await second).status).toBe("succeeded"); expect(await db.select().from(costEvents).where(eq(costEvents.companyId, f.companyId))).toHaveLength(2); }); + it("keeps an admission's budget observation when another decision reads the same credential mid-admission", async () => { + const f = await fixture(); + await db.insert(budgetPolicies).values({ companyId: f.companyId, scopeType: "company", scopeId: f.companyId, + windowKind: "calendar_month_utc", amount: 1, reservationCents: "1", notifyEnabled: false }); + let unlock!: () => void, locked!: (pid: number) => void, finishProviders!: () => void; + const lockGate = new Promise(resolve => { unlock = resolve; }); + const acquired = new Promise(resolve => { locked = resolve; }); + const providerGate = new Promise(resolve => { finishProviders = resolve; }); + const holding = db.transaction(async tx => { + await tx.select().from(companyDecisionModels).where(eq(companyDecisionModels.companyId, f.companyId)).for("no key update"); + const [row] = await tx.execute<{ pid: number }>(sql`select pg_backend_pid() as pid`); + locked(row.pid); await lockGate; + }); + const pid = await acquired; + const waitingAdmissions = (count: number) => vi.waitFor(async () => { + const [row] = await db.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity where ${pid} = any(pg_blocking_pids(pid))`); + expect(row.count).toBe(count); + }, { timeout: 10_000 }); + let providersEntered = 0, bothEntered!: () => void; + const bothProvidersEntered = new Promise(resolve => { bothEntered = resolve; }); + f.provider.mockImplementation(async () => { + if (++providersEntered === 2) bothEntered(); + await providerGate; return structuredClone(outcome); + }); + // Order the two admissions so the second one's credential read happens + // after the first admission took its budget observation. Reading a + // credential must not invalidate that observation. + const first = f.service.decide(f.context, DECISION_TEST_REQUEST); + let second: ReturnType | undefined; + try { + await waitingAdmissions(1); + second = f.service.decide(f.context, DECISION_TEST_REQUEST); + await waitingAdmissions(2); + // An obligation committed after both observations. An admission that + // keeps its observation ignores it; one that restarts is denied by it. + await db.insert(budgetReservations).values({ companyId: f.companyId, decisionInvocationId: randomUUID(), amountCents: "1" }); + unlock(); await holding; + // Before the providers finish, a decision settles only if admission denied it. + const settledEarly = (name: string, pending: typeof first) => pending.then(result => `${name} settled before its provider call: ${JSON.stringify(result)}`); + expect(await Promise.race([bothProvidersEntered.then(() => "both admitted"), settledEarly("first", first), settledEarly("second", second)])).toBe("both admitted"); + expect(f.provider).toHaveBeenCalledTimes(2); + const holds = await db.select().from(budgetReservations).where(and(eq(budgetReservations.companyId, f.companyId), sql`${budgetReservations.decisionInvocationId} in (select id from decision_invocations)`)); + expect(holds).toHaveLength(2); + expect(holds.every(row => row.state === "held" && row.amountCents === "1.0000000")).toBe(true); + } finally { unlock(); finishProviders(); await holding; await Promise.allSettled([first, second]); } + expect((await first).status).toBe("succeeded"); expect((await second!).status).toBe("succeeded"); + expect(await db.select().from(costEvents).where(eq(costEvents.companyId, f.companyId))).toHaveLength(2); + }); it("denies new work when an already committed reservation consumes the observed budget", async () => { const f = await fixture(); await db.insert(budgetPolicies).values({ companyId: f.companyId, scopeType: "company", scopeId: f.companyId, windowKind: "calendar_month_utc", amount: 1, reservationCents: "1", notifyEnabled: false }); diff --git a/server/src/__tests__/secrets-service.test.ts b/server/src/__tests__/secrets-service.test.ts index 7af2791881e..77125a82c0e 100644 --- a/server/src/__tests__/secrets-service.test.ts +++ b/server/src/__tests__/secrets-service.test.ts @@ -2660,6 +2660,37 @@ describeEmbeddedPostgres("secretService", () => { ).rejects.toThrow(/not bound/i); }); + it("records a read as \"Last used\" without writing the secret row that admissions lock", async () => { + const companyId = await seedCompany(); + const svc = secretService(db); + const secret = await svc.create(companyId, { + name: `last-used-${randomUUID()}`, + provider: "local_encrypted", + value: "runtime-secret", + }); + await svc.createBinding({ companyId, secretId: secret.id, targetType: "system", targetId: "system", configPath: "env.API_KEY" }); + const [before] = await db.select().from(companySecrets).where(eq(companySecrets.id, secret.id)); + expect((await svc.list(companyId)).find((row) => row.id === secret.id)?.lastResolvedAt).toBeNull(); + + await svc.resolveSecretValue(companyId, secret.id, "latest", { consumerType: "system", consumerId: "system", configPath: "env.API_KEY" }); + + // The row keeps its version: a concurrent REPEATABLE READ admission that + // share-locks it to fence rotation is not aborted by this read. + const [after] = await db.select().from(companySecrets).where(eq(companySecrets.id, secret.id)); + expect(after).toEqual(before); + const listed = (await svc.list(companyId)).find((row) => row.id === secret.id); + expect(listed?.lastResolvedAt).toBeInstanceOf(Date); + expect(listed!.lastResolvedAt!.getTime()).toBeGreaterThanOrEqual(before!.createdAt.getTime()); + + // A read made without a consumer context also counts as use. + const unattributed = await svc.create(companyId, { name: `last-used-unattributed-${randomUUID()}`, provider: "local_encrypted", value: "runtime-secret" }); + const [unattributedBefore] = await db.select().from(companySecrets).where(eq(companySecrets.id, unattributed.id)); + await svc.resolveSecretValue(companyId, unattributed.id, "latest"); + const [unattributedAfter] = await db.select().from(companySecrets).where(eq(companySecrets.id, unattributed.id)); + expect(unattributedAfter).toEqual(unattributedBefore); + expect((await svc.list(companyId)).find((row) => row.id === unattributed.id)?.lastResolvedAt).toBeInstanceOf(Date); + }); + it("preserves the original resolution error when failure access logging fails", async () => { const companyId = await seedCompany(); const svc = secretService(db); diff --git a/server/src/services/ai-connection-runtime.ts b/server/src/services/ai-connection-runtime.ts index 03ccb040d50..9ac8f911e05 100644 --- a/server/src/services/ai-connection-runtime.ts +++ b/server/src/services/ai-connection-runtime.ts @@ -284,8 +284,11 @@ export async function prepareManagedAiRuntime( .from(connectionGrants).where(and(eq(connectionGrants.companyId, input.companyId), eq(connectionGrants.id, selection.grant.id))); const currentRef = currentGrant?.refs.find(ref => ref.configPath === "ai.credential"); if (currentGrant?.status !== "active" || currentRef?.secretId !== credentialRef.secretId || currentRef.versionSelector !== credentialRef.versionSelector) return undefined; + // The share lock waits behind a rotating writer, so the read after + // resolution sees a concurrent rotation instead of the pre-rotation + // version. A lock takes no new row version and aborts no admission. return (await db.select({ epoch: companySecrets.aiSessionEpoch, version: companySecrets.latestVersion }) - .from(companySecrets).where(and(eq(companySecrets.companyId, input.companyId), eq(companySecrets.id, credentialRef.secretId))).limit(1))[0]; + .from(companySecrets).where(and(eq(companySecrets.companyId, input.companyId), eq(companySecrets.id, credentialRef.secretId))).for("share"))[0]; }; const { value, freshness } = await (async () => { if (noAuth) return { value: "", freshness: undefined }; diff --git a/server/src/services/ai-connections.ts b/server/src/services/ai-connections.ts index 25b812ffc3f..5805ea231c4 100644 --- a/server/src/services/ai-connections.ts +++ b/server/src/services/ai-connections.ts @@ -615,11 +615,12 @@ export function aiConnectionService(db: Db) { } else { value = await secrets.resolveSecretValue(row.connection.companyId, secret.id, "latest", context); } - // Resolution records lastResolvedAt and can wait behind a rotating writer. - // Re-read after that wait instead of handing a run the pre-rotation token. + // Wait behind a rotating writer (the share lock), then re-read instead of + // handing a run the pre-rotation token. A lock takes no new row version, + // so it never aborts a concurrent REPEATABLE READ admission. const [latest] = await db.select().from(companySecrets).where(and( eq(companySecrets.id, secret.id), eq(companySecrets.companyId, row.connection.companyId), - )); + )).for("share"); if (!latest || latest.status !== "active") throw unprocessable("Reconnect this AI account"); if (latest.latestVersion !== secret.latestVersion) { if (retry >= 2) throw unprocessable("AI credentials are changing. Retry this execution.", { code: "ai_connection_busy" }); diff --git a/server/src/services/secrets.ts b/server/src/services/secrets.ts index 65f88cd18c2..6c0d8d44e59 100644 --- a/server/src/services/secrets.ts +++ b/server/src/services/secrets.ts @@ -88,6 +88,12 @@ import { withAccountHomeSecretMutationLock, } from "@paperclipai/adapter-codex-local/server"; +function latestDate(a: Date | null, b: Date | null): Date | null { + if (!a) return b; + if (!b) return a; + return a.getTime() >= b.getTime() ? a : b; +} + const ENV_KEY_RE = /^[A-Za-z_][A-Za-z0-9_]*$/; const AGENT_ACCESS_CONFIG_PATH_PREFIX = "access."; // System consumer id for a durable orphan-sandbox teardown. The cleanup sweep @@ -642,6 +648,8 @@ type SecretConsumerContext = { allowedBindingIds?: string[] | null; }; +const UNATTRIBUTED_SECRET_READ: SecretConsumerContext = { consumerType: "system", consumerId: "system", actorType: "system" }; + type SecretBindingContext = Omit & { consumerType: SecretBindingTargetType; }; @@ -1365,6 +1373,19 @@ export function secretService(db: Db | DbTransaction) { }; } + /** Newest successful resolution per secret, read backwards on (secret_id, created_at). */ + async function lastSuccessfulResolutions(secretIds: string[]): Promise> { + if (secretIds.length === 0) return new Map(); + const rows = await db.execute<{ secret_id: string; used_at: string | Date | null }>(sql` + select ids.secret_id, ( + select e.created_at from ${secretAccessEvents} e + where e.secret_id = ids.secret_id and e.outcome = 'success' + order by e.created_at desc limit 1 + ) as used_at + from unnest(array[${sql.join(secretIds.map((id) => sql`${id}`), sql`, `)}]::uuid[]) as ids(secret_id)`); + return new Map(rows.flatMap((row) => row.used_at ? [[row.secret_id, new Date(row.used_at)] as const] : [])); + } + async function resolveSecretValueInternal( companyId: string, secretId: string, @@ -1428,12 +1449,12 @@ export function secretService(db: Db | DbTransaction) { if (isTailscaleNodeState(value) && systemPurpose !== "tailscale_network") { throw forbidden("Network node state requires its server-only consumer", { code: "secret_system_only" }); } + // The access event is the usage record; "Last used" is derived from it + // in list(). A read must not write the secret row: admission paths + // (decisions, fast responses, browser use, prompt-injection checks) + // share-lock it under REPEATABLE READ to fence rotation, and a new row + // version would abort them with a serialization failure. if (!readOnlyEnvironmentInspection) await Promise.all([ - db - .update(companySecrets) - .set({ lastResolvedAt: new Date(), updatedAt: new Date() }) - .where(eq(companySecrets.id, secret.id)) - .catch(() => undefined), recordAccessEvent({ companyId, secretId: secret.id, @@ -1441,7 +1462,8 @@ export function secretService(db: Db | DbTransaction) { secretScope: secret.scope, version: resolvedVersion, provider: providerId, - context: accessContext, + // A read without a consumer still counts as "Last used". + context: accessContext ?? UNATTRIBUTED_SECRET_READ, credentialOwnerUserId: secret.ownerUserId ?? null, credentialSubjectType: secret.scope === "user" ? "user" : null, credentialSubjectId: secret.ownerUserId ?? null, @@ -3873,8 +3895,10 @@ export function secretService(db: Db | DbTransaction) { .groupBy(companySecretBindings.secretId), ]); const countsBySecretId = new Map(referenceCounts.map((row) => [row.secretId, row.count])); + const lastUsedBySecretId = await lastSuccessfulResolutions(secrets.map((secret) => secret.id)); return secrets.map((secret) => ({ ...secret, + lastResolvedAt: latestDate(secret.lastResolvedAt, lastUsedBySecretId.get(secret.id) ?? null), referenceCount: countsBySecretId.get(secret.id) ?? 0, })); }, From a4c570aede7b6cb36eecc0c7163f67ee90283f60 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 19:46:56 -0600 Subject: [PATCH 08/11] feat(mcp): ask an app's form questions in the task chat and send the answer back (MCP elicitation) (#21466) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents use connected MCP apps through the tool gateway. > - An MCP app can ask the person a question in the middle of a tool call (MCP elicitation). There are two kinds: a form, or a link to open. > - Today a form question is a dead end: > - A 2026-07-28 server's form `input_required` stops with "needs information Paperclip can't collect yet". > - An older server's `elicitation/create` makes a card, but the answer is never sent back, and the server waits on an open stream. > - Paperclip does not declare form support, so compliant servers never ask. > - This pull request shows the question as a card in the task chat. The card names the app, and only a person can answer it. When the person answers, the agent wakes. The agent's repeated call sends the answer to the app exactly once, for both protocol eras. > - The benefit: apps that need a choice or a detail from the person now finish their work. Restarts lose nothing, and no call is left waiting. ## Linked Issues or Issue Description Refs #21140 (stateless MCP client; this PR closes its "form elicitation" follow-up). Refs #21263 (MCP Tasks; this PR shares its card code). Refs #21389 (MCP Tasks follow-ups: the card answer validation item). **Is your feature request related to a problem?** When a connected MCP app asks the person a form question during a tool call, the call stops and cannot continue. A 2026-07-28 server gets a "needs input" error. An older server gets a card whose answer never reaches it. **Describe the solution you'd like** Show the question in the task chat. Let a person answer, skip, or leave it. Wake the agent when the question is resolved. Send the answer on the agent's repeated call, once, with the server's exact state. **Describe alternatives you've considered** Holding the tool call open until the person answers. We rejected this: it holds a stream and a run slot, and it does not survive deploys or restarts. The plan (`doc/plans/2026-11-13-mcp-elicitation.md`) records nine design decisions with their alternatives. ## What Changed - **Plan:** `doc/plans/2026-11-13-mcp-elicitation.md` covers the spec, the inventory, the decisions, the journey, tests, and risks. - **New module `server/src/services/mcp-elicitation.ts`.** It is the single mapping for direct calls and MCP Tasks: - It turns each spec form field into a real card field: text with defaults, number, yes/no, single choice (`enum` and `oneOf`), and multiple choice (`items.enum` and `items.anyOf`). - It turns card answers back into typed `content`. - Answered sends `accept`. Skip sends `decline`. Withdrawn, closed, or expired sends `cancel`. - It refuses forms that ask for passwords, keys, tokens, card numbers, or one-time codes. - It keeps app text plain, so links in it are not clickable. - It seals `requestState` with AES-256-GCM, bound to the company and connection. - **New table `tool_mcp_elicitations`** (migration 0479). It is company-scoped, with one row per question round. It holds a keyed call identity, the sealed state, the questions and their cards, a claim time, and a 24-hour expiry. - **Gateway, stateless servers:** - Paperclip declares `elicitation: {form, url}` on task tool calls. - A form-only `input_required` saves the round, makes the cards, and stops with a wait message. - A repeated call before the person answers stops again without contacting the app. - After the answer, the repeated call claims the round and sends `inputResponses` and the exact `requestState`, once. - A failed send releases the claim. A crash mid-send lets the claim lapse. - **Gateway, older servers:** - Paperclip declares form support at `initialize`. - The first question gets `cancel` at once, so the server never waits. - When the same question (same words and schema) arrives on the repeated call, Paperclip answers it from the card. - **Gateway, mixed rounds and retries:** - A form beside a URL step shows the link first. - A repeated call that stopped for a question runs again in the same run. It is no longer replayed by the per-run idempotency key. - **Cards:** the title is "*App* needs your input". The resolver policy is `human_only`. "Send to *App*" sends the answer. A user comment does not supersede the card. - **Activity log:** new events `tool_gateway.elicitation_sent`, `tool_gateway.elicitation_refused`, and `tool_gateway.elicitation_expired`. They record the action only, never the values. - **MCP Tasks:** task cards use the same mapping and gain the app name, real fields, and `human_only`. A secret or an unsupported form stays unsupported. - **Storybook:** `Tasks and conversations/Questions and approvals/MCP App Questions` has five states: Asking, Answered, Skipped, Expired, and TwoQuestions. The fixtures are generated from the server mapper's output. - **Docs:** `doc/connections/MCP-ELICITATION.md` (linked from the README) is new. `MCP-PROTOCOL-VERSIONS.md` and `MCP-TASKS.md` are updated. ## Verification - `server/src/__tests__/mcp-elicitation.test.ts` has 16 tests. They pass on the branch. - **Red/green:** 9 of the 9 gateway tests fail with master's `tool-gateway.ts` and `mcp-http.ts`. The 7 pure unit tests cover the new module. - The tests use a fake 2026-07-28 server and a fake 2025-06-18 server that asks over the open stream and waits for the reply. - Covered: answer, skip (`decline`), withdraw (`cancel`), two questions at once, a restart while waiting, a crash mid-send, another company with the same app and arguments, expiry, a refused password field, an agent that tries to answer (rejected), and capabilities. - Existing suites: `mcp-tasks`, `mcp-protocol-versions`, `tool-gateway-service`, `tool-gateway`, and `remote-mcp-connectors`. See the results comment below. - Storybook: the five stories render in headless Chromium, and the play tests pass. The only console errors are the 404s from the backend-less API calls that every story makes. - `tsc --noEmit` is clean for `server` and `ui`. `node scripts/storybook-catalog.mjs` passes. ## Risks - **Older servers may start asking questions.** Declaring form support at `initialize` can make older servers ask where they did not before. Mitigation: the first question always gets `cancel` at once, so no call hangs, and the agent gets a clear next step. - **Per-call cost.** Each 2026-07-28 HTTP tool call in a task makes one indexed lookup for a waiting round. Nothing is added before the first token or to agent boot. - **Migration.** One additive table with no backfill, so it is safe to apply online. - **Changed Tasks behavior.** A task card that is withdrawn or closed now sends `cancel` instead of `decline`, as the spec defines. A skipped card still sends `decline`. - **Stale cards.** Expiry is checked only when the agent calls again. A card can sit past 24 hours until the next call. Then it is withdrawn and the app asks again. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code with tool use (shell, file edits, headless browser), with extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/connections/MCP-ELICITATION.md | 127 +++ doc/connections/MCP-PROTOCOL-VERSIONS.md | 15 +- doc/connections/MCP-TASKS.md | 4 +- doc/connections/README.md | 6 +- doc/plans/2026-11-13-mcp-elicitation.md | 342 ++++++++ .../db/src/migrations/0483_busy_tombstone.sql | 30 + ...{0478_snapshot.json => 0483_snapshot.json} | 434 +++++++++- packages/db/src/migrations/meta/_journal.json | 7 + packages/db/src/schema/index.ts | 1 + packages/db/src/schema/tool_access.ts | 55 ++ .../issue-thread-interaction-routes.test.ts | 28 + server/src/__tests__/mcp-elicitation.test.ts | 696 ++++++++++++++++ .../__tests__/mcp-protocol-versions.test.ts | 2 + .../__tests__/tool-gateway-service.test.ts | 7 +- server/src/routes/issues.ts | 13 + server/src/services/mcp-elicitation.ts | 365 +++++++++ server/src/services/mcp-http.ts | 35 +- server/src/services/tool-gateway.ts | 764 +++++++++++++----- ui/src/fixtures/mcpElicitationFixtures.ts | 218 +++++ .../mcp-elicitation-question-card.stories.tsx | 99 +++ 20 files changed, 3030 insertions(+), 218 deletions(-) create mode 100644 doc/connections/MCP-ELICITATION.md create mode 100644 doc/plans/2026-11-13-mcp-elicitation.md create mode 100644 packages/db/src/migrations/0483_busy_tombstone.sql rename packages/db/src/migrations/meta/{0478_snapshot.json => 0483_snapshot.json} (99%) create mode 100644 server/src/__tests__/mcp-elicitation.test.ts create mode 100644 server/src/services/mcp-elicitation.ts create mode 100644 ui/src/fixtures/mcpElicitationFixtures.ts create mode 100644 ui/storybook/stories/questions-and-approvals/mcp-elicitation-question-card.stories.tsx diff --git a/doc/connections/MCP-ELICITATION.md b/doc/connections/MCP-ELICITATION.md new file mode 100644 index 00000000000..ea228cf10e7 --- /dev/null +++ b/doc/connections/MCP-ELICITATION.md @@ -0,0 +1,127 @@ +# MCP elicitation: an app's question, answered in the task chat + +When a connected MCP app needs to ask the person something in the middle of a tool call, Paperclip handles it in five steps: + +1. The question appears in the task chat as a card that names the app. +2. The agent's call stops with a message telling it to wait. +3. A person answers the card. +4. Answering wakes the agent. +5. The agent repeats the call with the same arguments, and Paperclip sends the answer to the app exactly once. + +Nothing blocks while the person thinks, and nothing is held only in memory. A deploy or restart while a question waits loses nothing. + +The design and its alternatives are in [`doc/plans/2026-11-13-mcp-elicitation.md`](../plans/2026-11-13-mcp-elicitation.md). + +## What is supported + +| App asks | Paperclip does | +| --- | --- | +| A form question, from a stateless (2026-07-28) server (`input_required`) | Shows a question card. The repeated call sends `inputResponses` together with the server's exact `requestState` | +| A form question, from an older (2025-06-18 / 2025-11-25) server (`elicitation/create` on the open stream) | Answers `cancel` at once, so the server never waits, and shows the card. When the same question arrives on the repeated call, Paperclip answers it from the card | +| A URL step (any version) | Shows the provider link, as before. Paperclip never claims the person's consent ([MCP-PROTOCOL-VERSIONS.md](./MCP-PROTOCOL-VERSIONS.md)) | +| A form beside a URL step | The link goes first. On the next call the app asks for the form again | +| A form question inside a background MCP Task | Shows the same card. The answer goes back with `tasks/update` ([MCP-TASKS.md](./MCP-TASKS.md)) | +| A password, key, token, card number or one-time code in a form | Refused. No card is shown, and the agent is told the app must use a secure link | +| A schema outside the spec, or a form with no task | Shows no card. The agent gets a clear reason | + +## Capability + +Paperclip declares `elicitation: { form: {}, url: {} }` in two places: + +- **2026-07-28:** per request, only on HTTP `tools/call` requests that belong to a task, because that is where a person can answer. +- **Older HTTP sessions:** at `initialize`, only on the session that tool calls use, because that is the path that answers questions. Sessions that only read (resources, prompts, listing) declare none, so they are never asked a question they would leave waiting. A 2025-06-18 session gets `{}`, which means form only. Questions that arrive outside a task are answered `cancel` at once. + +Stdio apps keep URL only. + +## How the card works + +- **Shows which app is asking.** The card's title is "*App* needs your input", using the connection's name. The app's message is shown as plain text: links are not clickable, and secrets are redacted. +- **Only people can answer** (`resolverPolicy: "human_only"`). An agent answering its own tool's question would be consent given on nobody's behalf. +- **Lets the person review before sending.** The form is the standard question form: fields can be edited, and nothing is sent until **Send to *App***. **Skip** declines. +- **Is checked on submit.** Required fields, lengths, number ranges and the date format are validated before the card is resolved. Email and web address show a hint and are checked by the app, because the form only checks fixed-width patterns. +- **Wakes the agent either way.** Answering wakes the agent through the question-response outbox. **Skip** on an app's question also wakes it (ordinary questions do not), because the decline only reaches the app on the agent's next call. + +Each form field maps to one card field: + +| Schema | Card field | +| --- | --- | +| `string` (`minLength`, `maxLength`, `default`) | Text. The default is filled in | +| `string` with `format` `email` / `date` / `date-time` / `uri` | Text with a hint. `date` is validated | +| `number`, `integer` (`minimum`, `maximum`, `default`) | Number field | +| `boolean` (`default`) | Yes / No. The default is marked as recommended | +| `enum`, or `oneOf` with `const` + `title` | Single choice. Titles are the labels | +| `array` with `items.enum` or `items.anyOf` (`minItems`, `maxItems`, `default`) | Multiple choice | +| No properties | One **Continue** choice | + +Paperclip refuses the following shapes rather than showing part of the form: + +- Nested objects. +- Arrays of anything but choices. +- Unknown formats. +- More than 10 fields. +- More than 100 choices. +- More than 4 form questions in one round. + +## What goes back to the app + +| The card | Sent | +| --- | --- | +| Answered | `{ action: "accept", content }`, with values typed to the schema (choices are sent as their `const`) | +| Skipped | `{ action: "decline" }` | +| Withdrawn, task closed, expired, or deleted | `{ action: "cancel" }` | + +## What the agent sees + +| Moment | Result | +| --- | --- | +| The app asks | `elicitation_required`: "*App* asked a question, and it is waiting for a person in the task chat. Paperclip wakes you when it is answered; then call this tool again with the same arguments to continue." | +| Repeated before every card is resolved | The same message. The app is not contacted | +| Repeated after | The app's own result for the answer, decline or cancel | +| A secret was asked for | `elicitation_sensitive_refused` | +| An unsupported form | `elicitation_unsupported` | +| 10 questions already waiting on the task | `elicitation_limit_reached` | + +The agent never receives the person's answers from Paperclip, only the app's result. Within one run, a repeated side-effecting call normally replays the earlier result. A call that stopped for a question runs again instead. + +## Storage and recovery + +- **Table.** `tool_mcp_elicitations` holds one row per question round. It is company-scoped and records: + - The task, agent, run, connection and grant. + - A keyed call identity, covering endpoint, tool, arguments, grant, agent and any forwarded routing headers (such as a tenant). One tenant's answers never go out with another tenant's headers. + - The server's `requestState`, sealed with AES-256-GCM and bound to the company and connection. + - Each question's key, words, schema and card. +- **Answers.** Answers live only on the cards. +- **Saving.** The round is saved before its cards, and each card has a fixed key. If the process stops between the two, the next call makes the missing card under the same key, so no question is lost or shown twice. +- **Who can collect a round.** Only the identical call can collect it: same company, task, connection, grant, agent and call identity. +- **Older apps that ask several questions.** An answer is spent only when the app finishes the whole call. An app that asks Q1 and then Q2 gets Q1's answer again on the next call, without a second card. An app that refuses the reply (an HTTP error) keeps the answer for the next call. +- **Keyed retries.** A call with an unsent round runs again (instead of replaying its last result, even a failed send) only if it has the same tool and arguments as the saved round. +- **One round per call.** At most one waiting round exists per call (per question, for older servers), enforced by a unique index. Two matching calls at once share its cards. +- **Ask-first actions.** An approved action that stops for a question keeps its approval. The repeated call continues the same approved action and sends the answer, with no second approval. +- **Claims.** A repeated call claims the round before sending. If the process dies mid-send, the claim lapses after the tool timeout plus a minute, and the next call sends again. A failed send releases the claim at once. +- **Expiry.** Rounds expire after 24 hours, checked when the call is next made; no background poller is involved. Expired cards are withdrawn, so nobody answers a dead question, and the call starts over. +- **Activity log.** The log records `tool_gateway.elicitation_requested`, `tool_gateway.elicitation_sent` (the actions only, never values), `tool_gateway.elicitation_refused` (field names only) and `tool_gateway.elicitation_expired`. Answers are logged as `issue.thread_interaction_answered`. + +## MCP Apps + +A tool call made from an MCP App view runs in the task's session. Its questions appear in the same chat. + +## Not supported yet + +- Enforcing `minItems` and `maxItems` on the card. Today they show as help text, and the app checks them when the answer arrives. +- Sampling and roots requests. +- Form questions from stdio apps. +- Forms rendered inside an MCP App view. +- `input_required` on `resources/read` and `prompts/get`. +- Answers to the vendor `_meta.elicitation` result shape, which has no channel for them. + +## Tests + +`server/src/__tests__/mcp-elicitation.test.ts` covers the following: + +- The schema mapping and answer typing. +- The sensitive-field refusal. +- Capabilities. +- A fake stateless server: answer, skip, two questions at once, restart while waiting, a crash mid-send, a second company, expiry, and an agent trying to answer. +- A fake 2025-06-18 server that asks over the open stream. + +Stories: `Tasks and conversations/Questions and approvals/MCP App Questions`. diff --git a/doc/connections/MCP-PROTOCOL-VERSIONS.md b/doc/connections/MCP-PROTOCOL-VERSIONS.md index 9a0a114d4c5..806707aa70e 100644 --- a/doc/connections/MCP-PROTOCOL-VERSIONS.md +++ b/doc/connections/MCP-PROTOCOL-VERSIONS.md @@ -102,10 +102,15 @@ stateless MCP) is probed again once. agent, tool, and arguments) with a new JSON-RPC id. It survives an attempt the server rejected (for example a `401` before a token refresh). The call stops with `provider_retry_requested` so the agent retries. - - Form, sampling, or roots requests, or more than 8 links: the call stops - with `provider_input_required` and the message "This app needs information - Paperclip can't collect yet." Paperclip does not declare form, sampling, - or roots, so compliant servers should not send them. + - Form requests only, on a call that belongs to a task: a question card in + the task chat; the repeated call sends `inputResponses` and the exact + `requestState`. See [MCP-ELICITATION.md](./MCP-ELICITATION.md). Form + beside a URL request: the link goes first. + - Sampling or roots requests, form requests outside a task, or more than 8 + links: the call stops with `provider_input_required` and the message + "This app needs information Paperclip can't collect yet." Paperclip + declares form only on task calls and never declares sampling or roots, + so compliant servers should not send them. - `"input_required"` on a context read (`resources/read`, `prompts/get`) or a stdio call stops with `provider_input_required`. - Resource not found is `-32602` from modern servers and `-32002` from legacy @@ -160,8 +165,6 @@ Tracked as follow-ups: - Recording that a person finished a provider URL step, so Paperclip can answer it with `accept` and the provider's `requestState`. -- Collecting form input requests (MRTR form elicitation) in the task chat and - answering them on the retry. - Retrying a `requestState`-only result automatically instead of asking the agent to retry. - `subscriptions/listen` change notifications (cache invalidation relies on diff --git a/doc/connections/MCP-TASKS.md b/doc/connections/MCP-TASKS.md index 9d99072e02d..bf2219916bc 100644 --- a/doc/connections/MCP-TASKS.md +++ b/doc/connections/MCP-TASKS.md @@ -58,7 +58,7 @@ Paperclip polls, which the specification makes the default. | Attach | A call with the same endpoint, tool, full arguments, grant, agent, task, and actor attaches to the undelivered task instead of sending `tools/call` again. So does a retry with the same idempotency key, including after a crash mid-wait; the same key with other arguments is refused and leaves the original retry intact. One holder at a time (a lease): a second simultaneous caller is told to call again. | `findAttachableMcpTask`, `mcpTaskForKeyedRetry`, `claimMcpTask` | | Deliver | A completed task counts as delivered only once its result is recorded on the call. A crash in between leaves it collectable again. | `markMcpTaskDelivered` | | Restart | Background work reads pending rows at startup and polls each at its next deadline, with each request bounded to 30 s. It rechecks the same access a live call needs (grant, audience, delegation). When a task settles, or asks a new question, after its run has ended, the assignee is woken once; the agent's repeated call collects the result or shows the question. | `pollDueMcpTasks`, `server/src/services/mcp-task-work.ts` | -| Input | URL requests show the provider link, as for a direct request; Paperclip never answers them for the person. Form requests become a question card on the task; only the person's answer is sent with `tasks/update`. Other requests (sampling, roots) stop with a clear "needs input" error. | `planMcpTaskInput` | +| Input | URL requests show the provider link, as for a direct request; Paperclip never answers them for the person. Form requests become a question card on the task, built and answered as in [MCP-ELICITATION.md](./MCP-ELICITATION.md) (human-only, typed fields; skip sends `decline`, other endings `cancel`); only the person's answer is sent with `tasks/update`. Other requests (sampling, roots) stop with a clear "needs input" error. | `planMcpTaskInput` | | Cancel | Stopping the run sends `tasks/cancel` and marks the row cancelled. Closing the task or stopping its run does the same for a task in the background; a later call reports the cancellation and never revives it. A caller that only disconnects keeps the task. | `cancelMcpTaskRecord`, `mcpTaskStopReason` | | Finish | `failed` becomes a tool error with the provider's message. An unknown task (`-32602`) or a passed TTL becomes a clear error telling the agent to run the action again. Nothing waits forever: every task stops being tracked after its TTL, and seven days after Paperclip first saw it at most. | `provider_task_failed`, `provider_task_expired` | @@ -120,8 +120,6 @@ returns to approved, and its next execution attaches to the same task. - Sampling and roots input requests. - Showing a long-running provider action's progress (`statusMessage`) in the task chat. -- Validating question-card answers against the provider's form schema (types, - required fields) and letting the person correct them before `tasks/update`. ## Tests diff --git a/doc/connections/README.md b/doc/connections/README.md index f2d7edca5d9..31fb2959d61 100644 --- a/doc/connections/README.md +++ b/doc/connections/README.md @@ -19,8 +19,10 @@ MCP protocol versions (2026-07-28 stateless MCP and older handshake servers): MCP extensions: [MCP Apps](./MCP-APPS.md) (interactive views in the task chat), [MCP Skills](./MCP-SKILLS.md) (skills that connected apps publish, reviewed -by the board and delivered to agents), and [MCP Tasks](./MCP-TASKS.md) (long -provider calls that survive timeouts, restarts, and deploys). +by the board and delivered to agents), [MCP Tasks](./MCP-TASKS.md) (long +provider calls that survive timeouts, restarts, and deploys), and +[MCP elicitation](./MCP-ELICITATION.md) (an app's question to the person, +answered in the task chat). Long-term memory: [Experimental memory connectors](./MEMORY.md). diff --git a/doc/plans/2026-11-13-mcp-elicitation.md b/doc/plans/2026-11-13-mcp-elicitation.md new file mode 100644 index 00000000000..9f063000bfe --- /dev/null +++ b/doc/plans/2026-11-13-mcp-elicitation.md @@ -0,0 +1,342 @@ +# MCP elicitation: an app's question becomes a task-chat card, and the agent's next call carries the answer + +Status: plan, 2026-11-13. Owner: the mcp-elicitation session. Builds on #21119 and #21266 (MCP Apps), #21140 (stateless 2026-07-28 client), #21219 (Skills over MCP) and #21263 (MCP Tasks). + +## Summary + +**Recommendation: the run yields while it waits.** When a connected MCP app asks the person a question in the middle of a tool call, Paperclip does four things: + +1. It saves the question durably. +2. It shows the question as a question card in the task chat, titled with the app's name. +3. It tells the agent to wait. +4. When someone answers, it wakes the agent through the existing answer-delivery outbox. + +The agent then calls the tool again with the same arguments, and Paperclip sends the answer to the app exactly once. Nothing blocks and nothing stays in memory, so a restart at any point leaves no stuck call. + +The flow covers both protocol eras: + +- **Stateless servers (2026-07-28)** return `input_required`. Paperclip repeats the call with `inputResponses` and the exact `requestState`. +- **Older servers (2025-06-18 and 2025-11-25)** send `elicitation/create` over an open stream. + - The first time, Paperclip answers `cancel` at once, so the server is never left hanging. + - Once the person answers, the repeated call answers the same question from the card. + +Three rules hold across both eras: + +- Only people can answer. Agents cannot. +- Forms that ask for passwords, keys or card numbers are refused and never shown. +- Choosing **Skip** sends `decline`. A withdrawn, closed or expired card sends `cancel`. + +## Foreword + +- **Context.** Today Paperclip handles only part of what an MCP app can ask: + - **URL requests** already become a provider link that the person opens. + - **Form questions from background MCP Tasks** already become a question card, and the answer goes back with `tasks/update`. +- **Need.** Every other form question is a dead end: + - A stateless server's form `input_required` ends in a "needs information Paperclip can't collect yet" error. + - An older server's `elicitation/create` creates a card, but nothing ever reads the answer back. The card key `mcp-elicitation:` is never looked up again, so the server asks again on every retry. + - Paperclip declares only `elicitation: { url: {} }`. A compliant stateless server therefore never asks a form question at all. +- **Task.** Make form elicitation work end to end in the task chat for both eras. The work must be restart-safe, company-scoped and logged in the activity log. +- **Object.** The document is organized as follows: + - §1: what the spec requires. + - §2: what exists today. + - §3: the nine design decisions. + - §4: the user journey. + - §5: tests. + - §6: risks and what is out of scope. + +## 1. The spec: two transports carry one question model + +Form and URL requests share one data model. Only the transport differs between eras. + +### 1.1 Data model (client/elicitation, 2026-07-28) + +- **Request.** `elicitation/create` has these parameters: + - `mode`: `"form"` (the default when omitted) or `"url"`. + - `message`: required. + - Form mode adds `requestedSchema`. + - URL mode adds `url` (and `elicitationId` on 2025-11-25). +- **Form schema.** A flat object whose properties are primitives: + - **string**: `minLength`, `maxLength`, and `format` (one of `email`, `uri`, `date`, `date-time`). + - **number / integer**: `minimum` and `maximum`. + - **boolean**. + - **Enums (SEP-1330):** + - Single select: `enum`, or `oneOf: [{const, title}]`. + - Multi select: `type: "array"` with `items.enum` or `items.anyOf: [{const, title}]`, plus `minItems` and `maxItems`. + - **Defaults:** every type may carry a `default` (SEP-1034), which clients SHOULD pre-fill. +- **Response.** One of three actions: + - `{ action: "accept", content }`. + - `{ action: "decline" }`: the person explicitly said no. + - `{ action: "cancel" }`: the request was dismissed with no choice made. + - For URL mode, `accept` carries no content. It means "the person agreed to open the link", not "the step is done". +- **Capability.** The client declares `elicitation: { form: {}, url: {} }`, per request on 2026-07-28 and at `initialize` on older versions. `{}` alone means form only. A server MUST NOT use a mode that the client did not declare. + +### 1.2 Transport + +- **2026-07-28 (stateless).** Multi round-trip requests replace server-initiated requests. + - On `tools/call`, `resources/read` or `prompts/get`, the server returns `resultType: "input_required"`. The result carries an `inputRequests` map (server-chosen keys) and/or an opaque `requestState`. + - The client builds every requested input, then retries the *same* request with a new JSON-RPC `id`. The retry carries `inputResponses` (same keys) and echoes `requestState` byte for byte. The client MUST NOT inspect or alter `requestState`, and MUST NOT reuse it on any other request. + - Retries may be delayed arbitrarily. Servers SHOULD re-ask for anything missing rather than fail. +- **2025-06-18 / 2025-11-25 (sessions).** The server sends a JSON-RPC request `elicitation/create` inside the open `tools/call` SSE stream. The client POSTs the JSON-RPC response while the stream stays open. + - In 2025-11-25, URL mode can also arrive as error `-32042` carrying `data.elicitations`. + +### 1.3 Security rules Paperclip must meet + +- **Sensitive data.** Servers MUST NOT request secrets through forms: passwords, API keys, tokens or payment credentials. Such requests MUST use URL mode. +- **Consent and identity.** Clients MUST: + - Show which server is asking. + - Offer clear decline and cancel options. + - Let the person review and edit a form before it is sent. + - Before opening a URL, show the full address and get consent. + - Clients SHOULD also validate answers against the schema. +- **Links.** Clients MUST NOT prefetch URLs. They SHOULD NOT render links inside form text as clickable. + +## 2. Inventory: the pieces exist; the answer path does not + +### 2.1 Gateway (`server/src/services/tool-gateway.ts`) + +| Piece | Where | Today | +| --- | --- | --- | +| `extractMcpElicitationRequest` | `:6498` | Recognizes `elicitation/create` and vendor `_meta.elicitation` shapes | +| `elicitationQuestions` | `:6541` | Up to 10 properties. Enum becomes single select; everything else becomes "Provide answer" plus free text. Ignores type, format, default and multi select | +| `createElicitationCard` | `:6603` | `ask_user_questions` card titled "Tool needs input". No app name; resolver policy left at the default `anyone`, so agents may answer | +| `requestElicitationForRecordedToolCall` | `:6629` | Marks the invocation `awaiting_approval`, writes a call event and an audit entry, then throws 409 `elicitation_required` | +| Legacy stream `onRequest` | `:7928` | URL request: POSTs `cancel` and starts the link flow. Form request: creates a card and throws **without answering the server** | +| `mcpInputRequiredPending` | `:704` | URL-only rounds become the link flow; a bare `requestState` is kept in memory; **form rounds return `answerable: false`** | +| Modern `input_required` branch | `:8040` | A form round ends in `provider_input_required` ("can't collect yet") | +| `mcpContinuations` | `:1406` | In-memory `requestState` cache (lost on restart) | +| Capabilities (`server/src/services/mcp-http.ts:57`) | | Modern: `elicitation: { url: {} }`. Legacy: none | + +### 2.2 MCP Tasks (#21263, `doc/connections/MCP-TASKS.md`) + +- `planMcpTaskInput` (`:7125`) looks up the card by `mcp-task-input::`: + - Pending: wait. + - Answered with no `outcome`: `accept` with `elicitationContent`. + - Anything else: `decline`. +- The answer goes back with `tasks/update`. +- **This plan reuses its card builder and answer mapping and does not duplicate them.** Both move into one module so that Tasks cards gain the same fields and app name. + +### 2.3 Question cards (`issue_thread_interactions`, kind `ask_user_questions`) + +- **Shape.** The payload can carry a canonical `questionSet` (`packages/shared/src/types/issue.ts:1385`): + - `answerMode`: `single_select`, `multi_select` or `text`. + - `initialText` for defaults. + - `recommended` on options. + - `textValidation` with `minLength`, `maxLength`, `pattern`, `inputType` (number or integer), `minimum` and `maximum`. +- **Validation.** On submit, the server validates answers against the `questionSet` (`question-interaction-answers.ts:69`, 422 `invalid_question_response`). +- **Statuses.** + - Answered: `answered`. + - Skip: `cancelled` with `outcome: "skipped"`. + - Withdraw or close: `cancelled` with an outcome. + - Superseded: `expired`. + - Dismiss changes nothing. +- **Who may answer.** The respond route checks issue access, then the interaction's resolver policy (`issue-thread-interaction-resolution.ts`). `human_only` keeps agents out. +- **Wake on answer.** `answerQuestions` writes an `issue_question_response_deliveries` outbox row. The delivery worker wakes the assignee (`question-response-delivery.ts:1170`) with retries. The wake is durable. +- **UI.** + - `TaskChatInteractionCard` shows "Unanswered question" and opens `AskUserQuestionsCard` / `QuestionForm` above the composer. + - The header shows `interaction.title`. + - Text, number and integer fields with validation already exist. + +### 2.4 Waiting on a person elsewhere + +- **Native runtime requests** (`native-runtime/runtime-request-resolution-authority.ts`) hold a live harness turn open and fall back to a durable card. +- **Tool approvals** (`tool_action_requests`) yield the call. A later identical call collects the approval. +- Elicitation follows the approval pattern (yield, then collect), not the harness pattern. A remote HTTP call cannot be held across a restart. + +## 3. Design decisions + +Each decision states the chosen option first, then why. + +### D1. Waiting: the run yields; the agent's repeat call collects the answer + +- **Chosen.** The call ends at once with 409 `elicitation_required` and a plain message: + + > "Linear asked a question in the task chat. Paperclip wakes you when it is answered. Then call this tool again with the same arguments. Don't ask the question yourself." + + An answer wakes the assignee through the durable question-response outbox. A repeat call before every card is resolved returns the same message without contacting the app. +- **Why not block with a time limit?** Blocking would: + - Hold an HTTP stream and a run slot for minutes. + - Die on deploys and restarts. + - Need a second wake path anyway when the time limit passes. + + Yielding reuses the outbox, which already survives restarts, and matches the approval pattern. +- **Recovery.** Everything the retry needs lives in Postgres: + - The `tool_mcp_elicitations` row, with `requestState` sealed. + - The cards. + + A send is guarded by a lease (`claimed_at`). If the process dies mid-send, the next call reclaims the row once the lease is older than the tool timeout plus a margin. The answer is not lost and is not sent twice. + +### D2. Who may answer: any person with access to the task, never an agent; the card names the app + +- **Chosen.** `resolverPolicy: "human_only"` on every elicitation card. This applies to direct and Tasks questions alike. +- **Why not only the responsible user?** That person may be away, and other questions in the task chat already accept any member with access. Narrowing to one person would strand the run. +- **Why not agents?** Letting an agent answer its own tool's question defeats user consent. +- **Showing the app.** + - Card title: "*App name* needs your input". The app name comes from the connection's display name. + - Card description: the server's `message`, redacted, as plain text, so links are not clickable. + - Submit label: "Send to *App*". + - The person sees no raw IDs. + +### D3. Mapping: every spec type becomes a real field; anything else is refused clearly + +| Schema | Card field (`questionSet`) | +| --- | --- | +| `string` (+ `minLength`/`maxLength`) | `text` with those limits. `default` becomes `initialText` | +| `string` with `format` `email` / `date` / `date-time` / `uri` | `text` with a format hint. `pattern` for `email` and `date` | +| `number` / `integer` (+ `minimum`/`maximum`) | `text` with `inputType` and range. `default` becomes `initialText` | +| `boolean` | `single_select` Yes / No. `default` marks the recommended option | +| `enum`, `oneOf[{const,title}]` | `single_select`. Titles are labels; `default` is recommended | +| `array` of `items.enum` / `items.anyOf` | `multi_select`, with `minItems`/`maxItems` as help text | +| No properties | One `single_select` "Continue" option, sending `accept` with `{}` | + +- **Answers back.** Answers map back to typed `content`: enum options return their `const` value, and numbers and booleans are typed. +- **Required fields.** Required follows `schema.required`. +- **Refused shapes.** Nested objects, arrays of non-enums, unknown types, more than 10 properties, more than 4 form requests in one round, or more than 100 options. Paperclip shows no card. The agent gets `elicitation_unsupported`: + + > "*App* asked a question Paperclip can't show here. Complete this step in *App*, then try again." + + A legacy server gets `cancel`. A stateless server needs no reply. + +### D4. Sensitive fields: refuse, don't warn + +- **Chosen.** If any field's name, title or description matches the secret pattern, the whole request is refused before a card exists. This covers password, passcode, secret, API key, access or refresh token, credential, private key, card number, CVV/CVC, SSN, one-time code and 2FA/MFA. A field marked `format: "password"` or `writeOnly` is refused the same way. +- **Why refuse rather than warn?** The spec forbids such forms. A warning still puts a secret into the task thread and the activity history, where it stays. +- **What happens on refusal.** + - The agent gets `elicitation_sensitive_refused`: + + > "*App* asked for a password or key in a form. Paperclip doesn't collect secrets that way; the app must use a secure link." + + - A legacy server gets `cancel`. + - The refusal is logged with field names only, never values. + +### D5. What the agent sees + +| Moment | Agent sees | +| --- | --- | +| Question asked | 409 `elicitation_required` with the wait-and-retry message, `interactionIds` | +| Retried too early | The same message. The app is not contacted | +| Answered, skipped or closed, then retried | The app's real result for `accept` / `decline` / `cancel` (the app decides what declining means) | +| App asks again | A new question; the cycle repeats | +| Round expired before an answer | The call goes out fresh, so the app asks again | + +The agent never sees the answer values through Paperclip's message. Only the app's own result reaches it. + +### D6. Persistence: one company-scoped table, sealed state, logged actions, 24-hour expiry + +- **Table `tool_mcp_elicitations`.** One row per question round. Its columns: + - Scope: `company_id`, `issue_id`, `agent_id`, `run_id`, `connection_id`, `grant_id`, `invocation_id`. + - `call_identity`: an HMAC over endpoint, tool, arguments, grant and agent. + - `era`: `modern` or `legacy`. + - `sealed_request_state`: AES-256-GCM, bound to company and connection. + - `questions`: jsonb `[{key, questionHash, interactionId, mode}]`. It holds no answers; the card is the single source of truth. + - `status`: `waiting`, `sent`, `expired` or `refused`. + - Bookkeeping: `claimed_at`, `expires_at`, `created_at`, `updated_at`. +- **Lookups.** Every lookup filters by company, issue, connection, grant, agent and call identity. +- **Expiry.** Rows expire after 24 hours. Expiry is checked lazily on the next call, so no background poller is added (this follows the sleep-aware-worker rule). On expiry, pending cards are withdrawn through the interaction service, so nobody answers a dead question. The app's own `requestState` TTL is usually shorter; if the app rejects stale state, it SHOULD re-ask. +- **Cap.** At most 10 waiting rounds per task. Beyond that the call fails with "too many open questions" instead of spamming cards. +- **Activity log.** The log records these events: + - `tool_gateway.elicitation_requested` (exists). + - `tool_gateway.elicitation_sent`, with the actions only. + - `tool_gateway.elicitation_refused`. + - `tool_gateway.elicitation_expired`. + - The existing `issue.thread_interaction_answered`. + + Answer values are never logged. + +### D7. Retry semantics: stateless servers are fully supported; older servers are answered from the card + +- **Stateless (2026-07-28).** + - Paperclip declares `elicitation: { form: {}, url: {} }` on HTTP `tools/call` when the call belongs to a task. That is the only place a card can be shown. + - **Form rounds.** On `input_required` with form requests, Paperclip saves the round and creates the cards, which ends the call. + - **The repeat call.** It finds the waiting round, checks that every card is resolved and claims the row. It then sends one `tools/call` with a new id, `inputResponses` for every key, and the unsealed `requestState`. + - **Mixed rounds.** If a round mixes URL and form requests, the URL link flow goes first and is not persisted. Paperclip never claims URL consent that nobody gave. The fresh call that follows lets the server re-ask for the form. + - **Bare `requestState`.** A round with only `requestState` keeps today's in-memory retry. +- **Legacy (2025-06-18 / 2025-11-25).** Paperclip declares `elicitation: { form: {}, url: {} }` at `initialize` (`{}` on 2025-06-18). + - **When a form `elicitation/create` arrives on the stream,** Paperclip computes `questionHash`, a hash of the message and the canonical schema. + - **If an answered, unsent round matches** this call identity and hash, Paperclip claims it, POSTs `accept` / `decline` / `cancel` and keeps reading the stream. + - **Otherwise** it saves a round, creates the card and POSTs `cancel`, so the server stops waiting. Then it ends the call with `elicitation_required`. + - **Why answer ahead from a saved card?** The person consented to that exact question for that exact call. Holding the stream open would break D1. +- **Never supported.** The vendor `_meta.elicitation` result shape keeps today's card-only behavior: it has no channel for answers. + +### D8. Tasks interplay: one mapping, two transports + +- The schema mapping, sensitive check, card builder and answer mapping move to `server/src/services/mcp-elicitation.ts`. MCP Tasks uses them unchanged in behavior, and its cards gain the app name, real fields and `human_only`. +- **With `enableMcpTasks` off,** `tools/call` never returns a task, and D7 handles every question. +- **With it on,** + - A task's own `input_required` keeps its `tasks/update` path. + - A non-task `input_required` uses D7. + - The two never share a card key: tasks use `mcp-task-input:`, direct calls use `mcp-elicit:`. + +### D9. MCP Apps: a view's tool call asks in the same chat + +A tool call triggered from an MCP App view already runs in the task's gateway session, so D7 applies unchanged: + +- The card appears in the chat beside the view. +- The view's call receives the wait message. +- Answering wakes the agent, which repeats the call. + +Out of scope: a form rendered inside the view. + +## 4. User journey and UI states + +1. **Asking.** The agent calls Linear's `create_issue`, and Linear asks "Which team should own this?" The task chat shows "Unanswered question". Opening it shows the form: + - Title: "Linear needs your input". + - The app's message as plain text. + - The fields. + - **Send to Linear** and **Skip**. +2. **Answered.** The receipt shows the answers. The agent wakes, repeats the call and posts the result. +3. **Declined.** The person chooses **Skip**. The receipt reads "Skipped". The app receives `decline`, and the agent reports what the app did. +4. **Expired.** If nobody answers for 24 hours and the agent repeats the call, the card is withdrawn and its receipt reads "Withdrawn". The app asks again on a new card. +5. **Failed.** If the app rejects the answer (for example, a stale state), the agent receives the app's error and explains it. If the app asked for a secret, no card appears; the agent explains that the app must use a secure link. + +**Storybook.** `ui/storybook/stories/questions-and-approvals/mcp-elicitation-question-card.stories.tsx` covers these stories: + +- Asking, with all field types. +- Answered. +- Skipped. +- Withdrawn or expired. +- Two questions at once. + +The fixtures are generated from the server's own mapper output, so the stories show the real card. + +## 5. Test plan (red on master, green on the branch) + +| # | Test | Proves | +| --- | --- | --- | +| T1 | Fake 2026-07-28 server: `input_required` form produces a card and `elicitation_required`; answer, repeat the call, and the server receives `inputResponses` + exact `requestState` | D1, D7 modern | +| T2 | Same, with Skip producing `decline` and a withdrawn card producing `cancel` | D5 actions | +| T3 | Repeat before an answer: no upstream request | D1 | +| T4 | Two form keys in one round produce two cards; a single send carries both answers | Two questions at once | +| T5 | New gateway instance (simulated restart) between ask and repeat; a stale `claimed_at` is reclaimed | Restart-safe | +| T6 | Fake legacy server over SSE: first ask receives `cancel` immediately and gets a card; after the answer, the repeat call's stream receives `accept` + content | D7 legacy | +| T7 | A password field is refused: no card, `elicitation_sensitive_refused`, legacy receives `cancel` | D4 | +| T8 | Company B's session with the same tool and arguments does not collect company A's round; an agent actor cannot answer the card | Scope, permissions | +| T9 | Unit: every schema type maps and round-trips (SEP-1330 single/multi, titled/untitled, SEP-1034 defaults) | D3 | +| T10 | MCP Tasks suites stay green, and a Tasks card now has `human_only` and the app title | D8 | +| T11 | Capabilities: modern declares `form` only when the call has a task; legacy `initialize` declares form | D7 | + +**Live demo.** A local fake 2026-07-28 server, connected as a custom remote MCP app in a dev instance. An agent calls it, the question appears in the task chat, the answer is submitted in the browser, and the agent finishes. This is recorded in the PR. + +## 6. Risks and scope + +- **Older servers that never asked may start asking.** Declaring form on `initialize` invites questions. Mitigation: the first ask is always answered `cancel` at once, so no call hangs, and the agent gets a clear next step. +- **Answer-ahead could answer a different intent.** It matches only the same task, connection, grant, agent, tool, arguments, message and schema, within 24 hours, once. +- **Stale cards.** Expiry is lazy. A card can sit unanswered past 24 hours until the agent next calls; answering it then just wakes the agent, which re-asks. This is acceptable without a poller. +- **Migration.** One additive table, with no backfill. It is online-safe. + +**Out of scope (follow-ups):** + +- Sampling and roots requests. +- Form elicitation from stdio apps (they keep URL only). +- Forms rendered inside an MCP App view. +- `resources/read` and `prompts/get` input rounds. +- The vendor `_meta.elicitation` answer path. + +## Conclusion + +The plan ships form elicitation for both MCP eras without blocking: + +- The question becomes a named, human-only card. +- The answer travels on the agent's repeat call, sent once from durable state. +- Secrets and unsupported shapes are refused up front. + +**Next:** build `mcp-elicitation.ts`, the table, the gateway wiring and the tests (T1–T11). Then add the stories and `doc/connections/MCP-ELICITATION.md`, and open the PR. diff --git a/packages/db/src/migrations/0483_busy_tombstone.sql b/packages/db/src/migrations/0483_busy_tombstone.sql new file mode 100644 index 00000000000..1c406d014e7 --- /dev/null +++ b/packages/db/src/migrations/0483_busy_tombstone.sql @@ -0,0 +1,30 @@ +CREATE TABLE "tool_mcp_elicitations" ( + "id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL, + "company_id" uuid NOT NULL, + "issue_id" uuid NOT NULL, + "agent_id" uuid, + "run_id" uuid, + "invocation_id" uuid NOT NULL, + "connection_id" uuid NOT NULL, + "grant_id" uuid NOT NULL, + "catalog_tool_name" text NOT NULL, + "call_identity" text NOT NULL, + "era" text NOT NULL, + "sealed_request_state" text, + "questions" jsonb NOT NULL, + "arguments_identity" text NOT NULL, + "dedupe_key" text NOT NULL, + "status" text NOT NULL, + "claimed_until" timestamp with time zone, + "expires_at" timestamp with time zone NOT NULL, + "sent_at" timestamp with time zone, + "created_at" timestamp with time zone DEFAULT now() NOT NULL, + "updated_at" timestamp with time zone DEFAULT now() NOT NULL +); +--> statement-breakpoint +ALTER TABLE "tool_mcp_elicitations" ADD CONSTRAINT "tool_mcp_elicitations_company_id_companies_id_fk" FOREIGN KEY ("company_id") REFERENCES "public"."companies"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint +ALTER TABLE "tool_mcp_elicitations" ADD CONSTRAINT "tool_mcp_elicitations_invocation_id_tool_invocations_id_fk" FOREIGN KEY ("invocation_id") REFERENCES "public"."tool_invocations"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint +ALTER TABLE "tool_mcp_elicitations" ADD CONSTRAINT "tool_mcp_elicitations_connection_id_tool_connections_id_fk" FOREIGN KEY ("connection_id") REFERENCES "public"."tool_connections"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint +CREATE INDEX "tool_mcp_elicitations_call_idx" ON "tool_mcp_elicitations" USING btree ("company_id","issue_id","connection_id","call_identity","status");--> statement-breakpoint +CREATE INDEX "tool_mcp_elicitations_invocation_idx" ON "tool_mcp_elicitations" USING btree ("company_id","invocation_id");--> statement-breakpoint +CREATE UNIQUE INDEX "tool_mcp_elicitations_waiting_uq" ON "tool_mcp_elicitations" USING btree ("company_id","issue_id","dedupe_key") WHERE "tool_mcp_elicitations"."status" = 'waiting'; \ No newline at end of file diff --git a/packages/db/src/migrations/meta/0478_snapshot.json b/packages/db/src/migrations/meta/0483_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0478_snapshot.json rename to packages/db/src/migrations/meta/0483_snapshot.json index d7f274a6422..3eb96902c08 100644 --- a/packages/db/src/migrations/meta/0478_snapshot.json +++ b/packages/db/src/migrations/meta/0483_snapshot.json @@ -1,6 +1,6 @@ { - "id": "349cdd89-db04-4340-a5d3-3b877bf7227d", - "prevId": "d5d69127-12de-4281-9816-c20e013bfeb7", + "id": "ceaa38c4-681b-430e-9578-0854650d85b3", + "prevId": "8b353a38-62df-4834-be55-7e9ebefad5c5", "version": "7", "dialect": "postgresql", "tables": { @@ -18092,6 +18092,12 @@ "notNull": true, "default": "'company'" }, + "system_purpose": { + "name": "system_purpose", + "type": "text", + "primaryKey": false, + "notNull": false + }, "owner_user_id": { "name": "owner_user_id", "type": "text", @@ -35261,6 +35267,12 @@ "notNull": true, "default": 1 }, + "native_source_cursors": { + "name": "native_source_cursors", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, "native_phase": { "name": "native_phase", "type": "text", @@ -55947,6 +55959,153 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.tailscale_connections": { + "name": "tailscale_connections", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "secret_id": { + "name": "secret_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "credential_version": { + "name": "credential_version", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "credential_hash": { + "name": "credential_hash", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_by_user_id": { + "name": "created_by_user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "oauth_state_hash": { + "name": "oauth_state_hash", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "oauth_expires_at": { + "name": "oauth_expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "tailscale_connections_company_idx": { + "name": "tailscale_connections_company_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "tailscale_connections_oauth_state_idx": { + "name": "tailscale_connections_oauth_state_idx", + "columns": [ + { + "expression": "oauth_state_hash", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "tailscale_connections_company_id_companies_id_fk": { + "name": "tailscale_connections_company_id_companies_id_fk", + "tableFrom": "tailscale_connections", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "tailscale_connections_secret_id_company_secrets_id_fk": { + "name": "tailscale_connections_secret_id_company_secrets_id_fk", + "tableFrom": "tailscale_connections", + "tableTo": "company_secrets", + "columnsFrom": [ + "secret_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.tool_access_audit_events": { "name": "tool_access_audit_events", "schema": "", @@ -59954,6 +60113,277 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.tool_mcp_elicitations": { + "name": "tool_mcp_elicitations", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "issue_id": { + "name": "issue_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "agent_id": { + "name": "agent_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "run_id": { + "name": "run_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "invocation_id": { + "name": "invocation_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "connection_id": { + "name": "connection_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "grant_id": { + "name": "grant_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "catalog_tool_name": { + "name": "catalog_tool_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "call_identity": { + "name": "call_identity", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "era": { + "name": "era", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "sealed_request_state": { + "name": "sealed_request_state", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "questions": { + "name": "questions", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "arguments_identity": { + "name": "arguments_identity", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "dedupe_key": { + "name": "dedupe_key", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "claimed_until": { + "name": "claimed_until", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "sent_at": { + "name": "sent_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "tool_mcp_elicitations_call_idx": { + "name": "tool_mcp_elicitations_call_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "issue_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "connection_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "call_identity", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "tool_mcp_elicitations_invocation_idx": { + "name": "tool_mcp_elicitations_invocation_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "invocation_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "tool_mcp_elicitations_waiting_uq": { + "name": "tool_mcp_elicitations_waiting_uq", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "issue_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "dedupe_key", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"tool_mcp_elicitations\".\"status\" = 'waiting'", + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "tool_mcp_elicitations_company_id_companies_id_fk": { + "name": "tool_mcp_elicitations_company_id_companies_id_fk", + "tableFrom": "tool_mcp_elicitations", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "tool_mcp_elicitations_invocation_id_tool_invocations_id_fk": { + "name": "tool_mcp_elicitations_invocation_id_tool_invocations_id_fk", + "tableFrom": "tool_mcp_elicitations", + "tableTo": "tool_invocations", + "columnsFrom": [ + "invocation_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "tool_mcp_elicitations_connection_id_tool_connections_id_fk": { + "name": "tool_mcp_elicitations_connection_id_tool_connections_id_fk", + "tableFrom": "tool_mcp_elicitations", + "tableTo": "tool_connections", + "columnsFrom": [ + "connection_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.tool_mcp_gateway_tokens": { "name": "tool_mcp_gateway_tokens", "schema": "", diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index 150d999e51d..54d4380b27f 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -3368,6 +3368,13 @@ "when": 1794604948355, "tag": "0482_aberrant_rumiko_fujikawa", "breakpoints": true + }, + { + "idx": 483, + "version": "7", + "when": 1794611818460, + "tag": "0483_busy_tombstone", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/index.ts b/packages/db/src/schema/index.ts index 66e06586c9a..7eef891d0d3 100644 --- a/packages/db/src/schema/index.ts +++ b/packages/db/src/schema/index.ts @@ -194,6 +194,7 @@ export { toolMcpSkills, toolMcpSkillFiles, toolMcpTasks, + toolMcpElicitations, toolActionRequests, toolCallEvents, toolRateLimitCounters, diff --git a/packages/db/src/schema/tool_access.ts b/packages/db/src/schema/tool_access.ts index 88474beeceb..621597fd6d3 100644 --- a/packages/db/src/schema/tool_access.ts +++ b/packages/db/src/schema/tool_access.ts @@ -966,6 +966,61 @@ export const toolMcpTasks = pgTable( ], ); +/** + * A provider's form question waiting on a person (MCP elicitation). One row per + * question round: a stateless server's `input_required` result, or one older + * server's `elicitation/create`. The question cards hold the answers; this row + * holds what the repeated call needs to send them back, exactly once. + */ +export const toolMcpElicitations = pgTable( + "tool_mcp_elicitations", + { + id: uuid("id").primaryKey().defaultRandom(), + companyId: uuid("company_id").notNull().references(() => companies.id, { onDelete: "cascade" }), + issueId: uuid("issue_id").notNull(), + agentId: uuid("agent_id"), + runId: uuid("run_id"), + /** The call that was asked. */ + invocationId: uuid("invocation_id").notNull().references(() => toolInvocations.id, { onDelete: "cascade" }), + connectionId: uuid("connection_id").notNull().references(() => toolConnections.id, { onDelete: "cascade" }), + /** Only the same grant's repeated call may send the answers. */ + grantId: uuid("grant_id").notNull(), + catalogToolName: text("catalog_tool_name").notNull(), + /** Keyed identity of endpoint, tool, arguments, grant, and agent; only an identical call collects the round. */ + callIdentity: text("call_identity").notNull(), + /** modern: a stateless input_required round. legacy: one server-initiated question. */ + era: text("era").notNull(), + /** The server's opaque requestState, sealed (it may carry authority). */ + sealedRequestState: text("sealed_request_state"), + /** One entry per form question: the server's key, its words and schema (shown on the card, never an answer), and its card. */ + questions: jsonb("questions").$type | null; + }>>().notNull(), + /** Keyed identity of the tool name and full arguments; a keyed retry runs again only for the same request. */ + argumentsIdentity: text("arguments_identity").notNull(), + /** One waiting round per call (and per question, for older servers): matching calls at once share its cards. */ + dedupeKey: text("dedupe_key").notNull(), + /** waiting | sent | expired */ + status: text("status").notNull(), + /** A repeated call sending the answers holds the round until then. */ + claimedUntil: timestamp("claimed_until", { withTimezone: true }), + expiresAt: timestamp("expires_at", { withTimezone: true }).notNull(), + sentAt: timestamp("sent_at", { withTimezone: true }), + createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(), + updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(), + }, + (table) => [ + index("tool_mcp_elicitations_call_idx").on(table.companyId, table.issueId, table.connectionId, table.callIdentity, table.status), + index("tool_mcp_elicitations_invocation_idx").on(table.companyId, table.invocationId), + uniqueIndex("tool_mcp_elicitations_waiting_uq").on(table.companyId, table.issueId, table.dedupeKey) + .where(sql`${table.status} = 'waiting'`), + ], +); + /** * Verified skill files, written only by the server after a size and SHA-256 * check and never changed in place. Rows are keyed by connection and digest, diff --git a/server/src/__tests__/issue-thread-interaction-routes.test.ts b/server/src/__tests__/issue-thread-interaction-routes.test.ts index f36beac95c1..73b3bf5b8a6 100644 --- a/server/src/__tests__/issue-thread-interaction-routes.test.ts +++ b/server/src/__tests__/issue-thread-interaction-routes.test.ts @@ -1124,6 +1124,34 @@ describe("issue thread interaction routes", () => { expect(mockHeartbeatService.wakeup).not.toHaveBeenCalled(); }); + it.each([ + ["an app's question (MCP elicitation)", "mcp-elicit:round-1:abc", 1], + ["an ordinary question", "agent-question-1", 0], + ] as const)("wakes the assignee when a person skips %s, so the app receives the decline", async (_name, idempotencyKey, wakes) => { + mockInteractionService.skipInteraction.mockResolvedValue({ + id: "interaction-skip", + companyId: "company-1", + issueId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", + kind: "ask_user_questions", + status: "cancelled", + continuationPolicy: "wake_assignee", + idempotencyKey, + sourceCommentId: null, + sourceRunId: RUN_2, + payload: { version: 1, questions: [] }, + result: { version: 1, answers: [], outcome: "skipped" }, + }); + const app = await createApp(); + + const res = await request(app) + .post("/api/issues/aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa/interactions/interaction-skip/skip") + .send({}); + + expect(res.status).toBe(200); + expect(mockHeartbeatService.wakeup).toHaveBeenCalledTimes(wakes); + if (wakes) expect(mockHeartbeatService.wakeup).toHaveBeenCalledWith(ASSIGNEE_AGENT_ID, expect.anything()); + }); + it("submits item verdicts and emits one continuation wake with resolved item ids", async () => { const app = await createApp(); diff --git a/server/src/__tests__/mcp-elicitation.test.ts b/server/src/__tests__/mcp-elicitation.test.ts new file mode 100644 index 00000000000..a43c8944794 --- /dev/null +++ b/server/src/__tests__/mcp-elicitation.test.ts @@ -0,0 +1,696 @@ +import { randomUUID } from "node:crypto"; +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { + activityLog, + agents, + approvals, + toolActionRequests, + companies, + companyMemberships, + createDb, + heartbeatRuns, + issues, + issueThreadInteractions, + projects, + toolConnections, + toolInvocations, + toolMcpElicitations, +} from "@paperclipai/db"; +import { and, eq } from "drizzle-orm"; +import { paperclipQuestionSetPayloadSchema } from "@paperclipai/shared"; +import { forgetMcpHttpSessions, MCP_MODERN_PROTOCOL_VERSION, mcpClientCapabilities, mcpHttpRequestInit } from "../services/mcp-http.js"; +import { + buildElicitationQuestionSet, + elicitationContent, + elicitationQuestionHash, + elicitationResponseForCard, + elicitationSensitiveFields, + openElicitationState, + sealElicitationState, +} from "../services/mcp-elicitation.js"; +import { mcpInputRequiredPending } from "../services/tool-gateway.js"; +import { mcpInputRequired } from "../services/mcp-http.js"; +import { createToolGatewayService } from "../services/tool-gateway.js"; +import { toolAccessService } from "../services/tool-access.js"; +import { issueThreadInteractionService } from "../services/issue-thread-interactions.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; + +const META_VERSION = "io.modelcontextprotocol/protocolVersion"; +const META_CAPABILITIES = "io.modelcontextprotocol/clientCapabilities"; + +/** Every spec form type (SEP-1330 enums, SEP-1034 defaults). */ +const TICKET_SCHEMA = { + type: "object", + required: ["title", "priority"], + properties: { + title: { type: "string", title: "Ticket title", minLength: 3, maxLength: 80, default: "Untitled" }, + contact: { type: "string", format: "email", title: "Contact email" }, + priority: { type: "string", title: "Priority", oneOf: [{ const: "p1", title: "Urgent" }, { const: "p2", title: "Normal" }], default: "p2" }, + team: { type: "string", title: "Team", enum: ["ops", "eng"] }, + labels: { type: "array", title: "Labels", minItems: 1, maxItems: 2, items: { anyOf: [{ const: "bug", title: "Bug" }, { const: "ux", title: "UX" }, { const: "perf", title: "Speed" }] }, default: ["bug"] }, + tags: { type: "array", title: "Tags", items: { type: "string", enum: ["a", "b"] } }, + urgent: { type: "boolean", title: "Page someone?", default: false }, + count: { type: "integer", title: "How many?", minimum: 1, maximum: 5, default: 1 }, + budget: { type: "number", title: "Budget" }, + }, +}; + +describe("MCP form elicitation mapping", () => { + it("maps every spec field type to a real card field, with defaults", () => { + const built = buildElicitationQuestionSet({ appName: "Linear", request: { message: "Tell us about the ticket", requestedSchema: TICKET_SCHEMA } }); + expect(built.ok).toBe(true); + if (!built.ok) return; + // The card passes the same validation every question card does. + expect(paperclipQuestionSetPayloadSchema.safeParse(built.questionSet).success).toBe(true); + expect(built.questionSet).toMatchObject({ title: "Linear needs your input", description: "Tell us about the ticket", submitLabel: "Send to Linear" }); + const byId = Object.fromEntries(built.questionSet.questions.map((question) => [question.id, question])); + expect(byId.title).toMatchObject({ answerMode: "text", required: true, initialText: "Untitled", textValidation: { minLength: 3, maxLength: 80 } }); + // The question form checks only fixed-width patterns: an email is a hint, a date is checked. + expect(byId.contact).toMatchObject({ answerMode: "text", required: false, helpText: "An email address." }); + expect(byId.contact).not.toHaveProperty("textValidation"); + expect(buildElicitationQuestionSet({ appName: "A", request: { message: "m", requestedSchema: { properties: { d: { type: "string", format: "date" } } } } })) + .toMatchObject({ ok: true, questionSet: { questions: [{ textValidation: { pattern: "^\\d\\d\\d\\d-\\d\\d-\\d\\d$" } }] } }); + expect(byId.priority).toMatchObject({ answerMode: "single_select", options: [{ label: "Urgent" }, { label: "Normal", recommended: true }] }); + expect(byId.team).toMatchObject({ answerMode: "single_select", options: [{ label: "ops" }, { label: "eng" }] }); + expect(byId.labels).toMatchObject({ answerMode: "multi_select", helpText: "Choose at least 1. Choose at most 2.", options: [{ label: "Bug", recommended: true }, { label: "UX" }, { label: "Speed" }] }); + expect(byId.tags).toMatchObject({ answerMode: "multi_select" }); + expect(byId.urgent).toMatchObject({ answerMode: "single_select", options: [{ label: "Yes" }, { label: "No", recommended: true }] }); + expect(byId.count).toMatchObject({ answerMode: "text", initialText: "1", textValidation: { inputType: "integer", minimum: 1, maximum: 5 } }); + expect(byId.budget).toMatchObject({ answerMode: "text", textValidation: { inputType: "number" } }); + }); + + it("turns card answers back into typed content on the provider's schema", () => { + const request = { message: "m", requestedSchema: TICKET_SCHEMA }; + const content = elicitationContent(request, { answers: [ + { questionId: "title", optionIds: ["paperclip_text_answer"], otherText: " Login broken " }, + { questionId: "priority", optionIds: ["choice_1"] }, + { questionId: "labels", optionIds: ["choice_1", "choice_3"] }, + { questionId: "urgent", optionIds: ["choice_1"] }, + { questionId: "count", optionIds: [], otherText: "3" }, + { questionId: "budget", optionIds: [], otherText: "12.5" }, + ] }); + expect(content).toEqual({ title: "Login broken", priority: "p1", labels: ["bug", "perf"], urgent: true, count: 3, budget: 12.5 }); + }); + + it("answers accept, decline (skipped), or cancel (anything else ending the card)", () => { + const request = { message: "Continue?", requestedSchema: { type: "object", properties: {} } }; + expect(elicitationResponseForCard(request, { status: "pending", result: null })).toBeNull(); + expect(elicitationResponseForCard(request, { status: "answered", result: { version: 1, answers: [{ questionId: "continue", optionIds: ["continue"] }] } })) + .toEqual({ action: "accept", content: {} }); + expect(elicitationResponseForCard(request, { status: "cancelled", result: { version: 1, answers: [], outcome: "skipped" } })).toEqual({ action: "decline" }); + expect(elicitationResponseForCard(request, { status: "cancelled", result: { version: 1, answers: [], outcome: "withdrawn" } })).toEqual({ action: "cancel" }); + expect(elicitationResponseForCard(request, { status: "expired", result: null })).toEqual({ action: "cancel" }); + }); + + it("reads a card saved before typed fields (an MCP Task waiting across the upgrade)", () => { + const request = { message: "m", requestedSchema: { type: "object", properties: { + workspace: { enum: ["ops", "engineering"] }, page: { type: "boolean" }, count: { type: "integer" }, + } } }; + const payload = { version: 1, questions: [ + { id: "workspace", prompt: "workspace", selectionMode: "single", options: [{ id: "ops", label: "ops" }, { id: "engineering", label: "engineering" }] }, + { id: "page", prompt: "page", selectionMode: "single", options: [{ id: "answer", label: "Provide answer" }] }, + { id: "count", prompt: "count", selectionMode: "single", options: [{ id: "answer", label: "Provide answer" }] }, + ] }; + const result = { version: 1, answers: [ + { questionId: "workspace", optionIds: ["engineering"] }, + { questionId: "page", optionIds: [], otherText: "yes" }, + { questionId: "count", optionIds: [], otherText: "4" }, + ] }; + expect(elicitationResponseForCard(request, { status: "answered", result, payload })) + .toEqual({ action: "accept", content: { workspace: "engineering", page: true, count: 4 } }); + }); + + it("refuses forms that ask for secrets, and shapes outside the spec", () => { + expect(elicitationSensitiveFields({ properties: { password: { type: "string" } } })).toEqual(["password"]); + expect(elicitationSensitiveFields({ properties: { key: { type: "string", title: "Your API key" } } })).toEqual(["key"]); + expect(elicitationSensitiveFields({ properties: { code: { type: "string", description: "The one-time code we texted you" } } })).toEqual(["code"]); + expect(elicitationSensitiveFields({ properties: { x: { type: "string", format: "password" } } })).toEqual(["x"]); + expect(elicitationSensitiveFields({ properties: { name: { type: "string" }, max_tokens: { type: "integer" } } })).toEqual([]); + const ask = (requestedSchema: Record) => buildElicitationQuestionSet({ appName: "App", request: { message: "m", requestedSchema } }); + expect(ask({ properties: { card_number: { type: "string" } } })).toEqual({ ok: false, reason: "sensitive", fields: ["card_number"] }); + // The message itself asking for a secret is refused, whatever the field is called. + expect(buildElicitationQuestionSet({ appName: "App", request: { message: "Type your account password in the notes field", requestedSchema: { properties: { notes: { type: "string" } } } } })) + .toEqual({ ok: false, reason: "sensitive", fields: ["message"] }); + expect(buildElicitationQuestionSet({ appName: "App", request: { message: "How many tokens should the summary use?", requestedSchema: { properties: { n: { type: "integer" } } } } }).ok).toBe(true); + expect(ask({ properties: { nested: { type: "object", properties: {} } } })).toEqual({ ok: false, reason: "unsupported" }); + expect(ask({ properties: { list: { type: "array", items: { type: "string" } } } })).toEqual({ ok: false, reason: "unsupported" }); + expect(ask({ properties: Object.fromEntries(Array.from({ length: 11 }, (_, index) => [`f${index}`, { type: "string" }])) })).toEqual({ ok: false, reason: "unsupported" }); + }); + + it("keeps provider text plain: no clickable links, no markup", () => { + const built = buildElicitationQuestionSet({ appName: "App", request: { message: "Open [this](https://evil.example.com) or https://x.example.com", requestedSchema: { properties: {} } } }); + expect(built.ok && built.questionSet.description).toBe("Open [this] (https:​//evil.example.com) or bhttps:​//x.example.com/b"); + }); + + it("seals the server's state to the company and connection that received it", () => { + const scope = { companyId: "c1", connectionId: "k1" }; + const sealed = sealElicitationState("secret", scope, "opaque-state"); + expect(sealed).not.toContain("opaque-state"); + expect(openElicitationState("secret", scope, sealed)).toBe("opaque-state"); + expect(openElicitationState("secret", { ...scope, companyId: "c2" }, sealed)).toBeNull(); + expect(openElicitationState("other", scope, sealed)).toBeNull(); + }); + + it("identifies a question by its words and schema, not key order", () => { + expect(elicitationQuestionHash({ message: "m", requestedSchema: { a: 1, b: { c: 2, d: 3 } } })) + .toBe(elicitationQuestionHash({ message: "m", requestedSchema: { b: { d: 3, c: 2 }, a: 1 } })); + expect(elicitationQuestionHash({ message: "m", requestedSchema: {} })).not.toBe(elicitationQuestionHash({ message: "n", requestedSchema: {} })); + }); + + it("declares form questions only where a person can answer them", () => { + expect(mcpClientCapabilities(MCP_MODERN_PROTOCOL_VERSION).elicitation).toEqual({ url: {} }); + expect(mcpClientCapabilities(MCP_MODERN_PROTOCOL_VERSION, { elicitationForm: true }).elicitation).toEqual({ form: {}, url: {} }); + expect(mcpClientCapabilities("2025-11-25", { elicitationForm: true }).elicitation).toEqual({ form: {}, url: {} }); + expect(mcpClientCapabilities("2025-06-18", { elicitationForm: true }).elicitation).toEqual({}); + expect(mcpClientCapabilities("2025-03-26", { elicitationForm: true })).not.toHaveProperty("elicitation"); + // Older stdio sessions keep what they had. + expect(mcpClientCapabilities("2025-11-25")).not.toHaveProperty("elicitation"); + const body = JSON.parse(mcpHttpRequestInit({ "MCP-Protocol-Version": MCP_MODERN_PROTOCOL_VERSION }, { id: 1, method: "tools/call", params: { name: "x" } }, { elicitationForm: true }).body); + expect(body.params._meta[META_CAPABILITIES].elicitation).toEqual({ form: {}, url: {} }); + }); + + it("shows the link first when a round mixes a link and a form, and never claims consent", () => { + const mixed = mcpInputRequiredPending(mcpInputRequired({ resultType: "input_required", requestState: "s", inputRequests: { + consent: { method: "elicitation/create", params: { mode: "url", message: "Approve", url: "https://auth.example.com/consent" } }, + name: { method: "elicitation/create", params: { message: "Name?", requestedSchema: { type: "object", properties: {} } } }, + } })!); + expect(mixed).toMatchObject({ answerable: true, upstreamPending: { links: [{ url: "https://auth.example.com/consent" }] } }); + expect(mixed).not.toHaveProperty("retryState"); + }); +}); + +type Recorded = { method: string; headers: Headers; body: Record }; + +/** A stateless (2026-07-28) server whose tools ask the person questions. */ +function askingServer() { + const requests: Recorded[] = []; + let refuseAnswers = 0; + const respond = (id: unknown, result: Record) => Response.json({ jsonrpc: "2.0", id, result: { resultType: "complete", ...result } }); + const form = (message: string, requestedSchema: Record) => ({ method: "elicitation/create", params: { mode: "form", message, requestedSchema } }); + const handle = async (_url: string, init: RequestInit): Promise => { + if ((init.method ?? "GET").toUpperCase() !== "POST") return new Response(null, { status: 404 }); + const body = JSON.parse(String(init.body)); + requests.push({ method: body.method, headers: new Headers(init.headers), body }); + if (body.params?._meta?.[META_VERSION] !== MCP_MODERN_PROTOCOL_VERSION) return Response.json({ jsonrpc: "2.0", id: body.id, error: { code: -32020, message: "metadata" } }, { status: 400 }); + if (body.method === "server/discover") return respond(body.id, { supportedVersions: [MCP_MODERN_PROTOCOL_VERSION], capabilities: { tools: {} } }); + if (body.method === "tools/list") { + return respond(body.id, { tools: ["create_ticket", "two_questions", "ask_secret", "two_rounds"].map((name) => ({ name, description: name, inputSchema: { type: "object", properties: {} }, annotations: { readOnlyHint: true } })) }); + } + if (body.method !== "tools/call") return Response.json({ jsonrpc: "2.0", id: body.id, error: { code: -32601, message: "nope" } }); + const { name, inputResponses, requestState } = body.params; + if (inputResponses && refuseAnswers > 0) { refuseAnswers -= 1; return new Response(null, { status: 503 }); } + if (name === "two_rounds" && inputResponses?.first) { + return respond(body.id, { resultType: "input_required", requestState: "round-2", inputRequests: { second: form("And the due date?", { type: "object", properties: { due: { type: "string", format: "date" } }, required: ["due"] }) } }); + } + if (name === "two_rounds" && !inputResponses) { + return respond(body.id, { resultType: "input_required", requestState: "round-1", inputRequests: { first: form("Which team?", { type: "object", properties: { team: { type: "string", enum: ["ops", "eng"] } }, required: ["team"] }) } }); + } + if (inputResponses) return respond(body.id, { content: [{ type: "text", text: JSON.stringify({ inputResponses, requestState: requestState ?? null }) }] }); + if (name === "create_ticket") { + return respond(body.id, { resultType: "input_required", requestState: `state-for-${body.params.arguments?.ref ?? "none"}`, inputRequests: { details: form("Tell us about the ticket", TICKET_SCHEMA) } }); + } + if (name === "two_questions") { + return respond(body.id, { resultType: "input_required", requestState: "two-state", inputRequests: { + who: form("Who should own it?", { type: "object", properties: { owner: { type: "string", title: "Owner" } }, required: ["owner"] }), + go: form("Ship it now?", { type: "object", properties: {} }), + } }); + } + return respond(body.id, { resultType: "input_required", inputRequests: { secret: form("Sign in", { type: "object", properties: { password: { type: "string" } } }) } }); + }; + return { requests, handle, calls: () => requests.filter((request) => request.method === "tools/call"), refuseNextAnswer: () => { refuseAnswers = 1; } }; +} + +/** + * A 2025-06-18 server with a session that asks over the open tools/call stream + * and waits for the client's reply before it finishes, like the SDK servers. + * It asks its questions in turn and stops at the first one not accepted. + */ +function legacyAskingServer(options: { questions?: Array<{ message: string; key: string; values: string[]; askAgainIf?: string }>; refuseReplies?: number } = {}) { + const requests: Recorded[] = []; + const replies: Array> = []; + const questions = options.questions ?? [{ message: "Which team?", key: "team", values: ["ops", "eng"] }]; + let refuseReplies = options.refuseReplies ?? 0; + let sessions = 0; + let asks = 0; + const waiting = new Map) => void>(); + const sse = (message: unknown) => new TextEncoder().encode(`data: ${JSON.stringify(message)}\n\n`); + const handle = async (_url: string, init: RequestInit): Promise => { + if ((init.method ?? "GET").toUpperCase() !== "POST") return new Response(null, { status: 404 }); + const body = JSON.parse(String(init.body)); + const headers = new Headers(init.headers); + requests.push({ method: body.method ?? "(reply)", headers, body }); + if (body.method === "initialize") { + return Response.json({ jsonrpc: "2.0", id: body.id, result: { protocolVersion: "2025-06-18", capabilities: { tools: {} }, serverInfo: { name: "legacy", version: "1" } } }, + { headers: { "Mcp-Session-Id": `legacy-${++sessions}` } }); + } + if (!headers.get("mcp-session-id")) return Response.json({ jsonrpc: "2.0", id: null, error: { code: -32000, message: "Bad Request: No valid session ID provided" } }, { status: 400 }); + if (body.method === "notifications/initialized") return new Response(null, { status: 202 }); + if (!body.method && body.id) { + if (refuseReplies > 0) { refuseReplies -= 1; return new Response(null, { status: 500 }); } + replies.push(body); + waiting.get(String(body.id))?.(body); + return new Response(null, { status: 202 }); + } + if (body.method === "tools/list") return Response.json({ jsonrpc: "2.0", id: body.id, result: { tools: [{ name: "create_ticket", description: "x", inputSchema: { type: "object", properties: {} }, annotations: { readOnlyHint: true } }] } }); + if (body.method === "tools/call") { + const stream = new ReadableStream({ + async start(controller) { + const answers: unknown[] = []; + for (const question of questions) { + const askId = `ask-${++asks}`; + const reply = new Promise>((resolve) => waiting.set(askId, resolve)); + controller.enqueue(sse({ jsonrpc: "2.0", id: askId, method: "elicitation/create", params: { message: question.message, + requestedSchema: { type: "object", properties: { [question.key]: { type: "string", enum: question.values } }, required: [question.key] } } })); + let answer = await reply; + if (answer.result?.action !== "accept") { controller.close(); return; } + // The app asks the same question again to have the answer corrected. + if (question.askAgainIf !== undefined && answer.result.content?.[question.key] === question.askAgainIf) { + const againId = `ask-${++asks}`; + const again = new Promise>((resolve) => waiting.set(againId, resolve)); + controller.enqueue(sse({ jsonrpc: "2.0", id: againId, method: "elicitation/create", params: { message: question.message, + requestedSchema: { type: "object", properties: { [question.key]: { type: "string", enum: question.values } }, required: [question.key] } } })); + answer = await again; + if (answer.result?.action !== "accept") { controller.close(); return; } + } + answers.push(answer.result); + } + const text = answers.length === 1 ? `answer: ${JSON.stringify(answers[0])}` : `answers: ${JSON.stringify(answers)}`; + controller.enqueue(sse({ jsonrpc: "2.0", id: body.id, result: { content: [{ type: "text", text }] } })); + controller.close(); + }, + }); + return new Response(stream, { headers: { "content-type": "text/event-stream" } }); + } + return Response.json({ jsonrpc: "2.0", id: body.id, error: { code: -32601, message: "Method not found" } }); + }; + return { requests, replies, handle, calls: () => requests.filter((request) => request.method === "tools/call"), refuseNextReply: () => { refuseReplies = 1; } }; +} + +const support = await getEmbeddedPostgresTestSupport(); +(support.supported ? describe : describe.skip)("MCP form elicitation end to end", () => { + const actor = { actorType: "user" as const, actorId: "mcp-elicitation-user", actorSource: "local_implicit" as const }; + let fixture: Awaited>; + let db: ReturnType; + let keyDir: string; + beforeAll(async () => { + keyDir = await mkdtemp(join(tmpdir(), "mcp-elicitation-secrets-")); + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY_FILE", join(keyDir, "key")); + fixture = await startEmbeddedPostgresTestDatabase("mcp-elicitation-"); + db = createDb(fixture.connectionString); + }, 30_000); + afterAll(async () => { + await fixture?.cleanup(); + vi.unstubAllEnvs(); + if (keyDir) await rm(keyDir, { recursive: true, force: true }); + }); + + async function setup(handle: (url: string, init: RequestInit) => Promise, input: { askFirst?: string[] } = {}) { + const [org] = await db.insert(companies).values({ name: `Elicit ${randomUUID()}`, issuePrefix: `E${randomUUID().slice(0, 5)}` }).returning(); + await db.insert(companyMemberships).values({ companyId: org!.id, principalType: "user", principalId: actor.actorId, status: "active", membershipRole: "admin" }); + const [agent] = await db.insert(agents).values({ companyId: org!.id, name: "Ticket agent", role: "engineer", adapterType: "process", adapterConfig: {}, runtimeConfig: {} }).returning(); + const [project] = await db.insert(projects).values({ companyId: org!.id, name: "Tickets" }).returning(); + const [issue] = await db.insert(issues).values({ companyId: org!.id, projectId: project!.id, title: "File the bug", status: "in_progress", assigneeAgentId: agent!.id }).returning(); + const [run] = await db.insert(heartbeatRuns).values({ companyId: org!.id, agentId: agent!.id, invocationSource: "assignment", status: "running", contextSnapshot: { issueId: issue!.id, projectId: project!.id } }).returning(); + const options = { deploymentMode: "local_trusted" as const, deploymentExposure: "private" as const, + remoteHttpEndpointLookup: async () => [{ address: "8.8.8.8", family: 4 as const }], remoteHttpRequest: handle }; + const service = toolAccessService(db, options); + const connected = await service.connectGalleryApp(org!.id, { link: `https://ask-${randomUUID().slice(0, 8)}.example.com/mcp`, name: "Tracker" }, actor); + await service.finishGalleryAppConnection(org!.id, connected.connectionId, { + enabledCatalogEntryIds: connected.catalog.map((entry) => entry.id), + askFirstCatalogEntryIds: connected.catalog.filter((entry) => input.askFirst?.includes(entry.toolName)).map((entry) => entry.id), access: "all_agents", + }, actor); + const gatewayOptions = { ...options, toolActionSigningSecret: "elicitation-signing-key" }; + let gateway = createToolGatewayService(db, gatewayOptions); + let session = await gateway.createSession({ companyId: org!.id, agentId: agent!.id, runId: run!.id }); + const names = (await gateway.listToolsForSession(session.token)).map((tool) => tool.name); + const toolName = (name: string) => names.find((candidate) => candidate.endsWith(`:${name.replace(/_/g, "-")}`))!; + const call = (name: string, parameters: Record = { ref: "main" }, extra: { idempotencyKey?: string; tenant?: string } = {}) => + gateway.executeTool({ sessionToken: session.token, tool: toolName(name), parameters, timeoutMs: 5_000, + idempotencyKey: extra.idempotencyKey, ...(extra.tenant ? { callerHeaders: { "x-tenant-id": extra.tenant } } : {}) }); + const cards = () => db.select().from(issueThreadInteractions) + .where(and(eq(issueThreadInteractions.companyId, org!.id), eq(issueThreadInteractions.issueId, issue!.id))) + .orderBy(issueThreadInteractions.createdAt); + const interactions = issueThreadInteractionService(db); + const answer = (cardId: string, answers: Array<{ questionId: string; optionIds: string[]; otherText?: string }>) => + interactions.answerQuestions({ id: issue!.id, companyId: org!.id }, cardId, { answers }, { userId: actor.actorId }); + const skip = (cardId: string) => interactions.skipInteraction({ id: issue!.id, companyId: org!.id }, cardId, {}, { userId: actor.actorId }); + const rounds = () => db.select().from(toolMcpElicitations).where(eq(toolMcpElicitations.companyId, org!.id)); + return { + org: org!, agent: agent!, issue: issue!, run: run!, connected, call, cards, answer, skip, rounds, interactions, + gateway: () => gateway, + /** A call from another run of the agent on the same task, with its own session. */ + callerInAnotherRun: async () => { + const [other] = await db.insert(heartbeatRuns).values({ companyId: org!.id, agentId: agent!.id, invocationSource: "assignment", status: "running", contextSnapshot: { issueId: issue!.id } }).returning(); + const otherSession = await gateway.createSession({ companyId: org!.id, agentId: agent!.id, runId: other!.id }); + return (name: string, parameters: Record) => + gateway.executeTool({ sessionToken: otherSession.token, tool: toolName(name), parameters, timeoutMs: 5_000 }); + }, + /** The agent's next run on the task (a woken agent). */ + nextRun: async () => { + const [next] = await db.insert(heartbeatRuns).values({ companyId: org!.id, agentId: agent!.id, invocationSource: "assignment", status: "running", contextSnapshot: { issueId: issue!.id } }).returning(); + session = await gateway.createSession({ companyId: org!.id, agentId: agent!.id, runId: next!.id }); + }, + /** A new process: only the database survives. */ + restart: async () => { + forgetMcpHttpSessions(connected.connectionId); + gateway = createToolGatewayService(db, gatewayOptions); + session = await gateway.createSession({ companyId: org!.id, agentId: agent!.id, runId: run!.id }); + }, + }; + } + + it("puts a stateless server's form on a human-only card and sends the answer with the exact state on the repeated call", async () => { + const server = askingServer(); + const ctx = await setup(server.handle); + const first = await ctx.call("create_ticket").catch((error) => error); + expect(first).toMatchObject({ reasonCode: "elicitation_required", message: expect.stringContaining("Tracker asked a question") }); + expect(server.calls()[0]!.body.params._meta[META_CAPABILITIES].elicitation).toEqual({ form: {}, url: {} }); + const [card] = await ctx.cards(); + expect(card).toMatchObject({ kind: "ask_user_questions", status: "pending", title: "Tracker needs your input", requestedResolverPolicy: "human_only" }); + expect(card!.payload).toMatchObject({ questionSet: { description: "Tell us about the ticket", submitLabel: "Send to Tracker" } }); + const [round] = await ctx.rounds(); + expect(round).toMatchObject({ status: "waiting", era: "modern", issueId: ctx.issue.id }); + // The server's state is never stored readable. + expect(round!.sealedRequestState).not.toContain("state-for-main"); + const [invocation] = await db.select().from(toolInvocations).where(eq(toolInvocations.id, round!.invocationId)); + expect(invocation).toMatchObject({ status: "awaiting_approval", errorCode: "elicitation_required" }); + + // Repeating before anyone answers waits again and never reaches the app. + expect(await ctx.call("create_ticket").catch((error) => error)).toMatchObject({ reasonCode: "elicitation_required" }); + expect(server.calls()).toHaveLength(1); + + // An agent can't answer its own tool's question. + await expect(ctx.interactions.answerQuestions({ id: ctx.issue.id, companyId: ctx.org.id }, card!.id, + { answers: [{ questionId: "title", optionIds: ["paperclip_text_answer"], otherText: "Nope" }] }, { agentId: ctx.agent.id, runId: ctx.run.id })) + .rejects.toBeTruthy(); + + await ctx.answer(card!.id, [ + { questionId: "title", optionIds: ["paperclip_text_answer"], otherText: "Login broken" }, + { questionId: "priority", optionIds: ["choice_1"] }, + { questionId: "labels", optionIds: ["choice_2"] }, + { questionId: "count", optionIds: ["paperclip_text_answer"], otherText: "2" }, + ]); + const done = await ctx.call("create_ticket"); + expect(done).toMatchObject({ status: "completed" }); + const retried = server.calls()[1]!; + expect(retried.body.id).not.toBe(server.calls()[0]!.body.id); + expect(retried.body.params.requestState).toBe("state-for-main"); + expect(retried.body.params.inputResponses).toEqual({ details: { action: "accept", content: { title: "Login broken", priority: "p1", labels: ["ux"], count: 2 } } }); + expect((await ctx.rounds())[0]).toMatchObject({ status: "sent" }); + // Sent once: the same call in a later run is a fresh call. + await ctx.nextRun(); + await ctx.call("create_ticket").catch(() => undefined); + expect(server.calls()[2]!.body.params).not.toHaveProperty("inputResponses"); + const logged = await db.select().from(activityLog).where(and(eq(activityLog.companyId, ctx.org.id), eq(activityLog.action, "tool_gateway.elicitation_sent"))); + expect(logged).toHaveLength(1); + // Which kind of answer went back, never the words. + expect(JSON.stringify(logged[0]!.details)).not.toContain("Login broken"); + expect(logged[0]!.details).toMatchObject({ responses: ["accept"] }); + }); + + it("sends decline when the person skips, and survives a restart while waiting", async () => { + const server = askingServer(); + const ctx = await setup(server.handle); + await ctx.call("create_ticket", { ref: "skip" }).catch(() => undefined); + const [card] = await ctx.cards(); + await ctx.restart(); + await ctx.skip(card!.id); + await ctx.call("create_ticket", { ref: "skip" }); + expect(server.calls()[1]!.body.params).toMatchObject({ requestState: "state-for-skip", inputResponses: { details: { action: "decline" } } }); + }); + + it("asks two questions at once and sends both answers in one repeated call", async () => { + const server = askingServer(); + const ctx = await setup(server.handle); + expect(await ctx.call("two_questions").catch((error) => error)).toMatchObject({ reasonCode: "elicitation_required", details: { interactionIds: [expect.any(String), expect.any(String)] } }); + const [owner, go] = await ctx.cards(); + await ctx.answer(owner!.id, [{ questionId: "owner", optionIds: ["paperclip_text_answer"], otherText: "Ada" }]); + // One answered, one still waiting: no send. + expect(await ctx.call("two_questions").catch((error) => error)).toMatchObject({ reasonCode: "elicitation_required" }); + expect(server.calls()).toHaveLength(1); + await ctx.interactions.withdrawInteraction({ id: ctx.issue.id, companyId: ctx.org.id }, go!.id, {}, { userId: actor.actorId }).catch(async () => { + await db.update(issueThreadInteractions).set({ status: "cancelled", result: { version: 1, answers: [], outcome: "withdrawn" } }).where(eq(issueThreadInteractions.id, go!.id)); + }); + await ctx.call("two_questions"); + expect(server.calls()[1]!.body.params).toMatchObject({ + requestState: "two-state", + inputResponses: { who: { action: "accept", content: { owner: "Ada" } }, go: { action: "cancel" } }, + }); + }); + + it("refuses a form asking for a password: no card, a clear reason for the agent", async () => { + const server = askingServer(); + const ctx = await setup(server.handle); + expect(await ctx.call("ask_secret").catch((error) => error)).toMatchObject({ reasonCode: "elicitation_sensitive_refused", message: expect.stringContaining("secure link") }); + expect(await ctx.cards()).toHaveLength(0); + expect(await ctx.rounds()).toHaveLength(0); + }); + + it("reclaims a send interrupted by a crash, and never lets another company collect the round", async () => { + const server = askingServer(); + const ctx = await setup(server.handle); + await ctx.call("create_ticket", { ref: "crash" }).catch(() => undefined); + const [card] = await ctx.cards(); + await ctx.answer(card!.id, [{ questionId: "title", optionIds: ["paperclip_text_answer"], otherText: "After crash" }, { questionId: "priority", optionIds: ["choice_2"] }]); + // A process died mid-send: the claim is held but its time has passed. + const [round] = await ctx.rounds(); + await db.update(toolMcpElicitations).set({ claimedUntil: new Date(Date.now() - 1_000) }).where(eq(toolMcpElicitations.id, round!.id)); + // Another company with the same app and arguments starts its own call. + const other = await setup(server.handle); + await other.call("create_ticket", { ref: "crash" }).catch(() => undefined); + expect(server.calls().at(-1)!.body.params).not.toHaveProperty("inputResponses"); + await ctx.restart(); + await ctx.call("create_ticket", { ref: "crash" }); + expect(server.calls().at(-1)!.body.params).toMatchObject({ requestState: "state-for-crash", inputResponses: { details: { action: "accept", content: { title: "After crash", priority: "p2" } } } }); + }); + + it("withdraws an expired round's card and starts the call over", async () => { + const server = askingServer(); + const ctx = await setup(server.handle); + await ctx.call("create_ticket", { ref: "late" }).catch(() => undefined); + await db.update(toolMcpElicitations).set({ expiresAt: new Date(Date.now() - 1_000) }).where(eq(toolMcpElicitations.companyId, ctx.org.id)); + expect(await ctx.call("create_ticket", { ref: "late" }).catch((error) => error)).toMatchObject({ reasonCode: "elicitation_required" }); + expect(server.calls()[1]!.body.params).not.toHaveProperty("inputResponses"); + const [stale, fresh] = await ctx.cards(); + expect(stale!.status).not.toBe("pending"); + expect(fresh!.status).toBe("pending"); + expect((await ctx.rounds()).map((round) => round.status).sort()).toEqual(["expired", "waiting"]); + }); + + it("answers an older server's question: cancel at once the first time, then the person's answer on the repeated call", async () => { + const server = legacyAskingServer(); + const ctx = await setup(server.handle); + // The listing session only reads: it declares no questions. + expect(server.requests.find((request) => request.method === "initialize")!.body.params.capabilities).not.toHaveProperty("elicitation"); + expect(await ctx.call("create_ticket").catch((error) => error)).toMatchObject({ reasonCode: "elicitation_required" }); + // The tool call's own session asked for 2025-11-25, which knows form and URL questions. + const initialize = server.requests.filter((request) => request.method === "initialize").at(-1)!; + expect(initialize.body.params.capabilities.elicitation).toEqual({ form: {}, url: {} }); + // The server was never left waiting. + expect(server.replies.map((reply) => reply.result)).toEqual([{ action: "cancel" }]); + const [card] = await ctx.cards(); + expect(card).toMatchObject({ title: "Tracker needs your input", requestedResolverPolicy: "human_only" }); + await ctx.answer(card!.id, [{ questionId: "team", optionIds: ["choice_2"] }]); + const done = await ctx.call("create_ticket"); + expect(done).toMatchObject({ status: "completed", result: { content: 'answer: {"action":"accept","content":{"team":"eng"}}' } }); + expect(server.replies.map((reply) => reply.result)).toEqual([{ action: "cancel" }, { action: "accept", content: { team: "eng" } }]); + expect((await ctx.rounds())[0]).toMatchObject({ era: "legacy", status: "sent" }); + // Spent: the same call in a later run asks again. + await ctx.nextRun(); + expect(await ctx.call("create_ticket").catch((error) => error)).toMatchObject({ reasonCode: "elicitation_required" }); + expect(await ctx.cards()).toHaveLength(2); + }); + + it("never lets one forwarded tenant collect another tenant's answers", async () => { + const server = askingServer(); + const ctx = await setup(server.handle); + const [connection] = await db.select().from(toolConnections).where(eq(toolConnections.id, ctx.connected.connectionId)); + await db.update(toolConnections).set({ config: { ...connection!.config, headerPolicy: { passthrough: { allow: ["x-tenant-id"] } } } }) + .where(eq(toolConnections.id, ctx.connected.connectionId)); + await ctx.call("create_ticket", { ref: "t" }, { tenant: "acme" }).catch(() => undefined); + const [card] = await ctx.cards(); + await ctx.answer(card!.id, [{ questionId: "title", optionIds: ["paperclip_text_answer"], otherText: "Acme only" }, { questionId: "priority", optionIds: ["choice_1"] }]); + await ctx.call("create_ticket", { ref: "t" }, { tenant: "globex" }).catch(() => undefined); + const globex = server.calls().at(-1)!; + expect(globex.headers.get("x-tenant-id")).toBe("globex"); + expect(globex.body.params).not.toHaveProperty("inputResponses"); + await ctx.call("create_ticket", { ref: "t" }, { tenant: "acme" }); + const acme = server.calls().at(-1)!; + expect(acme.headers.get("x-tenant-id")).toBe("acme"); + expect(acme.body.params.inputResponses).toMatchObject({ details: { action: "accept", content: { title: "Acme only" } } }); + }); + + it("makes a missing card again under the same key when the process stopped before showing it", async () => { + const server = askingServer(); + const ctx = await setup(server.handle); + await ctx.call("create_ticket", { ref: "lost" }).catch(() => undefined); + const [round] = await ctx.rounds(); + // The round was saved, but its card never reached the task. + await db.delete(issueThreadInteractions).where(eq(issueThreadInteractions.issueId, ctx.issue.id)); + expect(await ctx.call("create_ticket", { ref: "lost" }).catch((error) => error)).toMatchObject({ reasonCode: "elicitation_required" }); + expect(server.calls()).toHaveLength(1); + const [card] = await ctx.cards(); + expect(card).toMatchObject({ idempotencyKey: round!.questions[0]!.cardKey, status: "pending" }); + await ctx.answer(card!.id, [{ questionId: "title", optionIds: ["paperclip_text_answer"], otherText: "Recovered" }, { questionId: "priority", optionIds: ["choice_2"] }]); + await ctx.call("create_ticket", { ref: "lost" }); + expect(server.calls()[1]!.body.params.inputResponses).toMatchObject({ details: { action: "accept", content: { title: "Recovered" } } }); + }); + + it("runs a keyed retry again only for the same request", async () => { + const server = askingServer(); + const ctx = await setup(server.handle); + await ctx.call("create_ticket", { ref: "a" }, { idempotencyKey: "ticket-key" }).catch(() => undefined); + const other = await ctx.call("create_ticket", { ref: "b" }, { idempotencyKey: "ticket-key" }).catch((error) => error); + expect(other).not.toMatchObject({ status: "completed" }); + expect(server.calls()).toHaveLength(1); + expect((await ctx.rounds())[0]).toMatchObject({ status: "waiting" }); + }); + + it("keeps an older server's answer when the server refuses the reply", async () => { + const server = legacyAskingServer(); + const ctx = await setup(server.handle); + await ctx.call("create_ticket").catch(() => undefined); + const [card] = await ctx.cards(); + await ctx.answer(card!.id, [{ questionId: "team", optionIds: ["choice_1"] }]); + server.refuseNextReply(); + await ctx.nextRun(); + expect(await ctx.call("create_ticket").catch((error) => error)).toMatchObject({ reasonCode: "mcp_remote_status" }); + // Not spent and not held: the next call sends it again. + expect((await ctx.rounds())[0]).toMatchObject({ status: "waiting", claimedUntil: null }); + await ctx.nextRun(); + expect(await ctx.call("create_ticket")).toMatchObject({ status: "completed", result: { content: 'answer: {"action":"accept","content":{"team":"ops"}}' } }); + expect((await ctx.rounds())[0]).toMatchObject({ status: "sent" }); + }); + + it("keeps an older server's earlier answers until the whole call finishes", async () => { + const server = legacyAskingServer({ questions: [ + { message: "Which team?", key: "team", values: ["ops", "eng"] }, + { message: "Which size?", key: "size", values: ["s", "m"] }, + ] }); + const ctx = await setup(server.handle); + await ctx.call("create_ticket").catch(() => undefined); + const [teamCard] = await ctx.cards(); + await ctx.answer(teamCard!.id, [{ questionId: "team", optionIds: ["choice_2"] }]); + await ctx.nextRun(); + // The team answer goes back, then the app asks its second question. + expect(await ctx.call("create_ticket").catch((error) => error)).toMatchObject({ reasonCode: "elicitation_required" }); + const [, sizeCard] = await ctx.cards(); + await ctx.answer(sizeCard!.id, [{ questionId: "size", optionIds: ["choice_1"] }]); + await ctx.nextRun(); + const done = await ctx.call("create_ticket"); + expect(done).toMatchObject({ status: "completed", result: { content: 'answers: [{"action":"accept","content":{"team":"eng"}},{"action":"accept","content":{"size":"s"}}]' } }); + // Nobody was asked the first question twice. + expect(await ctx.cards()).toHaveLength(2); + expect((await ctx.rounds()).map((round) => round.status)).toEqual(["sent", "sent"]); + }); + + it("sends the answer again in the same run after the app failed to take it", async () => { + const server = askingServer(); + const ctx = await setup(server.handle); + await ctx.call("create_ticket", { ref: "flaky" }).catch(() => undefined); + const [card] = await ctx.cards(); + await ctx.answer(card!.id, [{ questionId: "title", optionIds: ["paperclip_text_answer"], otherText: "Second try" }, { questionId: "priority", optionIds: ["choice_1"] }]); + server.refuseNextAnswer(); + expect(await ctx.call("create_ticket", { ref: "flaky" }).catch((error) => error)).toMatchObject({ reasonCode: "mcp_remote_status" }); + // Same run, same request: not a replay of the failure; the answer goes out again. + expect(await ctx.call("create_ticket", { ref: "flaky" })).toMatchObject({ status: "completed" }); + expect(server.calls()).toHaveLength(3); + expect(server.calls()[2]!.body.params.inputResponses).toMatchObject({ details: { action: "accept", content: { title: "Second try" } } }); + expect((await ctx.rounds())[0]).toMatchObject({ status: "sent" }); + }); + + it("shows one card when two matching calls ask at the same time", async () => { + const server = askingServer(); + const ctx = await setup(server.handle); + const first = await ctx.callerInAnotherRun(); + const second = await ctx.callerInAnotherRun(); + const results = await Promise.all([ + first("create_ticket", { ref: "race" }).catch((error) => error), + second("create_ticket", { ref: "race" }).catch((error) => error), + ]); + for (const result of results) expect(result).toMatchObject({ reasonCode: "elicitation_required" }); + expect(await ctx.cards()).toHaveLength(1); + expect((await ctx.rounds()).filter((round) => round.status === "waiting")).toHaveLength(1); + }); + + it("continues a stateless app's follow-up question after the first answer", async () => { + const server = askingServer(); + const ctx = await setup(server.handle); + await ctx.call("two_rounds").catch(() => undefined); + const [first] = await ctx.cards(); + await ctx.answer(first!.id, [{ questionId: "team", optionIds: ["choice_2"] }]); + expect(await ctx.call("two_rounds").catch((error) => error)).toMatchObject({ reasonCode: "elicitation_required" }); + // The first round is spent and the follow-up is saved; nothing is lost between them. + expect((await ctx.rounds()).map((round) => round.status).sort()).toEqual(["sent", "waiting"]); + const [, second] = await ctx.cards(); + await ctx.answer(second!.id, [{ questionId: "due", optionIds: ["paperclip_text_answer"], otherText: "2026-12-01" }]); + await ctx.call("two_rounds"); + expect(server.calls().at(-1)!.body.params).toMatchObject({ requestState: "round-2", inputResponses: { second: { action: "accept", content: { due: "2026-12-01" } } } }); + }); + + it("asks again on a fresh card when an older app repeats a question to correct the answer", async () => { + const server = legacyAskingServer({ questions: [{ message: "Which team?", key: "team", values: ["ops", "eng"], askAgainIf: "ops" }] }); + const ctx = await setup(server.handle); + await ctx.call("create_ticket").catch(() => undefined); + const [first] = await ctx.cards(); + await ctx.answer(first!.id, [{ questionId: "team", optionIds: ["choice_1"] }]); + await ctx.nextRun(); + // The app takes "ops", then asks again: a second card, not a stall. + expect(await ctx.call("create_ticket").catch((error) => error)).toMatchObject({ reasonCode: "elicitation_required" }); + const cards = await ctx.cards(); + expect(cards).toHaveLength(2); + await ctx.answer(cards[1]!.id, [{ questionId: "team", optionIds: ["choice_2"] }]); + await ctx.nextRun(); + expect(await ctx.call("create_ticket")).toMatchObject({ status: "completed", result: { content: 'answer: {"action":"accept","content":{"team":"eng"}}' } }); + }); + + it("advances to a follow-up question even when the task is at its open-question limit", async () => { + const server = askingServer(); + const ctx = await setup(server.handle); + await ctx.call("two_rounds").catch(() => undefined); + const [round] = await ctx.rounds(); + // Nine other questions are waiting on the task: ten in all, the limit. + await db.insert(toolMcpElicitations).values(Array.from({ length: 9 }, (_, index) => ({ + ...round!, id: randomUUID(), callIdentity: `other-call-${index}`, dedupeKey: `other-${index}`, createdAt: new Date(), updatedAt: new Date(), + }))); + const [first] = await ctx.cards(); + await ctx.answer(first!.id, [{ questionId: "team", optionIds: ["choice_1"] }]); + expect(await ctx.call("two_rounds").catch((error) => error)).toMatchObject({ reasonCode: "elicitation_required" }); + // The answered round made room for its own follow-up. + expect(await ctx.cards()).toHaveLength(2); + expect((await ctx.rounds()).filter((row) => row.status === "waiting")).toHaveLength(10); + }); + + it("keeps an approved action's approval while its question waits, and sends the answer without asking again", async () => { + const server = askingServer(); + const ctx = await setup(server.handle, { askFirst: ["create_ticket"] }); + await expect(ctx.call("create_ticket", { ref: "approved" })).rejects.toMatchObject({ reasonCode: "approval_required" }); + const [pending] = await db.select().from(toolActionRequests).where(eq(toolActionRequests.companyId, ctx.org.id)); + if (pending!.approvalId) await db.update(approvals).set({ status: "approved", decidedByUserId: actor.actorId, decidedAt: new Date() }).where(eq(approvals.id, pending!.approvalId)); + // The approved execution reaches the app, which asks a question. + await ctx.gateway().approveActionRequest({ companyId: ctx.org.id, actionRequestId: pending!.id, actor: { userId: actor.actorId } }).catch((error) => error); + expect(server.calls()).toHaveLength(1); + const [request] = await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending!.id)); + expect(request).toMatchObject({ status: "approved" }); + const card = (await ctx.cards()).find((candidate) => candidate.kind === "ask_user_questions")!; + await ctx.answer(card.id, [{ questionId: "title", optionIds: ["paperclip_text_answer"], otherText: "Approved and answered" }, { questionId: "priority", optionIds: ["choice_1"] }]); + // The repeated call continues the approved action: no second approval, the answer goes out. + // An approved action's continuation is reported with its result (status "replayed"). + const done = await ctx.call("create_ticket", { ref: "approved" }); + expect(JSON.stringify(done.result)).toContain("Approved and answered"); + expect(server.calls()).toHaveLength(2); + expect(server.calls()[1]!.body.params.inputResponses).toMatchObject({ details: { action: "accept", content: { title: "Approved and answered" } } }); + }); + + it("marks an app's question card so skipping it wakes the agent (route test: issue-thread-interaction-routes)", async () => { + const server = askingServer(); + const ctx = await setup(server.handle); + await ctx.call("create_ticket", { ref: "skip-wake" }).catch(() => undefined); + const [card] = await ctx.cards(); + const { isAppQuestionCardKey } = await import("../services/mcp-elicitation.js"); + expect(isAppQuestionCardKey(card!.idempotencyKey)).toBe(true); + }); +}); diff --git a/server/src/__tests__/mcp-protocol-versions.test.ts b/server/src/__tests__/mcp-protocol-versions.test.ts index e7c53664b68..2f75002a31f 100644 --- a/server/src/__tests__/mcp-protocol-versions.test.ts +++ b/server/src/__tests__/mcp-protocol-versions.test.ts @@ -504,6 +504,8 @@ const support = await getEmbeddedPostgresTestSupport(); forgetMcpHttpSessions(connected.connectionId); expect(await call("lookup")).toMatchObject({ decision: "allowed", result: { data: { isError: false } } }); expect(legacy.methods()).toEqual(["tools/call", "initialize", "notifications/initialized", "tools/call"]); + // The fallback opens the tool call's own session, which can answer form questions (2025-06-18: `{}`). + expect(legacy.requests.find((request) => request.method === "initialize")!.body.params.capabilities.elicitation).toEqual({}); expect(await savedConfig()).toMatchObject({ mcpProtocolVersion: "2025-06-18" }); }); diff --git a/server/src/__tests__/tool-gateway-service.test.ts b/server/src/__tests__/tool-gateway-service.test.ts index 2772b3fcc3c..bf5f679e858 100644 --- a/server/src/__tests__/tool-gateway-service.test.ts +++ b/server/src/__tests__/tool-gateway-service.test.ts @@ -3080,14 +3080,17 @@ describeEmbeddedPostgres("tool gateway service", () => { status: "pending", issueId: session.issueId, }); + // The card names the app, only people answer it, and the provider's + // enum becomes a real choice (see MCP-ELICITATION.md). + expect(interaction).toMatchObject({ requestedResolverPolicy: "human_only", title: expect.stringMatching(/needs your input$/) }); expect(interaction.payload).toMatchObject({ - title: "Which workspace should be used?", + questionSet: { description: "Which workspace should be used?" }, questions: [ { id: "workspace", prompt: "Workspace", required: true, - options: [{ id: "ops", label: "ops" }, { id: "engineering", label: "engineering" }], + options: [{ id: "choice_1", label: "ops" }, { id: "choice_2", label: "engineering" }], }, ], }); diff --git a/server/src/routes/issues.ts b/server/src/routes/issues.ts index d30fe1881fa..c3045825194 100644 --- a/server/src/routes/issues.ts +++ b/server/src/routes/issues.ts @@ -1,4 +1,5 @@ import { assertAgentInteraction } from "../services/agent-interaction-policy.js"; +import { isAppQuestionCardKey } from "../services/mcp-elicitation.js"; import { assertQueuedCommentInteractionDelivery, recheckQueuedCommentInteractionDelivery } from "../services/agent-interaction-delivery.js"; import { listStalledDependencies } from "../services/recovery/blocker-stall-wakeups.js"; import { confirmMcpAttachmentCleanup, recordMcpAttachmentCleanup, type McpAttachmentCleanup } from "../services/public-mcp/upload-cleanup.js"; @@ -19366,6 +19367,18 @@ export function issueRoutes( // Skip intentionally does not enqueue an interaction continuation. A // subsequent ordinary message uses the existing comment steering path. + // An app's question is the exception: skipping is the person's "no", + // which only reaches the app on the agent's next call, so wake it. + if (isAppQuestionCardKey(interaction.idempotencyKey)) { + await queueResolvedInteractionContinuationWakeup({ + db, + heartbeat, + issue, + interaction, + actor, + source: "issue.interaction.skip", + }); + } res.json(interaction); }, ); diff --git a/server/src/services/mcp-elicitation.ts b/server/src/services/mcp-elicitation.ts new file mode 100644 index 00000000000..c52d33df26a --- /dev/null +++ b/server/src/services/mcp-elicitation.ts @@ -0,0 +1,365 @@ +import { createCipheriv, createDecipheriv, createHash, hkdfSync, randomBytes } from "node:crypto"; +import type { PaperclipQuestionSetPayload, PaperclipQuestionSetQuestion } from "@paperclipai/shared"; +import { redactSensitiveText } from "../redaction.js"; + +/** + * MCP form elicitation (spec 2026-07-28, client/elicitation): the provider's + * question becomes a task-chat question card, and the person's answer on that + * card becomes the elicitation result. Used by direct tool calls (stateless + * `input_required` and older server-initiated `elicitation/create`) and by + * MCP Tasks, so every path shows the same card and sends the same answer. + */ + +export type McpFormElicitation = { + message: string; + requestedSchema: Record | null; +}; + +export type McpElicitationResponse = + | { action: "accept"; content: Record } + | { action: "decline" } + | { action: "cancel" }; + +/** At most this many fields per question, and form questions per round. */ +export const MCP_ELICITATION_MAX_FIELDS = 10; +export const MCP_ELICITATION_MAX_FORMS_PER_ROUND = 4; +const MAX_OPTIONS = 100; + +type Built = + | { ok: true; questionSet: PaperclipQuestionSetPayload } + | { ok: false; reason: "sensitive"; fields: string[] } + | { ok: false; reason: "unsupported" }; + +function record(value: unknown): Record | null { + return value && typeof value === "object" && !Array.isArray(value) ? value as Record : null; +} + +function text(value: unknown): string | undefined { + return typeof value === "string" && value.trim() ? value.trim() : undefined; +} + +/** + * Provider text is shown as plain words. Links in a form are not clickable + * (spec: Safe URL Handling), and secrets that slipped in are redacted. + */ +/** True for a card made from an app's question (direct or MCP Task), by its idempotency key. */ +export function isAppQuestionCardKey(key: string | null | undefined): boolean { + return typeof key === "string" && (key.startsWith("mcp-elicit:") || key.startsWith("mcp-task-input:") || key.startsWith("mcp-elicitation:")); +} + +export function elicitationPlainText(value: string, max: number): string { + return redactSensitiveText(value) + .replace(/[<>]/g, "") + .replace(/\]\(/g, "] (") + .replace(/:\/\//g, ":​//") + .slice(0, max); +} + +// Secrets must go through a URL (spec: Form Mode Security). Matched on the +// field's name, title, and description. +const SENSITIVE_FIELD = + /pass(?:word|wd|code|phrase)|secret|api[\s_-]?key|(?:access|refresh|auth|bearer|session|api)[\s_-]?token|\btokens?\b|credential|private[\s_-]?key|card[\s_-]?(?:number|no\b)|\bcvv\b|\bcvc\b|security[\s_-]?code|\bssn\b|social[\s_-]?security|one[\s_-]?time[\s_-]?(?:code|password)|\botp\b|\b2fa\b|\bmfa\b/i; + +// The same secrets, asked for in the form's own words ("type your password +// in the notes field"). A bare "token" is left out: in prose it is too common. +const SENSITIVE_MESSAGE = + /pass(?:word|wd|code|phrase)|secret|api[\s_-]?key|(?:access|refresh|auth|bearer|session|api)[\s_-]?token|credential|private[\s_-]?key|card[\s_-]?(?:number|no\b)|\bcvv\b|\bcvc\b|security[\s_-]?code|\bssn\b|social[\s_-]?security|one[\s_-]?time[\s_-]?(?:code|password)|\botp\b|\b2fa\b|\bmfa\b/i; + +/** Field keys that ask for a secret; empty when the form is safe to show. */ +export function elicitationSensitiveFields(schema: Record | null): string[] { + const properties = record(schema?.properties) ?? {}; + return Object.entries(properties).filter(([key, raw]) => { + const property = record(raw) ?? {}; + if (property.format === "password" || property.writeOnly === true) return true; + return [key, property.title, property.description].some((value) => typeof value === "string" && SENSITIVE_FIELD.test(value)); + }).map(([key]) => key); +} + +type Choice = { value: unknown; label: string }; + +/** Enum choices from `enum`, `oneOf`/`anyOf` with `const`+`title` (SEP-1330). */ +function choicesOf(schema: Record): Choice[] | null { + if (Array.isArray(schema.enum)) { + const titles = Array.isArray(schema.enumNames) ? schema.enumNames : []; + return schema.enum.map((value, index) => ({ value, label: String(text(titles[index]) ?? value) })); + } + const variants = Array.isArray(schema.oneOf) ? schema.oneOf : Array.isArray(schema.anyOf) ? schema.anyOf : null; + if (!variants) return null; + const choices: Choice[] = []; + for (const variant of variants) { + const entry = record(variant); + if (!entry || !("const" in entry)) return null; + choices.push({ value: entry.const, label: String(text(entry.title) ?? entry.const) }); + } + return choices; +} + +const primitive = (value: unknown) => typeof value === "string" || typeof value === "number" || typeof value === "boolean"; +const optionId = (index: number) => `choice_${index + 1}`; + +type Field = { key: string; property: Record; question: PaperclipQuestionSetQuestion; choices?: Choice[]; kind: "text" | "number" | "integer" | "boolean" | "single" | "multi" }; + +// Patterns stay within the fixed-width subset the question form can check +// safely (no repetition), so an email is a hint and the app validates it. +const FORMAT_HINT: Record = { + email: { help: "An email address." }, + uri: { help: "A web address." }, + date: { help: "A date, like 2026-11-13.", pattern: "^\\d\\d\\d\\d-\\d\\d-\\d\\d$" }, + "date-time": { help: "A date and time, like 2026-11-13T09:30:00Z." }, +}; + +function fieldFor(key: string, raw: unknown, required: boolean): Field | null { + const property = record(raw); + if (!property) return null; + const prompt = elicitationPlainText(text(property.title) ?? key, 4000); + const description = text(property.description); + const help = (extra?: string) => { + const value = [description ? elicitationPlainText(description, 3000) : null, extra].filter(Boolean).join(" "); + return value ? { helpText: value.slice(0, 4000) } : {}; + }; + const base = { id: key.slice(0, 160), prompt, required }; + const select = (choices: Choice[], multi: boolean, defaults: unknown[], extra?: string): Field | null => { + if (!choices.length || choices.length > MAX_OPTIONS || !choices.every((choice) => primitive(choice.value))) return null; + return { + key, property, choices, kind: multi ? "multi" : "single", + question: { + ...base, ...help(extra), answerMode: multi ? "multi_select" : "single_select", + options: choices.map((choice, index) => ({ + id: optionId(index), label: elicitationPlainText(choice.label, 1000) || `Option ${index + 1}`, + ...(defaults.some((value) => value === choice.value) ? { recommended: true } : {}), + })), + }, + }; + }; + const type = property.type; + if (type === "array") { + const items = record(property.items); + const choices = items ? choicesOf(items) : null; + if (!choices) return null; + const limits = [ + typeof property.minItems === "number" ? `Choose at least ${property.minItems}.` : null, + typeof property.maxItems === "number" ? `Choose at most ${property.maxItems}.` : null, + ].filter(Boolean).join(" "); + return select(choices, true, Array.isArray(property.default) ? property.default : [], limits || undefined); + } + const choices = choicesOf(property); + if (choices) return select(choices, false, property.default === undefined ? [] : [property.default]); + if (type === "boolean") { + return select([{ value: true, label: "Yes" }, { value: false, label: "No" }], false, property.default === undefined ? [] : [property.default]); + } + if (type === "number" || type === "integer") { + const minimum = typeof property.minimum === "number" && Number.isFinite(property.minimum) ? property.minimum : undefined; + const maximum = typeof property.maximum === "number" && Number.isFinite(property.maximum) ? property.maximum : undefined; + if (minimum !== undefined && maximum !== undefined && minimum > maximum) return null; + return { + key, property, kind: type, + question: { + ...base, ...help(), answerMode: "text", + ...(typeof property.default === "number" ? { initialText: String(property.default) } : {}), + textValidation: { inputType: type, ...(minimum !== undefined ? { minimum } : {}), ...(maximum !== undefined ? { maximum } : {}) }, + }, + }; + } + if (type === "string" || type === undefined) { + const format = typeof property.format === "string" ? FORMAT_HINT[property.format] : undefined; + if (typeof property.format === "string" && !format) return null; + const minLength = Number.isInteger(property.minLength) && (property.minLength as number) >= 0 ? Math.min(property.minLength as number, 100_000) : undefined; + const maxLength = Number.isInteger(property.maxLength) && (property.maxLength as number) >= 0 ? Math.min(property.maxLength as number, 100_000) : undefined; + if (minLength !== undefined && maxLength !== undefined && minLength > maxLength) return null; + const validation = { + ...(minLength !== undefined ? { minLength } : {}), + ...(maxLength !== undefined ? { maxLength } : {}), + ...(format?.pattern ? { pattern: format.pattern } : {}), + }; + return { + key, property, kind: "text", + question: { + ...base, ...help(format?.help), answerMode: "text", + ...(typeof property.default === "string" ? { initialText: property.default.slice(0, 100_000) } : {}), + ...(Object.keys(validation).length ? { textValidation: validation } : {}), + }, + }; + } + return null; +} + +function fieldsOf(schema: Record | null): Field[] | null { + if (schema && schema.type !== undefined && schema.type !== "object") return null; + const properties = record(schema?.properties) ?? {}; + const entries = Object.entries(properties); + if (entries.length > MCP_ELICITATION_MAX_FIELDS) return null; + const required = new Set(Array.isArray(schema?.required) ? schema.required.filter((key): key is string => typeof key === "string") : []); + const fields: Field[] = []; + for (const [key, raw] of entries) { + const field = fieldFor(key, raw, required.has(key)); + if (!field) return null; + fields.push(field); + } + return fields; +} + +/** The single "Continue" choice a form with no fields becomes. */ +const CONTINUE_QUESTION_ID = "continue"; + +/** + * The card for one form question, or why Paperclip will not show it. A form + * asking for a secret is refused before anything is shown; a schema outside + * the spec's flat primitive subset is refused rather than shown partly. + */ +export function buildElicitationQuestionSet(input: { appName: string; request: McpFormElicitation }): Built { + const sensitive = elicitationSensitiveFields(input.request.requestedSchema); + // A form whose message asks for a secret is refused too, whatever its fields are called. + if (SENSITIVE_MESSAGE.test(input.request.message)) sensitive.unshift("message"); + if (sensitive.length) return { ok: false, reason: "sensitive", fields: sensitive }; + const fields = fieldsOf(input.request.requestedSchema); + if (!fields) return { ok: false, reason: "unsupported" }; + const appName = elicitationPlainText(input.appName, 120) || "The app"; + const description = elicitationPlainText(input.request.message, 4000) || `${appName} needs more information to continue.`; + const questions: PaperclipQuestionSetQuestion[] = fields.length + ? fields.map((field) => field.question) + : [{ id: CONTINUE_QUESTION_ID, prompt: description, required: true, answerMode: "single_select", options: [{ id: "continue", label: "Continue" }] }]; + return { + ok: true, + questionSet: { + schema: "paperclip.question_set.v1", + title: `${appName} needs your input`, + description, + submitLabel: `Send to ${appName}`.slice(0, 200), + questions, + }, + }; +} + +type CardState = { status: string; result: unknown; payload?: unknown }; + +/** + * The elicitation result for a resolved card, or null while it waits. + * Answered: `accept` with typed content. Skipped: `decline` (the person said + * no). Anything else that ended the card (withdrawn, task closed, expired, + * cancelled): `cancel`, since nobody made a choice. + */ +export function elicitationResponseForCard(request: McpFormElicitation, card: CardState | null | undefined): McpElicitationResponse | null { + if (!card || card.status === "pending") return null; + const result = record(card.result); + if (card.status === "answered" && result?.outcome === undefined) { + // A card saved before typed fields (an MCP Task waiting across the + // upgrade) keeps its own options; read it the way it was built. + const payload = record(card.payload); + const content = payload && !payload.questionSet && Array.isArray(payload.questions) + ? legacyCardContent(request, payload.questions, result) + : elicitationContent(request, result); + return { action: "accept", content }; + } + if (card.status === "cancelled" && result?.outcome === "skipped") return { action: "decline" }; + return { action: "cancel" }; +} + +/** Map a person's card answers back onto the provider's schema, typed. */ +export function elicitationContent(request: McpFormElicitation, result: unknown): Record { + const answers = (Array.isArray(record(result)?.answers) ? record(result)!.answers as unknown[] : []) + .map(record).filter((answer): answer is Record => Boolean(answer)); + const fields = fieldsOf(request.requestedSchema) ?? []; + const content: Record = {}; + for (const field of fields) { + const answer = answers.find((candidate) => candidate.questionId === field.question.id); + if (!answer) continue; + const optionIds = Array.isArray(answer.optionIds) ? answer.optionIds.filter((id): id is string => typeof id === "string") : []; + const chosen = (field.choices ?? []).filter((_, index) => optionIds.includes(optionId(index))).map((choice) => choice.value); + const typed = typeof answer.otherText === "string" ? answer.otherText.trim() : ""; + if (field.kind === "multi") { content[field.key] = chosen; continue; } + if (field.kind === "single" || field.kind === "boolean") { if (chosen.length) content[field.key] = chosen[0]; continue; } + if (!typed) continue; + if (field.kind === "number" || field.kind === "integer") { + const value = Number(typed); + if (Number.isFinite(value) && (field.kind === "number" || Number.isInteger(value))) content[field.key] = value; + continue; + } + content[field.key] = typed; + } + return content; +} + +/** + * Answers on a card built before typed fields: enum choices were listed in + * the schema's order, and everything else was a written answer. + */ +function legacyCardContent(request: McpFormElicitation, storedQuestions: unknown[], result: unknown): Record { + const answers = (Array.isArray(record(result)?.answers) ? record(result)!.answers as unknown[] : []) + .map(record).filter((answer): answer is Record => Boolean(answer)); + const properties = record(request.requestedSchema?.properties); + const textOf = (answer: Record | undefined) => typeof answer?.otherText === "string" ? answer.otherText.trim() : ""; + if (!properties || !Object.keys(properties).length) { + const text = textOf(answers.find((answer) => answer.questionId === "response")); + return text ? { response: text } : {}; + } + const content: Record = {}; + for (const [key, raw] of Object.entries(properties).slice(0, MCP_ELICITATION_MAX_FIELDS)) { + const property = record(raw) ?? {}; + const answer = answers.find((candidate) => candidate.questionId === key.slice(0, 120)); + if (!answer) continue; + const enumValues = Array.isArray(property.enum) ? property.enum : []; + const stored = storedQuestions.map(record).find((question) => question?.id === key.slice(0, 120)); + const options = Array.isArray(stored?.options) ? stored!.options.map(record) : []; + const optionIds = Array.isArray(answer.optionIds) ? answer.optionIds : []; + const index = options.findIndex((option) => typeof option?.id === "string" && optionIds.includes(option.id)); + if (index >= 0 && index < enumValues.length) { content[key] = enumValues[index]; continue; } + const text = textOf(answer); + if (!text) continue; + if (property.type === "number" || property.type === "integer") { + const value = Number(text); + if (Number.isFinite(value) && (property.type === "number" || Number.isInteger(value))) content[key] = value; + } else if (property.type === "boolean") { + if (/^(true|yes|y|1)$/i.test(text)) content[key] = true; + else if (/^(false|no|n|0)$/i.test(text)) content[key] = false; + } else { + content[key] = text; + } + } + return content; +} + +/** Identifies one question by its words and schema, for answering an older server's repeat. */ +export function elicitationQuestionHash(request: McpFormElicitation): string { + const canonical = (value: unknown): unknown => Array.isArray(value) + ? value.map(canonical) + : record(value) ? Object.fromEntries(Object.keys(value as object).sort().map((key) => [key, canonical((value as Record)[key])])) : value; + return createHash("sha256").update(JSON.stringify(["paperclip:mcp-elicitation-question:v1", request.message, canonical(request.requestedSchema ?? {})])).digest("hex"); +} + +/** A form request from a stateless `inputRequests` entry or an older `elicitation/create` message. */ +export function formElicitationOf(params: Record): McpFormElicitation | null { + if (params.mode !== undefined && params.mode !== "form") return null; + return { + message: typeof params.message === "string" ? params.message : "", + requestedSchema: record(params.requestedSchema), + }; +} + +/* + * The server's requestState is opaque and may carry authority, so it is sealed + * at rest, bound to the company and connection that received it. + */ +const STATE_CONTEXT = "paperclip:mcp-elicitation-state:v1"; +const stateKey = (secret: string) => Buffer.from(hkdfSync("sha256", secret, Buffer.alloc(0), STATE_CONTEXT, 32)); +const stateAad = (scope: { companyId: string; connectionId: string }) => Buffer.from(JSON.stringify([STATE_CONTEXT, scope.companyId, scope.connectionId])); + +export function sealElicitationState(secret: string, scope: { companyId: string; connectionId: string }, value: string): string { + const iv = randomBytes(12); + const cipher = createCipheriv("aes-256-gcm", stateKey(secret), iv); + cipher.setAAD(stateAad(scope)); + const ciphertext = Buffer.concat([cipher.update(value, "utf8"), cipher.final()]); + return `v1.${iv.toString("base64url")}.${ciphertext.toString("base64url")}.${cipher.getAuthTag().toString("base64url")}`; +} + +export function openElicitationState(secret: string, scope: { companyId: string; connectionId: string }, sealed: string): string | null { + const [version, iv, ciphertext, tag] = sealed.split("."); + if (version !== "v1" || !iv || ciphertext === undefined || !tag) return null; + try { + const decipher = createDecipheriv("aes-256-gcm", stateKey(secret), Buffer.from(iv, "base64url")); + decipher.setAAD(stateAad(scope)); + decipher.setAuthTag(Buffer.from(tag, "base64url")); + return Buffer.concat([decipher.update(Buffer.from(ciphertext, "base64url")), decipher.final()]).toString("utf8"); + } catch { + return null; + } +} diff --git a/server/src/services/mcp-http.ts b/server/src/services/mcp-http.ts index d5f74e936d8..8fbdb7fab90 100644 --- a/server/src/services/mcp-http.ts +++ b/server/src/services/mcp-http.ts @@ -54,16 +54,25 @@ export function isModernMcpProtocolVersion(version: unknown): version is string * they may then return URL input requests, which Paperclip shows as a link. * Legacy sessions keep the handshake capabilities they have always received. */ -export function mcpClientCapabilities(protocolVersion: string, options: { declareTasks?: boolean } = {}): Record { +export function mcpClientCapabilities(protocolVersion: string, options: { declareTasks?: boolean; elicitationForm?: boolean } = {}): Record { + const version: string = protocolVersion; // The skills and tasks extensions are defined against stateless MCP only. // Tasks are declared only where Paperclip can wait on one: a provider tool // call over HTTP and the task requests themselves (spec: per request). - return isModernMcpProtocolVersion(protocolVersion) - ? { elicitation: { url: {} }, extensions: { - ...MCP_APPS_CLIENT_CAPABILITY, ...MCP_SKILLS_CLIENT_CAPABILITY, - ...(options.declareTasks ? { [MCP_TASKS_EXTENSION_ID]: {} } : {}), - } } - : { extensions: MCP_APPS_CLIENT_CAPABILITY }; + // Form questions are declared only where a person can answer them: an HTTP + // tool call that belongs to a task, or an HTTP session (whose questions + // outside a task are answered "cancel" at once). See MCP-ELICITATION.md. + if (isModernMcpProtocolVersion(protocolVersion)) { + return { elicitation: options.elicitationForm ? { form: {}, url: {} } : { url: {} }, extensions: { + ...MCP_APPS_CLIENT_CAPABILITY, ...MCP_SKILLS_CLIENT_CAPABILITY, + ...(options.declareTasks ? { [MCP_TASKS_EXTENSION_ID]: {} } : {}), + } }; + } + // 2025-06-18 knows only form questions (`{}`); 2025-11-25 added URL mode. + const legacyElicitation = !options.elicitationForm ? null + : version >= "2025-11-25" ? { form: {}, url: {} } + : version >= "2025-06-18" ? {} : null; + return { ...(legacyElicitation ? { elicitation: legacyElicitation } : {}), extensions: MCP_APPS_CLIENT_CAPABILITY }; } /** Newest version both sides support, or null when there is none. */ @@ -200,7 +209,7 @@ export type McpJsonRpcRequest = { id?: string | number; method: string; params?: export function mcpHttpRequestInit( headers: Record | undefined, message: McpJsonRpcRequest, - options: { toolInputSchema?: unknown; declareTasks?: boolean } = {}, + options: { toolInputSchema?: unknown; declareTasks?: boolean; elicitationForm?: boolean } = {}, ): { method: "POST"; headers: Record; body: string } { const requestHeaders = mcpHttpRequestHeaders(headers); const protocolVersion = mcpProtocolVersionOf(headers); @@ -211,7 +220,7 @@ export function mcpHttpRequestInit( _meta: { ...record(params._meta), [META_PROTOCOL_VERSION]: protocolVersion, - [META_CLIENT_CAPABILITIES]: mcpClientCapabilities(protocolVersion, { declareTasks: options.declareTasks }), + [META_CLIENT_CAPABILITIES]: mcpClientCapabilities(protocolVersion, { declareTasks: options.declareTasks, elicitationForm: options.elicitationForm }), [META_CLIENT_INFO]: MCP_CLIENT_INFO, }, }; @@ -436,6 +445,12 @@ type McpInitializeInput = { requestId: string; /** Identifies the server (connection + endpoint) so its era is probed once. */ eraKey?: string; + /** + * Declare form questions at `initialize`. Only for sessions whose every + * request answers a server's `elicitation/create` (tool calls); a session + * that only reads must not invite a question it would leave waiting. + */ + elicitationForm?: boolean; }; type DiscoverOutcome = { protocolVersion: string; capabilities?: Record } | { legacy: true }; @@ -543,7 +558,7 @@ async function initializeLegacyMcpHttpSession(input: McpInitializeInput, request method: "initialize", params: { protocolVersion: requestedVersion, - capabilities: mcpClientCapabilities(requestedVersion), + capabilities: mcpClientCapabilities(requestedVersion, { elicitationForm: input.elicitationForm }), clientInfo: MCP_CLIENT_INFO, }, }), diff --git a/server/src/services/tool-gateway.ts b/server/src/services/tool-gateway.ts index e43ab1da975..90ac0b99ff1 100644 --- a/server/src/services/tool-gateway.ts +++ b/server/src/services/tool-gateway.ts @@ -77,7 +77,7 @@ import { getAppDefinitionForApplicationKey, isConnectorFeatureEnabled } from "@p import { instanceSettingsService } from "./instance-settings.js"; import { redactSensitiveText } from "../redaction.js"; import { spawn } from "node:child_process"; -import { createHash, randomBytes, randomUUID } from "node:crypto"; +import { createHash, createHmac, randomBytes, randomUUID } from "node:crypto"; import { isDeepStrictEqual } from "node:util"; import { and, @@ -87,6 +87,7 @@ import { gt, inArray, isNull, + lt, lte, ne, or, @@ -125,6 +126,7 @@ import { toolMcpGatewayTokens, toolMcpSkills, toolMcpTasks, + toolMcpElicitations, toolPolicies, toolProfileBindings, toolProfileEntries, @@ -202,6 +204,17 @@ import { mcpHttpRequestHeaders, parseMcpHttpResponseBody, } from "./mcp-http.js"; +import { + buildElicitationQuestionSet, + elicitationQuestionHash, + elicitationResponseForCard, + formElicitationOf, + MCP_ELICITATION_MAX_FORMS_PER_ROUND, + openElicitationState, + sealElicitationState, + type McpElicitationResponse, + type McpFormElicitation, +} from "./mcp-elicitation.js"; import { projectedConnectionHeaders, projectedConnectionToolArguments, @@ -581,6 +594,9 @@ type RemoteHttpErrorEvidence = { // that passes while they are recorded must not turn them into a timeout. const REMOTE_TOOL_CALL_WORKFLOW_REASON_CODES = new Set([ "elicitation_required", + "elicitation_sensitive_refused", + "elicitation_unsupported", + "elicitation_limit_reached", "provider_interaction_required", "provider_input_required", "provider_retry_requested", @@ -699,7 +715,8 @@ const MCP_CACHEABLE_CONTEXT_METHODS = new Set(["resources/list", "resources/read * person action is recorded, so claiming "accept" would assert consent nobody * gave. The next call goes out fresh and the server, which tracks out-of-band * steps itself, decides. A bare `requestState` (try again) is kept for the - * retry because it asserts nothing. Form, sampling, and roots are not collected yet. + * retry because it asserts nothing. Form questions in a task go to a person + * first (see askElicitationRound); sampling and roots are not collected. */ export function mcpInputRequiredPending(inputRequired: McpInputRequired): { answerable: boolean; @@ -709,7 +726,11 @@ export function mcpInputRequiredPending(inputRequired: McpInputRequired): { } { const links: ToolUpstreamPending["links"] = []; let message: string | undefined; + let forms = 0; for (const request of Object.values(inputRequired.inputRequests)) { + // A form beside a link waits: the link goes first, and the server asks + // for the form again on the next call (spec: re-ask for missing input). + if (request.method === "elicitation/create" && (request.params.mode === undefined || request.params.mode === "form")) { forms += 1; continue; } if (request.method !== "elicitation/create" || request.params.mode !== "url") return { answerable: false }; const checked = checkOAuthEndpointUrl(request.params.url); if (!checked.ok) return { answerable: false }; @@ -717,7 +738,7 @@ export function mcpInputRequiredPending(inputRequired: McpInputRequired): { if (!message && typeof request.params.message === "string") message = redactSensitiveText(request.params.message).slice(0, 4000); } if (!links.length) { - return inputRequired.requestState ? { answerable: true, retryState: inputRequired.requestState } : { answerable: false }; + return inputRequired.requestState && !forms ? { answerable: true, retryState: inputRequired.requestState } : { answerable: false }; } // The link flow shows up to eight; more is a clear stop, never a partial step. if (links.length > MCP_MAX_INPUT_LINKS) return { answerable: false }; @@ -6521,103 +6542,50 @@ export function createToolGatewayService( return { message, requestedSchema, raw: candidate }; } - function enumOptions( - values: unknown[], - ): Array<{ id: string; label: string }> { - return values.slice(0, 10).map((value, index) => { - const label = - typeof value === "string" || - typeof value === "number" || - typeof value === "boolean" - ? String(value) - : `Option ${index + 1}`; - return { - id: slugSegment(label, `option-${index + 1}`).slice(0, 120), - label: label.slice(0, 120), - }; - }); - } - - function elicitationQuestions(request: McpElicitationRequest) { - const schema = request.requestedSchema; - const properties = asRecord(schema?.properties); - const required = Array.isArray(schema?.required) - ? new Set( - schema.required.filter( - (item): item is string => typeof item === "string", - ), - ) - : new Set(); - const questions: Array<{ - id: string; - prompt: string; - helpText?: string | null; - selectionMode: "single" | "multi"; - required?: boolean; - options: Array<{ - id: string; - label: string; - description?: string | null; - }>; - }> = []; - if (properties) { - for (const [key, rawProperty] of Object.entries(properties)) { - if (questions.length >= 10) break; - const property = asRecord(rawProperty) ?? {}; - const enumValues = Array.isArray(property.enum) ? property.enum : []; - const options = - enumValues.length > 0 - ? enumOptions(enumValues) - : [{ id: "answer", label: "Provide answer" }]; - questions.push({ - id: key.slice(0, 120), - prompt: ( - stringValue(property.title) ?? - stringValue(property.description) ?? - key - ).slice(0, 500), - helpText: - enumValues.length > 0 - ? null - : "Use Other to enter the requested value.", - selectionMode: "single", - required: required.has(key), - options, - }); - } + /** The connection's name, as the person knows the app. */ + async function elicitationAppName(companyId: string, connectionId: string | null | undefined, tool?: ToolGatewayDescriptor) { + const named = tool?.applicationDisplayName?.trim(); + if (named) return named; + if (connectionId) { + const [connection] = await db.select({ name: toolConnections.name }).from(toolConnections) + .where(and(eq(toolConnections.id, connectionId), eq(toolConnections.companyId, companyId))).limit(1); + if (connection?.name.trim()) return connection.name.trim(); } - if (questions.length > 0) return questions; - return [ - { - id: "response", - prompt: request.message.slice(0, 500), - helpText: "Use Other to enter the requested response.", - selectionMode: "single" as const, - required: true, - options: [{ id: "answer", label: "Provide response" }], - }, - ]; + return "The app"; } - /** The question card a provider's input request becomes; answering it wakes the assignee. */ - function createElicitationCard(input: { - companyId: string; issueId: string; agentId: string | null; toolName: string; - request: McpElicitationRequest; idempotencyKey: string; + /** Why a provider's form was not shown; the agent reads this, the person sees no card. */ + function elicitationRefusedError(appName: string, built: { reason: "sensitive" | "unsupported" }, invocationId?: string) { + return built.reason === "sensitive" + ? new ToolGatewayHttpError(409, + `${appName} asked for a password, key, or other secret in a form. Paperclip doesn't collect secrets that way, so nothing was shown. The app must ask for it through a secure link instead.`, + "elicitation_sensitive_refused", invocationId ? { invocationId } : undefined) + : new ToolGatewayHttpError(409, + `${appName} asked a question Paperclip can't show in the task chat. Complete this step in ${appName}, then try again.`, + "elicitation_unsupported", invocationId ? { invocationId } : undefined); + } + + /** + * The question card a provider's form becomes. Only people answer it: an + * agent answering its own tool's question would consent on nobody's behalf. + * Answering wakes the assignee through the question-response outbox. + */ + async function createElicitationCard(input: { + companyId: string; issueId: string; agentId: string | null; appName: string; + request: McpFormElicitation; idempotencyKey: string; }) { + const built = buildElicitationQuestionSet({ appName: input.appName, request: input.request }); + if (!built.ok) throw elicitationRefusedError(input.appName, built); return interactions.create( { id: input.issueId, companyId: input.companyId }, { kind: "ask_user_questions", idempotencyKey: input.idempotencyKey, - title: "Tool needs input", - summary: `${input.toolName} asked for more information before it can continue.`, + title: built.questionSet.title, + summary: built.questionSet.description, continuationPolicy: "wake_assignee", - payload: { - version: 1, - title: input.request.message.slice(0, 240), - submitLabel: "Send response", - questions: elicitationQuestions(input.request), - }, + resolverPolicy: "human_only", + payload: { version: 1, supersedeOnUserComment: false, questionSet: built.questionSet }, }, { agentId: input.agentId }, // Each provider question is independent: a new one must never expire @@ -6626,40 +6594,31 @@ export function createToolGatewayService( ); } - async function requestElicitationForRecordedToolCall(input: { - session: ToolGatewaySession; - tool: ToolGatewayDescriptor; - invocationId: string; - request: McpElicitationRequest; - /** One card per provider input request; defaults to one per call. */ - idempotencyKey?: string; - /** The name the card shows; a task's cards use one name on every path. */ - cardToolName?: string; + /** What the agent reads while a person has the app's question. */ + function elicitationWaitError(appName: string, details: Record) { + return new ToolGatewayHttpError( + 409, + `${appName} asked a question, and it is waiting for a person in the task chat. Paperclip wakes you when it is answered; then call this tool again with the same arguments to continue. Don't ask the question yourself.`, + "elicitation_required", + details, + ); + } + + async function markInvocationAwaitingElicitation(invocationId: string) { + await db.update(toolInvocations).set({ + status: "awaiting_approval", + errorCode: "elicitation_required", + errorMessage: "The app asked a question; Paperclip put it in the task chat for a person to answer.", + updatedAt: new Date(), + }).where(eq(toolInvocations.id, invocationId)); + } + + /** Record that the call stopped for a person's answer, then end it. */ + async function stopForElicitation(input: { + session: ToolGatewaySession; tool: ToolGatewayDescriptor; invocationId: string; + appName: string; interactionIds: string[]; requests: McpFormElicitation[]; }): Promise { - if (!input.session.issueId) { - throw new ToolGatewayHttpError( - 409, - "MCP elicitation is not supported for non-interactive gateway clients", - "elicitation_not_supported", - { invocationId: input.invocationId, tool: input.tool.name }, - ); - } - const interaction = await createElicitationCard({ - companyId: input.session.companyId, issueId: input.session.issueId, agentId: input.session.agentId, - toolName: input.cardToolName ?? input.tool.name, request: input.request, - idempotencyKey: input.idempotencyKey ?? `mcp-elicitation:${input.invocationId}`, - }); - const now = new Date(); - await db - .update(toolInvocations) - .set({ - status: "awaiting_approval", - errorCode: "elicitation_required", - errorMessage: - "Remote MCP tool requested elicitation; Paperclip created an issue interaction for the response.", - updatedAt: now, - }) - .where(eq(toolInvocations.id, input.invocationId)); + await markInvocationAwaitingElicitation(input.invocationId); await writeToolCallEvent({ invocationId: input.invocationId, session: input.session, @@ -6669,11 +6628,10 @@ export function createToolGatewayService( policyDecision: "defer_runtime", reasonCode: "elicitation_required", metadata: { - interactionId: interaction.id, - elicitation: { - message: input.request.message, - requestedSchema: input.request.requestedSchema, - }, + interactionId: input.interactionIds[0] ?? null, + interactionIds: input.interactionIds, + // The question's words and shape, never an answer. + elicitation: input.requests.map((request) => ({ message: request.message, requestedSchema: request.requestedSchema })), }, tool: input.tool, }); @@ -6686,23 +6644,332 @@ export function createToolGatewayService( action: "tool_gateway.elicitation_requested", details: { invocationId: input.invocationId, - interactionId: interaction.id, + interactionId: input.interactionIds[0] ?? null, + interactionIds: input.interactionIds, decision: "defer_runtime", reasonCode: "elicitation_required", tool: input.tool.name, ...toolAuditMetadata(input.tool), }, }); - throw new ToolGatewayHttpError( - 409, - "MCP tool requested additional input", - "elicitation_required", - { - invocationId: input.invocationId, - interactionId: interaction.id, - tool: input.tool.name, + throw elicitationWaitError(input.appName, { + invocationId: input.invocationId, + interactionId: input.interactionIds[0] ?? null, + interactionIds: input.interactionIds, + tool: input.tool.name, + }); + } + + /** + * A provider question with no way to send an answer back (a vendor result + * shape, or an MCP Task's question shown inline). The card is shown and the + * call ends; answering wakes the agent. + */ + async function requestElicitationForRecordedToolCall(input: { + session: ToolGatewaySession; + tool: ToolGatewayDescriptor; + invocationId: string; + request: McpElicitationRequest; + /** One card per provider input request; defaults to one per call. */ + idempotencyKey?: string; + connectionId?: string | null; + }): Promise { + if (!input.session.issueId) { + throw new ToolGatewayHttpError( + 409, + "MCP elicitation is not supported for non-interactive gateway clients", + "elicitation_not_supported", + { invocationId: input.invocationId, tool: input.tool.name }, + ); + } + const appName = await elicitationAppName(input.session.companyId, input.connectionId ?? input.tool.connectionId, input.tool); + const interaction = await createElicitationCard({ + companyId: input.session.companyId, issueId: input.session.issueId, agentId: input.session.agentId, + appName, request: input.request, + idempotencyKey: input.idempotencyKey ?? `mcp-elicitation:${input.invocationId}`, + }); + return stopForElicitation({ + session: input.session, tool: input.tool, invocationId: input.invocationId, + appName, interactionIds: [interaction.id], requests: [input.request], + }); + } + + /** How long a question round can wait for its answers before the call starts over. */ + const MCP_ELICITATION_TTL_MS = 24 * 60 * 60_000; + /** Open question rounds per task; more is a clear stop, never a wall of cards. */ + const MCP_ELICITATION_MAX_OPEN_ROUNDS = 10; + type McpElicitationRow = typeof toolMcpElicitations.$inferSelect; + type McpElicitationCall = { + session: ToolGatewaySession & { issueId: string }; + tool: ToolGatewayDescriptor; + connectionId: string; + grantId: string; + /** Keyed identity of the call, including forwarded routing headers; only the same call collects its round. */ + callIdentity: string; + /** Keyed identity of the full arguments, checked before a keyed retry runs again. */ + argumentsIdentity: string; + appName: string; + }; + const mcpElicitationScope = (row: Pick) => ({ companyId: row.companyId, connectionId: row.connectionId }); + const mcpElicitationCallIdentity = (continuationKey: string, routingIdentity: string) => + createHmac("sha256", mcpTaskSecret()).update("paperclip:mcp-elicitation-call:v1\0").update(continuationKey).update("\0").update(routingIdentity).digest("hex"); + const mcpElicitationCardKey = (roundId: string, key: string) => + `mcp-elicit:${roundId}:${createHash("sha256").update(key).digest("hex").slice(0, 32)}`; + + /** + * Save a round of form questions, then put each on a card, then end the + * call. The round is saved first: a crash before a card exists is repaired + * on the next call, which makes the missing card under the same key. + * Forms are checked before anything is saved or shown. + */ + async function askElicitationRound(call: McpElicitationCall, input: { + invocationId: string; + era: "modern" | "legacy"; + requestState: string | null; + forms: Array<{ key: string; request: McpFormElicitation }>; + /** An answered round this call just sent: spent in the same transaction that saves the next one. */ + spend?: { row: McpElicitationRow; responses: Record }; + }): Promise { + const { session } = call; + // The app already has the answered round's answers, even if this next one is refused. + const spendFirst = async () => { if (input.spend) await settleElicitationRound(input.spend.row, call, input.spend.responses, input.invocationId); }; + for (const form of input.forms) { + const built = buildElicitationQuestionSet({ appName: call.appName, request: form.request }); + if (!built.ok) { + await spendFirst(); + await writeAudit({ + session, companyId: session.companyId, agentId: session.agentId, runId: session.runId, issueId: session.issueId, + action: "tool_gateway.elicitation_refused", + details: { + invocationId: input.invocationId, tool: call.tool.name, reason: built.reason, + ...(built.reason === "sensitive" ? { fields: built.fields.slice(0, 10) } : {}), + }, + }); + throw elicitationRefusedError(call.appName, built, input.invocationId); + } + } + if (input.forms.length > MCP_ELICITATION_MAX_FORMS_PER_ROUND) { + await spendFirst(); + throw elicitationRefusedError(call.appName, { reason: "unsupported" }, input.invocationId); + } + const [open] = await db.select({ count: sql`count(*)::int` }).from(toolMcpElicitations).where(and( + eq(toolMcpElicitations.companyId, session.companyId), eq(toolMcpElicitations.issueId, session.issueId), + eq(toolMcpElicitations.status, "waiting"), gt(toolMcpElicitations.expiresAt, new Date()), + // The answered round this one replaces is not counted: it is spent as this is saved. + ...(input.spend ? [ne(toolMcpElicitations.id, input.spend.row.id)] : []), + )); + if ((open?.count ?? 0) >= MCP_ELICITATION_MAX_OPEN_ROUNDS) { + await spendFirst(); + throw new ToolGatewayHttpError(409, + `This task already has ${MCP_ELICITATION_MAX_OPEN_ROUNDS} app questions waiting for a person. Wait for those to be answered before trying again.`, + "elicitation_limit_reached", { invocationId: input.invocationId }); + } + // One waiting round per call (per question for older servers, which ask + // one at a time): two matching calls at once share one set of cards. + const dedupeKey = createHash("sha256").update(JSON.stringify([ + input.era, call.connectionId, call.grantId, call.callIdentity, + ...(input.era === "legacy" ? input.forms.map((form) => elicitationQuestionHash(form.request)) : []), + ])).digest("hex"); + let row: McpElicitationRow | undefined; + for (let attempt = 0; attempt < 2 && !row; attempt += 1) { + const id = randomUUID(); + const questions: McpElicitationRow["questions"] = input.forms.map((form) => ({ + key: form.key, questionHash: elicitationQuestionHash(form.request), cardKey: mcpElicitationCardKey(id, form.key), + message: form.request.message, requestedSchema: form.request.requestedSchema, + })); + // The answered round is spent only together with saving the next one, + // so a stop between the two never loses the way to continue. + // If saving fails, the answered round is released, not left held, so the next call sends it again. + [row] = await db.transaction(async (tx) => { + if (input.spend && attempt === 0) { + await tx.update(toolMcpElicitations).set({ status: "sent", sentAt: new Date(), claimedUntil: null, updatedAt: new Date() }) + .where(eq(toolMcpElicitations.id, input.spend.row.id)); + } + return tx.insert(toolMcpElicitations).values({ + id, companyId: session.companyId, issueId: session.issueId, agentId: session.agentId, runId: session.runId, + invocationId: input.invocationId, connectionId: call.connectionId, grantId: call.grantId, + catalogToolName: call.tool.name, callIdentity: call.callIdentity, argumentsIdentity: call.argumentsIdentity, + dedupeKey, era: input.era, + sealedRequestState: input.requestState === null ? null + : sealElicitationState(mcpTaskSecret(), { companyId: session.companyId, connectionId: call.connectionId }, input.requestState), + questions, status: "waiting", expiresAt: new Date(Date.now() + MCP_ELICITATION_TTL_MS), + }).onConflictDoNothing().returning(); + }).catch(async (error: unknown) => { + if (input.spend) await releaseElicitationRound(input.spend.row).catch(() => undefined); + throw error; + }); + if (input.spend && attempt === 0) await writeElicitationSentAudit(input.spend.row, call, input.spend.responses, input.invocationId); + if (row) break; + // Another matching call saved the round first: show its cards. An + // expired one is ended so this call can save a fresh round. + const [existing] = await db.select().from(toolMcpElicitations).where(and( + eq(toolMcpElicitations.companyId, session.companyId), eq(toolMcpElicitations.issueId, session.issueId), + eq(toolMcpElicitations.dedupeKey, dedupeKey), eq(toolMcpElicitations.status, "waiting"), + )).limit(1); + if (existing && existing.expiresAt > new Date()) row = existing; + else if (existing) await expireElicitationRound(existing, call); + } + if (!row) throw elicitationWaitError(call.appName, { invocationId: input.invocationId, tool: call.tool.name }); + const cards = await ensureElicitationCards(row, call); + return stopForElicitation({ + session, tool: call.tool, invocationId: input.invocationId, appName: call.appName, + interactionIds: cards.map((card) => card.id), requests: input.forms.map((form) => form.request), + }); + } + + /** + * The round's cards, found by their keys. A card that is missing (the + * process stopped between saving the round and showing it) is made now + * under the same key, so a question is never lost or shown twice. + */ + async function ensureElicitationCards(row: McpElicitationRow, call: McpElicitationCall) { + const keys = row.questions.map((question) => question.cardKey); + const found = keys.length ? await db.select({ + id: issueThreadInteractions.id, idempotencyKey: issueThreadInteractions.idempotencyKey, + status: issueThreadInteractions.status, result: issueThreadInteractions.result, payload: issueThreadInteractions.payload, + }).from(issueThreadInteractions).where(and( + eq(issueThreadInteractions.companyId, row.companyId), eq(issueThreadInteractions.issueId, row.issueId), + inArray(issueThreadInteractions.idempotencyKey, keys), + )) : []; + const cards: Array<{ id: string; status: string; result: unknown; payload: unknown }> = []; + let made = false; + for (const question of row.questions) { + let card: { id: string; status: string; result: unknown; payload: unknown } | undefined = + found.find((candidate) => candidate.idempotencyKey === question.cardKey); + if (!card) { + const created = await createElicitationCard({ + companyId: row.companyId, issueId: row.issueId, agentId: row.agentId, appName: call.appName, + request: { message: question.message, requestedSchema: question.requestedSchema }, idempotencyKey: question.cardKey, + }); + card = { id: created.id, status: created.status, result: created.result ?? null, payload: created.payload }; + made = true; + } + cards.push(card); + } + if (made || row.questions.some((question, index) => question.interactionId !== cards[index]!.id)) { + const questions = row.questions.map((question, index) => ({ ...question, interactionId: cards[index]!.id })); + await db.update(toolMcpElicitations).set({ questions, updatedAt: new Date() }).where(eq(toolMcpElicitations.id, row.id)); + row.questions = questions; + } + return cards; + } + + /** The card's answer for each question of a round; null while any waits. */ + async function elicitationRoundResponses(row: McpElicitationRow, call: McpElicitationCall) { + const cards = await ensureElicitationCards(row, call); + const responses: Record = {}; + for (const [index, question] of row.questions.entries()) { + const response = elicitationResponseForCard({ message: question.message, requestedSchema: question.requestedSchema }, cards[index]); + if (!response) return null; + responses[question.key] = response; + } + return responses; + } + + + /** End a round nobody answered in time; its cards are withdrawn so nobody answers a dead question. */ + async function expireElicitationRound(row: McpElicitationRow, call: McpElicitationCall) { + const [expired] = await db.update(toolMcpElicitations).set({ status: "expired", claimedUntil: null, updatedAt: new Date() }) + .where(and(eq(toolMcpElicitations.id, row.id), eq(toolMcpElicitations.status, "waiting"))).returning({ id: toolMcpElicitations.id }); + if (!expired) return; + for (const question of row.questions) { + if (!question.interactionId) continue; + await interactions.withdrawInteraction({ id: row.issueId, companyId: row.companyId }, question.interactionId, + { reason: "The app's question expired before it was answered." }, { agentId: row.agentId, runId: call.session.runId }) + .catch(() => undefined); + } + await writeAudit({ + session: call.session, companyId: row.companyId, agentId: call.session.agentId, runId: call.session.runId, issueId: row.issueId, + action: "tool_gateway.elicitation_expired", + details: { roundId: row.id, invocationId: row.invocationId, tool: row.catalogToolName }, + }); + } + + /** + * The waiting round for this exact call. Before every card is resolved the + * call stops again without contacting the app. Once they are, the round is + * claimed (so two repeated calls never both send) and its answers returned. + */ + async function collectElicitationRound(call: McpElicitationCall, input: { + invocationId: string; era: "modern" | "legacy"; questionHash?: string; claimMs: number; + }): Promise<{ row: McpElicitationRow; requestState: string | null; responses: Record } | null> { + const { session } = call; + const rows = await db.select().from(toolMcpElicitations).where(and( + eq(toolMcpElicitations.companyId, session.companyId), eq(toolMcpElicitations.issueId, session.issueId), + eq(toolMcpElicitations.connectionId, call.connectionId), eq(toolMcpElicitations.grantId, call.grantId), + eq(toolMcpElicitations.callIdentity, call.callIdentity), eq(toolMcpElicitations.era, input.era), + eq(toolMcpElicitations.status, "waiting"), + )).orderBy(desc(toolMcpElicitations.createdAt)).limit(MCP_ELICITATION_MAX_OPEN_ROUNDS); + const now = new Date(); + for (const row of rows) { + if (input.questionHash && !row.questions.some((question) => question.questionHash === input.questionHash)) continue; + if (row.expiresAt <= now) { await expireElicitationRound(row, call); continue; } + const responses = await elicitationRoundResponses(row, call); + if (!responses) { + await markInvocationAwaitingElicitation(input.invocationId); + throw elicitationWaitError(call.appName, { + invocationId: input.invocationId, interactionId: row.questions[0]?.interactionId ?? null, + interactionIds: row.questions.map((question) => question.interactionId).filter(Boolean), tool: call.tool.name, + }); + } + const [claimed] = await db.update(toolMcpElicitations) + .set({ claimedUntil: new Date(Date.now() + input.claimMs), updatedAt: new Date() }) + .where(and(eq(toolMcpElicitations.id, row.id), eq(toolMcpElicitations.status, "waiting"), + or(isNull(toolMcpElicitations.claimedUntil), lt(toolMcpElicitations.claimedUntil, new Date())))) + .returning(); + if (!claimed) { + // Another call is sending these answers right now. + await markInvocationAwaitingElicitation(input.invocationId); + throw elicitationWaitError(call.appName, { invocationId: input.invocationId, tool: call.tool.name }); + } + const requestState = claimed.sealedRequestState === null ? null + : openElicitationState(mcpTaskSecret(), mcpElicitationScope(claimed), claimed.sealedRequestState); + if (claimed.sealedRequestState !== null && requestState === null) { + // Not readable here (a rotated secret): start over rather than send a guess. + await db.update(toolMcpElicitations).set({ status: "expired", claimedUntil: null, updatedAt: new Date() }).where(eq(toolMcpElicitations.id, claimed.id)); + return null; + } + return { row: claimed, requestState, responses }; + } + return null; + } + + /** The round's answers reached the app and the call finished: it is spent. */ + async function settleElicitationRound(row: McpElicitationRow, call: McpElicitationCall, responses: Record, invocationId: string) { + await db.update(toolMcpElicitations).set({ status: "sent", sentAt: new Date(), claimedUntil: null, updatedAt: new Date() }) + .where(eq(toolMcpElicitations.id, row.id)); + await writeElicitationSentAudit(row, call, responses, invocationId); + } + + async function writeElicitationSentAudit(row: McpElicitationRow, call: McpElicitationCall, responses: Record, invocationId: string) { + await writeAudit({ + session: call.session, companyId: row.companyId, agentId: call.session.agentId, runId: call.session.runId, issueId: row.issueId, + action: "tool_gateway.elicitation_sent", + details: { + roundId: row.id, invocationId, askedInvocationId: row.invocationId, tool: row.catalogToolName, + // Which kind of answer went back, never what the person wrote. + responses: Object.values(responses).map((response) => response.action), }, - ); + }); + } + + /** The app did not finish with these answers: the next identical call uses them again. */ + async function releaseElicitationRound(row: McpElicitationRow) { + await db.update(toolMcpElicitations).set({ claimedUntil: null, updatedAt: new Date() }) + .where(and(eq(toolMcpElicitations.id, row.id), eq(toolMcpElicitations.status, "waiting"))); + } + + /** + * A keyed retry of a call that stopped for an app's question runs again only + * when it is the same request: same tool and full arguments as the round. + */ + async function elicitationRetryMatches(companyId: string, invocationId: string, toolName: string, parameters: unknown) { + const [row] = await db.select({ id: toolMcpElicitations.id }).from(toolMcpElicitations).where(and( + eq(toolMcpElicitations.companyId, companyId), eq(toolMcpElicitations.invocationId, invocationId), + eq(toolMcpElicitations.catalogToolName, toolName), eq(toolMcpElicitations.argumentsIdentity, mcpTaskArgumentsIdentity(parameters)), + eq(toolMcpElicitations.status, "waiting"), + )).limit(1); + return Boolean(row); } function normalizeMcpContent(content: unknown): string { @@ -7071,40 +7338,6 @@ export function createToolGatewayService( return row.id; } - /** Map a person's question-card answers back onto the provider's form schema. */ - function elicitationContent(request: McpElicitationRequest, result: unknown): Record { - const answers = (Array.isArray(asRecord(result)?.answers) ? asRecord(result)!.answers as unknown[] : []) - .map((answer) => asRecord(answer)).filter((answer): answer is Record => Boolean(answer)); - const textOf = (answer: Record | undefined) => typeof answer?.otherText === "string" ? answer.otherText.trim() : ""; - const properties = asRecord(request.requestedSchema?.properties); - if (!properties || Object.keys(properties).length === 0) { - const text = textOf(answers.find((answer) => answer.questionId === "response")); - return text ? { response: text } : {}; - } - const content: Record = {}; - for (const [key, raw] of Object.entries(properties).slice(0, 10)) { - const property = asRecord(raw) ?? {}; - const answer = answers.find((candidate) => candidate.questionId === key.slice(0, 120)); - if (!answer) continue; - const enumValues = Array.isArray(property.enum) ? property.enum : []; - const optionIds = Array.isArray(answer.optionIds) ? answer.optionIds : []; - const index = enumOptions(enumValues).findIndex((option) => optionIds.includes(option.id)); - if (index >= 0) { content[key] = enumValues[index]; continue; } - const text = textOf(answer); - if (!text) continue; - if (property.type === "number" || property.type === "integer") { - const value = Number(text); - if (Number.isFinite(value) && (property.type === "number" || Number.isInteger(value))) content[key] = value; - } else if (property.type === "boolean") { - if (/^(true|yes|y|1)$/i.test(text)) content[key] = true; - else if (/^(false|no|n|0)$/i.test(text)) content[key] = false; - } else { - content[key] = text; - } - } - return content; - } - type McpTaskInputPlan = { /** Responses ready to send with tasks/update. */ responses: Record; @@ -7144,17 +7377,17 @@ export function createToolGatewayService( raw: request.params, }, }; - const [card] = await db.select({ status: issueThreadInteractions.status, result: issueThreadInteractions.result }) + // A secret or a shape the card can't show is never put on a card. + if (!buildElicitationQuestionSet({ appName: "app", request: form.request }).ok) return { ...plan, unsupported: true }; + const [card] = await db.select({ status: issueThreadInteractions.status, result: issueThreadInteractions.result, payload: issueThreadInteractions.payload }) .from(issueThreadInteractions).where(and( eq(issueThreadInteractions.companyId, row.companyId), eq(issueThreadInteractions.issueId, row.issueId), eq(issueThreadInteractions.idempotencyKey, mcpTaskInputKey(row, key)), )).limit(1); - if (!card || card.status === "pending") { plan.waitingForm ??= { ...form, hasCard: Boolean(card) }; continue; } - const outcome = asRecord(card.result)?.outcome; - plan.responses[key] = card.status === "answered" && outcome === undefined - ? { action: "accept", content: elicitationContent(form.request, card.result) } - : { action: "decline" }; + const response = elicitationResponseForCard(form.request, card); + if (!response) { plan.waitingForm ??= { ...form, hasCard: Boolean(card) }; continue; } + plan.responses[key] = response; } if (plan.links.length > MCP_MAX_INPUT_LINKS) plan.unsupported = true; return plan; @@ -7278,7 +7511,8 @@ export function createToolGatewayService( // A form counts as shown only once its card exists. if (plan.waitingForm && !plan.waitingForm.hasCard && row.issueId) { await createElicitationCard({ - companyId: row.companyId, issueId: row.issueId, agentId: row.agentId, toolName: row.catalogToolName, + companyId: row.companyId, issueId: row.issueId, agentId: row.agentId, + appName: await elicitationAppName(row.companyId, row.connectionId, wait.tool), request: plan.waitingForm.request, idempotencyKey: mcpTaskInputKey(row, plan.waitingForm.key), }); } @@ -7295,7 +7529,7 @@ export function createToolGatewayService( await requestElicitationForRecordedToolCall({ session: wait.session, tool: wait.tool, invocationId: wait.invocationId, request: plan.waitingForm.request, idempotencyKey: mcpTaskInputKey(row, plan.waitingForm.key), - cardToolName: row.catalogToolName, + connectionId: row.connectionId, }); } } @@ -7452,7 +7686,8 @@ export function createToolGatewayService( : Infinity; if (!plan.unsupported && plan.waitingForm && row.issueId && cardCount < MCP_TASK_MAX_CARDS) { const card = await createElicitationCard({ - companyId: row.companyId, issueId: row.issueId, agentId: row.agentId, toolName: row.catalogToolName, + companyId: row.companyId, issueId: row.issueId, agentId: row.agentId, + appName: await elicitationAppName(row.companyId, row.connectionId), request: plan.waitingForm.request, idempotencyKey: mcpTaskInputKey(row, plan.waitingForm.key), }); surfaced.add(plan.waitingForm.key); @@ -7584,6 +7819,11 @@ export function createToolGatewayService( // The cached MCP session this call used, so a failure resets only this // identity's session and leaves other agents on the connection alone. let mcpSession: { scope: string; headers: Record; sessionId: string } | undefined; + // Answers to an app's question that this call is carrying back, until the app has them. + let elicitationClaim: Awaited> = null; + // An older app may ask several questions in turn; each answer is spent + // only when the whole call finishes, so a later stop never loses one. + const legacyAnswered: Array<{ row: McpElicitationRow; responses: Record }> = []; try { const dispatchRemote = async (target: string, init: RequestInit) => { // Refresh can reduce consent. Recheck the selected grant after refreshing @@ -7675,13 +7915,17 @@ export function createToolGatewayService( if (isModernMcpProtocolVersion(knownVersion)) { requestHeaders = { ...headers, "MCP-Protocol-Version": knownVersion }; } else if (connectionRequiresMcpSession(connection)) { - const scope = `${connection.id}:grant:${grant.id}:actor:${session.agentId}:${endpoint}`; + // Tool calls answer an older server's questions, so their session + // declares form questions; it is kept apart from sessions that only + // read (resources, prompts, listing), which declare none. + const scope = `${connection.id}:grant:${grant.id}:actor:${session.agentId}:elicit-form:${endpoint}`; requestHeaders = await getMcpHttpSession({ scope, send: sendForNegotiation, headers, requestId, eraKey, + elicitationForm: true, }); const sessionId = new Headers(requestHeaders).get("mcp-session-id"); if (sessionId) mcpSession = { scope, headers, sessionId }; @@ -7692,11 +7936,84 @@ export function createToolGatewayService( // Tied to the server that issued the state: a changed URL starts fresh. const continuationKey = `${connection.id}:${createHash("sha256").update(String(endpoint)).digest("hex")}:${grant.id}:${session.agentId}:${upstreamToolName}:${createHash("sha256").update(JSON.stringify(parameters ?? {})).digest("hex")}`; const continuation = mcpContinuations.get(continuationKey); + // A person can answer the app's questions only when the call belongs to a task. + const elicitationCall: McpElicitationCall | null = session.issueId ? { + session: session as ToolGatewaySession & { issueId: string }, tool, + connectionId: connection.id, grantId: grant.id, + // Forwarded routing headers (such as a tenant) are part of the call: + // one tenant's answers never go out with another tenant's headers. + callIdentity: mcpElicitationCallIdentity(continuationKey, mcpTaskArgumentsIdentity(mcpTaskCallerHeaders(connection, callerHeaders))), + argumentsIdentity: mcpTaskArgumentsIdentity(parameters), + appName: tool.applicationDisplayName?.trim() || connection.name.trim() || "The app", + } : null; + // A question round this exact call left for a person (stateless MCP). + // Until every card is resolved the call stops again without contacting + // the app; then this call carries the answers and the app's exact state. + if (elicitationCall && isModernMcpProtocolVersion(mcpProtocolVersionOf(requestHeaders))) { + elicitationClaim = await collectElicitationRound(elicitationCall, { invocationId, era: "modern", claimMs: ms + 60_000 }); + } + /** + * An older server asks over the open stream and waits for the reply. + * The first time, the question goes on a card and the server is told + * "cancel" at once, so it never hangs. When the person has answered, + * the same question on this same call is answered from the card, once. + */ + const answerLegacyElicitation = async (messageId: unknown, form: McpFormElicitation) => { + const respond = (result: McpElicitationResponse) => dispatchRemote(endpoint, { + method: "POST", headers: mcpHttpRequestHeaders(requestHeaders), + body: JSON.stringify({ jsonrpc: "2.0", id: messageId, result }), + }); + const cancel = () => respond({ action: "cancel" }).catch(() => undefined); + if (!elicitationCall) { + await cancel(); + throw new ToolGatewayHttpError(409, "MCP elicitation is not supported for non-interactive gateway clients", + "elicitation_not_supported", { invocationId, tool: tool.name }); + } + const questionHash = elicitationQuestionHash(form); + // The app asks a question it already had an answer to in this call + // (to have it corrected, or a second time): that answer is spent and + // the person is asked again on a fresh card. + const earlier = legacyAnswered.findIndex((used) => used.row.questions.some((question) => question.questionHash === questionHash)); + if (earlier >= 0) { + const [used] = legacyAnswered.splice(earlier, 1); + try { + await askElicitationRound(elicitationCall, { invocationId, era: "legacy", requestState: null, forms: [{ key: "question", request: form }], spend: used }); + } finally { + await cancel(); + } + } + let answered: Awaited>; + try { + answered = await collectElicitationRound(elicitationCall, { invocationId, era: "legacy", questionHash, claimMs: ms + 60_000 }); + } catch (error) { + await cancel(); + throw error; + } + if (answered) { + const key = answered.row.questions.find((question) => question.questionHash === questionHash)!.key; + const response = answered.responses[key]!; + legacyAnswered.push({ row: answered.row, responses: { [key]: response } }); + const delivered = await respond(response); + // An app that refused the reply never got the answer: keep it for the next call. + if (!delivered.ok) { + throw new ToolGatewayHttpError(502, `${elicitationCall.appName} did not accept the answer to its question. Try the call again.`, + "mcp_remote_status", { status: delivered.status, connectionId: connection.id, catalogEntryId: entry.id }); + } + return; + } + try { + await askElicitationRound(elicitationCall, { invocationId, era: "legacy", requestState: null, forms: [{ key: "question", request: form }] }); + } finally { + await cancel(); + } + }; let callRequestId = requestId; let callParams: Record = { name: upstreamToolName, arguments: parameters, - ...(continuation ? { requestState: continuation.requestState } : {}), + ...(elicitationClaim + ? { inputResponses: elicitationClaim.responses, ...(elicitationClaim.requestState !== null ? { requestState: elicitationClaim.requestState } : {}) } + : continuation ? { requestState: continuation.requestState } : {}), }; // The guard runs inside this call and the connection is pinned to the // address it approved, so an operator-supplied hostname cannot be rebound @@ -7709,6 +8026,8 @@ export function createToolGatewayService( toolInputSchema: entry.inputSchema ?? undefined, // Paperclip can wait on a task here, durably (see waitForMcpTask). declareTasks: tasksOn, + // A person can answer the app's form questions in the task chat. + elicitationForm: Boolean(elicitationCall), }), redirect: "manual", signal: controller.signal, @@ -7808,8 +8127,9 @@ export function createToolGatewayService( if (isModernMcpProtocolVersion(retryVersion)) { requestHeaders = { ...headers, "MCP-Protocol-Version": retryVersion }; } else { - const scope = `${connection.id}:grant:${grant.id}:actor:${session.agentId}:${endpoint}`; - requestHeaders = await getMcpHttpSession({ scope, send: sendForNegotiation, headers, requestId: `${requestId}-version`, eraKey }); + // The same form-capable tool session as a first call (see above). + const scope = `${connection.id}:grant:${grant.id}:actor:${session.agentId}:elicit-form:${endpoint}`; + requestHeaders = await getMcpHttpSession({ scope, send: sendForNegotiation, headers, requestId: `${requestId}-version`, eraKey, elicitationForm: true }); const sessionId = new Headers(requestHeaders).get("mcp-session-id"); if (sessionId) mcpSession = { scope, headers, sessionId }; } @@ -7938,8 +8258,12 @@ export function createToolGatewayService( }); throw new ToolGatewayHttpError(409, "This tool needs authorization in the provider.", "provider_interaction_required", { upstreamPending: pending, invocationId }); } - const request = extractMcpElicitationRequest(message); - if (request) await requestElicitationForRecordedToolCall({ session, tool, invocationId, request }); + const form = message.method === "elicitation/create" ? formElicitationOf(asRecord(message.params) ?? {}) : null; + if (form) await answerLegacyElicitation(message.id, form); + else { + const request = extractMcpElicitationRequest(message); + if (request) await requestElicitationForRecordedToolCall({ session, tool, invocationId, request }); + } }, })) : await readBoundedRemoteResponse(response, (bytes) => { @@ -7992,6 +8316,18 @@ export function createToolGatewayService( // spent (a new input request saves its own below). A rejected attempt, // such as a 401 before an OAuth refresh, keeps it for the explicit retry. if (continuation) mcpContinuations.delete(continuationKey); + // The app answered this call. If it asks a new form round, the answered + // round is spent together with saving that one (askElicitationRound). + const nextInput = mcpInputRequired(asRecord(payloadRecord.result)); + const asksNextForms = Boolean(nextInput && Object.values(nextInput.inputRequests).length + && Object.values(nextInput.inputRequests).every((request) => request.method === "elicitation/create" && formElicitationOf(request.params))); + if (elicitationClaim && !asksNextForms) { + const sent = elicitationClaim; + elicitationClaim = null; + await settleElicitationRound(sent.row, elicitationCall!, sent.responses, invocationId); + } + // The older app finished the call: the answers it used are spent. + for (const used of legacyAnswered.splice(0)) await settleElicitationRound(used.row, elicitationCall!, used.responses, invocationId); const upstreamPending = extractRemoteMcpPending(payloadRecord, String(connection.config.sourceTemplateKey ?? ""), entry.toolName); if (upstreamPending) { await retainUpstreamHandoff(invocationId, upstreamPending); @@ -8042,6 +8378,18 @@ export function createToolGatewayService( if (inputRequired) { // Stateless servers ask for input by returning it as a result. The call // is retried later, with the answers and the server's opaque state. + // Form questions alone go to a person in the task chat; the repeated + // call carries their answers (see collectElicitationRound). + const forms = Object.entries(inputRequired.inputRequests).flatMap(([key, request]) => { + const form = request.method === "elicitation/create" ? formElicitationOf(request.params) : null; + return form ? [{ key, request: form }] : []; + }); + if (elicitationCall && forms.length && forms.length === Object.keys(inputRequired.inputRequests).length) { + const spend = elicitationClaim ?? undefined; + // From here the answered round is spent with the next one saved; never released. + elicitationClaim = null; + await askElicitationRound(elicitationCall, { invocationId, era: "modern", requestState: inputRequired.requestState, forms, spend }); + } const pending = mcpInputRequiredPending(inputRequired); if (!pending.answerable) { throw new ToolGatewayHttpError( @@ -8091,6 +8439,9 @@ export function createToolGatewayService( return { result, failureInstructions, headerSummary, execution }; } } catch (error) { + // The app never got the answers this call carried: the next identical call sends them. + if (elicitationClaim) await releaseElicitationRound(elicitationClaim.row).catch(() => undefined); + for (const used of legacyAnswered.splice(0)) await releaseElicitationRound(used.row).catch(() => undefined); execution.error = remoteHttpErrorEvidence(error); execution.deadline = { timeoutMs: ms, aborted: controller.signal.aborted }; if (execution.response && execution.stream) execution.response.bodySizeBytes = execution.stream.bytesReceived; @@ -9586,20 +9937,31 @@ export function createToolGatewayService( }; } - /** Return a claimed approved action whose provider task is still running to approved, under the same claim. */ - async function resumeApprovedActionForPendingTask(input: { actionRequestId: string; invocationId: string; claimUpdatedAt: Date; message: string }) { + /** + * Return a claimed approved action to approved, under the same claim, when + * its call stopped without finishing: the provider's task is still running, + * or the app asked the person a question. The next execution continues it + * (attaches to the task, or sends the answer) without asking for approval again. + */ + async function resumeApprovedActionForPendingTask(input: { + actionRequestId: string; invocationId: string; claimUpdatedAt: Date; message: string; + errorCode?: "provider_task_pending" | "elicitation_required"; + }) { const now = new Date(); + const errorCode = input.errorCode ?? "provider_task_pending"; return db.transaction(async (tx) => { const [invocation] = await tx.select({ status: toolInvocations.status }).from(toolInvocations) .where(eq(toolInvocations.id, input.invocationId)).for("update").limit(1); - if (invocation?.status !== "executing") return false; + // A question stop already marked the invocation as waiting. + const resumable = invocation?.status === "executing" || (errorCode === "elicitation_required" && invocation?.status === "awaiting_approval"); + if (!resumable) return false; const [actionRequest] = await tx.select({ companyId: toolActionRequests.companyId, status: toolActionRequests.status, updatedAt: toolActionRequests.updatedAt }) .from(toolActionRequests).where(eq(toolActionRequests.id, input.actionRequestId)).for("update").limit(1); if (actionRequest?.status !== "executing" || actionRequest.updatedAt.getTime() !== input.claimUpdatedAt.getTime()) return false; await notifyToolReviewChange(tx, actionRequest.companyId); await tx.update(toolInvocations).set({ - status: "awaiting_approval", startedAt: null, errorCode: "provider_task_pending", errorMessage: input.message, updatedAt: now, - }).where(and(eq(toolInvocations.id, input.invocationId), eq(toolInvocations.status, "executing"))); + status: "awaiting_approval", startedAt: null, errorCode, errorMessage: input.message, updatedAt: now, + }).where(and(eq(toolInvocations.id, input.invocationId), inArray(toolInvocations.status, ["executing", "awaiting_approval"]))); await tx.update(toolActionRequests).set({ status: "approved", updatedAt: now }) .where(and(eq(toolActionRequests.id, input.actionRequestId), eq(toolActionRequests.status, "executing"), eq(toolActionRequests.updatedAt, input.claimUpdatedAt))); return true; @@ -10173,6 +10535,12 @@ export function createToolGatewayService( && await resumeApprovedActionForPendingTask({ actionRequestId: claimed.id, invocationId: invocation.id, claimUpdatedAt: claimed.updatedAt, message: error.message })) { throw error; } + // The app asked the person a question: keep the approval, so the + // repeated call carries the answer instead of failing the action. + if (error instanceof ToolGatewayHttpError && error.reasonCode === "elicitation_required" + && await resumeApprovedActionForPendingTask({ actionRequestId: claimed.id, invocationId: invocation.id, claimUpdatedAt: claimed.updatedAt, message: error.message, errorCode: "elicitation_required" })) { + throw error; + } const { reasonCode } = await markApprovedActionFailed({ actionRequestId: claimed.id, invocationId: invocation.id, @@ -12885,7 +13253,17 @@ export function createToolGatewayService( callerHeaders: input.callerHeaders, }) : null; - const retryingMcpTask = Boolean(retryingMcpTaskRecordId); + // The same call repeated after the app asked a person a question runs + // again: it carries the answers back, or waits again without the app. + // An explicit key may name a different request: only the same tool + // and arguments as the saved question round run again. + const retryingElicitation = recorded.replayed && accessDecision.allowed && !retryingSlackRateLimit && !retryingGitHubPublication + && !retryingMcpTaskRecordId && recorded.invocation.agentId === session.agentId + // Whatever the last attempt ended with (a failed send records its + // own error), a matching unsent round means the answer still has + // to reach the app. + && await elicitationRetryMatches(session.companyId, recorded.invocation.id, tool.name, effectiveParameters); + const retryingMcpTask = Boolean(retryingMcpTaskRecordId) || retryingElicitation; if (retryingGitHubPublication) { // Invocation hashes describe shortened audit previews. The durable // publication checks the complete saved request and current authority. diff --git a/ui/src/fixtures/mcpElicitationFixtures.ts b/ui/src/fixtures/mcpElicitationFixtures.ts new file mode 100644 index 00000000000..07d52b1245a --- /dev/null +++ b/ui/src/fixtures/mcpElicitationFixtures.ts @@ -0,0 +1,218 @@ +import { questionSetToAskUserQuestionsPayload, type AskUserQuestionsInteraction, type PaperclipQuestionSetPayload } from "@paperclipai/shared"; +import { issueThreadInteractionFixtureMeta } from "./issueThreadInteractionFixtures"; + +/* + * Question cards an MCP app's form question becomes (MCP elicitation). The + * question sets are the exact output of the server's mapper + * (server/src/services/mcp-elicitation.ts, buildElicitationQuestionSet) for + * the schemas in doc/connections/MCP-ELICITATION.md, so these stories show + * the card people really see. + */ + +/** Every spec field type: text with a default, titled single select, multi select, yes/no, integer. */ +export const mcpTicketQuestionSet: PaperclipQuestionSetPayload = { + "schema": "paperclip.question_set.v1", + "title": "Linear needs your input", + "description": "Linear needs a few details to file this ticket.", + "submitLabel": "Send to Linear", + "questions": [ + { + "id": "title", + "prompt": "Ticket title", + "required": true, + "answerMode": "text", + "initialText": "Login button does nothing", + "textValidation": { + "minLength": 3, + "maxLength": 80 + } + }, + { + "id": "priority", + "prompt": "Priority", + "required": true, + "answerMode": "single_select", + "options": [ + { + "id": "choice_1", + "label": "Urgent" + }, + { + "id": "choice_2", + "label": "Normal", + "recommended": true + }, + { + "id": "choice_3", + "label": "Later" + } + ] + }, + { + "id": "labels", + "prompt": "Labels", + "required": false, + "helpText": "Choose at most 2.", + "answerMode": "multi_select", + "options": [ + { + "id": "choice_1", + "label": "Bug", + "recommended": true + }, + { + "id": "choice_2", + "label": "Design" + }, + { + "id": "choice_3", + "label": "Speed" + } + ] + }, + { + "id": "page", + "prompt": "Page the on-call engineer?", + "required": false, + "answerMode": "single_select", + "options": [ + { + "id": "choice_1", + "label": "Yes" + }, + { + "id": "choice_2", + "label": "No", + "recommended": true + } + ] + }, + { + "id": "estimate", + "prompt": "Estimate (points)", + "required": false, + "answerMode": "text", + "textValidation": { + "inputType": "integer", + "minimum": 1, + "maximum": 8 + } + } + ] +}; + +/** A plain single-choice question. */ +export const mcpTeamQuestionSet: PaperclipQuestionSetPayload = { + "schema": "paperclip.question_set.v1", + "title": "Linear needs your input", + "description": "Which team should own this?", + "submitLabel": "Send to Linear", + "questions": [ + { + "id": "team", + "prompt": "Team", + "required": true, + "answerMode": "single_select", + "options": [ + { + "id": "choice_1", + "label": "Platform" + }, + { + "id": "choice_2", + "label": "Growth" + } + ] + } + ] +}; + +/** A form with no fields: the app asks to go ahead. */ +export const mcpConfirmQuestionSet: PaperclipQuestionSetPayload = { + "schema": "paperclip.question_set.v1", + "title": "Linear needs your input", + "description": "Create the ticket now?", + "submitLabel": "Send to Linear", + "questions": [ + { + "id": "continue", + "prompt": "Create the ticket now?", + "required": true, + "answerMode": "single_select", + "options": [ + { + "id": "continue", + "label": "Continue" + } + ] + } + ] +}; + +function mcpQuestionCard(id: string, questionSet: PaperclipQuestionSetPayload, overrides: Partial = {}): AskUserQuestionsInteraction { + return { + id, + companyId: issueThreadInteractionFixtureMeta.companyId, + issueId: issueThreadInteractionFixtureMeta.issueId, + kind: "ask_user_questions", + title: questionSet.title ?? null, + summary: questionSet.description ?? null, + status: "pending", + continuationPolicy: "wake_assignee", + createdByAgentId: "agent-codex", + createdByUserId: null, + resolvedByAgentId: null, + resolvedByUserId: null, + createdAt: new Date("2026-11-13T15:02:00.000Z"), + updatedAt: new Date("2026-11-13T15:02:00.000Z"), + resolvedAt: null, + payload: { ...questionSetToAskUserQuestionsPayload(questionSet), supersedeOnUserComment: false }, + result: null, + ...overrides, + } as AskUserQuestionsInteraction; +} + +const resolvedAt = new Date("2026-11-13T15:06:00.000Z"); + +export const mcpAskingCard = mcpQuestionCard("mcp-elicit-asking", mcpTicketQuestionSet); + +export const mcpAnsweredCard = mcpQuestionCard("mcp-elicit-answered", mcpTicketQuestionSet, { + status: "answered", + resolvedByUserId: issueThreadInteractionFixtureMeta.currentUserId, + resolvedAt, + updatedAt: resolvedAt, + result: { + version: 1, + answers: [ + { questionId: "title", optionIds: ["paperclip_text_answer"], otherText: "Login button does nothing on Safari" }, + { questionId: "priority", optionIds: ["choice_1"] }, + { questionId: "labels", optionIds: ["choice_1", "choice_3"] }, + { questionId: "page", optionIds: ["choice_2"] }, + { questionId: "estimate", optionIds: ["paperclip_text_answer"], otherText: "3" }, + ], + }, +}); + +/** Skip sends "decline" to the app. */ +export const mcpSkippedCard = mcpQuestionCard("mcp-elicit-skipped", mcpTeamQuestionSet, { + status: "cancelled", + resolvedByUserId: issueThreadInteractionFixtureMeta.currentUserId, + resolvedAt, + updatedAt: resolvedAt, + result: { version: 1, answers: [], outcome: "skipped" }, +}); + +/** Nobody answered within a day: the card is withdrawn and the app is asked again. */ +export const mcpExpiredCard = mcpQuestionCard("mcp-elicit-expired", mcpConfirmQuestionSet, { + status: "cancelled", + resolvedByAgentId: "agent-codex", + resolvedAt, + updatedAt: resolvedAt, + result: { version: 1, answers: [], outcome: "withdrawn", reason: "The app's question expired before it was answered." }, +}); + +/** Two questions in one round: each is its own card, and both answers go back together. */ +export const mcpTwoQuestionCards = [ + mcpQuestionCard("mcp-elicit-two-a", mcpTeamQuestionSet), + mcpQuestionCard("mcp-elicit-two-b", mcpConfirmQuestionSet), +]; diff --git a/ui/storybook/stories/questions-and-approvals/mcp-elicitation-question-card.stories.tsx b/ui/storybook/stories/questions-and-approvals/mcp-elicitation-question-card.stories.tsx new file mode 100644 index 00000000000..fb6ef1a84f0 --- /dev/null +++ b/ui/storybook/stories/questions-and-approvals/mcp-elicitation-question-card.stories.tsx @@ -0,0 +1,99 @@ +import { useState } from "react"; +import type { Meta, StoryObj } from "@storybook/react-vite"; +import { expect, userEvent, within } from "storybook/test"; +import type { IssueThreadInteraction } from "@paperclipai/shared"; +import { TaskChatInteractionCard } from "@/components/task-chat/TaskChatInteractionCard"; +import { issueThreadInteractionFixtureMeta } from "@/fixtures/issueThreadInteractionFixtures"; +import { + mcpAnsweredCard, + mcpAskingCard, + mcpExpiredCard, + mcpSkippedCard, + mcpTwoQuestionCards, +} from "@/fixtures/mcpElicitationFixtures"; +import { storybookAgentMap } from "../../fixtures/paperclipData"; + +/** + * An MCP app asked the person a question in the middle of a tool call (MCP + * elicitation, doc/connections/MCP-ELICITATION.md). The card names the app, + * shows its message as plain text, and only people can answer it. + */ +function ElicitationCards({ interactions, presentation }: { interactions: IssueThreadInteraction[]; presentation: "takeover" | "timeline" }) { + const [sent, setSent] = useState([]); + return ( +
+ {interactions.map((interaction) => ( + undefined} + onRejectInteraction={async () => undefined} + onSubmitInteractionAnswers={async () => { setSent((current) => [...current, interaction.id]); }} + onSubmitInteractionVerdicts={async () => undefined} + /> + ))} + {sent.length ?

Sent to the app

: null} +
+ ); +} + +const meta = { + id: "questions-and-approvals-mcp-elicitation-question-card", + title: "Tasks and conversations/Questions and approvals/MCP App Questions", + component: ElicitationCards, + parameters: { + layout: "fullscreen", + paperclip: { + "capabilities": [ + "questions-and-approvals/questions" + ], + "implementation": "production", + "boundary": "fixtures", + "notes": "Production question card with question sets produced by the server's MCP elicitation mapper. Sending answers back to the app, waking the agent, and the human-only rule are covered by server tests (server/src/__tests__/mcp-elicitation.test.ts)." + }, + }, +} satisfies Meta; + +export default meta; +type Story = StoryObj; + +/** Asking: every spec field type, with the app's defaults filled in. */ +export const Asking: Story = { + args: { interactions: [mcpAskingCard], presentation: "takeover" }, + play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + await expect(await canvas.findByText("Linear needs your input")).toBeInTheDocument(); + await expect(canvas.getByText("Linear needs a few details to file this ticket.")).toBeInTheDocument(); + // The app's default is filled in and stays editable. + await expect(canvas.getByText("Login button does nothing")).toBeInTheDocument(); + await expect(canvas.getByText("1 of 5")).toBeInTheDocument(); + }, +}; + +/** Answered: the receipt shows what went back to the app. */ +export const Answered: Story = { + args: { interactions: [mcpAnsweredCard], presentation: "timeline" }, +}; + +/** Skipped: the app was told the person declined. */ +export const Skipped: Story = { + args: { interactions: [mcpSkippedCard], presentation: "timeline" }, +}; + +/** Expired: nobody answered within a day, so the card was withdrawn and the app asks again. */ +export const Expired: Story = { + args: { interactions: [mcpExpiredCard], presentation: "timeline" }, +}; + +/** Two questions at once: each card is answered on its own; both answers go back together. */ +export const TwoQuestions: Story = { + args: { interactions: mcpTwoQuestionCards, presentation: "takeover" }, + play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + await expect(await canvas.findByText("Which team should own this?")).toBeInTheDocument(); + await userEvent.click(canvas.getByRole("radio", { name: /Platform/ })); + }, +}; From 1f388ed385804865ab28eefb4cba0ca9da80113c Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 19:47:42 -0600 Subject: [PATCH 09/11] Start follow-up runs normally when a warm sandbox's window ends during its idle close (#21504) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm sandbox keeps an agent's Runner alive between runs, so the next run on the task starts fast > - When a run ends, the server closes the idle warm Runner in-process. A separate SQL worker stops the sandbox when its warm window ends (#21410, merged today) > - Both steps start at the same moment. The sandbox stop wins, so the idle close fails. The failed owner stays registered, and the next run throws `native_instruction_materialization_handoff_unavailable` before any provider work > - Recovery must just work: a follow-up run must never fail because a predecessor's warm copy is gone > - This pull request lets the idle close finish before the sandbox stops, prepares instructions fresh when the warm copy cannot be handed off, and releases an owner whose sandbox the host already gave up > - The benefit is that automatic follow-up runs start normally after a warm window ends ## Linked Issues or Issue Description Refs #21472 (base of this stacked PR), #21410 (introduced the warm-window stop). **What happened?** In a Daytona two-server chaos wave (`w-20261113T200824-s7106`), three automatic follow-up runs failed before any provider work with `native_instruction_materialization_handoff_unavailable`: run `5f68e194`, run `d6bd5a97` and run `d1198e40`. None of them was at a release swap. In each case, the predecessor's idle close had failed first with `provider_state_unreadable` / "Daytona sandbox lease … is no longer active". The cause: the warm-window stop from #21410 stopped the sandbox at the same moment that the idle close started. For the first run's predecessor, the close was requested at 20:30:34 and the sandbox got SIGTERM at 20:30:37.8. **What did you expect?** The follow-up run prepares its instructions fresh and continues. The warm-window stop does not interrupt the host's own idle close. **Steps to reproduce** 1. Run a warm Daytona task until it ends, so its warm window starts. 2. Let the warm window end while the idle timer closes the Runner. 3. Start a follow-up run on the same task. It fails with `native_instruction_materialization_handoff_unavailable`. Two of the three tasks then ended stranded. That downstream part (the claim-gate cancellation with `execution_reconciliation_required` and the stranded notice) is owned and fixed by another session. It is **not** in this PR. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts` `claimWarmNativeInstructionCopy`: after it retires a stale warm owner whose copy cannot be adopted, it returns `null`. The caller then prepares instructions fresh. Before, it threw `native_instruction_materialization_handoff_unavailable`. - `server/src/services/warm-sandbox-expiry.ts`: the warm-window stop now waits `WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS` (2 minutes) past the window. The in-process idle close (close grace plus the 30 s proof read) finishes first. - `native-session-executor.ts` idle timer: when an idle close fails, it reads the lease rows for that sandbox. If every row is no longer `active` or `retained` (the host gave the sandbox up), it releases the in-process owner. The next run then prepares fresh. - `doc/execution-semantics.md`: one paragraph about the stop margin and fresh preparation. ## Verification Red → green (each test fails on the base `35ba76d0ef` and passes on this branch): - `native-session-executor.test.ts` › "hands off collection before probing or retiring a warm owner: unavailable" — red: throws `handoff_unavailable`; green: returns `null` (fresh preparation). - `native-session-executor.test.ts` › "forgets a warm owner whose idle close failed only once the host gave up its sandbox: pending_cleanup / released / retained" — red on all three; green: the owner is released for `pending_cleanup` and `released`, and kept (retired by the successor) for `retained`. - `warm-sandbox-expiry.test.ts` › "leaves the host's idle close of the warm runner time to finish before stopping its sandbox" — red: the sandbox stops 3 s after the window; green: it stops only after the margin. Two existing tests ("keeps a sandbox still inside its warm window, and wakes when the window ends", "reconciles a warm sandbox left running across a restart") now expect the deadline with the margin. Local runs on this branch: - `server`: `tsc --noEmit` clean. - `vitest run src/__tests__/warm-sandbox-expiry.test.ts src/services/heartbeat/instruction-working-copy.test.ts`: 24/24 passed. - `vitest run src/services/native-runtime/native-session-executor.test.ts -t "hands off collection before probing or retiring a warm owner|forgets a warm owner whose idle close failed"`: 10/10 passed. Not run: a Daytona wave with this change. ## Risks - Warm sandboxes now stop up to 2 minutes later than the configured window. This is a small cost increase per warm sandbox. I chose a fixed margin over an in-process "is a close running?" check because the stop is pure SQL. The margin works when two servers run at the same time (the server that runs the stop may not be the server that owns the Runner), and it needs no busy-loop or cross-server signal. - The margin does not cover a re-armed idle timer or a close that takes longer than 2 minutes. Fresh preparation (fix 1) and the owner release (fix 3) cover those cases. - Not run on Daytona. The evidence is from the wave above and the unit tests. - In the wave, a `restore_failed` copy-back onto the just-restarted sandbox was also seen. I did not find its root cause. It is not changed here. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test runs) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/execution-semantics.md | 8 +++ .../src/__tests__/warm-sandbox-expiry.test.ts | 30 +++++++++--- .../native-session-executor.test.ts | 49 ++++++++++++++++++- .../native-runtime/native-session-executor.ts | 24 ++++++++- server/src/services/warm-sandbox-expiry.ts | 4 +- 5 files changed, 103 insertions(+), 12 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 03909f22e00..c5801b8d8ec 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1639,6 +1639,14 @@ the configured interval until the provider confirms the new one. An explicit `0` (auto-stop disabled), a longer configured interval, per-turn, and non-reusable sandboxes are unchanged by the warm floor, and idle sandboxes still auto-stop. +The host's own warm-window stop of a retained sandbox waits 2 minutes past the +warm window, so the host's idle close of the warm runner (suspend and its proof +read) finishes before the sandbox stops. When an idle close still fails and every +lease row for that sandbox is no longer active or retained, the host has given the +sandbox up: the warm owner is released and the next run on the task prepares its +instructions fresh. A follow-up whose predecessor's warm instruction copy cannot be +handed off retires that owner and prepares fresh instead of failing. + A remote runner also keeps working while its controller is offline, and the host reattaches it within the remote runner reconnect grace (7 minutes). Daytona counts only SDK and lifecycle calls as activity, so with no controller its idle auto-stop diff --git a/server/src/__tests__/warm-sandbox-expiry.test.ts b/server/src/__tests__/warm-sandbox-expiry.test.ts index 0100bc54c37..0097287b743 100644 --- a/server/src/__tests__/warm-sandbox-expiry.test.ts +++ b/server/src/__tests__/warm-sandbox-expiry.test.ts @@ -8,6 +8,7 @@ import { heartbeatService, type HeartbeatEnvironmentRuntime } from "../services/ import { nextWarmSandboxStopAt, stopWarmSandboxesPastWindow, + WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS, WARM_SANDBOX_IDLE_TIMEOUT_KEY, warmSandboxWindowMetadata, } from "../services/warm-sandbox-expiry.js"; @@ -81,14 +82,14 @@ suite("warm sandbox expiry", () => { expect(await stopWarmSandboxesPastWindow(db)).toEqual({ stopping: 0 }); expect(await row(leaseId)).toEqual(before); - const windowEnd = before.lastUsedAt.getTime() + 5 * MINUTE; + const windowEnd = before.lastUsedAt.getTime() + 5 * MINUTE + WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS; expect(await nextWarmSandboxStopAt(db)).toBe(windowEnd); // The cleanup worker sleeps until exactly that deadline: no interval scan. expect(await nextEnvironmentCleanupAt(db)).toBe(windowEnd); }); it("stops a sandbox once its warm window has passed and keeps it for the next turn", async () => { - const leaseId = await warmRetained({ releasedAgoMs: 6 * MINUTE, windowMs: 5 * MINUTE }); + const leaseId = await warmRetained({ releasedAgoMs: 8 * MINUTE, windowMs: 5 * MINUTE }); expect(await stopWarmSandboxesPastWindow(db)).toEqual({ stopping: 1 }); const stopping = await row(leaseId); @@ -99,8 +100,21 @@ suite("warm sandbox expiry", () => { expect(await nextWarmSandboxStopAt(db)).toBeNull(); }); + it("leaves the host's idle close of the warm runner time to finish before stopping its sandbox", async () => { + const closing = await warmRetained({ releasedAgoMs: 5 * MINUTE + 3_000, windowMs: 5 * MINUTE }); + const before = await row(closing); + + expect(await stopWarmSandboxesPastWindow(db)).toEqual({ stopping: 0 }); + expect(await row(closing)).toEqual(before); + expect(await nextWarmSandboxStopAt(db)).toBeGreaterThanOrEqual(before.lastUsedAt.getTime() + 5 * MINUTE + 90_000); + + await db.update(environmentLeases).set({ lastUsedAt: new Date(Date.now() - 5 * MINUTE - WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS - 1_000) }) + .where(eq(environmentLeases.id, closing)); + expect(await stopWarmSandboxesPastWindow(db)).toEqual({ stopping: 1 }); + }); + it("stops the sandbox at the provider through the cleanup sweep", async () => { - const leaseId = await warmRetained({ releasedAgoMs: 6 * MINUTE, windowMs: 5 * MINUTE }); + const leaseId = await warmRetained({ releasedAgoMs: 8 * MINUTE, windowMs: 5 * MINUTE }); const teardown = vi.fn(async ({ lease }: { lease: { providerLeaseId: string | null } }) => ({ providerLeaseId: lease.providerLeaseId, state: "stopped" })); @@ -118,7 +132,7 @@ suite("warm sandbox expiry", () => { it("keeps a sandbox past its window while a live run holds it", async () => { const providerLeaseId = `sandbox://fake/${randomUUID()}`; - const leaseId = await warmRetained({ releasedAgoMs: 6 * MINUTE, providerLeaseId }); + const leaseId = await warmRetained({ releasedAgoMs: 8 * MINUTE, providerLeaseId }); const liveRun = await run("running"); await db.insert(environmentLeases).values({ companyId, environmentId, issueId, heartbeatRunId: liveRun, status: "active", leasePolicy: "reuse_by_environment", provider: "fake", providerLeaseId, @@ -132,7 +146,7 @@ suite("warm sandbox expiry", () => { }); it("keeps a sandbox past its window while a run on the task is starting", async () => { - const leaseId = await warmRetained({ releasedAgoMs: 6 * MINUTE }); + const leaseId = await warmRetained({ releasedAgoMs: 8 * MINUTE }); await run("running"); const before = await row(leaseId); @@ -141,7 +155,7 @@ suite("warm sandbox expiry", () => { }); it("keeps a sandbox whose own run has not ended", async () => { - const leaseId = await warmRetained({ releasedAgoMs: 6 * MINUTE, runStatus: "running" }); + const leaseId = await warmRetained({ releasedAgoMs: 8 * MINUTE, runStatus: "running" }); expect(await stopWarmSandboxesPastWindow(db)).toEqual({ stopping: 0 }); expect((await row(leaseId)).status).toBe("retained"); @@ -150,7 +164,7 @@ suite("warm sandbox expiry", () => { it("leaves a plugin Runner task lease to its own cleanup", async () => { // Its release path does not use the stop-only contract, so expiry must not // hand it over: the plugin could free the allocation and the task's files. - const leaseId = await warmRetained({ releasedAgoMs: 6 * MINUTE }); + const leaseId = await warmRetained({ releasedAgoMs: 8 * MINUTE }); await db.update(environmentLeases).set({ metadata: { driver: "plugin", pluginId: "fake-plugin", runnerTaskLease: true, [WARM_SANDBOX_IDLE_TIMEOUT_KEY]: 5 * MINUTE } }).where(eq(environmentLeases.id, leaseId)); const before = await row(leaseId); @@ -178,7 +192,7 @@ suite("warm sandbox expiry", () => { expect(await row(leaked)).toMatchObject({ status: "released", cleanupStatus: "success" }); expect((await row(recent)).status).toBe("retained"); const recentRow = await row(recent); - expect(await nextWarmSandboxStopAt(db)).toBe(recentRow.lastUsedAt.getTime() + 24 * HOUR); + expect(await nextWarmSandboxStopAt(db)).toBe(recentRow.lastUsedAt.getTime() + 24 * HOUR + WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS); }); it("records the run's warm window on the lease only for a warm run", () => { diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 7ddf5de47d2..c08975a60e1 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -11503,6 +11503,51 @@ describe("native warm session supervision", () => { } }); + it.each([ + { lease: "pending_cleanup", forgotten: true }, + { lease: "released", forgotten: true }, + { lease: "retained", forgotten: false }, + ])("forgets a warm owner whose idle close failed only once the host gave up its sandbox: $lease", async ({ lease, forgotten }) => { + const identity = `warm-idle-close-sandbox-${lease}`; + const warmExecution = { ...execution, binding: { ...execution.binding, runId: identity, executionWorkspaceId: `${identity}-workspace` }, + session: { ...execution.session, normalizedSessionId: identity, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 50 } }, + } as NativeExecutionInputV1; + let allowClose = false; + const session = { close: vi.fn(async () => { if (!allowClose) throw new Error("runner_harness_state_mismatch: provider_state_unreadable"); }) }; + state.execute.mockReset().mockImplementation(async (options) => { + await options.onSession?.(session); + return { result: { summary: "completed" }, terminal: { runTerminalState: "succeeded" }, + turnId: identity, normalizedSessionId: identity, providerSessionId: identity, + driverKind: "test", driverVersion: "1", nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null }; + }); + const copy = (runId: string) => ({ runId, root: `/tmp/${identity}`, preparationKey: "same", hasChanges: async () => false, + collectStopped: async () => undefined, retirementFailed: async () => undefined }); + let leaseRows: Array<{ status: string }> = []; + const base = leaseDb(warmExecution); + const db = { ...base, select: (fields?: unknown) => ({ from: (table: unknown) => table === environmentLeases + ? { where: async () => leaseRows } : base.select(fields as never).from(table as never) }) } as unknown as Db; + try { + await executePaperclipNativeSession({ db, execution: warmExecution, runnerInstanceId: identity, + runnerExecutionTarget: { kind: "remote", transport: "sandbox", environmentId: identity, remoteCwd: `/tmp/${identity}`, + sandboxLeaseAcquisition: { providerLeaseId: `${identity}-sandbox` } } as never, + instructionWorkingCopy: copy(identity) }); + leaseRows = [{ status: lease }]; + await vi.waitFor(() => expect(session.close).toHaveBeenCalledTimes(1)); + const followUpRunId = `${identity}-follow-up`; + const adopt = vi.fn(async () => null); + allowClose = true; + await expect(claimWarmNativeInstructionCopy({ priorExecution: warmExecution, + companyId: warmExecution.binding.companyId, agentId: warmExecution.binding.agentId, + executionWorkspaceId: warmExecution.binding.executionWorkspaceId, workspace: warmExecution.workspace, + environmentId: identity, runId: followUpRunId, preparationKey: "same", adopt })).resolves.toBeNull(); + expect(adopt).toHaveBeenCalledTimes(forgotten ? 0 : 1); + expect(session.close).toHaveBeenCalledTimes(forgotten ? 1 : 2); + } finally { + allowClose = true; + await closeWarmNativeSessionsForEnvironment({ environmentId: identity, reason: "test cleanup" }); + } + }); + it("admits the follow-up's execution only after the previous owner's final cleanup hands its scopes back", async () => { const identity = "warm-release-scope-handback"; const warmExecution = { ...execution, binding: { ...execution.binding, runId: identity, executionWorkspaceId: `${identity}-workspace` }, @@ -11895,8 +11940,8 @@ describe("native warm session supervision", () => { const claiming = claimWarmNativeInstructionCopy({ priorExecution: first, companyId: first.binding.companyId, agentId: first.binding.agentId, executionWorkspaceId: first.binding.executionWorkspaceId, workspace: first.workspace, environmentId: identity, runId: current.runId, preparationKey: current.preparationKey, forceRetirement: scenario === "reset", adopt }); - if (["unavailable", "auth", "wrong-capability", "retirement-failure"].includes(scenario)) { - await expect(claiming).rejects.toThrow(({ unavailable: "handoff_unavailable", auth: "authorization denied", + if (["auth", "wrong-capability", "retirement-failure"].includes(scenario)) { + await expect(claiming).rejects.toThrow(({ auth: "authorization denied", "wrong-capability": "handoff_mismatch", "retirement-failure": "stop unconfirmed" } as Record)[scenario]); } else expect(await claiming).toBeNull(); expect(priorCopy.hasChanges).not.toHaveBeenCalled(); diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 279c8c81ed0..fe7843e2a5d 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -716,6 +716,7 @@ type WarmNativeSession = { pluginTaskId?: string; /** The sandbox that holds this owner's Runner, when it runs in one. */ providerLeaseId?: string; + sandboxLeaseDb?: Db; /** A successor's retirement of this owner failed; it stays registered for its own sandbox. */ retirementUnproven?: boolean; observabilityCompatibilityKey?: string; @@ -1002,7 +1003,7 @@ export async function claimWarmNativeInstructionCopy(input: { const adopted = await input.adopt(entry.instructionWorkingCopy.runId); if (!adopted) { await retire("warm instruction materialization cannot be handed off"); - throw new Error("native_instruction_materialization_handoff_unavailable"); + return null; } if (adopted.runId !== input.runId || adopted.preparationKey !== input.preparationKey) throw new Error("native_instruction_materialization_handoff_mismatch"); entry.instructionWorkingCopy = adopted; @@ -1088,6 +1089,16 @@ export async function closeWarmNativeSessionsForEnvironment(input: { return closeIdleWarmNativeSessions(input); } +async function warmSandboxGivenUpByHost(entry: WarmNativeSession): Promise { + if (!entry.sandboxLeaseDb || !entry.providerLeaseId || !entry.environmentId) return false; + const leases = await entry.sandboxLeaseDb.select({ status: environmentLeases.status }).from(environmentLeases).where(and( + eq(environmentLeases.companyId, entry.companyId), + eq(environmentLeases.environmentId, entry.environmentId), + eq(environmentLeases.providerLeaseId, entry.providerLeaseId), + )); + return leases.length > 0 && leases.every(lease => lease.status !== "active" && lease.status !== "retained"); +} + async function warmSandboxDestroyed(db: Db, entry: WarmNativeSession): Promise { if (!entry.providerLeaseId || !entry.environmentId) return false; const leases = await db.select().from(environmentLeases).where(and( @@ -8777,6 +8788,16 @@ function armWarmNativeSessionIdleTimer( logger.warn({ runId: current.runId, issueId: current.issueId, environmentId: current.environmentId, providerLeaseId: current.providerLeaseId ?? null, closeFailure: describeWarmCloseFailure(error) }, released ? "idle warm native session detach failed" : "idle warm native session close failed"); + if (released) return; + return warmSandboxGivenUpByHost(current).then(givenUp => { + if (!givenUp) return; + logger.warn({ runId: current.runId, issueId: current.issueId, environmentId: current.environmentId, + providerLeaseId: current.providerLeaseId ?? null }, + "warm native session's sandbox was stopped by the host; released its owner"); + forgetWarmNativeSession(sessionId, current); + }, proofError => { + logger.warn({ err: proofError, runId: current.runId }, "could not read the warm session's sandbox lease"); + }); }) .finally(() => { current.busy = false; @@ -12602,6 +12623,7 @@ async function executePaperclipNativeSessionAttempt( pluginTaskId: usesPluginTask && input.runnerExecutionTarget?.kind === "remote" && input.runnerExecutionTarget.transport === "sandbox" ? input.runnerExecutionTarget.sandboxLeaseAcquisition?.providerLeaseId : undefined, providerLeaseId: sandboxProviderLeaseId(input.runnerExecutionTarget) ?? undefined, + sandboxLeaseDb: input.db, runId: input.execution.binding.runId, issueId: input.execution.binding.issueId, instructionWorkingCopy: input.instructionWorkingCopy?.checkpointWarm ? undefined : input.instructionWorkingCopy, diff --git a/server/src/services/warm-sandbox-expiry.ts b/server/src/services/warm-sandbox-expiry.ts index 2f4ab6171d5..dcbfb598c18 100644 --- a/server/src/services/warm-sandbox-expiry.ts +++ b/server/src/services/warm-sandbox-expiry.ts @@ -40,6 +40,8 @@ export const WARM_SANDBOX_IDLE_TIMEOUT_KEY = "warmSandboxIdleTimeoutMs"; /** Marker on the stop-and-retain intent: the warm window ended. */ export const WARM_WINDOW_EXPIRED_KEY = "warmWindowExpired"; +export const WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS = 2 * 60_000; + const STOP_PAGE_SIZE = 25; function positiveWindowSql(value: ReturnType) { @@ -52,7 +54,7 @@ export function warmWindowEndsAtSql() { const window = sql`least(coalesce( ${positiveWindowSql(sql`${environmentLeases.metadata}->${WARM_SANDBOX_IDLE_TIMEOUT_KEY}`)}, ${PAPERCLIP_RUNNER_IDLE_TIMEOUT_MAX_MS}), ${PAPERCLIP_RUNNER_IDLE_TIMEOUT_MAX_MS})`; - return sql`(extract(epoch from ${environmentLeases.lastUsedAt}) * 1000 + ${window})`; + return sql`(extract(epoch from ${environmentLeases.lastUsedAt}) * 1000 + ${window} + ${WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS})`; } /** From 81d06cfc97a6f9fb9726fc4443f6278bc2e982e2 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 19:47:58 -0600 Subject: [PATCH 10/11] feat(mcp-tasks): add a demo MCP server that uses Tasks, test against it, and keep audit records out of agent errors (#21464) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents call tools in connected apps through the tool gateway, which speaks MCP to each app > - MCP Tasks (#21263) lets an app answer a long tool call with a durable task. It shipped behind the experimental setting `enableMcpTasks`, off by default > - No public app returns tasks without credentials, so nobody had seen the feature work against a real server, and it was hard to understand > - This pull request adds a small demo app that uses Tasks, tests the gateway against it over HTTP, and records a live end-to-end run > - The live run found one bug: every task stop sent the call's audit record, with a server stack trace, to the agent. This pull request fixes it > - The benefit is a repeatable way to try and test Tasks, plain-language docs, and evidence for the decision to turn the setting on ## Linked Issues or Issue Description Refs #21389 Refs #21263 ## What Changed - Add an MCP Tasks demo server to `packages/kv-demo-mcp-server` (new bin `paperclip-tasks-demo-mcp-server`; run `node packages/kv-demo-mcp-server/dist/tasks/main.js` after a build). It is stateless MCP 2026-07-28 with `server/discover`, and it declares `io.modelcontextprotocol/tasks`. It has six tools: `slow_report`, `needs_approval` (form question), `will_fail`, `cancellable`, `expires`, and `quick_echo`. It saves task state to a file, so it can restart mid-task. It needs no credentials. It has a status page at `/` and JSON at `/api/state`. - I put it in the existing demo package, so there is no new workspace package and no `pnpm-lock.yaml` change. - Add `server/src/__tests__/mcp-tasks-demo-server.test.ts`. It runs the real gateway against the real demo server over HTTP, with one test per tool plus a restart. - Fix: `server/src/routes/tool-gateway.ts` no longer sends `details.execution` to callers. That record has request IDs and the server-side error stack. The audit still keeps it. Add `server/src/__tests__/tool-gateway-agent-errors.test.ts` (red before the fix, green after). - Docs: `doc/connections/MCP-TASKS.md` now has "In plain words", "What a person sees", "Try it yourself", the live-run results, and a recommendation. The package README documents the demo server. - `enableMcpTasks` stays off by default. ## Verification - `packages/kv-demo-mcp-server`: `npx vitest run` gives 25/25 pass (13 new demo-server tests). `tsc -p tsconfig.test.json` is clean. - `server`: `mcp-tasks-demo-server.test.ts` gives 8/8 pass. `tool-gateway-agent-errors.test.ts` gives 2/2 pass, and it failed 2/2 before the fix. - Existing suites that cover the changed route and Tasks: `mcp-tasks.test.ts` and `tool-gateway-discovery-http.test.ts` (results in a comment below). - `pnpm verify` exits 2, because the doc is a global CI input. Required CI gives the broad coverage. - Live end-to-end run on a local `test-drive` instance with `enableMcpTasks` on. The agent was Codex `gpt-6-luna` at low effort. The demo was connected through "Connect your own MCP server": | Behaviour | Result | `tools/call` count at the app | | --- | --- | --- | | Polling (`slow_report`, 45 s) | Delivered after 22 polls | 1 | | Question (`needs_approval`) | The card was answered in the UI. `tasks/update` was sent, and the task finished | 1 | | Failure (`will_fail`) | `provider_task_failed` with the app's message | 1 | | Cancel (`cancellable`) | Stop on the run sent `tasks/cancel` | 1 | | Expiry (`expires`, 15 s) | `provider_task_expired`, no hang | 1 | | Restart (`slow_report`, 120 s) | The server was down for 2 min 23 s. The result was delivered once, and the agent was woken once | 1 | After the run, all 6 `tool_mcp_tasks` rows were settled and none had a next poll. ## Risks - Low risk. The demo server is new code in a demo package, and the bin is additive. - The route change removes `execution` from tool-gateway error responses. No UI code or test reads that field from these responses. The audit and tool-call events still record it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/connections/MCP-TASKS.md | 118 ++++++++ packages/kv-demo-mcp-server/README.md | 50 ++++ packages/kv-demo-mcp-server/package.json | 6 +- packages/kv-demo-mcp-server/src/index.ts | 4 + .../kv-demo-mcp-server/src/tasks/http.test.ts | 220 +++++++++++++++ packages/kv-demo-mcp-server/src/tasks/http.ts | 257 ++++++++++++++++++ packages/kv-demo-mcp-server/src/tasks/main.ts | 34 +++ .../kv-demo-mcp-server/src/tasks/render.ts | 64 +++++ .../kv-demo-mcp-server/src/tasks/store.ts | 157 +++++++++++ .../kv-demo-mcp-server/src/tasks/tools.ts | 207 ++++++++++++++ .../__tests__/mcp-tasks-demo-server.test.ts | 197 ++++++++++++++ .../tool-gateway-agent-errors.test.ts | 51 ++++ server/src/routes/tool-gateway.ts | 14 +- 13 files changed, 1375 insertions(+), 4 deletions(-) create mode 100644 packages/kv-demo-mcp-server/src/tasks/http.test.ts create mode 100644 packages/kv-demo-mcp-server/src/tasks/http.ts create mode 100644 packages/kv-demo-mcp-server/src/tasks/main.ts create mode 100644 packages/kv-demo-mcp-server/src/tasks/render.ts create mode 100644 packages/kv-demo-mcp-server/src/tasks/store.ts create mode 100644 packages/kv-demo-mcp-server/src/tasks/tools.ts create mode 100644 server/src/__tests__/mcp-tasks-demo-server.test.ts create mode 100644 server/src/__tests__/tool-gateway-agent-errors.test.ts diff --git a/doc/connections/MCP-TASKS.md b/doc/connections/MCP-TASKS.md index bf2219916bc..bc1e4da5148 100644 --- a/doc/connections/MCP-TASKS.md +++ b/doc/connections/MCP-TASKS.md @@ -7,6 +7,65 @@ hands the agent the final result exactly as a quick call would return it. If the call outlives its timeout, or Paperclip restarts or deploys mid-call, the work is not lost and is never started a second time. +## In plain words + +Without Tasks, an agent that asks an app to do something slow has to hold the +line until the app finishes. If the wait is too long, or Paperclip restarts, +the answer is lost, and calling again may start the work a second time. + +With Tasks, the app answers straight away with a ticket ("task") and does the +work in the background. Paperclip keeps the ticket and checks on it every few +seconds. When the work is done, the agent gets the result as if the call had +simply been slow. If Paperclip restarts, it picks the ticket up again. It never +asks the app to start the same work twice. + +A task can also stop and ask a question ("approve this deploy?"). Paperclip +turns that into a question card on the task, and the person's answer goes back +to the app. + +### What a person sees + +| Situation | In the task chat | +| --- | --- | +| A long action is running | The agent's turn shows the tool as running. The agent keeps waiting, or is told to call again later. | +| The app asks a question | A **Tool needs input** card with the app's question. Answering it wakes the agent, which carries on. | +| The app reports a failure | The agent receives a tool error with the app's message. | +| Someone presses **Stop** on the run | Paperclip tells the app to cancel the work. | +| The app no longer has the task | The agent is told the result is gone and to run the action again. | +| Paperclip restarts mid-action | Nothing to do. The run resumes and the result arrives once. | + +### Try it yourself + +The repository includes a demo app that uses Tasks. It needs no accounts or +keys, and its "work" is a timer. The steps take about 10 minutes. + +1. Start the demo app: + + ```sh + pnpm --filter @paperclipai/kv-demo-mcp-server build + node packages/kv-demo-mcp-server/dist/tasks/main.js + ``` + + Open `http://127.0.0.1:8849/` to watch its tasks live. +2. In Paperclip, open **Instance settings → Experimental** and turn on + **Durable app actions (MCP Tasks)**. +3. Open **Apps → Connect your own MCP server**, paste + `http://127.0.0.1:8849/mcp`, and choose **Check link**. Six actions appear. +4. Give an agent a task such as: *"Call the connected app tool `slow_report` + with `{"topic": "uptime", "seconds": 45}`. If it says the app is still + working, call it again with the same arguments. Post the report."* The + agent must be one that receives app tools: a Paperclip runner agent or a + Codex agent. Plain local OpenCode and Claude agents do not get app tools + yet. +5. Try the other tools the same way: `needs_approval` (answer the card), + `will_fail`, `cancellable` (press **Stop** while it runs), and `expires`. +6. To see a restart, start `slow_report` with `"seconds": 120`, stop the + Paperclip server, and start it again. The report still arrives once. On the + demo's status page, `slow_report` was started one time. + +The package [README](../../packages/kv-demo-mcp-server/README.md) lists the +demo's options. + ## Turning it on MCP Tasks are behind the experimental setting **Durable app actions (MCP @@ -121,6 +180,60 @@ returns to approved, and its next execution attaches to the same task. - Showing a long-running provider action's progress (`statusMessage`) in the task chat. +## Live check with the demo app (2026-11-13) + +One local Paperclip (`test-drive`, embedded database, `enableMcpTasks` on), the +demo app connected through **Connect your own MCP server**, and a Codex agent +on a small model. Each behaviour was one task given to the agent in plain +words. + +| Behaviour | Result | Demo app `tools/call` count | Stored task row | +| --- | --- | --- | --- | +| Polling (`slow_report`, 45 s) | Report delivered; 22 `tasks/get` at the app's 2 s interval | 1 | `completed`, delivered | +| Question (`needs_approval`) | Card shown; the person's answer woke the agent; `tasks/update` sent; result delivered | 1 | `completed`, delivered | +| Failure (`will_fail`) | Tool error with the app's message (`provider_task_failed`) | 1 | `failed` | +| Cancel (`cancellable`) | **Stop** on the run sent `tasks/cancel`; the app stopped the work | 1 | `cancelled` (`run_cancelled`) | +| Expiry (`expires`, 15 s TTL) | Clear "run the action again" error (`provider_task_expired`), no hang | 1 | `expired` (`ttl_expired`) | +| Restart (`slow_report`, 120 s) | Server killed 6 s into the task and down for 2 min 23 s. On start it polled the stored task, woke the agent once, and the new run collected the report | 1 | `completed`, delivered, woken once | + +Afterwards no task row had a next poll, so background polling stopped. + +The run found one bug, fixed with the demo: every task stop (still working, +expired, failed, cancelled) returned the call's audit record to the agent, +including the server-side stack trace and request IDs. The agent pasted it into +the task. Agent-facing errors now leave that record out; the audit keeps it. + +What the run also showed, outside Tasks itself: + +- Only Paperclip runner and Codex agents receive app tools. A plain local + OpenCode agent did not see the demo's tools and called the app directly with + `curl`, bypassing Paperclip. +- The running-tool row and the agent's error text show the internal tool name + of a custom connection, including the connection ID + (`mcp.app-gallery-link--:slow-report`). +- Pressing **Stop** shows "Couldn't finish this attempt", which reads as a + failure. +- The running row shows no progress for a long action (`statusMessage` is not + displayed; listed under Follow-ups). + +## Is it ready to be on by default? + +**Not yet.** The demo showed every behaviour working with a real HTTP server, +including a restart, with each action started exactly once. Three of the four +findings in [#21389](https://github.com/paperclipai/paperclip/issues/21389) +still stand, and the demo does not exercise them: + +1. Turning the setting off and on again leaves stored tasks idle until the + next unrelated wake. +2. A cloud instance that turns it on through managed configuration does not + count pending tasks as an idle-sleep blocker. +3. A failed settings read counts as "off". A retry can then skip its stored + task and send `tools/call` again, which breaks the "never started twice" + promise. + +Fix those three and turn it on by default. Validating question-card answers +(the fourth finding) and showing progress can follow. + ## Tests `server/src/__tests__/mcp-tasks.test.ts` runs a fake 2026-07-28 server that @@ -129,3 +242,8 @@ attaching call, a keyed retry, a restart mid-task (fresh gateway, no second `tools/call`, one wake, one delivery), URL and form input, cancellation by the run and by closing the task, failure, an unknown task, TTL expiry, another agent never attaching, and synchronous calls left unchanged. + +`server/src/__tests__/mcp-tasks-demo-server.test.ts` runs the same gateway +against the real demo app (`packages/kv-demo-mcp-server/src/tasks`) over HTTP, +one test per demo tool, plus a restart. The demo's own tests are in +`packages/kv-demo-mcp-server/src/tasks/http.test.ts`. diff --git a/packages/kv-demo-mcp-server/README.md b/packages/kv-demo-mcp-server/README.md index 7934bd25b4c..ea1daaa90e1 100644 --- a/packages/kv-demo-mcp-server/README.md +++ b/packages/kv-demo-mcp-server/README.md @@ -205,3 +205,53 @@ records stay intact unless you also archive the connection. The KV demo is a fixture, not a piece of infrastructure. Treat each session as disposable: start it, run the demo, kill it. + +## MCP Tasks demo server + +The same package ships a second server, `paperclip-tasks-demo-mcp-server`, for +trying [MCP Tasks](../../doc/connections/MCP-TASKS.md): long tool calls that +return a task, which the client polls until it finishes. It speaks the +stateless MCP revision 2026-07-28 (`server/discover`, no session) and declares +the `io.modelcontextprotocol/tasks` extension. It needs no credentials and +calls nothing outside its own process: the "work" is a timer. + +| Tool | What it shows | +| --- | --- | +| `slow_report` | Returns a task, then a report after `seconds` (default 20). Polling. | +| `needs_approval` | Asks a form question (approve or reject); finishes after the answer arrives with `tasks/update`. | +| `will_fail` | Fails after `seconds` (default 5). A failed task. | +| `cancellable` | Runs for `seconds` (default 600) and stops on `tasks/cancel`. | +| `expires` | Its TTL (`ttlSeconds`, default 15) passes before it finishes, so the server forgets it. | +| `quick_echo` | Answers at once, without a task. | + +It returns a task only to a request that declares the extension. Without it, +the long tools answer with an error that says the client does not support +MCP Tasks. + +Start it from the repo root: + +```sh +pnpm --filter @paperclipai/kv-demo-mcp-server build +node packages/kv-demo-mcp-server/dist/tasks/main.js +``` + +It listens on `http://127.0.0.1:8849`: + +- **MCP endpoint**: `POST /mcp` +- **Status page**: `GET /`, which lists every task, how many times each tool was + started, and the latest protocol events. It reloads every 2 seconds. +- **JSON state**: `GET /api/state`, the same data for scripts. + +Task state is saved to a file (by default `~/.paperclip-tasks-demo/state.json`), +so you can stop and start the server mid-task. A task's progress is worked out +from the clock, so a restart neither resets nor repeats it. + +| Variable | Default | Purpose | +| --- | --- | --- | +| `TASKS_DEMO_PORT` (or `PORT`) | `8849` | Port to listen on. | +| `TASKS_DEMO_HOST` | `127.0.0.1` | Interface to bind. | +| `TASKS_DEMO_STATE_FILE` | `~/.paperclip-tasks-demo/state.json` | Where task state is saved. Its folder must be yours and not writable by others. `memory` keeps it in memory only. | +| `TASKS_DEMO_TOKEN` | unset | When set, `/mcp` and `/api/state` require `Authorization: Bearer `. | + +See "Try it yourself" in [MCP-TASKS.md](../../doc/connections/MCP-TASKS.md) for +the full walkthrough with Paperclip. diff --git a/packages/kv-demo-mcp-server/package.json b/packages/kv-demo-mcp-server/package.json index 2a39a1c6b89..f7a28baa5c3 100644 --- a/packages/kv-demo-mcp-server/package.json +++ b/packages/kv-demo-mcp-server/package.json @@ -13,7 +13,8 @@ }, "type": "module", "bin": { - "paperclip-kv-demo-mcp-server": "./dist/main.js" + "paperclip-kv-demo-mcp-server": "./dist/main.js", + "paperclip-tasks-demo-mcp-server": "./dist/tasks/main.js" }, "exports": { ".": "./src/index.ts" @@ -21,7 +22,8 @@ "publishConfig": { "access": "public", "bin": { - "paperclip-kv-demo-mcp-server": "./dist/main.js" + "paperclip-kv-demo-mcp-server": "./dist/main.js", + "paperclip-tasks-demo-mcp-server": "./dist/tasks/main.js" }, "exports": { ".": { diff --git a/packages/kv-demo-mcp-server/src/index.ts b/packages/kv-demo-mcp-server/src/index.ts index 98710d03880..a67a5fb8ed9 100644 --- a/packages/kv-demo-mcp-server/src/index.ts +++ b/packages/kv-demo-mcp-server/src/index.ts @@ -40,3 +40,7 @@ export { createKvDemoHttpServer } from "./http.js"; export type { KvDemoHttpServer, KvDemoHttpOptions } from "./http.js"; export { readConfigFromEnv } from "./config.js"; export type { KvDemoConfig } from "./config.js"; +export { createTasksDemoHttpServer, handleTasksDemoRpc, PROTOCOL_VERSION as TASKS_DEMO_PROTOCOL_VERSION, TASKS_EXTENSION_ID } from "./tasks/http.js"; +export type { TasksDemoHttpServer, TasksDemoHttpOptions } from "./tasks/http.js"; +export { DemoTaskStore } from "./tasks/store.js"; +export type { DemoTask, DemoTaskStatus, DemoState } from "./tasks/store.js"; diff --git a/packages/kv-demo-mcp-server/src/tasks/http.test.ts b/packages/kv-demo-mcp-server/src/tasks/http.test.ts new file mode 100644 index 00000000000..c66293d4b79 --- /dev/null +++ b/packages/kv-demo-mcp-server/src/tasks/http.test.ts @@ -0,0 +1,220 @@ +import { chmodSync, mkdtempSync, readFileSync, rmSync, symlinkSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import { PROTOCOL_VERSION, TASKS_EXTENSION_ID, createTasksDemoHttpServer, type TasksDemoHttpServer } from "./http.js"; +import { DemoTaskStore } from "./store.js"; + +let clock = Date.parse("2026-11-13T12:00:00Z"); +const now = () => clock; +const servers: TasksDemoHttpServer[] = []; +const dirs: string[] = []; + +afterEach(async () => { + await Promise.all(servers.splice(0).map((server) => server.close())); + for (const dir of dirs.splice(0)) rmSync(dir, { recursive: true, force: true }); +}); + +async function start(store = new DemoTaskStore({ now })) { + const server = createTasksDemoHttpServer({ store }); + servers.push(server); + const port = await server.listen(0); + return { server, url: `http://127.0.0.1:${port}` }; +} + +/** Send one request the way Paperclip's stateless client does. */ +async function rpc(url: string, method: string, params: Record = {}, options: { tasks?: boolean; name?: string; version?: string } = {}) { + const name = options.name ?? (typeof params.taskId === "string" ? params.taskId : typeof params.name === "string" ? params.name : undefined); + const response = await fetch(`${url}/mcp`, { + method: "POST", + headers: { + "content-type": "application/json", + "mcp-protocol-version": options.version ?? PROTOCOL_VERSION, + "mcp-method": method, + ...(name !== undefined ? { "mcp-name": name } : {}), + }, + body: JSON.stringify({ + jsonrpc: "2.0", id: 1, method, + params: { + ...params, + _meta: { + "io.modelcontextprotocol/protocolVersion": options.version ?? PROTOCOL_VERSION, + "io.modelcontextprotocol/clientCapabilities": { extensions: options.tasks === false ? {} : { [TASKS_EXTENSION_ID]: {} } }, + }, + }, + }), + }); + return { status: response.status, body: await response.json() as { result?: Record; error?: { code: number; message: string; data?: any } } }; +} + +const call = (url: string, name: string, args: Record = {}, tasks = true) => rpc(url, "tools/call", { name, arguments: args }, { tasks }); +const get = (url: string, taskId: string) => rpc(url, "tasks/get", { taskId }); + +describe("MCP Tasks demo server", () => { + it("advertises the Tasks extension on a stateless server and refuses the old handshake", async () => { + const { url } = await start(); + const discover = await rpc(url, "server/discover"); + expect(discover.body.result).toMatchObject({ resultType: "complete", supportedVersions: [PROTOCOL_VERSION], capabilities: { extensions: { [TASKS_EXTENSION_ID]: {} } } }); + const legacy = await rpc(url, "initialize", {}, { version: "2025-11-25" }); + expect(legacy.status).toBe(400); + expect(legacy.body.error).toMatchObject({ code: -32022, data: { supported: [PROTOCOL_VERSION] } }); + const tools = await rpc(url, "tools/list"); + expect(tools.body.result!.tools.map((tool: { name: string }) => tool.name)).toEqual(["slow_report", "needs_approval", "will_fail", "cancellable", "expires", "quick_echo"]); + }); + + it("returns a task only to a request that declared the extension", async () => { + const { url } = await start(); + const plain = await call(url, "slow_report", {}, false); + expect(plain.body.result).toMatchObject({ resultType: "complete", isError: true }); + const quick = await call(url, "quick_echo", { text: "hi" }); + expect(quick.body.result).toMatchObject({ resultType: "complete", content: [{ type: "text", text: "echo: hi" }] }); + const tasks = await rpc(url, "tasks/get", { taskId: "anything" }, { tasks: false }); + expect(tasks.body.error?.code).toBe(-32021); + }); + + it("slow_report: works, then completes with its result after the delay", async () => { + const { url, server } = await start(); + const created = await call(url, "slow_report", { topic: "uptime", seconds: 30 }); + expect(created.body.result).toMatchObject({ resultType: "task", status: "working", ttlMs: 30_000 + 3_600_000, pollIntervalMs: 2_000 }); + const taskId = created.body.result!.taskId as string; + clock += 15_000; + expect((await get(url, taskId)).body.result).toMatchObject({ resultType: "complete", status: "working", statusMessage: "50% done (15 of 30 s)" }); + clock += 15_000; + const done = await get(url, taskId); + expect(done.body.result).toMatchObject({ status: "completed", result: { isError: false, structuredContent: { topic: "uptime", seconds: 30 } } }); + expect(server.store.snapshot().toolCalls).toEqual({ slow_report: 1 }); + }); + + it("requires Mcp-Name to carry the task ID", async () => { + const { url } = await start(); + const taskId = (await call(url, "slow_report")).body.result!.taskId as string; + const wrong = await rpc(url, "tasks/get", { taskId }, { name: "other" }); + expect(wrong.body.error?.code).toBe(-32020); + }); + + it("needs_approval: asks a form question, then finishes after tasks/update", async () => { + const { url } = await start(); + const created = await call(url, "needs_approval", { action: "deploy" }); + const taskId = created.body.result!.taskId as string; + const waiting = await get(url, taskId); + expect(waiting.body.result).toMatchObject({ + status: "input_required", + inputRequests: { approval: { method: "elicitation/create", params: { mode: "form", requestedSchema: { required: ["decision"] } } } }, + }); + // An answer the form does not allow asks again, under a new key: a client + // treats a key it already answered as done and would never show it again. + await rpc(url, "tasks/update", { taskId, inputResponses: { approval: { action: "accept", content: { decision: "maybe" } } } }); + const again = (await get(url, taskId)).body.result!; + expect(again).toMatchObject({ status: "input_required", inputRequests: { "approval-2": { params: { message: "Please choose approve or reject." } } } }); + expect(again.inputRequests).not.toHaveProperty("approval"); + // An answer to the old key is ignored. + await rpc(url, "tasks/update", { taskId, inputResponses: { approval: { action: "accept", content: { decision: "approve" } } } }); + expect((await get(url, taskId)).body.result).toMatchObject({ status: "input_required" }); + await rpc(url, "tasks/update", { taskId, inputResponses: { "approval-2": { action: "accept", content: { decision: "approve", note: "ship it" } } } }); + expect((await get(url, taskId)).body.result).toMatchObject({ status: "working" }); + clock += 3_000; + expect((await get(url, taskId)).body.result).toMatchObject({ status: "completed", result: { structuredContent: { approved: true, action: "deploy", note: "ship it" } } }); + }); + + it("needs_approval: a rejection or decline completes without doing anything", async () => { + const { url } = await start(); + for (const response of [{ action: "accept", content: { decision: "reject" } }, { action: "decline" }]) { + const taskId = (await call(url, "needs_approval")).body.result!.taskId as string; + await rpc(url, "tasks/update", { taskId, inputResponses: { approval: response } }); + expect((await get(url, taskId)).body.result).toMatchObject({ status: "completed", result: { structuredContent: { approved: false } } }); + } + }); + + it("keeps even the longest jobs long enough to collect their result", async () => { + const { url } = await start(); + const report = (await call(url, "slow_report", { seconds: 3600 })).body.result!.taskId as string; + const failure = (await call(url, "will_fail", { seconds: 3600 })).body.result!.taskId as string; + clock += 3_600_000 + 60_000; + expect((await get(url, report)).body.result).toMatchObject({ status: "completed" }); + expect((await get(url, failure)).body.result).toMatchObject({ status: "failed" }); + }); + + it("shows what finished or expired while no client was polling", async () => { + const { url } = await start(); + await call(url, "slow_report", { seconds: 10 }); + await call(url, "expires", { ttlSeconds: 5 }); + clock += 10_000; + const state = await (await fetch(`${url}/api/state`)).json() as { tasks: Array<{ tool: string; status: string }> }; + expect(state.tasks.map((task) => [task.tool, task.status])).toEqual([["slow_report", "completed"]]); + expect(await (await fetch(url)).text()).toContain("completed"); + }); + + it("does not record calls to tools it does not have", async () => { + const { url, server } = await start(); + for (const name of ["nope-1", "nope-2"]) expect((await call(url, name)).body.error?.code).toBe(-32602); + expect(server.store.snapshot().toolCalls).toEqual({}); + }); + + it("will_fail: ends failed with a JSON-RPC error", async () => { + const { url } = await start(); + const taskId = (await call(url, "will_fail", { seconds: 2 })).body.result!.taskId as string; + clock += 2_000; + expect((await get(url, taskId)).body.result).toMatchObject({ status: "failed", error: { code: -32603, message: expect.stringContaining("on purpose") } }); + }); + + it("cancellable: tasks/cancel stops it, and a second cancel is refused", async () => { + const { url } = await start(); + const taskId = (await call(url, "cancellable")).body.result!.taskId as string; + expect((await rpc(url, "tasks/cancel", { taskId })).body.result).toEqual({ resultType: "complete" }); + // Past when the work would have finished: it stays cancelled. + clock += 600_000; + expect((await get(url, taskId)).body.result).toMatchObject({ status: "cancelled" }); + expect((await rpc(url, "tasks/cancel", { taskId })).body.error?.code).toBe(-32602); + }); + + it("expires: is forgotten once its TTL passes, before it finishes", async () => { + const { url } = await start(); + const created = await call(url, "expires", { ttlSeconds: 5 }); + expect(created.body.result).toMatchObject({ ttlMs: 5_000 }); + const taskId = created.body.result!.taskId as string; + clock += 4_000; + expect((await get(url, taskId)).body.result).toMatchObject({ status: "working" }); + clock += 1_000; + expect((await get(url, taskId)).body.error).toMatchObject({ code: -32602, message: expect.stringContaining("not found") }); + }); + + it("keeps tasks across a restart when given a state file", async () => { + const dir = mkdtempSync(join(tmpdir(), "tasks-demo-")); + dirs.push(dir); + const file = join(dir, "state.json"); + const first = await start(new DemoTaskStore({ file, now })); + const taskId = (await call(first.url, "slow_report", { seconds: 10 })).body.result!.taskId as string; + await first.server.close(); + servers.splice(servers.indexOf(first.server), 1); + clock += 10_000; + const second = await start(new DemoTaskStore({ file, now })); + expect((await get(second.url, taskId)).body.result).toMatchObject({ status: "completed" }); + expect(second.server.store.snapshot().toolCalls).toEqual({ slow_report: 1 }); + }); + + it("refuses a state directory other users can write to, and writes through no planted file", async () => { + const shared = mkdtempSync(join(tmpdir(), "tasks-demo-shared-")); + dirs.push(shared); + chmodSync(shared, 0o777); + expect(() => new DemoTaskStore({ file: join(shared, "state.json"), now })).toThrow(/others cannot write/); + + const dir = mkdtempSync(join(tmpdir(), "tasks-demo-")); + dirs.push(dir); + const victim = join(dir, "victim.txt"); + writeFileSync(victim, "keep me"); + // A link at the old predictable temp name must not be followed. + symlinkSync(victim, join(dir, `state.json.${process.pid}.tmp`)); + const store = new DemoTaskStore({ file: join(dir, "state.json"), now }); + store.countToolCall("slow_report"); + expect(readFileSync(victim, "utf8")).toBe("keep me"); + expect(JSON.parse(readFileSync(join(dir, "state.json"), "utf8")).toolCalls).toEqual({ slow_report: 1 }); + }); + + it("shows tasks on its status page", async () => { + const { url } = await start(); + await call(url, "slow_report"); + const html = await (await fetch(url)).text(); + expect(html).toContain("slow_report"); + expect(html).toContain("working"); + }); +}); diff --git a/packages/kv-demo-mcp-server/src/tasks/http.ts b/packages/kv-demo-mcp-server/src/tasks/http.ts new file mode 100644 index 00000000000..cd0f9dd6973 --- /dev/null +++ b/packages/kv-demo-mcp-server/src/tasks/http.ts @@ -0,0 +1,257 @@ +import { createServer, type IncomingMessage, type Server, type ServerResponse } from "node:http"; +import { DemoTaskStore, type DemoTask } from "./store.js"; +import { DEMO_TOOLS, advanceDemoTask, answerDemoTask, callDemoTool, statusMessageFor } from "./tools.js"; +import { renderTasksPage } from "./render.js"; + +/** + * A stateless MCP server (protocol 2026-07-28) that supports the MCP Tasks + * extension. It speaks JSON-RPC over plain HTTP POST: no session, no SSE, and + * every request carries its protocol version and client capabilities in + * `params._meta`, as the 2026-07-28 revision requires. + */ +export const PROTOCOL_VERSION = "2026-07-28"; +export const TASKS_EXTENSION_ID = "io.modelcontextprotocol/tasks"; +const META_VERSION = "io.modelcontextprotocol/protocolVersion"; +const META_CAPABILITIES = "io.modelcontextprotocol/clientCapabilities"; + +const HEADER_MISMATCH = -32020; +const MISSING_CAPABILITY = -32021; +const UNSUPPORTED_VERSION = -32022; +const INVALID_PARAMS = -32602; +const METHOD_NOT_FOUND = -32601; + +export interface TasksDemoHttpOptions { + store?: DemoTaskStore; + /** Optional shared secret required on every route except the status page. */ + token?: string | null; +} + +export interface TasksDemoHttpServer { + server: Server; + store: DemoTaskStore; + listen: (port: number, host?: string) => Promise; + close: () => Promise; +} + +type RpcRequest = { jsonrpc?: string; id?: string | number | null; method?: unknown; params?: Record }; +type Reply = { status: number; body: unknown } | null; + +function record(value: unknown): Record | null { + return value && typeof value === "object" && !Array.isArray(value) ? value as Record : null; +} + +function ok(id: unknown, result: Record): Reply { + return { status: 200, body: { jsonrpc: "2.0", id, result: { resultType: "complete", ...result } } }; +} + +function fail(status: number, id: unknown, code: number, message: string, data?: unknown): Reply { + return { status, body: { jsonrpc: "2.0", id: id ?? null, error: { code, message, ...(data !== undefined ? { data } : {}) } } }; +} + +/** `Mcp-Name` may be RFC 2047-style base64 (`=?base64?...?=`) for non-ASCII values. */ +function decodeHeaderValue(value: string | undefined): string | null { + if (value === undefined) return null; + const match = /^=\?base64\?(.*)\?=$/i.exec(value); + return match ? Buffer.from(match[1]!, "base64").toString("utf8") : value; +} + +function taskView(store: DemoTaskStore, task: DemoTask): Record { + const statusMessage = statusMessageFor(store, task); + return { + taskId: task.taskId, + status: task.status, + ...(statusMessage ? { statusMessage } : {}), + createdAt: new Date(task.createdAt).toISOString(), + lastUpdatedAt: new Date(task.lastUpdatedAt).toISOString(), + ttlMs: task.ttlMs, + pollIntervalMs: task.pollIntervalMs, + ...(task.status === "completed" ? { result: task.result } : {}), + ...(task.status === "failed" ? { error: task.error } : {}), + ...(task.status === "input_required" ? { inputRequests: task.inputRequests } : {}), + }; +} + +/** Handle one JSON-RPC request. Exported for tests. */ +export function handleTasksDemoRpc(store: DemoTaskStore, message: RpcRequest, headers: IncomingMessage["headers"]): Reply { + if (!record(message) || typeof message.method !== "string") return fail(400, null, -32600, "Invalid JSON-RPC request."); + const { id, method } = message; + if (id === undefined) return null; // A notification: nothing to answer. + const params = record(message.params) ?? {}; + const meta = record(params._meta) ?? {}; + const headerVersion = typeof headers["mcp-protocol-version"] === "string" ? headers["mcp-protocol-version"] : null; + + // A handshake-era client: tell it which version this server speaks. + if (method === "initialize") return fail(400, id, UNSUPPORTED_VERSION, "This server speaks only stateless MCP.", { supported: [PROTOCOL_VERSION] }); + if (headerVersion !== PROTOCOL_VERSION || meta[META_VERSION] !== PROTOCOL_VERSION) { + if (headerVersion && headerVersion !== PROTOCOL_VERSION) return fail(400, id, UNSUPPORTED_VERSION, `Unsupported protocol version ${headerVersion}.`, { supported: [PROTOCOL_VERSION] }); + return fail(400, id, HEADER_MISMATCH, "MCP-Protocol-Version header and params._meta protocolVersion must both be 2026-07-28."); + } + const mcpMethod = headers["mcp-method"]; + if (typeof mcpMethod === "string" && mcpMethod !== method) return fail(400, id, HEADER_MISMATCH, "Mcp-Method does not match the request."); + const declaredTasks = Boolean(record(record(record(meta[META_CAPABILITIES])?.extensions)?.[TASKS_EXTENSION_ID])); + const clientInfo = record(meta["io.modelcontextprotocol/clientInfo"]); + const client = typeof clientInfo?.name === "string" ? clientInfo.name : "unknown client"; + + switch (method) { + case "server/discover": + return ok(id, { + supportedVersions: [PROTOCOL_VERSION], + capabilities: { tools: {}, extensions: { [TASKS_EXTENSION_ID]: {} } }, + serverInfo: { name: "paperclip-tasks-demo", version: "0.1.0" }, + instructions: "Demo tools for MCP Tasks. Each long tool returns a task; nothing here touches real systems.", + }); + case "ping": + return ok(id, {}); + case "tools/list": + return ok(id, { tools: DEMO_TOOLS }); + case "tools/call": { + const name = params.name; + if (typeof name !== "string") return fail(200, id, INVALID_PARAMS, "tools/call needs a tool name."); + const mcpName = decodeHeaderValue(headers["mcp-name"] as string | undefined); + if (mcpName !== null && mcpName !== name) return fail(400, id, HEADER_MISMATCH, "Mcp-Name does not match the tool name."); + // Only known tools are counted, so made-up names cannot grow the saved state. + if (!DEMO_TOOLS.some((tool) => tool.name === name)) return fail(200, id, INVALID_PARAMS, `Unknown tool: ${name.slice(0, 64)}`); + store.countToolCall(name); + const outcome = callDemoTool(store, name, record(params.arguments) ?? {}, declaredTasks); + if (!outcome) return fail(200, id, INVALID_PARAMS, `Unknown tool: ${name.slice(0, 64)}`); + if (outcome.kind === "result") { + store.log({ method, tool: name, detail: declaredTasks ? "answered without a task" : `answered without a task: ${client} did not declare ${TASKS_EXTENSION_ID}` }); + return ok(id, outcome.result); + } + store.log({ method, tool: name, taskId: outcome.task.taskId, detail: `returned task (${outcome.task.status}) to ${client}` }); + const view = taskView(store, outcome.task); + return { status: 200, body: { jsonrpc: "2.0", id, result: { resultType: "task", ...view } } }; + } + case "tasks/get": + case "tasks/update": + case "tasks/cancel": { + if (!declaredTasks) return fail(400, id, MISSING_CAPABILITY, `${method} requires the ${TASKS_EXTENSION_ID} extension.`); + const taskId = params.taskId; + if (typeof taskId !== "string") return fail(200, id, INVALID_PARAMS, `${method} needs a taskId.`); + if (decodeHeaderValue(headers["mcp-name"] as string | undefined) !== taskId) return fail(400, id, HEADER_MISMATCH, "Mcp-Name must carry the task ID."); + const found = store.get(taskId); + if (!found) { + store.log({ method, taskId, detail: "task not found" }); + return fail(200, id, INVALID_PARAMS, "Failed to retrieve task: Task not found"); + } + const task = advanceDemoTask(store, found); + if (method === "tasks/get") { + task.polls += 1; + store.touch(); + store.log({ method, tool: task.tool, taskId, detail: task.status }); + return ok(id, taskView(store, task)); + } + if (method === "tasks/update") { + answerDemoTask(store, task, record(params.inputResponses) ?? {}); + store.log({ method, tool: task.tool, taskId, detail: `now ${task.status}` }); + return ok(id, {}); + } + if (task.status === "completed" || task.status === "failed" || task.status === "cancelled") { + store.log({ method, tool: task.tool, taskId, detail: `already ${task.status}` }); + return fail(200, id, INVALID_PARAMS, `Cannot cancel a task that is already ${task.status}.`); + } + store.update(task, { status: "cancelled", inputRequests: undefined, settlesAt: null }); + store.log({ method, tool: task.tool, taskId, detail: "cancelled; work stopped" }); + return ok(id, {}); + } + default: + return fail(404, id, METHOD_NOT_FOUND, `Method not found: ${method}`); + } +} + +/** + * State as of now: settle tasks whose time has come and forget expired ones, + * so the views show what finished while no client was polling. + */ +function refreshedSnapshot(store: DemoTaskStore) { + for (const { taskId } of store.snapshot().tasks) { + const task = store.get(taskId); + if (task) advanceDemoTask(store, task); + } + return store.snapshot(); +} + +function send(res: ServerResponse, status: number, body: unknown, type = "application/json"): void { + const payload = typeof body === "string" ? body : JSON.stringify(body); + res.writeHead(status, { "content-type": type, "content-length": Buffer.byteLength(payload) }); + res.end(payload); +} + +async function readBody(req: IncomingMessage): Promise { + const chunks: Buffer[] = []; + let size = 0; + for await (const chunk of req) { + size += (chunk as Buffer).length; + if (size > 1_000_000) throw new Error("Request body too large."); + chunks.push(chunk as Buffer); + } + return Buffer.concat(chunks).toString("utf8"); +} + +export function createTasksDemoHttpServer(options: TasksDemoHttpOptions = {}): TasksDemoHttpServer { + const store = options.store ?? new DemoTaskStore(); + const token = options.token?.trim() || null; + + const server = createServer((req, res) => { + void (async () => { + try { + const url = new URL(req.url ?? "/", "http://localhost"); + if (req.method === "GET" && (url.pathname === "/" || url.pathname === "/index.html")) { + send(res, 200, renderTasksPage(token ? null : refreshedSnapshot(store), store.now()), "text/html; charset=utf-8"); + return; + } + if (token && req.headers.authorization !== `Bearer ${token}`) { + send(res, 401, { error: "Unauthorized. Provide the TASKS_DEMO_TOKEN as a Bearer token." }); + return; + } + if (req.method === "GET" && url.pathname === "/api/state") { + send(res, 200, refreshedSnapshot(store)); + return; + } + if (url.pathname === "/mcp") { + if (req.method !== "POST") { + res.writeHead(405, { allow: "POST" }).end(); + return; + } + let message: RpcRequest; + try { + message = JSON.parse(await readBody(req)) as RpcRequest; + } catch { + send(res, 400, { jsonrpc: "2.0", id: null, error: { code: -32700, message: "Parse error" } }); + return; + } + const reply = handleTasksDemoRpc(store, message, req.headers); + if (!reply) { + res.writeHead(202).end(); + return; + } + send(res, reply.status, reply.body); + return; + } + send(res, 404, { error: "Not found." }); + } catch (error) { + if (!res.headersSent) send(res, 500, { error: error instanceof Error ? error.message : "Internal error." }); + else res.end(); + } + })(); + }); + + return { + server, + store, + listen: (port, host = "127.0.0.1") => + new Promise((resolve, reject) => { + server.once("error", reject); + server.listen(port, host, () => { + server.off("error", reject); + const address = server.address(); + resolve(typeof address === "object" && address ? address.port : port); + }); + }), + close: () => + new Promise((resolve, reject) => { + server.closeAllConnections(); + server.close((error) => (error ? reject(error) : resolve())); + }), + }; +} diff --git a/packages/kv-demo-mcp-server/src/tasks/main.ts b/packages/kv-demo-mcp-server/src/tasks/main.ts new file mode 100644 index 00000000000..d1cdadff5db --- /dev/null +++ b/packages/kv-demo-mcp-server/src/tasks/main.ts @@ -0,0 +1,34 @@ +#!/usr/bin/env node +import { homedir } from "node:os"; +import { join } from "node:path"; +import { createTasksDemoHttpServer } from "./http.js"; +import { DemoTaskStore } from "./store.js"; + +function port(raw: string | undefined): number { + if (!raw) return 8849; + const value = Number.parseInt(raw, 10); + if (!Number.isInteger(value) || value < 0 || value > 65535) throw new Error(`Invalid port: ${raw}`); + return value; +} + +async function main(): Promise { + const env = process.env; + // State lives in a file by default so the server can be stopped and started mid-task. + const stateFile = env.TASKS_DEMO_STATE_FILE === "memory" ? null : env.TASKS_DEMO_STATE_FILE || join(homedir(), ".paperclip-tasks-demo", "state.json"); + const host = env.TASKS_DEMO_HOST?.trim() || "127.0.0.1"; + const token = env.TASKS_DEMO_TOKEN?.trim() || null; + const { listen } = createTasksDemoHttpServer({ store: new DemoTaskStore({ file: stateFile }), token }); + const bound = await listen(port(env.PORT ?? env.TASKS_DEMO_PORT), host); + const base = `http://${host}:${bound}`; + console.error(`MCP Tasks demo server listening on ${base}`); + console.error(` MCP endpoint: ${base}/mcp`); + console.error(` Status page: ${base}/`); + console.error(` JSON state: ${base}/api/state`); + console.error(` Task state: ${stateFile ?? "in memory (lost on restart)"}`); + console.error(` Auth: ${token ? "TASKS_DEMO_TOKEN required (Bearer header)" : "none (set TASKS_DEMO_TOKEN to require one)"}`); +} + +void main().catch((error) => { + console.error("Failed to start the MCP Tasks demo server:", error instanceof Error ? error.message : error); + process.exit(1); +}); diff --git a/packages/kv-demo-mcp-server/src/tasks/render.ts b/packages/kv-demo-mcp-server/src/tasks/render.ts new file mode 100644 index 00000000000..3e509db50ed --- /dev/null +++ b/packages/kv-demo-mcp-server/src/tasks/render.ts @@ -0,0 +1,64 @@ +import type { DemoState } from "./store.js"; + +function escapeHtml(value: string): string { + return value.replace(/&/g, "&").replace(//g, ">").replace(/"/g, """).replace(/'/g, "'"); +} + +function time(ms: number): string { + return new Date(ms).toISOString().slice(11, 19); +} + +/** + * A plain status page: every task, how many times each tool was started, and + * the latest protocol events. It reloads itself every two seconds. + */ +export function renderTasksPage(state: DemoState | null, now: number): string { + const body = state === null + ? `

This server requires a token, so its state is not shown here. Read /api/state with the token instead.

` + : ` +

Tasks

+ + + ${state.tasks.length === 0 + ? `` + : [...state.tasks].reverse().map((task) => ` + + + + + + + `).join("")} +
ToolStatusCreatedChecksTTL leftTask ID
No tasks yet. Ask an agent to call slow_report.
${escapeHtml(task.tool)}${escapeHtml(task.status)}${time(task.createdAt)}${task.polls}${Math.max(0, Math.round((task.createdAt + task.ttlMs - now) / 1000))} s${escapeHtml(task.taskId)}
+

Times each tool was started (tools/call)

+

${Object.entries(state.toolCalls).map(([tool, count]) => `${escapeHtml(tool)}: ${count}`).join(" · ") || "none yet"}

+

Latest events

+ + + ${[...state.events].reverse().slice(0, 60).map((event) => ` + + `).join("")} +
TimeEventToolDetail
${time(event.at)}${escapeHtml(event.method)}${escapeHtml(event.tool ?? "")}${escapeHtml(event.detail)}
`; + return ` + + + + + +MCP Tasks demo + + + +

MCP Tasks demo server

+

MCP endpoint: /mcp (protocol 2026-07-28, extension io.modelcontextprotocol/tasks). This page reloads every 2 seconds.

+${body} + +`; +} diff --git a/packages/kv-demo-mcp-server/src/tasks/store.ts b/packages/kv-demo-mcp-server/src/tasks/store.ts new file mode 100644 index 00000000000..9c43c7afbd0 --- /dev/null +++ b/packages/kv-demo-mcp-server/src/tasks/store.ts @@ -0,0 +1,157 @@ +import { randomUUID } from "node:crypto"; +import { lstatSync, mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs"; +import { basename, dirname, join } from "node:path"; + +/** Task states from the MCP Tasks extension (2026-07-28). */ +export type DemoTaskStatus = "working" | "input_required" | "completed" | "failed" | "cancelled"; + +export type DemoInputRequest = { method: string; params: Record }; + +export interface DemoTask { + taskId: string; + tool: string; + arguments: Record; + status: DemoTaskStatus; + /** Epoch ms. */ + createdAt: number; + lastUpdatedAt: number; + ttlMs: number; + pollIntervalMs: number; + /** When a `working` task settles on its own (epoch ms). */ + settlesAt: number | null; + /** How a `working` task settles at `settlesAt`. */ + outcome: "complete" | "fail"; + inputRequests?: Record; + /** The person's answer from `tasks/update`, kept for the result. */ + answer?: Record; + result?: Record; + error?: { code: number; message: string }; + /** How many `tasks/get` requests this task has answered. */ + polls: number; +} + +export interface DemoEvent { + at: number; + method: string; + tool?: string; + taskId?: string; + detail: string; +} + +export interface DemoState { + tasks: DemoTask[]; + /** Every `tools/call` received, by tool name. Proves "started exactly once". */ + toolCalls: Record; + events: DemoEvent[]; +} + +const MAX_EVENTS = 200; +const MAX_TASKS = 500; + +/** + * In-memory task state, optionally mirrored to a JSON file so the server can + * be stopped and started again mid-task. A task's progress is derived from the + * clock (`settlesAt`), so a restart never resets or repeats work. + */ +/** + * The state directory must be a real directory owned by this user and not + * writable by anyone else, so no other local user can plant files in it. + */ +function assertPrivateDirectory(dir: string): void { + mkdirSync(dir, { recursive: true, mode: 0o700 }); + const info = lstatSync(dir); + const uid = process.getuid?.(); + if (!info.isDirectory() || (uid !== undefined && info.uid !== uid) || (info.mode & 0o022) !== 0) { + throw new Error(`Task state directory ${dir} must be a directory you own that others cannot write to. Choose another TASKS_DEMO_STATE_FILE.`); + } +} + +export class DemoTaskStore { + private state: DemoState = { tasks: [], toolCalls: {}, events: [] }; + + constructor( + private readonly options: { file?: string | null; now?: () => number } = {}, + ) { + if (options.file) { + assertPrivateDirectory(dirname(options.file)); + this.load(options.file); + } + } + + now(): number { + return this.options.now ? this.options.now() : Date.now(); + } + + private load(file: string): void { + try { + const parsed = JSON.parse(readFileSync(file, "utf8")) as Partial; + this.state = { + tasks: Array.isArray(parsed.tasks) ? parsed.tasks : [], + toolCalls: parsed.toolCalls && typeof parsed.toolCalls === "object" ? parsed.toolCalls : {}, + events: Array.isArray(parsed.events) ? parsed.events : [], + }; + } catch { + // No file yet, or unreadable: start empty. + } + } + + private save(): void { + const file = this.options.file; + if (!file) return; + // A fresh, unguessable temp name, created exclusively: nothing another + // user placed in the directory (such as a symlink) is ever written through. + const temp = join(dirname(file), `.${basename(file)}.${randomUUID()}.tmp`); + writeFileSync(temp, JSON.stringify(this.state, null, 2), { flag: "wx", mode: 0o600 }); + renameSync(temp, file); + } + + log(event: Omit): void { + this.state.events.push({ at: this.now(), ...event }); + if (this.state.events.length > MAX_EVENTS) this.state.events.splice(0, this.state.events.length - MAX_EVENTS); + this.save(); + } + + countToolCall(tool: string): void { + this.state.toolCalls[tool] = (this.state.toolCalls[tool] ?? 0) + 1; + this.save(); + } + + create(input: Omit): DemoTask { + const now = this.now(); + const task: DemoTask = { ...input, taskId: `demo-${randomUUID()}`, createdAt: now, lastUpdatedAt: now, polls: 0 }; + this.state.tasks.push(task); + if (this.state.tasks.length > MAX_TASKS) this.state.tasks.splice(0, this.state.tasks.length - MAX_TASKS); + this.save(); + return task; + } + + /** + * The task as it is now, or null when unknown or past its TTL. An expired + * task is forgotten, as the specification allows. + */ + get(taskId: string): DemoTask | null { + const task = this.state.tasks.find((candidate) => candidate.taskId === taskId); + if (!task) return null; + const now = this.now(); + if (now >= task.createdAt + task.ttlMs) { + this.state.tasks = this.state.tasks.filter((candidate) => candidate !== task); + this.log({ method: "expired", tool: task.tool, taskId, detail: `TTL of ${Math.round(task.ttlMs / 1000)} s passed; task forgotten` }); + return null; + } + return task; + } + + update(task: DemoTask, patch: Partial): DemoTask { + Object.assign(task, patch, { lastUpdatedAt: this.now() }); + this.save(); + return task; + } + + touch(): void { + this.save(); + } + + snapshot(): DemoState { + return structuredClone(this.state); + } +} diff --git a/packages/kv-demo-mcp-server/src/tasks/tools.ts b/packages/kv-demo-mcp-server/src/tasks/tools.ts new file mode 100644 index 00000000000..9f8b126abb8 --- /dev/null +++ b/packages/kv-demo-mcp-server/src/tasks/tools.ts @@ -0,0 +1,207 @@ +import type { DemoTask, DemoTaskStore } from "./store.js"; + +/** + * The demo tools. Each one shows one MCP Tasks behaviour. None of them touch + * anything outside this process: the "work" is the clock running. + */ +export const DEMO_TOOLS = [ + { + name: "slow_report", + description: "Writes a short report after a delay (default 20 seconds). Returns a task right away, then the report when the delay has passed. Shows polling.", + inputSchema: { + type: "object", + properties: { + topic: { type: "string", description: "What the report is about." }, + seconds: { type: "integer", minimum: 1, maximum: 3600, description: "How long the report takes. Default 20." }, + }, + }, + annotations: { readOnlyHint: true, title: "Slow report" }, + }, + { + name: "needs_approval", + description: "Pretends to perform an action that a person must approve first. The task asks a question (approve or reject) and finishes after the person answers. Shows input_required and tasks/update.", + inputSchema: { + type: "object", + properties: { action: { type: "string", description: "The action that needs approval." } }, + }, + annotations: { readOnlyHint: true, title: "Needs approval" }, + }, + { + name: "will_fail", + description: "Starts work that fails after a few seconds (default 5). Shows a failed task.", + inputSchema: { + type: "object", + properties: { seconds: { type: "integer", minimum: 1, maximum: 3600, description: "Seconds before it fails. Default 5." } }, + }, + annotations: { readOnlyHint: true, title: "Will fail" }, + }, + { + name: "cancellable", + description: "Starts long work (default 10 minutes) that stops cleanly when cancelled. Shows tasks/cancel.", + inputSchema: { + type: "object", + properties: { seconds: { type: "integer", minimum: 1, maximum: 86400, description: "How long the work runs. Default 600." } }, + }, + annotations: { readOnlyHint: true, title: "Cancellable" }, + }, + { + name: "expires", + description: "Starts work that outlives its own short TTL (default 15 seconds), so the server forgets the task before it finishes. Shows an expired task.", + inputSchema: { + type: "object", + properties: { ttlSeconds: { type: "integer", minimum: 1, maximum: 3600, description: "The task's TTL. Default 15." } }, + }, + annotations: { readOnlyHint: true, title: "Expires" }, + }, + { + name: "quick_echo", + description: "Answers immediately, without a task. Shows that ordinary calls are unchanged.", + inputSchema: { + type: "object", + properties: { text: { type: "string" } }, + }, + annotations: { readOnlyHint: true, title: "Quick echo" }, + }, +] as const; + +export type DemoToolName = (typeof DEMO_TOOLS)[number]["name"]; + +const HOUR_MS = 60 * 60_000; +const POLL_INTERVAL_MS = 2_000; +/** How long `needs_approval` keeps working after the person answers. */ +export const APPROVAL_FOLLOW_UP_MS = 3_000; + +export const APPROVAL_FORM = { + method: "elicitation/create", + params: { + mode: "form", + message: "Approve this demo action?", + requestedSchema: { + type: "object", + properties: { + decision: { type: "string", title: "Decision", enum: ["approve", "reject"] }, + note: { type: "string", title: "Note for the agent (optional)" }, + }, + required: ["decision"], + }, + }, +} as const; + +function intArg(args: Record, key: string, fallback: number, max: number): number { + const value = args[key]; + return typeof value === "number" && Number.isInteger(value) && value >= 1 ? Math.min(value, max) : fallback; +} + +function text(value: string, structured?: Record, isError = false): Record { + return { content: [{ type: "text", text: value }], ...(structured ? { structuredContent: structured } : {}), isError }; +} + +export type ToolCallOutcome = + | { kind: "result"; result: Record } + | { kind: "task"; task: DemoTask }; + +/** + * Run a tool. `tasks` says whether this request declared the Tasks extension; + * a server must never return a task to a request that did not. + */ +export function callDemoTool(store: DemoTaskStore, name: string, args: Record, tasks: boolean): ToolCallOutcome | null { + const now = store.now(); + const base = { tool: name, arguments: args, ttlMs: HOUR_MS, pollIntervalMs: POLL_INTERVAL_MS, outcome: "complete" as const }; + switch (name) { + case "quick_echo": + return { kind: "result", result: text(`echo: ${typeof args.text === "string" ? args.text : ""}`) }; + case "slow_report": { + const seconds = intArg(args, "seconds", 20, 3600); + if (!tasks) return { kind: "result", result: text(`This client does not support MCP Tasks, so the ${seconds}-second report was skipped.`, undefined, true) }; + return { kind: "task", task: store.create({ ...base, status: "working", settlesAt: now + seconds * 1000, ttlMs: seconds * 1000 + HOUR_MS }) }; + } + case "needs_approval": + if (!tasks) return { kind: "result", result: text("This action needs a person's approval, which requires a client that supports MCP Tasks.", undefined, true) }; + return { kind: "task", task: store.create({ ...base, status: "input_required", settlesAt: null, inputRequests: { approval: structuredClone(APPROVAL_FORM) as never } }) }; + case "will_fail": { + const seconds = intArg(args, "seconds", 5, 3600); + if (!tasks) return { kind: "result", result: text("The demo failure needs a client that supports MCP Tasks.", undefined, true) }; + return { kind: "task", task: store.create({ ...base, status: "working", settlesAt: now + seconds * 1000, outcome: "fail", ttlMs: seconds * 1000 + HOUR_MS }) }; + } + case "cancellable": { + const seconds = intArg(args, "seconds", 600, 86400); + if (!tasks) return { kind: "result", result: text("The cancellable demo needs a client that supports MCP Tasks.", undefined, true) }; + return { kind: "task", task: store.create({ ...base, status: "working", settlesAt: now + seconds * 1000, ttlMs: Math.max(HOUR_MS, seconds * 2000) }) }; + } + case "expires": { + const ttlSeconds = intArg(args, "ttlSeconds", 15, 3600); + if (!tasks) return { kind: "result", result: text("The expiry demo needs a client that supports MCP Tasks.", undefined, true) }; + // Works ten times longer than it is kept, so it is always forgotten first. + return { kind: "task", task: store.create({ ...base, status: "working", ttlMs: ttlSeconds * 1000, settlesAt: now + ttlSeconds * 10_000 }) }; + } + default: + return null; + } +} + +/** Settle a `working` task whose time has come. Called on every read. */ +export function advanceDemoTask(store: DemoTaskStore, task: DemoTask): DemoTask { + if (task.status !== "working" || task.settlesAt === null || store.now() < task.settlesAt) return task; + if (task.outcome === "fail") { + store.update(task, { status: "failed", error: { code: -32603, message: "The demo job crashed on purpose (will_fail)." } }); + store.log({ method: "settled", tool: task.tool, taskId: task.taskId, detail: "failed" }); + return task; + } + store.update(task, { status: "completed", result: resultFor(task) }); + store.log({ method: "settled", tool: task.tool, taskId: task.taskId, detail: "completed" }); + return task; +} + +function resultFor(task: DemoTask): Record { + const seconds = Math.round(((task.settlesAt ?? task.createdAt) - task.createdAt) / 1000); + if (task.tool === "needs_approval") { + const note = typeof task.answer?.note === "string" && task.answer.note ? ` Note: ${task.answer.note}` : ""; + const action = typeof task.arguments.action === "string" ? task.arguments.action : "the demo action"; + return text(`Approved: ${action} was carried out.${note}`, { approved: true, action, note: task.answer?.note ?? null }); + } + if (task.tool === "slow_report") { + const topic = typeof task.arguments.topic === "string" && task.arguments.topic ? task.arguments.topic : "nothing in particular"; + return text(`Report on ${topic}: finished after ${seconds} seconds and ${task.polls} status checks.`, { topic, seconds }); + } + return text(`${task.tool} finished after ${seconds} seconds.`, { seconds }); +} + +/** A human-readable progress line for `statusMessage`. */ +export function statusMessageFor(store: DemoTaskStore, task: DemoTask): string | undefined { + if (task.status === "input_required") return "Waiting for a person to approve or reject."; + if (task.status !== "working" || task.settlesAt === null) return undefined; + const total = Math.max(1, task.settlesAt - task.createdAt); + const done = Math.min(total, Math.max(0, store.now() - task.createdAt)); + return `${Math.floor((done / total) * 100)}% done (${Math.round(done / 1000)} of ${Math.round(total / 1000)} s)`; +} + +/** + * Apply a person's answer from `tasks/update`. Answers the task is not + * waiting for are ignored, as the specification requires. + */ +export function answerDemoTask(store: DemoTaskStore, task: DemoTask, inputResponses: Record): void { + if (task.status !== "input_required" || !task.inputRequests) return; + const key = Object.keys(task.inputRequests)[0]; + const response = key ? inputResponses[key] as { action?: unknown; content?: Record } | undefined : undefined; + if (!response || typeof response !== "object") return; + const content = response.content && typeof response.content === "object" ? response.content : {}; + if (response.action !== "accept" || content.decision === "reject") { + store.update(task, { + status: "completed", + inputRequests: undefined, + answer: content, + result: text("Rejected: nothing was done.", { approved: false }), + }); + store.log({ method: "answered", tool: task.tool, taskId: task.taskId, detail: `rejected (${String(response.action)})` }); + return; + } + if (content.decision !== "approve") { + // Ask again under a new key: a client treats an answered key as done. + const attempt = Number(/^approval-(\d+)$/.exec(key!)?.[1] ?? 1) + 1; + store.update(task, { inputRequests: { [`approval-${attempt}`]: { ...structuredClone(APPROVAL_FORM), params: { ...structuredClone(APPROVAL_FORM.params), message: "Please choose approve or reject." } } as never } }); + store.log({ method: "answered", tool: task.tool, taskId: task.taskId, detail: "invalid answer; asked again" }); + return; + } + store.update(task, { status: "working", inputRequests: undefined, answer: content, settlesAt: store.now() + APPROVAL_FOLLOW_UP_MS }); + store.log({ method: "answered", tool: task.tool, taskId: task.taskId, detail: "approved" }); +} diff --git a/server/src/__tests__/mcp-tasks-demo-server.test.ts b/server/src/__tests__/mcp-tasks-demo-server.test.ts new file mode 100644 index 00000000000..48cdc91519e --- /dev/null +++ b/server/src/__tests__/mcp-tasks-demo-server.test.ts @@ -0,0 +1,197 @@ +import { randomUUID } from "node:crypto"; +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { + agents, + companies, + companyMemberships, + createDb, + heartbeatRuns, + issueThreadInteractions, + issues, + projects, + toolMcpTasks, +} from "@paperclipai/db"; +import { and, eq, lte, ne } from "drizzle-orm"; +import { forgetMcpHttpSessions } from "../services/mcp-http.js"; +import { createToolGatewayService } from "../services/tool-gateway.js"; +import { toolAccessService } from "../services/tool-access.js"; +import { instanceSettingsService } from "../services/instance-settings.js"; +import { adapterExecutionControls, createAdapterExecutionControl } from "../services/adapter-execution-control.js"; +import { + createTasksDemoHttpServer, + DemoTaskStore, + type TasksDemoHttpServer, +} from "../../../packages/kv-demo-mcp-server/src/index.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; + +/** + * MCP Tasks end to end against the real demo server + * (`packages/kv-demo-mcp-server/src/tasks`) over HTTP. The demo's clock is + * moved by hand, so a "30-second" job finishes the moment the test says so. + */ +const support = await getEmbeddedPostgresTestSupport(); +(support.supported ? describe : describe.skip)("MCP Tasks with the demo server", () => { + const actor = { actorType: "user" as const, actorId: "mcp-tasks-demo-user", actorSource: "local_implicit" as const }; + let fixture: Awaited>; + let db: ReturnType; + let keyDir: string; + let demo: TasksDemoHttpServer; + let demoUrl: string; + let clock = Date.now(); + + beforeAll(async () => { + keyDir = await mkdtemp(join(tmpdir(), "mcp-tasks-demo-secrets-")); + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY_FILE", join(keyDir, "key")); + fixture = await startEmbeddedPostgresTestDatabase("mcp-tasks-demo-"); + db = createDb(fixture.connectionString); + await instanceSettingsService(db).updateExperimental({ enableMcpTasks: true }); + demo = createTasksDemoHttpServer({ store: new DemoTaskStore({ now: () => clock }) }); + demoUrl = `http://127.0.0.1:${await demo.listen(0)}/mcp`; + }, 30_000); + afterAll(async () => { + await demo?.close(); + await fixture?.cleanup(); + vi.unstubAllEnvs(); + if (keyDir) await rm(keyDir, { recursive: true, force: true }); + }); + + const starts = (tool: string) => demo.store.snapshot().toolCalls[tool] ?? 0; + const demoTask = (tool: string) => demo.store.snapshot().tasks.filter((task) => task.tool === tool).at(-1); + const polled = (tool: string) => demoTask(tool)?.polls ?? 0; + + async function setup() { + const [org] = await db.insert(companies).values({ name: `Demo ${randomUUID()}`, issuePrefix: `D${randomUUID().slice(0, 5)}` }).returning(); + await db.insert(companyMemberships).values({ companyId: org.id, principalType: "user", principalId: actor.actorId, status: "active", membershipRole: "admin" }); + const [agent] = await db.insert(agents).values({ companyId: org.id, name: "Demo agent", role: "engineer", adapterType: "process", adapterConfig: {}, runtimeConfig: {} }).returning(); + const [project] = await db.insert(projects).values({ companyId: org.id, name: "Demo project" }).returning(); + const [issue] = await db.insert(issues).values({ companyId: org.id, projectId: project.id, title: "Try the demo", status: "in_progress", assigneeAgentId: agent.id }).returning(); + const [run] = await db.insert(heartbeatRuns).values({ companyId: org.id, agentId: agent.id, invocationSource: "assignment", status: "running", contextSnapshot: { issueId: issue.id, projectId: project.id } }).returning(); + // A public-looking link, forwarded to the local demo server. + const options = { deploymentMode: "local_trusted" as const, deploymentExposure: "private" as const, + remoteHttpEndpointLookup: async () => [{ address: "8.8.8.8", family: 4 as const }], + remoteHttpRequest: (_url: string, init: RequestInit) => fetch(demoUrl, init) }; + const service = toolAccessService(db, options); + const connected = await service.connectGalleryApp(org.id, { link: `https://tasks-demo-${randomUUID().slice(0, 8)}.example.com/mcp`, name: "Tasks demo" }, actor); + await service.finishGalleryAppConnection(org.id, connected.connectionId, { + enabledCatalogEntryIds: connected.catalog.map((entry) => entry.id), askFirstCatalogEntryIds: [], access: "all_agents", + }, actor); + const gatewayOptions = { ...options, toolActionSigningSecret: "tasks-demo-signing-key" }; + let gateway = createToolGatewayService(db, gatewayOptions); + const session = await gateway.createSession({ companyId: org.id, agentId: agent.id, runId: run.id }); + const names = (await gateway.listToolsForSession(session.token)).map((tool) => tool.name); + const toolName = (name: string) => names.find((candidate) => candidate.endsWith(`:${name.replace(/_/g, "-")}`))!; + const call = (name: string, parameters: Record = {}, timeoutMs = 10_000, sessionToken = session.token) => + gateway.executeTool({ sessionToken, tool: toolName(name), parameters, timeoutMs }); + const rows = () => db.select().from(toolMcpTasks).where(eq(toolMcpTasks.companyId, org.id)); + return { + org, agent, issue, run, call, rows, names, + gateway: () => gateway, + restart: () => { forgetMcpHttpSessions(connected.connectionId); gateway = createToolGatewayService(db, gatewayOptions); return gateway; }, + /** Background poll for this company only. */ + poll: async (wake: (input: unknown) => Promise) => { + await db.update(toolMcpTasks).set({ nextPollAt: new Date(Date.now() + 60 * 60_000) }) + .where(and(ne(toolMcpTasks.companyId, org.id), lte(toolMcpTasks.nextPollAt, new Date()))); + return gateway.pollDueMcpTasks({ wake }); + }, + }; + } + + it("lists every demo tool", async () => { + const { names } = await setup(); + expect(names.map((name) => name.split(":")[1]).sort()).toEqual(["cancellable", "expires", "needs-approval", "quick-echo", "slow-report", "will-fail"]); + }); + + it("slow_report: polls the task and returns the report like a direct result", async () => { + const { call, rows } = await setup(); + const before = starts("slow_report"); + const pending = call("slow_report", { topic: "uptime", seconds: 30 }); + await vi.waitFor(() => expect(polled("slow_report")).toBeGreaterThan(0), { timeout: 5_000 }); + clock += 30_000; + const result = await pending; + expect(result).toMatchObject({ status: "completed", result: { content: expect.stringContaining("Report on uptime"), data: { structuredContent: { topic: "uptime", seconds: 30 } } } }); + expect(starts("slow_report") - before).toBe(1); + expect((await rows())[0]).toMatchObject({ status: "completed" }); + }); + + it("slow_report: outlives the timeout and a Paperclip restart, then is delivered once without starting again", async () => { + const ctx = await setup(); + const before = starts("slow_report"); + await expect(ctx.call("slow_report", { topic: "restart", seconds: 60 }, 400)).rejects.toMatchObject({ reasonCode: "provider_task_pending" }); + const fresh = ctx.restart(); + await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, ctx.run.id)); + const [row] = await ctx.rows(); + await db.update(toolMcpTasks).set({ nextPollAt: new Date(Date.now() - 1_000), leaseExpiresAt: null, leaseOwner: null }).where(eq(toolMcpTasks.id, row!.id)); + clock += 60_000; + const wake = vi.fn(async () => undefined); + await ctx.poll(wake); + expect(wake).toHaveBeenCalledTimes(1); + const [recovered] = await db.insert(heartbeatRuns).values({ companyId: ctx.org.id, agentId: ctx.agent.id, invocationSource: "assignment", status: "running", contextSnapshot: { issueId: ctx.issue.id } }).returning(); + const session = await fresh.createSession({ companyId: ctx.org.id, agentId: ctx.agent.id, runId: recovered!.id }); + expect(await ctx.call("slow_report", { topic: "restart", seconds: 60 }, 5_000, session.token)) + .toMatchObject({ status: "completed", result: { content: expect.stringContaining("Report on restart") } }); + expect(starts("slow_report") - before).toBe(1); + // Asking again is new work: the first result was delivered exactly once. + const [delivered] = await ctx.rows(); + expect(delivered!.deliveredAt).toBeInstanceOf(Date); + }); + + it("needs_approval: the question becomes a card, and the person's choice is sent with tasks/update", async () => { + const { call, org, issue } = await setup(); + const first = await call("needs_approval", { action: "rotate the demo key" }).catch((error) => error); + expect(first).toMatchObject({ reasonCode: "elicitation_required" }); + const [card] = await db.select().from(issueThreadInteractions).where(and(eq(issueThreadInteractions.companyId, org.id), eq(issueThreadInteractions.issueId, issue.id))); + expect(card).toMatchObject({ kind: "ask_user_questions", status: "pending" }); + await db.update(issueThreadInteractions).set({ status: "answered", result: { version: 1, answers: [ + { questionId: "decision", optionIds: ["approve"] }, + { questionId: "note", optionIds: [], otherText: "go ahead" }, + ] } }).where(eq(issueThreadInteractions.id, card!.id)); + const pending = call("needs_approval", { action: "rotate the demo key" }); + await vi.waitFor(() => expect(demoTask("needs_approval")?.status).toBe("working"), { timeout: 5_000 }); + clock += 3_000; + expect(await pending).toMatchObject({ status: "completed", result: { content: "Approved: rotate the demo key was carried out. Note: go ahead" } }); + }); + + it("will_fail: becomes a tool error with the app's message", async () => { + const { call } = await setup(); + const pending = call("will_fail", { seconds: 5 }).catch((error) => error); + await vi.waitFor(() => expect(polled("will_fail")).toBeGreaterThan(0), { timeout: 5_000 }); + clock += 5_000; + const failure = await pending; + expect(failure).toMatchObject({ reasonCode: "provider_task_failed" }); + expect(failure.message).toContain("crashed on purpose"); + }); + + it("cancellable: stopping the run cancels the task in the app", async () => { + const { call, run, rows } = await setup(); + const control = createAdapterExecutionControl(); + adapterExecutionControls.set(run.id, control); + try { + const pending = call("cancellable").catch((error) => error); + await vi.waitFor(() => expect(polled("cancellable")).toBeGreaterThan(0), { timeout: 5_000 }); + control.controller.abort(); + expect(await pending).toMatchObject({ reasonCode: "provider_task_cancelled" }); + } finally { + adapterExecutionControls.delete(run.id); + } + expect(demoTask("cancellable")?.status).toBe("cancelled"); + expect((await rows())[0]).toMatchObject({ status: "cancelled", errorCode: "run_cancelled" }); + }); + + it("expires: a task the app forgets ends with a clear error, never a hang", async () => { + const { call, rows } = await setup(); + const pending = call("expires", { ttlSeconds: 15 }).catch((error) => error); + await vi.waitFor(() => expect(polled("expires")).toBeGreaterThan(0), { timeout: 5_000 }); + clock += 15_000; + expect(await pending).toMatchObject({ reasonCode: "provider_task_expired" }); + expect((await rows())[0]).toMatchObject({ status: "expired" }); + }); + + it("quick_echo: an ordinary call stays synchronous", async () => { + const { call, rows } = await setup(); + expect(await call("quick_echo", { text: "hi" })).toMatchObject({ status: "completed", result: { content: "echo: hi" } }); + expect(await rows()).toEqual([]); + }); +}); diff --git a/server/src/__tests__/tool-gateway-agent-errors.test.ts b/server/src/__tests__/tool-gateway-agent-errors.test.ts new file mode 100644 index 00000000000..9fa066e5fe3 --- /dev/null +++ b/server/src/__tests__/tool-gateway-agent-errors.test.ts @@ -0,0 +1,51 @@ +import express from "express"; +import request from "supertest"; +import { describe, expect, it } from "vitest"; +import type { Db } from "@paperclipai/db"; +import { mcpGatewayProtocolRoutes, toolGatewayRoutes } from "../routes/tool-gateway.js"; +import { ToolGatewayHttpError, type ToolGatewayService } from "../services/tool-gateway.js"; + +/** + * A tool error carries the call's execution record (request IDs, timings, and + * the server-side stack of the error) so the audit can store it. Agents get + * the message and reason, never that record: they paste errors into task + * comments, where a person would read Paperclip's internals. + */ +function failingGateway() { + const error = new ToolGatewayHttpError(410, + "The app no longer has this action's result: it expired or was removed. Run the action again to start it over.", + "provider_task_expired", + { + connectionId: "connection-1", + execution: { + transport: "mcp_remote", + request: { requestId: "paperclip-tool-123" }, + error: { name: "Error", message: "expired", code: null, stack: "Error: expired\n at mcpTaskExpiredError (/srv/paperclip/server/src/services/tool-gateway.ts:6823:12)", cause: null }, + }, + }); + return { executeTool: async () => { throw error; } } as unknown as ToolGatewayService; +} + +describe("tool errors returned to agents", () => { + it("leave out the execution record on the MCP gateway", async () => { + const app = express().use(express.json()).use(mcpGatewayProtocolRoutes(failingGateway())); + const response = await request(app).post("/mcp/gateways/gateway-1").set("Authorization", "Bearer token") + .send({ jsonrpc: "2.0", id: 7, method: "tools/call", params: { name: "mcp.demo:expires", arguments: {} } }); + expect(response.body.error).toMatchObject({ + message: expect.stringContaining("Run the action again"), + data: { reasonCode: "provider_task_expired", connectionId: "connection-1" }, + }); + expect(response.body.error.data).not.toHaveProperty("execution"); + expect(JSON.stringify(response.body)).not.toContain("tool-gateway.ts"); + }); + + it("leave out the execution record on the session tool call route", async () => { + const app = express().use(express.json()).use(toolGatewayRoutes({} as Db, failingGateway())); + const response = await request(app).post("/tool-gateway/tools/call").set("x-paperclip-tool-gateway-token", "token") + .send({ tool: "mcp.demo:expires", parameters: {} }); + expect(response.status).toBe(410); + expect(response.body).toMatchObject({ reasonCode: "provider_task_expired", connectionId: "connection-1" }); + expect(response.body).not.toHaveProperty("execution"); + expect(JSON.stringify(response.body)).not.toContain("tool-gateway.ts"); + }); +}); diff --git a/server/src/routes/tool-gateway.ts b/server/src/routes/tool-gateway.ts index b73c834dbd4..4837ad62bfd 100644 --- a/server/src/routes/tool-gateway.ts +++ b/server/src/routes/tool-gateway.ts @@ -56,6 +56,16 @@ function callerHeaders(req: { headers: Record): Record { + const { execution: _auditOnly, ...rest } = details; + return rest; +} + async function cancellableRequest(req: Request, res: Response, work: (signal: AbortSignal) => Promise) { const controller = new AbortController(); const aborted = () => controller.abort(); @@ -244,7 +254,7 @@ async function handleMcpGatewayProtocol( res.status(err.status).json({ jsonrpc: "2.0", id, - error: { code: err.status >= 500 ? -32603 : -32000, message: err.message, data: { reasonCode: err.reasonCode, ...err.details } }, + error: { code: err.status >= 500 ? -32603 : -32000, message: err.message, data: { reasonCode: err.reasonCode, ...callerErrorDetails(err.details) } }, }); return; } @@ -340,7 +350,7 @@ function sendGatewayError(res: import("express").Response, err: unknown) { res.status(err.status).json({ error: err.message, reasonCode: err.reasonCode, - ...err.details, + ...callerErrorDetails(err.details), }); return; } From 5d0b27acca7a5dbae1db666df25237d174b52449 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 20:07:31 -0600 Subject: [PATCH 11/11] fix(run-events): watch both late appends for rejection as soon as they start (#21536) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server stores each run event through `appendHeartbeatRunEvent` and `appendHeartbeatRunEventBatch` > - Pull request #21511 added a test that deletes a run while two appends wait for its row lock > - Both appends must fail, but the test attached the rejection check of the second append only after it awaited the first > - When the second append failed first, its rejection had no handler, and Vitest failed the whole shard with one unhandled error > - This pull request attaches both rejection checks when the appends start > - The benefit is that server test shards on master and on unrelated pull requests pass again ## Linked Issues or Issue Description Refs #21511 **What happened?** Server test shards failed on master and on unrelated pull requests. Every test passed. Vitest reported one unhandled error: `Error: heartbeat_run_event_binding_mismatch`, from `server/src/__tests__/heartbeat-run-event-atomic-append.test.ts`. **Expected behavior** The test awaits both expected failures. No rejection stays without a handler. **Steps to reproduce** Run the general server test shard that contains `heartbeat-run-event-atomic-append.test.ts` on master at `cefef8f11d`. Examples: master "verify / General tests (server (1/10))", pull request #21443 "ci / General tests (server (9/12))", pull request #21505 "ci / General tests (server (1/12))". ## What Changed - In the test "waits for Stop on the run row and appends after it, and fails without rows after a run deletion", the two `expect(...).rejects` checks are now created immediately after the two appends start. The test awaits them after it releases the deletion. - No product code changed. The product code rejects correctly; the test left one rejection without a handler for a short time. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-run-event-atomic-append.test.ts`: 25 of 25 tests pass, three runs in a row, no unhandled error. - The failure before this change is the master shard named above. It depends on which append rejects first, so I did not get a local reproduction on demand. ## Risks Low risk. The change is in one test. The assertions are the same. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 --- .../src/__tests__/heartbeat-run-event-atomic-append.test.ts | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/server/src/__tests__/heartbeat-run-event-atomic-append.test.ts b/server/src/__tests__/heartbeat-run-event-atomic-append.test.ts index e7da91ca4b7..b805d99e9a6 100644 --- a/server/src/__tests__/heartbeat-run-event-atomic-append.test.ts +++ b/server/src/__tests__/heartbeat-run-event-atomic-append.test.ts @@ -322,11 +322,13 @@ describeEmbeddedPostgres("atomic ordinary run events", () => { }); const late = appendHeartbeatRunEventBatch(db, [4, 5].map(sourceSeq => nativeEvent(runId, sourceSeq))); const lateSingle = appendHeartbeatRunEvent(db, nativeEvent(runId, 4)); + const lateRejected = expect(late).rejects.toThrow("heartbeat_run_event_binding_mismatch"); + const lateSingleRejected = expect(lateSingle).rejects.toThrow("heartbeat_run_event_binding_mismatch"); const lateSettledWhileHeld = await settlesWhileHeld(late); await deletion.release(); expect(lateSettledWhileHeld).toBe(false); - await expect(late).rejects.toThrow("heartbeat_run_event_binding_mismatch"); - await expect(lateSingle).rejects.toThrow("heartbeat_run_event_binding_mismatch"); + await lateRejected; + await lateSingleRejected; expect(await readEvents(runId)).toEqual([]); });