Refresh the SSH account branch from master

Keep the SSH base current so the stacked Tailscale migrations can follow the
latest upstream schema sequence. The merge preserves managed SSH account
selection, host-auth compatibility, and the PATH-only ACP prerequisite check.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin FoleyandPaperclip committed 2026-11-13 09:03:27 -08:00
commit e8a0316483
262 files changed
+13201 -1135

No files matched your search

-15
View File
@@ -424,18 +424,3 @@ input file times are normalized in both stages so fresh checkouts do not force
Cargo to rebuild unchanged source. Changes made by build scripts still reach
Cargo's normal validation. The final application copy excludes Cargo's target
directory as before. Cache misses only cost compilation time.
Pull requests that change the Dockerfile, Docker ignore rules, or Runner native
inputs also build the isolated `runner-build` target in `Docker Runner check`.
The check runs `bash scripts/check-docker-runner-cache.sh` against a disposable
copy of tracked source and the actual Docker ignore rules. It compiles a baseline
and exports a local cache, removes that builder, changes a Rust metadata constant,
and rebuilds on a fresh builder using only the exported cache. It requires a
cached dependency build, an unchanged dependency recipe, and changed metadata
from the real binary. It also verifies that a dependency declaration change
alters the recipe. The probe exports small metadata results instead of importing
a large test image into the Docker daemon. Temporary builders and cache files
are removed afterward. It catches missing embedded inputs before the post-merge
build. It uses a GitHub-hosted runner with read-only repository access and never
publishes images or registry caches. Allow up to 20 minutes for its cold build and
source rebuild.
+24 -1
View File
@@ -108,6 +108,27 @@ It:
- fails by default if npm leaves `latest` pointing at a canary; use `--allow-canary-latest` only when that state is intentional
- creates a git tag `canary/vYYYY.MDD.P-canary.N`
#### Busy master and lost runners
Master often receives a commit every few minutes, faster than one canary
publishes (about 60-90 minutes). The release workflow keeps one running canary
and one pending canary. A newer push replaces the pending one, so the canary
that runs next is always the newest master commit. Commits in between are not
published on their own; they ship inside the next canary.
Each canary waits for `Cloud source verified v1` from the Cloud readiness run
of its exact SHA (up to 90 minutes). Cloud readiness runs on AWS Spot workers.
When an attempt fails only because its workers were reclaimed, the canary gate
and the master image watcher re-run that attempt's failed jobs instead of
failing the release
([`scripts/cloud-readiness-runner-loss.mjs`](../scripts/cloud-readiness-runner-loss.mjs)).
A job counts as lost when its log shows the runner's shutdown or lost
communication error. Retries continue up to attempt 4 of the readiness run. A
real check failure, including a flaky test, is never retried: that canary fails
and the next push publishes. If a canary is missing after a quiet period, look
at the failed `Reuse exact-source verification` job. It names the readiness run
and lists each failed job as a lost runner or a failure.
After publication, the onboarding smoke checks out the same source SHA and
installs its test dependencies from the publisher's resolved lockfile with
`--frozen-lockfile`. The same-run artifact is captured before tracked files are
@@ -316,7 +337,9 @@ latest attempt. A green admission-only run with skipped build and publish jobs
is pending work. The source observer retries only producers that were strictly
deferred, continues watching after build admission, and waits for publication.
The final publication step rechecks both proofs before it creates image tags.
Failed or cancelled producers require inspection and are not retried automatically.
Failed or cancelled producers require inspection and are not retried automatically,
with one exception: a Cloud readiness attempt that failed only because Spot
runners were reclaimed is retried (see "Busy master and lost runners" below).
[`docker.yml`](../.github/workflows/docker.yml) publishes the standard image:
+115 -19
View File
@@ -63,7 +63,7 @@ for `0.25`), so Honeycomb's `COUNT` and percentiles are weighted correctly.
Volume per sampled session: one `page.load` span per navigation, up to three
`web_vital.*` spans per page lifetime, and one span per same-origin `/api/`
request made after the SDK loads.
request made during a measured navigation or after the SDK loads.
## Load-speed cost
@@ -77,10 +77,22 @@ fetching, browser idle; `ui/src/lib/when-page-settles.ts`). If the page never
settles (for example a request hangs), the chunk loads 35 seconds after tracing
starts anyway, so a hard load that ended as `timeout` is still exported.
Because the SDK loads after the first page settles, API requests made during
the hard load have no fetch spans. The hard load's `page.load` span is still
recorded: measurements taken before the SDK loads are buffered and exported
when it arrives.
Because the SDK loads after the first page settles, it cannot trace the hard
load's API requests live. Instead the main-bundle controller registers one
`PerformanceObserver` for `resource` entries and keeps only this
origin's `/api/` `fetch`/XHR entries, at most 500. `buffered: true` delivers
the requests that finished before tracing started (it starts once the session
request returns). After that, the observer receives every entry even when the
browser's resource buffer (250 entries by default) is full; only entries that
overflowed that buffer before tracing started are missed. When the SDK loads it turns
those entries into child spans of the `page.load` they belong to (see
[Backfilled API spans](#backfilled-api-spans)). The observer stops and the
entries are dropped once no navigation that started before the SDK loaded can
still claim them: every such navigation has ended, and either a navigation
that started after the SDK loaded has begun or 60 seconds have passed (a
navigation whose history change came before the SDK can commit after it).
Measurements taken before the SDK loads are buffered and exported when it
arrives.
## What "loaded" means
@@ -112,10 +124,35 @@ it renders to `PageLoadTracker`), so a new page is measured without page code.
changes are not navigations. A history replace while a navigation is still
open is a redirect and continues it.
Outcomes: `ready`; `interrupted` (another navigation started first); `timeout`
(not ready after 30 seconds). A navigation open when the tab closes is not
sent. `was_hidden = true` means the tab was hidden during the measurement;
browsers pause rendering in hidden tabs, so exclude these from SLOs.
Outcomes:
- `ready`: loaded, as defined above.
- `interrupted`: another navigation started first.
- `hidden`: the tab was hidden when the navigation started (a link opened in
a background tab) or became hidden before the page was ready. The span ends
at the moment the tab hid, so a load in a background tab has a duration of
about 0. Browsers do not paint hidden tabs, so such a load could never
become ready; this is the same rule `web-vitals` applies to LCP and FCP.
For a hard load, Chromium's `visibility-state` performance entries give the
time the tab first hid, even before tracing started; elsewhere a tab that is
hidden when tracing starts counts as hidden from navigation start. A
redirect while the tab is still hidden renames the hidden span rather than
starting another one. `was_hidden` is `true` on these spans.
- `timeout`: not ready 30 seconds after it started. The span always ends at
exactly start + 30 s, however late the browser ran the timer. Any end
observed after that deadline (for example a page that only became ready
after the computer woke up) is also reported as this timeout.
`paperclip.page.suspended = true` marks a span during which the computer slept
or the tab was frozen: the observed end was more than 5 seconds past the
30 s deadline, or two consecutive animation frames were more than 5 seconds
apart. Its duration does not describe the page; exclude it like `hidden`. A
suspend that ends before the next frame is not detectable, but it can never
stretch a span beyond 30 s.
A navigation open when the tab closes is not sent. For load-time percentiles
and SLOs, filter on `outcome = ready`; that excludes hidden tabs, timeouts and
interruptions, which you count separately.
Pages with an explicit content signal:
@@ -126,6 +163,23 @@ Pages with an explicit content signal:
An agent chat with no saved task has no content signal and uses the idle rule.
### Page views
Some pages show a different list depending on `?view=`. For those,
`paperclip.page.view` records the view (`ui/src/lib/browser-tracing/page-view.ts`),
read from the URL when the span ends, so it is the view the page resolved and
wrote into the URL. Only names from the code's view registries are sent; any
other value is `other`.
| Route | Values |
| --- | --- |
| `/:companyPrefix/issues` | The keys of `TASK_VIEW_KEYS` in `ui/src/lib/task-views.ts` (`mine`, `unread`, `blocked`, `recent`, `everything`, `needs_attention`, `all`, `active`, `backlog`, `done`), `default` when the URL has no view yet, else `other` |
| `/:companyPrefix/decisions` | `ATTENTION_PERSONAL_VIEWS` (`mine`, `unassigned`, `company`, `archived`, `all`), `mine` when the URL has no view, else `other` |
Switching views on the same page changes only the query string, so it is not a
new navigation and has no `page.load` span; the attribute describes the view a
navigation landed on.
To add one, call `usePageContentReady(ready)` in the page component. Use it
only when the idle rule is inaccurate for that page.
@@ -140,6 +194,30 @@ URLs, server addresses, status messages, and event attributes, is dropped.
| `page.load` | One per hard load or client navigation. |
| `web_vital.lcp`, `web_vital.inp`, `web_vital.cls` | Core Web Vitals, from Google's `web-vitals`, usually reported as the tab hides. Zero duration; the value is an attribute. |
| `GET`, `POST`, … | A `fetch` to this origin's `/api/`. Its `traceparent` header joins the server's request span to the same trace. While a `page.load` span is open it is the parent, so one trace shows the navigation, its API calls, and (when server tracing is on) their server spans. |
| `HTTP` | A request to this origin's `/api/` made before the SDK loaded, during a measured navigation (in practice the hard load), rebuilt from Resource Timing. See below. |
### Backfilled API spans
The hard load's API requests happen before the SDK exists. When the SDK loads,
it creates one `HTTP` child span under that load's `page.load` for each
same-origin `/api/` `fetch`/XHR Resource Timing entry that started inside the
load's window (and before the SDK started tracing requests live, so nothing is
counted twice). Each runs from the entry's `startTime` to its `responseEnd`.
They differ from live fetch spans in three ways:
- The name is `HTTP`, OpenTelemetry's name for a request whose method is
unknown: Resource Timing does not expose the method, so there is no
`http.request.method`.
- `http.response.status_code` is present only where the browser exposes
`responseStatus` (Chromium); it is never guessed.
- No `traceparent` was sent with these requests, so they do not join the
server's spans. Use `paperclip.api.area` and the timings; for the server
side, query server spans by time.
They carry `paperclip.api.timing_source = resource_timing`. The URL is reduced
to `paperclip.api.area` by the same sanitizer as live fetch spans.
### Attributes
Attributes (the complete allowlist):
@@ -147,16 +225,19 @@ Attributes (the complete allowlist):
| --- | --- | --- |
| `paperclip.page.route` | all | Route pattern, for example `/:companyPrefix/issues/:issueId`; `(unmatched)` when no route matches. Built only from route definitions, never from the URL. On fetch spans: the route current when the request started. On `web_vital.lcp`: the hard load's route. |
| `paperclip.page.load_type` | `page.load` | `hard`, `client` |
| `paperclip.page.outcome` | `page.load` | `ready`, `interrupted`, `timeout` |
| `paperclip.page.outcome` | `page.load` | `ready`, `interrupted`, `hidden`, `timeout` |
| `paperclip.page.ready_signal` | `page.load` | `content`, `idle`, `none` |
| `paperclip.page.was_hidden` | `page.load` | boolean |
| `paperclip.page.suspended` | `page.load` | `true` when the computer slept or the tab froze during the measurement; absent otherwise |
| `paperclip.page.view` | `page.load` on pages with views | The view from `?view=`, from a fixed list per page, else `other`; absent on other pages |
| `paperclip.page.navigation_type` | hard `page.load` | `navigate`, `reload`, `back_forward`, `prerender` |
| `paperclip.page.ttfb_ms`, `paperclip.page.dom_content_loaded_ms`, `paperclip.page.load_event_ms` | hard `page.load` | Navigation Timing milestones, ms from navigation start |
| `paperclip.web_vital.name` | `web_vital.*` | `LCP`, `INP`, `CLS` |
| `paperclip.web_vital.value` | `web_vital.*` | ms for LCP and INP; unitless for CLS |
| `paperclip.web_vital.rating` | `web_vital.*` | `good`, `needs-improvement`, `poor` |
| `paperclip.api.area` | fetch | First path segment after `/api/` when it is a fixed word (`issues`, `companies`, …), else `other`. Use the joined server span's route for detail. |
| `http.request.method`, `http.response.status_code`, `error.type` | fetch | OpenTelemetry HTTP semantics |
| `paperclip.api.timing_source` | backfilled `HTTP` | `resource_timing` |
| `http.request.method`, `http.response.status_code`, `error.type` | fetch | OpenTelemetry HTTP semantics (backfilled spans: status only, where the browser exposes it) |
| `SampleRate` | all, when sampling | 1-in-N weight |
Resource: `service.name`, `service.version` (the UI build commit),
@@ -176,17 +257,20 @@ when server tracing is enabled ([observability.md](observability.md#enabling-tra
## Example Honeycomb queries
Use the `paperclip-web` dataset (or your `service.name`) and the environment
that holds browser data. Exclude hidden tabs and unfinished loads.
that holds browser data. Load-time percentiles filter on `outcome = ready`,
which leaves out hidden tabs, timeouts, and interruptions. Also leave out
suspended spans: `paperclip.page.suspended` is only ever set to `true`, so
filter on it not existing.
**P95 load time of the all-tasks page**
**P95 load time of the all-tasks page, by view**
```text
VISUALIZE P95(duration_ms), COUNT
WHERE name = page.load
paperclip.page.route = /:companyPrefix/issues
paperclip.page.outcome = ready
paperclip.page.was_hidden = false
GROUP BY paperclip.page.load_type
paperclip.page.suspended does-not-exist
GROUP BY paperclip.page.load_type, paperclip.page.view
```
**P95 load time of an individual task page**
@@ -196,12 +280,22 @@ VISUALIZE P95(duration_ms), COUNT
WHERE name = page.load
paperclip.page.route = /:companyPrefix/issues/:issueId
paperclip.page.outcome = ready
paperclip.page.was_hidden = false
paperclip.page.suspended does-not-exist
GROUP BY paperclip.page.load_type
```
**API requests during hard loads of the all-tasks page**
```text
VISUALIZE P95(duration_ms), COUNT
WHERE paperclip.api.timing_source = resource_timing
paperclip.page.route = /:companyPrefix/issues
GROUP BY paperclip.api.area
```
Group by `service.version` to compare builds. For a timeout rate, count
`paperclip.page.outcome = timeout` against all `page.load` spans for the route.
`paperclip.page.outcome = timeout` against `page.load` spans for the route
whose outcome is not `hidden`.
For Web Vitals, `P75(paperclip.web_vital.value)` where
`name = web_vital.lcp` (or `inp`, `cls`), grouped by `paperclip.page.route`.
@@ -211,11 +305,13 @@ For Web Vitals, `P75(paperclip.web_vital.value)` where
```text
IF(AND(EQUALS($name, "page.load"),
EQUALS($paperclip.page.route, "/:companyPrefix/issues/:issueId"),
EQUALS($paperclip.page.was_hidden, false)),
IN($paperclip.page.outcome, "ready", "timeout"),
NOT(EQUALS($paperclip.page.suspended, true))),
AND(EQUALS($paperclip.page.outcome, "ready"), LTE($duration_ms, 2000)))
```
The expression is null for other events, so they do not count.
The expression is null for other events (including `hidden` and
`interrupted` loads), so they do not count; a timeout counts as a failure.
## Verifying a deployment
+45 -11
View File
@@ -62,7 +62,10 @@ listing. See [CONNECTION-SEARCH.md](./CONNECTION-SEARCH.md).
| Agent tool list | Tools whose visibility omits `"model"` are not listed to agents and cannot be called by them. | `isModelVisibleTool` in `server/src/services/tool-gateway.ts` |
| Recording | Each successful call to a tool with a view stores one `tool_mcp_app_views` row: issue, run, agent, connection, the redacted arguments and the redacted `CallToolResult`. The tool result `_meta` is kept here and never sent to the agent. Recording is best effort and never fails the agent's call. | `server/src/services/mcp-app-views.ts` |
| API | `GET /api/issues/:issueId/mcp-app-views` lists a task's views. `GET …/:viewId` returns the view, reading the `ui://` document live through the agent's connection. `POST …/:viewId/tool-calls` runs a view's `tools/call`. Reading follows task read access. Tool calls also need the `tools:use` or `tools:manage_connections` permission, and only reach tools of the same connection that allow `"app"`. They go through the same policy, Ask-first approval and audit path as a connection test call. | `server/src/routes/mcp-apps.ts` |
| Chat | A run's views render with that run's reply, outside the folded tool history, under a "Widget from <app>" label. Views are keyed by the reply's run, so settled history needs no transcript. A task's views load in one request and are indexed by run once. | `ui/src/components/mcp-app/`, `TaskChatRunnerTurn.tsx` and `TaskChatThreadView.tsx` |
| Chat | A run's views render with that run's reply, outside the folded tool history, under a "Widget from <app>" label with an "Open full screen" control. Views are keyed by the reply's run, so settled history needs no transcript. A task's views load in one request and are indexed by run once. | `ui/src/components/mcp-app/`, `TaskChatRunnerTurn.tsx` and `TaskChatThreadView.tsx` |
| View messages | A view's `ui/message` shows a card under the widget: "<App> wants to send: …" with Send and Dismiss. Nothing is sent until the viewer clicks Send. The text is then sent as the viewer's own comment through the chat's normal send path, exactly like a typed message. `POST …/:viewId/messages` then records `issue.mcp_app_message_sent` for that comment; it accepts only the caller's own comment on the same task from the last 10 minutes. | `McpAppView.tsx`, `server/src/routes/mcp-apps.ts` |
| View context | A view's `ui/update-model-context` is stored per view (`tool_mcp_app_views.model_context`, 16 KB at most) by `PUT …/:viewId/model-context`. Identical updates are deduped in the browser and on the server; a change logs `issue.mcp_app_context_updated`; an empty update clears it. The task's next run receives the three most recently updated contexts as an "App context (untrusted)" section of its wake prompt, quoted as JSON data and labelled as third-party data, not instructions. It is read in parallel with the other run preparation and never fails the run. | `server/src/services/mcp-app-views.ts`, `heartbeat/execution-configuration.ts`, `renderPaperclipWakePrompt` |
| Tool icons | Provider `icons` (SEP-973) on MCP tools are sanitized into `tool_catalog_entries.icons`, outside the version hash. Task chat tool rows and the app's actions list show them, falling back to the usual icons. | `server/src/services/tool-access/catalog/`, `ui/src/components/ProviderToolIcon.tsx` |
### Sandbox
@@ -87,26 +90,57 @@ until the view reports `initialized`.
The host answers view requests as follows:
- `ui/open-link` opens only `http(s)` URLs, in a new tab without an opener.
- `ui/message` is refused (method not found), because posting to the task on
the user's behalf needs its own consent flow.
- `ui/update-model-context` is accepted but not yet sent to the agent.
- Display mode is always `inline`.
- `ui/message` waits for the viewer's decision: the view hears success only
after the comment was sent, and "Message sending denied" when the viewer
dismissed it. One proposal per view can wait at a time; further ones are
refused until it is settled. Only text from the `user` role is accepted, up
to 8,000 characters.
- `ui/update-model-context` is applied in order and deduped. Text blocks and
`structuredContent` are kept; other content is dropped.
- `ui/request-display-mode` honors the view's declared `availableDisplayModes`.
The host offers `inline` and `fullscreen` (no picture-in-picture) and answers
with the resulting mode. Full screen is a Paperclip dialog; switching modes
starts the view again in its new place, since a moved iframe always reloads.
The view gets `host-context-changed` with the new mode and its container
(a fixed `height` and `width` in full screen, `maxHeight` inline) and again
when the full-screen container resizes.
- `message` and `updateModelContext` are advertised in `hostCapabilities` only
where the viewer can send messages on the task. Without that, `ui/message`
is refused (method not found) and context updates are accepted and dropped.
- Viewers who cannot comment on the task, agents and API keys cannot send a
view's messages or set its context.
### Tool icons
Icons are untrusted provider data, so they are sanitized when the catalog is
refreshed and again before rendering:
- Only `https:` URLs and `data:image/*` URIs are kept. SVG is accepted only as
a `data:` URI, never from a remote URL, so it cannot change after review.
- A `data:` icon is at most 32 KB; a URL at most 2,048 characters; at most
four icons per tool. Unknown mime types, sizes and themes are dropped.
- Icons are always drawn as `<img referrerPolicy="no-referrer">` at a fixed
size. Markup is never inlined, so an SVG's scripts never run. A broken icon
falls back to the taxonomy icon.
## Testing plan
| Level | What it proves | Command or location |
| --- | --- | --- |
| Shared unit | `_meta.ui` parsing (current and legacy keys, visibility, non-`ui://` rejected), resource parsing (text and base64, mime check), CSP sanitizing | `pnpm exec vitest run packages/shared/src/mcp-apps.test.ts` |
| UI unit | Host protocol: initialize result, nothing sent before `initialized`, tool input before tool result, `tools/call` proxy and errors, link and method refusals, teardown. CSP placed before the view's markup. The run index at a realistic size. | `pnpm exec vitest run ui/src/components/mcp-app/` |
| Server integration (embedded Postgres) | Circleback catalog OAuth (DCR) against a fake Circleback server. `ui_meta` stored. An app-only tool is hidden from the agent. An agent call records one view with input, structured content and `_meta`, and a call without a view records none. Routes list the view and serve its document and CSP. A view's tool call works, and a model-only tool is refused. Other tasks, other companies and agent actors are refused. | `pnpm exec vitest run server/src/__tests__/mcp-apps.test.ts` |
| Storybook | The view with the agent's reply, the ready view (Refresh calls a tool), loading, view unavailable, and a tool call that needs approval. The fixture view follows the protocol by hand. | `Tasks/MCP App views` (`ui/storybook/stories/tasks/task-chat-mcp-app.stories.tsx`) |
| UI unit | Host protocol: initialize result, nothing sent before `initialized`, tool input before tool result, `tools/call` proxy and errors, link and method refusals, teardown. `ui/message` waits for the viewer and allows one proposal at a time; context updates are ordered, deduped and bounded; display modes honor the view's declaration; capabilities are advertised only when implemented. The consent card and the full-screen dialog. CSP placed before the view's markup. The run index at a realistic size. | `pnpm exec vitest run ui/src/components/mcp-app/` |
| Server integration (embedded Postgres) | Circleback catalog OAuth (DCR) against a fake Circleback server. `ui_meta` stored. An app-only tool is hidden from the agent. An agent call records one view with input, structured content and `_meta`, and a call without a view records none. Routes list the view and serve its document and CSP. A view's tool call works, and a model-only tool is refused. Other tasks, other companies and agent actors are refused. A view's context is stored, deduped, bounded, logged, cleared, and rendered for the task's next run as untrusted data. A view's message is attributed only to the viewer's own recent comment on the same task. | `pnpm exec vitest run server/src/__tests__/mcp-apps.test.ts` |
| Storybook | The view with the agent's reply, the ready view (Refresh calls a tool), loading, view unavailable, a tool call that needs approval, a view asking to send a message, full screen, and tool rows with provider icons. The fixture view follows the protocol by hand. | `Tasks/MCP App views` (`ui/storybook/stories/tasks/task-chat-mcp-app.stories.tsx`) |
| Live end to end | Connect Circleback with a real account and ask an agent for a recap of a recent meeting. Check that the chat shows the meeting cards and that clicking a card opens Circleback. Done on 2026-11-12. The run found the view-size and placement bugs. | Manual, local `pnpm dev` |
## Limits and follow-ups
- Only `inline` display mode is supported. There is no fullscreen or
picture-in-picture yet.
- A view's `ui/message` and `ui/update-model-context` do not reach the agent yet.
- Picture-in-picture is not supported. Switching between inline and full
screen restarts the view, so state inside the widget resets.
- A view's message is attributed in the activity log by a second request after
the comment is sent. If that request fails, the comment stays, without the
attribution.
- Icons are shown for tools only, not yet for MCP resources or prompts.
- Views are rendered in the task chat only, not in other transcript surfaces.
- A view document can be at most 4 MB of HTML. Circleback's is about 3.5 MB.
- Local stdio MCP servers are recorded the same way, but no stdio provider with
+133
View File
@@ -0,0 +1,133 @@
# MCP Tasks: long provider calls that survive timeouts, restarts, and deploys
Some provider actions take minutes: a CI run, a deploy, a batch job. An MCP
server that supports **Tasks** answers such a `tools/call` with a durable task
handle instead of the result. Paperclip stores that handle, waits on it, and
hands the agent the final result exactly as a quick call would return it. If
the call outlives its timeout, or Paperclip restarts or deploys mid-call, the
work is not lost and is never started a second time.
## Turning it on
MCP Tasks are behind the experimental setting **Durable app actions (MCP
Tasks)** (`enableMcpTasks`), off by default. Find it under Instance settings →
Experimental.
- **Off:** Paperclip never declares `io.modelcontextprotocol/tasks`, so servers
never return a task and every call behaves as before. Nothing polls, attaches
to, or wakes for stored tasks, and they do not keep an idle host awake.
- **On:** everything below applies. Turning it off later stops new tasks;
stored ones resume if it is turned on again before they expire.
## The standard
- Specification: [MCP Tasks extension, 2026-07-28](https://github.com/modelcontextprotocol/ext-tasks)
([overview](https://modelcontextprotocol.io/extensions/tasks/overview)).
- Extension id: `io.modelcontextprotocol/tasks`. It is defined for stateless MCP
(2026-07-28), so Paperclip uses it only with servers on that version. See
[MCP-PROTOCOL-VERSIONS.md](./MCP-PROTOCOL-VERSIONS.md).
- Input requests use the multi round-trip shape
([MRTR](https://modelcontextprotocol.io/specification/2026-07-28/basic/patterns/mrtr)).
The protocol works like this:
1. **The client declares the extension per request** in
`_meta["io.modelcontextprotocol/clientCapabilities"].extensions`. A server
must never return a task to a request that did not declare it.
2. **`tools/call` may return a task** (`resultType: "task"`, with `taskId`,
`status`, `ttlMs`, `pollIntervalMs`). The server decides per call.
3. **`tasks/get {taskId}`** returns the task: `working`, `input_required` (with
`inputRequests`), `completed` (with `result`, the normal tool result),
`failed` (with a JSON-RPC `error`), or `cancelled`.
4. **`tasks/update {taskId, inputResponses}`** answers input requests.
5. **`tasks/cancel {taskId}`** asks the server to stop. Cancellation is cooperative.
6. Over HTTP, `Mcp-Name` carries the task ID on every `tasks/*` request so load
balancers route it to the instance holding the task.
Optional `notifications/tasks` over `subscriptions/listen` are not used;
Paperclip polls, which the specification makes the default.
## How Paperclip implements it
| Step | What happens | Code |
| --- | --- | --- |
| Declare | Only the HTTP `tools/call` and `tasks/*` requests declare the extension. Discovery, listings, and local (stdio) servers do not, so no other path can receive a task it cannot handle. | `mcpHttpRequestInit(..., { declareTasks })` in `server/src/services/mcp-http.ts` |
| Record | A returned task is stored before Paperclip waits on it: a `tool_mcp_tasks` row linked to the call, run, task, connection, and grant. | `createMcpTaskRecord` in `server/src/services/tool-gateway.ts` |
| Wait | The live call polls `tasks/get` at the server's `pollIntervalMs` (kept between 0.2 s and 5 s) until the normal tool timeout. A completed task's `result` then goes through the same validation, redaction, and recording as a direct result. | `waitForMcpTask` |
| Hand off | If the timeout passes or the caller disconnects, the agent is told the app is still working and to call the tool again with the same arguments. The row keeps the task and background polling takes over. | `provider_task_pending` |
| Attach | A call with the same endpoint, tool, full arguments, grant, agent, task, and actor attaches to the undelivered task instead of sending `tools/call` again. So does a retry with the same idempotency key, including after a crash mid-wait; the same key with other arguments is refused and leaves the original retry intact. One holder at a time (a lease): a second simultaneous caller is told to call again. | `findAttachableMcpTask`, `mcpTaskForKeyedRetry`, `claimMcpTask` |
| Deliver | A completed task counts as delivered only once its result is recorded on the call. A crash in between leaves it collectable again. | `markMcpTaskDelivered` |
| Restart | Background work reads pending rows at startup and polls each at its next deadline, with each request bounded to 30 s. It rechecks the same access a live call needs (grant, audience, delegation). When a task settles, or asks a new question, after its run has ended, the assignee is woken once; the agent's repeated call collects the result or shows the question. | `pollDueMcpTasks`, `server/src/services/mcp-task-work.ts` |
| Input | URL requests show the provider link, as for a direct request; Paperclip never answers them for the person. Form requests become a question card on the task; only the person's answer is sent with `tasks/update`. Other requests (sampling, roots) stop with a clear "needs input" error. | `planMcpTaskInput` |
| Cancel | Stopping the run sends `tasks/cancel` and marks the row cancelled. Closing the task or stopping its run does the same for a task in the background; a later call reports the cancellation and never revives it. A caller that only disconnects keeps the task. | `cancelMcpTaskRecord`, `mcpTaskStopReason` |
| Finish | `failed` becomes a tool error with the provider's message. An unknown task (`-32602`) or a passed TTL becomes a clear error telling the agent to run the action again. Nothing waits forever: every task stops being tracked after its TTL, and seven days after Paperclip first saw it at most. | `provider_task_failed`, `provider_task_expired` |
### What is stored
- The provider's task ID is sealed at rest (AES-256-GCM, a key derived from the
instance's tool-action secret, bound to the company and connection). The
specification lets servers treat task IDs as bearer tokens. Uniqueness uses a
keyed hash, never the ID.
- No provider result is stored in the task row. The result stays with the
provider until it is delivered, and the delivered result is recorded on the
call exactly as a direct result is.
- A task is read only with the grant that created it, after the same access
check a live call makes. If that access is removed, the connection is
disabled, or its endpoint changes, background polling stops and the task is
reported as no longer available.
- A provider's error body is read only up to 64 KiB.
### Agent-facing errors
| Reason code | When | What the agent should do |
| --- | --- | --- |
| `provider_task_pending` | Still working at the timeout, or the caller left | Call the same tool again with the same arguments |
| `provider_interaction_required` | The app needs the person to open a link | Wait for the person, then call again |
| `elicitation_required` | The app asked a question; a card is on the task | Wait for the answer, then call again |
| `provider_input_required` | The app needs input Paperclip cannot collect | Ask the person to finish in the app |
| `provider_task_failed` | The app reported a failure | Treat as a tool error |
| `provider_task_cancelled` | The app cancelled, or the run was stopped | Run again if still needed |
| `provider_task_expired` | The app no longer has the task | Run the action again |
## Background work
`registerMcpTaskWork` uses the shared deadline scheduler
(`registerDeadlineWork`). It is quiet when nothing is pending:
- **Trigger:** writing or releasing a task row notifies the `mcp-tasks` queue;
startup runs once and rebuilds from the rows.
- **Deadline:** the earliest `next_poll_at`, never earlier than a live call's lease.
- **Cadence:** the server's `pollIntervalMs`, kept between 5 s and 5 min, with
backoff after failures. A task waiting for a person is checked every 5 min:
the person's answer wakes the agent, whose call checks immediately.
- **Quiet:** once every task is delivered, settled and woken, cancelled, or
expired, no row has a next poll and no query or provider request runs.
This is the narrow exception in `rules/sleep-aware-background-work.mdc`: the
provider offers no wake Paperclip can receive, so an active task is polled.
Polling stops with the task. A pending task is an idle-sleep blocker
(`tool_mcp_tasks` in `idle-sleep-safety.ts`), so a host that sleeps when idle
stays awake until every task is settled. Recovery runs even when heartbeat
scheduling is off; only the agent wake needs the scheduler.
An approved action whose task outlives the wait keeps its approval: the action
returns to approved, and its next execution attaches to the same task.
## Follow-ups
- `notifications/tasks` over `subscriptions/listen`, to replace polling where
servers support it.
- Sampling and roots input requests.
- Showing a long-running provider action's progress (`statusMessage`) in the
task chat.
- Validating question-card answers against the provider's form schema (types,
required fields) and letting the person correct them before `tasks/update`.
## Tests
`server/src/__tests__/mcp-tasks.test.ts` runs a fake 2026-07-28 server that
returns tasks. It covers polling to completion, a timeout followed by an
attaching call, a keyed retry, a restart mid-task (fresh gateway, no second
`tools/call`, one wake, one delivery), URL and form input, cancellation by the
run and by closing the task, failure, an unknown task, TTL expiry, another
agent never attaching, and synchronous calls left unchanged.
+4 -3
View File
@@ -17,9 +17,10 @@ Runtime authentication: [AI Connections](./AI-CONNECTIONS.md).
MCP protocol versions (2026-07-28 stateless MCP and older handshake servers):
[MCP-PROTOCOL-VERSIONS.md](./MCP-PROTOCOL-VERSIONS.md).
MCP extensions: [MCP Apps](./MCP-APPS.md) (interactive views in the task chat)
and [MCP Skills](./MCP-SKILLS.md) (skills that connected apps publish, reviewed
by the board and delivered to agents).
MCP extensions: [MCP Apps](./MCP-APPS.md) (interactive views in the task chat),
[MCP Skills](./MCP-SKILLS.md) (skills that connected apps publish, reviewed
by the board and delivered to agents), and [MCP Tasks](./MCP-TASKS.md) (long
provider calls that survive timeouts, restarts, and deploys).
Long-term memory: [Experimental memory connectors](./MEMORY.md).
+28
View File
@@ -1648,6 +1648,8 @@ timeout retries still require a fresh allocation under the policy above.
A person's task Retry on a connected task first asks the chat retry owner to repeat the exact failed chat request. When the latest run is that person's own saved board turn that cannot be certified for replay, when no chat retry owner is available, or when that owner refuses (`chat_failed_run_retry_not_authorized`), the refusal and any writes it made are rolled back and the same Retry receipt becomes an ordinary board continuation: the clicking person is the responsible user, the run starts a fresh session from saved work, and no chat action, chat authority, or output replay is created. This is the same authority as a new message from that person. When the conversation still holds saved or unadmitted chat input for the task, Retry keeps the exact-replay path and its refusal, because that input continues through its own admission. Dispatch keeps every normal gate (current task access, executor containment, recovery holds, unadmitted chat input). Projection and the route therefore agree: a shown Retry on a connected task is accepted. Retry that Paperclip requests by itself on a person's behalf (restart recovery, provider limit reset) keeps the exact-replay requirement and still settles on the refusal. A budget resume continues the person's own Retry, so it takes the board continuation too. No migration is needed; tasks blocked by an earlier release become retryable at the next click.
A task whose run was interrupted and whose fresh continuation is recorded reads as continuing (`working`, no Retry) while the continuation waits to be delivered and while it waits in the agent's queue. Once the delivered continuation run has waited in the queue for 15 minutes, the task also offers Retry with `retryAction: "move_forward"`, labelled "Move it forward now". The Retry names that queued run as its observed run, so admission joins it to the run (`coalesced`) and starts no second run; a run that a person joined starts ahead of system recovery work. The offer ends once a person has joined the run or the run starts. The same action is offered while the continuation is recorded but not yet delivered (`continuationDelivery: "pending"`) once it has waited 15 minutes since it was recorded. The Retry then names the interrupted run as its observed run, and admission delivers the recorded continuation as the one run, with its recorded isolation; the delivery pass afterwards finds nothing left to start. When the continuation is delivered, the task moves to another run, or the system gives up the delivery between the offer and the click, the click starts no run (`stale` or `alreadyMovedOn`). No move-forward is offered for a continuation recorded without a delivery request (`not_requested`), when a person's Retry for the interrupted run already waits, or when the task's assignee changed; a held restore keeps its own blocker.
The exact replay can also be refused after the click is accepted. This happens when chat source authorization was deferred by a busy identity or endpoint lock, or when it later fails, so the chat retry action settles `failed` with no admitted run. The task Retry receipt is then not failed. If it is a person's Retry, its source run is still the latest run, its chat wake has no run, and no saved or unadmitted chat input remains, the same receipt drops its chat action and becomes the board continuation described above, recorded as `issue.retry_continued_from_saved_work`. Saved chat input whose source comment was written no later than the failed run's stop boundary (the user's recorded Stop request, otherwise `finishedAt`, the same boundary admission uses) does not block this: admission holds such input as `message_predates_stop`, so it can never start alone. The continuation adopts it (the input receipt is coalesced into the Retry receipt) and the task continues from saved work. Saved input that arrived after the stop keeps its own admission path, and the Retry still settles as failed. When the latest run is an admission that the recovery hold cancelled before dispatch (`execution_reconciliation_required`), the saved work that counts is the recovered work of the run whose recovery action held it. The deferred-receipt worker dispatches it with every normal gate. Retry that Paperclip requests by itself still settles as failed. When a failed run's saved work is recovered, the task's execution blocker offers Retry (also on a connected task) and says "Saved work recovered. Retry continues from it." A stopped (cancelled) run keeps "Send a message to continue."
### Saved chat Retry preparation
@@ -1843,6 +1845,13 @@ mid-turn can leave provider or tool processes that the fence cannot find, so it
keeps the fail-closed path below, and the task continues in a fresh session from
the saved work.
On every `reattach_remote_runner` attempt, a turn the Runner reports as
`provider_turn_lost_on_restore` with `recoverable: true` continues once in the same
conversation, as on a local dead-runner resume. The Runner reports that cause only
after it proved the old process ended and restored the conversation. If the turn
still ends the run, the run keeps `native_provider_terminal_failed`, and the safe
replacement lane continues the task.
The first unanswered read can come on the last of the three provider attempts,
for example when a server restart used the second attempt. The run then gets one
more attempt for the replacement only: the failure stays retryable, the claim
@@ -6016,6 +6025,25 @@ cannot. Process verification and the final state read share one recovery
deadline. A timeout, command failure, transport failure or invalid saved JSON on
that final read keeps its closed recovery stage and reason in the diagnostic.
A Runner that no controller reaches before its connection lease expires, or
before its reconnect grace or deadline ends, stops its provider, saves its state
with lifecycle `recoverable_failure` and `lease_expired_requires_bootstrap`,
`transport_reconnect_grace_exceeded` or `transport_reconnect_deadline_exceeded`,
and exits. Only a new process with a fresh bootstrap ticket can reconnect, and
that process continues from the same cursor. A reattachment that finds this exit
re-reads the state after the process is proven gone. When it has the same six
identity fields, the same lifecycle and code, and no pending terminal delivery
or warm transition, the reattachment launches a Runner from that state, even
mid-turn and with events still in the outbox. Sandbox preparation reads that
state once more and keeps it in place, as it does for an adopted live Runner;
it does not require a `suspended` state or an idle provider checkpoint for this
exit. The new Runner restores the
conversation: it continues the turn, or reports the lost turn as recoverable
after it proves that the old provider process ended. This path admits the same
attempts as a reattachment, including the extra attempt of an unanswered-Runner
replacement. On staging, a deploy that kept a reattachment from reconnecting for
more than the lease ended these runs with `runner_remote_process_identity_unavailable`.
Reattachment keeps the complete durable PRP binding, including its turn and item
attachment IDs. A healthy warm handoff can create new attachment IDs within the
retained Runner. Replacing that run's controller must preserve those IDs; deriving
@@ -40,6 +40,7 @@ Remedies below mean completing the owning protocol or supplying a separately tes
| `agents.lifecycle` | [agents](../../packages/db/src/schema/agents.ts); [../modules/agent-lifecycle/index.ts](../../server/src/modules/agent-lifecycle/index.ts). active / recovery. | Preparing/verifying/pausing/resuming/terminating/cleaning-up phases. Fixture: Pausing agent versus stable paused state. | Domain settlement; preserve the owning evidence and records. |
| `decision_invocations` | [decision_invocations](../../packages/db/src/schema/decision_models.ts); [decision-models.ts](../../server/src/services/decision-models.ts). active. | Running decision invocation. This existing predicate is narrower than a universal unknown-state guard. Fixture: Running invocation versus completed invocation. | Domain completion; preserve the owning evidence and records. |
| `prompt_injection_invocations` | [prompt_injection_invocations](../../packages/db/src/schema/prompt_injection_prevention.ts); [prompt-injection-prevention.ts](../../server/src/services/prompt-injection-prevention.ts). active. | Running prompt-injection invocation; current SQL checks running specifically. Fixture: Running invocation versus finished invocation. | Domain completion; preserve the owning evidence and records. |
| `tool_mcp_tasks` | [tool_mcp_tasks](../../packages/db/src/schema/tool_access.ts); [mcp-task-work.ts](../../server/src/services/mcp-task-work.ts). active. | Provider task with a next poll (MCP Tasks); the provider cannot wake Paperclip. Fixture: pending task versus delivered task. | Domain completion: delivery, expiry, cancellation, or the 7-day cap. |
| `agent_api_keys` | [agent_api_keys](../../packages/db/src/schema/agent_api_keys.ts); [agents.ts](../../server/src/services/agents.ts). ingress configuration. | Unrevoked machine credential can accept autonomous inbound work. Fixture: Saved live key versus revoked key. | Incoming-work proof; preserve the owning evidence and records. |
| `board_api_keys` | [board_api_keys](../../packages/db/src/schema/board_api_keys.ts); [board-auth.ts](../../server/src/services/board-auth.ts). ingress configuration. | Unrevoked key whose expiry is absent or future. Fixture: Live key versus revoked/expired key. | Incoming-work proof; preserve the owning evidence and records. |
| `agent_wakeup_requests` | [agent_wakeup_requests](../../packages/db/src/schema/agent_wakeup_requests.ts); [heartbeat/queue.ts](../../server/src/services/heartbeat/queue.ts). active / handoff. | Unsettled wake; coalesced wake additionally requires finished receipt and exact company/agent finished run. Fixture: Deferred wake versus exact completed/coalesced receipt. | Domain handoff; preserve the owning evidence and records. |
+9 -6
View File
@@ -984,15 +984,18 @@ never affects the response or the turn.
Each `item.completed` usage event (`payload.kind: "usage"`) carries two fields
in `payload.usage`:
- `upstreamProvider`: the host that served the most recent model call when the
usage was reported, such as `Novita` or `DeepInfra`.
- `upstreamProvider`: the host that served the most recently finished model
call when the usage was reported, such as `Novita` or `DeepInfra`.
- `providerRequestId`: that call's OpenRouter generation ID, such as
`gen-1762981234-AbCdEfGh`.
The proxy clears the recorded host when each new model request starts, and a
session clears it when it attaches another run. A call whose host the proxy
could not read therefore carries neither field. It never inherits an earlier
call's or an earlier run's values.
The proxy records a call's host when that call's response ends, not when the
request starts. OpenCode reports a step's usage only a few milliseconds before
it sends the next model request, so a step's usage still names its own call's
host while the next call is in flight. A call whose host the proxy could not
read records no host when it ends, and a session clears the recorded host when
it attaches another run. Such a call therefore carries neither field. It never
inherits an earlier call's or an earlier run's values.
The runner's internal `usage.reported` event carries the same values, using its
existing `providerRequestId` field and an optional `upstreamProvider` field.