mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents call tools from connected apps through the Paperclip MCP gateway > - Some tool calls take a long time: exports, builds, large searches > - The MCP progress utility lets an app report how far a call is, but Paperclip ignored it > - So the task chat showed only a spinner until the call finished, with no sign of life > - This pull request reads the app's progress and shows it on the tool's row while the call runs > - The benefit is that people can see that a long call is working, how far it is, and how long it has run ## Linked Issues or Issue Description **Subsystem affected** Connected apps (MCP gateway) and the task chat. **Problem or motivation** A long tool call from a connected MCP app shows only a spinner in the task chat. Paperclip does not send a `progressToken`, and it drops `notifications/progress` from the app. **Proposed solution** Send a `progressToken` on each `tools/call` that an agent run makes. Read the app's progress notifications from the call's response stream. Show a progress bar (or an indeterminate sweep), the latest message, and elapsed time on the running tool row. Keep the latest state so that a reload mid-call shows it. **Alternatives considered** A new transport for progress. The existing company live-events socket already carries per-run events with run read checks, so this change uses it. **Additional context** Spec: https://modelcontextprotocol.io/specification/2026-07-28/basic/utilities/progress ## What Changed - `mcp-http.ts`: `mcpHttpRequestInit` can add `_meta.progressToken`. `readMcpHttpResponse` gives each notification on the response stream to an observer before the result. The observer cannot fail or delay the call. New `mcpProgressNotificationOf` matches a notification to its token. - `tool-gateway.ts`: remote HTTP and local stdio `tools/call` from an agent run send a per-call token and forward progress to a reporter. Calls without a run send no token. - New `server/src/services/tool-call-progress.ts`: checks each update (finite, increasing, plain single-line text of at most 200 characters). It publishes a `heartbeat.run.tool_progress` live event at most every 250 ms (latest wins) and saves the latest update on the invocation at most every 1 s. Nothing is awaited by the call. A final `done` event goes out when the call ends. - DB: nullable `tool_invocations.progress` jsonb column (migration `0479`, `ADD COLUMN` only). A save never writes onto a finished call. - Live delivery: the new event follows its run like transcript output. A client gets it only for runs it follows, after the run read check, in the run's company only. - New route `GET /api/issues/:issueId/tool-call-progress` (task read check) returns the latest progress of the task's running calls, for a reload mid-call. Documented in `docs/api/issues.md`. - UI: `ToolCallProgressProvider` (one per chat) keeps the latest update per call. A running tool row subscribes and shows a bar or sweep, the message, the percent and elapsed time (design tokens, reduced motion honored). Finished rows drop it and never fetch or re-render for progress. `LiveUpdatesProvider` ignores the event for query invalidation. - `packages/kv-demo-mcp-server`: new `slow_with_progress` demo tool (`steps`, `stepMs`, `withTotal`) in its own file. - Storybook: `Tasks/Tool progress` stories (running with total, running without total, done). - Docs: new `doc/connections/MCP-PROGRESS.md`; `MCP-PROTOCOL-VERSIONS.md` no longer lists progress as unsupported. ## Verification - Red/green: on the original source, 5 of the new server tests fail (token in `_meta`, SSE with interleaved progress and result, notification matching, run-interest delivery, gateway end-to-end). With this change they pass. - `server/src/__tests__/mcp-http.test.ts` (13), `tool-call-progress.test.ts` (9: throttling, latest wins, bounded text, out-of-order drop, finish, failures never throw, company/run/tool scoping), `tool-call-progress-store.test.ts` (2, embedded Postgres: company and task scoping, no write onto a finished call), `realtime/live-event-delivery.test.ts` (run interest), `tool-gateway.test.ts` end-to-end against the real SDK kv demo server over Streamable HTTP: 6 provider updates over about 1.2 s give 3 to 7 throttled events, all for the right company, run, task and tool, the last one `done`; no event reaches another company; the invocation keeps the progress; the result is unchanged. - UI: `tool-call-progress.test.tsx` (10: name matching, row states, plain-text rendering, drop on complete, restore after reload, no fetch for finished rows). All 67 task-chat test files pass. - `packages/kv-demo-mcp-server`: 13 tests, including progress over HTTP with and without a total. - Storybook: checked the three stories in a browser. The bar fills to 40%, the sweep animates, the elapsed time ticks, and the done row shows no progress. - `pnpm verify`: see the CI checks for the final revision. ## Risks - Low. The migration adds one nullable column. - An SSE response is now bounded per message, not per stream, so handled notifications (such as progress) do not use up the result's size limit. Each message is still bounded. - Progress code runs only while a run's tool call is in flight. It adds a few bytes to the request and throttled, fire-and-forget work. Agent boot and time to first token do not change. - A row finds its call by run and tool name. If one run calls the same tool twice at the same time, both rows show the newer call's progress. - Progress from a call that became an MCP Task is not shown. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
196 lines
7.6 KiB
TypeScript
196 lines
7.6 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { and, desc, eq, gte, isNotNull, isNull } from "drizzle-orm";
|
|
import { toolInvocations, type Db } from "@paperclipai/db";
|
|
import {
|
|
normalizeToolCallProgress,
|
|
type LiveEventType,
|
|
type ToolCallProgress,
|
|
type ToolCallProgressSnapshot,
|
|
} from "@paperclipai/shared";
|
|
import { logger } from "../middleware/logger.js";
|
|
import { publishLiveEvent } from "./live-events.js";
|
|
import { assignedToolName } from "./native-runtime/assigned-mcp-tools.js";
|
|
|
|
/**
|
|
* Live progress for a connected-app tool call that an agent run made.
|
|
*
|
|
* The provider streams MCP `notifications/progress` on the call's response.
|
|
* Each update is published to the run's viewers at most every
|
|
* PUBLISH_INTERVAL_MS (the latest one wins) and saved on the invocation at
|
|
* most every SAVE_INTERVAL_MS, so a reload mid-call shows the latest state.
|
|
* Nothing here is awaited by the call: the result is never delayed.
|
|
*/
|
|
export const TOOL_CALL_PROGRESS_PUBLISH_INTERVAL_MS = 250;
|
|
export const TOOL_CALL_PROGRESS_SAVE_INTERVAL_MS = 1_000;
|
|
export const TOOL_CALL_PROGRESS_EVENT: LiveEventType = "heartbeat.run.tool_progress";
|
|
/** Progress older than this is not shown after a reload (the call is long gone). */
|
|
const SNAPSHOT_MAX_AGE_MS = 24 * 60 * 60_000;
|
|
|
|
export interface ToolCallProgressReporter {
|
|
/** Sent as the request's `_meta.progressToken`. */
|
|
readonly token: string;
|
|
/** Records a provider update. Never throws, never blocks. */
|
|
report(update: { progress: unknown; total?: unknown; message?: unknown }): void;
|
|
/** The call ended: stop timers and tell viewers to drop the progress. */
|
|
finish(): void;
|
|
}
|
|
|
|
type Publish = (input: { companyId: string; type: LiveEventType; payload: Record<string, unknown> }) => void;
|
|
type Save = (progress: ToolCallProgress) => Promise<void>;
|
|
|
|
/** Leading-edge throttle that keeps only the latest value for its trailing run. */
|
|
function latestThrottle<T>(intervalMs: number, run: (value: T) => void) {
|
|
let lastAt = -Infinity;
|
|
let pending: { value: T } | null = null;
|
|
let timer: ReturnType<typeof setTimeout> | null = null;
|
|
const flush = () => {
|
|
timer = null;
|
|
if (!pending) return;
|
|
const { value } = pending;
|
|
pending = null;
|
|
lastAt = Date.now();
|
|
run(value);
|
|
};
|
|
return {
|
|
push(value: T) {
|
|
pending = { value };
|
|
if (timer) return;
|
|
const wait = lastAt + intervalMs - Date.now();
|
|
if (wait <= 0) flush();
|
|
else { timer = setTimeout(flush, wait); timer.unref?.(); }
|
|
},
|
|
cancel() {
|
|
if (timer) clearTimeout(timer);
|
|
timer = null;
|
|
pending = null;
|
|
},
|
|
};
|
|
}
|
|
|
|
export function createToolCallProgressReporter(input: {
|
|
companyId: string;
|
|
runId: string;
|
|
issueId: string | null;
|
|
agentId: string | null;
|
|
invocationId: string;
|
|
toolName: string;
|
|
startedAt?: Date;
|
|
publish?: Publish;
|
|
save?: Save;
|
|
}): ToolCallProgressReporter {
|
|
const publish: Publish = input.publish ?? ((event) => { publishLiveEvent(event); });
|
|
const startedAt = (input.startedAt ?? new Date()).toISOString();
|
|
const token = `paperclip-progress-${randomUUID()}`;
|
|
let finished = false;
|
|
let last: ToolCallProgress | null = null;
|
|
const snapshot = (progress: ToolCallProgress, done: boolean): ToolCallProgressSnapshot => ({
|
|
...progress, invocationId: input.invocationId, runId: input.runId, toolName: input.toolName,
|
|
nativeToolName: assignedToolName(input.toolName), startedAt, done,
|
|
});
|
|
const send = (value: ToolCallProgressSnapshot) => {
|
|
try {
|
|
publish({
|
|
companyId: input.companyId,
|
|
type: TOOL_CALL_PROGRESS_EVENT,
|
|
payload: { ...value, issueId: input.issueId, agentId: input.agentId },
|
|
});
|
|
} catch (err) {
|
|
logger.debug({ err, invocationId: input.invocationId }, "tool call progress publish failed");
|
|
}
|
|
};
|
|
const live = latestThrottle<ToolCallProgress>(TOOL_CALL_PROGRESS_PUBLISH_INTERVAL_MS, (progress) => send(snapshot(progress, false)));
|
|
// At most one save in flight; while it runs only the newest update waits.
|
|
let saving = false;
|
|
let queuedSave: ToolCallProgress | null = null;
|
|
const save = (progress: ToolCallProgress) => {
|
|
if (!input.save || finished) return;
|
|
if (saving) { queuedSave = progress; return; }
|
|
saving = true;
|
|
input.save(progress)
|
|
.catch((err) => logger.debug({ err, invocationId: input.invocationId }, "tool call progress save failed"))
|
|
.finally(() => {
|
|
saving = false;
|
|
const next = queuedSave;
|
|
queuedSave = null;
|
|
if (next) save(next);
|
|
});
|
|
};
|
|
const saved = latestThrottle<ToolCallProgress>(TOOL_CALL_PROGRESS_SAVE_INTERVAL_MS, save);
|
|
return {
|
|
token,
|
|
report(update) {
|
|
if (finished) return;
|
|
const progress = normalizeToolCallProgress(update);
|
|
// Progress must increase; a lower value is out of order and is dropped.
|
|
if (!progress || (last && progress.progress < last.progress)) return;
|
|
last = progress;
|
|
live.push(progress);
|
|
if (input.save) saved.push(progress);
|
|
},
|
|
finish() {
|
|
if (finished) return;
|
|
finished = true;
|
|
live.cancel();
|
|
saved.cancel();
|
|
queuedSave = null;
|
|
if (last) send(snapshot(last, true));
|
|
},
|
|
};
|
|
}
|
|
|
|
/** Saves a running call's latest progress. A finished call is never touched. */
|
|
export function saveToolCallProgress(db: Db, companyId: string, invocationId: string): Save {
|
|
return async (progress) => {
|
|
await db.update(toolInvocations)
|
|
.set({ progress })
|
|
.where(and(eq(toolInvocations.id, invocationId), eq(toolInvocations.companyId, companyId), isNull(toolInvocations.completedAt)));
|
|
};
|
|
}
|
|
|
|
/** Finished calls are reported for this long, so a chat that missed their end learns it. */
|
|
const FINISHED_REPORT_MS = 60 * 60_000;
|
|
|
|
/**
|
|
* The latest progress of a task's running tool calls, for a reload mid-call,
|
|
* plus its recently finished calls marked `done`. A chat drops a call only on
|
|
* `done`: a running call may not have saved its first update yet.
|
|
*/
|
|
export async function listToolCallProgress(db: Db, companyId: string, issueId: string, now = new Date()): Promise<ToolCallProgressSnapshot[]> {
|
|
const columns = {
|
|
id: toolInvocations.id,
|
|
runId: toolInvocations.runId,
|
|
toolName: toolInvocations.toolName,
|
|
progress: toolInvocations.progress,
|
|
startedAt: toolInvocations.startedAt,
|
|
createdAt: toolInvocations.createdAt,
|
|
completedAt: toolInvocations.completedAt,
|
|
};
|
|
const task = and(eq(toolInvocations.companyId, companyId), eq(toolInvocations.issueId, issueId), isNotNull(toolInvocations.runId));
|
|
const [running, finished] = await Promise.all([
|
|
db.select(columns).from(toolInvocations).where(and(task,
|
|
isNull(toolInvocations.completedAt),
|
|
isNotNull(toolInvocations.progress),
|
|
gte(toolInvocations.createdAt, new Date(now.getTime() - SNAPSHOT_MAX_AGE_MS)),
|
|
)).orderBy(desc(toolInvocations.createdAt)).limit(50),
|
|
db.select(columns).from(toolInvocations).where(and(task,
|
|
gte(toolInvocations.completedAt, new Date(now.getTime() - FINISHED_REPORT_MS)),
|
|
)).orderBy(desc(toolInvocations.completedAt)).limit(50),
|
|
]);
|
|
return [...running, ...finished].flatMap((row) => {
|
|
const done = row.completedAt !== null;
|
|
const savedAt = row.progress ? Date.parse(String(row.progress.updatedAt)) : NaN;
|
|
const at = done ? row.completedAt! : Number.isFinite(savedAt) ? new Date(savedAt) : now;
|
|
const progress = normalizeToolCallProgress(row.progress ?? (done ? { progress: 0 } : {}), at);
|
|
if (!progress || !row.runId) return [];
|
|
return [{
|
|
...progress,
|
|
invocationId: row.id,
|
|
runId: row.runId,
|
|
toolName: row.toolName,
|
|
nativeToolName: assignedToolName(row.toolName),
|
|
startedAt: (row.startedAt ?? row.createdAt).toISOString(),
|
|
done,
|
|
}];
|
|
});
|
|
}
|