mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-19 16:29:42 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents read Gmail, Calendar, Drive, Docs, Sheets and Slides through the built-in Google Workspace connection > - Paperclip now has its own REST-backed Google tools in `server/src/google-workspace-rest-tools/`. No service uses them yet > - That folder guessed a result size limit of 250,000 bytes. It applied the limit only to Drive, Docs, Sheets and Slides. Gmail and Calendar had no limit > - The guess was wrong in both directions. It refused documents that Paperclip can deliver, and the folder should not know Paperclip's limits > - This pull request makes the limit a number that the gateway gives. The folder applies it to every tool in all six services through one check. It also brings three Google planning docs up to date > - The benefit is that a REST result is refused only when the gateway would also refuse the same result from Google's own MCP server ## Linked Issues or Issue Description Refs #22027, #22028, #22029, #22026 **Is your feature request related to a problem?** The REST tools refused results over a size the folder chose itself (250,000 bytes). Paperclip can deliver larger results: the runner saves a large result as an artifact and gives the agent a pointer. Gmail and Calendar results had no limit at all. **Describe the solution you'd like** The gateway gives the limit. The folder applies the given number to the final serialized result of every tool, with one helper. A result over the limit is an error that says how to ask for less. It is never cut short. **Describe alternatives you've considered** A larger number chosen from the runner's 1 GiB capture limit. Google's own transfer limit (20 MB per response) and memory use make that a poor choice. Also, results would then differ between the REST path and Google's MCP path. ### Limits a tool result passes on the way to an agent | Point | Limit | What happens over it | | --- | --- | --- | | Google response read by `google-http.ts` | 20 MB (`MAX_RESPONSE_BYTES`) | Tool error. Nothing returned | | REST folder result (new) | the number the gateway passes: 999,885 bytes | Tool error with "read less" advice. The gateway returns `tool_error` (502) | | Gateway read of a remote MCP response (`MAX_REMOTE_MCP_RESPONSE_BYTES` in `tool-gateway.ts`) | 1,000,000 bytes per JSON body or event-stream message | `mcp_remote_response_too_large` (502). This is the "1,000,000 bytes are rejected" statement in `doc/prompt-injection-prevention.md`. It applies to the Google MCP path only; the REST path has no HTTP read | | Injection screening model check | 256 KiB model request | Not a delivery limit. A larger result is checked by the built-in patterns only | | Gateway HTTP route (`POST /tool-gateway/.../tools/call`) | none in Paperclip | `res.json(result)` | | Runner (`captureRunnerToolResult`) | 500,000 bytes (`RUNNER_TOOL_RESULT_MAX_BYTES`) | Saved as an artifact. The agent gets a pointer and reads it with `read_tool_result` | | Runner capture | 1 GiB (`RUNNER_API_RESPONSE_MAX_BYTES`) | `SemanticToolOutcomeUnknownError` | The gateway wraps the result as `{ content, data: { content, structuredContent, ... } }`, so the payload occurs again. Measured: a Gmail result of 676,459 bytes from the folder became 1,017,386 bytes delivered (ratio 1.50). The ratio cannot be more than 2. ### The budget `GOOGLE_WORKSPACE_REST_MAX_RESULT_BYTES` in `tool-gateway.ts` = `MAX_REMOTE_MCP_RESPONSE_BYTES` less the JSON-RPC envelope and event-stream framing (`event: message\ndata: {"jsonrpc":"2.0","id":"paperclip-tool-<uuid>-version","result":...}`) = **999,885 bytes**. A result that Google's MCP server could deliver through the gateway is also delivered over REST, and the reverse. With the wrapper, the largest delivered result is less than 2 MB. The runner saves it as an artifact (far below 1 GiB). So the folder never refuses a result that the runner can give as a pointer. Practical limits at that budget (plain ASCII; escaped and non-ASCII characters count more): | Tool | Largest content | | --- | --- | | Gmail `get_message` | about 500 KB plain body, or about 250 KB each for plain plus HTML | | Calendar `get_event` | about 500 KB description | | Drive `read_file_content` | about 500 KB text | | Drive `download_file_content` | about 375 KB file bytes | | Docs `read_doc` | about 240 KB document text | | Slides `read_presentation` | about 250 KB text | | Sheets `get_values` | about 500 KB cell text, at most 50,000 cells | ## What Changed - `tool.ts`: `defineTool` takes `whenTooLarge` advice per read tool (required for reads, refused for writes) and a `ToolLimits` argument. It measures the final `CallToolResult` once (`withinResultBudget`) and replaces an oversized result with a `too_large` tool error. `FALLBACK_MAX_RESULT_BYTES` (250,000) applies only when a direct caller gives no limit. - `callGoogleWorkspaceRestTool` and `createGoogleWorkspaceRestServer` take `maxResultBytes`. A value that is not a positive integer is refused before any request. - All 23 read tools state their advice: a smaller `pageSize` for list tools, `messageFormat` or `view` for Gmail, a narrower range for Sheets, `fields` for Slides. Write tools are not checked: a completed write is never reported as an error, so an agent is not led to repeat it. - Drive, Docs, Slides and Sheets no longer have their own byte checks (`boundedStructuredResult`, `boundedResult`, `sizedResult`, `MAX_CONTENT_BYTES` removed). Drive downloads stop reading at the given limit. The Sheets 50,000-cell limit stays. - Two Drive tool descriptions no longer state a fixed number. - Adapter `google-workspace-rest-dispatch.ts`: `maxResultBytes` is a required input and is passed through. - `tool-gateway.ts`: new `GOOGLE_WORKSPACE_REST_MAX_RESULT_BYTES`, passed on the REST branch. - Folder README: new "Result size" section with the limits and costs; old 250,000-byte text replaced. - Docs: `doc/memory/google-connector-memory.md`, the production checklist and the code-fix plan now list #22026 to #22029, the REST status (36 of 36, none switched on), the November 17 decisions, the known differences from Google's tools, open risks and next steps. ## Verification - `cd server && npx vitest run src/google-workspace-rest-tools` passes (53 files, 1,361 tests with the adapter test). - New `result-budget.test.ts`: for Gmail (read, list), Calendar (list, read), Drive, Docs, Slides and Sheets, a result just under a given budget is returned and one just over is refused with the tool's advice. Without a budget, the fallback applies; with a larger budget, the same result is returned. Drive downloads use the given cap. A Gmail list tells the caller to lower `pageSize`. A Gmail draft result over the budget is still returned, not an error. `defineTool` refuses a read tool without advice and a write tool with it. - `npx vitest run src/services/google-workspace-rest-dispatch.test.ts`: the adapter passes the gateway's number, and the folder applies exactly that number. - `npx vitest run src/__tests__/google-workspace-rest-transport.test.ts` (24 tests): with Gmail on REST, a 330 KB email passes the gateway with screening, is more than 500,000 bytes delivered, and `captureRunnerToolResult` turns it into a `read_tool_result` pointer. A 520 KB email is refused with `tool_error` and the advice, after one Google request. - `npx tsc --noEmit -p server/tsconfig.json`: no errors. - `pnpm verify --explain`: broader verification required (a doc under `doc/` is a global CI input). CI supplies that coverage. - Time per call: the check is one `JSON.stringify` of the result. About 5 µs for a 4 KB result, 18 µs for 42 KB, 0.24 ms for 700 KB. Drive, Docs, Sheets and Slides had the same pass before, so they add nothing. Gmail and Calendar add this one pass. ## Risks - Gmail and Calendar results now have a limit. Direct callers that pass no limit get the 250,000-byte fallback. The gateway always passes its number. - Drive, Docs, Slides and Sheets accept results up to about 4 times larger than before through the gateway. Large results reach agents as artifact pointers. - Error text for oversized results changed to one shared form. - No service is switched to REST, so users see no change today. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, file edits, GitHub CLI) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
185 lines
7.9 KiB
TypeScript
185 lines
7.9 KiB
TypeScript
import { z } from "zod";
|
||
import { GoogleToolError } from "./errors.js";
|
||
import type { GoogleHttp, GoogleRequest } from "./google-http.js";
|
||
|
||
export const MAX_BATCH_REQUESTS = 500;
|
||
export const OBJECT_PLACEHOLDER = "";
|
||
|
||
export const fileIdArg = (kind: string) => z.string().regex(/^[A-Za-z0-9_-]{1,256}$/, `Must be a Google ${kind} ID`);
|
||
export const revisionIdArg = z.string().min(1).max(1024);
|
||
|
||
export const batchRequestArg = z.record(z.string(), z.unknown()).refine(
|
||
(value) => Object.keys(value).length === 1,
|
||
"Each request must name exactly one kind of update, such as insertText",
|
||
);
|
||
|
||
const CONFLICT_STATUSES = new Set(["FAILED_PRECONDITION", "ABORTED"]);
|
||
const CONFLICT_REASON = /REVISION|CONDITION_NOT_MET|conditionNotMet/i;
|
||
const CONFLICT_MESSAGE = /\brequired revision\b|\brevision id\b|\blatest revision\b/i;
|
||
|
||
export function isRevisionConflict(error: GoogleToolError): boolean {
|
||
if (error.kind !== "invalid_argument" || (error.httpStatus !== 400 && error.httpStatus !== 409)) return false;
|
||
if (error.google.status && CONFLICT_STATUSES.has(error.google.status)) return true;
|
||
if ((error.google.reasons ?? []).some((reason) => CONFLICT_REASON.test(reason))) return true;
|
||
return CONFLICT_MESSAGE.test(error.message);
|
||
}
|
||
|
||
export type RequestKindRule =
|
||
| { allowed: true; refuse?: (body: Record<string, unknown>) => string | undefined; example?: Record<string, unknown> }
|
||
| { allowed: false; reason: string };
|
||
|
||
export type RequestKindRules = Readonly<Record<string, RequestKindRule>>;
|
||
|
||
const isRecord = (value: unknown): value is Record<string, unknown> => typeof value === "object" && value !== null && !Array.isArray(value);
|
||
const GOOGLE_FILE_HOSTS = new Set(["drive.google.com", "docs.google.com"]);
|
||
|
||
export function googleFileUrl(value: unknown): boolean {
|
||
if (typeof value !== "string") return false;
|
||
try {
|
||
return GOOGLE_FILE_HOSTS.has(new URL(value).hostname.toLowerCase());
|
||
} catch {
|
||
return false;
|
||
}
|
||
}
|
||
|
||
export const refuseGoogleFileUrl = (...path: string[]) => (body: Record<string, unknown>): string | undefined => {
|
||
let value: unknown = body;
|
||
for (const key of path) value = isRecord(value) ? value[key] : undefined;
|
||
return googleFileUrl(value) ? `takes its picture from a Google Drive or Docs link (${path.join(".")}), which is another file.` : undefined;
|
||
};
|
||
|
||
export function allowedKinds(rules: RequestKindRules): string[] {
|
||
return Object.entries(rules).filter(([, rule]) => rule.allowed).map(([kind]) => kind);
|
||
}
|
||
|
||
export function refusedKinds(rules: RequestKindRules): string[] {
|
||
return Object.entries(rules).filter(([, rule]) => !rule.allowed).map(([kind]) => kind);
|
||
}
|
||
|
||
export function assertRequestKinds(requests: ReadonlyArray<Record<string, unknown>>, rules: RequestKindRules): void {
|
||
const problems: string[] = [];
|
||
for (const [index, request] of requests.entries()) {
|
||
const kind = Object.keys(request)[0]!;
|
||
const rule = Object.hasOwn(rules, kind) ? rules[kind] : undefined;
|
||
if (!rule) problems.push(`requests[${index}].${kind} is not a request kind this tool accepts.`);
|
||
else if (!rule.allowed) problems.push(`requests[${index}].${kind} ${rule.reason}`);
|
||
else {
|
||
const body = request[kind];
|
||
const reason = rule.refuse?.(isRecord(body) ? body : {});
|
||
if (reason) problems.push(`requests[${index}].${kind} ${reason}`);
|
||
}
|
||
}
|
||
if (problems.length) {
|
||
throw new GoogleToolError("invalid_argument", `${problems.join(" ")} This tool only changes the one file it is given, so nothing was changed.`);
|
||
}
|
||
}
|
||
|
||
export interface TableCellText {
|
||
text: string;
|
||
rowSpan?: number;
|
||
columnSpan?: number;
|
||
row?: number;
|
||
column?: number;
|
||
}
|
||
|
||
export const MERGED_LEFT = "(merged left)";
|
||
export const MERGED_UP = "(merged up)";
|
||
|
||
const span = (value: number | undefined) => (value !== undefined && value > 1 ? Math.floor(value) : 1);
|
||
|
||
export function tableGrid(rows: readonly (readonly TableCellText[])[], columns: number): string[][] {
|
||
const grid: Array<Array<string | undefined>> = rows.map(() => []);
|
||
for (const [listedRow, cells] of rows.entries()) {
|
||
const positional = cells.length === columns && cells.every((cell) => cell.column === undefined);
|
||
let column = 0;
|
||
for (const [cellIndex, cell] of cells.entries()) {
|
||
const rowIndex = cell.row !== undefined && cell.row >= 0 && cell.row < rows.length ? cell.row : listedRow;
|
||
if (positional) column = cellIndex;
|
||
else if (cell.column !== undefined) column = cell.column;
|
||
else while (grid[rowIndex]![column] !== undefined) column += 1;
|
||
if (positional && grid[rowIndex]![column] !== undefined && cell.text === "") continue;
|
||
grid[rowIndex]![column] = cell.text;
|
||
for (let down = 0; down < span(cell.rowSpan) && rowIndex + down < rows.length; down += 1) {
|
||
for (let across = 0; across < span(cell.columnSpan); across += 1) {
|
||
if (down === 0 && across === 0) continue;
|
||
grid[rowIndex + down]![column + across] = down === 0 ? MERGED_LEFT : MERGED_UP;
|
||
}
|
||
}
|
||
column += span(cell.columnSpan);
|
||
}
|
||
}
|
||
const width = Math.max(columns, ...grid.map((row) => row.length));
|
||
return grid.map((row) => Array.from({ length: width }, (_, index) => row[index] ?? ""));
|
||
}
|
||
|
||
export function escapeTableCell(text: string): string {
|
||
return text.replace(/\\/g, "\\\\").replace(/\|/g, "\\|").replace(/\r?\n/g, "<br>");
|
||
}
|
||
|
||
export function markdownTable(rows: readonly (readonly TableCellText[])[], columns: number): string[] {
|
||
if (rows.length === 0) return ["[empty table]"];
|
||
const grid = tableGrid(rows, columns).map((cells) => cells.map(escapeTableCell));
|
||
const lines = grid.map((cells) => `| ${cells.join(" | ")} |`);
|
||
return [lines[0]!, `| ${grid[0]!.map(() => "---").join(" | ")} |`, ...lines.slice(1)];
|
||
}
|
||
|
||
export async function readWithComments<T>(http: GoogleHttp, request: GoogleRequest, plain: GoogleRequest | undefined, unavailable: string): Promise<T> {
|
||
try {
|
||
return await http.send<T>(request);
|
||
} catch (error) {
|
||
if (!plain || !(error instanceof GoogleToolError) || error.kind !== "permission_denied") throw error;
|
||
await http.send(plain);
|
||
throw new GoogleToolError("permission_denied", unavailable, error.httpStatus);
|
||
}
|
||
}
|
||
|
||
export async function sendBatchUpdate<T>(http: GoogleHttp, request: GoogleRequest, what: string, readTool: string): Promise<T> {
|
||
try {
|
||
return await http.send<T>(request);
|
||
} catch (error) {
|
||
if (error instanceof GoogleToolError && isRevisionConflict(error)) {
|
||
throw new GoogleToolError(
|
||
"invalid_argument",
|
||
`The ${what} is no longer at the revision you read (or that revision ID is not valid), so nothing was changed. Read it again with ${readTool} and retry with its new revisionId.`,
|
||
error.httpStatus,
|
||
);
|
||
}
|
||
throw error;
|
||
}
|
||
}
|
||
|
||
export interface RestCommentPost {
|
||
content?: string;
|
||
author?: { displayName?: string; anonymous?: boolean };
|
||
deleted?: boolean;
|
||
}
|
||
|
||
export interface RestCommentThread {
|
||
commentId?: string;
|
||
anchorId?: string;
|
||
status?: string;
|
||
plainTextQuote?: string;
|
||
headPost?: RestCommentPost;
|
||
replies?: RestCommentPost[];
|
||
}
|
||
|
||
function postLine(post: RestCommentPost): string {
|
||
if (post.deleted) return "(deleted)";
|
||
const author = post.author?.displayName || (post.author?.anonymous ? "Anonymous" : "Unknown author");
|
||
return `${author}: ${(post.content ?? "").replace(/\s*\n\s*/g, " ").trim()}`;
|
||
}
|
||
|
||
export function renderComments(threads: readonly RestCommentThread[], where: (thread: RestCommentThread) => string | undefined): string[] {
|
||
return threads.map((thread) => {
|
||
const place = where(thread);
|
||
const quote = thread.plainTextQuote ? ` on "${thread.plainTextQuote.replace(/\s+/g, " ").trim()}"` : "";
|
||
const state = thread.status === "RESOLVED" ? " (resolved)" : "";
|
||
const head = `- ${thread.headPost ? postLine(thread.headPost) : "(no text)"}${quote}${place ? ` [${place}]` : ""}${state}`;
|
||
return [head, ...(thread.replies ?? []).map((reply) => ` - ${postLine(reply)}`)].join("\n");
|
||
});
|
||
}
|
||
|
||
export function plural(count: number, one: string, many: string): string {
|
||
return count === 1 ? `1 ${one}` : `${count} ${many}`;
|
||
}
|