mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents change Paperclip records through the `call_api` tool, and the native runtime runs each tool call through a repeated-identical-failure guard (#21360, #21398) > - Some model hosts drop arguments. A model then sends the same failing call again and again, and each attempt costs a model turn > - In chat-eval cohort #14 (OpenCode, `deepseek-v4.1-flash`), two cases sent a body-less `PATCH /api/issues/{id}` 7 and 10 times with one input hash > - The guard counted these refusals and stopped running the call after the second failure. But its correction showed only a `"body": {...}` placeholder, so the model had no concrete call to copy. Also, the guard did not count a route's own 400/422 when it came back as an `ok: false` result > - This pull request puts a copyable call in each correction and counts returned 400/422 rejections. It never counts transient failures > - The benefit is that a model gets the exact call to send on the second failure, and the third identical validation failure does not run, whether the rejection is thrown or returned ## Linked Issues or Issue Description Refs #21360, #21398 **What happened?** In chat-eval cohort #14, `questions-and-approvals.review-plan` (run 470002dc) and `tasks.private-task` (run bb76a832) sent `call_api PATCH /api/issues/{id}` with no `body` 7 and 10 times, all with the same input hash. The pre-dispatch check (`assertBodyPresent`) refused each call and returned HTTP 400. The guard corrected the second failure and refused the rest. But the correction said only `Resend with a top-level body argument: {"operationId": ..., "pathParams": {...}, "body": {...}}`. The model kept sending the identical call, and each refusal cost another model turn: 2 min 18 s in review-plan and 24 s in private-task. Also, when the route itself returns a 400/422 validation response, `call_api` gives back `{ ok: false, status: 422, ... }` as a result and does not throw. The guard did not count that, so an identical invalid request reached the route every time. **Expected behavior** On the second identical deterministic failure, the agent gets a correction with a call it can copy that passes validation. On the third, Paperclip refuses to run the call and tells the agent to change the arguments or to stop and report. Transient failures (5xx, timeouts, rate limits, conflicts) always run again. **Steps to reproduce** 1. Run a native-runtime agent on a host that drops the `body` argument. 2. Have it update a task with `call_api PATCH /api/issues/{id}`. 3. Look at the tool results. They contain only a `{...}` placeholder. A route's 422 returned as `ok: false` is never counted. **Paperclip version or commit** masterd4498ae7ac(cohort #14), reproduced againstecdeed5e7e. ## What Changed - `runner-api-client.ts`: Add `runnerApiCallExample`. It builds a copyable call from the operation's catalog schema: the path parameters the model sent, plus a body with the required fields. If no field is required, the body has one free-text field, for example `{"operationId":"PATCH /api/issues/{id}","pathParams":{"id":"<sent id>"},"body":{"title":"<title>"}}`. The "requires body" correction and the "body must be a JSON object, not a JSON-encoded string" correction now include this example in place of the `{...}` placeholder. - `repeated-tool-failure-guard.ts`: Count a result with `ok: false` and HTTP 400/422 as a deterministic failure, in the same way as a thrown 400/422. The first returned rejection passes through unchanged. The second gets a `guidance` note that says the call will not run again. The third and later identical calls are refused without running, and the refusal quotes the response. - Never count transient results: 5xx, 408, 429, 409, 404, and a missing status (timeouts or unknown outcome). - A route can reject a request because of the state it found. So the guard forgets a returned rejection after a different call succeeds. A refusal before sending depends only on the arguments, and the guard keeps it for the run, as before. - Tests: the guard tests cover returned 422 (correct, then refuse), a changed call that still runs, returned 500/503/429/408/409/404/no-status and thrown 429/409/timeout that keep running, and a returned rejection forgotten after a state change. The runner API tests check that the example in each recorded body-less correction is sent when copied as is, that the issue-update example passes `updateIssueSchema`, and that every catalog write with a required body (more than 200) has an example its request schema accepts. ## Verification - `cd server && npx vitest run src/services/native-runtime/repeated-tool-failure-guard.test.ts src/services/native-runtime/runner-api.test.ts`: 2190 passed. - Red/green: with the master guard, the 2 returned-rejection tests fail. With the master client, 6 example tests fail. With this change, all pass. - `node scripts/typecheck-layers.mjs` (server): no errors. - Per-call cost (median of 5 × 20,000 calls, local M-series): guard on a successful call 2.13 µs before, 2.19 µs after. Transient-failure path 2.12 µs. Building an example takes 0.67 µs and happens only on a refused call. Nothing is added before the turn starts. The guard is created with the run's tool authority, as before. - Paid narrow run: see the comment below. ## Risks - Low. The change affects only calls that fail validation twice with identical arguments. - A route that returns 400/422 because of state can succeed later with the same arguments. To cover that, the guard forgets a returned rejection after any different call succeeds. A model that retries with no call in between is refused after two attempts and must change the arguments or report. - The example uses `<field>` placeholders for free text and for ids that the model must look up. If a model copies a placeholder verbatim, the route stores that text. The correction says to send the fields the model means to change. - This does not make a model stop resending a refused call. It only gives better guidance and keeps the refused call from running. ## Model Used - Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use, code execution, and extended reasoning). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
986 lines
55 KiB
TypeScript
986 lines
55 KiB
TypeScript
import { describe, expect, it, vi } from "vitest";
|
|
import {
|
|
runnerApiCatalog,
|
|
runnerApiOperation,
|
|
searchRunnerApi,
|
|
} from "./runner-api-catalog.js";
|
|
import Ajv from "ajv";
|
|
import { updateIssueSchema } from "@paperclipai/shared";
|
|
import {
|
|
assertRunnerApiDispatchable,
|
|
executeRunnerApi,
|
|
readBoundedResponse,
|
|
runnerApiCallExample,
|
|
RUNNER_API_CALL_HEADER,
|
|
runnerApiUrl,
|
|
validateRunnerApiCall,
|
|
type RunnerApiCall,
|
|
type RunnerApiIo,
|
|
} from "./runner-api-client.js";
|
|
import { runnerApiSensitiveResponse } from "./runner-api-sensitive-response.js";
|
|
import { RUNNER_API_RESPONSE_MAX_BYTES } from "./runner-api-response-limits.js";
|
|
import { runnerBridgeDefinitions } from "./runner-bridge-tools.js";
|
|
import { CAPABILITY_SEMANTIC_TOOL_CATALOG } from "../../vendor/paperclip-runner/index.js";
|
|
|
|
const context = {
|
|
companyId: "company-a",
|
|
issueId: "issue-a",
|
|
issueIdentifier: "API-1",
|
|
runId: "run-a",
|
|
workMode: "standard",
|
|
};
|
|
const projects = "GET /api/companies/{companyId}/projects";
|
|
const emptyRequestBody = (operation: { requestBody?: unknown }) =>
|
|
(operation.requestBody as { content?: Record<string, { schema?: { type?: string } }> } | undefined)?.content?.["application/json"]?.schema?.type === "array" ? [] : {};
|
|
const createProject = "POST /api/companies/{companyId}/projects";
|
|
const io = (fetcher: typeof fetch): RunnerApiIo => ({
|
|
apiUrl: "http://127.0.0.1:3100",
|
|
token: "private-agent-token",
|
|
fetch: fetcher,
|
|
readFile: async () => ({
|
|
bytes: Buffer.from("test"),
|
|
filename: "proof.txt",
|
|
contentType: "text/plain",
|
|
}),
|
|
saveResponse: async (bytes, contentType) => ({
|
|
artifactId: "artifact-a",
|
|
byteSize: Buffer.isBuffer(bytes) ? bytes.length : bytes.byteSize,
|
|
contentType,
|
|
}),
|
|
});
|
|
|
|
describe("capped list pagination", () => {
|
|
const listIssues = "GET /api/companies/{companyId}/issues";
|
|
const page = (headers: Record<string, string>, rows = [{ id: "a" }, { id: "b" }]) =>
|
|
vi.fn(async () => new Response(JSON.stringify(rows), { headers: { "content-type": "application/json", ...headers } }));
|
|
|
|
it("tells the agent a capped list is incomplete and how to fetch the next page", async () => {
|
|
const fetcher = page({ "x-paperclip-has-more": "true", "x-paperclip-limit": "2", "x-paperclip-next-offset": "2" });
|
|
const result = await executeRunnerApi({ operationId: listIssues, query: { limit: 2 } }, context, io(fetcher));
|
|
expect(result).toMatchObject({
|
|
ok: true,
|
|
data: [{ id: "a" }, { id: "b" }],
|
|
pagination: { hasMore: true, truncated: true, limit: 2, nextOffset: 2 },
|
|
});
|
|
expect((result as unknown as { pagination: { guidance: string } }).pagination.guidance).toContain("offset=2");
|
|
});
|
|
|
|
it("carries the cursor and the signal when the capped body is saved as an artifact", async () => {
|
|
const rows = Array.from({ length: 600 }, (_, index) => ({ id: `row-${index}`, title: "x".repeat(80) }));
|
|
const fetcher = page({ "x-paperclip-has-more": "true", "x-paperclip-limit": "600", "x-paperclip-next-offset": "600", "x-paperclip-next-after-id": "row-599" }, rows);
|
|
const result = await executeRunnerApi({ operationId: listIssues }, context, io(fetcher));
|
|
expect(result).toMatchObject({ ok: true, artifact: { artifactId: "artifact-a" }, pagination: { hasMore: true, nextOffset: 600, nextAfterId: "row-599" } });
|
|
});
|
|
|
|
it("reports a complete list as complete", async () => {
|
|
const result = await executeRunnerApi({ operationId: listIssues }, context, io(page({ "x-paperclip-has-more": "false", "x-paperclip-limit": "500" })));
|
|
expect(result).toMatchObject({ ok: true, pagination: { hasMore: false, truncated: false, nextOffset: null } });
|
|
expect((result as unknown as { pagination: object }).pagination).not.toHaveProperty("guidance");
|
|
});
|
|
|
|
it("adds nothing for routes that do not page", async () => {
|
|
const result = await executeRunnerApi({ operationId: projects }, context, io(page({})));
|
|
expect(result).not.toHaveProperty("pagination");
|
|
});
|
|
});
|
|
|
|
describe("runner call identity", () => {
|
|
it.each([[createProject, "call-key-a"], [projects, null]] as const)("sends the call key only with mutations: %s", async (operationId, expected) => {
|
|
const fetcher = vi.fn(async (_url: string | URL | Request, _init?: RequestInit) => new Response("{}", { headers: { "content-type": "application/json" } }));
|
|
await executeRunnerApi({ operationId, ...(operationId === createProject ? { body: { name: "Keyed" } } : {}) }, context,
|
|
{ ...io(fetcher as unknown as typeof fetch), callKey: "call-key-a" });
|
|
expect(new Headers(fetcher.mock.calls[0]![1]!.headers).get(RUNNER_API_CALL_HEADER)).toBe(expected);
|
|
});
|
|
});
|
|
|
|
describe("bounded response capture receipts", () => {
|
|
it.each([projects, createProject])("reports oversize evidence without retrying %s", async operationId => {
|
|
const fetcher = vi.fn(async () => new Response("", { headers: { "content-length": String(RUNNER_API_RESPONSE_MAX_BYTES + 1) } }));
|
|
const result = await executeRunnerApi({ operationId, ...(operationId === createProject ? { body: { name: "Oversize" } } : {}) }, context, io(fetcher));
|
|
expect(result).toMatchObject({ ok: false, error: "api_response_too_large", maxResponseBytes: RUNNER_API_RESPONSE_MAX_BYTES,
|
|
outcome: operationId === projects ? "read_failed" : "unknown" });
|
|
expect(fetcher).toHaveBeenCalledTimes(1);
|
|
});
|
|
it("settles the budget and removes the file even when saving fails", async () => {
|
|
const bytes = Buffer.alloc(32 * 1024);
|
|
const settle = vi.fn(async () => {});
|
|
const input = io(async () => new Response(bytes));
|
|
input.reserveResponseCapture = async () => settle;
|
|
input.saveResponse = async () => { throw new Error("storage unavailable"); };
|
|
await expect(executeRunnerApi({ operationId: projects }, context, input)).rejects.toThrow("storage unavailable");
|
|
expect(settle).toHaveBeenCalledExactlyOnceWith(bytes.length, true);
|
|
});
|
|
});
|
|
|
|
describe("connection setup response protection", () => {
|
|
it.each([
|
|
["POST", "/api/chat-endpoints/{endpointId}/setup"],
|
|
["POST", "/api/chat-endpoints/{endpointId}/setup-secret"],
|
|
["POST", "/api/chat-endpoints/{endpointId}/github/setup"],
|
|
["POST", "/api/chat-endpoints/{endpointId}/slack/resume"],
|
|
["POST", "/api/companies/{companyId}/adapters/{type}/login-sessions"],
|
|
["GET", "/api/companies/{companyId}/setup-token-login-sessions/{sessionId}/prompt"],
|
|
["POST", "/api/tool-gateway/gateways/{gatewayId}/tokens"],
|
|
["POST", "/api/companies/{companyId}/ai-connections/local/check"],
|
|
["POST", "/api/tool-connections/{connectionId}/test-calls"],
|
|
["GET", "/api/tool-connections/{connectionId}/test-calls/{actionRequestId}"],
|
|
])("protects %s %s receipts and history", (method, path) => {
|
|
expect(runnerApiSensitiveResponse(method, path)).toBe(true);
|
|
});
|
|
it("keeps ordinary connection metadata shareable", () => {
|
|
expect(runnerApiSensitiveResponse("GET", "/api/tool-connections/{connectionId}")).toBe(false);
|
|
});
|
|
});
|
|
|
|
describe("runner API catalog", () => {
|
|
it("discovers the real handoff schema and recovery endpoints with standard-mode mutation authority", () => {
|
|
const operation = runnerApiOperation("POST /api/companies/{companyId}/case-handoffs");
|
|
expect(operation.dedicatedTools).toEqual(["handoff_case", "create_case"]);
|
|
expect(operation.allowedModes).toEqual(["standard"]);
|
|
expect(operation.requestBody?.content["application/json"].schema).toMatchObject({ oneOf: [
|
|
{ additionalProperties: false, required: expect.arrayContaining(["case", "toAgentId", "phase", "instructions", "idempotencyKey", "workflow_document_text"]),
|
|
properties: { case: { required: ["create"] }, workflow_document_text: { type: "string", maxLength: 200000, pattern: "\\S" } } },
|
|
{ additionalProperties: false, required: expect.arrayContaining(["case", "toAgentId", "phase", "instructions", "idempotencyKey", "workflowId"]),
|
|
properties: { case: { required: ["create"] }, workflowId: { type: "string", format: "uuid" } } },
|
|
{ additionalProperties: false, required: expect.arrayContaining(["case", "toAgentId", "phase", "instructions", "idempotencyKey"]),
|
|
properties: { case: { required: ["id"] } } },
|
|
{ additionalProperties: false, required: expect.arrayContaining(["to", "case", "instructions", "idempotencyKey"]),
|
|
properties: { to: { enum: ["done"] } } },
|
|
] });
|
|
const branches = operation.requestBody!.content["application/json"].schema.oneOf;
|
|
expect(branches[1].required).not.toContain("workflow_document_text");
|
|
expect(branches[3].properties).not.toHaveProperty("workflow_document_text");
|
|
expect(operation.responses).toHaveProperty("202");
|
|
expect(runnerApiOperation("GET /api/cases/{caseId}/handoffs").parameters).toEqual(expect.arrayContaining([expect.objectContaining({ name: "cursor" })]));
|
|
for (const action of ["retry", "cancel"]) expect(runnerApiOperation(`POST /api/cases/{caseId}/handoffs/{handoffId}/${action}`).allowedModes).toEqual(["standard"]);
|
|
});
|
|
it("maps dedicated case data tools and describes the optional metadata projection", () => {
|
|
for (const [operationId, tool] of [
|
|
["GET /api/cases/{id}", "read_case"], ["PATCH /api/cases/{id}", "update_case"],
|
|
["GET /api/companies/{companyId}/workflows", "list_workflows"], ["POST /api/companies/{companyId}/workflows", "create_workflow"],
|
|
["GET /api/workflows/{id}", "read_workflow"], ["PATCH /api/workflows/{id}", "update_workflow"],
|
|
]) expect(runnerApiOperation(operationId!).dedicatedTools).toEqual([tool]);
|
|
const legacyCreate = runnerApiOperation("POST /api/companies/{companyId}/cases");
|
|
expect(legacyCreate.dedicatedTools).toEqual([]);
|
|
expect(legacyCreate.requestBody?.content["application/json"].schema.properties).not.toHaveProperty("createOnly");
|
|
for (const method of ["GET", "PUT"]) expect(runnerApiOperation(`${method} /api/cases/{id}/documents/{key}`).dedicatedTools).toEqual([]);
|
|
expect(runnerApiOperation("GET /api/cases/{id}").parameters).toEqual(expect.arrayContaining([
|
|
expect.objectContaining({ name: "includeDocumentBodies", in: "query" }),
|
|
]));
|
|
});
|
|
it.each(["standard", "ask", "planning", "skill_test"])("permits the title-only API in %s mode", workMode => {
|
|
const operationId = "PUT /api/issues/{id}/title";
|
|
const { operation } = validateRunnerApiCall({
|
|
operationId,
|
|
pathParams: { id: context.issueId },
|
|
body: { title: "Fix sign-in redirect", onlyIfProvisional: true },
|
|
}, { ...context, workMode });
|
|
expect(operation.allowedModes).toContain(workMode);
|
|
expect(operation.dedicatedTools).toEqual(["set_task_title"]);
|
|
expect(operation.requestBody?.content["application/json"].schema.additionalProperties).toBe(false);
|
|
});
|
|
it("advertises the conversational recording exception without general approval authority", () => {
|
|
const operationId = "POST /api/issues/{id}/interactions/{interactionId}/resolve-from-comment";
|
|
const operation = runnerApiOperation(operationId);
|
|
expect(operation.allowedModes).toContain("planning");
|
|
expect(operation.allowedModes).not.toContain("ask");
|
|
expect(operation.runnerRestrictions?.join(" ")).toContain("conversational confirmation");
|
|
expect(runnerApiOperation("POST /api/issues/{id}/interactions/{interactionId}/accept").callPolicy).toBe("rest");
|
|
expect(runnerApiOperation(createProject).allowedModes).not.toContain("planning");
|
|
});
|
|
it("accounts for unique operations with resolved request contracts", () => {
|
|
const catalog = runnerApiCatalog();
|
|
expect(catalog.length).toBeGreaterThan(400);
|
|
expect(new Set(catalog.map((entry) => entry.operationId)).size).toBe(
|
|
catalog.length,
|
|
);
|
|
expect(JSON.stringify(catalog)).not.toContain('"$ref"');
|
|
expect(runnerApiOperation("GET /api/companies/{companyId}/decisions").authorization.actor).toBe("board");
|
|
expect(runnerApiOperation("DELETE /api/issues/{id}/documents/{key}").authorization.actor).toBe("board");
|
|
// Agents read their own commentary (or all with commentary:view), so the catalog must not say board-only.
|
|
expect(runnerApiOperation("GET /api/companies/{companyId}/agent-commentary").authorization.actor).toBe("board_or_agent");
|
|
expect(runnerApiOperation("GET /api/companies/{companyId}/agent-commentary/{id}").authorization.actor).toBe("board_or_agent");
|
|
expect(runnerApiOperation("DELETE /api/issues/{id}/documents/{key}").dedicatedTools).toEqual([]);
|
|
expect(runnerApiOperation(createProject).requestBody?.content["application/json"].schema.required).toContain("name");
|
|
expect(runnerApiOperation(createProject).dedicatedTools).toEqual(["create_project"]);
|
|
expect(runnerApiOperation(projects).dedicatedTools).toEqual(["list_projects"]);
|
|
expect(runnerApiOperation("GET /api/companies/{companyId}/project-repositories").dedicatedTools).toEqual(["list_project_repositories"]);
|
|
expect(runnerApiOperation("GET /api/agents/{id}/instructions-bundle/file").dedicatedTools).toEqual(["read_agent_instructions"]);
|
|
expect(runnerApiOperation("PUT /api/agents/{id}/instructions-bundle/file").dedicatedTools).toEqual(["update_agent_instructions"]);
|
|
expect(runnerApiOperation("GET /api/agents/{id}/instructions-bundle/history").dedicatedTools).toEqual(["get_agent_instruction_history"]);
|
|
expect(runnerApiOperation("POST /api/agents/{id}/instructions-bundle/restore").dedicatedTools).toEqual(["restore_agent_instructions"]);
|
|
expect(runnerApiOperation("POST /api/companies/{companyId}/agent-hires").dedicatedTools).toEqual(["hire_agent"]);
|
|
expect(runnerApiOperation("POST /api/companies/{companyId}/agent-hires").dedicatedToolGuidance).toContain("inherits the caller's native runtime");
|
|
});
|
|
it.each(runnerApiCatalog().filter(operation => operation.transport === "rest"))("resolves the catalog route $operationId inside the bound origin", operation => {
|
|
const pathParams = Object.fromEntries(operation.parameters.filter(parameter => parameter.in === "path").map(parameter => [parameter.name, parameter.name === "companyId" ? context.companyId : "fixture-id"]));
|
|
const url = runnerApiUrl(operation, { operationId: operation.operationId, pathParams }, context, "https://paperclip.test");
|
|
expect(url.origin).toBe("https://paperclip.test");
|
|
expect(url.pathname).not.toContain("{");
|
|
expect(operation.responses).toBeDefined();
|
|
expect(operation.authorization.actor).toBeTruthy();
|
|
});
|
|
|
|
it("exposes team preview and install bodies through the actual adapter override search", () => {
|
|
const results = searchRunnerApi({ query: "adapterOverrides install team under manager parentAgentId", limit: 10 }).results;
|
|
const preview = results.find(row => row.operationId === "POST /api/companies/{companyId}/teams/catalog/{catalogId}/preview")!;
|
|
const install = results.find(row => row.operationId === "POST /api/companies/{companyId}/teams/catalog/{catalogId}/install")!;
|
|
const previewBody = preview.requestBody?.content["application/json"].schema;
|
|
const installBody = install.requestBody?.content["application/json"].schema;
|
|
expect(previewBody).toMatchObject({ additionalProperties: false, properties: { targetManagerAgentId: expect.any(Object), nameOverrides: expect.any(Object) } });
|
|
expect(installBody).toMatchObject({ additionalProperties: false, properties: { targetManagerAgentId: expect.any(Object), adapterOverrides: expect.any(Object), secretValues: expect.any(Object) } });
|
|
expect(previewBody.properties).not.toHaveProperty("adapterOverrides");
|
|
});
|
|
|
|
it("discovers proposal revision without confusing it with hiring, approval decisions, or reads", () => {
|
|
const resubmit = "POST /api/approvals/{id}/resubmit";
|
|
for (const query of [
|
|
"revise changes-requested hire proposal",
|
|
"resubmit approval after changes",
|
|
"Change the proposed hire to the cheaper model and send it back for approval.",
|
|
]) expect(searchRunnerApi({ query }).results.map(row => row.operationId), query).toContain(resubmit);
|
|
for (const [query, expected] of [
|
|
["GET /api/approvals/{id}", "GET /api/approvals/{id}"],
|
|
["get approval", "GET /api/approvals/{id}"],
|
|
["hire agent", "POST /api/companies/{companyId}/agent-hires"],
|
|
["approve approval", "POST /api/approvals/{id}/approve"],
|
|
["reject approval", "POST /api/approvals/{id}/reject"],
|
|
[resubmit, resubmit],
|
|
]) expect(searchRunnerApi({ query }).results[0]?.operationId, query).toBe(expected);
|
|
const operation = runnerApiOperation(resubmit);
|
|
expect(operation.requestBody?.content?.["application/json"]?.schema.properties).toHaveProperty("payload");
|
|
expect(operation.description).toContain("returns to pending");
|
|
});
|
|
|
|
it("keeps natural-language function words from hiding the requested API resource", () => {
|
|
expect(searchRunnerApi({ query: "list comments for an issue" }).results[0]?.operationId)
|
|
.toBe("GET /api/issues/{id}/comments");
|
|
for (const [query, operation] of [
|
|
["queued comments for an issue", "GET /api/issues/{id}/queued-comments"],
|
|
["activity for an issue", "GET /api/issues/{id}/activity"],
|
|
["runs for an issue", "GET /api/issues/{id}/runs"],
|
|
["case for task", "GET /api/issues/{issueId}/cases"],
|
|
]) expect(searchRunnerApi({ query }).results.map(row => row.operationId)).toContain(operation);
|
|
expect(searchRunnerApi({ query: "for an the" }).results).toEqual([]);
|
|
});
|
|
|
|
it("ranks natural language, explains dedicated alternatives, and supports exact lookup", () => {
|
|
expect(
|
|
searchRunnerApi({ query: "create project" }).results.map(
|
|
(entry) => entry.operationId,
|
|
),
|
|
).toContain(createProject);
|
|
expect(
|
|
searchRunnerApi({ query: "GET /api/companies/{companyId}/issues" })
|
|
.results[0].dedicatedTools,
|
|
).toContain("search_tasks");
|
|
expect(searchRunnerApi({ query: "nothing-zzzzzzzzzz" }).total).toBe(0);
|
|
});
|
|
it("finds the actual instance-restart operation and explains its authority boundary", () => {
|
|
const operation = searchRunnerApi({ query: "restart instance", limit: 1 }).results[0]!;
|
|
expect(operation.operationId).toBe("POST /api/health/dev-server/restart");
|
|
expect(operation.authorization.actor).toBe("board");
|
|
expect(operation.description).toMatch(/authenticated.*board/i);
|
|
expect(operation.description).toMatch(/agent credential/i);
|
|
expect(operation.description).toMatch(/supervisor/i);
|
|
});
|
|
it("finds the teams catalog when asked for a ready-made team", () => {
|
|
const operation = searchRunnerApi({ query: "ready-made team", limit: 1 }).results[0]!;
|
|
expect(operation.operationId).toBe("GET /api/teams/catalog");
|
|
expect(operation.description).toMatch(/only source of ready-made teams/i);
|
|
expect(operation.description).toContain("no teams catalog page");
|
|
expect(operation.parameters.map(parameter => parameter.name)).toContain("q");
|
|
const preview = runnerApiOperation("POST /api/companies/{companyId}/teams/catalog/{catalogId}/preview")!;
|
|
expect(preview.description).toMatch(/creates no resources/i);
|
|
expect(preview.allowedModes).toEqual(expect.arrayContaining(["standard", "ask", "planning"]));
|
|
expect(runnerApiOperation("POST /api/companies/{companyId}/teams/catalog/{catalogId}/install")!.allowedModes).not.toContain("ask");
|
|
const askContext = { ...context, workMode: "ask" };
|
|
expect(validateRunnerApiCall({ operationId: preview.operationId, pathParams: { catalogId: "team" }, body: {} }, askContext).operation.operationId).toBe(preview.operationId);
|
|
expect(() => validateRunnerApiCall({ operationId: "POST /api/companies/{companyId}/teams/catalog/{catalogId}/install", pathParams: { catalogId: "team" }, body: {} }, askContext)).toThrow();
|
|
const install = runnerApiOperation("POST /api/companies/{companyId}/teams/catalog/{catalogId}/install")!;
|
|
expect(install.description).toMatch(/permission to create agents/i);
|
|
});
|
|
it("discovers where to save project defaults without rewriting existing tasks", () => {
|
|
const operation = searchRunnerApi({ query: "save project defaults", limit: 1 }).results[0]!;
|
|
expect(operation.operationId).toBe("PATCH /api/projects/{id}");
|
|
expect(operation.description).toContain("goalIds");
|
|
expect(operation.description).toContain("description");
|
|
expect(operation.description).toMatch(/existing tasks/i);
|
|
expect(operation.requestBody?.content["application/json"].schema.properties).toHaveProperty("goalIds");
|
|
});
|
|
it("paginates without duplicates and rejects stale or mismatched cursors", () => {
|
|
const first = searchRunnerApi({ query: "project", limit: 1 });
|
|
const second = searchRunnerApi({
|
|
query: "project",
|
|
limit: 1,
|
|
cursor: first.nextCursor!,
|
|
});
|
|
expect(second.results[0].operationId).not.toBe(
|
|
first.results[0].operationId,
|
|
);
|
|
expect(() =>
|
|
searchRunnerApi({ query: "agent", cursor: first.nextCursor! }),
|
|
).toThrow("different query");
|
|
expect(() => searchRunnerApi({ query: "project", limit: 50 })).toThrow();
|
|
});
|
|
});
|
|
|
|
describe("runner API request boundary", () => {
|
|
it.each(runnerApiCatalog().filter(operation => operation.transport === "rest"))(
|
|
"dispatches $operationId with agent identity and preserves route authorization",
|
|
async operation => {
|
|
const request = vi.fn<typeof fetch>(async () =>
|
|
Response.json({ error: "Route permission required" }, { status: 403 }),
|
|
);
|
|
const pathParams = Object.fromEntries(
|
|
[...operation.path.matchAll(/\{([^}]+)\}/g)].map(([, name]) =>
|
|
[name, name === "companyId" ? context.companyId : "fixture"]),
|
|
);
|
|
const result = await executeRunnerApi(
|
|
// A write missing its required body is rejected before dispatch; send an empty one.
|
|
{ operationId: operation.operationId, pathParams, ...(operation.method !== "GET" && operation.requestBody?.required ? { body: emptyRequestBody(operation) } : {}) }, context, io(request),
|
|
);
|
|
expect(operation.callPolicy).toBe("rest");
|
|
expect(result).toMatchObject({ ok: false, status: 403, data: { error: "Route permission required" } });
|
|
expect(request).toHaveBeenCalledOnce();
|
|
const headers = new Headers(request.mock.calls[0][1]?.headers);
|
|
expect(headers.get("Authorization")).toBe("Bearer private-agent-token");
|
|
expect(headers.get("X-Paperclip-Run-Id")).toBe(context.runId);
|
|
},
|
|
);
|
|
it.each([
|
|
"POST /api/mcp/project-tools",
|
|
"POST /api/agents/{id}/claude-login",
|
|
])(
|
|
"directs authentication handshake %s to its existing client",
|
|
async (operationId) => {
|
|
const request = vi.fn<typeof fetch>();
|
|
expect(runnerApiOperation(operationId).transport).toBe("protocol");
|
|
await expect(
|
|
executeRunnerApi({ operationId }, context, io(request)),
|
|
).rejects.toThrow("existing protocol client");
|
|
expect(request).not.toHaveBeenCalled();
|
|
},
|
|
);
|
|
it.each([
|
|
"POST /api/companies/{companyId}/adapters/{type}/login-sessions",
|
|
"POST /api/agents/me/connections/{connectionId}/start-authorization",
|
|
])("calls JSON authentication initiation through REST: %s", async operationId => {
|
|
const request = vi.fn<typeof fetch>(async () => Response.json({ error: "Route permission required" }, { status: 403 }));
|
|
expect(runnerApiOperation(operationId).transport).toBe("rest");
|
|
const result = await executeRunnerApi({ operationId, pathParams: operationId.includes("{type}") ? { type: "codex_local" } : { connectionId: "connection-a" }, body: {} }, context, io(request));
|
|
expect(result).toMatchObject({ status: 403 });
|
|
expect(request).toHaveBeenCalledOnce();
|
|
});
|
|
it.each(
|
|
runnerApiCatalog().filter(
|
|
(operation) =>
|
|
!["GET", "HEAD", "OPTIONS"].includes(operation.method) &&
|
|
operation.path.includes("/routines/{id}/description/annotations"),
|
|
),
|
|
)("preserves routine collaboration $operationId", async (operation) => {
|
|
const request = vi.fn<typeof fetch>(async () =>
|
|
Response.json({ id: "thread", status: "open" }),
|
|
);
|
|
const pathParams = Object.fromEntries(
|
|
operation.parameters
|
|
.filter((parameter) => parameter.in === "path")
|
|
.map((parameter) => [parameter.name, "fixture"]),
|
|
);
|
|
await expect(
|
|
executeRunnerApi(
|
|
{ operationId: operation.operationId, pathParams, body: operation.method === "PATCH" ? { status: "resolved" } : { body: "Note" } },
|
|
context,
|
|
io(request),
|
|
),
|
|
).resolves.toMatchObject({ status: 200 });
|
|
expect(request).toHaveBeenCalledOnce();
|
|
expect(operation.callPolicy).toBe("rest");
|
|
});
|
|
it("keeps routine metadata readable", () => {
|
|
expect(
|
|
validateRunnerApiCall(
|
|
{ operationId: "GET /api/companies/{companyId}/routines" },
|
|
context,
|
|
).operation.callPolicy,
|
|
).toBe("rest");
|
|
});
|
|
it.each([
|
|
["PATCH /api/issues/{id}", { status: "blocked", assigneeAgentId: "teammate", blockedByIssueIds: ["dependency"], reopen: true }],
|
|
["PATCH /api/agents/{id}", { status: "idle" }],
|
|
["POST /api/issues/{id}/comments", { body: "Continue", resume: true }],
|
|
])("passes REST payload fields to route authorization for %s", async (operationId, body) => {
|
|
const request = vi.fn<typeof fetch>(async () => Response.json({ accepted: true }));
|
|
expect(await executeRunnerApi({ operationId: operationId as string, pathParams: { id: "fixture" }, body }, context, io(request)))
|
|
.toMatchObject({ ok: true, status: 200 });
|
|
expect(JSON.parse(request.mock.calls[0][1]?.body as string)).toEqual(body);
|
|
});
|
|
it("retains safe secret metadata discovery", () => {
|
|
for (const operationId of [
|
|
"GET /api/agents/me/secrets",
|
|
"GET /api/companies/{companyId}/secrets/catalog",
|
|
]) {
|
|
expect(
|
|
validateRunnerApiCall({ operationId }, context).operation.callPolicy,
|
|
).toBe("rest");
|
|
}
|
|
});
|
|
it("binds the company and encodes query scalars", () => {
|
|
const input = {
|
|
operationId: projects,
|
|
query: { q: "hello & goodbye", limit: 2, active: false },
|
|
};
|
|
const url = runnerApiUrl(
|
|
runnerApiOperation(projects),
|
|
input,
|
|
context,
|
|
"https://paperclip.test/api",
|
|
);
|
|
expect(url.origin).toBe("https://paperclip.test");
|
|
expect(url.pathname).toBe("/api/companies/company-a/projects");
|
|
expect(url.searchParams.get("q")).toBe("hello & goodbye");
|
|
});
|
|
it.each(["../secrets", ".", "..", "%2e%2e", "abc/def", "abc\\def"])(
|
|
"rejects path injection %s",
|
|
(id) => {
|
|
const input = {
|
|
operationId: "GET /api/projects/{id}",
|
|
pathParams: { id },
|
|
};
|
|
expect(() =>
|
|
runnerApiUrl(
|
|
runnerApiOperation(input.operationId),
|
|
input,
|
|
context,
|
|
"https://paperclip.test",
|
|
),
|
|
).toThrow();
|
|
},
|
|
);
|
|
it("rejects unknown inputs, foreign companies, and mode bypasses", () => {
|
|
expect(() =>
|
|
validateRunnerApiCall(
|
|
{ operationId: projects, headers: { Authorization: "board" } },
|
|
context,
|
|
),
|
|
).toThrow();
|
|
expect(() =>
|
|
validateRunnerApiCall(
|
|
{ operationId: projects, pathParams: { companyId: "foreign" } },
|
|
context,
|
|
),
|
|
).toThrow("another company");
|
|
for (const workMode of ["planning", "ask"]) {
|
|
expect(() =>
|
|
validateRunnerApiCall(
|
|
{ operationId: createProject, body: { name: "bad" } },
|
|
{ ...context, workMode },
|
|
),
|
|
).toThrow("only reads");
|
|
expect(
|
|
validateRunnerApiCall(
|
|
{ operationId: projects },
|
|
{ ...context, workMode },
|
|
).operation.method,
|
|
).toBe("GET");
|
|
}
|
|
});
|
|
it.each([
|
|
["POST /api/agents/me/secrets/{key}/value", { key: "FIXTURE_TOKEN" }],
|
|
["POST /api/companies/{companyId}/exports", {}],
|
|
["POST /api/companies/{companyId}/export", {}],
|
|
])("returns credential responses for %s above the inline limit without plaintext artifacts", async (operationId, pathParams) => {
|
|
const value = "fixture-credential-".repeat(3000);
|
|
const transport = io(async () => Response.json({ value }));
|
|
const save = vi.fn(transport.saveResponse);
|
|
const reserve = vi.fn(async () => vi.fn(async () => {}));
|
|
const result = await executeRunnerApi({ operationId, pathParams }, context, { ...transport, saveResponse: save, reserveResponseCapture: reserve });
|
|
expect(result).toMatchObject({ ok: true, status: 200, data: { value } });
|
|
expect(result).not.toHaveProperty("artifact");
|
|
expect(save).not.toHaveBeenCalled();
|
|
expect(reserve).not.toHaveBeenCalled();
|
|
});
|
|
it("withholds a credential response when the run is revoked during HTTP", async () => {
|
|
let authorized = true;
|
|
const request = vi.fn<typeof fetch>(async () => {
|
|
authorized = false;
|
|
return Response.json({ value: "private-fixture-value" });
|
|
});
|
|
const result = await executeRunnerApi({ operationId: "POST /api/agents/me/secrets/{key}/value", pathParams: { key: "fixture-key" } }, context, {
|
|
...io(request), beforeDispatch: async () => { if (!authorized) throw new Error("Run revoked"); },
|
|
});
|
|
expect(result).toMatchObject({ ok: false, error: "api_transport_failure", outcome: "unknown" });
|
|
expect(JSON.stringify(result)).not.toContain("private-fixture-value");
|
|
expect(request).toHaveBeenCalledOnce();
|
|
});
|
|
it("forwards only server-owned authentication and preserves API denials", async () => {
|
|
const request = vi.fn<typeof fetch>(async () =>
|
|
Response.json({ error: "Board access required" }, { status: 403 }),
|
|
);
|
|
const result = await executeRunnerApi(
|
|
{ operationId: projects },
|
|
context,
|
|
io(request),
|
|
);
|
|
expect(result).toMatchObject({
|
|
ok: false,
|
|
status: 403,
|
|
data: { error: "Board access required" },
|
|
});
|
|
const options = request.mock.calls[0][1]!;
|
|
expect(new Headers(options.headers).get("Authorization")).toBe(
|
|
"Bearer private-agent-token",
|
|
);
|
|
expect(new Headers(options.headers).get("X-Paperclip-Run-Id")).toBe(
|
|
context.runId,
|
|
);
|
|
expect(options.redirect).toBe("manual");
|
|
expect(JSON.stringify(result)).not.toContain("private-agent-token");
|
|
});
|
|
it("never retries a mutation after a transport failure", async () => {
|
|
const request = vi.fn<typeof fetch>(async () => {
|
|
throw new Error("socket reset");
|
|
});
|
|
expect(
|
|
await executeRunnerApi(
|
|
{ operationId: createProject, body: { name: "created?" } },
|
|
context,
|
|
io(request),
|
|
),
|
|
).toMatchObject({ outcome: "unknown", status: null });
|
|
expect(request).toHaveBeenCalledTimes(1);
|
|
});
|
|
it.each([500, 502, 503, 408, 302])(
|
|
"does not claim a mutation was unapplied after HTTP %s",
|
|
async (status) => {
|
|
const request = vi.fn<typeof fetch>(async () =>
|
|
Response.json(
|
|
{ error: "Request interrupted after possible commit" },
|
|
{ status },
|
|
),
|
|
);
|
|
expect(
|
|
await executeRunnerApi(
|
|
{ operationId: createProject, body: { name: "Maybe created" } },
|
|
context,
|
|
io(request),
|
|
),
|
|
).toMatchObject({ status, outcome: "unknown", ok: false });
|
|
expect(request).toHaveBeenCalledTimes(1);
|
|
},
|
|
);
|
|
it("retains uncertainty when a successful mutation returns malformed JSON", async () => {
|
|
const request = vi.fn<typeof fetch>(
|
|
async () =>
|
|
new Response("truncated{", {
|
|
status: 201,
|
|
headers: { "content-type": "application/json" },
|
|
}),
|
|
);
|
|
expect(
|
|
await executeRunnerApi(
|
|
{ operationId: createProject, body: { name: "Maybe created" } },
|
|
context,
|
|
io(request),
|
|
),
|
|
).toMatchObject({
|
|
status: 201,
|
|
outcome: "unknown",
|
|
error: "invalid_json_response",
|
|
});
|
|
});
|
|
it("rejects a string-encoded object before HTTP and allows a corrected request", async () => {
|
|
const request = vi.fn<typeof fetch>(async () =>
|
|
Response.json({ name: "Borealis" }, { status: 201 }),
|
|
);
|
|
await expect(
|
|
executeRunnerApi(
|
|
{ operationId: createProject, body: '{"name":"Borealis"}' },
|
|
context,
|
|
io(request),
|
|
),
|
|
).rejects.toThrow("not a JSON-encoded string");
|
|
expect(request).not.toHaveBeenCalled();
|
|
expect(
|
|
await executeRunnerApi(
|
|
{ operationId: createProject, body: { name: "Borealis" } },
|
|
context,
|
|
io(request),
|
|
),
|
|
).toMatchObject({ status: 201 });
|
|
expect(request).toHaveBeenCalledTimes(1);
|
|
});
|
|
it("does not follow redirects or pretend empty responses failed", async () => {
|
|
expect(
|
|
await executeRunnerApi(
|
|
{ operationId: projects },
|
|
context,
|
|
io(
|
|
async () =>
|
|
new Response(null, {
|
|
status: 302,
|
|
headers: { Location: "https://foreign.test" },
|
|
}),
|
|
),
|
|
),
|
|
).toMatchObject({ ok: false, error: "api_redirect_not_followed" });
|
|
expect(
|
|
await executeRunnerApi(
|
|
{ operationId: projects },
|
|
context,
|
|
io(async () => new Response(null, { status: 204 })),
|
|
),
|
|
).toMatchObject({ ok: true, status: 204, data: null });
|
|
});
|
|
it("uploads multipart artifacts and returns download references", async () => {
|
|
const request = vi.fn<typeof fetch>(
|
|
async () =>
|
|
new Response("download", {
|
|
headers: { "content-type": "application/octet-stream" },
|
|
}),
|
|
);
|
|
const result = await executeRunnerApi(
|
|
{
|
|
operationId: createProject,
|
|
files: [{ artifactId: "a", field: "package" }],
|
|
body: { meta: { name: "example" } },
|
|
},
|
|
context,
|
|
io(request),
|
|
);
|
|
const body = request.mock.calls[0][1]!.body as FormData;
|
|
expect(body.get("meta")).toBe('{"name":"example"}');
|
|
expect(await (body.get("package") as File).text()).toBe("test");
|
|
expect(
|
|
new Headers(request.mock.calls[0][1]!.headers).has("Content-Type"),
|
|
).toBe(false);
|
|
expect(result).toMatchObject({
|
|
artifact: { artifactId: "artifact-a", byteSize: 8 },
|
|
preview: null,
|
|
});
|
|
});
|
|
it("encodes text and raw file bodies without pretending they are JSON", async () => {
|
|
const request = vi.fn<typeof fetch>(
|
|
async () => new Response(null, { status: 204 }),
|
|
);
|
|
await executeRunnerApi(
|
|
{
|
|
operationId: createProject,
|
|
body: "plain text",
|
|
contentType: "text/plain",
|
|
},
|
|
context,
|
|
io(request),
|
|
);
|
|
expect(request.mock.calls[0][1]?.body).toBe("plain text");
|
|
await executeRunnerApi(
|
|
{
|
|
operationId: createProject,
|
|
files: [{ path: "sample.bin" }],
|
|
contentType: "application/octet-stream",
|
|
},
|
|
context,
|
|
io(request),
|
|
);
|
|
expect(
|
|
Buffer.from(request.mock.calls[1][1]?.body as Uint8Array).toString(),
|
|
).toBe("test");
|
|
await expect(
|
|
executeRunnerApi(
|
|
{
|
|
operationId: createProject,
|
|
body: "unexpected",
|
|
files: [{ artifactId: "a" }],
|
|
contentType: "application/octet-stream",
|
|
},
|
|
context,
|
|
io(request),
|
|
),
|
|
).rejects.toThrow();
|
|
});
|
|
it("keeps protocol clients separate from REST dispatch", () => {
|
|
const protocols = runnerApiCatalog().filter(operation => operation.transport === "protocol");
|
|
expect(protocols.some(operation => operation.path.endsWith("/events/ws"))).toBe(true);
|
|
for (const operation of protocols) {
|
|
expect(() => validateRunnerApiCall({ operationId: operation.operationId }, context)).toThrow("protocol client");
|
|
}
|
|
});
|
|
it("keeps skill-test mode consistent with the advertised tool contract", () => {
|
|
expect(
|
|
validateRunnerApiCall(
|
|
{ operationId: createProject, body: { name: "Skill fixture" } },
|
|
{ ...context, workMode: "skill_test" },
|
|
).operation.method,
|
|
).toBe("POST");
|
|
});
|
|
it("passes uploaded JSON bodies through the same route validation as REST", async () => {
|
|
const request = vi.fn<typeof fetch>(async () => Response.json({ error: "Board access required" }, { status: 403 }));
|
|
const transport = io(request);
|
|
transport.readFile = async () => ({ bytes: Buffer.from('{"status":"idle"}'), filename: "update.json", contentType: "application/json" });
|
|
expect(await executeRunnerApi({ operationId: "PATCH /api/agents/{id}", pathParams: { id: "fixture" }, files: [{ path: "update.json" }], contentType: "application/json" }, context, transport))
|
|
.toMatchObject({ ok: false, status: 403 });
|
|
expect(Buffer.from(request.mock.calls[0][1]?.body as Uint8Array).toString()).toBe('{"status":"idle"}');
|
|
});
|
|
it("reads text windows from a live response and supplies stable snapshots", async () => {
|
|
const text = "prefix\n" + "🧭é\n".repeat(6000) + "END-OF-EVIDENCE";
|
|
const saveResponse = vi.fn(io(fetch).saveResponse);
|
|
const request = vi.fn<typeof fetch>(async () => new Response(text, { headers: { "content-type": "application/json", etag: '"content-identity"' } }));
|
|
let offsetBytes = 0;
|
|
let received = "";
|
|
do {
|
|
const result = await executeRunnerApi({ operationId: projects, responseText: { offsetBytes, limitBytes: 4096 } }, context, { ...io(request), saveResponse }) as any;
|
|
expect(result).toMatchObject({ ok: true, etag: '"content-identity"', responseText: { offsetBytes, totalBytes: Buffer.byteLength(text) } });
|
|
expect(Buffer.byteLength(result.data)).toBeLessThanOrEqual(4096);
|
|
received += result.data;
|
|
offsetBytes = result.responseText.nextOffsetBytes;
|
|
} while (offsetBytes !== null);
|
|
expect(received).toBe(text);
|
|
expect(saveResponse).toHaveBeenCalled();
|
|
expect(request.mock.calls.every(([, options]) => new Headers(options?.headers).get("Authorization") === "Bearer private-agent-token")).toBe(true);
|
|
});
|
|
it.each(["text/plain", "application/octet-stream"])("does not duplicate an unpaged multi-gigabyte asset (%s)", async contentType => {
|
|
const totalBytes = 3 * 1024 * 1024 * 1024;
|
|
const saveResponse = vi.fn(io(fetch).saveResponse);
|
|
const request = vi.fn<typeof fetch>(async (_url, init) => {
|
|
expect(new Headers(init?.headers).get("range")).toBe("bytes=0-24576");
|
|
return new Response(Buffer.alloc(24577, 65), { status: 206, headers: {
|
|
"content-type": contentType, "content-range": `bytes 0-24576/${totalBytes}`, etag: `"${"a".repeat(64)}"`,
|
|
} });
|
|
});
|
|
const result = await executeRunnerApi({ operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: "existing" } }, context, { ...io(request), saveResponse });
|
|
expect(result).toMatchObject({ ok: true, status: 206, byteSize: totalBytes, artifact: { artifactId: "existing", byteSize: totalBytes, sha256: "a".repeat(64) } });
|
|
expect(saveResponse).not.toHaveBeenCalled();
|
|
expect(request).toHaveBeenCalledTimes(1);
|
|
});
|
|
it("reads a page beyond ten MiB in a multi-gigabyte asset", async () => {
|
|
const offsetBytes = 3 * 1024 * 1024 * 1024;
|
|
const totalBytes = offsetBytes + 100;
|
|
const request = vi.fn<typeof fetch>(async (_url, init) => {
|
|
expect(new Headers(init?.headers).get("range")).toBe(`bytes=${offsetBytes - 1}-${offsetBytes + 4}`);
|
|
return new Response("abcdef", { status: 206, headers: {
|
|
"content-type": "text/plain", "content-range": `bytes ${offsetBytes - 1}-${offsetBytes + 4}/${totalBytes}`,
|
|
} });
|
|
});
|
|
expect(await executeRunnerApi({ operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: "large" }, responseText: { offsetBytes, limitBytes: 4 } }, context, io(request))).toMatchObject({
|
|
ok: true, data: "bcde", responseText: { offsetBytes, nextOffsetBytes: offsetBytes + 4, totalBytes },
|
|
});
|
|
});
|
|
it.each([true, false])("saves responses above ten MiB (known length: %s)", async knownLength => {
|
|
const bytes = Buffer.alloc(12 * 1024 * 1024, 65);
|
|
const request = vi.fn<typeof fetch>(async () => new Response(bytes, { headers: {
|
|
"content-type": "text/plain", ...(knownLength ? { "content-length": String(bytes.length) } : {}),
|
|
} }));
|
|
const result = await executeRunnerApi({ operationId: projects }, context, io(request));
|
|
expect(result).toMatchObject({ ok: true, artifact: { byteSize: bytes.length }, byteSize: bytes.length });
|
|
});
|
|
it("pages a twelve MiB asset with linear transfer and UTF-8 boundaries", async () => {
|
|
const bytes = Buffer.from("🧭é".repeat(Math.floor(12 * 1024 * 1024 / 6)));
|
|
let transferred = 0;
|
|
const request = vi.fn<typeof fetch>(async (_url, init) => {
|
|
const range = new Headers(init?.headers).get("range");
|
|
const match = /^bytes=(\d+)-(\d+)$/.exec(range ?? "");
|
|
if (!match) { transferred += bytes.length; return new Response(bytes, { headers: { "content-type": "text/plain" } }); }
|
|
const start = Number(match[1]), end = Math.min(Number(match[2]), bytes.length - 1);
|
|
const part = bytes.subarray(start, end + 1); transferred += part.length;
|
|
return new Response(part, { status: 206, headers: { "content-type": "text/plain", "content-range": `bytes ${start}-${end}/${bytes.length}` } });
|
|
});
|
|
let offsetBytes: number | null = 0;
|
|
const parts: string[] = [];
|
|
do {
|
|
const page: any = await executeRunnerApi({ operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: "large" }, responseText: { offsetBytes } }, context, io(request));
|
|
expect(page.ok).toBe(true);
|
|
parts.push(page.data); offsetBytes = page.responseText.nextOffsetBytes;
|
|
} while (offsetBytes !== null);
|
|
expect(parts.join("")).toBe(bytes.toString());
|
|
expect(transferred).toBeLessThan(bytes.length + request.mock.calls.length * 5);
|
|
expect(request.mock.calls.every(([, init]) => new Headers(init?.headers).has("range"))).toBe(true);
|
|
});
|
|
it.each([
|
|
["bytes 4-8/20", "12345"], // wrong start
|
|
["bytes 3-7/20", "12345"], // early end
|
|
["bytes 3-8/20", "12345"], // truncated body
|
|
["bytes 3-8/9007199254740992", "123456"], // imprecise total
|
|
["bytes 3-8/*", "123456"], // unknown total
|
|
[null, "123456"],
|
|
])("rejects inconsistent partial asset receipts: %s", async (range, body) => {
|
|
const result = await executeRunnerApi({ operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: "saved" }, responseText: { offsetBytes: 4, limitBytes: 4 } }, context, io(async () => new Response(body, { status: 206, headers: { "content-type": "text/plain", ...(range ? { "content-range": range } : {}) } })));
|
|
expect(result).toMatchObject({ ok: false, error: "response_text_invalid_range" });
|
|
});
|
|
it("accepts EOF and rejects an offset inside a UTF-8 code point in ranged assets", async () => {
|
|
const call = (offsetBytes: number, bytes: Buffer, range: string) => executeRunnerApi({ operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: "saved" }, responseText: { offsetBytes, limitBytes: 4 } }, context, io(async () => new Response(new Uint8Array(bytes), { status: 206, headers: { "content-type": "text/plain", "content-range": range } })));
|
|
expect(await call(2, Buffer.from([0xa9]), "bytes 1-1/2")).toMatchObject({ ok: true, data: "", responseText: { nextOffsetBytes: null, totalBytes: 2 } });
|
|
expect(await call(1, Buffer.from("é"), "bytes 0-1/2")).toMatchObject({ ok: false, error: "response_text_invalid_offset" });
|
|
});
|
|
it("does not silently accept an asset server that ignores the byte range", async () => {
|
|
const result = await executeRunnerApi({ operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: "saved" }, responseText: { limitBytes: 4 } }, context, io(async () => new Response("unbounded", { headers: { "content-type": "text/plain" } })));
|
|
expect(result).toMatchObject({ ok: false, error: "api_transport_failure" });
|
|
});
|
|
it.each([{ offsetBytes: -1 }, { limitBytes: 0 }, { limitBytes: 3 }, { limitBytes: 24577 }, { offsetBytes: 0.5 }, { offsetBytes: Number.MAX_SAFE_INTEGER + 1 }])("rejects invalid text windows before dispatch: %j", async responseText => {
|
|
const request = vi.fn<typeof fetch>();
|
|
await expect(executeRunnerApi({ operationId: projects, responseText }, context, io(request))).rejects.toThrow("Invalid call_api arguments");
|
|
expect(request).not.toHaveBeenCalled();
|
|
});
|
|
it("does not page mutation responses by repeating the mutation", async () => {
|
|
const request = vi.fn<typeof fetch>();
|
|
await expect(executeRunnerApi({ operationId: createProject, body: { name: "Once" }, responseText: { offsetBytes: 4 } }, context, io(request))).rejects.toThrow("This request was not sent");
|
|
expect(request).not.toHaveBeenCalled();
|
|
});
|
|
it.each([{ limitBytes: 8192 }, { offsetBytes: 0, limitBytes: 4096 }])("refuses a new write whose responseText sets a window before sending: %j", async responseText => {
|
|
const request = vi.fn<typeof fetch>(async () => Response.json({ id: "company-a", name: "Acorn Studio" }));
|
|
const call = { operationId: "PATCH /api/companies/{companyId}", pathParams: { companyId: "company-a" }, body: { name: "Acorn Studio" }, responseText };
|
|
const error = await executeRunnerApi(call, context, io(request)).then(() => null, (caught: Error) => caught);
|
|
expect(request).not.toHaveBeenCalled();
|
|
expect(error?.message).toContain("This request was not sent: responseText applies only to GET reads and never carries request fields.");
|
|
expect(error?.message).toContain("Resend PATCH /api/companies/{companyId} once without responseText, with its request fields in body");
|
|
expect(validateRunnerApiCall(call, context).input).not.toHaveProperty("responseText");
|
|
});
|
|
it.each([
|
|
{ responseText: {} },
|
|
{ responseText: { offsetBytes: 0 } },
|
|
{ saveResponse: false },
|
|
{ responseText: {}, saveResponse: false },
|
|
])("sends a write once when its read options are empty or default: %j", async options => {
|
|
const request = vi.fn<typeof fetch>(async () => Response.json({ ok: true }));
|
|
const result = await executeRunnerApi({ operationId: "POST /api/issues/{id}/inbox-archive", pathParams: { id: "issue-a" }, body: {}, ...options }, context, io(request));
|
|
expect(request).toHaveBeenCalledTimes(1);
|
|
const [url, init] = request.mock.calls[0]!;
|
|
expect(String(url)).toMatch(/\/api\/issues\/issue-a\/inbox-archive$/);
|
|
expect(init?.method).toBe("POST");
|
|
expect(JSON.parse(String(init?.body))).toEqual({});
|
|
expect(result).toMatchObject({ ok: true, status: 200 });
|
|
expect(result).not.toHaveProperty("responseText");
|
|
});
|
|
it("still sends the same write once without responseText", async () => {
|
|
const request = vi.fn<typeof fetch>(async () => Response.json({ id: "company-a", name: "Acorn Studio" }));
|
|
const result = await executeRunnerApi({ operationId: "PATCH /api/companies/{companyId}", pathParams: { companyId: "company-a" }, body: { name: "Acorn Studio" } }, context, io(request));
|
|
expect(request).toHaveBeenCalledTimes(1);
|
|
expect(JSON.parse(String(request.mock.calls[0]![1]?.body))).toEqual({ name: "Acorn Studio" });
|
|
expect(result).toMatchObject({ ok: true, status: 200, data: { name: "Acorn Studio" } });
|
|
});
|
|
it("rejects a PATCH with no body instead of sending a no-op update", async () => {
|
|
const request = vi.fn<typeof fetch>();
|
|
const call = { operationId: "PATCH /api/companies/{companyId}", pathParams: { companyId: "company-a" }, query: {}, responseText: { limitBytes: 8192 } };
|
|
await expect(executeRunnerApi(call, context, io(request))).rejects.toThrow(/not sent.*requires body.*name.*Received arguments: operationId, pathParams, query, responseText\./);
|
|
expect(() => validateRunnerApiCall({ operationId: "PATCH /api/companies/{companyId}", pathParams: { companyId: "company-a" }, body: {} }, context)).not.toThrow();
|
|
// Recovery validates saved calls without dispatch: an older accepted body-less PATCH stays readable.
|
|
expect(() => validateRunnerApiCall({ operationId: "PATCH /api/companies/{companyId}", pathParams: { companyId: "company-a" } }, context)).not.toThrow();
|
|
expect(request).not.toHaveBeenCalled();
|
|
});
|
|
// Recorded cohort writes: the model sent read-only options and no body.
|
|
it.each([
|
|
{ operationId: "POST /api/issues/{id}/interactions/{interactionId}/resolve-from-comment", pathParams: { id: "issue-a", interactionId: "interaction-a" }, query: {}, saveResponse: false },
|
|
{ operationId: "PATCH /api/issues/{id}", pathParams: { id: "issue-a" }, query: {}, saveResponse: false },
|
|
{ operationId: "POST /api/issues/{id}/comments", pathParams: { id: "issue-a" }, query: {}, responseText: { limitBytes: 8192 }, saveResponse: false },
|
|
] as RunnerApiCall[])("refuses a write with no body before sending and names the shape to resend: $operationId", async call => {
|
|
const request = vi.fn<typeof fetch>();
|
|
const error = await executeRunnerApi(call, context, io(request)).then(() => null, (caught: Error) => caught);
|
|
expect(request).not.toHaveBeenCalled();
|
|
expect(error?.message).toContain(`This request was not sent: ${call.operationId} requires body`);
|
|
expect(error?.message).toContain(`Received arguments: ${Object.keys(call).sort().join(", ")}.`);
|
|
expect(error?.message).toContain('Resend with a top-level body argument');
|
|
expect(error?.message).toMatch(/saveResponse never carr(y|ies) request fields/);
|
|
// The example in the correction is a call that is sent when copied as is.
|
|
const example = exampleIn(error!.message);
|
|
expect(example.pathParams).toEqual(call.pathParams);
|
|
const sent = vi.fn<typeof fetch>(async () => Response.json({ ok: true }));
|
|
await expect(executeRunnerApi(example as RunnerApiCall, context, io(sent))).resolves.toMatchObject({ ok: true });
|
|
expect(sent).toHaveBeenCalledTimes(1);
|
|
});
|
|
it("gives a copyable issue update that the issue route accepts", () => {
|
|
const example = JSON.parse(runnerApiCallExample(runnerApiOperation("PATCH /api/issues/{id}"), { id: "issue-a" }));
|
|
expect(example).toEqual({ operationId: "PATCH /api/issues/{id}", pathParams: { id: "issue-a" }, body: { title: "<title>" } });
|
|
expect(updateIssueSchema.safeParse(example.body).success).toBe(true);
|
|
});
|
|
it("gives every write that requires a body an example its request schema accepts", () => {
|
|
const AjvCtor = ((Ajv as unknown as { default?: typeof Ajv }).default ?? Ajv) as unknown as new (options: object) => { validate(schema: unknown, data: unknown): boolean; errorsText(): string };
|
|
const ajv = new AjvCtor({ strict: false, validateFormats: false, allErrors: true });
|
|
const rejected: string[] = [];
|
|
let checked = 0;
|
|
for (const operation of runnerApiCatalog()) {
|
|
const schema = operation.requestBody?.content?.["application/json"]?.schema;
|
|
if (!operation.requestBody?.required || !schema || operation.transport !== "rest") continue;
|
|
checked++;
|
|
const example = JSON.parse(runnerApiCallExample(operation, {}));
|
|
const { input } = validateRunnerApiCall(example, { ...context, workMode: operation.allowedModes[0] ?? "standard" });
|
|
assertRunnerApiDispatchable(operation, input, example);
|
|
if (!ajv.validate(jsonSchemaFromOpenApi(schema), example.body)) rejected.push(`${operation.operationId}: ${ajv.errorsText()}`);
|
|
}
|
|
expect(checked).toBeGreaterThan(200);
|
|
expect(rejected).toEqual([]);
|
|
});
|
|
it("still sends writes whose body is present or optional", async () => {
|
|
const request = vi.fn<typeof fetch>(async () => Response.json({ ok: true }));
|
|
await executeRunnerApi({ operationId: "POST /api/issues/{id}/comments", pathParams: { id: "issue-a" }, body: { body: "Done" } }, context, io(request));
|
|
expect(request).toHaveBeenCalledTimes(1);
|
|
const optional = runnerApiCatalog().find(operation => operation.method === "POST" && operation.requestBody
|
|
&& !((operation.requestBody as any).content?.["application/json"]?.schema?.required?.length) && operation.allowedModes.includes("standard")
|
|
&& !operation.path.includes("{") && operation.transport === "rest");
|
|
if (optional) {
|
|
request.mockClear();
|
|
await executeRunnerApi({ operationId: optional.operationId }, context, io(request)).catch(() => undefined);
|
|
expect(request).toHaveBeenCalledTimes(1);
|
|
}
|
|
});
|
|
it("keeps read-only save options off the call_api schema the model sees", () => {
|
|
const callApi = CAPABILITY_SEMANTIC_TOOL_CATALOG.find(descriptor => descriptor.operationId === "call_api")!;
|
|
const properties = Object.keys(callApi.inputSchema.properties ?? {});
|
|
expect(properties).toContain("body");
|
|
expect(properties).not.toContain("saveResponse");
|
|
expect(callApi.description).toContain("save_api_response");
|
|
const save = runnerBridgeDefinitions({ workspace: false, skills: false, api: true, mode: "standard" }).find(tool => tool.name === "save_api_response");
|
|
expect(Object.keys((save?.inputSchema as any).properties).sort()).toEqual(["operationId", "pathParams", "query"]);
|
|
expect(runnerBridgeDefinitions({ workspace: false, skills: false, api: false, mode: "standard" }).some(tool => tool.name === "save_api_response")).toBe(false);
|
|
});
|
|
it("rejects binary, invalid UTF-8 and offsets that split text", async () => {
|
|
for (const [body, type, offsetBytes] of [
|
|
[Buffer.from("binary"), "application/octet-stream", 0],
|
|
[Buffer.from([0xff]), "text/plain", 0],
|
|
[Buffer.concat([Buffer.from([0xc2]), Buffer.alloc(24576, 0x80)]), "text/plain", 0],
|
|
[Buffer.from("é"), "text/plain", 1],
|
|
[Buffer.from("short"), "text/plain", 6],
|
|
] as const) {
|
|
const saveResponse = vi.fn(io(fetch).saveResponse);
|
|
const result = await executeRunnerApi({ operationId: projects, responseText: { offsetBytes } }, context, { ...io(async () => new Response(body, { headers: { "content-type": type } })), saveResponse });
|
|
expect(result).toMatchObject({ ok: false, error: expect.stringContaining("response_text") });
|
|
expect(saveResponse).not.toHaveBeenCalled();
|
|
}
|
|
});
|
|
it("preserves denials for bounded text reads", async () => {
|
|
expect(await executeRunnerApi({ operationId: projects, responseText: {} }, context, io(async () => Response.json({ error: "denied" }, { status: 403 })))).toMatchObject({ ok: false, status: 403, data: '{"error":"denied"}' });
|
|
});
|
|
it("bounds streamed responses even without content-length", async () => {
|
|
await expect(
|
|
readBoundedResponse(new Response("too large"), 3),
|
|
).rejects.toThrow("transfer limit");
|
|
});
|
|
});
|
|
|
|
/** The JSON call that follows "for example: " in a correction. */
|
|
function exampleIn(message: string): Record<string, unknown> {
|
|
const start = message.indexOf("{", message.indexOf("for example: "));
|
|
let depth = 0;
|
|
for (let index = start; index < message.length; index++) {
|
|
if (message[index] === "{") depth++;
|
|
if (message[index] === "}" && --depth === 0) return JSON.parse(message.slice(start, index + 1)) as Record<string, unknown>;
|
|
}
|
|
throw new Error("no example in message");
|
|
}
|
|
|
|
/** OpenAPI 3.0 schema keywords as JSON Schema, so Ajv can check them. */
|
|
function jsonSchemaFromOpenApi(value: unknown): unknown {
|
|
if (Array.isArray(value)) return value.map(jsonSchemaFromOpenApi);
|
|
if (!value || typeof value !== "object") return value;
|
|
// Values the model must supply itself (ids, secrets, codes, evidence) appear as
|
|
// placeholders, so their text constraints are not checked; structure is.
|
|
// The schemas come from Zod, whose unions take the first match: anyOf, not oneOf.
|
|
const { nullable, exclusiveMinimum, exclusiveMaximum, minimum, maximum, pattern: _pattern, minLength: _minLength, oneOf, ...rest } = value as Record<string, unknown>;
|
|
const schema: Record<string, unknown> = Object.fromEntries(Object.entries(rest).map(([key, entry]) => [key, jsonSchemaFromOpenApi(entry)]));
|
|
if (oneOf) schema.anyOf = jsonSchemaFromOpenApi(oneOf);
|
|
if (exclusiveMinimum === true) schema.exclusiveMinimum = minimum; else if (minimum !== undefined) schema.minimum = minimum;
|
|
if (typeof exclusiveMinimum === "number") schema.exclusiveMinimum = exclusiveMinimum;
|
|
if (exclusiveMaximum === true) schema.exclusiveMaximum = maximum; else if (maximum !== undefined) schema.maximum = maximum;
|
|
if (typeof exclusiveMaximum === "number") schema.exclusiveMaximum = exclusiveMaximum;
|
|
if (nullable === true && typeof schema.type === "string") schema.type = [schema.type, "null"];
|
|
return schema;
|
|
}
|