Files
PaperClipAI/server/src/services/native-runtime/runner-api.integration.test.ts
T
DottaandClaude Opus 5.5 b8207c4ea9 fix(runner): evict finished API snapshots instead of blocking reads at the company quota (#21133)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs read the Paperclip API through `call_api`. Large
responses are saved as "API snapshot" assets so the agent can read them
in pages.
> - Each company has a 20 GiB snapshot quota. Before a large read, the
server reserves 1 GiB of worst-case space against that quota.
> - Snapshot assets were never removed. Reservations from interrupted
captures (crash, restart, ambiguous upload) stayed at 1 GiB each until
"an operator reconciles" them. Nothing did that.
> - After a morning of high retry volume, the quota filled. Every large
read then failed with `api_response_company_storage_limit`, and agents
blocked their own tasks and asked an operator to free storage.
> - This pull request makes company admission reclaim space by itself:
it releases stale reservations of finished runs and evicts the oldest
snapshots of finished runs, oldest first, only as much as the new read
needs.
> - The benefit is that a read never fails because old snapshots filled
the quota, and no person must clear storage by hand.

## Linked Issues or Issue Description

**What happened?**

On staging, about 97 tasks of one agent had successful runs but ended in
`blocked`. In each run, `GET` calls through `call_api` for large records
(for example heartbeat runs and their events) failed with
`api_response_company_storage_limit`: "Company API snapshot storage
quota is exhausted. Read an existing asset or ask the operator to remove
old snapshots or increase the quota." The agent then reported a
task-wide blocker owned by "the instance storage operator". At least 15
tasks had a final comment such as "Blocked: company API snapshot storage
is exhausted". The error message itself told the agent that an operator
must act.

**Expected behavior**

A read must not fail because old snapshots filled the company quota. The
server must reclaim the space automatically. Only snapshots that active
runs still hold, or that users attached to issues, cases or logos, may
stay.

**Steps to reproduce**

1. Set `PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES` to 2 GiB.
2. Store a 1 GiB `runner-api` snapshot asset from a finished run, or
leave two 1 GiB unattached reservations from finished runs that are
older than 20 minutes.
3. In a running native run, call `call_api` for a response larger than
the inline limit.
4. Before this change, the call fails with
`api_response_company_storage_limit`. After this change, the call
succeeds and the oldest finished snapshot is evicted.

**Paperclip version or commit**

Staging deploy `704b3f4d63`. Fix is based on `origin/master` at
`8bdd53c11f`.

**Deployment mode**

Hosted (staging cloud), native runtime with Daytona sandboxes.

**Agent adapter(s) involved**

Native Paperclip runner (Codex app server, `gpt-6.1-sol`). The defect is
in the server-side `call_api` capture admission, so it affects every
native adapter.

## What Changed

- Added
`server/src/services/native-runtime/runner-api-capture-retention.ts`. It
measures company snapshot usage and reclaims space under the existing
per-company advisory lock.
- Reclaim step 1: release unattached reservations older than twice the
capture deadline (20 minutes) whose run is missing or not `queued`,
`running` or `scheduled_retry`.
- Reclaim step 2: evict snapshot assets oldest first until the new
reservation fits. A snapshot is eligible only when it is older than 1
hour, every run that captured it has finished (succeeded, interrupted,
failed, cancelled or timed out), and no issue attachment, case
attachment or company logo links to it. Each eviction writes an
`asset.evicted` activity-log row with the asset id and the owning run
id.
- A capture upload that finishes after its reservation was reclaimed is
rejected in the publish transaction. The same transaction records the
object for deletion, and the bounded cleanup removes it. The upload has
a 10-minute abort signal.
- Failed deletes are logged per company and counted with the
`runner_api_capture_delete_failed` metric. When cleanup keeps failing,
the read error says so in plain words.
- Eviction credits deletes that are already pending before it chooses
more snapshots. Eviction is durable. In the same transaction that
deletes an asset row, it inserts a reservation row with
`pending_delete_object_key`, so the bytes stay counted. Admission then
deletes the stored objects in pages, under a 30-second abort signal, and
removes each row only after its delete succeeds. A failed or interrupted
delete is retried at the next over-quota admission. No background poller
is added.
- Added the `pending_delete_object_key` column and an index on
`runner_api_response_reservations.asset_id` (migration 0472).
- `#reserveApiResponseCapture` now runs reclaim only when the quota
would be exceeded, so reads below the quota do no extra work. After
marking evictions it finishes the deletes and retries admission once.
- When no space can be reclaimed, the error message now names what holds
the space (runs still working, files attached to tasks, deletes in
progress). It no longer tells the agent to ask an operator.
- Replaced the test that asserted old snapshots block reads with tests
for eviction, multi-page eviction, durable delete retry, a bounded
stalled delete, stale-reservation release, and protection of active and
attached snapshots.

## Verification

- Red: the three new tests in `runner-api.integration.test.ts` fail on
`origin/master` (two get `api_response_company_storage_limit`, one gets
the old "ask the operator" message).
- Green: `cd server && npx vitest run
src/services/native-runtime/runner-api.integration.test.ts` passes 70/70
with the fix.
- The multi-page test failed first with
`api_response_company_storage_limit` because pending deletes ran in one
200-row batch. The paged delete loop fixed it.
- `cd server && npx tsc --noEmit -p .` passes.

## Risks

- An agent that reads a saved snapshot of a finished run after it was
evicted gets a not-found response. It can read the API again. Eviction
happens only under quota pressure, oldest first.
- A released stale reservation can leave an unknown storage object from
an ambiguous upload. These bytes were never linked to an asset, so they
were already unreachable. This is the same storage the old code kept
counted forever.
- Eviction runs inside the company capture lock. The candidate query is
bounded to pages of 200 rows and uses the new `asset_id` index.
- Migration 0472 adds one nullable column (`ADD COLUMN IF NOT EXISTS`)
and builds one index with `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. It
is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`. It is additive and
safe for old and new servers during a rolling deploy.
- During a storage outage, an over-quota read still fails until deletes
succeed. The bytes stay counted, so the quota is never exceeded.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, browser reads of staging).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:30:38 -06:00

1593 lines
118 KiB
TypeScript

import { existsSync } from "node:fs";
import { createHash, randomUUID } from "node:crypto";
import { chmod, writeFile, symlink, mkdir, open } from "node:fs/promises";
import { join, dirname } from "node:path";
import { and, eq, isNotNull } from "drizzle-orm";
import { agents, agentApiKeys, agentWakeupRequests, activityLog, assets, authUsers, issueAttachments, cases, caseAttachments, companyLogos, toolRuntimeMetricCounters, documents, heartbeatRuns, issues, projects, routineDocuments, routineRevisions, routines, runnerApiResponseReservations, companySkills, companySkillVersions, companySkillTestRuns, companyMemberships, principalPermissionGrants, budgetPolicies, companies } from "@paperclipai/db";
import { initializeRunIdentity } from "../run-identity.js";
import { beforeAll, afterAll, beforeEach, afterEach, describe, expect, it, vi } from "vitest";
import * as nativeSetupReadiness from "./setup-readiness.js";
import { secretService } from "../secrets.js";
import { authorizationService } from "../authorization.js";
import { startRunnerApiTestServer } from "../../__tests__/helpers/runner-api-server.js";
import { createRunnerdCodexTransport, defaultCapabilityRunnerdBinary } from "../../vendor/paperclip-runner/index.js";
import { runnerApiCatalog } from "./runner-api-catalog.js";
import { registerRunnerPrpAuthority } from "../../realtime/runner-prp-ws.js";
import { createLocalAgentJwt } from "../../agent-auth-jwt.js";
import { RUNNER_API_RESPONSE_MAX_BYTES, RUNNER_API_RESPONSE_RUN_MAX_BYTES } from "./runner-api-response-limits.js";
import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js";
import { finishPendingRunnerApiCaptureDeletes, RUNNER_API_CAPTURE_DELETE_FAILURE_METRIC } from "./runner-api-capture-retention.js";
import { prepareHeartbeatTaskContext } from "../heartbeat/task-context.js";
import { createHeartbeatRunPreparation } from "../heartbeat/run-preparation.js";
import { createAssignedMcpTools } from "./assigned-mcp-tools.js";
import type { ToolGatewayService } from "../tool-gateway.js";
import { RUNNER_TOOL_RESULT_MAX_BYTES } from "./runner-tool-results.js";
describe("runner API against real HTTP routes", () => {
let server: Awaited<ReturnType<typeof startRunnerApiTestServer>>;
const nativeAuthProbe = vi.fn<typeof nativeSetupReadiness.testNativeRunnerAuthentication>(async () => ({
adapterType: "paperclip_runner", status: "fail", testedAt: new Date().toISOString(),
checks: [{ code: "codex_hello_probe_auth_required", level: "error", message: "The API authority fixture has no provider credentials." }],
}));
let restoreNativeAuthProbe: (() => void) | undefined;
const oldSecret = process.env.PAPERCLIP_AGENT_JWT_SECRET;
const oldMasterKeyFile = process.env.PAPERCLIP_SECRETS_MASTER_KEY_FILE;
beforeEach(() => {
vi.stubEnv("PAPERCLIP_RUNNER_API_TOOLS_ENABLED", undefined);
vi.stubEnv("PAPERCLIP_RUNNER_API_TOOLS_COMPANY_IDS", undefined);
});
afterEach(() => vi.unstubAllEnvs());
beforeAll(async () => {
// API authorization and replay use the real lifecycle; external model login is unavailable here.
const probe = vi.spyOn(nativeSetupReadiness, "testNativeRunnerAuthentication").mockImplementation(nativeAuthProbe);
restoreNativeAuthProbe = () => probe.mockRestore();
process.env.PAPERCLIP_AGENT_JWT_SECRET = randomUUID();
server = await startRunnerApiTestServer();
process.env.PAPERCLIP_SECRETS_MASTER_KEY_FILE = join(server.root, "master.key");
}, 60_000);
afterAll(async () => {
try { await server?.close(); }
finally {
restoreNativeAuthProbe?.();
if (oldMasterKeyFile === undefined) delete process.env.PAPERCLIP_SECRETS_MASTER_KEY_FILE;
else process.env.PAPERCLIP_SECRETS_MASTER_KEY_FILE = oldMasterKeyFile;
if (oldSecret === undefined) delete process.env.PAPERCLIP_AGENT_JWT_SECRET;
else process.env.PAPERCLIP_AGENT_JWT_SECRET = oldSecret;
}
});
it.each(["http_response", "runner_response"])("counts committed effects independently after losing the %s", async boundary => {
const fixture = await server.fixture();
const name = `Recovery effect ${randomUUID()}`;
const call = { tool: "call_api", callId: "one-project-effect", arguments: {
operationId: "POST /api/companies/{companyId}/projects", body: { name },
} };
const effects = () => server.db.select({ id: projects.id, name: projects.name }).from(projects)
.where(and(eq(projects.companyId, fixture.companyId), eq(projects.name, name)));
const fetchActual = globalThis.fetch;
let dispatched = 0;
const network = vi.spyOn(globalThis, "fetch").mockImplementation(async (url, init) => {
const response = await fetchActual(url, init);
if (String(url) === `${fixture.apiUrl}/api/companies/${fixture.companyId}/projects` && init?.method === "POST") {
dispatched += 1;
expect(response.ok).toBe(true);
expect(await effects()).toHaveLength(1);
if (boundary === "http_response") throw new TypeError("fixture lost the committed HTTP response");
}
return response;
});
try {
const first = await fixture.authority.execute(call);
const ledgerBeforeRecovery = await effects();
expect(ledgerBeforeRecovery).toHaveLength(1);
const replacement = new PaperclipRunnerToolAuthority(server.db, fixture);
const recovered = await replacement.recover(call);
if (boundary === "http_response") {
expect(first).toMatchObject({ outcome: "unknown" });
expect(recovered).toBeNull();
expect(await replacement.execute(call)).toMatchObject({ outcome: "unknown" });
} else {
expect(first).toMatchObject({ ok: true, status: 201 });
expect(recovered).toEqual({ result: first });
expect(await replacement.execute(call)).toEqual(first);
}
expect(await effects()).toEqual(ledgerBeforeRecovery);
expect(dispatched).toBe(1);
await server.db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, fixture.runId));
await expect(replacement.recover(call)).rejects.toThrow("binding_not_authorized");
expect(await effects()).toEqual(ledgerBeforeRecovery);
expect(dispatched).toBe(1);
} finally { network.mockRestore(); }
});
it.each(["standard", "low_trust_review"])("delivers oversized assigned results without repeating the action (%s)", async trust => {
const fixture = await server.fixture({ apiToolsEnabled: false });
if (trust === "low_trust_review") {
await server.db.update(agents).set({ permissions: { trustPreset: trust, authorizationPolicy: {
trustBoundary: { mode: trust, companyId: fixture.companyId, issueIds: [fixture.issueId], allowedAgentIds: [fixture.agentId] },
} } }).where(eq(agents.id, fixture.agentId));
}
const original = { data: { content: "🧭é".repeat(120_000), last: "complete-result" } };
const executeTool = vi.fn().mockResolvedValue({ status: "completed", result: original });
const assignedMcpTools = await createAssignedMcpTools({
gatewayPublicId: "fixture", bearerToken: "fixture",
gateway: {
listToolsForNamedGateway: async () => [{ name: "app:write", displayName: "Write", description: "Fixture action",
parametersSchema: { type: "object" }, pluginId: "fixture", providerType: "mcp_remote_http", risk: "write" }],
executeTool,
} as unknown as ToolGatewayService,
});
const binding = { ...fixture.authority.binding, assignedMcpTools };
const authority = new PaperclipRunnerToolAuthority(server.db, binding);
const result: any = await authority.execute({ tool: assignedMcpTools.definitions()[0]!.name as string, callId: "large-action", arguments: {} });
expect(Buffer.byteLength(JSON.stringify(result))).toBeLessThan(32 * 1024);
expect(result).toMatchObject({ artifact: { artifactId: expect.any(String) }, readWith: { tool: "read_tool_result" } });
// Recreate the authority to prove that pages do not depend on a live cache.
const reopened = new PaperclipRunnerToolAuthority(server.db, binding);
expect(reopened.definitions().some(tool => tool.name === "read_tool_result")).toBe(true);
let offsetBytes: number | null = 0;
let text = "";
let pages = 0;
do {
const page: any = await reopened.execute({ tool: "read_tool_result", callId: randomUUID(),
arguments: { artifactId: result.artifact.artifactId, offsetBytes } });
expect(page).not.toHaveProperty("operationId");
expect(page.ok).toBe(true);
expect(Buffer.byteLength(JSON.stringify(page))).toBeLessThanOrEqual(RUNNER_TOOL_RESULT_MAX_BYTES);
if (pages === 0) expect(Buffer.byteLength(page.data)).toBeGreaterThan(490_000);
pages++;
text += page.data;
offsetBytes = page.responseText.nextOffsetBytes;
} while (offsetBytes !== null);
expect(JSON.parse(text)).toEqual(original);
expect(pages).toBe(2);
expect(executeTool).toHaveBeenCalledOnce();
expect((await fixture.snapshot()).assets.filter(asset => asset.id === result.artifact.artifactId)).toHaveLength(1);
const foreign = await server.fixture();
const foreignAuthority = new PaperclipRunnerToolAuthority(server.db, { ...foreign.authority.binding, assignedMcpTools });
await expect(foreignAuthority.execute({ tool: "read_tool_result", callId: "foreign",
arguments: { artifactId: result.artifact.artifactId } })).rejects.toThrow();
await expect(reopened.execute({ tool: "read_tool_result", callId: "unrelated",
arguments: { artifactId: fixture.artifactId } })).rejects.toThrow("not available to this task");
await expect(reopened.execute({ tool: "call_api", callId: "disabled",
arguments: { operationId: "GET /api/companies/{companyId}/projects" } })).rejects.toThrow("not_advertised");
await server.db.update(agents).set({ status: "paused" }).where(eq(agents.id, fixture.agentId));
await expect(reopened.execute({ tool: "read_tool_result", callId: "paused",
arguments: { artifactId: result.artifact.artifactId } })).rejects.toThrow("binding_not_authorized");
});
it.skipIf(!process.env.PAPERCLIP_REQUIRE_RUNNER_API_INTEGRATION && !existsSync(defaultCapabilityRunnerdBinary())).each(["current", "legacy_http"])("runs runnerd → PRP → authority → actual authenticated HTTP (%s receipt)", async (receiptFormat) => {
const fixture = await server.fixture();
const provider = join(server.root, `scripted-api-provider-${receiptFormat}.mjs`);
await writeFile(provider, `#!${process.execPath}
import { createInterface } from 'node:readline';
const send = value => process.stdout.write(JSON.stringify(value)+'\\n');
let step = 0;
const calls = [{tool:'search_api',arguments:{query:'list projects'}},{tool:'call_api',arguments:{operationId:'GET /api/companies/{companyId}/projects'}},{tool:'call_api',arguments:{operationId:'GET /api/projects/{id}',pathParams:{id:'${fixture.foreignProjectId}'}}}];
const next = () => { const c=calls[step++]; if(c) send({id:'call-'+step,method:'item/tool/call',params:{threadId:'api-thread',turnId:'api-turn',itemId:'api-item-'+step,callId:'api-call-'+step,...c}}); else send({method:'turn/completed',params:{turn:{id:'api-turn',status:'completed'}}}); };
for await (const line of createInterface({input:process.stdin})) {
const m=JSON.parse(line);
if(!m.method) {if(String(m.id).startsWith('call-')) {
if(step > 1) {
const envelope=JSON.parse(m.result.contentItems[0].text);
if(envelope.operationId!=='call_api'||envelope.callId!=='api-call-'+step) throw new Error('API receipt lost semantic call identity');
const receipt=envelope.result;
if(step===2 && (receipt.status!==200||receipt.data[0].name!=='Aurora')) throw new Error('Provider did not receive successful HTTP result');
if(step===3 && (receipt.status!==404||receipt.ok!==false)) throw new Error('Provider did not receive HTTP denial');
}
next();
} continue;}
if(m.method==='initialize') send({id:m.id,result:{userAgent:'scripted-api-provider'}});
else if(m.method==='thread/start') send({id:m.id,result:{thread:{id:'api-thread',sessionId:'api-session'}}});
else if(m.method==='turn/start') {send({id:m.id,result:{turn:{id:'api-turn',status:'inProgress'}}});send({method:'turn/started',params:{turn:{id:'api-turn'}}});next();}
else if(m.id!==undefined) send({id:m.id,result:{}});
}
`);
await chmod(provider, 0o700);
const bundle = createRunnerdCodexTransport({
runnerBinary: defaultCapabilityRunnerdBinary(), codexCommand: provider, codexArgs: [],
stateDirectory: join(server.root, `scripted-runner-${receiptFormat}`), lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null },
prpIdentity: { runnerInstanceId: "api-test", environmentLeaseId: "api-test-lease", runId: fixture.runId, normalizedSessionId: "api-test-session", turnId: "api-test-turn", itemId: "api-test-item" },
controlPlaneRegistration: prp => registerRunnerPrpAuthority({ companyId: fixture.companyId, runId: fixture.runId, authority: prp }),
});
const results: any[] = [];
bundle.transport.setServerRequestHandler(async request => {
const params = request.params as any;
const result = await fixture.authority.execute({ tool: params.tool, arguments: params.arguments, callId: params.callId });
results.push(result);
// Match production's dynamicToolResponse: an extra test-only envelope
// hides collisions between API response fields and PRP tool identity.
const { apiOperationId, ...receipt } = result as Record<string, unknown>;
const wireResult = receiptFormat === "legacy_http" && params.tool === "call_api"
? { ...receipt, operationId: apiOperationId }
: result;
return { success: true, contentItems: [{ type: "inputText", text: JSON.stringify(wireResult) }] };
});
try {
await bundle.transport.request("initialize", {});
await bundle.transport.request("thread/start", { cwd: fixture.workspace, dynamicTools: await fixture.authority.definitions() });
await bundle.transport.request("turn/start", { input: [{ type: "text", text: "Find the project" }] });
let timeout: ReturnType<typeof setTimeout> | undefined;
try {
await Promise.race([
(async () => { for await (const notification of bundle.transport.notifications()) if (notification.method === "turn/completed") return; })(),
new Promise<never>((_, reject) => { timeout = setTimeout(() => reject(new Error("Native provider did not continue after the API tool result")), 10_000); }),
]);
} finally { clearTimeout(timeout); }
expect(results).toHaveLength(3);
expect(results[1]).toMatchObject({ status: 200, apiOperationId: "GET /api/companies/{companyId}/projects", data: [{ id: fixture.projectId, name: "Aurora" }] });
expect(results[2]).toMatchObject({ ok: false, status: 404 });
expect(bundle.evidence().diagnostics).toContain("runnerd authenticated to the durable PRP control plane");
} finally { await bundle.transport.close(); }
}, 30_000);
it("cannot opt into API tools through a binding when the operator flag is false", async () => {
const fixture = await server.fixture({ apiToolsEnabled: true });
process.env.PAPERCLIP_RUNNER_API_TOOLS_ENABLED = "false";
try {
const names = (await fixture.authority.definitions()).map(tool => tool.name);
expect(names).toContain("get_task_context");
expect(names).not.toContain("search_api");
expect(names).not.toContain("call_api");
await expect(fixture.authority.execute({ tool: "call_api", callId: "disabled", arguments: { operationId: "GET /api/companies/{companyId}/projects" } })).rejects.toThrow("not_advertised");
} finally { delete process.env.PAPERCLIP_RUNNER_API_TOOLS_ENABLED; }
});
it("uses route permissions for lifecycle payloads instead of fallback-only denials", async () => {
const fixture = await server.fixture({ disableWakeOnDemand: true });
const result = await fixture.authority.execute({ tool: "call_api", callId: "block-via-rest", arguments: {
operationId: "PATCH /api/issues/{id}", pathParams: { id: fixture.blockerId },
body: { status: "blocked", billingCode: "REST-PARITY", blockedByIssueIds: [fixture.issueId] },
} });
expect(result).toMatchObject({ ok: true, status: 200, data: { status: "blocked", billingCode: "REST-PARITY" } });
expect((await fixture.snapshot()).issues.find(row => row.id === fixture.blockerId))
.toMatchObject({ status: "blocked", billingCode: "REST-PARITY" });
});
it.each(["company owner", "unrelated parent", "no fallback"])(
"keeps the authenticated native run responsible user over %s", async fallback => {
const fixture = await server.fixture({ connectionScenario: "fresh", disableWakeOnDemand: true });
const ownerUserId = randomUUID();
if (fallback !== "no fallback") {
await server.db.insert(authUsers).values({ id: ownerUserId, name: "Company owner", email: `${ownerUserId}@fixture.invalid`,
emailVerified: true, createdAt: new Date(), updatedAt: new Date() });
await server.db.insert(companyMemberships).values({ companyId: fixture.companyId, principalType: "user",
principalId: ownerUserId, status: "active", membershipRole: "owner" });
}
if (fallback === "unrelated parent") {
await server.db.update(issues).set({ responsibleUserId: ownerUserId }).where(eq(issues.id, fixture.blockerId));
}
const call = { tool: "call_api", callId: "bound-routine", arguments: {
operationId: "POST /api/companies/{companyId}/routines", body: {
title: "Native run owned routine", assigneeAgentId: fixture.agentId, status: "paused",
...(fallback === "unrelated parent" ? { parentIssueId: fixture.blockerId } : {}),
actorResponsibleUserId: ownerUserId, responsibleUserId: ownerUserId, createdByUserId: ownerUserId,
},
} };
const created = await fixture.authority.execute(call);
expect(created).toMatchObject({ ok: true, status: 201 });
expect(await new PaperclipRunnerToolAuthority(server.db, fixture).execute(call)).toEqual(created);
const [routine] = await server.db.select({ id: routines.id, responsibleUserId: routines.responsibleUserId,
createdByAgentId: routines.createdByAgentId, createdByUserId: routines.createdByUserId }).from(routines)
.where(eq(routines.companyId, fixture.companyId));
expect(routine).toEqual({ id: expect.any(String), responsibleUserId: fixture.responsibleUserId,
createdByAgentId: fixture.agentId, createdByUserId: null });
const revisions = await server.db.select({ responsibleUserId: routineRevisions.responsibleUserId,
createdByRunId: routineRevisions.createdByRunId }).from(routineRevisions)
.where(eq(routineRevisions.routineId, routine.id));
expect(revisions).toEqual([{ responsibleUserId: fixture.responsibleUserId, createdByRunId: fixture.runId }]);
await expect(fixture.authority.execute({ ...call, callId: "foreign-routine", arguments: {
...call.arguments, pathParams: { companyId: fixture.foreignCompanyId },
} })).rejects.toThrow("another company");
await server.db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, fixture.runId));
await expect(fixture.authority.execute({ ...call, callId: "cancelled-routine" })).rejects.toThrow("binding_not_authorized");
expect(await server.db.select({ id: routines.id }).from(routines).where(eq(routines.companyId, fixture.companyId)))
.toEqual([{ id: routine.id }]);
},
);
it("uses the current run identity when an agent JWT carries a stale user claim", async () => {
const fixture = await server.fixture({ connectionScenario: "fresh", disableWakeOnDemand: true });
const token = createLocalAgentJwt(fixture.agentId, fixture.companyId, "paperclip_runner", fixture.runId, "stale-user");
const response = await fetch(`${fixture.apiUrl}/api/companies/${fixture.companyId}/routines`, {
method: "POST", headers: { Authorization: `Bearer ${token}`, "Content-Type": "application/json" },
body: JSON.stringify({ title: "Current identity routine", assigneeAgentId: fixture.agentId, status: "paused" }),
});
expect(response.status).toBe(201);
expect(await response.json()).toMatchObject({ responsibleUserId: fixture.responsibleUserId,
createdByAgentId: fixture.agentId, createdByUserId: null });
});
it("rejects routine creation after the bound responsible user loses company membership", async () => {
const fixture = await server.fixture({ connectionScenario: "fresh", disableWakeOnDemand: true });
await server.db.delete(companyMemberships).where(and(eq(companyMemberships.companyId, fixture.companyId),
eq(companyMemberships.principalId, fixture.responsibleUserId!)));
await expect(fixture.authority.execute({ tool: "call_api", callId: "revoked-user-routine", arguments: {
operationId: "POST /api/companies/{companyId}/routines", body: {
title: "Revoked authority routine", assigneeAgentId: fixture.agentId, status: "paused",
},
} })).rejects.toMatchObject({ status: 403 });
expect(await server.db.select({ id: routines.id }).from(routines).where(eq(routines.companyId, fixture.companyId))).toEqual([]);
});
it.each([true, false])("keeps agent API-key responsibility binding (present=%s)", async bound => {
const fixture = await server.fixture({ connectionScenario: "fresh", disableWakeOnDemand: true });
const token = `pcp_fixture_${randomUUID()}`;
await server.db.insert(agentApiKeys).values({ companyId: fixture.companyId, agentId: fixture.agentId,
name: "Routine API fixture", keyHash: createHash("sha256").update(token).digest("hex"),
responsibleUserId: bound ? fixture.responsibleUserId : null });
const response = await fetch(`${fixture.apiUrl}/api/companies/${fixture.companyId}/routines`, {
method: "POST", headers: { Authorization: `Bearer ${token}`, "Content-Type": "application/json" },
body: JSON.stringify({ title: "API-key routine", assigneeAgentId: fixture.agentId, status: "paused" }),
});
expect(response.status).toBe(bound ? 201 : 403);
const rows = await server.db.select({ responsibleUserId: routines.responsibleUserId,
createdByAgentId: routines.createdByAgentId, createdByUserId: routines.createdByUserId }).from(routines)
.where(eq(routines.companyId, fixture.companyId));
expect(rows).toEqual(bound ? [{ responsibleUserId: fixture.responsibleUserId,
createdByAgentId: fixture.agentId, createdByUserId: null }] : []);
});
it("manages owned routines and triggers with durable replay, attribution and owner denials", async () => {
const fixture = await server.fixture({ disableWakeOnDemand: true, conversation: true });
await server.db.update(issues).set({ responsibleUserId: fixture.responsibleUserId }).where(eq(issues.id, fixture.issueId));
const args = { operationId: "POST /api/companies/{companyId}/routines", body: {
title: "REST parity routine", assigneeAgentId: fixture.agentId, projectId: fixture.projectId, parentIssueId: fixture.issueId, status: "paused",
} };
const call = (callId: string, arguments_: unknown) => fixture.authority.execute({ tool: "call_api", callId, arguments: arguments_ }) as Promise<any>;
const created = await call("routine-create", args);
expect(created).toMatchObject({ ok: true, status: 201, data: { title: "REST parity routine", assigneeAgentId: fixture.agentId } });
expect(await call("routine-create", args)).toEqual(created);
const routineId = created.data.id;
const trigger = await call("routine-trigger", { operationId: "POST /api/routines/{id}/triggers", pathParams: { id: routineId }, body: { kind: "schedule", cronExpression: "0 9 * * *", timezone: "UTC", enabled: false } });
expect(trigger).toMatchObject({ ok: true, status: 201, data: { trigger: { routineId, enabled: false } } });
expect(await call("routine-update", { operationId: "PATCH /api/routines/{id}", pathParams: { id: routineId }, body: { title: "Updated through fallback" } }))
.toMatchObject({ ok: true, status: 200, data: { title: "Updated through fallback" } });
const [other] = await server.db.insert(agents).values({ companyId: fixture.companyId, name: "Other routine owner", status: "active" }).returning();
expect(await call("routine-other-owner", { ...args, body: { ...args.body, assigneeAgentId: other.id } }))
.toMatchObject({ ok: false, status: 403 });
const snapshot = await fixture.snapshot();
expect(snapshot.activity).toEqual(expect.arrayContaining([
expect.objectContaining({ action: "routine.created", agentId: fixture.agentId, runId: fixture.runId }),
expect.objectContaining({ action: "runner.api_called", agentId: fixture.agentId, runId: fixture.runId, details: expect.objectContaining({ operationId: args.operationId, status: 201 }) }),
]));
expect(await server.db.select().from(routines).where(eq(routines.companyId, fixture.companyId))).toHaveLength(1);
});
it("starts a permitted skill test through the fallback using the existing assignment grants", async () => {
const fixture = await server.fixture({ conversation: true, disableWakeOnDemand: true });
await server.db.insert(companyMemberships).values({ companyId: fixture.companyId, principalType: "agent", principalId: fixture.agentId, status: "active", membershipRole: "member" });
await server.db.insert(principalPermissionGrants).values([
{ companyId: fixture.companyId, principalType: "agent", principalId: fixture.agentId, permissionKey: "tasks:assign" },
{ companyId: fixture.companyId, principalType: "user", principalId: fixture.responsibleUserId!, permissionKey: "tasks:assign" },
]);
const skillDir = join(server.root, `parity-skill-${randomUUID()}`);
await mkdir(skillDir);
await writeFile(join(skillDir, "SKILL.md"), "# Parity skill\nReturn the supplied fixture input.\n");
const [skill] = await server.db.insert(companySkills).values({
companyId: fixture.companyId, key: `company/${fixture.companyId}/parity`, slug: "parity", name: "Parity skill",
markdown: "# Parity skill\n", sourceType: "local_path", sourceLocator: skillDir,
trustLevel: "markdown_only", compatibility: "compatible", fileInventory: [{ path: "SKILL.md", kind: "skill" }], metadata: { sourceKind: "managed_local" },
}).returning();
const call = { tool: "call_api", callId: "skill-test-create", arguments: {
operationId: "POST /api/companies/{companyId}/skills/{skillId}/test-runs", pathParams: { skillId: skill.id },
body: { agentId: fixture.agentId, content: "Fixture input" },
} };
const result = await fixture.authority.execute(call) as any;
expect(result).toMatchObject({ ok: true, status: 201, data: { skillId: skill.id, agentId: fixture.agentId } });
expect(await fixture.authority.execute(call)).toEqual(result);
expect((await fixture.snapshot()).issues.some(issue => issue.id === result.data.issueId)).toBe(true);
});
it("keeps saved API responses within the issuing actor's scope", async () => {
const fixture = await server.fixture();
const body = "Private API response\n".repeat(2000);
const saved = await fixture.authority.execute({ tool: "call_api", callId: "private-snapshot", arguments: {
operationId: "PUT /api/issues/{id}/documents/{key}", pathParams: { id: fixture.issueId, key: "private-api-result" },
body: { title: "Private API result", body, format: "markdown", baseRevisionId: null },
} }) as any;
expect(saved).toMatchObject({ ok: true, status: 201, artifact: { artifactId: expect.any(String) } });
const url = `${fixture.apiUrl}/api/assets/${saved.artifact.artifactId}/content`;
const owner = createLocalAgentJwt(fixture.agentId, fixture.companyId, "paperclip_runner", fixture.runId);
expect((await fetch(url, { headers: { Authorization: `Bearer ${owner}` } })).status).toBe(200);
const [peer] = await server.db.insert(agents).values({ companyId: fixture.companyId, name: "Peer snapshot reader", status: "active", role: "ceo", adapterType: "paperclip_runner" }).returning();
const [peerRun] = await server.db.insert(heartbeatRuns).values({ companyId: fixture.companyId, agentId: peer.id, status: "running", runtimeMode: "native", nativeIssueId: fixture.blockerId, contextSnapshot: { issueId: fixture.blockerId } }).returning();
await server.db.update(issues).set({ assigneeAgentId: peer.id, status: "in_progress", executionRunId: peerRun.id,
description: `![private](/api/assets/${saved.artifact.artifactId}/content)`,
}).where(eq(issues.id, fixture.blockerId));
const token = createLocalAgentJwt(peer.id, fixture.companyId, "codex_local", peerRun.id);
expect((await fetch(url, { headers: { Authorization: `Bearer ${token}` } })).status).toBe(404);
const peerAuthority = new PaperclipRunnerToolAuthority(server.db, {
...fixture.authority.binding, agentId: peer.id, runId: peerRun.id, issueId: fixture.blockerId,
});
await expect(peerAuthority.execute({ tool: "call_api", callId: "peer-snapshot-upload", arguments: {
operationId: "POST /api/companies/{companyId}/issues/{issueId}/attachments", pathParams: { issueId: fixture.blockerId },
files: [{ artifactId: saved.artifact.artifactId }],
} })).rejects.toThrow("Artifact is not available to this actor");
const exported = await peerAuthority.execute({ tool: "call_api", callId: "peer-snapshot-export", arguments: {
operationId: "POST /api/companies/{companyId}/exports", body: { include: { issues: true }, issues: [fixture.blockerId] },
} }) as any;
expect(exported).toMatchObject({ ok: true, status: 200 });
for (const [path, file] of Object.entries(exported.data.files)) {
if (!path.startsWith("blobs/")) continue;
expect(Buffer.from((file as { data: string }).data, "base64").toString("utf8")).not.toContain("Private API response");
}
expect(await peerAuthority.execute({ tool: "call_api", callId: "peer-snapshot-logo", arguments: {
operationId: "PATCH /api/companies/{companyId}/branding", body: { logoAssetId: saved.artifact.artifactId },
} })).toMatchObject({ ok: false, status: 422 });
const ownUpload = await fixture.authority.execute({ tool: "call_api", callId: "own-snapshot-upload", arguments: {
operationId: "POST /api/companies/{companyId}/issues/{issueId}/attachments", pathParams: { issueId: fixture.issueId },
files: [{ artifactId: saved.artifact.artifactId }],
} });
expect(ownUpload).toMatchObject({ ok: true, status: 201 });
});
it.each(["skill_test", "low_trust_review"])("allows task image reads without company browsing (%s)", async scope => {
const fixture = await server.fixture();
if (scope === "skill_test") {
await server.db.update(heartbeatRuns).set({ contextSnapshot: { issueId: fixture.issueId,
paperclipAgentJwtScope: { kind: "skill_test", issueId: fixture.issueId },
} }).where(eq(heartbeatRuns.id, fixture.runId));
} else {
await server.db.update(agents).set({ permissions: { trustPreset: "low_trust_review", authorizationPolicy: {
trustBoundary: { mode: "low_trust_review", companyId: fixture.companyId, issueIds: [fixture.issueId], allowedAgentIds: [fixture.agentId] },
} } }).where(eq(agents.id, fixture.agentId));
}
const keyScope = scope === "skill_test" ? { kind: "skill_test" as const, issueId: fixture.issueId } : { kind: "standard" as const };
const actor = { type: "agent" as const, agentId: fixture.agentId, companyId: fixture.companyId, runId: fixture.runId, keyScope };
expect((await authorizationService(server.db).decide({ actor, action: "company_scope:read",
resource: { type: "company", companyId: fixture.companyId },
})).allowed).toBe(false);
const imageIds: string[] = [];
for (const issueId of [fixture.issueId, fixture.blockerId]) {
const stored = await server.storage.putFile({ companyId: fixture.companyId, namespace: `issues/${issueId}`,
originalFilename: "image.png", contentType: "image/png", body: Buffer.from("task-image-fixture"),
});
const [image] = await server.db.insert(assets).values({ companyId: fixture.companyId, ...stored, createdByAgentId: fixture.agentId }).returning();
imageIds.push(image.id);
}
const token = createLocalAgentJwt(fixture.agentId, fixture.companyId, "paperclip_runner", fixture.runId, null, keyScope);
expect((await fetch(`${fixture.apiUrl}/api/assets/${imageIds[0]}/content`, { headers: { Authorization: `Bearer ${token}` } })).status).toBe(200);
expect((await fetch(`${fixture.apiUrl}/api/assets/${imageIds[1]}/content`, { headers: { Authorization: `Bearer ${token}` } })).status).toBe(404);
expect(await fixture.authority.execute({ tool: "call_api", callId: "scoped-image", arguments: {
operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: imageIds[0] },
} })).toMatchObject({ ok: true, status: 206 });
await expect(fixture.authority.execute({ tool: "call_api", callId: "scoped-image-upload", arguments: {
operationId: "POST /api/companies/{companyId}/issues/{issueId}/attachments", pathParams: { issueId: fixture.issueId },
files: [{ artifactId: imageIds[1] }],
} })).rejects.toThrow("Artifact is not available to this actor");
});
it("keeps restored webhook secrets inline and encrypts their replay receipts", async () => {
const fixture = await server.fixture({ disableWakeOnDemand: true, conversation: true });
vi.stubEnv("PAPERCLIP_API_URL", fixture.apiUrl);
await server.db.update(issues).set({ responsibleUserId: fixture.responsibleUserId }).where(eq(issues.id, fixture.issueId));
const call = (callId: string, args: unknown) => fixture.authority.execute({ tool: "call_api", callId, arguments: args }) as Promise<any>;
const created = await call("webhook-routine", { operationId: "POST /api/companies/{companyId}/routines", body: {
title: "Restored webhook", assigneeAgentId: fixture.agentId, projectId: fixture.projectId, parentIssueId: fixture.issueId, status: "paused",
} });
expect(created).toMatchObject({ status: 201 });
const routineId = created.data.id;
await server.db.update(routines).set({ description: "Restored routine description\n".repeat(2000) }).where(eq(routines.id, routineId));
const trigger = await call("webhook-trigger", { operationId: "POST /api/routines/{id}/triggers", pathParams: { id: routineId },
body: { kind: "webhook", signingMode: "bearer", replayWindowSec: 300 },
});
expect(trigger).toMatchObject({ status: 201 });
expect(await call("delete-webhook", { operationId: "DELETE /api/routine-triggers/{id}", pathParams: { id: trigger.data.trigger.id } }))
.toMatchObject({ status: 204 });
const args = { operationId: "POST /api/routines/{id}/revisions/{revisionId}/restore",
pathParams: { id: routineId, revisionId: trigger.data.revision.id },
};
const restored = await call("restore-webhook", args);
expect(restored).toMatchObject({ ok: true, status: 200, data: { secretMaterials: [expect.objectContaining({ webhookSecret: expect.any(String) })] } });
expect(restored).not.toHaveProperty("artifact");
expect(await call("restore-webhook", args)).toEqual(restored);
const value = restored.data.secretMaterials[0].webhookSecret;
const [run] = await server.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
expect(JSON.stringify(run.resultJson)).not.toContain(value);
expect(Object.values((run.resultJson as any).apiToolReceipts).find((receipt: any) => receipt.operationId === args.operationId))
.toHaveProperty("encryptedResult");
expect((await fixture.snapshot()).assets.filter(asset => asset.objectKey.includes("/runner-api/"))).toEqual([]);
});
// s7504 run cb7994f9 (Codex, CHA-5): while a succeeded run's accounting held
// new company work, this live run's write_task_document was refused with
// "Company or budget authority unavailable", so it proposed a blocked result.
it("lets a running run write during a temporary accounting hold, but not after a hard stop", async () => {
const fixture = await server.fixture({ disableWakeOnDemand: true, conversation: true });
await server.db.insert(budgetPolicies).values({ companyId: fixture.companyId, scopeType: "company", scopeId: fixture.companyId,
metric: "billed_cents", windowKind: "calendar_month_utc", amount: 3000, notifyEnabled: false, unpricedUsagePolicy: "block" });
// Another run ended without succeeding and awaits accounting: new work waits.
await server.db.insert(heartbeatRuns).values({ companyId: fixture.companyId, agentId: fixture.agentId, invocationSource: "automation",
status: "failed", finishedAt: new Date(), costAccountingPending: true, usageJson: { accountingReceiptReady: false } });
const write = (key: string) => fixture.authority.execute({ tool: "write_task_document", callId: `hold-${key}`,
arguments: { taskId: fixture.issueId, key, title: "Brief", body: "Three bullets", baseRevisionId: null } });
await expect(write("brief")).resolves.toMatchObject({ ok: true });
await server.db.update(companies).set({ status: "paused", pauseReason: "budget", pausedAt: new Date() })
.where(eq(companies.id, fixture.companyId));
await expect(write("after-stop")).rejects.toThrow("Company or budget authority unavailable");
});
it.each(["comment_on_task", "write_task_document", "hire_agent"])("recovers completed %s receipts without repeating the write", async tool => {
const fixture = await server.fixture({ disableWakeOnDemand: true, conversation: true });
if (tool === "hire_agent") {
await server.db.insert(companyMemberships).values({ companyId: fixture.companyId, principalType: "agent", principalId: fixture.agentId, status: "active", membershipRole: "member" });
await server.db.insert(principalPermissionGrants).values([
{ companyId: fixture.companyId, principalType: "agent", principalId: fixture.agentId, permissionKey: "agents:create" },
{ companyId: fixture.companyId, principalType: "user", principalId: fixture.responsibleUserId!, permissionKey: "agents:create" },
]);
}
const args = tool === "hire_agent" ? { name: "Recovery teammate", instructions: "Inspect the assigned task." }
: tool === "comment_on_task" ? { taskId: fixture.issueId, body: " Persisted comment " }
: { taskId: fixture.issueId, key: "recovery", title: "Recovery document", body: "Persisted document", baseRevisionId: null };
const call = { tool, callId: "completed-bridge-write", arguments: args };
const result = await fixture.authority.execute(call);
expect(result).toMatchObject({ ok: true, status: expect.any(Number), apiOperationId: expect.any(String) });
if (tool === "hire_agent") {
await expect.poll(() => nativeAuthProbe.mock.calls
.filter(([context]) => context.companyId === fixture.companyId)
.map(([context, provider]) => ({ companyId: context.companyId, provider, configProvider: context.config.provider })),
{ timeout: 5_000 }).toEqual([{ companyId: fixture.companyId, provider: "codex", configProvider: "codex" }]);
}
const reopened = new PaperclipRunnerToolAuthority(server.db, fixture.authority.binding);
const effects = async () => {
const snapshot = await fixture.snapshot();
return { comments: snapshot.comments, revisions: snapshot.documentRevisions, approvals: snapshot.approvals,
agents: await server.db.select({ id: agents.id, name: agents.name }).from(agents).where(eq(agents.companyId, fixture.companyId)).orderBy(agents.id),
calls: snapshot.activity.filter(row => row.action === "runner.api_called") };
};
const before = await effects();
expect(before.calls).toHaveLength(1);
expect(await reopened.recover(call)).toEqual({ result });
expect(await reopened.recover({ ...call, arguments: { ...args, ...(tool === "hire_agent" ? { name: "Changed teammate" } : { body: "Changed body" }) } })).toBeNull();
expect(await reopened.recover({ ...call, callId: "missing-receipt" })).toBeNull();
const disabled = new PaperclipRunnerToolAuthority(server.db, { ...fixture.authority.binding, apiToolsEnabled: false });
expect(await disabled.recover(call)).toBeNull();
const [run] = await server.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
const pending = structuredClone(run.resultJson!);
for (const receipt of Object.values(pending.apiToolReceipts as Record<string, { state: string }>)) receipt.state = "pending";
await server.db.update(heartbeatRuns).set({ resultJson: pending }).where(eq(heartbeatRuns.id, fixture.runId));
expect(await reopened.recover(call)).toBeNull();
await server.db.update(heartbeatRuns).set({ resultJson: run.resultJson }).where(eq(heartbeatRuns.id, fixture.runId));
await server.db.update(agents).set({ status: "paused" }).where(eq(agents.id, fixture.agentId));
await expect(reopened.recover(call)).rejects.toThrow("paperclip_runner_tool_binding_not_authorized");
await server.db.update(agents).set({ status: "active" }).where(eq(agents.id, fixture.agentId));
expect(await effects()).toEqual(before);
});
it("recovers a body-less PATCH saved before the body rule while rejecting a new one before dispatch", async () => {
const fixture = await server.fixture();
const canonical = (value: unknown): string => Array.isArray(value) ? `[${value.map(canonical).join(",")}]`
: value && typeof value === "object" ? `{${Object.keys(value).sort().map(key => `${JSON.stringify(key)}:${canonical((value as Record<string, unknown>)[key])}`).join(",")}}`
: JSON.stringify(value) ?? "null";
const sha = (value: string) => createHash("sha256").update(value).digest("hex");
const bodyless = { operationId: "PATCH /api/projects/{id}", pathParams: { id: fixture.projectId } };
const call = { tool: "call_api", callId: "saved-bodyless-patch", arguments: bodyless };
// An older server accepted and completed this call; seed its receipt from a real completed PATCH.
const result = await fixture.authority.execute({ ...call, arguments: { ...bodyless, body: { name: "Saved before upgrade" } } });
expect(result).toMatchObject({ ok: true, status: 200 });
const [run] = await server.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
const receipts = structuredClone((run.resultJson as Record<string, any>).apiToolReceipts);
receipts[sha(call.callId)].digest = sha(canonical(bodyless));
await server.db.update(heartbeatRuns).set({ resultJson: { ...run.resultJson, apiToolReceipts: receipts } }).where(eq(heartbeatRuns.id, fixture.runId));
const reopened = new PaperclipRunnerToolAuthority(server.db, fixture.authority.binding);
const network = vi.spyOn(globalThis, "fetch");
try {
expect(await reopened.recover(call)).toEqual({ result });
expect(await reopened.execute(call)).toEqual(result);
const rejected = { ...call, callId: "new-bodyless-patch" };
await expect(reopened.execute(rejected)).rejects.toThrow(/not sent.*requires body/);
await expect(reopened.execute(rejected)).rejects.toThrow(/not sent.*requires body/);
expect(network).not.toHaveBeenCalled();
} finally { network.mockRestore(); }
// The rejected call reserved no receipt, so its callId stays usable for the corrected call.
const [after] = await server.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
expect(Object.keys((after.resultJson as Record<string, any>).apiToolReceipts)).not.toContain(sha("new-bodyless-patch"));
expect(await reopened.execute({ tool: "call_api", callId: "new-bodyless-patch", arguments: { ...bodyless, body: { name: "Corrected" } } }))
.toMatchObject({ ok: true, status: 200, data: { name: "Corrected" } });
});
it("returns and replays scoped secret values without plaintext receipts or response assets", async () => {
const fixture = await server.fixture();
const value = "disposable-private-credential-".repeat(1500);
const secrets = secretService(server.db);
const secret = await secrets.create(fixture.companyId, { name: "parity/private-value", key: "PARITY_TOKEN", provider: "local_encrypted", value });
await secrets.createBinding({ companyId: fixture.companyId, secretId: secret.id, targetType: "agent", targetId: fixture.agentId, configPath: "env.PARITY_TOKEN" });
const call = { tool: "call_api", callId: "secret-value", arguments: { operationId: "POST /api/agents/me/secrets/{key}/value", pathParams: { key: secret.key } } };
const result = await fixture.authority.execute(call);
expect(result).toMatchObject({ ok: true, status: 200, data: { key: secret.key, value } });
expect(await fixture.authority.execute(call)).toEqual(result);
expect(await fixture.authority.recover(call)).toEqual({ result });
const disabled = new PaperclipRunnerToolAuthority(server.db, { ...fixture.authority.binding, apiToolsEnabled: false });
expect(await disabled.recover(call)).toBeNull();
await server.db.update(agents).set({ status: "paused" }).where(eq(agents.id, fixture.agentId));
await expect(fixture.authority.recover(call)).rejects.toThrow("paperclip_runner_tool_binding_not_authorized");
await server.db.update(agents).set({ status: "active" }).where(eq(agents.id, fixture.agentId));
const snapshot = await fixture.snapshot();
expect(snapshot.assets.filter(asset => asset.objectKey.includes("/runner-api/"))).toEqual([]);
const [run] = await server.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
expect(JSON.stringify(run.resultJson)).not.toContain(value);
expect(Object.values((run.resultJson as any).apiToolReceipts)[0]).toHaveProperty("encryptedResult");
expect(snapshot.activity.filter(row => row.action === "runner.api_called")).toHaveLength(1);
const otherAgentId = randomUUID();
await server.db.insert(agents).values({ id: otherAgentId, companyId: fixture.companyId, name: "Unbound peer", status: "active" });
const [peerRun] = await server.db.insert(heartbeatRuns).values({ companyId: fixture.companyId, agentId: otherAgentId, status: "running", runtimeMode: "native", nativeIssueId: fixture.blockerId, contextSnapshot: { issueId: fixture.blockerId } }).returning();
await server.db.update(issues).set({ assigneeAgentId: otherAgentId, status: "in_progress", executionRunId: peerRun.id }).where(eq(issues.id, fixture.blockerId));
const token = createLocalAgentJwt(otherAgentId, fixture.companyId, "paperclip_runner", peerRun.id);
expect((await fetch(`${fixture.apiUrl}/api/agents/me/secrets/${secret.key}/value`, { method: "POST", headers: { Authorization: `Bearer ${token}`, "X-Paperclip-Run-Id": peerRun.id } })).status).toBe(403);
});
it.each([
["POST /api/companies/{companyId}/exports", 1],
["POST /api/companies/{companyId}/export", 2500],
])("keeps authorized company export credentials private and replayable for %s", async (operationId, repeats) => {
const fixture = await server.fixture();
await server.db.update(agents).set({ role: "ceo" }).where(eq(agents.id, fixture.agentId));
const credential = "disposable-export-credential-".repeat(repeats);
const [httpAgent] = await server.db.insert(agents).values({
companyId: fixture.companyId, name: "Exported HTTP fixture", adapterType: "http",
adapterConfig: { url: "https://fixture.invalid", headers: { Authorization: credential } },
}).returning();
const call = { tool: "call_api", callId: "private-export", arguments: {
operationId, body: { include: { agents: true }, agents: [httpAgent.id] },
} };
const result = await fixture.authority.execute(call) as any;
expect(result).toMatchObject({ ok: true, status: 200 });
expect(JSON.stringify(result.data)).toContain(credential);
expect(result).not.toHaveProperty("artifact");
expect(await fixture.authority.execute(call)).toEqual(result);
const snapshot = await fixture.snapshot();
expect(snapshot.assets.filter(asset => asset.objectKey.includes("/runner-api/"))).toEqual([]);
expect(snapshot.activity.filter(row => row.action === "runner.api_called")).toHaveLength(1);
const [run] = await server.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
expect(JSON.stringify(run.resultJson)).not.toContain(credential);
expect(Object.values((run.resultJson as any).apiToolReceipts)[0]).toHaveProperty("encryptedResult");
});
async function reprepareRun(fixture: { runId: string; issueId: string; agentId: string }) {
const [run] = await server.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
const [issue] = await server.db.select().from(issues).where(eq(issues.id, fixture.issueId));
const [agent] = await server.db.select().from(agents).where(eq(agents.id, fixture.agentId));
const context = structuredClone(run.contextSnapshot ?? {});
await prepareHeartbeatTaskContext(server.db, { run, agent,
task: { issueRef: issue, issueContext: null, context, taskSession: null },
policy: { recovering: true, trustPreset: { kind: "standard" }, experimentalInstanceSettings: {} },
services: { issuesSvc: { getAncestors: async () => [] },
getPinnedSkillTestContext: createHeartbeatRunPreparation(server.db).getPinnedSkillTestContext },
});
await server.db.update(heartbeatRuns).set({ contextSnapshot: context }).where(eq(heartbeatRuns.id, fixture.runId));
}
it("preserves the REST skill-test credential scope after a task mode edit and restart preparation", async () => {
const fixture = await server.fixture();
await server.db.update(heartbeatRuns).set({ contextSnapshot: { issueId: fixture.issueId,
paperclipAgentJwtScope: { kind: "skill_test", issueId: fixture.issueId },
} }).where(eq(heartbeatRuns.id, fixture.runId));
await server.db.update(issues).set({ workMode: "skill_test" }).where(eq(issues.id, fixture.issueId));
await server.db.update(issues).set({ workMode: "standard" }).where(eq(issues.id, fixture.issueId));
await expect(reprepareRun(fixture)).rejects.toThrow("The skill test's saved version is unavailable");
const [skill] = await server.db.insert(companySkills).values({ companyId: fixture.companyId,
key: `company/${fixture.companyId}/scope-test`, slug: "scope-test", name: "Scope test", markdown: "# Scope test", sourceType: "managed" }).returning();
const [version] = await server.db.insert(companySkillVersions).values({ companyId: fixture.companyId,
companySkillId: skill.id, revisionNumber: 1, fileInventory: [{ path: "SKILL.md", kind: "skill", content: "# Scope test" }] }).returning();
await server.db.insert(companySkillTestRuns).values({ companyId: fixture.companyId, skillId: skill.id,
skillVersionId: version.id, agentId: fixture.agentId, issueId: fixture.issueId, inputSnapshot: "Exercise the saved scope" });
await reprepareRun(fixture);
const [prepared] = await server.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
expect(prepared.contextSnapshot).toMatchObject({ paperclipAgentJwtScope: { kind: "skill_test", issueId: fixture.issueId },
paperclipSkillTest: { skillKey: skill.key, skillVersionId: version.id } });
const body = { kind: "secret", name: "parity/skill-test", value: "disposable-skill-test-value", justification: "Must remain denied" };
const token = createLocalAgentJwt(fixture.agentId, fixture.companyId, "paperclip_runner", fixture.runId,
fixture.responsibleUserId, { kind: "skill_test", issueId: fixture.issueId });
const direct = await fetch(`${fixture.apiUrl}/api/agents/me/secret-proposals`, {
method: "POST", headers: { Authorization: `Bearer ${token}`, "Content-Type": "application/json", "X-Paperclip-Run-Id": fixture.runId }, body: JSON.stringify(body),
});
expect(direct.status).toBe(403);
const result = await fixture.authority.execute({ tool: "call_api", callId: "skill-test-secret-denied", arguments: {
operationId: "POST /api/agents/me/secret-proposals", body,
} });
expect(result).toMatchObject({ ok: false, status: direct.status, data: await direct.json() });
});
it("retains standard REST credential scope after a task becomes a skill test and restart preparation", async () => {
const fixture = await server.fixture();
await server.db.update(heartbeatRuns).set({ contextSnapshot: { issueId: fixture.issueId,
paperclipAgentJwtScope: { kind: "standard" },
} }).where(eq(heartbeatRuns.id, fixture.runId));
await server.db.update(issues).set({ workMode: "skill_test" }).where(eq(issues.id, fixture.issueId));
await reprepareRun(fixture);
const token = createLocalAgentJwt(fixture.agentId, fixture.companyId, "paperclip_runner", fixture.runId, null, { kind: "standard" });
const direct = await fetch(`${fixture.apiUrl}/api/companies/${fixture.companyId}/projects`, { headers: { Authorization: `Bearer ${token}` } });
expect(direct.status).toBe(200);
expect(await fixture.authority.execute({ tool: "call_api", callId: "standard-scope", arguments: {
operationId: "GET /api/companies/{companyId}/projects",
} })).toMatchObject({ ok: true, status: 200, data: await direct.json() });
});
it("creates and reads secret proposals while preserving redaction and board review", async () => {
const fixture = await server.fixture();
const keyFile = process.env.PAPERCLIP_SECRETS_MASTER_KEY_FILE;
process.env.PAPERCLIP_SECRETS_MASTER_KEY_FILE = join(server.root, "proposal-master.key");
try {
const args = { operationId: "POST /api/agents/me/secret-proposals", body: {
kind: "secret", name: "parity/example", value: "disposable-parity-secret", justification: "Test existing agent REST permission",
} };
const call = (callId: string, arguments_: unknown) => fixture.authority.execute({ tool: "call_api", callId, arguments: arguments_ }) as Promise<any>;
const created = await call("proposal-create", args);
expect(created).toMatchObject({ ok: true, status: 201, data: { status: "pending", kind: "secret" } });
expect(await call("proposal-create", args)).toEqual(created);
const listed = await call("proposal-list", { operationId: "GET /api/agents/me/secret-proposals" });
expect(listed).toMatchObject({ ok: true, status: 200, data: { proposals: [expect.objectContaining({ id: created.data.id })] } });
expect(JSON.stringify([created, listed])).not.toContain(args.body.value);
expect(await call("proposal-approve-denied", { operationId: "POST /api/companies/{companyId}/secret-proposals/{id}/approve", pathParams: { id: created.data.id } }))
.toMatchObject({ ok: false, status: 403 });
const [run] = await server.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
expect(JSON.stringify(run.resultJson)).not.toContain(args.body.value);
} finally {
if (keyFile === undefined) delete process.env.PAPERCLIP_SECRETS_MASTER_KEY_FILE;
else process.env.PAPERCLIP_SECRETS_MASTER_KEY_FILE = keyFile;
}
});
it("supports routine annotation collaboration without changing scheduling", async () => {
const fixture = await server.fixture();
const [document] = await server.db.select().from(documents).where(eq(documents.companyId, fixture.companyId));
const [routine] = await server.db.insert(routines).values({ companyId: fixture.companyId, projectId: fixture.projectId, title: "Review schedule", description: document.latestBody, assigneeAgentId: fixture.agentId, status: "paused" }).returning();
await server.db.insert(routineDocuments).values({ companyId: fixture.companyId, routineId: routine.id, documentId: document.id, key: "description" });
const exact = "silver-wren", start = document.latestBody.indexOf(exact);
const base = "/api/routines/{id}/description/annotations";
const call = (callId: string, operationId: string, body: unknown, threadId?: string) => fixture.authority.execute({ tool: "call_api", callId, arguments: { operationId, pathParams: { id: routine.id, ...(threadId ? { threadId } : {}) }, body } }) as Promise<any>;
const created = await call("annotation-create", `POST ${base}`, { baseRevisionId: document.latestRevisionId, baseRevisionNumber: document.latestRevisionNumber, selector: { quote: { exact, prefix: document.latestBody.slice(0, start), suffix: document.latestBody.slice(start + exact.length) }, position: { normalizedStart: start, normalizedEnd: start + exact.length, markdownStart: start, markdownEnd: start + exact.length } }, body: "Please clarify this note" });
expect(created).toMatchObject({ status: 201, data: { routineId: routine.id, status: "open" } });
const threadId = created.data.id;
expect(await call("annotation-comment", `POST ${base}/{threadId}/comments`, { body: "Clarified note" }, threadId)).toMatchObject({ status: 201, data: { body: "Clarified note" } });
expect(await call("annotation-resolve", `PATCH ${base}/{threadId}`, { status: "resolved" }, threadId)).toMatchObject({ status: 200, data: { status: "resolved" } });
expect(await call("annotation-reopen", `PATCH ${base}/{threadId}`, { status: "open" }, threadId)).toMatchObject({ status: 200, data: { status: "open" } });
const [unchanged] = await server.db.select().from(routines).where(eq(routines.id, routine.id));
expect(unchanged).toMatchObject({ status: "paused", lastTriggeredAt: null, lastEnqueuedAt: null });
expect((await fixture.snapshot()).activity).toEqual(expect.arrayContaining([expect.objectContaining({ action: "routine.document_annotation_thread_created", agentId: fixture.agentId, runId: fixture.runId })]));
});
it("matches direct agent REST authorization for formerly restricted routes", async () => {
const fixture = await server.fixture({ disableWakeOnDemand: true });
const token = createLocalAgentJwt(fixture.agentId, fixture.companyId, "paperclip_runner", fixture.runId);
for (const [operationId, path, body] of [
["POST /api/agents/{id}/pause", `/api/agents/${fixture.agentId}/pause`, {}],
["POST /api/agents/{id}/resume", `/api/agents/${fixture.agentId}/resume`, {}],
["POST /api/companies/{companyId}/exports", `/api/companies/${fixture.companyId}/exports`, {}],
["POST /api/companies/{companyId}/invites", `/api/companies/${fixture.companyId}/invites`, {}],
["POST /api/approvals/{id}/approve", `/api/approvals/${fixture.approvalId}/approve`, {}],
["POST /api/issues/{id}/recovery-actions/resolve", `/api/issues/${fixture.blockerId}/recovery-actions/resolve`, {}],
["POST /api/execution-workspaces/{id}/runtime-services/{action}", `/api/execution-workspaces/${randomUUID()}/runtime-services/stop`, {}],
["POST /api/companies/{companyId}/skills/{skillId}/test-runs", `/api/companies/${fixture.companyId}/skills/${randomUUID()}/test-runs`, { agentId: fixture.agentId, content: "Test input" }],
] as const) {
const direct = await fetch(`${fixture.apiUrl}${path}`, { method: "POST", headers: { Authorization: `Bearer ${token}`, "X-Paperclip-Run-Id": fixture.runId, "Content-Type": "application/json" }, body: JSON.stringify(body) });
const names = [...operationId.matchAll(/\{([^}]+)\}/g)].map(([, name]) => name);
const template = operationId.slice(5).split("/");
const actual = path.split("/");
const values = template.flatMap((segment, index) => segment.startsWith("{") ? [actual[index]] : []);
const result = await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId, pathParams: Object.fromEntries(names.map((name, index) => [name, values[index]])), body,
} }) as any;
expect(result.status, operationId).toBe(direct.status);
expect(result.ok, operationId).toBe(direct.ok);
if (!direct.ok) expect(result.data, operationId).toEqual(await direct.json());
}
});
it("preserves route validation, authorization and audit; replays mutations once", async () => {
const fixture = await server.fixture();
const call = (callId: string, args: unknown) => fixture.authority.execute({ tool: "call_api", callId, arguments: args });
await expect(call("foreign", { operationId: "GET /api/projects/{id}", pathParams: { id: fixture.foreignProjectId } })).resolves.toMatchObject({ ok: false, status: 404 });
await expect(call("invalid", { operationId: "POST /api/companies/{companyId}/projects", body: {} })).resolves.toMatchObject({ ok: false, status: 400 });
const args = { operationId: "POST /api/companies/{companyId}/projects", body: { name: "Created through HTTP" } };
const result = await call("create", args);
expect(result).toMatchObject({ status: 201, data: { name: "Created through HTTP" } });
expect(await call("create", args)).toEqual(result);
await expect(call("create", { ...args, body: { name: "Different" } })).rejects.toThrow("reused");
const snapshot = await fixture.snapshot();
expect(snapshot.projects.filter(p => p.name === "Created through HTTP")).toHaveLength(1);
expect(snapshot.activity).toEqual(expect.arrayContaining([
expect.objectContaining({ action: "project.created", agentId: fixture.agentId }),
expect.objectContaining({ action: "runner.api_called", agentId: fixture.agentId, runId: fixture.runId, details: expect.objectContaining({ operationId: args.operationId, status: 201 }) }),
]));
});
it("blocks stale bindings and Ask/Plan mutations before HTTP", async () => {
for (const mode of ["ask", "planning"] as const) {
const fixture = await server.fixture({ mode });
await expect(fixture.authority.execute({ tool: "call_api", callId: "read", arguments: { operationId: "GET /api/companies/{companyId}/projects" } })).resolves.toMatchObject({ status: 200 });
await expect(fixture.authority.execute({ tool: "call_api", callId: "write", arguments: { operationId: "POST /api/companies/{companyId}/projects", body: { name: "Denied" } } })).rejects.toThrow("only reads");
await server.db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, fixture.runId));
await expect(fixture.authority.execute({ tool: "search_api", callId: "stale", arguments: { query: "projects" } })).rejects.toThrow("binding_not_authorized");
expect((await fixture.snapshot()).projects.some(p => p.name === "Denied")).toBe(false);
}
});
it("revokes advertised API tools without disabling dedicated operations", async () => {
const fixture = await server.fixture();
expect(fixture.authority.definitions().some(tool => tool.name === "call_api")).toBe(true);
vi.stubEnv("PAPERCLIP_RUNNER_API_TOOLS_ENABLED", "false");
try {
expect(fixture.authority.definitions().some(tool => tool.name === "search_api")).toBe(false);
await expect(fixture.authority.execute({ tool: "call_api", callId: "revoked", arguments: {
operationId: "POST /api/companies/{companyId}/projects", body: { name: "Must not exist" },
} })).rejects.toThrow("not_advertised");
await expect(fixture.authority.execute({ tool: "get_task_context", callId: "dedicated-after-stop", arguments: {} }))
.resolves.toMatchObject({ activeTask: { id: fixture.issueId } });
expect((await fixture.snapshot()).projects.some(project => project.name === "Must not exist")).toBe(false);
} finally { vi.unstubAllEnvs(); }
});
it("retains an uncertain mutation receipt without dispatching it again", async () => {
const fixture = await server.fixture();
const args = { operationId: "POST /api/companies/{companyId}/projects", body: { name: "Uncertain" } };
let rejectNetwork!: (error: Error) => void;
const fetcher = vi.spyOn(globalThis, "fetch").mockImplementation(() => new Promise((_resolve, reject) => { rejectNetwork = reject; }));
try {
const call = () => fixture.authority.execute({ tool: "call_api", callId: "uncertain", arguments: args });
const pending = call();
await vi.waitFor(() => expect(fetcher).toHaveBeenCalledTimes(1));
expect(await call()).toMatchObject({ status: null, outcome: "unknown", error: "api_outcome_unknown" });
rejectNetwork(new Error("lost response"));
const result = await pending;
expect(result).toMatchObject({ status: null, outcome: "unknown" });
expect(await call()).toEqual(result);
expect(fetcher).toHaveBeenCalledTimes(1);
} finally { fetcher.mockRestore(); }
expect((await fixture.snapshot()).projects.some(p => p.name === "Uncertain")).toBe(false);
});
it.each([403, 422])("a deliberate retry can execute an exact mutation previously rejected with %s", async status => {
const fixture = await server.fixture();
const args = { operationId: "POST /api/companies/{companyId}/projects", body: { name: "Authorized on retry" } };
const rejected = vi.spyOn(globalThis, "fetch").mockResolvedValueOnce(Response.json({ error: "Rejected before mutation" }, { status }));
try {
expect(await fixture.authority.execute({ tool: "call_api", callId: "rejected", arguments: args }))
.toMatchObject({ ok: false, status });
} finally { rejected.mockRestore(); }
await server.db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date() })
.where(eq(heartbeatRuns.id, fixture.runId));
const id = randomUUID(), intentId = randomUUID();
await server.db.insert(heartbeatRuns).values({ id, companyId: fixture.companyId, agentId: fixture.agentId,
status: "running", runtimeMode: "native", nativeIssueId: fixture.issueId, responsibleUserId: fixture.responsibleUserId,
contextSnapshot: { issueId: fixture.issueId, previousRunId: fixture.runId } });
await server.db.update(issues).set({ executionRunId: id }).where(eq(issues.id, fixture.issueId));
await server.db.insert(agentWakeupRequests).values({ id: intentId, companyId: fixture.companyId,
agentId: fixture.agentId, runId: id, source: "on_demand", reason: "task_retry", status: "coalesced",
requestedByActorType: "user", requestedByActorId: fixture.responsibleUserId ?? "board",
payload: { issueId: fixture.issueId, taskRetry: { schema: "paperclip.task-retry.v1", generationId: intentId,
sourceRunId: fixture.runId, actorId: fixture.responsibleUserId ?? "board", strategy: "fresh_isolated_workspace" } } });
if (fixture.responsibleUserId) await initializeRunIdentity(server.db, { companyId: fixture.companyId, runId: id,
issueId: fixture.issueId, responsibleUserId: fixture.responsibleUserId, cause: "instruction" });
const authority = new PaperclipRunnerToolAuthority(server.db, { ...fixture, runId: id });
const network = vi.spyOn(globalThis, "fetch");
try {
const result = await authority.execute({ tool: "call_api", callId: "accepted-retry", arguments: args });
expect(result).toMatchObject({ ok: true, status: 201 });
expect(await authority.execute({ tool: "call_api", callId: "accepted-retry", arguments: args })).toEqual(result);
expect(network).toHaveBeenCalledTimes(1);
} finally { network.mockRestore(); }
expect((await fixture.snapshot()).projects.filter(project => project.name === "Authorized on retry")).toHaveLength(1);
});
it.each(["task_retry", "automatic_successor"])("%s reuses the completed effect and fences a different unresolved effect across run identities", async mode => {
const fixture = await server.fixture();
const args = { operationId: "POST /api/companies/{companyId}/projects", body: { name: "Already created" } };
const completed = await fixture.authority.execute({ tool: "call_api", callId: "first-publication", arguments: args });
expect(completed).toMatchObject({ status: 201 });
const [original] = await server.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
await server.db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(),
resultJson: { ...original.resultJson, apiToolReceipts: { ...(original.resultJson?.apiToolReceipts as object),
unresolved: { operationId: "POST /api/companies/{companyId}/routines", state: "pending", digest: "unknown" } } },
}).where(eq(heartbeatRuns.id, fixture.runId));
let id = randomUUID();
const intentId = randomUUID();
await server.db.insert(heartbeatRuns).values({ id, companyId: fixture.companyId, agentId: fixture.agentId,
status: "running", runtimeMode: "native", nativeIssueId: fixture.issueId, responsibleUserId: fixture.responsibleUserId,
contextSnapshot: { issueId: fixture.issueId, previousRunId: fixture.runId } });
await server.db.update(issues).set({ executionRunId: id }).where(eq(issues.id, fixture.issueId));
await server.db.insert(agentWakeupRequests).values({ id: intentId, companyId: fixture.companyId,
agentId: fixture.agentId, runId: id, source: "on_demand", reason: "task_retry", status: "coalesced",
requestedByActorType: "user", requestedByActorId: fixture.responsibleUserId ?? "board",
payload: { issueId: fixture.issueId, taskRetry: { schema: "paperclip.task-retry.v1", generationId: intentId,
sourceRunId: fixture.runId, actorId: fixture.responsibleUserId ?? "board", strategy: "fresh_isolated_workspace" } } });
if (mode === "automatic_successor") {
const parentId = id;
id = randomUUID();
await server.db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date() })
.where(eq(heartbeatRuns.id, parentId));
await server.db.insert(heartbeatRuns).values({ id, companyId: fixture.companyId, agentId: fixture.agentId,
status: "running", runtimeMode: "native", nativeIssueId: fixture.issueId, retryOfRunId: parentId,
responsibleUserId: fixture.responsibleUserId, invocationSource: "automation", contextSnapshot: { issueId: fixture.issueId } });
await server.db.update(issues).set({ executionRunId: id }).where(eq(issues.id, fixture.issueId));
}
if (fixture.responsibleUserId) await initializeRunIdentity(server.db, { companyId: fixture.companyId, runId: id,
issueId: fixture.issueId, responsibleUserId: fixture.responsibleUserId, cause: "instruction" });
const authority = new PaperclipRunnerToolAuthority(server.db, { ...fixture, runId: id });
const network = vi.spyOn(globalThis, "fetch");
try {
expect(await authority.execute({ tool: "call_api", callId: "replacement-publication", arguments: args })).toEqual(completed);
expect(await authority.execute({ tool: "call_api", callId: "uncertain-other-operation", arguments: {
operationId: "POST /api/companies/{companyId}/routines", body: { title: "Uncertain", assigneeAgentId: fixture.agentId },
} })).toMatchObject({ outcome: "unknown", error: "api_outcome_unknown" });
expect(network).not.toHaveBeenCalled();
expect(await authority.execute({ tool: "call_api", callId: "independent-read", arguments: {
operationId: "GET /api/companies/{companyId}/projects",
} })).toMatchObject({ status: 200 });
expect(network).toHaveBeenCalledTimes(1);
} finally { network.mockRestore(); }
expect((await fixture.snapshot()).projects.filter(project => project.name === "Already created")).toHaveLength(1);
});
it("revalidates the active run after reading an upload", async () => {
const fixture = await server.fixture();
const original = server.storage.getObject.bind(server.storage);
const getObject = vi.spyOn(server.storage, "getObject").mockImplementation(async (...args) => {
const object = await original(...args);
await server.db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, fixture.runId));
return object;
});
try {
await expect(fixture.authority.execute({ tool: "call_api", callId: "expired-during-upload", arguments: {
operationId: "POST /api/companies/{companyId}/issues/{issueId}/attachments", pathParams: { issueId: fixture.issueId }, files: [{ artifactId: fixture.artifactId }],
} })).rejects.toThrow("binding_not_authorized");
expect((await fixture.snapshot()).assets).toHaveLength(2);
} finally { getObject.mockRestore(); }
});
it("retrieves every byte of a saved large response without creating an artifact loop", async () => {
const fixture = await server.fixture();
const foreign = await server.fixture();
const text = JSON.stringify({ padding: "🧭é".repeat(6000), evidence: "last-field-is-readable" });
const stored = await server.storage.putFile({ companyId: fixture.companyId, namespace: "eval", originalFilename: "evidence.json", contentType: "application/json", body: Buffer.from(text) });
const [asset] = await server.db.insert(assets).values({ ...stored, companyId: fixture.companyId, createdByAgentId: fixture.agentId }).returning();
const call = (authority: typeof fixture.authority, assetId: string, responseText?: object) => authority.execute({ tool: "call_api", callId: randomUUID(), arguments: { operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId }, ...(responseText ? { responseText } : {}) } }) as Promise<any>;
const initial = await call(fixture.authority, asset.id);
expect(initial).toMatchObject({ status: 206, artifact: { artifactId: asset.id, sha256: asset.sha256, byteSize: Buffer.byteLength(text) } });
const before = (await fixture.snapshot()).assets.length;
const reads = vi.spyOn(server.storage, "getObject");
let offsetBytes = 0;
let result = "";
try {
do {
const page = await call(fixture.authority, initial.artifact.artifactId, { offsetBytes, limitBytes: 4096 });
expect(page).toMatchObject({ status: 206, responseText: { offsetBytes, totalBytes: Buffer.byteLength(text) } });
result += page.data;
offsetBytes = page.responseText.nextOffsetBytes;
} while (offsetBytes !== null);
expect(JSON.parse(result)).toEqual(JSON.parse(text));
expect(reads.mock.calls.length).toBeGreaterThan(1);
expect(reads.mock.calls.every(([, , options]) => options?.range)).toBe(true);
const storageBytes = reads.mock.calls.reduce((total, [, , options]) => total + options!.range!.end - options!.range!.start + 1, 0);
expect(storageBytes).toBeLessThan(Buffer.byteLength(text) + reads.mock.calls.length * 5);
expect((await fixture.snapshot()).assets).toHaveLength(before);
expect(await call(foreign.authority, initial.artifact.artifactId, { offsetBytes: 0 })).toMatchObject({ ok: false, status: 404 });
} finally {
reads.mockRestore();
}
});
it("captures a live response above ten MiB and reads its saved snapshot to EOF", async () => {
const fixture = await server.fixture();
const foreign = await server.fixture();
const description = "A".repeat(12 * 1024 * 1024) + "END-OF-LARGE-RESPONSE";
await server.db.update(projects).set({ description }).where(eq(projects.id, fixture.projectId));
const initial: any = await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId }, responseText: { limitBytes: 8192 },
} });
expect(initial).toMatchObject({ ok: true, status: 200, responseText: { offsetBytes: 0, nextOffsetBytes: 8192 } });
expect(initial.artifact.byteSize).toBeGreaterThan(12 * 1024 * 1024);
const saved = await server.db.select().from(assets).where(eq(assets.id, initial.artifact.artifactId)).then(rows => rows[0]);
const object = await server.storage.getObject(fixture.companyId, saved.objectKey);
const chunks: Buffer[] = [];
for await (const chunk of object.stream) chunks.push(chunk);
const expected = Buffer.concat(chunks);
expect(JSON.parse(expected.toString()).description).toBe(description);
const before = (await fixture.snapshot()).assets.length;
for (const offsetBytes of [0, 10 * 1024 * 1024 + 1, expected.length - 8192, expected.length]) {
const page: any = await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: saved.id }, responseText: { offsetBytes, limitBytes: 8192 },
} });
expect(page).toMatchObject({ ok: true, status: 206, data: expected.subarray(offsetBytes, offsetBytes + 8192).toString(),
responseText: { offsetBytes, totalBytes: expected.length, nextOffsetBytes: offsetBytes + 8192 < expected.length ? offsetBytes + 8192 : null } });
expect(page.artifact).toBeUndefined();
}
expect((await fixture.snapshot()).assets).toHaveLength(before);
expect(await foreign.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: saved.id }, responseText: { offsetBytes: 10 * 1024 * 1024 + 1 },
} })).toMatchObject({ ok: false, status: 404 });
const mutation = { tool: "call_api", callId: randomUUID(), arguments: {
operationId: "PATCH /api/projects/{id}", pathParams: { id: fixture.projectId }, body: { name: "Updated once" },
} };
const receipt: any = await fixture.authority.execute(mutation);
expect(receipt).toMatchObject({ ok: true, status: 200, artifact: { contentType: "application/json; charset=utf-8" } });
expect(receipt.artifact.byteSize).toBeGreaterThan(12 * 1024 * 1024);
expect(await fixture.authority.execute(mutation)).toEqual(receipt);
const after = await fixture.snapshot();
expect(after.assets).toHaveLength(before + 1);
expect(after.activity.filter(row => row.action === "project.updated")).toHaveLength(1);
const [run] = await server.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
expect((run.resultJson as Record<string, unknown>).apiResponseCaptureBytes).toBe(initial.artifact.byteSize + receipt.artifact.byteSize);
}, 30_000);
it("enforces the durable run budget while permitting small reads and saved pages", async () => {
const fixture = await server.fixture();
await server.db.update(heartbeatRuns).set({ resultJson: { apiResponseCaptureBytes: RUNNER_API_RESPONSE_RUN_MAX_BYTES } }).where(eq(heartbeatRuns.id, fixture.runId));
const call = () => fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
} });
expect(await call()).toMatchObject({ ok: true });
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
const before = (await fixture.snapshot()).assets.length;
expect(await call()).toMatchObject({ ok: false, error: "api_response_capture_limit", outcome: "read_failed" });
expect((await fixture.snapshot()).assets).toHaveLength(before);
expect(await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: fixture.artifactId }, responseText: { limitBytes: 4 },
} })).toMatchObject({ ok: true });
});
it("reserves the run budget atomically across simultaneous captures", async () => {
const fixture = await server.fixture();
const used = RUNNER_API_RESPONSE_RUN_MAX_BYTES - RUNNER_API_RESPONSE_MAX_BYTES;
await server.db.update(heartbeatRuns).set({ resultJson: { apiResponseCaptureBytes: used } }).where(eq(heartbeatRuns.id, fixture.runId));
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
let unblock!: () => void;
let started!: () => void;
const gate = new Promise<void>(resolve => { unblock = resolve; });
const saving = new Promise<void>(resolve => { started = resolve; });
const putFile = server.storage.putFile.bind(server.storage);
const spy = vi.spyOn(server.storage, "putFile").mockImplementation(async input => {
if (input.namespace === "runner-api") { started(); await gate; }
return putFile(input);
});
const call = () => fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
} });
const first = call();
try {
await saving;
expect(await call()).toMatchObject({ ok: false, error: "api_response_capture_limit" });
} finally { unblock(); spy.mockRestore(); }
const result: any = await first;
expect(result).toMatchObject({ ok: true });
const [run] = await server.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
expect((run.resultJson as Record<string, unknown>).apiResponseCaptureBytes).toBe(used + result.artifact.byteSize);
expect(await call()).toMatchObject({ ok: false, error: "api_response_capture_limit" });
});
it("evicts the oldest finished snapshots instead of blocking a read when the company quota is full", async () => {
const fixture = await server.fixture();
vi.stubEnv("PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES", String(2 * RUNNER_API_RESPONSE_MAX_BYTES));
const stored = await server.storage.putFile({ companyId: fixture.companyId, namespace: "runner-api", originalFilename: "response.json",
contentType: "application/json", body: Buffer.from("{}") });
const [oldest] = await server.db.insert(assets).values({ ...stored, companyId: fixture.companyId, byteSize: RUNNER_API_RESPONSE_MAX_BYTES,
createdByAgentId: fixture.agentId, createdAt: new Date(Date.now() - 2 * 86_400_000) }).returning();
const finishedRunId = randomUUID();
await server.db.insert(heartbeatRuns).values({ id: finishedRunId, companyId: fixture.companyId, agentId: fixture.agentId,
status: "succeeded", runtimeMode: "native", nativeIssueId: fixture.blockerId, contextSnapshot: { issueId: fixture.blockerId } });
const [newer] = await server.db.insert(assets).values({ companyId: fixture.companyId, provider: "local_disk",
objectKey: `${fixture.companyId}/runner-api/newer-snapshot`, contentType: "text/plain", byteSize: RUNNER_API_RESPONSE_MAX_BYTES / 2,
sha256: "newer-snapshot", createdByAgentId: fixture.agentId, createdAt: new Date(Date.now() - 86_400_000) }).returning();
await server.db.insert(runnerApiResponseReservations).values({ companyId: fixture.companyId, runId: finishedRunId, assetId: newer.id, reservedBytes: newer.byteSize });
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
const authority = new PaperclipRunnerToolAuthority(server.db, fixture);
const result: any = await authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
} });
expect(result).toMatchObject({ ok: true });
const remaining = await server.db.select({ id: assets.id }).from(assets).where(eq(assets.companyId, fixture.companyId));
expect(remaining.map(row => row.id)).not.toContain(oldest.id);
expect(remaining.map(row => row.id)).toContain(newer.id);
expect(remaining.map(row => row.id)).toContain(result.artifact.artifactId);
expect((await server.storage.headObject(fixture.companyId, oldest.objectKey)).exists).toBe(false);
});
it("keeps an evicted snapshot counted until its stored object is deleted, then retries the delete", async () => {
const fixture = await server.fixture();
vi.stubEnv("PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES", String(2 * RUNNER_API_RESPONSE_MAX_BYTES));
const stored = await server.storage.putFile({ companyId: fixture.companyId, namespace: "runner-api", originalFilename: "response.json",
contentType: "application/json", body: Buffer.from("{}") });
const [oldest] = await server.db.insert(assets).values({ ...stored, companyId: fixture.companyId, byteSize: RUNNER_API_RESPONSE_MAX_BYTES * 1.5,
createdByAgentId: fixture.agentId, createdAt: new Date(Date.now() - 86_400_000) }).returning();
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
const call = () => fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
} });
const spy = vi.spyOn(server.storage, "deleteObject").mockRejectedValue(new Error("storage unavailable"));
try {
expect(await call()).toMatchObject({ ok: false, error: "api_response_company_storage_limit" });
} finally { spy.mockRestore(); }
expect(await server.db.select().from(runnerApiResponseReservations).where(eq(runnerApiResponseReservations.companyId, fixture.companyId))).toEqual([
expect.objectContaining({ assetId: null, pendingDeleteObjectKey: oldest.objectKey, reservedBytes: oldest.byteSize }),
]);
expect((await server.storage.headObject(fixture.companyId, oldest.objectKey)).exists).toBe(true);
expect(await call()).toMatchObject({ ok: true });
expect((await server.storage.headObject(fixture.companyId, oldest.objectKey)).exists).toBe(false);
const reservations = await server.db.select().from(runnerApiResponseReservations).where(eq(runnerApiResponseReservations.companyId, fixture.companyId));
expect(reservations.every(row => row.pendingDeleteObjectKey === null)).toBe(true);
});
it("reclaims enough space when eviction spans more than one page of small snapshots", async () => {
const fixture = await server.fixture();
vi.stubEnv("PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES", String(2 * RUNNER_API_RESPONSE_MAX_BYTES));
const size = 4 * 1024 * 1024;
const start = Date.now() - 86_400_000;
const small = await server.db.insert(assets).values(Array.from({ length: 500 }, (_, index) => ({ companyId: fixture.companyId, provider: "local_disk",
objectKey: `${fixture.companyId}/runner-api/small-${index}`, contentType: "text/plain", byteSize: size,
sha256: `small-${index}`, createdByAgentId: fixture.agentId, createdAt: new Date(start + index * 1000) }))).returning({ id: assets.id });
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
expect(await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
} })).toMatchObject({ ok: true });
const remaining = new Set((await server.db.select({ id: assets.id }).from(assets).where(eq(assets.companyId, fixture.companyId))).map(row => row.id));
const evicted = small.filter(row => !remaining.has(row.id)).length;
expect(evicted).toBeGreaterThan(200);
expect(evicted).toBeLessThan(500);
expect(small.slice(0, evicted).every(row => !remaining.has(row.id))).toBe(true);
expect(await server.db.select().from(runnerApiResponseReservations).where(and(eq(runnerApiResponseReservations.companyId, fixture.companyId),
isNotNull(runnerApiResponseReservations.pendingDeleteObjectKey)))).toEqual([]);
});
it("bounds a stalled snapshot delete and keeps it for a later retry", async () => {
const fixture = await server.fixture();
const objectKey = `${fixture.companyId}/runner-api/stalled-delete`;
await server.db.insert(runnerApiResponseReservations).values({ companyId: fixture.companyId, runId: null, reservedBytes: 1, pendingDeleteObjectKey: objectKey });
const stalled = { ...server.storage, deleteObject: (_companyId: string, _key: string, options?: { signal?: AbortSignal }) =>
new Promise<void>((_, reject) => options?.signal?.addEventListener("abort", () => reject(options.signal!.reason))) };
const startedAt = Date.now();
expect(await finishPendingRunnerApiCaptureDeletes(server.db, stalled as typeof server.storage, fixture.companyId, 50)).toEqual({ deleted: 0, failed: 1 });
expect(await server.db.select().from(toolRuntimeMetricCounters).where(and(eq(toolRuntimeMetricCounters.companyId, fixture.companyId),
eq(toolRuntimeMetricCounters.metric, RUNNER_API_CAPTURE_DELETE_FAILURE_METRIC)))).toEqual([expect.objectContaining({ count: 1 })]);
expect(Date.now() - startedAt).toBeLessThan(5_000);
expect(await server.db.select().from(runnerApiResponseReservations).where(eq(runnerApiResponseReservations.companyId, fixture.companyId))).toEqual([
expect.objectContaining({ pendingDeleteObjectKey: objectKey }),
]);
const deleteSpy = vi.spyOn(server.storage, "deleteObject");
try {
expect(await finishPendingRunnerApiCaptureDeletes(server.db, server.storage, fixture.companyId)).toEqual({ deleted: 1, failed: 0 });
expect(deleteSpy).toHaveBeenCalledWith(fixture.companyId, objectKey, expect.objectContaining({ allVersions: true }));
} finally { deleteSpy.mockRestore(); }
expect(await server.db.select().from(runnerApiResponseReservations).where(eq(runnerApiResponseReservations.companyId, fixture.companyId))).toEqual([]);
});
it("retries earlier pending deletes before evicting more snapshots", async () => {
const fixture = await server.fixture();
vi.stubEnv("PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES", String(2 * RUNNER_API_RESPONSE_MAX_BYTES));
const stored = await server.storage.putFile({ companyId: fixture.companyId, namespace: "runner-api", originalFilename: "response.json",
contentType: "application/json", body: Buffer.from("{}") });
await server.db.insert(runnerApiResponseReservations).values({ companyId: fixture.companyId, runId: null,
reservedBytes: RUNNER_API_RESPONSE_MAX_BYTES, pendingDeleteObjectKey: stored.objectKey });
const [survivor] = await server.db.insert(assets).values({ companyId: fixture.companyId, provider: "local_disk",
objectKey: `${fixture.companyId}/runner-api/survivor`, contentType: "text/plain", byteSize: RUNNER_API_RESPONSE_MAX_BYTES / 2,
sha256: "survivor", createdByAgentId: fixture.agentId, createdAt: new Date(Date.now() - 86_400_000) }).returning();
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
expect(await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
} })).toMatchObject({ ok: true });
expect((await server.db.select({ id: assets.id }).from(assets).where(eq(assets.companyId, fixture.companyId))).map(row => row.id)).toContain(survivor.id);
expect((await server.storage.headObject(fixture.companyId, stored.objectKey)).exists).toBe(false);
});
it("says storage cleanup is failing when old snapshot deletes keep failing", async () => {
const fixture = await server.fixture();
vi.stubEnv("PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES", String(2 * RUNNER_API_RESPONSE_MAX_BYTES));
await server.db.insert(runnerApiResponseReservations).values({ companyId: fixture.companyId, runId: null,
reservedBytes: RUNNER_API_RESPONSE_MAX_BYTES * 1.5, pendingDeleteObjectKey: `${fixture.companyId}/runner-api/undeletable`,
createdAt: new Date(Date.now() - 3_600_000) });
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
const spy = vi.spyOn(server.storage, "deleteObject").mockRejectedValue(new Error("storage unavailable"));
let result: any;
try {
result = await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
} });
} finally { spy.mockRestore(); }
expect(result).toMatchObject({ ok: false, error: "api_response_company_storage_limit" });
expect(result.message).toMatch(/keeps failing to delete/);
expect(result.message).not.toMatch(/operator|reservation|snapshot|quota/i);
});
it("rejects and removes an upload that finishes after its reservation was reclaimed", async () => {
const fixture = await server.fixture();
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
const before = (await server.db.select({ id: assets.id }).from(assets).where(eq(assets.companyId, fixture.companyId))).length;
const putFile = server.storage.putFile.bind(server.storage);
let objectKey = "";
let uploadSignal: AbortSignal | undefined;
const spy = vi.spyOn(server.storage, "putFile").mockImplementationOnce(async input => {
uploadSignal = input.signal;
const saved = await putFile(input);
objectKey = saved.objectKey;
await server.db.delete(runnerApiResponseReservations).where(eq(runnerApiResponseReservations.companyId, fixture.companyId));
return saved;
});
try {
await expect(fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
} })).rejects.toThrow("expired before it could be stored");
} finally { spy.mockRestore(); }
expect(uploadSignal).toBeInstanceOf(AbortSignal);
expect((await server.db.select({ id: assets.id }).from(assets).where(eq(assets.companyId, fixture.companyId))).length).toBe(before);
expect((await server.storage.headObject(fixture.companyId, objectKey)).exists).toBe(false);
expect(await server.db.select().from(runnerApiResponseReservations).where(eq(runnerApiResponseReservations.companyId, fixture.companyId))).toEqual([]);
});
it("evicts only expired snapshots of finished runs and records each eviction", async () => {
const fixture = await server.fixture();
vi.stubEnv("PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES", String(2 * RUNNER_API_RESPONSE_MAX_BYTES));
const finishedRunId = randomUUID();
await server.db.insert(heartbeatRuns).values({ id: finishedRunId, companyId: fixture.companyId, agentId: fixture.agentId,
status: "cancelled", runtimeMode: "native", nativeIssueId: fixture.blockerId, contextSnapshot: { issueId: fixture.blockerId } });
const [recent] = await server.db.insert(assets).values({ companyId: fixture.companyId, provider: "local_disk",
objectKey: `${fixture.companyId}/runner-api/recent-finished`, contentType: "text/plain", byteSize: RUNNER_API_RESPONSE_MAX_BYTES * 1.5,
sha256: "recent-finished", createdByAgentId: fixture.agentId, createdAt: new Date(Date.now() - 5 * 60_000) }).returning();
await server.db.insert(runnerApiResponseReservations).values({ companyId: fixture.companyId, runId: finishedRunId, assetId: recent.id, reservedBytes: recent.byteSize });
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
const call = () => fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
} });
expect(await call()).toMatchObject({ ok: false, error: "api_response_company_storage_limit" });
expect((await server.db.select({ id: assets.id }).from(assets).where(eq(assets.id, recent.id)))).toHaveLength(1);
await server.db.update(assets).set({ createdAt: new Date(Date.now() - 2 * 3_600_000) }).where(eq(assets.id, recent.id));
expect(await call()).toMatchObject({ ok: true });
expect((await server.db.select({ id: assets.id }).from(assets).where(eq(assets.id, recent.id)))).toHaveLength(0);
expect(await server.db.select().from(activityLog).where(and(eq(activityLog.companyId, fixture.companyId), eq(activityLog.action, "asset.evicted")))).toEqual([
expect.objectContaining({ actorType: "system", entityType: "asset", entityId: recent.id, runId: finishedRunId }),
]);
});
it("runs another eviction pass when a concurrent capture takes the freed space", async () => {
const fixture = await server.fixture();
vi.stubEnv("PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES", String(2 * RUNNER_API_RESPONSE_MAX_BYTES));
const expired = async (name: string, ageMs: number) => (await server.db.insert(assets).values({ companyId: fixture.companyId, provider: "local_disk",
objectKey: `${fixture.companyId}/runner-api/${name}`, contentType: "text/plain", byteSize: RUNNER_API_RESPONSE_MAX_BYTES,
sha256: name, createdByAgentId: fixture.agentId, createdAt: new Date(Date.now() - ageMs) }).returning())[0];
const older = await expired("older", 3 * 3_600_000);
const newer = await expired("newer", 2 * 3_600_000);
const otherRunId = randomUUID();
await server.db.insert(heartbeatRuns).values({ id: otherRunId, companyId: fixture.companyId, agentId: fixture.agentId,
status: "running", runtimeMode: "native", nativeIssueId: fixture.blockerId, contextSnapshot: { issueId: fixture.blockerId } });
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
const deleteObject = server.storage.deleteObject.bind(server.storage);
let stolen = false;
const spy = vi.spyOn(server.storage, "deleteObject").mockImplementation(async (companyId, key, options) => {
if (!stolen) {
stolen = true;
await server.db.insert(runnerApiResponseReservations).values({ companyId: fixture.companyId, runId: otherRunId, reservedBytes: RUNNER_API_RESPONSE_MAX_BYTES });
}
return deleteObject(companyId, key, options);
});
let result: any;
try {
result = await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
} });
} finally { spy.mockRestore(); }
expect(stolen).toBe(true);
expect(result).toMatchObject({ ok: true });
const remaining = (await server.db.select({ id: assets.id }).from(assets).where(eq(assets.companyId, fixture.companyId))).map(row => row.id);
expect(remaining).not.toContain(older.id);
expect(remaining).not.toContain(newer.id);
});
it("releases stale reservations left by interrupted captures of finished runs", async () => {
const fixture = await server.fixture();
vi.stubEnv("PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES", String(2 * RUNNER_API_RESPONSE_MAX_BYTES));
const finishedRunId = randomUUID();
await server.db.insert(heartbeatRuns).values({ id: finishedRunId, companyId: fixture.companyId, agentId: fixture.agentId,
status: "failed", runtimeMode: "native", nativeIssueId: fixture.blockerId, contextSnapshot: { issueId: fixture.blockerId } });
const longAgo = new Date(Date.now() - 3_600_000);
await server.db.insert(runnerApiResponseReservations).values([
{ companyId: fixture.companyId, runId: null, reservedBytes: RUNNER_API_RESPONSE_MAX_BYTES, createdAt: longAgo },
{ companyId: fixture.companyId, runId: finishedRunId, reservedBytes: RUNNER_API_RESPONSE_MAX_BYTES, createdAt: longAgo },
]);
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
const call = () => fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
} });
expect(await call()).toMatchObject({ ok: true });
expect(await call()).toMatchObject({ ok: true });
const reservations = await server.db.select().from(runnerApiResponseReservations).where(eq(runnerApiResponseReservations.companyId, fixture.companyId));
expect(reservations.every(row => row.runId === fixture.runId && row.assetId)).toBe(true);
});
it("keeps snapshots of active runs, attached snapshots, and in-flight reservations when the company quota is full", async () => {
const fixture = await server.fixture();
vi.stubEnv("PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES", String(2 * RUNNER_API_RESPONSE_MAX_BYTES));
const activeRunId = randomUUID();
await server.db.insert(heartbeatRuns).values({ id: activeRunId, companyId: fixture.companyId, agentId: fixture.agentId,
status: "running", runtimeMode: "native", nativeIssueId: fixture.blockerId, contextSnapshot: { issueId: fixture.blockerId } });
const longAgo = new Date(Date.now() - 86_400_000);
const [held] = await server.db.insert(assets).values({ companyId: fixture.companyId, provider: "local_disk",
objectKey: `${fixture.companyId}/runner-api/active-run-snapshot`, contentType: "text/plain", byteSize: RUNNER_API_RESPONSE_MAX_BYTES / 2,
sha256: "active-run-snapshot", createdByAgentId: fixture.agentId, createdAt: longAgo }).returning();
await server.db.insert(runnerApiResponseReservations).values([
{ companyId: fixture.companyId, runId: activeRunId, assetId: held.id, reservedBytes: held.byteSize },
{ companyId: fixture.companyId, runId: null, reservedBytes: RUNNER_API_RESPONSE_MAX_BYTES / 2 },
]);
const [attached] = await server.db.insert(assets).values({ companyId: fixture.companyId, provider: "local_disk",
objectKey: `${fixture.companyId}/runner-api/attached-snapshot`, contentType: "text/plain", byteSize: RUNNER_API_RESPONSE_MAX_BYTES / 2,
sha256: "attached-snapshot", createdByAgentId: fixture.agentId, createdAt: longAgo }).returning();
await server.db.insert(issueAttachments).values({ companyId: fixture.companyId, issueId: fixture.issueId, assetId: attached.id });
const linked = async (name: string) => (await server.db.insert(assets).values({ companyId: fixture.companyId, provider: "local_disk",
objectKey: `${fixture.companyId}/runner-api/${name}`, contentType: "text/plain", byteSize: 1024 * 1024,
sha256: name, createdByAgentId: fixture.agentId, createdAt: new Date(longAgo.getTime() - 1000) }).returning())[0];
const caseAsset = await linked("case-snapshot");
const logoAsset = await linked("logo-snapshot");
const caseId = randomUUID();
await server.db.insert(cases).values({ id: caseId, companyId: fixture.companyId, caseNumber: 1, identifier: `CASE-${caseId}`, caseType: "work", title: "Retention" });
await server.db.insert(caseAttachments).values({ companyId: fixture.companyId, caseId, assetId: caseAsset.id });
await server.db.insert(companyLogos).values({ companyId: fixture.companyId, assetId: logoAsset.id });
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
const result: any = await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
} });
expect(result).toMatchObject({ ok: false, error: "api_response_company_storage_limit" });
expect(result.message).not.toMatch(/operator/i);
const remaining = await server.db.select({ id: assets.id }).from(assets).where(eq(assets.companyId, fixture.companyId));
expect(remaining.map(row => row.id)).toEqual(expect.arrayContaining([held.id, attached.id, caseAsset.id, logoAsset.id]));
expect(await server.db.select().from(runnerApiResponseReservations).where(eq(runnerApiResponseReservations.companyId, fixture.companyId))).toHaveLength(2);
});
it("keeps company storage reserved after an ambiguous upload failure", async () => {
const fixture = await server.fixture();
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
const spy = vi.spyOn(server.storage, "putFile").mockRejectedValueOnce(new Error("storage disconnected"));
try {
await expect(fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
} })).rejects.toThrow("storage disconnected");
} finally { spy.mockRestore(); }
expect(await server.db.select().from(runnerApiResponseReservations).where(eq(runnerApiResponseReservations.companyId, fixture.companyId))).toEqual([
expect.objectContaining({ assetId: null, reservedBytes: RUNNER_API_RESPONSE_MAX_BYTES }),
]);
});
it.each([false, true])("reconciles metadata failure without losing committed assets (commit=%s)", async committed => {
const fixture = await server.fixture();
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
const putFile = server.storage.putFile.bind(server.storage);
const transaction = server.db.transaction.bind(server.db);
let objectKey = "";
let transactionSpy: ReturnType<typeof vi.spyOn> | undefined;
const spy = vi.spyOn(server.storage, "putFile").mockImplementationOnce(async input => {
const saved = await putFile(input);
objectKey = saved.objectKey;
transactionSpy = vi.spyOn(server.db, "transaction").mockImplementationOnce(async (fn, config) => {
if (committed) await transaction(fn, config);
throw new Error("metadata transaction failed");
});
return saved;
});
try {
await expect(fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
} })).rejects.toThrow("metadata transaction failed");
} finally { spy.mockRestore(); transactionSpy?.mockRestore(); }
const reservations = await server.db.select().from(runnerApiResponseReservations).where(eq(runnerApiResponseReservations.companyId, fixture.companyId));
expect(reservations).toHaveLength(committed ? 1 : 0);
if (committed) expect(reservations[0].assetId).toBeTruthy();
expect((await server.storage.headObject(fixture.companyId, objectKey)).exists).toBe(committed);
});
it("shares company admission across simultaneous runs", async () => {
const fixture = await server.fixture();
vi.stubEnv("PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES", String(RUNNER_API_RESPONSE_MAX_BYTES));
const otherRunId = randomUUID();
await server.db.insert(heartbeatRuns).values({ id: otherRunId, companyId: fixture.companyId, agentId: fixture.agentId,
status: "running", runtimeMode: "native", nativeIssueId: fixture.blockerId, contextSnapshot: { issueId: fixture.blockerId } });
await server.db.update(issues).set({ status: "in_progress", executionRunId: otherRunId }).where(eq(issues.id, fixture.blockerId));
const other = new PaperclipRunnerToolAuthority(server.db, { ...fixture, issueId: fixture.blockerId, runId: otherRunId });
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
const arguments_ = { operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId } };
const results: any[] = await Promise.all([fixture.authority, other].map(authority => authority.execute({ tool: "call_api", callId: randomUUID(), arguments: arguments_ })));
expect(results.filter(result => result.ok)).toHaveLength(1);
expect(results.find(result => !result.ok)).toMatchObject({ error: "api_response_company_storage_limit" });
});
it("releases a cleaned failed capture's company reservation but retains its run charge", async () => {
const fixture = await server.fixture();
let emitted = false;
const stream = new ReadableStream({
async pull(target) {
if (!emitted) { emitted = true; target.enqueue(Buffer.alloc(32 * 1024)); }
else { await new Promise(resolve => setTimeout(resolve, 100)); target.error(new Error("connection reset")); }
},
});
const spy = vi.spyOn(globalThis, "fetch").mockResolvedValueOnce(new Response(stream, { headers: { "content-type": "text/plain" } }));
try {
expect(await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
} })).toMatchObject({ ok: false, error: "api_transport_failure" });
} finally { spy.mockRestore(); }
expect(await server.db.select().from(runnerApiResponseReservations).where(eq(runnerApiResponseReservations.companyId, fixture.companyId))).toEqual([]);
const [run] = await server.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
expect((run.resultJson as Record<string, unknown>).apiResponseCaptureBytes).toBe(RUNNER_API_RESPONSE_MAX_BYTES);
});
it("persists and range-reads asset sizes above two GiB", async () => {
const fixture = await server.fixture();
const offsetBytes = 3 * 1024 * 1024 * 1024;
const objectKey = `${fixture.companyId}/eval/sparse-large-response.txt`;
const path = join(server.root, "storage", objectKey);
await mkdir(dirname(path), { recursive: true });
const file = await open(path, "wx");
try { await file.write(Buffer.from("readable"), 0, 8, offsetBytes); }
finally { await file.close(); }
const [asset] = await server.db.insert(assets).values({ companyId: fixture.companyId, provider: "local_disk", objectKey,
contentType: "text/plain", byteSize: offsetBytes + 8, sha256: "sparse-fixture", createdByAgentId: fixture.agentId }).returning();
expect(asset.byteSize).toBe(offsetBytes + 8);
const page = await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: asset.id }, responseText: { offsetBytes, limitBytes: 8 },
} });
expect(page).toMatchObject({ ok: true, data: "readable", responseText: { offsetBytes, totalBytes: offsetBytes + 8, nextOffsetBytes: null } });
});
it("contains workspace files, checks artifact ownership, and persists downloads", async () => {
const fixture = await server.fixture();
const foreign = await server.fixture();
const upload = (files: unknown) => fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: { operationId: "POST /api/companies/{companyId}/issues/{issueId}/attachments", pathParams: { issueId: fixture.issueId }, files } });
await expect(upload([{ artifactId: foreign.artifactId }])).rejects.toThrow("not available");
await expect(upload([{ path: "../outside.txt" }])).rejects.toThrow();
const outside = join(server.root, "outside.txt");
await writeFile(outside, "must-not-upload");
await symlink(outside, join(fixture.workspace, "escape.txt"));
await expect(upload([{ path: "escape.txt" }])).rejects.toThrow();
await expect(upload([{ path: "sample.txt" }])).resolves.toMatchObject({ status: 201 });
const download = await fixture.authority.execute({ tool: "call_api", callId: "download", arguments: { operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: fixture.binaryArtifactId } } }) as any;
expect(download).toMatchObject({ status: 206, byteSize: 32000, artifact: { artifactId: fixture.binaryArtifactId, byteSize: 32000 } });
expect((await fixture.snapshot()).assets).toEqual(expect.arrayContaining([expect.objectContaining({ id: download.artifact.artifactId, companyId: fixture.companyId, createdByAgentId: fixture.agentId })]));
});
it.each(["sample.txt", "/home/agent/proof.txt"])("uploads remote task bytes at %s and fails closed without the reader", async (sourcePath) => {
const fixture=await server.fixture();
const bytes=Buffer.from("proof exists only on the execution host\n");
const read=vi.fn(async(path:string,maxBytes:number)=>{expect(path).toBe(sourcePath);expect(maxBytes).toBeGreaterThan(bytes.length);return bytes;});
const binding={...fixture.authority.binding,executionTargetKind:"remote" as const,readRemoteApiFile:read};
const authority=new PaperclipRunnerToolAuthority(server.db,binding);
const call={tool:"call_api" as const,callId:randomUUID(),arguments:{operationId:"POST /api/companies/{companyId}/issues/{issueId}/attachments",pathParams:{issueId:fixture.issueId},files:[{path:sourcePath}]}};
await expect(authority.execute(call)).resolves.toMatchObject({status:201});
expect(read).toHaveBeenCalledOnce();
expect((await fixture.snapshot()).assets).toEqual(expect.arrayContaining([expect.objectContaining({sha256:createHash("sha256").update(bytes).digest("hex"),byteSize:bytes.length})]));
const unsupported=new PaperclipRunnerToolAuthority(server.db,{...binding,readRemoteApiFile:undefined});
await expect(unsupported.execute({...call,callId:randomUUID()})).rejects.toThrow("Remote workspace file uploads are unavailable");
const changing=new PaperclipRunnerToolAuthority(server.db,{...binding,readRemoteApiFile:async()=>{await server.db.update(heartbeatRuns).set({status:"cancelled"}).where(eq(heartbeatRuns.id,fixture.runId));return bytes;}});
await expect(changing.execute({...call,callId:randomUUID()})).rejects.toThrow();
});
it("does not accept caller-supplied identity in an API comment", async () => {
const fixture = await server.fixture();
const result = await fixture.authority.execute({ tool: "call_api", callId: "comment", arguments: { operationId: "POST /api/issues/{id}/comments", pathParams: { id: fixture.issueId }, body: { body: "Identity proof", authorAgentId: randomUUID(), authorUserId: "spoofed", runId: randomUUID() } } });
expect(result).toMatchObject({ status: 201 });
expect((await fixture.snapshot()).comments).toEqual(expect.arrayContaining([expect.objectContaining({ body: "Identity proof", authorAgentId: fixture.agentId, authorUserId: null })]));
});
it("rejects a stale source-run question over public HTTP and accepts it from the next run", async () => {
const localTrustedServer = await startRunnerApiTestServer({ deploymentMode: "local_trusted" });
try {
const fixture = await localTrustedServer.fixture({
disableWakeOnDemand: true,
contextSnapshot: { paperclipWake: { comments: [] } },
});
const sourceToken = createLocalAgentJwt(
fixture.agentId,
fixture.companyId,
"paperclip_runner",
fixture.runId,
);
expect(sourceToken).toBeTruthy();
const requestHeaders = (token: string) => ({
authorization: `Bearer ${token}`,
"content-type": "application/json",
"x-paperclip-run-id": fixture.runId,
});
const humanCommentResponse = await fetch(
`${fixture.apiUrl}/api/issues/${fixture.issueId}/comments`,
{
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ body: "Human direction committed after the old wake." }),
},
);
expect(humanCommentResponse.status).toBe(201);
const humanComment = await humanCommentResponse.json() as { id: string; authorUserId?: string | null; authorAgentId?: string | null; createdByRunId?: string | null };
expect(humanComment).toMatchObject({ authorUserId: "local-board", authorAgentId: null, createdByRunId: null });
const questionPayload = {
kind: "ask_user_questions",
idempotencyKey: `stale-source-question-${fixture.runId}`,
continuationPolicy: "wake_assignee",
payload: {
version: 1,
title: "One real question",
questions: [{
id: "scope",
prompt: "Choose a scope.",
selectionMode: "single",
required: true,
options: [{ id: "small", label: "Small" }],
}],
},
};
const staleResponse = await fetch(
`${fixture.apiUrl}/api/issues/${fixture.issueId}/interactions`,
{ method: "POST", headers: requestHeaders(sourceToken!), body: JSON.stringify(questionPayload) },
);
expect(staleResponse.status).toBe(409);
const staleBody = await staleResponse.json() as { details?: { reason?: string; commentIds?: string[] } };
expect(staleBody.details).toMatchObject({ reason: "newer_comment_not_delivered", commentIds: [humanComment.id] });
const afterStale = await fetch(`${fixture.apiUrl}/api/issues/${fixture.issueId}/interactions`);
expect(afterStale.status).toBe(200);
expect(await afterStale.json()).toEqual([]);
const successorRunId = randomUUID();
await localTrustedServer.db.insert(heartbeatRuns).values({
id: successorRunId,
companyId: fixture.companyId,
agentId: fixture.agentId,
status: "running",
runtimeMode: "native",
nativeIssueId: fixture.issueId,
invocationSource: "continuation",
triggerDetail: "comment",
contextSnapshot: { issueId: fixture.issueId, paperclipWake: { comments: [{ id: humanComment.id }] } },
});
await localTrustedServer.db.update(issues).set({ executionRunId: successorRunId, checkoutRunId: successorRunId }).where(eq(issues.id, fixture.issueId));
const successorToken = createLocalAgentJwt(
fixture.agentId,
fixture.companyId,
"paperclip_runner",
successorRunId,
);
expect(successorToken).toBeTruthy();
const acceptedResponse = await fetch(
`${fixture.apiUrl}/api/issues/${fixture.issueId}/interactions`,
{
method: "POST",
headers: { ...requestHeaders(successorToken!), "x-paperclip-run-id": successorRunId },
body: JSON.stringify({ ...questionPayload, idempotencyKey: `fresh-source-question-${successorRunId}` }),
},
);
const acceptedText = await acceptedResponse.text();
expect(acceptedResponse.status, acceptedText).toBe(201);
const accepted = JSON.parse(acceptedText) as { id: string; status: string; sourceRunId: string };
expect(accepted).toMatchObject({ status: "pending", sourceRunId: successorRunId });
const answerResponse = await fetch(
`${fixture.apiUrl}/api/issues/${fixture.issueId}/interactions/${accepted.id}/respond`,
{
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ answers: [{ questionId: "scope", optionIds: ["small"] }] }),
},
);
expect(answerResponse.status).toBe(200);
expect((await answerResponse.json() as { status: string }).status).toBe("answered");
} finally {
await localTrustedServer.close();
}
});
it("creates child tasks after seeding and preserves company numbering", async () => {
const fixture = await server.fixture();
await expect(fixture.authority.execute({ tool: "create_task", callId: "child", arguments: { title: "Verify release notes", description: "Check before shipping.", assigneeActorId: null, idempotencyKey: "child" } })).resolves.toBeDefined();
const children = (await fixture.snapshot()).issues.filter(issue => issue.parentId === fixture.issueId);
expect(children).toHaveLength(1);
expect(children[0].issueNumber).toBe(3);
});
it("resets fixture data and identities for paired comparisons", async () => {
const baseline = await server.fixture({ reset: true, apiToolsEnabled: false });
await server.db.update(issues).set({ billingCode: "previous-attempt" }).where(eq(issues.id, baseline.issueId));
const treatment = await server.fixture({ reset: true });
expect(treatment.companyId).toBe(baseline.companyId);
expect(treatment.issueId).toBe(baseline.issueId);
expect((await treatment.snapshot()).issues[0].billingCode).toBeNull();
const originalTools = (await baseline.authority.definitions()).map(tool => tool.name);
const treatmentTools = (await treatment.authority.definitions()).map(tool => tool.name);
const apiTools = ["hire_agent", "search_api", "call_api", "get_task", "comment_on_task", "list_task_documents", "read_task_document", "write_task_document", "save_api_response"];
expect(treatmentTools.filter(name => !originalTools.includes(name))).toEqual(apiTools);
expect(treatmentTools.filter(name => !apiTools.includes(String(name)))).toEqual(originalTools);
});
});