Add bulk keyed issue document reads for native helpers (#22105)

## Thinking Path

> - Paperclip manages work for AI agents.
> - Native helpers read task documents through authorized API tools.
> - A scan of 146 cases needs one saved response and one download per
case.
> - The document service can read the same key across authorized tasks
with bounded queries.
> - This pull request adds a bounded bulk read with the existing task
access policy.
> - Helpers can transfer one batch and check current revisions before
fetching changed bodies.

## Linked Issues or Issue Description

**Subsystem affected**

Task document REST API, shared query contracts, and native API
discovery.

**Problem or motivation**

Per-task document reads make native sync scans cost hundreds of tool
calls. Task list timestamps do not prove that document content is
unchanged.

**Proposed solution**

Add `GET
/api/companies/{companyId}/issue-documents?key=notes&issueIds=...`. Read
up to 200 task UUIDs under current company and task permissions. Return
current bodies and revision metadata, or metadata alone with
`includeBody=false`.

**Alternatives considered**

Generic request batching would add tool execution and partial failure
rules. One bulk resource read removes the per-case relay cost with the
existing tools. Do not skip documents based only on task-row timestamps.
Keep managed skill files immutable; cleanup of copied skill test
fixtures belongs to those tests.

**Roadmap alignment**

This improves existing task documents and native API access. It adds no
new workflow or UI. The roadmap has no duplicate bulk document read
work. No duplicate open pull request was found.

## What Changed

- Add a bulk keyed document route and a company-scoped document reader.
Reuse the current batched task-read policy and completed-task read
receipts.
- Preserve request order and deduplicate IDs. Return `document: null`
for readable tasks without the key. Omit unreadable, hidden, foreign and
missing tasks without revealing which exist.
- Bound requests to 200 IDs and responses to 8 MiB of JSON bytes. Check
encoded body sizes in SQL before fetching bodies in the same read
snapshot. Build JSON within the byte budget. Oversized responses fail
with 413 and are never saved as partial artifacts.
- Admit at most two active reads per actor and eight per server
instance. Extra requests receive 429 with Retry-After. Keep the slot
until both the database work and response settle.
- Batch completed-handoff lookups and group child receipts by parent.
Extend the existing event batch writer for ordinary non-waking receipts
with atomic sequence allocation. Stored receipt shapes stay unchanged
for existing consumers.
- Expose revision-only reads without selecting document bodies. Document
native discovery, one save plus one download, and complete-inventory
checks.
- Add real HTTP, database and artifact tests for 146 open and completed
cases, 200 maximum-size bodies, metadata overhead, access failures,
query limits and JSON expansion. Cover admission limits, failure
cleanup, and concurrent event writers. Refresh the shared SDK source
pins.

## Verification

Final revision: `93dc8b029600211d79a050650f3dff82e5dad52f`.

- [CI run 44212220917, attempt
3](https://github.com/paperclipai/paperclip/actions/runs/44212220917)
passed. This includes typecheck, build, general tests, Runner checks,
serialized server suites, all e2e shards, and the final verify gate.
-
[Build](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134506305267),
[e2e](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134506309117),
and
[verify](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134510062263)
are green.
- [Linux CI job
134506309291](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134506309291)
passed all 16 bulk document integration tests. This includes the native
transfer of all 146 documents with the saved size and hash,
completed-case receipt batching, and size-only rejection of 200
maximum-size documents.
- [Greptile
review](https://github.com/paperclipai/paperclip/pull/22105#issuecomment-6965449032)
is 5/5 on the final revision. All review threads are resolved. Security
checks pass. GitHub reports CLEAN and MERGEABLE.
- Local server typecheck passed on macOS with Node 25.6.1. The focused
run below passed 177 tests; the Linux-only transfer test was skipped
locally and then passed in CI.

```sh
pnpm --filter @paperclipai/server typecheck
cd server
pnpm exec vitest run \
  src/routes/issue-document-batch.test.ts \
  src/services/native-runtime/issue-document-batch.integration.test.ts \
  src/__tests__/documents-service.test.ts \
  src/__tests__/openapi-routes.test.ts \
  src/__tests__/heartbeat-run-event-atomic-append.test.ts \
  src/__tests__/chat-completion-delivery.test.ts \
  src/__tests__/native-child-task-read-routes.test.ts
```

- Shared typecheck and build passed. From `packages/shared/`, `pnpm exec
vitest run src/validators/issue.test.ts` passed 61 tests before the
rebase; the batch query contract did not change in the rebase.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`
passed. Runtime and lockfile pins are unchanged.
- `pnpm verify --explain` returned exit 2 because the API docs and
shared contracts require broader CI. The successful CI run above
supplies that coverage.

Failure and recovery record:

- Both performance findings reproduced before the fixes. Scale tests now
verify a size-only query for 200 large documents and one handoff lookup
plus two receipt-write statements for 146 completed cases. Concurrent
writers retain contiguous unique sequences and decoded payloads.
- Initial local runs found missing shared exports and OpenAPI route
coverage. These are fixed. Fixtures now respect Linux-only installation,
thrown cross-company denials, and the ES2022 TypeScript target. Rebase
conflicts preserved the onboarding coverage from master.
- CI attempt 1 interrupted Build, e2e shard 7 and six general-test
shards with runner shutdown signals. Attempt 2 recovered the tests but
its [Build
job](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134499680425)
also received a shutdown signal and exited 143. No compiler or test
assertion failure was found in those interrupted job logs. The reason
the runners stopped remains unknown. Attempt 3 recovered Build and its
dependent verify gate on the same revision. Earlier interruptions are
not counted as passes.

## Risks

- Additive API only. No migration or existing endpoint response change.
- Helpers must adopt the batch response to gain the tool-call reduction.
They must verify that every requested task is present before treating
the scan as complete. Unreadable tasks are intentionally omitted.
- Split large document batches when the route returns 413. Metadata
remains available without bodies. Concurrency limits are per server
instance and can refuse simultaneous reads with 429; callers retry after
an active read finishes.
- This does not change company-owned helper inventory expiry or cleanup
tests, installed skill permissions, or generic tool batching. It does
not deploy to a tenant.

## Model Used

OpenAI `gpt-6.1-sol` through Codex in T3 Code, with xhigh reasoning,
tool use, and code execution. Context window size was not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin FoleyandPaperclip authored and GitHub committed 2026-11-17 17:42:18 -08:00
1 parent bf55063567
commit ea154ce55e
20 files changed
+634 -30

No files matched your search

+30
View File
@@ -343,6 +343,36 @@ GET /api/issues/{issueId}/documents
GET /api/issues/{issueId}/documents/{key}
```
### Bulk Get By Key
```
GET /api/companies/{companyId}/issue-documents?key=sentry-case&issueIds={uuid1},{uuid2}
```
Read one document key across 1–200 task UUIDs. Items follow the requested order;
duplicate IDs appear once. The result is `{ key, items: [{ issueId, document }] }`.
Each document has the same body and revision metadata as Get By Key. A readable
task without the key returns `document: null`. Missing, hidden, foreign-company,
and unreadable tasks are omitted without revealing whether they exist. Callers
must check that every requested ID is present before treating a scan as complete.
Task bridge keys cannot use this company-wide route. Break-glass reads are not supported.
Use `includeBody=false` for current revision IDs, revision numbers, and update
times without loading document bodies. A helper with cached documents can compare
their revisions, then fetch bodies only for changed or newly added documents.
Task-row timestamps alone do not prove that a document is unchanged.
Responses use `Cache-Control: private, no-store`. A response over 8 MiB returns
`413`; split the IDs into smaller requests. The response is never truncated.
Body sizes are checked in the database before loading them. Reads use one
snapshot so a concurrent edit cannot grow an admitted body set. At most two
reads per actor and eight per server instance can run at once; extra reads
return `429` with `Retry-After: 1`.
Native agents can discover this GET with `search_api`, save the whole batch with
one `save_api_response`, and transfer it with one `download_artifact`. The saved
artifact supplies its exact `artifactId`, `byteSize`, and `sha256` for the download.
This replaces per-task document relays; helpers must consume the batch result.
### Create Or Update
```
+2
View File
@@ -6,6 +6,8 @@ export const COMPANY_STATUSES = ["active", "paused", "archived"] as const;
export const ISSUE_SUMMARIES_MAX_IDS = 50;
/** Tasks per chat bootstrap request. The list preloads larger sets in chunks. */
export const ISSUE_CHAT_BOOTSTRAP_MAX_IDS = 10;
/** Tasks per keyed bulk document read; keeps a routine's case inventory in one request. */
export const ISSUE_DOCUMENT_BATCH_MAX_IDS = 200;
export type CompanyStatus = (typeof COMPANY_STATUSES)[number];
export const DEPLOYMENT_MODES = ["local_trusted", "authenticated"] as const;
+3
View File
@@ -426,6 +426,7 @@ export {
ISSUE_DISPLAY_STATUSES,
ISSUE_SUMMARIES_MAX_IDS,
ISSUE_CHAT_BOOTSTRAP_MAX_IDS,
ISSUE_DOCUMENT_BATCH_MAX_IDS,
INBOX_MINE_ISSUE_STATUSES,
INBOX_MINE_ISSUE_STATUS_FILTER,
ISSUE_PRIORITIES,
@@ -1285,6 +1286,7 @@ export type {
IssueThreadInteractionResult,
IssueDocument,
IssueDocumentSummary,
IssueDocumentBatch,
DocumentRevision,
ExactDocumentRevision,
DocumentFormat,
@@ -2176,6 +2178,7 @@ export {
workspaceFileWorkspaceKindSchema,
issueDocumentFormatSchema,
issueDocumentKeySchema,
issueDocumentBatchQuerySchema,
upsertIssueDocumentSchema,
restoreIssueDocumentRevisionSchema,
createIssueTreeHoldSchema,
+1
View File
@@ -874,6 +874,7 @@ export type {
IssueThreadInteractionResult,
IssueDocument,
IssueDocumentSummary,
IssueDocumentBatch,
DocumentRevision,
ExactDocumentRevision,
DocumentFormat,
+6
View File
@@ -198,6 +198,12 @@ export interface IssueDocument extends IssueDocumentSummary {
body: string;
}
/** Unreadable or nonexistent tasks are omitted. A readable task without the key has document: null. */
export interface IssueDocumentBatch {
key: string;
items: Array<{ issueId: string; document: IssueDocument | IssueDocumentSummary | null }>;
}
export interface DocumentRevision {
id: string;
companyId: string;
+1
View File
@@ -517,6 +517,7 @@ export {
createIssueAttachmentMetadataSchema,
issueDocumentFormatSchema,
issueDocumentKeySchema,
issueDocumentBatchQuerySchema,
upsertIssueDocumentSchema,
restoreIssueDocumentRevisionSchema,
upsertIssueWatchdogSchema,
@@ -4,6 +4,7 @@ import {
addIssueCommentSchema,
paperclipQuestionSetPayloadSchema,
issueCommentMetadataSchema,
issueDocumentBatchQuerySchema,
createIssueSchema,
issueBlockedInboxAttentionSchema,
resolveIssueRecoveryActionSchema,
@@ -16,6 +17,16 @@ import {
import { createAgentSchema } from "./agent.js";
describe("issue validators", () => {
it("bounds document batches before deduplication and normalizes keys and IDs", () => {
const id = "AAAAAAAA-AAAA-4AAA-8AAA-AAAAAAAAAAAA";
expect(issueDocumentBatchQuerySchema.parse({ key: " SENTRY-CASE ", issueIds: ` ${id},${id} ` })).toEqual({
key: "sentry-case", issueIds: [id.toLowerCase()], includeBody: true,
});
expect(issueDocumentBatchQuerySchema.parse({ key: "notes", issueIds: Array(200).fill(id).join(","), includeBody: "false" }).includeBody).toBe(false);
expect(issueDocumentBatchQuerySchema.safeParse({ key: "notes", issueIds: Array(201).fill(id).join(",") }).success).toBe(false);
expect(issueDocumentBatchQuerySchema.safeParse({ key: "notes", issueIds: [id] }).success).toBe(false);
});
it.each(["BASH_ENV", "ZDOTDIR", "GH_CONFIG_DIR", "PAPERCLIP_GITHUB_LAUNCHER_DIR"])("rejects task-owned managed shell selector %s", key => {
for (const name of [key, key.toLowerCase()]) {
const overrides = {adapterConfig:{env:{[name]:"/workspace/startup"}}};
+10 -1
View File
@@ -2,7 +2,7 @@ import { CHAT_PROVIDERS } from "../types/chat-channels.js";
import { taskWorkspaceSelectionSchema } from "./task-workspace.js";
import { z } from "zod";
import { maxStringLength } from "./text.js";
import { ISSUE_CHAT_BOOTSTRAP_MAX_IDS, ISSUE_SUMMARIES_MAX_IDS } from "../constants.js";
import { ISSUE_CHAT_BOOTSTRAP_MAX_IDS, ISSUE_DOCUMENT_BATCH_MAX_IDS, ISSUE_SUMMARIES_MAX_IDS } from "../constants.js";
import { normalizeIssueIdentifier } from "../issue-references.js";
import {
@@ -1336,6 +1336,15 @@ export const issueDocumentKeySchema = z
"Document key must be lowercase letters, numbers, _ or -",
);
export const issueDocumentBatchQuerySchema = z.object({
key: z.string().trim().toLowerCase().pipe(issueDocumentKeySchema),
issueIds: z.string().max(ISSUE_DOCUMENT_BATCH_MAX_IDS * 40)
.transform(value => value.split(",").map(id => id.trim().toLowerCase()))
.pipe(z.array(z.string().guid()).min(1).max(ISSUE_DOCUMENT_BATCH_MAX_IDS))
.transform(ids => [...new Set(ids)]),
includeBody: z.enum(["true", "false"]).default("true").transform(value => value === "true"),
}).strict();
export const suggestedTaskDraftSchema = z
.object({
clientKey: z.string().trim().min(1).max(120),
@@ -66,6 +66,42 @@ describeEmbeddedPostgres("atomic ordinary run events", () => {
}
});
it("appends ordinary read receipts in one statement with contiguous sequences under a concurrent writer", async () => {
const runId = await makeRun();
const statements: string[] = [];
const recorded = drizzle(db.$client, { schema: db._.fullSchema, logger: { logQuery: query => statements.push(query) } }) as typeof db;
const inputs = Array.from({ length: 146 }, (_, index) => ({
companyId, agentId, runId, eventType: "chat_handoff.completed_read", stream: "system", level: "info",
payload: { taskId: `task-${index}`, statusVersion: index, nested: ["before\u0000after"] },
}));
const [receipts] = await Promise.all([
appendHeartbeatRunEventBatch(recorded, inputs),
appendHeartbeatRunEvent(otherDb, { companyId, agentId, runId, eventType: "run.performance.span", payload: { concurrent: true } }),
]);
expect(statements).toHaveLength(1);
expect(receipts).toHaveLength(146);
for (const [index, receipt] of receipts.entries()) {
expect(receipt).toMatchObject({ disposition: "committed", highestContiguousSourceSeq: 0, row: { payload: inputs[index].payload } });
expect(receipt.row.seq).toBe(receipts[0].row.seq + index);
}
const events = await readEvents(runId);
expect(events.map(event => event.seq)).toEqual(Array.from({ length: 147 }, (_, index) => index + 1));
expect((await readRun(runId)).nextEventSeq).toBe(148);
});
it("refuses ordinary batches with mixed bindings or special event effects before writing", async () => {
const runId = await makeRun();
const input = { companyId, agentId, runId, eventType: "child_task.read" };
for (const other of [
{ ...input, runId: "30000000-0000-4000-8000-000000000099" },
{ ...input, companyId: "30000000-0000-4000-8000-000000000099" },
{ ...input, eventType: "native.process_start_requested" },
]) await expect(appendHeartbeatRunEventBatch(db, [input, other])).rejects.toThrow("heartbeat_run_event_batch_unsupported");
expect(await readEvents(runId)).toEqual([]);
expect((await readRun(runId)).nextEventSeq).toBe(1);
expect(await appendHeartbeatRunEventBatch(db, [])).toEqual([]);
});
it("persists a decoded receipt in one executed statement", async () => {
const runId = await makeRun();
const statements: string[] = [];
@@ -79,6 +79,7 @@ const apiPrefixes: Record<string, string> = {
"instance-server-metrics.ts": "/api",
"instance-settings.ts": "/api",
"issues.ts": "/api",
"issue-document-batch.ts": "/api",
"issue-tree-control.ts": "/api",
"llms.ts": "/api",
"link-previews.ts": "/api",
@@ -1325,6 +1326,18 @@ describe("openapi routes", () => {
.toMatchObject({ type: "object", required: ["started"], properties: { started: { type: "boolean" } } });
});
it("documents bounded bulk keyed document reads for native discovery", () => {
const { spec } = loadSpecRoutes();
const operation = spec.paths["/api/companies/{companyId}/issue-documents"].get;
expect(operation.parameters).toEqual(expect.arrayContaining([
expect.objectContaining({ in: "query", name: "key", required: true }),
expect.objectContaining({ in: "query", name: "issueIds", required: true, schema: expect.objectContaining({ type: "string", maxLength: 8000 }) }),
expect.objectContaining({ in: "query", name: "includeBody", required: false, schema: expect.objectContaining({ enum: ["true", "false"] }) }),
]));
expect(operation.responses["413"]).toBeDefined();
expect(operation.description).toContain("save_api_response");
});
it("covers the mounted server routes exactly", () => {
const {
routes: actualRoutes,
@@ -0,0 +1,91 @@
import express from "express";
import request from "supertest";
import type { Db } from "@paperclipai/db";
import { describe, expect, it, vi } from "vitest";
import { registerIssueDocumentBatchRoute } from "./issue-document-batch.js";
vi.mock("../services/authorization.js", () => ({ issueReadSqlCondition: async () => undefined }));
vi.mock("../services/documents.js", () => ({
ISSUE_DOCUMENT_BATCH_MAX_BYTES: 8 * 1024 * 1024,
documentService: () => ({ getIssueDocumentsByKey: async () => [] }),
}));
const companyId = "11111111-1111-4111-8111-111111111111";
const url = `/api/companies/${companyId}/issue-documents?key=notes&issueIds=22222222-2222-4222-8222-222222222222`;
function deferred<T>() {
let resolve!: (value: T) => void;
let reject!: (error: unknown) => void;
const promise = new Promise<T>((yes, no) => { resolve = yes; reject = no; });
return { promise, resolve, reject };
}
function fixture() {
const app = express(), router = express.Router();
const entered: Array<() => void> = [];
const gates: Array<ReturnType<typeof deferred<[]>>> = [];
const pending: Array<Promise<request.Response>> = [];
const db = { select: () => ({ from: () => ({ where: async () => [] }) }) } as unknown as Db;
app.use((req, _res, next) => {
req.actor = { type: "agent", companyId, agentId: req.header("actor")!, source: "agent_key" };
next();
});
registerIssueDocumentBatchRoute(router, db, {
filterReadable: async () => {
const gate = deferred<[]>();
gates.push(gate);
entered.shift()!();
return gate.promise;
},
recordReads: async () => undefined,
});
app.use("/api", router);
app.use((_error: unknown, _req: express.Request, res: express.Response, _next: express.NextFunction) => { res.status(500).json({ error: "test read failed" }); });
return {
app,
async start(actor: string) {
const signal = deferred<void>();
entered.push(signal.resolve);
const response = request(app).get(url).set("actor", actor).then(value => value);
pending.push(response);
await signal.promise;
return { response, gate: gates.at(-1)! };
},
async close() {
for (const gate of gates) gate.resolve([]);
await Promise.allSettled(pending);
},
};
}
describe("bulk document read admission", () => {
it("caps each actor and releases failed reads", async () => {
const f = fixture();
try {
const first = await f.start("one"), second = await f.start("one");
const refused = await request(f.app).get(url).set("actor", "one");
expect(refused.status).toBe(429);
expect(refused.headers["retry-after"]).toBe("1");
first.gate.reject(new Error("read failed"));
expect((await first.response).status).toBe(500);
const replacement = await f.start("one");
replacement.gate.resolve([]);
second.gate.resolve([]);
expect((await replacement.response).status).toBe(200);
} finally { await f.close(); }
});
it("caps total reads across actors", async () => {
const f = fixture();
try {
const reads = [];
for (let index = 0; index < 8; index++) reads.push(await f.start(`actor-${index}`));
expect((await request(f.app).get(url).set("actor", "ninth")).status).toBe(429);
reads[0].gate.resolve([]);
await reads[0].response;
const replacement = await f.start("ninth");
replacement.gate.resolve([]);
expect((await replacement.response).status).toBe(200);
} finally { await f.close(); }
});
});
+96
View File
@@ -0,0 +1,96 @@
import type { Request, Response, Router } from "express";
import { and, eq, inArray, isNull } from "drizzle-orm";
import { issues, type Db } from "@paperclipai/db";
import { issueDocumentBatchQuerySchema } from "@paperclipai/shared";
import { issueReadSqlCondition } from "../services/authorization.js";
import { documentService, ISSUE_DOCUMENT_BATCH_MAX_BYTES } from "../services/documents.js";
import { assertCompanyAccess } from "./authz.js";
type ReadableIssue = Pick<typeof issues.$inferSelect,
"id" | "companyId" | "projectId" | "parentId" | "assigneeAgentId" | "assigneeUserId" | "status" | "updatedAt" | "lastStatusDecisionId" | "statusVersion">;
const MAX_ACTOR_READS = 2;
const MAX_SERVER_READS = 8;
function createBatchReadAdmission() {
const active = new Map<string, number>();
let total = 0;
return (req: Request, res: Response): (() => void) | null => {
const actor = `${req.actor.type}:${req.actor.agentId ?? req.actor.userId ?? req.actor.source}`;
if (total >= MAX_SERVER_READS || (active.get(actor) ?? 0) >= MAX_ACTOR_READS) {
res.setHeader("Retry-After", "1");
res.status(429).json({ error: "Too many document batch reads; retry after an active read finishes" });
return null;
}
total++;
active.set(actor, (active.get(actor) ?? 0) + 1);
let workDone = false, responseDone = false, released = false;
const release = () => {
if (released || !workDone || !responseDone) return;
released = true;
total--;
const remaining = active.get(actor)! - 1;
if (remaining) active.set(actor, remaining);
else active.delete(actor);
};
const finish = () => { responseDone = true; release(); };
res.once("finish", finish);
res.once("close", finish);
// A disconnect cannot free database capacity while its read is still live.
return () => { workDone = true; release(); };
};
}
export function registerIssueDocumentBatchRoute(router: Router, db: Db, options: {
filterReadable: (req: Request, rows: ReadableIssue[]) => Promise<ReadableIssue[]>;
recordReads: (req: Request, issues: ReadableIssue[]) => Promise<void>;
}) {
const admit = createBatchReadAdmission();
router.get("/companies/:companyId/issue-documents", async (req, res) => {
const companyId = req.params.companyId as string;
assertCompanyAccess(req, companyId);
if (req.actor.type === "agent" && req.actor.source === "agent_key" && req.actor.keyScope?.kind === "task_bridge") {
res.status(403).json({ error: "Task bridge keys cannot use company-wide issue list APIs" });
return;
}
const parsed = issueDocumentBatchQuerySchema.safeParse(req.query);
if (!parsed.success) {
res.status(400).json({ error: "Invalid issue document batch query", details: parsed.error.issues });
return;
}
const { key, issueIds, includeBody } = parsed.data;
const settle = admit(req, res);
if (!settle) return;
try {
res.setHeader("Cache-Control", "private, no-store");
const candidates = await db.select({
id: issues.id, companyId: issues.companyId, projectId: issues.projectId, parentId: issues.parentId,
assigneeAgentId: issues.assigneeAgentId, assigneeUserId: issues.assigneeUserId, status: issues.status,
updatedAt: issues.updatedAt, lastStatusDecisionId: issues.lastStatusDecisionId, statusVersion: issues.statusVersion,
}).from(issues).where(and(
eq(issues.companyId, companyId), inArray(issues.id, issueIds), isNull(issues.hiddenAt),
await issueReadSqlCondition(db, req.actor),
));
const readable = await options.filterReadable(req, candidates);
const byId = new Map(readable.map(issue => [issue.id, issue]));
const docs = await documentService(db).getIssueDocumentsByKey(companyId, readable.map(issue => issue.id), key, includeBody);
const byIssueId = new Map(docs.map(doc => [doc.issueId, doc]));
const items = issueIds.filter(id => byId.has(id)).map(issueId => ({ issueId, document: byIssueId.get(issueId) ?? null }));
const parts = [`{"key":${JSON.stringify(key)},"items":[`];
let bytes = Buffer.byteLength(parts[0]) + 2; // closing array and object
for (const [index, item] of items.entries()) {
const part = `${index ? "," : ""}${JSON.stringify(item)}`;
bytes += Buffer.byteLength(part);
if (bytes > ISSUE_DOCUMENT_BATCH_MAX_BYTES) {
res.status(413).json({ error: "Document batch exceeds 8 MiB; split issueIds into smaller requests or use includeBody=false", maxResponseBytes: ISSUE_DOCUMENT_BATCH_MAX_BYTES });
return;
}
parts.push(part);
}
if (includeBody) {
await options.recordReads(req, readable.filter(issue => byIssueId.has(issue.id)));
}
res.type("json").send(parts.join("") + "]}");
} finally { settle(); }
});
}
+15 -2
View File
@@ -18,6 +18,7 @@ import { parseObject } from "../adapters/utils.js";
import { prepareRestrictedWorkspaceRetry } from "../services/restricted-workspace-retry.js";
import { ISSUE_SUMMARIES_MAX_IDS, issueChatBootstrapQuerySchema, issueSummariesQuerySchema, type IssueSummary } from "@paperclipai/shared";
import { readIssueChatBootstrapItems } from "../services/issue-chat-bootstrap.js";
import { registerIssueDocumentBatchRoute } from "./issue-document-batch.js";
import { issueRunPresenceService } from "../services/issue-run-presence.js";
import { activityService } from "../services/activity.js";
import { listIssueNavigationState, listIssueSummaries } from "../services/issue-summaries.js";
@@ -25,8 +26,8 @@ import { listIssueStatusBlockerAttention } from "../services/issue-status-presen
import { assertNoCaseHandoffMutation } from "../services/heartbeat/case-handoff-lifecycle.js";
import { assertHandedOffRunTaskOwnership } from "../run-task-ownership.js";
import { recordNativeHumanSelfHandoff } from "../services/native-runtime/native-human-self-handoff.js";
import { recordChildTaskRead } from "../services/native-runtime/native-child-completion-delivery.js";
import { recordCompletedHandoffRead } from "../services/chat-completion-delivery.js";
import { recordChildTaskRead, recordChildTaskReads } from "../services/native-runtime/native-child-completion-delivery.js";
import { recordCompletedHandoffRead, recordCompletedHandoffReads } from "../services/chat-completion-delivery.js";
import { recordDeclinedInteractionRead } from "../services/chat-interaction-outcome-report.js";
import { runRetainsIssueIdentity } from "../services/task-repository-acquisition.js";
import { withConversationReadState } from "../services/agent-chat-read-state.js";
@@ -11446,6 +11447,18 @@ export function issueRoutes(
},
);
registerIssueDocumentBatchRoute(router, db, {
filterReadable: filterIssuesForActor,
recordReads: async (req, issues) => {
if (req.actor.type !== "agent" || !req.actor.agentId || !req.actor.runId) return;
const input = { companyId: req.params.companyId as string, agentId: req.actor.agentId, runId: req.actor.runId };
await recordCompletedHandoffReads(db, { ...input, tasks: issues })
.catch(error => logger.warn({ err: error, runId: input.runId }, "could not record the completed handoffs a run read"));
await recordChildTaskReads(db, { ...input, children: issues })
.catch(error => logger.warn({ err: error, runId: input.runId }, "could not record the child tasks a run read"));
},
});
router.get("/issues/:id/documents", async (req, res) => {
const id = req.params.id as string;
const issue = await getAccessibleResource(
+11 -1
View File
@@ -1,4 +1,4 @@
import { catalogTeamInstallSchema, catalogTeamPreviewSchema, updateWorkspacePreferenceSchema } from "@paperclipai/shared";
import { catalogTeamInstallSchema, catalogTeamPreviewSchema, updateWorkspacePreferenceSchema, issueDocumentBatchQuerySchema } from "@paperclipai/shared";
import { MUSE_ASSET_VERSION, museStopBoundarySchema, museDetectorCleanupAttestationSchema, createGrokInvitationSchema, configureGrokRoutineSchema, verifyGrokRoutineSchema, recoverGrokRoutineSchema, pairGrokBindingSchema, stopGrokWorkSchema, retryGrokDeliverySchema, updateFastResponseSchema } from "@paperclipai/shared";
import { slackTestMessageSchema, slackTestMessageResultSchema, slackManagerAuthorizationSchema, slackSetupOptionsSchema, slackManagedProvisionSchema, slackManagedResultSchema, slackRegistrationSchema, slackSetupActionSchema, slackInstallAuthorizationSchema, slackRegistrationStateSchema, slackAppConfigurationSchema, slackAvatarStateSchema, slackAccountStateSchema } from "@paperclipai/shared";
import { experimentalApiMetadata } from "./experimental-api-metadata.js";
@@ -4800,6 +4800,16 @@ registry.registerPath({
responses: { 200: r.ok(), 401: r.unauthorized },
});
registry.registerPath({
method: "get",
path: "/api/companies/{companyId}/issue-documents",
tags: ["issues"],
summary: "Bulk read one document key across issue IDs",
description: "Read current documents for 1–200 comma-separated task UUIDs in one request. Company and task read permissions apply. Unreadable, hidden, foreign and nonexistent tasks are omitted; a readable task without this key returns document:null. Items follow first-requested ID order with duplicates removed. includeBody defaults to true; includeBody=false returns revision metadata and updatedAt without loading bodies, for cache validation. No break-glass access. Responses over 8 MiB fail with 413; split the IDs into smaller requests. Native helpers can use save_api_response then download_artifact to relay the complete batch without per-task calls. Verify every requested ID is present before treating the inventory as complete.",
request: { params: z.object({ companyId: z.string().uuid() }), query: issueDocumentBatchQuerySchema },
responses: { 200: r.ok(), 400: r.badRequest, 401: r.unauthorized, 403: r.forbidden, 413: { description: "Split the document batch into smaller requests" }, 429: { description: "At most two active reads per actor and eight per server instance; retry after an active read finishes", headers: { "Retry-After": { schema: { type: "string", enum: ["1"] } } } } },
});
registry.registerPath({
method: "get",
path: "/api/issues/{id}/documents",
@@ -5,7 +5,7 @@ import { and, asc, desc, eq, gte, inArray, isNotNull, isNull, lte, ne, sql } fro
import { agents, agentWakeupRequests, chatCompletionDeliveries as deliveries, chatTaskHandoffs as handoffs,
heartbeatRunEvents, heartbeatRuns, issueComments, issueDocuments, issueThreadInteractions, issues, type Db } from "@paperclipai/db";
import { extractIssueReferenceMatches } from "@paperclipai/shared";
import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
import { appendHeartbeatRunEventBatch } from "./heartbeat-run-events.js";
import { conflict, forbidden, notFound } from "../errors.js";
export const CHAT_COMPLETION_WAKE_REASON = "chat_task_completed";
@@ -148,13 +148,24 @@ export async function recordCompletedHandoffRead(db: Db, input: {
companyId: string; agentId: string; runId: string;
task: { id: string; status: string; statusVersion?: number | null };
}) {
if (input.task.status !== "done" || typeof input.task.statusVersion !== "number") return;
const [handoff] = await db.select({ agentId: handoffs.agentId }).from(handoffs)
.where(and(eq(handoffs.taskId, input.task.id), eq(handoffs.companyId, input.companyId)));
if (handoff?.agentId !== input.agentId) return;
await appendHeartbeatRunEvent(db, { companyId: input.companyId, runId: input.runId, agentId: input.agentId,
await recordCompletedHandoffReads(db, { ...input, tasks: [input.task] });
}
/** One lookup and one event append statement for a batch; payloads stay compatible with single reads. */
export async function recordCompletedHandoffReads(db: Db, input: {
companyId: string; agentId: string; runId: string;
tasks: ReadonlyArray<{ id: string; status: string; statusVersion?: number | null }>;
}) {
const completed = input.tasks.filter(task => task.status === "done" && typeof task.statusVersion === "number");
if (!completed.length) return;
const rows = await db.select({ taskId: handoffs.taskId }).from(handoffs)
.where(and(inArray(handoffs.taskId, completed.map(task => task.id)), eq(handoffs.companyId, input.companyId), eq(handoffs.agentId, input.agentId)));
const matching = new Set(rows.map(row => row.taskId));
await appendHeartbeatRunEventBatch(db, completed.filter(task => matching.has(task.id)).map(task => ({
companyId: input.companyId, runId: input.runId, agentId: input.agentId,
eventType: COMPLETED_HANDOFF_READ_EVENT, stream: "system", level: "info",
payload: { taskId: input.task.id, statusVersion: input.task.statusVersion } });
payload: { taskId: task.id, statusVersion: task.statusVersion },
})));
}
/** A reply in this conversation already reported this completion when a turn of the same
+34 -3
View File
@@ -2,15 +2,18 @@ import { notifyStatusCardIssueChange } from "./status-card-notifications.js";
import { assertAgentInteraction } from "./agent-interaction-policy.js";
import { notifyIssueRecovery, notifyIssueGraphRecovery } from "./issue-recovery-notifications.js";
import { randomUUID } from "node:crypto";
import { and, asc, desc, eq, sql } from "drizzle-orm";
import { and, asc, desc, eq, inArray, sql } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import { documentRevisions, documents, issueDocuments, issues } from "@paperclipai/db";
import { isSystemIssueDocumentKey, issueDocumentKeySchema } from "@paperclipai/shared";
import { conflict, notFound, unprocessable } from "../errors.js";
import { ISSUE_DOCUMENT_BATCH_MAX_IDS, isSystemIssueDocumentKey, issueDocumentKeySchema } from "@paperclipai/shared";
import { conflict, notFound, payloadTooLarge, unprocessable } from "../errors.js";
import { isUniqueViolation } from "../db-errors.js";
import { insertRowsInChunks } from "./batch-insert.js";
import type { ImportIssueDocumentRow } from "./import-write-types.js";
// Stay below download_artifact's 10 MiB limit, including JSON escaping and metadata.
export const ISSUE_DOCUMENT_BATCH_MAX_BYTES = 8 * 1024 * 1024;
function normalizeDocumentKey(key: string) {
const normalized = key.trim().toLowerCase();
const parsed = issueDocumentKeySchema.safeParse(normalized);
@@ -172,6 +175,34 @@ export function documentService(db: Db) {
return row ? mapIssueDocumentRow(row, true) : null;
},
/** Caller authorizes the task set first. One query reads only the requested key. */
getIssueDocumentsByKey: async (companyId: string, issueIds: string[], rawKey: string, includeBody = true) => {
const key = normalizeDocumentKey(rawKey);
if (issueIds.length > ISSUE_DOCUMENT_BATCH_MAX_IDS) throw unprocessable("Too many issue document IDs");
if (!issueIds.length) return [];
const join = and(eq(issueDocuments.documentId, documents.id), eq(documents.companyId, companyId));
const condition = and(eq(issueDocuments.companyId, companyId), inArray(issueDocuments.issueId, issueIds), eq(issueDocuments.key, key));
const read = async (connection: Db) => {
if (includeBody) {
// PostgreSQL counts encoded JSON one body at a time; no bodies cross
// into the app before the entire set fits. The same snapshot below
// prevents a concurrent edit from growing the admitted response.
const [size] = await connection.select({ bytes: sql<string>`coalesce(sum(octet_length(to_json(${documents.latestBody})::text)), 0)` })
.from(issueDocuments).innerJoin(documents, join).where(condition);
if (Number(size.bytes) > ISSUE_DOCUMENT_BATCH_MAX_BYTES) {
throw payloadTooLarge("Document batch exceeds 8 MiB; split issueIds into smaller requests or use includeBody=false", { maxResponseBytes: ISSUE_DOCUMENT_BATCH_MAX_BYTES });
}
}
const rows = await connection
.select({ ...issueDocumentSelect, latestBody: includeBody ? documents.latestBody : sql<string>`''` })
.from(issueDocuments).innerJoin(documents, join).where(condition);
return rows.map(row => mapIssueDocumentRow(row, includeBody));
};
return includeBody
? db.transaction(tx => read(tx as unknown as Db), { isolationLevel: "repeatable read", accessMode: "read only" })
: read(db);
},
/** Bounded receipt read: never load every historical body for one revision. */
getIssueDocumentRevision: async (issueId: string, rawKey: string, revisionId: string) => {
const key = normalizeDocumentKey(rawKey);
+35 -2
View File
@@ -180,7 +180,9 @@ async function appendRunEvent(
}
/**
* Appends native source events of one run and one source atomically, in the
* Appends ordinary non-waking events of one run in one statement, preserving
* each event's existing payload and sequence. Special writers keep the single
* append path. Also appends native source events of one run and one source in the
* given order, and returns one receipt per event. A stream that continues its
* source's stored cursor with new events takes one statement for the whole
* batch. Any other batch (a replay, a gap, an event that registers a wake, a
@@ -205,7 +207,38 @@ async function appendRunEventBatch(
): Promise<AppendHeartbeatRunEventResult[]> {
const first = inputs[0]!;
const source = first.nativeSource;
if (!source || inputs.some(input => !input.nativeSource || input.runUpdate || input.retryExhaustion
if (!source) {
if (inputs.some(input => input.nativeSource || input.runUpdate || input.retryExhaustion || input.assertWritable
|| input.companyId !== first.companyId || input.runId !== first.runId || input.agentId !== first.agentId
|| Object.values(runEventWakes(input)).some(Boolean))) throw new Error("heartbeat_run_event_batch_unsupported");
assertWritable?.();
const rows = JSON.stringify(inputs.map(input => ({
eventType: input.eventType, stream: input.stream ?? null, level: input.level ?? null, color: input.color ?? null,
message: input.message ?? null, payload: input.payload ? heartbeatRunEvents.payload.mapToDriverValue(input.payload) : null,
})));
const inserted = await db.execute<Record<string, unknown>>(sql`
with allocated_run_events as (
update heartbeat_runs
set next_event_seq = next_event_seq + ${inputs.length}::integer,
updated_at = ${sql.param(new Date(), heartbeatRuns.updatedAt)}
where id = ${first.runId} and company_id = ${first.companyId} and agent_id = ${first.agentId}
returning company_id, id as run_id, agent_id, next_event_seq - ${inputs.length}::integer as first_seq
)
insert into heartbeat_run_events
(company_id, run_id, agent_id, seq, event_type, stream, level, color, message, payload)
select allocated.company_id, allocated.run_id, allocated.agent_id, allocated.first_seq + event.ordinal - 1,
event.value ->> 'eventType', event.value ->> 'stream', event.value ->> 'level', event.value ->> 'color',
event.value ->> 'message', (event.value ->> 'payload')::jsonb
from allocated_run_events allocated cross join jsonb_array_elements(${rows}::jsonb) with ordinality as event(value, ordinal)
order by event.ordinal
returning *
`);
if (inserted.length !== inputs.length) throw new Error("heartbeat_run_event_binding_mismatch");
assertWritable?.();
return inserted.map(decodeRunEventRow).sort((a, b) => a.seq - b.seq)
.map(row => ({ row, disposition: "committed" as const, highestContiguousSourceSeq: 0 }));
}
if (inputs.some(input => !input.nativeSource || input.runUpdate || input.retryExhaustion
|| input.companyId !== first.companyId || input.runId !== first.runId || input.agentId !== first.agentId
|| input.nativeSource.sourceInstanceId !== source.sourceInstanceId)) {
throw new Error("heartbeat_run_event_batch_unsupported");
@@ -0,0 +1,194 @@
import { createHash, randomUUID } from "node:crypto";
import { readFile } from "node:fs/promises";
import { join } from "node:path";
import { eq, sql } from "drizzle-orm";
import { drizzle } from "drizzle-orm/postgres-js";
import { agentApiKeys, assets, chatTaskHandoffs, documents, heartbeatRunEvents, issueDocuments, issues } from "@paperclipai/db";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { startRunnerApiTestServer } from "../../__tests__/helpers/runner-api-server.js";
import { documentService } from "../documents.js";
import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js";
const operationId = "GET /api/companies/{companyId}/issue-documents";
describe("bulk issue documents through native API tools", () => {
let server: Awaited<ReturnType<typeof startRunnerApiTestServer>>;
const oldSecret = process.env.PAPERCLIP_AGENT_JWT_SECRET;
beforeAll(async () => {
process.env.PAPERCLIP_AGENT_JWT_SECRET = randomUUID();
server = await startRunnerApiTestServer();
}, 60000);
afterAll(async () => {
await server?.close();
if (oldSecret === undefined) delete process.env.PAPERCLIP_AGENT_JWT_SECRET;
else process.env.PAPERCLIP_AGENT_JWT_SECRET = oldSecret;
});
async function saveCases(completed = false) {
const f = await server.fixture({ conversation: completed });
const ids = Array.from({ length: 146 }, () => randomUUID());
await server.db.insert(issues).values(ids.map(id => ({ id, companyId: f.companyId, title: "Sync case", status: completed ? "done" : "todo", ...(completed ? { parentId: f.issueId } : {}) })));
if (completed) await server.db.insert(chatTaskHandoffs).values(ids.map(taskId => ({
taskId, companyId: f.companyId, conversationId: f.issueId, agentId: f.agentId, sessionGeneration: 0,
})));
const rows = await server.db.insert(documents).values(ids.map((id, index) => ({
companyId: f.companyId, format: "markdown", title: "Case", latestBody: `Case ${index}: é🧭\n\"quoted\" \\escaped`,
}))).returning();
await server.db.insert(issueDocuments).values(rows.map((row, index) => ({ companyId: f.companyId, issueId: ids[index], documentId: row.id, key: "sentry-case" })));
const authority = new PaperclipRunnerToolAuthority(server.db, { ...f, workspaceRoot: f.workspace });
const execute = (tool: string, args: object) => authority.execute({ tool, callId: randomUUID(), arguments: args }) as Promise<any>;
const found = await execute("search_api", { query: "bulk issue documents by key", limit: 10 });
expect(JSON.stringify(found)).toContain(operationId);
const queries: string[] = [];
const previousDebug = server.db.$client.options.debug;
server.db.$client.options.debug = (_connection, query) => { queries.push(query); };
let saved: any;
try {
saved = await execute("save_api_response", { operationId, pathParams: { companyId: f.companyId }, query: { key: "sentry-case", issueIds: ids.join(",") } });
} finally { server.db.$client.options.debug = previousDebug; }
expect(saved).toMatchObject({ ok: true, status: 200, artifact: { byteSize: expect.any(Number), sha256: expect.any(String) } });
const [asset] = await server.db.select().from(assets).where(eq(assets.id, saved.artifact.artifactId));
const object = await server.storage.getObject(f.companyId, asset.objectKey);
const chunks: Buffer[] = [];
for await (const chunk of object.stream) chunks.push(Buffer.from(chunk));
return { f, ids, rows, saved, execute, queries, bytes: Buffer.concat(chunks) };
}
it("saves 146 complete documents in one lossless response artifact", async () => {
const { ids, rows, saved, bytes } = await saveCases();
expect(bytes.length).toBe(saved.artifact.byteSize);
expect(createHash("sha256").update(bytes).digest("hex")).toBe(saved.artifact.sha256);
const result = JSON.parse(bytes.toString());
expect(result.key).toBe("sentry-case");
expect(result.items.map((item: any) => item.issueId)).toEqual(ids);
expect(result.items.map((item: any) => item.document.body)).toEqual(rows.map(row => row.latestBody));
for (const item of result.items) expect(item.document).toMatchObject({ issueId: item.issueId, key: "sentry-case", latestRevisionNumber: expect.any(Number), updatedAt: expect.any(String) });
});
it.skipIf(process.platform !== "linux")("downloads all 146 documents in one native transfer with the saved hash and size", async () => {
const { f, saved, execute, bytes } = await saveCases();
await execute("download_artifact", { artifactId: saved.artifact.artifactId, byteSize: saved.artifact.byteSize, sha256: saved.artifact.sha256, destination: "cases.json" });
expect(await readFile(join(f.workspace, "cases.json"))).toEqual(bytes);
});
it("reads 146 completed cases with one handoff lookup and batched receipt writes", async () => {
const { f, ids, queries } = await saveCases(true);
expect(queries.filter(query => /from "chat_task_handoffs"/.test(query))).toHaveLength(1);
const writes = queries.filter(query => /insert into heartbeat_run_events/.test(query));
expect(writes).toHaveLength(2);
const events = await server.db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.runId));
const handoffs = events.filter(event => event.eventType === "chat_handoff.completed_read");
expect(new Set(handoffs.map(event => event.payload!.taskId))).toEqual(new Set(ids));
const children = events.filter(event => event.eventType === "child_task.read");
expect(children).toHaveLength(1);
expect(new Set(Object.keys(children[0].payload!.children as object))).toEqual(new Set(ids));
expect(new Set(events.map(event => event.seq)).size).toBe(events.length);
});
it("returns current revision metadata without bodies and identifies readable tasks with no document", async () => {
const f = await server.fixture();
const service = documentService(server.db);
const first = await service.getIssueDocumentByKey(f.issueId, "notes");
await service.upsertIssueDocument({ issueId: f.issueId, key: "notes", format: "markdown", body: "Updated content", baseRevisionId: first!.latestRevisionId });
const current = await service.getIssueDocumentByKey(f.issueId, "notes");
const receipt = await f.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId, pathParams: { companyId: f.companyId }, query: { key: " NOTES ", issueIds: [f.blockerId, f.issueId, f.issueId].join(","), includeBody: false },
} }) as any;
expect(receipt).toMatchObject({ ok: true, data: { key: "notes", items: [
{ issueId: f.blockerId, document: null },
{ issueId: f.issueId, document: { latestRevisionId: current!.latestRevisionId, latestRevisionNumber: 2 } },
] } });
expect(receipt.data.items[1].document).not.toHaveProperty("body");
});
it("omits private, hidden, foreign and missing tasks without disclosing which exist", async () => {
const f = await server.fixture();
const foreign = await server.fixture();
const privateId = randomUUID(), hiddenId = randomUUID(), missingId = randomUUID();
await server.db.insert(issues).values([
{ id: privateId, companyId: f.companyId, title: "Private", visibility: "private", status: "todo" },
{ id: hiddenId, companyId: f.companyId, title: "Hidden", hiddenAt: new Date(), status: "todo" },
]);
for (const id of [privateId, hiddenId]) await documentService(server.db).upsertIssueDocument({ issueId: id, key: "notes", format: "markdown", body: "Never leak" });
const result = await f.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId, pathParams: { companyId: f.companyId }, query: { key: "notes", issueIds: [f.issueId, privateId, hiddenId, foreign.issueId, missingId].join(",") },
} }) as any;
expect(result).toMatchObject({ ok: true, data: { items: [{ issueId: f.issueId, document: { body: "Document verification code: silver-wren." } }] } });
expect(JSON.stringify(result)).not.toContain("Never leak");
await expect(f.authority.execute({ tool: "save_api_response", callId: randomUUID(), arguments: {
operationId, pathParams: { companyId: foreign.companyId }, query: { key: "notes", issueIds: foreign.issueId },
} })).rejects.toMatchObject({ status: 403 });
const anonymous = await fetch(`${server.apiUrl}/api/companies/${f.companyId}/issue-documents?key=notes&issueIds=${f.issueId}`);
expect(anonymous.status).toBe(401);
});
it("enforces company boundaries and denies task bridge keys on the HTTP route", async () => {
const f = await server.fixture({ connectionScenario: "fresh", disableWakeOnDemand: true });
const token = `pcp_fixture_${randomUUID()}`;
const [key] = await server.db.insert(agentApiKeys).values({
companyId: f.companyId, agentId: f.agentId, name: "Batch fixture",
keyHash: createHash("sha256").update(token).digest("hex"), responsibleUserId: f.responsibleUserId,
}).returning();
const get = (companyId: string) => fetch(`${server.apiUrl}/api/companies/${companyId}/issue-documents?key=notes&issueIds=${f.issueId}`, { headers: { Authorization: `Bearer ${token}` } });
expect((await get(f.companyId)).status).toBe(200);
expect((await get(f.foreignCompanyId)).status).toBe(403);
await server.db.update(agentApiKeys).set({ scopeConfig: { kind: "task_bridge", projectId: f.projectId } }).where(eq(agentApiKeys.id, key.id));
expect((await get(f.companyId)).status).toBe(403);
});
it.each([
{}, { key: "notes" }, { key: "notes", issueIds: "" }, { key: "notes", issueIds: "not-a-uuid" },
{ key: "notes", issueIds: Array.from({ length: 201 }, () => randomUUID()).join(",") },
{ key: "../notes", issueIds: randomUUID() }, { key: "notes", issueIds: randomUUID(), includeBody: "yes" },
{ key: "notes", issueIds: randomUUID(), breakGlass: true },
])("rejects invalid or unbounded query %#", async query => {
const f = await server.fixture();
const result = await f.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: { operationId, pathParams: { companyId: f.companyId }, query } }) as any;
expect(result).toMatchObject({ ok: false, status: 400 });
});
it("rejects an oversized batch without saving a partial artifact, while metadata remains readable", async () => {
const f = await server.fixture();
const document = await documentService(server.db).getIssueDocumentByKey(f.issueId, "notes");
// Escaping expands these bodies in JSON. The cap must use wire bytes.
await server.db.update(documents).set({ latestBody: "\"".repeat(4 * 1024 * 1024) }).where(eq(documents.id, document!.id));
const query = { key: "notes", issueIds: f.issueId };
const result = await f.authority.execute({ tool: "save_api_response", callId: randomUUID(), arguments: { operationId, pathParams: { companyId: f.companyId }, query } }) as any;
expect(result).toMatchObject({ ok: false, status: 413 });
expect(result).not.toHaveProperty("artifact");
const metadata = await f.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: { operationId, pathParams: { companyId: f.companyId }, query: { ...query, includeBody: false } } }) as any;
expect(metadata).toMatchObject({ ok: true, data: { items: [{ issueId: f.issueId, document: { latestRevisionId: document!.latestRevisionId } }] } });
expect(metadata.data.items[0].document).not.toHaveProperty("body");
});
it("rejects many large documents using a size-only query before loading any bodies", async () => {
const f = await server.fixture();
const ids = Array.from({ length: 200 }, () => randomUUID());
await server.db.insert(issues).values(ids.map(id => ({ id, companyId: f.companyId, title: "Large case", status: "todo" })));
const rows = await server.db.insert(documents).values(ids.map(() => ({ companyId: f.companyId, latestBody: sql<string>`repeat('x', 524288)` }))).returning({ id: documents.id });
await server.db.insert(issueDocuments).values(rows.map((row, index) => ({ companyId: f.companyId, issueId: ids[index], documentId: row.id, key: "sentry-case" })));
const queries: string[] = [];
const measured = drizzle(server.db.$client, { schema: server.db._.fullSchema, logger: { logQuery: query => queries.push(query) } }) as typeof server.db;
let status: number | undefined;
try { await documentService(measured).getIssueDocumentsByKey(f.companyId, ids, "sentry-case"); }
catch (error) { status = (error as { status: number }).status; }
expect(status).toBe(413);
const reads = queries.filter(query => /^select /i.test(query));
expect(reads).toHaveLength(1);
expect(reads[0]).toContain("octet_length");
expect(reads[0].split(" from ")[0]).not.toContain('"documents"."id"');
const result = await f.authority.execute({ tool: "save_api_response", callId: randomUUID(), arguments: {
operationId, pathParams: { companyId: f.companyId }, query: { key: "sentry-case", issueIds: ids.join(",") },
} }) as any;
expect(result).toMatchObject({ ok: false, status: 413 });
expect(result).not.toHaveProperty("artifact");
// Bodies alone fit, but their metadata pushes the response over budget.
// The incremental JSON builder must refuse this before joining a response.
await server.db.update(documents).set({ latestBody: "x".repeat(524000) }).where(eq(documents.companyId, f.companyId));
const nearLimit = await f.authority.execute({ tool: "save_api_response", callId: randomUUID(), arguments: {
operationId, pathParams: { companyId: f.companyId }, query: { key: "sentry-case", issueIds: ids.slice(0, 16).join(",") },
} }) as any;
expect(nearLimit).toMatchObject({ ok: false, status: 413 });
expect(nearLimit).not.toHaveProperty("artifact");
});
});
@@ -1,6 +1,6 @@
import { and, eq, inArray, isNull, ne, or, sql } from "drizzle-orm";
import { agentWakeupRequests, heartbeatRunEvents, heartbeatRuns, statusDecisions, type Db } from "@paperclipai/db";
import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js";
import { appendHeartbeatRunEventBatch } from "../heartbeat-run-events.js";
export interface NativeChildCompletionRecipient {
companyId: string;
@@ -17,12 +17,25 @@ export async function recordChildTaskRead(db: Db, input: {
children: ReadonlyArray<{ id: string; parentId: string | null; status: string; updatedAt: Date; lastStatusDecisionId?: string | null }>;
}): Promise<void> {
const completed = input.children.filter(child => child.parentId === input.parentIssueId && child.status === "done");
if (completed.length === 0) return;
await appendHeartbeatRunEvent(db, { companyId: input.companyId, runId: input.runId, agentId: input.agentId,
eventType: CHILD_TASK_READ_EVENT, stream: "system", level: "info",
payload: { parentIssueId: input.parentIssueId,
children: Object.fromEntries(completed.map(child => [child.id, { status: child.status, updatedAt: child.updatedAt.toISOString(),
statusDecisionId: child.lastStatusDecisionId ?? null }])) } });
await recordChildTaskReads(db, { ...input, children: completed });
}
/** Group completed children by parent, then persist the existing receipt shape in one statement. */
export async function recordChildTaskReads(db: Db, input: {
companyId: string; agentId: string; runId: string;
children: ReadonlyArray<{ id: string; parentId: string | null; status: string; updatedAt: Date; lastStatusDecisionId?: string | null }>;
}): Promise<void> {
const groups = new Map<string, Record<string, unknown>>();
for (const child of input.children) {
if (!child.parentId || child.status !== "done") continue;
const children = groups.get(child.parentId) ?? {};
children[child.id] = { status: child.status, updatedAt: child.updatedAt.toISOString(), statusDecisionId: child.lastStatusDecisionId ?? null };
groups.set(child.parentId, children);
}
await appendHeartbeatRunEventBatch(db, [...groups].map(([parentIssueId, children]) => ({
companyId: input.companyId, runId: input.runId, agentId: input.agentId,
eventType: CHILD_TASK_READ_EVENT, stream: "system", level: "info", payload: { parentIssueId, children },
})));
}
/** New child results belong to a fresh parent turn, including across a Done claim. */
@@ -229,7 +229,7 @@
"packages/shared/src/connector-availability.ts": "82aec0dfb2e9b8cb04e21c4bcd9aab74b087c76a",
"packages/shared/src/connector-identity.test.ts": "d58357707faefa7c98bb906cdaeb4db39d31081a",
"packages/shared/src/connector-identity.ts": "5ede83c9c5225dbe8af9cf3a49c74e477ef5b955",
"packages/shared/src/constants.ts": "9762160217907b1a3cc33f3dc9f86b46afae2cdd",
"packages/shared/src/constants.ts": "049c9b0cfd2f222aa361bbf07a75d6150760acdc",
"packages/shared/src/crash-point-registry.ts": "3cbf2d290bc111061376b43020848623e493737c",
"packages/shared/src/crash-points.test.ts": "3564a1f62d22f54a3c2fb0a72b3a6cde17c957d2",
"packages/shared/src/crash-points.ts": "c15259f39610e352eac5423a491548ec1bae94fe",
@@ -268,7 +268,7 @@
"packages/shared/src/home-paths.ts": "fa4f1c440653b76b3dbb7a51026d557287deb45d",
"packages/shared/src/humanize-connection.test.ts": "ba9e90a44c63452b0bf9f733adb90241921e8eca",
"packages/shared/src/humanize-connection.ts": "fbd414476a0ef7106a7c80d8572208781b091699",
"packages/shared/src/index.ts": "bc4373e3956c06613f3214706824a93050a757cf",
"packages/shared/src/index.ts": "414cc5429e2566c9f5cdc24379b05a5916385a3e",
"packages/shared/src/issue-activity-identity.test.ts": "3dc294640f54c2be67943e3e0149cf4eff2c95ca",
"packages/shared/src/issue-activity-identity.ts": "1b0f07ec3fe0506aaaf80a4fa9893f3baf8f0075",
"packages/shared/src/issue-attribution.test.ts": "84563685e7f3c48029243cf1cc24a3a09ee3cde9",
@@ -412,10 +412,10 @@
"packages/shared/src/types/inbox-agent-policy.ts": "cb9d41d9681e80b834889268ad96015a39df92f6",
"packages/shared/src/types/inbox-dismissal.ts": "58fb7013847396ac9971eb5799dd0bf0486e0371",
"packages/shared/src/types/inbox-mine.ts": "1d302b531d34842189704abd2f75f8743d09490d",
"packages/shared/src/types/index.ts": "e753d49a5cb6ae09a4b2d62bc7f832012b499c08",
"packages/shared/src/types/index.ts": "708503f0fe1ad9ed807ccb70cab4d77f3eac6cf2",
"packages/shared/src/types/instance.ts": "d47ae3080fbf096e71c51e707efe7e827a6562c7",
"packages/shared/src/types/issue-tree-control.ts": "75489fffd491b8350b05caf57d7796419797d9df",
"packages/shared/src/types/issue.ts": "d594c0037a8d9882dc35d3ecd7cb8c276d7e2831",
"packages/shared/src/types/issue.ts": "27ae255864444760df489d1fc2df081ccd214889",
"packages/shared/src/types/live.ts": "b90e55e4de8600c6fc17d7fa8d89079d4ae655ec",
"packages/shared/src/types/native-finalization.ts": "c026a73003ae232fea4f7624d561493b383df358",
"packages/shared/src/types/plugin.ts": "af85e6aa845c2ef6956d0151ff11332947bf3949",
@@ -489,13 +489,13 @@
"packages/shared/src/validators/folder.ts": "b486444ab09ec3d5a9afec7dbd2e71c1bc2beb67",
"packages/shared/src/validators/goal.ts": "602724134915e1e83c50115abba36b288c0abd3d",
"packages/shared/src/validators/inbox-agent-policy.ts": "29cef157f4d9b564fd813868038aa8c813df4a7b",
"packages/shared/src/validators/index.ts": "c8cbf1aea7577e7f7a07b4ca733dabb9b23b0458",
"packages/shared/src/validators/index.ts": "450d622613cece45c4e5cc968a7d8e68c7e903dc",
"packages/shared/src/validators/instance.test.ts": "3faa9593a9a05f74429d9b1d2251906a03e9a903",
"packages/shared/src/validators/instance.ts": "40c23e4579a1a777d77588a3058db03b46bccb14",
"packages/shared/src/validators/issue-chat-bootstrap.test.ts": "e1ebb9aa94da22c825bc3ddd32ce2cfc98bfe645",
"packages/shared/src/validators/issue-tree-control.ts": "7144ef86a83e008ac0626e1d62a1d6164c46f438",
"packages/shared/src/validators/issue.test.ts": "9a10bf1112d7c85f80cc1fd5db9920dda8974643",
"packages/shared/src/validators/issue.ts": "9877f071ad93952070507a79aec91fbf312d9a84",
"packages/shared/src/validators/issue.test.ts": "c66108342e62370064b9f3e92652a71c71528012",
"packages/shared/src/validators/issue.ts": "1b016c9453f5276eb61526de2b27d4841e88a008",
"packages/shared/src/validators/json-schema.ts": "b6cba6a3f76b49afce1c7dc1c2f086ab946c6133",
"packages/shared/src/validators/native-finalization.test.ts": "40c68bdd1d0e8a77cd70c4576fae53334fe5be0d",
"packages/shared/src/validators/native-finalization.ts": "d4e674de6ac7d36385e6f6c2172fa16ffc9b1830",