mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-19 16:29:42 +01:00
Add bulk keyed issue document reads for native helpers (#22105)
## Thinking Path
> - Paperclip manages work for AI agents.
> - Native helpers read task documents through authorized API tools.
> - A scan of 146 cases needs one saved response and one download per
case.
> - The document service can read the same key across authorized tasks
with bounded queries.
> - This pull request adds a bounded bulk read with the existing task
access policy.
> - Helpers can transfer one batch and check current revisions before
fetching changed bodies.
## Linked Issues or Issue Description
**Subsystem affected**
Task document REST API, shared query contracts, and native API
discovery.
**Problem or motivation**
Per-task document reads make native sync scans cost hundreds of tool
calls. Task list timestamps do not prove that document content is
unchanged.
**Proposed solution**
Add `GET
/api/companies/{companyId}/issue-documents?key=notes&issueIds=...`. Read
up to 200 task UUIDs under current company and task permissions. Return
current bodies and revision metadata, or metadata alone with
`includeBody=false`.
**Alternatives considered**
Generic request batching would add tool execution and partial failure
rules. One bulk resource read removes the per-case relay cost with the
existing tools. Do not skip documents based only on task-row timestamps.
Keep managed skill files immutable; cleanup of copied skill test
fixtures belongs to those tests.
**Roadmap alignment**
This improves existing task documents and native API access. It adds no
new workflow or UI. The roadmap has no duplicate bulk document read
work. No duplicate open pull request was found.
## What Changed
- Add a bulk keyed document route and a company-scoped document reader.
Reuse the current batched task-read policy and completed-task read
receipts.
- Preserve request order and deduplicate IDs. Return `document: null`
for readable tasks without the key. Omit unreadable, hidden, foreign and
missing tasks without revealing which exist.
- Bound requests to 200 IDs and responses to 8 MiB of JSON bytes. Check
encoded body sizes in SQL before fetching bodies in the same read
snapshot. Build JSON within the byte budget. Oversized responses fail
with 413 and are never saved as partial artifacts.
- Admit at most two active reads per actor and eight per server
instance. Extra requests receive 429 with Retry-After. Keep the slot
until both the database work and response settle.
- Batch completed-handoff lookups and group child receipts by parent.
Extend the existing event batch writer for ordinary non-waking receipts
with atomic sequence allocation. Stored receipt shapes stay unchanged
for existing consumers.
- Expose revision-only reads without selecting document bodies. Document
native discovery, one save plus one download, and complete-inventory
checks.
- Add real HTTP, database and artifact tests for 146 open and completed
cases, 200 maximum-size bodies, metadata overhead, access failures,
query limits and JSON expansion. Cover admission limits, failure
cleanup, and concurrent event writers. Refresh the shared SDK source
pins.
## Verification
Final revision: `93dc8b029600211d79a050650f3dff82e5dad52f`.
- [CI run 44212220917, attempt
3](https://github.com/paperclipai/paperclip/actions/runs/44212220917)
passed. This includes typecheck, build, general tests, Runner checks,
serialized server suites, all e2e shards, and the final verify gate.
-
[Build](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134506305267),
[e2e](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134506309117),
and
[verify](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134510062263)
are green.
- [Linux CI job
134506309291](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134506309291)
passed all 16 bulk document integration tests. This includes the native
transfer of all 146 documents with the saved size and hash,
completed-case receipt batching, and size-only rejection of 200
maximum-size documents.
- [Greptile
review](https://github.com/paperclipai/paperclip/pull/22105#issuecomment-6965449032)
is 5/5 on the final revision. All review threads are resolved. Security
checks pass. GitHub reports CLEAN and MERGEABLE.
- Local server typecheck passed on macOS with Node 25.6.1. The focused
run below passed 177 tests; the Linux-only transfer test was skipped
locally and then passed in CI.
```sh
pnpm --filter @paperclipai/server typecheck
cd server
pnpm exec vitest run \
src/routes/issue-document-batch.test.ts \
src/services/native-runtime/issue-document-batch.integration.test.ts \
src/__tests__/documents-service.test.ts \
src/__tests__/openapi-routes.test.ts \
src/__tests__/heartbeat-run-event-atomic-append.test.ts \
src/__tests__/chat-completion-delivery.test.ts \
src/__tests__/native-child-task-read-routes.test.ts
```
- Shared typecheck and build passed. From `packages/shared/`, `pnpm exec
vitest run src/validators/issue.test.ts` passed 61 tests before the
rebase; the batch query contract did not change in the rebase.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`
passed. Runtime and lockfile pins are unchanged.
- `pnpm verify --explain` returned exit 2 because the API docs and
shared contracts require broader CI. The successful CI run above
supplies that coverage.
Failure and recovery record:
- Both performance findings reproduced before the fixes. Scale tests now
verify a size-only query for 200 large documents and one handoff lookup
plus two receipt-write statements for 146 completed cases. Concurrent
writers retain contiguous unique sequences and decoded payloads.
- Initial local runs found missing shared exports and OpenAPI route
coverage. These are fixed. Fixtures now respect Linux-only installation,
thrown cross-company denials, and the ES2022 TypeScript target. Rebase
conflicts preserved the onboarding coverage from master.
- CI attempt 1 interrupted Build, e2e shard 7 and six general-test
shards with runner shutdown signals. Attempt 2 recovered the tests but
its [Build
job](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134499680425)
also received a shutdown signal and exited 143. No compiler or test
assertion failure was found in those interrupted job logs. The reason
the runners stopped remains unknown. Attempt 3 recovered Build and its
dependent verify gate on the same revision. Earlier interruptions are
not counted as passes.
## Risks
- Additive API only. No migration or existing endpoint response change.
- Helpers must adopt the batch response to gain the tool-call reduction.
They must verify that every requested task is present before treating
the scan as complete. Unreadable tasks are intentionally omitted.
- Split large document batches when the route returns 413. Metadata
remains available without bodies. Concurrency limits are per server
instance and can refuse simultaneous reads with 429; callers retry after
an active read finishes.
- This does not change company-owned helper inventory expiry or cleanup
tests, installed skill permissions, or generic tool batching. It does
not deploy to a tenant.
## Model Used
OpenAI `gpt-6.1-sol` through Codex in T3 Code, with xhigh reasoning,
tool use, and code execution. Context window size was not exposed.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
bf55063567
commit
ea154ce55e
20 files changed
+634
-30
No files matched your search
@@ -0,0 +1,91 @@
|
||||
import express from "express";
|
||||
import request from "supertest";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { registerIssueDocumentBatchRoute } from "./issue-document-batch.js";
|
||||
|
||||
vi.mock("../services/authorization.js", () => ({ issueReadSqlCondition: async () => undefined }));
|
||||
vi.mock("../services/documents.js", () => ({
|
||||
ISSUE_DOCUMENT_BATCH_MAX_BYTES: 8 * 1024 * 1024,
|
||||
documentService: () => ({ getIssueDocumentsByKey: async () => [] }),
|
||||
}));
|
||||
|
||||
const companyId = "11111111-1111-4111-8111-111111111111";
|
||||
const url = `/api/companies/${companyId}/issue-documents?key=notes&issueIds=22222222-2222-4222-8222-222222222222`;
|
||||
|
||||
function deferred<T>() {
|
||||
let resolve!: (value: T) => void;
|
||||
let reject!: (error: unknown) => void;
|
||||
const promise = new Promise<T>((yes, no) => { resolve = yes; reject = no; });
|
||||
return { promise, resolve, reject };
|
||||
}
|
||||
|
||||
function fixture() {
|
||||
const app = express(), router = express.Router();
|
||||
const entered: Array<() => void> = [];
|
||||
const gates: Array<ReturnType<typeof deferred<[]>>> = [];
|
||||
const pending: Array<Promise<request.Response>> = [];
|
||||
const db = { select: () => ({ from: () => ({ where: async () => [] }) }) } as unknown as Db;
|
||||
app.use((req, _res, next) => {
|
||||
req.actor = { type: "agent", companyId, agentId: req.header("actor")!, source: "agent_key" };
|
||||
next();
|
||||
});
|
||||
registerIssueDocumentBatchRoute(router, db, {
|
||||
filterReadable: async () => {
|
||||
const gate = deferred<[]>();
|
||||
gates.push(gate);
|
||||
entered.shift()!();
|
||||
return gate.promise;
|
||||
},
|
||||
recordReads: async () => undefined,
|
||||
});
|
||||
app.use("/api", router);
|
||||
app.use((_error: unknown, _req: express.Request, res: express.Response, _next: express.NextFunction) => { res.status(500).json({ error: "test read failed" }); });
|
||||
return {
|
||||
app,
|
||||
async start(actor: string) {
|
||||
const signal = deferred<void>();
|
||||
entered.push(signal.resolve);
|
||||
const response = request(app).get(url).set("actor", actor).then(value => value);
|
||||
pending.push(response);
|
||||
await signal.promise;
|
||||
return { response, gate: gates.at(-1)! };
|
||||
},
|
||||
async close() {
|
||||
for (const gate of gates) gate.resolve([]);
|
||||
await Promise.allSettled(pending);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
describe("bulk document read admission", () => {
|
||||
it("caps each actor and releases failed reads", async () => {
|
||||
const f = fixture();
|
||||
try {
|
||||
const first = await f.start("one"), second = await f.start("one");
|
||||
const refused = await request(f.app).get(url).set("actor", "one");
|
||||
expect(refused.status).toBe(429);
|
||||
expect(refused.headers["retry-after"]).toBe("1");
|
||||
first.gate.reject(new Error("read failed"));
|
||||
expect((await first.response).status).toBe(500);
|
||||
const replacement = await f.start("one");
|
||||
replacement.gate.resolve([]);
|
||||
second.gate.resolve([]);
|
||||
expect((await replacement.response).status).toBe(200);
|
||||
} finally { await f.close(); }
|
||||
});
|
||||
|
||||
it("caps total reads across actors", async () => {
|
||||
const f = fixture();
|
||||
try {
|
||||
const reads = [];
|
||||
for (let index = 0; index < 8; index++) reads.push(await f.start(`actor-${index}`));
|
||||
expect((await request(f.app).get(url).set("actor", "ninth")).status).toBe(429);
|
||||
reads[0].gate.resolve([]);
|
||||
await reads[0].response;
|
||||
const replacement = await f.start("ninth");
|
||||
replacement.gate.resolve([]);
|
||||
expect((await replacement.response).status).toBe(200);
|
||||
} finally { await f.close(); }
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,96 @@
|
||||
import type { Request, Response, Router } from "express";
|
||||
import { and, eq, inArray, isNull } from "drizzle-orm";
|
||||
import { issues, type Db } from "@paperclipai/db";
|
||||
import { issueDocumentBatchQuerySchema } from "@paperclipai/shared";
|
||||
import { issueReadSqlCondition } from "../services/authorization.js";
|
||||
import { documentService, ISSUE_DOCUMENT_BATCH_MAX_BYTES } from "../services/documents.js";
|
||||
import { assertCompanyAccess } from "./authz.js";
|
||||
|
||||
type ReadableIssue = Pick<typeof issues.$inferSelect,
|
||||
"id" | "companyId" | "projectId" | "parentId" | "assigneeAgentId" | "assigneeUserId" | "status" | "updatedAt" | "lastStatusDecisionId" | "statusVersion">;
|
||||
|
||||
const MAX_ACTOR_READS = 2;
|
||||
const MAX_SERVER_READS = 8;
|
||||
|
||||
function createBatchReadAdmission() {
|
||||
const active = new Map<string, number>();
|
||||
let total = 0;
|
||||
return (req: Request, res: Response): (() => void) | null => {
|
||||
const actor = `${req.actor.type}:${req.actor.agentId ?? req.actor.userId ?? req.actor.source}`;
|
||||
if (total >= MAX_SERVER_READS || (active.get(actor) ?? 0) >= MAX_ACTOR_READS) {
|
||||
res.setHeader("Retry-After", "1");
|
||||
res.status(429).json({ error: "Too many document batch reads; retry after an active read finishes" });
|
||||
return null;
|
||||
}
|
||||
total++;
|
||||
active.set(actor, (active.get(actor) ?? 0) + 1);
|
||||
let workDone = false, responseDone = false, released = false;
|
||||
const release = () => {
|
||||
if (released || !workDone || !responseDone) return;
|
||||
released = true;
|
||||
total--;
|
||||
const remaining = active.get(actor)! - 1;
|
||||
if (remaining) active.set(actor, remaining);
|
||||
else active.delete(actor);
|
||||
};
|
||||
const finish = () => { responseDone = true; release(); };
|
||||
res.once("finish", finish);
|
||||
res.once("close", finish);
|
||||
// A disconnect cannot free database capacity while its read is still live.
|
||||
return () => { workDone = true; release(); };
|
||||
};
|
||||
}
|
||||
|
||||
export function registerIssueDocumentBatchRoute(router: Router, db: Db, options: {
|
||||
filterReadable: (req: Request, rows: ReadableIssue[]) => Promise<ReadableIssue[]>;
|
||||
recordReads: (req: Request, issues: ReadableIssue[]) => Promise<void>;
|
||||
}) {
|
||||
const admit = createBatchReadAdmission();
|
||||
router.get("/companies/:companyId/issue-documents", async (req, res) => {
|
||||
const companyId = req.params.companyId as string;
|
||||
assertCompanyAccess(req, companyId);
|
||||
if (req.actor.type === "agent" && req.actor.source === "agent_key" && req.actor.keyScope?.kind === "task_bridge") {
|
||||
res.status(403).json({ error: "Task bridge keys cannot use company-wide issue list APIs" });
|
||||
return;
|
||||
}
|
||||
const parsed = issueDocumentBatchQuerySchema.safeParse(req.query);
|
||||
if (!parsed.success) {
|
||||
res.status(400).json({ error: "Invalid issue document batch query", details: parsed.error.issues });
|
||||
return;
|
||||
}
|
||||
const { key, issueIds, includeBody } = parsed.data;
|
||||
const settle = admit(req, res);
|
||||
if (!settle) return;
|
||||
try {
|
||||
res.setHeader("Cache-Control", "private, no-store");
|
||||
const candidates = await db.select({
|
||||
id: issues.id, companyId: issues.companyId, projectId: issues.projectId, parentId: issues.parentId,
|
||||
assigneeAgentId: issues.assigneeAgentId, assigneeUserId: issues.assigneeUserId, status: issues.status,
|
||||
updatedAt: issues.updatedAt, lastStatusDecisionId: issues.lastStatusDecisionId, statusVersion: issues.statusVersion,
|
||||
}).from(issues).where(and(
|
||||
eq(issues.companyId, companyId), inArray(issues.id, issueIds), isNull(issues.hiddenAt),
|
||||
await issueReadSqlCondition(db, req.actor),
|
||||
));
|
||||
const readable = await options.filterReadable(req, candidates);
|
||||
const byId = new Map(readable.map(issue => [issue.id, issue]));
|
||||
const docs = await documentService(db).getIssueDocumentsByKey(companyId, readable.map(issue => issue.id), key, includeBody);
|
||||
const byIssueId = new Map(docs.map(doc => [doc.issueId, doc]));
|
||||
const items = issueIds.filter(id => byId.has(id)).map(issueId => ({ issueId, document: byIssueId.get(issueId) ?? null }));
|
||||
const parts = [`{"key":${JSON.stringify(key)},"items":[`];
|
||||
let bytes = Buffer.byteLength(parts[0]) + 2; // closing array and object
|
||||
for (const [index, item] of items.entries()) {
|
||||
const part = `${index ? "," : ""}${JSON.stringify(item)}`;
|
||||
bytes += Buffer.byteLength(part);
|
||||
if (bytes > ISSUE_DOCUMENT_BATCH_MAX_BYTES) {
|
||||
res.status(413).json({ error: "Document batch exceeds 8 MiB; split issueIds into smaller requests or use includeBody=false", maxResponseBytes: ISSUE_DOCUMENT_BATCH_MAX_BYTES });
|
||||
return;
|
||||
}
|
||||
parts.push(part);
|
||||
}
|
||||
if (includeBody) {
|
||||
await options.recordReads(req, readable.filter(issue => byIssueId.has(issue.id)));
|
||||
}
|
||||
res.type("json").send(parts.join("") + "]}");
|
||||
} finally { settle(); }
|
||||
});
|
||||
}
|
||||
@@ -18,6 +18,7 @@ import { parseObject } from "../adapters/utils.js";
|
||||
import { prepareRestrictedWorkspaceRetry } from "../services/restricted-workspace-retry.js";
|
||||
import { ISSUE_SUMMARIES_MAX_IDS, issueChatBootstrapQuerySchema, issueSummariesQuerySchema, type IssueSummary } from "@paperclipai/shared";
|
||||
import { readIssueChatBootstrapItems } from "../services/issue-chat-bootstrap.js";
|
||||
import { registerIssueDocumentBatchRoute } from "./issue-document-batch.js";
|
||||
import { issueRunPresenceService } from "../services/issue-run-presence.js";
|
||||
import { activityService } from "../services/activity.js";
|
||||
import { listIssueNavigationState, listIssueSummaries } from "../services/issue-summaries.js";
|
||||
@@ -25,8 +26,8 @@ import { listIssueStatusBlockerAttention } from "../services/issue-status-presen
|
||||
import { assertNoCaseHandoffMutation } from "../services/heartbeat/case-handoff-lifecycle.js";
|
||||
import { assertHandedOffRunTaskOwnership } from "../run-task-ownership.js";
|
||||
import { recordNativeHumanSelfHandoff } from "../services/native-runtime/native-human-self-handoff.js";
|
||||
import { recordChildTaskRead } from "../services/native-runtime/native-child-completion-delivery.js";
|
||||
import { recordCompletedHandoffRead } from "../services/chat-completion-delivery.js";
|
||||
import { recordChildTaskRead, recordChildTaskReads } from "../services/native-runtime/native-child-completion-delivery.js";
|
||||
import { recordCompletedHandoffRead, recordCompletedHandoffReads } from "../services/chat-completion-delivery.js";
|
||||
import { recordDeclinedInteractionRead } from "../services/chat-interaction-outcome-report.js";
|
||||
import { runRetainsIssueIdentity } from "../services/task-repository-acquisition.js";
|
||||
import { withConversationReadState } from "../services/agent-chat-read-state.js";
|
||||
@@ -11446,6 +11447,18 @@ export function issueRoutes(
|
||||
},
|
||||
);
|
||||
|
||||
registerIssueDocumentBatchRoute(router, db, {
|
||||
filterReadable: filterIssuesForActor,
|
||||
recordReads: async (req, issues) => {
|
||||
if (req.actor.type !== "agent" || !req.actor.agentId || !req.actor.runId) return;
|
||||
const input = { companyId: req.params.companyId as string, agentId: req.actor.agentId, runId: req.actor.runId };
|
||||
await recordCompletedHandoffReads(db, { ...input, tasks: issues })
|
||||
.catch(error => logger.warn({ err: error, runId: input.runId }, "could not record the completed handoffs a run read"));
|
||||
await recordChildTaskReads(db, { ...input, children: issues })
|
||||
.catch(error => logger.warn({ err: error, runId: input.runId }, "could not record the child tasks a run read"));
|
||||
},
|
||||
});
|
||||
|
||||
router.get("/issues/:id/documents", async (req, res) => {
|
||||
const id = req.params.id as string;
|
||||
const issue = await getAccessibleResource(
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { catalogTeamInstallSchema, catalogTeamPreviewSchema, updateWorkspacePreferenceSchema } from "@paperclipai/shared";
|
||||
import { catalogTeamInstallSchema, catalogTeamPreviewSchema, updateWorkspacePreferenceSchema, issueDocumentBatchQuerySchema } from "@paperclipai/shared";
|
||||
import { MUSE_ASSET_VERSION, museStopBoundarySchema, museDetectorCleanupAttestationSchema, createGrokInvitationSchema, configureGrokRoutineSchema, verifyGrokRoutineSchema, recoverGrokRoutineSchema, pairGrokBindingSchema, stopGrokWorkSchema, retryGrokDeliverySchema, updateFastResponseSchema } from "@paperclipai/shared";
|
||||
import { slackTestMessageSchema, slackTestMessageResultSchema, slackManagerAuthorizationSchema, slackSetupOptionsSchema, slackManagedProvisionSchema, slackManagedResultSchema, slackRegistrationSchema, slackSetupActionSchema, slackInstallAuthorizationSchema, slackRegistrationStateSchema, slackAppConfigurationSchema, slackAvatarStateSchema, slackAccountStateSchema } from "@paperclipai/shared";
|
||||
import { experimentalApiMetadata } from "./experimental-api-metadata.js";
|
||||
@@ -4800,6 +4800,16 @@ registry.registerPath({
|
||||
responses: { 200: r.ok(), 401: r.unauthorized },
|
||||
});
|
||||
|
||||
registry.registerPath({
|
||||
method: "get",
|
||||
path: "/api/companies/{companyId}/issue-documents",
|
||||
tags: ["issues"],
|
||||
summary: "Bulk read one document key across issue IDs",
|
||||
description: "Read current documents for 1–200 comma-separated task UUIDs in one request. Company and task read permissions apply. Unreadable, hidden, foreign and nonexistent tasks are omitted; a readable task without this key returns document:null. Items follow first-requested ID order with duplicates removed. includeBody defaults to true; includeBody=false returns revision metadata and updatedAt without loading bodies, for cache validation. No break-glass access. Responses over 8 MiB fail with 413; split the IDs into smaller requests. Native helpers can use save_api_response then download_artifact to relay the complete batch without per-task calls. Verify every requested ID is present before treating the inventory as complete.",
|
||||
request: { params: z.object({ companyId: z.string().uuid() }), query: issueDocumentBatchQuerySchema },
|
||||
responses: { 200: r.ok(), 400: r.badRequest, 401: r.unauthorized, 403: r.forbidden, 413: { description: "Split the document batch into smaller requests" }, 429: { description: "At most two active reads per actor and eight per server instance; retry after an active read finishes", headers: { "Retry-After": { schema: { type: "string", enum: ["1"] } } } } },
|
||||
});
|
||||
|
||||
registry.registerPath({
|
||||
method: "get",
|
||||
path: "/api/issues/{id}/documents",
|
||||
|
||||
Reference in new issue
Block a user