fix(runner): stream and page large API responses within capture budgets (#14301)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use governed API tools to inspect task evidence.
> - Large API results become saved assets with short previews.
> - Reading an asset through the same tool used to create another asset,
so the agent could not reach the rest of the evidence.
> - The 10 MiB response cap also blocked useful large results. Removing
all bounds allowed excessive disk use.
> - This pull request streams responses up to 1 GiB and makes saved text
readable in bounded pages. It adds durable run budgets and capture
admission limits.
> - Agents can inspect complete evidence while tool results, memory use,
and capture work stay bounded.

## Linked Issues or Issue Description

**What happened?**
A large response became an asset. Reading that asset returned another
asset and the same preview. Responses above 10 MiB failed before the
agent could read any page.

**Expected behavior**
The agent can fetch a large response and read its saved text to EOF.
Each page stays bounded. New snapshots have a generous finite limit and
a durable run budget. Existing larger assets remain readable through
byte ranges.

**Steps to reproduce**
1. Call a GET operation that returns more than 10 MiB of text or JSON.
2. Before the fix, the tool returns `api_transport_failure`.
3. With this change, responses up to 1 GiB become streamed snapshots
with artifact references.
4. Read `GET /api/assets/{assetId}/content` with `responseText:
{offsetBytes: 0, limitBytes: 8192}`. Follow `nextOffsetBytes` until
null.

Related work: #14186 added the API fallback tools. #14218 bounded API
discovery.

## What Changed

- Add authenticated UTF-8 text windows to `call_api`, with byte offsets
and total size. Keep each page at or below 24 KiB.
- Stream new responses above 24 KiB through private temporary files into
company-owned assets. Bound each capture to 1 GiB of decoded bytes.
Reject oversized declared lengths before reading and count streamed
bytes before writing.
- Reserve capture budget in the run record before spilling. Allow 4 GiB
per run. Settle successful captures to their actual size. Failed or
interrupted captures retain their full 1 GiB reservation. Run restarts
do not reset the budget.
- Enforce a 20 GiB company snapshot quota with database reservations.
Count legacy snapshots and unfinished storage work across runs and
processes. Asset deletion frees quota.
- Limit large captures to two per company and four per server process.
Hold slots through storage upload and temporary-file cleanup. Use a
10-minute download deadline and 30-second connection/idle-read timeouts.
- Return explicit size, budget, busy, and timeout errors. Preserve
unknown outcomes for mutations whose response cannot be captured.
- Read saved assets through authenticated storage ranges, with at most
two extra bytes for UTF-8 and EOF handling. Unpaged reads return the
existing asset and digest with a bounded preview. Reads create no copies
and do not consume capture budget.
- Keep existing assets above 1 GiB readable in pages. Use safe integer
offsets and PostgreSQL `bigint` asset sizes.
- Stream large S3 uploads through ordered multipart requests. Abort
failed uploads and remove partial local files.
- Revalidate run authority during downloads. Keep company authorization,
GET-only text paging, redirect denial, and mutation replay receipts.
- Document the separate 10 MiB upload limits. This PR does not raise
memory-buffered attachment ingestion limits. Future large video uploads
need streamed ingestion and storage quotas.

## Verification

- Full workspace `pnpm -r typecheck` and `pnpm build` pass after
rebasing on master.
- Focused API and response tests: 1,761 pass. Cover declared and chunked
oversize responses, incorrect Content-Length, exact-limit success,
active-stream deadline, cancellation, cleanup, concurrency admission,
and mutation outcome handling.
- Real HTTP integration: 28 tests pass, including runnerd → PRP →
authority → HTTP, a 12 MiB snapshot, final-page/EOF reads, cross-company
denial, a persisted 3 GiB sparse asset, and large mutation receipt
replay.
- The HTTP suite verifies durable run-budget accounting, simultaneous
runs competing for company quota, legacy snapshot accounting, deletion
refunds, failed-storage reservations, cleaned-failure refunds,
metadata-rollback cleanup refunds, preservation after a lost commit
acknowledgement, and small/saved reads after capture-budget exhaustion.
- A standalone proof streams exactly 1 GiB through the production
capture helper, verifies the final bytes, and removes its temporary
file. It uses repeated 256 KiB chunks and records a peak process RSS of
191 MiB.
- Earlier storage verification covers exact S3 multipart boundaries,
cleanup/abort failures, and a 17 MiB transfer through the real AWS SDK
to a local HTTP S3 endpoint. No cloud S3 qualification was run for this
follow-up.
- The local full test run was interrupted for the company-quota changes.
A later targeted run hit exhausted macOS shared-memory slots before
tests started; two unattached PostgreSQL segments with dead owners were
reclaimed before retrying. All 55 current-head checks pass at
`aebb80ceeeee77d5a56b67bfffd835f2f846878c`, including the full CI test
suite, typecheck, build, browser suites, security scan, and Greptile
(5/5). There are no unresolved review threads. The combined rebased test
catalog also passes (48 tests).
- Earlier paging acceptance passed Daytona and separate staging at
`7739879e9`. Those runs predate the streaming and budget changes.

## Risks

- The 1 GiB response cap and 10-minute active-download deadline are
intentional product limits. Larger live results must use endpoint
pagination or a direct file workflow. Existing larger assets remain
readable through bounded ranges.
- A durable 20 GiB company snapshot quota counts stored runner-api
assets and active/orphan reservations across runs and processes. The
operator can set PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES to a
finite value of at least 1 GiB. Deleting snapshots frees capacity;
possible orphan storage must be reconciled before releasing its
reservation.
- A failed capture uses its full reservation. A new large capture needs
a full 1 GiB available, even if it later completes at a smaller size.
Small reads and existing asset pages remain available.
- Concurrency limits apply per server process. The run byte budget is
shared through the database.
- The `integer` to `bigint` migration rewrites asset metadata and takes
an exclusive table lock. File bytes stay in storage.
- A live endpoint is fetched once before returning its snapshot.
Continue reading the saved artifact for stable pages. Mutations may
commit before any size or transport error; inspect state before
retrying.
- Attachment uploads and native file handoffs still default to 10 MiB.
Raising buffered ingestion paths to GiB sizes is separate work.

## Model Used

OpenAI Codex, based on GPT-6, with code execution and repository tools.
The runtime does not expose an exact serving model variant or
context-window size. The earlier paging work also used browser testing
and subagents.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-09-28 10:36:32 -05:00
1 parent cea8dda472
commit 3447609d22
40 files changed
+2889 -73

No files matched your search

+20
View File
@@ -432,3 +432,23 @@ cleanup authority; it does not prove that remote inference has stopped. Recovery
revokes the previous boot identity with a conditional update. Its own claim also
expires so another sweep can finish cleanup after a restart. Historical rows keep
null ownership fields and follow the previous recovery path.
## Large API response snapshots
`assets.byte_size` uses PostgreSQL `bigint` so saved responses and byte ranges can
exceed 2 GiB. The API and Drizzle mapping continue to expose a JavaScript number;
response readers validate safe integer offsets. The type-widening migration
rewrites the asset metadata table and needs an exclusive table lock. File bytes
remain in local or object storage.
### Runner API response reservations
`runner_api_response_reservations` holds company-scoped API snapshot reservations.
Before a capture spills, the server locks company admission and counts stored
`runner-api` assets plus unattached reservations against a 20 GiB default quota.
A committed asset replaces its reservation in that total. The asset foreign key
cascades on deletion, while deleting a run sets `run_id` to null so an orphan
reservation cannot silently disappear. Failed cleanup or an ambiguous storage
write requires operator reconciliation before an unattached reservation is
removed. The table stores no response bodies. See `doc/runner-api-tools.md` for
limits and the operator override.
+1 -1
View File
@@ -414,7 +414,7 @@ Operational policy:
- `provider` enum/text (`local_disk | s3`)
- `object_key` text not null
- `content_type` text not null
- `byte_size` int not null
- `byte_size` bigint not null (safe integer byte count in the API)
- `sha256` text not null
- `original_filename` text null
- `created_by_agent_id` uuid fk null
+4
View File
@@ -262,3 +262,7 @@ named existing controls on legacy and native Codex. Discover it with
results and follow-up coverage are recorded in that suite's guide.
Continuation accounting has an explicit-only eight-cell Product E2E [baseline suite](../tests/runner-e2e/CONTINUATION-ACCOUNTING.md), complementing the deterministic lifecycle inventory.
The explicit-only Product E2E `api-response-reading` suite verifies retrieval of
large saved API responses on local and Daytona native Codex runs. See the
[Runner E2E guide](../tests/runner-e2e/README.md#bounded-api-response-reading).
+95 -4
View File
@@ -51,14 +51,105 @@ exactly one authorized `artifactId` or task-workspace `path`, and an optional
multipart `field`. No arbitrary URL, headers, authentication, or remote file URL
can be supplied. Routes still validate payloads and enforce permissions.
Requests have a 30-second HTTP timeout, 16 KiB URL limit and 10 MiB payload/response
transfer limit. Responses above 24 KiB and binary responses become company-owned
assets with retrievable references; text previews are limited to 2,000 bytes.
Requests have a 16 KiB URL limit and 10 MiB request/upload limit. **New response
captures are limited to 1 GiB of decoded bytes.** This is separate from the 24 KiB
inline/page limit. The receiver rejects an oversized Content-Length before
reading and counts actual bytes before writing, including chunked or compressed
responses. Oversized responses return `api_response_too_large`; narrow the query
or use the endpoint's own pagination. A mutation may already have committed, so
inspect its state rather than retrying it to obtain a smaller response.
Connection setup and stalled response reads time out after 30 seconds. An active
capture has a 10-minute total download deadline. Responses above 24 KiB stream
into a private temporary file, then into a company-owned asset. Memory stays
bounded by the inline prefix and stream buffers. Binary responses also become
assets; text previews are limited to 2,000 bytes.
Large captures reserve 1 GiB against a **durable 4 GiB per-run capture budget**
before creating a file. A completed capture settles to its actual byte count;
failed/interrupted captures retain the full reservation to bound retry loops.
A process restart does not reset that budget. Small inline responses and reads
of existing assets need no reservation. Concurrent large captures are limited
to **two per company and four per server process**, including the storage upload
and temporary-file cleanup. Budget exhaustion returns `api_response_capture_limit`;
concurrency exhaustion returns `api_response_capture_busy` without queuing more
large transfers. A **20 GiB company-wide quota** counts all stored `runner-api` snapshots plus
unattached reservations. Admission uses a company database lock, so runs and
server processes share the same quota. Existing snapshots from before this
change count too. Attaching an asset converts its reservation to actual stored
bytes; deleting the asset frees that capacity. Small binary snapshots also need
storage admission. Ordinary inline text/JSON and existing-asset pages do not.
Operators can set `PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES` to a positive
safe integer of at least 1 GiB. Invalid values fall back to 20 GiB. No zero or
unlimited setting is accepted. This quota covers API snapshots, not all company
attachments. Storage capacity and backend limits still apply.
Handled pre-storage failures release the company reservation after temporary
file cleanup, but retain the run's charge against retry loops. After a metadata
transaction fails, a locking read must prove the asset was not committed before
the uploaded object is removed. Confirmed cleanup refunds the company quota. A crash, failed
cleanup, or ambiguous storage write keeps an unattached reservation. Operators
must reconcile possible orphan files/objects before deleting that reservation
from `runner_api_response_reservations`; no automatic expiry silently refunds
possibly occupied space. Linked rows are removed with their asset. Deleting a
run does not release its unattached company reservations.
Temporary files are removed on success or failure. Long captures revalidate the
active run at least every MiB or at the next chunk after one second, and again
before returning the snapshot. Stopping the run stops its download.
To inspect saved text without creating another artifact, call its authorized
content operation with `responseText`:
```json
{"operationId":"GET /api/assets/{assetId}/content","pathParams":{"assetId":"RETURNED_ARTIFACT_ID"},"responseText":{"offsetBytes":0,"limitBytes":8192}}
```
The result contains `data` as text (including JSON), plus `responseText` with
`offsetBytes`, `nextOffsetBytes`, and `totalBytes`. Continue at `nextOffsetBytes`
until it is null. Windows end at UTF-8 boundaries. The limit defaults to 24 KiB
and accepts 4–24,576 bytes. The server rejects binary content, invalid UTF-8,
and offsets inside a code point or beyond the response. This option only works
with GET; never repeat a mutation to retrieve another part of its response.
Read the saved artifact for a stable snapshot instead of paging a changing live
response. A text-window call against a live response above 24 KiB also returns
that snapshot's artifact reference; continue on its content operation. A saved
asset page never creates another asset. An unpaged asset read also fetches only
a bounded preview and returns the existing reference instead of copying the file. Offsets and total sizes use safe integer
byte counts, including values above 2 GiB. Existing assets larger than the 1 GiB
capture limit remain readable because each request transfers only a bounded range.
Request/upload limits and all route
authorization remain.
Saved asset pages use authenticated HTTP byte ranges. The storage provider reads
only the requested window, with at most two extra bytes for UTF-8/EOF handling.
The client validates `Content-Range`, the total size, and the received byte count;
it rejects unsupported or inconsistent ranges instead of downloading the whole
asset for every page. Other GET routes are fetched once in full to create the
snapshot, so use the returned asset for subsequent pages. S3 uses streamed
multipart uploads for large snapshots. Asset sizes are stored as PostgreSQL
`bigint`, preserving the existing numeric API shape.
### Media files and upload limits
The 1 GiB limit covers new snapshots returned by `call_api`, such as large JSON
exports or binary API downloads. It does not raise attachment upload limits or
limit files an agent creates and edits inside its workspace. Saved asset downloads
stream from storage and support byte ranges, including video seeking.
`PAPERCLIP_ATTACHMENT_MAX_BYTES` separately defaults to 10 MiB for uploads and
native file handoffs. `call_api` uploads also have their own 10 MiB limit. Several
upload and handoff paths buffer complete files in memory; raising those defaults
to GiB sizes requires streaming ingestion and corresponding admission/budget
controls first. For a future video attachment workflow, 2 GiB per streamed file
is a reasonable default, with an operator override and storage quotas. Do not
claim that this response-paging change enables GiB attachment uploads.
Tool responses identify the HTTP route with `apiOperationId`. The native protocol
reserves `operationId` and `callId` for semantic tool-call identity; API metadata
must not masquerade as that envelope. Saved mutation receipts are normalized at
the tool boundary as well, without repeating their HTTP request.
All redirects are refused. Oversized or interrupted mutation responses have an
All redirects are refused. Interrupted mutation responses have an
unknown outcome, requiring inspection before another mutation.
Mutation responses with HTTP 5xx, HTTP 408, redirects, or malformed JSON also
retain an unknown outcome. A server may have committed the write before it
@@ -0,0 +1 @@
ALTER TABLE "assets" ALTER COLUMN "byte_size" SET DATA TYPE bigint;
@@ -0,0 +1,13 @@
CREATE TABLE IF NOT EXISTS "runner_api_response_reservations" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
"company_id" uuid NOT NULL,
"run_id" uuid,
"asset_id" uuid,
"reserved_bytes" bigint NOT NULL,
"created_at" timestamp with time zone DEFAULT now() NOT NULL
);
--> statement-breakpoint
DO $$ BEGIN ALTER TABLE "runner_api_response_reservations" ADD CONSTRAINT "runner_api_response_reservations_company_id_companies_id_fk" FOREIGN KEY ("company_id") REFERENCES "public"."companies"("id") ON DELETE cascade ON UPDATE no action; EXCEPTION WHEN duplicate_object THEN NULL; END $$;--> statement-breakpoint
DO $$ BEGIN ALTER TABLE "runner_api_response_reservations" ADD CONSTRAINT "runner_api_response_reservations_run_id_heartbeat_runs_id_fk" FOREIGN KEY ("run_id") REFERENCES "public"."heartbeat_runs"("id") ON DELETE set null ON UPDATE no action; EXCEPTION WHEN duplicate_object THEN NULL; END $$;--> statement-breakpoint
DO $$ BEGIN ALTER TABLE "runner_api_response_reservations" ADD CONSTRAINT "runner_api_response_reservations_asset_id_assets_id_fk" FOREIGN KEY ("asset_id") REFERENCES "public"."assets"("id") ON DELETE cascade ON UPDATE no action; EXCEPTION WHEN duplicate_object THEN NULL; END $$;--> statement-breakpoint
CREATE INDEX IF NOT EXISTS "runner_api_response_reservations_company_idx" ON "runner_api_response_reservations" USING btree ("company_id");
@@ -1,6 +1,6 @@
{
"id": "8d202a5a-5d6b-44d9-9336-3bb39a8b99bc",
"prevId": "861a2815-1418-4c71-a77b-d0e2e9566413",
"id": "c27cbbec-6948-4b2b-8db8-36d0bf3830fd",
"prevId": "7db2ceec-ed62-4968-90c9-4c79206cfe34",
"version": "7",
"dialect": "postgresql",
"tables": {
@@ -2917,7 +2917,7 @@
},
"byte_size": {
"name": "byte_size",
"type": "integer",
"type": "bigint",
"primaryKey": false,
"notNull": true
},
@@ -3302,6 +3302,13 @@
"primaryKey": false,
"notNull": false
},
"keyboard_shortcuts": {
"name": "keyboard_shortcuts",
"type": "boolean",
"primaryKey": false,
"notNull": true,
"default": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
@@ -5701,6 +5708,12 @@
"notNull": true,
"default": false
},
"communication_guidance": {
"name": "communication_guidance",
"type": "text",
"primaryKey": false,
"notNull": false
},
"state": {
"name": "state",
"type": "text",
@@ -6611,6 +6624,13 @@
"primaryKey": false,
"notNull": false
},
"communication_instructions": {
"name": "communication_instructions",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "''"
},
"allow_direct_messages": {
"name": "allow_direct_messages",
"type": "boolean",
@@ -8096,6 +8116,519 @@
},
"isRLSEnabled": false
},
"public.chat_github_configurations": {
"name": "chat_github_configurations",
"schema": "",
"columns": {
"endpoint_id": {
"name": "endpoint_id",
"type": "uuid",
"primaryKey": true,
"notNull": true
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"revision": {
"name": "revision",
"type": "integer",
"primaryKey": false,
"notNull": true,
"default": 1
},
"configuration": {
"name": "configuration",
"type": "jsonb",
"primaryKey": false,
"notNull": true
},
"updated_by_user_id": {
"name": "updated_by_user_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {},
"foreignKeys": {
"chat_github_configurations_company_id_endpoint_id_chat_endpoints_company_id_id_fk": {
"name": "chat_github_configurations_company_id_endpoint_id_chat_endpoints_company_id_id_fk",
"tableFrom": "chat_github_configurations",
"tableTo": "chat_endpoints",
"columnsFrom": [
"company_id",
"endpoint_id"
],
"columnsTo": [
"company_id",
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {
"chat_github_config_revision_check": {
"name": "chat_github_config_revision_check",
"value": "\"chat_github_configurations\".\"revision\" > 0"
}
},
"isRLSEnabled": false
},
"public.chat_github_registrations": {
"name": "chat_github_registrations",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"endpoint_id": {
"name": "endpoint_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"user_id": {
"name": "user_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"state_hash": {
"name": "state_hash",
"type": "text",
"primaryKey": false,
"notNull": true
},
"trusted_origin": {
"name": "trusted_origin",
"type": "text",
"primaryKey": false,
"notNull": true
},
"status": {
"name": "status",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'pending'"
},
"expires_at": {
"name": "expires_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true
},
"consumed_at": {
"name": "consumed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"chat_github_registration_state_uq": {
"name": "chat_github_registration_state_uq",
"columns": [
{
"expression": "state_hash",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"chat_github_registration_endpoint_idx": {
"name": "chat_github_registration_endpoint_idx",
"columns": [
{
"expression": "endpoint_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "expires_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"chat_github_registrations_company_id_endpoint_id_chat_endpoints_company_id_id_fk": {
"name": "chat_github_registrations_company_id_endpoint_id_chat_endpoints_company_id_id_fk",
"tableFrom": "chat_github_registrations",
"tableTo": "chat_endpoints",
"columnsFrom": [
"company_id",
"endpoint_id"
],
"columnsTo": [
"company_id",
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {
"chat_github_registration_status_check": {
"name": "chat_github_registration_status_check",
"value": "\"chat_github_registrations\".\"status\" in ('pending', 'exchanging', 'completed', 'failed')"
}
},
"isRLSEnabled": false
},
"public.chat_github_reviews": {
"name": "chat_github_reviews",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"endpoint_id": {
"name": "endpoint_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"issue_id": {
"name": "issue_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"run_id": {
"name": "run_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"repository_id": {
"name": "repository_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"repository": {
"name": "repository",
"type": "text",
"primaryKey": false,
"notNull": true
},
"pull_number": {
"name": "pull_number",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"head_sha": {
"name": "head_sha",
"type": "text",
"primaryKey": false,
"notNull": true
},
"delivery_id": {
"name": "delivery_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"configuration_revision": {
"name": "configuration_revision",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"policy_snapshot": {
"name": "policy_snapshot",
"type": "jsonb",
"primaryKey": false,
"notNull": true
},
"event": {
"name": "event",
"type": "jsonb",
"primaryKey": false,
"notNull": true
},
"state": {
"name": "state",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'queued'"
},
"assessment": {
"name": "assessment",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"conclusion": {
"name": "conclusion",
"type": "text",
"primaryKey": false,
"notNull": false
},
"check_id": {
"name": "check_id",
"type": "text",
"primaryKey": false,
"notNull": false
},
"check_url": {
"name": "check_url",
"type": "text",
"primaryKey": false,
"notNull": false
},
"summary_id": {
"name": "summary_id",
"type": "text",
"primaryKey": false,
"notNull": false
},
"summary_url": {
"name": "summary_url",
"type": "text",
"primaryKey": false,
"notNull": false
},
"publication_receipts": {
"name": "publication_receipts",
"type": "jsonb",
"primaryKey": false,
"notNull": true,
"default": "'{}'::jsonb"
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"chat_github_reviews_delivery_uq": {
"name": "chat_github_reviews_delivery_uq",
"columns": [
{
"expression": "endpoint_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "delivery_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"chat_github_reviews_pull_idx": {
"name": "chat_github_reviews_pull_idx",
"columns": [
{
"expression": "endpoint_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "repository_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "pull_number",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "created_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"chat_github_reviews_task_idx": {
"name": "chat_github_reviews_task_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "issue_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "run_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"chat_github_reviews_run_id_heartbeat_runs_id_fk": {
"name": "chat_github_reviews_run_id_heartbeat_runs_id_fk",
"tableFrom": "chat_github_reviews",
"tableTo": "heartbeat_runs",
"columnsFrom": [
"run_id"
],
"columnsTo": [
"id"
],
"onDelete": "set null",
"onUpdate": "no action"
},
"chat_github_reviews_company_id_endpoint_id_chat_endpoints_company_id_id_fk": {
"name": "chat_github_reviews_company_id_endpoint_id_chat_endpoints_company_id_id_fk",
"tableFrom": "chat_github_reviews",
"tableTo": "chat_endpoints",
"columnsFrom": [
"company_id",
"endpoint_id"
],
"columnsTo": [
"company_id",
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"chat_github_reviews_company_id_issue_id_issues_company_id_id_fk": {
"name": "chat_github_reviews_company_id_issue_id_issues_company_id_id_fk",
"tableFrom": "chat_github_reviews",
"tableTo": "issues",
"columnsFrom": [
"company_id",
"issue_id"
],
"columnsTo": [
"company_id",
"id"
],
"onDelete": "restrict",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {
"chat_github_reviews_company_id_uq": {
"name": "chat_github_reviews_company_id_uq",
"nullsNotDistinct": false,
"columns": [
"company_id",
"id"
]
}
},
"policies": {},
"checkConstraints": {
"chat_github_review_pull_number_check": {
"name": "chat_github_review_pull_number_check",
"value": "\"chat_github_reviews\".\"pull_number\" > 0"
},
"chat_github_review_state_check": {
"name": "chat_github_review_state_check",
"value": "\"chat_github_reviews\".\"state\" in ('queued', 'running', 'completed', 'incomplete', 'error', 'superseded', 'manual_required')"
}
},
"isRLSEnabled": false
},
"public.chat_teams_file_transfers": {
"name": "chat_teams_file_transfers",
"schema": "",
@@ -1,6 +1,6 @@
{
"id": "861a2815-1418-4c71-a77b-d0e2e9566413",
"prevId": "4770ad6d-2cdc-45c4-af89-c300fd134d91",
"id": "33120bf3-9cac-4909-ae94-d514ddc1e9e2",
"prevId": "c27cbbec-6948-4b2b-8db8-36d0bf3830fd",
"version": "7",
"dialect": "postgresql",
"tables": {
@@ -2917,7 +2917,7 @@
},
"byte_size": {
"name": "byte_size",
"type": "integer",
"type": "bigint",
"primaryKey": false,
"notNull": true
},
@@ -3302,6 +3302,13 @@
"primaryKey": false,
"notNull": false
},
"keyboard_shortcuts": {
"name": "keyboard_shortcuts",
"type": "boolean",
"primaryKey": false,
"notNull": true,
"default": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
@@ -5701,6 +5708,12 @@
"notNull": true,
"default": false
},
"communication_guidance": {
"name": "communication_guidance",
"type": "text",
"primaryKey": false,
"notNull": false
},
"state": {
"name": "state",
"type": "text",
@@ -6611,6 +6624,13 @@
"primaryKey": false,
"notNull": false
},
"communication_instructions": {
"name": "communication_instructions",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "''"
},
"allow_direct_messages": {
"name": "allow_direct_messages",
"type": "boolean",
@@ -8096,6 +8116,519 @@
},
"isRLSEnabled": false
},
"public.chat_github_configurations": {
"name": "chat_github_configurations",
"schema": "",
"columns": {
"endpoint_id": {
"name": "endpoint_id",
"type": "uuid",
"primaryKey": true,
"notNull": true
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"revision": {
"name": "revision",
"type": "integer",
"primaryKey": false,
"notNull": true,
"default": 1
},
"configuration": {
"name": "configuration",
"type": "jsonb",
"primaryKey": false,
"notNull": true
},
"updated_by_user_id": {
"name": "updated_by_user_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {},
"foreignKeys": {
"chat_github_configurations_company_id_endpoint_id_chat_endpoints_company_id_id_fk": {
"name": "chat_github_configurations_company_id_endpoint_id_chat_endpoints_company_id_id_fk",
"tableFrom": "chat_github_configurations",
"tableTo": "chat_endpoints",
"columnsFrom": [
"company_id",
"endpoint_id"
],
"columnsTo": [
"company_id",
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {
"chat_github_config_revision_check": {
"name": "chat_github_config_revision_check",
"value": "\"chat_github_configurations\".\"revision\" > 0"
}
},
"isRLSEnabled": false
},
"public.chat_github_registrations": {
"name": "chat_github_registrations",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"endpoint_id": {
"name": "endpoint_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"user_id": {
"name": "user_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"state_hash": {
"name": "state_hash",
"type": "text",
"primaryKey": false,
"notNull": true
},
"trusted_origin": {
"name": "trusted_origin",
"type": "text",
"primaryKey": false,
"notNull": true
},
"status": {
"name": "status",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'pending'"
},
"expires_at": {
"name": "expires_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true
},
"consumed_at": {
"name": "consumed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"chat_github_registration_state_uq": {
"name": "chat_github_registration_state_uq",
"columns": [
{
"expression": "state_hash",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"chat_github_registration_endpoint_idx": {
"name": "chat_github_registration_endpoint_idx",
"columns": [
{
"expression": "endpoint_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "expires_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"chat_github_registrations_company_id_endpoint_id_chat_endpoints_company_id_id_fk": {
"name": "chat_github_registrations_company_id_endpoint_id_chat_endpoints_company_id_id_fk",
"tableFrom": "chat_github_registrations",
"tableTo": "chat_endpoints",
"columnsFrom": [
"company_id",
"endpoint_id"
],
"columnsTo": [
"company_id",
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {
"chat_github_registration_status_check": {
"name": "chat_github_registration_status_check",
"value": "\"chat_github_registrations\".\"status\" in ('pending', 'exchanging', 'completed', 'failed')"
}
},
"isRLSEnabled": false
},
"public.chat_github_reviews": {
"name": "chat_github_reviews",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"endpoint_id": {
"name": "endpoint_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"issue_id": {
"name": "issue_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"run_id": {
"name": "run_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"repository_id": {
"name": "repository_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"repository": {
"name": "repository",
"type": "text",
"primaryKey": false,
"notNull": true
},
"pull_number": {
"name": "pull_number",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"head_sha": {
"name": "head_sha",
"type": "text",
"primaryKey": false,
"notNull": true
},
"delivery_id": {
"name": "delivery_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"configuration_revision": {
"name": "configuration_revision",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"policy_snapshot": {
"name": "policy_snapshot",
"type": "jsonb",
"primaryKey": false,
"notNull": true
},
"event": {
"name": "event",
"type": "jsonb",
"primaryKey": false,
"notNull": true
},
"state": {
"name": "state",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'queued'"
},
"assessment": {
"name": "assessment",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"conclusion": {
"name": "conclusion",
"type": "text",
"primaryKey": false,
"notNull": false
},
"check_id": {
"name": "check_id",
"type": "text",
"primaryKey": false,
"notNull": false
},
"check_url": {
"name": "check_url",
"type": "text",
"primaryKey": false,
"notNull": false
},
"summary_id": {
"name": "summary_id",
"type": "text",
"primaryKey": false,
"notNull": false
},
"summary_url": {
"name": "summary_url",
"type": "text",
"primaryKey": false,
"notNull": false
},
"publication_receipts": {
"name": "publication_receipts",
"type": "jsonb",
"primaryKey": false,
"notNull": true,
"default": "'{}'::jsonb"
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"chat_github_reviews_delivery_uq": {
"name": "chat_github_reviews_delivery_uq",
"columns": [
{
"expression": "endpoint_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "delivery_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"chat_github_reviews_pull_idx": {
"name": "chat_github_reviews_pull_idx",
"columns": [
{
"expression": "endpoint_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "repository_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "pull_number",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "created_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"chat_github_reviews_task_idx": {
"name": "chat_github_reviews_task_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "issue_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "run_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"chat_github_reviews_run_id_heartbeat_runs_id_fk": {
"name": "chat_github_reviews_run_id_heartbeat_runs_id_fk",
"tableFrom": "chat_github_reviews",
"tableTo": "heartbeat_runs",
"columnsFrom": [
"run_id"
],
"columnsTo": [
"id"
],
"onDelete": "set null",
"onUpdate": "no action"
},
"chat_github_reviews_company_id_endpoint_id_chat_endpoints_company_id_id_fk": {
"name": "chat_github_reviews_company_id_endpoint_id_chat_endpoints_company_id_id_fk",
"tableFrom": "chat_github_reviews",
"tableTo": "chat_endpoints",
"columnsFrom": [
"company_id",
"endpoint_id"
],
"columnsTo": [
"company_id",
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"chat_github_reviews_company_id_issue_id_issues_company_id_id_fk": {
"name": "chat_github_reviews_company_id_issue_id_issues_company_id_id_fk",
"tableFrom": "chat_github_reviews",
"tableTo": "issues",
"columnsFrom": [
"company_id",
"issue_id"
],
"columnsTo": [
"company_id",
"id"
],
"onDelete": "restrict",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {
"chat_github_reviews_company_id_uq": {
"name": "chat_github_reviews_company_id_uq",
"nullsNotDistinct": false,
"columns": [
"company_id",
"id"
]
}
},
"policies": {},
"checkConstraints": {
"chat_github_review_pull_number_check": {
"name": "chat_github_review_pull_number_check",
"value": "\"chat_github_reviews\".\"pull_number\" > 0"
},
"chat_github_review_state_check": {
"name": "chat_github_review_state_check",
"value": "\"chat_github_reviews\".\"state\" in ('queued', 'running', 'completed', 'incomplete', 'error', 'superseded', 'manual_required')"
}
},
"isRLSEnabled": false
},
"public.chat_teams_file_transfers": {
"name": "chat_teams_file_transfers",
"schema": "",
@@ -37416,6 +37949,26 @@
"notNull": true,
"default": true
},
"setup_pending": {
"name": "setup_pending",
"type": "boolean",
"primaryKey": false,
"notNull": true,
"default": false
},
"archived": {
"name": "archived",
"type": "boolean",
"primaryKey": false,
"notNull": true,
"default": false
},
"last_webhook_delivery": {
"name": "last_webhook_delivery",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"cron_expression": {
"name": "cron_expression",
"type": "text",
@@ -37677,6 +38230,100 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.routine_webhook_test_receipts": {
"name": "routine_webhook_test_receipts",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"trigger_id": {
"name": "trigger_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"delivery_key_hash": {
"name": "delivery_key_hash",
"type": "text",
"primaryKey": false,
"notNull": true
},
"received_at": {
"name": "received_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"routine_webhook_test_receipts_delivery_uq": {
"name": "routine_webhook_test_receipts_delivery_uq",
"columns": [
{
"expression": "trigger_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "delivery_key_hash",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"routine_webhook_test_receipts_company_id_companies_id_fk": {
"name": "routine_webhook_test_receipts_company_id_companies_id_fk",
"tableFrom": "routine_webhook_test_receipts",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"routine_webhook_test_receipts_trigger_id_routine_triggers_id_fk": {
"name": "routine_webhook_test_receipts_trigger_id_routine_triggers_id_fk",
"tableFrom": "routine_webhook_test_receipts",
"tableTo": "routine_triggers",
"columnsFrom": [
"trigger_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.routines": {
"name": "routines",
"schema": "",
@@ -38292,6 +38939,113 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.runner_api_response_reservations": {
"name": "runner_api_response_reservations",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"run_id": {
"name": "run_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"asset_id": {
"name": "asset_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"reserved_bytes": {
"name": "reserved_bytes",
"type": "bigint",
"primaryKey": false,
"notNull": true
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"runner_api_response_reservations_company_idx": {
"name": "runner_api_response_reservations_company_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"runner_api_response_reservations_company_id_companies_id_fk": {
"name": "runner_api_response_reservations_company_id_companies_id_fk",
"tableFrom": "runner_api_response_reservations",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"runner_api_response_reservations_run_id_heartbeat_runs_id_fk": {
"name": "runner_api_response_reservations_run_id_heartbeat_runs_id_fk",
"tableFrom": "runner_api_response_reservations",
"tableTo": "heartbeat_runs",
"columnsFrom": [
"run_id"
],
"columnsTo": [
"id"
],
"onDelete": "set null",
"onUpdate": "no action"
},
"runner_api_response_reservations_asset_id_assets_id_fk": {
"name": "runner_api_response_reservations_asset_id_assets_id_fk",
"tableFrom": "runner_api_response_reservations",
"tableTo": "assets",
"columnsFrom": [
"asset_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.secret_access_events": {
"name": "secret_access_events",
"schema": "",
+14
View File
@@ -1982,6 +1982,20 @@
"when": 1790439010225,
"tag": "0284_petite_genesis",
"breakpoints": true
},
{
"idx": 285,
"version": "7",
"when": 1790603883247,
"tag": "0285_heavy_oracle",
"breakpoints": true
},
{
"idx": 286,
"version": "7",
"when": 1790607525758,
"tag": "0286_complete_alice",
"breakpoints": true
}
]
}
+2 -2
View File
@@ -1,4 +1,4 @@
import { pgTable, uuid, text, integer, timestamp, index, uniqueIndex } from "drizzle-orm/pg-core";
import { pgTable, uuid, text, bigint, timestamp, index, uniqueIndex } from "drizzle-orm/pg-core";
import { companies } from "./companies.js";
import { agents } from "./agents.js";
@@ -10,7 +10,7 @@ export const assets = pgTable(
provider: text("provider").notNull(),
objectKey: text("object_key").notNull(),
contentType: text("content_type").notNull(),
byteSize: integer("byte_size").notNull(),
byteSize: bigint("byte_size", { mode: "number" }).notNull(),
sha256: text("sha256").notNull(),
originalFilename: text("original_filename"),
createdByAgentId: uuid("created_by_agent_id").references(() => agents.id),
+1
View File
@@ -115,6 +115,7 @@ export {
export { feedbackExports } from "./feedback_exports.js";
export { issueReadStates } from "./issue_read_states.js";
export { assets } from "./assets.js";
export { runnerApiResponseReservations } from "./runner_api_response_reservations.js";
export { issueAttachments } from "./issue_attachments.js";
export { documents } from "./documents.js";
export { documentRevisions } from "./document_revisions.js";
@@ -0,0 +1,13 @@
import { pgTable, uuid, bigint, timestamp, index } from "drizzle-orm/pg-core";
import { companies } from "./companies.js";
import { heartbeatRuns } from "./heartbeat_runs.js";
import { assets } from "./assets.js";
export const runnerApiResponseReservations = pgTable("runner_api_response_reservations", {
id: uuid("id").primaryKey().defaultRandom(),
companyId: uuid("company_id").notNull().references(() => companies.id, { onDelete: "cascade" }),
runId: uuid("run_id").references(() => heartbeatRuns.id, { onDelete: "set null" }),
assetId: uuid("asset_id").references(() => assets.id, { onDelete: "cascade" }),
reservedBytes: bigint("reserved_bytes", { mode: "number" }).notNull(),
createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(),
}, table => ({ companyIdx: index("runner_api_response_reservations_company_idx").on(table.companyId) }));
@@ -1 +1 @@
[{"annotations":{"exposure":"always","operationId":"get_task_context","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read the active task and actor, including the exact approved Markdown revision when this issue has an accepted plan.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"get_task_context","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"get_task_history","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read bounded comments on the active task.","inputSchema":{"additionalProperties":false,"properties":{"limit":{"default":50,"maximum":200,"minimum":1,"type":"integer"}},"required":[],"type":"object"},"name":"get_task_history","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"list_documents","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"List revisioned documents on the active task.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_documents","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"read_document","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read the current revision of one active-task document.","inputSchema":{"additionalProperties":false,"properties":{"key":{"description":"Stable issue-document key.","maxLength":120,"minLength":1,"type":"string"}},"required":["key"],"type":"object"},"name":"read_document","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"list_document_revisions","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read bounded revision history for one active-task document.","inputSchema":{"additionalProperties":false,"properties":{"key":{"description":"Stable issue-document key.","maxLength":120,"minLength":1,"type":"string"},"limit":{"default":50,"maximum":200,"minimum":1,"type":"integer"}},"required":["key"],"type":"object"},"name":"list_document_revisions","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"report_progress","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Append a durable progress comment to the active task.","inputSchema":{"additionalProperties":false,"properties":{"body":{"description":"Multiline progress update.","maxLength":20000,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","body"],"type":"object"},"name":"report_progress","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"answer_status_question","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Append the answer to a status-only wake without changing task disposition.","inputSchema":{"additionalProperties":false,"properties":{"body":{"description":"Concise status answer.","maxLength":20000,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","body"],"type":"object"},"name":"answer_status_question","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"write_document","requiredClaimLine truncated
[{"annotations":{"exposure":"always","operationId":"get_task_context","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read the active task and actor, including the exact approved Markdown revision when this issue has an accepted plan.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"get_task_context","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"get_task_history","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read bounded comments on the active task.","inputSchema":{"additionalProperties":false,"properties":{"limit":{"default":50,"maximum":200,"minimum":1,"type":"integer"}},"required":[],"type":"object"},"name":"get_task_history","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"list_documents","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"List revisioned documents on the active task.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_documents","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"read_document","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read the current revision of one active-task document.","inputSchema":{"additionalProperties":false,"properties":{"key":{"description":"Stable issue-document key.","maxLength":120,"minLength":1,"type":"string"}},"required":["key"],"type":"object"},"name":"read_document","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"list_document_revisions","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read bounded revision history for one active-task document.","inputSchema":{"additionalProperties":false,"properties":{"key":{"description":"Stable issue-document key.","maxLength":120,"minLength":1,"type":"string"},"limit":{"default":50,"maximum":200,"minimum":1,"type":"integer"}},"required":["key"],"type":"object"},"name":"list_document_revisions","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"report_progress","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Append a durable progress comment to the active task.","inputSchema":{"additionalProperties":false,"properties":{"body":{"description":"Multiline progress update.","maxLength":20000,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","body"],"type":"object"},"name":"report_progress","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"answer_status_question","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Append the answer to a status-only wake without changing task disposition.","inputSchema":{"additionalProperties":false,"properties":{"body":{"description":"Concise status answer.","maxLength":20000,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","body"],"type":"object"},"name":"answer_status_question","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"write_document","requiredClaimLine truncated
@@ -112,7 +112,7 @@
"planning",
"skill_test"
],
"description": "Fallback only: call a discovered Paperclip API operation when dedicated tools lack the required operation or parameters. Uses your existing permissions. Prefer dedicated tools; never bypass a denial or runner lifecycle tool.",
"description": "Fallback only: call a discovered Paperclip API operation when dedicated tools lack the required operation or parameters. Uses your existing permissions. Prefer dedicated tools; never bypass a denial or runner lifecycle tool. For large text responses, read the returned artifact with GET /api/assets/{assetId}/content and responseText; follow nextOffsetBytes until null.",
"effect": "write",
"inputSchema": {
"additionalProperties": false,
@@ -199,6 +199,25 @@
"query": {
"additionalProperties": true,
"type": "object"
},
"responseText": {
"additionalProperties": false,
"description": "GET only: return a bounded UTF-8 text window inline, including JSON as text, without saving another artifact. Offsets and limits are bytes. Use the returned nextOffsetBytes to continue; null means complete. Prefer reading a saved artifact for a stable snapshot. New live responses have a 1 GiB capture limit and a 4 GiB per-run capture budget. Existing larger assets remain readable in bounded pages. If a live response returns an artifact, continue on its content operation for a stable snapshot.",
"properties": {
"limitBytes": {
"default": 24576,
"maximum": 24576,
"minimum": 4,
"type": "integer"
},
"offsetBytes": {
"default": 0,
"maximum": 9007199254740991,
"minimum": 0,
"type": "integer"
}
},
"type": "object"
}
},
"required": [
@@ -24,7 +24,7 @@
"prpVersion": 1,
"nativeExecutionVersion": 1,
"catalogVersion": 1,
"catalogSha256": "sha256:1fa8ecf5d3c28e62a32e818de95633ff3efe5d4af1a100310516ff9b5ab2358d",
"catalogSha256": "sha256:a19534304480dfd3d357cc902e785af05158ce1496f4d324184112732ff94c9c",
"driverContractVersion": 1,
"driverKind": "paperclip-deterministic",
"driverVersion": "1.0.0"
@@ -160,7 +160,7 @@
},
{
"path": "fixtures/evals/native-execution-seeded.json",
"sha256": "5f3144ea856ad27778f47de73eb542703b1d9df2abad2c427838fbaf69c3d59f",
"sha256": "c6367c7c7e3dd01fed341b7a83b0008c9d5d1efdfa85bc70e1013bc304a008c7",
"expectation": "accept",
"compatibilityCase": "canonical"
},
@@ -28,7 +28,7 @@ export const callApiAction = {
},
"documentation": {
"title": "Call the Paperclip API",
"description": "Fallback only: call a discovered Paperclip API operation when dedicated tools lack the required operation or parameters. Uses your existing permissions. Prefer dedicated tools; never bypass a denial or runner lifecycle tool.",
"description": "Fallback only: call a discovered Paperclip API operation when dedicated tools lack the required operation or parameters. Uses your existing permissions. Prefer dedicated tools; never bypass a denial or runner lifecycle tool. For large text responses, read the returned artifact with GET /api/assets/{assetId}/content and responseText; follow nextOffsetBytes until null.",
"note": "Production HTTP fallback; does not grant privileges or replace dedicated tools."
},
"examples": {
@@ -51,7 +51,7 @@ export const callApiAction = {
"operationId": "call_api",
"version": 1,
"title": "Call the Paperclip API",
"description": "Fallback only: call a discovered Paperclip API operation when dedicated tools lack the required operation or parameters. Uses your existing permissions. Prefer dedicated tools; never bypass a denial or runner lifecycle tool.",
"description": "Fallback only: call a discovered Paperclip API operation when dedicated tools lack the required operation or parameters. Uses your existing permissions. Prefer dedicated tools; never bypass a denial or runner lifecycle tool. For large text responses, read the returned artifact with GET /api/assets/{assetId}/content and responseText; follow nextOffsetBytes until null.",
"exposure": "optional",
"requiredClaims": [
"api:call"
@@ -96,6 +96,15 @@ export const callApiAction = {
"type": "string",
"maxLength": 120
},
"responseText": {
"description": "GET only: return a bounded UTF-8 text window inline, including JSON as text, without saving another artifact. Offsets and limits are bytes. Use the returned nextOffsetBytes to continue; null means complete. Prefer reading a saved artifact for a stable snapshot. New live responses have a 1 GiB capture limit and a 4 GiB per-run capture budget. Existing larger assets remain readable in bounded pages. If a live response returns an artifact, continue on its content operation for a stable snapshot.",
"type": "object",
"properties": {
"offsetBytes": { "type": "integer", "minimum": 0, "maximum": 9007199254740991, "default": 0 },
"limitBytes": { "type": "integer", "minimum": 4, "maximum": 24576, "default": 24576 }
},
"additionalProperties": false
},
"files": {
"type": "array",
"maxItems": 10,
+1
View File
@@ -4,6 +4,7 @@ export interface AssetImage {
provider: string;
objectKey: string;
contentType: string;
/** Safe integer byte count; persisted as bigint to support files above 2 GiB. */
byteSize: number;
sha256: string;
originalFilename: string | null;
+23
View File
@@ -421,6 +421,29 @@ describe("GET /api/assets/:assetId/content", () => {
getAssetByIdMock.mockReset();
});
it("reads only the requested storage range for a large asset", async () => {
const storage = createStorageService("text/plain");
getAssetByIdMock.mockResolvedValue({ ...createAsset(), contentType: "text/plain", byteSize: 10 * 1024 * 1024 });
vi.mocked(storage.getObject).mockImplementation(async (_company, _key, options) => {
expect(options).toEqual({ range: { start: 8192, end: 12288 } });
return { stream: Readable.from(Buffer.alloc(4097, 65)), contentLength: 4097 };
});
const res = await requestApp(await createApp(storage), baseUrl => request(baseUrl).get("/api/assets/asset-1/content").set("Range", "bytes=8192-12288"));
expect(res.status).toBe(206);
expect(res.headers["content-range"]).toBe("bytes 8192-12288/10485760");
expect(res.headers["content-length"]).toBe("4097");
expect(res.text).toBe("A".repeat(4097));
});
it.each(["bytes=999-1000", "bytes=0-1,3-4", "bytes=1x-3", "bytes=-", "items=0-3"])("rejects invalid asset ranges before storage reads: %s", async range => {
const storage = createStorageService();
getAssetByIdMock.mockResolvedValue(createAsset());
const res = await requestApp(await createApp(storage), baseUrl => request(baseUrl).get("/api/assets/asset-1/content").set("Range", range));
expect(res.status).toBe(416);
expect(res.headers["content-range"]).toBe("bytes */40");
expect(storage.getObject).not.toHaveBeenCalled();
});
it("downloads script-capable HTML with nosniff and a sandbox CSP", async () => {
const html = Buffer.from("<script>globalThis.__assetXss = true</script>");
const storage = createStorageService("text/html");
@@ -1,4 +1,6 @@
import { afterEach, describe, expect, it } from "vitest";
import { Readable } from "node:stream";
import { createHash } from "node:crypto";
import os from "node:os";
import path from "node:path";
import { promises as fs } from "node:fs";
@@ -42,6 +44,27 @@ describe("local disk storage provider", () => {
expect(stored.sha256).toHaveLength(64);
});
it("writes streamed files with exact metadata", async () => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-storage-"));
tempRoots.push(root);
const service = createStorageService(createLocalDiskStorageProvider(root));
const bytes = Buffer.from("streamed response");
const sha256 = createHash("sha256").update(bytes).digest("hex");
const stored = await service.putFile({ companyId: "company-1", namespace: "runner-api", originalFilename: "response.txt",
contentType: "text/plain", body: Readable.from([bytes.subarray(0, 4), bytes.subarray(4)]), byteSize: bytes.length, sha256 });
expect(stored).toMatchObject({ byteSize: bytes.length, sha256 });
expect(await readStreamToBuffer((await service.getObject("company-1", stored.objectKey)).stream)).toEqual(bytes);
});
it("removes partial local uploads when their source fails", async () => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-storage-"));
tempRoots.push(root);
const provider = createLocalDiskStorageProvider(root);
const body = Readable.from((async function* () { yield Buffer.from("partial"); throw new Error("source failed"); })());
await expect(provider.putObject({ objectKey: "company-1/failed", contentType: "text/plain", contentLength: 20, body })).rejects.toThrow("source failed");
expect(await fs.readdir(path.join(root, "company-1"))).toEqual([]);
});
it("streams only requested byte ranges", async () => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-storage-"));
tempRoots.push(root);
@@ -0,0 +1,48 @@
import { AbortMultipartUploadCommand, CompleteMultipartUploadCommand, CreateMultipartUploadCommand, UploadPartCommand, S3Client } from "@aws-sdk/client-s3";
import { Readable } from "node:stream";
import { describe, expect, it, vi } from "vitest";
import { putS3Multipart } from "../storage/s3-multipart.js";
describe("streamed S3 multipart uploads", () => {
it("streams exact ordered parts without buffering the full response", async () => {
const client = new S3Client({ region: "us-east-1" });
const lengths: number[] = [];
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
if (command instanceof UploadPartCommand) {
expect(command.input.Body).toBeInstanceOf(Readable);
let length = 0;
for await (const chunk of command.input.Body as Readable) {
expect(chunk.length).toBeLessThanOrEqual(64 * 1024);
length += chunk.length;
}
expect(length).toBe(command.input.ContentLength);
lengths.push(length);
return { ETag: `part-${command.input.PartNumber}` };
}
expect(command).toBeInstanceOf(CompleteMultipartUploadCommand);
expect(command.input.MultipartUpload.Parts).toEqual([1, 2, 3].map(PartNumber => ({ PartNumber, ETag: `part-${PartNumber}` })));
return {};
});
const body = Readable.from((async function* () { for (let i = 0; i < 272; i++) yield Buffer.alloc(64 * 1024); })());
try {
await putS3Multipart(client, "bucket", "key", { objectKey: "key", contentType: "text/plain", contentLength: 17 * 1024 * 1024, body });
expect(lengths).toEqual([8, 8, 1].map(mb => mb * 1024 * 1024));
} finally { send.mockRestore(); client.destroy(); }
});
it("aborts an incomplete upload and closes its source on failure", async () => {
const client = new S3Client({ region: "us-east-1" });
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
if (command instanceof UploadPartCommand) throw new Error("storage unavailable");
expect(command).toBeInstanceOf(AbortMultipartUploadCommand);
return {};
});
const body = Readable.from([Buffer.from("abc")]);
try {
await expect(putS3Multipart(client, "bucket", "key", { objectKey: "key", contentType: "text/plain", contentLength: 3, body })).rejects.toThrow("storage unavailable");
expect(send).toHaveBeenCalledTimes(3);
expect(body.destroyed).toBe(true);
} finally { send.mockRestore(); client.destroy(); }
});
});
+22 -2
View File
@@ -328,13 +328,32 @@ export function assetRoutes(db: Db, storage: StorageService) {
const asset = await getAccessibleResource(req, res, svc.getById(assetId), "Asset not found");
if (!asset) return;
const object = await storage.getObject(asset.companyId, asset.objectKey);
// Use the persisted size only after resource authorization. Single ranges
// keep saved API text pages bounded all the way to disk or object storage.
const rawRange = req.headers.range;
const rangeSyntax = rawRange && /^bytes=(\d*)-(\d*)$/i.exec(rawRange);
const emptyRead = asset.byteSize === 0 && rangeSyntax?.[1] === "0";
const ranges = rawRange && !emptyRead ? req.range(asset.byteSize) : undefined;
res.setHeader("Accept-Ranges", "bytes");
if (/^[a-f0-9]{64}$/.test(asset.sha256)) res.setHeader("ETag", `"${asset.sha256}"`);
if (rawRange && (!rangeSyntax || (!rangeSyntax[1] && !rangeSyntax[2])
|| (!emptyRead && (!Array.isArray(ranges) || ranges.length !== 1)))) {
res.setHeader("Content-Range", `bytes */${asset.byteSize}`);
res.status(416).end();
return;
}
const range = Array.isArray(ranges) ? ranges[0] : undefined;
const object = await storage.getObject(asset.companyId, asset.objectKey, range ? { range } : undefined);
const responseContentType = asset.contentType || object.contentType || "application/octet-stream";
const mediaType = responseContentType.split(";", 1)[0]?.trim().toLowerCase() ?? "";
const inlineSafe = mediaType !== SVG_CONTENT_TYPE
&& isInlineAttachmentContentType(mediaType);
res.setHeader("Content-Type", responseContentType);
res.setHeader("Content-Length", String(asset.byteSize || object.contentLength || 0));
res.setHeader("Content-Length", String(range ? range.end - range.start + 1 : asset.byteSize || object.contentLength || 0));
if (range) {
res.status(206);
res.setHeader("Content-Range", `bytes ${range.start}-${range.end}/${asset.byteSize}`);
}
res.setHeader("Cache-Control", "private, max-age=60");
res.setHeader("X-Content-Type-Options", "nosniff");
if (!inlineSafe) {
@@ -349,6 +368,7 @@ export function assetRoutes(db: Db, storage: StorageService) {
object.stream.on("error", (err) => {
next(err);
});
res.on("close", () => object.stream.destroy());
object.stream.pipe(res);
});
@@ -1,3 +1,4 @@
import { createReadStream } from "node:fs";
import { publicChatTaskUrl } from "../chat-task-url.js";
import type { createAssignedMcpTools } from "./assigned-mcp-tools.js";
import { assertAssignableAgent } from "../agent-assignability.js";
@@ -27,10 +28,13 @@ import { workspaceFileResourceService } from "../workspace-file-resources.js";
import { badRequest, forbidden } from "../../errors.js";
import { searchRunnerApi } from "./runner-api-catalog.js";
import { executeRunnerApi, validateRunnerApiCall, RUNNER_API_MAX_BYTES, type RunnerApiFile } from "./runner-api-client.js";
import { acquireRunnerApiResponseSlot, runnerApiCompanyCaptureMaxBytes, RUNNER_API_RESPONSE_MAX_BYTES, RUNNER_API_RESPONSE_RUN_MAX_BYTES, RunnerApiResponseLimitError } from "./runner-api-response-limits.js";
import { and, desc, eq, isNull, notInArray, sql } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
activityLog,
assets,
runnerApiResponseReservations,
agents,
agentWakeupRequests,
chatEndpoints,
@@ -527,6 +531,8 @@ export class PaperclipRunnerToolAuthority {
const token = createLocalAgentJwt(this.binding.agentId, this.binding.companyId, bound.actor.adapterType, this.binding.runId, bound.run.responsibleUserId);
if (!token) throw new Error("Paperclip run authentication is unavailable");
const execute = async () => {
let reservationId: string | undefined;
let storageAttempted = false;
const current = await this.#boundContext();
if (!runnerApiToolsEnabled(this.binding.companyId, this.binding.apiToolsEnabled)) throw new Error("paperclip_runner_tool_not_advertised");
return executeRunnerApi(input, { ...context, workMode: current.issue.workMode }, {
@@ -537,10 +543,34 @@ export class PaperclipRunnerToolAuthority {
validateRunnerApiCall(input, { ...context, workMode: fresh.issue.workMode });
},
readFile: (file) => this.#readApiFile(file),
reserveResponseCapture: async () => {
const reservation = await this.#reserveApiResponseCapture();
reservationId = reservation.id;
return (completedBytes, cleaned) => reservation.settle(completedBytes, cleaned, storageAttempted);
},
saveResponse: async (bytes, contentType) => {
if (!reservationId) throw new Error("API response storage reservation is missing");
const storage = this.binding.storage ?? getStorageService();
const saved = await storage.putFile({ companyId: this.binding.companyId, namespace: "runner-api", originalFilename: contentType.includes("json") ? "response.json" : "response.bin", contentType, body: bytes });
const asset = await assetService(this.db).create(this.binding.companyId, { ...saved, createdByAgentId: this.binding.agentId });
storageAttempted = true;
const saved = await storage.putFile({ companyId: this.binding.companyId, namespace: "runner-api", originalFilename: contentType.includes("json") ? "response.json" : "response.bin", contentType, ...(Buffer.isBuffer(bytes) ? { body: bytes } : { body: createReadStream(bytes.path), byteSize: bytes.byteSize, sha256: bytes.sha256 }) });
const asset = await this.db.transaction(async tx => {
await this.#lockApiCaptureCompany(tx as unknown as Db);
const asset = await assetService(tx as unknown as Db).create(this.binding.companyId, { ...saved, createdByAgentId: this.binding.agentId });
await tx.update(runnerApiResponseReservations).set({ assetId: asset.id, reservedBytes: asset.byteSize }).where(and(eq(runnerApiResponseReservations.id, reservationId!), eq(runnerApiResponseReservations.companyId, this.binding.companyId)));
return asset;
}).catch(async error => {
// A failed commit acknowledgement can still leave a committed
// asset. A locking read waits out that transaction before proving
// the reservation is unlinked and safe to compensate.
const reservation = await this.db.select({ assetId: runnerApiResponseReservations.assetId })
.from(runnerApiResponseReservations).where(and(eq(runnerApiResponseReservations.id, reservationId!), eq(runnerApiResponseReservations.companyId, this.binding.companyId))).for("update")
.then(rows => rows[0]).catch(() => undefined);
if (reservation?.assetId === null) {
const removed = await storage.deleteObject(this.binding.companyId, saved.objectKey).then(() => true, () => false);
if (removed) storageAttempted = false;
}
throw error;
});
const activity = await persistActivity(this.db, { companyId: this.binding.companyId, actorType: "agent", actorId: this.binding.agentId, agentId: this.binding.agentId, runId: this.binding.runId, issueId: this.binding.issueId, action: "asset.created", entityType: "asset", entityId: asset.id, details: { source: "runner.call_api", byteSize: saved.byteSize } });
publishActivity(activity.publication);
return { artifactId: asset.id, url: `/api/assets/${asset.id}/content`, contentType, byteSize: saved.byteSize, sha256: saved.sha256 };
@@ -589,6 +619,68 @@ export class PaperclipRunnerToolAuthority {
return result;
}
async #lockApiCaptureCompany(tx: Db): Promise<void> {
await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`runner-api-capture:${this.binding.companyId}`}, 0))`);
}
async #reserveApiResponseCapture() {
const release = acquireRunnerApiResponseSlot(this.binding.companyId);
let id: string;
try {
id = await this.db.transaction(async tx => {
await this.#lockApiCaptureCompany(tx as unknown as Db);
const locked = await this.#lockAuthorizedMutationContext(tx as unknown as Db);
const resultJson = record(locked.run.resultJson);
const used = resultJson.apiResponseCaptureBytes ?? 0;
if (typeof used !== "number" || !Number.isSafeInteger(used) || used < 0
|| used + RUNNER_API_RESPONSE_MAX_BYTES > RUNNER_API_RESPONSE_RUN_MAX_BYTES) {
throw new RunnerApiResponseLimitError("api_response_capture_limit", "This run has insufficient capture budget (4 GiB per run). Read an existing asset or narrow the query.");
}
// One company lock covers all runs/processes. Stored assets include
// snapshots from before reservations existed; attached reservations
// are excluded so completed snapshots are counted exactly once.
const [usage] = await tx.select({ bytes: sql<string>`
coalesce((select sum(${assets.byteSize}) from ${assets}
where ${assets.companyId} = ${this.binding.companyId}
and ${assets.objectKey} like ${`${this.binding.companyId}/runner-api/%`}), 0)
+ coalesce((select sum(${runnerApiResponseReservations.reservedBytes}) from ${runnerApiResponseReservations}
where ${runnerApiResponseReservations.companyId} = ${this.binding.companyId}
and ${runnerApiResponseReservations.assetId} is null), 0)` }).from(heartbeatRuns).where(eq(heartbeatRuns.id, this.binding.runId));
const companyBytes = Number(usage.bytes);
if (!Number.isSafeInteger(companyBytes) || companyBytes < 0 || companyBytes + RUNNER_API_RESPONSE_MAX_BYTES > runnerApiCompanyCaptureMaxBytes()) {
throw new RunnerApiResponseLimitError("api_response_company_storage_limit", "Company API snapshot storage quota is exhausted. Read an existing asset or ask the operator to remove old snapshots or increase the quota.");
}
// Reserve worst-case bytes durably before creating a temporary file.
// Interrupted/failed captures keep the reservation, preventing retry
// loops and process restarts from resetting this run's disk-I/O budget.
await tx.update(heartbeatRuns).set({ resultJson: { ...resultJson, apiResponseCaptureBytes: used + RUNNER_API_RESPONSE_MAX_BYTES } }).where(eq(heartbeatRuns.id, this.binding.runId));
const [reservation] = await tx.insert(runnerApiResponseReservations).values({ companyId: this.binding.companyId, runId: this.binding.runId, reservedBytes: RUNNER_API_RESPONSE_MAX_BYTES }).returning({ id: runnerApiResponseReservations.id });
return reservation.id;
});
} catch (error) { release(); throw error; }
let settled = false;
return { id, settle: async (completedBytes?: number, cleaned?: boolean, storageAttempted?: boolean) => {
if (settled) return;
settled = true;
try {
await this.db.transaction(async tx => {
await this.#lockApiCaptureCompany(tx as unknown as Db);
// Reclaim a handled pre-storage failure only after disk cleanup.
// Crashes, failed cleanup, and ambiguous storage failures keep their
// reservation until an operator reconciles the possible orphan data.
if (cleaned && !storageAttempted) await tx.delete(runnerApiResponseReservations).where(and(eq(runnerApiResponseReservations.id, id), isNull(runnerApiResponseReservations.assetId)));
if (completedBytes === undefined) return;
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, this.binding.runId), eq(heartbeatRuns.companyId, this.binding.companyId))).for("update");
if (!run) return;
const resultJson = record(run.resultJson);
const used = resultJson.apiResponseCaptureBytes;
if (typeof used !== "number" || !Number.isSafeInteger(used) || used < RUNNER_API_RESPONSE_MAX_BYTES) return;
await tx.update(heartbeatRuns).set({ resultJson: { ...resultJson, apiResponseCaptureBytes: used - RUNNER_API_RESPONSE_MAX_BYTES + completedBytes } }).where(eq(heartbeatRuns.id, run.id));
});
} finally { release(); }
} };
}
async #readApiFile(file: RunnerApiFile): Promise<{ bytes: Buffer; filename: string; contentType: string }> {
if (file.artifactId) {
const asset = await assetService(this.db).getById(file.artifactId);
@@ -1,8 +1,11 @@
import { z } from "zod";
import { badRequest, forbidden, unprocessable } from "../../errors.js";
import { runnerApiOperation, type RunnerApiOperation } from "./runner-api-catalog.js";
import { captureRunnerApiResponse, type RunnerApiResponseBody } from "./runner-api-response.js";
import { runnerApiRestriction } from "./runner-api-policy.js";
import { RunnerApiResponseLimitError, RunnerApiResponseCleanupError, RUNNER_API_RESPONSE_MAX_BYTES } from "./runner-api-response-limits.js";
/** Request/upload bound only. Response capture has a separate streamed limit. */
export const RUNNER_API_MAX_BYTES = 10 * 1024 * 1024;
export const RUNNER_API_INLINE_BYTES = 24 * 1024;
export const RUNNER_API_TIMEOUT_MS = 30_000;
@@ -18,6 +21,10 @@ export const runnerApiCallSchema = z.object({
body: z.unknown().optional(),
contentType: z.string().max(120).optional(),
files: z.array(fileSchema).max(10).optional(),
responseText: z.object({
offsetBytes: z.number().int().min(0).max(Number.MAX_SAFE_INTEGER).optional(),
limitBytes: z.number().int().min(4).max(RUNNER_API_INLINE_BYTES).optional(),
}).strict().optional(),
}).strict();
export type RunnerApiCall = z.infer<typeof runnerApiCallSchema>;
export type RunnerApiFile = z.infer<typeof fileSchema>;
@@ -34,8 +41,10 @@ export interface RunnerApiIo {
fetch?: typeof fetch;
/** Revalidate server-owned authority after asynchronous file preparation. */
beforeDispatch?(): Promise<void>;
/** Reserve before spilling. Settle completed bytes; failed captures retain their reservation. */
reserveResponseCapture?(): Promise<(completedBytes?: number, cleanupSucceeded?: boolean) => Promise<void>>;
readFile(file: RunnerApiFile): Promise<{ bytes: Buffer; filename: string; contentType: string }>;
saveResponse(bytes: Buffer, contentType: string): Promise<Record<string, unknown>>;
saveResponse(bytes: RunnerApiResponseBody, contentType: string): Promise<Record<string, unknown>>;
}
export function validateRunnerApiCall(value: unknown, context: RunnerApiContext): { input: RunnerApiCall; operation: RunnerApiOperation } {
@@ -44,6 +53,7 @@ export function validateRunnerApiCall(value: unknown, context: RunnerApiContext)
const input = parsed.data;
const operation = runnerApiOperation(input.operationId);
if (operation.transport !== "rest") throw unprocessable("This endpoint requires its existing protocol client; call_api supports REST only");
if (input.responseText && operation.method !== "GET") throw badRequest("responseText requires a GET operation; read the saved artifact instead of repeating a mutation");
const restriction = runnerApiRestriction(operation.method, operation.path);
if (restriction) throw forbidden(restriction);
if (!operation.allowedModes.includes(context.workMode)) throw forbidden("call_api permits only reads in Ask and Plan modes; use the permitted dedicated tools");
@@ -139,10 +149,19 @@ export async function readBoundedResponse(response: Response, maxBytes = RUNNER_
}
export async function executeRunnerApi(input: RunnerApiCall, context: RunnerApiContext, io: RunnerApiIo) {
const { operation } = validateRunnerApiCall(input, context);
const { operation, input: validated } = validateRunnerApiCall(input, context);
input = validated;
const url = runnerApiUrl(operation, input, context, io.apiUrl);
if (!io.token) throw new Error("Paperclip run authentication is unavailable");
const headers = new Headers({ Authorization: `Bearer ${io.token}`, "X-Paperclip-Run-Id": context.runId });
const textOffset = input.responseText?.offsetBytes ?? 0;
const textLimit = input.responseText?.limitBytes ?? RUNNER_API_INLINE_BYTES;
const assetTextRange = operation.operationId === "GET /api/assets/{assetId}/content"
? { start: Math.max(0, textOffset - 1), end: Math.min(Number.MAX_SAFE_INTEGER, textOffset + textLimit) }
: undefined;
// One byte before the requested offset permits an EOF read; one byte after
// the page lets the decoder trim an incomplete UTF-8 code point.
if (assetTextRange) headers.set("Range", `bytes=${assetTextRange.start}-${assetTextRange.end}`);
let body: BodyInit | undefined;
const contentType = input.contentType ?? (input.files?.length ? "multipart/form-data" : "application/json");
if (/\r|\n/.test(contentType)) throw badRequest("Invalid content type");
@@ -177,29 +196,118 @@ export async function executeRunnerApi(input: RunnerApiCall, context: RunnerApiC
}
let response: Response;
let bytes: Buffer;
let captured: Awaited<ReturnType<typeof captureRunnerApiResponse>> | undefined;
let settleCapture: ((completedBytes?: number, cleanupSucceeded?: boolean) => Promise<void>) | undefined;
const disposeCapture = async (priorCleanupSucceeded = true) => {
let cleaned = false;
try { await captured?.dispose(); cleaned = priorCleanupSucceeded; }
finally { await settleCapture?.(captured?.byteSize, cleaned); }
};
const saveResponse = async (body: RunnerApiResponseBody, type: string) => {
// Small binary responses also create assets and need storage admission.
settleCapture ??= await io.reserveResponseCapture?.();
return io.saveResponse(body, type);
};
await io.beforeDispatch?.();
const controller = new AbortController();
// Bound connection setup and stalled body reads, not the total download time.
let timeout = setTimeout(() => controller.abort(), RUNNER_API_TIMEOUT_MS);
try {
response = await (io.fetch ?? fetch)(url, { method: operation.method, headers, body, redirect: "manual", signal: AbortSignal.timeout(RUNNER_API_TIMEOUT_MS) });
bytes = await readBoundedResponse(response);
} catch {
return { ok: false, status: null, operationId: operation.operationId, error: "api_transport_failure", outcome: ["GET", "HEAD", "OPTIONS"].includes(operation.method) ? "read_failed" : "unknown", guidance: "Inspect current state before retrying a mutation; it may already have succeeded." };
}
const type = response.headers.get("content-type") ?? "application/octet-stream";
const mutation = !["GET", "HEAD", "OPTIONS"].includes(operation.method);
// A server can commit a write before an error response or redirect. Preserve
// the HTTP evidence, but never describe that write as definitely not applied.
const uncertain = mutation && (response.status >= 500 || response.status === 408 || (response.status >= 300 && response.status < 400));
const uncertainty = { outcome: "unknown", guidance: "Inspect current state before retrying this mutation; it may already have succeeded." };
const base = { ok: response.ok, status: response.status, operationId: operation.operationId, contentType: type, retryAfter: response.headers.get("retry-after"), ...(uncertain ? uncertainty : {}) };
if (response.status >= 300 && response.status < 400) return { ...base, ok: false, error: "api_redirect_not_followed" };
if (!bytes.length) return { ...base, data: null };
if (bytes.length > RUNNER_API_INLINE_BYTES || !/json|^text\//i.test(type)) {
return { ...base, artifact: await io.saveResponse(bytes, type), byteSize: bytes.length, preview: /json|^text\//i.test(type) ? bytes.subarray(0, 2000).toString("utf8") : null };
}
const text = bytes.toString("utf8");
if (/json/i.test(type)) {
try { return { ...base, data: JSON.parse(text) as unknown }; }
catch { return { ...base, ok: false, error: "invalid_json_response", data: text, ...(mutation ? uncertainty : {}) }; }
}
return { ...base, data: text };
response = await (io.fetch ?? fetch)(url, { method: operation.method, headers, body, redirect: "manual", signal: controller.signal });
clearTimeout(timeout);
if (assetTextRange && response.ok) {
timeout = setTimeout(() => controller.abort(), RUNNER_API_TIMEOUT_MS);
bytes = await readBoundedResponse(response, textLimit + 2);
} else {
captured = await captureRunnerApiResponse(response, RUNNER_API_INLINE_BYTES, controller, RUNNER_API_TIMEOUT_MS, io.beforeDispatch, {
beforeSpill: async () => { settleCapture = await io.reserveResponseCapture?.(); },
});
// Read only the requested window (+ one UTF-8 lookahead byte) into memory.
bytes = await captured.read(input.responseText && response.ok ? textOffset : 0,
input.responseText ? textLimit + 1 : RUNNER_API_INLINE_BYTES);
}
} catch (error) {
await disposeCapture(!(error instanceof RunnerApiResponseCleanupError));
return { ok: false, status: null, operationId: operation.operationId,
error: error instanceof RunnerApiResponseLimitError ? error.code : "api_transport_failure",
...(error instanceof RunnerApiResponseLimitError ? { message: error.message, maxResponseBytes: RUNNER_API_RESPONSE_MAX_BYTES } : {}),
outcome: ["GET", "HEAD", "OPTIONS"].includes(operation.method) ? "read_failed" : "unknown", guidance: "Inspect current state before retrying a mutation; it may already have succeeded." };
} finally { clearTimeout(timeout); }
try {
const type = response.headers.get("content-type") ?? "application/octet-stream";
const mutation = !["GET", "HEAD", "OPTIONS"].includes(operation.method);
// A server can commit a write before an error response or redirect. Preserve
// the HTTP evidence, but never describe that write as definitely not applied.
const uncertain = mutation && (response.status >= 500 || response.status === 408 || (response.status >= 300 && response.status < 400));
const uncertainty = { outcome: "unknown", guidance: "Inspect current state before retrying this mutation; it may already have succeeded." };
const base = { ok: response.ok, status: response.status, operationId: operation.operationId, contentType: type, retryAfter: response.headers.get("retry-after"), ...(uncertain ? uncertainty : {}) };
if (response.status >= 300 && response.status < 400) return { ...base, ok: false, error: "api_redirect_not_followed" };
let totalBytes = captured?.byteSize ?? bytes.length;
let baseOffset = captured && response.ok ? textOffset : 0;
if (assetTextRange && response.ok) {
const range = /^bytes (\d+)-(\d+)\/(\d+)$/.exec(response.headers.get("content-range") ?? "");
if (response.status === 200 && bytes.length === 0 && textOffset === 0) {
// An empty asset has no satisfiable byte range; its route returns 200.
} else if (response.status !== 206 || !range) {
return { ...base, ok: false, error: "response_text_invalid_range" };
} else {
const [start, end, total] = range.slice(1).map(Number);
if (![start, end, total].every(Number.isSafeInteger) || start !== assetTextRange.start
|| total < 1 || end !== Math.min(assetTextRange.end, total - 1)
|| end < start || bytes.length !== end - start + 1) {
return { ...base, ok: false, error: "response_text_invalid_range" };
}
baseOffset = start;
totalBytes = total;
}
}
if (input.responseText) {
if (!/json|^text\//i.test(type)) return { ...base, ok: false, error: "response_text_requires_text_content" };
// HTTP denials are evidence, not text pages of an authorized asset.
const offsetBytes = response.ok ? textOffset : 0;
const localOffset = offsetBytes - baseOffset;
if (offsetBytes > totalBytes || localOffset < 0 || (localOffset < bytes.length && (bytes[localOffset] & 0xc0) === 0x80)) {
return { ...base, ok: false, error: "response_text_invalid_offset", byteSize: totalBytes };
}
let end = Math.min(bytes.length, localOffset + textLimit);
// Do not cut a UTF-8 code point. The next offset always starts a whole one.
while (end < bytes.length && end > localOffset && (bytes[end] & 0xc0) === 0x80) end--;
if (end === localOffset && end < bytes.length) return { ...base, ok: false, error: "response_text_invalid_utf8" };
let data: string;
try {
data = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }).decode(bytes.subarray(localOffset, end));
} catch {
return { ...base, ok: false, error: "response_text_invalid_utf8" };
}
const nextOffset = baseOffset + end;
const artifact = captured && captured.byteSize > RUNNER_API_INLINE_BYTES
? await saveResponse(captured.body, type) : undefined;
return { ...base, data, ...(artifact ? { artifact, guidance: "Continue reading this saved artifact with GET /api/assets/{assetId}/content and responseText.nextOffsetBytes for a stable snapshot." } : {}), responseText: { offsetBytes, nextOffsetBytes: nextOffset < totalBytes ? nextOffset : null, totalBytes } };
}
if (assetTextRange && response.ok && (totalBytes > RUNNER_API_INLINE_BYTES || !/json|^text\//i.test(type))) {
// This content is already an authorized, stable asset. Return its existing
// reference after a bounded preview; never download and duplicate it.
const assetId = input.pathParams!.assetId;
const sha256 = /^"([a-f0-9]{64})"$/.exec(response.headers.get("etag") ?? "")?.[1];
return { ...base, artifact: { artifactId: assetId, url: `/api/assets/${assetId}/content`, contentType: type, byteSize: totalBytes, ...(sha256 ? { sha256 } : {}) },
byteSize: totalBytes, preview: /json|^text\//i.test(type) ? bytes.subarray(0, 2000).toString("utf8") : null,
guidance: "Read this asset with responseText: { offsetBytes: 0 }. Continue with responseText.nextOffsetBytes until null." };
}
if (!totalBytes) return { ...base, data: null };
if (totalBytes > RUNNER_API_INLINE_BYTES || !/json|^text\//i.test(type)) {
return { ...base, artifact: await saveResponse(captured?.body ?? bytes, type), byteSize: totalBytes, preview: /json|^text\//i.test(type) ? bytes.subarray(0, 2000).toString("utf8") : null,
...(/json|^text\//i.test(type) ? { guidance: "Read the saved artifact with GET /api/assets/{assetId}/content and responseText: { offsetBytes: 0 }. Continue with responseText.nextOffsetBytes until null. Do not repeat a mutation to read its response." } : {}) };
}
const text = bytes.toString("utf8");
if (/json/i.test(type)) {
try { return { ...base, data: JSON.parse(text) as unknown }; }
catch { return { ...base, ok: false, error: "invalid_json_response", data: text, ...(mutation ? uncertainty : {}) }; }
}
return { ...base, data: text };
} catch (error) {
if (!(error instanceof RunnerApiResponseLimitError)) throw error;
return { ok: false, status: response.status, operationId: operation.operationId, error: error.code, message: error.message,
outcome: ["GET", "HEAD", "OPTIONS"].includes(operation.method) ? "read_failed" : "unknown",
guidance: "Inspect current state before retrying a mutation; it may already have succeeded." };
} finally { await disposeCapture(); }
}
@@ -0,0 +1,39 @@
export const RUNNER_API_RESPONSE_MAX_BYTES = 1024 * 1024 * 1024;
export const RUNNER_API_RESPONSE_RUN_MAX_BYTES = 4 * RUNNER_API_RESPONSE_MAX_BYTES;
export const RUNNER_API_RESPONSE_DEADLINE_MS = 10 * 60_000;
export function runnerApiCompanyCaptureMaxBytes(): number {
const configured = Number(process.env.PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES);
return Number.isSafeInteger(configured) && configured >= RUNNER_API_RESPONSE_MAX_BYTES
? configured : 20 * RUNNER_API_RESPONSE_MAX_BYTES;
}
/** Cleanup failure must keep the durable reservation for possible orphan bytes. */
export class RunnerApiResponseCleanupError extends Error {
constructor(cause: unknown) { super("Failed to remove captured API response", { cause }); }
}
export class RunnerApiResponseLimitError extends Error {
constructor(readonly code: string, message: string) { super(message); }
}
// Hold a slot until the temporary file is removed, including storage upload.
// These limits are per server process; the run byte budget is database-backed.
const activeByCompany = new Map<string, number>();
let active = 0;
export function acquireRunnerApiResponseSlot(companyId: string): () => void {
const companyActive = activeByCompany.get(companyId) ?? 0;
if (active >= 4 || companyActive >= 2) {
throw new RunnerApiResponseLimitError("api_response_capture_busy", "Large API response capture is busy. Wait for another capture to finish before retrying a read.");
}
active++;
activeByCompany.set(companyId, companyActive + 1);
let released = false;
return () => {
if (released) return;
released = true;
active--;
const remaining = activeByCompany.get(companyId)! - 1;
if (remaining) activeByCompany.set(companyId, remaining);
else activeByCompany.delete(companyId);
};
}
@@ -0,0 +1,142 @@
import { createHash } from "node:crypto";
import { access, readdir } from "node:fs/promises";
import { tmpdir } from "node:os";
import { describe, expect, it, vi } from "vitest";
import { captureRunnerApiResponse } from "./runner-api-response.js";
import { acquireRunnerApiResponseSlot, runnerApiCompanyCaptureMaxBytes, RUNNER_API_RESPONSE_MAX_BYTES } from "./runner-api-response-limits.js";
const controller = () => new AbortController();
describe("streamed API response capture", () => {
it("rejects an oversized Content-Length before reading or reserving disk", async () => {
let cancelled = false;
let reserved = false;
const abort = controller();
const response = new Response(new ReadableStream({ cancel() { cancelled = true; } }), { headers: { "content-length": String(RUNNER_API_RESPONSE_MAX_BYTES + 1) } });
await expect(captureRunnerApiResponse(response, 24 * 1024, abort, 1000, undefined, {
beforeSpill: async () => { reserved = true; },
})).rejects.toMatchObject({ code: "api_response_too_large" });
expect(cancelled).toBe(true);
expect(abort.signal.aborted).toBe(true);
expect(reserved).toBe(false);
});
it.each([undefined, "1"])("counts streamed bytes with Content-Length %s and removes partial files", async contentLength => {
const before = new Set(await readdir(tmpdir()));
let cancelled = false;
let reserved = 0;
const response = new Response(new ReadableStream({
pull(target) { target.enqueue(Buffer.alloc(4096)); },
cancel() { cancelled = true; },
}), { headers: contentLength ? { "content-length": contentLength } : {} });
await expect(captureRunnerApiResponse(response, 4096, controller(), 1000, undefined, {
maxBytes: 16 * 1024,
beforeSpill: async () => { reserved++; },
})).rejects.toMatchObject({ code: "api_response_too_large" });
expect(cancelled).toBe(true);
expect(reserved).toBe(1);
expect((await readdir(tmpdir())).filter(name => name.startsWith("paperclip-api-response-") && !before.has(name))).toEqual([]);
});
it("accepts exactly the capture limit", async () => {
const captured = await captureRunnerApiResponse(new Response(Buffer.alloc(8192)), 4096, controller(), 1000, undefined, { maxBytes: 8192 });
try { expect(captured.byteSize).toBe(8192); }
finally { await captured.dispose(); }
});
it("stops an active endless stream at the total deadline", async () => {
let cancelled = false;
const response = new Response(new ReadableStream({
async pull(target) {
await new Promise(resolve => setTimeout(resolve, 5));
if (!cancelled) target.enqueue(Buffer.alloc(4096));
},
cancel() { cancelled = true; },
}));
await expect(captureRunnerApiResponse(response, 4096, controller(), 1000, undefined, { deadlineMs: 30 })).rejects.toMatchObject({ code: "api_response_timeout" });
expect(cancelled).toBe(true);
});
it("spills a chunked response and reads only requested bytes, preserving its digest", async () => {
const chunk = Buffer.alloc(64 * 1024, 65);
const digest = createHash("sha256");
let emitted = 0;
const response = new Response(new ReadableStream({
pull(target) {
if (emitted++ === 200) return target.close();
digest.update(chunk);
target.enqueue(chunk);
},
}));
const captured = await captureRunnerApiResponse(response, 24 * 1024, controller(), 1000);
expect(Buffer.isBuffer(captured.body)).toBe(false);
if (Buffer.isBuffer(captured.body)) throw new Error("Expected a file");
const path = captured.body.path;
try {
expect(captured.body).toMatchObject({ byteSize: 200 * chunk.length, sha256: digest.digest("hex") });
expect(await captured.read(11 * 1024 * 1024, 4)).toEqual(Buffer.from("AAAA"));
expect(await captured.read(captured.byteSize, 4)).toHaveLength(0);
} finally { await captured.dispose(); }
await expect(access(path)).rejects.toThrow();
});
it("keeps small bodies inline", async () => {
const captured = await captureRunnerApiResponse(new Response("small"), 24 * 1024, controller(), 1000);
try { expect(captured.body).toEqual(Buffer.from("small")); }
finally { await captured.dispose(); }
});
it("stops a long capture when its run loses authority", async () => {
let checks = 0;
let cancelled = false;
const response = new Response(new ReadableStream({
pull(target) { target.enqueue(Buffer.alloc(64 * 1024)); },
cancel() { cancelled = true; },
}));
await expect(captureRunnerApiResponse(response, 24 * 1024, controller(), 1000, async () => {
if (++checks === 2) throw new Error("run stopped");
})).rejects.toThrow("run stopped");
expect(cancelled).toBe(true);
expect(checks).toBe(2);
});
it("allows active downloads longer than the idle timeout", async () => {
let chunks = 0;
const response = new Response(new ReadableStream({
async pull(target) {
await new Promise(resolve => setTimeout(resolve, 20));
if (chunks++ === 5) target.close();
else target.enqueue(Buffer.from("a"));
},
}));
const captured = await captureRunnerApiResponse(response, 24 * 1024, controller(), 100);
try { expect(captured.body).toEqual(Buffer.from("aaaaa")); }
finally { await captured.dispose(); }
});
it("cancels stalled bodies and aborts the HTTP request", async () => {
let cancelled = false;
const response = new Response(new ReadableStream({ cancel() { cancelled = true; } }));
const abort = controller();
await expect(captureRunnerApiResponse(response, 24 * 1024, abort, 10)).rejects.toThrow("timed out");
expect(cancelled).toBe(true);
expect(abort.signal.aborted).toBe(true);
});
});
describe("large response concurrency", () => {
it("only accepts finite positive operator quotas, never unlimited settings", () => {
try {
for (const value of ["0", "-1", "Infinity", "NaN", "1", "9007199254740992"]) {
vi.stubEnv("PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES", value);
expect(runnerApiCompanyCaptureMaxBytes()).toBe(20 * RUNNER_API_RESPONSE_MAX_BYTES);
}
vi.stubEnv("PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES", String(2 * RUNNER_API_RESPONSE_MAX_BYTES));
expect(runnerApiCompanyCaptureMaxBytes()).toBe(2 * RUNNER_API_RESPONSE_MAX_BYTES);
} finally { vi.unstubAllEnvs(); }
});
it("limits each company and the server and releases slots idempotently", () => {
const releases: Array<() => void> = [];
try {
releases.push(acquireRunnerApiResponseSlot("a"), acquireRunnerApiResponseSlot("a"));
expect(() => acquireRunnerApiResponseSlot("a")).toThrow("busy");
releases.push(acquireRunnerApiResponseSlot("b"), acquireRunnerApiResponseSlot("b"));
expect(() => acquireRunnerApiResponseSlot("c")).toThrow("busy");
releases[0](); releases[0]();
releases.push(acquireRunnerApiResponseSlot("c"));
expect(() => acquireRunnerApiResponseSlot("d")).toThrow("busy");
} finally { for (const release of releases) release(); }
acquireRunnerApiResponseSlot("a")();
});
});
@@ -0,0 +1,113 @@
import { createHash } from "node:crypto";
import { mkdtemp, open, rm, type FileHandle } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { RUNNER_API_RESPONSE_MAX_BYTES, RUNNER_API_RESPONSE_DEADLINE_MS, RunnerApiResponseLimitError, RunnerApiResponseCleanupError } from "./runner-api-response-limits.js";
export type RunnerApiResponseBody = Buffer | { path: string; byteSize: number; sha256: string };
/** Spill bounded responses to a private temporary file.
* Only the inline prefix and one HTTP chunk are held in memory. The caller must
* dispose the capture after saving it or reading its requested text window.
*/
export async function captureRunnerApiResponse(response: Response, inlineBytes: number, controller: AbortController, idleTimeoutMs: number, checkAuthority?: () => Promise<void>, options: {
maxBytes?: number;
deadlineMs?: number;
beforeSpill?: () => Promise<void>;
} = {}) {
const maxBytes = options.maxBytes ?? RUNNER_API_RESPONSE_MAX_BYTES;
const deadline = Date.now() + (options.deadlineMs ?? RUNNER_API_RESPONSE_DEADLINE_MS);
const reader = response.body?.getReader();
let directory: string | undefined;
let file: FileHandle | undefined;
let path: string | undefined;
let byteSize = 0;
const prefix: Buffer[] = [];
const hash = createHash("sha256");
const dispose = async () => {
try {
try { await file?.close(); }
finally {
file = undefined;
if (directory) await rm(directory, { recursive: true, force: true });
}
} catch (error) { throw new RunnerApiResponseCleanupError(error); }
};
try {
if (Number(response.headers.get("content-length")) > maxBytes) {
throw new RunnerApiResponseLimitError("api_response_too_large", `API response exceeds the ${maxBytes}-byte capture limit. Narrow the query or use the endpoint's pagination.`);
}
let checkedAt = Date.now();
let checkedBytes = 0;
await checkAuthority?.();
if (reader) {
while (true) {
controller.signal.throwIfAborted();
if (Date.now() >= deadline) throw new RunnerApiResponseLimitError("api_response_timeout", "API response exceeded its capture deadline.");
let timeout: ReturnType<typeof setTimeout> | undefined;
const next = await Promise.race([
reader.read(),
new Promise<never>((_, reject) => {
timeout = setTimeout(() => {
controller.abort();
reject(new RunnerApiResponseLimitError("api_response_timeout", "API response read timed out or exceeded its capture deadline."));
}, Math.min(idleTimeoutMs, Math.max(0, deadline - Date.now())));
}),
]).finally(() => clearTimeout(timeout));
if (next.done) break;
// Count decoded bytes before copying, hashing, or writing the chunk.
// Content-Length can be missing, false, or describe compressed bytes.
if (next.value.byteLength > maxBytes - byteSize) {
throw new RunnerApiResponseLimitError("api_response_too_large", `API response exceeds the ${maxBytes}-byte capture limit. Narrow the query or use the endpoint's pagination.`);
}
const chunk = Buffer.from(next.value);
if (!Number.isSafeInteger(byteSize + chunk.length)) throw new Error("API response byte offset exceeds numeric precision");
hash.update(chunk);
if (!file && byteSize + chunk.length > inlineBytes) {
await options.beforeSpill?.();
directory = await mkdtemp(join(tmpdir(), "paperclip-api-response-"));
path = join(directory, "body");
file = await open(path, "wx+", 0o600);
for (const part of prefix) await file.writeFile(part);
prefix.length = 0;
}
if (file) await file.writeFile(chunk);
else prefix.push(chunk);
byteSize += chunk.length;
if (checkAuthority && (Date.now() - checkedAt >= 1000 || byteSize - checkedBytes >= 1024 * 1024)) {
await checkAuthority();
checkedAt = Date.now();
checkedBytes = byteSize;
}
}
}
if (Date.now() >= deadline) throw new RunnerApiResponseLimitError("api_response_timeout", "API response exceeded its capture deadline.");
await checkAuthority?.();
const bytes = file ? undefined : Buffer.concat(prefix);
const body: RunnerApiResponseBody = path ? { path, byteSize, sha256: hash.digest("hex") } : bytes!;
return {
body,
byteSize,
async read(offset: number, limit: number): Promise<Buffer> {
const length = Math.min(limit, Math.max(0, byteSize - offset));
if (!file) return bytes!.subarray(offset, offset + length);
const result = Buffer.alloc(length);
let read = 0;
while (read < length) {
const next = await file.read(result, read, length - read, offset + read);
if (!next.bytesRead) throw new Error("Saved API response was truncated");
read += next.bytesRead;
}
return result;
},
dispose,
};
} catch (error) {
controller.abort();
await dispose();
throw error;
} finally {
await reader?.cancel().catch(() => {});
reader?.releaseLock();
}
}
@@ -1,14 +1,16 @@
import { existsSync } from "node:fs";
import { randomUUID } from "node:crypto";
import { chmod, writeFile, symlink } from "node:fs/promises";
import { join } from "node:path";
import { chmod, writeFile, symlink, mkdir, open } from "node:fs/promises";
import { join, dirname } from "node:path";
import { eq } from "drizzle-orm";
import { documents, heartbeatRuns, issues, routineDocuments, routines } from "@paperclipai/db";
import { assets, documents, heartbeatRuns, issues, projects, routineDocuments, routines, runnerApiResponseReservations } from "@paperclipai/db";
import { beforeAll, afterAll, beforeEach, afterEach, describe, expect, it, vi } from "vitest";
import { startRunnerApiTestServer } from "../../__tests__/helpers/runner-api-server.js";
import { createRunnerdCodexTransport, defaultCapabilityRunnerdBinary } from "../../vendor/paperclip-runner/index.js";
import { runnerApiCatalog } from "./runner-api-catalog.js";
import { registerRunnerPrpAuthority } from "../../realtime/runner-prp-ws.js";
import { RUNNER_API_RESPONSE_MAX_BYTES, RUNNER_API_RESPONSE_RUN_MAX_BYTES } from "./runner-api-response-limits.js";
import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js";
describe("runner API against real HTTP routes", () => {
let server: Awaited<ReturnType<typeof startRunnerApiTestServer>>;
@@ -248,6 +250,243 @@ else if(m.id!==undefined) send({id:m.id,result:{}});
} finally { getObject.mockRestore(); }
});
it("retrieves every byte of a saved large response without creating an artifact loop", async () => {
const fixture = await server.fixture();
const foreign = await server.fixture();
const text = JSON.stringify({ padding: "🧭é".repeat(6000), evidence: "last-field-is-readable" });
const stored = await server.storage.putFile({ companyId: fixture.companyId, namespace: "eval", originalFilename: "evidence.json", contentType: "application/json", body: Buffer.from(text) });
const [asset] = await server.db.insert(assets).values({ ...stored, companyId: fixture.companyId, createdByAgentId: fixture.agentId }).returning();
const call = (authority: typeof fixture.authority, assetId: string, responseText?: object) => authority.execute({ tool: "call_api", callId: randomUUID(), arguments: { operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId }, ...(responseText ? { responseText } : {}) } }) as Promise<any>;
const initial = await call(fixture.authority, asset.id);
expect(initial).toMatchObject({ status: 206, artifact: { artifactId: asset.id, sha256: asset.sha256, byteSize: Buffer.byteLength(text) } });
const before = (await fixture.snapshot()).assets.length;
const reads = vi.spyOn(server.storage, "getObject");
let offsetBytes = 0;
let result = "";
try {
do {
const page = await call(fixture.authority, initial.artifact.artifactId, { offsetBytes, limitBytes: 4096 });
expect(page).toMatchObject({ status: 206, responseText: { offsetBytes, totalBytes: Buffer.byteLength(text) } });
result += page.data;
offsetBytes = page.responseText.nextOffsetBytes;
} while (offsetBytes !== null);
expect(JSON.parse(result)).toEqual(JSON.parse(text));
expect(reads.mock.calls.length).toBeGreaterThan(1);
expect(reads.mock.calls.every(([, , options]) => options?.range)).toBe(true);
const storageBytes = reads.mock.calls.reduce((total, [, , options]) => total + options!.range!.end - options!.range!.start + 1, 0);
expect(storageBytes).toBeLessThan(Buffer.byteLength(text) + reads.mock.calls.length * 5);
expect((await fixture.snapshot()).assets).toHaveLength(before);
expect(await call(foreign.authority, initial.artifact.artifactId, { offsetBytes: 0 })).toMatchObject({ ok: false, status: 404 });
} finally {
reads.mockRestore();
}
});
it("captures a live response above ten MiB and reads its saved snapshot to EOF", async () => {
const fixture = await server.fixture();
const foreign = await server.fixture();
const description = "A".repeat(12 * 1024 * 1024) + "END-OF-LARGE-RESPONSE";
await server.db.update(projects).set({ description }).where(eq(projects.id, fixture.projectId));
const initial: any = await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId }, responseText: { limitBytes: 8192 },
} });
expect(initial).toMatchObject({ ok: true, status: 200, responseText: { offsetBytes: 0, nextOffsetBytes: 8192 } });
expect(initial.artifact.byteSize).toBeGreaterThan(12 * 1024 * 1024);
const saved = await server.db.select().from(assets).where(eq(assets.id, initial.artifact.artifactId)).then(rows => rows[0]);
const object = await server.storage.getObject(fixture.companyId, saved.objectKey);
const chunks: Buffer[] = [];
for await (const chunk of object.stream) chunks.push(chunk);
const expected = Buffer.concat(chunks);
expect(JSON.parse(expected.toString()).description).toBe(description);
const before = (await fixture.snapshot()).assets.length;
for (const offsetBytes of [0, 10 * 1024 * 1024 + 1, expected.length - 8192, expected.length]) {
const page: any = await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: saved.id }, responseText: { offsetBytes, limitBytes: 8192 },
} });
expect(page).toMatchObject({ ok: true, status: 206, data: expected.subarray(offsetBytes, offsetBytes + 8192).toString(),
responseText: { offsetBytes, totalBytes: expected.length, nextOffsetBytes: offsetBytes + 8192 < expected.length ? offsetBytes + 8192 : null } });
expect(page.artifact).toBeUndefined();
}
expect((await fixture.snapshot()).assets).toHaveLength(before);
expect(await foreign.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: saved.id }, responseText: { offsetBytes: 10 * 1024 * 1024 + 1 },
} })).toMatchObject({ ok: false, status: 404 });
const mutation = { tool: "call_api", callId: randomUUID(), arguments: {
operationId: "PATCH /api/projects/{id}", pathParams: { id: fixture.projectId }, body: { name: "Updated once" },
} };
const receipt: any = await fixture.authority.execute(mutation);
expect(receipt).toMatchObject({ ok: true, status: 200, artifact: { contentType: "application/json; charset=utf-8" } });
expect(receipt.artifact.byteSize).toBeGreaterThan(12 * 1024 * 1024);
expect(await fixture.authority.execute(mutation)).toEqual(receipt);
const after = await fixture.snapshot();
expect(after.assets).toHaveLength(before + 1);
expect(after.activity.filter(row => row.action === "project.updated")).toHaveLength(1);
const [run] = await server.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
expect((run.resultJson as Record<string, unknown>).apiResponseCaptureBytes).toBe(initial.artifact.byteSize + receipt.artifact.byteSize);
}, 30_000);
it("enforces the durable run budget while permitting small reads and saved pages", async () => {
const fixture = await server.fixture();
await server.db.update(heartbeatRuns).set({ resultJson: { apiResponseCaptureBytes: RUNNER_API_RESPONSE_RUN_MAX_BYTES } }).where(eq(heartbeatRuns.id, fixture.runId));
const call = () => fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
} });
expect(await call()).toMatchObject({ ok: true });
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
const before = (await fixture.snapshot()).assets.length;
expect(await call()).toMatchObject({ ok: false, error: "api_response_capture_limit", outcome: "read_failed" });
expect((await fixture.snapshot()).assets).toHaveLength(before);
expect(await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: fixture.artifactId }, responseText: { limitBytes: 4 },
} })).toMatchObject({ ok: true });
});
it("reserves the run budget atomically across simultaneous captures", async () => {
const fixture = await server.fixture();
const used = RUNNER_API_RESPONSE_RUN_MAX_BYTES - RUNNER_API_RESPONSE_MAX_BYTES;
await server.db.update(heartbeatRuns).set({ resultJson: { apiResponseCaptureBytes: used } }).where(eq(heartbeatRuns.id, fixture.runId));
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
let unblock!: () => void;
let started!: () => void;
const gate = new Promise<void>(resolve => { unblock = resolve; });
const saving = new Promise<void>(resolve => { started = resolve; });
const putFile = server.storage.putFile.bind(server.storage);
const spy = vi.spyOn(server.storage, "putFile").mockImplementation(async input => {
if (input.namespace === "runner-api") { started(); await gate; }
return putFile(input);
});
const call = () => fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
} });
const first = call();
try {
await saving;
expect(await call()).toMatchObject({ ok: false, error: "api_response_capture_limit" });
} finally { unblock(); spy.mockRestore(); }
const result: any = await first;
expect(result).toMatchObject({ ok: true });
const [run] = await server.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
expect((run.resultJson as Record<string, unknown>).apiResponseCaptureBytes).toBe(used + result.artifact.byteSize);
expect(await call()).toMatchObject({ ok: false, error: "api_response_capture_limit" });
});
it("counts old snapshots and other-run reservations against the company quota, and reclaims deletions", async () => {
const fixture = await server.fixture();
vi.stubEnv("PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES", String(2 * RUNNER_API_RESPONSE_MAX_BYTES));
const [oldSnapshot] = await server.db.insert(assets).values({ companyId: fixture.companyId, provider: "local_disk",
objectKey: `${fixture.companyId}/runner-api/old-snapshot`, contentType: "text/plain", byteSize: RUNNER_API_RESPONSE_MAX_BYTES,
sha256: "old-snapshot", createdByAgentId: fixture.agentId }).returning();
const [reservation] = await server.db.insert(runnerApiResponseReservations).values({ companyId: fixture.companyId, runId: null, reservedBytes: RUNNER_API_RESPONSE_MAX_BYTES }).returning();
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
// A fresh authority instance cannot reset reservations from another run/process.
const authority = new PaperclipRunnerToolAuthority(server.db, fixture);
const call = () => authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
} });
expect(await call()).toMatchObject({ ok: false, error: "api_response_company_storage_limit" });
await server.db.delete(assets).where(eq(assets.id, oldSnapshot.id));
const first: any = await call();
expect(first).toMatchObject({ ok: true });
expect(await call()).toMatchObject({ ok: false, error: "api_response_company_storage_limit" });
await server.db.delete(assets).where(eq(assets.id, first.artifact.artifactId));
expect(await server.db.select().from(runnerApiResponseReservations).where(eq(runnerApiResponseReservations.companyId, fixture.companyId))).toEqual([expect.objectContaining({ id: reservation.id })]);
expect(await call()).toMatchObject({ ok: true });
});
it("keeps company storage reserved after an ambiguous upload failure", async () => {
const fixture = await server.fixture();
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
const spy = vi.spyOn(server.storage, "putFile").mockRejectedValueOnce(new Error("storage disconnected"));
try {
await expect(fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
} })).rejects.toThrow("storage disconnected");
} finally { spy.mockRestore(); }
expect(await server.db.select().from(runnerApiResponseReservations).where(eq(runnerApiResponseReservations.companyId, fixture.companyId))).toEqual([
expect.objectContaining({ assetId: null, reservedBytes: RUNNER_API_RESPONSE_MAX_BYTES }),
]);
});
it.each([false, true])("reconciles metadata failure without losing committed assets (commit=%s)", async committed => {
const fixture = await server.fixture();
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
const putFile = server.storage.putFile.bind(server.storage);
const transaction = server.db.transaction.bind(server.db);
let objectKey = "";
let transactionSpy: ReturnType<typeof vi.spyOn> | undefined;
const spy = vi.spyOn(server.storage, "putFile").mockImplementationOnce(async input => {
const saved = await putFile(input);
objectKey = saved.objectKey;
transactionSpy = vi.spyOn(server.db, "transaction").mockImplementationOnce(async (fn, config) => {
if (committed) await transaction(fn, config);
throw new Error("metadata transaction failed");
});
return saved;
});
try {
await expect(fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
} })).rejects.toThrow("metadata transaction failed");
} finally { spy.mockRestore(); transactionSpy?.mockRestore(); }
const reservations = await server.db.select().from(runnerApiResponseReservations).where(eq(runnerApiResponseReservations.companyId, fixture.companyId));
expect(reservations).toHaveLength(committed ? 1 : 0);
if (committed) expect(reservations[0].assetId).toBeTruthy();
expect((await server.storage.headObject(fixture.companyId, objectKey)).exists).toBe(committed);
});
it("shares company admission across simultaneous runs", async () => {
const fixture = await server.fixture();
vi.stubEnv("PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES", String(RUNNER_API_RESPONSE_MAX_BYTES));
const otherRunId = randomUUID();
await server.db.insert(heartbeatRuns).values({ id: otherRunId, companyId: fixture.companyId, agentId: fixture.agentId,
status: "running", runtimeMode: "native", nativeIssueId: fixture.blockerId, contextSnapshot: { issueId: fixture.blockerId } });
await server.db.update(issues).set({ status: "in_progress", executionRunId: otherRunId }).where(eq(issues.id, fixture.blockerId));
const other = new PaperclipRunnerToolAuthority(server.db, { ...fixture, issueId: fixture.blockerId, runId: otherRunId });
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
const arguments_ = { operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId } };
const results: any[] = await Promise.all([fixture.authority, other].map(authority => authority.execute({ tool: "call_api", callId: randomUUID(), arguments: arguments_ })));
expect(results.filter(result => result.ok)).toHaveLength(1);
expect(results.find(result => !result.ok)).toMatchObject({ error: "api_response_company_storage_limit" });
});
it("releases a cleaned failed capture's company reservation but retains its run charge", async () => {
const fixture = await server.fixture();
let emitted = false;
const stream = new ReadableStream({
async pull(target) {
if (!emitted) { emitted = true; target.enqueue(Buffer.alloc(32 * 1024)); }
else { await new Promise(resolve => setTimeout(resolve, 100)); target.error(new Error("connection reset")); }
},
});
const spy = vi.spyOn(globalThis, "fetch").mockResolvedValueOnce(new Response(stream, { headers: { "content-type": "text/plain" } }));
try {
expect(await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
} })).toMatchObject({ ok: false, error: "api_transport_failure" });
} finally { spy.mockRestore(); }
expect(await server.db.select().from(runnerApiResponseReservations).where(eq(runnerApiResponseReservations.companyId, fixture.companyId))).toEqual([]);
const [run] = await server.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
expect((run.resultJson as Record<string, unknown>).apiResponseCaptureBytes).toBe(RUNNER_API_RESPONSE_MAX_BYTES);
});
it("persists and range-reads asset sizes above two GiB", async () => {
const fixture = await server.fixture();
const offsetBytes = 3 * 1024 * 1024 * 1024;
const objectKey = `${fixture.companyId}/eval/sparse-large-response.txt`;
const path = join(server.root, "storage", objectKey);
await mkdir(dirname(path), { recursive: true });
const file = await open(path, "wx");
try { await file.write(Buffer.from("readable"), 0, 8, offsetBytes); }
finally { await file.close(); }
const [asset] = await server.db.insert(assets).values({ companyId: fixture.companyId, provider: "local_disk", objectKey,
contentType: "text/plain", byteSize: offsetBytes + 8, sha256: "sparse-fixture", createdByAgentId: fixture.agentId }).returning();
expect(asset.byteSize).toBe(offsetBytes + 8);
const page = await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: asset.id }, responseText: { offsetBytes, limitBytes: 8 },
} });
expect(page).toMatchObject({ ok: true, data: "readable", responseText: { offsetBytes, totalBytes: offsetBytes + 8, nextOffsetBytes: null } });
});
it("contains workspace files, checks artifact ownership, and persists downloads", async () => {
const fixture = await server.fixture();
const foreign = await server.fixture();
@@ -260,7 +499,7 @@ else if(m.id!==undefined) send({id:m.id,result:{}});
await expect(upload([{ path: "escape.txt" }])).rejects.toThrow();
await expect(upload([{ path: "sample.txt" }])).resolves.toMatchObject({ status: 201 });
const download = await fixture.authority.execute({ tool: "call_api", callId: "download", arguments: { operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: fixture.binaryArtifactId } } }) as any;
expect(download).toMatchObject({ status: 200, byteSize: 32000, artifact: { byteSize: 32000 } });
expect(download).toMatchObject({ status: 206, byteSize: 32000, artifact: { artifactId: fixture.binaryArtifactId, byteSize: 32000 } });
expect((await fixture.snapshot()).assets).toEqual(expect.arrayContaining([expect.objectContaining({ id: download.artifact.artifactId, companyId: fixture.companyId, createdByAgentId: fixture.agentId })]));
});
@@ -11,6 +11,7 @@ import {
validateRunnerApiCall,
type RunnerApiIo,
} from "./runner-api-client.js";
import { RUNNER_API_RESPONSE_MAX_BYTES } from "./runner-api-response-limits.js";
const context = {
companyId: "company-a",
@@ -32,11 +33,30 @@ const io = (fetcher: typeof fetch): RunnerApiIo => ({
}),
saveResponse: async (bytes, contentType) => ({
artifactId: "artifact-a",
byteSize: bytes.length,
byteSize: Buffer.isBuffer(bytes) ? bytes.length : bytes.byteSize,
contentType,
}),
});
describe("bounded response capture receipts", () => {
it.each([projects, createProject])("reports oversize evidence without retrying %s", async operationId => {
const fetcher = vi.fn(async () => new Response("", { headers: { "content-length": String(RUNNER_API_RESPONSE_MAX_BYTES + 1) } }));
const result = await executeRunnerApi({ operationId }, context, io(fetcher));
expect(result).toMatchObject({ ok: false, error: "api_response_too_large", maxResponseBytes: RUNNER_API_RESPONSE_MAX_BYTES,
outcome: operationId === projects ? "read_failed" : "unknown" });
expect(fetcher).toHaveBeenCalledTimes(1);
});
it("settles the budget and removes the file even when saving fails", async () => {
const bytes = Buffer.alloc(32 * 1024);
const settle = vi.fn(async () => {});
const input = io(async () => new Response(bytes));
input.reserveResponseCapture = async () => settle;
input.saveResponse = async () => { throw new Error("storage unavailable"); };
await expect(executeRunnerApi({ operationId: projects }, context, input)).rejects.toThrow("storage unavailable");
expect(settle).toHaveBeenCalledExactlyOnceWith(bytes.length, true);
});
});
describe("runner API catalog", () => {
it("accounts for unique operations with resolved request contracts", () => {
const catalog = runnerApiCatalog();
@@ -591,6 +611,127 @@ describe("runner API request boundary", () => {
).toThrow("inline JSON object");
}
});
it("reads text windows from a live response and supplies stable snapshots", async () => {
const text = "prefix\n" + "🧭é\n".repeat(6000) + "END-OF-EVIDENCE";
const saveResponse = vi.fn(io(fetch).saveResponse);
const request = vi.fn<typeof fetch>(async () => new Response(text, { headers: { "content-type": "application/json" } }));
let offsetBytes = 0;
let received = "";
do {
const result = await executeRunnerApi({ operationId: projects, responseText: { offsetBytes, limitBytes: 4096 } }, context, { ...io(request), saveResponse }) as any;
expect(result).toMatchObject({ ok: true, responseText: { offsetBytes, totalBytes: Buffer.byteLength(text) } });
expect(Buffer.byteLength(result.data)).toBeLessThanOrEqual(4096);
received += result.data;
offsetBytes = result.responseText.nextOffsetBytes;
} while (offsetBytes !== null);
expect(received).toBe(text);
expect(saveResponse).toHaveBeenCalled();
expect(request.mock.calls.every(([, options]) => new Headers(options?.headers).get("Authorization") === "Bearer private-agent-token")).toBe(true);
});
it.each(["text/plain", "application/octet-stream"])("does not duplicate an unpaged multi-gigabyte asset (%s)", async contentType => {
const totalBytes = 3 * 1024 * 1024 * 1024;
const saveResponse = vi.fn(io(fetch).saveResponse);
const request = vi.fn<typeof fetch>(async (_url, init) => {
expect(new Headers(init?.headers).get("range")).toBe("bytes=0-24576");
return new Response(Buffer.alloc(24577, 65), { status: 206, headers: {
"content-type": contentType, "content-range": `bytes 0-24576/${totalBytes}`, etag: `"${"a".repeat(64)}"`,
} });
});
const result = await executeRunnerApi({ operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: "existing" } }, context, { ...io(request), saveResponse });
expect(result).toMatchObject({ ok: true, status: 206, byteSize: totalBytes, artifact: { artifactId: "existing", byteSize: totalBytes, sha256: "a".repeat(64) } });
expect(saveResponse).not.toHaveBeenCalled();
expect(request).toHaveBeenCalledTimes(1);
});
it("reads a page beyond ten MiB in a multi-gigabyte asset", async () => {
const offsetBytes = 3 * 1024 * 1024 * 1024;
const totalBytes = offsetBytes + 100;
const request = vi.fn<typeof fetch>(async (_url, init) => {
expect(new Headers(init?.headers).get("range")).toBe(`bytes=${offsetBytes - 1}-${offsetBytes + 4}`);
return new Response("abcdef", { status: 206, headers: {
"content-type": "text/plain", "content-range": `bytes ${offsetBytes - 1}-${offsetBytes + 4}/${totalBytes}`,
} });
});
expect(await executeRunnerApi({ operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: "large" }, responseText: { offsetBytes, limitBytes: 4 } }, context, io(request))).toMatchObject({
ok: true, data: "bcde", responseText: { offsetBytes, nextOffsetBytes: offsetBytes + 4, totalBytes },
});
});
it.each([true, false])("saves responses above ten MiB (known length: %s)", async knownLength => {
const bytes = Buffer.alloc(12 * 1024 * 1024, 65);
const request = vi.fn<typeof fetch>(async () => new Response(bytes, { headers: {
"content-type": "text/plain", ...(knownLength ? { "content-length": String(bytes.length) } : {}),
} }));
const result = await executeRunnerApi({ operationId: projects }, context, io(request));
expect(result).toMatchObject({ ok: true, artifact: { byteSize: bytes.length }, byteSize: bytes.length });
});
it("pages a twelve MiB asset with linear transfer and UTF-8 boundaries", async () => {
const bytes = Buffer.from("🧭é".repeat(Math.floor(12 * 1024 * 1024 / 6)));
let transferred = 0;
const request = vi.fn<typeof fetch>(async (_url, init) => {
const range = new Headers(init?.headers).get("range");
const match = /^bytes=(\d+)-(\d+)$/.exec(range ?? "");
if (!match) { transferred += bytes.length; return new Response(bytes, { headers: { "content-type": "text/plain" } }); }
const start = Number(match[1]), end = Math.min(Number(match[2]), bytes.length - 1);
const part = bytes.subarray(start, end + 1); transferred += part.length;
return new Response(part, { status: 206, headers: { "content-type": "text/plain", "content-range": `bytes ${start}-${end}/${bytes.length}` } });
});
let offsetBytes: number | null = 0;
const parts: string[] = [];
do {
const page: any = await executeRunnerApi({ operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: "large" }, responseText: { offsetBytes } }, context, io(request));
expect(page.ok).toBe(true);
parts.push(page.data); offsetBytes = page.responseText.nextOffsetBytes;
} while (offsetBytes !== null);
expect(parts.join("")).toBe(bytes.toString());
expect(transferred).toBeLessThan(bytes.length + request.mock.calls.length * 5);
expect(request.mock.calls.every(([, init]) => new Headers(init?.headers).has("range"))).toBe(true);
});
it.each([
["bytes 4-8/20", "12345"], // wrong start
["bytes 3-7/20", "12345"], // early end
["bytes 3-8/20", "12345"], // truncated body
["bytes 3-8/9007199254740992", "123456"], // imprecise total
["bytes 3-8/*", "123456"], // unknown total
[null, "123456"],
])("rejects inconsistent partial asset receipts: %s", async (range, body) => {
const result = await executeRunnerApi({ operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: "saved" }, responseText: { offsetBytes: 4, limitBytes: 4 } }, context, io(async () => new Response(body, { status: 206, headers: { "content-type": "text/plain", ...(range ? { "content-range": range } : {}) } })));
expect(result).toMatchObject({ ok: false, error: "response_text_invalid_range" });
});
it("accepts EOF and rejects an offset inside a UTF-8 code point in ranged assets", async () => {
const call = (offsetBytes: number, bytes: Buffer, range: string) => executeRunnerApi({ operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: "saved" }, responseText: { offsetBytes, limitBytes: 4 } }, context, io(async () => new Response(new Uint8Array(bytes), { status: 206, headers: { "content-type": "text/plain", "content-range": range } })));
expect(await call(2, Buffer.from([0xa9]), "bytes 1-1/2")).toMatchObject({ ok: true, data: "", responseText: { nextOffsetBytes: null, totalBytes: 2 } });
expect(await call(1, Buffer.from("é"), "bytes 0-1/2")).toMatchObject({ ok: false, error: "response_text_invalid_offset" });
});
it("does not silently accept an asset server that ignores the byte range", async () => {
const result = await executeRunnerApi({ operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: "saved" }, responseText: { limitBytes: 4 } }, context, io(async () => new Response("unbounded", { headers: { "content-type": "text/plain" } })));
expect(result).toMatchObject({ ok: false, error: "api_transport_failure" });
});
it.each([{ offsetBytes: -1 }, { limitBytes: 0 }, { limitBytes: 3 }, { limitBytes: 24577 }, { offsetBytes: 0.5 }, { offsetBytes: Number.MAX_SAFE_INTEGER + 1 }])("rejects invalid text windows before dispatch: %j", async responseText => {
const request = vi.fn<typeof fetch>();
await expect(executeRunnerApi({ operationId: projects, responseText }, context, io(request))).rejects.toThrow("Invalid call_api arguments");
expect(request).not.toHaveBeenCalled();
});
it("does not page mutation responses by repeating the mutation", async () => {
const request = vi.fn<typeof fetch>();
await expect(executeRunnerApi({ operationId: createProject, body: { name: "Once" }, responseText: {} }, context, io(request))).rejects.toThrow("GET");
expect(request).not.toHaveBeenCalled();
});
it("rejects binary, invalid UTF-8 and offsets that split text", async () => {
for (const [body, type, offsetBytes] of [
[Buffer.from("binary"), "application/octet-stream", 0],
[Buffer.from([0xff]), "text/plain", 0],
[Buffer.concat([Buffer.from([0xc2]), Buffer.alloc(24576, 0x80)]), "text/plain", 0],
[Buffer.from("é"), "text/plain", 1],
[Buffer.from("short"), "text/plain", 6],
] as const) {
const saveResponse = vi.fn(io(fetch).saveResponse);
const result = await executeRunnerApi({ operationId: projects, responseText: { offsetBytes } }, context, { ...io(async () => new Response(body, { headers: { "content-type": type } })), saveResponse });
expect(result).toMatchObject({ ok: false, error: expect.stringContaining("response_text") });
expect(saveResponse).not.toHaveBeenCalled();
}
});
it("preserves denials for bounded text reads", async () => {
expect(await executeRunnerApi({ operationId: projects, responseText: {} }, context, io(async () => Response.json({ error: "denied" }, { status: 403 })))).toMatchObject({ ok: false, status: 403, data: '{"error":"denied"}' });
});
it("bounds streamed responses even without content-length", async () => {
await expect(
readBoundedResponse(new Response("too large"), 3),
+4 -2
View File
@@ -47,8 +47,10 @@ export function createLocalDiskStorageProvider(baseDir: string): StorageProvider
await fs.mkdir(dir, { recursive: true });
const tempPath = `${targetPath}.tmp-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`;
await fs.writeFile(tempPath, input.body);
await fs.rename(tempPath, targetPath);
try {
await fs.writeFile(tempPath, input.body);
await fs.rename(tempPath, targetPath);
} finally { await fs.rm(tempPath, { force: true }); }
},
async getObject(input): Promise<GetObjectResult> {
+52
View File
@@ -0,0 +1,52 @@
import { AbortMultipartUploadCommand, CompleteMultipartUploadCommand, CreateMultipartUploadCommand, UploadPartCommand, type S3Client } from "@aws-sdk/client-s3";
import { Readable } from "node:stream";
import type { PutObjectInput } from "./types.js";
/** Each part is streamed too: multipart must not become another whole-file
* buffer or a fixed part-count ceiling below the storage provider's limits. */
export async function putS3Multipart(client: S3Client, bucket: string, key: string, input: PutObjectInput) {
const source = input.body instanceof Readable ? input.body : Readable.from([input.body]);
const iterator = source[Symbol.asyncIterator]();
let pending = Buffer.alloc(0);
let uploadId: string | undefined;
// Scale part sizes to the S3 part-count limit while retaining stream backpressure.
const partSize = Math.max(8 * 1024 * 1024, Math.ceil(input.contentLength / 10_000));
try {
uploadId = (await client.send(new CreateMultipartUploadCommand({ Bucket: bucket, Key: key, ContentType: input.contentType }))).UploadId;
if (!uploadId) throw new Error("S3 did not return a multipart upload id");
const parts: { ETag: string; PartNumber: number }[] = [];
for (let offset = 0; offset < input.contentLength;) {
const length = Math.min(partSize, input.contentLength - offset);
let remaining = length;
const body = Readable.from((async function* () {
while (remaining > 0) {
if (!pending.length) {
const next = await iterator.next();
if (next.done) throw new Error("S3 upload source was truncated");
pending = Buffer.isBuffer(next.value) ? next.value : Buffer.from(next.value);
if (!pending.length) continue;
}
const count = Math.min(remaining, pending.length);
const chunk = pending.subarray(0, count);
pending = pending.subarray(count);
remaining -= count;
yield chunk;
}
})());
try {
const result = await client.send(new UploadPartCommand({ Bucket: bucket, Key: key, UploadId: uploadId, PartNumber: parts.length + 1, Body: body, ContentLength: length }));
if (remaining || !result.ETag) throw new Error("S3 upload part is incomplete");
parts.push({ ETag: result.ETag, PartNumber: parts.length + 1 });
} finally { body.destroy(); }
offset += length;
}
if (pending.length || !(await iterator.next()).done) throw new Error("S3 upload source exceeds its declared size");
await client.send(new CompleteMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId, MultipartUpload: { Parts: parts } }));
} catch (error) {
if (uploadId) await client.send(new AbortMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId })).catch(() => {});
throw error;
} finally {
source.destroy();
await iterator.return?.();
}
}
+5
View File
@@ -5,6 +5,7 @@ import {
HeadObjectCommand,
PutObjectCommand,
} from "@aws-sdk/client-s3";
import { putS3Multipart } from "./s3-multipart.js";
import { Readable } from "node:stream";
import type { StorageProvider, GetObjectResult, HeadObjectResult } from "./types.js";
import { notFound, unprocessable } from "../errors.js";
@@ -81,6 +82,10 @@ export function createS3StorageProvider(config: S3ProviderConfig): StorageProvid
async putObject(input) {
const key = buildKey(prefix, input.objectKey);
if (input.contentLength > 16 * 1024 * 1024) {
await putS3Multipart(client, bucket, key, input);
return;
}
await client.send(
new PutObjectCommand({
Bucket: bucket,
+22 -11
View File
@@ -1,5 +1,6 @@
import { createHash, randomUUID } from "node:crypto";
import path from "node:path";
import { Readable } from "node:stream";
import type { StorageService, StorageProvider, PutFileInput, PutFileResult } from "./types.js";
import { badRequest, forbidden, unprocessable } from "../errors.js";
@@ -79,10 +80,13 @@ function assertPutFileInput(input: PutFileInput): void {
if (!input.contentType || input.contentType.trim().length === 0) {
throw unprocessable("contentType is required");
}
if (!(input.body instanceof Buffer)) {
throw unprocessable("body must be a Buffer");
if (!(input.body instanceof Buffer) && !(input.body instanceof Readable)) {
throw unprocessable("body must be a Buffer or Readable");
}
if (input.body.length <= 0) {
if (!Buffer.isBuffer(input.body) && (!("byteSize" in input) || !Number.isSafeInteger(input.byteSize) || input.byteSize < 0 || !/^[0-9a-f]{64}$/.test(input.sha256))) {
throw unprocessable("Streamed files require an exact byte size and SHA-256");
}
if (("byteSize" in input ? input.byteSize : input.body.length) <= 0) {
throw unprocessable("File is empty");
}
}
@@ -94,21 +98,28 @@ export function createStorageService(provider: StorageProvider): StorageService
async putFile(input: PutFileInput): Promise<PutFileResult> {
assertPutFileInput(input);
const objectKey = buildObjectKey(input.companyId, input.namespace, input.originalFilename);
const byteSize = input.body.length;
const byteSize = "byteSize" in input ? input.byteSize : input.body.length;
const contentType = input.contentType.trim().toLowerCase();
await provider.putObject({
objectKey,
body: input.body,
contentType,
contentLength: byteSize,
});
try {
await provider.putObject({
objectKey,
body: input.body,
contentType,
contentLength: byteSize,
});
} catch (error) {
await provider.deleteObject({ objectKey }).catch(() => {});
throw error;
} finally {
if (input.body instanceof Readable) input.body.destroy();
}
return {
provider: provider.id,
objectKey,
contentType,
byteSize,
sha256: hashBuffer(input.body),
sha256: "sha256" in input ? input.sha256 : hashBuffer(input.body),
originalFilename: input.originalFilename,
};
},
+2 -3
View File
@@ -42,13 +42,12 @@ export interface StorageProvider {
deleteObject(input: GetObjectInput): Promise<void>;
}
export interface PutFileInput {
export type PutFileInput = {
companyId: string;
namespace: string;
originalFilename: string | null;
contentType: string;
body: Buffer;
}
} & ({ body: Buffer } | { body: Readable; byteSize: number; sha256: string });
export interface PutFileResult {
provider: StorageProviderId;
+12
View File
@@ -1094,3 +1094,15 @@ keys with low request limits. It can only lower the configured campaign limit.
Keep subscription qualification separate from API-key results.
The explicit-only eight-cell [continuation accounting baseline](CONTINUATION-ACCOUNTING.md) tests productive work, bounded repair, restart and late gates with real providers.
### Bounded API response reading
`api-response-reading` is an explicit-only, two-cell native Codex suite (local
and Daytona). Each cell creates synthetic diagnostic evidence over 24 KiB via
the public API, outside the agent assignment. The browser starts one ordinary
task which must read the saved response in bounded text pages and persist the
exact hidden evidence code. The oracle also requires completed API tool events;
missing events or a narrative completion cannot pass. Existing run, copyback,
screenshot, billing and environment cleanup checks apply. Use
`--id api-response-reading.runner-codex.daytona.saved-text-pages` with an
immutable Daytona image; no private hooks or fixture database writes are used.
@@ -0,0 +1,102 @@
import { createHash } from "node:crypto";
import { describe, expect, it, vi } from "vitest";
import type { RunnerApi } from "./api.js";
import { apiResponseReadingTask, gradeApiResponsePaging, readResponseProof, responseEvidenceCode, responseEvidenceDescription } from "./api-response-reading.js";
describe("large API response evidence fixture", () => {
it("keeps evidence beyond both preview and inline limits and out of the assignment", () => {
const source = responseEvidenceDescription("fixture");
const code = responseEvidenceCode("fixture");
expect(Buffer.byteLength(source)).toBeGreaterThan(24 * 1024);
expect(source.indexOf(code)).toBeGreaterThan(24 * 1024);
expect(apiResponseReadingTask.buildPrompt("fixture")).not.toContain(code);
expect(apiResponseReadingTask.buildPrompt("fixture")).toContain("responseText.nextOffsetBytes");
});
it("downloads exact run-attributed proof bytes rather than trusting the agent's report", async () => {
const bytes = Buffer.from("independent evidence\n");
const proof = { id: "proof", issueId: "issue", originatingRunId: "run", originalFilename: "api-response-proof.txt", byteSize: bytes.length, sha256: createHash("sha256").update(bytes).digest("hex") };
const download = vi.fn().mockResolvedValue({ ok: () => true, body: async () => bytes });
const get = vi.fn().mockResolvedValue([proof]);
const api = { get, request: { get: download } } as unknown as RunnerApi;
expect((await readResponseProof(api, "issue", "run")).content).toBe(bytes.toString());
expect(download).toHaveBeenCalledWith("/api/attachments/proof/content?download=1");
get.mockResolvedValue([{ ...proof, originatingRunId: "other" }]);
await expect(readResponseProof(api, "issue", "run")).rejects.toThrow("observed 0");
get.mockResolvedValue([proof, proof]);
await expect(readResponseProof(api, "issue", "run")).rejects.toThrow("observed 2");
get.mockResolvedValue([{ ...proof, sha256: "mismatch" }]);
await expect(readResponseProof(api, "issue", "run")).rejects.toThrow("disagrees");
});
});
describe("saved API response pagination oracle", () => {
const sourceIssueId = "source-issue";
const assetId = "source-asset";
const bytes = Buffer.from(JSON.stringify({ id: sourceIssueId, description: "synthetic padding ".repeat(2000) }));
const operation = "GET /api/assets/{assetId}/content";
const event = (eventType: string, payload: unknown) => ({ eventType, payload: { prpEvent: { payload } } });
const call = (id: string, input: unknown, result: unknown, status = "completed") => [
event("item.started", { item: { id, type: "tool_use", name: "call_api", input } }),
event("item.completed", { item: { id, type: "tool_result", tool_use_id: id, result } }),
event("tool.execution.completed", { name: "call_api", executionId: id, status }),
];
const source = () => call("source-read", { operationId: "GET /api/issues/{id}", pathParams: { id: sourceIssueId } }, {
ok: true, status: 200, apiOperationId: "GET /api/issues/{id}", artifact: {
artifactId: assetId, byteSize: bytes.length, sha256: createHash("sha256").update(bytes).digest("hex"),
url: `/api/assets/${assetId}/content`,
},
});
const page = (offset: number, id = `page-${offset}`, artifactId = assetId) => {
const end = Math.min(offset + 8192, bytes.length);
return call(id, { operationId: operation, pathParams: { assetId: artifactId }, responseText: { offsetBytes: offset, limitBytes: 8192 } }, {
ok: true, status: 200, apiOperationId: operation, data: bytes.subarray(offset, end).toString(),
responseText: { offsetBytes: offset, totalBytes: bytes.length, nextOffsetBytes: end === bytes.length ? null : end },
});
};
const pages = () => Array.from({ length: Math.ceil(bytes.length / 8192) }, (_, i) => page(i * 8192)).flat();
it("accepts correlated source artifact pages through EOF, including a harmless identical retry", () => {
expect(gradeApiResponsePaging([...source(), ...page(0, "retry"), ...pages()], sourceIssueId).passed).toBe(true);
});
it("rejects unrelated completed calls", () => {
const unrelated = [1, 2, 3].flatMap(i => call(`unrelated-${i}`, { operationId: "GET /api/agents/me" }, { ok: true, status: 200 }));
expect(gradeApiResponsePaging(unrelated, sourceIssueId).passed).toBe(false);
});
it("rejects repeating the first page without reaching EOF", () => {
expect(gradeApiResponsePaging([...source(), ...page(0), ...page(0, "again"), ...page(0, "again-2")], sourceIssueId).passed).toBe(false);
});
it("rejects a gap even when the last page claims EOF", () => {
expect(gradeApiResponsePaging([...source(), ...pages().filter((_e, i) => i < 3 || i >= 6)], sourceIssueId).passed).toBe(false);
});
it("rejects reading a different artifact", () => {
const other = Array.from({ length: Math.ceil(bytes.length / 8192) }, (_, i) => page(i * 8192, `other-${i}`, "other-asset")).flat();
expect(gradeApiResponsePaging([...source(), ...other], sourceIssueId).passed).toBe(false);
});
it("requires the exact seeded source issue", () => {
expect(gradeApiResponsePaging([...source(), ...pages()], "other-issue").passed).toBe(false);
});
it("requires a matched successful completion for every page", () => {
const incomplete = pages().filter(e => e.eventType !== "tool.execution.completed");
expect(gradeApiResponsePaging([...source(), ...incomplete], sourceIssueId).passed).toBe(false);
});
it.each([
["requested offset", { offsetBytes: 1, totalBytes: bytes.length, nextOffsetBytes: 8192 }],
["next offset", { offsetBytes: 0, totalBytes: bytes.length, nextOffsetBytes: 8193 }],
["total size", { offsetBytes: 0, totalBytes: bytes.length + 1, nextOffsetBytes: 8192 }],
["premature EOF", { offsetBytes: 0, totalBytes: bytes.length, nextOffsetBytes: null }],
])("rejects mismatched %s metadata", (_label, responseText) => {
const malformed = call("bad-page", {
operationId: operation, pathParams: { assetId }, responseText: { offsetBytes: 0, limitBytes: 8192 },
}, { ok: true, status: 206, apiOperationId: operation, data: bytes.subarray(0, 8192).toString(), responseText });
expect(gradeApiResponsePaging([...source(), ...pages(), ...malformed], sourceIssueId).passed).toBe(false);
});
it("rejects contradictory duplicate bytes and a wrong source artifact digest", () => {
const corrupt = call("corrupt-page", {
operationId: operation, pathParams: { assetId }, responseText: { offsetBytes: 0, limitBytes: 8192 },
}, {
ok: true, status: 206, apiOperationId: operation, data: "x".repeat(8192),
responseText: { offsetBytes: 0, totalBytes: bytes.length, nextOffsetBytes: 8192 },
});
expect(gradeApiResponsePaging([...source(), ...pages(), ...corrupt], sourceIssueId).passed).toBe(false);
expect(gradeApiResponsePaging([...source(), ...corrupt, ...pages().slice(3)], sourceIssueId).passed).toBe(false);
});
});
+134
View File
@@ -0,0 +1,134 @@
import { createHash } from "node:crypto";
import type { RunnerTaskFixture } from "./types.js";
import type { RunnerApi } from "./api.js";
import type { IssueAttachment } from "../../packages/shared/src/types/issue.js";
/** Verify delivered bytes through the same public download path the user receives. */
export async function readResponseProof(api: RunnerApi, issueId: string, runId: string) {
const attachments = await api.get<IssueAttachment[]>(`/api/issues/${issueId}/attachments`);
const proofs = attachments.filter(attachment =>
attachment.issueId === issueId && attachment.originatingRunId === runId &&
attachment.originalFilename === "api-response-proof.txt");
if (proofs.length !== 1) throw new Error(`Expected one proof attachment from the tested run; observed ${proofs.length}`);
const proof = proofs[0]!;
const response = await api.request.get(`/api/attachments/${encodeURIComponent(proof.id)}/content?download=1`);
if (!response.ok()) throw new Error(`Proof download returned ${response.status()}`);
const bytes = await response.body();
if (bytes.length !== proof.byteSize || createHash("sha256").update(bytes).digest("hex") !== proof.sha256) {
throw new Error("Downloaded proof disagrees with stored attachment bytes");
}
return { attachmentId: proof.id, content: bytes.toString("utf8"), sha256: proof.sha256 };
}
export function responseEvidenceCode(nonce: string) {
return createHash("sha256").update(`bounded-response-evidence:${nonce}`).digest("hex");
}
export function responseEvidenceDescription(nonce: string) {
return `${"Synthetic diagnostic padding.\n".repeat(1400)}\nEvidence code: ${responseEvidenceCode(nonce)}\n`;
}
/** Correlate trusted PRP call inputs, results, and completion receipts. */
export function gradeApiResponsePaging(events: readonly { eventType?: string; payload?: unknown }[], sourceIssueId: string) {
const object = (value: unknown): Record<string, unknown> =>
value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
const inputs = new Map<string, Record<string, unknown>>();
const results = new Map<string, Record<string, unknown>>();
const completed = new Set<string>();
for (const event of events) {
const payload = object(object(object(event.payload).prpEvent).payload);
const item = object(payload.item);
if (event.eventType === "item.started" && item.type === "tool_use" && item.name === "call_api" && typeof item.id === "string") {
inputs.set(item.id, object(item.input));
}
if (event.eventType === "item.completed" && item.type === "tool_result" && typeof item.id === "string" &&
(item.tool_use_id === undefined || item.tool_use_id === item.id)) results.set(item.id, object(item.result));
if (event.eventType === "tool.execution.completed" && payload.name === "call_api" && payload.status === "completed" && typeof payload.executionId === "string") {
completed.add(payload.executionId);
}
}
const calls = [...inputs].flatMap(([id, input]) => {
const result = results.get(id);
return completed.has(id) && result?.ok === true && [200, 206].includes(Number(result.status)) ? [{ input, result }] : [];
});
const sourceOperation = "GET /api/issues/{id}";
const assetOperation = "GET /api/assets/{assetId}/content";
const sources = calls.filter(call => call.input.operationId === sourceOperation && call.result.apiOperationId === sourceOperation &&
object(call.input.pathParams).id === sourceIssueId);
let failure = "No successful call_api source read with a saved response artifact";
for (const source of sources) {
const artifact = object(source.result.artifact);
const assetId = artifact.artifactId;
const total = artifact.byteSize;
if (typeof assetId !== "string" || typeof total !== "number" || !Number.isSafeInteger(total) || total <= 24 * 1024 || total > 1024 * 1024 ||
typeof artifact.sha256 !== "string" || !/^[a-f0-9]{64}$/.test(artifact.sha256) || artifact.url !== `/api/assets/${assetId}/content`) continue;
const pages = new Map<number, { bytes: Buffer; next: number | null }>();
let malformed = false;
for (const call of calls) {
if (call.input.operationId !== assetOperation || object(call.input.pathParams).assetId !== assetId) continue;
const requested = object(call.input.responseText);
const returned = object(call.result.responseText);
const offset = requested.offsetBytes;
const limit = requested.limitBytes;
const next = returned.nextOffsetBytes;
const data = call.result.data;
if (call.result.apiOperationId !== assetOperation || typeof offset !== "number" || !Number.isSafeInteger(offset) || offset < 0 || offset >= total ||
typeof limit !== "number" || !Number.isSafeInteger(limit) || limit < 1 || limit > 8192 || returned.offsetBytes !== offset || returned.totalBytes !== total ||
typeof data !== "string") { malformed = true; break; }
const bytes = Buffer.from(data, "utf8");
const end = offset + bytes.length;
if (bytes.length === 0 || bytes.length > limit || end > total || (end === total ? next !== null : next !== end)) {
malformed = true; break;
}
const prior = pages.get(offset);
if (prior && (!prior.bytes.equals(bytes) || prior.next !== next)) { malformed = true; break; }
pages.set(offset, { bytes, next: next as number | null });
}
if (malformed) { failure = "Artifact paging request, returned offsets, bytes, or EOF disagree"; continue; }
const chunks: Buffer[] = [];
const offsets: number[] = [];
let offset = 0;
let eofReached = false;
while (pages.has(offset)) {
const page = pages.get(offset)!;
chunks.push(page.bytes); offsets.push(offset);
if (page.next === null) { eofReached = true; break; }
offset = page.next;
}
if (!eofReached || offsets.length < 2) { failure = "Saved artifact pages do not cover offset zero through EOF without gaps"; continue; }
const bytes = Buffer.concat(chunks);
const digest = createHash("sha256").update(bytes).digest("hex");
if (bytes.length !== total || digest !== artifact.sha256) { failure = "Paged bytes disagree with the source response artifact digest"; continue; }
try {
if (object(JSON.parse(bytes.toString("utf8"))).id !== sourceIssueId) { failure = "Paged artifact belongs to another source issue"; continue; }
} catch { failure = "Paged source response is not complete JSON"; continue; }
return { passed: true, sourceIssueId, artifactId: assetId, totalBytes: total, pageCount: offsets.length, offsets, eofReached, sha256: digest };
}
return { passed: false, sourceIssueId, failure };
}
export const apiResponseReadingTask: RunnerTaskFixture = {
id: "saved-text-pages", label: "Read saved large API evidence", groups: [],
workMode: "standard", flow: "single_turn", expectedRunCount: 1,
attemptTimeoutMs: { local: 10 * 60_000, daytona: 10 * 60_000 },
expectedTerminalState: { issue: "done", run: "succeeded" },
buildTitle: nonce => `Read large API evidence ${nonce}`,
// Delivered files render as an attachment card, which can replace the finish summary.
buildVisibleMarker: () => "api-response-proof.txt",
buildPrompt: nonce => [
"Inspect diagnostic evidence task {{API_RESPONSE_SOURCE_ID}} using the Paperclip API tools.",
"Discover GET /api/issues/{id}, call it for that task, and retain the returned response artifact.",
"Read the saved artifact through GET /api/assets/{assetId}/content with responseText: {offsetBytes:0,limitBytes:8192}.",
"Continue using responseText.nextOffsetBytes until null. Extract the Evidence code at the end of its description.",
"The large response must be read with bounded responseText pages. Do not use other tools or API routes to obtain the evidence; if bounded reading fails, report the failure instead of substituting a different reader.",
"Write only that code followed by a newline to api-response-proof.txt in the current workspace and deliver that file as an attachment. Do not edit the evidence task or create child tasks.",
`Verify the proof file and finish this task with paperclip_finish, reportedWorkDisposition done, and summary API_RESPONSE_READ_${nonce}.`,
].join("\n"),
buildMatchers: (nonce, execution) => [
{ kind: "file_exact", path: "api-response-proof.txt", expected: `${responseEvidenceCode(nonce)}\n` },
{ kind: "message_contains", expected: "api-response-proof.txt" },
{ kind: "issue_status", expected: "done" },
{ kind: "run_status", expected: "succeeded" },
{ kind: "runtime_mode", expected: execution.profile.expectedRuntimeMode },
{ kind: "environment", expected: execution.environment.id },
],
};
+3 -3
View File
@@ -71,10 +71,10 @@ describe("runner E2E catalog", () => {
expect(localIntegrityTasks).toHaveLength(2);
expect(openRouterBreadthTasks).toHaveLength(3);
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
8, 46, 23, 47, 52, 28, 18, 6, 6, 4, 42, 14, 10, 2,
2, 8, 46, 23, 47, 52, 28, 18, 6, 6, 4, 42, 14, 10, 2,
]);
expect(validateRunnerCatalog()).toHaveLength(306);
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(306);
expect(validateRunnerCatalog()).toHaveLength(308);
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(308);
expect(
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
).toHaveLength(42);
+9
View File
@@ -1,3 +1,4 @@
import { apiResponseReadingTask } from "./api-response-reading.js";
import { accountingTasks } from "./accounting-cases.js";
import { continuationTasks } from "./continuation-cases.js";
import { lifecycleLiveTasks, lifecycleLiveDefinitionDigest } from "./lifecycle-live-cases.js";
@@ -911,6 +912,14 @@ const everydayProfiles = [
].map(productionStoryProfile);
export const runnerSuites: readonly RunnerSuiteFixture[] = [
{
id: "api-response-reading", label: "Bounded API response reading", manualOnly: true,
description: "Retrieve evidence beyond a saved API preview through authorized bounded text windows.",
groups: [], environments: runnerEnvironments,
profiles: runnerProfiles.filter(profile => profile.id === "runner-codex"),
tasks: [apiResponseReadingTask], expectedMatrixSize: 2,
definitionMetadata: { version: 1, grading: "hidden-evidence-exact-copy-and-api-tool-events", scheduling: "explicit-only" },
},
{
id: "continuation-accounting", label: "Continuation accounting baseline", manualOnly: true,
description: "Structured productive steps, bounded repair, restart and late gates; comments cannot buy more attempts.",
+30 -1
View File
@@ -1,3 +1,4 @@
import { gradeApiResponsePaging, readResponseProof, responseEvidenceDescription } from "./api-response-reading.js";
import { observeBrowserBootstrap } from "./browser-bootstrap-diagnostics.js";
import { runAccountingFlow } from "./accounting-flow.js";
import type { Issue } from "../../packages/shared/src/types/issue.js";
@@ -534,6 +535,7 @@ for (const execution of executions) {
const marker = execution.task.buildVisibleMarker(nonce);
const title = execution.task.buildTitle(nonce);
let prompt = execution.task.buildPrompt(nonce);
let apiResponseSourceId: string | undefined;
let lifecycleBlockerId: string | null = null;
const credentials = credentialValues();
const secrets = normalizedSecrets(Object.values(credentials));
@@ -549,6 +551,7 @@ for (const execution of executions) {
let selectedRuns: RunRecord[] = [];
let runtimeLeases: EnvironmentLeaseRecord[] = [];
let matcherResults: MatcherResult[] = [];
let downloadedResponseProof: Awaited<ReturnType<typeof readResponseProof>> | undefined;
let firstTaskEvidence: RunnerE2EResult["firstTask"];
let turnTimings: NonNullable<RunnerE2EResult["turnTimings"]> | undefined;
const turnSubmissionTimesMs: number[] = [];
@@ -794,6 +797,15 @@ for (const execution of executions) {
daytonaImage: process.env.PAPERCLIP_E2E_DAYTONA_IMAGE,
});
if (execution.suite.id === "api-response-reading") {
const source = await api.post<{ id: string }>(`/api/companies/${fixtures.company.id}/issues`, {
title: `Synthetic diagnostic evidence ${nonce}`,
description: responseEvidenceDescription(nonce), status: "backlog",
});
apiResponseSourceId = source.id;
prompt = prompt.replaceAll("{{API_RESPONSE_SOURCE_ID}}", source.id);
}
if (execution.suite.id === "lifecycle-baseline" && lifecycleLiveCase(execution.task.id)?.family === "blocker") {
const prerequisite = await api.post<{ id: string }>(`/api/companies/${fixtures.company.id}/issues`, {
title: `Supply dataset ${nonce}`,
@@ -1885,6 +1897,12 @@ for (const execution of executions) {
}
}
if (execution.suite.id === "api-response-reading") {
const paging = gradeApiResponsePaging(runEventsByRun.flatMap(captured => captured.events), apiResponseSourceId ?? "");
await writeSanitizedJson(snapshotsDir, "api-response-pagination.json", paging, secrets);
if (!paging.passed) invariantFailures.push(`Bounded response paging was not proven: ${paging.failure}`);
}
const context = record(run.contextSnapshot);
const environmentContext = record(context.paperclipEnvironment);
const workspaceContext = record(context.paperclipWorkspace);
@@ -1938,6 +1956,11 @@ for (const execution of executions) {
]),
),
);
if (execution.suite.id === "api-response-reading") {
downloadedResponseProof = await readResponseProof(api, issue.id, run.id);
fileObservations["api-response-proof.txt"] = downloadedResponseProof.content;
await writeSanitizedJson(snapshotsDir, "downloaded-response-proof.json", downloadedResponseProof, secrets);
}
matcherResults = await Promise.all(
taskMatchers.map((matcher) =>
evaluateMatcher(matcher, {
@@ -2398,7 +2421,13 @@ for (const execution of executions) {
const visibleAgentReplies = page
.getByTestId("task-chat-thread")
.getByTestId("task-chat-agent-bubble");
if (execution.task.flow === "warm_three_turn") {
if (execution.suite.id === "api-response-reading") {
// File delivery renders a card instead of an exact summary bubble.
// Prove the visible link points to the same independently checked bytes.
const proofLink = page.getByRole("link", { name: "Open api-response-proof.txt", exact: true }).first();
await expect(proofLink).toBeVisible({ timeout: 30_000 });
await expect(proofLink).toHaveAttribute("href", `/api/attachments/${downloadedResponseProof!.attachmentId}/content`);
} else if (execution.task.flow === "warm_three_turn") {
// Prove the persisted user-facing response is visible, independently
// of the byte-for-byte workspace checks and lease continuity checks.
expect(finalRunMessage.trim()).not.toBe("");