mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-04 02:32:07 +01:00
fix(runner): stream and page large API responses within capture budgets (#14301)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use governed API tools to inspect task evidence.
> - Large API results become saved assets with short previews.
> - Reading an asset through the same tool used to create another asset,
so the agent could not reach the rest of the evidence.
> - The 10 MiB response cap also blocked useful large results. Removing
all bounds allowed excessive disk use.
> - This pull request streams responses up to 1 GiB and makes saved text
readable in bounded pages. It adds durable run budgets and capture
admission limits.
> - Agents can inspect complete evidence while tool results, memory use,
and capture work stay bounded.
## Linked Issues or Issue Description
**What happened?**
A large response became an asset. Reading that asset returned another
asset and the same preview. Responses above 10 MiB failed before the
agent could read any page.
**Expected behavior**
The agent can fetch a large response and read its saved text to EOF.
Each page stays bounded. New snapshots have a generous finite limit and
a durable run budget. Existing larger assets remain readable through
byte ranges.
**Steps to reproduce**
1. Call a GET operation that returns more than 10 MiB of text or JSON.
2. Before the fix, the tool returns `api_transport_failure`.
3. With this change, responses up to 1 GiB become streamed snapshots
with artifact references.
4. Read `GET /api/assets/{assetId}/content` with `responseText:
{offsetBytes: 0, limitBytes: 8192}`. Follow `nextOffsetBytes` until
null.
Related work: #14186 added the API fallback tools. #14218 bounded API
discovery.
## What Changed
- Add authenticated UTF-8 text windows to `call_api`, with byte offsets
and total size. Keep each page at or below 24 KiB.
- Stream new responses above 24 KiB through private temporary files into
company-owned assets. Bound each capture to 1 GiB of decoded bytes.
Reject oversized declared lengths before reading and count streamed
bytes before writing.
- Reserve capture budget in the run record before spilling. Allow 4 GiB
per run. Settle successful captures to their actual size. Failed or
interrupted captures retain their full 1 GiB reservation. Run restarts
do not reset the budget.
- Enforce a 20 GiB company snapshot quota with database reservations.
Count legacy snapshots and unfinished storage work across runs and
processes. Asset deletion frees quota.
- Limit large captures to two per company and four per server process.
Hold slots through storage upload and temporary-file cleanup. Use a
10-minute download deadline and 30-second connection/idle-read timeouts.
- Return explicit size, budget, busy, and timeout errors. Preserve
unknown outcomes for mutations whose response cannot be captured.
- Read saved assets through authenticated storage ranges, with at most
two extra bytes for UTF-8 and EOF handling. Unpaged reads return the
existing asset and digest with a bounded preview. Reads create no copies
and do not consume capture budget.
- Keep existing assets above 1 GiB readable in pages. Use safe integer
offsets and PostgreSQL `bigint` asset sizes.
- Stream large S3 uploads through ordered multipart requests. Abort
failed uploads and remove partial local files.
- Revalidate run authority during downloads. Keep company authorization,
GET-only text paging, redirect denial, and mutation replay receipts.
- Document the separate 10 MiB upload limits. This PR does not raise
memory-buffered attachment ingestion limits. Future large video uploads
need streamed ingestion and storage quotas.
## Verification
- Full workspace `pnpm -r typecheck` and `pnpm build` pass after
rebasing on master.
- Focused API and response tests: 1,761 pass. Cover declared and chunked
oversize responses, incorrect Content-Length, exact-limit success,
active-stream deadline, cancellation, cleanup, concurrency admission,
and mutation outcome handling.
- Real HTTP integration: 28 tests pass, including runnerd → PRP →
authority → HTTP, a 12 MiB snapshot, final-page/EOF reads, cross-company
denial, a persisted 3 GiB sparse asset, and large mutation receipt
replay.
- The HTTP suite verifies durable run-budget accounting, simultaneous
runs competing for company quota, legacy snapshot accounting, deletion
refunds, failed-storage reservations, cleaned-failure refunds,
metadata-rollback cleanup refunds, preservation after a lost commit
acknowledgement, and small/saved reads after capture-budget exhaustion.
- A standalone proof streams exactly 1 GiB through the production
capture helper, verifies the final bytes, and removes its temporary
file. It uses repeated 256 KiB chunks and records a peak process RSS of
191 MiB.
- Earlier storage verification covers exact S3 multipart boundaries,
cleanup/abort failures, and a 17 MiB transfer through the real AWS SDK
to a local HTTP S3 endpoint. No cloud S3 qualification was run for this
follow-up.
- The local full test run was interrupted for the company-quota changes.
A later targeted run hit exhausted macOS shared-memory slots before
tests started; two unattached PostgreSQL segments with dead owners were
reclaimed before retrying. All 55 current-head checks pass at
`aebb80ceeeee77d5a56b67bfffd835f2f846878c`, including the full CI test
suite, typecheck, build, browser suites, security scan, and Greptile
(5/5). There are no unresolved review threads. The combined rebased test
catalog also passes (48 tests).
- Earlier paging acceptance passed Daytona and separate staging at
`7739879e9`. Those runs predate the streaming and budget changes.
## Risks
- The 1 GiB response cap and 10-minute active-download deadline are
intentional product limits. Larger live results must use endpoint
pagination or a direct file workflow. Existing larger assets remain
readable through bounded ranges.
- A durable 20 GiB company snapshot quota counts stored runner-api
assets and active/orphan reservations across runs and processes. The
operator can set PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES to a
finite value of at least 1 GiB. Deleting snapshots frees capacity;
possible orphan storage must be reconciled before releasing its
reservation.
- A failed capture uses its full reservation. A new large capture needs
a full 1 GiB available, even if it later completes at a smaller size.
Small reads and existing asset pages remain available.
- Concurrency limits apply per server process. The run byte budget is
shared through the database.
- The `integer` to `bigint` migration rewrites asset metadata and takes
an exclusive table lock. File bytes stay in storage.
- A live endpoint is fetched once before returning its snapshot.
Continue reading the saved artifact for stable pages. Mutations may
commit before any size or transport error; inspect state before
retrying.
- Attachment uploads and native file handoffs still default to 10 MiB.
Raising buffered ingestion paths to GiB sizes is separate work.
## Model Used
OpenAI Codex, based on GPT-6, with code execution and repository tools.
The runtime does not expose an exact serving model variant or
context-window size. The earlier paging work also used browser testing
and subagents.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
cea8dda472
commit
3447609d22
40 files changed
+2889
-73
No files matched your search
@@ -432,3 +432,23 @@ cleanup authority; it does not prove that remote inference has stopped. Recovery
|
||||
revokes the previous boot identity with a conditional update. Its own claim also
|
||||
expires so another sweep can finish cleanup after a restart. Historical rows keep
|
||||
null ownership fields and follow the previous recovery path.
|
||||
|
||||
## Large API response snapshots
|
||||
|
||||
`assets.byte_size` uses PostgreSQL `bigint` so saved responses and byte ranges can
|
||||
exceed 2 GiB. The API and Drizzle mapping continue to expose a JavaScript number;
|
||||
response readers validate safe integer offsets. The type-widening migration
|
||||
rewrites the asset metadata table and needs an exclusive table lock. File bytes
|
||||
remain in local or object storage.
|
||||
|
||||
### Runner API response reservations
|
||||
|
||||
`runner_api_response_reservations` holds company-scoped API snapshot reservations.
|
||||
Before a capture spills, the server locks company admission and counts stored
|
||||
`runner-api` assets plus unattached reservations against a 20 GiB default quota.
|
||||
A committed asset replaces its reservation in that total. The asset foreign key
|
||||
cascades on deletion, while deleting a run sets `run_id` to null so an orphan
|
||||
reservation cannot silently disappear. Failed cleanup or an ambiguous storage
|
||||
write requires operator reconciliation before an unattached reservation is
|
||||
removed. The table stores no response bodies. See `doc/runner-api-tools.md` for
|
||||
limits and the operator override.
|
||||
@@ -414,7 +414,7 @@ Operational policy:
|
||||
- `provider` enum/text (`local_disk | s3`)
|
||||
- `object_key` text not null
|
||||
- `content_type` text not null
|
||||
- `byte_size` int not null
|
||||
- `byte_size` bigint not null (safe integer byte count in the API)
|
||||
- `sha256` text not null
|
||||
- `original_filename` text null
|
||||
- `created_by_agent_id` uuid fk null
|
||||
|
||||
@@ -262,3 +262,7 @@ named existing controls on legacy and native Codex. Discover it with
|
||||
results and follow-up coverage are recorded in that suite's guide.
|
||||
|
||||
Continuation accounting has an explicit-only eight-cell Product E2E [baseline suite](../tests/runner-e2e/CONTINUATION-ACCOUNTING.md), complementing the deterministic lifecycle inventory.
|
||||
|
||||
The explicit-only Product E2E `api-response-reading` suite verifies retrieval of
|
||||
large saved API responses on local and Daytona native Codex runs. See the
|
||||
[Runner E2E guide](../tests/runner-e2e/README.md#bounded-api-response-reading).
|
||||
+95
-4
@@ -51,14 +51,105 @@ exactly one authorized `artifactId` or task-workspace `path`, and an optional
|
||||
multipart `field`. No arbitrary URL, headers, authentication, or remote file URL
|
||||
can be supplied. Routes still validate payloads and enforce permissions.
|
||||
|
||||
Requests have a 30-second HTTP timeout, 16 KiB URL limit and 10 MiB payload/response
|
||||
transfer limit. Responses above 24 KiB and binary responses become company-owned
|
||||
assets with retrievable references; text previews are limited to 2,000 bytes.
|
||||
Requests have a 16 KiB URL limit and 10 MiB request/upload limit. **New response
|
||||
captures are limited to 1 GiB of decoded bytes.** This is separate from the 24 KiB
|
||||
inline/page limit. The receiver rejects an oversized Content-Length before
|
||||
reading and counts actual bytes before writing, including chunked or compressed
|
||||
responses. Oversized responses return `api_response_too_large`; narrow the query
|
||||
or use the endpoint's own pagination. A mutation may already have committed, so
|
||||
inspect its state rather than retrying it to obtain a smaller response.
|
||||
|
||||
Connection setup and stalled response reads time out after 30 seconds. An active
|
||||
capture has a 10-minute total download deadline. Responses above 24 KiB stream
|
||||
into a private temporary file, then into a company-owned asset. Memory stays
|
||||
bounded by the inline prefix and stream buffers. Binary responses also become
|
||||
assets; text previews are limited to 2,000 bytes.
|
||||
|
||||
Large captures reserve 1 GiB against a **durable 4 GiB per-run capture budget**
|
||||
before creating a file. A completed capture settles to its actual byte count;
|
||||
failed/interrupted captures retain the full reservation to bound retry loops.
|
||||
A process restart does not reset that budget. Small inline responses and reads
|
||||
of existing assets need no reservation. Concurrent large captures are limited
|
||||
to **two per company and four per server process**, including the storage upload
|
||||
and temporary-file cleanup. Budget exhaustion returns `api_response_capture_limit`;
|
||||
concurrency exhaustion returns `api_response_capture_busy` without queuing more
|
||||
large transfers. A **20 GiB company-wide quota** counts all stored `runner-api` snapshots plus
|
||||
unattached reservations. Admission uses a company database lock, so runs and
|
||||
server processes share the same quota. Existing snapshots from before this
|
||||
change count too. Attaching an asset converts its reservation to actual stored
|
||||
bytes; deleting the asset frees that capacity. Small binary snapshots also need
|
||||
storage admission. Ordinary inline text/JSON and existing-asset pages do not.
|
||||
|
||||
Operators can set `PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES` to a positive
|
||||
safe integer of at least 1 GiB. Invalid values fall back to 20 GiB. No zero or
|
||||
unlimited setting is accepted. This quota covers API snapshots, not all company
|
||||
attachments. Storage capacity and backend limits still apply.
|
||||
|
||||
Handled pre-storage failures release the company reservation after temporary
|
||||
file cleanup, but retain the run's charge against retry loops. After a metadata
|
||||
transaction fails, a locking read must prove the asset was not committed before
|
||||
the uploaded object is removed. Confirmed cleanup refunds the company quota. A crash, failed
|
||||
cleanup, or ambiguous storage write keeps an unattached reservation. Operators
|
||||
must reconcile possible orphan files/objects before deleting that reservation
|
||||
from `runner_api_response_reservations`; no automatic expiry silently refunds
|
||||
possibly occupied space. Linked rows are removed with their asset. Deleting a
|
||||
run does not release its unattached company reservations.
|
||||
|
||||
Temporary files are removed on success or failure. Long captures revalidate the
|
||||
active run at least every MiB or at the next chunk after one second, and again
|
||||
before returning the snapshot. Stopping the run stops its download.
|
||||
|
||||
To inspect saved text without creating another artifact, call its authorized
|
||||
content operation with `responseText`:
|
||||
|
||||
```json
|
||||
{"operationId":"GET /api/assets/{assetId}/content","pathParams":{"assetId":"RETURNED_ARTIFACT_ID"},"responseText":{"offsetBytes":0,"limitBytes":8192}}
|
||||
```
|
||||
|
||||
The result contains `data` as text (including JSON), plus `responseText` with
|
||||
`offsetBytes`, `nextOffsetBytes`, and `totalBytes`. Continue at `nextOffsetBytes`
|
||||
until it is null. Windows end at UTF-8 boundaries. The limit defaults to 24 KiB
|
||||
and accepts 4–24,576 bytes. The server rejects binary content, invalid UTF-8,
|
||||
and offsets inside a code point or beyond the response. This option only works
|
||||
with GET; never repeat a mutation to retrieve another part of its response.
|
||||
Read the saved artifact for a stable snapshot instead of paging a changing live
|
||||
response. A text-window call against a live response above 24 KiB also returns
|
||||
that snapshot's artifact reference; continue on its content operation. A saved
|
||||
asset page never creates another asset. An unpaged asset read also fetches only
|
||||
a bounded preview and returns the existing reference instead of copying the file. Offsets and total sizes use safe integer
|
||||
byte counts, including values above 2 GiB. Existing assets larger than the 1 GiB
|
||||
capture limit remain readable because each request transfers only a bounded range.
|
||||
Request/upload limits and all route
|
||||
authorization remain.
|
||||
Saved asset pages use authenticated HTTP byte ranges. The storage provider reads
|
||||
only the requested window, with at most two extra bytes for UTF-8/EOF handling.
|
||||
The client validates `Content-Range`, the total size, and the received byte count;
|
||||
it rejects unsupported or inconsistent ranges instead of downloading the whole
|
||||
asset for every page. Other GET routes are fetched once in full to create the
|
||||
snapshot, so use the returned asset for subsequent pages. S3 uses streamed
|
||||
multipart uploads for large snapshots. Asset sizes are stored as PostgreSQL
|
||||
`bigint`, preserving the existing numeric API shape.
|
||||
|
||||
### Media files and upload limits
|
||||
|
||||
The 1 GiB limit covers new snapshots returned by `call_api`, such as large JSON
|
||||
exports or binary API downloads. It does not raise attachment upload limits or
|
||||
limit files an agent creates and edits inside its workspace. Saved asset downloads
|
||||
stream from storage and support byte ranges, including video seeking.
|
||||
|
||||
`PAPERCLIP_ATTACHMENT_MAX_BYTES` separately defaults to 10 MiB for uploads and
|
||||
native file handoffs. `call_api` uploads also have their own 10 MiB limit. Several
|
||||
upload and handoff paths buffer complete files in memory; raising those defaults
|
||||
to GiB sizes requires streaming ingestion and corresponding admission/budget
|
||||
controls first. For a future video attachment workflow, 2 GiB per streamed file
|
||||
is a reasonable default, with an operator override and storage quotas. Do not
|
||||
claim that this response-paging change enables GiB attachment uploads.
|
||||
|
||||
Tool responses identify the HTTP route with `apiOperationId`. The native protocol
|
||||
reserves `operationId` and `callId` for semantic tool-call identity; API metadata
|
||||
must not masquerade as that envelope. Saved mutation receipts are normalized at
|
||||
the tool boundary as well, without repeating their HTTP request.
|
||||
All redirects are refused. Oversized or interrupted mutation responses have an
|
||||
All redirects are refused. Interrupted mutation responses have an
|
||||
unknown outcome, requiring inspection before another mutation.
|
||||
Mutation responses with HTTP 5xx, HTTP 408, redirects, or malformed JSON also
|
||||
retain an unknown outcome. A server may have committed the write before it
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
ALTER TABLE "assets" ALTER COLUMN "byte_size" SET DATA TYPE bigint;
|
||||
@@ -0,0 +1,13 @@
|
||||
CREATE TABLE IF NOT EXISTS "runner_api_response_reservations" (
|
||||
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
|
||||
"company_id" uuid NOT NULL,
|
||||
"run_id" uuid,
|
||||
"asset_id" uuid,
|
||||
"reserved_bytes" bigint NOT NULL,
|
||||
"created_at" timestamp with time zone DEFAULT now() NOT NULL
|
||||
);
|
||||
--> statement-breakpoint
|
||||
DO $$ BEGIN ALTER TABLE "runner_api_response_reservations" ADD CONSTRAINT "runner_api_response_reservations_company_id_companies_id_fk" FOREIGN KEY ("company_id") REFERENCES "public"."companies"("id") ON DELETE cascade ON UPDATE no action; EXCEPTION WHEN duplicate_object THEN NULL; END $$;--> statement-breakpoint
|
||||
DO $$ BEGIN ALTER TABLE "runner_api_response_reservations" ADD CONSTRAINT "runner_api_response_reservations_run_id_heartbeat_runs_id_fk" FOREIGN KEY ("run_id") REFERENCES "public"."heartbeat_runs"("id") ON DELETE set null ON UPDATE no action; EXCEPTION WHEN duplicate_object THEN NULL; END $$;--> statement-breakpoint
|
||||
DO $$ BEGIN ALTER TABLE "runner_api_response_reservations" ADD CONSTRAINT "runner_api_response_reservations_asset_id_assets_id_fk" FOREIGN KEY ("asset_id") REFERENCES "public"."assets"("id") ON DELETE cascade ON UPDATE no action; EXCEPTION WHEN duplicate_object THEN NULL; END $$;--> statement-breakpoint
|
||||
CREATE INDEX IF NOT EXISTS "runner_api_response_reservations_company_idx" ON "runner_api_response_reservations" USING btree ("company_id");
|
||||
+536
-3
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"id": "8d202a5a-5d6b-44d9-9336-3bb39a8b99bc",
|
||||
"prevId": "861a2815-1418-4c71-a77b-d0e2e9566413",
|
||||
"id": "c27cbbec-6948-4b2b-8db8-36d0bf3830fd",
|
||||
"prevId": "7db2ceec-ed62-4968-90c9-4c79206cfe34",
|
||||
"version": "7",
|
||||
"dialect": "postgresql",
|
||||
"tables": {
|
||||
@@ -2917,7 +2917,7 @@
|
||||
},
|
||||
"byte_size": {
|
||||
"name": "byte_size",
|
||||
"type": "integer",
|
||||
"type": "bigint",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
@@ -3302,6 +3302,13 @@
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"keyboard_shortcuts": {
|
||||
"name": "keyboard_shortcuts",
|
||||
"type": "boolean",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": false
|
||||
},
|
||||
"created_at": {
|
||||
"name": "created_at",
|
||||
"type": "timestamp with time zone",
|
||||
@@ -5701,6 +5708,12 @@
|
||||
"notNull": true,
|
||||
"default": false
|
||||
},
|
||||
"communication_guidance": {
|
||||
"name": "communication_guidance",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"state": {
|
||||
"name": "state",
|
||||
"type": "text",
|
||||
@@ -6611,6 +6624,13 @@
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"communication_instructions": {
|
||||
"name": "communication_instructions",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "''"
|
||||
},
|
||||
"allow_direct_messages": {
|
||||
"name": "allow_direct_messages",
|
||||
"type": "boolean",
|
||||
@@ -8096,6 +8116,519 @@
|
||||
},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.chat_github_configurations": {
|
||||
"name": "chat_github_configurations",
|
||||
"schema": "",
|
||||
"columns": {
|
||||
"endpoint_id": {
|
||||
"name": "endpoint_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": true,
|
||||
"notNull": true
|
||||
},
|
||||
"company_id": {
|
||||
"name": "company_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"revision": {
|
||||
"name": "revision",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": 1
|
||||
},
|
||||
"configuration": {
|
||||
"name": "configuration",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"updated_by_user_id": {
|
||||
"name": "updated_by_user_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"created_at": {
|
||||
"name": "created_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
},
|
||||
"updated_at": {
|
||||
"name": "updated_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
}
|
||||
},
|
||||
"indexes": {},
|
||||
"foreignKeys": {
|
||||
"chat_github_configurations_company_id_endpoint_id_chat_endpoints_company_id_id_fk": {
|
||||
"name": "chat_github_configurations_company_id_endpoint_id_chat_endpoints_company_id_id_fk",
|
||||
"tableFrom": "chat_github_configurations",
|
||||
"tableTo": "chat_endpoints",
|
||||
"columnsFrom": [
|
||||
"company_id",
|
||||
"endpoint_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"company_id",
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
},
|
||||
"compositePrimaryKeys": {},
|
||||
"uniqueConstraints": {},
|
||||
"policies": {},
|
||||
"checkConstraints": {
|
||||
"chat_github_config_revision_check": {
|
||||
"name": "chat_github_config_revision_check",
|
||||
"value": "\"chat_github_configurations\".\"revision\" > 0"
|
||||
}
|
||||
},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.chat_github_registrations": {
|
||||
"name": "chat_github_registrations",
|
||||
"schema": "",
|
||||
"columns": {
|
||||
"id": {
|
||||
"name": "id",
|
||||
"type": "uuid",
|
||||
"primaryKey": true,
|
||||
"notNull": true,
|
||||
"default": "gen_random_uuid()"
|
||||
},
|
||||
"company_id": {
|
||||
"name": "company_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"endpoint_id": {
|
||||
"name": "endpoint_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"user_id": {
|
||||
"name": "user_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"state_hash": {
|
||||
"name": "state_hash",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"trusted_origin": {
|
||||
"name": "trusted_origin",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"status": {
|
||||
"name": "status",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "'pending'"
|
||||
},
|
||||
"expires_at": {
|
||||
"name": "expires_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"consumed_at": {
|
||||
"name": "consumed_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"created_at": {
|
||||
"name": "created_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"chat_github_registration_state_uq": {
|
||||
"name": "chat_github_registration_state_uq",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "state_hash",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": true,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"chat_github_registration_endpoint_idx": {
|
||||
"name": "chat_github_registration_endpoint_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "endpoint_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "expires_at",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
"chat_github_registrations_company_id_endpoint_id_chat_endpoints_company_id_id_fk": {
|
||||
"name": "chat_github_registrations_company_id_endpoint_id_chat_endpoints_company_id_id_fk",
|
||||
"tableFrom": "chat_github_registrations",
|
||||
"tableTo": "chat_endpoints",
|
||||
"columnsFrom": [
|
||||
"company_id",
|
||||
"endpoint_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"company_id",
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
},
|
||||
"compositePrimaryKeys": {},
|
||||
"uniqueConstraints": {},
|
||||
"policies": {},
|
||||
"checkConstraints": {
|
||||
"chat_github_registration_status_check": {
|
||||
"name": "chat_github_registration_status_check",
|
||||
"value": "\"chat_github_registrations\".\"status\" in ('pending', 'exchanging', 'completed', 'failed')"
|
||||
}
|
||||
},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.chat_github_reviews": {
|
||||
"name": "chat_github_reviews",
|
||||
"schema": "",
|
||||
"columns": {
|
||||
"id": {
|
||||
"name": "id",
|
||||
"type": "uuid",
|
||||
"primaryKey": true,
|
||||
"notNull": true,
|
||||
"default": "gen_random_uuid()"
|
||||
},
|
||||
"company_id": {
|
||||
"name": "company_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"endpoint_id": {
|
||||
"name": "endpoint_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"issue_id": {
|
||||
"name": "issue_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"run_id": {
|
||||
"name": "run_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"repository_id": {
|
||||
"name": "repository_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"repository": {
|
||||
"name": "repository",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"pull_number": {
|
||||
"name": "pull_number",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"head_sha": {
|
||||
"name": "head_sha",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"delivery_id": {
|
||||
"name": "delivery_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"configuration_revision": {
|
||||
"name": "configuration_revision",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"policy_snapshot": {
|
||||
"name": "policy_snapshot",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"event": {
|
||||
"name": "event",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"state": {
|
||||
"name": "state",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "'queued'"
|
||||
},
|
||||
"assessment": {
|
||||
"name": "assessment",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"conclusion": {
|
||||
"name": "conclusion",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"check_id": {
|
||||
"name": "check_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"check_url": {
|
||||
"name": "check_url",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"summary_id": {
|
||||
"name": "summary_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"summary_url": {
|
||||
"name": "summary_url",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"publication_receipts": {
|
||||
"name": "publication_receipts",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "'{}'::jsonb"
|
||||
},
|
||||
"created_at": {
|
||||
"name": "created_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
},
|
||||
"updated_at": {
|
||||
"name": "updated_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"chat_github_reviews_delivery_uq": {
|
||||
"name": "chat_github_reviews_delivery_uq",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "endpoint_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "delivery_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": true,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"chat_github_reviews_pull_idx": {
|
||||
"name": "chat_github_reviews_pull_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "endpoint_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "repository_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "pull_number",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "created_at",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"chat_github_reviews_task_idx": {
|
||||
"name": "chat_github_reviews_task_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "company_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "issue_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "run_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
"chat_github_reviews_run_id_heartbeat_runs_id_fk": {
|
||||
"name": "chat_github_reviews_run_id_heartbeat_runs_id_fk",
|
||||
"tableFrom": "chat_github_reviews",
|
||||
"tableTo": "heartbeat_runs",
|
||||
"columnsFrom": [
|
||||
"run_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "set null",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
"chat_github_reviews_company_id_endpoint_id_chat_endpoints_company_id_id_fk": {
|
||||
"name": "chat_github_reviews_company_id_endpoint_id_chat_endpoints_company_id_id_fk",
|
||||
"tableFrom": "chat_github_reviews",
|
||||
"tableTo": "chat_endpoints",
|
||||
"columnsFrom": [
|
||||
"company_id",
|
||||
"endpoint_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"company_id",
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
"chat_github_reviews_company_id_issue_id_issues_company_id_id_fk": {
|
||||
"name": "chat_github_reviews_company_id_issue_id_issues_company_id_id_fk",
|
||||
"tableFrom": "chat_github_reviews",
|
||||
"tableTo": "issues",
|
||||
"columnsFrom": [
|
||||
"company_id",
|
||||
"issue_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"company_id",
|
||||
"id"
|
||||
],
|
||||
"onDelete": "restrict",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
},
|
||||
"compositePrimaryKeys": {},
|
||||
"uniqueConstraints": {
|
||||
"chat_github_reviews_company_id_uq": {
|
||||
"name": "chat_github_reviews_company_id_uq",
|
||||
"nullsNotDistinct": false,
|
||||
"columns": [
|
||||
"company_id",
|
||||
"id"
|
||||
]
|
||||
}
|
||||
},
|
||||
"policies": {},
|
||||
"checkConstraints": {
|
||||
"chat_github_review_pull_number_check": {
|
||||
"name": "chat_github_review_pull_number_check",
|
||||
"value": "\"chat_github_reviews\".\"pull_number\" > 0"
|
||||
},
|
||||
"chat_github_review_state_check": {
|
||||
"name": "chat_github_review_state_check",
|
||||
"value": "\"chat_github_reviews\".\"state\" in ('queued', 'running', 'completed', 'incomplete', 'error', 'superseded', 'manual_required')"
|
||||
}
|
||||
},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.chat_teams_file_transfers": {
|
||||
"name": "chat_teams_file_transfers",
|
||||
"schema": "",
|
||||
+757
-3
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"id": "861a2815-1418-4c71-a77b-d0e2e9566413",
|
||||
"prevId": "4770ad6d-2cdc-45c4-af89-c300fd134d91",
|
||||
"id": "33120bf3-9cac-4909-ae94-d514ddc1e9e2",
|
||||
"prevId": "c27cbbec-6948-4b2b-8db8-36d0bf3830fd",
|
||||
"version": "7",
|
||||
"dialect": "postgresql",
|
||||
"tables": {
|
||||
@@ -2917,7 +2917,7 @@
|
||||
},
|
||||
"byte_size": {
|
||||
"name": "byte_size",
|
||||
"type": "integer",
|
||||
"type": "bigint",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
@@ -3302,6 +3302,13 @@
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"keyboard_shortcuts": {
|
||||
"name": "keyboard_shortcuts",
|
||||
"type": "boolean",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": false
|
||||
},
|
||||
"created_at": {
|
||||
"name": "created_at",
|
||||
"type": "timestamp with time zone",
|
||||
@@ -5701,6 +5708,12 @@
|
||||
"notNull": true,
|
||||
"default": false
|
||||
},
|
||||
"communication_guidance": {
|
||||
"name": "communication_guidance",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"state": {
|
||||
"name": "state",
|
||||
"type": "text",
|
||||
@@ -6611,6 +6624,13 @@
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"communication_instructions": {
|
||||
"name": "communication_instructions",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "''"
|
||||
},
|
||||
"allow_direct_messages": {
|
||||
"name": "allow_direct_messages",
|
||||
"type": "boolean",
|
||||
@@ -8096,6 +8116,519 @@
|
||||
},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.chat_github_configurations": {
|
||||
"name": "chat_github_configurations",
|
||||
"schema": "",
|
||||
"columns": {
|
||||
"endpoint_id": {
|
||||
"name": "endpoint_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": true,
|
||||
"notNull": true
|
||||
},
|
||||
"company_id": {
|
||||
"name": "company_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"revision": {
|
||||
"name": "revision",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": 1
|
||||
},
|
||||
"configuration": {
|
||||
"name": "configuration",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"updated_by_user_id": {
|
||||
"name": "updated_by_user_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"created_at": {
|
||||
"name": "created_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
},
|
||||
"updated_at": {
|
||||
"name": "updated_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
}
|
||||
},
|
||||
"indexes": {},
|
||||
"foreignKeys": {
|
||||
"chat_github_configurations_company_id_endpoint_id_chat_endpoints_company_id_id_fk": {
|
||||
"name": "chat_github_configurations_company_id_endpoint_id_chat_endpoints_company_id_id_fk",
|
||||
"tableFrom": "chat_github_configurations",
|
||||
"tableTo": "chat_endpoints",
|
||||
"columnsFrom": [
|
||||
"company_id",
|
||||
"endpoint_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"company_id",
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
},
|
||||
"compositePrimaryKeys": {},
|
||||
"uniqueConstraints": {},
|
||||
"policies": {},
|
||||
"checkConstraints": {
|
||||
"chat_github_config_revision_check": {
|
||||
"name": "chat_github_config_revision_check",
|
||||
"value": "\"chat_github_configurations\".\"revision\" > 0"
|
||||
}
|
||||
},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.chat_github_registrations": {
|
||||
"name": "chat_github_registrations",
|
||||
"schema": "",
|
||||
"columns": {
|
||||
"id": {
|
||||
"name": "id",
|
||||
"type": "uuid",
|
||||
"primaryKey": true,
|
||||
"notNull": true,
|
||||
"default": "gen_random_uuid()"
|
||||
},
|
||||
"company_id": {
|
||||
"name": "company_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"endpoint_id": {
|
||||
"name": "endpoint_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"user_id": {
|
||||
"name": "user_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"state_hash": {
|
||||
"name": "state_hash",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"trusted_origin": {
|
||||
"name": "trusted_origin",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"status": {
|
||||
"name": "status",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "'pending'"
|
||||
},
|
||||
"expires_at": {
|
||||
"name": "expires_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"consumed_at": {
|
||||
"name": "consumed_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"created_at": {
|
||||
"name": "created_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"chat_github_registration_state_uq": {
|
||||
"name": "chat_github_registration_state_uq",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "state_hash",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": true,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"chat_github_registration_endpoint_idx": {
|
||||
"name": "chat_github_registration_endpoint_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "endpoint_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "expires_at",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
"chat_github_registrations_company_id_endpoint_id_chat_endpoints_company_id_id_fk": {
|
||||
"name": "chat_github_registrations_company_id_endpoint_id_chat_endpoints_company_id_id_fk",
|
||||
"tableFrom": "chat_github_registrations",
|
||||
"tableTo": "chat_endpoints",
|
||||
"columnsFrom": [
|
||||
"company_id",
|
||||
"endpoint_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"company_id",
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
},
|
||||
"compositePrimaryKeys": {},
|
||||
"uniqueConstraints": {},
|
||||
"policies": {},
|
||||
"checkConstraints": {
|
||||
"chat_github_registration_status_check": {
|
||||
"name": "chat_github_registration_status_check",
|
||||
"value": "\"chat_github_registrations\".\"status\" in ('pending', 'exchanging', 'completed', 'failed')"
|
||||
}
|
||||
},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.chat_github_reviews": {
|
||||
"name": "chat_github_reviews",
|
||||
"schema": "",
|
||||
"columns": {
|
||||
"id": {
|
||||
"name": "id",
|
||||
"type": "uuid",
|
||||
"primaryKey": true,
|
||||
"notNull": true,
|
||||
"default": "gen_random_uuid()"
|
||||
},
|
||||
"company_id": {
|
||||
"name": "company_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"endpoint_id": {
|
||||
"name": "endpoint_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"issue_id": {
|
||||
"name": "issue_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"run_id": {
|
||||
"name": "run_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"repository_id": {
|
||||
"name": "repository_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"repository": {
|
||||
"name": "repository",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"pull_number": {
|
||||
"name": "pull_number",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"head_sha": {
|
||||
"name": "head_sha",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"delivery_id": {
|
||||
"name": "delivery_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"configuration_revision": {
|
||||
"name": "configuration_revision",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"policy_snapshot": {
|
||||
"name": "policy_snapshot",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"event": {
|
||||
"name": "event",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"state": {
|
||||
"name": "state",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "'queued'"
|
||||
},
|
||||
"assessment": {
|
||||
"name": "assessment",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"conclusion": {
|
||||
"name": "conclusion",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"check_id": {
|
||||
"name": "check_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"check_url": {
|
||||
"name": "check_url",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"summary_id": {
|
||||
"name": "summary_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"summary_url": {
|
||||
"name": "summary_url",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"publication_receipts": {
|
||||
"name": "publication_receipts",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "'{}'::jsonb"
|
||||
},
|
||||
"created_at": {
|
||||
"name": "created_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
},
|
||||
"updated_at": {
|
||||
"name": "updated_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"chat_github_reviews_delivery_uq": {
|
||||
"name": "chat_github_reviews_delivery_uq",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "endpoint_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "delivery_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": true,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"chat_github_reviews_pull_idx": {
|
||||
"name": "chat_github_reviews_pull_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "endpoint_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "repository_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "pull_number",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "created_at",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"chat_github_reviews_task_idx": {
|
||||
"name": "chat_github_reviews_task_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "company_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "issue_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "run_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
"chat_github_reviews_run_id_heartbeat_runs_id_fk": {
|
||||
"name": "chat_github_reviews_run_id_heartbeat_runs_id_fk",
|
||||
"tableFrom": "chat_github_reviews",
|
||||
"tableTo": "heartbeat_runs",
|
||||
"columnsFrom": [
|
||||
"run_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "set null",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
"chat_github_reviews_company_id_endpoint_id_chat_endpoints_company_id_id_fk": {
|
||||
"name": "chat_github_reviews_company_id_endpoint_id_chat_endpoints_company_id_id_fk",
|
||||
"tableFrom": "chat_github_reviews",
|
||||
"tableTo": "chat_endpoints",
|
||||
"columnsFrom": [
|
||||
"company_id",
|
||||
"endpoint_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"company_id",
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
"chat_github_reviews_company_id_issue_id_issues_company_id_id_fk": {
|
||||
"name": "chat_github_reviews_company_id_issue_id_issues_company_id_id_fk",
|
||||
"tableFrom": "chat_github_reviews",
|
||||
"tableTo": "issues",
|
||||
"columnsFrom": [
|
||||
"company_id",
|
||||
"issue_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"company_id",
|
||||
"id"
|
||||
],
|
||||
"onDelete": "restrict",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
},
|
||||
"compositePrimaryKeys": {},
|
||||
"uniqueConstraints": {
|
||||
"chat_github_reviews_company_id_uq": {
|
||||
"name": "chat_github_reviews_company_id_uq",
|
||||
"nullsNotDistinct": false,
|
||||
"columns": [
|
||||
"company_id",
|
||||
"id"
|
||||
]
|
||||
}
|
||||
},
|
||||
"policies": {},
|
||||
"checkConstraints": {
|
||||
"chat_github_review_pull_number_check": {
|
||||
"name": "chat_github_review_pull_number_check",
|
||||
"value": "\"chat_github_reviews\".\"pull_number\" > 0"
|
||||
},
|
||||
"chat_github_review_state_check": {
|
||||
"name": "chat_github_review_state_check",
|
||||
"value": "\"chat_github_reviews\".\"state\" in ('queued', 'running', 'completed', 'incomplete', 'error', 'superseded', 'manual_required')"
|
||||
}
|
||||
},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.chat_teams_file_transfers": {
|
||||
"name": "chat_teams_file_transfers",
|
||||
"schema": "",
|
||||
@@ -37416,6 +37949,26 @@
|
||||
"notNull": true,
|
||||
"default": true
|
||||
},
|
||||
"setup_pending": {
|
||||
"name": "setup_pending",
|
||||
"type": "boolean",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": false
|
||||
},
|
||||
"archived": {
|
||||
"name": "archived",
|
||||
"type": "boolean",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": false
|
||||
},
|
||||
"last_webhook_delivery": {
|
||||
"name": "last_webhook_delivery",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"cron_expression": {
|
||||
"name": "cron_expression",
|
||||
"type": "text",
|
||||
@@ -37677,6 +38230,100 @@
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.routine_webhook_test_receipts": {
|
||||
"name": "routine_webhook_test_receipts",
|
||||
"schema": "",
|
||||
"columns": {
|
||||
"id": {
|
||||
"name": "id",
|
||||
"type": "uuid",
|
||||
"primaryKey": true,
|
||||
"notNull": true,
|
||||
"default": "gen_random_uuid()"
|
||||
},
|
||||
"company_id": {
|
||||
"name": "company_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"trigger_id": {
|
||||
"name": "trigger_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"delivery_key_hash": {
|
||||
"name": "delivery_key_hash",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"received_at": {
|
||||
"name": "received_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"routine_webhook_test_receipts_delivery_uq": {
|
||||
"name": "routine_webhook_test_receipts_delivery_uq",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "trigger_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "delivery_key_hash",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": true,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
"routine_webhook_test_receipts_company_id_companies_id_fk": {
|
||||
"name": "routine_webhook_test_receipts_company_id_companies_id_fk",
|
||||
"tableFrom": "routine_webhook_test_receipts",
|
||||
"tableTo": "companies",
|
||||
"columnsFrom": [
|
||||
"company_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
"routine_webhook_test_receipts_trigger_id_routine_triggers_id_fk": {
|
||||
"name": "routine_webhook_test_receipts_trigger_id_routine_triggers_id_fk",
|
||||
"tableFrom": "routine_webhook_test_receipts",
|
||||
"tableTo": "routine_triggers",
|
||||
"columnsFrom": [
|
||||
"trigger_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
},
|
||||
"compositePrimaryKeys": {},
|
||||
"uniqueConstraints": {},
|
||||
"policies": {},
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.routines": {
|
||||
"name": "routines",
|
||||
"schema": "",
|
||||
@@ -38292,6 +38939,113 @@
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.runner_api_response_reservations": {
|
||||
"name": "runner_api_response_reservations",
|
||||
"schema": "",
|
||||
"columns": {
|
||||
"id": {
|
||||
"name": "id",
|
||||
"type": "uuid",
|
||||
"primaryKey": true,
|
||||
"notNull": true,
|
||||
"default": "gen_random_uuid()"
|
||||
},
|
||||
"company_id": {
|
||||
"name": "company_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"run_id": {
|
||||
"name": "run_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"asset_id": {
|
||||
"name": "asset_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"reserved_bytes": {
|
||||
"name": "reserved_bytes",
|
||||
"type": "bigint",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"created_at": {
|
||||
"name": "created_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"runner_api_response_reservations_company_idx": {
|
||||
"name": "runner_api_response_reservations_company_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "company_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
"runner_api_response_reservations_company_id_companies_id_fk": {
|
||||
"name": "runner_api_response_reservations_company_id_companies_id_fk",
|
||||
"tableFrom": "runner_api_response_reservations",
|
||||
"tableTo": "companies",
|
||||
"columnsFrom": [
|
||||
"company_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
"runner_api_response_reservations_run_id_heartbeat_runs_id_fk": {
|
||||
"name": "runner_api_response_reservations_run_id_heartbeat_runs_id_fk",
|
||||
"tableFrom": "runner_api_response_reservations",
|
||||
"tableTo": "heartbeat_runs",
|
||||
"columnsFrom": [
|
||||
"run_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "set null",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
"runner_api_response_reservations_asset_id_assets_id_fk": {
|
||||
"name": "runner_api_response_reservations_asset_id_assets_id_fk",
|
||||
"tableFrom": "runner_api_response_reservations",
|
||||
"tableTo": "assets",
|
||||
"columnsFrom": [
|
||||
"asset_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
},
|
||||
"compositePrimaryKeys": {},
|
||||
"uniqueConstraints": {},
|
||||
"policies": {},
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.secret_access_events": {
|
||||
"name": "secret_access_events",
|
||||
"schema": "",
|
||||
+14
@@ -1982,6 +1982,20 @@
|
||||
"when": 1790439010225,
|
||||
"tag": "0284_petite_genesis",
|
||||
"breakpoints": true
|
||||
},
|
||||
{
|
||||
"idx": 285,
|
||||
"version": "7",
|
||||
"when": 1790603883247,
|
||||
"tag": "0285_heavy_oracle",
|
||||
"breakpoints": true
|
||||
},
|
||||
{
|
||||
"idx": 286,
|
||||
"version": "7",
|
||||
"when": 1790607525758,
|
||||
"tag": "0286_complete_alice",
|
||||
"breakpoints": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
import { pgTable, uuid, text, integer, timestamp, index, uniqueIndex } from "drizzle-orm/pg-core";
|
||||
import { pgTable, uuid, text, bigint, timestamp, index, uniqueIndex } from "drizzle-orm/pg-core";
|
||||
import { companies } from "./companies.js";
|
||||
import { agents } from "./agents.js";
|
||||
|
||||
@@ -10,7 +10,7 @@ export const assets = pgTable(
|
||||
provider: text("provider").notNull(),
|
||||
objectKey: text("object_key").notNull(),
|
||||
contentType: text("content_type").notNull(),
|
||||
byteSize: integer("byte_size").notNull(),
|
||||
byteSize: bigint("byte_size", { mode: "number" }).notNull(),
|
||||
sha256: text("sha256").notNull(),
|
||||
originalFilename: text("original_filename"),
|
||||
createdByAgentId: uuid("created_by_agent_id").references(() => agents.id),
|
||||
|
||||
@@ -115,6 +115,7 @@ export {
|
||||
export { feedbackExports } from "./feedback_exports.js";
|
||||
export { issueReadStates } from "./issue_read_states.js";
|
||||
export { assets } from "./assets.js";
|
||||
export { runnerApiResponseReservations } from "./runner_api_response_reservations.js";
|
||||
export { issueAttachments } from "./issue_attachments.js";
|
||||
export { documents } from "./documents.js";
|
||||
export { documentRevisions } from "./document_revisions.js";
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
import { pgTable, uuid, bigint, timestamp, index } from "drizzle-orm/pg-core";
|
||||
import { companies } from "./companies.js";
|
||||
import { heartbeatRuns } from "./heartbeat_runs.js";
|
||||
import { assets } from "./assets.js";
|
||||
|
||||
export const runnerApiResponseReservations = pgTable("runner_api_response_reservations", {
|
||||
id: uuid("id").primaryKey().defaultRandom(),
|
||||
companyId: uuid("company_id").notNull().references(() => companies.id, { onDelete: "cascade" }),
|
||||
runId: uuid("run_id").references(() => heartbeatRuns.id, { onDelete: "set null" }),
|
||||
assetId: uuid("asset_id").references(() => assets.id, { onDelete: "cascade" }),
|
||||
reservedBytes: bigint("reserved_bytes", { mode: "number" }).notNull(),
|
||||
createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(),
|
||||
}, table => ({ companyIdx: index("runner_api_response_reservations_company_idx").on(table.companyId) }));
|
||||
@@ -1 +1 @@
|
||||
[{"annotations":{"exposure":"always","operationId":"get_task_context","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read the active task and actor, including the exact approved Markdown revision when this issue has an accepted plan.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"get_task_context","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"get_task_history","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read bounded comments on the active task.","inputSchema":{"additionalProperties":false,"properties":{"limit":{"default":50,"maximum":200,"minimum":1,"type":"integer"}},"required":[],"type":"object"},"name":"get_task_history","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"list_documents","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"List revisioned documents on the active task.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_documents","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"read_document","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read the current revision of one active-task document.","inputSchema":{"additionalProperties":false,"properties":{"key":{"description":"Stable issue-document key.","maxLength":120,"minLength":1,"type":"string"}},"required":["key"],"type":"object"},"name":"read_document","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"list_document_revisions","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read bounded revision history for one active-task document.","inputSchema":{"additionalProperties":false,"properties":{"key":{"description":"Stable issue-document key.","maxLength":120,"minLength":1,"type":"string"},"limit":{"default":50,"maximum":200,"minimum":1,"type":"integer"}},"required":["key"],"type":"object"},"name":"list_document_revisions","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"report_progress","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Append a durable progress comment to the active task.","inputSchema":{"additionalProperties":false,"properties":{"body":{"description":"Multiline progress update.","maxLength":20000,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","body"],"type":"object"},"name":"report_progress","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"answer_status_question","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Append the answer to a status-only wake without changing task disposition.","inputSchema":{"additionalProperties":false,"properties":{"body":{"description":"Concise status answer.","maxLength":20000,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","body"],"type":"object"},"name":"answer_status_question","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"write_document","requiredClaimLine truncated
|
||||
[{"annotations":{"exposure":"always","operationId":"get_task_context","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read the active task and actor, including the exact approved Markdown revision when this issue has an accepted plan.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"get_task_context","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"get_task_history","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read bounded comments on the active task.","inputSchema":{"additionalProperties":false,"properties":{"limit":{"default":50,"maximum":200,"minimum":1,"type":"integer"}},"required":[],"type":"object"},"name":"get_task_history","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"list_documents","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"List revisioned documents on the active task.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_documents","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"read_document","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read the current revision of one active-task document.","inputSchema":{"additionalProperties":false,"properties":{"key":{"description":"Stable issue-document key.","maxLength":120,"minLength":1,"type":"string"}},"required":["key"],"type":"object"},"name":"read_document","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"list_document_revisions","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read bounded revision history for one active-task document.","inputSchema":{"additionalProperties":false,"properties":{"key":{"description":"Stable issue-document key.","maxLength":120,"minLength":1,"type":"string"},"limit":{"default":50,"maximum":200,"minimum":1,"type":"integer"}},"required":["key"],"type":"object"},"name":"list_document_revisions","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"report_progress","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Append a durable progress comment to the active task.","inputSchema":{"additionalProperties":false,"properties":{"body":{"description":"Multiline progress update.","maxLength":20000,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","body"],"type":"object"},"name":"report_progress","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"answer_status_question","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Append the answer to a status-only wake without changing task disposition.","inputSchema":{"additionalProperties":false,"properties":{"body":{"description":"Concise status answer.","maxLength":20000,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","body"],"type":"object"},"name":"answer_status_question","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"write_document","requiredClaimLine truncated
|
||||
@@ -112,7 +112,7 @@
|
||||
"planning",
|
||||
"skill_test"
|
||||
],
|
||||
"description": "Fallback only: call a discovered Paperclip API operation when dedicated tools lack the required operation or parameters. Uses your existing permissions. Prefer dedicated tools; never bypass a denial or runner lifecycle tool.",
|
||||
"description": "Fallback only: call a discovered Paperclip API operation when dedicated tools lack the required operation or parameters. Uses your existing permissions. Prefer dedicated tools; never bypass a denial or runner lifecycle tool. For large text responses, read the returned artifact with GET /api/assets/{assetId}/content and responseText; follow nextOffsetBytes until null.",
|
||||
"effect": "write",
|
||||
"inputSchema": {
|
||||
"additionalProperties": false,
|
||||
@@ -199,6 +199,25 @@
|
||||
"query": {
|
||||
"additionalProperties": true,
|
||||
"type": "object"
|
||||
},
|
||||
"responseText": {
|
||||
"additionalProperties": false,
|
||||
"description": "GET only: return a bounded UTF-8 text window inline, including JSON as text, without saving another artifact. Offsets and limits are bytes. Use the returned nextOffsetBytes to continue; null means complete. Prefer reading a saved artifact for a stable snapshot. New live responses have a 1 GiB capture limit and a 4 GiB per-run capture budget. Existing larger assets remain readable in bounded pages. If a live response returns an artifact, continue on its content operation for a stable snapshot.",
|
||||
"properties": {
|
||||
"limitBytes": {
|
||||
"default": 24576,
|
||||
"maximum": 24576,
|
||||
"minimum": 4,
|
||||
"type": "integer"
|
||||
},
|
||||
"offsetBytes": {
|
||||
"default": 0,
|
||||
"maximum": 9007199254740991,
|
||||
"minimum": 0,
|
||||
"type": "integer"
|
||||
}
|
||||
},
|
||||
"type": "object"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
|
||||
@@ -24,7 +24,7 @@
|
||||
"prpVersion": 1,
|
||||
"nativeExecutionVersion": 1,
|
||||
"catalogVersion": 1,
|
||||
"catalogSha256": "sha256:1fa8ecf5d3c28e62a32e818de95633ff3efe5d4af1a100310516ff9b5ab2358d",
|
||||
"catalogSha256": "sha256:a19534304480dfd3d357cc902e785af05158ce1496f4d324184112732ff94c9c",
|
||||
"driverContractVersion": 1,
|
||||
"driverKind": "paperclip-deterministic",
|
||||
"driverVersion": "1.0.0"
|
||||
|
||||
@@ -160,7 +160,7 @@
|
||||
},
|
||||
{
|
||||
"path": "fixtures/evals/native-execution-seeded.json",
|
||||
"sha256": "5f3144ea856ad27778f47de73eb542703b1d9df2abad2c427838fbaf69c3d59f",
|
||||
"sha256": "c6367c7c7e3dd01fed341b7a83b0008c9d5d1efdfa85bc70e1013bc304a008c7",
|
||||
"expectation": "accept",
|
||||
"compatibilityCase": "canonical"
|
||||
},
|
||||
|
||||
@@ -28,7 +28,7 @@ export const callApiAction = {
|
||||
},
|
||||
"documentation": {
|
||||
"title": "Call the Paperclip API",
|
||||
"description": "Fallback only: call a discovered Paperclip API operation when dedicated tools lack the required operation or parameters. Uses your existing permissions. Prefer dedicated tools; never bypass a denial or runner lifecycle tool.",
|
||||
"description": "Fallback only: call a discovered Paperclip API operation when dedicated tools lack the required operation or parameters. Uses your existing permissions. Prefer dedicated tools; never bypass a denial or runner lifecycle tool. For large text responses, read the returned artifact with GET /api/assets/{assetId}/content and responseText; follow nextOffsetBytes until null.",
|
||||
"note": "Production HTTP fallback; does not grant privileges or replace dedicated tools."
|
||||
},
|
||||
"examples": {
|
||||
@@ -51,7 +51,7 @@ export const callApiAction = {
|
||||
"operationId": "call_api",
|
||||
"version": 1,
|
||||
"title": "Call the Paperclip API",
|
||||
"description": "Fallback only: call a discovered Paperclip API operation when dedicated tools lack the required operation or parameters. Uses your existing permissions. Prefer dedicated tools; never bypass a denial or runner lifecycle tool.",
|
||||
"description": "Fallback only: call a discovered Paperclip API operation when dedicated tools lack the required operation or parameters. Uses your existing permissions. Prefer dedicated tools; never bypass a denial or runner lifecycle tool. For large text responses, read the returned artifact with GET /api/assets/{assetId}/content and responseText; follow nextOffsetBytes until null.",
|
||||
"exposure": "optional",
|
||||
"requiredClaims": [
|
||||
"api:call"
|
||||
@@ -96,6 +96,15 @@ export const callApiAction = {
|
||||
"type": "string",
|
||||
"maxLength": 120
|
||||
},
|
||||
"responseText": {
|
||||
"description": "GET only: return a bounded UTF-8 text window inline, including JSON as text, without saving another artifact. Offsets and limits are bytes. Use the returned nextOffsetBytes to continue; null means complete. Prefer reading a saved artifact for a stable snapshot. New live responses have a 1 GiB capture limit and a 4 GiB per-run capture budget. Existing larger assets remain readable in bounded pages. If a live response returns an artifact, continue on its content operation for a stable snapshot.",
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"offsetBytes": { "type": "integer", "minimum": 0, "maximum": 9007199254740991, "default": 0 },
|
||||
"limitBytes": { "type": "integer", "minimum": 4, "maximum": 24576, "default": 24576 }
|
||||
},
|
||||
"additionalProperties": false
|
||||
},
|
||||
"files": {
|
||||
"type": "array",
|
||||
"maxItems": 10,
|
||||
|
||||
@@ -4,6 +4,7 @@ export interface AssetImage {
|
||||
provider: string;
|
||||
objectKey: string;
|
||||
contentType: string;
|
||||
/** Safe integer byte count; persisted as bigint to support files above 2 GiB. */
|
||||
byteSize: number;
|
||||
sha256: string;
|
||||
originalFilename: string | null;
|
||||
|
||||
@@ -421,6 +421,29 @@ describe("GET /api/assets/:assetId/content", () => {
|
||||
getAssetByIdMock.mockReset();
|
||||
});
|
||||
|
||||
it("reads only the requested storage range for a large asset", async () => {
|
||||
const storage = createStorageService("text/plain");
|
||||
getAssetByIdMock.mockResolvedValue({ ...createAsset(), contentType: "text/plain", byteSize: 10 * 1024 * 1024 });
|
||||
vi.mocked(storage.getObject).mockImplementation(async (_company, _key, options) => {
|
||||
expect(options).toEqual({ range: { start: 8192, end: 12288 } });
|
||||
return { stream: Readable.from(Buffer.alloc(4097, 65)), contentLength: 4097 };
|
||||
});
|
||||
const res = await requestApp(await createApp(storage), baseUrl => request(baseUrl).get("/api/assets/asset-1/content").set("Range", "bytes=8192-12288"));
|
||||
expect(res.status).toBe(206);
|
||||
expect(res.headers["content-range"]).toBe("bytes 8192-12288/10485760");
|
||||
expect(res.headers["content-length"]).toBe("4097");
|
||||
expect(res.text).toBe("A".repeat(4097));
|
||||
});
|
||||
|
||||
it.each(["bytes=999-1000", "bytes=0-1,3-4", "bytes=1x-3", "bytes=-", "items=0-3"])("rejects invalid asset ranges before storage reads: %s", async range => {
|
||||
const storage = createStorageService();
|
||||
getAssetByIdMock.mockResolvedValue(createAsset());
|
||||
const res = await requestApp(await createApp(storage), baseUrl => request(baseUrl).get("/api/assets/asset-1/content").set("Range", range));
|
||||
expect(res.status).toBe(416);
|
||||
expect(res.headers["content-range"]).toBe("bytes */40");
|
||||
expect(storage.getObject).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("downloads script-capable HTML with nosniff and a sandbox CSP", async () => {
|
||||
const html = Buffer.from("<script>globalThis.__assetXss = true</script>");
|
||||
const storage = createStorageService("text/html");
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { Readable } from "node:stream";
|
||||
import { createHash } from "node:crypto";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { promises as fs } from "node:fs";
|
||||
@@ -42,6 +44,27 @@ describe("local disk storage provider", () => {
|
||||
expect(stored.sha256).toHaveLength(64);
|
||||
});
|
||||
|
||||
it("writes streamed files with exact metadata", async () => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-storage-"));
|
||||
tempRoots.push(root);
|
||||
const service = createStorageService(createLocalDiskStorageProvider(root));
|
||||
const bytes = Buffer.from("streamed response");
|
||||
const sha256 = createHash("sha256").update(bytes).digest("hex");
|
||||
const stored = await service.putFile({ companyId: "company-1", namespace: "runner-api", originalFilename: "response.txt",
|
||||
contentType: "text/plain", body: Readable.from([bytes.subarray(0, 4), bytes.subarray(4)]), byteSize: bytes.length, sha256 });
|
||||
expect(stored).toMatchObject({ byteSize: bytes.length, sha256 });
|
||||
expect(await readStreamToBuffer((await service.getObject("company-1", stored.objectKey)).stream)).toEqual(bytes);
|
||||
});
|
||||
|
||||
it("removes partial local uploads when their source fails", async () => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-storage-"));
|
||||
tempRoots.push(root);
|
||||
const provider = createLocalDiskStorageProvider(root);
|
||||
const body = Readable.from((async function* () { yield Buffer.from("partial"); throw new Error("source failed"); })());
|
||||
await expect(provider.putObject({ objectKey: "company-1/failed", contentType: "text/plain", contentLength: 20, body })).rejects.toThrow("source failed");
|
||||
expect(await fs.readdir(path.join(root, "company-1"))).toEqual([]);
|
||||
});
|
||||
|
||||
it("streams only requested byte ranges", async () => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-storage-"));
|
||||
tempRoots.push(root);
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
import { AbortMultipartUploadCommand, CompleteMultipartUploadCommand, CreateMultipartUploadCommand, UploadPartCommand, S3Client } from "@aws-sdk/client-s3";
|
||||
import { Readable } from "node:stream";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { putS3Multipart } from "../storage/s3-multipart.js";
|
||||
|
||||
describe("streamed S3 multipart uploads", () => {
|
||||
it("streams exact ordered parts without buffering the full response", async () => {
|
||||
const client = new S3Client({ region: "us-east-1" });
|
||||
const lengths: number[] = [];
|
||||
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
|
||||
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
|
||||
if (command instanceof UploadPartCommand) {
|
||||
expect(command.input.Body).toBeInstanceOf(Readable);
|
||||
let length = 0;
|
||||
for await (const chunk of command.input.Body as Readable) {
|
||||
expect(chunk.length).toBeLessThanOrEqual(64 * 1024);
|
||||
length += chunk.length;
|
||||
}
|
||||
expect(length).toBe(command.input.ContentLength);
|
||||
lengths.push(length);
|
||||
return { ETag: `part-${command.input.PartNumber}` };
|
||||
}
|
||||
expect(command).toBeInstanceOf(CompleteMultipartUploadCommand);
|
||||
expect(command.input.MultipartUpload.Parts).toEqual([1, 2, 3].map(PartNumber => ({ PartNumber, ETag: `part-${PartNumber}` })));
|
||||
return {};
|
||||
});
|
||||
const body = Readable.from((async function* () { for (let i = 0; i < 272; i++) yield Buffer.alloc(64 * 1024); })());
|
||||
try {
|
||||
await putS3Multipart(client, "bucket", "key", { objectKey: "key", contentType: "text/plain", contentLength: 17 * 1024 * 1024, body });
|
||||
expect(lengths).toEqual([8, 8, 1].map(mb => mb * 1024 * 1024));
|
||||
} finally { send.mockRestore(); client.destroy(); }
|
||||
});
|
||||
it("aborts an incomplete upload and closes its source on failure", async () => {
|
||||
const client = new S3Client({ region: "us-east-1" });
|
||||
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
|
||||
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
|
||||
if (command instanceof UploadPartCommand) throw new Error("storage unavailable");
|
||||
expect(command).toBeInstanceOf(AbortMultipartUploadCommand);
|
||||
return {};
|
||||
});
|
||||
const body = Readable.from([Buffer.from("abc")]);
|
||||
try {
|
||||
await expect(putS3Multipart(client, "bucket", "key", { objectKey: "key", contentType: "text/plain", contentLength: 3, body })).rejects.toThrow("storage unavailable");
|
||||
expect(send).toHaveBeenCalledTimes(3);
|
||||
expect(body.destroyed).toBe(true);
|
||||
} finally { send.mockRestore(); client.destroy(); }
|
||||
});
|
||||
});
|
||||
@@ -328,13 +328,32 @@ export function assetRoutes(db: Db, storage: StorageService) {
|
||||
const asset = await getAccessibleResource(req, res, svc.getById(assetId), "Asset not found");
|
||||
if (!asset) return;
|
||||
|
||||
const object = await storage.getObject(asset.companyId, asset.objectKey);
|
||||
// Use the persisted size only after resource authorization. Single ranges
|
||||
// keep saved API text pages bounded all the way to disk or object storage.
|
||||
const rawRange = req.headers.range;
|
||||
const rangeSyntax = rawRange && /^bytes=(\d*)-(\d*)$/i.exec(rawRange);
|
||||
const emptyRead = asset.byteSize === 0 && rangeSyntax?.[1] === "0";
|
||||
const ranges = rawRange && !emptyRead ? req.range(asset.byteSize) : undefined;
|
||||
res.setHeader("Accept-Ranges", "bytes");
|
||||
if (/^[a-f0-9]{64}$/.test(asset.sha256)) res.setHeader("ETag", `"${asset.sha256}"`);
|
||||
if (rawRange && (!rangeSyntax || (!rangeSyntax[1] && !rangeSyntax[2])
|
||||
|| (!emptyRead && (!Array.isArray(ranges) || ranges.length !== 1)))) {
|
||||
res.setHeader("Content-Range", `bytes */${asset.byteSize}`);
|
||||
res.status(416).end();
|
||||
return;
|
||||
}
|
||||
const range = Array.isArray(ranges) ? ranges[0] : undefined;
|
||||
const object = await storage.getObject(asset.companyId, asset.objectKey, range ? { range } : undefined);
|
||||
const responseContentType = asset.contentType || object.contentType || "application/octet-stream";
|
||||
const mediaType = responseContentType.split(";", 1)[0]?.trim().toLowerCase() ?? "";
|
||||
const inlineSafe = mediaType !== SVG_CONTENT_TYPE
|
||||
&& isInlineAttachmentContentType(mediaType);
|
||||
res.setHeader("Content-Type", responseContentType);
|
||||
res.setHeader("Content-Length", String(asset.byteSize || object.contentLength || 0));
|
||||
res.setHeader("Content-Length", String(range ? range.end - range.start + 1 : asset.byteSize || object.contentLength || 0));
|
||||
if (range) {
|
||||
res.status(206);
|
||||
res.setHeader("Content-Range", `bytes ${range.start}-${range.end}/${asset.byteSize}`);
|
||||
}
|
||||
res.setHeader("Cache-Control", "private, max-age=60");
|
||||
res.setHeader("X-Content-Type-Options", "nosniff");
|
||||
if (!inlineSafe) {
|
||||
@@ -349,6 +368,7 @@ export function assetRoutes(db: Db, storage: StorageService) {
|
||||
object.stream.on("error", (err) => {
|
||||
next(err);
|
||||
});
|
||||
res.on("close", () => object.stream.destroy());
|
||||
object.stream.pipe(res);
|
||||
});
|
||||
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { createReadStream } from "node:fs";
|
||||
import { publicChatTaskUrl } from "../chat-task-url.js";
|
||||
import type { createAssignedMcpTools } from "./assigned-mcp-tools.js";
|
||||
import { assertAssignableAgent } from "../agent-assignability.js";
|
||||
@@ -27,10 +28,13 @@ import { workspaceFileResourceService } from "../workspace-file-resources.js";
|
||||
import { badRequest, forbidden } from "../../errors.js";
|
||||
import { searchRunnerApi } from "./runner-api-catalog.js";
|
||||
import { executeRunnerApi, validateRunnerApiCall, RUNNER_API_MAX_BYTES, type RunnerApiFile } from "./runner-api-client.js";
|
||||
import { acquireRunnerApiResponseSlot, runnerApiCompanyCaptureMaxBytes, RUNNER_API_RESPONSE_MAX_BYTES, RUNNER_API_RESPONSE_RUN_MAX_BYTES, RunnerApiResponseLimitError } from "./runner-api-response-limits.js";
|
||||
import { and, desc, eq, isNull, notInArray, sql } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import {
|
||||
activityLog,
|
||||
assets,
|
||||
runnerApiResponseReservations,
|
||||
agents,
|
||||
agentWakeupRequests,
|
||||
chatEndpoints,
|
||||
@@ -527,6 +531,8 @@ export class PaperclipRunnerToolAuthority {
|
||||
const token = createLocalAgentJwt(this.binding.agentId, this.binding.companyId, bound.actor.adapterType, this.binding.runId, bound.run.responsibleUserId);
|
||||
if (!token) throw new Error("Paperclip run authentication is unavailable");
|
||||
const execute = async () => {
|
||||
let reservationId: string | undefined;
|
||||
let storageAttempted = false;
|
||||
const current = await this.#boundContext();
|
||||
if (!runnerApiToolsEnabled(this.binding.companyId, this.binding.apiToolsEnabled)) throw new Error("paperclip_runner_tool_not_advertised");
|
||||
return executeRunnerApi(input, { ...context, workMode: current.issue.workMode }, {
|
||||
@@ -537,10 +543,34 @@ export class PaperclipRunnerToolAuthority {
|
||||
validateRunnerApiCall(input, { ...context, workMode: fresh.issue.workMode });
|
||||
},
|
||||
readFile: (file) => this.#readApiFile(file),
|
||||
reserveResponseCapture: async () => {
|
||||
const reservation = await this.#reserveApiResponseCapture();
|
||||
reservationId = reservation.id;
|
||||
return (completedBytes, cleaned) => reservation.settle(completedBytes, cleaned, storageAttempted);
|
||||
},
|
||||
saveResponse: async (bytes, contentType) => {
|
||||
if (!reservationId) throw new Error("API response storage reservation is missing");
|
||||
const storage = this.binding.storage ?? getStorageService();
|
||||
const saved = await storage.putFile({ companyId: this.binding.companyId, namespace: "runner-api", originalFilename: contentType.includes("json") ? "response.json" : "response.bin", contentType, body: bytes });
|
||||
const asset = await assetService(this.db).create(this.binding.companyId, { ...saved, createdByAgentId: this.binding.agentId });
|
||||
storageAttempted = true;
|
||||
const saved = await storage.putFile({ companyId: this.binding.companyId, namespace: "runner-api", originalFilename: contentType.includes("json") ? "response.json" : "response.bin", contentType, ...(Buffer.isBuffer(bytes) ? { body: bytes } : { body: createReadStream(bytes.path), byteSize: bytes.byteSize, sha256: bytes.sha256 }) });
|
||||
const asset = await this.db.transaction(async tx => {
|
||||
await this.#lockApiCaptureCompany(tx as unknown as Db);
|
||||
const asset = await assetService(tx as unknown as Db).create(this.binding.companyId, { ...saved, createdByAgentId: this.binding.agentId });
|
||||
await tx.update(runnerApiResponseReservations).set({ assetId: asset.id, reservedBytes: asset.byteSize }).where(and(eq(runnerApiResponseReservations.id, reservationId!), eq(runnerApiResponseReservations.companyId, this.binding.companyId)));
|
||||
return asset;
|
||||
}).catch(async error => {
|
||||
// A failed commit acknowledgement can still leave a committed
|
||||
// asset. A locking read waits out that transaction before proving
|
||||
// the reservation is unlinked and safe to compensate.
|
||||
const reservation = await this.db.select({ assetId: runnerApiResponseReservations.assetId })
|
||||
.from(runnerApiResponseReservations).where(and(eq(runnerApiResponseReservations.id, reservationId!), eq(runnerApiResponseReservations.companyId, this.binding.companyId))).for("update")
|
||||
.then(rows => rows[0]).catch(() => undefined);
|
||||
if (reservation?.assetId === null) {
|
||||
const removed = await storage.deleteObject(this.binding.companyId, saved.objectKey).then(() => true, () => false);
|
||||
if (removed) storageAttempted = false;
|
||||
}
|
||||
throw error;
|
||||
});
|
||||
const activity = await persistActivity(this.db, { companyId: this.binding.companyId, actorType: "agent", actorId: this.binding.agentId, agentId: this.binding.agentId, runId: this.binding.runId, issueId: this.binding.issueId, action: "asset.created", entityType: "asset", entityId: asset.id, details: { source: "runner.call_api", byteSize: saved.byteSize } });
|
||||
publishActivity(activity.publication);
|
||||
return { artifactId: asset.id, url: `/api/assets/${asset.id}/content`, contentType, byteSize: saved.byteSize, sha256: saved.sha256 };
|
||||
@@ -589,6 +619,68 @@ export class PaperclipRunnerToolAuthority {
|
||||
return result;
|
||||
}
|
||||
|
||||
async #lockApiCaptureCompany(tx: Db): Promise<void> {
|
||||
await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`runner-api-capture:${this.binding.companyId}`}, 0))`);
|
||||
}
|
||||
|
||||
async #reserveApiResponseCapture() {
|
||||
const release = acquireRunnerApiResponseSlot(this.binding.companyId);
|
||||
let id: string;
|
||||
try {
|
||||
id = await this.db.transaction(async tx => {
|
||||
await this.#lockApiCaptureCompany(tx as unknown as Db);
|
||||
const locked = await this.#lockAuthorizedMutationContext(tx as unknown as Db);
|
||||
const resultJson = record(locked.run.resultJson);
|
||||
const used = resultJson.apiResponseCaptureBytes ?? 0;
|
||||
if (typeof used !== "number" || !Number.isSafeInteger(used) || used < 0
|
||||
|| used + RUNNER_API_RESPONSE_MAX_BYTES > RUNNER_API_RESPONSE_RUN_MAX_BYTES) {
|
||||
throw new RunnerApiResponseLimitError("api_response_capture_limit", "This run has insufficient capture budget (4 GiB per run). Read an existing asset or narrow the query.");
|
||||
}
|
||||
// One company lock covers all runs/processes. Stored assets include
|
||||
// snapshots from before reservations existed; attached reservations
|
||||
// are excluded so completed snapshots are counted exactly once.
|
||||
const [usage] = await tx.select({ bytes: sql<string>`
|
||||
coalesce((select sum(${assets.byteSize}) from ${assets}
|
||||
where ${assets.companyId} = ${this.binding.companyId}
|
||||
and ${assets.objectKey} like ${`${this.binding.companyId}/runner-api/%`}), 0)
|
||||
+ coalesce((select sum(${runnerApiResponseReservations.reservedBytes}) from ${runnerApiResponseReservations}
|
||||
where ${runnerApiResponseReservations.companyId} = ${this.binding.companyId}
|
||||
and ${runnerApiResponseReservations.assetId} is null), 0)` }).from(heartbeatRuns).where(eq(heartbeatRuns.id, this.binding.runId));
|
||||
const companyBytes = Number(usage.bytes);
|
||||
if (!Number.isSafeInteger(companyBytes) || companyBytes < 0 || companyBytes + RUNNER_API_RESPONSE_MAX_BYTES > runnerApiCompanyCaptureMaxBytes()) {
|
||||
throw new RunnerApiResponseLimitError("api_response_company_storage_limit", "Company API snapshot storage quota is exhausted. Read an existing asset or ask the operator to remove old snapshots or increase the quota.");
|
||||
}
|
||||
// Reserve worst-case bytes durably before creating a temporary file.
|
||||
// Interrupted/failed captures keep the reservation, preventing retry
|
||||
// loops and process restarts from resetting this run's disk-I/O budget.
|
||||
await tx.update(heartbeatRuns).set({ resultJson: { ...resultJson, apiResponseCaptureBytes: used + RUNNER_API_RESPONSE_MAX_BYTES } }).where(eq(heartbeatRuns.id, this.binding.runId));
|
||||
const [reservation] = await tx.insert(runnerApiResponseReservations).values({ companyId: this.binding.companyId, runId: this.binding.runId, reservedBytes: RUNNER_API_RESPONSE_MAX_BYTES }).returning({ id: runnerApiResponseReservations.id });
|
||||
return reservation.id;
|
||||
});
|
||||
} catch (error) { release(); throw error; }
|
||||
let settled = false;
|
||||
return { id, settle: async (completedBytes?: number, cleaned?: boolean, storageAttempted?: boolean) => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
try {
|
||||
await this.db.transaction(async tx => {
|
||||
await this.#lockApiCaptureCompany(tx as unknown as Db);
|
||||
// Reclaim a handled pre-storage failure only after disk cleanup.
|
||||
// Crashes, failed cleanup, and ambiguous storage failures keep their
|
||||
// reservation until an operator reconciles the possible orphan data.
|
||||
if (cleaned && !storageAttempted) await tx.delete(runnerApiResponseReservations).where(and(eq(runnerApiResponseReservations.id, id), isNull(runnerApiResponseReservations.assetId)));
|
||||
if (completedBytes === undefined) return;
|
||||
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, this.binding.runId), eq(heartbeatRuns.companyId, this.binding.companyId))).for("update");
|
||||
if (!run) return;
|
||||
const resultJson = record(run.resultJson);
|
||||
const used = resultJson.apiResponseCaptureBytes;
|
||||
if (typeof used !== "number" || !Number.isSafeInteger(used) || used < RUNNER_API_RESPONSE_MAX_BYTES) return;
|
||||
await tx.update(heartbeatRuns).set({ resultJson: { ...resultJson, apiResponseCaptureBytes: used - RUNNER_API_RESPONSE_MAX_BYTES + completedBytes } }).where(eq(heartbeatRuns.id, run.id));
|
||||
});
|
||||
} finally { release(); }
|
||||
} };
|
||||
}
|
||||
|
||||
async #readApiFile(file: RunnerApiFile): Promise<{ bytes: Buffer; filename: string; contentType: string }> {
|
||||
if (file.artifactId) {
|
||||
const asset = await assetService(this.db).getById(file.artifactId);
|
||||
|
||||
@@ -1,8 +1,11 @@
|
||||
import { z } from "zod";
|
||||
import { badRequest, forbidden, unprocessable } from "../../errors.js";
|
||||
import { runnerApiOperation, type RunnerApiOperation } from "./runner-api-catalog.js";
|
||||
import { captureRunnerApiResponse, type RunnerApiResponseBody } from "./runner-api-response.js";
|
||||
import { runnerApiRestriction } from "./runner-api-policy.js";
|
||||
import { RunnerApiResponseLimitError, RunnerApiResponseCleanupError, RUNNER_API_RESPONSE_MAX_BYTES } from "./runner-api-response-limits.js";
|
||||
|
||||
/** Request/upload bound only. Response capture has a separate streamed limit. */
|
||||
export const RUNNER_API_MAX_BYTES = 10 * 1024 * 1024;
|
||||
export const RUNNER_API_INLINE_BYTES = 24 * 1024;
|
||||
export const RUNNER_API_TIMEOUT_MS = 30_000;
|
||||
@@ -18,6 +21,10 @@ export const runnerApiCallSchema = z.object({
|
||||
body: z.unknown().optional(),
|
||||
contentType: z.string().max(120).optional(),
|
||||
files: z.array(fileSchema).max(10).optional(),
|
||||
responseText: z.object({
|
||||
offsetBytes: z.number().int().min(0).max(Number.MAX_SAFE_INTEGER).optional(),
|
||||
limitBytes: z.number().int().min(4).max(RUNNER_API_INLINE_BYTES).optional(),
|
||||
}).strict().optional(),
|
||||
}).strict();
|
||||
export type RunnerApiCall = z.infer<typeof runnerApiCallSchema>;
|
||||
export type RunnerApiFile = z.infer<typeof fileSchema>;
|
||||
@@ -34,8 +41,10 @@ export interface RunnerApiIo {
|
||||
fetch?: typeof fetch;
|
||||
/** Revalidate server-owned authority after asynchronous file preparation. */
|
||||
beforeDispatch?(): Promise<void>;
|
||||
/** Reserve before spilling. Settle completed bytes; failed captures retain their reservation. */
|
||||
reserveResponseCapture?(): Promise<(completedBytes?: number, cleanupSucceeded?: boolean) => Promise<void>>;
|
||||
readFile(file: RunnerApiFile): Promise<{ bytes: Buffer; filename: string; contentType: string }>;
|
||||
saveResponse(bytes: Buffer, contentType: string): Promise<Record<string, unknown>>;
|
||||
saveResponse(bytes: RunnerApiResponseBody, contentType: string): Promise<Record<string, unknown>>;
|
||||
}
|
||||
|
||||
export function validateRunnerApiCall(value: unknown, context: RunnerApiContext): { input: RunnerApiCall; operation: RunnerApiOperation } {
|
||||
@@ -44,6 +53,7 @@ export function validateRunnerApiCall(value: unknown, context: RunnerApiContext)
|
||||
const input = parsed.data;
|
||||
const operation = runnerApiOperation(input.operationId);
|
||||
if (operation.transport !== "rest") throw unprocessable("This endpoint requires its existing protocol client; call_api supports REST only");
|
||||
if (input.responseText && operation.method !== "GET") throw badRequest("responseText requires a GET operation; read the saved artifact instead of repeating a mutation");
|
||||
const restriction = runnerApiRestriction(operation.method, operation.path);
|
||||
if (restriction) throw forbidden(restriction);
|
||||
if (!operation.allowedModes.includes(context.workMode)) throw forbidden("call_api permits only reads in Ask and Plan modes; use the permitted dedicated tools");
|
||||
@@ -139,10 +149,19 @@ export async function readBoundedResponse(response: Response, maxBytes = RUNNER_
|
||||
}
|
||||
|
||||
export async function executeRunnerApi(input: RunnerApiCall, context: RunnerApiContext, io: RunnerApiIo) {
|
||||
const { operation } = validateRunnerApiCall(input, context);
|
||||
const { operation, input: validated } = validateRunnerApiCall(input, context);
|
||||
input = validated;
|
||||
const url = runnerApiUrl(operation, input, context, io.apiUrl);
|
||||
if (!io.token) throw new Error("Paperclip run authentication is unavailable");
|
||||
const headers = new Headers({ Authorization: `Bearer ${io.token}`, "X-Paperclip-Run-Id": context.runId });
|
||||
const textOffset = input.responseText?.offsetBytes ?? 0;
|
||||
const textLimit = input.responseText?.limitBytes ?? RUNNER_API_INLINE_BYTES;
|
||||
const assetTextRange = operation.operationId === "GET /api/assets/{assetId}/content"
|
||||
? { start: Math.max(0, textOffset - 1), end: Math.min(Number.MAX_SAFE_INTEGER, textOffset + textLimit) }
|
||||
: undefined;
|
||||
// One byte before the requested offset permits an EOF read; one byte after
|
||||
// the page lets the decoder trim an incomplete UTF-8 code point.
|
||||
if (assetTextRange) headers.set("Range", `bytes=${assetTextRange.start}-${assetTextRange.end}`);
|
||||
let body: BodyInit | undefined;
|
||||
const contentType = input.contentType ?? (input.files?.length ? "multipart/form-data" : "application/json");
|
||||
if (/\r|\n/.test(contentType)) throw badRequest("Invalid content type");
|
||||
@@ -177,29 +196,118 @@ export async function executeRunnerApi(input: RunnerApiCall, context: RunnerApiC
|
||||
}
|
||||
let response: Response;
|
||||
let bytes: Buffer;
|
||||
let captured: Awaited<ReturnType<typeof captureRunnerApiResponse>> | undefined;
|
||||
let settleCapture: ((completedBytes?: number, cleanupSucceeded?: boolean) => Promise<void>) | undefined;
|
||||
const disposeCapture = async (priorCleanupSucceeded = true) => {
|
||||
let cleaned = false;
|
||||
try { await captured?.dispose(); cleaned = priorCleanupSucceeded; }
|
||||
finally { await settleCapture?.(captured?.byteSize, cleaned); }
|
||||
};
|
||||
const saveResponse = async (body: RunnerApiResponseBody, type: string) => {
|
||||
// Small binary responses also create assets and need storage admission.
|
||||
settleCapture ??= await io.reserveResponseCapture?.();
|
||||
return io.saveResponse(body, type);
|
||||
};
|
||||
await io.beforeDispatch?.();
|
||||
const controller = new AbortController();
|
||||
// Bound connection setup and stalled body reads, not the total download time.
|
||||
let timeout = setTimeout(() => controller.abort(), RUNNER_API_TIMEOUT_MS);
|
||||
try {
|
||||
response = await (io.fetch ?? fetch)(url, { method: operation.method, headers, body, redirect: "manual", signal: AbortSignal.timeout(RUNNER_API_TIMEOUT_MS) });
|
||||
bytes = await readBoundedResponse(response);
|
||||
} catch {
|
||||
return { ok: false, status: null, operationId: operation.operationId, error: "api_transport_failure", outcome: ["GET", "HEAD", "OPTIONS"].includes(operation.method) ? "read_failed" : "unknown", guidance: "Inspect current state before retrying a mutation; it may already have succeeded." };
|
||||
}
|
||||
const type = response.headers.get("content-type") ?? "application/octet-stream";
|
||||
const mutation = !["GET", "HEAD", "OPTIONS"].includes(operation.method);
|
||||
// A server can commit a write before an error response or redirect. Preserve
|
||||
// the HTTP evidence, but never describe that write as definitely not applied.
|
||||
const uncertain = mutation && (response.status >= 500 || response.status === 408 || (response.status >= 300 && response.status < 400));
|
||||
const uncertainty = { outcome: "unknown", guidance: "Inspect current state before retrying this mutation; it may already have succeeded." };
|
||||
const base = { ok: response.ok, status: response.status, operationId: operation.operationId, contentType: type, retryAfter: response.headers.get("retry-after"), ...(uncertain ? uncertainty : {}) };
|
||||
if (response.status >= 300 && response.status < 400) return { ...base, ok: false, error: "api_redirect_not_followed" };
|
||||
if (!bytes.length) return { ...base, data: null };
|
||||
if (bytes.length > RUNNER_API_INLINE_BYTES || !/json|^text\//i.test(type)) {
|
||||
return { ...base, artifact: await io.saveResponse(bytes, type), byteSize: bytes.length, preview: /json|^text\//i.test(type) ? bytes.subarray(0, 2000).toString("utf8") : null };
|
||||
}
|
||||
const text = bytes.toString("utf8");
|
||||
if (/json/i.test(type)) {
|
||||
try { return { ...base, data: JSON.parse(text) as unknown }; }
|
||||
catch { return { ...base, ok: false, error: "invalid_json_response", data: text, ...(mutation ? uncertainty : {}) }; }
|
||||
}
|
||||
return { ...base, data: text };
|
||||
response = await (io.fetch ?? fetch)(url, { method: operation.method, headers, body, redirect: "manual", signal: controller.signal });
|
||||
clearTimeout(timeout);
|
||||
if (assetTextRange && response.ok) {
|
||||
timeout = setTimeout(() => controller.abort(), RUNNER_API_TIMEOUT_MS);
|
||||
bytes = await readBoundedResponse(response, textLimit + 2);
|
||||
} else {
|
||||
captured = await captureRunnerApiResponse(response, RUNNER_API_INLINE_BYTES, controller, RUNNER_API_TIMEOUT_MS, io.beforeDispatch, {
|
||||
beforeSpill: async () => { settleCapture = await io.reserveResponseCapture?.(); },
|
||||
});
|
||||
// Read only the requested window (+ one UTF-8 lookahead byte) into memory.
|
||||
bytes = await captured.read(input.responseText && response.ok ? textOffset : 0,
|
||||
input.responseText ? textLimit + 1 : RUNNER_API_INLINE_BYTES);
|
||||
}
|
||||
} catch (error) {
|
||||
await disposeCapture(!(error instanceof RunnerApiResponseCleanupError));
|
||||
return { ok: false, status: null, operationId: operation.operationId,
|
||||
error: error instanceof RunnerApiResponseLimitError ? error.code : "api_transport_failure",
|
||||
...(error instanceof RunnerApiResponseLimitError ? { message: error.message, maxResponseBytes: RUNNER_API_RESPONSE_MAX_BYTES } : {}),
|
||||
outcome: ["GET", "HEAD", "OPTIONS"].includes(operation.method) ? "read_failed" : "unknown", guidance: "Inspect current state before retrying a mutation; it may already have succeeded." };
|
||||
} finally { clearTimeout(timeout); }
|
||||
try {
|
||||
const type = response.headers.get("content-type") ?? "application/octet-stream";
|
||||
const mutation = !["GET", "HEAD", "OPTIONS"].includes(operation.method);
|
||||
// A server can commit a write before an error response or redirect. Preserve
|
||||
// the HTTP evidence, but never describe that write as definitely not applied.
|
||||
const uncertain = mutation && (response.status >= 500 || response.status === 408 || (response.status >= 300 && response.status < 400));
|
||||
const uncertainty = { outcome: "unknown", guidance: "Inspect current state before retrying this mutation; it may already have succeeded." };
|
||||
const base = { ok: response.ok, status: response.status, operationId: operation.operationId, contentType: type, retryAfter: response.headers.get("retry-after"), ...(uncertain ? uncertainty : {}) };
|
||||
if (response.status >= 300 && response.status < 400) return { ...base, ok: false, error: "api_redirect_not_followed" };
|
||||
let totalBytes = captured?.byteSize ?? bytes.length;
|
||||
let baseOffset = captured && response.ok ? textOffset : 0;
|
||||
if (assetTextRange && response.ok) {
|
||||
const range = /^bytes (\d+)-(\d+)\/(\d+)$/.exec(response.headers.get("content-range") ?? "");
|
||||
if (response.status === 200 && bytes.length === 0 && textOffset === 0) {
|
||||
// An empty asset has no satisfiable byte range; its route returns 200.
|
||||
} else if (response.status !== 206 || !range) {
|
||||
return { ...base, ok: false, error: "response_text_invalid_range" };
|
||||
} else {
|
||||
const [start, end, total] = range.slice(1).map(Number);
|
||||
if (![start, end, total].every(Number.isSafeInteger) || start !== assetTextRange.start
|
||||
|| total < 1 || end !== Math.min(assetTextRange.end, total - 1)
|
||||
|| end < start || bytes.length !== end - start + 1) {
|
||||
return { ...base, ok: false, error: "response_text_invalid_range" };
|
||||
}
|
||||
baseOffset = start;
|
||||
totalBytes = total;
|
||||
}
|
||||
}
|
||||
if (input.responseText) {
|
||||
if (!/json|^text\//i.test(type)) return { ...base, ok: false, error: "response_text_requires_text_content" };
|
||||
// HTTP denials are evidence, not text pages of an authorized asset.
|
||||
const offsetBytes = response.ok ? textOffset : 0;
|
||||
const localOffset = offsetBytes - baseOffset;
|
||||
if (offsetBytes > totalBytes || localOffset < 0 || (localOffset < bytes.length && (bytes[localOffset] & 0xc0) === 0x80)) {
|
||||
return { ...base, ok: false, error: "response_text_invalid_offset", byteSize: totalBytes };
|
||||
}
|
||||
let end = Math.min(bytes.length, localOffset + textLimit);
|
||||
// Do not cut a UTF-8 code point. The next offset always starts a whole one.
|
||||
while (end < bytes.length && end > localOffset && (bytes[end] & 0xc0) === 0x80) end--;
|
||||
if (end === localOffset && end < bytes.length) return { ...base, ok: false, error: "response_text_invalid_utf8" };
|
||||
let data: string;
|
||||
try {
|
||||
data = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }).decode(bytes.subarray(localOffset, end));
|
||||
} catch {
|
||||
return { ...base, ok: false, error: "response_text_invalid_utf8" };
|
||||
}
|
||||
const nextOffset = baseOffset + end;
|
||||
const artifact = captured && captured.byteSize > RUNNER_API_INLINE_BYTES
|
||||
? await saveResponse(captured.body, type) : undefined;
|
||||
return { ...base, data, ...(artifact ? { artifact, guidance: "Continue reading this saved artifact with GET /api/assets/{assetId}/content and responseText.nextOffsetBytes for a stable snapshot." } : {}), responseText: { offsetBytes, nextOffsetBytes: nextOffset < totalBytes ? nextOffset : null, totalBytes } };
|
||||
}
|
||||
if (assetTextRange && response.ok && (totalBytes > RUNNER_API_INLINE_BYTES || !/json|^text\//i.test(type))) {
|
||||
// This content is already an authorized, stable asset. Return its existing
|
||||
// reference after a bounded preview; never download and duplicate it.
|
||||
const assetId = input.pathParams!.assetId;
|
||||
const sha256 = /^"([a-f0-9]{64})"$/.exec(response.headers.get("etag") ?? "")?.[1];
|
||||
return { ...base, artifact: { artifactId: assetId, url: `/api/assets/${assetId}/content`, contentType: type, byteSize: totalBytes, ...(sha256 ? { sha256 } : {}) },
|
||||
byteSize: totalBytes, preview: /json|^text\//i.test(type) ? bytes.subarray(0, 2000).toString("utf8") : null,
|
||||
guidance: "Read this asset with responseText: { offsetBytes: 0 }. Continue with responseText.nextOffsetBytes until null." };
|
||||
}
|
||||
if (!totalBytes) return { ...base, data: null };
|
||||
if (totalBytes > RUNNER_API_INLINE_BYTES || !/json|^text\//i.test(type)) {
|
||||
return { ...base, artifact: await saveResponse(captured?.body ?? bytes, type), byteSize: totalBytes, preview: /json|^text\//i.test(type) ? bytes.subarray(0, 2000).toString("utf8") : null,
|
||||
...(/json|^text\//i.test(type) ? { guidance: "Read the saved artifact with GET /api/assets/{assetId}/content and responseText: { offsetBytes: 0 }. Continue with responseText.nextOffsetBytes until null. Do not repeat a mutation to read its response." } : {}) };
|
||||
}
|
||||
const text = bytes.toString("utf8");
|
||||
if (/json/i.test(type)) {
|
||||
try { return { ...base, data: JSON.parse(text) as unknown }; }
|
||||
catch { return { ...base, ok: false, error: "invalid_json_response", data: text, ...(mutation ? uncertainty : {}) }; }
|
||||
}
|
||||
return { ...base, data: text };
|
||||
} catch (error) {
|
||||
if (!(error instanceof RunnerApiResponseLimitError)) throw error;
|
||||
return { ok: false, status: response.status, operationId: operation.operationId, error: error.code, message: error.message,
|
||||
outcome: ["GET", "HEAD", "OPTIONS"].includes(operation.method) ? "read_failed" : "unknown",
|
||||
guidance: "Inspect current state before retrying a mutation; it may already have succeeded." };
|
||||
} finally { await disposeCapture(); }
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
export const RUNNER_API_RESPONSE_MAX_BYTES = 1024 * 1024 * 1024;
|
||||
export const RUNNER_API_RESPONSE_RUN_MAX_BYTES = 4 * RUNNER_API_RESPONSE_MAX_BYTES;
|
||||
export const RUNNER_API_RESPONSE_DEADLINE_MS = 10 * 60_000;
|
||||
export function runnerApiCompanyCaptureMaxBytes(): number {
|
||||
const configured = Number(process.env.PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES);
|
||||
return Number.isSafeInteger(configured) && configured >= RUNNER_API_RESPONSE_MAX_BYTES
|
||||
? configured : 20 * RUNNER_API_RESPONSE_MAX_BYTES;
|
||||
}
|
||||
|
||||
/** Cleanup failure must keep the durable reservation for possible orphan bytes. */
|
||||
export class RunnerApiResponseCleanupError extends Error {
|
||||
constructor(cause: unknown) { super("Failed to remove captured API response", { cause }); }
|
||||
}
|
||||
|
||||
export class RunnerApiResponseLimitError extends Error {
|
||||
constructor(readonly code: string, message: string) { super(message); }
|
||||
}
|
||||
|
||||
// Hold a slot until the temporary file is removed, including storage upload.
|
||||
// These limits are per server process; the run byte budget is database-backed.
|
||||
const activeByCompany = new Map<string, number>();
|
||||
let active = 0;
|
||||
export function acquireRunnerApiResponseSlot(companyId: string): () => void {
|
||||
const companyActive = activeByCompany.get(companyId) ?? 0;
|
||||
if (active >= 4 || companyActive >= 2) {
|
||||
throw new RunnerApiResponseLimitError("api_response_capture_busy", "Large API response capture is busy. Wait for another capture to finish before retrying a read.");
|
||||
}
|
||||
active++;
|
||||
activeByCompany.set(companyId, companyActive + 1);
|
||||
let released = false;
|
||||
return () => {
|
||||
if (released) return;
|
||||
released = true;
|
||||
active--;
|
||||
const remaining = activeByCompany.get(companyId)! - 1;
|
||||
if (remaining) activeByCompany.set(companyId, remaining);
|
||||
else activeByCompany.delete(companyId);
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,142 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { access, readdir } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { captureRunnerApiResponse } from "./runner-api-response.js";
|
||||
import { acquireRunnerApiResponseSlot, runnerApiCompanyCaptureMaxBytes, RUNNER_API_RESPONSE_MAX_BYTES } from "./runner-api-response-limits.js";
|
||||
|
||||
const controller = () => new AbortController();
|
||||
describe("streamed API response capture", () => {
|
||||
it("rejects an oversized Content-Length before reading or reserving disk", async () => {
|
||||
let cancelled = false;
|
||||
let reserved = false;
|
||||
const abort = controller();
|
||||
const response = new Response(new ReadableStream({ cancel() { cancelled = true; } }), { headers: { "content-length": String(RUNNER_API_RESPONSE_MAX_BYTES + 1) } });
|
||||
await expect(captureRunnerApiResponse(response, 24 * 1024, abort, 1000, undefined, {
|
||||
beforeSpill: async () => { reserved = true; },
|
||||
})).rejects.toMatchObject({ code: "api_response_too_large" });
|
||||
expect(cancelled).toBe(true);
|
||||
expect(abort.signal.aborted).toBe(true);
|
||||
expect(reserved).toBe(false);
|
||||
});
|
||||
it.each([undefined, "1"])("counts streamed bytes with Content-Length %s and removes partial files", async contentLength => {
|
||||
const before = new Set(await readdir(tmpdir()));
|
||||
let cancelled = false;
|
||||
let reserved = 0;
|
||||
const response = new Response(new ReadableStream({
|
||||
pull(target) { target.enqueue(Buffer.alloc(4096)); },
|
||||
cancel() { cancelled = true; },
|
||||
}), { headers: contentLength ? { "content-length": contentLength } : {} });
|
||||
await expect(captureRunnerApiResponse(response, 4096, controller(), 1000, undefined, {
|
||||
maxBytes: 16 * 1024,
|
||||
beforeSpill: async () => { reserved++; },
|
||||
})).rejects.toMatchObject({ code: "api_response_too_large" });
|
||||
expect(cancelled).toBe(true);
|
||||
expect(reserved).toBe(1);
|
||||
expect((await readdir(tmpdir())).filter(name => name.startsWith("paperclip-api-response-") && !before.has(name))).toEqual([]);
|
||||
});
|
||||
it("accepts exactly the capture limit", async () => {
|
||||
const captured = await captureRunnerApiResponse(new Response(Buffer.alloc(8192)), 4096, controller(), 1000, undefined, { maxBytes: 8192 });
|
||||
try { expect(captured.byteSize).toBe(8192); }
|
||||
finally { await captured.dispose(); }
|
||||
});
|
||||
it("stops an active endless stream at the total deadline", async () => {
|
||||
let cancelled = false;
|
||||
const response = new Response(new ReadableStream({
|
||||
async pull(target) {
|
||||
await new Promise(resolve => setTimeout(resolve, 5));
|
||||
if (!cancelled) target.enqueue(Buffer.alloc(4096));
|
||||
},
|
||||
cancel() { cancelled = true; },
|
||||
}));
|
||||
await expect(captureRunnerApiResponse(response, 4096, controller(), 1000, undefined, { deadlineMs: 30 })).rejects.toMatchObject({ code: "api_response_timeout" });
|
||||
expect(cancelled).toBe(true);
|
||||
});
|
||||
it("spills a chunked response and reads only requested bytes, preserving its digest", async () => {
|
||||
const chunk = Buffer.alloc(64 * 1024, 65);
|
||||
const digest = createHash("sha256");
|
||||
let emitted = 0;
|
||||
const response = new Response(new ReadableStream({
|
||||
pull(target) {
|
||||
if (emitted++ === 200) return target.close();
|
||||
digest.update(chunk);
|
||||
target.enqueue(chunk);
|
||||
},
|
||||
}));
|
||||
const captured = await captureRunnerApiResponse(response, 24 * 1024, controller(), 1000);
|
||||
expect(Buffer.isBuffer(captured.body)).toBe(false);
|
||||
if (Buffer.isBuffer(captured.body)) throw new Error("Expected a file");
|
||||
const path = captured.body.path;
|
||||
try {
|
||||
expect(captured.body).toMatchObject({ byteSize: 200 * chunk.length, sha256: digest.digest("hex") });
|
||||
expect(await captured.read(11 * 1024 * 1024, 4)).toEqual(Buffer.from("AAAA"));
|
||||
expect(await captured.read(captured.byteSize, 4)).toHaveLength(0);
|
||||
} finally { await captured.dispose(); }
|
||||
await expect(access(path)).rejects.toThrow();
|
||||
});
|
||||
it("keeps small bodies inline", async () => {
|
||||
const captured = await captureRunnerApiResponse(new Response("small"), 24 * 1024, controller(), 1000);
|
||||
try { expect(captured.body).toEqual(Buffer.from("small")); }
|
||||
finally { await captured.dispose(); }
|
||||
});
|
||||
it("stops a long capture when its run loses authority", async () => {
|
||||
let checks = 0;
|
||||
let cancelled = false;
|
||||
const response = new Response(new ReadableStream({
|
||||
pull(target) { target.enqueue(Buffer.alloc(64 * 1024)); },
|
||||
cancel() { cancelled = true; },
|
||||
}));
|
||||
await expect(captureRunnerApiResponse(response, 24 * 1024, controller(), 1000, async () => {
|
||||
if (++checks === 2) throw new Error("run stopped");
|
||||
})).rejects.toThrow("run stopped");
|
||||
expect(cancelled).toBe(true);
|
||||
expect(checks).toBe(2);
|
||||
});
|
||||
it("allows active downloads longer than the idle timeout", async () => {
|
||||
let chunks = 0;
|
||||
const response = new Response(new ReadableStream({
|
||||
async pull(target) {
|
||||
await new Promise(resolve => setTimeout(resolve, 20));
|
||||
if (chunks++ === 5) target.close();
|
||||
else target.enqueue(Buffer.from("a"));
|
||||
},
|
||||
}));
|
||||
const captured = await captureRunnerApiResponse(response, 24 * 1024, controller(), 100);
|
||||
try { expect(captured.body).toEqual(Buffer.from("aaaaa")); }
|
||||
finally { await captured.dispose(); }
|
||||
});
|
||||
it("cancels stalled bodies and aborts the HTTP request", async () => {
|
||||
let cancelled = false;
|
||||
const response = new Response(new ReadableStream({ cancel() { cancelled = true; } }));
|
||||
const abort = controller();
|
||||
await expect(captureRunnerApiResponse(response, 24 * 1024, abort, 10)).rejects.toThrow("timed out");
|
||||
expect(cancelled).toBe(true);
|
||||
expect(abort.signal.aborted).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("large response concurrency", () => {
|
||||
it("only accepts finite positive operator quotas, never unlimited settings", () => {
|
||||
try {
|
||||
for (const value of ["0", "-1", "Infinity", "NaN", "1", "9007199254740992"]) {
|
||||
vi.stubEnv("PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES", value);
|
||||
expect(runnerApiCompanyCaptureMaxBytes()).toBe(20 * RUNNER_API_RESPONSE_MAX_BYTES);
|
||||
}
|
||||
vi.stubEnv("PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES", String(2 * RUNNER_API_RESPONSE_MAX_BYTES));
|
||||
expect(runnerApiCompanyCaptureMaxBytes()).toBe(2 * RUNNER_API_RESPONSE_MAX_BYTES);
|
||||
} finally { vi.unstubAllEnvs(); }
|
||||
});
|
||||
it("limits each company and the server and releases slots idempotently", () => {
|
||||
const releases: Array<() => void> = [];
|
||||
try {
|
||||
releases.push(acquireRunnerApiResponseSlot("a"), acquireRunnerApiResponseSlot("a"));
|
||||
expect(() => acquireRunnerApiResponseSlot("a")).toThrow("busy");
|
||||
releases.push(acquireRunnerApiResponseSlot("b"), acquireRunnerApiResponseSlot("b"));
|
||||
expect(() => acquireRunnerApiResponseSlot("c")).toThrow("busy");
|
||||
releases[0](); releases[0]();
|
||||
releases.push(acquireRunnerApiResponseSlot("c"));
|
||||
expect(() => acquireRunnerApiResponseSlot("d")).toThrow("busy");
|
||||
} finally { for (const release of releases) release(); }
|
||||
acquireRunnerApiResponseSlot("a")();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,113 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { mkdtemp, open, rm, type FileHandle } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { RUNNER_API_RESPONSE_MAX_BYTES, RUNNER_API_RESPONSE_DEADLINE_MS, RunnerApiResponseLimitError, RunnerApiResponseCleanupError } from "./runner-api-response-limits.js";
|
||||
|
||||
export type RunnerApiResponseBody = Buffer | { path: string; byteSize: number; sha256: string };
|
||||
|
||||
/** Spill bounded responses to a private temporary file.
|
||||
* Only the inline prefix and one HTTP chunk are held in memory. The caller must
|
||||
* dispose the capture after saving it or reading its requested text window.
|
||||
*/
|
||||
export async function captureRunnerApiResponse(response: Response, inlineBytes: number, controller: AbortController, idleTimeoutMs: number, checkAuthority?: () => Promise<void>, options: {
|
||||
maxBytes?: number;
|
||||
deadlineMs?: number;
|
||||
beforeSpill?: () => Promise<void>;
|
||||
} = {}) {
|
||||
const maxBytes = options.maxBytes ?? RUNNER_API_RESPONSE_MAX_BYTES;
|
||||
const deadline = Date.now() + (options.deadlineMs ?? RUNNER_API_RESPONSE_DEADLINE_MS);
|
||||
const reader = response.body?.getReader();
|
||||
let directory: string | undefined;
|
||||
let file: FileHandle | undefined;
|
||||
let path: string | undefined;
|
||||
let byteSize = 0;
|
||||
const prefix: Buffer[] = [];
|
||||
const hash = createHash("sha256");
|
||||
const dispose = async () => {
|
||||
try {
|
||||
try { await file?.close(); }
|
||||
finally {
|
||||
file = undefined;
|
||||
if (directory) await rm(directory, { recursive: true, force: true });
|
||||
}
|
||||
} catch (error) { throw new RunnerApiResponseCleanupError(error); }
|
||||
};
|
||||
try {
|
||||
if (Number(response.headers.get("content-length")) > maxBytes) {
|
||||
throw new RunnerApiResponseLimitError("api_response_too_large", `API response exceeds the ${maxBytes}-byte capture limit. Narrow the query or use the endpoint's pagination.`);
|
||||
}
|
||||
let checkedAt = Date.now();
|
||||
let checkedBytes = 0;
|
||||
await checkAuthority?.();
|
||||
if (reader) {
|
||||
while (true) {
|
||||
controller.signal.throwIfAborted();
|
||||
if (Date.now() >= deadline) throw new RunnerApiResponseLimitError("api_response_timeout", "API response exceeded its capture deadline.");
|
||||
let timeout: ReturnType<typeof setTimeout> | undefined;
|
||||
const next = await Promise.race([
|
||||
reader.read(),
|
||||
new Promise<never>((_, reject) => {
|
||||
timeout = setTimeout(() => {
|
||||
controller.abort();
|
||||
reject(new RunnerApiResponseLimitError("api_response_timeout", "API response read timed out or exceeded its capture deadline."));
|
||||
}, Math.min(idleTimeoutMs, Math.max(0, deadline - Date.now())));
|
||||
}),
|
||||
]).finally(() => clearTimeout(timeout));
|
||||
if (next.done) break;
|
||||
// Count decoded bytes before copying, hashing, or writing the chunk.
|
||||
// Content-Length can be missing, false, or describe compressed bytes.
|
||||
if (next.value.byteLength > maxBytes - byteSize) {
|
||||
throw new RunnerApiResponseLimitError("api_response_too_large", `API response exceeds the ${maxBytes}-byte capture limit. Narrow the query or use the endpoint's pagination.`);
|
||||
}
|
||||
const chunk = Buffer.from(next.value);
|
||||
if (!Number.isSafeInteger(byteSize + chunk.length)) throw new Error("API response byte offset exceeds numeric precision");
|
||||
hash.update(chunk);
|
||||
if (!file && byteSize + chunk.length > inlineBytes) {
|
||||
await options.beforeSpill?.();
|
||||
directory = await mkdtemp(join(tmpdir(), "paperclip-api-response-"));
|
||||
path = join(directory, "body");
|
||||
file = await open(path, "wx+", 0o600);
|
||||
for (const part of prefix) await file.writeFile(part);
|
||||
prefix.length = 0;
|
||||
}
|
||||
if (file) await file.writeFile(chunk);
|
||||
else prefix.push(chunk);
|
||||
byteSize += chunk.length;
|
||||
if (checkAuthority && (Date.now() - checkedAt >= 1000 || byteSize - checkedBytes >= 1024 * 1024)) {
|
||||
await checkAuthority();
|
||||
checkedAt = Date.now();
|
||||
checkedBytes = byteSize;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (Date.now() >= deadline) throw new RunnerApiResponseLimitError("api_response_timeout", "API response exceeded its capture deadline.");
|
||||
await checkAuthority?.();
|
||||
const bytes = file ? undefined : Buffer.concat(prefix);
|
||||
const body: RunnerApiResponseBody = path ? { path, byteSize, sha256: hash.digest("hex") } : bytes!;
|
||||
return {
|
||||
body,
|
||||
byteSize,
|
||||
async read(offset: number, limit: number): Promise<Buffer> {
|
||||
const length = Math.min(limit, Math.max(0, byteSize - offset));
|
||||
if (!file) return bytes!.subarray(offset, offset + length);
|
||||
const result = Buffer.alloc(length);
|
||||
let read = 0;
|
||||
while (read < length) {
|
||||
const next = await file.read(result, read, length - read, offset + read);
|
||||
if (!next.bytesRead) throw new Error("Saved API response was truncated");
|
||||
read += next.bytesRead;
|
||||
}
|
||||
return result;
|
||||
},
|
||||
dispose,
|
||||
};
|
||||
} catch (error) {
|
||||
controller.abort();
|
||||
await dispose();
|
||||
throw error;
|
||||
} finally {
|
||||
await reader?.cancel().catch(() => {});
|
||||
reader?.releaseLock();
|
||||
}
|
||||
}
|
||||
@@ -1,14 +1,16 @@
|
||||
import { existsSync } from "node:fs";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { chmod, writeFile, symlink } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { chmod, writeFile, symlink, mkdir, open } from "node:fs/promises";
|
||||
import { join, dirname } from "node:path";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { documents, heartbeatRuns, issues, routineDocuments, routines } from "@paperclipai/db";
|
||||
import { assets, documents, heartbeatRuns, issues, projects, routineDocuments, routines, runnerApiResponseReservations } from "@paperclipai/db";
|
||||
import { beforeAll, afterAll, beforeEach, afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { startRunnerApiTestServer } from "../../__tests__/helpers/runner-api-server.js";
|
||||
import { createRunnerdCodexTransport, defaultCapabilityRunnerdBinary } from "../../vendor/paperclip-runner/index.js";
|
||||
import { runnerApiCatalog } from "./runner-api-catalog.js";
|
||||
import { registerRunnerPrpAuthority } from "../../realtime/runner-prp-ws.js";
|
||||
import { RUNNER_API_RESPONSE_MAX_BYTES, RUNNER_API_RESPONSE_RUN_MAX_BYTES } from "./runner-api-response-limits.js";
|
||||
import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js";
|
||||
|
||||
describe("runner API against real HTTP routes", () => {
|
||||
let server: Awaited<ReturnType<typeof startRunnerApiTestServer>>;
|
||||
@@ -248,6 +250,243 @@ else if(m.id!==undefined) send({id:m.id,result:{}});
|
||||
} finally { getObject.mockRestore(); }
|
||||
});
|
||||
|
||||
it("retrieves every byte of a saved large response without creating an artifact loop", async () => {
|
||||
const fixture = await server.fixture();
|
||||
const foreign = await server.fixture();
|
||||
const text = JSON.stringify({ padding: "🧭é".repeat(6000), evidence: "last-field-is-readable" });
|
||||
const stored = await server.storage.putFile({ companyId: fixture.companyId, namespace: "eval", originalFilename: "evidence.json", contentType: "application/json", body: Buffer.from(text) });
|
||||
const [asset] = await server.db.insert(assets).values({ ...stored, companyId: fixture.companyId, createdByAgentId: fixture.agentId }).returning();
|
||||
const call = (authority: typeof fixture.authority, assetId: string, responseText?: object) => authority.execute({ tool: "call_api", callId: randomUUID(), arguments: { operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId }, ...(responseText ? { responseText } : {}) } }) as Promise<any>;
|
||||
const initial = await call(fixture.authority, asset.id);
|
||||
expect(initial).toMatchObject({ status: 206, artifact: { artifactId: asset.id, sha256: asset.sha256, byteSize: Buffer.byteLength(text) } });
|
||||
const before = (await fixture.snapshot()).assets.length;
|
||||
const reads = vi.spyOn(server.storage, "getObject");
|
||||
let offsetBytes = 0;
|
||||
let result = "";
|
||||
try {
|
||||
do {
|
||||
const page = await call(fixture.authority, initial.artifact.artifactId, { offsetBytes, limitBytes: 4096 });
|
||||
expect(page).toMatchObject({ status: 206, responseText: { offsetBytes, totalBytes: Buffer.byteLength(text) } });
|
||||
result += page.data;
|
||||
offsetBytes = page.responseText.nextOffsetBytes;
|
||||
} while (offsetBytes !== null);
|
||||
expect(JSON.parse(result)).toEqual(JSON.parse(text));
|
||||
expect(reads.mock.calls.length).toBeGreaterThan(1);
|
||||
expect(reads.mock.calls.every(([, , options]) => options?.range)).toBe(true);
|
||||
const storageBytes = reads.mock.calls.reduce((total, [, , options]) => total + options!.range!.end - options!.range!.start + 1, 0);
|
||||
expect(storageBytes).toBeLessThan(Buffer.byteLength(text) + reads.mock.calls.length * 5);
|
||||
expect((await fixture.snapshot()).assets).toHaveLength(before);
|
||||
expect(await call(foreign.authority, initial.artifact.artifactId, { offsetBytes: 0 })).toMatchObject({ ok: false, status: 404 });
|
||||
} finally {
|
||||
reads.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it("captures a live response above ten MiB and reads its saved snapshot to EOF", async () => {
|
||||
const fixture = await server.fixture();
|
||||
const foreign = await server.fixture();
|
||||
const description = "A".repeat(12 * 1024 * 1024) + "END-OF-LARGE-RESPONSE";
|
||||
await server.db.update(projects).set({ description }).where(eq(projects.id, fixture.projectId));
|
||||
const initial: any = await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
|
||||
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId }, responseText: { limitBytes: 8192 },
|
||||
} });
|
||||
expect(initial).toMatchObject({ ok: true, status: 200, responseText: { offsetBytes: 0, nextOffsetBytes: 8192 } });
|
||||
expect(initial.artifact.byteSize).toBeGreaterThan(12 * 1024 * 1024);
|
||||
const saved = await server.db.select().from(assets).where(eq(assets.id, initial.artifact.artifactId)).then(rows => rows[0]);
|
||||
const object = await server.storage.getObject(fixture.companyId, saved.objectKey);
|
||||
const chunks: Buffer[] = [];
|
||||
for await (const chunk of object.stream) chunks.push(chunk);
|
||||
const expected = Buffer.concat(chunks);
|
||||
expect(JSON.parse(expected.toString()).description).toBe(description);
|
||||
const before = (await fixture.snapshot()).assets.length;
|
||||
for (const offsetBytes of [0, 10 * 1024 * 1024 + 1, expected.length - 8192, expected.length]) {
|
||||
const page: any = await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
|
||||
operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: saved.id }, responseText: { offsetBytes, limitBytes: 8192 },
|
||||
} });
|
||||
expect(page).toMatchObject({ ok: true, status: 206, data: expected.subarray(offsetBytes, offsetBytes + 8192).toString(),
|
||||
responseText: { offsetBytes, totalBytes: expected.length, nextOffsetBytes: offsetBytes + 8192 < expected.length ? offsetBytes + 8192 : null } });
|
||||
expect(page.artifact).toBeUndefined();
|
||||
}
|
||||
expect((await fixture.snapshot()).assets).toHaveLength(before);
|
||||
expect(await foreign.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
|
||||
operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: saved.id }, responseText: { offsetBytes: 10 * 1024 * 1024 + 1 },
|
||||
} })).toMatchObject({ ok: false, status: 404 });
|
||||
const mutation = { tool: "call_api", callId: randomUUID(), arguments: {
|
||||
operationId: "PATCH /api/projects/{id}", pathParams: { id: fixture.projectId }, body: { name: "Updated once" },
|
||||
} };
|
||||
const receipt: any = await fixture.authority.execute(mutation);
|
||||
expect(receipt).toMatchObject({ ok: true, status: 200, artifact: { contentType: "application/json; charset=utf-8" } });
|
||||
expect(receipt.artifact.byteSize).toBeGreaterThan(12 * 1024 * 1024);
|
||||
expect(await fixture.authority.execute(mutation)).toEqual(receipt);
|
||||
const after = await fixture.snapshot();
|
||||
expect(after.assets).toHaveLength(before + 1);
|
||||
expect(after.activity.filter(row => row.action === "project.updated")).toHaveLength(1);
|
||||
const [run] = await server.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
|
||||
expect((run.resultJson as Record<string, unknown>).apiResponseCaptureBytes).toBe(initial.artifact.byteSize + receipt.artifact.byteSize);
|
||||
}, 30_000);
|
||||
|
||||
it("enforces the durable run budget while permitting small reads and saved pages", async () => {
|
||||
const fixture = await server.fixture();
|
||||
await server.db.update(heartbeatRuns).set({ resultJson: { apiResponseCaptureBytes: RUNNER_API_RESPONSE_RUN_MAX_BYTES } }).where(eq(heartbeatRuns.id, fixture.runId));
|
||||
const call = () => fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
|
||||
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
|
||||
} });
|
||||
expect(await call()).toMatchObject({ ok: true });
|
||||
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
|
||||
const before = (await fixture.snapshot()).assets.length;
|
||||
expect(await call()).toMatchObject({ ok: false, error: "api_response_capture_limit", outcome: "read_failed" });
|
||||
expect((await fixture.snapshot()).assets).toHaveLength(before);
|
||||
expect(await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
|
||||
operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: fixture.artifactId }, responseText: { limitBytes: 4 },
|
||||
} })).toMatchObject({ ok: true });
|
||||
});
|
||||
|
||||
it("reserves the run budget atomically across simultaneous captures", async () => {
|
||||
const fixture = await server.fixture();
|
||||
const used = RUNNER_API_RESPONSE_RUN_MAX_BYTES - RUNNER_API_RESPONSE_MAX_BYTES;
|
||||
await server.db.update(heartbeatRuns).set({ resultJson: { apiResponseCaptureBytes: used } }).where(eq(heartbeatRuns.id, fixture.runId));
|
||||
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
|
||||
let unblock!: () => void;
|
||||
let started!: () => void;
|
||||
const gate = new Promise<void>(resolve => { unblock = resolve; });
|
||||
const saving = new Promise<void>(resolve => { started = resolve; });
|
||||
const putFile = server.storage.putFile.bind(server.storage);
|
||||
const spy = vi.spyOn(server.storage, "putFile").mockImplementation(async input => {
|
||||
if (input.namespace === "runner-api") { started(); await gate; }
|
||||
return putFile(input);
|
||||
});
|
||||
const call = () => fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
|
||||
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
|
||||
} });
|
||||
const first = call();
|
||||
try {
|
||||
await saving;
|
||||
expect(await call()).toMatchObject({ ok: false, error: "api_response_capture_limit" });
|
||||
} finally { unblock(); spy.mockRestore(); }
|
||||
const result: any = await first;
|
||||
expect(result).toMatchObject({ ok: true });
|
||||
const [run] = await server.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
|
||||
expect((run.resultJson as Record<string, unknown>).apiResponseCaptureBytes).toBe(used + result.artifact.byteSize);
|
||||
expect(await call()).toMatchObject({ ok: false, error: "api_response_capture_limit" });
|
||||
});
|
||||
|
||||
it("counts old snapshots and other-run reservations against the company quota, and reclaims deletions", async () => {
|
||||
const fixture = await server.fixture();
|
||||
vi.stubEnv("PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES", String(2 * RUNNER_API_RESPONSE_MAX_BYTES));
|
||||
const [oldSnapshot] = await server.db.insert(assets).values({ companyId: fixture.companyId, provider: "local_disk",
|
||||
objectKey: `${fixture.companyId}/runner-api/old-snapshot`, contentType: "text/plain", byteSize: RUNNER_API_RESPONSE_MAX_BYTES,
|
||||
sha256: "old-snapshot", createdByAgentId: fixture.agentId }).returning();
|
||||
const [reservation] = await server.db.insert(runnerApiResponseReservations).values({ companyId: fixture.companyId, runId: null, reservedBytes: RUNNER_API_RESPONSE_MAX_BYTES }).returning();
|
||||
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
|
||||
// A fresh authority instance cannot reset reservations from another run/process.
|
||||
const authority = new PaperclipRunnerToolAuthority(server.db, fixture);
|
||||
const call = () => authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
|
||||
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
|
||||
} });
|
||||
expect(await call()).toMatchObject({ ok: false, error: "api_response_company_storage_limit" });
|
||||
await server.db.delete(assets).where(eq(assets.id, oldSnapshot.id));
|
||||
const first: any = await call();
|
||||
expect(first).toMatchObject({ ok: true });
|
||||
expect(await call()).toMatchObject({ ok: false, error: "api_response_company_storage_limit" });
|
||||
await server.db.delete(assets).where(eq(assets.id, first.artifact.artifactId));
|
||||
expect(await server.db.select().from(runnerApiResponseReservations).where(eq(runnerApiResponseReservations.companyId, fixture.companyId))).toEqual([expect.objectContaining({ id: reservation.id })]);
|
||||
expect(await call()).toMatchObject({ ok: true });
|
||||
});
|
||||
|
||||
it("keeps company storage reserved after an ambiguous upload failure", async () => {
|
||||
const fixture = await server.fixture();
|
||||
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
|
||||
const spy = vi.spyOn(server.storage, "putFile").mockRejectedValueOnce(new Error("storage disconnected"));
|
||||
try {
|
||||
await expect(fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
|
||||
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
|
||||
} })).rejects.toThrow("storage disconnected");
|
||||
} finally { spy.mockRestore(); }
|
||||
expect(await server.db.select().from(runnerApiResponseReservations).where(eq(runnerApiResponseReservations.companyId, fixture.companyId))).toEqual([
|
||||
expect.objectContaining({ assetId: null, reservedBytes: RUNNER_API_RESPONSE_MAX_BYTES }),
|
||||
]);
|
||||
});
|
||||
|
||||
it.each([false, true])("reconciles metadata failure without losing committed assets (commit=%s)", async committed => {
|
||||
const fixture = await server.fixture();
|
||||
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
|
||||
const putFile = server.storage.putFile.bind(server.storage);
|
||||
const transaction = server.db.transaction.bind(server.db);
|
||||
let objectKey = "";
|
||||
let transactionSpy: ReturnType<typeof vi.spyOn> | undefined;
|
||||
const spy = vi.spyOn(server.storage, "putFile").mockImplementationOnce(async input => {
|
||||
const saved = await putFile(input);
|
||||
objectKey = saved.objectKey;
|
||||
transactionSpy = vi.spyOn(server.db, "transaction").mockImplementationOnce(async (fn, config) => {
|
||||
if (committed) await transaction(fn, config);
|
||||
throw new Error("metadata transaction failed");
|
||||
});
|
||||
return saved;
|
||||
});
|
||||
try {
|
||||
await expect(fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
|
||||
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
|
||||
} })).rejects.toThrow("metadata transaction failed");
|
||||
} finally { spy.mockRestore(); transactionSpy?.mockRestore(); }
|
||||
const reservations = await server.db.select().from(runnerApiResponseReservations).where(eq(runnerApiResponseReservations.companyId, fixture.companyId));
|
||||
expect(reservations).toHaveLength(committed ? 1 : 0);
|
||||
if (committed) expect(reservations[0].assetId).toBeTruthy();
|
||||
expect((await server.storage.headObject(fixture.companyId, objectKey)).exists).toBe(committed);
|
||||
});
|
||||
|
||||
it("shares company admission across simultaneous runs", async () => {
|
||||
const fixture = await server.fixture();
|
||||
vi.stubEnv("PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES", String(RUNNER_API_RESPONSE_MAX_BYTES));
|
||||
const otherRunId = randomUUID();
|
||||
await server.db.insert(heartbeatRuns).values({ id: otherRunId, companyId: fixture.companyId, agentId: fixture.agentId,
|
||||
status: "running", runtimeMode: "native", nativeIssueId: fixture.blockerId, contextSnapshot: { issueId: fixture.blockerId } });
|
||||
await server.db.update(issues).set({ status: "in_progress", executionRunId: otherRunId }).where(eq(issues.id, fixture.blockerId));
|
||||
const other = new PaperclipRunnerToolAuthority(server.db, { ...fixture, issueId: fixture.blockerId, runId: otherRunId });
|
||||
await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId));
|
||||
const arguments_ = { operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId } };
|
||||
const results: any[] = await Promise.all([fixture.authority, other].map(authority => authority.execute({ tool: "call_api", callId: randomUUID(), arguments: arguments_ })));
|
||||
expect(results.filter(result => result.ok)).toHaveLength(1);
|
||||
expect(results.find(result => !result.ok)).toMatchObject({ error: "api_response_company_storage_limit" });
|
||||
});
|
||||
|
||||
it("releases a cleaned failed capture's company reservation but retains its run charge", async () => {
|
||||
const fixture = await server.fixture();
|
||||
let emitted = false;
|
||||
const stream = new ReadableStream({
|
||||
async pull(target) {
|
||||
if (!emitted) { emitted = true; target.enqueue(Buffer.alloc(32 * 1024)); }
|
||||
else { await new Promise(resolve => setTimeout(resolve, 100)); target.error(new Error("connection reset")); }
|
||||
},
|
||||
});
|
||||
const spy = vi.spyOn(globalThis, "fetch").mockResolvedValueOnce(new Response(stream, { headers: { "content-type": "text/plain" } }));
|
||||
try {
|
||||
expect(await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
|
||||
operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId },
|
||||
} })).toMatchObject({ ok: false, error: "api_transport_failure" });
|
||||
} finally { spy.mockRestore(); }
|
||||
expect(await server.db.select().from(runnerApiResponseReservations).where(eq(runnerApiResponseReservations.companyId, fixture.companyId))).toEqual([]);
|
||||
const [run] = await server.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
|
||||
expect((run.resultJson as Record<string, unknown>).apiResponseCaptureBytes).toBe(RUNNER_API_RESPONSE_MAX_BYTES);
|
||||
});
|
||||
|
||||
it("persists and range-reads asset sizes above two GiB", async () => {
|
||||
const fixture = await server.fixture();
|
||||
const offsetBytes = 3 * 1024 * 1024 * 1024;
|
||||
const objectKey = `${fixture.companyId}/eval/sparse-large-response.txt`;
|
||||
const path = join(server.root, "storage", objectKey);
|
||||
await mkdir(dirname(path), { recursive: true });
|
||||
const file = await open(path, "wx");
|
||||
try { await file.write(Buffer.from("readable"), 0, 8, offsetBytes); }
|
||||
finally { await file.close(); }
|
||||
const [asset] = await server.db.insert(assets).values({ companyId: fixture.companyId, provider: "local_disk", objectKey,
|
||||
contentType: "text/plain", byteSize: offsetBytes + 8, sha256: "sparse-fixture", createdByAgentId: fixture.agentId }).returning();
|
||||
expect(asset.byteSize).toBe(offsetBytes + 8);
|
||||
const page = await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: {
|
||||
operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: asset.id }, responseText: { offsetBytes, limitBytes: 8 },
|
||||
} });
|
||||
expect(page).toMatchObject({ ok: true, data: "readable", responseText: { offsetBytes, totalBytes: offsetBytes + 8, nextOffsetBytes: null } });
|
||||
});
|
||||
|
||||
it("contains workspace files, checks artifact ownership, and persists downloads", async () => {
|
||||
const fixture = await server.fixture();
|
||||
const foreign = await server.fixture();
|
||||
@@ -260,7 +499,7 @@ else if(m.id!==undefined) send({id:m.id,result:{}});
|
||||
await expect(upload([{ path: "escape.txt" }])).rejects.toThrow();
|
||||
await expect(upload([{ path: "sample.txt" }])).resolves.toMatchObject({ status: 201 });
|
||||
const download = await fixture.authority.execute({ tool: "call_api", callId: "download", arguments: { operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: fixture.binaryArtifactId } } }) as any;
|
||||
expect(download).toMatchObject({ status: 200, byteSize: 32000, artifact: { byteSize: 32000 } });
|
||||
expect(download).toMatchObject({ status: 206, byteSize: 32000, artifact: { artifactId: fixture.binaryArtifactId, byteSize: 32000 } });
|
||||
expect((await fixture.snapshot()).assets).toEqual(expect.arrayContaining([expect.objectContaining({ id: download.artifact.artifactId, companyId: fixture.companyId, createdByAgentId: fixture.agentId })]));
|
||||
});
|
||||
|
||||
|
||||
@@ -11,6 +11,7 @@ import {
|
||||
validateRunnerApiCall,
|
||||
type RunnerApiIo,
|
||||
} from "./runner-api-client.js";
|
||||
import { RUNNER_API_RESPONSE_MAX_BYTES } from "./runner-api-response-limits.js";
|
||||
|
||||
const context = {
|
||||
companyId: "company-a",
|
||||
@@ -32,11 +33,30 @@ const io = (fetcher: typeof fetch): RunnerApiIo => ({
|
||||
}),
|
||||
saveResponse: async (bytes, contentType) => ({
|
||||
artifactId: "artifact-a",
|
||||
byteSize: bytes.length,
|
||||
byteSize: Buffer.isBuffer(bytes) ? bytes.length : bytes.byteSize,
|
||||
contentType,
|
||||
}),
|
||||
});
|
||||
|
||||
describe("bounded response capture receipts", () => {
|
||||
it.each([projects, createProject])("reports oversize evidence without retrying %s", async operationId => {
|
||||
const fetcher = vi.fn(async () => new Response("", { headers: { "content-length": String(RUNNER_API_RESPONSE_MAX_BYTES + 1) } }));
|
||||
const result = await executeRunnerApi({ operationId }, context, io(fetcher));
|
||||
expect(result).toMatchObject({ ok: false, error: "api_response_too_large", maxResponseBytes: RUNNER_API_RESPONSE_MAX_BYTES,
|
||||
outcome: operationId === projects ? "read_failed" : "unknown" });
|
||||
expect(fetcher).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
it("settles the budget and removes the file even when saving fails", async () => {
|
||||
const bytes = Buffer.alloc(32 * 1024);
|
||||
const settle = vi.fn(async () => {});
|
||||
const input = io(async () => new Response(bytes));
|
||||
input.reserveResponseCapture = async () => settle;
|
||||
input.saveResponse = async () => { throw new Error("storage unavailable"); };
|
||||
await expect(executeRunnerApi({ operationId: projects }, context, input)).rejects.toThrow("storage unavailable");
|
||||
expect(settle).toHaveBeenCalledExactlyOnceWith(bytes.length, true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("runner API catalog", () => {
|
||||
it("accounts for unique operations with resolved request contracts", () => {
|
||||
const catalog = runnerApiCatalog();
|
||||
@@ -591,6 +611,127 @@ describe("runner API request boundary", () => {
|
||||
).toThrow("inline JSON object");
|
||||
}
|
||||
});
|
||||
it("reads text windows from a live response and supplies stable snapshots", async () => {
|
||||
const text = "prefix\n" + "🧭é\n".repeat(6000) + "END-OF-EVIDENCE";
|
||||
const saveResponse = vi.fn(io(fetch).saveResponse);
|
||||
const request = vi.fn<typeof fetch>(async () => new Response(text, { headers: { "content-type": "application/json" } }));
|
||||
let offsetBytes = 0;
|
||||
let received = "";
|
||||
do {
|
||||
const result = await executeRunnerApi({ operationId: projects, responseText: { offsetBytes, limitBytes: 4096 } }, context, { ...io(request), saveResponse }) as any;
|
||||
expect(result).toMatchObject({ ok: true, responseText: { offsetBytes, totalBytes: Buffer.byteLength(text) } });
|
||||
expect(Buffer.byteLength(result.data)).toBeLessThanOrEqual(4096);
|
||||
received += result.data;
|
||||
offsetBytes = result.responseText.nextOffsetBytes;
|
||||
} while (offsetBytes !== null);
|
||||
expect(received).toBe(text);
|
||||
expect(saveResponse).toHaveBeenCalled();
|
||||
expect(request.mock.calls.every(([, options]) => new Headers(options?.headers).get("Authorization") === "Bearer private-agent-token")).toBe(true);
|
||||
});
|
||||
it.each(["text/plain", "application/octet-stream"])("does not duplicate an unpaged multi-gigabyte asset (%s)", async contentType => {
|
||||
const totalBytes = 3 * 1024 * 1024 * 1024;
|
||||
const saveResponse = vi.fn(io(fetch).saveResponse);
|
||||
const request = vi.fn<typeof fetch>(async (_url, init) => {
|
||||
expect(new Headers(init?.headers).get("range")).toBe("bytes=0-24576");
|
||||
return new Response(Buffer.alloc(24577, 65), { status: 206, headers: {
|
||||
"content-type": contentType, "content-range": `bytes 0-24576/${totalBytes}`, etag: `"${"a".repeat(64)}"`,
|
||||
} });
|
||||
});
|
||||
const result = await executeRunnerApi({ operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: "existing" } }, context, { ...io(request), saveResponse });
|
||||
expect(result).toMatchObject({ ok: true, status: 206, byteSize: totalBytes, artifact: { artifactId: "existing", byteSize: totalBytes, sha256: "a".repeat(64) } });
|
||||
expect(saveResponse).not.toHaveBeenCalled();
|
||||
expect(request).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
it("reads a page beyond ten MiB in a multi-gigabyte asset", async () => {
|
||||
const offsetBytes = 3 * 1024 * 1024 * 1024;
|
||||
const totalBytes = offsetBytes + 100;
|
||||
const request = vi.fn<typeof fetch>(async (_url, init) => {
|
||||
expect(new Headers(init?.headers).get("range")).toBe(`bytes=${offsetBytes - 1}-${offsetBytes + 4}`);
|
||||
return new Response("abcdef", { status: 206, headers: {
|
||||
"content-type": "text/plain", "content-range": `bytes ${offsetBytes - 1}-${offsetBytes + 4}/${totalBytes}`,
|
||||
} });
|
||||
});
|
||||
expect(await executeRunnerApi({ operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: "large" }, responseText: { offsetBytes, limitBytes: 4 } }, context, io(request))).toMatchObject({
|
||||
ok: true, data: "bcde", responseText: { offsetBytes, nextOffsetBytes: offsetBytes + 4, totalBytes },
|
||||
});
|
||||
});
|
||||
it.each([true, false])("saves responses above ten MiB (known length: %s)", async knownLength => {
|
||||
const bytes = Buffer.alloc(12 * 1024 * 1024, 65);
|
||||
const request = vi.fn<typeof fetch>(async () => new Response(bytes, { headers: {
|
||||
"content-type": "text/plain", ...(knownLength ? { "content-length": String(bytes.length) } : {}),
|
||||
} }));
|
||||
const result = await executeRunnerApi({ operationId: projects }, context, io(request));
|
||||
expect(result).toMatchObject({ ok: true, artifact: { byteSize: bytes.length }, byteSize: bytes.length });
|
||||
});
|
||||
it("pages a twelve MiB asset with linear transfer and UTF-8 boundaries", async () => {
|
||||
const bytes = Buffer.from("🧭é".repeat(Math.floor(12 * 1024 * 1024 / 6)));
|
||||
let transferred = 0;
|
||||
const request = vi.fn<typeof fetch>(async (_url, init) => {
|
||||
const range = new Headers(init?.headers).get("range");
|
||||
const match = /^bytes=(\d+)-(\d+)$/.exec(range ?? "");
|
||||
if (!match) { transferred += bytes.length; return new Response(bytes, { headers: { "content-type": "text/plain" } }); }
|
||||
const start = Number(match[1]), end = Math.min(Number(match[2]), bytes.length - 1);
|
||||
const part = bytes.subarray(start, end + 1); transferred += part.length;
|
||||
return new Response(part, { status: 206, headers: { "content-type": "text/plain", "content-range": `bytes ${start}-${end}/${bytes.length}` } });
|
||||
});
|
||||
let offsetBytes: number | null = 0;
|
||||
const parts: string[] = [];
|
||||
do {
|
||||
const page: any = await executeRunnerApi({ operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: "large" }, responseText: { offsetBytes } }, context, io(request));
|
||||
expect(page.ok).toBe(true);
|
||||
parts.push(page.data); offsetBytes = page.responseText.nextOffsetBytes;
|
||||
} while (offsetBytes !== null);
|
||||
expect(parts.join("")).toBe(bytes.toString());
|
||||
expect(transferred).toBeLessThan(bytes.length + request.mock.calls.length * 5);
|
||||
expect(request.mock.calls.every(([, init]) => new Headers(init?.headers).has("range"))).toBe(true);
|
||||
});
|
||||
it.each([
|
||||
["bytes 4-8/20", "12345"], // wrong start
|
||||
["bytes 3-7/20", "12345"], // early end
|
||||
["bytes 3-8/20", "12345"], // truncated body
|
||||
["bytes 3-8/9007199254740992", "123456"], // imprecise total
|
||||
["bytes 3-8/*", "123456"], // unknown total
|
||||
[null, "123456"],
|
||||
])("rejects inconsistent partial asset receipts: %s", async (range, body) => {
|
||||
const result = await executeRunnerApi({ operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: "saved" }, responseText: { offsetBytes: 4, limitBytes: 4 } }, context, io(async () => new Response(body, { status: 206, headers: { "content-type": "text/plain", ...(range ? { "content-range": range } : {}) } })));
|
||||
expect(result).toMatchObject({ ok: false, error: "response_text_invalid_range" });
|
||||
});
|
||||
it("accepts EOF and rejects an offset inside a UTF-8 code point in ranged assets", async () => {
|
||||
const call = (offsetBytes: number, bytes: Buffer, range: string) => executeRunnerApi({ operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: "saved" }, responseText: { offsetBytes, limitBytes: 4 } }, context, io(async () => new Response(new Uint8Array(bytes), { status: 206, headers: { "content-type": "text/plain", "content-range": range } })));
|
||||
expect(await call(2, Buffer.from([0xa9]), "bytes 1-1/2")).toMatchObject({ ok: true, data: "", responseText: { nextOffsetBytes: null, totalBytes: 2 } });
|
||||
expect(await call(1, Buffer.from("é"), "bytes 0-1/2")).toMatchObject({ ok: false, error: "response_text_invalid_offset" });
|
||||
});
|
||||
it("does not silently accept an asset server that ignores the byte range", async () => {
|
||||
const result = await executeRunnerApi({ operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: "saved" }, responseText: { limitBytes: 4 } }, context, io(async () => new Response("unbounded", { headers: { "content-type": "text/plain" } })));
|
||||
expect(result).toMatchObject({ ok: false, error: "api_transport_failure" });
|
||||
});
|
||||
it.each([{ offsetBytes: -1 }, { limitBytes: 0 }, { limitBytes: 3 }, { limitBytes: 24577 }, { offsetBytes: 0.5 }, { offsetBytes: Number.MAX_SAFE_INTEGER + 1 }])("rejects invalid text windows before dispatch: %j", async responseText => {
|
||||
const request = vi.fn<typeof fetch>();
|
||||
await expect(executeRunnerApi({ operationId: projects, responseText }, context, io(request))).rejects.toThrow("Invalid call_api arguments");
|
||||
expect(request).not.toHaveBeenCalled();
|
||||
});
|
||||
it("does not page mutation responses by repeating the mutation", async () => {
|
||||
const request = vi.fn<typeof fetch>();
|
||||
await expect(executeRunnerApi({ operationId: createProject, body: { name: "Once" }, responseText: {} }, context, io(request))).rejects.toThrow("GET");
|
||||
expect(request).not.toHaveBeenCalled();
|
||||
});
|
||||
it("rejects binary, invalid UTF-8 and offsets that split text", async () => {
|
||||
for (const [body, type, offsetBytes] of [
|
||||
[Buffer.from("binary"), "application/octet-stream", 0],
|
||||
[Buffer.from([0xff]), "text/plain", 0],
|
||||
[Buffer.concat([Buffer.from([0xc2]), Buffer.alloc(24576, 0x80)]), "text/plain", 0],
|
||||
[Buffer.from("é"), "text/plain", 1],
|
||||
[Buffer.from("short"), "text/plain", 6],
|
||||
] as const) {
|
||||
const saveResponse = vi.fn(io(fetch).saveResponse);
|
||||
const result = await executeRunnerApi({ operationId: projects, responseText: { offsetBytes } }, context, { ...io(async () => new Response(body, { headers: { "content-type": type } })), saveResponse });
|
||||
expect(result).toMatchObject({ ok: false, error: expect.stringContaining("response_text") });
|
||||
expect(saveResponse).not.toHaveBeenCalled();
|
||||
}
|
||||
});
|
||||
it("preserves denials for bounded text reads", async () => {
|
||||
expect(await executeRunnerApi({ operationId: projects, responseText: {} }, context, io(async () => Response.json({ error: "denied" }, { status: 403 })))).toMatchObject({ ok: false, status: 403, data: '{"error":"denied"}' });
|
||||
});
|
||||
it("bounds streamed responses even without content-length", async () => {
|
||||
await expect(
|
||||
readBoundedResponse(new Response("too large"), 3),
|
||||
|
||||
@@ -47,8 +47,10 @@ export function createLocalDiskStorageProvider(baseDir: string): StorageProvider
|
||||
await fs.mkdir(dir, { recursive: true });
|
||||
|
||||
const tempPath = `${targetPath}.tmp-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`;
|
||||
await fs.writeFile(tempPath, input.body);
|
||||
await fs.rename(tempPath, targetPath);
|
||||
try {
|
||||
await fs.writeFile(tempPath, input.body);
|
||||
await fs.rename(tempPath, targetPath);
|
||||
} finally { await fs.rm(tempPath, { force: true }); }
|
||||
},
|
||||
|
||||
async getObject(input): Promise<GetObjectResult> {
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
import { AbortMultipartUploadCommand, CompleteMultipartUploadCommand, CreateMultipartUploadCommand, UploadPartCommand, type S3Client } from "@aws-sdk/client-s3";
|
||||
import { Readable } from "node:stream";
|
||||
import type { PutObjectInput } from "./types.js";
|
||||
|
||||
/** Each part is streamed too: multipart must not become another whole-file
|
||||
* buffer or a fixed part-count ceiling below the storage provider's limits. */
|
||||
export async function putS3Multipart(client: S3Client, bucket: string, key: string, input: PutObjectInput) {
|
||||
const source = input.body instanceof Readable ? input.body : Readable.from([input.body]);
|
||||
const iterator = source[Symbol.asyncIterator]();
|
||||
let pending = Buffer.alloc(0);
|
||||
let uploadId: string | undefined;
|
||||
// Scale part sizes to the S3 part-count limit while retaining stream backpressure.
|
||||
const partSize = Math.max(8 * 1024 * 1024, Math.ceil(input.contentLength / 10_000));
|
||||
try {
|
||||
uploadId = (await client.send(new CreateMultipartUploadCommand({ Bucket: bucket, Key: key, ContentType: input.contentType }))).UploadId;
|
||||
if (!uploadId) throw new Error("S3 did not return a multipart upload id");
|
||||
const parts: { ETag: string; PartNumber: number }[] = [];
|
||||
for (let offset = 0; offset < input.contentLength;) {
|
||||
const length = Math.min(partSize, input.contentLength - offset);
|
||||
let remaining = length;
|
||||
const body = Readable.from((async function* () {
|
||||
while (remaining > 0) {
|
||||
if (!pending.length) {
|
||||
const next = await iterator.next();
|
||||
if (next.done) throw new Error("S3 upload source was truncated");
|
||||
pending = Buffer.isBuffer(next.value) ? next.value : Buffer.from(next.value);
|
||||
if (!pending.length) continue;
|
||||
}
|
||||
const count = Math.min(remaining, pending.length);
|
||||
const chunk = pending.subarray(0, count);
|
||||
pending = pending.subarray(count);
|
||||
remaining -= count;
|
||||
yield chunk;
|
||||
}
|
||||
})());
|
||||
try {
|
||||
const result = await client.send(new UploadPartCommand({ Bucket: bucket, Key: key, UploadId: uploadId, PartNumber: parts.length + 1, Body: body, ContentLength: length }));
|
||||
if (remaining || !result.ETag) throw new Error("S3 upload part is incomplete");
|
||||
parts.push({ ETag: result.ETag, PartNumber: parts.length + 1 });
|
||||
} finally { body.destroy(); }
|
||||
offset += length;
|
||||
}
|
||||
if (pending.length || !(await iterator.next()).done) throw new Error("S3 upload source exceeds its declared size");
|
||||
await client.send(new CompleteMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId, MultipartUpload: { Parts: parts } }));
|
||||
} catch (error) {
|
||||
if (uploadId) await client.send(new AbortMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId })).catch(() => {});
|
||||
throw error;
|
||||
} finally {
|
||||
source.destroy();
|
||||
await iterator.return?.();
|
||||
}
|
||||
}
|
||||
@@ -5,6 +5,7 @@ import {
|
||||
HeadObjectCommand,
|
||||
PutObjectCommand,
|
||||
} from "@aws-sdk/client-s3";
|
||||
import { putS3Multipart } from "./s3-multipart.js";
|
||||
import { Readable } from "node:stream";
|
||||
import type { StorageProvider, GetObjectResult, HeadObjectResult } from "./types.js";
|
||||
import { notFound, unprocessable } from "../errors.js";
|
||||
@@ -81,6 +82,10 @@ export function createS3StorageProvider(config: S3ProviderConfig): StorageProvid
|
||||
|
||||
async putObject(input) {
|
||||
const key = buildKey(prefix, input.objectKey);
|
||||
if (input.contentLength > 16 * 1024 * 1024) {
|
||||
await putS3Multipart(client, bucket, key, input);
|
||||
return;
|
||||
}
|
||||
await client.send(
|
||||
new PutObjectCommand({
|
||||
Bucket: bucket,
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import path from "node:path";
|
||||
import { Readable } from "node:stream";
|
||||
import type { StorageService, StorageProvider, PutFileInput, PutFileResult } from "./types.js";
|
||||
import { badRequest, forbidden, unprocessable } from "../errors.js";
|
||||
|
||||
@@ -79,10 +80,13 @@ function assertPutFileInput(input: PutFileInput): void {
|
||||
if (!input.contentType || input.contentType.trim().length === 0) {
|
||||
throw unprocessable("contentType is required");
|
||||
}
|
||||
if (!(input.body instanceof Buffer)) {
|
||||
throw unprocessable("body must be a Buffer");
|
||||
if (!(input.body instanceof Buffer) && !(input.body instanceof Readable)) {
|
||||
throw unprocessable("body must be a Buffer or Readable");
|
||||
}
|
||||
if (input.body.length <= 0) {
|
||||
if (!Buffer.isBuffer(input.body) && (!("byteSize" in input) || !Number.isSafeInteger(input.byteSize) || input.byteSize < 0 || !/^[0-9a-f]{64}$/.test(input.sha256))) {
|
||||
throw unprocessable("Streamed files require an exact byte size and SHA-256");
|
||||
}
|
||||
if (("byteSize" in input ? input.byteSize : input.body.length) <= 0) {
|
||||
throw unprocessable("File is empty");
|
||||
}
|
||||
}
|
||||
@@ -94,21 +98,28 @@ export function createStorageService(provider: StorageProvider): StorageService
|
||||
async putFile(input: PutFileInput): Promise<PutFileResult> {
|
||||
assertPutFileInput(input);
|
||||
const objectKey = buildObjectKey(input.companyId, input.namespace, input.originalFilename);
|
||||
const byteSize = input.body.length;
|
||||
const byteSize = "byteSize" in input ? input.byteSize : input.body.length;
|
||||
const contentType = input.contentType.trim().toLowerCase();
|
||||
await provider.putObject({
|
||||
objectKey,
|
||||
body: input.body,
|
||||
contentType,
|
||||
contentLength: byteSize,
|
||||
});
|
||||
try {
|
||||
await provider.putObject({
|
||||
objectKey,
|
||||
body: input.body,
|
||||
contentType,
|
||||
contentLength: byteSize,
|
||||
});
|
||||
} catch (error) {
|
||||
await provider.deleteObject({ objectKey }).catch(() => {});
|
||||
throw error;
|
||||
} finally {
|
||||
if (input.body instanceof Readable) input.body.destroy();
|
||||
}
|
||||
|
||||
return {
|
||||
provider: provider.id,
|
||||
objectKey,
|
||||
contentType,
|
||||
byteSize,
|
||||
sha256: hashBuffer(input.body),
|
||||
sha256: "sha256" in input ? input.sha256 : hashBuffer(input.body),
|
||||
originalFilename: input.originalFilename,
|
||||
};
|
||||
},
|
||||
|
||||
@@ -42,13 +42,12 @@ export interface StorageProvider {
|
||||
deleteObject(input: GetObjectInput): Promise<void>;
|
||||
}
|
||||
|
||||
export interface PutFileInput {
|
||||
export type PutFileInput = {
|
||||
companyId: string;
|
||||
namespace: string;
|
||||
originalFilename: string | null;
|
||||
contentType: string;
|
||||
body: Buffer;
|
||||
}
|
||||
} & ({ body: Buffer } | { body: Readable; byteSize: number; sha256: string });
|
||||
|
||||
export interface PutFileResult {
|
||||
provider: StorageProviderId;
|
||||
|
||||
@@ -1094,3 +1094,15 @@ keys with low request limits. It can only lower the configured campaign limit.
|
||||
Keep subscription qualification separate from API-key results.
|
||||
|
||||
The explicit-only eight-cell [continuation accounting baseline](CONTINUATION-ACCOUNTING.md) tests productive work, bounded repair, restart and late gates with real providers.
|
||||
|
||||
### Bounded API response reading
|
||||
|
||||
`api-response-reading` is an explicit-only, two-cell native Codex suite (local
|
||||
and Daytona). Each cell creates synthetic diagnostic evidence over 24 KiB via
|
||||
the public API, outside the agent assignment. The browser starts one ordinary
|
||||
task which must read the saved response in bounded text pages and persist the
|
||||
exact hidden evidence code. The oracle also requires completed API tool events;
|
||||
missing events or a narrative completion cannot pass. Existing run, copyback,
|
||||
screenshot, billing and environment cleanup checks apply. Use
|
||||
`--id api-response-reading.runner-codex.daytona.saved-text-pages` with an
|
||||
immutable Daytona image; no private hooks or fixture database writes are used.
|
||||
@@ -0,0 +1,102 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import type { RunnerApi } from "./api.js";
|
||||
import { apiResponseReadingTask, gradeApiResponsePaging, readResponseProof, responseEvidenceCode, responseEvidenceDescription } from "./api-response-reading.js";
|
||||
|
||||
describe("large API response evidence fixture", () => {
|
||||
it("keeps evidence beyond both preview and inline limits and out of the assignment", () => {
|
||||
const source = responseEvidenceDescription("fixture");
|
||||
const code = responseEvidenceCode("fixture");
|
||||
expect(Buffer.byteLength(source)).toBeGreaterThan(24 * 1024);
|
||||
expect(source.indexOf(code)).toBeGreaterThan(24 * 1024);
|
||||
expect(apiResponseReadingTask.buildPrompt("fixture")).not.toContain(code);
|
||||
expect(apiResponseReadingTask.buildPrompt("fixture")).toContain("responseText.nextOffsetBytes");
|
||||
});
|
||||
it("downloads exact run-attributed proof bytes rather than trusting the agent's report", async () => {
|
||||
const bytes = Buffer.from("independent evidence\n");
|
||||
const proof = { id: "proof", issueId: "issue", originatingRunId: "run", originalFilename: "api-response-proof.txt", byteSize: bytes.length, sha256: createHash("sha256").update(bytes).digest("hex") };
|
||||
const download = vi.fn().mockResolvedValue({ ok: () => true, body: async () => bytes });
|
||||
const get = vi.fn().mockResolvedValue([proof]);
|
||||
const api = { get, request: { get: download } } as unknown as RunnerApi;
|
||||
expect((await readResponseProof(api, "issue", "run")).content).toBe(bytes.toString());
|
||||
expect(download).toHaveBeenCalledWith("/api/attachments/proof/content?download=1");
|
||||
get.mockResolvedValue([{ ...proof, originatingRunId: "other" }]);
|
||||
await expect(readResponseProof(api, "issue", "run")).rejects.toThrow("observed 0");
|
||||
get.mockResolvedValue([proof, proof]);
|
||||
await expect(readResponseProof(api, "issue", "run")).rejects.toThrow("observed 2");
|
||||
get.mockResolvedValue([{ ...proof, sha256: "mismatch" }]);
|
||||
await expect(readResponseProof(api, "issue", "run")).rejects.toThrow("disagrees");
|
||||
});
|
||||
});
|
||||
|
||||
describe("saved API response pagination oracle", () => {
|
||||
const sourceIssueId = "source-issue";
|
||||
const assetId = "source-asset";
|
||||
const bytes = Buffer.from(JSON.stringify({ id: sourceIssueId, description: "synthetic padding ".repeat(2000) }));
|
||||
const operation = "GET /api/assets/{assetId}/content";
|
||||
const event = (eventType: string, payload: unknown) => ({ eventType, payload: { prpEvent: { payload } } });
|
||||
const call = (id: string, input: unknown, result: unknown, status = "completed") => [
|
||||
event("item.started", { item: { id, type: "tool_use", name: "call_api", input } }),
|
||||
event("item.completed", { item: { id, type: "tool_result", tool_use_id: id, result } }),
|
||||
event("tool.execution.completed", { name: "call_api", executionId: id, status }),
|
||||
];
|
||||
const source = () => call("source-read", { operationId: "GET /api/issues/{id}", pathParams: { id: sourceIssueId } }, {
|
||||
ok: true, status: 200, apiOperationId: "GET /api/issues/{id}", artifact: {
|
||||
artifactId: assetId, byteSize: bytes.length, sha256: createHash("sha256").update(bytes).digest("hex"),
|
||||
url: `/api/assets/${assetId}/content`,
|
||||
},
|
||||
});
|
||||
const page = (offset: number, id = `page-${offset}`, artifactId = assetId) => {
|
||||
const end = Math.min(offset + 8192, bytes.length);
|
||||
return call(id, { operationId: operation, pathParams: { assetId: artifactId }, responseText: { offsetBytes: offset, limitBytes: 8192 } }, {
|
||||
ok: true, status: 200, apiOperationId: operation, data: bytes.subarray(offset, end).toString(),
|
||||
responseText: { offsetBytes: offset, totalBytes: bytes.length, nextOffsetBytes: end === bytes.length ? null : end },
|
||||
});
|
||||
};
|
||||
const pages = () => Array.from({ length: Math.ceil(bytes.length / 8192) }, (_, i) => page(i * 8192)).flat();
|
||||
it("accepts correlated source artifact pages through EOF, including a harmless identical retry", () => {
|
||||
expect(gradeApiResponsePaging([...source(), ...page(0, "retry"), ...pages()], sourceIssueId).passed).toBe(true);
|
||||
});
|
||||
it("rejects unrelated completed calls", () => {
|
||||
const unrelated = [1, 2, 3].flatMap(i => call(`unrelated-${i}`, { operationId: "GET /api/agents/me" }, { ok: true, status: 200 }));
|
||||
expect(gradeApiResponsePaging(unrelated, sourceIssueId).passed).toBe(false);
|
||||
});
|
||||
it("rejects repeating the first page without reaching EOF", () => {
|
||||
expect(gradeApiResponsePaging([...source(), ...page(0), ...page(0, "again"), ...page(0, "again-2")], sourceIssueId).passed).toBe(false);
|
||||
});
|
||||
it("rejects a gap even when the last page claims EOF", () => {
|
||||
expect(gradeApiResponsePaging([...source(), ...pages().filter((_e, i) => i < 3 || i >= 6)], sourceIssueId).passed).toBe(false);
|
||||
});
|
||||
it("rejects reading a different artifact", () => {
|
||||
const other = Array.from({ length: Math.ceil(bytes.length / 8192) }, (_, i) => page(i * 8192, `other-${i}`, "other-asset")).flat();
|
||||
expect(gradeApiResponsePaging([...source(), ...other], sourceIssueId).passed).toBe(false);
|
||||
});
|
||||
it("requires the exact seeded source issue", () => {
|
||||
expect(gradeApiResponsePaging([...source(), ...pages()], "other-issue").passed).toBe(false);
|
||||
});
|
||||
it("requires a matched successful completion for every page", () => {
|
||||
const incomplete = pages().filter(e => e.eventType !== "tool.execution.completed");
|
||||
expect(gradeApiResponsePaging([...source(), ...incomplete], sourceIssueId).passed).toBe(false);
|
||||
});
|
||||
it.each([
|
||||
["requested offset", { offsetBytes: 1, totalBytes: bytes.length, nextOffsetBytes: 8192 }],
|
||||
["next offset", { offsetBytes: 0, totalBytes: bytes.length, nextOffsetBytes: 8193 }],
|
||||
["total size", { offsetBytes: 0, totalBytes: bytes.length + 1, nextOffsetBytes: 8192 }],
|
||||
["premature EOF", { offsetBytes: 0, totalBytes: bytes.length, nextOffsetBytes: null }],
|
||||
])("rejects mismatched %s metadata", (_label, responseText) => {
|
||||
const malformed = call("bad-page", {
|
||||
operationId: operation, pathParams: { assetId }, responseText: { offsetBytes: 0, limitBytes: 8192 },
|
||||
}, { ok: true, status: 206, apiOperationId: operation, data: bytes.subarray(0, 8192).toString(), responseText });
|
||||
expect(gradeApiResponsePaging([...source(), ...pages(), ...malformed], sourceIssueId).passed).toBe(false);
|
||||
});
|
||||
it("rejects contradictory duplicate bytes and a wrong source artifact digest", () => {
|
||||
const corrupt = call("corrupt-page", {
|
||||
operationId: operation, pathParams: { assetId }, responseText: { offsetBytes: 0, limitBytes: 8192 },
|
||||
}, {
|
||||
ok: true, status: 206, apiOperationId: operation, data: "x".repeat(8192),
|
||||
responseText: { offsetBytes: 0, totalBytes: bytes.length, nextOffsetBytes: 8192 },
|
||||
});
|
||||
expect(gradeApiResponsePaging([...source(), ...pages(), ...corrupt], sourceIssueId).passed).toBe(false);
|
||||
expect(gradeApiResponsePaging([...source(), ...corrupt, ...pages().slice(3)], sourceIssueId).passed).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,134 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import type { RunnerTaskFixture } from "./types.js";
|
||||
import type { RunnerApi } from "./api.js";
|
||||
import type { IssueAttachment } from "../../packages/shared/src/types/issue.js";
|
||||
|
||||
/** Verify delivered bytes through the same public download path the user receives. */
|
||||
export async function readResponseProof(api: RunnerApi, issueId: string, runId: string) {
|
||||
const attachments = await api.get<IssueAttachment[]>(`/api/issues/${issueId}/attachments`);
|
||||
const proofs = attachments.filter(attachment =>
|
||||
attachment.issueId === issueId && attachment.originatingRunId === runId &&
|
||||
attachment.originalFilename === "api-response-proof.txt");
|
||||
if (proofs.length !== 1) throw new Error(`Expected one proof attachment from the tested run; observed ${proofs.length}`);
|
||||
const proof = proofs[0]!;
|
||||
const response = await api.request.get(`/api/attachments/${encodeURIComponent(proof.id)}/content?download=1`);
|
||||
if (!response.ok()) throw new Error(`Proof download returned ${response.status()}`);
|
||||
const bytes = await response.body();
|
||||
if (bytes.length !== proof.byteSize || createHash("sha256").update(bytes).digest("hex") !== proof.sha256) {
|
||||
throw new Error("Downloaded proof disagrees with stored attachment bytes");
|
||||
}
|
||||
return { attachmentId: proof.id, content: bytes.toString("utf8"), sha256: proof.sha256 };
|
||||
}
|
||||
|
||||
export function responseEvidenceCode(nonce: string) {
|
||||
return createHash("sha256").update(`bounded-response-evidence:${nonce}`).digest("hex");
|
||||
}
|
||||
export function responseEvidenceDescription(nonce: string) {
|
||||
return `${"Synthetic diagnostic padding.\n".repeat(1400)}\nEvidence code: ${responseEvidenceCode(nonce)}\n`;
|
||||
}
|
||||
|
||||
/** Correlate trusted PRP call inputs, results, and completion receipts. */
|
||||
export function gradeApiResponsePaging(events: readonly { eventType?: string; payload?: unknown }[], sourceIssueId: string) {
|
||||
const object = (value: unknown): Record<string, unknown> =>
|
||||
value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
|
||||
const inputs = new Map<string, Record<string, unknown>>();
|
||||
const results = new Map<string, Record<string, unknown>>();
|
||||
const completed = new Set<string>();
|
||||
for (const event of events) {
|
||||
const payload = object(object(object(event.payload).prpEvent).payload);
|
||||
const item = object(payload.item);
|
||||
if (event.eventType === "item.started" && item.type === "tool_use" && item.name === "call_api" && typeof item.id === "string") {
|
||||
inputs.set(item.id, object(item.input));
|
||||
}
|
||||
if (event.eventType === "item.completed" && item.type === "tool_result" && typeof item.id === "string" &&
|
||||
(item.tool_use_id === undefined || item.tool_use_id === item.id)) results.set(item.id, object(item.result));
|
||||
if (event.eventType === "tool.execution.completed" && payload.name === "call_api" && payload.status === "completed" && typeof payload.executionId === "string") {
|
||||
completed.add(payload.executionId);
|
||||
}
|
||||
}
|
||||
const calls = [...inputs].flatMap(([id, input]) => {
|
||||
const result = results.get(id);
|
||||
return completed.has(id) && result?.ok === true && [200, 206].includes(Number(result.status)) ? [{ input, result }] : [];
|
||||
});
|
||||
const sourceOperation = "GET /api/issues/{id}";
|
||||
const assetOperation = "GET /api/assets/{assetId}/content";
|
||||
const sources = calls.filter(call => call.input.operationId === sourceOperation && call.result.apiOperationId === sourceOperation &&
|
||||
object(call.input.pathParams).id === sourceIssueId);
|
||||
let failure = "No successful call_api source read with a saved response artifact";
|
||||
for (const source of sources) {
|
||||
const artifact = object(source.result.artifact);
|
||||
const assetId = artifact.artifactId;
|
||||
const total = artifact.byteSize;
|
||||
if (typeof assetId !== "string" || typeof total !== "number" || !Number.isSafeInteger(total) || total <= 24 * 1024 || total > 1024 * 1024 ||
|
||||
typeof artifact.sha256 !== "string" || !/^[a-f0-9]{64}$/.test(artifact.sha256) || artifact.url !== `/api/assets/${assetId}/content`) continue;
|
||||
const pages = new Map<number, { bytes: Buffer; next: number | null }>();
|
||||
let malformed = false;
|
||||
for (const call of calls) {
|
||||
if (call.input.operationId !== assetOperation || object(call.input.pathParams).assetId !== assetId) continue;
|
||||
const requested = object(call.input.responseText);
|
||||
const returned = object(call.result.responseText);
|
||||
const offset = requested.offsetBytes;
|
||||
const limit = requested.limitBytes;
|
||||
const next = returned.nextOffsetBytes;
|
||||
const data = call.result.data;
|
||||
if (call.result.apiOperationId !== assetOperation || typeof offset !== "number" || !Number.isSafeInteger(offset) || offset < 0 || offset >= total ||
|
||||
typeof limit !== "number" || !Number.isSafeInteger(limit) || limit < 1 || limit > 8192 || returned.offsetBytes !== offset || returned.totalBytes !== total ||
|
||||
typeof data !== "string") { malformed = true; break; }
|
||||
const bytes = Buffer.from(data, "utf8");
|
||||
const end = offset + bytes.length;
|
||||
if (bytes.length === 0 || bytes.length > limit || end > total || (end === total ? next !== null : next !== end)) {
|
||||
malformed = true; break;
|
||||
}
|
||||
const prior = pages.get(offset);
|
||||
if (prior && (!prior.bytes.equals(bytes) || prior.next !== next)) { malformed = true; break; }
|
||||
pages.set(offset, { bytes, next: next as number | null });
|
||||
}
|
||||
if (malformed) { failure = "Artifact paging request, returned offsets, bytes, or EOF disagree"; continue; }
|
||||
const chunks: Buffer[] = [];
|
||||
const offsets: number[] = [];
|
||||
let offset = 0;
|
||||
let eofReached = false;
|
||||
while (pages.has(offset)) {
|
||||
const page = pages.get(offset)!;
|
||||
chunks.push(page.bytes); offsets.push(offset);
|
||||
if (page.next === null) { eofReached = true; break; }
|
||||
offset = page.next;
|
||||
}
|
||||
if (!eofReached || offsets.length < 2) { failure = "Saved artifact pages do not cover offset zero through EOF without gaps"; continue; }
|
||||
const bytes = Buffer.concat(chunks);
|
||||
const digest = createHash("sha256").update(bytes).digest("hex");
|
||||
if (bytes.length !== total || digest !== artifact.sha256) { failure = "Paged bytes disagree with the source response artifact digest"; continue; }
|
||||
try {
|
||||
if (object(JSON.parse(bytes.toString("utf8"))).id !== sourceIssueId) { failure = "Paged artifact belongs to another source issue"; continue; }
|
||||
} catch { failure = "Paged source response is not complete JSON"; continue; }
|
||||
return { passed: true, sourceIssueId, artifactId: assetId, totalBytes: total, pageCount: offsets.length, offsets, eofReached, sha256: digest };
|
||||
}
|
||||
return { passed: false, sourceIssueId, failure };
|
||||
}
|
||||
|
||||
export const apiResponseReadingTask: RunnerTaskFixture = {
|
||||
id: "saved-text-pages", label: "Read saved large API evidence", groups: [],
|
||||
workMode: "standard", flow: "single_turn", expectedRunCount: 1,
|
||||
attemptTimeoutMs: { local: 10 * 60_000, daytona: 10 * 60_000 },
|
||||
expectedTerminalState: { issue: "done", run: "succeeded" },
|
||||
buildTitle: nonce => `Read large API evidence ${nonce}`,
|
||||
// Delivered files render as an attachment card, which can replace the finish summary.
|
||||
buildVisibleMarker: () => "api-response-proof.txt",
|
||||
buildPrompt: nonce => [
|
||||
"Inspect diagnostic evidence task {{API_RESPONSE_SOURCE_ID}} using the Paperclip API tools.",
|
||||
"Discover GET /api/issues/{id}, call it for that task, and retain the returned response artifact.",
|
||||
"Read the saved artifact through GET /api/assets/{assetId}/content with responseText: {offsetBytes:0,limitBytes:8192}.",
|
||||
"Continue using responseText.nextOffsetBytes until null. Extract the Evidence code at the end of its description.",
|
||||
"The large response must be read with bounded responseText pages. Do not use other tools or API routes to obtain the evidence; if bounded reading fails, report the failure instead of substituting a different reader.",
|
||||
"Write only that code followed by a newline to api-response-proof.txt in the current workspace and deliver that file as an attachment. Do not edit the evidence task or create child tasks.",
|
||||
`Verify the proof file and finish this task with paperclip_finish, reportedWorkDisposition done, and summary API_RESPONSE_READ_${nonce}.`,
|
||||
].join("\n"),
|
||||
buildMatchers: (nonce, execution) => [
|
||||
{ kind: "file_exact", path: "api-response-proof.txt", expected: `${responseEvidenceCode(nonce)}\n` },
|
||||
{ kind: "message_contains", expected: "api-response-proof.txt" },
|
||||
{ kind: "issue_status", expected: "done" },
|
||||
{ kind: "run_status", expected: "succeeded" },
|
||||
{ kind: "runtime_mode", expected: execution.profile.expectedRuntimeMode },
|
||||
{ kind: "environment", expected: execution.environment.id },
|
||||
],
|
||||
};
|
||||
@@ -71,10 +71,10 @@ describe("runner E2E catalog", () => {
|
||||
expect(localIntegrityTasks).toHaveLength(2);
|
||||
expect(openRouterBreadthTasks).toHaveLength(3);
|
||||
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
|
||||
8, 46, 23, 47, 52, 28, 18, 6, 6, 4, 42, 14, 10, 2,
|
||||
2, 8, 46, 23, 47, 52, 28, 18, 6, 6, 4, 42, 14, 10, 2,
|
||||
]);
|
||||
expect(validateRunnerCatalog()).toHaveLength(306);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(306);
|
||||
expect(validateRunnerCatalog()).toHaveLength(308);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(308);
|
||||
expect(
|
||||
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
|
||||
).toHaveLength(42);
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { apiResponseReadingTask } from "./api-response-reading.js";
|
||||
import { accountingTasks } from "./accounting-cases.js";
|
||||
import { continuationTasks } from "./continuation-cases.js";
|
||||
import { lifecycleLiveTasks, lifecycleLiveDefinitionDigest } from "./lifecycle-live-cases.js";
|
||||
@@ -911,6 +912,14 @@ const everydayProfiles = [
|
||||
].map(productionStoryProfile);
|
||||
|
||||
export const runnerSuites: readonly RunnerSuiteFixture[] = [
|
||||
{
|
||||
id: "api-response-reading", label: "Bounded API response reading", manualOnly: true,
|
||||
description: "Retrieve evidence beyond a saved API preview through authorized bounded text windows.",
|
||||
groups: [], environments: runnerEnvironments,
|
||||
profiles: runnerProfiles.filter(profile => profile.id === "runner-codex"),
|
||||
tasks: [apiResponseReadingTask], expectedMatrixSize: 2,
|
||||
definitionMetadata: { version: 1, grading: "hidden-evidence-exact-copy-and-api-tool-events", scheduling: "explicit-only" },
|
||||
},
|
||||
{
|
||||
id: "continuation-accounting", label: "Continuation accounting baseline", manualOnly: true,
|
||||
description: "Structured productive steps, bounded repair, restart and late gates; comments cannot buy more attempts.",
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { gradeApiResponsePaging, readResponseProof, responseEvidenceDescription } from "./api-response-reading.js";
|
||||
import { observeBrowserBootstrap } from "./browser-bootstrap-diagnostics.js";
|
||||
import { runAccountingFlow } from "./accounting-flow.js";
|
||||
import type { Issue } from "../../packages/shared/src/types/issue.js";
|
||||
@@ -534,6 +535,7 @@ for (const execution of executions) {
|
||||
const marker = execution.task.buildVisibleMarker(nonce);
|
||||
const title = execution.task.buildTitle(nonce);
|
||||
let prompt = execution.task.buildPrompt(nonce);
|
||||
let apiResponseSourceId: string | undefined;
|
||||
let lifecycleBlockerId: string | null = null;
|
||||
const credentials = credentialValues();
|
||||
const secrets = normalizedSecrets(Object.values(credentials));
|
||||
@@ -549,6 +551,7 @@ for (const execution of executions) {
|
||||
let selectedRuns: RunRecord[] = [];
|
||||
let runtimeLeases: EnvironmentLeaseRecord[] = [];
|
||||
let matcherResults: MatcherResult[] = [];
|
||||
let downloadedResponseProof: Awaited<ReturnType<typeof readResponseProof>> | undefined;
|
||||
let firstTaskEvidence: RunnerE2EResult["firstTask"];
|
||||
let turnTimings: NonNullable<RunnerE2EResult["turnTimings"]> | undefined;
|
||||
const turnSubmissionTimesMs: number[] = [];
|
||||
@@ -794,6 +797,15 @@ for (const execution of executions) {
|
||||
daytonaImage: process.env.PAPERCLIP_E2E_DAYTONA_IMAGE,
|
||||
});
|
||||
|
||||
if (execution.suite.id === "api-response-reading") {
|
||||
const source = await api.post<{ id: string }>(`/api/companies/${fixtures.company.id}/issues`, {
|
||||
title: `Synthetic diagnostic evidence ${nonce}`,
|
||||
description: responseEvidenceDescription(nonce), status: "backlog",
|
||||
});
|
||||
apiResponseSourceId = source.id;
|
||||
prompt = prompt.replaceAll("{{API_RESPONSE_SOURCE_ID}}", source.id);
|
||||
}
|
||||
|
||||
if (execution.suite.id === "lifecycle-baseline" && lifecycleLiveCase(execution.task.id)?.family === "blocker") {
|
||||
const prerequisite = await api.post<{ id: string }>(`/api/companies/${fixtures.company.id}/issues`, {
|
||||
title: `Supply dataset ${nonce}`,
|
||||
@@ -1885,6 +1897,12 @@ for (const execution of executions) {
|
||||
}
|
||||
}
|
||||
|
||||
if (execution.suite.id === "api-response-reading") {
|
||||
const paging = gradeApiResponsePaging(runEventsByRun.flatMap(captured => captured.events), apiResponseSourceId ?? "");
|
||||
await writeSanitizedJson(snapshotsDir, "api-response-pagination.json", paging, secrets);
|
||||
if (!paging.passed) invariantFailures.push(`Bounded response paging was not proven: ${paging.failure}`);
|
||||
}
|
||||
|
||||
const context = record(run.contextSnapshot);
|
||||
const environmentContext = record(context.paperclipEnvironment);
|
||||
const workspaceContext = record(context.paperclipWorkspace);
|
||||
@@ -1938,6 +1956,11 @@ for (const execution of executions) {
|
||||
]),
|
||||
),
|
||||
);
|
||||
if (execution.suite.id === "api-response-reading") {
|
||||
downloadedResponseProof = await readResponseProof(api, issue.id, run.id);
|
||||
fileObservations["api-response-proof.txt"] = downloadedResponseProof.content;
|
||||
await writeSanitizedJson(snapshotsDir, "downloaded-response-proof.json", downloadedResponseProof, secrets);
|
||||
}
|
||||
matcherResults = await Promise.all(
|
||||
taskMatchers.map((matcher) =>
|
||||
evaluateMatcher(matcher, {
|
||||
@@ -2398,7 +2421,13 @@ for (const execution of executions) {
|
||||
const visibleAgentReplies = page
|
||||
.getByTestId("task-chat-thread")
|
||||
.getByTestId("task-chat-agent-bubble");
|
||||
if (execution.task.flow === "warm_three_turn") {
|
||||
if (execution.suite.id === "api-response-reading") {
|
||||
// File delivery renders a card instead of an exact summary bubble.
|
||||
// Prove the visible link points to the same independently checked bytes.
|
||||
const proofLink = page.getByRole("link", { name: "Open api-response-proof.txt", exact: true }).first();
|
||||
await expect(proofLink).toBeVisible({ timeout: 30_000 });
|
||||
await expect(proofLink).toHaveAttribute("href", `/api/attachments/${downloadedResponseProof!.attachmentId}/content`);
|
||||
} else if (execution.task.flow === "warm_three_turn") {
|
||||
// Prove the persisted user-facing response is visible, independently
|
||||
// of the byte-for-byte workspace checks and lease continuity checks.
|
||||
expect(finalRunMessage.trim()).not.toBe("");
|
||||
|
||||
Reference in new issue
Block a user