diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 1d21aa87f81..3c82a93a2b7 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -432,3 +432,23 @@ cleanup authority; it does not prove that remote inference has stopped. Recovery revokes the previous boot identity with a conditional update. Its own claim also expires so another sweep can finish cleanup after a restart. Historical rows keep null ownership fields and follow the previous recovery path. + +## Large API response snapshots + +`assets.byte_size` uses PostgreSQL `bigint` so saved responses and byte ranges can +exceed 2 GiB. The API and Drizzle mapping continue to expose a JavaScript number; +response readers validate safe integer offsets. The type-widening migration +rewrites the asset metadata table and needs an exclusive table lock. File bytes +remain in local or object storage. + +### Runner API response reservations + +`runner_api_response_reservations` holds company-scoped API snapshot reservations. +Before a capture spills, the server locks company admission and counts stored +`runner-api` assets plus unattached reservations against a 20 GiB default quota. +A committed asset replaces its reservation in that total. The asset foreign key +cascades on deletion, while deleting a run sets `run_id` to null so an orphan +reservation cannot silently disappear. Failed cleanup or an ambiguous storage +write requires operator reconciliation before an unattached reservation is +removed. The table stores no response bodies. See `doc/runner-api-tools.md` for +limits and the operator override. diff --git a/doc/SPEC-implementation.md b/doc/SPEC-implementation.md index cb18da50f92..56fbb5b7627 100644 --- a/doc/SPEC-implementation.md +++ b/doc/SPEC-implementation.md @@ -414,7 +414,7 @@ Operational policy: - `provider` enum/text (`local_disk | s3`) - `object_key` text not null - `content_type` text not null - - `byte_size` int not null + - `byte_size` bigint not null (safe integer byte count in the API) - `sha256` text not null - `original_filename` text null - `created_by_agent_id` uuid fk null diff --git a/doc/evals.md b/doc/evals.md index 8b0c49baf29..f343f07c01d 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -262,3 +262,7 @@ named existing controls on legacy and native Codex. Discover it with results and follow-up coverage are recorded in that suite's guide. Continuation accounting has an explicit-only eight-cell Product E2E [baseline suite](../tests/runner-e2e/CONTINUATION-ACCOUNTING.md), complementing the deterministic lifecycle inventory. + +The explicit-only Product E2E `api-response-reading` suite verifies retrieval of +large saved API responses on local and Daytona native Codex runs. See the +[Runner E2E guide](../tests/runner-e2e/README.md#bounded-api-response-reading). diff --git a/doc/runner-api-tools.md b/doc/runner-api-tools.md index f095e14783c..62451ba6b2c 100644 --- a/doc/runner-api-tools.md +++ b/doc/runner-api-tools.md @@ -51,14 +51,105 @@ exactly one authorized `artifactId` or task-workspace `path`, and an optional multipart `field`. No arbitrary URL, headers, authentication, or remote file URL can be supplied. Routes still validate payloads and enforce permissions. -Requests have a 30-second HTTP timeout, 16 KiB URL limit and 10 MiB payload/response -transfer limit. Responses above 24 KiB and binary responses become company-owned -assets with retrievable references; text previews are limited to 2,000 bytes. +Requests have a 16 KiB URL limit and 10 MiB request/upload limit. **New response +captures are limited to 1 GiB of decoded bytes.** This is separate from the 24 KiB +inline/page limit. The receiver rejects an oversized Content-Length before +reading and counts actual bytes before writing, including chunked or compressed +responses. Oversized responses return `api_response_too_large`; narrow the query +or use the endpoint's own pagination. A mutation may already have committed, so +inspect its state rather than retrying it to obtain a smaller response. + +Connection setup and stalled response reads time out after 30 seconds. An active +capture has a 10-minute total download deadline. Responses above 24 KiB stream +into a private temporary file, then into a company-owned asset. Memory stays +bounded by the inline prefix and stream buffers. Binary responses also become +assets; text previews are limited to 2,000 bytes. + +Large captures reserve 1 GiB against a **durable 4 GiB per-run capture budget** +before creating a file. A completed capture settles to its actual byte count; +failed/interrupted captures retain the full reservation to bound retry loops. +A process restart does not reset that budget. Small inline responses and reads +of existing assets need no reservation. Concurrent large captures are limited +to **two per company and four per server process**, including the storage upload +and temporary-file cleanup. Budget exhaustion returns `api_response_capture_limit`; +concurrency exhaustion returns `api_response_capture_busy` without queuing more +large transfers. A **20 GiB company-wide quota** counts all stored `runner-api` snapshots plus +unattached reservations. Admission uses a company database lock, so runs and +server processes share the same quota. Existing snapshots from before this +change count too. Attaching an asset converts its reservation to actual stored +bytes; deleting the asset frees that capacity. Small binary snapshots also need +storage admission. Ordinary inline text/JSON and existing-asset pages do not. + +Operators can set `PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES` to a positive +safe integer of at least 1 GiB. Invalid values fall back to 20 GiB. No zero or +unlimited setting is accepted. This quota covers API snapshots, not all company +attachments. Storage capacity and backend limits still apply. + +Handled pre-storage failures release the company reservation after temporary +file cleanup, but retain the run's charge against retry loops. After a metadata +transaction fails, a locking read must prove the asset was not committed before +the uploaded object is removed. Confirmed cleanup refunds the company quota. A crash, failed +cleanup, or ambiguous storage write keeps an unattached reservation. Operators +must reconcile possible orphan files/objects before deleting that reservation +from `runner_api_response_reservations`; no automatic expiry silently refunds +possibly occupied space. Linked rows are removed with their asset. Deleting a +run does not release its unattached company reservations. + +Temporary files are removed on success or failure. Long captures revalidate the +active run at least every MiB or at the next chunk after one second, and again +before returning the snapshot. Stopping the run stops its download. + +To inspect saved text without creating another artifact, call its authorized +content operation with `responseText`: + +```json +{"operationId":"GET /api/assets/{assetId}/content","pathParams":{"assetId":"RETURNED_ARTIFACT_ID"},"responseText":{"offsetBytes":0,"limitBytes":8192}} +``` + +The result contains `data` as text (including JSON), plus `responseText` with +`offsetBytes`, `nextOffsetBytes`, and `totalBytes`. Continue at `nextOffsetBytes` +until it is null. Windows end at UTF-8 boundaries. The limit defaults to 24 KiB +and accepts 4–24,576 bytes. The server rejects binary content, invalid UTF-8, +and offsets inside a code point or beyond the response. This option only works +with GET; never repeat a mutation to retrieve another part of its response. +Read the saved artifact for a stable snapshot instead of paging a changing live +response. A text-window call against a live response above 24 KiB also returns +that snapshot's artifact reference; continue on its content operation. A saved +asset page never creates another asset. An unpaged asset read also fetches only +a bounded preview and returns the existing reference instead of copying the file. Offsets and total sizes use safe integer +byte counts, including values above 2 GiB. Existing assets larger than the 1 GiB +capture limit remain readable because each request transfers only a bounded range. +Request/upload limits and all route +authorization remain. +Saved asset pages use authenticated HTTP byte ranges. The storage provider reads +only the requested window, with at most two extra bytes for UTF-8/EOF handling. +The client validates `Content-Range`, the total size, and the received byte count; +it rejects unsupported or inconsistent ranges instead of downloading the whole +asset for every page. Other GET routes are fetched once in full to create the +snapshot, so use the returned asset for subsequent pages. S3 uses streamed +multipart uploads for large snapshots. Asset sizes are stored as PostgreSQL +`bigint`, preserving the existing numeric API shape. + +### Media files and upload limits + +The 1 GiB limit covers new snapshots returned by `call_api`, such as large JSON +exports or binary API downloads. It does not raise attachment upload limits or +limit files an agent creates and edits inside its workspace. Saved asset downloads +stream from storage and support byte ranges, including video seeking. + +`PAPERCLIP_ATTACHMENT_MAX_BYTES` separately defaults to 10 MiB for uploads and +native file handoffs. `call_api` uploads also have their own 10 MiB limit. Several +upload and handoff paths buffer complete files in memory; raising those defaults +to GiB sizes requires streaming ingestion and corresponding admission/budget +controls first. For a future video attachment workflow, 2 GiB per streamed file +is a reasonable default, with an operator override and storage quotas. Do not +claim that this response-paging change enables GiB attachment uploads. + Tool responses identify the HTTP route with `apiOperationId`. The native protocol reserves `operationId` and `callId` for semantic tool-call identity; API metadata must not masquerade as that envelope. Saved mutation receipts are normalized at the tool boundary as well, without repeating their HTTP request. -All redirects are refused. Oversized or interrupted mutation responses have an +All redirects are refused. Interrupted mutation responses have an unknown outcome, requiring inspection before another mutation. Mutation responses with HTTP 5xx, HTTP 408, redirects, or malformed JSON also retain an unknown outcome. A server may have committed the write before it diff --git a/packages/db/src/migrations/0285_heavy_oracle.sql b/packages/db/src/migrations/0285_heavy_oracle.sql new file mode 100644 index 00000000000..bf064548742 --- /dev/null +++ b/packages/db/src/migrations/0285_heavy_oracle.sql @@ -0,0 +1 @@ +ALTER TABLE "assets" ALTER COLUMN "byte_size" SET DATA TYPE bigint; \ No newline at end of file diff --git a/packages/db/src/migrations/0286_complete_alice.sql b/packages/db/src/migrations/0286_complete_alice.sql new file mode 100644 index 00000000000..695640919a1 --- /dev/null +++ b/packages/db/src/migrations/0286_complete_alice.sql @@ -0,0 +1,13 @@ +CREATE TABLE IF NOT EXISTS "runner_api_response_reservations" ( + "id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL, + "company_id" uuid NOT NULL, + "run_id" uuid, + "asset_id" uuid, + "reserved_bytes" bigint NOT NULL, + "created_at" timestamp with time zone DEFAULT now() NOT NULL +); +--> statement-breakpoint +DO $$ BEGIN ALTER TABLE "runner_api_response_reservations" ADD CONSTRAINT "runner_api_response_reservations_company_id_companies_id_fk" FOREIGN KEY ("company_id") REFERENCES "public"."companies"("id") ON DELETE cascade ON UPDATE no action; EXCEPTION WHEN duplicate_object THEN NULL; END $$;--> statement-breakpoint +DO $$ BEGIN ALTER TABLE "runner_api_response_reservations" ADD CONSTRAINT "runner_api_response_reservations_run_id_heartbeat_runs_id_fk" FOREIGN KEY ("run_id") REFERENCES "public"."heartbeat_runs"("id") ON DELETE set null ON UPDATE no action; EXCEPTION WHEN duplicate_object THEN NULL; END $$;--> statement-breakpoint +DO $$ BEGIN ALTER TABLE "runner_api_response_reservations" ADD CONSTRAINT "runner_api_response_reservations_asset_id_assets_id_fk" FOREIGN KEY ("asset_id") REFERENCES "public"."assets"("id") ON DELETE cascade ON UPDATE no action; EXCEPTION WHEN duplicate_object THEN NULL; END $$;--> statement-breakpoint +CREATE INDEX IF NOT EXISTS "runner_api_response_reservations_company_idx" ON "runner_api_response_reservations" USING btree ("company_id"); diff --git a/packages/db/src/migrations/meta/0281_snapshot.json b/packages/db/src/migrations/meta/0285_snapshot.json similarity index 98% rename from packages/db/src/migrations/meta/0281_snapshot.json rename to packages/db/src/migrations/meta/0285_snapshot.json index 92941576375..74f739c25ff 100644 --- a/packages/db/src/migrations/meta/0281_snapshot.json +++ b/packages/db/src/migrations/meta/0285_snapshot.json @@ -1,6 +1,6 @@ { - "id": "8d202a5a-5d6b-44d9-9336-3bb39a8b99bc", - "prevId": "861a2815-1418-4c71-a77b-d0e2e9566413", + "id": "c27cbbec-6948-4b2b-8db8-36d0bf3830fd", + "prevId": "7db2ceec-ed62-4968-90c9-4c79206cfe34", "version": "7", "dialect": "postgresql", "tables": { @@ -2917,7 +2917,7 @@ }, "byte_size": { "name": "byte_size", - "type": "integer", + "type": "bigint", "primaryKey": false, "notNull": true }, @@ -3302,6 +3302,13 @@ "primaryKey": false, "notNull": false }, + "keyboard_shortcuts": { + "name": "keyboard_shortcuts", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, "created_at": { "name": "created_at", "type": "timestamp with time zone", @@ -5701,6 +5708,12 @@ "notNull": true, "default": false }, + "communication_guidance": { + "name": "communication_guidance", + "type": "text", + "primaryKey": false, + "notNull": false + }, "state": { "name": "state", "type": "text", @@ -6611,6 +6624,13 @@ "primaryKey": false, "notNull": false }, + "communication_instructions": { + "name": "communication_instructions", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "''" + }, "allow_direct_messages": { "name": "allow_direct_messages", "type": "boolean", @@ -8096,6 +8116,519 @@ }, "isRLSEnabled": false }, + "public.chat_github_configurations": { + "name": "chat_github_configurations", + "schema": "", + "columns": { + "endpoint_id": { + "name": "endpoint_id", + "type": "uuid", + "primaryKey": true, + "notNull": true + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "revision": { + "name": "revision", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 1 + }, + "configuration": { + "name": "configuration", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "updated_by_user_id": { + "name": "updated_by_user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": {}, + "foreignKeys": { + "chat_github_configurations_company_id_endpoint_id_chat_endpoints_company_id_id_fk": { + "name": "chat_github_configurations_company_id_endpoint_id_chat_endpoints_company_id_id_fk", + "tableFrom": "chat_github_configurations", + "tableTo": "chat_endpoints", + "columnsFrom": [ + "company_id", + "endpoint_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "chat_github_config_revision_check": { + "name": "chat_github_config_revision_check", + "value": "\"chat_github_configurations\".\"revision\" > 0" + } + }, + "isRLSEnabled": false + }, + "public.chat_github_registrations": { + "name": "chat_github_registrations", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "endpoint_id": { + "name": "endpoint_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "state_hash": { + "name": "state_hash", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "trusted_origin": { + "name": "trusted_origin", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "consumed_at": { + "name": "consumed_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "chat_github_registration_state_uq": { + "name": "chat_github_registration_state_uq", + "columns": [ + { + "expression": "state_hash", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "chat_github_registration_endpoint_idx": { + "name": "chat_github_registration_endpoint_idx", + "columns": [ + { + "expression": "endpoint_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "chat_github_registrations_company_id_endpoint_id_chat_endpoints_company_id_id_fk": { + "name": "chat_github_registrations_company_id_endpoint_id_chat_endpoints_company_id_id_fk", + "tableFrom": "chat_github_registrations", + "tableTo": "chat_endpoints", + "columnsFrom": [ + "company_id", + "endpoint_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "chat_github_registration_status_check": { + "name": "chat_github_registration_status_check", + "value": "\"chat_github_registrations\".\"status\" in ('pending', 'exchanging', 'completed', 'failed')" + } + }, + "isRLSEnabled": false + }, + "public.chat_github_reviews": { + "name": "chat_github_reviews", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "endpoint_id": { + "name": "endpoint_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "issue_id": { + "name": "issue_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "run_id": { + "name": "run_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "repository_id": { + "name": "repository_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "repository": { + "name": "repository", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "pull_number": { + "name": "pull_number", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "head_sha": { + "name": "head_sha", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "delivery_id": { + "name": "delivery_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "configuration_revision": { + "name": "configuration_revision", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "policy_snapshot": { + "name": "policy_snapshot", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "event": { + "name": "event", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "state": { + "name": "state", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'queued'" + }, + "assessment": { + "name": "assessment", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "conclusion": { + "name": "conclusion", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "check_id": { + "name": "check_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "check_url": { + "name": "check_url", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "summary_id": { + "name": "summary_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "summary_url": { + "name": "summary_url", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "publication_receipts": { + "name": "publication_receipts", + "type": "jsonb", + "primaryKey": false, + "notNull": true, + "default": "'{}'::jsonb" + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "chat_github_reviews_delivery_uq": { + "name": "chat_github_reviews_delivery_uq", + "columns": [ + { + "expression": "endpoint_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "delivery_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "chat_github_reviews_pull_idx": { + "name": "chat_github_reviews_pull_idx", + "columns": [ + { + "expression": "endpoint_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "repository_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "pull_number", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "chat_github_reviews_task_idx": { + "name": "chat_github_reviews_task_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "issue_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "chat_github_reviews_run_id_heartbeat_runs_id_fk": { + "name": "chat_github_reviews_run_id_heartbeat_runs_id_fk", + "tableFrom": "chat_github_reviews", + "tableTo": "heartbeat_runs", + "columnsFrom": [ + "run_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "chat_github_reviews_company_id_endpoint_id_chat_endpoints_company_id_id_fk": { + "name": "chat_github_reviews_company_id_endpoint_id_chat_endpoints_company_id_id_fk", + "tableFrom": "chat_github_reviews", + "tableTo": "chat_endpoints", + "columnsFrom": [ + "company_id", + "endpoint_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "chat_github_reviews_company_id_issue_id_issues_company_id_id_fk": { + "name": "chat_github_reviews_company_id_issue_id_issues_company_id_id_fk", + "tableFrom": "chat_github_reviews", + "tableTo": "issues", + "columnsFrom": [ + "company_id", + "issue_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "restrict", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "chat_github_reviews_company_id_uq": { + "name": "chat_github_reviews_company_id_uq", + "nullsNotDistinct": false, + "columns": [ + "company_id", + "id" + ] + } + }, + "policies": {}, + "checkConstraints": { + "chat_github_review_pull_number_check": { + "name": "chat_github_review_pull_number_check", + "value": "\"chat_github_reviews\".\"pull_number\" > 0" + }, + "chat_github_review_state_check": { + "name": "chat_github_review_state_check", + "value": "\"chat_github_reviews\".\"state\" in ('queued', 'running', 'completed', 'incomplete', 'error', 'superseded', 'manual_required')" + } + }, + "isRLSEnabled": false + }, "public.chat_teams_file_transfers": { "name": "chat_teams_file_transfers", "schema": "", diff --git a/packages/db/src/migrations/meta/0280_snapshot.json b/packages/db/src/migrations/meta/0286_snapshot.json similarity index 98% rename from packages/db/src/migrations/meta/0280_snapshot.json rename to packages/db/src/migrations/meta/0286_snapshot.json index ea3ae5e50dd..0f02bbe75dc 100644 --- a/packages/db/src/migrations/meta/0280_snapshot.json +++ b/packages/db/src/migrations/meta/0286_snapshot.json @@ -1,6 +1,6 @@ { - "id": "861a2815-1418-4c71-a77b-d0e2e9566413", - "prevId": "4770ad6d-2cdc-45c4-af89-c300fd134d91", + "id": "33120bf3-9cac-4909-ae94-d514ddc1e9e2", + "prevId": "c27cbbec-6948-4b2b-8db8-36d0bf3830fd", "version": "7", "dialect": "postgresql", "tables": { @@ -2917,7 +2917,7 @@ }, "byte_size": { "name": "byte_size", - "type": "integer", + "type": "bigint", "primaryKey": false, "notNull": true }, @@ -3302,6 +3302,13 @@ "primaryKey": false, "notNull": false }, + "keyboard_shortcuts": { + "name": "keyboard_shortcuts", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, "created_at": { "name": "created_at", "type": "timestamp with time zone", @@ -5701,6 +5708,12 @@ "notNull": true, "default": false }, + "communication_guidance": { + "name": "communication_guidance", + "type": "text", + "primaryKey": false, + "notNull": false + }, "state": { "name": "state", "type": "text", @@ -6611,6 +6624,13 @@ "primaryKey": false, "notNull": false }, + "communication_instructions": { + "name": "communication_instructions", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "''" + }, "allow_direct_messages": { "name": "allow_direct_messages", "type": "boolean", @@ -8096,6 +8116,519 @@ }, "isRLSEnabled": false }, + "public.chat_github_configurations": { + "name": "chat_github_configurations", + "schema": "", + "columns": { + "endpoint_id": { + "name": "endpoint_id", + "type": "uuid", + "primaryKey": true, + "notNull": true + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "revision": { + "name": "revision", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 1 + }, + "configuration": { + "name": "configuration", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "updated_by_user_id": { + "name": "updated_by_user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": {}, + "foreignKeys": { + "chat_github_configurations_company_id_endpoint_id_chat_endpoints_company_id_id_fk": { + "name": "chat_github_configurations_company_id_endpoint_id_chat_endpoints_company_id_id_fk", + "tableFrom": "chat_github_configurations", + "tableTo": "chat_endpoints", + "columnsFrom": [ + "company_id", + "endpoint_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "chat_github_config_revision_check": { + "name": "chat_github_config_revision_check", + "value": "\"chat_github_configurations\".\"revision\" > 0" + } + }, + "isRLSEnabled": false + }, + "public.chat_github_registrations": { + "name": "chat_github_registrations", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "endpoint_id": { + "name": "endpoint_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "state_hash": { + "name": "state_hash", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "trusted_origin": { + "name": "trusted_origin", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "consumed_at": { + "name": "consumed_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "chat_github_registration_state_uq": { + "name": "chat_github_registration_state_uq", + "columns": [ + { + "expression": "state_hash", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "chat_github_registration_endpoint_idx": { + "name": "chat_github_registration_endpoint_idx", + "columns": [ + { + "expression": "endpoint_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "chat_github_registrations_company_id_endpoint_id_chat_endpoints_company_id_id_fk": { + "name": "chat_github_registrations_company_id_endpoint_id_chat_endpoints_company_id_id_fk", + "tableFrom": "chat_github_registrations", + "tableTo": "chat_endpoints", + "columnsFrom": [ + "company_id", + "endpoint_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "chat_github_registration_status_check": { + "name": "chat_github_registration_status_check", + "value": "\"chat_github_registrations\".\"status\" in ('pending', 'exchanging', 'completed', 'failed')" + } + }, + "isRLSEnabled": false + }, + "public.chat_github_reviews": { + "name": "chat_github_reviews", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "endpoint_id": { + "name": "endpoint_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "issue_id": { + "name": "issue_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "run_id": { + "name": "run_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "repository_id": { + "name": "repository_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "repository": { + "name": "repository", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "pull_number": { + "name": "pull_number", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "head_sha": { + "name": "head_sha", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "delivery_id": { + "name": "delivery_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "configuration_revision": { + "name": "configuration_revision", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "policy_snapshot": { + "name": "policy_snapshot", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "event": { + "name": "event", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "state": { + "name": "state", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'queued'" + }, + "assessment": { + "name": "assessment", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "conclusion": { + "name": "conclusion", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "check_id": { + "name": "check_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "check_url": { + "name": "check_url", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "summary_id": { + "name": "summary_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "summary_url": { + "name": "summary_url", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "publication_receipts": { + "name": "publication_receipts", + "type": "jsonb", + "primaryKey": false, + "notNull": true, + "default": "'{}'::jsonb" + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "chat_github_reviews_delivery_uq": { + "name": "chat_github_reviews_delivery_uq", + "columns": [ + { + "expression": "endpoint_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "delivery_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "chat_github_reviews_pull_idx": { + "name": "chat_github_reviews_pull_idx", + "columns": [ + { + "expression": "endpoint_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "repository_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "pull_number", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "chat_github_reviews_task_idx": { + "name": "chat_github_reviews_task_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "issue_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "chat_github_reviews_run_id_heartbeat_runs_id_fk": { + "name": "chat_github_reviews_run_id_heartbeat_runs_id_fk", + "tableFrom": "chat_github_reviews", + "tableTo": "heartbeat_runs", + "columnsFrom": [ + "run_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "chat_github_reviews_company_id_endpoint_id_chat_endpoints_company_id_id_fk": { + "name": "chat_github_reviews_company_id_endpoint_id_chat_endpoints_company_id_id_fk", + "tableFrom": "chat_github_reviews", + "tableTo": "chat_endpoints", + "columnsFrom": [ + "company_id", + "endpoint_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "chat_github_reviews_company_id_issue_id_issues_company_id_id_fk": { + "name": "chat_github_reviews_company_id_issue_id_issues_company_id_id_fk", + "tableFrom": "chat_github_reviews", + "tableTo": "issues", + "columnsFrom": [ + "company_id", + "issue_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "restrict", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "chat_github_reviews_company_id_uq": { + "name": "chat_github_reviews_company_id_uq", + "nullsNotDistinct": false, + "columns": [ + "company_id", + "id" + ] + } + }, + "policies": {}, + "checkConstraints": { + "chat_github_review_pull_number_check": { + "name": "chat_github_review_pull_number_check", + "value": "\"chat_github_reviews\".\"pull_number\" > 0" + }, + "chat_github_review_state_check": { + "name": "chat_github_review_state_check", + "value": "\"chat_github_reviews\".\"state\" in ('queued', 'running', 'completed', 'incomplete', 'error', 'superseded', 'manual_required')" + } + }, + "isRLSEnabled": false + }, "public.chat_teams_file_transfers": { "name": "chat_teams_file_transfers", "schema": "", @@ -37416,6 +37949,26 @@ "notNull": true, "default": true }, + "setup_pending": { + "name": "setup_pending", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "archived": { + "name": "archived", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "last_webhook_delivery": { + "name": "last_webhook_delivery", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, "cron_expression": { "name": "cron_expression", "type": "text", @@ -37677,6 +38230,100 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.routine_webhook_test_receipts": { + "name": "routine_webhook_test_receipts", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "trigger_id": { + "name": "trigger_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "delivery_key_hash": { + "name": "delivery_key_hash", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "received_at": { + "name": "received_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "routine_webhook_test_receipts_delivery_uq": { + "name": "routine_webhook_test_receipts_delivery_uq", + "columns": [ + { + "expression": "trigger_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "delivery_key_hash", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "routine_webhook_test_receipts_company_id_companies_id_fk": { + "name": "routine_webhook_test_receipts_company_id_companies_id_fk", + "tableFrom": "routine_webhook_test_receipts", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "routine_webhook_test_receipts_trigger_id_routine_triggers_id_fk": { + "name": "routine_webhook_test_receipts_trigger_id_routine_triggers_id_fk", + "tableFrom": "routine_webhook_test_receipts", + "tableTo": "routine_triggers", + "columnsFrom": [ + "trigger_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.routines": { "name": "routines", "schema": "", @@ -38292,6 +38939,113 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.runner_api_response_reservations": { + "name": "runner_api_response_reservations", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "run_id": { + "name": "run_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "asset_id": { + "name": "asset_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "reserved_bytes": { + "name": "reserved_bytes", + "type": "bigint", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "runner_api_response_reservations_company_idx": { + "name": "runner_api_response_reservations_company_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "runner_api_response_reservations_company_id_companies_id_fk": { + "name": "runner_api_response_reservations_company_id_companies_id_fk", + "tableFrom": "runner_api_response_reservations", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "runner_api_response_reservations_run_id_heartbeat_runs_id_fk": { + "name": "runner_api_response_reservations_run_id_heartbeat_runs_id_fk", + "tableFrom": "runner_api_response_reservations", + "tableTo": "heartbeat_runs", + "columnsFrom": [ + "run_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "runner_api_response_reservations_asset_id_assets_id_fk": { + "name": "runner_api_response_reservations_asset_id_assets_id_fk", + "tableFrom": "runner_api_response_reservations", + "tableTo": "assets", + "columnsFrom": [ + "asset_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.secret_access_events": { "name": "secret_access_events", "schema": "", diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index a46ca4cfb44..264cefe4465 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -1982,6 +1982,20 @@ "when": 1790439010225, "tag": "0284_petite_genesis", "breakpoints": true + }, + { + "idx": 285, + "version": "7", + "when": 1790603883247, + "tag": "0285_heavy_oracle", + "breakpoints": true + }, + { + "idx": 286, + "version": "7", + "when": 1790607525758, + "tag": "0286_complete_alice", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/assets.ts b/packages/db/src/schema/assets.ts index 648473ddecb..12074cd58e7 100644 --- a/packages/db/src/schema/assets.ts +++ b/packages/db/src/schema/assets.ts @@ -1,4 +1,4 @@ -import { pgTable, uuid, text, integer, timestamp, index, uniqueIndex } from "drizzle-orm/pg-core"; +import { pgTable, uuid, text, bigint, timestamp, index, uniqueIndex } from "drizzle-orm/pg-core"; import { companies } from "./companies.js"; import { agents } from "./agents.js"; @@ -10,7 +10,7 @@ export const assets = pgTable( provider: text("provider").notNull(), objectKey: text("object_key").notNull(), contentType: text("content_type").notNull(), - byteSize: integer("byte_size").notNull(), + byteSize: bigint("byte_size", { mode: "number" }).notNull(), sha256: text("sha256").notNull(), originalFilename: text("original_filename"), createdByAgentId: uuid("created_by_agent_id").references(() => agents.id), diff --git a/packages/db/src/schema/index.ts b/packages/db/src/schema/index.ts index 81ea3bc33bb..db17f2aa695 100644 --- a/packages/db/src/schema/index.ts +++ b/packages/db/src/schema/index.ts @@ -115,6 +115,7 @@ export { export { feedbackExports } from "./feedback_exports.js"; export { issueReadStates } from "./issue_read_states.js"; export { assets } from "./assets.js"; +export { runnerApiResponseReservations } from "./runner_api_response_reservations.js"; export { issueAttachments } from "./issue_attachments.js"; export { documents } from "./documents.js"; export { documentRevisions } from "./document_revisions.js"; diff --git a/packages/db/src/schema/runner_api_response_reservations.ts b/packages/db/src/schema/runner_api_response_reservations.ts new file mode 100644 index 00000000000..32519a7a866 --- /dev/null +++ b/packages/db/src/schema/runner_api_response_reservations.ts @@ -0,0 +1,13 @@ +import { pgTable, uuid, bigint, timestamp, index } from "drizzle-orm/pg-core"; +import { companies } from "./companies.js"; +import { heartbeatRuns } from "./heartbeat_runs.js"; +import { assets } from "./assets.js"; + +export const runnerApiResponseReservations = pgTable("runner_api_response_reservations", { + id: uuid("id").primaryKey().defaultRandom(), + companyId: uuid("company_id").notNull().references(() => companies.id, { onDelete: "cascade" }), + runId: uuid("run_id").references(() => heartbeatRuns.id, { onDelete: "set null" }), + assetId: uuid("asset_id").references(() => assets.id, { onDelete: "cascade" }), + reservedBytes: bigint("reserved_bytes", { mode: "number" }).notNull(), + createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(), +}, table => ({ companyIdx: index("runner_api_response_reservations_company_idx").on(table.companyId) })); diff --git a/packages/paperclip-runner/generated/capability/semantic-tool-contracts.json b/packages/paperclip-runner/generated/capability/semantic-tool-contracts.json index 1558a1ef65a..9ca2c4ba8d1 100644 --- a/packages/paperclip-runner/generated/capability/semantic-tool-contracts.json +++ b/packages/paperclip-runner/generated/capability/semantic-tool-contracts.json @@ -1 +1 @@ -[{"annotations":{"exposure":"always","operationId":"get_task_context","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read the active task and actor, including the exact approved Markdown revision when this issue has an accepted plan.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"get_task_context","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"get_task_history","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read bounded comments on the active task.","inputSchema":{"additionalProperties":false,"properties":{"limit":{"default":50,"maximum":200,"minimum":1,"type":"integer"}},"required":[],"type":"object"},"name":"get_task_history","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"list_documents","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"List revisioned documents on the active task.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_documents","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"read_document","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read the current revision of one active-task document.","inputSchema":{"additionalProperties":false,"properties":{"key":{"description":"Stable issue-document key.","maxLength":120,"minLength":1,"type":"string"}},"required":["key"],"type":"object"},"name":"read_document","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"list_document_revisions","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read bounded revision history for one active-task document.","inputSchema":{"additionalProperties":false,"properties":{"key":{"description":"Stable issue-document key.","maxLength":120,"minLength":1,"type":"string"},"limit":{"default":50,"maximum":200,"minimum":1,"type":"integer"}},"required":["key"],"type":"object"},"name":"list_document_revisions","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"report_progress","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Append a durable progress comment to the active task.","inputSchema":{"additionalProperties":false,"properties":{"body":{"description":"Multiline progress update.","maxLength":20000,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","body"],"type":"object"},"name":"report_progress","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"answer_status_question","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Append the answer to a status-only wake without changing task disposition.","inputSchema":{"additionalProperties":false,"properties":{"body":{"description":"Concise status answer.","maxLength":20000,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","body"],"type":"object"},"name":"answer_status_question","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"write_document","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create or update an active-task document with optimistic revision safety.","inputSchema":{"additionalProperties":false,"properties":{"baseRevisionId":{"description":"Current revision id, or null when creating.","maxLength":20000,"type":["string","null"]},"body":{"description":"Markdown document body.","maxLength":200000,"minLength":1,"type":"string"},"changeSummary":{"description":"Optional revision summary.","maxLength":20000,"type":["string","null"]},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"key":{"description":"Stable issue-document key.","maxLength":120,"minLength":1,"type":"string"},"title":{"description":"Document title.","maxLength":300,"minLength":1,"type":"string"}},"required":["idempotencyKey","key","title","body","baseRevisionId"],"type":"object"},"name":"write_document","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"request_human_input","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create a durable human question or approval card on the current Paperclip task bound to this run; Paperclip renders it and authenticates the response. Use questions with continuationPolicy='wake_assignee' when an answer is needed, including otherwise tool-free chat turns. Supply a stable idempotencyKey and reuse it on retries. For one question at a time, ask only the next unanswered question and wait for its real answer. Never infer answers, answer your own card, or treat clarification as approval. Preserve existing review gates. Call this tool before claiming a question was asked; if creation fails, report the failure. Do not fabricate answer links or Markdown buttons, post duplicate cards, or substitute call_api. Use payload.questions for choices and payload.questionSet for text fields; see the payload schema for formats.","inputSchema":{"additionalProperties":false,"properties":{"continuationPolicy":{"enum":["none","wake_assignee","wake_assignee_on_accept"]},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"interactionKind":{"enum":["confirmation","checkbox","questions","suggest_tasks","item_verdicts"]},"payload":{"additionalProperties":true,"description":"Kind-specific interaction data. For interactionKind='questions', use version:1 and questions:[{id,prompt,selectionMode:'single'|'multi',required?,options:[{id,label,description?,freeText?}]}]. Choice questions need at least two distinct meaningful options. For an open-ended text answer, ALSO include questionSet:{schema:'paperclip.question_set.v1',questions:[{id,prompt,answerMode:'text',required?}]} with no options or customAnswer in its text questions. Keep matching IDs/prompts in both arrays; the required compatibility questions entry uses selectionMode:'single' and options:[{id:'describe',label:'Your answer',freeText:true}]. Without questionSet this incorrectly renders as a one-option choice. Never use a lone Other or describe option as the presentation. Option keys are id/label, not value. For confirmation, payload may be {}. Keep IDs stable across retries.","type":"object"},"prompt":{"description":"Question or decision prompt.","maxLength":10000,"minLength":1,"type":"string"},"targetRevisionId":{"description":"Optional bound document revision.","maxLength":20000,"type":["string","null"]},"title":{"description":"Interaction card title.","maxLength":300,"minLength":1,"type":"string"}},"required":["idempotencyKey","interactionKind","title","prompt","continuationPolicy"],"type":"object"},"name":"request_human_input","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"register_deliverable","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Register attachment metadata and its artifact work product without credentials or bytes in the tool result.","inputSchema":{"additionalProperties":false,"properties":{"byteSize":{"maximum":100000000,"minimum":0,"type":"integer"},"contentRef":{"description":"Opaque package-local content reference.","maxLength":2000,"minLength":1,"type":"string"},"contentType":{"description":"Media type.","maxLength":200,"minLength":1,"type":"string"},"filename":{"description":"Display filename.","maxLength":500,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"sha256":{"pattern":"^[a-fA-F0-9]{64}$","type":"string"},"title":{"description":"Work-product title.","maxLength":500,"minLength":1,"type":"string"}},"required":["idempotencyKey","filename","contentType","byteSize","sha256","contentRef","title"],"type":"object"},"name":"register_deliverable","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"finish_task","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Finish the active task with a durable summary.","inputSchema":{"additionalProperties":false,"properties":{"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"summary":{"description":"Completion summary.","maxLength":20000,"minLength":1,"type":"string"}},"required":["idempotencyKey","summary"],"type":"object"},"name":"finish_task","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"block_task","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Block the active task with a durable reason and optional first-class dependencies.","inputSchema":{"additionalProperties":false,"properties":{"blockedByTaskIds":{"description":"Internal task ids that block this task.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"reason":{"description":"Block reason.","maxLength":20000,"minLength":1,"type":"string"}},"required":["idempotencyKey","reason"],"type":"object"},"name":"block_task","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"request_review","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Move the active task to review with a durable summary.","inputSchema":{"additionalProperties":false,"properties":{"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"summary":{"description":"Review handoff summary.","maxLength":20000,"minLength":1,"type":"string"}},"required":["idempotencyKey","summary"],"type":"object"},"name":"request_review","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"list_agents","requiredClaims":["discovery:agents:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"List redacted actor profiles.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_agents","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"get_agent","requiredClaims":["discovery:agents:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read one redacted actor profile.","inputSchema":{"additionalProperties":false,"properties":{"actorId":{"description":"Actor id.","maxLength":200,"minLength":1,"type":"string"}},"required":["actorId"],"type":"object"},"name":"get_agent","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"hire_agent","requiredClaims":["delegation:agents:create"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create a persistent native Runner teammate with an identity and persona. The teammate reports to the caller and inherits the caller's native runtime; provider, adapter, environment, and credential settings are selected by Paperclip and are never caller-supplied here. Reuse a suitable teammate from list_agents when possible.","inputSchema":{"additionalProperties":false,"properties":{"capabilities":{"maxLength":2000,"type":["string","null"]},"instructions":{"maxLength":20000,"type":["string","null"]},"name":{"maxLength":200,"minLength":1,"type":"string"},"role":{"enum":["ceo","cto","cmo","cfo","security","engineer","designer","pm","qa","devops","researcher","general"],"type":"string"},"title":{"maxLength":300,"type":["string","null"]}},"required":["name"],"type":"object"},"name":"hire_agent","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"search_tasks","requiredClaims":["discovery:tasks:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Search tasks by text and status within the run company.","inputSchema":{"additionalProperties":false,"properties":{"limit":{"default":50,"maximum":200,"minimum":1,"type":"integer"},"query":{"maxLength":500,"type":"string"},"statuses":{"items":{"enum":["backlog","todo","in_progress","in_review","done","blocked","cancelled"]},"maxItems":7,"type":"array","uniqueItems":true}},"required":[],"type":"object"},"name":"search_tasks","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"list_approvals","requiredClaims":["governance:approvals:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"List approvals in the run company.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_approvals","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"get_approval","requiredClaims":["governance:approvals:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read one approval without protected data.","inputSchema":{"additionalProperties":false,"properties":{"approvalId":{"description":"Approval id.","maxLength":200,"minLength":1,"type":"string"}},"required":["approvalId"],"type":"object"},"name":"get_approval","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"get_approval_context","requiredClaims":["governance:approvals:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read one approval, its comments, and linked tasks.","inputSchema":{"additionalProperties":false,"properties":{"approvalId":{"description":"Approval id.","maxLength":200,"minLength":1,"type":"string"}},"required":["approvalId"],"type":"object"},"name":"get_approval_context","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"get_workspace_runtime","requiredClaims":["workspace:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read active-task workspace services.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"get_workspace_runtime","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"control_workspace_service","requiredClaims":["workspace:control"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Start, stop, or fault one active-task workspace service.","inputSchema":{"additionalProperties":false,"properties":{"action":{"enum":["start","stop","fail"]},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"serviceId":{"description":"Workspace service id.","maxLength":200,"minLength":1,"type":"string"},"url":{"description":"Optional service URL.","maxLength":20000,"type":["string","null"]}},"required":["idempotencyKey","serviceId","action"],"type":"object"},"name":"control_workspace_service","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"set_dependencies","requiredClaims":["dependencies:write"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Replace the active task's first-class blocker set.","inputSchema":{"additionalProperties":false,"properties":{"blockedByTaskIds":{"description":"Replacement blocker task ids.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","blockedByTaskIds"],"type":"object"},"name":"set_dependencies","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"create_task","requiredClaims":["delegation:tasks:create"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create a project task from a conversation, or a child from an ordinary task. Persist initialPlan before execution. Set status to backlog when the user wants to save or plan work without starting it; backlog tasks never wake an agent. Omitted status means todo, subject to blockers.","inputSchema":{"additionalProperties":false,"properties":{"assigneeActorId":{"description":"Optional agent assignee. Omit to assign the current agent.","maxLength":20000,"type":["string","null"]},"blockedByTaskIds":{"description":"Initial blocker task ids.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"description":{"description":"Child task description.","maxLength":20000,"type":["string","null"]},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"initialPlan":{"description":"Relevant markdown plan saved on the new task before execution starts.","maxLength":200000,"type":["string","null"]},"priority":{"enum":["critical","high","medium","low"]},"projectId":{"description":"Project ID for the task.","type":["string","null"]},"status":{"description":"Initial status. Use backlog to save work without executing it. Defaults to todo (blocked when dependencies are unresolved).","enum":["backlog","todo"]},"title":{"description":"Child task title.","maxLength":500,"minLength":1,"type":"string"}},"required":["idempotencyKey","title"],"type":"object"},"name":"create_task","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"},"task":{"additionalProperties":false,"properties":{"assigneeActorId":{"type":["string","null"]},"id":{"minLength":1,"type":"string"},"identifier":{"type":["string","null"]},"parentId":{"minLength":1,"type":["string","null"]},"projectId":{"minLength":1,"type":["string","null"]},"status":{"minLength":1,"type":"string"}},"required":["id","identifier","parentId","status","assigneeActorId"],"type":"object"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds","task"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"request_approval","requiredClaims":["governance:approvals:request"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create a governed approval and waiting posture.","inputSchema":{"additionalProperties":false,"properties":{"approvalType":{"description":"Stable approval type.","maxLength":200,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"payload":{"additionalProperties":true,"type":"object"}},"required":["idempotencyKey","approvalType","payload"],"type":"object"},"name":"request_approval","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"decide_approval","requiredClaims":["governance:approvals:decide"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Decide an approval as an explicitly authorized approver.","inputSchema":{"additionalProperties":false,"properties":{"approvalId":{"description":"Approval id.","maxLength":200,"minLength":1,"type":"string"},"decision":{"enum":["approved","rejected","cancelled"]},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"note":{"description":"Decision note.","maxLength":20000,"minLength":1,"type":"string"}},"required":["idempotencyKey","approvalId","decision","note"],"type":"object"},"name":"decide_approval","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"comment_on_approval","requiredClaims":["governance:approvals:comment"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Add a durable comment to an approval.","inputSchema":{"additionalProperties":false,"properties":{"approvalId":{"description":"Approval id.","maxLength":200,"minLength":1,"type":"string"},"body":{"description":"Approval comment.","maxLength":20000,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","approvalId","body"],"type":"object"},"name":"comment_on_approval","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"schedule_wake","requiredClaims":["control_plane:wakes"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Schedule a deterministic continuation wake.","inputSchema":{"additionalProperties":false,"properties":{"delayTicks":{"maximum":10000,"minimum":1,"type":"integer"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"payload":{"additionalProperties":true,"type":"object"},"reason":{"enum":["manual","issue_commented","interaction_resolved","approval_resolved","blockers_resolved","scheduled_retry","resume"]}},"required":["idempotencyKey","reason","delayTicks"],"type":"object"},"name":"schedule_wake","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"generic_api_request","requiredClaims":["test:generic_api_request"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Test-only escape hatch. Disabled unless the scenario and explicit claim both enable it.","inputSchema":{"additionalProperties":false,"properties":{"body":{"additionalProperties":true,"type":"object"},"method":{"enum":["GET","POST","PATCH"]},"path":{"maxLength":500,"pattern":"^/mock/","type":"string"}},"required":["method","path"],"type":"object"},"name":"generic_api_request","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"search_api","requiredClaims":["api:discover"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Fallback only: discover Paperclip API operations when the available dedicated tools cannot express the task. Prefer dedicated tools for common operations; do not search before using them. For a persistent hire, first list_agents to reuse a suitable teammate. Search agent-hires for the hiring schema and agent-configurations for compatible adapter/runtime settings, then use call_api with the returned operationId. Supply role-specific instructionsBundle.files as a filename-to-content record. Use the returned agent.id for delegation and obey any pending approval. Provider helper threads are temporary workers, not Paperclip hires. If a hire response is uncertain, reconcile with list_agents before retrying.","inputSchema":{"additionalProperties":false,"properties":{"cursor":{"maxLength":200,"type":"string"},"limit":{"default":5,"maximum":10,"minimum":1,"type":"integer"},"query":{"maxLength":500,"minLength":1,"type":"string"}},"required":["query"],"type":"object"},"name":"search_api","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"call_api","requiredClaims":["api:call"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Fallback only: call a discovered Paperclip API operation when dedicated tools lack the required operation or parameters. Uses your existing permissions. Prefer dedicated tools; never bypass a denial or runner lifecycle tool.","inputSchema":{"additionalProperties":false,"properties":{"body":{"anyOf":[{"additionalProperties":true,"type":"object"},{"items":{},"type":"array"},{"type":"string"},{"type":"number"},{"type":"boolean"},{"type":"null"}],"description":"Request value matching the discovered schema. For JSON object or array requests, pass the object or array directly, never a JSON-encoded string. Strings are for text bodies or endpoints whose schema explicitly accepts a string."},"contentType":{"maxLength":120,"type":"string"},"files":{"items":{"additionalProperties":false,"oneOf":[{"properties":{"artifactId":{}},"required":["artifactId"]},{"properties":{"path":{}},"required":["path"]}],"properties":{"artifactId":{"type":"string"},"field":{"type":"string"},"path":{"description":"File relative to the active issue workspace. Remote files must first be uploaded as an artifact.","type":"string"}},"type":"object"},"maxItems":10,"type":"array"},"operationId":{"description":"Exact operationId returned by search_api, for example GET /api/projects/{id}. Do not guess identifiers.","maxLength":500,"minLength":1,"type":"string"},"pathParams":{"additionalProperties":{"type":"string"},"type":"object"},"query":{"additionalProperties":true,"type":"object"}},"required":["operationId"],"type":"object"},"name":"call_api","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"create_project","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create a project after considering existing projects and available repositories. repositoryIds and repositoryUrls accept multiple existing repositories. Use HTTPS GitHub repositoryUrls when an accessible repo is not in the catalog; this registers project repositories, not remote GitHub repositories. Non-code projects may omit repositories. Cannot combine repositoryIds/repositoryUrls with workspace. Reuse the idempotency key on retries.","inputSchema":{"additionalProperties":false,"properties":{"archivedAt":{"description":"Archive timestamp.","maxLength":20000,"type":["string","null"]},"color":{"description":"Project color.","maxLength":20000,"type":["string","null"]},"description":{"description":"Project outcome and context.","maxLength":20000,"type":["string","null"]},"env":{"additionalProperties":true,"type":"object"},"executionWorkspacePolicy":{"additionalProperties":true,"type":"object"},"goalId":{"description":"Goal ID.","maxLength":20000,"type":["string","null"]},"goalIds":{"description":"Goal IDs.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"icon":{"description":"Project icon.","enum":["folder","rocket","code","terminal","database","globe","package","boxes","box","layers","briefcase","compass","target","flame","zap","star","bug","wrench","hammer","lightbulb","sparkles","shield","lock","search","cog","brain","cpu","git-branch","file-code","puzzle","gem","atom","heart","mail","message-square","crown","radar","telescope","hexagon",null],"type":["string","null"]},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"leadAgentId":{"description":"Lead agent ID.","maxLength":20000,"type":["string","null"]},"name":{"description":"Project name.","maxLength":500,"minLength":1,"type":"string"},"repositoryIds":{"description":"Authorized repository IDs from list_project_repositories; may contain multiple repositories.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"repositoryUrls":{"description":"Existing HTTPS GitHub repository URLs, including repos absent from the catalog.","items":{"maxLength":2000,"pattern":"^https://github\\.com/(?!\\.{1,2}/)[A-Za-z0-9_.-]+/(?!\\.{1,2}/?$)[A-Za-z0-9_.-]+/?$","type":"string"},"maxItems":100,"type":"array"},"status":{"enum":["backlog","planned","in_progress","completed","cancelled"]},"targetDate":{"description":"Target date.","maxLength":20000,"type":["string","null"]},"workspace":{"additionalProperties":true,"type":"object"}},"required":["idempotencyKey","name"],"type":"object"},"name":"create_project","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"list_project_repositories","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"List authorized repositories with stable IDs and names. Consider appropriate repositories before creating a project; never invent IDs.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_project_repositories","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"list_projects","requiredClaims":["discovery:projects:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Inspect available company projects before selecting a project for new work.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_projects","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"create_skill","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create a reusable single-file skill in the company library. Supply a complete SKILL.md whose name and description match the inputs. This saves the skill and shows a card; it does not assign the skill to any agent. Reuse idempotencyKey on retries.","inputSchema":{"additionalProperties":false,"properties":{"description":{"maxLength":2000,"minLength":1,"type":"string"},"idempotencyKey":{"maxLength":240,"minLength":1,"type":"string"},"markdown":{"description":"Complete SKILL.md including name and description frontmatter and substantive instructions.","maxLength":200000,"minLength":1,"type":"string"},"name":{"description":"Lowercase skill name, matching SKILL.md frontmatter.","maxLength":120,"minLength":1,"pattern":"^[a-z0-9]+(?:-[a-z0-9]+)*$","type":"string"},"slug":{"description":"Optional; must equal name.","maxLength":120,"minLength":1,"pattern":"^[a-z0-9]+(?:-[a-z0-9]+)*$","type":"string"}},"required":["idempotencyKey","name","description","markdown"],"type":"object"},"name":"create_skill","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"reassign_task","requiredClaims":["delegation:tasks:assign"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Reassign an existing task to another company agent. Read search_tasks first and supply its current assignee and statusVersion to prevent overwriting a concurrent change. Preserve the task, documents, dependencies, and blocked/backlog status. Active work is stopped before handoff. Use a stable idempotency key for retries. Cannot reassign this run’s own task, conversations, completed tasks, or pending reviews; use create_task for delegation from the current task.","inputSchema":{"additionalProperties":false,"properties":{"assigneeActorId":{"description":"New company agent ID from list_agents.","minLength":1,"type":"string"},"expectedAssigneeActorId":{"description":"Current assigneeAgentId from search_tasks; null means unassigned.","type":["string","null"]},"expectedStatusVersion":{"description":"Current statusVersion from search_tasks.","minimum":0,"type":"integer"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"reason":{"description":"Why the task should move and context for the new owner.","maxLength":20000,"minLength":1,"type":"string"},"taskId":{"description":"Existing task ID from search_tasks.","minLength":1,"type":"string"}},"required":["idempotencyKey","taskId","assigneeActorId","expectedAssigneeActorId","expectedStatusVersion","reason"],"type":"object"},"name":"reassign_task","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}}] +[{"annotations":{"exposure":"always","operationId":"get_task_context","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read the active task and actor, including the exact approved Markdown revision when this issue has an accepted plan.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"get_task_context","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"get_task_history","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read bounded comments on the active task.","inputSchema":{"additionalProperties":false,"properties":{"limit":{"default":50,"maximum":200,"minimum":1,"type":"integer"}},"required":[],"type":"object"},"name":"get_task_history","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"list_documents","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"List revisioned documents on the active task.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_documents","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"read_document","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read the current revision of one active-task document.","inputSchema":{"additionalProperties":false,"properties":{"key":{"description":"Stable issue-document key.","maxLength":120,"minLength":1,"type":"string"}},"required":["key"],"type":"object"},"name":"read_document","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"list_document_revisions","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read bounded revision history for one active-task document.","inputSchema":{"additionalProperties":false,"properties":{"key":{"description":"Stable issue-document key.","maxLength":120,"minLength":1,"type":"string"},"limit":{"default":50,"maximum":200,"minimum":1,"type":"integer"}},"required":["key"],"type":"object"},"name":"list_document_revisions","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"always","operationId":"report_progress","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Append a durable progress comment to the active task.","inputSchema":{"additionalProperties":false,"properties":{"body":{"description":"Multiline progress update.","maxLength":20000,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","body"],"type":"object"},"name":"report_progress","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"answer_status_question","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Append the answer to a status-only wake without changing task disposition.","inputSchema":{"additionalProperties":false,"properties":{"body":{"description":"Concise status answer.","maxLength":20000,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","body"],"type":"object"},"name":"answer_status_question","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"write_document","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create or update an active-task document with optimistic revision safety.","inputSchema":{"additionalProperties":false,"properties":{"baseRevisionId":{"description":"Current revision id, or null when creating.","maxLength":20000,"type":["string","null"]},"body":{"description":"Markdown document body.","maxLength":200000,"minLength":1,"type":"string"},"changeSummary":{"description":"Optional revision summary.","maxLength":20000,"type":["string","null"]},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"key":{"description":"Stable issue-document key.","maxLength":120,"minLength":1,"type":"string"},"title":{"description":"Document title.","maxLength":300,"minLength":1,"type":"string"}},"required":["idempotencyKey","key","title","body","baseRevisionId"],"type":"object"},"name":"write_document","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"request_human_input","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create a durable human question or approval card on the current Paperclip task bound to this run; Paperclip renders it and authenticates the response. Use questions with continuationPolicy='wake_assignee' when an answer is needed, including otherwise tool-free chat turns. Supply a stable idempotencyKey and reuse it on retries. For one question at a time, ask only the next unanswered question and wait for its real answer. Never infer answers, answer your own card, or treat clarification as approval. Preserve existing review gates. Call this tool before claiming a question was asked; if creation fails, report the failure. Do not fabricate answer links or Markdown buttons, post duplicate cards, or substitute call_api. Use payload.questions for choices and payload.questionSet for text fields; see the payload schema for formats.","inputSchema":{"additionalProperties":false,"properties":{"continuationPolicy":{"enum":["none","wake_assignee","wake_assignee_on_accept"]},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"interactionKind":{"enum":["confirmation","checkbox","questions","suggest_tasks","item_verdicts"]},"payload":{"additionalProperties":true,"description":"Kind-specific interaction data. For interactionKind='questions', use version:1 and questions:[{id,prompt,selectionMode:'single'|'multi',required?,options:[{id,label,description?,freeText?}]}]. Choice questions need at least two distinct meaningful options. For an open-ended text answer, ALSO include questionSet:{schema:'paperclip.question_set.v1',questions:[{id,prompt,answerMode:'text',required?}]} with no options or customAnswer in its text questions. Keep matching IDs/prompts in both arrays; the required compatibility questions entry uses selectionMode:'single' and options:[{id:'describe',label:'Your answer',freeText:true}]. Without questionSet this incorrectly renders as a one-option choice. Never use a lone Other or describe option as the presentation. Option keys are id/label, not value. For confirmation, payload may be {}. Keep IDs stable across retries.","type":"object"},"prompt":{"description":"Question or decision prompt.","maxLength":10000,"minLength":1,"type":"string"},"targetRevisionId":{"description":"Optional bound document revision.","maxLength":20000,"type":["string","null"]},"title":{"description":"Interaction card title.","maxLength":300,"minLength":1,"type":"string"}},"required":["idempotencyKey","interactionKind","title","prompt","continuationPolicy"],"type":"object"},"name":"request_human_input","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"register_deliverable","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Register attachment metadata and its artifact work product without credentials or bytes in the tool result.","inputSchema":{"additionalProperties":false,"properties":{"byteSize":{"maximum":100000000,"minimum":0,"type":"integer"},"contentRef":{"description":"Opaque package-local content reference.","maxLength":2000,"minLength":1,"type":"string"},"contentType":{"description":"Media type.","maxLength":200,"minLength":1,"type":"string"},"filename":{"description":"Display filename.","maxLength":500,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"sha256":{"pattern":"^[a-fA-F0-9]{64}$","type":"string"},"title":{"description":"Work-product title.","maxLength":500,"minLength":1,"type":"string"}},"required":["idempotencyKey","filename","contentType","byteSize","sha256","contentRef","title"],"type":"object"},"name":"register_deliverable","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"finish_task","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Finish the active task with a durable summary.","inputSchema":{"additionalProperties":false,"properties":{"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"summary":{"description":"Completion summary.","maxLength":20000,"minLength":1,"type":"string"}},"required":["idempotencyKey","summary"],"type":"object"},"name":"finish_task","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"block_task","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Block the active task with a durable reason and optional first-class dependencies.","inputSchema":{"additionalProperties":false,"properties":{"blockedByTaskIds":{"description":"Internal task ids that block this task.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"reason":{"description":"Block reason.","maxLength":20000,"minLength":1,"type":"string"}},"required":["idempotencyKey","reason"],"type":"object"},"name":"block_task","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"always","operationId":"request_review","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Move the active task to review with a durable summary.","inputSchema":{"additionalProperties":false,"properties":{"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"summary":{"description":"Review handoff summary.","maxLength":20000,"minLength":1,"type":"string"}},"required":["idempotencyKey","summary"],"type":"object"},"name":"request_review","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"list_agents","requiredClaims":["discovery:agents:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"List redacted actor profiles.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_agents","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"get_agent","requiredClaims":["discovery:agents:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read one redacted actor profile.","inputSchema":{"additionalProperties":false,"properties":{"actorId":{"description":"Actor id.","maxLength":200,"minLength":1,"type":"string"}},"required":["actorId"],"type":"object"},"name":"get_agent","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"hire_agent","requiredClaims":["delegation:agents:create"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create a persistent native Runner teammate with an identity and persona. The teammate reports to the caller and inherits the caller's native runtime; provider, adapter, environment, and credential settings are selected by Paperclip and are never caller-supplied here. Reuse a suitable teammate from list_agents when possible.","inputSchema":{"additionalProperties":false,"properties":{"capabilities":{"maxLength":2000,"type":["string","null"]},"instructions":{"maxLength":20000,"type":["string","null"]},"name":{"maxLength":200,"minLength":1,"type":"string"},"role":{"enum":["ceo","cto","cmo","cfo","security","engineer","designer","pm","qa","devops","researcher","general"],"type":"string"},"title":{"maxLength":300,"type":["string","null"]}},"required":["name"],"type":"object"},"name":"hire_agent","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"search_tasks","requiredClaims":["discovery:tasks:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Search tasks by text and status within the run company.","inputSchema":{"additionalProperties":false,"properties":{"limit":{"default":50,"maximum":200,"minimum":1,"type":"integer"},"query":{"maxLength":500,"type":"string"},"statuses":{"items":{"enum":["backlog","todo","in_progress","in_review","done","blocked","cancelled"]},"maxItems":7,"type":"array","uniqueItems":true}},"required":[],"type":"object"},"name":"search_tasks","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"list_approvals","requiredClaims":["governance:approvals:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"List approvals in the run company.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_approvals","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"get_approval","requiredClaims":["governance:approvals:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read one approval without protected data.","inputSchema":{"additionalProperties":false,"properties":{"approvalId":{"description":"Approval id.","maxLength":200,"minLength":1,"type":"string"}},"required":["approvalId"],"type":"object"},"name":"get_approval","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"get_approval_context","requiredClaims":["governance:approvals:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read one approval, its comments, and linked tasks.","inputSchema":{"additionalProperties":false,"properties":{"approvalId":{"description":"Approval id.","maxLength":200,"minLength":1,"type":"string"}},"required":["approvalId"],"type":"object"},"name":"get_approval_context","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"get_workspace_runtime","requiredClaims":["workspace:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Read active-task workspace services.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"get_workspace_runtime","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"control_workspace_service","requiredClaims":["workspace:control"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Start, stop, or fault one active-task workspace service.","inputSchema":{"additionalProperties":false,"properties":{"action":{"enum":["start","stop","fail"]},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"serviceId":{"description":"Workspace service id.","maxLength":200,"minLength":1,"type":"string"},"url":{"description":"Optional service URL.","maxLength":20000,"type":["string","null"]}},"required":["idempotencyKey","serviceId","action"],"type":"object"},"name":"control_workspace_service","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"set_dependencies","requiredClaims":["dependencies:write"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Replace the active task's first-class blocker set.","inputSchema":{"additionalProperties":false,"properties":{"blockedByTaskIds":{"description":"Replacement blocker task ids.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","blockedByTaskIds"],"type":"object"},"name":"set_dependencies","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"create_task","requiredClaims":["delegation:tasks:create"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create a project task from a conversation, or a child from an ordinary task. Persist initialPlan before execution. Set status to backlog when the user wants to save or plan work without starting it; backlog tasks never wake an agent. Omitted status means todo, subject to blockers.","inputSchema":{"additionalProperties":false,"properties":{"assigneeActorId":{"description":"Optional agent assignee. Omit to assign the current agent.","maxLength":20000,"type":["string","null"]},"blockedByTaskIds":{"description":"Initial blocker task ids.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"description":{"description":"Child task description.","maxLength":20000,"type":["string","null"]},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"initialPlan":{"description":"Relevant markdown plan saved on the new task before execution starts.","maxLength":200000,"type":["string","null"]},"priority":{"enum":["critical","high","medium","low"]},"projectId":{"description":"Project ID for the task.","type":["string","null"]},"status":{"description":"Initial status. Use backlog to save work without executing it. Defaults to todo (blocked when dependencies are unresolved).","enum":["backlog","todo"]},"title":{"description":"Child task title.","maxLength":500,"minLength":1,"type":"string"}},"required":["idempotencyKey","title"],"type":"object"},"name":"create_task","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"},"task":{"additionalProperties":false,"properties":{"assigneeActorId":{"type":["string","null"]},"id":{"minLength":1,"type":"string"},"identifier":{"type":["string","null"]},"parentId":{"minLength":1,"type":["string","null"]},"projectId":{"minLength":1,"type":["string","null"]},"status":{"minLength":1,"type":"string"}},"required":["id","identifier","parentId","status","assigneeActorId"],"type":"object"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds","task"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"request_approval","requiredClaims":["governance:approvals:request"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create a governed approval and waiting posture.","inputSchema":{"additionalProperties":false,"properties":{"approvalType":{"description":"Stable approval type.","maxLength":200,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"payload":{"additionalProperties":true,"type":"object"}},"required":["idempotencyKey","approvalType","payload"],"type":"object"},"name":"request_approval","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"decide_approval","requiredClaims":["governance:approvals:decide"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Decide an approval as an explicitly authorized approver.","inputSchema":{"additionalProperties":false,"properties":{"approvalId":{"description":"Approval id.","maxLength":200,"minLength":1,"type":"string"},"decision":{"enum":["approved","rejected","cancelled"]},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"note":{"description":"Decision note.","maxLength":20000,"minLength":1,"type":"string"}},"required":["idempotencyKey","approvalId","decision","note"],"type":"object"},"name":"decide_approval","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"comment_on_approval","requiredClaims":["governance:approvals:comment"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Add a durable comment to an approval.","inputSchema":{"additionalProperties":false,"properties":{"approvalId":{"description":"Approval id.","maxLength":200,"minLength":1,"type":"string"},"body":{"description":"Approval comment.","maxLength":20000,"minLength":1,"type":"string"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"}},"required":["idempotencyKey","approvalId","body"],"type":"object"},"name":"comment_on_approval","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"schedule_wake","requiredClaims":["control_plane:wakes"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Schedule a deterministic continuation wake.","inputSchema":{"additionalProperties":false,"properties":{"delayTicks":{"maximum":10000,"minimum":1,"type":"integer"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"payload":{"additionalProperties":true,"type":"object"},"reason":{"enum":["manual","issue_commented","interaction_resolved","approval_resolved","blockers_resolved","scheduled_retry","resume"]}},"required":["idempotencyKey","reason","delayTicks"],"type":"object"},"name":"schedule_wake","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}},{"annotations":{"exposure":"optional","operationId":"generic_api_request","requiredClaims":["test:generic_api_request"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Test-only escape hatch. Disabled unless the scenario and explicit claim both enable it.","inputSchema":{"additionalProperties":false,"properties":{"body":{"additionalProperties":true,"type":"object"},"method":{"enum":["GET","POST","PATCH"]},"path":{"maxLength":500,"pattern":"^/mock/","type":"string"}},"required":["method","path"],"type":"object"},"name":"generic_api_request","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"search_api","requiredClaims":["api:discover"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Fallback only: discover Paperclip API operations when the available dedicated tools cannot express the task. Prefer dedicated tools for common operations; do not search before using them. For a persistent hire, first list_agents to reuse a suitable teammate. Search agent-hires for the hiring schema and agent-configurations for compatible adapter/runtime settings, then use call_api with the returned operationId. Supply role-specific instructionsBundle.files as a filename-to-content record. Use the returned agent.id for delegation and obey any pending approval. Provider helper threads are temporary workers, not Paperclip hires. If a hire response is uncertain, reconcile with list_agents before retrying.","inputSchema":{"additionalProperties":false,"properties":{"cursor":{"maxLength":200,"type":"string"},"limit":{"default":5,"maximum":10,"minimum":1,"type":"integer"},"query":{"maxLength":500,"minLength":1,"type":"string"}},"required":["query"],"type":"object"},"name":"search_api","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"call_api","requiredClaims":["api:call"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Fallback only: call a discovered Paperclip API operation when dedicated tools lack the required operation or parameters. Uses your existing permissions. Prefer dedicated tools; never bypass a denial or runner lifecycle tool. For large text responses, read the returned artifact with GET /api/assets/{assetId}/content and responseText; follow nextOffsetBytes until null.","inputSchema":{"additionalProperties":false,"properties":{"body":{"anyOf":[{"additionalProperties":true,"type":"object"},{"items":{},"type":"array"},{"type":"string"},{"type":"number"},{"type":"boolean"},{"type":"null"}],"description":"Request value matching the discovered schema. For JSON object or array requests, pass the object or array directly, never a JSON-encoded string. Strings are for text bodies or endpoints whose schema explicitly accepts a string."},"contentType":{"maxLength":120,"type":"string"},"files":{"items":{"additionalProperties":false,"oneOf":[{"properties":{"artifactId":{}},"required":["artifactId"]},{"properties":{"path":{}},"required":["path"]}],"properties":{"artifactId":{"type":"string"},"field":{"type":"string"},"path":{"description":"File relative to the active issue workspace. Remote files must first be uploaded as an artifact.","type":"string"}},"type":"object"},"maxItems":10,"type":"array"},"operationId":{"description":"Exact operationId returned by search_api, for example GET /api/projects/{id}. Do not guess identifiers.","maxLength":500,"minLength":1,"type":"string"},"pathParams":{"additionalProperties":{"type":"string"},"type":"object"},"query":{"additionalProperties":true,"type":"object"},"responseText":{"additionalProperties":false,"description":"GET only: return a bounded UTF-8 text window inline, including JSON as text, without saving another artifact. Offsets and limits are bytes. Use the returned nextOffsetBytes to continue; null means complete. Prefer reading a saved artifact for a stable snapshot. New live responses have a 1 GiB capture limit and a 4 GiB per-run capture budget. Existing larger assets remain readable in bounded pages. If a live response returns an artifact, continue on its content operation for a stable snapshot.","properties":{"limitBytes":{"default":24576,"maximum":24576,"minimum":4,"type":"integer"},"offsetBytes":{"default":0,"maximum":9007199254740991,"minimum":0,"type":"integer"}},"type":"object"}},"required":["operationId"],"type":"object"},"name":"call_api","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"create_project","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create a project after considering existing projects and available repositories. repositoryIds and repositoryUrls accept multiple existing repositories. Use HTTPS GitHub repositoryUrls when an accessible repo is not in the catalog; this registers project repositories, not remote GitHub repositories. Non-code projects may omit repositories. Cannot combine repositoryIds/repositoryUrls with workspace. Reuse the idempotency key on retries.","inputSchema":{"additionalProperties":false,"properties":{"archivedAt":{"description":"Archive timestamp.","maxLength":20000,"type":["string","null"]},"color":{"description":"Project color.","maxLength":20000,"type":["string","null"]},"description":{"description":"Project outcome and context.","maxLength":20000,"type":["string","null"]},"env":{"additionalProperties":true,"type":"object"},"executionWorkspacePolicy":{"additionalProperties":true,"type":"object"},"goalId":{"description":"Goal ID.","maxLength":20000,"type":["string","null"]},"goalIds":{"description":"Goal IDs.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"icon":{"description":"Project icon.","enum":["folder","rocket","code","terminal","database","globe","package","boxes","box","layers","briefcase","compass","target","flame","zap","star","bug","wrench","hammer","lightbulb","sparkles","shield","lock","search","cog","brain","cpu","git-branch","file-code","puzzle","gem","atom","heart","mail","message-square","crown","radar","telescope","hexagon",null],"type":["string","null"]},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"leadAgentId":{"description":"Lead agent ID.","maxLength":20000,"type":["string","null"]},"name":{"description":"Project name.","maxLength":500,"minLength":1,"type":"string"},"repositoryIds":{"description":"Authorized repository IDs from list_project_repositories; may contain multiple repositories.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"repositoryUrls":{"description":"Existing HTTPS GitHub repository URLs, including repos absent from the catalog.","items":{"maxLength":2000,"pattern":"^https://github\\.com/(?!\\.{1,2}/)[A-Za-z0-9_.-]+/(?!\\.{1,2}/?$)[A-Za-z0-9_.-]+/?$","type":"string"},"maxItems":100,"type":"array"},"status":{"enum":["backlog","planned","in_progress","completed","cancelled"]},"targetDate":{"description":"Target date.","maxLength":20000,"type":["string","null"]},"workspace":{"additionalProperties":true,"type":"object"}},"required":["idempotencyKey","name"],"type":"object"},"name":"create_project","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"list_project_repositories","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"List authorized repositories with stable IDs and names. Consider appropriate repositories before creating a project; never invent IDs.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_project_repositories","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"list_projects","requiredClaims":["discovery:projects:read"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Inspect available company projects before selecting a project for new work.","inputSchema":{"additionalProperties":false,"properties":{},"required":[],"type":"object"},"name":"list_projects","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"create_skill","requiredClaims":[],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Create a reusable single-file skill in the company library. Supply a complete SKILL.md whose name and description match the inputs. This saves the skill and shows a card; it does not assign the skill to any agent. Reuse idempotencyKey on retries.","inputSchema":{"additionalProperties":false,"properties":{"description":{"maxLength":2000,"minLength":1,"type":"string"},"idempotencyKey":{"maxLength":240,"minLength":1,"type":"string"},"markdown":{"description":"Complete SKILL.md including name and description frontmatter and substantive instructions.","maxLength":200000,"minLength":1,"type":"string"},"name":{"description":"Lowercase skill name, matching SKILL.md frontmatter.","maxLength":120,"minLength":1,"pattern":"^[a-z0-9]+(?:-[a-z0-9]+)*$","type":"string"},"slug":{"description":"Optional; must equal name.","maxLength":120,"minLength":1,"pattern":"^[a-z0-9]+(?:-[a-z0-9]+)*$","type":"string"}},"required":["idempotencyKey","name","description","markdown"],"type":"object"},"name":"create_skill","outputSchema":{"additionalProperties":true,"type":"object"}},{"annotations":{"exposure":"optional","operationId":"reassign_task","requiredClaims":["delegation:tasks:assign"],"semanticContract":"paperclip.semantic-tool.v1","version":1},"description":"Reassign an existing task to another company agent. Read search_tasks first and supply its current assignee and statusVersion to prevent overwriting a concurrent change. Preserve the task, documents, dependencies, and blocked/backlog status. Active work is stopped before handoff. Use a stable idempotency key for retries. Cannot reassign this run’s own task, conversations, completed tasks, or pending reviews; use create_task for delegation from the current task.","inputSchema":{"additionalProperties":false,"properties":{"assigneeActorId":{"description":"New company agent ID from list_agents.","minLength":1,"type":"string"},"expectedAssigneeActorId":{"description":"Current assigneeAgentId from search_tasks; null means unassigned.","type":["string","null"]},"expectedStatusVersion":{"description":"Current statusVersion from search_tasks.","minimum":0,"type":"integer"},"idempotencyKey":{"description":"Caller-stable retry key.","maxLength":240,"minLength":1,"type":"string"},"reason":{"description":"Why the task should move and context for the new owner.","maxLength":20000,"minLength":1,"type":"string"},"taskId":{"description":"Existing task ID from search_tasks.","minLength":1,"type":"string"}},"required":["idempotencyKey","taskId","assigneeActorId","expectedAssigneeActorId","expectedStatusVersion","reason"],"type":"object"},"name":"reassign_task","outputSchema":{"additionalProperties":false,"properties":{"commandId":{"description":"Stable command identifier.","maxLength":200,"minLength":1,"type":"string"},"disposition":{"enum":["applied","duplicate"]},"entityRefs":{"description":"Entities affected by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"scheduledWakeIds":{"description":"Wake identifiers scheduled by the operation.","items":{"minLength":1,"type":"string"},"maxItems":200,"type":"array","uniqueItems":true},"stateRevision":{"minimum":0,"type":"integer"}},"required":["commandId","disposition","stateRevision","entityRefs","scheduledWakeIds"],"type":"object"}}] diff --git a/packages/paperclip-runner/generated/semantic-action-catalog.json b/packages/paperclip-runner/generated/semantic-action-catalog.json index 2df6b716fb4..adeb3bb5332 100644 --- a/packages/paperclip-runner/generated/semantic-action-catalog.json +++ b/packages/paperclip-runner/generated/semantic-action-catalog.json @@ -112,7 +112,7 @@ "planning", "skill_test" ], - "description": "Fallback only: call a discovered Paperclip API operation when dedicated tools lack the required operation or parameters. Uses your existing permissions. Prefer dedicated tools; never bypass a denial or runner lifecycle tool.", + "description": "Fallback only: call a discovered Paperclip API operation when dedicated tools lack the required operation or parameters. Uses your existing permissions. Prefer dedicated tools; never bypass a denial or runner lifecycle tool. For large text responses, read the returned artifact with GET /api/assets/{assetId}/content and responseText; follow nextOffsetBytes until null.", "effect": "write", "inputSchema": { "additionalProperties": false, @@ -199,6 +199,25 @@ "query": { "additionalProperties": true, "type": "object" + }, + "responseText": { + "additionalProperties": false, + "description": "GET only: return a bounded UTF-8 text window inline, including JSON as text, without saving another artifact. Offsets and limits are bytes. Use the returned nextOffsetBytes to continue; null means complete. Prefer reading a saved artifact for a stable snapshot. New live responses have a 1 GiB capture limit and a 4 GiB per-run capture budget. Existing larger assets remain readable in bounded pages. If a live response returns an artifact, continue on its content operation for a stable snapshot.", + "properties": { + "limitBytes": { + "default": 24576, + "maximum": 24576, + "minimum": 4, + "type": "integer" + }, + "offsetBytes": { + "default": 0, + "maximum": 9007199254740991, + "minimum": 0, + "type": "integer" + } + }, + "type": "object" } }, "required": [ diff --git a/packages/paperclip-runner/protocol/fixtures/evals/native-execution-seeded.json b/packages/paperclip-runner/protocol/fixtures/evals/native-execution-seeded.json index 4ff19e56d72..74e4b4be817 100644 --- a/packages/paperclip-runner/protocol/fixtures/evals/native-execution-seeded.json +++ b/packages/paperclip-runner/protocol/fixtures/evals/native-execution-seeded.json @@ -24,7 +24,7 @@ "prpVersion": 1, "nativeExecutionVersion": 1, "catalogVersion": 1, - "catalogSha256": "sha256:1fa8ecf5d3c28e62a32e818de95633ff3efe5d4af1a100310516ff9b5ab2358d", + "catalogSha256": "sha256:a19534304480dfd3d357cc902e785af05158ce1496f4d324184112732ff94c9c", "driverContractVersion": 1, "driverKind": "paperclip-deterministic", "driverVersion": "1.0.0" diff --git a/packages/paperclip-runner/protocol/manifest.json b/packages/paperclip-runner/protocol/manifest.json index 83ac18ab79d..bd8c300f1d9 100644 --- a/packages/paperclip-runner/protocol/manifest.json +++ b/packages/paperclip-runner/protocol/manifest.json @@ -160,7 +160,7 @@ }, { "path": "fixtures/evals/native-execution-seeded.json", - "sha256": "5f3144ea856ad27778f47de73eb542703b1d9df2abad2c427838fbaf69c3d59f", + "sha256": "c6367c7c7e3dd01fed341b7a83b0008c9d5d1efdfa85bc70e1013bc304a008c7", "expectation": "accept", "compatibilityCase": "canonical" }, diff --git a/packages/paperclip-runner/src/protocol-actions/call-api.ts b/packages/paperclip-runner/src/protocol-actions/call-api.ts index ffbed93c8c9..6ce1236ddb7 100644 --- a/packages/paperclip-runner/src/protocol-actions/call-api.ts +++ b/packages/paperclip-runner/src/protocol-actions/call-api.ts @@ -28,7 +28,7 @@ export const callApiAction = { }, "documentation": { "title": "Call the Paperclip API", - "description": "Fallback only: call a discovered Paperclip API operation when dedicated tools lack the required operation or parameters. Uses your existing permissions. Prefer dedicated tools; never bypass a denial or runner lifecycle tool.", + "description": "Fallback only: call a discovered Paperclip API operation when dedicated tools lack the required operation or parameters. Uses your existing permissions. Prefer dedicated tools; never bypass a denial or runner lifecycle tool. For large text responses, read the returned artifact with GET /api/assets/{assetId}/content and responseText; follow nextOffsetBytes until null.", "note": "Production HTTP fallback; does not grant privileges or replace dedicated tools." }, "examples": { @@ -51,7 +51,7 @@ export const callApiAction = { "operationId": "call_api", "version": 1, "title": "Call the Paperclip API", - "description": "Fallback only: call a discovered Paperclip API operation when dedicated tools lack the required operation or parameters. Uses your existing permissions. Prefer dedicated tools; never bypass a denial or runner lifecycle tool.", + "description": "Fallback only: call a discovered Paperclip API operation when dedicated tools lack the required operation or parameters. Uses your existing permissions. Prefer dedicated tools; never bypass a denial or runner lifecycle tool. For large text responses, read the returned artifact with GET /api/assets/{assetId}/content and responseText; follow nextOffsetBytes until null.", "exposure": "optional", "requiredClaims": [ "api:call" @@ -96,6 +96,15 @@ export const callApiAction = { "type": "string", "maxLength": 120 }, + "responseText": { + "description": "GET only: return a bounded UTF-8 text window inline, including JSON as text, without saving another artifact. Offsets and limits are bytes. Use the returned nextOffsetBytes to continue; null means complete. Prefer reading a saved artifact for a stable snapshot. New live responses have a 1 GiB capture limit and a 4 GiB per-run capture budget. Existing larger assets remain readable in bounded pages. If a live response returns an artifact, continue on its content operation for a stable snapshot.", + "type": "object", + "properties": { + "offsetBytes": { "type": "integer", "minimum": 0, "maximum": 9007199254740991, "default": 0 }, + "limitBytes": { "type": "integer", "minimum": 4, "maximum": 24576, "default": 24576 } + }, + "additionalProperties": false + }, "files": { "type": "array", "maxItems": 10, diff --git a/packages/shared/src/types/asset.ts b/packages/shared/src/types/asset.ts index 2520de4055c..d148d6cba0e 100644 --- a/packages/shared/src/types/asset.ts +++ b/packages/shared/src/types/asset.ts @@ -4,6 +4,7 @@ export interface AssetImage { provider: string; objectKey: string; contentType: string; + /** Safe integer byte count; persisted as bigint to support files above 2 GiB. */ byteSize: number; sha256: string; originalFilename: string | null; diff --git a/server/src/__tests__/assets.test.ts b/server/src/__tests__/assets.test.ts index 9c8308fb92d..fbc6976498c 100644 --- a/server/src/__tests__/assets.test.ts +++ b/server/src/__tests__/assets.test.ts @@ -421,6 +421,29 @@ describe("GET /api/assets/:assetId/content", () => { getAssetByIdMock.mockReset(); }); + it("reads only the requested storage range for a large asset", async () => { + const storage = createStorageService("text/plain"); + getAssetByIdMock.mockResolvedValue({ ...createAsset(), contentType: "text/plain", byteSize: 10 * 1024 * 1024 }); + vi.mocked(storage.getObject).mockImplementation(async (_company, _key, options) => { + expect(options).toEqual({ range: { start: 8192, end: 12288 } }); + return { stream: Readable.from(Buffer.alloc(4097, 65)), contentLength: 4097 }; + }); + const res = await requestApp(await createApp(storage), baseUrl => request(baseUrl).get("/api/assets/asset-1/content").set("Range", "bytes=8192-12288")); + expect(res.status).toBe(206); + expect(res.headers["content-range"]).toBe("bytes 8192-12288/10485760"); + expect(res.headers["content-length"]).toBe("4097"); + expect(res.text).toBe("A".repeat(4097)); + }); + + it.each(["bytes=999-1000", "bytes=0-1,3-4", "bytes=1x-3", "bytes=-", "items=0-3"])("rejects invalid asset ranges before storage reads: %s", async range => { + const storage = createStorageService(); + getAssetByIdMock.mockResolvedValue(createAsset()); + const res = await requestApp(await createApp(storage), baseUrl => request(baseUrl).get("/api/assets/asset-1/content").set("Range", range)); + expect(res.status).toBe(416); + expect(res.headers["content-range"]).toBe("bytes */40"); + expect(storage.getObject).not.toHaveBeenCalled(); + }); + it("downloads script-capable HTML with nosniff and a sandbox CSP", async () => { const html = Buffer.from(""); const storage = createStorageService("text/html"); diff --git a/server/src/__tests__/storage-local-provider.test.ts b/server/src/__tests__/storage-local-provider.test.ts index b72cdbaded8..8ff225fcf46 100644 --- a/server/src/__tests__/storage-local-provider.test.ts +++ b/server/src/__tests__/storage-local-provider.test.ts @@ -1,4 +1,6 @@ import { afterEach, describe, expect, it } from "vitest"; +import { Readable } from "node:stream"; +import { createHash } from "node:crypto"; import os from "node:os"; import path from "node:path"; import { promises as fs } from "node:fs"; @@ -42,6 +44,27 @@ describe("local disk storage provider", () => { expect(stored.sha256).toHaveLength(64); }); + it("writes streamed files with exact metadata", async () => { + const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-storage-")); + tempRoots.push(root); + const service = createStorageService(createLocalDiskStorageProvider(root)); + const bytes = Buffer.from("streamed response"); + const sha256 = createHash("sha256").update(bytes).digest("hex"); + const stored = await service.putFile({ companyId: "company-1", namespace: "runner-api", originalFilename: "response.txt", + contentType: "text/plain", body: Readable.from([bytes.subarray(0, 4), bytes.subarray(4)]), byteSize: bytes.length, sha256 }); + expect(stored).toMatchObject({ byteSize: bytes.length, sha256 }); + expect(await readStreamToBuffer((await service.getObject("company-1", stored.objectKey)).stream)).toEqual(bytes); + }); + + it("removes partial local uploads when their source fails", async () => { + const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-storage-")); + tempRoots.push(root); + const provider = createLocalDiskStorageProvider(root); + const body = Readable.from((async function* () { yield Buffer.from("partial"); throw new Error("source failed"); })()); + await expect(provider.putObject({ objectKey: "company-1/failed", contentType: "text/plain", contentLength: 20, body })).rejects.toThrow("source failed"); + expect(await fs.readdir(path.join(root, "company-1"))).toEqual([]); + }); + it("streams only requested byte ranges", async () => { const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-storage-")); tempRoots.push(root); diff --git a/server/src/__tests__/storage-s3-multipart.test.ts b/server/src/__tests__/storage-s3-multipart.test.ts new file mode 100644 index 00000000000..b3bbd6ae4b7 --- /dev/null +++ b/server/src/__tests__/storage-s3-multipart.test.ts @@ -0,0 +1,48 @@ +import { AbortMultipartUploadCommand, CompleteMultipartUploadCommand, CreateMultipartUploadCommand, UploadPartCommand, S3Client } from "@aws-sdk/client-s3"; +import { Readable } from "node:stream"; +import { describe, expect, it, vi } from "vitest"; +import { putS3Multipart } from "../storage/s3-multipart.js"; + +describe("streamed S3 multipart uploads", () => { + it("streams exact ordered parts without buffering the full response", async () => { + const client = new S3Client({ region: "us-east-1" }); + const lengths: number[] = []; + const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => { + if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" }; + if (command instanceof UploadPartCommand) { + expect(command.input.Body).toBeInstanceOf(Readable); + let length = 0; + for await (const chunk of command.input.Body as Readable) { + expect(chunk.length).toBeLessThanOrEqual(64 * 1024); + length += chunk.length; + } + expect(length).toBe(command.input.ContentLength); + lengths.push(length); + return { ETag: `part-${command.input.PartNumber}` }; + } + expect(command).toBeInstanceOf(CompleteMultipartUploadCommand); + expect(command.input.MultipartUpload.Parts).toEqual([1, 2, 3].map(PartNumber => ({ PartNumber, ETag: `part-${PartNumber}` }))); + return {}; + }); + const body = Readable.from((async function* () { for (let i = 0; i < 272; i++) yield Buffer.alloc(64 * 1024); })()); + try { + await putS3Multipart(client, "bucket", "key", { objectKey: "key", contentType: "text/plain", contentLength: 17 * 1024 * 1024, body }); + expect(lengths).toEqual([8, 8, 1].map(mb => mb * 1024 * 1024)); + } finally { send.mockRestore(); client.destroy(); } + }); + it("aborts an incomplete upload and closes its source on failure", async () => { + const client = new S3Client({ region: "us-east-1" }); + const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => { + if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" }; + if (command instanceof UploadPartCommand) throw new Error("storage unavailable"); + expect(command).toBeInstanceOf(AbortMultipartUploadCommand); + return {}; + }); + const body = Readable.from([Buffer.from("abc")]); + try { + await expect(putS3Multipart(client, "bucket", "key", { objectKey: "key", contentType: "text/plain", contentLength: 3, body })).rejects.toThrow("storage unavailable"); + expect(send).toHaveBeenCalledTimes(3); + expect(body.destroyed).toBe(true); + } finally { send.mockRestore(); client.destroy(); } + }); +}); diff --git a/server/src/routes/assets.ts b/server/src/routes/assets.ts index 3f9d4ee0c3c..5ac78e0cc2a 100644 --- a/server/src/routes/assets.ts +++ b/server/src/routes/assets.ts @@ -328,13 +328,32 @@ export function assetRoutes(db: Db, storage: StorageService) { const asset = await getAccessibleResource(req, res, svc.getById(assetId), "Asset not found"); if (!asset) return; - const object = await storage.getObject(asset.companyId, asset.objectKey); + // Use the persisted size only after resource authorization. Single ranges + // keep saved API text pages bounded all the way to disk or object storage. + const rawRange = req.headers.range; + const rangeSyntax = rawRange && /^bytes=(\d*)-(\d*)$/i.exec(rawRange); + const emptyRead = asset.byteSize === 0 && rangeSyntax?.[1] === "0"; + const ranges = rawRange && !emptyRead ? req.range(asset.byteSize) : undefined; + res.setHeader("Accept-Ranges", "bytes"); + if (/^[a-f0-9]{64}$/.test(asset.sha256)) res.setHeader("ETag", `"${asset.sha256}"`); + if (rawRange && (!rangeSyntax || (!rangeSyntax[1] && !rangeSyntax[2]) + || (!emptyRead && (!Array.isArray(ranges) || ranges.length !== 1)))) { + res.setHeader("Content-Range", `bytes */${asset.byteSize}`); + res.status(416).end(); + return; + } + const range = Array.isArray(ranges) ? ranges[0] : undefined; + const object = await storage.getObject(asset.companyId, asset.objectKey, range ? { range } : undefined); const responseContentType = asset.contentType || object.contentType || "application/octet-stream"; const mediaType = responseContentType.split(";", 1)[0]?.trim().toLowerCase() ?? ""; const inlineSafe = mediaType !== SVG_CONTENT_TYPE && isInlineAttachmentContentType(mediaType); res.setHeader("Content-Type", responseContentType); - res.setHeader("Content-Length", String(asset.byteSize || object.contentLength || 0)); + res.setHeader("Content-Length", String(range ? range.end - range.start + 1 : asset.byteSize || object.contentLength || 0)); + if (range) { + res.status(206); + res.setHeader("Content-Range", `bytes ${range.start}-${range.end}/${asset.byteSize}`); + } res.setHeader("Cache-Control", "private, max-age=60"); res.setHeader("X-Content-Type-Options", "nosniff"); if (!inlineSafe) { @@ -349,6 +368,7 @@ export function assetRoutes(db: Db, storage: StorageService) { object.stream.on("error", (err) => { next(err); }); + res.on("close", () => object.stream.destroy()); object.stream.pipe(res); }); diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts index 82093f10255..dab0e16d0fe 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts @@ -1,3 +1,4 @@ +import { createReadStream } from "node:fs"; import { publicChatTaskUrl } from "../chat-task-url.js"; import type { createAssignedMcpTools } from "./assigned-mcp-tools.js"; import { assertAssignableAgent } from "../agent-assignability.js"; @@ -27,10 +28,13 @@ import { workspaceFileResourceService } from "../workspace-file-resources.js"; import { badRequest, forbidden } from "../../errors.js"; import { searchRunnerApi } from "./runner-api-catalog.js"; import { executeRunnerApi, validateRunnerApiCall, RUNNER_API_MAX_BYTES, type RunnerApiFile } from "./runner-api-client.js"; +import { acquireRunnerApiResponseSlot, runnerApiCompanyCaptureMaxBytes, RUNNER_API_RESPONSE_MAX_BYTES, RUNNER_API_RESPONSE_RUN_MAX_BYTES, RunnerApiResponseLimitError } from "./runner-api-response-limits.js"; import { and, desc, eq, isNull, notInArray, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { activityLog, + assets, + runnerApiResponseReservations, agents, agentWakeupRequests, chatEndpoints, @@ -527,6 +531,8 @@ export class PaperclipRunnerToolAuthority { const token = createLocalAgentJwt(this.binding.agentId, this.binding.companyId, bound.actor.adapterType, this.binding.runId, bound.run.responsibleUserId); if (!token) throw new Error("Paperclip run authentication is unavailable"); const execute = async () => { + let reservationId: string | undefined; + let storageAttempted = false; const current = await this.#boundContext(); if (!runnerApiToolsEnabled(this.binding.companyId, this.binding.apiToolsEnabled)) throw new Error("paperclip_runner_tool_not_advertised"); return executeRunnerApi(input, { ...context, workMode: current.issue.workMode }, { @@ -537,10 +543,34 @@ export class PaperclipRunnerToolAuthority { validateRunnerApiCall(input, { ...context, workMode: fresh.issue.workMode }); }, readFile: (file) => this.#readApiFile(file), + reserveResponseCapture: async () => { + const reservation = await this.#reserveApiResponseCapture(); + reservationId = reservation.id; + return (completedBytes, cleaned) => reservation.settle(completedBytes, cleaned, storageAttempted); + }, saveResponse: async (bytes, contentType) => { + if (!reservationId) throw new Error("API response storage reservation is missing"); const storage = this.binding.storage ?? getStorageService(); - const saved = await storage.putFile({ companyId: this.binding.companyId, namespace: "runner-api", originalFilename: contentType.includes("json") ? "response.json" : "response.bin", contentType, body: bytes }); - const asset = await assetService(this.db).create(this.binding.companyId, { ...saved, createdByAgentId: this.binding.agentId }); + storageAttempted = true; + const saved = await storage.putFile({ companyId: this.binding.companyId, namespace: "runner-api", originalFilename: contentType.includes("json") ? "response.json" : "response.bin", contentType, ...(Buffer.isBuffer(bytes) ? { body: bytes } : { body: createReadStream(bytes.path), byteSize: bytes.byteSize, sha256: bytes.sha256 }) }); + const asset = await this.db.transaction(async tx => { + await this.#lockApiCaptureCompany(tx as unknown as Db); + const asset = await assetService(tx as unknown as Db).create(this.binding.companyId, { ...saved, createdByAgentId: this.binding.agentId }); + await tx.update(runnerApiResponseReservations).set({ assetId: asset.id, reservedBytes: asset.byteSize }).where(and(eq(runnerApiResponseReservations.id, reservationId!), eq(runnerApiResponseReservations.companyId, this.binding.companyId))); + return asset; + }).catch(async error => { + // A failed commit acknowledgement can still leave a committed + // asset. A locking read waits out that transaction before proving + // the reservation is unlinked and safe to compensate. + const reservation = await this.db.select({ assetId: runnerApiResponseReservations.assetId }) + .from(runnerApiResponseReservations).where(and(eq(runnerApiResponseReservations.id, reservationId!), eq(runnerApiResponseReservations.companyId, this.binding.companyId))).for("update") + .then(rows => rows[0]).catch(() => undefined); + if (reservation?.assetId === null) { + const removed = await storage.deleteObject(this.binding.companyId, saved.objectKey).then(() => true, () => false); + if (removed) storageAttempted = false; + } + throw error; + }); const activity = await persistActivity(this.db, { companyId: this.binding.companyId, actorType: "agent", actorId: this.binding.agentId, agentId: this.binding.agentId, runId: this.binding.runId, issueId: this.binding.issueId, action: "asset.created", entityType: "asset", entityId: asset.id, details: { source: "runner.call_api", byteSize: saved.byteSize } }); publishActivity(activity.publication); return { artifactId: asset.id, url: `/api/assets/${asset.id}/content`, contentType, byteSize: saved.byteSize, sha256: saved.sha256 }; @@ -589,6 +619,68 @@ export class PaperclipRunnerToolAuthority { return result; } + async #lockApiCaptureCompany(tx: Db): Promise { + await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`runner-api-capture:${this.binding.companyId}`}, 0))`); + } + + async #reserveApiResponseCapture() { + const release = acquireRunnerApiResponseSlot(this.binding.companyId); + let id: string; + try { + id = await this.db.transaction(async tx => { + await this.#lockApiCaptureCompany(tx as unknown as Db); + const locked = await this.#lockAuthorizedMutationContext(tx as unknown as Db); + const resultJson = record(locked.run.resultJson); + const used = resultJson.apiResponseCaptureBytes ?? 0; + if (typeof used !== "number" || !Number.isSafeInteger(used) || used < 0 + || used + RUNNER_API_RESPONSE_MAX_BYTES > RUNNER_API_RESPONSE_RUN_MAX_BYTES) { + throw new RunnerApiResponseLimitError("api_response_capture_limit", "This run has insufficient capture budget (4 GiB per run). Read an existing asset or narrow the query."); + } + // One company lock covers all runs/processes. Stored assets include + // snapshots from before reservations existed; attached reservations + // are excluded so completed snapshots are counted exactly once. + const [usage] = await tx.select({ bytes: sql` + coalesce((select sum(${assets.byteSize}) from ${assets} + where ${assets.companyId} = ${this.binding.companyId} + and ${assets.objectKey} like ${`${this.binding.companyId}/runner-api/%`}), 0) + + coalesce((select sum(${runnerApiResponseReservations.reservedBytes}) from ${runnerApiResponseReservations} + where ${runnerApiResponseReservations.companyId} = ${this.binding.companyId} + and ${runnerApiResponseReservations.assetId} is null), 0)` }).from(heartbeatRuns).where(eq(heartbeatRuns.id, this.binding.runId)); + const companyBytes = Number(usage.bytes); + if (!Number.isSafeInteger(companyBytes) || companyBytes < 0 || companyBytes + RUNNER_API_RESPONSE_MAX_BYTES > runnerApiCompanyCaptureMaxBytes()) { + throw new RunnerApiResponseLimitError("api_response_company_storage_limit", "Company API snapshot storage quota is exhausted. Read an existing asset or ask the operator to remove old snapshots or increase the quota."); + } + // Reserve worst-case bytes durably before creating a temporary file. + // Interrupted/failed captures keep the reservation, preventing retry + // loops and process restarts from resetting this run's disk-I/O budget. + await tx.update(heartbeatRuns).set({ resultJson: { ...resultJson, apiResponseCaptureBytes: used + RUNNER_API_RESPONSE_MAX_BYTES } }).where(eq(heartbeatRuns.id, this.binding.runId)); + const [reservation] = await tx.insert(runnerApiResponseReservations).values({ companyId: this.binding.companyId, runId: this.binding.runId, reservedBytes: RUNNER_API_RESPONSE_MAX_BYTES }).returning({ id: runnerApiResponseReservations.id }); + return reservation.id; + }); + } catch (error) { release(); throw error; } + let settled = false; + return { id, settle: async (completedBytes?: number, cleaned?: boolean, storageAttempted?: boolean) => { + if (settled) return; + settled = true; + try { + await this.db.transaction(async tx => { + await this.#lockApiCaptureCompany(tx as unknown as Db); + // Reclaim a handled pre-storage failure only after disk cleanup. + // Crashes, failed cleanup, and ambiguous storage failures keep their + // reservation until an operator reconciles the possible orphan data. + if (cleaned && !storageAttempted) await tx.delete(runnerApiResponseReservations).where(and(eq(runnerApiResponseReservations.id, id), isNull(runnerApiResponseReservations.assetId))); + if (completedBytes === undefined) return; + const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, this.binding.runId), eq(heartbeatRuns.companyId, this.binding.companyId))).for("update"); + if (!run) return; + const resultJson = record(run.resultJson); + const used = resultJson.apiResponseCaptureBytes; + if (typeof used !== "number" || !Number.isSafeInteger(used) || used < RUNNER_API_RESPONSE_MAX_BYTES) return; + await tx.update(heartbeatRuns).set({ resultJson: { ...resultJson, apiResponseCaptureBytes: used - RUNNER_API_RESPONSE_MAX_BYTES + completedBytes } }).where(eq(heartbeatRuns.id, run.id)); + }); + } finally { release(); } + } }; + } + async #readApiFile(file: RunnerApiFile): Promise<{ bytes: Buffer; filename: string; contentType: string }> { if (file.artifactId) { const asset = await assetService(this.db).getById(file.artifactId); diff --git a/server/src/services/native-runtime/runner-api-client.ts b/server/src/services/native-runtime/runner-api-client.ts index dec7582012f..0b8110c0de8 100644 --- a/server/src/services/native-runtime/runner-api-client.ts +++ b/server/src/services/native-runtime/runner-api-client.ts @@ -1,8 +1,11 @@ import { z } from "zod"; import { badRequest, forbidden, unprocessable } from "../../errors.js"; import { runnerApiOperation, type RunnerApiOperation } from "./runner-api-catalog.js"; +import { captureRunnerApiResponse, type RunnerApiResponseBody } from "./runner-api-response.js"; import { runnerApiRestriction } from "./runner-api-policy.js"; +import { RunnerApiResponseLimitError, RunnerApiResponseCleanupError, RUNNER_API_RESPONSE_MAX_BYTES } from "./runner-api-response-limits.js"; +/** Request/upload bound only. Response capture has a separate streamed limit. */ export const RUNNER_API_MAX_BYTES = 10 * 1024 * 1024; export const RUNNER_API_INLINE_BYTES = 24 * 1024; export const RUNNER_API_TIMEOUT_MS = 30_000; @@ -18,6 +21,10 @@ export const runnerApiCallSchema = z.object({ body: z.unknown().optional(), contentType: z.string().max(120).optional(), files: z.array(fileSchema).max(10).optional(), + responseText: z.object({ + offsetBytes: z.number().int().min(0).max(Number.MAX_SAFE_INTEGER).optional(), + limitBytes: z.number().int().min(4).max(RUNNER_API_INLINE_BYTES).optional(), + }).strict().optional(), }).strict(); export type RunnerApiCall = z.infer; export type RunnerApiFile = z.infer; @@ -34,8 +41,10 @@ export interface RunnerApiIo { fetch?: typeof fetch; /** Revalidate server-owned authority after asynchronous file preparation. */ beforeDispatch?(): Promise; + /** Reserve before spilling. Settle completed bytes; failed captures retain their reservation. */ + reserveResponseCapture?(): Promise<(completedBytes?: number, cleanupSucceeded?: boolean) => Promise>; readFile(file: RunnerApiFile): Promise<{ bytes: Buffer; filename: string; contentType: string }>; - saveResponse(bytes: Buffer, contentType: string): Promise>; + saveResponse(bytes: RunnerApiResponseBody, contentType: string): Promise>; } export function validateRunnerApiCall(value: unknown, context: RunnerApiContext): { input: RunnerApiCall; operation: RunnerApiOperation } { @@ -44,6 +53,7 @@ export function validateRunnerApiCall(value: unknown, context: RunnerApiContext) const input = parsed.data; const operation = runnerApiOperation(input.operationId); if (operation.transport !== "rest") throw unprocessable("This endpoint requires its existing protocol client; call_api supports REST only"); + if (input.responseText && operation.method !== "GET") throw badRequest("responseText requires a GET operation; read the saved artifact instead of repeating a mutation"); const restriction = runnerApiRestriction(operation.method, operation.path); if (restriction) throw forbidden(restriction); if (!operation.allowedModes.includes(context.workMode)) throw forbidden("call_api permits only reads in Ask and Plan modes; use the permitted dedicated tools"); @@ -139,10 +149,19 @@ export async function readBoundedResponse(response: Response, maxBytes = RUNNER_ } export async function executeRunnerApi(input: RunnerApiCall, context: RunnerApiContext, io: RunnerApiIo) { - const { operation } = validateRunnerApiCall(input, context); + const { operation, input: validated } = validateRunnerApiCall(input, context); + input = validated; const url = runnerApiUrl(operation, input, context, io.apiUrl); if (!io.token) throw new Error("Paperclip run authentication is unavailable"); const headers = new Headers({ Authorization: `Bearer ${io.token}`, "X-Paperclip-Run-Id": context.runId }); + const textOffset = input.responseText?.offsetBytes ?? 0; + const textLimit = input.responseText?.limitBytes ?? RUNNER_API_INLINE_BYTES; + const assetTextRange = operation.operationId === "GET /api/assets/{assetId}/content" + ? { start: Math.max(0, textOffset - 1), end: Math.min(Number.MAX_SAFE_INTEGER, textOffset + textLimit) } + : undefined; + // One byte before the requested offset permits an EOF read; one byte after + // the page lets the decoder trim an incomplete UTF-8 code point. + if (assetTextRange) headers.set("Range", `bytes=${assetTextRange.start}-${assetTextRange.end}`); let body: BodyInit | undefined; const contentType = input.contentType ?? (input.files?.length ? "multipart/form-data" : "application/json"); if (/\r|\n/.test(contentType)) throw badRequest("Invalid content type"); @@ -177,29 +196,118 @@ export async function executeRunnerApi(input: RunnerApiCall, context: RunnerApiC } let response: Response; let bytes: Buffer; + let captured: Awaited> | undefined; + let settleCapture: ((completedBytes?: number, cleanupSucceeded?: boolean) => Promise) | undefined; + const disposeCapture = async (priorCleanupSucceeded = true) => { + let cleaned = false; + try { await captured?.dispose(); cleaned = priorCleanupSucceeded; } + finally { await settleCapture?.(captured?.byteSize, cleaned); } + }; + const saveResponse = async (body: RunnerApiResponseBody, type: string) => { + // Small binary responses also create assets and need storage admission. + settleCapture ??= await io.reserveResponseCapture?.(); + return io.saveResponse(body, type); + }; await io.beforeDispatch?.(); + const controller = new AbortController(); + // Bound connection setup and stalled body reads, not the total download time. + let timeout = setTimeout(() => controller.abort(), RUNNER_API_TIMEOUT_MS); try { - response = await (io.fetch ?? fetch)(url, { method: operation.method, headers, body, redirect: "manual", signal: AbortSignal.timeout(RUNNER_API_TIMEOUT_MS) }); - bytes = await readBoundedResponse(response); - } catch { - return { ok: false, status: null, operationId: operation.operationId, error: "api_transport_failure", outcome: ["GET", "HEAD", "OPTIONS"].includes(operation.method) ? "read_failed" : "unknown", guidance: "Inspect current state before retrying a mutation; it may already have succeeded." }; - } - const type = response.headers.get("content-type") ?? "application/octet-stream"; - const mutation = !["GET", "HEAD", "OPTIONS"].includes(operation.method); - // A server can commit a write before an error response or redirect. Preserve - // the HTTP evidence, but never describe that write as definitely not applied. - const uncertain = mutation && (response.status >= 500 || response.status === 408 || (response.status >= 300 && response.status < 400)); - const uncertainty = { outcome: "unknown", guidance: "Inspect current state before retrying this mutation; it may already have succeeded." }; - const base = { ok: response.ok, status: response.status, operationId: operation.operationId, contentType: type, retryAfter: response.headers.get("retry-after"), ...(uncertain ? uncertainty : {}) }; - if (response.status >= 300 && response.status < 400) return { ...base, ok: false, error: "api_redirect_not_followed" }; - if (!bytes.length) return { ...base, data: null }; - if (bytes.length > RUNNER_API_INLINE_BYTES || !/json|^text\//i.test(type)) { - return { ...base, artifact: await io.saveResponse(bytes, type), byteSize: bytes.length, preview: /json|^text\//i.test(type) ? bytes.subarray(0, 2000).toString("utf8") : null }; - } - const text = bytes.toString("utf8"); - if (/json/i.test(type)) { - try { return { ...base, data: JSON.parse(text) as unknown }; } - catch { return { ...base, ok: false, error: "invalid_json_response", data: text, ...(mutation ? uncertainty : {}) }; } - } - return { ...base, data: text }; + response = await (io.fetch ?? fetch)(url, { method: operation.method, headers, body, redirect: "manual", signal: controller.signal }); + clearTimeout(timeout); + if (assetTextRange && response.ok) { + timeout = setTimeout(() => controller.abort(), RUNNER_API_TIMEOUT_MS); + bytes = await readBoundedResponse(response, textLimit + 2); + } else { + captured = await captureRunnerApiResponse(response, RUNNER_API_INLINE_BYTES, controller, RUNNER_API_TIMEOUT_MS, io.beforeDispatch, { + beforeSpill: async () => { settleCapture = await io.reserveResponseCapture?.(); }, + }); + // Read only the requested window (+ one UTF-8 lookahead byte) into memory. + bytes = await captured.read(input.responseText && response.ok ? textOffset : 0, + input.responseText ? textLimit + 1 : RUNNER_API_INLINE_BYTES); + } + } catch (error) { + await disposeCapture(!(error instanceof RunnerApiResponseCleanupError)); + return { ok: false, status: null, operationId: operation.operationId, + error: error instanceof RunnerApiResponseLimitError ? error.code : "api_transport_failure", + ...(error instanceof RunnerApiResponseLimitError ? { message: error.message, maxResponseBytes: RUNNER_API_RESPONSE_MAX_BYTES } : {}), + outcome: ["GET", "HEAD", "OPTIONS"].includes(operation.method) ? "read_failed" : "unknown", guidance: "Inspect current state before retrying a mutation; it may already have succeeded." }; + } finally { clearTimeout(timeout); } + try { + const type = response.headers.get("content-type") ?? "application/octet-stream"; + const mutation = !["GET", "HEAD", "OPTIONS"].includes(operation.method); + // A server can commit a write before an error response or redirect. Preserve + // the HTTP evidence, but never describe that write as definitely not applied. + const uncertain = mutation && (response.status >= 500 || response.status === 408 || (response.status >= 300 && response.status < 400)); + const uncertainty = { outcome: "unknown", guidance: "Inspect current state before retrying this mutation; it may already have succeeded." }; + const base = { ok: response.ok, status: response.status, operationId: operation.operationId, contentType: type, retryAfter: response.headers.get("retry-after"), ...(uncertain ? uncertainty : {}) }; + if (response.status >= 300 && response.status < 400) return { ...base, ok: false, error: "api_redirect_not_followed" }; + let totalBytes = captured?.byteSize ?? bytes.length; + let baseOffset = captured && response.ok ? textOffset : 0; + if (assetTextRange && response.ok) { + const range = /^bytes (\d+)-(\d+)\/(\d+)$/.exec(response.headers.get("content-range") ?? ""); + if (response.status === 200 && bytes.length === 0 && textOffset === 0) { + // An empty asset has no satisfiable byte range; its route returns 200. + } else if (response.status !== 206 || !range) { + return { ...base, ok: false, error: "response_text_invalid_range" }; + } else { + const [start, end, total] = range.slice(1).map(Number); + if (![start, end, total].every(Number.isSafeInteger) || start !== assetTextRange.start + || total < 1 || end !== Math.min(assetTextRange.end, total - 1) + || end < start || bytes.length !== end - start + 1) { + return { ...base, ok: false, error: "response_text_invalid_range" }; + } + baseOffset = start; + totalBytes = total; + } + } + if (input.responseText) { + if (!/json|^text\//i.test(type)) return { ...base, ok: false, error: "response_text_requires_text_content" }; + // HTTP denials are evidence, not text pages of an authorized asset. + const offsetBytes = response.ok ? textOffset : 0; + const localOffset = offsetBytes - baseOffset; + if (offsetBytes > totalBytes || localOffset < 0 || (localOffset < bytes.length && (bytes[localOffset] & 0xc0) === 0x80)) { + return { ...base, ok: false, error: "response_text_invalid_offset", byteSize: totalBytes }; + } + let end = Math.min(bytes.length, localOffset + textLimit); + // Do not cut a UTF-8 code point. The next offset always starts a whole one. + while (end < bytes.length && end > localOffset && (bytes[end] & 0xc0) === 0x80) end--; + if (end === localOffset && end < bytes.length) return { ...base, ok: false, error: "response_text_invalid_utf8" }; + let data: string; + try { + data = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }).decode(bytes.subarray(localOffset, end)); + } catch { + return { ...base, ok: false, error: "response_text_invalid_utf8" }; + } + const nextOffset = baseOffset + end; + const artifact = captured && captured.byteSize > RUNNER_API_INLINE_BYTES + ? await saveResponse(captured.body, type) : undefined; + return { ...base, data, ...(artifact ? { artifact, guidance: "Continue reading this saved artifact with GET /api/assets/{assetId}/content and responseText.nextOffsetBytes for a stable snapshot." } : {}), responseText: { offsetBytes, nextOffsetBytes: nextOffset < totalBytes ? nextOffset : null, totalBytes } }; + } + if (assetTextRange && response.ok && (totalBytes > RUNNER_API_INLINE_BYTES || !/json|^text\//i.test(type))) { + // This content is already an authorized, stable asset. Return its existing + // reference after a bounded preview; never download and duplicate it. + const assetId = input.pathParams!.assetId; + const sha256 = /^"([a-f0-9]{64})"$/.exec(response.headers.get("etag") ?? "")?.[1]; + return { ...base, artifact: { artifactId: assetId, url: `/api/assets/${assetId}/content`, contentType: type, byteSize: totalBytes, ...(sha256 ? { sha256 } : {}) }, + byteSize: totalBytes, preview: /json|^text\//i.test(type) ? bytes.subarray(0, 2000).toString("utf8") : null, + guidance: "Read this asset with responseText: { offsetBytes: 0 }. Continue with responseText.nextOffsetBytes until null." }; + } + if (!totalBytes) return { ...base, data: null }; + if (totalBytes > RUNNER_API_INLINE_BYTES || !/json|^text\//i.test(type)) { + return { ...base, artifact: await saveResponse(captured?.body ?? bytes, type), byteSize: totalBytes, preview: /json|^text\//i.test(type) ? bytes.subarray(0, 2000).toString("utf8") : null, + ...(/json|^text\//i.test(type) ? { guidance: "Read the saved artifact with GET /api/assets/{assetId}/content and responseText: { offsetBytes: 0 }. Continue with responseText.nextOffsetBytes until null. Do not repeat a mutation to read its response." } : {}) }; + } + const text = bytes.toString("utf8"); + if (/json/i.test(type)) { + try { return { ...base, data: JSON.parse(text) as unknown }; } + catch { return { ...base, ok: false, error: "invalid_json_response", data: text, ...(mutation ? uncertainty : {}) }; } + } + return { ...base, data: text }; + } catch (error) { + if (!(error instanceof RunnerApiResponseLimitError)) throw error; + return { ok: false, status: response.status, operationId: operation.operationId, error: error.code, message: error.message, + outcome: ["GET", "HEAD", "OPTIONS"].includes(operation.method) ? "read_failed" : "unknown", + guidance: "Inspect current state before retrying a mutation; it may already have succeeded." }; + } finally { await disposeCapture(); } } diff --git a/server/src/services/native-runtime/runner-api-response-limits.ts b/server/src/services/native-runtime/runner-api-response-limits.ts new file mode 100644 index 00000000000..7ae151a1994 --- /dev/null +++ b/server/src/services/native-runtime/runner-api-response-limits.ts @@ -0,0 +1,39 @@ +export const RUNNER_API_RESPONSE_MAX_BYTES = 1024 * 1024 * 1024; +export const RUNNER_API_RESPONSE_RUN_MAX_BYTES = 4 * RUNNER_API_RESPONSE_MAX_BYTES; +export const RUNNER_API_RESPONSE_DEADLINE_MS = 10 * 60_000; +export function runnerApiCompanyCaptureMaxBytes(): number { + const configured = Number(process.env.PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES); + return Number.isSafeInteger(configured) && configured >= RUNNER_API_RESPONSE_MAX_BYTES + ? configured : 20 * RUNNER_API_RESPONSE_MAX_BYTES; +} + +/** Cleanup failure must keep the durable reservation for possible orphan bytes. */ +export class RunnerApiResponseCleanupError extends Error { + constructor(cause: unknown) { super("Failed to remove captured API response", { cause }); } +} + +export class RunnerApiResponseLimitError extends Error { + constructor(readonly code: string, message: string) { super(message); } +} + +// Hold a slot until the temporary file is removed, including storage upload. +// These limits are per server process; the run byte budget is database-backed. +const activeByCompany = new Map(); +let active = 0; +export function acquireRunnerApiResponseSlot(companyId: string): () => void { + const companyActive = activeByCompany.get(companyId) ?? 0; + if (active >= 4 || companyActive >= 2) { + throw new RunnerApiResponseLimitError("api_response_capture_busy", "Large API response capture is busy. Wait for another capture to finish before retrying a read."); + } + active++; + activeByCompany.set(companyId, companyActive + 1); + let released = false; + return () => { + if (released) return; + released = true; + active--; + const remaining = activeByCompany.get(companyId)! - 1; + if (remaining) activeByCompany.set(companyId, remaining); + else activeByCompany.delete(companyId); + }; +} diff --git a/server/src/services/native-runtime/runner-api-response.test.ts b/server/src/services/native-runtime/runner-api-response.test.ts new file mode 100644 index 00000000000..35b40c6db8b --- /dev/null +++ b/server/src/services/native-runtime/runner-api-response.test.ts @@ -0,0 +1,142 @@ +import { createHash } from "node:crypto"; +import { access, readdir } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { describe, expect, it, vi } from "vitest"; +import { captureRunnerApiResponse } from "./runner-api-response.js"; +import { acquireRunnerApiResponseSlot, runnerApiCompanyCaptureMaxBytes, RUNNER_API_RESPONSE_MAX_BYTES } from "./runner-api-response-limits.js"; + +const controller = () => new AbortController(); +describe("streamed API response capture", () => { + it("rejects an oversized Content-Length before reading or reserving disk", async () => { + let cancelled = false; + let reserved = false; + const abort = controller(); + const response = new Response(new ReadableStream({ cancel() { cancelled = true; } }), { headers: { "content-length": String(RUNNER_API_RESPONSE_MAX_BYTES + 1) } }); + await expect(captureRunnerApiResponse(response, 24 * 1024, abort, 1000, undefined, { + beforeSpill: async () => { reserved = true; }, + })).rejects.toMatchObject({ code: "api_response_too_large" }); + expect(cancelled).toBe(true); + expect(abort.signal.aborted).toBe(true); + expect(reserved).toBe(false); + }); + it.each([undefined, "1"])("counts streamed bytes with Content-Length %s and removes partial files", async contentLength => { + const before = new Set(await readdir(tmpdir())); + let cancelled = false; + let reserved = 0; + const response = new Response(new ReadableStream({ + pull(target) { target.enqueue(Buffer.alloc(4096)); }, + cancel() { cancelled = true; }, + }), { headers: contentLength ? { "content-length": contentLength } : {} }); + await expect(captureRunnerApiResponse(response, 4096, controller(), 1000, undefined, { + maxBytes: 16 * 1024, + beforeSpill: async () => { reserved++; }, + })).rejects.toMatchObject({ code: "api_response_too_large" }); + expect(cancelled).toBe(true); + expect(reserved).toBe(1); + expect((await readdir(tmpdir())).filter(name => name.startsWith("paperclip-api-response-") && !before.has(name))).toEqual([]); + }); + it("accepts exactly the capture limit", async () => { + const captured = await captureRunnerApiResponse(new Response(Buffer.alloc(8192)), 4096, controller(), 1000, undefined, { maxBytes: 8192 }); + try { expect(captured.byteSize).toBe(8192); } + finally { await captured.dispose(); } + }); + it("stops an active endless stream at the total deadline", async () => { + let cancelled = false; + const response = new Response(new ReadableStream({ + async pull(target) { + await new Promise(resolve => setTimeout(resolve, 5)); + if (!cancelled) target.enqueue(Buffer.alloc(4096)); + }, + cancel() { cancelled = true; }, + })); + await expect(captureRunnerApiResponse(response, 4096, controller(), 1000, undefined, { deadlineMs: 30 })).rejects.toMatchObject({ code: "api_response_timeout" }); + expect(cancelled).toBe(true); + }); + it("spills a chunked response and reads only requested bytes, preserving its digest", async () => { + const chunk = Buffer.alloc(64 * 1024, 65); + const digest = createHash("sha256"); + let emitted = 0; + const response = new Response(new ReadableStream({ + pull(target) { + if (emitted++ === 200) return target.close(); + digest.update(chunk); + target.enqueue(chunk); + }, + })); + const captured = await captureRunnerApiResponse(response, 24 * 1024, controller(), 1000); + expect(Buffer.isBuffer(captured.body)).toBe(false); + if (Buffer.isBuffer(captured.body)) throw new Error("Expected a file"); + const path = captured.body.path; + try { + expect(captured.body).toMatchObject({ byteSize: 200 * chunk.length, sha256: digest.digest("hex") }); + expect(await captured.read(11 * 1024 * 1024, 4)).toEqual(Buffer.from("AAAA")); + expect(await captured.read(captured.byteSize, 4)).toHaveLength(0); + } finally { await captured.dispose(); } + await expect(access(path)).rejects.toThrow(); + }); + it("keeps small bodies inline", async () => { + const captured = await captureRunnerApiResponse(new Response("small"), 24 * 1024, controller(), 1000); + try { expect(captured.body).toEqual(Buffer.from("small")); } + finally { await captured.dispose(); } + }); + it("stops a long capture when its run loses authority", async () => { + let checks = 0; + let cancelled = false; + const response = new Response(new ReadableStream({ + pull(target) { target.enqueue(Buffer.alloc(64 * 1024)); }, + cancel() { cancelled = true; }, + })); + await expect(captureRunnerApiResponse(response, 24 * 1024, controller(), 1000, async () => { + if (++checks === 2) throw new Error("run stopped"); + })).rejects.toThrow("run stopped"); + expect(cancelled).toBe(true); + expect(checks).toBe(2); + }); + it("allows active downloads longer than the idle timeout", async () => { + let chunks = 0; + const response = new Response(new ReadableStream({ + async pull(target) { + await new Promise(resolve => setTimeout(resolve, 20)); + if (chunks++ === 5) target.close(); + else target.enqueue(Buffer.from("a")); + }, + })); + const captured = await captureRunnerApiResponse(response, 24 * 1024, controller(), 100); + try { expect(captured.body).toEqual(Buffer.from("aaaaa")); } + finally { await captured.dispose(); } + }); + it("cancels stalled bodies and aborts the HTTP request", async () => { + let cancelled = false; + const response = new Response(new ReadableStream({ cancel() { cancelled = true; } })); + const abort = controller(); + await expect(captureRunnerApiResponse(response, 24 * 1024, abort, 10)).rejects.toThrow("timed out"); + expect(cancelled).toBe(true); + expect(abort.signal.aborted).toBe(true); + }); +}); + +describe("large response concurrency", () => { + it("only accepts finite positive operator quotas, never unlimited settings", () => { + try { + for (const value of ["0", "-1", "Infinity", "NaN", "1", "9007199254740992"]) { + vi.stubEnv("PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES", value); + expect(runnerApiCompanyCaptureMaxBytes()).toBe(20 * RUNNER_API_RESPONSE_MAX_BYTES); + } + vi.stubEnv("PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES", String(2 * RUNNER_API_RESPONSE_MAX_BYTES)); + expect(runnerApiCompanyCaptureMaxBytes()).toBe(2 * RUNNER_API_RESPONSE_MAX_BYTES); + } finally { vi.unstubAllEnvs(); } + }); + it("limits each company and the server and releases slots idempotently", () => { + const releases: Array<() => void> = []; + try { + releases.push(acquireRunnerApiResponseSlot("a"), acquireRunnerApiResponseSlot("a")); + expect(() => acquireRunnerApiResponseSlot("a")).toThrow("busy"); + releases.push(acquireRunnerApiResponseSlot("b"), acquireRunnerApiResponseSlot("b")); + expect(() => acquireRunnerApiResponseSlot("c")).toThrow("busy"); + releases[0](); releases[0](); + releases.push(acquireRunnerApiResponseSlot("c")); + expect(() => acquireRunnerApiResponseSlot("d")).toThrow("busy"); + } finally { for (const release of releases) release(); } + acquireRunnerApiResponseSlot("a")(); + }); +}); diff --git a/server/src/services/native-runtime/runner-api-response.ts b/server/src/services/native-runtime/runner-api-response.ts new file mode 100644 index 00000000000..4416043eeca --- /dev/null +++ b/server/src/services/native-runtime/runner-api-response.ts @@ -0,0 +1,113 @@ +import { createHash } from "node:crypto"; +import { mkdtemp, open, rm, type FileHandle } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { RUNNER_API_RESPONSE_MAX_BYTES, RUNNER_API_RESPONSE_DEADLINE_MS, RunnerApiResponseLimitError, RunnerApiResponseCleanupError } from "./runner-api-response-limits.js"; + +export type RunnerApiResponseBody = Buffer | { path: string; byteSize: number; sha256: string }; + +/** Spill bounded responses to a private temporary file. + * Only the inline prefix and one HTTP chunk are held in memory. The caller must + * dispose the capture after saving it or reading its requested text window. + */ +export async function captureRunnerApiResponse(response: Response, inlineBytes: number, controller: AbortController, idleTimeoutMs: number, checkAuthority?: () => Promise, options: { + maxBytes?: number; + deadlineMs?: number; + beforeSpill?: () => Promise; +} = {}) { + const maxBytes = options.maxBytes ?? RUNNER_API_RESPONSE_MAX_BYTES; + const deadline = Date.now() + (options.deadlineMs ?? RUNNER_API_RESPONSE_DEADLINE_MS); + const reader = response.body?.getReader(); + let directory: string | undefined; + let file: FileHandle | undefined; + let path: string | undefined; + let byteSize = 0; + const prefix: Buffer[] = []; + const hash = createHash("sha256"); + const dispose = async () => { + try { + try { await file?.close(); } + finally { + file = undefined; + if (directory) await rm(directory, { recursive: true, force: true }); + } + } catch (error) { throw new RunnerApiResponseCleanupError(error); } + }; + try { + if (Number(response.headers.get("content-length")) > maxBytes) { + throw new RunnerApiResponseLimitError("api_response_too_large", `API response exceeds the ${maxBytes}-byte capture limit. Narrow the query or use the endpoint's pagination.`); + } + let checkedAt = Date.now(); + let checkedBytes = 0; + await checkAuthority?.(); + if (reader) { + while (true) { + controller.signal.throwIfAborted(); + if (Date.now() >= deadline) throw new RunnerApiResponseLimitError("api_response_timeout", "API response exceeded its capture deadline."); + let timeout: ReturnType | undefined; + const next = await Promise.race([ + reader.read(), + new Promise((_, reject) => { + timeout = setTimeout(() => { + controller.abort(); + reject(new RunnerApiResponseLimitError("api_response_timeout", "API response read timed out or exceeded its capture deadline.")); + }, Math.min(idleTimeoutMs, Math.max(0, deadline - Date.now()))); + }), + ]).finally(() => clearTimeout(timeout)); + if (next.done) break; + // Count decoded bytes before copying, hashing, or writing the chunk. + // Content-Length can be missing, false, or describe compressed bytes. + if (next.value.byteLength > maxBytes - byteSize) { + throw new RunnerApiResponseLimitError("api_response_too_large", `API response exceeds the ${maxBytes}-byte capture limit. Narrow the query or use the endpoint's pagination.`); + } + const chunk = Buffer.from(next.value); + if (!Number.isSafeInteger(byteSize + chunk.length)) throw new Error("API response byte offset exceeds numeric precision"); + hash.update(chunk); + if (!file && byteSize + chunk.length > inlineBytes) { + await options.beforeSpill?.(); + directory = await mkdtemp(join(tmpdir(), "paperclip-api-response-")); + path = join(directory, "body"); + file = await open(path, "wx+", 0o600); + for (const part of prefix) await file.writeFile(part); + prefix.length = 0; + } + if (file) await file.writeFile(chunk); + else prefix.push(chunk); + byteSize += chunk.length; + if (checkAuthority && (Date.now() - checkedAt >= 1000 || byteSize - checkedBytes >= 1024 * 1024)) { + await checkAuthority(); + checkedAt = Date.now(); + checkedBytes = byteSize; + } + } + } + if (Date.now() >= deadline) throw new RunnerApiResponseLimitError("api_response_timeout", "API response exceeded its capture deadline."); + await checkAuthority?.(); + const bytes = file ? undefined : Buffer.concat(prefix); + const body: RunnerApiResponseBody = path ? { path, byteSize, sha256: hash.digest("hex") } : bytes!; + return { + body, + byteSize, + async read(offset: number, limit: number): Promise { + const length = Math.min(limit, Math.max(0, byteSize - offset)); + if (!file) return bytes!.subarray(offset, offset + length); + const result = Buffer.alloc(length); + let read = 0; + while (read < length) { + const next = await file.read(result, read, length - read, offset + read); + if (!next.bytesRead) throw new Error("Saved API response was truncated"); + read += next.bytesRead; + } + return result; + }, + dispose, + }; + } catch (error) { + controller.abort(); + await dispose(); + throw error; + } finally { + await reader?.cancel().catch(() => {}); + reader?.releaseLock(); + } +} diff --git a/server/src/services/native-runtime/runner-api.integration.test.ts b/server/src/services/native-runtime/runner-api.integration.test.ts index 805a618c6b3..9a30b5cb049 100644 --- a/server/src/services/native-runtime/runner-api.integration.test.ts +++ b/server/src/services/native-runtime/runner-api.integration.test.ts @@ -1,14 +1,16 @@ import { existsSync } from "node:fs"; import { randomUUID } from "node:crypto"; -import { chmod, writeFile, symlink } from "node:fs/promises"; -import { join } from "node:path"; +import { chmod, writeFile, symlink, mkdir, open } from "node:fs/promises"; +import { join, dirname } from "node:path"; import { eq } from "drizzle-orm"; -import { documents, heartbeatRuns, issues, routineDocuments, routines } from "@paperclipai/db"; +import { assets, documents, heartbeatRuns, issues, projects, routineDocuments, routines, runnerApiResponseReservations } from "@paperclipai/db"; import { beforeAll, afterAll, beforeEach, afterEach, describe, expect, it, vi } from "vitest"; import { startRunnerApiTestServer } from "../../__tests__/helpers/runner-api-server.js"; import { createRunnerdCodexTransport, defaultCapabilityRunnerdBinary } from "../../vendor/paperclip-runner/index.js"; import { runnerApiCatalog } from "./runner-api-catalog.js"; import { registerRunnerPrpAuthority } from "../../realtime/runner-prp-ws.js"; +import { RUNNER_API_RESPONSE_MAX_BYTES, RUNNER_API_RESPONSE_RUN_MAX_BYTES } from "./runner-api-response-limits.js"; +import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js"; describe("runner API against real HTTP routes", () => { let server: Awaited>; @@ -248,6 +250,243 @@ else if(m.id!==undefined) send({id:m.id,result:{}}); } finally { getObject.mockRestore(); } }); + it("retrieves every byte of a saved large response without creating an artifact loop", async () => { + const fixture = await server.fixture(); + const foreign = await server.fixture(); + const text = JSON.stringify({ padding: "🧭é".repeat(6000), evidence: "last-field-is-readable" }); + const stored = await server.storage.putFile({ companyId: fixture.companyId, namespace: "eval", originalFilename: "evidence.json", contentType: "application/json", body: Buffer.from(text) }); + const [asset] = await server.db.insert(assets).values({ ...stored, companyId: fixture.companyId, createdByAgentId: fixture.agentId }).returning(); + const call = (authority: typeof fixture.authority, assetId: string, responseText?: object) => authority.execute({ tool: "call_api", callId: randomUUID(), arguments: { operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId }, ...(responseText ? { responseText } : {}) } }) as Promise; + const initial = await call(fixture.authority, asset.id); + expect(initial).toMatchObject({ status: 206, artifact: { artifactId: asset.id, sha256: asset.sha256, byteSize: Buffer.byteLength(text) } }); + const before = (await fixture.snapshot()).assets.length; + const reads = vi.spyOn(server.storage, "getObject"); + let offsetBytes = 0; + let result = ""; + try { + do { + const page = await call(fixture.authority, initial.artifact.artifactId, { offsetBytes, limitBytes: 4096 }); + expect(page).toMatchObject({ status: 206, responseText: { offsetBytes, totalBytes: Buffer.byteLength(text) } }); + result += page.data; + offsetBytes = page.responseText.nextOffsetBytes; + } while (offsetBytes !== null); + expect(JSON.parse(result)).toEqual(JSON.parse(text)); + expect(reads.mock.calls.length).toBeGreaterThan(1); + expect(reads.mock.calls.every(([, , options]) => options?.range)).toBe(true); + const storageBytes = reads.mock.calls.reduce((total, [, , options]) => total + options!.range!.end - options!.range!.start + 1, 0); + expect(storageBytes).toBeLessThan(Buffer.byteLength(text) + reads.mock.calls.length * 5); + expect((await fixture.snapshot()).assets).toHaveLength(before); + expect(await call(foreign.authority, initial.artifact.artifactId, { offsetBytes: 0 })).toMatchObject({ ok: false, status: 404 }); + } finally { + reads.mockRestore(); + } + }); + + it("captures a live response above ten MiB and reads its saved snapshot to EOF", async () => { + const fixture = await server.fixture(); + const foreign = await server.fixture(); + const description = "A".repeat(12 * 1024 * 1024) + "END-OF-LARGE-RESPONSE"; + await server.db.update(projects).set({ description }).where(eq(projects.id, fixture.projectId)); + const initial: any = await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: { + operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId }, responseText: { limitBytes: 8192 }, + } }); + expect(initial).toMatchObject({ ok: true, status: 200, responseText: { offsetBytes: 0, nextOffsetBytes: 8192 } }); + expect(initial.artifact.byteSize).toBeGreaterThan(12 * 1024 * 1024); + const saved = await server.db.select().from(assets).where(eq(assets.id, initial.artifact.artifactId)).then(rows => rows[0]); + const object = await server.storage.getObject(fixture.companyId, saved.objectKey); + const chunks: Buffer[] = []; + for await (const chunk of object.stream) chunks.push(chunk); + const expected = Buffer.concat(chunks); + expect(JSON.parse(expected.toString()).description).toBe(description); + const before = (await fixture.snapshot()).assets.length; + for (const offsetBytes of [0, 10 * 1024 * 1024 + 1, expected.length - 8192, expected.length]) { + const page: any = await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: { + operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: saved.id }, responseText: { offsetBytes, limitBytes: 8192 }, + } }); + expect(page).toMatchObject({ ok: true, status: 206, data: expected.subarray(offsetBytes, offsetBytes + 8192).toString(), + responseText: { offsetBytes, totalBytes: expected.length, nextOffsetBytes: offsetBytes + 8192 < expected.length ? offsetBytes + 8192 : null } }); + expect(page.artifact).toBeUndefined(); + } + expect((await fixture.snapshot()).assets).toHaveLength(before); + expect(await foreign.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: { + operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: saved.id }, responseText: { offsetBytes: 10 * 1024 * 1024 + 1 }, + } })).toMatchObject({ ok: false, status: 404 }); + const mutation = { tool: "call_api", callId: randomUUID(), arguments: { + operationId: "PATCH /api/projects/{id}", pathParams: { id: fixture.projectId }, body: { name: "Updated once" }, + } }; + const receipt: any = await fixture.authority.execute(mutation); + expect(receipt).toMatchObject({ ok: true, status: 200, artifact: { contentType: "application/json; charset=utf-8" } }); + expect(receipt.artifact.byteSize).toBeGreaterThan(12 * 1024 * 1024); + expect(await fixture.authority.execute(mutation)).toEqual(receipt); + const after = await fixture.snapshot(); + expect(after.assets).toHaveLength(before + 1); + expect(after.activity.filter(row => row.action === "project.updated")).toHaveLength(1); + const [run] = await server.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + expect((run.resultJson as Record).apiResponseCaptureBytes).toBe(initial.artifact.byteSize + receipt.artifact.byteSize); + }, 30_000); + + it("enforces the durable run budget while permitting small reads and saved pages", async () => { + const fixture = await server.fixture(); + await server.db.update(heartbeatRuns).set({ resultJson: { apiResponseCaptureBytes: RUNNER_API_RESPONSE_RUN_MAX_BYTES } }).where(eq(heartbeatRuns.id, fixture.runId)); + const call = () => fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: { + operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId }, + } }); + expect(await call()).toMatchObject({ ok: true }); + await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId)); + const before = (await fixture.snapshot()).assets.length; + expect(await call()).toMatchObject({ ok: false, error: "api_response_capture_limit", outcome: "read_failed" }); + expect((await fixture.snapshot()).assets).toHaveLength(before); + expect(await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: { + operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: fixture.artifactId }, responseText: { limitBytes: 4 }, + } })).toMatchObject({ ok: true }); + }); + + it("reserves the run budget atomically across simultaneous captures", async () => { + const fixture = await server.fixture(); + const used = RUNNER_API_RESPONSE_RUN_MAX_BYTES - RUNNER_API_RESPONSE_MAX_BYTES; + await server.db.update(heartbeatRuns).set({ resultJson: { apiResponseCaptureBytes: used } }).where(eq(heartbeatRuns.id, fixture.runId)); + await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId)); + let unblock!: () => void; + let started!: () => void; + const gate = new Promise(resolve => { unblock = resolve; }); + const saving = new Promise(resolve => { started = resolve; }); + const putFile = server.storage.putFile.bind(server.storage); + const spy = vi.spyOn(server.storage, "putFile").mockImplementation(async input => { + if (input.namespace === "runner-api") { started(); await gate; } + return putFile(input); + }); + const call = () => fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: { + operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId }, + } }); + const first = call(); + try { + await saving; + expect(await call()).toMatchObject({ ok: false, error: "api_response_capture_limit" }); + } finally { unblock(); spy.mockRestore(); } + const result: any = await first; + expect(result).toMatchObject({ ok: true }); + const [run] = await server.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + expect((run.resultJson as Record).apiResponseCaptureBytes).toBe(used + result.artifact.byteSize); + expect(await call()).toMatchObject({ ok: false, error: "api_response_capture_limit" }); + }); + + it("counts old snapshots and other-run reservations against the company quota, and reclaims deletions", async () => { + const fixture = await server.fixture(); + vi.stubEnv("PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES", String(2 * RUNNER_API_RESPONSE_MAX_BYTES)); + const [oldSnapshot] = await server.db.insert(assets).values({ companyId: fixture.companyId, provider: "local_disk", + objectKey: `${fixture.companyId}/runner-api/old-snapshot`, contentType: "text/plain", byteSize: RUNNER_API_RESPONSE_MAX_BYTES, + sha256: "old-snapshot", createdByAgentId: fixture.agentId }).returning(); + const [reservation] = await server.db.insert(runnerApiResponseReservations).values({ companyId: fixture.companyId, runId: null, reservedBytes: RUNNER_API_RESPONSE_MAX_BYTES }).returning(); + await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId)); + // A fresh authority instance cannot reset reservations from another run/process. + const authority = new PaperclipRunnerToolAuthority(server.db, fixture); + const call = () => authority.execute({ tool: "call_api", callId: randomUUID(), arguments: { + operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId }, + } }); + expect(await call()).toMatchObject({ ok: false, error: "api_response_company_storage_limit" }); + await server.db.delete(assets).where(eq(assets.id, oldSnapshot.id)); + const first: any = await call(); + expect(first).toMatchObject({ ok: true }); + expect(await call()).toMatchObject({ ok: false, error: "api_response_company_storage_limit" }); + await server.db.delete(assets).where(eq(assets.id, first.artifact.artifactId)); + expect(await server.db.select().from(runnerApiResponseReservations).where(eq(runnerApiResponseReservations.companyId, fixture.companyId))).toEqual([expect.objectContaining({ id: reservation.id })]); + expect(await call()).toMatchObject({ ok: true }); + }); + + it("keeps company storage reserved after an ambiguous upload failure", async () => { + const fixture = await server.fixture(); + await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId)); + const spy = vi.spyOn(server.storage, "putFile").mockRejectedValueOnce(new Error("storage disconnected")); + try { + await expect(fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: { + operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId }, + } })).rejects.toThrow("storage disconnected"); + } finally { spy.mockRestore(); } + expect(await server.db.select().from(runnerApiResponseReservations).where(eq(runnerApiResponseReservations.companyId, fixture.companyId))).toEqual([ + expect.objectContaining({ assetId: null, reservedBytes: RUNNER_API_RESPONSE_MAX_BYTES }), + ]); + }); + + it.each([false, true])("reconciles metadata failure without losing committed assets (commit=%s)", async committed => { + const fixture = await server.fixture(); + await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId)); + const putFile = server.storage.putFile.bind(server.storage); + const transaction = server.db.transaction.bind(server.db); + let objectKey = ""; + let transactionSpy: ReturnType | undefined; + const spy = vi.spyOn(server.storage, "putFile").mockImplementationOnce(async input => { + const saved = await putFile(input); + objectKey = saved.objectKey; + transactionSpy = vi.spyOn(server.db, "transaction").mockImplementationOnce(async (fn, config) => { + if (committed) await transaction(fn, config); + throw new Error("metadata transaction failed"); + }); + return saved; + }); + try { + await expect(fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: { + operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId }, + } })).rejects.toThrow("metadata transaction failed"); + } finally { spy.mockRestore(); transactionSpy?.mockRestore(); } + const reservations = await server.db.select().from(runnerApiResponseReservations).where(eq(runnerApiResponseReservations.companyId, fixture.companyId)); + expect(reservations).toHaveLength(committed ? 1 : 0); + if (committed) expect(reservations[0].assetId).toBeTruthy(); + expect((await server.storage.headObject(fixture.companyId, objectKey)).exists).toBe(committed); + }); + + it("shares company admission across simultaneous runs", async () => { + const fixture = await server.fixture(); + vi.stubEnv("PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES", String(RUNNER_API_RESPONSE_MAX_BYTES)); + const otherRunId = randomUUID(); + await server.db.insert(heartbeatRuns).values({ id: otherRunId, companyId: fixture.companyId, agentId: fixture.agentId, + status: "running", runtimeMode: "native", nativeIssueId: fixture.blockerId, contextSnapshot: { issueId: fixture.blockerId } }); + await server.db.update(issues).set({ status: "in_progress", executionRunId: otherRunId }).where(eq(issues.id, fixture.blockerId)); + const other = new PaperclipRunnerToolAuthority(server.db, { ...fixture, issueId: fixture.blockerId, runId: otherRunId }); + await server.db.update(projects).set({ description: "x".repeat(32 * 1024) }).where(eq(projects.id, fixture.projectId)); + const arguments_ = { operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId } }; + const results: any[] = await Promise.all([fixture.authority, other].map(authority => authority.execute({ tool: "call_api", callId: randomUUID(), arguments: arguments_ }))); + expect(results.filter(result => result.ok)).toHaveLength(1); + expect(results.find(result => !result.ok)).toMatchObject({ error: "api_response_company_storage_limit" }); + }); + + it("releases a cleaned failed capture's company reservation but retains its run charge", async () => { + const fixture = await server.fixture(); + let emitted = false; + const stream = new ReadableStream({ + async pull(target) { + if (!emitted) { emitted = true; target.enqueue(Buffer.alloc(32 * 1024)); } + else { await new Promise(resolve => setTimeout(resolve, 100)); target.error(new Error("connection reset")); } + }, + }); + const spy = vi.spyOn(globalThis, "fetch").mockResolvedValueOnce(new Response(stream, { headers: { "content-type": "text/plain" } })); + try { + expect(await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: { + operationId: "GET /api/projects/{id}", pathParams: { id: fixture.projectId }, + } })).toMatchObject({ ok: false, error: "api_transport_failure" }); + } finally { spy.mockRestore(); } + expect(await server.db.select().from(runnerApiResponseReservations).where(eq(runnerApiResponseReservations.companyId, fixture.companyId))).toEqual([]); + const [run] = await server.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + expect((run.resultJson as Record).apiResponseCaptureBytes).toBe(RUNNER_API_RESPONSE_MAX_BYTES); + }); + + it("persists and range-reads asset sizes above two GiB", async () => { + const fixture = await server.fixture(); + const offsetBytes = 3 * 1024 * 1024 * 1024; + const objectKey = `${fixture.companyId}/eval/sparse-large-response.txt`; + const path = join(server.root, "storage", objectKey); + await mkdir(dirname(path), { recursive: true }); + const file = await open(path, "wx"); + try { await file.write(Buffer.from("readable"), 0, 8, offsetBytes); } + finally { await file.close(); } + const [asset] = await server.db.insert(assets).values({ companyId: fixture.companyId, provider: "local_disk", objectKey, + contentType: "text/plain", byteSize: offsetBytes + 8, sha256: "sparse-fixture", createdByAgentId: fixture.agentId }).returning(); + expect(asset.byteSize).toBe(offsetBytes + 8); + const page = await fixture.authority.execute({ tool: "call_api", callId: randomUUID(), arguments: { + operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: asset.id }, responseText: { offsetBytes, limitBytes: 8 }, + } }); + expect(page).toMatchObject({ ok: true, data: "readable", responseText: { offsetBytes, totalBytes: offsetBytes + 8, nextOffsetBytes: null } }); + }); + it("contains workspace files, checks artifact ownership, and persists downloads", async () => { const fixture = await server.fixture(); const foreign = await server.fixture(); @@ -260,7 +499,7 @@ else if(m.id!==undefined) send({id:m.id,result:{}}); await expect(upload([{ path: "escape.txt" }])).rejects.toThrow(); await expect(upload([{ path: "sample.txt" }])).resolves.toMatchObject({ status: 201 }); const download = await fixture.authority.execute({ tool: "call_api", callId: "download", arguments: { operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: fixture.binaryArtifactId } } }) as any; - expect(download).toMatchObject({ status: 200, byteSize: 32000, artifact: { byteSize: 32000 } }); + expect(download).toMatchObject({ status: 206, byteSize: 32000, artifact: { artifactId: fixture.binaryArtifactId, byteSize: 32000 } }); expect((await fixture.snapshot()).assets).toEqual(expect.arrayContaining([expect.objectContaining({ id: download.artifact.artifactId, companyId: fixture.companyId, createdByAgentId: fixture.agentId })])); }); diff --git a/server/src/services/native-runtime/runner-api.test.ts b/server/src/services/native-runtime/runner-api.test.ts index 047bf1a9248..5cc31752644 100644 --- a/server/src/services/native-runtime/runner-api.test.ts +++ b/server/src/services/native-runtime/runner-api.test.ts @@ -11,6 +11,7 @@ import { validateRunnerApiCall, type RunnerApiIo, } from "./runner-api-client.js"; +import { RUNNER_API_RESPONSE_MAX_BYTES } from "./runner-api-response-limits.js"; const context = { companyId: "company-a", @@ -32,11 +33,30 @@ const io = (fetcher: typeof fetch): RunnerApiIo => ({ }), saveResponse: async (bytes, contentType) => ({ artifactId: "artifact-a", - byteSize: bytes.length, + byteSize: Buffer.isBuffer(bytes) ? bytes.length : bytes.byteSize, contentType, }), }); +describe("bounded response capture receipts", () => { + it.each([projects, createProject])("reports oversize evidence without retrying %s", async operationId => { + const fetcher = vi.fn(async () => new Response("", { headers: { "content-length": String(RUNNER_API_RESPONSE_MAX_BYTES + 1) } })); + const result = await executeRunnerApi({ operationId }, context, io(fetcher)); + expect(result).toMatchObject({ ok: false, error: "api_response_too_large", maxResponseBytes: RUNNER_API_RESPONSE_MAX_BYTES, + outcome: operationId === projects ? "read_failed" : "unknown" }); + expect(fetcher).toHaveBeenCalledTimes(1); + }); + it("settles the budget and removes the file even when saving fails", async () => { + const bytes = Buffer.alloc(32 * 1024); + const settle = vi.fn(async () => {}); + const input = io(async () => new Response(bytes)); + input.reserveResponseCapture = async () => settle; + input.saveResponse = async () => { throw new Error("storage unavailable"); }; + await expect(executeRunnerApi({ operationId: projects }, context, input)).rejects.toThrow("storage unavailable"); + expect(settle).toHaveBeenCalledExactlyOnceWith(bytes.length, true); + }); +}); + describe("runner API catalog", () => { it("accounts for unique operations with resolved request contracts", () => { const catalog = runnerApiCatalog(); @@ -591,6 +611,127 @@ describe("runner API request boundary", () => { ).toThrow("inline JSON object"); } }); + it("reads text windows from a live response and supplies stable snapshots", async () => { + const text = "prefix\n" + "🧭é\n".repeat(6000) + "END-OF-EVIDENCE"; + const saveResponse = vi.fn(io(fetch).saveResponse); + const request = vi.fn(async () => new Response(text, { headers: { "content-type": "application/json" } })); + let offsetBytes = 0; + let received = ""; + do { + const result = await executeRunnerApi({ operationId: projects, responseText: { offsetBytes, limitBytes: 4096 } }, context, { ...io(request), saveResponse }) as any; + expect(result).toMatchObject({ ok: true, responseText: { offsetBytes, totalBytes: Buffer.byteLength(text) } }); + expect(Buffer.byteLength(result.data)).toBeLessThanOrEqual(4096); + received += result.data; + offsetBytes = result.responseText.nextOffsetBytes; + } while (offsetBytes !== null); + expect(received).toBe(text); + expect(saveResponse).toHaveBeenCalled(); + expect(request.mock.calls.every(([, options]) => new Headers(options?.headers).get("Authorization") === "Bearer private-agent-token")).toBe(true); + }); + it.each(["text/plain", "application/octet-stream"])("does not duplicate an unpaged multi-gigabyte asset (%s)", async contentType => { + const totalBytes = 3 * 1024 * 1024 * 1024; + const saveResponse = vi.fn(io(fetch).saveResponse); + const request = vi.fn(async (_url, init) => { + expect(new Headers(init?.headers).get("range")).toBe("bytes=0-24576"); + return new Response(Buffer.alloc(24577, 65), { status: 206, headers: { + "content-type": contentType, "content-range": `bytes 0-24576/${totalBytes}`, etag: `"${"a".repeat(64)}"`, + } }); + }); + const result = await executeRunnerApi({ operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: "existing" } }, context, { ...io(request), saveResponse }); + expect(result).toMatchObject({ ok: true, status: 206, byteSize: totalBytes, artifact: { artifactId: "existing", byteSize: totalBytes, sha256: "a".repeat(64) } }); + expect(saveResponse).not.toHaveBeenCalled(); + expect(request).toHaveBeenCalledTimes(1); + }); + it("reads a page beyond ten MiB in a multi-gigabyte asset", async () => { + const offsetBytes = 3 * 1024 * 1024 * 1024; + const totalBytes = offsetBytes + 100; + const request = vi.fn(async (_url, init) => { + expect(new Headers(init?.headers).get("range")).toBe(`bytes=${offsetBytes - 1}-${offsetBytes + 4}`); + return new Response("abcdef", { status: 206, headers: { + "content-type": "text/plain", "content-range": `bytes ${offsetBytes - 1}-${offsetBytes + 4}/${totalBytes}`, + } }); + }); + expect(await executeRunnerApi({ operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: "large" }, responseText: { offsetBytes, limitBytes: 4 } }, context, io(request))).toMatchObject({ + ok: true, data: "bcde", responseText: { offsetBytes, nextOffsetBytes: offsetBytes + 4, totalBytes }, + }); + }); + it.each([true, false])("saves responses above ten MiB (known length: %s)", async knownLength => { + const bytes = Buffer.alloc(12 * 1024 * 1024, 65); + const request = vi.fn(async () => new Response(bytes, { headers: { + "content-type": "text/plain", ...(knownLength ? { "content-length": String(bytes.length) } : {}), + } })); + const result = await executeRunnerApi({ operationId: projects }, context, io(request)); + expect(result).toMatchObject({ ok: true, artifact: { byteSize: bytes.length }, byteSize: bytes.length }); + }); + it("pages a twelve MiB asset with linear transfer and UTF-8 boundaries", async () => { + const bytes = Buffer.from("🧭é".repeat(Math.floor(12 * 1024 * 1024 / 6))); + let transferred = 0; + const request = vi.fn(async (_url, init) => { + const range = new Headers(init?.headers).get("range"); + const match = /^bytes=(\d+)-(\d+)$/.exec(range ?? ""); + if (!match) { transferred += bytes.length; return new Response(bytes, { headers: { "content-type": "text/plain" } }); } + const start = Number(match[1]), end = Math.min(Number(match[2]), bytes.length - 1); + const part = bytes.subarray(start, end + 1); transferred += part.length; + return new Response(part, { status: 206, headers: { "content-type": "text/plain", "content-range": `bytes ${start}-${end}/${bytes.length}` } }); + }); + let offsetBytes: number | null = 0; + const parts: string[] = []; + do { + const page: any = await executeRunnerApi({ operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: "large" }, responseText: { offsetBytes } }, context, io(request)); + expect(page.ok).toBe(true); + parts.push(page.data); offsetBytes = page.responseText.nextOffsetBytes; + } while (offsetBytes !== null); + expect(parts.join("")).toBe(bytes.toString()); + expect(transferred).toBeLessThan(bytes.length + request.mock.calls.length * 5); + expect(request.mock.calls.every(([, init]) => new Headers(init?.headers).has("range"))).toBe(true); + }); + it.each([ + ["bytes 4-8/20", "12345"], // wrong start + ["bytes 3-7/20", "12345"], // early end + ["bytes 3-8/20", "12345"], // truncated body + ["bytes 3-8/9007199254740992", "123456"], // imprecise total + ["bytes 3-8/*", "123456"], // unknown total + [null, "123456"], + ])("rejects inconsistent partial asset receipts: %s", async (range, body) => { + const result = await executeRunnerApi({ operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: "saved" }, responseText: { offsetBytes: 4, limitBytes: 4 } }, context, io(async () => new Response(body, { status: 206, headers: { "content-type": "text/plain", ...(range ? { "content-range": range } : {}) } }))); + expect(result).toMatchObject({ ok: false, error: "response_text_invalid_range" }); + }); + it("accepts EOF and rejects an offset inside a UTF-8 code point in ranged assets", async () => { + const call = (offsetBytes: number, bytes: Buffer, range: string) => executeRunnerApi({ operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: "saved" }, responseText: { offsetBytes, limitBytes: 4 } }, context, io(async () => new Response(new Uint8Array(bytes), { status: 206, headers: { "content-type": "text/plain", "content-range": range } }))); + expect(await call(2, Buffer.from([0xa9]), "bytes 1-1/2")).toMatchObject({ ok: true, data: "", responseText: { nextOffsetBytes: null, totalBytes: 2 } }); + expect(await call(1, Buffer.from("é"), "bytes 0-1/2")).toMatchObject({ ok: false, error: "response_text_invalid_offset" }); + }); + it("does not silently accept an asset server that ignores the byte range", async () => { + const result = await executeRunnerApi({ operationId: "GET /api/assets/{assetId}/content", pathParams: { assetId: "saved" }, responseText: { limitBytes: 4 } }, context, io(async () => new Response("unbounded", { headers: { "content-type": "text/plain" } }))); + expect(result).toMatchObject({ ok: false, error: "api_transport_failure" }); + }); + it.each([{ offsetBytes: -1 }, { limitBytes: 0 }, { limitBytes: 3 }, { limitBytes: 24577 }, { offsetBytes: 0.5 }, { offsetBytes: Number.MAX_SAFE_INTEGER + 1 }])("rejects invalid text windows before dispatch: %j", async responseText => { + const request = vi.fn(); + await expect(executeRunnerApi({ operationId: projects, responseText }, context, io(request))).rejects.toThrow("Invalid call_api arguments"); + expect(request).not.toHaveBeenCalled(); + }); + it("does not page mutation responses by repeating the mutation", async () => { + const request = vi.fn(); + await expect(executeRunnerApi({ operationId: createProject, body: { name: "Once" }, responseText: {} }, context, io(request))).rejects.toThrow("GET"); + expect(request).not.toHaveBeenCalled(); + }); + it("rejects binary, invalid UTF-8 and offsets that split text", async () => { + for (const [body, type, offsetBytes] of [ + [Buffer.from("binary"), "application/octet-stream", 0], + [Buffer.from([0xff]), "text/plain", 0], + [Buffer.concat([Buffer.from([0xc2]), Buffer.alloc(24576, 0x80)]), "text/plain", 0], + [Buffer.from("é"), "text/plain", 1], + [Buffer.from("short"), "text/plain", 6], + ] as const) { + const saveResponse = vi.fn(io(fetch).saveResponse); + const result = await executeRunnerApi({ operationId: projects, responseText: { offsetBytes } }, context, { ...io(async () => new Response(body, { headers: { "content-type": type } })), saveResponse }); + expect(result).toMatchObject({ ok: false, error: expect.stringContaining("response_text") }); + expect(saveResponse).not.toHaveBeenCalled(); + } + }); + it("preserves denials for bounded text reads", async () => { + expect(await executeRunnerApi({ operationId: projects, responseText: {} }, context, io(async () => Response.json({ error: "denied" }, { status: 403 })))).toMatchObject({ ok: false, status: 403, data: '{"error":"denied"}' }); + }); it("bounds streamed responses even without content-length", async () => { await expect( readBoundedResponse(new Response("too large"), 3), diff --git a/server/src/storage/local-disk-provider.ts b/server/src/storage/local-disk-provider.ts index 56b35495452..df3ed649fa6 100644 --- a/server/src/storage/local-disk-provider.ts +++ b/server/src/storage/local-disk-provider.ts @@ -47,8 +47,10 @@ export function createLocalDiskStorageProvider(baseDir: string): StorageProvider await fs.mkdir(dir, { recursive: true }); const tempPath = `${targetPath}.tmp-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`; - await fs.writeFile(tempPath, input.body); - await fs.rename(tempPath, targetPath); + try { + await fs.writeFile(tempPath, input.body); + await fs.rename(tempPath, targetPath); + } finally { await fs.rm(tempPath, { force: true }); } }, async getObject(input): Promise { diff --git a/server/src/storage/s3-multipart.ts b/server/src/storage/s3-multipart.ts new file mode 100644 index 00000000000..123852c8a09 --- /dev/null +++ b/server/src/storage/s3-multipart.ts @@ -0,0 +1,52 @@ +import { AbortMultipartUploadCommand, CompleteMultipartUploadCommand, CreateMultipartUploadCommand, UploadPartCommand, type S3Client } from "@aws-sdk/client-s3"; +import { Readable } from "node:stream"; +import type { PutObjectInput } from "./types.js"; + +/** Each part is streamed too: multipart must not become another whole-file + * buffer or a fixed part-count ceiling below the storage provider's limits. */ +export async function putS3Multipart(client: S3Client, bucket: string, key: string, input: PutObjectInput) { + const source = input.body instanceof Readable ? input.body : Readable.from([input.body]); + const iterator = source[Symbol.asyncIterator](); + let pending = Buffer.alloc(0); + let uploadId: string | undefined; + // Scale part sizes to the S3 part-count limit while retaining stream backpressure. + const partSize = Math.max(8 * 1024 * 1024, Math.ceil(input.contentLength / 10_000)); + try { + uploadId = (await client.send(new CreateMultipartUploadCommand({ Bucket: bucket, Key: key, ContentType: input.contentType }))).UploadId; + if (!uploadId) throw new Error("S3 did not return a multipart upload id"); + const parts: { ETag: string; PartNumber: number }[] = []; + for (let offset = 0; offset < input.contentLength;) { + const length = Math.min(partSize, input.contentLength - offset); + let remaining = length; + const body = Readable.from((async function* () { + while (remaining > 0) { + if (!pending.length) { + const next = await iterator.next(); + if (next.done) throw new Error("S3 upload source was truncated"); + pending = Buffer.isBuffer(next.value) ? next.value : Buffer.from(next.value); + if (!pending.length) continue; + } + const count = Math.min(remaining, pending.length); + const chunk = pending.subarray(0, count); + pending = pending.subarray(count); + remaining -= count; + yield chunk; + } + })()); + try { + const result = await client.send(new UploadPartCommand({ Bucket: bucket, Key: key, UploadId: uploadId, PartNumber: parts.length + 1, Body: body, ContentLength: length })); + if (remaining || !result.ETag) throw new Error("S3 upload part is incomplete"); + parts.push({ ETag: result.ETag, PartNumber: parts.length + 1 }); + } finally { body.destroy(); } + offset += length; + } + if (pending.length || !(await iterator.next()).done) throw new Error("S3 upload source exceeds its declared size"); + await client.send(new CompleteMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId, MultipartUpload: { Parts: parts } })); + } catch (error) { + if (uploadId) await client.send(new AbortMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId })).catch(() => {}); + throw error; + } finally { + source.destroy(); + await iterator.return?.(); + } +} diff --git a/server/src/storage/s3-provider.ts b/server/src/storage/s3-provider.ts index 48c305bbfbd..517ccf23d98 100644 --- a/server/src/storage/s3-provider.ts +++ b/server/src/storage/s3-provider.ts @@ -5,6 +5,7 @@ import { HeadObjectCommand, PutObjectCommand, } from "@aws-sdk/client-s3"; +import { putS3Multipart } from "./s3-multipart.js"; import { Readable } from "node:stream"; import type { StorageProvider, GetObjectResult, HeadObjectResult } from "./types.js"; import { notFound, unprocessable } from "../errors.js"; @@ -81,6 +82,10 @@ export function createS3StorageProvider(config: S3ProviderConfig): StorageProvid async putObject(input) { const key = buildKey(prefix, input.objectKey); + if (input.contentLength > 16 * 1024 * 1024) { + await putS3Multipart(client, bucket, key, input); + return; + } await client.send( new PutObjectCommand({ Bucket: bucket, diff --git a/server/src/storage/service.ts b/server/src/storage/service.ts index bccdfc9f031..bed07ec392b 100644 --- a/server/src/storage/service.ts +++ b/server/src/storage/service.ts @@ -1,5 +1,6 @@ import { createHash, randomUUID } from "node:crypto"; import path from "node:path"; +import { Readable } from "node:stream"; import type { StorageService, StorageProvider, PutFileInput, PutFileResult } from "./types.js"; import { badRequest, forbidden, unprocessable } from "../errors.js"; @@ -79,10 +80,13 @@ function assertPutFileInput(input: PutFileInput): void { if (!input.contentType || input.contentType.trim().length === 0) { throw unprocessable("contentType is required"); } - if (!(input.body instanceof Buffer)) { - throw unprocessable("body must be a Buffer"); + if (!(input.body instanceof Buffer) && !(input.body instanceof Readable)) { + throw unprocessable("body must be a Buffer or Readable"); } - if (input.body.length <= 0) { + if (!Buffer.isBuffer(input.body) && (!("byteSize" in input) || !Number.isSafeInteger(input.byteSize) || input.byteSize < 0 || !/^[0-9a-f]{64}$/.test(input.sha256))) { + throw unprocessable("Streamed files require an exact byte size and SHA-256"); + } + if (("byteSize" in input ? input.byteSize : input.body.length) <= 0) { throw unprocessable("File is empty"); } } @@ -94,21 +98,28 @@ export function createStorageService(provider: StorageProvider): StorageService async putFile(input: PutFileInput): Promise { assertPutFileInput(input); const objectKey = buildObjectKey(input.companyId, input.namespace, input.originalFilename); - const byteSize = input.body.length; + const byteSize = "byteSize" in input ? input.byteSize : input.body.length; const contentType = input.contentType.trim().toLowerCase(); - await provider.putObject({ - objectKey, - body: input.body, - contentType, - contentLength: byteSize, - }); + try { + await provider.putObject({ + objectKey, + body: input.body, + contentType, + contentLength: byteSize, + }); + } catch (error) { + await provider.deleteObject({ objectKey }).catch(() => {}); + throw error; + } finally { + if (input.body instanceof Readable) input.body.destroy(); + } return { provider: provider.id, objectKey, contentType, byteSize, - sha256: hashBuffer(input.body), + sha256: "sha256" in input ? input.sha256 : hashBuffer(input.body), originalFilename: input.originalFilename, }; }, diff --git a/server/src/storage/types.ts b/server/src/storage/types.ts index dab23fa4c06..15289f77a5d 100644 --- a/server/src/storage/types.ts +++ b/server/src/storage/types.ts @@ -42,13 +42,12 @@ export interface StorageProvider { deleteObject(input: GetObjectInput): Promise; } -export interface PutFileInput { +export type PutFileInput = { companyId: string; namespace: string; originalFilename: string | null; contentType: string; - body: Buffer; -} +} & ({ body: Buffer } | { body: Readable; byteSize: number; sha256: string }); export interface PutFileResult { provider: StorageProviderId; diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index ba580e0ee2f..04dfe551e25 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -1094,3 +1094,15 @@ keys with low request limits. It can only lower the configured campaign limit. Keep subscription qualification separate from API-key results. The explicit-only eight-cell [continuation accounting baseline](CONTINUATION-ACCOUNTING.md) tests productive work, bounded repair, restart and late gates with real providers. + +### Bounded API response reading + +`api-response-reading` is an explicit-only, two-cell native Codex suite (local +and Daytona). Each cell creates synthetic diagnostic evidence over 24 KiB via +the public API, outside the agent assignment. The browser starts one ordinary +task which must read the saved response in bounded text pages and persist the +exact hidden evidence code. The oracle also requires completed API tool events; +missing events or a narrative completion cannot pass. Existing run, copyback, +screenshot, billing and environment cleanup checks apply. Use +`--id api-response-reading.runner-codex.daytona.saved-text-pages` with an +immutable Daytona image; no private hooks or fixture database writes are used. diff --git a/tests/runner-e2e/api-response-reading.test.ts b/tests/runner-e2e/api-response-reading.test.ts new file mode 100644 index 00000000000..0ccfb053edc --- /dev/null +++ b/tests/runner-e2e/api-response-reading.test.ts @@ -0,0 +1,102 @@ +import { createHash } from "node:crypto"; +import { describe, expect, it, vi } from "vitest"; +import type { RunnerApi } from "./api.js"; +import { apiResponseReadingTask, gradeApiResponsePaging, readResponseProof, responseEvidenceCode, responseEvidenceDescription } from "./api-response-reading.js"; + +describe("large API response evidence fixture", () => { + it("keeps evidence beyond both preview and inline limits and out of the assignment", () => { + const source = responseEvidenceDescription("fixture"); + const code = responseEvidenceCode("fixture"); + expect(Buffer.byteLength(source)).toBeGreaterThan(24 * 1024); + expect(source.indexOf(code)).toBeGreaterThan(24 * 1024); + expect(apiResponseReadingTask.buildPrompt("fixture")).not.toContain(code); + expect(apiResponseReadingTask.buildPrompt("fixture")).toContain("responseText.nextOffsetBytes"); + }); + it("downloads exact run-attributed proof bytes rather than trusting the agent's report", async () => { + const bytes = Buffer.from("independent evidence\n"); + const proof = { id: "proof", issueId: "issue", originatingRunId: "run", originalFilename: "api-response-proof.txt", byteSize: bytes.length, sha256: createHash("sha256").update(bytes).digest("hex") }; + const download = vi.fn().mockResolvedValue({ ok: () => true, body: async () => bytes }); + const get = vi.fn().mockResolvedValue([proof]); + const api = { get, request: { get: download } } as unknown as RunnerApi; + expect((await readResponseProof(api, "issue", "run")).content).toBe(bytes.toString()); + expect(download).toHaveBeenCalledWith("/api/attachments/proof/content?download=1"); + get.mockResolvedValue([{ ...proof, originatingRunId: "other" }]); + await expect(readResponseProof(api, "issue", "run")).rejects.toThrow("observed 0"); + get.mockResolvedValue([proof, proof]); + await expect(readResponseProof(api, "issue", "run")).rejects.toThrow("observed 2"); + get.mockResolvedValue([{ ...proof, sha256: "mismatch" }]); + await expect(readResponseProof(api, "issue", "run")).rejects.toThrow("disagrees"); + }); +}); + +describe("saved API response pagination oracle", () => { + const sourceIssueId = "source-issue"; + const assetId = "source-asset"; + const bytes = Buffer.from(JSON.stringify({ id: sourceIssueId, description: "synthetic padding ".repeat(2000) })); + const operation = "GET /api/assets/{assetId}/content"; + const event = (eventType: string, payload: unknown) => ({ eventType, payload: { prpEvent: { payload } } }); + const call = (id: string, input: unknown, result: unknown, status = "completed") => [ + event("item.started", { item: { id, type: "tool_use", name: "call_api", input } }), + event("item.completed", { item: { id, type: "tool_result", tool_use_id: id, result } }), + event("tool.execution.completed", { name: "call_api", executionId: id, status }), + ]; + const source = () => call("source-read", { operationId: "GET /api/issues/{id}", pathParams: { id: sourceIssueId } }, { + ok: true, status: 200, apiOperationId: "GET /api/issues/{id}", artifact: { + artifactId: assetId, byteSize: bytes.length, sha256: createHash("sha256").update(bytes).digest("hex"), + url: `/api/assets/${assetId}/content`, + }, + }); + const page = (offset: number, id = `page-${offset}`, artifactId = assetId) => { + const end = Math.min(offset + 8192, bytes.length); + return call(id, { operationId: operation, pathParams: { assetId: artifactId }, responseText: { offsetBytes: offset, limitBytes: 8192 } }, { + ok: true, status: 200, apiOperationId: operation, data: bytes.subarray(offset, end).toString(), + responseText: { offsetBytes: offset, totalBytes: bytes.length, nextOffsetBytes: end === bytes.length ? null : end }, + }); + }; + const pages = () => Array.from({ length: Math.ceil(bytes.length / 8192) }, (_, i) => page(i * 8192)).flat(); + it("accepts correlated source artifact pages through EOF, including a harmless identical retry", () => { + expect(gradeApiResponsePaging([...source(), ...page(0, "retry"), ...pages()], sourceIssueId).passed).toBe(true); + }); + it("rejects unrelated completed calls", () => { + const unrelated = [1, 2, 3].flatMap(i => call(`unrelated-${i}`, { operationId: "GET /api/agents/me" }, { ok: true, status: 200 })); + expect(gradeApiResponsePaging(unrelated, sourceIssueId).passed).toBe(false); + }); + it("rejects repeating the first page without reaching EOF", () => { + expect(gradeApiResponsePaging([...source(), ...page(0), ...page(0, "again"), ...page(0, "again-2")], sourceIssueId).passed).toBe(false); + }); + it("rejects a gap even when the last page claims EOF", () => { + expect(gradeApiResponsePaging([...source(), ...pages().filter((_e, i) => i < 3 || i >= 6)], sourceIssueId).passed).toBe(false); + }); + it("rejects reading a different artifact", () => { + const other = Array.from({ length: Math.ceil(bytes.length / 8192) }, (_, i) => page(i * 8192, `other-${i}`, "other-asset")).flat(); + expect(gradeApiResponsePaging([...source(), ...other], sourceIssueId).passed).toBe(false); + }); + it("requires the exact seeded source issue", () => { + expect(gradeApiResponsePaging([...source(), ...pages()], "other-issue").passed).toBe(false); + }); + it("requires a matched successful completion for every page", () => { + const incomplete = pages().filter(e => e.eventType !== "tool.execution.completed"); + expect(gradeApiResponsePaging([...source(), ...incomplete], sourceIssueId).passed).toBe(false); + }); + it.each([ + ["requested offset", { offsetBytes: 1, totalBytes: bytes.length, nextOffsetBytes: 8192 }], + ["next offset", { offsetBytes: 0, totalBytes: bytes.length, nextOffsetBytes: 8193 }], + ["total size", { offsetBytes: 0, totalBytes: bytes.length + 1, nextOffsetBytes: 8192 }], + ["premature EOF", { offsetBytes: 0, totalBytes: bytes.length, nextOffsetBytes: null }], + ])("rejects mismatched %s metadata", (_label, responseText) => { + const malformed = call("bad-page", { + operationId: operation, pathParams: { assetId }, responseText: { offsetBytes: 0, limitBytes: 8192 }, + }, { ok: true, status: 206, apiOperationId: operation, data: bytes.subarray(0, 8192).toString(), responseText }); + expect(gradeApiResponsePaging([...source(), ...pages(), ...malformed], sourceIssueId).passed).toBe(false); + }); + it("rejects contradictory duplicate bytes and a wrong source artifact digest", () => { + const corrupt = call("corrupt-page", { + operationId: operation, pathParams: { assetId }, responseText: { offsetBytes: 0, limitBytes: 8192 }, + }, { + ok: true, status: 206, apiOperationId: operation, data: "x".repeat(8192), + responseText: { offsetBytes: 0, totalBytes: bytes.length, nextOffsetBytes: 8192 }, + }); + expect(gradeApiResponsePaging([...source(), ...pages(), ...corrupt], sourceIssueId).passed).toBe(false); + expect(gradeApiResponsePaging([...source(), ...corrupt, ...pages().slice(3)], sourceIssueId).passed).toBe(false); + }); +}); diff --git a/tests/runner-e2e/api-response-reading.ts b/tests/runner-e2e/api-response-reading.ts new file mode 100644 index 00000000000..b91db1c71ab --- /dev/null +++ b/tests/runner-e2e/api-response-reading.ts @@ -0,0 +1,134 @@ +import { createHash } from "node:crypto"; +import type { RunnerTaskFixture } from "./types.js"; +import type { RunnerApi } from "./api.js"; +import type { IssueAttachment } from "../../packages/shared/src/types/issue.js"; + +/** Verify delivered bytes through the same public download path the user receives. */ +export async function readResponseProof(api: RunnerApi, issueId: string, runId: string) { + const attachments = await api.get(`/api/issues/${issueId}/attachments`); + const proofs = attachments.filter(attachment => + attachment.issueId === issueId && attachment.originatingRunId === runId && + attachment.originalFilename === "api-response-proof.txt"); + if (proofs.length !== 1) throw new Error(`Expected one proof attachment from the tested run; observed ${proofs.length}`); + const proof = proofs[0]!; + const response = await api.request.get(`/api/attachments/${encodeURIComponent(proof.id)}/content?download=1`); + if (!response.ok()) throw new Error(`Proof download returned ${response.status()}`); + const bytes = await response.body(); + if (bytes.length !== proof.byteSize || createHash("sha256").update(bytes).digest("hex") !== proof.sha256) { + throw new Error("Downloaded proof disagrees with stored attachment bytes"); + } + return { attachmentId: proof.id, content: bytes.toString("utf8"), sha256: proof.sha256 }; +} + +export function responseEvidenceCode(nonce: string) { + return createHash("sha256").update(`bounded-response-evidence:${nonce}`).digest("hex"); +} +export function responseEvidenceDescription(nonce: string) { + return `${"Synthetic diagnostic padding.\n".repeat(1400)}\nEvidence code: ${responseEvidenceCode(nonce)}\n`; +} + +/** Correlate trusted PRP call inputs, results, and completion receipts. */ +export function gradeApiResponsePaging(events: readonly { eventType?: string; payload?: unknown }[], sourceIssueId: string) { + const object = (value: unknown): Record => + value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; + const inputs = new Map>(); + const results = new Map>(); + const completed = new Set(); + for (const event of events) { + const payload = object(object(object(event.payload).prpEvent).payload); + const item = object(payload.item); + if (event.eventType === "item.started" && item.type === "tool_use" && item.name === "call_api" && typeof item.id === "string") { + inputs.set(item.id, object(item.input)); + } + if (event.eventType === "item.completed" && item.type === "tool_result" && typeof item.id === "string" && + (item.tool_use_id === undefined || item.tool_use_id === item.id)) results.set(item.id, object(item.result)); + if (event.eventType === "tool.execution.completed" && payload.name === "call_api" && payload.status === "completed" && typeof payload.executionId === "string") { + completed.add(payload.executionId); + } + } + const calls = [...inputs].flatMap(([id, input]) => { + const result = results.get(id); + return completed.has(id) && result?.ok === true && [200, 206].includes(Number(result.status)) ? [{ input, result }] : []; + }); + const sourceOperation = "GET /api/issues/{id}"; + const assetOperation = "GET /api/assets/{assetId}/content"; + const sources = calls.filter(call => call.input.operationId === sourceOperation && call.result.apiOperationId === sourceOperation && + object(call.input.pathParams).id === sourceIssueId); + let failure = "No successful call_api source read with a saved response artifact"; + for (const source of sources) { + const artifact = object(source.result.artifact); + const assetId = artifact.artifactId; + const total = artifact.byteSize; + if (typeof assetId !== "string" || typeof total !== "number" || !Number.isSafeInteger(total) || total <= 24 * 1024 || total > 1024 * 1024 || + typeof artifact.sha256 !== "string" || !/^[a-f0-9]{64}$/.test(artifact.sha256) || artifact.url !== `/api/assets/${assetId}/content`) continue; + const pages = new Map(); + let malformed = false; + for (const call of calls) { + if (call.input.operationId !== assetOperation || object(call.input.pathParams).assetId !== assetId) continue; + const requested = object(call.input.responseText); + const returned = object(call.result.responseText); + const offset = requested.offsetBytes; + const limit = requested.limitBytes; + const next = returned.nextOffsetBytes; + const data = call.result.data; + if (call.result.apiOperationId !== assetOperation || typeof offset !== "number" || !Number.isSafeInteger(offset) || offset < 0 || offset >= total || + typeof limit !== "number" || !Number.isSafeInteger(limit) || limit < 1 || limit > 8192 || returned.offsetBytes !== offset || returned.totalBytes !== total || + typeof data !== "string") { malformed = true; break; } + const bytes = Buffer.from(data, "utf8"); + const end = offset + bytes.length; + if (bytes.length === 0 || bytes.length > limit || end > total || (end === total ? next !== null : next !== end)) { + malformed = true; break; + } + const prior = pages.get(offset); + if (prior && (!prior.bytes.equals(bytes) || prior.next !== next)) { malformed = true; break; } + pages.set(offset, { bytes, next: next as number | null }); + } + if (malformed) { failure = "Artifact paging request, returned offsets, bytes, or EOF disagree"; continue; } + const chunks: Buffer[] = []; + const offsets: number[] = []; + let offset = 0; + let eofReached = false; + while (pages.has(offset)) { + const page = pages.get(offset)!; + chunks.push(page.bytes); offsets.push(offset); + if (page.next === null) { eofReached = true; break; } + offset = page.next; + } + if (!eofReached || offsets.length < 2) { failure = "Saved artifact pages do not cover offset zero through EOF without gaps"; continue; } + const bytes = Buffer.concat(chunks); + const digest = createHash("sha256").update(bytes).digest("hex"); + if (bytes.length !== total || digest !== artifact.sha256) { failure = "Paged bytes disagree with the source response artifact digest"; continue; } + try { + if (object(JSON.parse(bytes.toString("utf8"))).id !== sourceIssueId) { failure = "Paged artifact belongs to another source issue"; continue; } + } catch { failure = "Paged source response is not complete JSON"; continue; } + return { passed: true, sourceIssueId, artifactId: assetId, totalBytes: total, pageCount: offsets.length, offsets, eofReached, sha256: digest }; + } + return { passed: false, sourceIssueId, failure }; +} + +export const apiResponseReadingTask: RunnerTaskFixture = { + id: "saved-text-pages", label: "Read saved large API evidence", groups: [], + workMode: "standard", flow: "single_turn", expectedRunCount: 1, + attemptTimeoutMs: { local: 10 * 60_000, daytona: 10 * 60_000 }, + expectedTerminalState: { issue: "done", run: "succeeded" }, + buildTitle: nonce => `Read large API evidence ${nonce}`, + // Delivered files render as an attachment card, which can replace the finish summary. + buildVisibleMarker: () => "api-response-proof.txt", + buildPrompt: nonce => [ + "Inspect diagnostic evidence task {{API_RESPONSE_SOURCE_ID}} using the Paperclip API tools.", + "Discover GET /api/issues/{id}, call it for that task, and retain the returned response artifact.", + "Read the saved artifact through GET /api/assets/{assetId}/content with responseText: {offsetBytes:0,limitBytes:8192}.", + "Continue using responseText.nextOffsetBytes until null. Extract the Evidence code at the end of its description.", + "The large response must be read with bounded responseText pages. Do not use other tools or API routes to obtain the evidence; if bounded reading fails, report the failure instead of substituting a different reader.", + "Write only that code followed by a newline to api-response-proof.txt in the current workspace and deliver that file as an attachment. Do not edit the evidence task or create child tasks.", + `Verify the proof file and finish this task with paperclip_finish, reportedWorkDisposition done, and summary API_RESPONSE_READ_${nonce}.`, + ].join("\n"), + buildMatchers: (nonce, execution) => [ + { kind: "file_exact", path: "api-response-proof.txt", expected: `${responseEvidenceCode(nonce)}\n` }, + { kind: "message_contains", expected: "api-response-proof.txt" }, + { kind: "issue_status", expected: "done" }, + { kind: "run_status", expected: "succeeded" }, + { kind: "runtime_mode", expected: execution.profile.expectedRuntimeMode }, + { kind: "environment", expected: execution.environment.id }, + ], +}; diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index ae70523d43a..ac64f1db0ef 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -71,10 +71,10 @@ describe("runner E2E catalog", () => { expect(localIntegrityTasks).toHaveLength(2); expect(openRouterBreadthTasks).toHaveLength(3); expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([ - 8, 46, 23, 47, 52, 28, 18, 6, 6, 4, 42, 14, 10, 2, + 2, 8, 46, 23, 47, 52, 28, 18, 6, 6, 4, 42, 14, 10, 2, ]); - expect(validateRunnerCatalog()).toHaveLength(306); - expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(306); + expect(validateRunnerCatalog()).toHaveLength(308); + expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(308); expect( runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"), ).toHaveLength(42); diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index 7fc74839073..e1a613cb2f9 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -1,3 +1,4 @@ +import { apiResponseReadingTask } from "./api-response-reading.js"; import { accountingTasks } from "./accounting-cases.js"; import { continuationTasks } from "./continuation-cases.js"; import { lifecycleLiveTasks, lifecycleLiveDefinitionDigest } from "./lifecycle-live-cases.js"; @@ -911,6 +912,14 @@ const everydayProfiles = [ ].map(productionStoryProfile); export const runnerSuites: readonly RunnerSuiteFixture[] = [ + { + id: "api-response-reading", label: "Bounded API response reading", manualOnly: true, + description: "Retrieve evidence beyond a saved API preview through authorized bounded text windows.", + groups: [], environments: runnerEnvironments, + profiles: runnerProfiles.filter(profile => profile.id === "runner-codex"), + tasks: [apiResponseReadingTask], expectedMatrixSize: 2, + definitionMetadata: { version: 1, grading: "hidden-evidence-exact-copy-and-api-tool-events", scheduling: "explicit-only" }, + }, { id: "continuation-accounting", label: "Continuation accounting baseline", manualOnly: true, description: "Structured productive steps, bounded repair, restart and late gates; comments cannot buy more attempts.", diff --git a/tests/runner-e2e/runner.spec.ts b/tests/runner-e2e/runner.spec.ts index 19923ae6f3a..1cd4ad18a30 100644 --- a/tests/runner-e2e/runner.spec.ts +++ b/tests/runner-e2e/runner.spec.ts @@ -1,3 +1,4 @@ +import { gradeApiResponsePaging, readResponseProof, responseEvidenceDescription } from "./api-response-reading.js"; import { observeBrowserBootstrap } from "./browser-bootstrap-diagnostics.js"; import { runAccountingFlow } from "./accounting-flow.js"; import type { Issue } from "../../packages/shared/src/types/issue.js"; @@ -534,6 +535,7 @@ for (const execution of executions) { const marker = execution.task.buildVisibleMarker(nonce); const title = execution.task.buildTitle(nonce); let prompt = execution.task.buildPrompt(nonce); + let apiResponseSourceId: string | undefined; let lifecycleBlockerId: string | null = null; const credentials = credentialValues(); const secrets = normalizedSecrets(Object.values(credentials)); @@ -549,6 +551,7 @@ for (const execution of executions) { let selectedRuns: RunRecord[] = []; let runtimeLeases: EnvironmentLeaseRecord[] = []; let matcherResults: MatcherResult[] = []; + let downloadedResponseProof: Awaited> | undefined; let firstTaskEvidence: RunnerE2EResult["firstTask"]; let turnTimings: NonNullable | undefined; const turnSubmissionTimesMs: number[] = []; @@ -794,6 +797,15 @@ for (const execution of executions) { daytonaImage: process.env.PAPERCLIP_E2E_DAYTONA_IMAGE, }); + if (execution.suite.id === "api-response-reading") { + const source = await api.post<{ id: string }>(`/api/companies/${fixtures.company.id}/issues`, { + title: `Synthetic diagnostic evidence ${nonce}`, + description: responseEvidenceDescription(nonce), status: "backlog", + }); + apiResponseSourceId = source.id; + prompt = prompt.replaceAll("{{API_RESPONSE_SOURCE_ID}}", source.id); + } + if (execution.suite.id === "lifecycle-baseline" && lifecycleLiveCase(execution.task.id)?.family === "blocker") { const prerequisite = await api.post<{ id: string }>(`/api/companies/${fixtures.company.id}/issues`, { title: `Supply dataset ${nonce}`, @@ -1885,6 +1897,12 @@ for (const execution of executions) { } } + if (execution.suite.id === "api-response-reading") { + const paging = gradeApiResponsePaging(runEventsByRun.flatMap(captured => captured.events), apiResponseSourceId ?? ""); + await writeSanitizedJson(snapshotsDir, "api-response-pagination.json", paging, secrets); + if (!paging.passed) invariantFailures.push(`Bounded response paging was not proven: ${paging.failure}`); + } + const context = record(run.contextSnapshot); const environmentContext = record(context.paperclipEnvironment); const workspaceContext = record(context.paperclipWorkspace); @@ -1938,6 +1956,11 @@ for (const execution of executions) { ]), ), ); + if (execution.suite.id === "api-response-reading") { + downloadedResponseProof = await readResponseProof(api, issue.id, run.id); + fileObservations["api-response-proof.txt"] = downloadedResponseProof.content; + await writeSanitizedJson(snapshotsDir, "downloaded-response-proof.json", downloadedResponseProof, secrets); + } matcherResults = await Promise.all( taskMatchers.map((matcher) => evaluateMatcher(matcher, { @@ -2398,7 +2421,13 @@ for (const execution of executions) { const visibleAgentReplies = page .getByTestId("task-chat-thread") .getByTestId("task-chat-agent-bubble"); - if (execution.task.flow === "warm_three_turn") { + if (execution.suite.id === "api-response-reading") { + // File delivery renders a card instead of an exact summary bubble. + // Prove the visible link points to the same independently checked bytes. + const proofLink = page.getByRole("link", { name: "Open api-response-proof.txt", exact: true }).first(); + await expect(proofLink).toBeVisible({ timeout: 30_000 }); + await expect(proofLink).toHaveAttribute("href", `/api/attachments/${downloadedResponseProof!.attachmentId}/content`); + } else if (execution.task.flow === "warm_three_turn") { // Prove the persisted user-facing response is visible, independently // of the byte-for-byte workspace checks and lease continuity checks. expect(finalRunMessage.trim()).not.toBe("");