From 49f0030f064ef6d19b44496d9d275b7fbcf8d8d2 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 01:17:57 -0600 Subject: [PATCH] fix(heartbeat): rank a person's joined comment first and refill free run slots (#21277) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat queue starts each agent's queued runs, up to the agent's `maxConcurrentRuns` > - #21234 starts person-initiated runs before system recovery runs, and it reads "person-initiated" from the run's own wake request > - When a person comments on a task that already has a queued system run, the comment joins that run. The run keeps its system wake, so dispatch ranks the person's message as system work > - Dispatch is event driven. The queued-run re-drive backstop covers only agents with no live run, so an agent with several slots keeps a free slot empty until one of its runs finishes, and runs can take hours > - This pull request makes dispatch read the person wakes that joined each queued run, and makes the re-drive backstop cover an agent that still has a free slot > - The benefit is that a person's comment starts first, and free slots fill within about a minute instead of waiting for an unrelated run to finish ## Linked Issues or Issue Description Refs #21234, #20986 **What happened?** On staging, one agent had `maxConcurrentRuns` 20, 18 running runs and 76 queued runs at 03:12Z. No run started on that agent from 03:01:33Z to 03:16:57Z (15.4 minutes). From 03:11:10Z, after a run finished, at least 2 slots were free. Other agents in the same company started runs during that time. Runs started again only after the next deploy restarted the server. Five queued runs came from a person: comments at 02:43Z, 02:44Z and 02:45Z, and Retries at 02:46Z, 02:49Z and 03:01Z. The comment runs had the wake reasons `issue_reopened_via_comment` and `issue_commented`, the invocation source `automation`, and no person on the run's own wake request. **Expected behavior** A run that carries a person's comment ranks as person-initiated, also when the comment joined an existing queued run. When an agent has a free slot and a queued run is ready, the run starts promptly, without waiting for another run on the agent to finish. **Steps to reproduce** 1. Give an agent `maxConcurrentRuns` 3. Start 2 runs on it. 2. Queue a third run while no dispatch signal reaches the queue (for example, a claim declines it while a gate is still closed, and the gate then clears). 3. See that `listUndispatchedQueuedRuns` returns nothing, because the agent has a live run. The run stays queued until one of the 2 running runs finishes. 4. Separately, queue a system run on a task, then post a person's comment on the task so that the comment wake coalesces into that queued run. Queue older system runs on other tasks of the same single-slot agent. 5. See that dispatch starts the older system runs first. **Root cause** 1. `coalesceIntoActiveExecutionRun` (`server/src/modules/wake-queue/adapters/postgres.ts`) merges the person's comment context into the queued run and records the person's wake as a separate `coalesced` row with `runId` set to that run. The run's own `wakeupRequestId` still points to the system wake. `queuedRunDispatcherForAgent` (`server/src/services/heartbeat/queue.ts`) only read the wake behind `wakeupRequestId`. The task routes do record the comment author as the requester (`requestedByActorType: "user"`). The person was lost only at dispatch time. 2. `listUndispatchedQueuedRunPage` (`server/src/services/queued-run-dispatch-liveness.ts`) skipped every agent with any `running` or `scheduled_retry` run. That rule assumes one slot per agent. For an agent with several slots, a lost or declined dispatch is not re-driven until a run on that agent finishes. This change is not in `claimQueuedRun`, the fresh-isolated preflight, the wake-queue promoter or `fresh-continuation-fallback.ts`. **Paperclip version or commit** Staging Core `ddc0aacad8` (contains #21234 `ade7e0643f`). This branch is based on master `6cb0dd0c6b`. **Deployment mode** Authenticated, public, Paperclip Cloud managed staging. **Agent adapter(s) involved** `paperclip_runner` (Codex native runtime). The fix is adapter-independent. ## What Changed - `queuedRunDispatcherForAgent` also reads person wakes whose `runId` is a queued run of the agent (by `run_id IN (queued run ids)`, so an earlier saved comment that a run adopted is included). A run with such a wake ranks as person-initiated and is never counted as system recovery. - `listUndispatchedQueuedRuns` now returns a queued run when its agent has fewer `running` plus `scheduled_retry` runs than its `maxConcurrentRuns`, and its task has no `running` or `scheduled_retry` run. The re-drive backstop and the ownership I7 check use this query. - The query counts each agent's live runs once, in one grouped subquery that is joined to the queued runs. - Migration 0476 adds `agent_wakeup_requests_company_run_idx` (`company_id`, `run_id`, partial on a non-null run) with `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. It is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS` and classifies as `online_compatible`. `doc/DATABASE.md` describes it. - Added `maxConcurrentRunsSql()` to `timer-policy.ts`. It reads the same limit as `parseHeartbeatPolicy`, including the single-run runner providers, which now share one constant. ## Verification - Red/green: `server/src/__tests__/recovery-wake-herd.test.ts` "starts a queued run that a person's comment joined ahead of older system work". It fails on master (`expected null not to be null`: the older system run started first) and passes with this change. - Red/green: `server/src/__tests__/ownership-reconciliation-report.test.ts` "re-drives a queued run while its agent still has a free slot, and not once every slot is running". It fails on master (`expected [] to deeply equal [ Array(1) ]`) and passes with this change. - Red/green: "starts a queued run that adopted a person's comment saved before every queued run". It fails with a `requestedAt` bound and passes with the `run_id` lookup. - Red/green: "counts an agent's live runs in one grouped aggregate, not once per queued run" runs EXPLAIN on the page query. It fails with a per-row correlated count and passes with the grouped subquery. - New: "uses the agent's single-run limit for a runner provider that runs one turn at a time". - Two existing tests now set `maxConcurrentRuns: 1` so that they keep testing "the agent's only slot is busy". - `npx vitest run src/__tests__/recovery-wake-herd.test.ts src/services/heartbeat/queued-run-dispatch-order.test.ts src/__tests__/ownership-reconciliation-report.test.ts src/services/queued-run-dispatch-liveness.test.ts src/services/heartbeat/dispatch-work.test.ts`: 71 passed. - `npx tsc --noEmit -p server/tsconfig.json`: no errors. ## Risks - The re-drive backstop now also covers multi-slot agents with live runs. Each queued run is re-driven at most on its existing backoff (1 minute, then doubling to 15 minutes). A queued run whose task already has a live run is not re-driven. - The ownership I7 report can now list a queued run on a multi-slot agent that has a free slot. This is the stall that this change targets. - One more indexed read per dispatch pass (`agent_wakeup_requests` by company and run id). The new index is built concurrently, so the previous release keeps writing during the build. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, read-only staging API reads in a browser tab). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/DATABASE.md | 6 + packages/db/src/client.ts | 1 + .../0476_wakeup_request_run_index.sql | 1 + ...{0471_snapshot.json => 0476_snapshot.json} | 629 +++++++++++++++++- packages/db/src/migrations/meta/_journal.json | 7 + .../db/src/schema/agent_wakeup_requests.ts | 4 + .../ownership-reconciliation-report.test.ts | 51 +- .../src/__tests__/recovery-wake-herd.test.ts | 46 ++ server/src/services/heartbeat/queue.ts | 12 +- server/src/services/heartbeat/timer-policy.ts | 14 +- .../services/queued-run-dispatch-liveness.ts | 28 +- 11 files changed, 785 insertions(+), 14 deletions(-) create mode 100644 packages/db/src/migrations/0476_wakeup_request_run_index.sql rename packages/db/src/migrations/meta/{0471_snapshot.json => 0476_snapshot.json} (99%) diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 5b9623f1be0..138f1128461 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -507,6 +507,12 @@ indexes also serve the comment and delivery `on delete set null` actions. Issue lists share one external review-wait read per issue between the review and blocked-inbox attention maps. +Migration 0476 builds `agent_wakeup_requests_company_run_idx` concurrently. It +keys wake requests by company and `run_id`, partial on a non-null run. Queued-run +dispatch finds the person wakes that joined or were adopted by each queued run +through it, so the read is bounded by the queue size, not by the agent's wake +history. + Migration 0457 builds `heartbeat_runs_company_native_session_idx` concurrently. It keys runs by company and `native_session_id`, partial on a non-null session. Native retention, recovery, resume and archive reads find a session's runs diff --git a/packages/db/src/client.ts b/packages/db/src/client.ts index 0a38039de4b..eaea1067bb6 100644 --- a/packages/db/src/client.ts +++ b/packages/db/src/client.ts @@ -690,6 +690,7 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([ "0461_created_from_legacy_origin_run_index.sql", "0470_heartbeat_runs_queued_created_index.sql", "0472_runner_api_capture_pending_deletes.sql", + "0476_wakeup_request_run_index.sql", "0469_zippy_photon.sql", diff --git a/packages/db/src/migrations/0476_wakeup_request_run_index.sql b/packages/db/src/migrations/0476_wakeup_request_run_index.sql new file mode 100644 index 00000000000..31425377c93 --- /dev/null +++ b/packages/db/src/migrations/0476_wakeup_request_run_index.sql @@ -0,0 +1 @@ +CREATE INDEX CONCURRENTLY IF NOT EXISTS "agent_wakeup_requests_company_run_idx" ON "agent_wakeup_requests" USING btree ("company_id","run_id") WHERE "agent_wakeup_requests"."run_id" IS NOT NULL; diff --git a/packages/db/src/migrations/meta/0471_snapshot.json b/packages/db/src/migrations/meta/0476_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0471_snapshot.json rename to packages/db/src/migrations/meta/0476_snapshot.json index 2016c014343..317026f43a0 100644 --- a/packages/db/src/migrations/meta/0471_snapshot.json +++ b/packages/db/src/migrations/meta/0476_snapshot.json @@ -1,6 +1,6 @@ { - "id": "671a422a-0b93-4155-b097-9d35a4fd2b2b", - "prevId": "f0e17e47-6217-4080-b6b6-9d83b33e9ced", + "id": "82dcdc57-0796-4162-9451-8ea08a72199e", + "prevId": "ee1f1f66-0cbe-4b18-b364-5e96708a1f0b", "version": "7", "dialect": "postgresql", "tables": { @@ -3954,6 +3954,28 @@ "method": "btree", "with": {} }, + "agent_wakeup_requests_company_run_idx": { + "name": "agent_wakeup_requests_company_run_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"agent_wakeup_requests\".\"run_id\" IS NOT NULL", + "concurrently": true, + "method": "btree", + "with": {} + }, "agent_wakeup_requests_review_path_recovery_idempotency_uq": { "name": "agent_wakeup_requests_review_path_recovery_idempotency_uq", "columns": [ @@ -53686,6 +53708,12 @@ "primaryKey": false, "notNull": true }, + "pending_delete_object_key": { + "name": "pending_delete_object_key", + "type": "text", + "primaryKey": false, + "notNull": false + }, "created_at": { "name": "created_at", "type": "timestamp with time zone", @@ -53709,6 +53737,21 @@ "concurrently": false, "method": "btree", "with": {} + }, + "runner_api_response_reservations_asset_idx": { + "name": "runner_api_response_reservations_asset_idx", + "columns": [ + { + "expression": "asset_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} } }, "foreignKeys": { @@ -53921,6 +53964,7 @@ } ], "isUnique": true, + "where": "\"sandbox_creation_operations\".\"state\" <> 'resolved'", "concurrently": false, "method": "btree", "with": {} @@ -57514,6 +57558,12 @@ "notNull": true, "default": "'{}'::jsonb" }, + "ui_meta": { + "name": "ui_meta", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, "risk_level": { "name": "risk_level", "type": "text", @@ -59639,6 +59689,253 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.tool_mcp_app_views": { + "name": "tool_mcp_app_views", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "invocation_id": { + "name": "invocation_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "issue_id": { + "name": "issue_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "run_id": { + "name": "run_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "agent_id": { + "name": "agent_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "responsible_user_id": { + "name": "responsible_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "connection_id": { + "name": "connection_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "catalog_entry_id": { + "name": "catalog_entry_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "application_name": { + "name": "application_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "tool_name": { + "name": "tool_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "upstream_tool_name": { + "name": "upstream_tool_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "resource_uri": { + "name": "resource_uri", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "tool_input": { + "name": "tool_input", + "type": "jsonb", + "primaryKey": false, + "notNull": true, + "default": "'{}'::jsonb" + }, + "tool_result": { + "name": "tool_result", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "tool_mcp_app_views_issue_idx": { + "name": "tool_mcp_app_views_issue_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "issue_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "tool_mcp_app_views_connection_idx": { + "name": "tool_mcp_app_views_connection_idx", + "columns": [ + { + "expression": "connection_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "tool_mcp_app_views_company_id_companies_id_fk": { + "name": "tool_mcp_app_views_company_id_companies_id_fk", + "tableFrom": "tool_mcp_app_views", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "tool_mcp_app_views_invocation_id_tool_invocations_id_fk": { + "name": "tool_mcp_app_views_invocation_id_tool_invocations_id_fk", + "tableFrom": "tool_mcp_app_views", + "tableTo": "tool_invocations", + "columnsFrom": [ + "invocation_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "tool_mcp_app_views_issue_id_issues_id_fk": { + "name": "tool_mcp_app_views_issue_id_issues_id_fk", + "tableFrom": "tool_mcp_app_views", + "tableTo": "issues", + "columnsFrom": [ + "issue_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "tool_mcp_app_views_run_id_heartbeat_runs_id_fk": { + "name": "tool_mcp_app_views_run_id_heartbeat_runs_id_fk", + "tableFrom": "tool_mcp_app_views", + "tableTo": "heartbeat_runs", + "columnsFrom": [ + "run_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "tool_mcp_app_views_agent_id_agents_id_fk": { + "name": "tool_mcp_app_views_agent_id_agents_id_fk", + "tableFrom": "tool_mcp_app_views", + "tableTo": "agents", + "columnsFrom": [ + "agent_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "tool_mcp_app_views_connection_id_tool_connections_id_fk": { + "name": "tool_mcp_app_views_connection_id_tool_connections_id_fk", + "tableFrom": "tool_mcp_app_views", + "tableTo": "tool_connections", + "columnsFrom": [ + "connection_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "tool_mcp_app_views_catalog_entry_id_tool_catalog_entries_id_fk": { + "name": "tool_mcp_app_views_catalog_entry_id_tool_catalog_entries_id_fk", + "tableFrom": "tool_mcp_app_views", + "tableTo": "tool_catalog_entries", + "columnsFrom": [ + "catalog_entry_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.tool_mcp_gateway_tokens": { "name": "tool_mcp_gateway_tokens", "schema": "", @@ -60353,6 +60650,334 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.tool_mcp_skill_files": { + "name": "tool_mcp_skill_files", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "connection_id": { + "name": "connection_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "file_uri": { + "name": "file_uri", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "digest": { + "name": "digest", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "size": { + "name": "size", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "content_base64": { + "name": "content_base64", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "tool_mcp_skill_files_digest_idx": { + "name": "tool_mcp_skill_files_digest_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "connection_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "file_uri", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "digest", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "tool_mcp_skill_files_company_id_companies_id_fk": { + "name": "tool_mcp_skill_files_company_id_companies_id_fk", + "tableFrom": "tool_mcp_skill_files", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "tool_mcp_skill_files_connection_id_tool_connections_id_fk": { + "name": "tool_mcp_skill_files_connection_id_tool_connections_id_fk", + "tableFrom": "tool_mcp_skill_files", + "tableTo": "tool_connections", + "columnsFrom": [ + "connection_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.tool_mcp_skills": { + "name": "tool_mcp_skills", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "connection_id": { + "name": "connection_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "skill_uri": { + "name": "skill_uri", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "''" + }, + "frontmatter": { + "name": "frontmatter", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "manifest": { + "name": "manifest", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "manifest_digest": { + "name": "manifest_digest", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending_review'" + }, + "status_reason": { + "name": "status_reason", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "approved_manifest_digest": { + "name": "approved_manifest_digest", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "allowed_tools_approved": { + "name": "allowed_tools_approved", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "reviewed_at": { + "name": "reviewed_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "reviewed_by_user_id": { + "name": "reviewed_by_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "last_seen_at": { + "name": "last_seen_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "tool_mcp_skills_connection_uri_idx": { + "name": "tool_mcp_skills_connection_uri_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "connection_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "skill_uri", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "tool_mcp_skills_company_status_idx": { + "name": "tool_mcp_skills_company_status_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "tool_mcp_skills_company_id_companies_id_fk": { + "name": "tool_mcp_skills_company_id_companies_id_fk", + "tableFrom": "tool_mcp_skills", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "tool_mcp_skills_connection_id_tool_connections_id_fk": { + "name": "tool_mcp_skills_connection_id_tool_connections_id_fk", + "tableFrom": "tool_mcp_skills", + "tableTo": "tool_connections", + "columnsFrom": [ + "connection_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.tool_oauth_states": { "name": "tool_oauth_states", "schema": "", diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index 0d61b8089e7..84592a4329b 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -3319,6 +3319,13 @@ "when": 1794532772498, "tag": "0475_sandbox_creation_owner_unresolved", "breakpoints": true + }, + { + "idx": 476, + "version": "7", + "when": 1794553127740, + "tag": "0476_wakeup_request_run_index", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/agent_wakeup_requests.ts b/packages/db/src/schema/agent_wakeup_requests.ts index 11c82c37122..5a4ae36796e 100644 --- a/packages/db/src/schema/agent_wakeup_requests.ts +++ b/packages/db/src/schema/agent_wakeup_requests.ts @@ -59,6 +59,10 @@ export const agentWakeupRequests = pgTable( table.agentId, table.requestedAt, ), + companyRunIdx: index("agent_wakeup_requests_company_run_idx") + .on(table.companyId, table.runId) + .concurrently() + .where(sql`${table.runId} IS NOT NULL`), reviewPathRecoveryIdempotencyUq: uniqueIndex( "agent_wakeup_requests_review_path_recovery_idempotency_uq", ) diff --git a/server/src/__tests__/ownership-reconciliation-report.test.ts b/server/src/__tests__/ownership-reconciliation-report.test.ts index 72868565b8b..9266fec6d78 100644 --- a/server/src/__tests__/ownership-reconciliation-report.test.ts +++ b/server/src/__tests__/ownership-reconciliation-report.test.ts @@ -34,7 +34,7 @@ import { applyOwnershipReportBackstops, registerOwnershipReconciliationWork } fr import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js"; import { notifyNativeFinalizationWork } from "../services/native-finalization-notifications.js"; import { instanceSettingsService } from "../services/instance-settings.js"; -import { listUndispatchedQueuedRuns, readUndispatchedQueuedRuns } from "../services/queued-run-dispatch-liveness.js"; +import { listUndispatchedQueuedRuns, readUndispatchedQueuedRuns, undispatchedQueuedRunPageQuery } from "../services/queued-run-dispatch-liveness.js"; import { releaseLapsedCoordinatorOwner } from "../services/workspace-recovery-stop-proof.js"; import { subscribeDatabaseWork } from "@paperclipai/db/work-signals"; import { @@ -1073,6 +1073,7 @@ describe("report-only ownership reconciliation", () => { }); it("reads undispatched queued runs in bounded slices that resume from a cursor, and skips an agent with a scheduled retry", async () => { + await db.update(agents).set({ runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } }).where(eq(agents.id, agentId)); const created = [30, 29, 28].map(minutesAgo); const runs = await db.insert(heartbeatRuns).values(created.map(createdAt => ({ companyId, agentId, status: "queued", invocationSource: "automation", createdAt }))).returning(); @@ -1086,6 +1087,51 @@ describe("report-only ownership reconciliation", () => { expect(await listUndispatchedQueuedRuns(db)).toEqual([]); }); + it("re-drives a queued run while its agent still has a free slot, and not once every slot is running", async () => { + await db.update(agents).set({ runtimeConfig: { heartbeat: { maxConcurrentRuns: 3 } } }).where(eq(agents.id, agentId)); + const tasks = await db.insert(issues).values([0, 1, 2, 3].map(index => + ({ companyId, title: `Task ${index}`, status: "in_progress" as const, assigneeAgentId: agentId }))).returning(); + const live = (index: number) => ({ companyId, agentId, issueId: tasks[index]!.id, status: "running" as const, + invocationSource: "automation" as const, contextSnapshot: { issueId: tasks[index]!.id }, startedAt: minutesAgo(90) }); + await db.insert(heartbeatRuns).values([live(0), live(1)]); + const [waiting] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "queued", invocationSource: "automation", + contextSnapshot: { issueId: tasks[3]!.id }, createdAt: minutesAgo(20) }).returning(); + await db.insert(heartbeatRuns).values({ companyId, agentId, status: "queued", invocationSource: "automation", + contextSnapshot: { issueId: tasks[0]!.id }, createdAt: minutesAgo(25) }); + expect((await listUndispatchedQueuedRuns(db)).map(run => run.runId)).toEqual([waiting!.id]); + await db.insert(heartbeatRuns).values(live(2)); + expect(await listUndispatchedQueuedRuns(db)).toEqual([]); + }); + + it("counts an agent's live runs in one grouped aggregate, not once per queued run", async () => { + await db.insert(heartbeatRuns).values(Array.from({ length: 5 }, () => + ({ companyId, agentId, status: "queued" as const, invocationSource: "automation" as const, createdAt: minutesAgo(20) }))); + await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running", invocationSource: "automation" }); + const query = undispatchedQueuedRunPageQuery(db, undefined, null, null, 500).toSQL(); + const [row] = await db.$client.unsafe(`explain (format json) ${query.sql}`, query.params as never[]); + type PlanNode = { "Node Type": string; "Parent Relationship"?: string; Plans?: PlanNode[] }; + const aggregates: PlanNode[] = []; + const subPlanAggregates: PlanNode[] = []; + const walk = (node: PlanNode, inSubPlan: boolean) => { + const subPlan = inSubPlan || node["Parent Relationship"] === "SubPlan"; + if (node["Node Type"] === "Aggregate") (subPlan ? subPlanAggregates : aggregates).push(node); + for (const child of node.Plans ?? []) walk(child, subPlan); + }; + walk((row as unknown as { "QUERY PLAN": Array<{ Plan: PlanNode }> })["QUERY PLAN"][0]!.Plan, false); + expect(aggregates).toHaveLength(1); + expect(subPlanAggregates).toEqual([]); + }); + + it("uses the agent's single-run limit for a runner provider that runs one turn at a time", async () => { + await db.update(agents).set({ adapterType: "paperclip_runner", adapterConfig: { provider: "openai_dot" }, + runtimeConfig: { heartbeat: { maxConcurrentRuns: 5 } } }).where(eq(agents.id, agentId)); + await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running", invocationSource: "automation", + startedAt: minutesAgo(90) }); + await db.insert(heartbeatRuns).values({ companyId, agentId, status: "queued", invocationSource: "automation", + createdAt: minutesAgo(20) }); + expect(await listUndispatchedQueuedRuns(db)).toEqual([]); + }); + it("keeps the earliest grace deadline when the queued-run check reaches its cap, and rescans once it ends", async () => { const createdAt = minutesAgo(1); await db.insert(heartbeatRuns).values(Array.from({ length: 2_000 }, () => @@ -1128,7 +1174,8 @@ describe("report-only ownership reconciliation", () => { const otherAgent = randomUUID(); const pausedAgent = randomUUID(); await db.insert(agents).values([ - { id: otherAgent, companyId, name: "Busy", role: "engineer", adapterType: "codex_local" }, + { id: otherAgent, companyId, name: "Busy", role: "engineer", adapterType: "codex_local", + runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } }, { id: pausedAgent, companyId, name: "Paused", role: "engineer", adapterType: "codex_local", status: "paused" }, ]); await db.insert(heartbeatRuns).values([ diff --git a/server/src/__tests__/recovery-wake-herd.test.ts b/server/src/__tests__/recovery-wake-herd.test.ts index 117fde197b7..07c27485ebe 100644 --- a/server/src/__tests__/recovery-wake-herd.test.ts +++ b/server/src/__tests__/recovery-wake-herd.test.ts @@ -111,6 +111,52 @@ suite("recovery wake herd", () => { } }, 120_000); + async function queuedSystemRun(agentId: string, createdMinutesAgo: number, wakeReason: string) { + const run = await queuedRecoveryRun(agentId, createdMinutesAgo); + const [row] = await db.update(heartbeatRuns).set({ contextSnapshot: { ...run.contextSnapshot, wakeReason } }) + .where(eq(heartbeatRuns.id, run.id)).returning(); + return row!; + } + + it("starts a queued run that a person's comment joined ahead of older system work", async () => { + const agentId = await seedCompany(1); + const older: Array = []; + for (let index = 0; index < 3; index += 1) older.push(await queuedSystemRun(agentId, 60 - index, "issue_assigned")); + const joined = await queuedSystemRun(agentId, 50, "issue_reopened_via_comment"); + await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "automation", triggerDetail: "system", + reason: "issue_execution_same_name", status: "coalesced", coalescedCount: 1, requestedByActorType: "user", + requestedByActorId: userId, runId: joined.id, finishedAt: new Date(), payload: { issueId: joined.issueId } }); + const heartbeat = heartbeatService(db); + try { + await heartbeat.resumeQueuedRuns(); + expect((await readRun(joined.id)).startedAt).not.toBeNull(); + for (const run of older) expect(await readRun(run.id)).toMatchObject({ status: "queued", startedAt: null }); + } finally { + await heartbeat.drainActiveRunExecutions(); + } + }, 120_000); + + it("starts a queued run that adopted a person's comment saved before every queued run", async () => { + const agentId = await seedCompany(1); + const [saved] = await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "automation", triggerDetail: "system", + reason: "issue_commented", status: "deferred_issue_execution", requestedByActorType: "user", requestedByActorId: userId, + requestedAt: new Date(Date.now() - 90 * 60_000), createdAt: new Date(Date.now() - 90 * 60_000), + updatedAt: new Date(Date.now() - 90 * 60_000), payload: {} }).returning(); + const older: Array = []; + for (let index = 0; index < 3; index += 1) older.push(await queuedSystemRun(agentId, 60 - index, "issue_assigned")); + const adopting = await queuedSystemRun(agentId, 50, "issue_commented"); + await db.update(agentWakeupRequests).set({ runId: adopting.id, status: "queued", updatedAt: new Date(), + payload: { issueId: adopting.issueId } }).where(eq(agentWakeupRequests.id, saved!.id)); + const heartbeat = heartbeatService(db); + try { + await heartbeat.resumeQueuedRuns(); + expect((await readRun(adopting.id)).startedAt).not.toBeNull(); + for (const run of older) expect(await readRun(run.id)).toMatchObject({ status: "queued", startedAt: null }); + } finally { + await heartbeat.drainActiveRunExecutions(); + } + }, 120_000); + async function pendingContinuation(agentId: string, endedMinutesAgo: number) { const issue = await task(agentId, "todo"); const source = await sourceRun(agentId, issue.id, endedMinutesAgo); diff --git a/server/src/services/heartbeat/queue.ts b/server/src/services/heartbeat/queue.ts index fe4adc984b1..1be0c728e84 100644 --- a/server/src/services/heartbeat/queue.ts +++ b/server/src/services/heartbeat/queue.ts @@ -2360,10 +2360,18 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende eq(agentWakeupRequests.requestedByActorType, "user"))) : []; const userRequestIds = new Set(userRequests.map(row => row.id)); + const joinedUserRequests = queuedRuns.length > 1 + ? await db.select({ runId: agentWakeupRequests.runId }).from(agentWakeupRequests).where(and( + eq(agentWakeupRequests.companyId, agent.companyId), inArray(agentWakeupRequests.runId, queuedRuns.map(run => run.id)), + eq(agentWakeupRequests.agentId, agent.id), eq(agentWakeupRequests.requestedByActorType, "user"))) + : []; + const userRunIds = new Set(joinedUserRequests.flatMap(row => row.runId ? [row.runId] : [])); + const requestedByUser = (run: typeof heartbeatRuns.$inferSelect) => + userRunIds.has(run.id) || Boolean(run.wakeupRequestId && userRequestIds.has(run.wakeupRequestId)); const sourceRunIdOf = (run: typeof heartbeatRuns.$inferSelect) => readNonEmptyString(parseObject(run.contextSnapshot).previousRunId) ?? run.retryOfRunId ?? null; const recoveryRuns = queuedRuns.filter(run => isSystemRecoveryRun({ wakeReason: wakeReasonOf(run), - requestedByUser: Boolean(run.wakeupRequestId && userRequestIds.has(run.wakeupRequestId)) })); + requestedByUser: requestedByUser(run) })); const sourceRunIds = [...new Set(recoveryRuns.flatMap(run => sourceRunIdOf(run) ?? []))]; const sourceRows = queuedRuns.length > 1 && sourceRunIds.length ? await db.select({ id: heartbeatRuns.id, finishedAt: heartbeatRuns.finishedAt }).from(heartbeatRuns) @@ -2392,7 +2400,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende id: run.id, createdAt: run.createdAt, wakeReason: wakeReasonOf(run), - requestedByUser: Boolean(run.wakeupRequestId && userRequestIds.has(run.wakeupRequestId)), + requestedByUser: requestedByUser(run), sourceEndedAt: sourceRunId ? sourceEndedAt.get(sourceRunId) ?? null : null, readinessRank: issueId ? (ready ? (issue?.status === "in_progress" ? 0 : 1) : 3) : 2, priorityRank: issueRunPriorityRank(issue?.priority), diff --git a/server/src/services/heartbeat/timer-policy.ts b/server/src/services/heartbeat/timer-policy.ts index 51b785011ce..2eadc93bd0a 100644 --- a/server/src/services/heartbeat/timer-policy.ts +++ b/server/src/services/heartbeat/timer-policy.ts @@ -9,6 +9,7 @@ const HEARTBEAT_MAX_CONCURRENT_RUNS_DEFAULT = AGENT_DEFAULT_MAX_CONCURRENT_RUNS; const HEARTBEAT_MAX_CONCURRENT_RUNS_MIN = 1; const HEARTBEAT_MAX_CONCURRENT_RUNS_MAX = 50; const TIMER_ACTIONABLE_ISSUE_STATUSES = ["todo", "in_progress"] as const; +const SINGLE_RUN_RUNNER_PROVIDERS = ["openai_dot", "grok_bot", "muse"]; function normalizeMaxConcurrentRuns(value: unknown) { const parsed = Math.floor( @@ -21,6 +22,17 @@ function normalizeMaxConcurrentRuns(value: unknown) { ); } +/** The same limit as `parseHeartbeatPolicy(agent).maxConcurrentRuns`, read in SQL. */ +export function maxConcurrentRunsSql(): SQL { + const configured = sql`${agents.runtimeConfig}->'heartbeat'->'maxConcurrentRuns'`; + return sql`(case + when ${agents.adapterType} = 'paperclip_runner' + and ${agents.adapterConfig}->>'provider' in (${sql.join(SINGLE_RUN_RUNNER_PROVIDERS.map(provider => sql`${provider}`), sql`, `)}) then 1 + when jsonb_typeof(${configured}) = 'number' then greatest(${HEARTBEAT_MAX_CONCURRENT_RUNS_MIN}, + least(${HEARTBEAT_MAX_CONCURRENT_RUNS_MAX}, floor((${configured})::text::numeric))) + else ${HEARTBEAT_MAX_CONCURRENT_RUNS_DEFAULT} end)`; +} + export function heartbeatTimerPolicyIsWellFormed(agent: typeof agents.$inferSelect): boolean { const record = (v: unknown): v is Record => v !== null && typeof v === "object" && !Array.isArray(v); if (!record(agent.runtimeConfig)) return false; @@ -48,7 +60,7 @@ export function parseHeartbeatPolicy(agent: typeof agents.$inferSelect) { intervalSec: Math.max(0, asNumber(heartbeat.intervalSec, 0)), wakeOnDemand: isHeartbeatWakeOnDemandEnabled(agent), maxConcurrentRuns: agent.adapterType === "paperclip_runner" && - ["openai_dot", "grok_bot", "muse"].includes(String(parseObject(agent.adapterConfig).provider)) + SINGLE_RUN_RUNNER_PROVIDERS.includes(String(parseObject(agent.adapterConfig).provider)) ? 1 : normalizeMaxConcurrentRuns(heartbeat.maxConcurrentRuns), skipTimerWhenNoActionableWork: asBoolean( heartbeat.skipTimerWhenNoActionableWork ?? diff --git a/server/src/services/queued-run-dispatch-liveness.ts b/server/src/services/queued-run-dispatch-liveness.ts index fe5d2ca09d1..5fc10406880 100644 --- a/server/src/services/queued-run-dispatch-liveness.ts +++ b/server/src/services/queued-run-dispatch-liveness.ts @@ -1,6 +1,7 @@ -import { and, asc, eq, gte, notInArray, sql } from "drizzle-orm"; +import { and, asc, eq, gte, inArray, notInArray, sql } from "drizzle-orm"; import { agents, companies, heartbeatRuns, type Db } from "@paperclipai/db"; import { instanceSettingsService, isTruthyRuntimeEnvValue, resolveWorktreeRunExecutionActivation } from "./instance-settings.js"; +import { maxConcurrentRunsSql } from "./heartbeat/timer-policy.js"; export const QUEUED_RUN_REDRIVE_AFTER_MS = 60_000; export const QUEUED_RUN_REDRIVE_MAX_INTERVAL_MS = 15 * 60_000; @@ -21,9 +22,14 @@ export type UndispatchedQueuedRun = { export type UndispatchedQueuedRunCursor = { at: string; runId: string }; type PageCursor = UndispatchedQueuedRunCursor; -async function listUndispatchedQueuedRunPage(db: Db, companyId: string | undefined, createdAfter: Date | null | undefined, - after: PageCursor | null, limit: number): - Promise> { +/** One page of undispatched queued runs. Live runs are counted once per agent. */ +export function undispatchedQueuedRunPageQuery(db: Db, companyId: string | undefined, createdAfter: Date | null | undefined, + after: PageCursor | null, limit: number) { + const liveRuns = db.select({ companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + count: sql`count(*)`.as("live_count") }).from(heartbeatRuns) + .where(and(inArray(heartbeatRuns.status, ["running", "scheduled_retry"]), + companyId ? eq(heartbeatRuns.companyId, companyId) : undefined)) + .groupBy(heartbeatRuns.companyId, heartbeatRuns.agentId).as("agent_live_runs"); return db.select({ runId: heartbeatRuns.id, companyId: heartbeatRuns.companyId, @@ -34,6 +40,7 @@ async function listUndispatchedQueuedRunPage(db: Db, companyId: string | undefin }).from(heartbeatRuns) .innerJoin(companies, eq(companies.id, heartbeatRuns.companyId)) .innerJoin(agents, and(eq(agents.id, heartbeatRuns.agentId), eq(agents.companyId, heartbeatRuns.companyId))) + .leftJoin(liveRuns, and(eq(liveRuns.companyId, heartbeatRuns.companyId), eq(liveRuns.agentId, heartbeatRuns.agentId))) .where(and( eq(heartbeatRuns.status, "queued"), companyId ? eq(heartbeatRuns.companyId, companyId) : undefined, @@ -42,14 +49,21 @@ async function listUndispatchedQueuedRunPage(db: Db, companyId: string | undefin eq(agents.lifecycleState, "ready"), notInArray(agents.status, NOT_READY_AGENT_STATUSES), after ? sql`(${heartbeatRuns.createdAt}, ${heartbeatRuns.id}) > (${after.at}::timestamptz, ${after.runId}::uuid)` : undefined, - sql`not exists (select 1 from ${heartbeatRuns} live - where live.company_id = ${heartbeatRuns.companyId} and live.agent_id = ${heartbeatRuns.agentId} - and live.status in ('running', 'scheduled_retry'))`, + sql`coalesce(${liveRuns.count}, 0) < ${maxConcurrentRunsSql()}`, + sql`not exists (select 1 from ${heartbeatRuns} task_live + where task_live.company_id = ${heartbeatRuns.companyId} + and task_live.context_snapshot->>'issueId' = coalesce(${heartbeatRuns.issueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId') + and task_live.status in ('running', 'scheduled_retry'))`, )) .orderBy(asc(heartbeatRuns.createdAt), asc(heartbeatRuns.id)) .limit(limit); } +async function listUndispatchedQueuedRunPage(db: Db, companyId: string | undefined, createdAfter: Date | null | undefined, + after: PageCursor | null, limit: number): Promise> { + return undispatchedQueuedRunPageQuery(db, companyId, createdAfter, after, limit); +} + /** Every undispatched queued run, read page by page with no cap; one * company's runs when `companyId` is given. `createdAfter` is the worktree * execution cutoff: dispatch never starts a run created before it. */