mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 21:34:48 +01:00
fix(heartbeat): rank a person's joined comment first and refill free run slots (#21277)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat queue starts each agent's queued runs, up to the agent's `maxConcurrentRuns` > - #21234 starts person-initiated runs before system recovery runs, and it reads "person-initiated" from the run's own wake request > - When a person comments on a task that already has a queued system run, the comment joins that run. The run keeps its system wake, so dispatch ranks the person's message as system work > - Dispatch is event driven. The queued-run re-drive backstop covers only agents with no live run, so an agent with several slots keeps a free slot empty until one of its runs finishes, and runs can take hours > - This pull request makes dispatch read the person wakes that joined each queued run, and makes the re-drive backstop cover an agent that still has a free slot > - The benefit is that a person's comment starts first, and free slots fill within about a minute instead of waiting for an unrelated run to finish ## Linked Issues or Issue Description Refs #21234, #20986 **What happened?** On staging, one agent had `maxConcurrentRuns` 20, 18 running runs and 76 queued runs at 03:12Z. No run started on that agent from 03:01:33Z to 03:16:57Z (15.4 minutes). From 03:11:10Z, after a run finished, at least 2 slots were free. Other agents in the same company started runs during that time. Runs started again only after the next deploy restarted the server. Five queued runs came from a person: comments at 02:43Z, 02:44Z and 02:45Z, and Retries at 02:46Z, 02:49Z and 03:01Z. The comment runs had the wake reasons `issue_reopened_via_comment` and `issue_commented`, the invocation source `automation`, and no person on the run's own wake request. **Expected behavior** A run that carries a person's comment ranks as person-initiated, also when the comment joined an existing queued run. When an agent has a free slot and a queued run is ready, the run starts promptly, without waiting for another run on the agent to finish. **Steps to reproduce** 1. Give an agent `maxConcurrentRuns` 3. Start 2 runs on it. 2. Queue a third run while no dispatch signal reaches the queue (for example, a claim declines it while a gate is still closed, and the gate then clears). 3. See that `listUndispatchedQueuedRuns` returns nothing, because the agent has a live run. The run stays queued until one of the 2 running runs finishes. 4. Separately, queue a system run on a task, then post a person's comment on the task so that the comment wake coalesces into that queued run. Queue older system runs on other tasks of the same single-slot agent. 5. See that dispatch starts the older system runs first. **Root cause** 1. `coalesceIntoActiveExecutionRun` (`server/src/modules/wake-queue/adapters/postgres.ts`) merges the person's comment context into the queued run and records the person's wake as a separate `coalesced` row with `runId` set to that run. The run's own `wakeupRequestId` still points to the system wake. `queuedRunDispatcherForAgent` (`server/src/services/heartbeat/queue.ts`) only read the wake behind `wakeupRequestId`. The task routes do record the comment author as the requester (`requestedByActorType: "user"`). The person was lost only at dispatch time. 2. `listUndispatchedQueuedRunPage` (`server/src/services/queued-run-dispatch-liveness.ts`) skipped every agent with any `running` or `scheduled_retry` run. That rule assumes one slot per agent. For an agent with several slots, a lost or declined dispatch is not re-driven until a run on that agent finishes. This change is not in `claimQueuedRun`, the fresh-isolated preflight, the wake-queue promoter or `fresh-continuation-fallback.ts`. **Paperclip version or commit** Staging Core `ddc0aacad8` (contains #21234 `ade7e0643f`). This branch is based on master `6cb0dd0c6b`. **Deployment mode** Authenticated, public, Paperclip Cloud managed staging. **Agent adapter(s) involved** `paperclip_runner` (Codex native runtime). The fix is adapter-independent. ## What Changed - `queuedRunDispatcherForAgent` also reads person wakes whose `runId` is a queued run of the agent (by `run_id IN (queued run ids)`, so an earlier saved comment that a run adopted is included). A run with such a wake ranks as person-initiated and is never counted as system recovery. - `listUndispatchedQueuedRuns` now returns a queued run when its agent has fewer `running` plus `scheduled_retry` runs than its `maxConcurrentRuns`, and its task has no `running` or `scheduled_retry` run. The re-drive backstop and the ownership I7 check use this query. - The query counts each agent's live runs once, in one grouped subquery that is joined to the queued runs. - Migration 0476 adds `agent_wakeup_requests_company_run_idx` (`company_id`, `run_id`, partial on a non-null run) with `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. It is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS` and classifies as `online_compatible`. `doc/DATABASE.md` describes it. - Added `maxConcurrentRunsSql()` to `timer-policy.ts`. It reads the same limit as `parseHeartbeatPolicy`, including the single-run runner providers, which now share one constant. ## Verification - Red/green: `server/src/__tests__/recovery-wake-herd.test.ts` "starts a queued run that a person's comment joined ahead of older system work". It fails on master (`expected null not to be null`: the older system run started first) and passes with this change. - Red/green: `server/src/__tests__/ownership-reconciliation-report.test.ts` "re-drives a queued run while its agent still has a free slot, and not once every slot is running". It fails on master (`expected [] to deeply equal [ Array(1) ]`) and passes with this change. - Red/green: "starts a queued run that adopted a person's comment saved before every queued run". It fails with a `requestedAt` bound and passes with the `run_id` lookup. - Red/green: "counts an agent's live runs in one grouped aggregate, not once per queued run" runs EXPLAIN on the page query. It fails with a per-row correlated count and passes with the grouped subquery. - New: "uses the agent's single-run limit for a runner provider that runs one turn at a time". - Two existing tests now set `maxConcurrentRuns: 1` so that they keep testing "the agent's only slot is busy". - `npx vitest run src/__tests__/recovery-wake-herd.test.ts src/services/heartbeat/queued-run-dispatch-order.test.ts src/__tests__/ownership-reconciliation-report.test.ts src/services/queued-run-dispatch-liveness.test.ts src/services/heartbeat/dispatch-work.test.ts`: 71 passed. - `npx tsc --noEmit -p server/tsconfig.json`: no errors. ## Risks - The re-drive backstop now also covers multi-slot agents with live runs. Each queued run is re-driven at most on its existing backoff (1 minute, then doubling to 15 minutes). A queued run whose task already has a live run is not re-driven. - The ownership I7 report can now list a queued run on a multi-slot agent that has a free slot. This is the stall that this change targets. - One more indexed read per dispatch pass (`agent_wakeup_requests` by company and run id). The new index is built concurrently, so the previous release keeps writing during the build. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, read-only staging API reads in a browser tab). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
74096c19f4
commit
49f0030f06
11 files changed
+785
-14
No files matched your search
@@ -507,6 +507,12 @@ indexes also serve the comment and delivery `on delete set null` actions. Issue
|
||||
lists share one external review-wait read per issue between the review and
|
||||
blocked-inbox attention maps.
|
||||
|
||||
Migration 0476 builds `agent_wakeup_requests_company_run_idx` concurrently. It
|
||||
keys wake requests by company and `run_id`, partial on a non-null run. Queued-run
|
||||
dispatch finds the person wakes that joined or were adopted by each queued run
|
||||
through it, so the read is bounded by the queue size, not by the agent's wake
|
||||
history.
|
||||
|
||||
Migration 0457 builds `heartbeat_runs_company_native_session_idx` concurrently.
|
||||
It keys runs by company and `native_session_id`, partial on a non-null session.
|
||||
Native retention, recovery, resume and archive reads find a session's runs
|
||||
|
||||
@@ -690,6 +690,7 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([
|
||||
"0461_created_from_legacy_origin_run_index.sql",
|
||||
"0470_heartbeat_runs_queued_created_index.sql",
|
||||
"0472_runner_api_capture_pending_deletes.sql",
|
||||
"0476_wakeup_request_run_index.sql",
|
||||
"0469_zippy_photon.sql",
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
CREATE INDEX CONCURRENTLY IF NOT EXISTS "agent_wakeup_requests_company_run_idx" ON "agent_wakeup_requests" USING btree ("company_id","run_id") WHERE "agent_wakeup_requests"."run_id" IS NOT NULL;
|
||||
+627
-2
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"id": "671a422a-0b93-4155-b097-9d35a4fd2b2b",
|
||||
"prevId": "f0e17e47-6217-4080-b6b6-9d83b33e9ced",
|
||||
"id": "82dcdc57-0796-4162-9451-8ea08a72199e",
|
||||
"prevId": "ee1f1f66-0cbe-4b18-b364-5e96708a1f0b",
|
||||
"version": "7",
|
||||
"dialect": "postgresql",
|
||||
"tables": {
|
||||
@@ -3954,6 +3954,28 @@
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"agent_wakeup_requests_company_run_idx": {
|
||||
"name": "agent_wakeup_requests_company_run_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "company_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "run_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"agent_wakeup_requests\".\"run_id\" IS NOT NULL",
|
||||
"concurrently": true,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"agent_wakeup_requests_review_path_recovery_idempotency_uq": {
|
||||
"name": "agent_wakeup_requests_review_path_recovery_idempotency_uq",
|
||||
"columns": [
|
||||
@@ -53686,6 +53708,12 @@
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"pending_delete_object_key": {
|
||||
"name": "pending_delete_object_key",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"created_at": {
|
||||
"name": "created_at",
|
||||
"type": "timestamp with time zone",
|
||||
@@ -53709,6 +53737,21 @@
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"runner_api_response_reservations_asset_idx": {
|
||||
"name": "runner_api_response_reservations_asset_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "asset_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
@@ -53921,6 +53964,7 @@
|
||||
}
|
||||
],
|
||||
"isUnique": true,
|
||||
"where": "\"sandbox_creation_operations\".\"state\" <> 'resolved'",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
@@ -57514,6 +57558,12 @@
|
||||
"notNull": true,
|
||||
"default": "'{}'::jsonb"
|
||||
},
|
||||
"ui_meta": {
|
||||
"name": "ui_meta",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"risk_level": {
|
||||
"name": "risk_level",
|
||||
"type": "text",
|
||||
@@ -59639,6 +59689,253 @@
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.tool_mcp_app_views": {
|
||||
"name": "tool_mcp_app_views",
|
||||
"schema": "",
|
||||
"columns": {
|
||||
"id": {
|
||||
"name": "id",
|
||||
"type": "uuid",
|
||||
"primaryKey": true,
|
||||
"notNull": true,
|
||||
"default": "gen_random_uuid()"
|
||||
},
|
||||
"company_id": {
|
||||
"name": "company_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"invocation_id": {
|
||||
"name": "invocation_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"issue_id": {
|
||||
"name": "issue_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"run_id": {
|
||||
"name": "run_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"agent_id": {
|
||||
"name": "agent_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"responsible_user_id": {
|
||||
"name": "responsible_user_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"connection_id": {
|
||||
"name": "connection_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"catalog_entry_id": {
|
||||
"name": "catalog_entry_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"application_name": {
|
||||
"name": "application_name",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"tool_name": {
|
||||
"name": "tool_name",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"upstream_tool_name": {
|
||||
"name": "upstream_tool_name",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"resource_uri": {
|
||||
"name": "resource_uri",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"tool_input": {
|
||||
"name": "tool_input",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "'{}'::jsonb"
|
||||
},
|
||||
"tool_result": {
|
||||
"name": "tool_result",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"created_at": {
|
||||
"name": "created_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"tool_mcp_app_views_issue_idx": {
|
||||
"name": "tool_mcp_app_views_issue_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "company_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "issue_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "created_at",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"tool_mcp_app_views_connection_idx": {
|
||||
"name": "tool_mcp_app_views_connection_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "connection_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
"tool_mcp_app_views_company_id_companies_id_fk": {
|
||||
"name": "tool_mcp_app_views_company_id_companies_id_fk",
|
||||
"tableFrom": "tool_mcp_app_views",
|
||||
"tableTo": "companies",
|
||||
"columnsFrom": [
|
||||
"company_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
"tool_mcp_app_views_invocation_id_tool_invocations_id_fk": {
|
||||
"name": "tool_mcp_app_views_invocation_id_tool_invocations_id_fk",
|
||||
"tableFrom": "tool_mcp_app_views",
|
||||
"tableTo": "tool_invocations",
|
||||
"columnsFrom": [
|
||||
"invocation_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "set null",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
"tool_mcp_app_views_issue_id_issues_id_fk": {
|
||||
"name": "tool_mcp_app_views_issue_id_issues_id_fk",
|
||||
"tableFrom": "tool_mcp_app_views",
|
||||
"tableTo": "issues",
|
||||
"columnsFrom": [
|
||||
"issue_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
"tool_mcp_app_views_run_id_heartbeat_runs_id_fk": {
|
||||
"name": "tool_mcp_app_views_run_id_heartbeat_runs_id_fk",
|
||||
"tableFrom": "tool_mcp_app_views",
|
||||
"tableTo": "heartbeat_runs",
|
||||
"columnsFrom": [
|
||||
"run_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "set null",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
"tool_mcp_app_views_agent_id_agents_id_fk": {
|
||||
"name": "tool_mcp_app_views_agent_id_agents_id_fk",
|
||||
"tableFrom": "tool_mcp_app_views",
|
||||
"tableTo": "agents",
|
||||
"columnsFrom": [
|
||||
"agent_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "set null",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
"tool_mcp_app_views_connection_id_tool_connections_id_fk": {
|
||||
"name": "tool_mcp_app_views_connection_id_tool_connections_id_fk",
|
||||
"tableFrom": "tool_mcp_app_views",
|
||||
"tableTo": "tool_connections",
|
||||
"columnsFrom": [
|
||||
"connection_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
"tool_mcp_app_views_catalog_entry_id_tool_catalog_entries_id_fk": {
|
||||
"name": "tool_mcp_app_views_catalog_entry_id_tool_catalog_entries_id_fk",
|
||||
"tableFrom": "tool_mcp_app_views",
|
||||
"tableTo": "tool_catalog_entries",
|
||||
"columnsFrom": [
|
||||
"catalog_entry_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "set null",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
},
|
||||
"compositePrimaryKeys": {},
|
||||
"uniqueConstraints": {},
|
||||
"policies": {},
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.tool_mcp_gateway_tokens": {
|
||||
"name": "tool_mcp_gateway_tokens",
|
||||
"schema": "",
|
||||
@@ -60353,6 +60650,334 @@
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.tool_mcp_skill_files": {
|
||||
"name": "tool_mcp_skill_files",
|
||||
"schema": "",
|
||||
"columns": {
|
||||
"id": {
|
||||
"name": "id",
|
||||
"type": "uuid",
|
||||
"primaryKey": true,
|
||||
"notNull": true,
|
||||
"default": "gen_random_uuid()"
|
||||
},
|
||||
"company_id": {
|
||||
"name": "company_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"connection_id": {
|
||||
"name": "connection_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"file_uri": {
|
||||
"name": "file_uri",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"digest": {
|
||||
"name": "digest",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"size": {
|
||||
"name": "size",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"content_base64": {
|
||||
"name": "content_base64",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"created_at": {
|
||||
"name": "created_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"tool_mcp_skill_files_digest_idx": {
|
||||
"name": "tool_mcp_skill_files_digest_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "company_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "connection_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "file_uri",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "digest",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": true,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
"tool_mcp_skill_files_company_id_companies_id_fk": {
|
||||
"name": "tool_mcp_skill_files_company_id_companies_id_fk",
|
||||
"tableFrom": "tool_mcp_skill_files",
|
||||
"tableTo": "companies",
|
||||
"columnsFrom": [
|
||||
"company_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
"tool_mcp_skill_files_connection_id_tool_connections_id_fk": {
|
||||
"name": "tool_mcp_skill_files_connection_id_tool_connections_id_fk",
|
||||
"tableFrom": "tool_mcp_skill_files",
|
||||
"tableTo": "tool_connections",
|
||||
"columnsFrom": [
|
||||
"connection_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
},
|
||||
"compositePrimaryKeys": {},
|
||||
"uniqueConstraints": {},
|
||||
"policies": {},
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.tool_mcp_skills": {
|
||||
"name": "tool_mcp_skills",
|
||||
"schema": "",
|
||||
"columns": {
|
||||
"id": {
|
||||
"name": "id",
|
||||
"type": "uuid",
|
||||
"primaryKey": true,
|
||||
"notNull": true,
|
||||
"default": "gen_random_uuid()"
|
||||
},
|
||||
"company_id": {
|
||||
"name": "company_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"connection_id": {
|
||||
"name": "connection_id",
|
||||
"type": "uuid",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"skill_uri": {
|
||||
"name": "skill_uri",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"name": {
|
||||
"name": "name",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"description": {
|
||||
"name": "description",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "''"
|
||||
},
|
||||
"frontmatter": {
|
||||
"name": "frontmatter",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"manifest": {
|
||||
"name": "manifest",
|
||||
"type": "jsonb",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"manifest_digest": {
|
||||
"name": "manifest_digest",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"status": {
|
||||
"name": "status",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "'pending_review'"
|
||||
},
|
||||
"status_reason": {
|
||||
"name": "status_reason",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"approved_manifest_digest": {
|
||||
"name": "approved_manifest_digest",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"allowed_tools_approved": {
|
||||
"name": "allowed_tools_approved",
|
||||
"type": "boolean",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": false
|
||||
},
|
||||
"reviewed_at": {
|
||||
"name": "reviewed_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"reviewed_by_user_id": {
|
||||
"name": "reviewed_by_user_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"last_seen_at": {
|
||||
"name": "last_seen_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
},
|
||||
"created_at": {
|
||||
"name": "created_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
},
|
||||
"updated_at": {
|
||||
"name": "updated_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"tool_mcp_skills_connection_uri_idx": {
|
||||
"name": "tool_mcp_skills_connection_uri_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "company_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "connection_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "skill_uri",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": true,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"tool_mcp_skills_company_status_idx": {
|
||||
"name": "tool_mcp_skills_company_status_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "company_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "status",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
"tool_mcp_skills_company_id_companies_id_fk": {
|
||||
"name": "tool_mcp_skills_company_id_companies_id_fk",
|
||||
"tableFrom": "tool_mcp_skills",
|
||||
"tableTo": "companies",
|
||||
"columnsFrom": [
|
||||
"company_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
},
|
||||
"tool_mcp_skills_connection_id_tool_connections_id_fk": {
|
||||
"name": "tool_mcp_skills_connection_id_tool_connections_id_fk",
|
||||
"tableFrom": "tool_mcp_skills",
|
||||
"tableTo": "tool_connections",
|
||||
"columnsFrom": [
|
||||
"connection_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
},
|
||||
"compositePrimaryKeys": {},
|
||||
"uniqueConstraints": {},
|
||||
"policies": {},
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.tool_oauth_states": {
|
||||
"name": "tool_oauth_states",
|
||||
"schema": "",
|
||||
+7
@@ -3319,6 +3319,13 @@
|
||||
"when": 1794532772498,
|
||||
"tag": "0475_sandbox_creation_owner_unresolved",
|
||||
"breakpoints": true
|
||||
},
|
||||
{
|
||||
"idx": 476,
|
||||
"version": "7",
|
||||
"when": 1794553127740,
|
||||
"tag": "0476_wakeup_request_run_index",
|
||||
"breakpoints": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -59,6 +59,10 @@ export const agentWakeupRequests = pgTable(
|
||||
table.agentId,
|
||||
table.requestedAt,
|
||||
),
|
||||
companyRunIdx: index("agent_wakeup_requests_company_run_idx")
|
||||
.on(table.companyId, table.runId)
|
||||
.concurrently()
|
||||
.where(sql`${table.runId} IS NOT NULL`),
|
||||
reviewPathRecoveryIdempotencyUq: uniqueIndex(
|
||||
"agent_wakeup_requests_review_path_recovery_idempotency_uq",
|
||||
)
|
||||
|
||||
@@ -34,7 +34,7 @@ import { applyOwnershipReportBackstops, registerOwnershipReconciliationWork } fr
|
||||
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
|
||||
import { notifyNativeFinalizationWork } from "../services/native-finalization-notifications.js";
|
||||
import { instanceSettingsService } from "../services/instance-settings.js";
|
||||
import { listUndispatchedQueuedRuns, readUndispatchedQueuedRuns } from "../services/queued-run-dispatch-liveness.js";
|
||||
import { listUndispatchedQueuedRuns, readUndispatchedQueuedRuns, undispatchedQueuedRunPageQuery } from "../services/queued-run-dispatch-liveness.js";
|
||||
import { releaseLapsedCoordinatorOwner } from "../services/workspace-recovery-stop-proof.js";
|
||||
import { subscribeDatabaseWork } from "@paperclipai/db/work-signals";
|
||||
import {
|
||||
@@ -1073,6 +1073,7 @@ describe("report-only ownership reconciliation", () => {
|
||||
});
|
||||
|
||||
it("reads undispatched queued runs in bounded slices that resume from a cursor, and skips an agent with a scheduled retry", async () => {
|
||||
await db.update(agents).set({ runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } }).where(eq(agents.id, agentId));
|
||||
const created = [30, 29, 28].map(minutesAgo);
|
||||
const runs = await db.insert(heartbeatRuns).values(created.map(createdAt =>
|
||||
({ companyId, agentId, status: "queued", invocationSource: "automation", createdAt }))).returning();
|
||||
@@ -1086,6 +1087,51 @@ describe("report-only ownership reconciliation", () => {
|
||||
expect(await listUndispatchedQueuedRuns(db)).toEqual([]);
|
||||
});
|
||||
|
||||
it("re-drives a queued run while its agent still has a free slot, and not once every slot is running", async () => {
|
||||
await db.update(agents).set({ runtimeConfig: { heartbeat: { maxConcurrentRuns: 3 } } }).where(eq(agents.id, agentId));
|
||||
const tasks = await db.insert(issues).values([0, 1, 2, 3].map(index =>
|
||||
({ companyId, title: `Task ${index}`, status: "in_progress" as const, assigneeAgentId: agentId }))).returning();
|
||||
const live = (index: number) => ({ companyId, agentId, issueId: tasks[index]!.id, status: "running" as const,
|
||||
invocationSource: "automation" as const, contextSnapshot: { issueId: tasks[index]!.id }, startedAt: minutesAgo(90) });
|
||||
await db.insert(heartbeatRuns).values([live(0), live(1)]);
|
||||
const [waiting] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "queued", invocationSource: "automation",
|
||||
contextSnapshot: { issueId: tasks[3]!.id }, createdAt: minutesAgo(20) }).returning();
|
||||
await db.insert(heartbeatRuns).values({ companyId, agentId, status: "queued", invocationSource: "automation",
|
||||
contextSnapshot: { issueId: tasks[0]!.id }, createdAt: minutesAgo(25) });
|
||||
expect((await listUndispatchedQueuedRuns(db)).map(run => run.runId)).toEqual([waiting!.id]);
|
||||
await db.insert(heartbeatRuns).values(live(2));
|
||||
expect(await listUndispatchedQueuedRuns(db)).toEqual([]);
|
||||
});
|
||||
|
||||
it("counts an agent's live runs in one grouped aggregate, not once per queued run", async () => {
|
||||
await db.insert(heartbeatRuns).values(Array.from({ length: 5 }, () =>
|
||||
({ companyId, agentId, status: "queued" as const, invocationSource: "automation" as const, createdAt: minutesAgo(20) })));
|
||||
await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running", invocationSource: "automation" });
|
||||
const query = undispatchedQueuedRunPageQuery(db, undefined, null, null, 500).toSQL();
|
||||
const [row] = await db.$client.unsafe(`explain (format json) ${query.sql}`, query.params as never[]);
|
||||
type PlanNode = { "Node Type": string; "Parent Relationship"?: string; Plans?: PlanNode[] };
|
||||
const aggregates: PlanNode[] = [];
|
||||
const subPlanAggregates: PlanNode[] = [];
|
||||
const walk = (node: PlanNode, inSubPlan: boolean) => {
|
||||
const subPlan = inSubPlan || node["Parent Relationship"] === "SubPlan";
|
||||
if (node["Node Type"] === "Aggregate") (subPlan ? subPlanAggregates : aggregates).push(node);
|
||||
for (const child of node.Plans ?? []) walk(child, subPlan);
|
||||
};
|
||||
walk((row as unknown as { "QUERY PLAN": Array<{ Plan: PlanNode }> })["QUERY PLAN"][0]!.Plan, false);
|
||||
expect(aggregates).toHaveLength(1);
|
||||
expect(subPlanAggregates).toEqual([]);
|
||||
});
|
||||
|
||||
it("uses the agent's single-run limit for a runner provider that runs one turn at a time", async () => {
|
||||
await db.update(agents).set({ adapterType: "paperclip_runner", adapterConfig: { provider: "openai_dot" },
|
||||
runtimeConfig: { heartbeat: { maxConcurrentRuns: 5 } } }).where(eq(agents.id, agentId));
|
||||
await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running", invocationSource: "automation",
|
||||
startedAt: minutesAgo(90) });
|
||||
await db.insert(heartbeatRuns).values({ companyId, agentId, status: "queued", invocationSource: "automation",
|
||||
createdAt: minutesAgo(20) });
|
||||
expect(await listUndispatchedQueuedRuns(db)).toEqual([]);
|
||||
});
|
||||
|
||||
it("keeps the earliest grace deadline when the queued-run check reaches its cap, and rescans once it ends", async () => {
|
||||
const createdAt = minutesAgo(1);
|
||||
await db.insert(heartbeatRuns).values(Array.from({ length: 2_000 }, () =>
|
||||
@@ -1128,7 +1174,8 @@ describe("report-only ownership reconciliation", () => {
|
||||
const otherAgent = randomUUID();
|
||||
const pausedAgent = randomUUID();
|
||||
await db.insert(agents).values([
|
||||
{ id: otherAgent, companyId, name: "Busy", role: "engineer", adapterType: "codex_local" },
|
||||
{ id: otherAgent, companyId, name: "Busy", role: "engineer", adapterType: "codex_local",
|
||||
runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } },
|
||||
{ id: pausedAgent, companyId, name: "Paused", role: "engineer", adapterType: "codex_local", status: "paused" },
|
||||
]);
|
||||
await db.insert(heartbeatRuns).values([
|
||||
|
||||
@@ -111,6 +111,52 @@ suite("recovery wake herd", () => {
|
||||
}
|
||||
}, 120_000);
|
||||
|
||||
async function queuedSystemRun(agentId: string, createdMinutesAgo: number, wakeReason: string) {
|
||||
const run = await queuedRecoveryRun(agentId, createdMinutesAgo);
|
||||
const [row] = await db.update(heartbeatRuns).set({ contextSnapshot: { ...run.contextSnapshot, wakeReason } })
|
||||
.where(eq(heartbeatRuns.id, run.id)).returning();
|
||||
return row!;
|
||||
}
|
||||
|
||||
it("starts a queued run that a person's comment joined ahead of older system work", async () => {
|
||||
const agentId = await seedCompany(1);
|
||||
const older: Array<typeof heartbeatRuns.$inferSelect> = [];
|
||||
for (let index = 0; index < 3; index += 1) older.push(await queuedSystemRun(agentId, 60 - index, "issue_assigned"));
|
||||
const joined = await queuedSystemRun(agentId, 50, "issue_reopened_via_comment");
|
||||
await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "automation", triggerDetail: "system",
|
||||
reason: "issue_execution_same_name", status: "coalesced", coalescedCount: 1, requestedByActorType: "user",
|
||||
requestedByActorId: userId, runId: joined.id, finishedAt: new Date(), payload: { issueId: joined.issueId } });
|
||||
const heartbeat = heartbeatService(db);
|
||||
try {
|
||||
await heartbeat.resumeQueuedRuns();
|
||||
expect((await readRun(joined.id)).startedAt).not.toBeNull();
|
||||
for (const run of older) expect(await readRun(run.id)).toMatchObject({ status: "queued", startedAt: null });
|
||||
} finally {
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
}
|
||||
}, 120_000);
|
||||
|
||||
it("starts a queued run that adopted a person's comment saved before every queued run", async () => {
|
||||
const agentId = await seedCompany(1);
|
||||
const [saved] = await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "automation", triggerDetail: "system",
|
||||
reason: "issue_commented", status: "deferred_issue_execution", requestedByActorType: "user", requestedByActorId: userId,
|
||||
requestedAt: new Date(Date.now() - 90 * 60_000), createdAt: new Date(Date.now() - 90 * 60_000),
|
||||
updatedAt: new Date(Date.now() - 90 * 60_000), payload: {} }).returning();
|
||||
const older: Array<typeof heartbeatRuns.$inferSelect> = [];
|
||||
for (let index = 0; index < 3; index += 1) older.push(await queuedSystemRun(agentId, 60 - index, "issue_assigned"));
|
||||
const adopting = await queuedSystemRun(agentId, 50, "issue_commented");
|
||||
await db.update(agentWakeupRequests).set({ runId: adopting.id, status: "queued", updatedAt: new Date(),
|
||||
payload: { issueId: adopting.issueId } }).where(eq(agentWakeupRequests.id, saved!.id));
|
||||
const heartbeat = heartbeatService(db);
|
||||
try {
|
||||
await heartbeat.resumeQueuedRuns();
|
||||
expect((await readRun(adopting.id)).startedAt).not.toBeNull();
|
||||
for (const run of older) expect(await readRun(run.id)).toMatchObject({ status: "queued", startedAt: null });
|
||||
} finally {
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
}
|
||||
}, 120_000);
|
||||
|
||||
async function pendingContinuation(agentId: string, endedMinutesAgo: number) {
|
||||
const issue = await task(agentId, "todo");
|
||||
const source = await sourceRun(agentId, issue.id, endedMinutesAgo);
|
||||
|
||||
@@ -2360,10 +2360,18 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
|
||||
eq(agentWakeupRequests.requestedByActorType, "user")))
|
||||
: [];
|
||||
const userRequestIds = new Set(userRequests.map(row => row.id));
|
||||
const joinedUserRequests = queuedRuns.length > 1
|
||||
? await db.select({ runId: agentWakeupRequests.runId }).from(agentWakeupRequests).where(and(
|
||||
eq(agentWakeupRequests.companyId, agent.companyId), inArray(agentWakeupRequests.runId, queuedRuns.map(run => run.id)),
|
||||
eq(agentWakeupRequests.agentId, agent.id), eq(agentWakeupRequests.requestedByActorType, "user")))
|
||||
: [];
|
||||
const userRunIds = new Set(joinedUserRequests.flatMap(row => row.runId ? [row.runId] : []));
|
||||
const requestedByUser = (run: typeof heartbeatRuns.$inferSelect) =>
|
||||
userRunIds.has(run.id) || Boolean(run.wakeupRequestId && userRequestIds.has(run.wakeupRequestId));
|
||||
const sourceRunIdOf = (run: typeof heartbeatRuns.$inferSelect) =>
|
||||
readNonEmptyString(parseObject(run.contextSnapshot).previousRunId) ?? run.retryOfRunId ?? null;
|
||||
const recoveryRuns = queuedRuns.filter(run => isSystemRecoveryRun({ wakeReason: wakeReasonOf(run),
|
||||
requestedByUser: Boolean(run.wakeupRequestId && userRequestIds.has(run.wakeupRequestId)) }));
|
||||
requestedByUser: requestedByUser(run) }));
|
||||
const sourceRunIds = [...new Set(recoveryRuns.flatMap(run => sourceRunIdOf(run) ?? []))];
|
||||
const sourceRows = queuedRuns.length > 1 && sourceRunIds.length
|
||||
? await db.select({ id: heartbeatRuns.id, finishedAt: heartbeatRuns.finishedAt }).from(heartbeatRuns)
|
||||
@@ -2392,7 +2400,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
|
||||
id: run.id,
|
||||
createdAt: run.createdAt,
|
||||
wakeReason: wakeReasonOf(run),
|
||||
requestedByUser: Boolean(run.wakeupRequestId && userRequestIds.has(run.wakeupRequestId)),
|
||||
requestedByUser: requestedByUser(run),
|
||||
sourceEndedAt: sourceRunId ? sourceEndedAt.get(sourceRunId) ?? null : null,
|
||||
readinessRank: issueId ? (ready ? (issue?.status === "in_progress" ? 0 : 1) : 3) : 2,
|
||||
priorityRank: issueRunPriorityRank(issue?.priority),
|
||||
|
||||
@@ -9,6 +9,7 @@ const HEARTBEAT_MAX_CONCURRENT_RUNS_DEFAULT = AGENT_DEFAULT_MAX_CONCURRENT_RUNS;
|
||||
const HEARTBEAT_MAX_CONCURRENT_RUNS_MIN = 1;
|
||||
const HEARTBEAT_MAX_CONCURRENT_RUNS_MAX = 50;
|
||||
const TIMER_ACTIONABLE_ISSUE_STATUSES = ["todo", "in_progress"] as const;
|
||||
const SINGLE_RUN_RUNNER_PROVIDERS = ["openai_dot", "grok_bot", "muse"];
|
||||
|
||||
function normalizeMaxConcurrentRuns(value: unknown) {
|
||||
const parsed = Math.floor(
|
||||
@@ -21,6 +22,17 @@ function normalizeMaxConcurrentRuns(value: unknown) {
|
||||
);
|
||||
}
|
||||
|
||||
/** The same limit as `parseHeartbeatPolicy(agent).maxConcurrentRuns`, read in SQL. */
|
||||
export function maxConcurrentRunsSql(): SQL<number> {
|
||||
const configured = sql`${agents.runtimeConfig}->'heartbeat'->'maxConcurrentRuns'`;
|
||||
return sql<number>`(case
|
||||
when ${agents.adapterType} = 'paperclip_runner'
|
||||
and ${agents.adapterConfig}->>'provider' in (${sql.join(SINGLE_RUN_RUNNER_PROVIDERS.map(provider => sql`${provider}`), sql`, `)}) then 1
|
||||
when jsonb_typeof(${configured}) = 'number' then greatest(${HEARTBEAT_MAX_CONCURRENT_RUNS_MIN},
|
||||
least(${HEARTBEAT_MAX_CONCURRENT_RUNS_MAX}, floor((${configured})::text::numeric)))
|
||||
else ${HEARTBEAT_MAX_CONCURRENT_RUNS_DEFAULT} end)`;
|
||||
}
|
||||
|
||||
export function heartbeatTimerPolicyIsWellFormed(agent: typeof agents.$inferSelect): boolean {
|
||||
const record = (v: unknown): v is Record<string, unknown> => v !== null && typeof v === "object" && !Array.isArray(v);
|
||||
if (!record(agent.runtimeConfig)) return false;
|
||||
@@ -48,7 +60,7 @@ export function parseHeartbeatPolicy(agent: typeof agents.$inferSelect) {
|
||||
intervalSec: Math.max(0, asNumber(heartbeat.intervalSec, 0)),
|
||||
wakeOnDemand: isHeartbeatWakeOnDemandEnabled(agent),
|
||||
maxConcurrentRuns: agent.adapterType === "paperclip_runner" &&
|
||||
["openai_dot", "grok_bot", "muse"].includes(String(parseObject(agent.adapterConfig).provider))
|
||||
SINGLE_RUN_RUNNER_PROVIDERS.includes(String(parseObject(agent.adapterConfig).provider))
|
||||
? 1 : normalizeMaxConcurrentRuns(heartbeat.maxConcurrentRuns),
|
||||
skipTimerWhenNoActionableWork: asBoolean(
|
||||
heartbeat.skipTimerWhenNoActionableWork ??
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { and, asc, eq, gte, notInArray, sql } from "drizzle-orm";
|
||||
import { and, asc, eq, gte, inArray, notInArray, sql } from "drizzle-orm";
|
||||
import { agents, companies, heartbeatRuns, type Db } from "@paperclipai/db";
|
||||
import { instanceSettingsService, isTruthyRuntimeEnvValue, resolveWorktreeRunExecutionActivation } from "./instance-settings.js";
|
||||
import { maxConcurrentRunsSql } from "./heartbeat/timer-policy.js";
|
||||
|
||||
export const QUEUED_RUN_REDRIVE_AFTER_MS = 60_000;
|
||||
export const QUEUED_RUN_REDRIVE_MAX_INTERVAL_MS = 15 * 60_000;
|
||||
@@ -21,9 +22,14 @@ export type UndispatchedQueuedRun = {
|
||||
export type UndispatchedQueuedRunCursor = { at: string; runId: string };
|
||||
type PageCursor = UndispatchedQueuedRunCursor;
|
||||
|
||||
async function listUndispatchedQueuedRunPage(db: Db, companyId: string | undefined, createdAfter: Date | null | undefined,
|
||||
after: PageCursor | null, limit: number):
|
||||
Promise<Array<UndispatchedQueuedRun & { cursorAt: string }>> {
|
||||
/** One page of undispatched queued runs. Live runs are counted once per agent. */
|
||||
export function undispatchedQueuedRunPageQuery(db: Db, companyId: string | undefined, createdAfter: Date | null | undefined,
|
||||
after: PageCursor | null, limit: number) {
|
||||
const liveRuns = db.select({ companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId,
|
||||
count: sql<number>`count(*)`.as("live_count") }).from(heartbeatRuns)
|
||||
.where(and(inArray(heartbeatRuns.status, ["running", "scheduled_retry"]),
|
||||
companyId ? eq(heartbeatRuns.companyId, companyId) : undefined))
|
||||
.groupBy(heartbeatRuns.companyId, heartbeatRuns.agentId).as("agent_live_runs");
|
||||
return db.select({
|
||||
runId: heartbeatRuns.id,
|
||||
companyId: heartbeatRuns.companyId,
|
||||
@@ -34,6 +40,7 @@ async function listUndispatchedQueuedRunPage(db: Db, companyId: string | undefin
|
||||
}).from(heartbeatRuns)
|
||||
.innerJoin(companies, eq(companies.id, heartbeatRuns.companyId))
|
||||
.innerJoin(agents, and(eq(agents.id, heartbeatRuns.agentId), eq(agents.companyId, heartbeatRuns.companyId)))
|
||||
.leftJoin(liveRuns, and(eq(liveRuns.companyId, heartbeatRuns.companyId), eq(liveRuns.agentId, heartbeatRuns.agentId)))
|
||||
.where(and(
|
||||
eq(heartbeatRuns.status, "queued"),
|
||||
companyId ? eq(heartbeatRuns.companyId, companyId) : undefined,
|
||||
@@ -42,14 +49,21 @@ async function listUndispatchedQueuedRunPage(db: Db, companyId: string | undefin
|
||||
eq(agents.lifecycleState, "ready"),
|
||||
notInArray(agents.status, NOT_READY_AGENT_STATUSES),
|
||||
after ? sql`(${heartbeatRuns.createdAt}, ${heartbeatRuns.id}) > (${after.at}::timestamptz, ${after.runId}::uuid)` : undefined,
|
||||
sql`not exists (select 1 from ${heartbeatRuns} live
|
||||
where live.company_id = ${heartbeatRuns.companyId} and live.agent_id = ${heartbeatRuns.agentId}
|
||||
and live.status in ('running', 'scheduled_retry'))`,
|
||||
sql`coalesce(${liveRuns.count}, 0) < ${maxConcurrentRunsSql()}`,
|
||||
sql`not exists (select 1 from ${heartbeatRuns} task_live
|
||||
where task_live.company_id = ${heartbeatRuns.companyId}
|
||||
and task_live.context_snapshot->>'issueId' = coalesce(${heartbeatRuns.issueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId')
|
||||
and task_live.status in ('running', 'scheduled_retry'))`,
|
||||
))
|
||||
.orderBy(asc(heartbeatRuns.createdAt), asc(heartbeatRuns.id))
|
||||
.limit(limit);
|
||||
}
|
||||
|
||||
async function listUndispatchedQueuedRunPage(db: Db, companyId: string | undefined, createdAfter: Date | null | undefined,
|
||||
after: PageCursor | null, limit: number): Promise<Array<UndispatchedQueuedRun & { cursorAt: string }>> {
|
||||
return undispatchedQueuedRunPageQuery(db, companyId, createdAfter, after, limit);
|
||||
}
|
||||
|
||||
/** Every undispatched queued run, read page by page with no cap; one
|
||||
* company's runs when `companyId` is given. `createdAfter` is the worktree
|
||||
* execution cutoff: dispatch never starts a run created before it. */
|
||||
|
||||
Reference in new issue
Block a user