fix(heartbeat): rank a person's joined comment first and refill free run slots (#21277)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts each agent's queued runs, up to the
agent's `maxConcurrentRuns`
> - #21234 starts person-initiated runs before system recovery runs, and
it reads "person-initiated" from the run's own wake request
> - When a person comments on a task that already has a queued system
run, the comment joins that run. The run keeps its system wake, so
dispatch ranks the person's message as system work
> - Dispatch is event driven. The queued-run re-drive backstop covers
only agents with no live run, so an agent with several slots keeps a
free slot empty until one of its runs finishes, and runs can take hours
> - This pull request makes dispatch read the person wakes that joined
each queued run, and makes the re-drive backstop cover an agent that
still has a free slot
> - The benefit is that a person's comment starts first, and free slots
fill within about a minute instead of waiting for an unrelated run to
finish

## Linked Issues or Issue Description

Refs #21234, #20986

**What happened?**

On staging, one agent had `maxConcurrentRuns` 20, 18 running runs and 76
queued runs at 03:12Z. No run started on that agent from 03:01:33Z to
03:16:57Z (15.4 minutes). From 03:11:10Z, after a run finished, at least
2 slots were free. Other agents in the same company started runs during
that time. Runs started again only after the next deploy restarted the
server.

Five queued runs came from a person: comments at 02:43Z, 02:44Z and
02:45Z, and Retries at 02:46Z, 02:49Z and 03:01Z. The comment runs had
the wake reasons `issue_reopened_via_comment` and `issue_commented`, the
invocation source `automation`, and no person on the run's own wake
request.

**Expected behavior**

A run that carries a person's comment ranks as person-initiated, also
when the comment joined an existing queued run. When an agent has a free
slot and a queued run is ready, the run starts promptly, without waiting
for another run on the agent to finish.

**Steps to reproduce**

1. Give an agent `maxConcurrentRuns` 3. Start 2 runs on it.
2. Queue a third run while no dispatch signal reaches the queue (for
example, a claim declines it while a gate is still closed, and the gate
then clears).
3. See that `listUndispatchedQueuedRuns` returns nothing, because the
agent has a live run. The run stays queued until one of the 2 running
runs finishes.
4. Separately, queue a system run on a task, then post a person's
comment on the task so that the comment wake coalesces into that queued
run. Queue older system runs on other tasks of the same single-slot
agent.
5. See that dispatch starts the older system runs first.

**Root cause**

1. `coalesceIntoActiveExecutionRun`
(`server/src/modules/wake-queue/adapters/postgres.ts`) merges the
person's comment context into the queued run and records the person's
wake as a separate `coalesced` row with `runId` set to that run. The
run's own `wakeupRequestId` still points to the system wake.
`queuedRunDispatcherForAgent` (`server/src/services/heartbeat/queue.ts`)
only read the wake behind `wakeupRequestId`. The task routes do record
the comment author as the requester (`requestedByActorType: "user"`).
The person was lost only at dispatch time.
2. `listUndispatchedQueuedRunPage`
(`server/src/services/queued-run-dispatch-liveness.ts`) skipped every
agent with any `running` or `scheduled_retry` run. That rule assumes one
slot per agent. For an agent with several slots, a lost or declined
dispatch is not re-driven until a run on that agent finishes.

This change is not in `claimQueuedRun`, the fresh-isolated preflight,
the wake-queue promoter or `fresh-continuation-fallback.ts`.

**Paperclip version or commit**

Staging Core `ddc0aacad8` (contains #21234 `ade7e0643f`). This branch is
based on master `6cb0dd0c6b`.

**Deployment mode**

Authenticated, public, Paperclip Cloud managed staging.

**Agent adapter(s) involved**

`paperclip_runner` (Codex native runtime). The fix is
adapter-independent.

## What Changed

- `queuedRunDispatcherForAgent` also reads person wakes whose `runId` is
a queued run of the agent (by `run_id IN (queued run ids)`, so an
earlier saved comment that a run adopted is included). A run with such a
wake ranks as person-initiated and is never counted as system recovery.
- `listUndispatchedQueuedRuns` now returns a queued run when its agent
has fewer `running` plus `scheduled_retry` runs than its
`maxConcurrentRuns`, and its task has no `running` or `scheduled_retry`
run. The re-drive backstop and the ownership I7 check use this query.
- The query counts each agent's live runs once, in one grouped subquery
that is joined to the queued runs.
- Migration 0476 adds `agent_wakeup_requests_company_run_idx`
(`company_id`, `run_id`, partial on a non-null run) with `CREATE INDEX
CONCURRENTLY IF NOT EXISTS`. It is listed in
`MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS` and classifies as
`online_compatible`. `doc/DATABASE.md` describes it.
- Added `maxConcurrentRunsSql()` to `timer-policy.ts`. It reads the same
limit as `parseHeartbeatPolicy`, including the single-run runner
providers, which now share one constant.

## Verification

- Red/green: `server/src/__tests__/recovery-wake-herd.test.ts` "starts a
queued run that a person's comment joined ahead of older system work".
It fails on master (`expected null not to be null`: the older system run
started first) and passes with this change.
- Red/green:
`server/src/__tests__/ownership-reconciliation-report.test.ts`
"re-drives a queued run while its agent still has a free slot, and not
once every slot is running". It fails on master (`expected [] to deeply
equal [ Array(1) ]`) and passes with this change.
- Red/green: "starts a queued run that adopted a person's comment saved
before every queued run". It fails with a `requestedAt` bound and passes
with the `run_id` lookup.
- Red/green: "counts an agent's live runs in one grouped aggregate, not
once per queued run" runs EXPLAIN on the page query. It fails with a
per-row correlated count and passes with the grouped subquery.
- New: "uses the agent's single-run limit for a runner provider that
runs one turn at a time".
- Two existing tests now set `maxConcurrentRuns: 1` so that they keep
testing "the agent's only slot is busy".
- `npx vitest run src/__tests__/recovery-wake-herd.test.ts
src/services/heartbeat/queued-run-dispatch-order.test.ts
src/__tests__/ownership-reconciliation-report.test.ts
src/services/queued-run-dispatch-liveness.test.ts
src/services/heartbeat/dispatch-work.test.ts`: 71 passed.
- `npx tsc --noEmit -p server/tsconfig.json`: no errors.

## Risks

- The re-drive backstop now also covers multi-slot agents with live
runs. Each queued run is re-driven at most on its existing backoff (1
minute, then doubling to 15 minutes). A queued run whose task already
has a live run is not re-driven.
- The ownership I7 report can now list a queued run on a multi-slot
agent that has a free slot. This is the stall that this change targets.
- One more indexed read per dispatch pass (`agent_wakeup_requests` by
company and run id). The new index is built concurrently, so the
previous release keeps writing during the build.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, read-only staging API reads in a
browser tab).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 01:17:57 -06:00
1 parent 74096c19f4
commit 49f0030f06
11 files changed
+785 -14

No files matched your search

+6
View File
@@ -507,6 +507,12 @@ indexes also serve the comment and delivery `on delete set null` actions. Issue
lists share one external review-wait read per issue between the review and
blocked-inbox attention maps.
Migration 0476 builds `agent_wakeup_requests_company_run_idx` concurrently. It
keys wake requests by company and `run_id`, partial on a non-null run. Queued-run
dispatch finds the person wakes that joined or were adopted by each queued run
through it, so the read is bounded by the queue size, not by the agent's wake
history.
Migration 0457 builds `heartbeat_runs_company_native_session_idx` concurrently.
It keys runs by company and `native_session_id`, partial on a non-null session.
Native retention, recovery, resume and archive reads find a session's runs
+1
View File
@@ -690,6 +690,7 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([
"0461_created_from_legacy_origin_run_index.sql",
"0470_heartbeat_runs_queued_created_index.sql",
"0472_runner_api_capture_pending_deletes.sql",
"0476_wakeup_request_run_index.sql",
"0469_zippy_photon.sql",
@@ -0,0 +1 @@
CREATE INDEX CONCURRENTLY IF NOT EXISTS "agent_wakeup_requests_company_run_idx" ON "agent_wakeup_requests" USING btree ("company_id","run_id") WHERE "agent_wakeup_requests"."run_id" IS NOT NULL;
@@ -1,6 +1,6 @@
{
"id": "671a422a-0b93-4155-b097-9d35a4fd2b2b",
"prevId": "f0e17e47-6217-4080-b6b6-9d83b33e9ced",
"id": "82dcdc57-0796-4162-9451-8ea08a72199e",
"prevId": "ee1f1f66-0cbe-4b18-b364-5e96708a1f0b",
"version": "7",
"dialect": "postgresql",
"tables": {
@@ -3954,6 +3954,28 @@
"method": "btree",
"with": {}
},
"agent_wakeup_requests_company_run_idx": {
"name": "agent_wakeup_requests_company_run_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "run_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"agent_wakeup_requests\".\"run_id\" IS NOT NULL",
"concurrently": true,
"method": "btree",
"with": {}
},
"agent_wakeup_requests_review_path_recovery_idempotency_uq": {
"name": "agent_wakeup_requests_review_path_recovery_idempotency_uq",
"columns": [
@@ -53686,6 +53708,12 @@
"primaryKey": false,
"notNull": true
},
"pending_delete_object_key": {
"name": "pending_delete_object_key",
"type": "text",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
@@ -53709,6 +53737,21 @@
"concurrently": false,
"method": "btree",
"with": {}
},
"runner_api_response_reservations_asset_idx": {
"name": "runner_api_response_reservations_asset_idx",
"columns": [
{
"expression": "asset_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
@@ -53921,6 +53964,7 @@
}
],
"isUnique": true,
"where": "\"sandbox_creation_operations\".\"state\" <> 'resolved'",
"concurrently": false,
"method": "btree",
"with": {}
@@ -57514,6 +57558,12 @@
"notNull": true,
"default": "'{}'::jsonb"
},
"ui_meta": {
"name": "ui_meta",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"risk_level": {
"name": "risk_level",
"type": "text",
@@ -59639,6 +59689,253 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.tool_mcp_app_views": {
"name": "tool_mcp_app_views",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"invocation_id": {
"name": "invocation_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"issue_id": {
"name": "issue_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"run_id": {
"name": "run_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"agent_id": {
"name": "agent_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"responsible_user_id": {
"name": "responsible_user_id",
"type": "text",
"primaryKey": false,
"notNull": false
},
"connection_id": {
"name": "connection_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"catalog_entry_id": {
"name": "catalog_entry_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"application_name": {
"name": "application_name",
"type": "text",
"primaryKey": false,
"notNull": true
},
"tool_name": {
"name": "tool_name",
"type": "text",
"primaryKey": false,
"notNull": true
},
"upstream_tool_name": {
"name": "upstream_tool_name",
"type": "text",
"primaryKey": false,
"notNull": true
},
"resource_uri": {
"name": "resource_uri",
"type": "text",
"primaryKey": false,
"notNull": true
},
"tool_input": {
"name": "tool_input",
"type": "jsonb",
"primaryKey": false,
"notNull": true,
"default": "'{}'::jsonb"
},
"tool_result": {
"name": "tool_result",
"type": "jsonb",
"primaryKey": false,
"notNull": true
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"tool_mcp_app_views_issue_idx": {
"name": "tool_mcp_app_views_issue_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "issue_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "created_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"tool_mcp_app_views_connection_idx": {
"name": "tool_mcp_app_views_connection_idx",
"columns": [
{
"expression": "connection_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"tool_mcp_app_views_company_id_companies_id_fk": {
"name": "tool_mcp_app_views_company_id_companies_id_fk",
"tableFrom": "tool_mcp_app_views",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"tool_mcp_app_views_invocation_id_tool_invocations_id_fk": {
"name": "tool_mcp_app_views_invocation_id_tool_invocations_id_fk",
"tableFrom": "tool_mcp_app_views",
"tableTo": "tool_invocations",
"columnsFrom": [
"invocation_id"
],
"columnsTo": [
"id"
],
"onDelete": "set null",
"onUpdate": "no action"
},
"tool_mcp_app_views_issue_id_issues_id_fk": {
"name": "tool_mcp_app_views_issue_id_issues_id_fk",
"tableFrom": "tool_mcp_app_views",
"tableTo": "issues",
"columnsFrom": [
"issue_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"tool_mcp_app_views_run_id_heartbeat_runs_id_fk": {
"name": "tool_mcp_app_views_run_id_heartbeat_runs_id_fk",
"tableFrom": "tool_mcp_app_views",
"tableTo": "heartbeat_runs",
"columnsFrom": [
"run_id"
],
"columnsTo": [
"id"
],
"onDelete": "set null",
"onUpdate": "no action"
},
"tool_mcp_app_views_agent_id_agents_id_fk": {
"name": "tool_mcp_app_views_agent_id_agents_id_fk",
"tableFrom": "tool_mcp_app_views",
"tableTo": "agents",
"columnsFrom": [
"agent_id"
],
"columnsTo": [
"id"
],
"onDelete": "set null",
"onUpdate": "no action"
},
"tool_mcp_app_views_connection_id_tool_connections_id_fk": {
"name": "tool_mcp_app_views_connection_id_tool_connections_id_fk",
"tableFrom": "tool_mcp_app_views",
"tableTo": "tool_connections",
"columnsFrom": [
"connection_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"tool_mcp_app_views_catalog_entry_id_tool_catalog_entries_id_fk": {
"name": "tool_mcp_app_views_catalog_entry_id_tool_catalog_entries_id_fk",
"tableFrom": "tool_mcp_app_views",
"tableTo": "tool_catalog_entries",
"columnsFrom": [
"catalog_entry_id"
],
"columnsTo": [
"id"
],
"onDelete": "set null",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.tool_mcp_gateway_tokens": {
"name": "tool_mcp_gateway_tokens",
"schema": "",
@@ -60353,6 +60650,334 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.tool_mcp_skill_files": {
"name": "tool_mcp_skill_files",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"connection_id": {
"name": "connection_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"file_uri": {
"name": "file_uri",
"type": "text",
"primaryKey": false,
"notNull": true
},
"digest": {
"name": "digest",
"type": "text",
"primaryKey": false,
"notNull": true
},
"size": {
"name": "size",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"content_base64": {
"name": "content_base64",
"type": "text",
"primaryKey": false,
"notNull": true
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"tool_mcp_skill_files_digest_idx": {
"name": "tool_mcp_skill_files_digest_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "connection_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "file_uri",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "digest",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"tool_mcp_skill_files_company_id_companies_id_fk": {
"name": "tool_mcp_skill_files_company_id_companies_id_fk",
"tableFrom": "tool_mcp_skill_files",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"tool_mcp_skill_files_connection_id_tool_connections_id_fk": {
"name": "tool_mcp_skill_files_connection_id_tool_connections_id_fk",
"tableFrom": "tool_mcp_skill_files",
"tableTo": "tool_connections",
"columnsFrom": [
"connection_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.tool_mcp_skills": {
"name": "tool_mcp_skills",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"connection_id": {
"name": "connection_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"skill_uri": {
"name": "skill_uri",
"type": "text",
"primaryKey": false,
"notNull": true
},
"name": {
"name": "name",
"type": "text",
"primaryKey": false,
"notNull": true
},
"description": {
"name": "description",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "''"
},
"frontmatter": {
"name": "frontmatter",
"type": "jsonb",
"primaryKey": false,
"notNull": true
},
"manifest": {
"name": "manifest",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"manifest_digest": {
"name": "manifest_digest",
"type": "text",
"primaryKey": false,
"notNull": false
},
"status": {
"name": "status",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'pending_review'"
},
"status_reason": {
"name": "status_reason",
"type": "text",
"primaryKey": false,
"notNull": false
},
"approved_manifest_digest": {
"name": "approved_manifest_digest",
"type": "text",
"primaryKey": false,
"notNull": false
},
"allowed_tools_approved": {
"name": "allowed_tools_approved",
"type": "boolean",
"primaryKey": false,
"notNull": true,
"default": false
},
"reviewed_at": {
"name": "reviewed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"reviewed_by_user_id": {
"name": "reviewed_by_user_id",
"type": "text",
"primaryKey": false,
"notNull": false
},
"last_seen_at": {
"name": "last_seen_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"tool_mcp_skills_connection_uri_idx": {
"name": "tool_mcp_skills_connection_uri_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "connection_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "skill_uri",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"tool_mcp_skills_company_status_idx": {
"name": "tool_mcp_skills_company_status_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "status",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"tool_mcp_skills_company_id_companies_id_fk": {
"name": "tool_mcp_skills_company_id_companies_id_fk",
"tableFrom": "tool_mcp_skills",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"tool_mcp_skills_connection_id_tool_connections_id_fk": {
"name": "tool_mcp_skills_connection_id_tool_connections_id_fk",
"tableFrom": "tool_mcp_skills",
"tableTo": "tool_connections",
"columnsFrom": [
"connection_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.tool_oauth_states": {
"name": "tool_oauth_states",
"schema": "",
+7
View File
@@ -3319,6 +3319,13 @@
"when": 1794532772498,
"tag": "0475_sandbox_creation_owner_unresolved",
"breakpoints": true
},
{
"idx": 476,
"version": "7",
"when": 1794553127740,
"tag": "0476_wakeup_request_run_index",
"breakpoints": true
}
]
}
@@ -59,6 +59,10 @@ export const agentWakeupRequests = pgTable(
table.agentId,
table.requestedAt,
),
companyRunIdx: index("agent_wakeup_requests_company_run_idx")
.on(table.companyId, table.runId)
.concurrently()
.where(sql`${table.runId} IS NOT NULL`),
reviewPathRecoveryIdempotencyUq: uniqueIndex(
"agent_wakeup_requests_review_path_recovery_idempotency_uq",
)
@@ -34,7 +34,7 @@ import { applyOwnershipReportBackstops, registerOwnershipReconciliationWork } fr
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
import { notifyNativeFinalizationWork } from "../services/native-finalization-notifications.js";
import { instanceSettingsService } from "../services/instance-settings.js";
import { listUndispatchedQueuedRuns, readUndispatchedQueuedRuns } from "../services/queued-run-dispatch-liveness.js";
import { listUndispatchedQueuedRuns, readUndispatchedQueuedRuns, undispatchedQueuedRunPageQuery } from "../services/queued-run-dispatch-liveness.js";
import { releaseLapsedCoordinatorOwner } from "../services/workspace-recovery-stop-proof.js";
import { subscribeDatabaseWork } from "@paperclipai/db/work-signals";
import {
@@ -1073,6 +1073,7 @@ describe("report-only ownership reconciliation", () => {
});
it("reads undispatched queued runs in bounded slices that resume from a cursor, and skips an agent with a scheduled retry", async () => {
await db.update(agents).set({ runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } }).where(eq(agents.id, agentId));
const created = [30, 29, 28].map(minutesAgo);
const runs = await db.insert(heartbeatRuns).values(created.map(createdAt =>
({ companyId, agentId, status: "queued", invocationSource: "automation", createdAt }))).returning();
@@ -1086,6 +1087,51 @@ describe("report-only ownership reconciliation", () => {
expect(await listUndispatchedQueuedRuns(db)).toEqual([]);
});
it("re-drives a queued run while its agent still has a free slot, and not once every slot is running", async () => {
await db.update(agents).set({ runtimeConfig: { heartbeat: { maxConcurrentRuns: 3 } } }).where(eq(agents.id, agentId));
const tasks = await db.insert(issues).values([0, 1, 2, 3].map(index =>
({ companyId, title: `Task ${index}`, status: "in_progress" as const, assigneeAgentId: agentId }))).returning();
const live = (index: number) => ({ companyId, agentId, issueId: tasks[index]!.id, status: "running" as const,
invocationSource: "automation" as const, contextSnapshot: { issueId: tasks[index]!.id }, startedAt: minutesAgo(90) });
await db.insert(heartbeatRuns).values([live(0), live(1)]);
const [waiting] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "queued", invocationSource: "automation",
contextSnapshot: { issueId: tasks[3]!.id }, createdAt: minutesAgo(20) }).returning();
await db.insert(heartbeatRuns).values({ companyId, agentId, status: "queued", invocationSource: "automation",
contextSnapshot: { issueId: tasks[0]!.id }, createdAt: minutesAgo(25) });
expect((await listUndispatchedQueuedRuns(db)).map(run => run.runId)).toEqual([waiting!.id]);
await db.insert(heartbeatRuns).values(live(2));
expect(await listUndispatchedQueuedRuns(db)).toEqual([]);
});
it("counts an agent's live runs in one grouped aggregate, not once per queued run", async () => {
await db.insert(heartbeatRuns).values(Array.from({ length: 5 }, () =>
({ companyId, agentId, status: "queued" as const, invocationSource: "automation" as const, createdAt: minutesAgo(20) })));
await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running", invocationSource: "automation" });
const query = undispatchedQueuedRunPageQuery(db, undefined, null, null, 500).toSQL();
const [row] = await db.$client.unsafe(`explain (format json) ${query.sql}`, query.params as never[]);
type PlanNode = { "Node Type": string; "Parent Relationship"?: string; Plans?: PlanNode[] };
const aggregates: PlanNode[] = [];
const subPlanAggregates: PlanNode[] = [];
const walk = (node: PlanNode, inSubPlan: boolean) => {
const subPlan = inSubPlan || node["Parent Relationship"] === "SubPlan";
if (node["Node Type"] === "Aggregate") (subPlan ? subPlanAggregates : aggregates).push(node);
for (const child of node.Plans ?? []) walk(child, subPlan);
};
walk((row as unknown as { "QUERY PLAN": Array<{ Plan: PlanNode }> })["QUERY PLAN"][0]!.Plan, false);
expect(aggregates).toHaveLength(1);
expect(subPlanAggregates).toEqual([]);
});
it("uses the agent's single-run limit for a runner provider that runs one turn at a time", async () => {
await db.update(agents).set({ adapterType: "paperclip_runner", adapterConfig: { provider: "openai_dot" },
runtimeConfig: { heartbeat: { maxConcurrentRuns: 5 } } }).where(eq(agents.id, agentId));
await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running", invocationSource: "automation",
startedAt: minutesAgo(90) });
await db.insert(heartbeatRuns).values({ companyId, agentId, status: "queued", invocationSource: "automation",
createdAt: minutesAgo(20) });
expect(await listUndispatchedQueuedRuns(db)).toEqual([]);
});
it("keeps the earliest grace deadline when the queued-run check reaches its cap, and rescans once it ends", async () => {
const createdAt = minutesAgo(1);
await db.insert(heartbeatRuns).values(Array.from({ length: 2_000 }, () =>
@@ -1128,7 +1174,8 @@ describe("report-only ownership reconciliation", () => {
const otherAgent = randomUUID();
const pausedAgent = randomUUID();
await db.insert(agents).values([
{ id: otherAgent, companyId, name: "Busy", role: "engineer", adapterType: "codex_local" },
{ id: otherAgent, companyId, name: "Busy", role: "engineer", adapterType: "codex_local",
runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } },
{ id: pausedAgent, companyId, name: "Paused", role: "engineer", adapterType: "codex_local", status: "paused" },
]);
await db.insert(heartbeatRuns).values([
@@ -111,6 +111,52 @@ suite("recovery wake herd", () => {
}
}, 120_000);
async function queuedSystemRun(agentId: string, createdMinutesAgo: number, wakeReason: string) {
const run = await queuedRecoveryRun(agentId, createdMinutesAgo);
const [row] = await db.update(heartbeatRuns).set({ contextSnapshot: { ...run.contextSnapshot, wakeReason } })
.where(eq(heartbeatRuns.id, run.id)).returning();
return row!;
}
it("starts a queued run that a person's comment joined ahead of older system work", async () => {
const agentId = await seedCompany(1);
const older: Array<typeof heartbeatRuns.$inferSelect> = [];
for (let index = 0; index < 3; index += 1) older.push(await queuedSystemRun(agentId, 60 - index, "issue_assigned"));
const joined = await queuedSystemRun(agentId, 50, "issue_reopened_via_comment");
await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "automation", triggerDetail: "system",
reason: "issue_execution_same_name", status: "coalesced", coalescedCount: 1, requestedByActorType: "user",
requestedByActorId: userId, runId: joined.id, finishedAt: new Date(), payload: { issueId: joined.issueId } });
const heartbeat = heartbeatService(db);
try {
await heartbeat.resumeQueuedRuns();
expect((await readRun(joined.id)).startedAt).not.toBeNull();
for (const run of older) expect(await readRun(run.id)).toMatchObject({ status: "queued", startedAt: null });
} finally {
await heartbeat.drainActiveRunExecutions();
}
}, 120_000);
it("starts a queued run that adopted a person's comment saved before every queued run", async () => {
const agentId = await seedCompany(1);
const [saved] = await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "automation", triggerDetail: "system",
reason: "issue_commented", status: "deferred_issue_execution", requestedByActorType: "user", requestedByActorId: userId,
requestedAt: new Date(Date.now() - 90 * 60_000), createdAt: new Date(Date.now() - 90 * 60_000),
updatedAt: new Date(Date.now() - 90 * 60_000), payload: {} }).returning();
const older: Array<typeof heartbeatRuns.$inferSelect> = [];
for (let index = 0; index < 3; index += 1) older.push(await queuedSystemRun(agentId, 60 - index, "issue_assigned"));
const adopting = await queuedSystemRun(agentId, 50, "issue_commented");
await db.update(agentWakeupRequests).set({ runId: adopting.id, status: "queued", updatedAt: new Date(),
payload: { issueId: adopting.issueId } }).where(eq(agentWakeupRequests.id, saved!.id));
const heartbeat = heartbeatService(db);
try {
await heartbeat.resumeQueuedRuns();
expect((await readRun(adopting.id)).startedAt).not.toBeNull();
for (const run of older) expect(await readRun(run.id)).toMatchObject({ status: "queued", startedAt: null });
} finally {
await heartbeat.drainActiveRunExecutions();
}
}, 120_000);
async function pendingContinuation(agentId: string, endedMinutesAgo: number) {
const issue = await task(agentId, "todo");
const source = await sourceRun(agentId, issue.id, endedMinutesAgo);
+10 -2
View File
@@ -2360,10 +2360,18 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
eq(agentWakeupRequests.requestedByActorType, "user")))
: [];
const userRequestIds = new Set(userRequests.map(row => row.id));
const joinedUserRequests = queuedRuns.length > 1
? await db.select({ runId: agentWakeupRequests.runId }).from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.companyId, agent.companyId), inArray(agentWakeupRequests.runId, queuedRuns.map(run => run.id)),
eq(agentWakeupRequests.agentId, agent.id), eq(agentWakeupRequests.requestedByActorType, "user")))
: [];
const userRunIds = new Set(joinedUserRequests.flatMap(row => row.runId ? [row.runId] : []));
const requestedByUser = (run: typeof heartbeatRuns.$inferSelect) =>
userRunIds.has(run.id) || Boolean(run.wakeupRequestId && userRequestIds.has(run.wakeupRequestId));
const sourceRunIdOf = (run: typeof heartbeatRuns.$inferSelect) =>
readNonEmptyString(parseObject(run.contextSnapshot).previousRunId) ?? run.retryOfRunId ?? null;
const recoveryRuns = queuedRuns.filter(run => isSystemRecoveryRun({ wakeReason: wakeReasonOf(run),
requestedByUser: Boolean(run.wakeupRequestId && userRequestIds.has(run.wakeupRequestId)) }));
requestedByUser: requestedByUser(run) }));
const sourceRunIds = [...new Set(recoveryRuns.flatMap(run => sourceRunIdOf(run) ?? []))];
const sourceRows = queuedRuns.length > 1 && sourceRunIds.length
? await db.select({ id: heartbeatRuns.id, finishedAt: heartbeatRuns.finishedAt }).from(heartbeatRuns)
@@ -2392,7 +2400,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
id: run.id,
createdAt: run.createdAt,
wakeReason: wakeReasonOf(run),
requestedByUser: Boolean(run.wakeupRequestId && userRequestIds.has(run.wakeupRequestId)),
requestedByUser: requestedByUser(run),
sourceEndedAt: sourceRunId ? sourceEndedAt.get(sourceRunId) ?? null : null,
readinessRank: issueId ? (ready ? (issue?.status === "in_progress" ? 0 : 1) : 3) : 2,
priorityRank: issueRunPriorityRank(issue?.priority),
+13 -1
View File
@@ -9,6 +9,7 @@ const HEARTBEAT_MAX_CONCURRENT_RUNS_DEFAULT = AGENT_DEFAULT_MAX_CONCURRENT_RUNS;
const HEARTBEAT_MAX_CONCURRENT_RUNS_MIN = 1;
const HEARTBEAT_MAX_CONCURRENT_RUNS_MAX = 50;
const TIMER_ACTIONABLE_ISSUE_STATUSES = ["todo", "in_progress"] as const;
const SINGLE_RUN_RUNNER_PROVIDERS = ["openai_dot", "grok_bot", "muse"];
function normalizeMaxConcurrentRuns(value: unknown) {
const parsed = Math.floor(
@@ -21,6 +22,17 @@ function normalizeMaxConcurrentRuns(value: unknown) {
);
}
/** The same limit as `parseHeartbeatPolicy(agent).maxConcurrentRuns`, read in SQL. */
export function maxConcurrentRunsSql(): SQL<number> {
const configured = sql`${agents.runtimeConfig}->'heartbeat'->'maxConcurrentRuns'`;
return sql<number>`(case
when ${agents.adapterType} = 'paperclip_runner'
and ${agents.adapterConfig}->>'provider' in (${sql.join(SINGLE_RUN_RUNNER_PROVIDERS.map(provider => sql`${provider}`), sql`, `)}) then 1
when jsonb_typeof(${configured}) = 'number' then greatest(${HEARTBEAT_MAX_CONCURRENT_RUNS_MIN},
least(${HEARTBEAT_MAX_CONCURRENT_RUNS_MAX}, floor((${configured})::text::numeric)))
else ${HEARTBEAT_MAX_CONCURRENT_RUNS_DEFAULT} end)`;
}
export function heartbeatTimerPolicyIsWellFormed(agent: typeof agents.$inferSelect): boolean {
const record = (v: unknown): v is Record<string, unknown> => v !== null && typeof v === "object" && !Array.isArray(v);
if (!record(agent.runtimeConfig)) return false;
@@ -48,7 +60,7 @@ export function parseHeartbeatPolicy(agent: typeof agents.$inferSelect) {
intervalSec: Math.max(0, asNumber(heartbeat.intervalSec, 0)),
wakeOnDemand: isHeartbeatWakeOnDemandEnabled(agent),
maxConcurrentRuns: agent.adapterType === "paperclip_runner" &&
["openai_dot", "grok_bot", "muse"].includes(String(parseObject(agent.adapterConfig).provider))
SINGLE_RUN_RUNNER_PROVIDERS.includes(String(parseObject(agent.adapterConfig).provider))
? 1 : normalizeMaxConcurrentRuns(heartbeat.maxConcurrentRuns),
skipTimerWhenNoActionableWork: asBoolean(
heartbeat.skipTimerWhenNoActionableWork ??
@@ -1,6 +1,7 @@
import { and, asc, eq, gte, notInArray, sql } from "drizzle-orm";
import { and, asc, eq, gte, inArray, notInArray, sql } from "drizzle-orm";
import { agents, companies, heartbeatRuns, type Db } from "@paperclipai/db";
import { instanceSettingsService, isTruthyRuntimeEnvValue, resolveWorktreeRunExecutionActivation } from "./instance-settings.js";
import { maxConcurrentRunsSql } from "./heartbeat/timer-policy.js";
export const QUEUED_RUN_REDRIVE_AFTER_MS = 60_000;
export const QUEUED_RUN_REDRIVE_MAX_INTERVAL_MS = 15 * 60_000;
@@ -21,9 +22,14 @@ export type UndispatchedQueuedRun = {
export type UndispatchedQueuedRunCursor = { at: string; runId: string };
type PageCursor = UndispatchedQueuedRunCursor;
async function listUndispatchedQueuedRunPage(db: Db, companyId: string | undefined, createdAfter: Date | null | undefined,
after: PageCursor | null, limit: number):
Promise<Array<UndispatchedQueuedRun & { cursorAt: string }>> {
/** One page of undispatched queued runs. Live runs are counted once per agent. */
export function undispatchedQueuedRunPageQuery(db: Db, companyId: string | undefined, createdAfter: Date | null | undefined,
after: PageCursor | null, limit: number) {
const liveRuns = db.select({ companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId,
count: sql<number>`count(*)`.as("live_count") }).from(heartbeatRuns)
.where(and(inArray(heartbeatRuns.status, ["running", "scheduled_retry"]),
companyId ? eq(heartbeatRuns.companyId, companyId) : undefined))
.groupBy(heartbeatRuns.companyId, heartbeatRuns.agentId).as("agent_live_runs");
return db.select({
runId: heartbeatRuns.id,
companyId: heartbeatRuns.companyId,
@@ -34,6 +40,7 @@ async function listUndispatchedQueuedRunPage(db: Db, companyId: string | undefin
}).from(heartbeatRuns)
.innerJoin(companies, eq(companies.id, heartbeatRuns.companyId))
.innerJoin(agents, and(eq(agents.id, heartbeatRuns.agentId), eq(agents.companyId, heartbeatRuns.companyId)))
.leftJoin(liveRuns, and(eq(liveRuns.companyId, heartbeatRuns.companyId), eq(liveRuns.agentId, heartbeatRuns.agentId)))
.where(and(
eq(heartbeatRuns.status, "queued"),
companyId ? eq(heartbeatRuns.companyId, companyId) : undefined,
@@ -42,14 +49,21 @@ async function listUndispatchedQueuedRunPage(db: Db, companyId: string | undefin
eq(agents.lifecycleState, "ready"),
notInArray(agents.status, NOT_READY_AGENT_STATUSES),
after ? sql`(${heartbeatRuns.createdAt}, ${heartbeatRuns.id}) > (${after.at}::timestamptz, ${after.runId}::uuid)` : undefined,
sql`not exists (select 1 from ${heartbeatRuns} live
where live.company_id = ${heartbeatRuns.companyId} and live.agent_id = ${heartbeatRuns.agentId}
and live.status in ('running', 'scheduled_retry'))`,
sql`coalesce(${liveRuns.count}, 0) < ${maxConcurrentRunsSql()}`,
sql`not exists (select 1 from ${heartbeatRuns} task_live
where task_live.company_id = ${heartbeatRuns.companyId}
and task_live.context_snapshot->>'issueId' = coalesce(${heartbeatRuns.issueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId')
and task_live.status in ('running', 'scheduled_retry'))`,
))
.orderBy(asc(heartbeatRuns.createdAt), asc(heartbeatRuns.id))
.limit(limit);
}
async function listUndispatchedQueuedRunPage(db: Db, companyId: string | undefined, createdAfter: Date | null | undefined,
after: PageCursor | null, limit: number): Promise<Array<UndispatchedQueuedRun & { cursorAt: string }>> {
return undispatchedQueuedRunPageQuery(db, companyId, createdAfter, after, limit);
}
/** Every undispatched queued run, read page by page with no cap; one
* company's runs when `companyId` is given. `createdAfter` is the worktree
* execution cutoff: dispatch never starts a run created before it. */