fix(recovery): stop re-checking settled native runs that cannot be released (#21590)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server releases the process identity (`process_pid`,
`process_group_id`, `process_started_at`) of a finished native run when
it can prove that the runner process stopped (#17374)
> - A background queue (`settledNativeProcesses`) finds these runs and
checks them. A check that cannot finish ("skipped" or "pending") writes
nothing, so the run stays in the queue and every recovery hint checks it
again
> - A finished run that a replaced container recorded can never pass the
check, because only the host that recorded the process can prove that it
stopped. Each deploy adds more of these runs
> - On one hosted instance this loop ran about 10 checks per second and
about 55 SQL statements per second, and the server main thread was about
95% busy with only a few live runs
> - This pull request moves each "cannot make progress" reason into the
queue's SQL predicate, adds a growing backoff for "pending" runs, and
makes one check cost one locking statement
> - The benefit is that runs which cannot progress cost nothing until
something that can change the answer happens, and a run that becomes
releasable is still released at once

## Linked Issues or Issue Description

No public issue exists. Bug report:

**What happened?**
The settled native process recovery queue checks the same finished
native runs again on every recovery hint and every 30 seconds, forever.
With 300 such runs and one company-wide hint per second it ran 9 checks
per second and 84 SQL statements per second (9.4 statements per check,
2.2 ms of server CPU per check) on embedded PostgreSQL. A hosted
instance showed the same rates. The runs that never finish are: runs
recorded by a host that is gone (each redeploy starts a new container
with a new hostname), runs whose coordinator phase is not settled, runs
with no checkpoint and no provider evidence, and runs whose local
process is still alive (they defer 30 seconds with no limit). Startup
discovery (`discoverSettledNativeRunProcesses`) also opens one
transaction for every such run on every boot.

**Expected behavior**
A run that cannot make progress is not read again until something that
can change the answer happens. A run that becomes releasable (its phase
settles, provider evidence arrives, its own host checks it, its sandbox
receipt arrives, its process exits) is released promptly.

**Steps to reproduce**
1. Create a failed native run with a recorded local process identity
from another host (or an unsettled coordinator phase, or no checkpoint
and no provider evidence, or a live process).
2. Register the residual recovery work and send company-wide
`notifyLiveRunRecovery` hints.
3. Count the run's `for update` checks: on `master` every hint checks
the run again.

`server/src/__tests__/settled-native-process-rechecks.test.ts` automates
these steps.

**Paperclip version or commit**
`master` at 3c3e6c2d75.

**Deployment mode**
Any. The cost grows with every deploy of a hosted instance.

## What Changed

- `server/src/services/native-local-process-stop.ts`
- New `settledNativeProcessCandidates`: one fixed-text statement that
finds releasable runs. It reads only runs recorded in this server's PID
space (`process_host_hash`), sandbox runs (`remote`, with a non-local
lease), and rows saved before the hash (their latest identity event must
name this host). It excludes runs with an unsettled coordinator phase
and runs with no checkpoint and no provider evidence. Each PID-space
branch reads its own range of the new partial index.
- `releaseSettledNativeRunProcess` now reads the run, phase, evidence,
lease facts and identity match in one `for update` statement (it was 6
statements). It reads the lease rows again only for a sandbox run. The
new `inspectSettledNativeRunProcess` also returns whether a remote stop
acknowledgement can apply.
- Startup discovery uses the same candidate statement, so it opens no
transaction for an excluded run.
- `server/src/services/heartbeat/live-recovery-work.ts` (settled kind
only)
- The load runs the candidate statement and a separate indexed statement
for cancelled `cursor_cloud_api` runs whose Stop is still `requested`.
- The check uses the loaded entry. It no longer runs the load again for
each entry.
- A "pending" run backs off 30 s, 60 s, 120 s, up to 15 min. A
run-scoped hint (identity event, lease cleanup, stop receipt) resets the
backoff.
- A "skipped" run that the predicate cannot exclude (for example a birth
time mismatch on this host) waits for a run-scoped hint or a restart.
- `server/src/services/heartbeat/recovery.ts` (the settled handler
only): returns the outcome. It calls `acknowledgeRemoteStop` only when
the run has a non-local lease or is a cancelled `cursor_cloud_api` run.
For any other run that function reads two rows and writes nothing.
- `packages/db`: migration 0487 builds
`heartbeat_runs_settled_process_identity_idx` and
`heartbeat_runs_pending_api_stop_idx` with `CREATE INDEX CONCURRENTLY IF
NOT EXISTS`. The migration is `online_compatible` and is listed in
`MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`. `doc/DATABASE.md` describes both
indexes.

End state for each reason that a check cannot finish:

| Reason | Before | After | What re-admits the run |
| --- | --- | --- | --- |
| Recorded by another PID space | checked on every hint | not read |
only a server in that PID space reads it (a live older server in a
deploy overlap still releases its own runs) |
| Coordinator phase not settled | checked on every hint | not read | the
phase write; the finalizer sends a company hint in the same transaction
|
| No checkpoint and no provider evidence | checked on every hint | not
read | the next hint after a provider or turn event |
| Live local process, unreleased lease, missing sandbox receipt | every
hint and every 30 s | backoff 30 s to 15 min | a run-scoped hint resets
the backoff |
| Other same-host skip | checked on every hint | parked | a run-scoped
hint (an identity event sends one) or a restart |

## Verification

- `pnpm exec vitest run
src/__tests__/settled-native-process-rechecks.test.ts` (new, 12 tests,
real PostgreSQL, real queue and coordinator, fake clock):
- each exclusion: replaced container, unsettled phase, no evidence: 0
checks over 4 hints (red on `master`: 3 to 5 checks);
- a live older server and a stopped older server (different hash): the
new server never reads the run and writes no receipt; this server's own
run is released at start;
- re-admission: phase settles, late provider evidence, legacy rows with
no hash (this host released, another host not read), sandbox run
released from any server with its receipt;
- live process: at most 4 checks over 8 hints (red: 17), released on the
run-scoped hint after the process exits;
- one check is BEGIN, one `for update` statement and COMMIT (3
statements); the next hints do not check it; a run-scoped hint checks it
once more (red: 5 statements and a check on every hint);
- a restart costs one check for each pending run; startup discovery with
50 excluded runs sends 1 statement (red: 302).
  - On `master` 11 of the 12 tests fail.
- Measurement on embedded PostgreSQL, 300 stuck runs, one company hint
per second for 300 s (fake clock), same harness before and after:
- before: 9.0 checks/s, 84.3 statements/s, 9.4 statements per check, 5.9
s CPU (2.2 ms per check);
- after: 0 checks/s, 3.2 statements/s for all seven residual queues
together, 0.16 s CPU.
- Load plan with 12,000 excluded finished runs (10% without a hash) in a
31,000-row `heartbeat_runs`: the candidate statement reads only index
entries for this server's hash, `remote` and `unknown`, and a bitmap
scan of the 1,200 legacy rows: 2.7 ms, 4,416 buffers. Without the index
the same exclusion read all 31,000 rows (31,127 buffers).
- Existing suites, one file at a time: all pass.
`live-recovery-work.test.ts` 51/51 (an earlier run under machine load
hit the `afterAll` hook timeout after all tests passed; the rerun passed
clean, and `native-safe-replacement.test.ts` hits the same hook timeout
on the base commit), `native-settled-process-release.test.ts` 31/31,
`native-local-process-stop.test.ts` 5/5, `heartbeat/recovery.test.ts`
107/107, `native-runner-restart-recovery.integration.test.ts` 231 passed
/ 14 skipped, `native-restart-recovery.test.ts` 49/49,
`native-controller-recovery.integration.test.ts` 30/30,
`retained-native-session-cleanup-work.test.ts` 29/29,
`native-provider-failure.integration.test.ts` 16/16,
`native-safe-replacement.test.ts` 70/70 (with the `afterAll` hook
timeout that also occurs on the base commit),
`native-remote-retry-admission.test.ts` 93/93.
- `pnpm check:module-boundaries`, `pnpm check:migrations` (packages/db),
server typecheck: pass.
- Not run: the full suite and browser suites.

## Risks

- Unresolved by design: a finished run whose server was replaced keeps
its process fields indefinitely and is never released, because only the
host that recorded a process can prove that it stopped, and the shared
prover refuses a controller-lease proof for a run with a process group.
This pull request only stops paying for it. These runs keep no stop
receipt (`native.local_process_stopped`), so what waits for that receipt
still waits: accounting close of the run's usage source, retained native
session cleanup, and instruction or directory working copies that wait
for proof that the run stopped. A follow-up must decide the ownership
rule for a replaced container.
- Rows saved before `process_host_hash` exist are read through the
legacy branch, which checks the latest identity event for each of them
on each load. On an instance with many such rows the load costs one
index lookup per row. The count can be measured with `select count(*)
from heartbeat_runs where process_host_hash is null and process_pid is
not null and status in
('failed','cancelled','timed_out','interrupted')`.
- A run that this server recorded but whose `process_host_hash` changed
(a new PID namespace on the same host, for example a container restart)
is no longer read here. Before, it was read and skipped, so no release
is lost.
- The backoff and parked state live in memory. A restart checks each
pending run once.
- The migration builds two partial indexes concurrently on
`heartbeat_runs`. It holds no long lock.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-14 02:29:23 -06:00
1 parent e8c79235ca
commit f452b449fd
11 files changed
+965 -81

No files matched your search

+10
View File
@@ -507,6 +507,16 @@ indexes also serve the comment and delivery `on delete set null` actions. Issue
lists share one external review-wait read per issue between the review and
blocked-inbox attention maps.
Migration 0487 builds two partial `heartbeat_runs` indexes concurrently for the
settled-process recovery queue. `heartbeat_runs_settled_process_identity_idx`
keys settled native runs that still hold a process identity by
`process_host_hash`, company and id. Each server reads only the runs recorded in
its own PID space, sandbox runs (`remote`) and rows saved before the hash, so
finished runs from replaced containers are not read at all.
`heartbeat_runs_pending_api_stop_idx` keys cancelled `cursor_cloud_api` runs
whose Stop is still `requested`. Without them, the queue reads every finished
run of a company on each hint.
Migration 0476 builds `agent_wakeup_requests_company_run_idx` concurrently. It
keys wake requests by company and `run_id`, partial on a non-null run. Queued-run
dispatch finds the person wakes that joined or were adopted by each queued run
+1
View File
@@ -748,6 +748,7 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([
"0472_runner_api_capture_pending_deletes.sql",
"0476_wakeup_request_run_index.sql",
"0486_heartbeat_runs_company_finished_index.sql",
"0487_heartbeat_runs_settled_process_indexes.sql",
"0469_zippy_photon.sql",
@@ -0,0 +1,5 @@
CREATE INDEX CONCURRENTLY IF NOT EXISTS "heartbeat_runs_settled_process_identity_idx" ON "heartbeat_runs" USING btree ("process_host_hash","company_id","id") WHERE "heartbeat_runs"."runtime_mode" = 'native' and "heartbeat_runs"."status" in ('failed', 'cancelled', 'timed_out', 'interrupted')
and "heartbeat_runs"."finished_at" is not null
and ("heartbeat_runs"."process_pid" is not null or "heartbeat_runs"."process_group_id" is not null or "heartbeat_runs"."process_started_at" is not null);--> statement-breakpoint
CREATE INDEX CONCURRENTLY IF NOT EXISTS "heartbeat_runs_pending_api_stop_idx" ON "heartbeat_runs" USING btree ("company_id","id") WHERE "heartbeat_runs"."status" = 'cancelled' and "heartbeat_runs"."driver_kind" = 'cursor_cloud_api'
and ("heartbeat_runs"."result_json"->'executionCancellation'->>'state') = 'requested';
@@ -1,6 +1,6 @@
{
"id": "8b353a38-62df-4834-be55-7e9ebefad5c5",
"prevId": "41eacc6a-6eab-4f01-965a-62aa0cd6778f",
"id": "73c8fa97-70fc-49c0-804e-e49cfdffd156",
"prevId": "0c3e74be-7936-4c97-9798-505daf53c16b",
"version": "7",
"dialect": "postgresql",
"tables": {
@@ -6582,6 +6582,111 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.automatic_run_brakes": {
"name": "automatic_run_brakes",
"schema": "",
"columns": {
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": true,
"notNull": true
},
"state": {
"name": "state",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'closed'"
},
"reason": {
"name": "reason",
"type": "jsonb",
"primaryKey": false,
"notNull": true,
"default": "'{}'::jsonb"
},
"state_changed_at": {
"name": "state_changed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"opened_at": {
"name": "opened_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"next_probe_at": {
"name": "next_probe_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"probe_run_id": {
"name": "probe_run_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"probe_successes": {
"name": "probe_successes",
"type": "integer",
"primaryKey": false,
"notNull": true,
"default": 0
},
"release_window_started_at": {
"name": "release_window_started_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"release_window_starts": {
"name": "release_window_starts",
"type": "integer",
"primaryKey": false,
"notNull": true,
"default": 0
},
"next_creation_at": {
"name": "next_creation_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {},
"foreignKeys": {
"automatic_run_brakes_company_id_companies_id_fk": {
"name": "automatic_run_brakes_company_id_companies_id_fk",
"tableFrom": "automatic_run_brakes",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.board_api_keys": {
"name": "board_api_keys",
"schema": "",
@@ -36029,6 +36134,78 @@
"method": "btree",
"with": {}
},
"heartbeat_runs_company_finished_idx": {
"name": "heartbeat_runs_company_finished_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "finished_at",
"isExpression": false,
"asc": false,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"heartbeat_runs\".\"finished_at\" is not null",
"concurrently": true,
"method": "btree",
"with": {}
},
"heartbeat_runs_settled_process_identity_idx": {
"name": "heartbeat_runs_settled_process_identity_idx",
"columns": [
{
"expression": "process_host_hash",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' and \"heartbeat_runs\".\"status\" in ('failed', 'cancelled', 'timed_out', 'interrupted')\n and \"heartbeat_runs\".\"finished_at\" is not null\n and (\"heartbeat_runs\".\"process_pid\" is not null or \"heartbeat_runs\".\"process_group_id\" is not null or \"heartbeat_runs\".\"process_started_at\" is not null)",
"concurrently": true,
"method": "btree",
"with": {}
},
"heartbeat_runs_pending_api_stop_idx": {
"name": "heartbeat_runs_pending_api_stop_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"heartbeat_runs\".\"status\" = 'cancelled' and \"heartbeat_runs\".\"driver_kind\" = 'cursor_cloud_api'\n and (\"heartbeat_runs\".\"result_json\"->'executionCancellation'->>'state') = 'requested'",
"concurrently": true,
"method": "btree",
"with": {}
},
"heartbeat_runs_company_created_at_desc_idx": {
"name": "heartbeat_runs_company_created_at_desc_idx",
"columns": [
@@ -59607,6 +59784,12 @@
"primaryKey": false,
"notNull": false
},
"progress": {
"name": "progress",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
@@ -60113,6 +60296,277 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.tool_mcp_elicitations": {
"name": "tool_mcp_elicitations",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"issue_id": {
"name": "issue_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"agent_id": {
"name": "agent_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"run_id": {
"name": "run_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"invocation_id": {
"name": "invocation_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"connection_id": {
"name": "connection_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"grant_id": {
"name": "grant_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"catalog_tool_name": {
"name": "catalog_tool_name",
"type": "text",
"primaryKey": false,
"notNull": true
},
"call_identity": {
"name": "call_identity",
"type": "text",
"primaryKey": false,
"notNull": true
},
"era": {
"name": "era",
"type": "text",
"primaryKey": false,
"notNull": true
},
"sealed_request_state": {
"name": "sealed_request_state",
"type": "text",
"primaryKey": false,
"notNull": false
},
"questions": {
"name": "questions",
"type": "jsonb",
"primaryKey": false,
"notNull": true
},
"arguments_identity": {
"name": "arguments_identity",
"type": "text",
"primaryKey": false,
"notNull": true
},
"dedupe_key": {
"name": "dedupe_key",
"type": "text",
"primaryKey": false,
"notNull": true
},
"status": {
"name": "status",
"type": "text",
"primaryKey": false,
"notNull": true
},
"claimed_until": {
"name": "claimed_until",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"expires_at": {
"name": "expires_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true
},
"sent_at": {
"name": "sent_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"tool_mcp_elicitations_call_idx": {
"name": "tool_mcp_elicitations_call_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "issue_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "connection_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "call_identity",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "status",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"tool_mcp_elicitations_invocation_idx": {
"name": "tool_mcp_elicitations_invocation_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "invocation_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"tool_mcp_elicitations_waiting_uq": {
"name": "tool_mcp_elicitations_waiting_uq",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "issue_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "dedupe_key",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"where": "\"tool_mcp_elicitations\".\"status\" = 'waiting'",
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"tool_mcp_elicitations_company_id_companies_id_fk": {
"name": "tool_mcp_elicitations_company_id_companies_id_fk",
"tableFrom": "tool_mcp_elicitations",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"tool_mcp_elicitations_invocation_id_tool_invocations_id_fk": {
"name": "tool_mcp_elicitations_invocation_id_tool_invocations_id_fk",
"tableFrom": "tool_mcp_elicitations",
"tableTo": "tool_invocations",
"columnsFrom": [
"invocation_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"tool_mcp_elicitations_connection_id_tool_connections_id_fk": {
"name": "tool_mcp_elicitations_connection_id_tool_connections_id_fk",
"tableFrom": "tool_mcp_elicitations",
"tableTo": "tool_connections",
"columnsFrom": [
"connection_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.tool_mcp_gateway_tokens": {
"name": "tool_mcp_gateway_tokens",
"schema": "",
+7
View File
@@ -3396,6 +3396,13 @@
"when": 1794623177597,
"tag": "0486_heartbeat_runs_company_finished_index",
"breakpoints": true
},
{
"idx": 487,
"version": "7",
"when": 1794639469687,
"tag": "0487_heartbeat_runs_settled_process_indexes",
"breakpoints": true
}
]
}
+10
View File
@@ -250,6 +250,16 @@ export const heartbeatRuns = pgTable(
),
companyFinishedIdx: index("heartbeat_runs_company_finished_idx").on(table.companyId, table.finishedAt.desc())
.where(sql`${table.finishedAt} is not null`).concurrently(),
settledProcessIdentityIdx: index("heartbeat_runs_settled_process_identity_idx")
.on(table.processHostHash, table.companyId, table.id)
.where(sql`${table.runtimeMode} = 'native' and ${table.status} in ('failed', 'cancelled', 'timed_out', 'interrupted')
and ${table.finishedAt} is not null
and (${table.processPid} is not null or ${table.processGroupId} is not null or ${table.processStartedAt} is not null)`)
.concurrently(),
pendingApiStopIdx: index("heartbeat_runs_pending_api_stop_idx").on(table.companyId, table.id)
.where(sql`${table.status} = 'cancelled' and ${table.driverKind} = 'cursor_cloud_api'
and (${table.resultJson}->'executionCancellation'->>'state') = 'requested'`)
.concurrently(),
companyCreatedAtDescIdx: index("heartbeat_runs_company_created_at_desc_idx").on(
table.companyId,
table.createdAt.desc(),
@@ -0,0 +1,296 @@
import { randomUUID } from "node:crypto";
import os from "node:os";
import { spawn } from "node:child_process";
import { once } from "node:events";
import { setImmediate as immediate } from "node:timers/promises";
import { and, eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { agents, applyPendingMigrations, authUsers, closeCheckpointDb, closeLivenessDb, companies, companyMemberships,
controllerInstances, createDb, environmentLeases, heartbeatRunEvents, heartbeatRuns, issues, nativeRunFinalizations } from "@paperclipai/db";
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { meterDatabaseStatements } from "./helpers/database-statement-meter.js";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { heartbeatService } from "../services/heartbeat.js";
import { notifyLiveRunRecovery } from "../services/live-run-recovery-notifications.js";
import { appendHeartbeatRunEvent } from "../services/heartbeat-run-events.js";
import { persistHeartbeatRunProcessMetadata } from "../services/heartbeat/run-lifecycle.js";
import { discoverSettledNativeRunProcesses } from "../services/native-local-process-stop.js";
import { remoteTerminationReceipt } from "../services/remote-execution-termination.js";
vi.setConfig({ testTimeout: 300_000, hookTimeout: 180_000 });
describe("settled native process re-checks", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let db: ReturnType<typeof createDb>;
let meter: ReturnType<typeof meterDatabaseStatements>;
let heartbeat: ReturnType<typeof heartbeatService>;
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator>;
const errors: unknown[] = [];
const passes = new Set<Promise<unknown>>();
const children: number[] = [];
const checks = new Map<string, number>();
const stops: Array<() => Promise<void>> = [];
beforeAll(async () => {
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_SETTLED_RECHECK_TEST_DATABASE_URL");
if (!external) temporary = await startEmbeddedPostgresTestDatabase("settled-recheck-");
else await applyPendingMigrations(external);
const url = external ?? temporary!.connectionString;
db = createDb(url, { maxConnections: 4, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
meter = meterDatabaseStatements(db, url);
await Promise.all(Array.from({ length: 4 }, () => db.execute(sql`select pg_sleep(0.01)`)));
db.$client.options.debug = (_connection, query, parameters) => {
if (!/for update/i.test(query) || !query.includes("sessionCheckpoint")) return;
const runId = (parameters as unknown[]).find(value => typeof value === "string" && checks.has(value)) as string | undefined;
if (runId) checks.set(runId, checks.get(runId)! + 1);
};
});
beforeEach(async () => {
await db.execute(sql`truncate companies, controller_instances cascade`);
errors.length = 0; checks.clear();
heartbeat = heartbeatService(db);
coordinator = newCoordinator();
});
afterEach(async () => {
for (const stop of stops.splice(0)) await stop();
await coordinator.stop(); await heartbeat.drainActiveRunExecutions();
for (const pid of children.splice(0)) { try { process.kill(-pid, "SIGKILL"); } catch { } }
vi.useRealTimers(); vi.restoreAllMocks(); expect(errors).toEqual([]);
});
afterAll(async () => {
meter?.uninstall();
if (db) { await closeLivenessDb(db, 1); await closeCheckpointDb(db, 1); }
await db?.$client.end({ timeout: 1 }); await temporary?.cleanup();
});
function newCoordinator() {
const created = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true,
onError: error => errors.push(error) });
const register = created.register.bind(created);
created.register = (queue, task) => register(queue, { ...task, run: signal => {
const pass = Promise.resolve(task.run(signal)); passes.add(pass);
void pass.finally(() => passes.delete(pass)); return pass;
} });
return created;
}
async function fixture() {
const companyId = randomUUID(), actorId = "settled-recheck-user";
await db.insert(authUsers).values({ id: actorId, name: "Owner", email: `${actorId}@example.test`,
createdAt: new Date(), updatedAt: new Date() }).onConflictDoNothing();
await db.insert(companies).values({ id: companyId, name: "Settled", issuePrefix: `SR${companyId.slice(0, 6)}`,
requireBoardApprovalForNewAgents: false, defaultResponsibleUserId: actorId });
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: actorId,
status: "active", membershipRole: "owner" });
const [agent] = await db.insert(agents).values({ companyId, name: "Worker", status: "idle", adapterType: "process",
adapterConfig: {}, runtimeConfig: { heartbeat: { enabled: false, wakeOnDemand: true } } }).returning();
const [issue] = await db.insert(issues).values({ companyId, title: "Work", status: "done", assigneeAgentId: agent.id }).returning();
return { companyId, agentId: agent.id, issueId: issue.id };
}
type Fixture = Awaited<ReturnType<typeof fixture>>;
async function exitedPid() {
const child = spawn(process.execPath, ["-e", "0"], { detached: true, stdio: "ignore" });
const pid = child.pid!; await once(child, "exit"); return pid;
}
function livePid() {
const child = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)"], { detached: true, stdio: "ignore" });
child.unref(); children.push(child.pid!); return child.pid!;
}
async function kill(pid: number) {
try { process.kill(-pid, "SIGKILL"); } catch { }
for (let attempt = 0; attempt < 100; attempt++) {
try { process.kill(pid, 0); } catch { return; }
await new Promise(resolve => setImmediate(resolve));
}
}
type Seed = { host?: { hostname: string; hash: string | null }; phase?: string | null; checkpoint?: boolean; live?: boolean;
birthMismatch?: boolean; remote?: "stopped" | "unconfirmed" };
async function seed(f: Fixture, options: Seed = {}) {
const pid = options.live ? livePid() : await exitedPid();
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, runtimeMode: "native",
nativeIssueId: f.issueId, status: "running", invocationSource: "assignment", triggerDetail: "system",
runnerProfileJson: options.checkpoint === false ? {} : { sessionCheckpoint: { providerSessionId: "thread" } } }).returning();
if (options.host || options.birthMismatch) {
const startedAt = new Date();
const host = options.host ?? { hostname: os.hostname(), hash: null };
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: run.id, agentId: f.agentId,
eventType: "native.process_identity_recorded", stream: "system", level: "info", message: "identity",
payload: { targetKind: options.remote ? "remote" : "local", hostname: host.hostname, processPid: pid, processGroupId: pid,
processStartedAt: new Date(startedAt.getTime() - (options.birthMismatch ? 5_000 : 0)).toISOString() },
runUpdate: { processPid: pid, processGroupId: pid, processStartedAt: startedAt, processHostHash: host.hash } });
} else {
await persistHeartbeatRunProcessMetadata(db, run.id, { pid, processGroupId: pid, startedAt: new Date().toISOString(), targetKind: "local" });
}
if (options.phase !== null) await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: f.issueId,
runId: run.id, phase: options.phase ?? "committed", controllerBootId: randomUUID() });
const [lease] = await db.insert(environmentLeases).values({ companyId: f.companyId, issueId: f.issueId, heartbeatRunId: run.id,
provider: options.remote ? "daytona" : "local", providerLeaseId: options.remote ? randomUUID() : null, leasePolicy: "ephemeral",
status: "released", releasedAt: new Date(), cleanupStatus: "success" }).returning();
if (options.remote === "stopped") await db.update(environmentLeases).set({ metadata: {
remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "stopped" }),
} }).where(eq(environmentLeases.id, lease.id));
await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date() }).where(eq(heartbeatRuns.id, run.id));
checks.set(run.id, 0);
return { runId: run.id, pid };
}
async function start(target = coordinator) {
if (!vi.isFakeTimers()) vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
await heartbeat.registerResidualRecoveryWork(target, () => true).ready;
await advance(2_000);
}
async function advance(ms: number) {
const until = Date.now() + ms;
while (Date.now() < until) {
await vi.advanceTimersByTimeAsync(Math.min(1_000, until - Date.now()));
await Promise.allSettled([...passes]); await immediate();
}
}
const companyHint = (companyId: string) => db.transaction(tx => notifyLiveRunRecovery(tx, { companyId }));
const runHint = (companyId: string, runId: string) => db.transaction(tx => notifyLiveRunRecovery(tx, { companyId, runId }));
async function hints(companyId: string, count: number) {
for (let index = 0; index < count; index++) { await companyHint(companyId); await advance(31_000); }
}
const processPid = async (id: string) => (await db.select({ pid: heartbeatRuns.processPid }).from(heartbeatRuns)
.where(eq(heartbeatRuns.id, id)))[0]!.pid;
const stopReceipts = async (id: string) => (await db.select().from(heartbeatRunEvents).where(and(
eq(heartbeatRunEvents.runId, id), eq(heartbeatRunEvents.eventType, "native.local_process_stopped")))).length;
it.each([
["recorded by a replaced container", { host: { hostname: "replaced-container", hash: "replaced-container-space" } }],
["whose coordinator phase is unsettled", { phase: "workspace_finalizing" }],
["without a checkpoint or provider evidence", { checkpoint: false }],
] as const)("never re-reads a terminal run %s on company hints", async (_name, options) => {
const f = await fixture();
const { runId, pid } = await seed(f, options as Seed);
await start();
await hints(f.companyId, 4);
expect(checks.get(runId)).toBe(0);
expect(await processPid(runId)).toBe(pid);
});
it.each(["live", "stopped"] as const)("leaves a %s older server's runs to that server", async state => {
const f = await fixture();
const bootId = randomUUID();
await db.insert(controllerInstances).values({ bootId, processSpace: "older-server-space", pid: 4242,
leaseExpiresAt: new Date(Date.now() + (state === "live" ? 60_000 : -600_000)),
stoppedAt: state === "stopped" ? new Date() : null });
const { runId, pid } = await seed(f, { host: { hostname: "older-server", hash: "older-server-space" } });
await db.update(nativeRunFinalizations).set({ controllerBootId: bootId, controllerHostHash: "older-server-space" })
.where(eq(nativeRunFinalizations.runId, runId));
const own = await seed(f);
await start();
expect(await processPid(own.runId)).toBeNull();
await hints(f.companyId, 2);
expect(checks.get(runId)).toBe(0);
expect(await processPid(runId)).toBe(pid);
expect(await stopReceipts(runId)).toBe(0);
});
it("re-admits a run once its coordinator phase settles", async () => {
const f = await fixture();
const { runId } = await seed(f, { phase: "workspace_finalizing" });
await start();
await hints(f.companyId, 1);
expect(await processPid(runId)).not.toBeNull();
await db.update(nativeRunFinalizations).set({ phase: "committed" }).where(eq(nativeRunFinalizations.runId, runId));
await hints(f.companyId, 1);
expect(await processPid(runId)).toBeNull();
expect(checks.get(runId)).toBe(1);
expect(await stopReceipts(runId)).toBe(1);
});
it("re-admits a run once late provider evidence arrives", async () => {
const f = await fixture();
const { runId } = await seed(f, { checkpoint: false });
await start();
await hints(f.companyId, 1);
expect(await processPid(runId)).not.toBeNull();
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId, agentId: f.agentId, eventType: "harness.ready",
stream: "system", level: "info", message: "ready" });
await hints(f.companyId, 1);
expect(await processPid(runId)).toBeNull();
expect(checks.get(runId)).toBe(1);
});
it("releases legacy rows without a PID-space hash only when this host recorded them", async () => {
const f = await fixture();
const own = await seed(f, { host: { hostname: os.hostname(), hash: null } });
const foreign = await seed(f, { host: { hostname: "legacy-other-host", hash: null } });
await start();
expect(await processPid(own.runId)).toBeNull();
await hints(f.companyId, 2);
expect(checks.get(foreign.runId)).toBe(0);
expect(await processPid(foreign.runId)).toBe(foreign.pid);
});
it("releases a sandbox run from any server once its termination receipt exists", async () => {
const f = await fixture();
const pending = await seed(f, { host: { hostname: "replaced-container", hash: "remote" }, remote: "unconfirmed" });
const stopped = await seed(f, { host: { hostname: "replaced-container", hash: "remote" }, remote: "stopped" });
await start();
expect(await processPid(stopped.runId)).toBeNull();
expect(await processPid(pending.runId)).not.toBeNull();
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.heartbeatRunId, pending.runId));
await db.update(environmentLeases).set({ metadata: {
remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "stopped" }),
} }).where(eq(environmentLeases.id, lease.id));
await runHint(f.companyId, pending.runId);
await advance(2_000);
expect(await processPid(pending.runId)).toBeNull();
});
it("backs off a live local process and releases it on the next run-scoped hint after it exits", async () => {
const f = await fixture();
const { runId, pid } = await seed(f, { live: true });
await start();
expect(checks.get(runId)).toBe(1);
await hints(f.companyId, 8);
expect(checks.get(runId)).toBeLessThanOrEqual(4);
expect(await processPid(runId)).toBe(pid);
await kill(pid);
await runHint(f.companyId, runId);
await advance(2_000);
expect(await processPid(runId)).toBeNull();
expect(await stopReceipts(runId)).toBe(1);
});
it("checks a same-host skip once with one locking statement, then only on its own trigger", async () => {
const f = await fixture();
const { runId } = await seed(f, { birthMismatch: true, host: { hostname: os.hostname(), hash: null } });
meter.startLog();
await start();
const log = meter.stopLog();
expect(checks.get(runId)).toBe(1);
const inCheck = log.filter(entry => entry.inTransaction || /^\s*begin/i.test(entry.sql));
expect(inCheck.filter(entry => /for update/i.test(entry.sql))).toHaveLength(1);
expect(inCheck.length).toBeLessThanOrEqual(3);
await hints(f.companyId, 3);
expect(checks.get(runId)).toBe(1);
await runHint(f.companyId, runId);
await advance(2_000);
expect(checks.get(runId)).toBe(2);
});
it("checks a pending run at most once after a restart and discovers no excluded run at startup", async () => {
const f = await fixture();
const { runId } = await seed(f, { live: true });
for (let index = 0; index < 50; index++) await seed(f, { host: { hostname: "replaced-container", hash: "replaced-container-space" } });
await start();
expect(checks.get(runId)).toBe(1);
await coordinator.stop();
const restarted = newCoordinator();
stops.push(() => restarted.stop());
await start(restarted);
expect(checks.get(runId)).toBe(2);
vi.useRealTimers();
const before = meter.read();
const counted = [...checks.values()].reduce((sum, value) => sum + value, 0);
await discoverSettledNativeRunProcesses(db, { skip: id => id === runId });
expect(meter.read().statements - before.statements).toBe(1);
expect([...checks.values()].reduce((sum, value) => sum + value, 0)).toBe(counted);
});
});
@@ -1,6 +1,6 @@
import { pendingFailedNativeRetryCancellationCondition } from "../native-runtime/native-cancellation-request.js";
import { legacyWorkspaceRecoveryNotMigrated } from "../legacy-workspace-recovery-migration.js";
import { settledNativeProcessIdentityCondition } from "../native-local-process-stop.js";
import { settledNativeProcessCandidates } from "../native-local-process-stop.js";
import { and, asc, eq, gt, inArray, isNotNull, isNull, or, sql } from "drizzle-orm";
import { agents, environmentLeases, heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db";
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
@@ -9,6 +9,8 @@ import type { LiveResourceRecoveryScope, LiveRunRecoveryScope } from "../live-ru
import { registerScopedDeadlineWork } from "../scoped-deadline-work.js";
const TERMINAL = ["succeeded", "failed", "cancelled", "timed_out", "interrupted"];
const SETTLED_PENDING_BASE_DELAY_MS = 30_000;
const SETTLED_PENDING_MAX_DELAY_MS = 15 * 60_000;
type Scope = LiveRunRecoveryScope & LiveResourceRecoveryScope;
type Entry = { id: string; companyId: string; runId?: string | null; dueAt: number };
type CheckScope = { companyId: string; runId?: string; leaseId?: string; agentId?: string;
@@ -27,6 +29,8 @@ export function registerLiveRecoveryWork(db: Db, coordinator: ReturnType<typeof
}, canRun: () => boolean) {
function register(queue: DeliveryQueue, kind: RunKind | "orphanLease" | "shutdown" | "agent") {
const cursors = new Map<string, string>();
const settledOutcomes = new Map<string, { companyId: string; parked: boolean; attempts: number; notBefore: number }>();
const settledScans = new Map<string, Set<string>>();
let enabled = false;
async function load(scope?: Scope) {
const key = scope ? JSON.stringify({ companyId: scope.companyId, runId: scope.runId,
@@ -63,12 +67,11 @@ export function registerLiveRecoveryWork(db: Db, coordinator: ReturnType<typeof
? row.runId ? row.updatedAt.getTime() : row.updatedAt.getTime() + 5 * 60_000
: Date.parse(String((row.metadata?.legacyShutdownWorkspaceSettlement as Record<string, unknown>)?.deadlineAt)) || Date.now(),
}));
} else if (kind === "settled") {
const page = await loadSettled(scope, cursor);
if (page.complete) cursors.delete(key); else cursors.set(key, page.rows.at(-1)!.id);
return { complete: page.complete, rows: applySettledOutcomes(key, scope, page.rows, page.complete) };
} else {
// Retirement may commit before Stop acknowledgement. This is only a
// work hint; the owner rechecks the complete provider/process proof.
const pendingApiStop = and(eq(heartbeatRuns.status, "cancelled"), eq(heartbeatRuns.driverKind, "cursor_cloud_api"),
sql`${heartbeatRuns.runnerProfileJson}->>'backend' = 'cursor_cloud_api'`,
sql`${heartbeatRuns.resultJson}->'executionCancellation'->>'state' = 'requested'`);
const candidates = await db.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId,
nextAttemptAt: nativeRunFinalizations.nextAttemptAt, leaseExpiresAt: nativeRunFinalizations.leaseExpiresAt })
.from(heartbeatRuns).leftJoin(nativeRunFinalizations, and(eq(nativeRunFinalizations.runId, heartbeatRuns.id),
@@ -83,12 +86,8 @@ export function registerLiveRecoveryWork(db: Db, coordinator: ReturnType<typeof
: kind === "question" ? and(or(inArray(heartbeatRuns.status, ["queued", "running", "scheduled_retry"]),
pendingFailedNativeRetryCancellationCondition()),
sql`${heartbeatRuns.contextSnapshot}->'nativeQuestionCancellation' is not null`)
: kind === "active" ? and(eq(heartbeatRuns.status, "running"),
sql`${heartbeatRuns.contextSnapshot}->'nativeQuestionCancellation' is null`)
: and(eq(heartbeatRuns.runtimeMode, "native"), inArray(heartbeatRuns.status, ["failed", "cancelled", "timed_out", "interrupted"]),
isNotNull(heartbeatRuns.finishedAt), or(pendingApiStop,
and(or(isNotNull(heartbeatRuns.processPid), isNotNull(heartbeatRuns.processGroupId),
isNotNull(heartbeatRuns.processStartedAt)), settledNativeProcessIdentityCondition(db)))),
: and(eq(heartbeatRuns.status, "running"),
sql`${heartbeatRuns.contextSnapshot}->'nativeQuestionCancellation' is null`),
)).orderBy(asc(heartbeatRuns.id)).limit(100);
rows = candidates.map(row => ({ id: row.id, companyId: row.companyId, dueAt: kind === "native"
? Math.max(Date.now(), row.nextAttemptAt?.getTime() ?? 0, row.leaseExpiresAt?.getTime() ?? 0) : Date.now() }));
@@ -97,6 +96,50 @@ export function registerLiveRecoveryWork(db: Db, coordinator: ReturnType<typeof
if (complete) cursors.delete(key); else cursors.set(key, rows.at(-1)!.id);
return { complete, rows };
}
async function loadSettled(scope: Scope | undefined, cursor: string | undefined) {
const bounds = and(scope ? eq(heartbeatRuns.companyId, scope.companyId) : undefined,
scope?.runId ? eq(heartbeatRuns.id, scope.runId) : undefined, cursor ? gt(heartbeatRuns.id, cursor) : undefined);
const select = (condition: ReturnType<typeof and>) => db.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId })
.from(heartbeatRuns).where(and(bounds, condition)).orderBy(asc(heartbeatRuns.id)).limit(100);
const processes = await settledNativeProcessCandidates(db, { companyId: scope?.companyId, runId: scope?.runId,
after: cursor, limit: 100 });
const stops = await select(and(eq(heartbeatRuns.status, "cancelled"), eq(heartbeatRuns.driverKind, "cursor_cloud_api"),
sql`(${heartbeatRuns.resultJson}->'executionCancellation'->>'state') = 'requested'`,
eq(heartbeatRuns.runtimeMode, "native"), isNotNull(heartbeatRuns.finishedAt),
sql`${heartbeatRuns.runnerProfileJson}->>'backend' = 'cursor_cloud_api'`));
const merged = [...new Map([...processes, ...stops].map(row => [row.id, row])).values()]
.sort((left, right) => left.id < right.id ? -1 : left.id > right.id ? 1 : 0);
const complete = processes.length < 100 && stops.length < 100 && merged.length <= 100;
return { complete, rows: merged.slice(0, 100).map(row => ({ ...row, dueAt: Date.now() })) };
}
function applySettledOutcomes(key: string, scope: Scope | undefined, rows: Entry[], complete: boolean) {
if (scope?.runId) settledOutcomes.delete(scope.runId);
const seen = settledScans.get(key) ?? new Set<string>();
for (const row of rows) seen.add(row.id);
if (!complete) settledScans.set(key, seen);
else {
settledScans.delete(key);
if (!scope?.runId) for (const [id, outcome] of settledOutcomes)
if ((!scope || outcome.companyId === scope.companyId) && !seen.has(id)) settledOutcomes.delete(id);
}
return rows.flatMap(row => {
const outcome = settledOutcomes.get(row.id);
if (outcome?.parked) return [];
return [{ ...row, dueAt: Math.max(row.dueAt, outcome?.notBefore ?? 0) }];
});
}
function recordSettledOutcome(entry: Entry, outcome: unknown): Entry | null | undefined {
if (outcome === "released") { settledOutcomes.delete(entry.id); return null; }
if (outcome === "skipped") {
settledOutcomes.set(entry.id, { companyId: entry.companyId, parked: true, attempts: 0, notBefore: 0 });
return null;
}
if (outcome !== "pending") return undefined;
const attempts = (settledOutcomes.get(entry.id)?.attempts ?? 0) + 1;
const notBefore = Date.now() + Math.min(SETTLED_PENDING_MAX_DELAY_MS, SETTLED_PENDING_BASE_DELAY_MS * 2 ** (attempts - 1));
settledOutcomes.set(entry.id, { companyId: entry.companyId, parked: false, attempts, notBefore });
return { ...entry, dueAt: notBefore };
}
return registerScopedDeadlineWork<Entry, Scope>(coordinator, {
queue,
reconstructionIntervalMs: 30_000,
@@ -114,10 +157,12 @@ export function registerLiveRecoveryWork(db: Db, coordinator: ReturnType<typeof
const currentScope = { companyId: entry.companyId,
...(kind === "agent" ? { agentId: entry.id } : kind === "orphanLease" || kind === "shutdown"
? { leaseId: entry.id } : { runId: entry.id }) };
const loaded = await load(currentScope);
const current = (Array.isArray(loaded) ? loaded : loaded.rows).find(row => row.id === entry.id);
if (!current) return null;
if (current.dueAt > Date.now()) return current;
if (kind !== "settled") {
const loaded = await load(currentScope);
const current = (Array.isArray(loaded) ? loaded : loaded.rows).find(row => row.id === entry.id);
if (!current) return null;
if (current.dueAt > Date.now()) return current;
}
let next: number | null = null;
const errors: unknown[] = [];
const scope: CheckScope = { companyId: entry.companyId,
@@ -126,8 +171,12 @@ export function registerLiveRecoveryWork(db: Db, coordinator: ReturnType<typeof
onDeferredDeadline: at => { next = next === null ? at : Math.min(next, at); },
onError: error => { errors.push(error); },
};
await owner[kind](scope);
const outcome = await owner[kind](scope);
if (errors.length) throw new AggregateError(errors, `Recovery obligation ${entry.id} failed`);
if (kind === "settled") {
const settled = recordSettledOutcome(entry, outcome);
if (settled !== undefined) return settled;
}
if (kind === "native") {
const refreshed = await load(currentScope);
const remaining = (Array.isArray(refreshed) ? refreshed : refreshed.rows).find(row => row.id === entry.id);
+5 -4
View File
@@ -39,7 +39,7 @@ import {
hasStopOnlyCleanup,
settleStopOnlyCleanup,
} from "../sandbox-stop-and-retain.js";
import { discoverSettledNativeRunProcesses, PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED, recordNativeLocalProcessStopInTransaction, releaseSettledNativeRunProcess, retryPendingSettledNativeRunProcesses } from "../native-local-process-stop.js";
import { discoverSettledNativeRunProcesses, inspectSettledNativeRunProcess, PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED, recordNativeLocalProcessStopInTransaction, retryPendingSettledNativeRunProcesses } from "../native-local-process-stop.js";
import { pendingCaseHandoffForRun } from "./case-handoff-lifecycle.js";
import { portableTaskPreparationLeaseId } from "../plugin-task-ownership.js";
import { retireInheritedRemoteRunnerIdentity } from "../native-runtime/native-inherited-runner-retry.js";
@@ -3109,10 +3109,11 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
if (!scope.runId || activeRunExecutions.has(scope.runId) || runningProcesses.has(scope.runId)
|| adapterExecutionControls.has(scope.runId)) {
scope.onDeferredDeadline(Date.now() + 30_000);
return;
return "deferred";
}
const outcome = await releaseSettledNativeRunProcess(db, scope.runId, scope);
if (outcome !== "pending") await acknowledgeRemoteStop(scope.runId, scope.companyId);
const { outcome, remoteStop } = await inspectSettledNativeRunProcess(db, scope.runId, scope);
if (outcome !== "pending" && remoteStop) await acknowledgeRemoteStop(scope.runId, scope.companyId);
return outcome;
},
orphanLease: scope => sweepOrphanedActiveLeases({ ...scope, backoffMs: 5 * 60_000 }),
shutdown: scope => sweepLegacyShutdownWorkspaceSettlements(db, new Date(), scope),
+108 -57
View File
@@ -1,13 +1,13 @@
import { notifyLiveRunRecovery } from "./live-run-recovery-notifications.js";
import { and, asc, desc, eq, exists, gt, inArray, isNotNull, isNull, or, sql } from "drizzle-orm";
import { environmentLeases, heartbeatRunEvents, heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db";
import { and, desc, eq, gt, inArray, isNotNull, isNull, or, sql } from "drizzle-orm";
import { environmentLeases, heartbeatRunEvents, heartbeatRuns, type Db } from "@paperclipai/db";
import { notifyRetainedNativeSessionCleanup } from "./retained-native-session-cleanup-notifications.js";
import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
import { nativeSessionProviderEvidence } from "./native-runtime/native-session-resume.js";
import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
import { NATIVE_SESSION_PROVIDER_EVENT_TYPES } from "./native-runtime/native-session-resume.js";
import os from "node:os";
import { currentProcessSpaceHash } from "./controller-process-identity.js";
import { currentProcessSpaceHash, UNKNOWN_PROCESS_SPACE } from "./controller-process-identity.js";
import { proveOwnerGone } from "./owner-proof.js";
export const PROCESS_START_REQUESTED = "native.process_start_requested";
@@ -69,6 +69,12 @@ export async function recordNativeLocalProcessStop(db: Db, run: Pick<typeof hear
if (leases.some(lease => lease.provider !== "local")) return false;
if ((run.processPid && !absent(run.processPid)) ||
(run.processGroupId && !absent(-run.processGroupId))) return false;
await appendLocalProcessStop(db, { ...run, processStartedAt: run.processStartedAt?.toISOString() ?? null });
return true;
}
async function appendLocalProcessStop(db: Db, run: { id: string; companyId: string; agentId: string;
processPid: number | null; processGroupId: number | null; processStartedAt: string | null }) {
await notifyDeliveryWork(db, DELIVERY_QUEUES.executionReplacement,
JSON.stringify({ companyId: run.companyId, runId: run.id }));
await appendHeartbeatRunEvent(db, {
@@ -79,10 +85,8 @@ export async function recordNativeLocalProcessStop(db: Db, run: Pick<typeof hear
stream: "system",
level: "info",
message: "Recovery verified the local process stopped before clearing its identity.",
payload: { processPid: run.processPid, processGroupId: run.processGroupId,
processStartedAt: run.processStartedAt?.toISOString() ?? null },
payload: { processPid: run.processPid, processGroupId: run.processGroupId, processStartedAt: run.processStartedAt },
});
return true;
}
/** Transaction form for callers that clear the process identity: a stopped
@@ -376,69 +380,122 @@ export async function readNativeLocalProcessStop(db: Db, companyId: string, runI
}
type SettledProcessRelease = "released" | "pending" | "skipped";
export type SettledProcessReleaseResult = { outcome: SettledProcessRelease; remoteStop: boolean };
export function settledNativeProcessIdentityCondition(db: Db) {
const identity = db.select({ payload: heartbeatRunEvents.payload }).from(heartbeatRunEvents).where(and(
eq(heartbeatRunEvents.companyId, heartbeatRuns.companyId), eq(heartbeatRunEvents.runId, heartbeatRuns.id),
eq(heartbeatRunEvents.eventType, PROCESS_IDENTITY_RECORDED), isNull(heartbeatRunEvents.sourceEventId),
)).orderBy(desc(heartbeatRunEvents.seq)).limit(1).as("settled_process_identity");
return or(isNotNull(heartbeatRuns.processStartedAt), exists(db.select({ id: environmentLeases.id })
.from(environmentLeases).where(and(eq(environmentLeases.companyId, heartbeatRuns.companyId),
eq(environmentLeases.heartbeatRunId, heartbeatRuns.id), sql`${environmentLeases.provider} <> 'local'`))),
and(gt(heartbeatRuns.processPid, 1), eq(heartbeatRuns.processPid, heartbeatRuns.processGroupId),
exists(db.select({ payload: identity.payload }).from(identity).where(and(
sql`${identity.payload}->>'targetKind' = 'local'`, sql`${identity.payload}->>'hostname' = ${os.hostname()}`,
sql`${identity.payload}->>'processPid' = ${heartbeatRuns.processPid}::text`,
sql`${identity.payload}->>'processGroupId' = ${heartbeatRuns.processGroupId}::text`,
)))));
const providerEventTypes = () => sql.join(NATIVE_SESSION_PROVIDER_EVENT_TYPES.map(type => sql`${type}`), sql`, `);
function settledNativeProcessReleaseCondition() {
const processSpace = currentProcessSpaceHash();
const ownSpace = processSpace === UNKNOWN_PROCESS_SPACE ? undefined : eq(heartbeatRuns.processHostHash, processSpace);
const remoteLease = sql`exists (select 1 from environment_leases settled_lease where settled_lease.company_id = ${heartbeatRuns.companyId}
and settled_lease.heartbeat_run_id = ${heartbeatRuns.id} and settled_lease.provider is distinct from 'local')`;
const unrecordedSpace = or(isNull(heartbeatRuns.processHostHash), eq(heartbeatRuns.processHostHash, UNKNOWN_PROCESS_SPACE));
return and(eq(heartbeatRuns.runtimeMode, "native"), inArray(heartbeatRuns.status, SETTLED_RUN_STATUSES),
isNotNull(heartbeatRuns.finishedAt),
or(isNotNull(heartbeatRuns.processPid), isNotNull(heartbeatRuns.processGroupId), isNotNull(heartbeatRuns.processStartedAt)),
or(ownSpace, eq(heartbeatRuns.processHostHash, "remote"), isNull(heartbeatRuns.processHostHash),
eq(heartbeatRuns.processHostHash, UNKNOWN_PROCESS_SPACE)),
or(ownSpace, remoteLease, and(unrecordedSpace, sql`(select settled_identity.payload->>'hostname' from heartbeat_run_events settled_identity
where settled_identity.company_id = ${heartbeatRuns.companyId} and settled_identity.run_id = ${heartbeatRuns.id}
and settled_identity.event_type = ${PROCESS_IDENTITY_RECORDED} and settled_identity.source_event_id is null
order by settled_identity.seq desc limit 1) = ${os.hostname()}::text`)),
sql`not exists (select 1 from native_run_finalizations settled_coordinator
where settled_coordinator.run_id = ${heartbeatRuns.id} and settled_coordinator.phase not in ('committed', 'terminal_failure'))`,
sql`(coalesce(${heartbeatRuns.runnerProfileJson}->'sessionCheckpoint', 'null'::jsonb) <> 'null'::jsonb
or exists (select 1 from heartbeat_run_events settled_evidence where settled_evidence.company_id = ${heartbeatRuns.companyId}
and settled_evidence.run_id = ${heartbeatRuns.id} and settled_evidence.event_type in (${providerEventTypes()})))`);
}
export async function settledNativeProcessCandidates(db: Pick<Db, "execute">, input: {
companyId?: string; runId?: string; after?: string | null; limit: number;
}): Promise<Array<{ id: string; companyId: string }>> {
const processSpace = currentProcessSpaceHash();
const condition = and(input.companyId ? eq(heartbeatRuns.companyId, input.companyId) : undefined,
input.runId ? eq(heartbeatRuns.id, input.runId) : undefined, input.after ? gt(heartbeatRuns.id, input.after) : undefined,
settledNativeProcessReleaseCondition());
const spaces = [processSpace === UNKNOWN_PROCESS_SPACE ? null : eq(heartbeatRuns.processHostHash, processSpace),
eq(heartbeatRuns.processHostHash, "remote"), eq(heartbeatRuns.processHostHash, UNKNOWN_PROCESS_SPACE),
isNull(heartbeatRuns.processHostHash)].filter(space => space !== null);
const branches = spaces.map(space => sql`(select ${heartbeatRuns.id} as id, ${heartbeatRuns.companyId} as company_id
from ${heartbeatRuns} where ${and(space, condition)} offset 0)`);
return await db.execute(sql`select id, company_id as "companyId" from (${sql.join(branches, sql` union all `)}) candidates
order by id limit ${input.limit}`) as unknown as Array<{ id: string; companyId: string }>;
}
type SettledProcessFacts = {
companyId: string; agentId: string; runtimeMode: string; status: string; finished: boolean;
processPid: number | null; processGroupId: number | null; processStartedAt: string | null; apiDriver: boolean | null;
hasSessionCheckpoint: boolean; phase: string | null; providerEvidence: boolean; remoteLeases: boolean;
localLeasePending: boolean; identityMatches: boolean | null;
};
function readSettledProcessFacts(tx: Db, runId: string) {
return tx.execute(sql`select r.company_id as "companyId", r.agent_id as "agentId", r.runtime_mode as "runtimeMode",
r.status, r.finished_at is not null as "finished", r.process_pid as "processPid", r.process_group_id as "processGroupId",
to_char(r.process_started_at at time zone 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') as "processStartedAt",
r.driver_kind = 'cursor_cloud_api' as "apiDriver",
coalesce(r.runner_profile_json->'sessionCheckpoint', 'null'::jsonb) <> 'null'::jsonb as "hasSessionCheckpoint",
(select f.phase from native_run_finalizations f where f.run_id = r.id limit 1) as "phase",
exists (select 1 from heartbeat_run_events e where e.company_id = r.company_id and e.run_id = r.id
and e.event_type in (${providerEventTypes()})) as "providerEvidence",
exists (select 1 from environment_leases l where l.company_id = r.company_id and l.heartbeat_run_id = r.id
and l.provider is distinct from 'local') as "remoteLeases",
exists (select 1 from environment_leases l where l.company_id = r.company_id and l.heartbeat_run_id = r.id
and (l.released_at is null or l.status = 'pending_cleanup' or l.cleanup_status = 'failed')) as "localLeasePending",
(select identity.payload->>'targetKind' = 'local' and identity.payload->>'hostname' = ${os.hostname()}::text
and identity.payload ? 'processPid' and identity.payload->'processPid' = coalesce(to_jsonb(r.process_pid), 'null'::jsonb)
and identity.payload ? 'processGroupId' and identity.payload->'processGroupId' = coalesce(to_jsonb(r.process_group_id), 'null'::jsonb)
and case when r.process_started_at is not null
then identity.payload->>'processStartedAt' = to_char(r.process_started_at at time zone 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"')
else coalesce(r.process_pid > 1 and r.process_group_id = r.process_pid, false) end
from (select e.payload from heartbeat_run_events e where e.company_id = r.company_id and e.run_id = r.id
and e.event_type = ${PROCESS_IDENTITY_RECORDED}::text and e.source_event_id is null order by e.seq desc limit 1) identity
) as "identityMatches"
from heartbeat_runs r where r.id = ${runId}::uuid for update of r`) as unknown as Promise<SettledProcessFacts[]>;
}
export async function releaseSettledNativeRunProcess(db: Db, runId: string, options: { onDeferredDeadline?: (at: number) => void } = {}): Promise<SettledProcessRelease> {
return (await inspectSettledNativeRunProcess(db, runId, options)).outcome;
}
export async function inspectSettledNativeRunProcess(db: Db, runId: string, options: { onDeferredDeadline?: (at: number) => void } = {}): Promise<SettledProcessReleaseResult> {
let remoteStop = false;
const outcome = await db.transaction(async transaction => {
const tx = transaction as unknown as Db;
const [run] = await tx.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId,
runtimeMode: heartbeatRuns.runtimeMode, status: heartbeatRuns.status, finishedAt: heartbeatRuns.finishedAt,
processPid: heartbeatRuns.processPid, processGroupId: heartbeatRuns.processGroupId, processStartedAt: heartbeatRuns.processStartedAt,
hasSessionCheckpoint: sql<boolean>`coalesce(${heartbeatRuns.runnerProfileJson}->'sessionCheckpoint', 'null'::jsonb) <> 'null'::jsonb`,
}).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for("update").limit(1);
if (!run || run.runtimeMode !== "native" || !SETTLED_RUN_STATUSES.includes(run.status) || !run.finishedAt ||
const [run] = await readSettledProcessFacts(tx, runId);
if (!run) return "skipped";
remoteStop = run.remoteLeases || (run.status === "cancelled" && run.apiDriver === true);
if (run.runtimeMode !== "native" || !SETTLED_RUN_STATUSES.includes(run.status) || !run.finished ||
(run.processPid === null && run.processGroupId === null && run.processStartedAt === null)) return "skipped";
const [coordinator] = await tx.select({ phase: nativeRunFinalizations.phase }).from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, run.id)).limit(1);
if (coordinator && !SETTLED_COORDINATOR_PHASES.includes(coordinator.phase)) return "skipped";
if (!run.hasSessionCheckpoint &&
!(await nativeSessionProviderEvidence(tx, [run.id])).has(run.id)) return "skipped";
const leases = await tx.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id),
));
if (leases.some(lease => lease.provider !== "local")) {
if (run.phase !== null && !SETTLED_COORDINATOR_PHASES.includes(run.phase)) return "skipped";
if (!run.hasSessionCheckpoint && !run.providerEvidence) return "skipped";
if (run.remoteLeases) {
const leases = await tx.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, runId),
));
if (!leases.every(hasRemoteTerminationReceipt)) return "pending";
} else {
if (leases.some(lease => !lease.releasedAt || lease.status === "pending_cleanup" || lease.cleanupStatus === "failed")) return "pending";
const [identity] = await tx.select({ payload: heartbeatRunEvents.payload }).from(heartbeatRunEvents).where(and(
eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id),
eq(heartbeatRunEvents.eventType, PROCESS_IDENTITY_RECORDED), isNull(heartbeatRunEvents.sourceEventId),
)).orderBy(desc(heartbeatRunEvents.seq)).limit(1);
if (identity?.payload?.targetKind !== "local" || identity.payload.hostname !== os.hostname() ||
identity.payload.processPid !== run.processPid || identity.payload.processGroupId !== run.processGroupId ||
(run.processStartedAt ? identity.payload.processStartedAt !== run.processStartedAt.toISOString()
: !Number.isSafeInteger(run.processPid) || run.processPid! <= 1 || run.processGroupId !== run.processPid)) return "skipped";
if (!(await recordNativeLocalProcessStop(tx, run))) {
if (run.localLeasePending) return "pending";
if (run.identityMatches !== true) return "skipped";
if ((!run.processPid && !run.processGroupId) || (run.processPid && !absent(run.processPid))
|| (run.processGroupId && !absent(-run.processGroupId))) {
options.onDeferredDeadline?.(Date.now() + 30_000);
return "pending";
}
await appendLocalProcessStop(tx, { id: runId, companyId: run.companyId, agentId: run.agentId,
processPid: run.processPid, processGroupId: run.processGroupId, processStartedAt: run.processStartedAt });
}
await notifyLiveRunRecovery(tx, { companyId: run.companyId, runId: run.id, agentId: run.agentId });
await notifyRetainedNativeSessionCleanup(tx, { companyId: run.companyId, runId: run.id });
await notifyLiveRunRecovery(tx, { companyId: run.companyId, runId, agentId: run.agentId });
await notifyRetainedNativeSessionCleanup(tx, { companyId: run.companyId, runId });
// Local and remote stops both close the run's usage source.
await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, run.companyId);
await tx.update(heartbeatRuns).set({
processPid: null, processGroupId: null, processStartedAt: null, updatedAt: new Date(),
}).where(eq(heartbeatRuns.id, run.id));
}).where(eq(heartbeatRuns.id, runId));
return "released" as const;
});
if (outcome !== "pending") pendingSettledProcessReleases.delete(runId);
return outcome;
return { outcome, remoteStop };
}
const MAX_SETTLED_PROCESS_RELEASE_ATTEMPTS = 20;
@@ -473,13 +530,7 @@ export async function discoverSettledNativeRunProcesses(db: Db, input: {
let after: string | null = null;
let released = 0;
do {
const candidates: Array<{ id: string }> = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.runtimeMode, "native"),
inArray(heartbeatRuns.status, SETTLED_RUN_STATUSES),
isNotNull(heartbeatRuns.finishedAt),
or(isNotNull(heartbeatRuns.processPid), isNotNull(heartbeatRuns.processGroupId), isNotNull(heartbeatRuns.processStartedAt)),
...(after ? [gt(heartbeatRuns.id, after)] : []),
)).orderBy(asc(heartbeatRuns.id)).limit(limit);
const candidates: Array<{ id: string }> = await settledNativeProcessCandidates(db, { after, limit });
for (const candidate of candidates) {
if (input.skip?.(candidate.id)) continue;
if (await settleNativeRunProcess(db, candidate.id) === "released") released += 1;
@@ -210,7 +210,7 @@ export function selectNativeSessionResumeRun<
return null;
}
const NATIVE_SESSION_PROVIDER_EVENT_TYPES = [
export const NATIVE_SESSION_PROVIDER_EVENT_TYPES = [
"harness.ready",
"session.started",
"session.resumed",