Files
PaperClipAI/server/src/services/native-runtime/native-safe-replacement.ts
T
DottaandClaude Opus 5.5 cec8c2b3f4 perf(recovery): name the run or task in run writers' recovery hints instead of the company (#21765)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs issue recovery queues on one Node main thread. The
`stranded-assignments` and `assigned-work-invariant` queues check
assigned tasks that have no live run. One stranded check costs about 19
database statements.
> - #21552 scoped run hints to the run's task graph. #21729 made the
5-minute backstop re-check only changed tasks. A hint with only a
company id still makes the queue re-check every open task of the
company, 25 per pass.
> - Under load on staging (7 native runs), the stranded check ran about
6.7 times per second, and its statement bundle was more than a third of
all statements. An activity fixture shows the cause: ordinary run
writers send a company-only hint once per run, and one company hint
costs about 850 checks. At 1,000 agents this work never stops.
> - This pull request makes those writers name the run (or the task)
they already know. A run that has no task now resolves to no tasks
instead of the whole company. The changed-tasks backstop no longer
counts an agent's own run heartbeat as a change to every task of that
agent.
> - The benefit: with 7 runs a minute plus one taskless run, stranded
checks drop from 18.4 to 1.0 per second and stranded-check statements
from 344 to 7 per second. Real company-wide changes (pause lifted,
budget, company status, tree holds) keep the full company pass.

## Linked Issues or Issue Description

Refs #21729, #21552, #21676, #21724

**What existing behavior does this improve?**

The issue recovery task queues (`stranded-assignments`,
`assigned-work-invariant`, `resolved-dependencies`, `stale-issue-locks`)
and the writers that wake them.

**Subsystem affected**

Server: issue recovery background work and the run lifecycle writers
that publish recovery hints.

**Current behavior**

Many run lifecycle writers know the run but publish a company-only hint,
for example `notifyExecutionRunWork(tx, { companyId })` in the native
finalizer and projection, `settleNativeTerminalWake`, the claimed-wake
settlement, and an environment lease release of a run without a task. A
queued run without a task publishes a company hint. A graph hint for a
run without a task resolves to "whole company". A comment sends a
company hint to `assigned-work-invariant` and `resolved-dependencies`.
Each of these re-checks all open tasks of the company. Also, the
changed-tasks backstop treats any change to the assignee agent's row as
a change to all of that agent's tasks, and every run start and finish
writes that row.

**Proposed behavior**

A writer that knows the run sends the run's graph hint. A writer that
knows the task sends the task hint. A run without a task affects no task
queue entry. An agent row change that is only the agent's own run
heartbeat does not mark its other tasks changed. Company-wide facts keep
the company hint.

**Reason and benefit**

The work is now proportional to activity, not to (activity x history).
See Verification.

## What Changed

- `server/src/services/issue-recovery-notifications.ts`:
- `notifyIssueRecovery` with a task sends the task hint to all four task
queues (before: only to `stranded-assignments` and `stale-issue-locks`;
`assigned-work-invariant` and `resolved-dependencies` got the company
hint). Only comments (`issue-comment-writes.ts`) call it with a task.
- `resolveIssueGraph` returns `[]` when a run-only graph has no task. A
missing task seed still returns `null` (whole company).
- `server/src/services/recovery/work.ts`: a load for an empty graph
returns no rows. The `assigned-work-invariant` queue emits the same
`check` debug event as `stranded-assignments` (with
`PAPERCLIP_DEBUG_WORK_SIGNALS=assigned-work-invariant`), with the load
reason.
- `server/src/services/recovery/changed-tasks.ts`: the assignee-agent
part of the changed-tasks condition skips an agent whose last write was
its run heartbeat (`last_heartbeat_at = updated_at`). A resume, pause or
configuration write (no heartbeat stamp) still counts.
- `server/src/services/heartbeat/queue.ts`: a queued run without a task
gets its id before the insert and publishes a run hint, not a company
hint. The daily-cap comment release publishes the run's graph hint.
- `server/src/services/environments.ts`, `work-intent-outbox.ts`,
`chat-inbound-wake-notice-notifications.ts`: a lease release passes its
run id. `promoteDeferredWorkIntents` scopes `lock_released` with only a
run to that run's graph. `notifyChatInboundWakeNotices` accepts an
optional `runId` for its settlement hint.
- Native and recovery run writers now pass the run id they already have
to `notifyExecutionRunWork`, `writeExecutionControl` or
`notifyExternalConversationSettlement`: `native-run-finalizer.ts` (3),
`native-run-projection.ts` (2), `native-session-executor.ts` (3),
`wake-request-claimed-settlement.ts`, `native-superseded-retry.ts`,
`native-safe-replacement.ts`, `native-restart-recovery.ts` (2),
`native-failed-workspace-recovery.ts`,
`native-recovery-preparation-failure.ts`,
`execution-control-reconciliation.ts`,
`execution-recovery-resolution.ts` (2). These are the native equivalents
of the legacy run paths in `heartbeat/queue.ts`, which already pass the
run id.
- Tests:
- `server/src/__tests__/issue-recovery-activity-hints.test.ts` (new):
staging-shaped company (780 in-review tasks with succeeded native runs,
40 in progress, 30 to-do, 50 agents) on embedded PostgreSQL, the real
recovery, dispatch and retry-cancellation workers, a fake clock, and
real process-adapter runs. Each run is woken by a user comment through
`heartbeat.wakeup`, starts, calls back to post an agent comment and move
the task to `in_review`, and finishes. One (or five) taskless runs a
minute run beside them. It counts checks by queue and reason, task-queue
signals by caller, and statements by owner. It asserts the workload
(runs started and succeeded, agent comments, run events, zero agent
write failures, at most 10% failed runs), then no company check and no
company-only task-queue signal. In CI it runs the 7-runs-a-minute
workload for 4 minutes (about 1 minute of real time). With
`RECOVERY_ACTIVITY_REPORT=<file>` it runs both workloads at full length,
attributes statements by stack, and writes the numbers below to that
file.
- `server/src/__tests__/issue-recovery-deadlines.test.ts`: pickup tests
for each narrowed writer (see Verification).

### Publishers of company-only (or `*`) hints to the recovery task
queues

"Per" is how often the writer fires on a busy instance. "Known" says
whether the run or task is known at the call site.

Narrowed in this pull request:

| Call site | Event | Per | Known | Now |
| --- | --- | --- | --- | --- |
| `heartbeat/queue.ts:5396` `enqueueWakeup` | queue a run without a task
(timer, manual, chat without a task) | each taskless run | run (id
generated) | run graph (no tasks) |
| `environments.ts:1780` `releaseLease` ->
`notifyChatInboundWakeNotices` | environment lease released | each run
without a task | run | run graph |
| `environments.ts:1792` `releaseLease` ->
`promoteDeferredWorkIntents(lock_released)` | lease released | each run
without a task | run | run graph |
| `environments.ts:1934` `releaseLeasesForRun` | leases of a run
released | each run without a task | run | run graph |
| `issue-comment-writes.ts:12` (`assigned-work-invariant`,
`resolved-dependencies` only) | comment | each comment | task | task |
| `native-runtime/native-run-projection.ts:54`
`settleNativeTerminalWake` | native run's wake claim settled | each
native run finish | run | run graph |
| `native-runtime/native-run-projection.ts:177` | native run status
projected | each native run finish | run | run graph |
| `native-runtime/native-run-finalizer.ts:408`, `:629`, `:2080` | native
finalization terminal / failure / projection | each native run finish |
run | run graph |
| `native-runtime/native-session-executor.ts:13018`, `:13085`, `:13284`
| native execution control deadline, cancelled retry, pre-result
settlement | each native run end | run | run graph |
| `recovery/wake-request-claimed-settlement.ts:67` | claimed wake
settled after the run | each run whose claim is reconciled | run | run
graph |
| `heartbeat/queue.ts:590` | daily-cap comment key release | each run
cancelled at the daily cap | run | run graph |
| `native-runtime/native-superseded-retry.ts:42`,
`native-safe-replacement.ts:613`, `native-restart-recovery.ts:1033`,
`:1935`, `native-failed-workspace-recovery.ts:164`,
`native-recovery-preparation-failure.ts:30`,
`execution-control-reconciliation.ts:270`,
`execution-recovery-resolution.ts:830`, `:965` | native recovery
terminal writes | each recovered run | run | run graph |
| `resolveIssueGraph` for any run-only hint of a run without a task (for
example `setWakeupStatus`, `run-lifecycle.ts:1570`) | run lifecycle |
each taskless run event | run | no tasks |

Left with the company hint (company-wide fact, or the graph is not
known):

| Call site | Event | Per | Why kept |
| --- | --- | --- | --- |
| `budgets.ts:371`, `:386`, `:429`, `:437`, `:454` | budget policy
changed, scope paused or resumed | operator or budget event |
eligibility of every task can change |
| `work-intent-outbox.ts:79` with `pause_lifted`, `tree_hold_released`,
`scheduling_resumed`, `reconciler_backstop`, and `accounting_settled`
with an active hard stop | hold release | operator event;
`accounting_settled` once per run only with a hard-stop policy | can
admit any task; guarded by "keeps the company pass for hold releases
that can admit any task" |
| `companies.ts:77`, `:350`, `:353`, `:361`, `:512`, `:522` | company
created, paused, resumed, archived | operator | company-wide |
| `agent-lifecycle.ts:65`, `:72`, `agent-lifecycle-driver.ts:29`,
`agent-configuration.ts:130`, `:159`, `agent-deletion.ts:61` | agent
paused, resumed, configured, deleted | operator | can change many tasks,
and other agents' review paths |
| `access.ts:570`, `:752`, `:912`, `:926`, `:960`, `:975`, `:1175`,
`board-claim.ts:129`, `:142`, `middleware/auth.ts:706` | membership and
authority changes | operator, first sign-in | company authority |
| `chat-channels/endpoints.ts:461`, `identity-links.ts:323`,
`resource-reconciliation.ts:290`, `slack/setup-services.ts:50`,
`chat-github-management.ts:182`, `:352`, `chat-work-notifications.ts:50`
| chat authority changes | operator | company authority |
| `instance-settings.ts:666`, `:669`, `:680` (`*`) | instance setting
changed | operator | instance-wide |
| `issue-tree-control.ts:809`, `:889`, `:1236`, `:1339` | tree hold
created or released | operator | many tasks |
| `company-deletion.ts:83`, `company-portability.ts:6560` | company
delete, import | rare | many tasks |
| `issues.ts:8015` (blockers replaced), `:11165` (task created),
`:12685`, `:12699` (task deleted), `:12991`, `:14618` | task graph
changes | per task create or blocker edit | graph moves; #21552 kept
these |
| `documents.ts:615`, `:673`, `:757`, `:808`, `:860` | document delete
and restore | rare | #21552 kept these |
| `task-watchdogs.ts:854`, `:1841` | watchdog changes | rare | #21552
kept these |
| `system-recovery.ts:17`, `:70`, `task-retry.ts:294`,
`workspace-recovery-capture-policy.ts:59`,
`workspace-recovery-stop-proof.ts:122` | system and workspace recovery |
per recovery | not shown to stay in one graph |
| `agent-conversations.ts:607`,
`chat-channels/telegram/attachment-failure.ts:83`,
`native-runtime/chat-attachment-reuse.ts:1754`,
`native-runner-file-handoff.ts:1317` | conversation and attachment paths
| per failure | not shown to stay in one graph |
| `wake-queue/adapters/postgres.ts:376`, `:410`, `:555`, `:1169`,
`routes/issues.ts:14552`, `issues.ts:12070`, `:12171`,
`heartbeat/run-control.ts:1128`, `:1770`, `projects.ts:100`, `:1051`,
`environments.ts:1617` (`notifyChatInboundWakeNotices` without a task) |
saved notice and wake changes | per manual wake or project change |
company admission state; not reached by run activity in the fixture |
| `heartbeat/run-state.ts:1746`,
`native-harness-backup-expiration.ts:356`, `:526`,
`explicit-native-continuation.ts:705` | session goal clear, backup
expiry, explicit continuation | rare | no run id at the call site |
| `fast-responses.ts:201`, `:559`, `:672`, `:1053`,
`chat-github-response-comments.ts:62`, `:229`, `:440`,
`chat-github-reviews.ts:1063` (`notifyChatResponseAvailability` with a
delivery only) | chat response availability | per chat response | no
task at the call site; not measured here (chat paths are not in the
fixture) |

Per-run-event path: appending run events, saving native checkpoints,
renewing leases and streaming deltas publish no recovery task-queue
hint. `heartbeat-run-events.ts` reaches only live-run recovery queues
(`native.process_*`, `turn.*` events), chat progress, response
presentation, and `notifyNativeStatusReassessment` for control-plane
evidence events only. Checkpoints and lease renewals reach
`notifyLiveRunRecovery`, which does not include the task queues. A run's
start and finish publish its graph hint from several writers (about 20
task-queue signals per legacy run in the fixture, for example
`notifyChatRunMilestones` in `run-identity.ts` activation,
`claimQueuedRun`, `run-completion.ts`, `setWakeupStatus`). They coalesce
into about 3 checks of the run's own task per run. That is not one hint,
but it is bounded by the run's graph. This pull request does not change
it.

## Verification

- Activity fixture (`issue-recovery-activity-hints.test.ts`), checks per
second by queue and reason, and statements per second. Each workload
minute is exactly 60 s of fake-clock time, and the rates use the elapsed
fake-clock time. "Before" is this branch's test against the
`origin/master` service files. Both versions did the same work (7
runs/min: 150 runs started, 143 to 145 succeeded, 133 to 135 agent
comments, about 1,280 run events; 50 runs/min: 629 to 630 started, 620
to 622 succeeded, about 590 agent comments, about 5,550 run events). The
test now asserts this workload and zero agent write failures before it
accepts the counts.

  | | master | this branch |
  | --- | --- | --- |
| 7 runs/min + 1 taskless run/min (600 s), stranded checks/s | 18.30
company + 0.07 hint | 0.91 hint + 0.06 backstop, 0 company |
  | 7 runs/min, statements/s in stranded checks | 343.9 | 7.2 |
  | 7 runs/min, all statements/s | 577.5 | 220.4 |
  | 7 runs/min, company-only task-queue signals | 710 | 0 |
| 50 runs/min + 5 taskless runs/min (360 s), stranded checks/s | 16.86
company + 0.31 hint | 1.05 hint, 0 company |
  | 50 runs/min, statements/s in stranded checks | 314.0 | 7.4 |
  | 50 runs/min, all statements/s | 1,717.6 | 1,387.5 |
  | 50 runs/min, company-only task-queue signals | 2,864 | 0 |

"Statements in stranded checks" counts statements whose stack includes
`reconcileStrandedAssignedIssues`. The other statements are the runs
themselves. Recovery load queries (graph lookups and hint loads) are
12.5 to 17.2/s at 7 runs and 69 to 72/s at 50 runs in both versions.
`stale-issue-locks` saved deadlines check about 1 to 1.6 tasks/s in both
versions.
- Without the taskless runs, master already showed the backstop degrade
under activity: with 7 runs a minute, 2.46 backstop checks/s, because
every run writes its agent's row and marks all of that agent's tasks
changed. With this branch: 0.27/s.
- New tests in `issue-recovery-deadlines.test.ts`, red on
`origin/master` service files and green here:
- "re-checks a run's own tasks for its settlement, lease and dispatch
writers, and nothing for a run without a task": for `notifyRunDispatch`,
`notifyExecutionRunWork`, `notifyExternalConversationSettlement`,
`notifyChatInboundWakeNotices` and
`promoteDeferredWorkIntents(lock_released)` with a run id, the run's
graph (task, parent, dependent) is checked within one pass and none of
300 backlog tasks is. For a run without a task, no task is checked. A
comment checks only its task in `stranded-assignments` and
`assigned-work-invariant`.
- "counts an agent's own run heartbeat as no change to its other tasks,
and a resume as a change".
- "resolves a run without a task to no tasks and a missing task to the
whole company".
- Unchanged and passing: the #21729 pickup tests in
`issue-recovery-idle-backstop.test.ts` (finished run, user comment,
raised budget, resumed agent within a minute; missed hint within the
backstop bound; outside change within the hourly pass), the #21552 tests
in `issue-recovery-deadlines.test.ts` (including "keeps the company pass
for hold releases that can admit any task" and "falls back to the
company pass when a run affects more tasks than a hint names").
- Regression, embedded PostgreSQL: `issue-recovery-deadlines`,
`issue-recovery-work-debug-attribution`, `issue-recovery-signal-fanout`,
`issue-recovery-work-lifecycle` (42 tests);
`native-run-projection-work`, `native-recovery-notice-work`,
`muse-terminal-settlement`, `heartbeat-held-comment-wake`,
`superseded-workspace-export-recovery`,
`native-recovery-preparation-failure`, `upgrade-readiness-post-result`,
`chat-wake-conversation-batch`, `environment-service`,
`retained-native-session-cleanup-work`, `slack-conversation-lifecycle`,
`workspace-recovery-dependency-finalize`,
`native-failed-run-owner-release`, `chat-run-milestone-work` (462
tests).
- `tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- **Recovery time bounds. The owner must see this.** No hint-driven
pickup becomes slower for a task in the run's graph. Two missed-hint
bounds change:
- An agent change that sends no hint and is followed by a run heartbeat
of the same agent inside the 5-minute window (for example a resume whose
hint was lost, after which the agent ran another task) is now found by
the hourly full pass, not the 5-minute changed-tasks pass. A normal
resume sends `pause_lifted` and is checked at once (tested).
- A task outside the run's graph whose recovery depended on the end of a
taskless run, or on the native finish writers above, is now found by its
own hint, the 5-minute changed-tasks pass (its own rows), or the hourly
full pass. Before, any run finish re-checked the whole company. I found
no recovery decision that waits on another run of the agent: concurrency
waits are queued runs that dispatch resumes.
- A run-only hint for a run row that does not exist resolves to no tasks
(it was the whole company). A run row is not deleted while its tasks are
open, except by company deletion, which sends the company hint.
- Native finish writers now send run-scoped hints to the live-run
recovery queues too (`notifyLiveRunRecovery`,
`notifyLiveResourceRecovery`, attention retention). The legacy run paths
already send these with the run id.
- The fixture uses process-adapter runs, not native runs. The native
writers are narrowed by code reading. Their effect on the hosted staging
instance is not verified.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 10:47:34 -06:00

895 lines
47 KiB
TypeScript

import { notifyScheduledRetryDispatch } from "../run-dispatch-notifications.js";
import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js";
import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js";
import { notifyExecutionRunWork } from "../execution-control-notifications.js";
import { logger } from "../../middleware/logger.js";
import { createHash, randomUUID } from "node:crypto";
import { stat } from "node:fs/promises";
import { and, asc, eq, gt, inArray, isNull, or, sql } from "drizzle-orm";
import {
agentWakeupRequests,
environmentLeases,
executionWorkspaces,
executionWorkspaceSnapshots,
heartbeatRunEvents,
heartbeatRuns,
issues,
issueRecoveryActions,
nativeRunFinalizations,
toolInvocations,
workspaceOperations,
type Db,
} from "@paperclipai/db";
import { readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema } from "@paperclipai/shared";
import { decideNativeReplacement } from "./native-replacement-evidence.js";
import { issueService } from "../issues.js";
import { issueRecoveryActionService } from "../issue-recovery-actions.js";
import { buildExecutionContinuation } from "../execution-continuation.js";
import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js";
import { hasRemoteTerminationReceipt } from "../remote-execution-termination.js";
import { NATIVE_PROVIDER_DURABLE_EVENT_TYPES } from "./native-retired-backup-source.js";
import { completedWorkspaceRecoveryCondition } from "../workspace-recovery-receipt.js";
import { nativeRunHasNoWorkspaceAccess } from "../workspace-restore-recovery-state.js";
import { verifiedWorkspaceSnapshotReuse } from "../remote-workspace-snapshot-reuse-receipt.js";
import { RECOVERY_CONTINUATION_MAX_HOLD_MS } from "../recovery-continuation-pacing.js";
export const NATIVE_SAFE_REPLACEMENT_REASON = "native_safe_replacement";
const record = (v: unknown): Record<string, unknown> =>
v && typeof v === "object" && !Array.isArray(v)
? (v as Record<string, unknown>)
: {};
const processAlive = (pid: number | null) => {
if (pid === null) return false;
try {
process.kill(pid, 0);
return true;
} catch (error) {
return (error as NodeJS.ErrnoException).code !== "ESRCH";
}
};
type ReplacementDb = Db | Parameters<Parameters<Db["transaction"]>[0]>[0];
const listenerUnavailable = "local_runtime_api_listener_unavailable";
function historicalListenerGuardPending() {
const profile = heartbeatRuns.runnerProfileJson, detail = nativeRunFinalizations.failureDetail;
const providerEvents = sql.join([...NATIVE_PROVIDER_DURABLE_EVENT_TYPES].map(value => sql`${value}`), sql`, `);
return and(
eq(nativeRunFinalizations.failureCode, "native_session_interrupted"),
eq(heartbeatRuns.errorCode, "native_session_interrupted"), eq(heartbeatRuns.error, listenerUnavailable),
sql`${detail}->>'message' = ${listenerUnavailable}`,
sql`${detail}->>'originalFailureCode' = 'native_session_interrupted'`,
sql`${detail}->>'recoveryMode' = 'ambiguous_state'`,
sql`${detail}->'providerSessionEstablished' = 'false'::jsonb`,
sql`${detail}->'providerEventsExist' = 'false'::jsonb`,
sql`${detail}->'checkpointExists' = 'false'::jsonb`,
eq(nativeRunFinalizations.attempt, 1),
isNull(nativeRunFinalizations.leaseOwner), isNull(nativeRunFinalizations.leaseExpiresAt),
isNull(nativeRunFinalizations.assessmentId), isNull(nativeRunFinalizations.decisionId),
isNull(heartbeatRuns.processPid), isNull(heartbeatRuns.processGroupId),
isNull(heartbeatRuns.sessionIdBefore), isNull(heartbeatRuns.sessionIdAfter), isNull(heartbeatRuns.retryOfRunId),
sql`not (coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) ?| array['cancellation', 'nativeCancellation'])`,
sql`coalesce(${profile}->'sessionCheckpoint', 'null'::jsonb) = 'null'::jsonb`,
sql`coalesce(${profile}->'nativeProviderPackAdmission', 'null'::jsonb) = 'null'::jsonb`,
sql`coalesce(${profile}#>>'{nativeProviderPackAdmissionState,state}', 'pending') = 'pending'`,
sql`${profile}#>>'{nativeExecutionInput,binding,runId}' = ${heartbeatRuns.id}::text`,
sql`${profile}#>>'{nativeExecutionInput,binding,companyId}' = ${heartbeatRuns.companyId}::text`,
sql`${profile}#>>'{nativeExecutionInput,binding,agentId}' = ${heartbeatRuns.agentId}::text`,
sql`${profile}#>>'{nativeExecutionInput,binding,issueId}' = ${nativeRunFinalizations.issueId}::text`,
sql`${profile}#>>'{nativeExecutionInput,session,normalizedSessionId}' = ${heartbeatRuns.nativeSessionId}::text`,
sql`not exists (select 1 from ${heartbeatRuns} prior where prior.company_id = ${heartbeatRuns.companyId}
and prior.native_session_id = ${heartbeatRuns.nativeSessionId} and prior.id <> ${heartbeatRuns.id})`,
sql`not exists (select 1 from ${heartbeatRunEvents} event where event.company_id = ${heartbeatRuns.companyId}
and event.run_id = ${heartbeatRuns.id} and (event.source_event_id is not null
or event.event_type in (${providerEvents}) or event.event_type ~ '^(turn|item|tool|semantic_tool|run[.]result)[.]'))`,
sql`not exists (select 1 from ${toolInvocations} invocation where invocation.company_id = ${heartbeatRuns.companyId}
and invocation.run_id = ${heartbeatRuns.id})`,
sql`not exists (select 1 from ${issueRecoveryActions}
where ${issueRecoveryActions.companyId} = ${heartbeatRuns.companyId}
and ${issueRecoveryActions.sourceIssueId} = ${nativeRunFinalizations.issueId}
and ${issueRecoveryActions.cause} = 'native_continuation_requires_reconciliation'
and ${issueRecoveryActions.evidence}->>'runId' = ${heartbeatRuns.id}::text
and ${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'
and not coalesce((${completedWorkspaceRecoveryCondition(heartbeatRuns.id)}), false))`,
sql`exists (select 1 from ${issues} task where task.company_id = ${heartbeatRuns.companyId}
and task.id = ${nativeRunFinalizations.issueId} and task.assignee_agent_id = ${heartbeatRuns.agentId}
and task.status in ('todo', 'in_progress', 'in_review', 'blocked')
and (task.status <> 'blocked' or exists (select 1 from ${issueRecoveryActions} hold
where hold.company_id = task.company_id and hold.source_issue_id = task.id and hold.kind = 'active_run_watchdog'
and hold.cause = ${nativeRunFinalizations.failureCode} and hold.evidence->>'runId' = ${heartbeatRuns.id}::text
and hold.evidence#>>'{nativeFailureBlock,runId}' = ${heartbeatRuns.id}::text
and hold.evidence#>'{nativeFailureBlock,statusVersion}' = to_jsonb(task.status_version))))`,
);
}
async function historicalListenerGuardEvidence(db: ReplacementDb, run: typeof heartbeatRuns.$inferSelect) {
const [evidence] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
.innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)).where(and(
eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.id, run.id), historicalListenerGuardPending(),
)).limit(1);
return Boolean(evidence);
}
/**
* A native run that failed before its first coordinator claim. The claim is the
* only gate to provider dispatch: it moves the coordinator past attempt 0 and
* records a controller generation. A run that never claimed started no provider
* process, turn or tool, so a fresh session can continue the task once its
* prepared files are saved. Any claim, process, provider or tool event, user
* Stop, or recorded provider work keeps today's reconciliation. One automatic
* continuation per chain: a continuation that itself fails is not continued.
*/
export const NATIVE_PRE_CLAIM_FAILURE_CODE = "native_continuation_requires_reconciliation";
function preClaimFailurePending() {
const providerEvents = sql.join([...NATIVE_PROVIDER_DURABLE_EVENT_TYPES].map(value => sql`${value}`), sql`, `);
return and(
eq(nativeRunFinalizations.failureCode, NATIVE_PRE_CLAIM_FAILURE_CODE),
eq(heartbeatRuns.runtimeMode, "native"),
eq(nativeRunFinalizations.attempt, 0), eq(nativeRunFinalizations.controllerGeneration, 0),
isNull(nativeRunFinalizations.leaseOwner), isNull(nativeRunFinalizations.leaseExpiresAt),
isNull(nativeRunFinalizations.assessmentId), isNull(nativeRunFinalizations.decisionId),
isNull(heartbeatRuns.processPid), isNull(heartbeatRuns.processGroupId),
isNull(heartbeatRuns.processStartedAt), isNull(heartbeatRuns.sessionIdAfter),
sql`${heartbeatRuns.nativeIssueId} = ${nativeRunFinalizations.issueId}`,
sql`coalesce(${heartbeatRuns.resultJson}#>'{executionRecovery,providerWorkStarted}', 'false'::jsonb) = 'false'::jsonb`,
sql`not (coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) ?| array['cancellation', 'nativeCancellation', 'executionCancellation'])`,
sql`coalesce(${heartbeatRuns.contextSnapshot}->>'nativePreClaimContinuationOf', '') = ''`,
sql`not exists (select 1 from ${heartbeatRunEvents} event where event.company_id = ${heartbeatRuns.companyId}
and event.run_id = ${heartbeatRuns.id} and (event.source_event_id is not null
or event.event_type in (${providerEvents}) or event.event_type ~ '^(turn|item|tool|semantic_tool|run[.]result)[.]'))`,
sql`not exists (select 1 from ${toolInvocations} invocation where invocation.company_id = ${heartbeatRuns.companyId}
and invocation.run_id = ${heartbeatRuns.id})`,
// Saved files come first. A no-replay hold settles only with its completed recovery.
sql`not exists (select 1 from ${issueRecoveryActions}
where ${issueRecoveryActions.companyId} = ${heartbeatRuns.companyId}
and ${issueRecoveryActions.sourceIssueId} = ${nativeRunFinalizations.issueId}
and ${issueRecoveryActions.evidence}->>'runId' = ${heartbeatRuns.id}::text
and (${issueRecoveryActions.evidence} ? 'explicitUserContinuation'
or (${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'
and not coalesce((${completedWorkspaceRecoveryCondition(heartbeatRuns.id)}), false))))`,
// The task has not moved on: a later run (such as a user's message) owns it.
sql`not exists (select 1 from ${heartbeatRuns} later where later.company_id = ${heartbeatRuns.companyId}
and (later.context_snapshot ->> 'issueId') = ${nativeRunFinalizations.issueId}::text
and later.created_at > ${heartbeatRuns.createdAt} and later.id <> ${heartbeatRuns.id})`,
sql`exists (select 1 from ${issues} task where task.company_id = ${heartbeatRuns.companyId}
and task.id = ${nativeRunFinalizations.issueId} and task.assignee_agent_id = ${heartbeatRuns.agentId}
and task.status in ('todo', 'in_progress', 'in_review', 'blocked')
and (task.status <> 'blocked' or exists (select 1 from ${issueRecoveryActions} hold
where hold.company_id = task.company_id and hold.source_issue_id = task.id and hold.kind = 'active_run_watchdog'
and hold.cause = ${nativeRunFinalizations.failureCode} and hold.evidence->>'runId' = ${heartbeatRuns.id}::text
and hold.evidence#>>'{nativeFailureBlock,runId}' = ${heartbeatRuns.id}::text
and hold.evidence#>'{nativeFailureBlock,statusVersion}' = to_jsonb(task.status_version))))`,
);
}
async function preClaimFailureEvidence(db: ReplacementDb, run: typeof heartbeatRuns.$inferSelect) {
const [evidence] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
.innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)).where(and(
eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.id, run.id), preClaimFailurePending(),
)).limit(1);
return Boolean(evidence);
}
/** A saved remote source is proved by its current ledger, never a controller path. */
export async function remoteWorkspaceReplacementEvidence(
db: ReplacementDb,
run: typeof heartbeatRuns.$inferSelect,
issueId: string,
leases: Array<typeof environmentLeases.$inferSelect>,
lock = false,
) {
const [task] = await db.select({ workspaceId: issues.executionWorkspaceId }).from(issues).where(and(
eq(issues.companyId, run.companyId), eq(issues.id, issueId),
));
const query = task?.workspaceId ? db.select().from(executionWorkspaces).where(and(
eq(executionWorkspaces.companyId, run.companyId), eq(executionWorkspaces.id, task.workspaceId),
)) : null;
const [workspace] = query ? await (lock ? query.for("update") : query) : [];
const authority = readRemoteWorkspaceAuthority(workspace?.metadata);
if (run.runnerProfileJson?.remoteWorkspace == null && !authority) return null;
if (lock) {
const [currentRun] = await db.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.id, run.id),
)).for("update");
if (!currentRun || currentRun.status !== "failed") return { workspacePreserved: false, providerStopped: false };
run = currentRun;
leases = await db.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id),
)).for("update");
}
const providerStopped = leases.length > 0 && leases.every(hasRemoteTerminationReceipt);
const denied = { workspacePreserved: false, providerStopped };
const parsed = remoteWorkspaceRunReferenceSchema.safeParse(run.runnerProfileJson?.remoteWorkspace);
if (!workspace || !authority || !parsed.success || parsed.data.state !== "saved" || !parsed.data.snapshotId
|| run.runnerProfileJson?.nativeWorkspaceSync != null || ["archived", "cleanup_failed"].includes(workspace.status)) return denied;
const reference = parsed.data;
const executionWorkspace = record(record(run.runnerProfileJson?.nativeExecutionInput).workspace);
if (workspace.id !== reference.workspaceId || executionWorkspace.cwd !== reference.remoteCwd
|| authority.dirty || authority.writerRunId || authority.epoch !== reference.epoch
|| authority.snapshotId !== reference.snapshotId) return denied;
const lease = leases.find(row => row.id === reference.leaseId && row.providerLeaseId === reference.providerLeaseId
&& (!row.executionWorkspaceId || row.executionWorkspaceId === workspace.id)
&& row.metadata?.remoteWorkspaceId === workspace.id && row.metadata?.remoteCwd === reference.remoteCwd);
if (!lease || !providerStopped) return denied;
if (authority.location ? authority.location.leaseId !== lease.id || authority.location.environmentId !== lease.environmentId
|| authority.location.providerLeaseId !== reference.providerLeaseId || authority.location.remoteCwd !== reference.remoteCwd
: record(lease.metadata?.remoteExecutionTermination).state !== "destroyed") return denied;
const snapshotQuery = db.select().from(executionWorkspaceSnapshots).where(and(
eq(executionWorkspaceSnapshots.companyId, run.companyId), eq(executionWorkspaceSnapshots.executionWorkspaceId, workspace.id),
eq(executionWorkspaceSnapshots.id, reference.snapshotId!), eq(executionWorkspaceSnapshots.status, "committed"),
));
const [snapshot] = await (lock ? snapshotQuery.for("update") : snapshotQuery);
if (!snapshot?.sha256 || !snapshot.objectKey || !snapshot.key || !snapshot.manifestDigest || !snapshot.byteSize) return denied;
const [operation] = snapshot.sourceRunId === run.id && snapshot.epoch === reference.epoch
? await db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
eq(workspaceOperations.id, snapshot.sourceOperationId), eq(workspaceOperations.companyId, run.companyId),
eq(workspaceOperations.executionWorkspaceId, workspace.id), eq(workspaceOperations.heartbeatRunId, run.id),
eq(workspaceOperations.issueId, issueId), eq(workspaceOperations.phase, "remote_workspace_snapshot"),
eq(workspaceOperations.status, "succeeded"),
sql`${workspaceOperations.metadata}->>'boundary' = 'final'`,
sql`${workspaceOperations.metadata}->>'epoch' = ${String(reference.epoch)}`,
sql`${workspaceOperations.metadata}->>'snapshotId' = ${snapshot.id}`,
sql`${workspaceOperations.metadata}->>'sha256' = ${snapshot.sha256}`,
)) : [];
const reused = operation ? null : await verifiedWorkspaceSnapshotReuse(db, {
companyId: run.companyId, issueId, runId: run.id, reference, snapshot });
return { workspacePreserved: Boolean(operation || reused), providerStopped };
}
export function nativeReplacementPending() {
return and(
eq(heartbeatRuns.status, "failed"),
eq(nativeRunFinalizations.phase, "terminal_failure"),
or(inArray(
nativeRunFinalizations.failureCode,
["native_provider_terminal_failed", "native_session_cleanup_quarantined", "provider_transport_failed"],
), historicalListenerGuardPending(), preClaimFailurePending()),
isNull(nativeRunFinalizations.resultId),
sql`coalesce(${nativeRunFinalizations.failureDetail}->>'successorRunId', '') = ''`,
sql`coalesce(${nativeRunFinalizations.failureDetail}->>'replacementDenied', '') = ''`,
);
}
export const NATIVE_OWNERSHIP_PROOF_BOUND_MS = RECOVERY_CONTINUATION_MAX_HOLD_MS;
const OWNERSHIP_UNVERIFIED = "provider_ownership_unverified";
const ownershipProofDeadline = () => sql`(coalesce(${heartbeatRuns.finishedAt}, ${nativeRunFinalizations.updatedAt})
+ ${NATIVE_OWNERSHIP_PROOF_BOUND_MS}::integer * interval '1 millisecond')`;
const ownershipProofWaiting = () => and(
eq(nativeRunFinalizations.failureCode, "native_provider_terminal_failed"), isNull(nativeRunFinalizations.leaseOwner));
async function ownershipProofExpired(db: ReplacementDb, run: typeof heartbeatRuns.$inferSelect) {
const [row] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
.innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)).where(and(
eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.id, run.id), nativeReplacementPending(),
ownershipProofWaiting(), sql`${ownershipProofDeadline()} <= now()`,
)).limit(1);
return Boolean(row);
}
async function markOwnershipProofWaiting(db: ReplacementDb, run: typeof heartbeatRuns.$inferSelect) {
await db.update(nativeRunFinalizations).set({
failureDetail: sql`coalesce(${nativeRunFinalizations.failureDetail}, '{}'::jsonb)
|| jsonb_build_object('ownershipUnverifiedAt', now())`,
}).where(and(eq(nativeRunFinalizations.companyId, run.companyId), eq(nativeRunFinalizations.runId, run.id),
ownershipProofWaiting(), sql`not (coalesce(${nativeRunFinalizations.failureDetail}, '{}'::jsonb) ? 'ownershipUnverifiedAt')`));
}
async function releaseOwnershipProofWake(db: ReplacementDb, run: typeof heartbeatRuns.$inferSelect) {
await db.update(nativeRunFinalizations).set({
failureDetail: sql`${nativeRunFinalizations.failureDetail} - 'ownershipUnverifiedAt'`,
}).where(and(eq(nativeRunFinalizations.companyId, run.companyId), eq(nativeRunFinalizations.runId, run.id),
sql`${nativeRunFinalizations.failureDetail} ? 'ownershipUnverifiedAt'`));
}
export async function nextNativeOwnershipProofDeadlineAt(db: Db): Promise<number | null> {
const [row] = await db.select({ at: sql<number | null>`(extract(epoch from min(${ownershipProofDeadline()})) * 1000)::float8` })
.from(heartbeatRuns).innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id))
.where(and(nativeReplacementPending(), ownershipProofWaiting(),
sql`${nativeRunFinalizations.failureDetail} ? 'ownershipUnverifiedAt'`));
return row?.at == null ? null : Math.ceil(Number(row.at));
}
const REPLACEMENT_SCAN_LIMIT = 100;
const replacementDiscovery = new WeakMap<object, { after: string | undefined; complete: boolean; generation: number; rescan: boolean }>();
function replacementDiscoveryState(db: Db) {
let state = replacementDiscovery.get(db);
if (!state) { state = { after: undefined, complete: false, generation: 0, rescan: false }; replacementDiscovery.set(db, state); }
return state;
}
export function invalidateNativeReplacementDiscovery(db: Db) {
const state = replacementDiscoveryState(db);
state.complete = false; state.rescan = true; state.generation++;
}
async function replacementHistoryReady(db: Db, run: typeof heartbeatRuns.$inferSelect, issueId: string, retryDatabaseErrors = false) {
try {
const continuation = await buildExecutionContinuation({ db, companyId: run.companyId, issueId,
agentId: run.agentId, context: record(run.contextSnapshot), summary: null, exposeLowTrustRaw: false });
return continuation.unresolvedInteractionIds.length === 0;
} catch (error) {
let cause: unknown = error;
for (let depth = 0; retryDatabaseErrors && depth < 4 && cause && typeof cause === "object"; depth++) {
const failure = cause as { code?: unknown; cause?: unknown };
if (typeof failure.code === "string" && /^[A-Z0-9_]+$/.test(failure.code)) throw error;
cause = failure.cause;
}
return null;
}
}
async function discoverNativeReplacements(db: Db, after: string | undefined, limit = 25) {
const candidates: Array<{ run: typeof heartbeatRuns.$inferSelect; coordinator: typeof nativeRunFinalizations.$inferSelect }> = [];
let scanned = 0;
for (;;) {
const rows = await db.select({ run: heartbeatRuns, coordinator: nativeRunFinalizations }).from(heartbeatRuns)
.innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id))
.where(and(nativeReplacementPending(), after ? gt(heartbeatRuns.id, after) : undefined))
.orderBy(asc(heartbeatRuns.id)).limit(Math.min(25, REPLACEMENT_SCAN_LIMIT - scanned));
if (!rows.length) return { candidates, scanned, after: undefined };
for (const row of rows) {
after = row.run.id; scanned++;
let ready = true;
if (row.coordinator.failureCode === "native_session_interrupted") {
ready = false;
if (row.run.runnerProfileJson?.remoteWorkspace != null) {
const leases = await db.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, row.run.companyId), eq(environmentLeases.heartbeatRunId, row.run.id),
));
if (leases.every(lease => lease.releasedAt !== null)) {
const evidence = await remoteWorkspaceReplacementEvidence(db, row.run, row.coordinator.issueId, leases);
ready = Boolean(evidence?.workspacePreserved && evidence.providerStopped)
&& await replacementHistoryReady(db, row.run, row.coordinator.issueId, true) === true;
}
}
}
if (row.coordinator.failureCode === NATIVE_PRE_CLAIM_FAILURE_CODE)
ready = await preClaimContinuationReady(db, row.run, row.coordinator.issueId);
if (ready) candidates.push(row);
if (candidates.length >= limit || scanned >= REPLACEMENT_SCAN_LIMIT) return { candidates, scanned, after };
}
if (rows.length < 25) return { candidates, scanned, after: undefined };
}
}
/** Saved files and a stopped sandbox, or a run that had no workspace to save. */
async function preClaimContinuationReady(db: Db, run: typeof heartbeatRuns.$inferSelect, issueId: string) {
const leases = await db.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id),
));
if (leases.some(lease => lease.releasedAt === null)) return false;
const evidence = await remoteWorkspaceReplacementEvidence(db, run, issueId, leases);
const workspaceReady = evidence ? evidence.workspacePreserved && evidence.providerStopped
: run.runnerProfileJson?.nativeWorkspaceSync == null && nativeRunHasNoWorkspaceAccess(run);
return workspaceReady && await replacementHistoryReady(db, run, issueId, true) === true;
}
function settleReplacementDiscovery(state: ReturnType<typeof replacementDiscoveryState>, after: string | undefined, generation: number) {
const rescan = state.rescan || generation !== state.generation;
state.after = after; state.complete = !after && !rescan;
state.rescan = Boolean(after && rescan);
}
export async function hasPendingNativeReplacements(db: Db) {
const state = replacementDiscoveryState(db);
if (state.complete) return false;
if (state.after) return true;
state.rescan = false;
const generation = state.generation;
const result = await discoverNativeReplacements(db, undefined, 1);
if (result.candidates.length) return true;
settleReplacementDiscovery(state, result.after, generation);
return !state.complete;
}
/** Retryable outbox: terminal_failure plus a new-format failure code is the durable candidate. */
export async function reconcileSafeNativeReplacements(
db: Db,
now = new Date(),
options: {
verifyStoppedSession?: (run: typeof heartbeatRuns.$inferSelect) => Promise<{
evidence: Record<string, unknown>;
retire: () => boolean;
} | null>;
/** Test fault injection at durability boundaries; never exposed by an API. */
failpoint?: (phase: "successor_inserted" | "lineage_committed") => void;
} = {},
) {
const discovery = replacementDiscoveryState(db);
if (discovery.complete) discovery.after = undefined;
discovery.complete = false;
if (!discovery.after) discovery.rescan = false;
const generation = discovery.generation;
const selection = await discoverNativeReplacements(db, discovery.after);
const candidates = selection.candidates;
let scheduled = 0, pendingFailures = 0;
for (const { run, coordinator } of candidates) {
try {
if (
coordinator.failureDetail?.successorRunId ||
coordinator.failureDetail?.replacementDenied
)
continue;
const execution = record(
record(run.runnerProfileJson).nativeExecutionInput,
);
const workspace = record(execution.workspace);
const leases = await db
.select()
.from(environmentLeases)
.where(
and(
eq(environmentLeases.companyId, run.companyId),
eq(environmentLeases.heartbeatRunId, run.id),
),
);
let overdue: boolean | undefined;
const ownershipOverdue = async () => overdue ??= await ownershipProofExpired(db, run);
const leaseHeld = leases.some((lease) => lease.releasedAt === null);
// Teardown is still in progress. The next sweep rechecks its durable outcome.
if (leaseHeld && !await ownershipOverdue()) {
if (coordinator.failureCode === "native_provider_terminal_failed") await markOwnershipProofWaiting(db, run);
continue;
}
const historicalListener = coordinator.failureCode === "native_session_interrupted";
if (historicalListener && !await historicalListenerGuardEvidence(db, run)) continue;
const preClaim = coordinator.failureCode === NATIVE_PRE_CLAIM_FAILURE_CODE;
if (preClaim && !await preClaimFailureEvidence(db, run)) continue;
const stoppedSession = coordinator.failureCode !== "native_provider_terminal_failed" && !historicalListener && !preClaim
? await options.verifyStoppedSession?.(run) ?? null : null;
// A transport label alone is not evidence. Keep inspecting these candidates
// as process cleanup and the final transcript become durable.
if (coordinator.failureCode !== "native_provider_terminal_failed" && !historicalListener && !preClaim && !stoppedSession) continue;
const invocations = await db
.select()
.from(toolInvocations)
.where(
and(
eq(toolInvocations.companyId, run.companyId),
eq(toolInvocations.runId, run.id),
),
);
const events = await db
.select({
eventType: heartbeatRunEvents.eventType,
payload: heartbeatRunEvents.payload,
})
.from(heartbeatRunEvents)
.where(
and(
eq(heartbeatRunEvents.companyId, run.companyId),
eq(heartbeatRunEvents.runId, run.id),
inArray(heartbeatRunEvents.eventType, [
"harness.diagnostic",
"tool.execution.started",
"workspace.changed",
"workspace.change.updated",
"delegation.started",
"terminal.input.sent",
"artifact.generated",
"item.started",
]),
),
);
const safeControlReads = new Set([
"connections_search",
"connection_request",
"paperclip_get_agent",
"paperclip_get_issue",
"paperclip_list_issues",
"paperclip_read_document",
]);
const uncertainProviderActions = events.flatMap((event) => {
const envelope = record(event.payload);
const p = envelope.prpEvent
? record(record(envelope.prpEvent).payload)
: envelope;
if (event.eventType === "harness.diagnostic")
return p.classification === "descendant"
? [`descendant ${String(p.receivedThreadId ?? "unknown")}`]
: [];
if (
[
"workspace.changed",
"workspace.change.updated",
"delegation.started",
"terminal.input.sent",
"artifact.generated",
].includes(event.eventType)
)
return [event.eventType];
if (event.eventType === "tool.execution.started") {
const name = typeof p.name === "string" ? p.name : "unknown tool";
const completedTaskControlCallIds = stoppedSession?.evidence.completedTaskControlCallIds;
if (name === "paperclip_finish" && Array.isArray(completedTaskControlCallIds) &&
completedTaskControlCallIds.includes(p.executionId)) return [];
const receiptedRead = invocations.some(
(row) =>
(row.id === p.executionId ||
row.correlationId === p.executionId) &&
row.toolName === name &&
row.riskLevel === "read" &&
row.status === "succeeded" &&
row.resultHash,
);
return p.transport === "process" ||
(!receiptedRead && !safeControlReads.has(name))
? [String(p.executionId ?? name)]
: [];
}
return [
"command_execution",
"commandExecution",
"file_change",
"fileChange",
"collabAgentToolCall",
"subAgentActivity",
].includes(String(p.kind ?? p.type))
? [String(p.kind ?? p.type)]
: [];
});
const history = await replacementHistoryReady(db, run, coordinator.issueId);
if (history === false) {
if (await ownershipOverdue()) await releaseOwnershipProofWake(db, run);
continue;
}
const historyComplete = history === true;
const remoteWorkspace = await remoteWorkspaceReplacementEvidence(db, run, coordinator.issueId, leases);
if (historicalListener && (!remoteWorkspace?.workspacePreserved || !remoteWorkspace.providerStopped || invocations.length)) continue;
const noWorkspaceToSave = preClaim && !remoteWorkspace
&& run.runnerProfileJson?.nativeWorkspaceSync == null && nativeRunHasNoWorkspaceAccess(run);
if (preClaim && (invocations.length || !noWorkspaceToSave
&& (!remoteWorkspace?.workspacePreserved || !remoteWorkspace.providerStopped))) continue;
const decision = decideNativeReplacement({
failedSession: true,
// A facade transport failure can conceal an authoritative protocol
// rejection. A stopped process and read receipts do not resolve that.
// Before its first claim the run could not dispatch provider work.
failureMeaningKnown:
preClaim || Boolean(stoppedSession) || (typeof coordinator.failureDetail?.originalFailureCode === "string" &&
![
"notification_transport_failed",
"provider_turn_failed",
"native_provider_terminal_failed",
].includes(coordinator.failureDetail.originalFailureCode)),
predecessorFenced:
coordinator.leaseOwner === null && run.status === "failed",
providerStopped:
!processAlive(run.processPid) &&
!processAlive(run.processGroupId ? -run.processGroupId : null) &&
(remoteWorkspace?.providerStopped ?? true) && !leaseHeld,
workspacePreserved: noWorkspaceToSave || (remoteWorkspace ? remoteWorkspace.workspacePreserved :
typeof workspace.cwd === "string" &&
(await stat(workspace.cwd).then(
(s) => s.isDirectory(),
() => false,
))),
historyComplete,
effectInventoryComplete:
preClaim || Boolean(stoppedSession) || (record(run.runnerProfileJson).recoveryEventInventoryVersion === 1 &&
record(execution.provider).kind === "codex"),
attempts: coordinator.attempt,
invocations,
apiReceipts: record(record(run.resultJson).apiToolReceipts),
uncertainProviderActions,
});
if (!decision.allowed) {
// Lack of containment can be temporary; do not prevent the next proof.
const ownershipBound = decision.cause === OWNERSHIP_UNVERIFIED;
if (ownershipBound && !await ownershipOverdue()) {
await markOwnershipProofWaiting(db, run);
continue;
}
const nextAction = ownershipBound
? "The previous session could not be confirmed stopped, so the task continues in a fresh run with its own workspace."
: decision.nextAction;
await db.transaction(async (tx) => {
await tx.execute(
sql`select set_config('statement_timeout', '15000', true), set_config('lock_timeout', '1000', true)`,
);
const [task] = await tx
.select()
.from(issues)
.where(
and(
eq(issues.id, coordinator.issueId),
eq(issues.companyId, run.companyId),
),
)
.for("update");
const [current] = await tx
.select()
.from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, run.id))
.for("update");
if (
!current ||
current.phase !== "terminal_failure" ||
current.failureDetail?.successorRunId ||
current.failureDetail?.replacementDenied ||
(ownershipBound && !await ownershipProofExpired(tx, run))
)
return;
await notifyExecutionRunWork(tx, { companyId: run.companyId, runId: run.id });
await tx
.update(nativeRunFinalizations)
.set({
failureDetail: {
...current.failureDetail,
replacementDenied: decision.cause,
nextAction,
},
updatedAt: now,
})
.where(eq(nativeRunFinalizations.runId, run.id));
if (
!task ||
task.assigneeAgentId !== run.agentId ||
["done", "cancelled"].includes(task.status)
)
return;
await issueRecoveryActionService(
tx as unknown as Db,
).upsertSourceScoped({
companyId: run.companyId,
sourceIssueId: coordinator.issueId,
kind: "active_run_watchdog",
ownerType: "board",
ownerAgentId: null,
returnOwnerAgentId: run.agentId,
cause: decision.cause,
fingerprint: `native-replacement:${run.id}`,
evidence: { runId: run.id, attempts: coordinator.attempt },
nextAction,
wakePolicy: null,
maxAttempts: 3,
supersedeOnIdentityChange: true,
});
});
continue;
}
const created = await db.transaction(async (tx) => {
await tx.execute(
sql`select set_config('statement_timeout', '15000', true), set_config('lock_timeout', '1000', true)`,
);
const [task] = await tx
.select()
.from(issues)
.where(
and(
eq(issues.id, coordinator.issueId),
eq(issues.companyId, run.companyId),
),
)
.for("update");
const [current] = await tx
.select()
.from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, run.id))
.for("update");
if (
!task ||
task.assigneeAgentId !== run.agentId ||
!["todo", "in_progress", "in_review", "blocked"].includes(task.status) ||
(task.executionRunId !== null && task.executionRunId !== run.id) ||
(task.checkoutRunId !== null && task.checkoutRunId !== run.id) ||
!current ||
current.phase !== "terminal_failure" ||
current.leaseOwner !== null ||
current.failureCode !== coordinator.failureCode ||
current.failureDetail?.successorRunId ||
current.failureDetail?.replacementDenied ||
current.attempt >= 3
)
return false;
let historicalRun: typeof heartbeatRuns.$inferSelect | undefined;
if (historicalListener || preClaim) {
const [currentRun] = await tx.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId),
)).for("update");
historicalRun = currentRun;
}
if (task.status === "blocked") {
const failureHolds = await tx.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, run.companyId),
eq(issueRecoveryActions.sourceIssueId, task.id),
eq(issueRecoveryActions.kind, "active_run_watchdog"),
eq(issueRecoveryActions.cause, current.failureCode!),
sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`,
)).for("update");
const ownsBlock = failureHolds.some(hold => {
const receipt = record(hold.evidence.nativeFailureBlock);
return receipt.runId === run.id && receipt.statusVersion === task.statusVersion;
});
if (!ownsBlock) return false;
}
const preservationHolds = historicalListener ? await tx.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, run.companyId), eq(issueRecoveryActions.sourceIssueId, task.id),
eq(issueRecoveryActions.kind, "active_run_watchdog"),
eq(issueRecoveryActions.cause, "native_continuation_requires_reconciliation"),
sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`,
sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'`,
)).for("update") : [];
if (historicalListener && (!historicalRun || !await historicalListenerGuardEvidence(tx, historicalRun))) return false;
if (preClaim && (!historicalRun || !await preClaimFailureEvidence(tx, historicalRun))) return false;
if (remoteWorkspace) {
const currentWorkspace = await remoteWorkspaceReplacementEvidence(tx, run, coordinator.issueId, leases, true);
if (!currentWorkspace?.workspacePreserved || !currentWorkspace.providerStopped) return false;
}
if (stoppedSession) {
const [currentRun] = await tx.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId),
)).for("update");
if (!currentRun || currentRun.status !== "failed" || currentRun.runnerInstanceId !== run.runnerInstanceId ||
currentRun.nativeSessionId !== run.nativeSessionId || currentRun.processPid || currentRun.processGroupId) return false;
}
if (task.status === "blocked") {
// Restore only this failure's unchanged projection. The normal issue
// service still enforces dependency readiness and assignee eligibility.
await issueService(tx as unknown as Db).update(task.id, { status: "in_progress" }, tx);
}
if (stoppedSession) {
// If the last ownership proof changes, roll back the status restoration.
if (!stoppedSession.retire()) throw new Error("native_replacement_stopped_session_changed");
await appendHeartbeatRunEvent(tx as unknown as Db, {
companyId: run.companyId, runId: run.id, agentId: run.agentId,
eventType: "native.stopped_text_turn_verified", stream: "system", level: "info",
message: "The previous runner and provider stopped. The interrupted turn had no external actions; any completion bookkeeping has a verified receipt.",
payload: stoppedSession.evidence,
});
}
const successorRunId = randomUUID();
const dueAt = new Date(now.getTime() + 30_000);
const predecessorContext = { ...record(run.contextSnapshot) };
// History comes from the failed source run. Consumed wake fields must
// not grant this automatic retry fresh comment/resume authority.
for (const key of [
"explicitUserContinuation", "wakeCommentId", "wakeCommentIds", "commentId",
"commentIds", "latestCommentId", "resumeIntent", "followUpRequested",
"paperclipWake", "paperclipWakeComment", "paperclipTaskMarkdown", "paperclipTaskMarkdownCompact",
"paperclipTaskMarkdownAssignment", "paperclipTaskMarkdownAssignmentCompact", "paperclipTurnContext",
]) delete predecessorContext[key];
const context = {
...predecessorContext,
issueId: task.id,
retryOfRunId: run.id,
wakeReason: NATIVE_SAFE_REPLACEMENT_REASON,
retryReason: NATIVE_SAFE_REPLACEMENT_REASON,
forceFreshSession: true,
recoveryIncidentRootRunId:
record(run.contextSnapshot).recoveryIncidentRootRunId ?? run.id,
// Bounds pre-claim continuation to one automatic successor per chain.
...(preClaim ? { nativePreClaimContinuationOf: run.id } : {}),
};
const [wake] = await tx
.insert(agentWakeupRequests)
.values({
companyId: run.companyId,
agentId: run.agentId,
source: "automation",
triggerDetail: "system",
reason: NATIVE_SAFE_REPLACEMENT_REASON,
status: "queued",
payload: context,
requestedByActorType: "system",
idempotencyKey: `native-safe-replacement:${run.id}`,
})
.returning();
await notifyExecutionRunWork(tx, { companyId: run.companyId, runId: run.id });
await notifyScheduledRetryDispatch(tx, { companyId: run.companyId, runId: run.id });
await tx.insert(heartbeatRuns).values({
id: successorRunId,
companyId: run.companyId,
agentId: run.agentId,
invocationSource: "automation",
triggerDetail: "system",
status: "scheduled_retry",
executionStatusDeliveryId: randomUUID(),
wakeupRequestId: wake!.id,
responsibleUserId: run.responsibleUserId,
contextSnapshot: context,
retryOfRunId: run.id,
scheduledRetryAt: dueAt,
scheduledRetryAttempt: preClaim ? 1 : current.attempt,
scheduledRetryReason: NATIVE_SAFE_REPLACEMENT_REASON,
});
options.failpoint?.("successor_inserted");
await tx
.update(agentWakeupRequests)
.set({ runId: successorRunId })
.where(eq(agentWakeupRequests.id, wake!.id));
await tx
.update(nativeRunFinalizations)
.set({
failureDetail: {
...current.failureDetail,
...(stoppedSession ? { stoppedTextTurn: stoppedSession.evidence } : {}),
...(preClaim ? { preClaimContinuation: true } : {}),
successorRunId,
nextAction:
"Continue in the linked fresh provider session after the retry delay.",
},
nextAttemptAt: dueAt,
updatedAt: now,
})
.where(eq(nativeRunFinalizations.runId, run.id));
await issueRecoveryActionService(
tx as unknown as Db,
).resolveActiveForIssue({
companyId: run.companyId,
sourceIssueId: task.id,
kind: "active_run_watchdog",
fingerprint: createHash("sha256")
.update(`${run.id}:${current.failureCode}`)
.digest("hex"),
status: "resolved",
outcome: "handed_back",
resolutionNote: `Safe continuation is scheduled in run ${successorRunId}.`,
});
// A previous sweep can have resolved the UI bookkeeping while retaining
// an effective no-replay hold. Retire only this exact failure's hold in
// the same transaction as its verified successor; unrelated holds remain.
const holds = await tx.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, run.companyId),
eq(issueRecoveryActions.sourceIssueId, task.id),
eq(issueRecoveryActions.kind, "active_run_watchdog"),
eq(issueRecoveryActions.cause, current.failureCode!),
sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`,
)).for("update");
holds.push(...preservationHolds);
if (holds.some(hold => hold.status === "active" || hold.status === "escalated")) {
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: run.companyId, issueId: task.id });
}
for (const hold of holds) {
await tx.update(issueRecoveryActions).set({
status: "resolved", outcome: "handed_back", resolvedAt: now, updatedAt: now,
wakePolicy: null, monitorPolicy: null,
nextAction: `Safe continuation is scheduled in run ${successorRunId}.`,
resolutionNote: preClaim
? "The run stopped before it started any provider work, and its saved files are recovered. A fresh session continues the task."
: "Verified process termination and action receipts allow a fresh session to continue.",
evidence: { ...hold.evidence, verifiedReplacement: { successorRunId, recordedAt: now.toISOString() },
...(hold.evidence.automaticRecovery ? { automaticRecovery: {
...record(hold.evidence.automaticRecovery), replay: "verified_safe_replacement", successorRunId,
} } : {}),
},
}).where(eq(issueRecoveryActions.id, hold.id));
}
await tx
.update(heartbeatRuns)
.set({ executionStatusDeliveryId: randomUUID() })
.where(eq(heartbeatRuns.id, run.id));
await notifyNativeStatusReassessment(tx, { companyId: run.companyId, issueId: task.id });
// The successor gets task authority only through the existing admission gate.
await tx
.update(issues)
.set({ executionRunId: null, checkoutRunId: null, updatedAt: now })
.where(
and(
eq(issues.id, task.id),
eq(issues.assigneeAgentId, run.agentId),
),
);
return true;
});
if (created) {
options.failpoint?.("lineage_committed");
scheduled += 1;
} else if (await ownershipOverdue()) await releaseOwnershipProofWake(db, run);
} catch (error) {
if (options.failpoint) throw error;
pendingFailures++;
logger.warn(
{ runId: run.id },
"Native replacement remains pending; another sweep will retry its durable decision",
);
}
}
settleReplacementDiscovery(discovery, selection.after, generation);
if (pendingFailures) {
discovery.complete = false; discovery.rescan = true;
throw new Error("native_replacement_decision_retry_required");
}
return { scanned: selection.scanned, scheduled };
}