From 6adc8b10493614063375fe58f82057baa461ad4f Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 7 Nov 2026 02:17:24 -0600 Subject: [PATCH] fix(recovery): keep restarted Retry attempts as task directories and settle failed re-preparation (#19759) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Task recovery lets a user click Retry after a run fails, and a native run restarts its own preparation after a control-plane restart > - A Retry for a task without a project runs in a fresh task directory under `isolated-workspace-attempts` > - When a restart hit that Retry during bootstrap, preparing it again treated the task directory as the base of a new git worktree and failed > - The failed run then kept its native recovery claim open, so the task showed Retry but every Retry click created no run > - This pull request keeps the restarted attempt in its task directory and settles a failed re-preparation > - The benefit is that a Retry finishes after a restart, and a failed one can be retried again ## Linked Issues or Issue Description Refs #19666 **What happened?** A task without a project failed after a restart (`native_session_interrupted`). The user clicked Retry. The Retry run started in a fresh task directory. A restart hit it during bootstrap. Restart recovery prepared the same run again and failed with `workspace_validation_failed`: "requested isolated_workspace with git_worktree, but base workspace `.../isolated-workspace-attempts///` is not a git checkout". The task rested blocked with Retry shown. A later Retry click created no run: the receipt stayed deferred with "New work must wait while a native run recovers its unfinished accounting." A second storm wave showed the same failure on a Claude C1 task. **Expected behavior** Preparing a run again after a restart reuses the task directory that run already bound. A git worktree base is only the project's checkout or the task's own worktree. A run whose re-preparation fails leaves a working Retry. **Paperclip version or commit** `master` at `51eb4ff705` and `131b566ca4` (both include #19666). Restart-chaos waves on 2026-11-07. **Steps to reproduce** 1. Use a task with no project and isolated workspaces enabled. Let a native run fail after a restart. 2. Click Retry. The run starts in `isolated-workspace-attempts///`. 3. Restart the control plane while the run bootstraps. 4. The re-preparation fails with `git_worktree_base_not_git_checkout`. Click Retry again: no run starts. Root cause: - `server/src/services/fresh-workspace-continuation.ts` `readFreshWorkspaceReadiness` returns `null` for a native run that already has admitted input and a bound workspace (from #19693). This is correct: the restart must resume the admitted workspace. - `server/src/services/heartbeat/environment-selection.ts` (`boundTaskDirectory`) excluded every native recovery binding from the task-directory path. With readiness no longer fresh, `useIsolatedTaskDirectory` was false. The default `isolated_workspace` strategy (`git_worktree`) applied, with the task directory as its base. - The restart recovery claim set the native coordinator to `observed` with its lease owner. `failRunSetup` failed the run, but nothing released that claim. `budgets.ts` reads an open coordinator as unfinished accounting, and Retry admission reads it as `controller_settling`. ## What Changed - Environment selection treats a native recovery binding as the task directory when it is the task's current binding and a bound isolated task directory. The directory strategy is kept; it never becomes a git worktree base. A recovery binding the issue no longer points to keeps its existing path. - New `settleFailedNativeRecoveryPreparation`. When a `bootstrap_incomplete` or `resume_dead_runner` restart claim fails its re-preparation before any provider dispatch, it settles the claimed coordinator as `terminal_failure` and wakes deferred work. It only changes a coordinator that is still `observed`, owned by that claim, without a result, for a failed run. Reattach claims (live runner) are not changed. - `doc/task-recovery.md` describes both rules. ## Verification - `environment-selection.test.ts`, RC-58 cases: a restarted fresh attempt bound to the task re-prepares as its task directory (`useIsolatedTaskDirectory` true). It failed before the fix and passes now. A binding the issue moved away from keeps the old behavior. - `native-runtime/native-recovery-preparation-failure.test.ts` (embedded Postgres): before settlement, Retry admission reports `controller_settling` (the dead Retry). After settlement the coordinator is `terminal_failure` with the run's error code, and admission no longer waits on the controller. Settlement is idempotent. Another owner, a reattach claim, a run that is not failed, and a coordinator finalizing elsewhere are left unchanged. - Local regression: `environment-selection`, `workspace-preparation`, `retry-workspace-continuity`, `native-recovery-preparation-failure`, `explicit-native-continuation` (588 tests passed). - Restart-chaos: local restart-chaos runs on `3a7abef4e5` passed, but neither one reproduced the exact failure (a restart during the Retry's bootstrap). The race depends on timing, so the unit tests above are the red/green proof for both defects. - `--replay w-20261107T011010-s7403/t02` (codex C1 K4 with the original fault sequence): PASS, R1-R6 and T. The replay's Retry did not cross a bootstrap restart. - Wave `w-20261107T064432-s7210` (codex+claude, C1+B1, K1/K4/K6, `--triggers retry`, concurrency 2): 8/8 PASS. It had 4 Retries, including C1 tasks with a fault fired right after the Retry click. No `git_worktree_base_not_git_checkout` and no dead Retry. The server log has no "incomplete native runner bootstrap" restart in this wave either. - `ce7e3f9b19` registers execution work before the settlement writes (Greptile finding). ## Risks - A native recovery now re-prepares a bound task directory through the task-directory path. That path still validates the binding, owner, status and directory identity before use. - Settling a failed re-preparation ends that run's recovery claim. It applies only when the previous runner is verified gone (`bootstrap_incomplete`, `resume_dead_runner`) and no provider dispatch started in this attempt. A Retry still passes its normal stop, ownership and admission checks. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/task-recovery.md | 9 +- server/src/services/heartbeat.ts | 6 ++ .../heartbeat/environment-selection.test.ts | 30 ++++++ .../heartbeat/environment-selection.ts | 6 +- ...ative-recovery-preparation-failure.test.ts | 95 +++++++++++++++++++ .../native-recovery-preparation-failure.ts | 48 ++++++++++ 6 files changed, 192 insertions(+), 2 deletions(-) create mode 100644 server/src/services/native-runtime/native-recovery-preparation-failure.test.ts create mode 100644 server/src/services/native-runtime/native-recovery-preparation-failure.ts diff --git a/doc/task-recovery.md b/doc/task-recovery.md index cac5d7aff46..55cebac9f1b 100644 --- a/doc/task-recovery.md +++ b/doc/task-recovery.md @@ -38,7 +38,14 @@ repeating the failed step. A Retry never silently drops the agent's work: Either choice consumes the generation's one alternative attempt. Preparing the same attempt again after a restart reuses the directory that attempt already -created and bound; it does not ask for another fresh directory. +created and bound; it does not ask for another fresh directory. A native +restart re-prepares from its admitted workspace. When that is the task's +current task directory, it keeps the directory strategy; a task directory is +never the base of a new git worktree. A git worktree base is always the +project's checkout or the task's own worktree. If a restart's re-preparation +still fails before the provider starts, the run settles its native coordinator +as a terminal failure, so a later Retry is admitted instead of waiting on an +open recovery claim. The existing exact-run retry endpoint retains its source-bound contract. Connected conversations use their existing inbound authorization owner. A board diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index c7bbd976b47..9c85c7515c5 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -23,6 +23,7 @@ import { readUpgradeReadiness } from "./heartbeat/upgrade-readiness.js"; import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; import { writeExecutionControl } from "./execution-control-notifications.js"; import { readFreshWorkspaceReadiness } from "./fresh-workspace-continuation.js"; +import { settleFailedNativeRecoveryPreparation } from "./native-runtime/native-recovery-preparation-failure.js"; import { promptInjectionPreventionService } from "./prompt-injection-prevention.js"; import { createHeartbeatAgentStatus } from "./heartbeat/agent-status.js"; import { heartbeatExternalProvider, nativeRuntimeHasInstructionFilesystem } from "./heartbeat/external-provider.js"; @@ -3810,6 +3811,11 @@ export function heartbeatService( }); legacyPreparationFailedBeforeDispatch = !legacyAdapterEntered && !nativeDispatchStarted; } + // A failed re-preparation must not keep its recovery claim open. + if (runOptions.nativeRestartRecovery && !legacyAdapterEntered && !nativeDispatchStarted) + await settleFailedNativeRecoveryPreparation(db, { companyId: run.companyId, runId: run.id, + claim: runOptions.nativeRestartRecovery }).catch(error => + logger.warn({ err: error, runId: run.id }, "Failed native recovery preparation could not be settled")); } } finally { let caseHandoffCleanupSettled = false; diff --git a/server/src/services/heartbeat/environment-selection.test.ts b/server/src/services/heartbeat/environment-selection.test.ts index b8a3981f7ad..d416429daea 100644 --- a/server/src/services/heartbeat/environment-selection.test.ts +++ b/server/src/services/heartbeat/environment-selection.test.ts @@ -13,6 +13,7 @@ import { instanceSettingsService } from "../instance-settings.js"; import { buildNativeExecutionInput } from "../native-runtime/native-execution-input.js"; import { nativeRuntimeContextFixture } from "../native-runtime/runtime-context.test-fixture.js"; import { selectHeartbeatEnvironment, type HeartbeatEnvironmentSelectionInput } from "./environment-selection.js"; +import { materializeIsolatedTaskDirectory } from "../isolated-task-directory.js"; const restoration = vi.hoisted(() => ({ removed: false })); vi.mock("../git-workspace-restoration.js", async importOriginal => { @@ -96,6 +97,35 @@ describe.skipIf(!support.supported)("heartbeat environment selection boundary", expect(result.persistedNativeExecutionInput?.binding.executionWorkspaceId).toBe(original.id); }); + it.each(["bound", "issue_moved"] as const)( + "re-prepares a restarted fresh Retry attempt as its task directory, never as a git worktree base (RC-58, %s)", async binding => { + const { input, issue, workspace, recover } = await fixture(); + // A project-less task's fresh Retry attempt directory, created and bound by the first preparation. + await db.update(issues).set({ projectId: null }).where(eq(issues.id, issue.id)); + const attemptCwd = await materializeIsolatedTaskDirectory({ companyId: input.agent.companyId, issueId: issue.id, attemptId: input.run.id }); + const created = await executionWorkspaceService(db).create({ companyId: input.agent.companyId, projectId: null, + sourceIssueId: issue.id, name: "Fresh attempt", mode: "isolated_workspace", strategyType: "project_primary", + providerType: "local_fs", cwd: attemptCwd }); + if (!created) throw new Error("Attempt workspace fixture was not created"); + await db.update(issues).set({ executionWorkspaceId: created.id, executionWorkspacePreference: "reuse_existing", + executionWorkspaceSettings: { mode: "isolated_workspace" } }).where(eq(issues.id, issue.id)); + const issueRef = { ...input.task.issueRef!, projectId: null, executionWorkspaceId: created.id, + executionWorkspacePreference: "reuse_existing" }; + input.task.issueRef = issueRef; + input.task.executionProjectId = null; + input.policy.requestedExecutionWorkspaceMode = "isolated_workspace"; + input.policy.issueExecutionWorkspaceSettings = { mode: "isolated_workspace" }; + if (binding === "issue_moved") issueRef.executionWorkspaceId = (await workspace()).id; + // The restart re-prepares from the admitted native input; readiness is no longer "fresh". + recover(created.id); + input.task.freshWorkspace = null; + const result = await selectHeartbeatEnvironment(db, input); + expect(result.nativeRecoveryExecutionWorkspaceId).toBe(created.id); + expect(result.reusableExistingExecutionWorkspace?.id).toBe(created.id); + // Only the task's current binding may resume as a task directory. + expect(result.useIsolatedTaskDirectory).toBe(binding === "bound"); + }); + it("selects the saved execution workspace without inspecting live Git delivery state", async () => { const { input, workspace, recover } = await fixture(); const saved = await workspace(); diff --git a/server/src/services/heartbeat/environment-selection.ts b/server/src/services/heartbeat/environment-selection.ts index e4076e2c0eb..43a8422c90f 100644 --- a/server/src/services/heartbeat/environment-selection.ts +++ b/server/src/services/heartbeat/environment-selection.ts @@ -494,7 +494,11 @@ export async function selectHeartbeatEnvironment(db: Db, input: HeartbeatEnviron ...(reusableExistingExecutionWorkspace?.providerType === "adapter_managed" ? [config.workspaceStrategy] : []), issueAssigneeOverrides?.adapterConfig?.workspaceStrategy, projectExecutionWorkspacePolicy?.workspaceStrategy, issueExecutionWorkspaceSettings?.workspaceStrategy]; - const boundTaskDirectory = !controllerOnly && !fresh && !nativeRecoveryExecutionWorkspaceId && issueId !== null && issueRef !== null && + // RC-58: a native restart re-prepares its admitted workspace. When that is + // the task's own bound directory (for example a fresh Retry attempt), it keeps + // its directory strategy; it never becomes the base of a new git worktree. + const boundTaskDirectory = !controllerOnly && !fresh && issueId !== null && issueRef !== null && + (!nativeRecoveryExecutionWorkspaceId || nativeRecoveryExecutionWorkspaceId === issueRef.executionWorkspaceId) && requestedShouldReuseExisting && requestedExecutionWorkspaceMode === "isolated_workspace" && !workspaceProjectContext?.hasWorkspace && !issueRef.projectWorkspaceId && taskWorkspaceIntent?.selection.kind !== "configured_source" && diff --git a/server/src/services/native-runtime/native-recovery-preparation-failure.test.ts b/server/src/services/native-runtime/native-recovery-preparation-failure.test.ts new file mode 100644 index 00000000000..11f26a5167d --- /dev/null +++ b/server/src/services/native-runtime/native-recovery-preparation-failure.test.ts @@ -0,0 +1,95 @@ +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { + agents, applyPendingMigrations, companies, createDb, heartbeatRunEvents, heartbeatRuns, issueComments, + issueRecoveryActions, issues, nativeRunFinalizations, type Db, +} from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js"; +import { admitExplicitNativeContinuation } from "../explicit-native-continuation.js"; +import { settleFailedNativeRecoveryPreparation } from "./native-recovery-preparation-failure.js"; + +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_NATIVE_RECOVERY_PREPARATION_TEST_DATABASE_URL"); +const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); + +/** RC-58: a failed re-preparation after a restart must leave a working Retry. */ +describe.skipIf(!support.supported)("failed native recovery preparation", () => { + let database: Awaited>; + let db: Db; + beforeAll(async () => { + if (externalUrl) { + await applyPendingMigrations(externalUrl); + database = { connectionString: externalUrl, cleanup: async () => {} } as typeof database; + } else database = await startEmbeddedPostgresTestDatabase("native-recovery-preparation-"); + db = createDb(database.connectionString); + }, 90_000); + afterAll(async () => { await db?.$client.end(); await database?.cleanup(); }); + + async function seed() { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(), commentId = randomUUID(); + const leaseOwner = `boot:2:${randomUUID()}`; + await db.insert(companies).values({ id: companyId, name: "Recovery preparation", defaultResponsibleUserId: "board", + issuePrefix: `P${companyId.replaceAll("-", "").slice(0, 8)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Native", role: "engineer", adapterType: "paperclip_runner", + status: "idle", runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } }); + await db.insert(issues).values({ id: issueId, companyId, title: "Write the report", status: "blocked", assigneeAgentId: agentId }); + // The restart claimed the run (bootstrap_incomplete); its re-preparation then failed. + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issueId, runtimeMode: "native", + status: "failed", errorCode: "workspace_validation_failed", nativePhase: "observed", contextSnapshot: { issueId }, + startedAt: new Date(Date.now() - 120_000), finishedAt: new Date(Date.now() - 60_000) }); + await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed", attempt: 1, + leaseOwner, leaseExpiresAt: new Date(Date.now() + 60_000), controllerGeneration: 2, recoveryState: "bootstrap_incomplete" }); + // Server-authored process trail: the first bootstrap's runner was verified stopped. + await db.insert(heartbeatRunEvents).values([ + { companyId, runId, agentId, seq: 1, eventType: "native.process_start_requested", stream: "system" }, + { companyId, runId, agentId, seq: 2, eventType: "native.process_identity_recorded", stream: "system" }, + { companyId, runId, agentId, seq: 3, eventType: "native.local_process_stopped", stream: "system" }, + ]); + await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issueId, kind: "active_run_watchdog", + cause: "native_continuation_requires_reconciliation", fingerprint: runId, status: "active", ownerType: "board", + returnOwnerAgentId: agentId, nextAction: "Retry the task.", evidence: { runId } }); + await db.insert(issueComments).values({ id: commentId, companyId, issueId, authorType: "user", authorUserId: "board", + body: "Please continue.", createdAt: new Date() }); + return { companyId, agentId, issueId, runId, commentId, leaseOwner }; + } + + async function admissionBlocks(f: Awaited>) { + const blocked: string[] = []; + await db.transaction(async tx => admitExplicitNativeContinuation({ db: tx as unknown as Db, companyId: f.companyId, + issueId: f.issueId, agentId: f.agentId, actorId: "board", actorType: "user", reason: "issue_commented", + commentId: f.commentId, successorRunId: randomUUID(), dryRun: true, onBlocked: reason => blocked.push(reason) })); + return blocked; + } + + it("settles its own recovery claim so a later Retry is no longer held by the controller", async () => { + const f = await seed(); + expect(await admissionBlocks(f)).toEqual(["controller_settling"]); + expect(await settleFailedNativeRecoveryPreparation(db, { companyId: f.companyId, runId: f.runId, + claim: { kind: "bootstrap_incomplete", leaseOwner: f.leaseOwner } })).toBe(true); + const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(coordinator).toMatchObject({ phase: "terminal_failure", leaseOwner: null, resultId: null, + failureCode: "workspace_validation_failed" }); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + expect(run).toMatchObject({ status: "failed", nativePhase: "terminal_failure" }); + expect(await admissionBlocks(f)).not.toContain("controller_settling"); + // Settling is idempotent. + expect(await settleFailedNativeRecoveryPreparation(db, { companyId: f.companyId, runId: f.runId, + claim: { kind: "bootstrap_incomplete", leaseOwner: f.leaseOwner } })).toBe(false); + }, 30_000); + + it.each(["other_owner", "reattach_claim", "run_not_failed", "finalizing_elsewhere"] as const)( + "leaves a coordinator it does not own unchanged: %s", async kind => { + const f = await seed(); + if (kind === "run_not_failed") await db.update(heartbeatRuns).set({ status: "running", finishedAt: null }) + .where(eq(heartbeatRuns.id, f.runId)); + if (kind === "finalizing_elsewhere") await db.update(nativeRunFinalizations).set({ phase: "committing" }) + .where(eq(nativeRunFinalizations.runId, f.runId)); + const claim = { kind: kind === "reattach_claim" ? "reattach_existing_runner" as const : "resume_dead_runner" as const, + leaseOwner: kind === "other_owner" ? `boot:3:${randomUUID()}` : f.leaseOwner }; + const [before] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(await settleFailedNativeRecoveryPreparation(db, { companyId: f.companyId, runId: f.runId, claim })).toBe(false); + const [after] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(after).toEqual(before); + }, 30_000); +}); diff --git a/server/src/services/native-runtime/native-recovery-preparation-failure.ts b/server/src/services/native-runtime/native-recovery-preparation-failure.ts new file mode 100644 index 00000000000..e82e7750b2b --- /dev/null +++ b/server/src/services/native-runtime/native-recovery-preparation-failure.ts @@ -0,0 +1,48 @@ +import { and, eq, isNull } from "drizzle-orm"; +import { heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import { notifyExecutionRunWork } from "../execution-control-notifications.js"; +import type { NativeRestartRecoveryClaim } from "./native-restart-recovery.js"; + +/** Claims whose previous runner process is verified gone. A reattach claim + * still has a live runner, so its coordinator keeps its own recovery owner. */ +const SETTLEABLE_CLAIMS = new Set(["bootstrap_incomplete", "resume_dead_runner"]); + +/** + * RC-58: a restart recovery claims the run's native coordinator, then prepares + * the run again. When that preparation fails before any provider dispatch, the + * run is already failed but the claim would keep the coordinator `observed` + * and owned. That reads as unfinished native accounting and an unsettled + * controller, so a later Retry can never be admitted. Settle exactly this + * claim as a terminal failure. A coordinator that another owner holds, or one + * with a result, is left unchanged. Returns whether this call settled it. + */ +export async function settleFailedNativeRecoveryPreparation(db: Db, input: { + companyId: string; runId: string; claim: Pick; +}): Promise { + if (!SETTLEABLE_CLAIMS.has(input.claim.kind)) return false; + return db.transaction(async tx => { + const [run] = await tx.select({ status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode }) + .from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId))) + .for("update"); + if (run?.status !== "failed") return false; + // Register the work before the writes it announces. A spare wake when no + // coordinator matches below is harmless. + await notifyExecutionRunWork(tx as unknown as Db, { companyId: input.companyId }); + const now = new Date(); + const [settled] = await tx.update(nativeRunFinalizations).set({ + phase: "terminal_failure", leaseOwner: null, leaseExpiresAt: null, nextAttemptAt: null, controlDeadlineAt: null, + failureCode: run.errorCode ?? "native_recovery_preparation_failed", + failureDetail: { recoveryPreparationFailed: { claim: input.claim.kind, errorCode: run.errorCode }, + nextAction: "Preparation failed before the provider was started again. Retry the task." }, + recoveryState: "blocked", updatedAt: now, + }).where(and( + eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, input.runId), + eq(nativeRunFinalizations.phase, "observed"), eq(nativeRunFinalizations.leaseOwner, input.claim.leaseOwner), + isNull(nativeRunFinalizations.resultId), + )).returning({ runId: nativeRunFinalizations.runId }); + if (!settled) return false; + await tx.update(heartbeatRuns).set({ nativePhase: "terminal_failure", nativePhaseUpdatedAt: now, executionControlDeadlineAt: null }) + .where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId))); + return true; + }); +}