fix(native): keep recovery workers off a run its live heartbeat is finishing (#22101)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run ends in two steps: the live heartbeat exports the
workspace, then it commits the accepted result and records the run's
exit code, log summary and usage
> - Two recovery workers can also do these steps: the workspace
finalization worker and the result commit worker
> - The runner posts its accepted result before its process returns.
That post wakes both workers at once
> - The workspace finalization worker then takes the export lock while
the heartbeat still waits for the runner. The heartbeat gets "busy",
stops, and leaves the run to the workers
> - On staging, 45 of 135 recent succeeded native runs had no exit code
and no log size because of this
> - This pull request makes the workers leave a run alone while its live
heartbeat in the same process still runs
> - The benefit is that the heartbeat finishes its own run again, with
exit code, logs and usage. The workers still recover a run that the
heartbeat leaves unfinished

## Linked Issues or Issue Description

Refs #22043

**What happened?**
About a third of succeeded native runs had a null `exit_code` and a null
`log_bytes`. Their traces show a `task.workspace_finalization` span
(only the workspace finalization worker makes it) and no
`execution.native_finalize` or `completion.*` spans. The live heartbeat
lost the workspace export lock to the worker, threw
`NativeWorkspaceFinalizationBusyError` and stopped without completing
the run.

**Expected behavior**
The live heartbeat exports the workspace, commits the result, and
records the exit code, log summary and usage. The workers act only when
the heartbeat cannot finish the run.

**Steps to reproduce**
1. Start a native run that syncs its workspace.
2. Let the runner post its accepted result before its process returns.
The result post wakes the workspace finalization worker at once.
3. The worker exports the workspace first. The heartbeat gets `acquired:
false` and stops. The result commit worker then commits the run without
the heartbeat's completion data.

**Relevant logs or output**
Staging run `cb1cfc23-79eb-4528-bcdc-dcf639430d31`: the worker restored
the workspace in about 51s. The worker then committed the status
decision and projected the run as succeeded. The heartbeat wrote no
completion data.

## What Changed

- Add `server/src/services/native-runtime/live-native-finalization.ts`.
It keeps an in-process reservation for each run whose live heartbeat is
still running.
- The heartbeat (`runtime-execution.ts`) reserves a native run before it
dispatches. It releases the reservation when its execution ends. The
release wakes both worker queues for that run.
- `withNativeWorkspaceFinalizationOwnership` returns `owner_active` to
every caller except the run's own heartbeat while the run is reserved.
The workspace finalization worker already waits for a release
notification on `owner_active`.
- The result commit worker skips a reserved run. The release
notification wakes it again.

## Verification

- New regression tests, red before the fix and green after:
- `native-workspace-finalization-work.test.ts`: "leaves a result posted
mid-run to the live heartbeat, then resumes after its release". Without
the fix the worker exported the workspace while the heartbeat held the
run.
- `native-result-commit-work.test.ts`: "leaves a live heartbeat's run to
it and commits only what the heartbeat leaves behind". Without the fix
the worker committed the run while the heartbeat held it.
- Both tests also prove that the workers still finish the run after the
heartbeat releases it without finishing.
- `pnpm verify` (affected tests) passes. Server typecheck passes.
- After deploy: measure the share of succeeded native runs on staging
with a null `exit_code`. It should go down toward 0%.

## Risks

- The reservation is in memory. A worker in another server process (for
example during a deploy overlap) can still race the heartbeat. That case
behaves as before this change.
- If the heartbeat stops in an unusual way, its `finally` block releases
the reservation and wakes the workers. If the process crashes, the
reservation goes away, and startup discovery recovers the run as before.
- While a heartbeat is still running after the runner posted its result,
the workers wait. Before, they could start the export at once. The
heartbeat does the same work, so the result reaches the user at about
the same time.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-17 21:21:32 -06:00
1 parent ea154ce55e
commit 6902c3f193
8 files changed
+141 -2

No files matched your search

@@ -16,6 +16,7 @@ import { agents, companies, completionContracts, createDb, heartbeatRuns, issues
statusDecisionEffects, statusDecisions, workspaceOperations, type Db } from "@paperclipai/db";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { nativeResultCommitWork } from "../services/native-runtime/native-result-commit-work.js";
import { reserveLiveNativeFinalization } from "../services/native-runtime/live-native-finalization.js";
import { finalizeNativeRun, NativeFinalizationDeferredError } from "../services/native-runtime/native-run-finalizer.js";
import { PaperclipControlPlanePort } from "../services/native-runtime/paperclip-control-plane-port.js";
import { NativeRunCoordinatorStore } from "../services/native-runtime/native-run-coordinator-store.js";
@@ -137,6 +138,22 @@ describe("accepted result commit lifecycle", () => {
await quiet(); await coordinator!.stop(); await start(); await flush(); await quiet();
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.runId, binding.runId))).toHaveLength(1);
});
it.each(["delivered", "lost"] as const)("leaves a live heartbeat's run to it and commits what it leaves behind (release notification %s)", async notification => {
await start(); await flush();
const binding = await seed();
const release = reserveLiveNativeFinalization(db, binding);
await complete(binding, "control_plane"); await barrier(binding);
await vi.advanceTimersByTimeAsync(5000); await delay(10);
// The reconciliation pass also leaves the reserved run to its heartbeat.
await reconcileNativeFinalizations(db, [binding.runId]);
expect((await state(binding.runId)).phase).not.toBe("committed");
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.runId, binding.runId))).toHaveLength(0);
if (notification === "lost") vi.spyOn(db, "transaction").mockRejectedValueOnce(new Error("notification transaction failed"));
await release(); await release();
if (notification === "lost") await vi.advanceTimersByTimeAsync(30_000);
await flush();
expect((await state(binding.runId)).phase).toBe("committed"); await quiet();
});
it("ignores routine company-only finalization hints and still commits accepted work", async () => {
await start(); await flush(); await quiet();
const previous = db.$client.options.debug;
@@ -16,6 +16,7 @@ import { nativeWorkspaceFinalizationWork } from "../services/native-runtime/nati
import { attemptNativeWorkspaceFinalization } from "../services/native-runtime/native-workspace-finalizer.js";
import { getRunExecutionTraceContext, measureRunPhase } from "../services/execution-preparation-timing.js";
import { withNativeWorkspaceFinalizationOwnership } from "../services/native-runtime/native-workspace-finalization-ownership.js";
import { reserveLiveNativeFinalization } from "../services/native-runtime/live-native-finalization.js";
import { resumeNativeWorkspaceAfterOwnerStop } from "../services/native-runtime/native-workspace-owner-recovery.js";
import * as platformProof from "../services/native-runtime/native-workspace-platform-proof.js";
import { reconcileNativeFinalizations } from "../services/native-runtime/native-finalization-reconciler.js";
@@ -320,6 +321,22 @@ describe("workspace finalization scheduled lifecycle", () => {
expect([...entries].sort((a, b) => a.dueAt - b.dueAt).map(item => item.id))
.toEqual([overdue.runId, interrupted.runId]);
});
it("leaves a result posted mid-run to the live heartbeat, then resumes after its release", async () => {
await start(); await flush();
const binding = await seed();
const release = reserveLiveNativeFinalization(db, binding);
const nested = reserveLiveNativeFinalization(db, binding);
await nested(); await nested();
// The runner posts its accepted result before its process returns.
await complete(binding); await flush(); await quiet();
expect(await successful(binding.runId)).toHaveLength(0);
const heartbeat = await withNativeWorkspaceFinalizationOwnership({ db, companyId, runId: binding.runId, liveHeartbeat: true },
async () => "owned");
expect(heartbeat).toEqual({ acquired: true, value: "owned" });
// A heartbeat that stopped before exporting leaves the work to the worker.
await release(); await flush();
expect(await successful(binding.runId)).toHaveLength(1); await quiet();
});
it("waits quietly for the actual local owner to release", async () => {
const binding = await seed(); await complete(binding);
await withNativeWorkspaceFinalizationOwnership({ db, companyId, runId: binding.runId }, async () => {
@@ -36,6 +36,9 @@ import { createHeartbeatRunState } from "./run-state.js";
import { buildEffectiveRunSessionConfigMetadata } from "./workspaces.js";
import { revokeHeartbeatRunGatewayTokens } from "./run-preparation.js";
import { executeHeartbeatRuntime, type HeartbeatRuntimeExecutionInput } from "./runtime-execution.js";
import { liveNativeFinalizationReserved } from "../native-runtime/live-native-finalization.js";
import { onNativeWorkspaceFinalizationRelease } from "../native-workspace-finalization-notifications.js";
import { withNativeWorkspaceFinalizationOwnership } from "../native-runtime/native-workspace-finalization-ownership.js";
import { prepareNativeHeartbeatRun } from "../native-runtime/prepare-native-run.js";
import { buildNativeCompletionContract } from "../native-runtime/completion-contracts.js";
@@ -412,6 +415,36 @@ describe.skipIf(!support.supported)("heartbeat runtime execution boundary", () =
expect(input.instructions.releaseInstructionCopy).toHaveBeenCalledOnce();
}
it.each(["returns", "throws"] as const)("keeps recovery workers off the run until the live heartbeat ends (runner %s)", async outcome => {
const { input } = await fixture(true);
let reservedDuringDispatch = false;
let workerOwnership: unknown;
vi.mocked(executePaperclipNativeSession).mockImplementationOnce(async () => {
// A runner posts its accepted result here, before it returns.
reservedDuringDispatch = liveNativeFinalizationReserved(input.run.id);
workerOwnership = await withNativeWorkspaceFinalizationOwnership({ db, companyId: input.run.companyId, runId: input.run.id },
async () => "taken");
if (outcome === "throws") throw new Error("runner failed after dispatch");
return { exitCode: 0, signal: null, timedOut: false };
});
const execution = executeHeartbeatRuntime(db, input);
if (outcome === "throws") await expect(execution).rejects.toThrow(); else await execution;
expect(reservedDuringDispatch).toBe(true);
expect(workerOwnership).toEqual({ acquired: false, reason: "owner_active" });
expect(liveNativeFinalizationReserved(input.run.id)).toBe(false);
expectCleanup(input);
});
it("still cleans up when waking the workers after the live heartbeat throws", async () => {
const { input } = await fixture(true);
const stop = onNativeWorkspaceFinalizationRelease(db, () => { throw new Error("release listener failed"); });
try {
await expect(executeHeartbeatRuntime(db, input)).resolves.toMatchObject({ dispatched: true });
} finally { stop(); }
expect(liveNativeFinalizationReserved(input.run.id)).toBe(false);
expectCleanup(input);
});
it("passes preparation renewal ownership to native dispatch", async () => {
const { input } = await fixture(true);
const acquired = vi.fn(async () => undefined);
@@ -60,6 +60,7 @@ import { heartbeatRuns, nativeRunFinalizations, workspaceOperations } from "@pap
import { conflict } from "../../errors.js";
import { getStartupTraceContext } from "../../instrumentation.js";
import { logger } from "../../middleware/logger.js";
import { reserveLiveNativeFinalization } from "../native-runtime/live-native-finalization.js";
import {
buildNativeProviderEnvironment,
executePaperclipNativeSession,
@@ -547,6 +548,11 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe
const runGoalControlRequestId = readNonEmptyString(
context.goalControlRequestId,
);
// The runner posts its accepted result before it returns. Keep the recovery
// workers off this run until this heartbeat has finished it.
const releaseLiveNativeFinalization = nativeRuntimeResolution.kind === "native"
? reserveLiveNativeFinalization(db, { companyId: run.companyId, runId: run.id, issueId: issueRef?.id })
: undefined;
try {
if (nativeRuntimeResolution.kind === "native") {
if (!nativeExecution || !nativeRunnerInstanceId)
@@ -1090,7 +1096,7 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe
};
if (nativeWorkspaceSync || remoteWorkspaceAuthority) {
const owned = await measureRunPhase("execution.workspace_ownership", () => withNativeWorkspaceFinalizationOwnership({
db, companyId: run.companyId, runId: run.id,
db, companyId: run.companyId, runId: run.id, liveHeartbeat: true,
...(remoteWorkspaceAuthority && nativeRuntimeResolution.kind !== "native" && persistedExecutionWorkspace
? { remoteWorkspaceId: persistedExecutionWorkspace.id } : {}),
}, completeWorkspace));
@@ -1326,6 +1332,8 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe
}
throw adapterErr;
} finally {
// Never throws, so a failed cleanup below cannot keep the workers waiting.
await releaseLiveNativeFinalization?.();
try {
if (!preserveRunGatewayTokens) await revokeHeartbeatRunGatewayTokens({
db,
@@ -0,0 +1,48 @@
import type { Db } from "@paperclipai/db";
import { notifyNativeResultCommit } from "../native-result-commit-notifications.js";
import { notifyNativeWorkspaceFinalization, releaseNativeWorkspaceFinalization } from "../native-workspace-finalization-notifications.js";
import { logger } from "../../middleware/logger.js";
// Runs whose live heartbeat in this process will export the workspace and
// commit the accepted result itself. A runner posts its result before its
// process returns, and that post wakes the recovery workers. Without this
// reservation a worker could take the run from the heartbeat that is still
// finishing it, and the run would lose its exit code, log summary and usage.
const reservedRuns = new Map<string, number>();
export function liveNativeFinalizationReserved(runId: string): boolean {
return reservedRuns.has(runId);
}
/**
* Hold a run for its live heartbeat. The returned release never throws. It
* wakes the workers for this run, so they resume anything the heartbeat left
* unfinished; the result worker also rechecks a reserved run on its own. A
* crashed process loses the reservation, and startup recovery takes over.
*/
export function reserveLiveNativeFinalization(db: Db, scope: { companyId: string; runId: string; issueId?: string }) {
reservedRuns.set(scope.runId, (reservedRuns.get(scope.runId) ?? 0) + 1);
let released = false;
return async () => {
if (released) return;
released = true;
const remaining = (reservedRuns.get(scope.runId) ?? 1) - 1;
if (remaining > 0) reservedRuns.set(scope.runId, remaining);
else reservedRuns.delete(scope.runId);
if (remaining > 0) return;
try {
await db.transaction(async (tx) => {
await notifyNativeWorkspaceFinalization(tx, scope);
await notifyNativeResultCommit(tx, scope);
});
} catch (error) {
// Startup and periodic discovery still find the run.
logger.warn({ err: error, runId: scope.runId }, "native finalization workers could not be woken after the live heartbeat");
}
try {
releaseNativeWorkspaceFinalization(db, scope);
} catch (error) {
logger.warn({ err: error, runId: scope.runId }, "native workspace finalization could not be woken after the live heartbeat");
}
};
}
@@ -1,4 +1,5 @@
import { sameRunResumeClaimEntry } from "./scheduled-resume-continuation.js";
import { liveNativeFinalizationReserved } from "./live-native-finalization.js";
import { appendBoundedRecoveryHistory } from "./native-restart-recovery.js";
import { notifyLiveRunRecovery } from "../live-run-recovery-notifications.js";
import { currentControllerProcessStamp } from "../controller-process-identity.js";
@@ -692,6 +693,8 @@ async function reconcileNativeFinalizationsPass(
const { operation } = outcome;
const workspaceFinalizeStatus = "succeeded" as const;
if (options.skipNativeResultCommit && row.coordinatorPhase !== "committed") continue;
// The run's live heartbeat commits it; its release wakes the result worker.
if (row.coordinatorPhase !== "committed" && liveNativeFinalizationReserved(row.runId)) continue;
try {
const finalized = await finalizeNativeRun({
db,
@@ -749,6 +752,8 @@ async function reconcileNativeFinalizationsPass(
continue;
}
if (options.skipNativeResultCommit && row.coordinatorPhase !== "committed") continue;
// The run's live heartbeat commits it; its release wakes the result worker.
if (row.coordinatorPhase !== "committed" && liveNativeFinalizationReserved(row.runId)) continue;
const replayDecision = resolveNativeReconciliationStatus({
facts: { canonicalReplay: true },
priorIssueStatus: row.issueStatus as NativeAuthoritativeIssueStatus,
@@ -9,6 +9,9 @@ import { pendingNativeStatusEffectsCondition } from "./native-pending-effect-wor
import { nativeRunnerOwnershipNotHeldCondition } from "./native-runner-ownership.js";
import { finalizeNativeRun, NativeFinalizationDeferredError, recordNativeFinalizationFailure } from "./native-run-finalizer.js";
import { readNativeWorkspaceSyncReference } from "./native-workspace-sync.js";
import { liveNativeFinalizationReserved } from "./live-native-finalization.js";
const LIVE_HEARTBEAT_RECHECK_MS = 30_000;
type Entry = { id: string; companyId: string; issueId: string; dueAt: number };
@@ -80,6 +83,9 @@ export function nativeResultCommitWork(db: Db, finalize = finalizeNativeRun) {
const [row] = await candidates(scope);
if (!row) return null;
const due = entry(row);
// The live heartbeat commits its own result. Its release wakes this queue;
// the recheck still finds the run if that notification is lost.
if (liveNativeFinalizationReserved(current.id)) return { ...due, dueAt: Math.max(due.dueAt, Date.now() + LIVE_HEARTBEAT_RECHECK_MS) };
if (due.dueAt > Date.now()) return due;
const [run] = await db.select({ profile: heartbeatRuns.runnerProfileJson }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.companyId, current.companyId), eq(heartbeatRuns.id, current.id)));
@@ -14,6 +14,7 @@ import { and, eq, inArray, or, sql } from "drizzle-orm";
import { executionWorkspaces, heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, workspaceOperations, withDedicatedDbConnection, type Db } from "@paperclipai/db";
import { currentNativeWorkspaceRecoveryResult } from "./native-workspace-recovery-binding.js";
import { withWorkspaceLockKeepalive } from "./native-workspace-lock-keepalive.js";
import { liveNativeFinalizationReserved } from "./live-native-finalization.js";
import {
currentNativeControllerIdentity,
type NativeControllerIdentity,
@@ -105,9 +106,13 @@ export function nativeWorkspaceFinalizationOwnerCanUpgrade(profile: Record<strin
* There is deliberately no wall-clock timeout that can steal a slow export.
*/
export async function withNativeWorkspaceFinalizationOwnership<T>(
input: { db: Db; companyId: string; runId: string; retainedRepairActionId?: string; remoteWorkspaceId?: string },
input: { db: Db; companyId: string; runId: string; retainedRepairActionId?: string; remoteWorkspaceId?: string;
/** The run's own live heartbeat; every other caller yields to it. */
liveHeartbeat?: boolean },
action: (ownership: NativeWorkspaceFinalizationOwnership) => Promise<T>,
): Promise<{ acquired: false; reason: NativeWorkspaceFinalizationWaitReason } | { acquired: true; value: T }> {
// Its release wakes the workers again.
if (!input.liveHeartbeat && liveNativeFinalizationReserved(input.runId)) return { acquired: false, reason: "owner_active" };
const identity = await currentNativeControllerIdentity();
const processGroupId = await currentProcessGroupId();
const deployment = currentDeploymentIdentity();