fix(native): keep a warm follow-up that inherited its predecessor's process identity waiting on the same run (#21774)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A deploy swaps servers with an overlap. A warm agent session stays
with the outgoing server that ran its last turn.
> - #21697 makes a follow-up wait while that outgoing server is still
alive, instead of starting a second Runner over its session.
> - Runtime selection copies the warm predecessor's process fields
(`processPid`, `processGroupId`, `processStartedAt`) onto the follow-up
and records `inheritedProcessIdentityRunId`.
> - The wait records itself with `deferNativeControllerRecovery`. That
function refused any run with process fields, so it refused every warm
follow-up that the wait stopped. The run failed with
`native_controller_recovery_wait_authority_changed`.
> - This pull request lets the deferral accept an inherited identity,
and clears those inherited fields in the same transaction so the usual
resumption claim can pick the run up.
> - The benefit is that a warm follow-up in a deploy overlap stays on
the same run and does not fail.

## Linked Issues or Issue Description

Refs #21697

**What happened?**
In a zero-downtime release-swap test, a follow-up comment arrived 0.25 s
after its predecessor finished on the outgoing server. The follow-up run
started on the incoming server 1.09 s later.
`waitForLiveWarmSessionOwner` found the outgoing server alive and asked
to wait. `deferNativeControllerRecovery` then refused with
`native_controller_recovery_wait_authority_changed`, because the run
carried the process fields that runtime selection copied from its warm
predecessor. The run failed after 2.7 s with
`native_continuation_requires_reconciliation`. Held recovery started a
fresh run. The task finished, but the run history shows one failed run.

**Expected behavior**
A warm follow-up whose only process identity is inherited from its
predecessor waits on the same run. The resumption owner claims it at the
deadline. A run whose own process has started is still refused.

**Steps to reproduce**
1. Run a warm remote agent turn on server A.
2. Switch new run starts to server B while A still serves.
3. Post a follow-up within about 1 s of the turn's finish.
4. The follow-up fails with
`native_controller_recovery_wait_authority_changed`.

## What Changed

- `server/src/services/native-runtime/native-controller-recovery.ts`
(`deferNativeControllerRecovery`):
- Process fields refuse the deferral only when they are the run's own.
Fields count as inherited when
`runnerProfileJson.inheritedProcessIdentityRunId` is set and the run has
no own `native.process_start_requested` or
`native.process_identity_recorded` event. `run-cost-accounting.ts` uses
the same proof.
- With an inherited identity, the same transaction clears `processPid`,
`processGroupId` and `processStartedAt`. It keeps
`inheritedProcessIdentityRunId`. `claimNativeSessionResumptions`
requires those fields to be null.
-
`server/src/services/native-runtime/native-controller-recovery.integration.test.ts`
(real Postgres):
- New: "keeps a warm follow-up that inherited its predecessor's process
identity on the same run until the outgoing owner's deadline".
- Two new cases in "does not publish an admission wait after authority
changes": `own-process` (process fields, no marker) and
`inherited-then-own-process` (marker plus an own process-start event).
Both stay refused and keep their fields.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-controller-recovery.integration.test.ts
-t "inherited its predecessor|does not publish an admission
wait|unstarted follow-up"`: 9 passed.
- Red on master: the new test fails with `Error:
native_controller_recovery_wait_authority_changed`. The own-process
cases pass on master and with the change.
- Green: the run is `retryable_failure` with `nextAttemptAt` at the
deadline, its process fields are null, no claim happens before the
deadline, and one claim happens at the deadline.

## Risks

- The follow-up now waits instead of failing. The wait ends when the
outgoing server saves a suspended checkpoint, when it exits, or at the
predecessor's finish plus its idle window plus 2 minutes. In the
observed case, the outgoing server's idle close could not save its
checkpoint (`native_runner_checkpoint_owner_lost`), so the wait would
end only when that server exits. See the follow-up note below.
- An own process is still refused. The proof uses the run's own
process-start events, the same proof as run cost accounting.

**Follow-up note (not in this PR):** The outgoing server's checkpoint
write checks the task's current run. A native successor that is parked
in this wait is not one of the accepted successor kinds, and the idle
close is not retried. So the wait does not end early from a checkpoint.
In the observed timeline, the wait is about 225 s (until the outgoing
server exited). The worst case with the default 5-minute idle window is
about 425 s.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code
with tool use (shell, file edit) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-15 09:19:23 -06:00
1 parent b9cf64f5bf
commit 2b23119ed2
5 files changed
+57 -8

No files matched your search

@@ -12,8 +12,8 @@ import os from "node:os";
import { currentProcessSpaceHash, UNKNOWN_PROCESS_SPACE } from "./controller-process-identity.js";
import { proveOwnerGone } from "./owner-proof.js";
export const PROCESS_START_REQUESTED = "native.process_start_requested";
export const PROCESS_IDENTITY_RECORDED = "native.process_identity_recorded";
import { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED } from "./native-process-start-events.js";
export { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED };
/** Server-authored: the runner-state fence refused this run before it asked for
* a runner, because a prior run's saved state could not be proven finished. */
export const PRIOR_OWNER_FENCE_REFUSED = "native.prior_owner_fence_refused";
@@ -0,0 +1,5 @@
export const PROCESS_START_REQUESTED = "native.process_start_requested";
export const PROCESS_IDENTITY_RECORDED = "native.process_identity_recorded";
/** Server-authored events (no source event ID) that prove a run started its own process. */
export const OWN_PROCESS_START_EVENT_TYPES = [PROCESS_START_REQUESTED, PROCESS_IDENTITY_RECORDED] as const;
@@ -433,6 +433,27 @@ describe.skipIf(!recoveryPostgresSupport.supported)("native controller durable d
expect(await claimNativeSessionResumptions({ db, runnerInstanceId: "successor", runIds: [runId], now: nextDeadline })).toEqual([]);
});
it("keeps a warm follow-up that inherited its predecessor's process identity on the same run until the outgoing owner's deadline", async () => {
const fixture = await seed(), runId = randomUUID();
const scope = { ...fixture.scope, runId };
const deadline = new Date(Date.now() + 60_000);
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: scope.issueId,
nativeSessionId: scope.normalizedSessionId, status: "running", runtimeMode: "native",
processPid: 609, processGroupId: 609, processStartedAt: new Date(Date.now() - 120_000),
runnerProfileJson: { nativeExecutionInput: { binding: scope }, inheritedProcessIdentityRunId: fixture.scope.runId } });
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, scope.issueId));
await db.insert(nativeRunFinalizations).values({ companyId, issueId: scope.issueId, runId, phase: "observed" });
await deferNativeControllerRecovery(db, { ...scope, retryAt: deadline });
const [finalization] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
expect(finalization).toMatchObject({ phase: "retryable_failure", attempt: 0, leaseOwner: null,
nextAttemptAt: deadline, failureCode: NATIVE_RETAINED_OWNER_WAIT });
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
expect(run).toMatchObject({ processPid: null, processGroupId: null, processStartedAt: null });
expect(run.runnerProfileJson).toMatchObject({ inheritedProcessIdentityRunId: fixture.scope.runId });
expect(await claimNativeSessionResumptions({ db, runnerInstanceId: "successor", runIds: [runId] })).toEqual([]);
expect(await claimNativeSessionResumptions({ db, runnerInstanceId: "successor", runIds: [runId], now: deadline })).toHaveLength(1);
});
it.each(["committed", "workspace_finalizing"])("defers a remote warm continuation while its %s predecessor has a finalization lease", async phase => {
const fixture = await seed(), runId = randomUUID(), environmentId = randomUUID();
const previous = parseNativeExecutionInput({
@@ -935,14 +956,21 @@ describe.skipIf(!recoveryPostgresSupport.supported)("native controller durable d
}
});
it.each(["live-owner", "expired-owner", "provider-attempt", "stopped", "operator-stop"])("does not publish an admission wait after authority changes: %s", async scenario => {
it.each(["live-owner", "expired-owner", "provider-attempt", "stopped", "operator-stop", "own-process", "inherited-then-own-process", "inherited-then-own-identity"])("does not publish an admission wait after authority changes: %s", async scenario => {
const fixture = await seed(), runId = randomUUID();
const scope = { ...fixture.scope, runId };
const ownProcess = scenario.startsWith("own-process") || scenario.startsWith("inherited-then-own");
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: scope.issueId,
status: scenario === "stopped" ? "cancelled" : "running", runtimeMode: "native",
resultJson: scenario === "operator-stop" ? { cancellation: { source: "operator", expected: true,
initiator: { type: "user" }, reason: "Stop", recordedAt: new Date().toISOString() } } : null,
runnerProfileJson: { nativeExecutionInput: { binding: scope } } });
...(ownProcess ? { processPid: 4242, processGroupId: 4242, processStartedAt: new Date() } : {}),
runnerProfileJson: { nativeExecutionInput: { binding: scope },
...(scenario.startsWith("inherited-then-own") ? { inheritedProcessIdentityRunId: fixture.scope.runId } : {}) } });
if (scenario.startsWith("inherited-then-own")) {
await db.insert(heartbeatRunEvents).values({ companyId, runId, agentId, seq: 1,
eventType: scenario === "inherited-then-own-process" ? "native.process_start_requested" : "native.process_identity_recorded" });
}
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, scope.issueId));
await db.insert(nativeRunFinalizations).values({ companyId, issueId: scope.issueId, runId, phase: "observed",
attempt: scenario === "provider-attempt" ? 1 : 0,
@@ -953,6 +981,9 @@ describe.skipIf(!recoveryPostgresSupport.supported)("native controller durable d
retryAt: new Date(Date.now() + 60_000) }))
.rejects.toThrow("wait_authority_changed");
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)))[0].phase).toBe("observed");
if (ownProcess) {
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0].processPid).toBe(4242);
}
});
it("reports bounded retained-pointer coverage without loading historical payloads", async () => {
@@ -1,10 +1,11 @@
import { posix } from "node:path";
import { and, eq, inArray, ne, or, sql } from "drizzle-orm";
import { environmentLeases, heartbeatRuns, issues, nativeRunFinalizations, nativeRunnerCheckpoints, type Db } from "@paperclipai/db";
import { and, eq, inArray, isNull, ne, or, sql } from "drizzle-orm";
import { environmentLeases, heartbeatRunEvents, heartbeatRuns, issues, nativeRunFinalizations, nativeRunnerCheckpoints, type Db } from "@paperclipai/db";
import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
import { captureLegacyDurablePrpCheckpoint, parseNativeExecutionInput, type DurablePrpCheckpoint, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js";
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
import { readRunCancellation } from "../run-cancellation.js";
import { OWN_PROCESS_START_EVENT_TYPES } from "../native-process-start-events.js";
import { nativeSha256 } from "./canonical.js";
import { findNativeSessionResumeRun, isNativeSessionId, isUnusedNativeSessionBootstrap, nativeSessionProviderEvidence } from "./native-session-resume.js";
import { insertMigratedNativeRunnerCheckpoint, loadNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
@@ -99,16 +100,27 @@ export async function deferNativeControllerRecovery(db: Db, input: {
const cancellation = record(run?.resultJson?.nativeCancellation);
const stop = readRunCancellation(run?.resultJson);
const [clock] = await tx.execute<{ now: Date }>(sql`select clock_timestamp() as now`);
const hasProcessIdentity = Boolean(run?.processPid || run?.processGroupId || run?.processStartedAt);
const inheritedProcessIdentity = hasProcessIdentity
&& typeof record(run?.runnerProfileJson).inheritedProcessIdentityRunId === "string"
&& !(await tx.select({ id: heartbeatRunEvents.id }).from(heartbeatRunEvents).where(and(
eq(heartbeatRunEvents.companyId, input.companyId), eq(heartbeatRunEvents.runId, input.runId),
isNull(heartbeatRunEvents.sourceEventId),
inArray(heartbeatRunEvents.eventType, [...OWN_PROCESS_START_EVENT_TYPES]))).limit(1)).length;
if (!run || run.runtimeMode !== "native" || run.status !== "running" || task?.executionRunId !== input.runId
|| !owner || owner.phase !== "observed" || owner.resultId || owner.attempt !== 0
|| owner.controllerGeneration !== 0 || owner.leaseOwner !== (input.leaseOwner ?? null)
|| (input.leaseOwner && (!owner.leaseExpiresAt || owner.leaseExpiresAt <= new Date(clock.now)))
|| run.processPid || run.processGroupId || run.processStartedAt || authority || progress.size
|| (hasProcessIdentity && !inheritedProcessIdentity) || authority || progress.size
|| !record(run.runnerProfileJson).nativeExecutionInput || run.resultJson?.startupCancellation
|| (stop?.source === "operator" && stop.expected)
|| (["run", "issue"].includes(String(cancellation.scope)) && ["pending", "acknowledged"].includes(String(cancellation.dispatchState)))) {
throw new Error("native_controller_recovery_wait_authority_changed");
}
if (inheritedProcessIdentity) {
await tx.update(heartbeatRuns).set({ processPid: null, processGroupId: null, processStartedAt: null, updatedAt: new Date() })
.where(eq(heartbeatRuns.id, input.runId));
}
await notifyNativeFinalizationWork(tx, input);
await tx.update(nativeRunFinalizations).set({
phase: "retryable_failure", leaseOwner: null, leaseExpiresAt: null, controlDeadlineAt: null,
+2 -1
View File
@@ -13,6 +13,7 @@ import { settleChargedImageReceiptsInTransaction } from "./native-runtime/genera
import type { BudgetServiceHooks } from "./budgets.js";
import { queueBudgetEvaluation } from "./budget-evaluation-work.js";
import { ensureRuntimeAccountingBaseline } from "./runtime-accounting-totals.js";
import { OWN_PROCESS_START_EVENT_TYPES } from "./native-process-start-events.js";
import { logger } from "../middleware/logger.js";
import { promises as fs } from "node:fs";
import { priceAnthropicReceipt } from "./anthropic-pricing.js";
@@ -141,7 +142,7 @@ const inheritedProcessIdentityOnly = sql`${heartbeatRuns.runnerProfileJson}->>'i
and not exists (select 1 from ${heartbeatRunEvents} own_process
where own_process.company_id = ${heartbeatRuns.companyId} and own_process.run_id = ${heartbeatRuns.id}
and own_process.source_event_id is null
and own_process.event_type in ('native.process_start_requested', 'native.process_identity_recorded'))`;
and own_process.event_type in (${sql.join(OWN_PROCESS_START_EVENT_TYPES.map(type => sql`${type}`), sql`, `)}))`;
const nativeProviderWorkNeverStarted = sql`${heartbeatRuns.runtimeMode} = 'native'
and ${heartbeatRuns.status} in ('failed', 'cancelled', 'timed_out', 'interrupted')