Files
PaperClipAI/server/src/services/native-runtime/native-warm-checkpoint-ownership.integration.test.ts
T
DottaandClaude Opus 5.5 776e900b48 fix(native): retire a finished run's idle Runner that still holds the sandbox Runner port before launching (#20995)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs execute through a Runner (`paperclip-runnerd`) inside a
remote sandbox. Daytona reaches every Runner through one fixed preview
port (43127), so a sandbox serves one listening Runner at a time.
> - A warm sandbox is reused by the next run of the same task. That run
often opens a new session, so it neither adopts the previous run's warm
Runner nor retires it. Retirement of another session's warm owner works
only through the in-memory warm registry of the server that launched it.
> - After a server restart (or on another server), that registry is
empty. The previous run's idle warm Runner keeps the port, and every new
launch fails with `runner_ingress_bind_conflict` until the bounded retry
runs out.
> - This pull request finds such a holder on the sandbox before launch
and proves from durable records whether it may be stopped. A finished
run's idle Runner is stopped through its verified process identity. Any
other holder sends the run to the fresh-sandbox fallback from #20863 and
is never stopped.
> - The benefit is that a follow-up run after a restart starts on its
reused sandbox instead of colliding with its predecessor and needing
Retry.

## Linked Issues or Issue Description

**What happened?**
Restart-chaos wave `w-20261111T230537-s6402` (Claude through ACPX,
Daytona warm sandboxes, `--ephemeral-server`, K1/K2/K15 with
`--k15-fresh-home`, seed 6402, master, no injected fault) needed Retry
on 4 of 6 tasks. Each failing run's first Runner failed with
`runner_ingress_bind_conflict: failed to bind listener: Address already
in use`. The harness's sandbox captures name the process holding port
43127 at the time. In every case it was the previous, already
`succeeded` run of the same task, started under the server generation
before the restart:

| Task | Failing run | Port 43127 held by | Holder run status |
| --- | --- | --- | --- |
| t01 | b4769dfd | 395b202d (Runner started 23:11:30) | succeeded
23:12:19 |
| t05 | b6d466fd | 0969ad46 (23:11:29) | succeeded 23:12:18 |
| t02 | 4097fc1f | 527a1659 (23:11:28) | succeeded 23:12:17 |
| t03 | 60d821af | 1c06bf75 (23:11:28) | succeeded 23:12:25 |

**Root cause**
Warm owners are tracked only in the process-local `warmNativeSessions`
map in `server/src/services/native-runtime/native-session-executor.ts`.
`claimWarmNativeInstructionCopy`,
`reserveWarmNativeInstructionDirectory` and the same-session admission
path retire a predecessor only when it is in that map. A restarted
server has an empty map. The follow-up run uses a new session, so
`inspectRemoteWarmSessionRecovery` (same session only) does not adopt it
either. The launch then meets the old Runner on the fixed port.
`stopStaleRemoteRunnerBeforeBootstrap` stops only the run's own Runner,
so the same-run retries collide again.

**Expected behavior**
A finished predecessor's idle warm Runner is retired before the next
Runner launches on the same sandbox, wherever it was launched. A holder
that cannot be proven safe to stop is never stopped, and the run moves
to a fresh sandbox.

## What Changed

- New
`server/src/services/native-runtime/sandbox-runner-port-holders.ts`:
`retireSandboxRunnerPortHolders` lists every other session's Runner on
the sandbox (session roots with a `runner-process.identity`; this run's
own root and moved-aside roots are skipped). It checks each one with the
existing verified process check (`REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT`:
pid, runner id in the command line and, when recorded, the boot
generation). Then, for a live holder:
- Registered in this server's warm registry: it is retired through
`retireWarmPredecessor`.
- Its company-scoped run row is `succeeded` or `cancelled`, for the same
agent, and its Runner state is idle (empty outbox, no pending terminal
delivery, and both saved fences `pendingProviderCleanup` and
`warmTransition` clear): it is stopped through the verified identity
(`TERM`, then `KILL`, each with a bounded wait). An activity log entry
`native.runner_port_holder_retired` is written.
- When the inventory command itself cannot run, the holder state is
unproven, so the run takes the fresh-sandbox fallback.
- Anything else (a live run, a failed or interrupted run that may still
be resumed, an unknown owner, work in the outbox, an unverifiable or
changed identity marker, or a stop that is not confirmed):
`NativePredecessorRetirementUnprovenError`. Before any process start,
the existing #20863 path marks the lease replace-before-reuse and
re-queues the run in the uncharged lane on a fresh sandbox.
- `executePaperclipNativeSessionAttempt` runs this check before
`native.process_start_requested`, and only for a resumed (reused)
sandbox launched through a shell runner without a warm session to reuse
and without a reattach recovery. A created or replacement sandbox cannot
hold another session's Runner, so it adds no command there.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/sandbox-runner-port-holders.integration.test.ts`
(embedded Postgres, real processes): two Runners contend for one real
port. The second bind first fails with `EADDRINUSE`, the same failure as
staging. After `retireSandboxRunnerPortHolders` the finished run's
holder has exited and the bind succeeds, with the activity entry
recorded. The commands run through a confined runner that refuses any
path outside the sandbox workspace base. A holder owned by a live run,
by a finished run with work in its outbox, by a cancelled run with
pending provider cleanup, or by no recorded run is never stopped: the
call raises `NativePredecessorRetirementUnprovenError`
(`runner_port_holder_owner_not_settled` /
`runner_port_holder_unknown_owner`) and the port stays held. This run's
own session, moved-aside roots and exited holders are ignored.
- `native-warm-checkpoint-ownership.integration.test.ts`: its fixture
sandbox runner now answers the inventory command with an empty list
(that sandbox has no other Runner); all 9 cells pass.
- `native-session-executor.test.ts`: 1406 of 1406 pass. Server `tsc
--noEmit` passes.
- Chaos results are in a comment below.

## Risks

- A finished run's idle warm Runner is stopped earlier than its idle
timeout. It held no unsaved work (its run is `succeeded` or `cancelled`
and its outbox is empty), and SIGTERM is the same stop as its idle
close. Only that warm reuse is lost.
- Failed and interrupted holders are not stopped, because their run may
still reattach. Such a run goes to a fresh sandbox instead.
- One extra sandbox command before launch on a reused sandbox for a cold
launch, no wait and no file changes. A warm reuse and a reattach skip
it.
- Not changed: a bind conflict from a holder that is not a Runner (no
identity file) still uses the existing same-run bootstrap budget.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, test runs). Default context window,
extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #20863, #20953, #20972, #20992.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 21:06:12 -06:00

256 lines
19 KiB
TypeScript

import { randomUUID } from "node:crypto";
import { mkdtemp, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { and, eq } from "drizzle-orm";
import { afterAll, beforeAll, expect, it, vi } from "vitest";
import { completionContracts, heartbeatRuns, issues, nativeRunFinalizations } from "@paperclipai/db";
import type { NativeExecutionInputV1 } from "@paperclipai/paperclip-runner";
import { createLocalDiskStorageProvider } from "../../storage/local-disk-provider.js";
import { createStorageService } from "../../storage/service.js";
import { describeEmbeddedPostgres, nativeRestartFixture } from "../../__tests__/helpers/native-restart.js";
import { CONTROL_PLANE_CONFORMANCE_RESULT } from "../../vendor/paperclip-runner/testing.js";
import { defaultCapabilityRunnerdBinary } from "../../vendor/paperclip-runner/index.js";
type ProviderTransport = ReturnType<NonNullable<NonNullable<Parameters<typeof import("../../vendor/paperclip-runner/index.js").createNativeSessionBackend>[1]>["codexTransportFactory"]>>;
const { FakeCodexTransport } = await import(new URL("../../../../packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.test-support.ts", import.meta.url).href) as { FakeCodexTransport: new () => ProviderTransport };
const { runnerdRecoveryInternals } = await import(new URL("../../../../packages/paperclip-runner/src/live/runnerd-codex-transport.ts", import.meta.url).href) as {
runnerdRecoveryInternals: { releaseRunnerProcessOwnership(input: { runnerSettled: boolean; checkpoint: typeof state.checkpoint;
release: () => Promise<void>; forceKill: () => Promise<void> }): Promise<void> };
};
const state = vi.hoisted(() => ({ execute: vi.fn(), checkpoint: null as null | ((settlement: "settled" | "unsettled") => Promise<void>), storage: null as unknown, transport: null as unknown }));
vi.mock("./native-harness-backup-storage.js", async original => ({
...await original<typeof import("./native-harness-backup-storage.js")>(),
durableNativeHarnessStorage: () => state.storage,
}));
vi.mock("./native-runner-execution-driver.js", async original => {
const actual = await original<typeof import("./native-runner-execution-driver.js")>();
return { ...actual, createNativeRunnerExecutionDriver: (input: Parameters<typeof actual.createNativeRunnerExecutionDriver>[0]) => {
if (input.checkpoint) state.checkpoint = input.checkpoint as typeof state.checkpoint;
return actual.createNativeRunnerExecutionDriver(input);
} };
});
vi.mock("../../vendor/paperclip-runner/index.js", async original => {
const actual = await original<typeof import("../../vendor/paperclip-runner/index.js")>();
return { ...actual, executeNativeSession: state.execute,
createNativeSessionBackend: (input: Parameters<typeof actual.createNativeSessionBackend>[0], options: Parameters<typeof actual.createNativeSessionBackend>[1]) => {
options?.codexTransportFactory?.();
return actual.createNativeSessionBackend(input, { ...options, codexTransportFactory: () => state.transport as ProviderTransport });
} };
});
import { closeWarmNativeSessionsForEnvironment, detachNativeSessionsForRestart, executePaperclipNativeSession, NativeControllerDetachedForRestartError } from "./native-session-executor.js";
import * as executionLease from "./native-execution-lease.js";
import { finalizeNativeRun } from "./native-run-finalizer.js";
import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
import { createPostgresWakeQueueAdapter } from "../../modules/wake-queue/adapters/postgres.js";
import { createReleaseIssueExecution } from "../../modules/wake-queue/application/use-cases.js";
describeEmbeddedPostgres("managed warm checkpoint ownership", () => {
const environment = nativeRestartFixture();
let root: string;
beforeAll(async () => { root = await mkdtemp(join(tmpdir(), "warm-checkpoint-owner-")); });
afterAll(async () => { state.storage = null; await rm(root, { recursive: true, force: true }); });
it.each(["normal", "delayed", "instruction_error", "fresh_successor", "wrong_boot", "superseded_owner", "detach_during_close", "detach_during_instruction_error", "detach_during_instruction_error_and_renewal_loss"])("keeps the producing coordinator claim until the managed failed-turn close joins: %s", async scenario => {
let fixture: Omit<Awaited<ReturnType<typeof environment.seedRun>>, "runId"> & { runId: string } = await environment.seedRun(`WARM-CHECKPOINT-OWNER-${scenario}`);
const { companyId, agentId } = environment.current;
const [contract] = await fixture.db.select().from(completionContracts).where(eq(completionContracts.issueId, fixture.issueId));
let execution: NativeExecutionInputV1 = {
schema: "paperclip.native-execution-input.v1",
binding: { companyId, agentId, issueId: fixture.issueId, runId: fixture.runId, executionWorkspaceId: randomUUID() },
provider: { kind: "codex", model: "test-model" },
task: { identifier: "WARM-1", title: "Retain checkpoint owner", description: null, prompt: "Return a failed turn.", workMode: "standard" },
workspace: { cwd: root, repoUrl: null, repoRef: null, branchName: null },
session: { normalizedSessionId: fixture.native.normalizedSessionId, driverKind: "codex_app_server", protocolVersion: 1,
lifecyclePolicy: { mode: "warm", idleTimeoutMs: 60_000 } },
completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1",
contract: { ...contract.contractJson, revision: String(contract.revision) } } as NativeExecutionInputV1["completionContract"],
interactionResponses: [], credentialBindings: [],
};
state.storage = createStorageService(createLocalDiskStorageProvider(join(root, "objects")));
const transport = new FakeCodexTransport();
const request = transport.request.bind(transport);
transport.request = async (method, params) => {
const response = await request(method, params);
return method === "thread/start" || method === "thread/resume"
? { ...response, cwd: root, thread: { ...response.thread as Record<string, unknown>, cwd: root } }
: response;
};
state.transport = transport;
const closeFailures: string[] = [];
const ownersAtClose: Array<{ phase: string; leaseOwner: string | null; resultId: string | null; leaseExpiresAt: Date | null }> = [];
const initiatingError = new Error("fixture instruction checkpoint failed private-canary-N45");
const invalidated = scenario === "wrong_boot" || scenario === "superseded_owner";
const renewalLoss = scenario === "detach_during_instruction_error_and_renewal_loss";
const instructionErrorWithDetach = scenario.startsWith("detach_during_instruction_error");
let originalBoot: string | null = null;
let replacementOwner: string | null = null;
let enteredClose!: () => void, releaseClose!: () => void;
const closeEntered = new Promise<void>(resolve => { enteredClose = resolve; });
const closeGate = new Promise<void>(resolve => { releaseClose = resolve; });
const detached = vi.fn(async () => {});
Object.assign(transport, { detachControllerForRestart: detached });
const renew = executionLease.startNativeSessionExecutionLeaseRenewal;
let renewalStopped = false;
const renewal = vi.spyOn(executionLease, "startNativeSessionExecutionLeaseRenewal").mockImplementation(input => {
const owned = renew(input);
return { ...owned, stop: async () => { renewalStopped = true; await owned.stop(); } };
});
transport.close = async () => {
const [owner] = await fixture.db.select({ phase: nativeRunFinalizations.phase, leaseOwner: nativeRunFinalizations.leaseOwner,
resultId: nativeRunFinalizations.resultId, leaseExpiresAt: nativeRunFinalizations.leaseExpiresAt }).from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId));
ownersAtClose.push(owner);
expect(renewalStopped).toBe(false);
if (scenario === "detach_during_close") { enteredClose(); await closeGate; }
if (scenario === "delayed") {
await expect.poll(async () => {
const [current] = await fixture.db.select({ expires: nativeRunFinalizations.leaseExpiresAt }).from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, fixture.runId));
return current.expires!.getTime();
}, { timeout: 25_000, interval: 250 }).toBeGreaterThan(owner.leaseExpiresAt!.getTime() + 10_000);
expect(renewalStopped).toBe(false);
}
try {
await runnerdRecoveryInternals.releaseRunnerProcessOwnership({ runnerSettled: false,
checkpoint: state.checkpoint, release: async () => {}, forceKill: async () => {} });
} catch (error) { closeFailures.push(String(error)); throw error; }
};
state.execute.mockReset().mockImplementation(async options => {
const session = await options.backend.openSession({ identity: { companyId, agentId, issueId: fixture.issueId,
runId: fixture.runId, sessionId: execution.session.normalizedSessionId }, workingDirectory: root });
await options.onSession(session);
await options.controlPlane.openRun({ identity: { companyId, agentId, issueId: fixture.issueId, runId: fixture.runId,
sessionId: execution.session.normalizedSessionId }, backendKind: "mock", sourceInstanceId: fixture.native.runnerInstanceId });
await options.controlPlane.appendEvent({ schema: "paperclip.prp.event.v1", runId: fixture.runId,
normalizedSessionId: execution.session.normalizedSessionId, turnId: fixture.native.turnId,
sourceInstanceId: fixture.native.runnerInstanceId, sourceEventId: "provider-overload", sourceSeq: 1,
sourceKind: "runner", eventType: "turn.failed", schemaVersion: 1, priority: 0, emittedAt: new Date().toISOString(),
payload: { status: "failed", error: { codexErrorInfo: "serverOverloaded" } } });
const terminal = { schema: "paperclip.prp.terminal.v1", runTerminalState: "failed", turnTerminalState: "failed", reportedWorkDisposition: "needs_review" };
const result = { ...CONTROL_PLANE_CONFORMANCE_RESULT, reportedWorkDisposition: "needs_review",
completionClaim: { ...CONTROL_PLANE_CONFORMANCE_RESULT.completionClaim, contractRevision: execution.completionContract.contract.revision } };
await options.controlPlane.completeRun({ result, terminal, turnId: fixture.native.turnId });
return { result, terminal, turnId: fixture.native.turnId, normalizedSessionId: execution.session.normalizedSessionId,
providerSessionId: "fixture-session", driverKind: "codex_app_server", driverVersion: "fixture", nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null };
});
const environmentId = randomUUID();
let acquisition: "created" | "resumed" = "created";
const providerLeaseId = randomUUID();
const runCurrent = async () => {
await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: { ...fixture.preparedProfile, nativeExecutionInput: execution } })
.where(eq(heartbeatRuns.id, fixture.runId));
return executePaperclipNativeSession({ db: fixture.db, execution, runnerInstanceId: fixture.native.runnerInstanceId,
useRunnerd: true, runnerIngressAuthorized: true,
runnerRemoteBinaryPath: defaultCapabilityRunnerdBinary(), runnerRemoteCodexPath: defaultCapabilityRunnerdBinary(),
runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: root, environmentId,
leaseId: randomUUID(), providerKey: "fixture", effectiveCapabilities: { runnerWebSocketIngress: true },
sandboxLeaseAcquisition: { outcome: acquisition, providerLeaseId },
runner: { execute: async (command: { args?: string[] }) => command.args?.[1] === "uname -s; uname -m"
? { exitCode: 0, timedOut: false, stdout: `${process.platform === "darwin" ? "Darwin" : "Linux"}\n${process.arch === "arm64" ? "arm64" : "x86_64"}\n`, stderr: "" }
: command.args?.[2] === "paperclip-runner-port-holders"
? { exitCode: 0, timedOut: false, stdout: "", stderr: "" }
: { exitCode: 1, timedOut: false, stdout: "", stderr: "" }, syncIn: async () => {} } } as never,
instructionWorkingCopy: { runId: fixture.runId, root, checkpointWarm: async () => {
if (instructionErrorWithDetach) { enteredClose(); await closeGate; throw initiatingError; }
if (scenario === "instruction_error") throw initiatingError;
if (invalidated) {
const [current] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId));
originalBoot = current.controllerBootId;
replacementOwner = scenario === "superseded_owner" ? randomUUID() : current.leaseOwner;
await fixture.db.update(nativeRunFinalizations).set({ controllerBootId: randomUUID(), leaseOwner: replacementOwner,
controllerGeneration: current.controllerGeneration + 1 }).where(eq(nativeRunFinalizations.runId, fixture.runId));
}
return true;
},
collectStopped: async () => {}, hasChanges: async () => false },
});
};
let running: ReturnType<typeof runCurrent> | undefined;
try {
running = runCurrent();
if (scenario.startsWith("detach_during_")) {
await Promise.race([closeEntered, running.then(() => { throw new Error("fixture checkpoint was not reached"); })]);
expect(renewalStopped).toBe(false);
const [beforeDetach] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId));
const detach = await detachNativeSessionsForRestart([fixture.runId]);
expect(detach).toEqual({ detachedRunIds: [fixture.runId], inactiveRunIds: [], unsupportedRunIds: [], pendingRunIds: [] });
expect(detached).toHaveBeenCalledOnce();
const [held] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId));
expect(held).toMatchObject({ phase: "workspace_finalizing", leaseOwner: beforeDetach.leaseOwner, resultId: beforeDetach.resultId });
if (renewalLoss) {
originalBoot = held.controllerBootId;
replacementOwner = randomUUID();
await fixture.db.update(nativeRunFinalizations).set({ leaseOwner: replacementOwner,
controllerGeneration: held.controllerGeneration + 1 }).where(eq(nativeRunFinalizations.runId, fixture.runId));
await expect.poll(() => renewal.mock.results[0].value.remainingMs(), { timeout: 25_000, interval: 250 }).toBe(0);
}
releaseClose();
await expect(running).rejects.toBeInstanceOf(NativeControllerDetachedForRestartError);
const failure = await running.catch(error => error);
expect(failure).toMatchObject({ diagnostic: { stage: "post_result_cleanup",
postExecutionCleanupFailed: instructionErrorWithDetach, renewalStopFailed: renewalLoss } });
expect(failure).not.toHaveProperty("cause");
expect(JSON.stringify(failure)).not.toContain("private-canary-N45");
} else if (invalidated) await expect(running).rejects.toThrow("native_session_lease_lost");
else if (scenario === "instruction_error") await expect(running).rejects.toBe(initiatingError);
else {
const result = await running;
expect(result.exitCode).toBe(1);
expect(result.errorCode).toBe("native_provider_overloaded");
}
expect(state.checkpoint).toBeTypeOf("function");
if (invalidated || renewalLoss) expect(closeFailures).toEqual(["Error: native_runner_checkpoint_owner_lost"]);
else expect(closeFailures).toEqual([]);
expect(ownersAtClose).toHaveLength(1);
expect(ownersAtClose[0].leaseOwner).not.toBeNull();
expect(renewalStopped).toBe(true);
const [released] = await fixture.db.select({ leaseOwner: nativeRunFinalizations.leaseOwner }).from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, fixture.runId));
expect(released.leaseOwner).toBe(invalidated || renewalLoss ? replacementOwner : scenario.startsWith("detach_during_") ? ownersAtClose[0].leaseOwner : null);
expect(state.execute).toHaveBeenCalledTimes(1);
if (scenario.startsWith("detach_during_")) {
const rows = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.nativeIssueId, fixture.issueId));
expect(rows).toHaveLength(1);
expect(rows[0].resultJson).not.toHaveProperty("nativeCancellation");
}
if (scenario === "fresh_successor") {
await fixture.db.update(heartbeatRuns).set({ responsibleUserId: "local-capacity-owner" }).where(eq(heartbeatRuns.id, fixture.runId));
await fixture.db.update(issues).set({ responsibleUserId: "local-capacity-owner" }).where(eq(issues.id, fixture.issueId));
await finalizeNativeRun({ db: fixture.db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
const [retry] = await fixture.db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.retryOfRunId, fixture.runId)));
expect(retry).toMatchObject({ status: "scheduled_retry", scheduledRetryReason: "native_provider_overloaded", contextSnapshot: { forceFreshSession: true } });
const unexpected = async () => { throw new Error("unexpected fixture escalation"); };
const release = createReleaseIssueExecution({ issueLock: createPostgresWakeQueueAdapter(fixture.db, {
resolveResponsibleUserId: unexpected, getRoutineEnv: unexpected, resolveSessionBeforeForWakeup: unexpected }),
recovery: { escalateStrandedAssignedIssue: unexpected, escalateStrandedRecoveryIssueInPlace: unexpected } });
expect((await release({ companyId, runId: fixture.runId, now: new Date() })).outcome.kind).toBe("released");
const [ready] = await fixture.db.update(heartbeatRuns).set({ status: "running", runtimeMode: "native", nativeIssueId: fixture.issueId })
.where(eq(heartbeatRuns.id, retry.id)).returning();
await fixture.db.update(issues).set({ executionRunId: retry.id }).where(eq(issues.id, fixture.issueId));
const [task] = await fixture.db.select().from(issues).where(eq(issues.id, fixture.issueId));
const native = await prepareNativeHeartbeatRun({ db: fixture.db, run: ready, issue: task, environmentLeaseId: randomUUID() });
await fixture.db.insert(nativeRunFinalizations).values({ runId: retry.id, companyId, issueId: fixture.issueId, phase: "observed" });
const previousSession = execution.session.normalizedSessionId;
execution = { ...execution, binding: { ...execution.binding, runId: retry.id }, session: { ...execution.session, normalizedSessionId: native.normalizedSessionId } };
expect(execution.session.normalizedSessionId).not.toBe(previousSession);
fixture = { ...fixture, runId: retry.id, native };
acquisition = "resumed";
renewalStopped = false;
const second = await runCurrent();
expect(second.errorCode).toBe("native_provider_overloaded");
expect(state.execute).toHaveBeenCalledTimes(2);
expect(ownersAtClose).toHaveLength(2);
expect(closeFailures).toEqual([]);
}
} finally {
releaseClose();
await running?.catch(() => undefined);
if (invalidated || renewalLoss) await fixture.db.update(nativeRunFinalizations).set({ controllerBootId: originalBoot, phase: "committed",
leaseOwner: null, leaseExpiresAt: null }).where(eq(nativeRunFinalizations.runId, fixture.runId));
await closeWarmNativeSessionsForEnvironment({ environmentId, reason: "fixture cleanup" });
renewal.mockRestore();
}
}, 45_000);
});