fix(native-runtime): a remote Runner reattachment continues a turn the Runner can recover (#21362)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime runs agent turns through a Runner, often in a
remote sandbox, and must recover runs after a restart without help from
a person
> - When a Runner process is lost, a new Runner restores the
conversation. If the provider turn is gone, the Runner sends
`turn.failed` with `provider_turn_lost_on_restore` and `recoverable:
true`. The Runner sends this only after it proves that the old process
ended
> - The runtime can continue such a turn in the same conversation, but
only when the server sets `resumeInterruptedTurn`. The server set it
only for a local `resume_dead_runner` claim or a scheduled resume, and a
scheduled remote resume is refused. So on a remote Runner reattachment
the runtime did not continue the turn, and the run ended
> - This pull request sets `resumeInterruptedTurn` for a
`reattach_remote_runner` claim too
> - The benefit is that a remote run continues the lost turn in the same
conversation immediately, instead of failing and then restarting 3 to 6
minutes later

## Linked Issues or Issue Description

**What happened?**
On hosted staging (build `d1d71a7755`, read-only trace), 10 runs in 3
hours ended with `native_provider_terminal_failed` and the message "The
runner restored the conversation after process loss, but the previous
turn is no longer active." The Runner marked each of these events
`recoverable: true`. Example: run 646e2efb. A new Runner process started
at 08:32:04. `session.resumed` came at 08:34:49. `turn.failed
provider_turn_lost_on_restore` (recoverable) came at 08:34:51. The run
failed at 08:35:21. Example: run 11fe2cdc. `session.resumed` came at
09:23:24. `turn.failed` came at 09:23:25. The run failed at 09:23:52. In
each case, a separate recovery path continued the task only 3 to 6
minutes later.

**Expected behavior**
When the Runner marks a lost turn as recoverable after a restore on a
remote reattachment, the run continues the turn in the same
conversation.

**Steps to reproduce**
1. Start a native run on a remote sandbox Runner.
2. Lose the Runner process while a turn is active, so that a new Runner
reattaches through a `reattach_remote_runner` claim.
3. The new Runner restores the conversation and reports
`provider_turn_lost_on_restore` with `recoverable: true`.
4. See that the run ends with `native_provider_terminal_failed` and does
not continue the turn.

**Agent adapter(s) involved**
Not adapter-specific (core bug). Seen with Codex on a Daytona remote
sandbox.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`:
`resumeInterruptedTurn` is now also true for a `reattach_remote_runner`
restart-recovery claim. The runtime still continues the turn only when
the Runner itself sends `turn.failed provider_turn_lost_on_restore` with
`recoverable: true` for the checkpointed active turn.
- `doc/execution-semantics.md`: one paragraph describes this behaviour,
and says that a lost turn that still ends the run keeps
`native_provider_terminal_failed`.
- Tests:
- `native-scheduled-resume-continuation.integration.test.ts`: "continues
a lost turn on a same-run remote Runner reattachment" (2 cases: a plain
reattachment and a `replaceUnansweredRunner` reattachment).
- New `native-recoverable-lost-turn.integration.test.ts`: a guard
through `executePaperclipNativeSession`. The runtime saves a
failed-terminal checkpoint and then raises the lost turn (recoverable
and not recoverable). The run ends as `native_provider_terminal_failed`,
and the hold cause stays one that the replacement lane continues.
- An earlier revision of this PR also mapped a recoverable lost turn to
`native_session_interrupted`. Review showed that this does not make the
run retryable, and the hold it writes is not picked up by any automatic
continuation. That hunk is removed.

## Verification

- Red without the product change: the two integration cases fail with
`passed` = `[false]` (expected `[true]`).
- Green with the change:
- `cd server && npx vitest run
src/services/native-runtime/native-scheduled-resume-continuation.integration.test.ts
src/services/native-runtime/native-recoverable-lost-turn.integration.test.ts`
— 14 passed.
- `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`
— passed (with the scheduled-resume file: 224 passed, 14 skipped,
pre-existing skips).
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts` — 1469
passed, 7 skipped.
  - `cd server && npx tsc --noEmit` — exit 0.
- Staging evidence (hosted staging, build `d1d71a7755`, read-only
trace): 10 runs in 3 hours ended with `native_provider_terminal_failed`
although the Runner marked the lost turn `recoverable: true`. Run
646e2efb: new Runner pid 08:32:04, `session.resumed` 08:34:49,
`turn.failed provider_turn_lost_on_restore` (recoverable) 08:34:51, run
failed 08:35:21. Run 11fe2cdc: `session.resumed` 09:23:24, `turn.failed`
09:23:25, run failed 09:23:52. The task continued only 3 to 6 minutes
later through a separate recovery path.

## Risks

- This change has not run live on staging or on a Daytona sandbox yet.
The evidence is the staging trace and the local tests.
- This PR does not introduce or widen the gap where a recoverable lost
turn still ends the run. On master, that run already ends as
`native_provider_terminal_failed`, and the replacement lane continues it
after its decision (3 to 6 minutes on staging). This PR removes the
run-ending case whenever the Runner reports the turn recoverable for the
checkpointed active turn on a remote reattachment. Where the run still
ends (for example, when the continuation itself is lost), the behaviour
is unchanged, and the guard test
`native-recoverable-lost-turn.integration.test.ts` proves that it stays
`native_provider_terminal_failed`.
- The remaining case is tracked with the owner of the recovery path:
recovery evidence will accept `provider_turn_lost_on_restore` /
`recoverable` directly, after their ownership-deadline PR #21364. An
earlier hunk here tried to shorten that case. An integration test
through the real path showed that it would create a board-owned
ambiguous hold that no automatic pass continues, so the hunk is removed.
- This change does not fix why the turn is lost. In staging, a retry
replaced a live Runner. A separate fix for that is in progress.
- The runtime continues a turn only when the Runner sends the
recoverable lost-turn signal. The Runner sends it only after it proves
that the old process ended. So no provider turn runs twice.
- No Runner image change. No schema change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude
Code with tool use (shell, file edit, code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 07:09:20 -06:00
1 parent d9b42b50fd
commit 5ba26600ef
4 files changed
+104 -3

No files matched your search

+7
View File
@@ -1845,6 +1845,13 @@ mid-turn can leave provider or tool processes that the fence cannot find, so it
keeps the fail-closed path below, and the task continues in a fresh session from
the saved work.
On every `reattach_remote_runner` attempt, a turn the Runner reports as
`provider_turn_lost_on_restore` with `recoverable: true` continues once in the same
conversation, as on a local dead-runner resume. The Runner reports that cause only
after it proved the old process ended and restored the conversation. If the turn
still ends the run, the run keeps `native_provider_terminal_failed`, and the safe
replacement lane continues the task.
The first unanswered read can come on the last of the three provider attempts,
for example when a server restart used the second attempt. The run then gets one
more attempt for the replacement only: the failure stays retryable, the claim
@@ -0,0 +1,75 @@
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { agents, companies, completionContracts, createDb, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations } from "@paperclipai/db";
import type { ControlPlanePort, NativeExecutionInputV1, PersistedNativeSession } from "@paperclipai/paperclip-runner";
import { EXECUTION_RECONCILIATION_CAUSES } from "@paperclipai/shared";
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
import { NativeProviderTerminalFailure } from "../../vendor/paperclip-runner/index.js";
const provider = vi.hoisted(() => ({ execute: vi.fn() }));
vi.mock("../../vendor/paperclip-runner/index.js", async (importOriginal) => ({
...(await importOriginal<typeof import("../../vendor/paperclip-runner/index.js")>()), executeNativeSession: provider.execute,
}));
import { executePaperclipNativeSession } from "./native-session-executor.js";
import { buildNativeCompletionContract } from "./completion-contracts.js";
import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
const LOST_TURN_MESSAGE = "The runner restored the conversation after process loss, but the previous turn is no longer active.";
describe("a lost turn that still ends the run after a restore", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
const companyId = randomUUID(), agentId = randomUUID();
beforeAll(async () => {
temporary = await startEmbeddedPostgresTestDatabase("native-recoverable-lost-turn-");
db = createDb(temporary.connectionString);
await db.insert(companies).values({ id: companyId, name: "Lost turn", issuePrefix: "LST" });
await db.insert(agents).values({ id: agentId, companyId, name: "Lost turn worker", status: "active", adapterType: "paperclip_runner",
adapterConfig: { provider: "codex", model: "test-model" } });
});
afterAll(async () => { await temporary?.cleanup(); });
it.each([true, false])("keeps a lost turn (recoverable %s) on the hold the replacement lane continues", async (recoverable) => {
const issueId = randomUUID(), runId = randomUUID();
await db.insert(issues).values({ id: issueId, companyId, title: "Lost turn task", status: "in_progress", assigneeAgentId: agentId });
const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native",
nativeIssueId: issueId, contextSnapshot: { issueId } }).returning();
const [issue] = await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)).returning();
const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId: randomUUID() });
const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issueId));
const contractInput = buildNativeCompletionContract(issue, { revision: contract.revision });
await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed" });
const execution: NativeExecutionInputV1 = {
schema: "paperclip.native-execution-input.v1", binding: { companyId, runId, issueId, agentId, executionWorkspaceId: runId },
provider: { kind: "codex", model: "test-model" },
task: { identifier: issueId, title: issue.title, description: null, prompt: "Continue the task.", workMode: "standard" },
workspace: { cwd: "/tmp", repoUrl: null, repoRef: null, branchName: null },
session: { normalizedSessionId: native.normalizedSessionId, driverKind: "codex_app_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1", contract: contractInput },
interactionResponses: [], credentialBindings: [],
};
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, runId));
const identity = { companyId, agentId, issueId, runId, sessionId: native.normalizedSessionId };
const snapshot: PersistedNativeSession = { backendKind: "mock", sessionId: native.normalizedSessionId, identity,
providerSessionId: "provider-thread", activeTurnId: null, pendingRuntimeRequests: [], goal: null, semanticResult: null,
terminal: { schema: "paperclip.prp.terminal.v1", turnTerminalState: "failed", runTerminalState: "failed", reportedWorkDisposition: "yielded" } };
provider.execute.mockReset().mockImplementation(async (options: { controlPlane: ControlPlanePort }) => {
await options.controlPlane.openRun({ identity, backendKind: "mock", sourceInstanceId: native.runnerInstanceId });
await options.controlPlane.checkpointSession?.(snapshot);
throw new NativeProviderTerminalFailure("provider_turn_lost_on_restore", recoverable, LOST_TURN_MESSAGE);
});
const unexpected = async () => { throw new Error("Unexpected provider call"); };
await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, backend: { descriptor: unexpected, openSession: unexpected } })
.catch(() => undefined);
expect(provider.execute).toHaveBeenCalledTimes(1);
const [saved] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
expect(saved.runnerProfileJson).toMatchObject({ sessionCheckpoint: { terminal: { runTerminalState: "failed" } } });
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
expect(coordinator).toMatchObject({ phase: "terminal_failure", failureCode: "native_provider_terminal_failed", nextAttemptAt: null });
const holds = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId));
expect(holds).toEqual([expect.objectContaining({ kind: "active_run_watchdog", cause: "native_provider_terminal_failed",
evidence: expect.objectContaining({ runId }) })]);
expect(EXECUTION_RECONCILIATION_CAUSES).toContain(holds[0]!.cause);
});
});
@@ -18,6 +18,7 @@ import { executePaperclipNativeSession } from "./native-session-executor.js";
import { buildNativeCompletionContract } from "./completion-contracts.js";
import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
import { claimNativeSessionResumptions } from "./native-finalization-reconciler.js";
import { currentNativeControllerIdentity, type NativeRestartRecoveryClaim } from "./native-restart-recovery.js";
const EXITED_PID = 4_194_304;
@@ -40,7 +41,8 @@ describe("a scheduled same-run resume reaching the provider runtime", () => {
checkpointExists: true, originalFailureCode: "native_session_recovery_failed" };
async function resumeOnce(options: { failureDetail: Record<string, unknown>; stopRecorded: boolean; viaReconciler?: boolean;
failedRelaunch?: boolean; unidentifiedStopRecorded?: boolean; liveRunnerForRun?: boolean }) {
failedRelaunch?: boolean; unidentifiedStopRecorded?: boolean; liveRunnerForRun?: boolean;
remoteReattach?: { replaceUnansweredRunner?: true } }) {
const issueId = randomUUID();
const runId = randomUUID();
const [issue] = await db.insert(issues).values({ id: issueId, companyId, title: "Resume the lost turn",
@@ -99,12 +101,23 @@ describe("a scheduled same-run resume reaching the provider runtime", () => {
? spawn(process.execPath, ["-e", "setTimeout(() => {}, 60_000)", "--", "--run-id", runId], { stdio: "ignore" }) : null;
if (liveRunner) await once(liveRunner, "spawn");
let leaseOwner: string | undefined;
let restartRecovery: NativeRestartRecoveryClaim | undefined;
if (options.remoteReattach) {
const controller = await currentNativeControllerIdentity();
leaseOwner = `reattach-${runId}`;
await db.update(nativeRunFinalizations).set({ phase: "observed", leaseOwner, leaseExpiresAt: new Date(Date.now() + 60_000),
controllerBootId: controller.bootId, controllerPid: controller.pid, controllerGeneration: 7 })
.where(eq(nativeRunFinalizations.runId, runId));
restartRecovery = { kind: "reattach_remote_runner", runId, leaseOwner, controllerGeneration: 7, providerAttempt: 2,
restartKind: "hard", recoveryRequestId: null, remote: { providerLeaseId: "sandbox", remoteCwd: "/workspace" },
...options.remoteReattach };
}
if (options.viaReconciler) {
const claims = await claimNativeSessionResumptions({ db, runnerInstanceId: native.runnerInstanceId, runIds: [runId] });
expect(claims).toEqual([expect.objectContaining({ runId })]);
leaseOwner = claims[0]!.leaseOwner;
}
await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, leaseOwner, backend: {
await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, leaseOwner, restartRecovery, backend: {
descriptor: async () => { throw new Error("Unexpected provider call"); },
openSession: async () => { throw new Error("Unexpected provider call"); },
} }).catch(() => undefined).finally(() => { liveRunner?.kill("SIGKILL"); });
@@ -135,6 +148,11 @@ describe("a scheduled same-run resume reaching the provider runtime", () => {
expect(passed).toEqual([false]);
});
it.each([{}, { replaceUnansweredRunner: true as const }])("continues a lost turn on a same-run remote Runner reattachment (%j)", async (remoteReattach) => {
const { passed } = await resumeOnce({ failureDetail: exactResume, stopRecorded: false, remoteReattach });
expect(passed).toEqual([true]);
});
it("keeps the lost turn failing when the reconciler claimed a resume without a stop receipt", async () => {
const { passed } = await resumeOnce({ failureDetail: exactResume, stopRecorded: false, viaReconciler: true });
expect(passed).toEqual([false]);
@@ -12321,7 +12321,8 @@ async function executePaperclipNativeSessionAttempt(
trace.activate(runnerSessionStartupScope);
const runExecution = () =>
executeNativeSession({
resumeInterruptedTurn: input.restartRecovery?.kind === "resume_dead_runner" || scheduledResumeContinuation,
resumeInterruptedTurn: input.restartRecovery?.kind === "resume_dead_runner"
|| input.restartRecovery?.kind === "reattach_remote_runner" || scheduledResumeContinuation,
getFreshSessionHandoff: input.getFreshSessionHandoff,
// A failed seal commits nothing: the messages stay queued for the
// turn boundary instead of failing this run before its turn.