From 5ba26600ef38a844aeca9713d4e4923c1f204d0a Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 07:09:20 -0600 Subject: [PATCH] fix(native-runtime): a remote Runner reattachment continues a turn the Runner can recover (#21362) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime runs agent turns through a Runner, often in a remote sandbox, and must recover runs after a restart without help from a person > - When a Runner process is lost, a new Runner restores the conversation. If the provider turn is gone, the Runner sends `turn.failed` with `provider_turn_lost_on_restore` and `recoverable: true`. The Runner sends this only after it proves that the old process ended > - The runtime can continue such a turn in the same conversation, but only when the server sets `resumeInterruptedTurn`. The server set it only for a local `resume_dead_runner` claim or a scheduled resume, and a scheduled remote resume is refused. So on a remote Runner reattachment the runtime did not continue the turn, and the run ended > - This pull request sets `resumeInterruptedTurn` for a `reattach_remote_runner` claim too > - The benefit is that a remote run continues the lost turn in the same conversation immediately, instead of failing and then restarting 3 to 6 minutes later ## Linked Issues or Issue Description **What happened?** On hosted staging (build `d1d71a7755`, read-only trace), 10 runs in 3 hours ended with `native_provider_terminal_failed` and the message "The runner restored the conversation after process loss, but the previous turn is no longer active." The Runner marked each of these events `recoverable: true`. Example: run 646e2efb. A new Runner process started at 08:32:04. `session.resumed` came at 08:34:49. `turn.failed provider_turn_lost_on_restore` (recoverable) came at 08:34:51. The run failed at 08:35:21. Example: run 11fe2cdc. `session.resumed` came at 09:23:24. `turn.failed` came at 09:23:25. The run failed at 09:23:52. In each case, a separate recovery path continued the task only 3 to 6 minutes later. **Expected behavior** When the Runner marks a lost turn as recoverable after a restore on a remote reattachment, the run continues the turn in the same conversation. **Steps to reproduce** 1. Start a native run on a remote sandbox Runner. 2. Lose the Runner process while a turn is active, so that a new Runner reattaches through a `reattach_remote_runner` claim. 3. The new Runner restores the conversation and reports `provider_turn_lost_on_restore` with `recoverable: true`. 4. See that the run ends with `native_provider_terminal_failed` and does not continue the turn. **Agent adapter(s) involved** Not adapter-specific (core bug). Seen with Codex on a Daytona remote sandbox. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: `resumeInterruptedTurn` is now also true for a `reattach_remote_runner` restart-recovery claim. The runtime still continues the turn only when the Runner itself sends `turn.failed provider_turn_lost_on_restore` with `recoverable: true` for the checkpointed active turn. - `doc/execution-semantics.md`: one paragraph describes this behaviour, and says that a lost turn that still ends the run keeps `native_provider_terminal_failed`. - Tests: - `native-scheduled-resume-continuation.integration.test.ts`: "continues a lost turn on a same-run remote Runner reattachment" (2 cases: a plain reattachment and a `replaceUnansweredRunner` reattachment). - New `native-recoverable-lost-turn.integration.test.ts`: a guard through `executePaperclipNativeSession`. The runtime saves a failed-terminal checkpoint and then raises the lost turn (recoverable and not recoverable). The run ends as `native_provider_terminal_failed`, and the hold cause stays one that the replacement lane continues. - An earlier revision of this PR also mapped a recoverable lost turn to `native_session_interrupted`. Review showed that this does not make the run retryable, and the hold it writes is not picked up by any automatic continuation. That hunk is removed. ## Verification - Red without the product change: the two integration cases fail with `passed` = `[false]` (expected `[true]`). - Green with the change: - `cd server && npx vitest run src/services/native-runtime/native-scheduled-resume-continuation.integration.test.ts src/services/native-runtime/native-recoverable-lost-turn.integration.test.ts` — 14 passed. - `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts` — passed (with the scheduled-resume file: 224 passed, 14 skipped, pre-existing skips). - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts` — 1469 passed, 7 skipped. - `cd server && npx tsc --noEmit` — exit 0. - Staging evidence (hosted staging, build `d1d71a7755`, read-only trace): 10 runs in 3 hours ended with `native_provider_terminal_failed` although the Runner marked the lost turn `recoverable: true`. Run 646e2efb: new Runner pid 08:32:04, `session.resumed` 08:34:49, `turn.failed provider_turn_lost_on_restore` (recoverable) 08:34:51, run failed 08:35:21. Run 11fe2cdc: `session.resumed` 09:23:24, `turn.failed` 09:23:25, run failed 09:23:52. The task continued only 3 to 6 minutes later through a separate recovery path. ## Risks - This change has not run live on staging or on a Daytona sandbox yet. The evidence is the staging trace and the local tests. - This PR does not introduce or widen the gap where a recoverable lost turn still ends the run. On master, that run already ends as `native_provider_terminal_failed`, and the replacement lane continues it after its decision (3 to 6 minutes on staging). This PR removes the run-ending case whenever the Runner reports the turn recoverable for the checkpointed active turn on a remote reattachment. Where the run still ends (for example, when the continuation itself is lost), the behaviour is unchanged, and the guard test `native-recoverable-lost-turn.integration.test.ts` proves that it stays `native_provider_terminal_failed`. - The remaining case is tracked with the owner of the recovery path: recovery evidence will accept `provider_turn_lost_on_restore` / `recoverable` directly, after their ownership-deadline PR #21364. An earlier hunk here tried to shorten that case. An integration test through the real path showed that it would create a board-owned ambiguous hold that no automatic pass continues, so the hunk is removed. - This change does not fix why the turn is lost. In staging, a retry replaced a live Runner. A separate fix for that is in progress. - The runtime continues a turn only when the Runner sends the recoverable lost-turn signal. The Runner sends it only after it proves that the old process ended. So no provider turn runs twice. - No Runner image change. No schema change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude Code with tool use (shell, file edit, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/execution-semantics.md | 7 ++ ...-recoverable-lost-turn.integration.test.ts | 75 +++++++++++++++++++ ...ed-resume-continuation.integration.test.ts | 22 +++++- .../native-runtime/native-session-executor.ts | 3 +- 4 files changed, 104 insertions(+), 3 deletions(-) create mode 100644 server/src/services/native-runtime/native-recoverable-lost-turn.integration.test.ts diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 41f8670bc5c..1e4190c8064 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1845,6 +1845,13 @@ mid-turn can leave provider or tool processes that the fence cannot find, so it keeps the fail-closed path below, and the task continues in a fresh session from the saved work. +On every `reattach_remote_runner` attempt, a turn the Runner reports as +`provider_turn_lost_on_restore` with `recoverable: true` continues once in the same +conversation, as on a local dead-runner resume. The Runner reports that cause only +after it proved the old process ended and restored the conversation. If the turn +still ends the run, the run keeps `native_provider_terminal_failed`, and the safe +replacement lane continues the task. + The first unanswered read can come on the last of the three provider attempts, for example when a server restart used the second attempt. The run then gets one more attempt for the replacement only: the failure stays retryable, the claim diff --git a/server/src/services/native-runtime/native-recoverable-lost-turn.integration.test.ts b/server/src/services/native-runtime/native-recoverable-lost-turn.integration.test.ts new file mode 100644 index 00000000000..7d20633bd38 --- /dev/null +++ b/server/src/services/native-runtime/native-recoverable-lost-turn.integration.test.ts @@ -0,0 +1,75 @@ +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { agents, companies, completionContracts, createDb, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations } from "@paperclipai/db"; +import type { ControlPlanePort, NativeExecutionInputV1, PersistedNativeSession } from "@paperclipai/paperclip-runner"; +import { EXECUTION_RECONCILIATION_CAUSES } from "@paperclipai/shared"; +import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { NativeProviderTerminalFailure } from "../../vendor/paperclip-runner/index.js"; + +const provider = vi.hoisted(() => ({ execute: vi.fn() })); +vi.mock("../../vendor/paperclip-runner/index.js", async (importOriginal) => ({ + ...(await importOriginal()), executeNativeSession: provider.execute, +})); +import { executePaperclipNativeSession } from "./native-session-executor.js"; +import { buildNativeCompletionContract } from "./completion-contracts.js"; +import { prepareNativeHeartbeatRun } from "./prepare-native-run.js"; + +const LOST_TURN_MESSAGE = "The runner restored the conversation after process loss, but the previous turn is no longer active."; + +describe("a lost turn that still ends the run after a restore", () => { + let temporary: Awaited>; + let db: ReturnType; + const companyId = randomUUID(), agentId = randomUUID(); + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("native-recoverable-lost-turn-"); + db = createDb(temporary.connectionString); + await db.insert(companies).values({ id: companyId, name: "Lost turn", issuePrefix: "LST" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Lost turn worker", status: "active", adapterType: "paperclip_runner", + adapterConfig: { provider: "codex", model: "test-model" } }); + }); + afterAll(async () => { await temporary?.cleanup(); }); + + it.each([true, false])("keeps a lost turn (recoverable %s) on the hold the replacement lane continues", async (recoverable) => { + const issueId = randomUUID(), runId = randomUUID(); + await db.insert(issues).values({ id: issueId, companyId, title: "Lost turn task", status: "in_progress", assigneeAgentId: agentId }); + const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native", + nativeIssueId: issueId, contextSnapshot: { issueId } }).returning(); + const [issue] = await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)).returning(); + const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId: randomUUID() }); + const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issueId)); + const contractInput = buildNativeCompletionContract(issue, { revision: contract.revision }); + await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed" }); + const execution: NativeExecutionInputV1 = { + schema: "paperclip.native-execution-input.v1", binding: { companyId, runId, issueId, agentId, executionWorkspaceId: runId }, + provider: { kind: "codex", model: "test-model" }, + task: { identifier: issueId, title: issue.title, description: null, prompt: "Continue the task.", workMode: "standard" }, + workspace: { cwd: "/tmp", repoUrl: null, repoRef: null, branchName: null }, + session: { normalizedSessionId: native.normalizedSessionId, driverKind: "codex_app_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } }, + completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1", contract: contractInput }, + interactionResponses: [], credentialBindings: [], + }; + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, runId)); + const identity = { companyId, agentId, issueId, runId, sessionId: native.normalizedSessionId }; + const snapshot: PersistedNativeSession = { backendKind: "mock", sessionId: native.normalizedSessionId, identity, + providerSessionId: "provider-thread", activeTurnId: null, pendingRuntimeRequests: [], goal: null, semanticResult: null, + terminal: { schema: "paperclip.prp.terminal.v1", turnTerminalState: "failed", runTerminalState: "failed", reportedWorkDisposition: "yielded" } }; + provider.execute.mockReset().mockImplementation(async (options: { controlPlane: ControlPlanePort }) => { + await options.controlPlane.openRun({ identity, backendKind: "mock", sourceInstanceId: native.runnerInstanceId }); + await options.controlPlane.checkpointSession?.(snapshot); + throw new NativeProviderTerminalFailure("provider_turn_lost_on_restore", recoverable, LOST_TURN_MESSAGE); + }); + const unexpected = async () => { throw new Error("Unexpected provider call"); }; + await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, backend: { descriptor: unexpected, openSession: unexpected } }) + .catch(() => undefined); + expect(provider.execute).toHaveBeenCalledTimes(1); + const [saved] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + expect(saved.runnerProfileJson).toMatchObject({ sessionCheckpoint: { terminal: { runTerminalState: "failed" } } }); + const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)); + expect(coordinator).toMatchObject({ phase: "terminal_failure", failureCode: "native_provider_terminal_failed", nextAttemptAt: null }); + const holds = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId)); + expect(holds).toEqual([expect.objectContaining({ kind: "active_run_watchdog", cause: "native_provider_terminal_failed", + evidence: expect.objectContaining({ runId }) })]); + expect(EXECUTION_RECONCILIATION_CAUSES).toContain(holds[0]!.cause); + }); +}); diff --git a/server/src/services/native-runtime/native-scheduled-resume-continuation.integration.test.ts b/server/src/services/native-runtime/native-scheduled-resume-continuation.integration.test.ts index dd94207b60d..bf447bebdba 100644 --- a/server/src/services/native-runtime/native-scheduled-resume-continuation.integration.test.ts +++ b/server/src/services/native-runtime/native-scheduled-resume-continuation.integration.test.ts @@ -18,6 +18,7 @@ import { executePaperclipNativeSession } from "./native-session-executor.js"; import { buildNativeCompletionContract } from "./completion-contracts.js"; import { prepareNativeHeartbeatRun } from "./prepare-native-run.js"; import { claimNativeSessionResumptions } from "./native-finalization-reconciler.js"; +import { currentNativeControllerIdentity, type NativeRestartRecoveryClaim } from "./native-restart-recovery.js"; const EXITED_PID = 4_194_304; @@ -40,7 +41,8 @@ describe("a scheduled same-run resume reaching the provider runtime", () => { checkpointExists: true, originalFailureCode: "native_session_recovery_failed" }; async function resumeOnce(options: { failureDetail: Record; stopRecorded: boolean; viaReconciler?: boolean; - failedRelaunch?: boolean; unidentifiedStopRecorded?: boolean; liveRunnerForRun?: boolean }) { + failedRelaunch?: boolean; unidentifiedStopRecorded?: boolean; liveRunnerForRun?: boolean; + remoteReattach?: { replaceUnansweredRunner?: true } }) { const issueId = randomUUID(); const runId = randomUUID(); const [issue] = await db.insert(issues).values({ id: issueId, companyId, title: "Resume the lost turn", @@ -99,12 +101,23 @@ describe("a scheduled same-run resume reaching the provider runtime", () => { ? spawn(process.execPath, ["-e", "setTimeout(() => {}, 60_000)", "--", "--run-id", runId], { stdio: "ignore" }) : null; if (liveRunner) await once(liveRunner, "spawn"); let leaseOwner: string | undefined; + let restartRecovery: NativeRestartRecoveryClaim | undefined; + if (options.remoteReattach) { + const controller = await currentNativeControllerIdentity(); + leaseOwner = `reattach-${runId}`; + await db.update(nativeRunFinalizations).set({ phase: "observed", leaseOwner, leaseExpiresAt: new Date(Date.now() + 60_000), + controllerBootId: controller.bootId, controllerPid: controller.pid, controllerGeneration: 7 }) + .where(eq(nativeRunFinalizations.runId, runId)); + restartRecovery = { kind: "reattach_remote_runner", runId, leaseOwner, controllerGeneration: 7, providerAttempt: 2, + restartKind: "hard", recoveryRequestId: null, remote: { providerLeaseId: "sandbox", remoteCwd: "/workspace" }, + ...options.remoteReattach }; + } if (options.viaReconciler) { const claims = await claimNativeSessionResumptions({ db, runnerInstanceId: native.runnerInstanceId, runIds: [runId] }); expect(claims).toEqual([expect.objectContaining({ runId })]); leaseOwner = claims[0]!.leaseOwner; } - await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, leaseOwner, backend: { + await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, leaseOwner, restartRecovery, backend: { descriptor: async () => { throw new Error("Unexpected provider call"); }, openSession: async () => { throw new Error("Unexpected provider call"); }, } }).catch(() => undefined).finally(() => { liveRunner?.kill("SIGKILL"); }); @@ -135,6 +148,11 @@ describe("a scheduled same-run resume reaching the provider runtime", () => { expect(passed).toEqual([false]); }); + it.each([{}, { replaceUnansweredRunner: true as const }])("continues a lost turn on a same-run remote Runner reattachment (%j)", async (remoteReattach) => { + const { passed } = await resumeOnce({ failureDetail: exactResume, stopRecorded: false, remoteReattach }); + expect(passed).toEqual([true]); + }); + it("keeps the lost turn failing when the reconciler claimed a resume without a stop receipt", async () => { const { passed } = await resumeOnce({ failureDetail: exactResume, stopRecorded: false, viaReconciler: true }); expect(passed).toEqual([false]); diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 3b6d3662c47..ef4122799ae 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -12321,7 +12321,8 @@ async function executePaperclipNativeSessionAttempt( trace.activate(runnerSessionStartupScope); const runExecution = () => executeNativeSession({ - resumeInterruptedTurn: input.restartRecovery?.kind === "resume_dead_runner" || scheduledResumeContinuation, + resumeInterruptedTurn: input.restartRecovery?.kind === "resume_dead_runner" + || input.restartRecovery?.kind === "reattach_remote_runner" || scheduledResumeContinuation, getFreshSessionHandoff: input.getFreshSessionHandoff, // A failed seal commits nothing: the messages stay queued for the // turn boundary instead of failing this run before its turn.