From 820c458465fa40137c81fe510f3e6dc573b4f895 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 28 Oct 2026 09:49:32 -0500 Subject: [PATCH] fix(runner): restore per-turn controller checkpoints before resume (#18018) ## Thinking Path > - Paperclip coordinates agents and preserves their work between turns. > - Remote native sessions save controller checkpoints in the database. > - External persistence leaves a small identity marker on local disk. > - A later `per_turn` continuation must restore that checkpoint before it reads the controller identity. > - The restore path handled warm sessions only, so valid per-turn continuations failed before provider execution. > - This PR restores the referenced checkpoint and retains the existing ownership and lifecycle checks. ## Linked Issues or Issue Description Related: #17439 added external controller persistence and the warm restore path. #18011 repairs warm preparation and fresh bootstrap checks; it does not cover ordinary per-turn continuation. **What happened?** A remote OpenCode run completed with `per_turn` lifecycle and saved its controller checkpoint in the database. Its next turn reused the normalized session. Admission read the remaining external-persistence marker as a full controller file and failed with `runner_state_identity_mismatch: durable_identity_unreadable`. The prior run's complete checkpoint and harness backup still existed. The failure occurred before provider work started. **Expected behavior** Restore the exact prior controller checkpoint before identity inspection. Then apply the existing company, task, session, prior-owner and suspended-state checks before a provider can start. **Steps to reproduce** 1. Complete a remote native OpenCode task with `per_turn` lifecycle and external controller persistence. 2. Send a follow-up that resumes the same saved session without a session reset. 3. Observe admission reject the valid external controller marker before provider execution. **Paperclip version or commit** Observed at `40c1ab36cdac70c85dc8fe5e912431834fdb691b`. The same warm-only restore remains on current master. **Deployment mode** Hosted Linux with Daytona and native OpenCode. The original model was `openrouter/deepseek/deepseek-v4.1-flash-1017`. ## What Changed - Restore external controller state for ordinary remote per-turn continuations through the existing checkpoint loader. - Bind that read to the current company and task, the marker's run, and the expected normalized session. - Keep local, warm and restart-recovery paths, prior-owner checks and harness backup checks intact. - Refresh a prior-run projection from the latest checkpoint after a rejected active-owner attempt. Keep current-run full state and the existing legacy file-backed fallback intact. - Add restoration and denial regressions, execution documentation and session verification recipes. ## Verification - Red: the real controller generated an external marker; a valid per-turn OpenCode continuation failed with the exact `durable_identity_unreadable` error before tool-authority construction. The repair passed the same case and its ownership guards. - Review regression red/green: a real controller advanced its checkpoint after an active-owner rejection. Before the follow-up fix, a later valid continuation retained a null attach seed. After the fix, it restored the newer attach template and prepare command. Legacy full-file roots with no database checkpoint retain their existing behavior. - At current head `1f6589233e6121d082a0c0c4444ea895b6e8cd7c`, 86 focused executor cases and all 14 execution-driver cases pass. Coverage includes checkpoint restoration, direct/backup suspension, Codex, legacy files, warm admission and canonical bootstrap, missing or malformed evidence, wrong bindings, active/foreign owners and unsuspended runners. Tests stop before provider launch. - The real PostgreSQL encrypted-checkpoint integration passes, including normalized-session refusal before write and stale-writer fencing. Independent source review found no actionable safety defect. - `node scripts/check-maps.mjs`, `git diff --check`, full `pnpm -r typecheck` and `pnpm build` pass on current head. - The local full suite did not complete: native PostgreSQL startup failed under the host sandbox, an initial Docker rerun lacked migrations, and the migrated retry crossed the review-fix revision change. Logs are retained; that mixed-revision run was stopped and its owned database removed. Full test evidence comes from current-head CI. - Current-head CI passes: 53 successful checks, two intended Storybook skips, none pending or failing. Greptile is 5/5 on that exact head with no open review threads. The only manual merge conflict with #18011 was in the verification recipes; both recipes and the unchanged per-turn restore helper are preserved. **Live continuation acceptance remains unverified, so this PR stays draft.** Two disposable OpenCode/Daytona attempts never reached turns two and three: - At `72b49208b1`, the first turn executed tools and produced the exact 33-byte workspace note, but exceeded its four-minute settlement deadline. The first completion lacked a registered attachment; the second decorated a typed attachment reference with filename/size/hash text. The strict UUID parser rejected that reference before checking a valid criterion reference. The scoped attachment, asset and publication receipt exist. Replaying the actual guard rejects the original input before a database lookup; correcting only that reference passes against the recovered row and receipt. This did not show a publication-persistence or controller defect. The task remained unsettled. - At `1f6589233e`, a separately approved replacement used a temporary prompt-only fixture change requiring exact `deliverable:` references, with metadata in `verification.detail`. Production code, grading and deadlines were unchanged; the fixture patch was removed afterward. This attempt had no recorded provider authentication, harness-ready, model, tool or result event. Ingress acquisition took 84.072 seconds; transport connection took 205.835 seconds within the 239.832-second failed native execution. A local settlement GET also timed out after 30 seconds. The underlying startup stall is unresolved; these observations do not establish a remote provider outage. Final recovery was blocked with `ambiguous_remote_recovery_evidence`; no successful settlement or continuation is claimed. Both attempts used a Darwin arm64 source host with Node 26.4.0, isolated PostgreSQL 18, a Linux x64 runner and OpenCode 1.18.34. The representative qualified model was `openrouter/deepseek/deepseek-v4-flash-0731`; the original incident used `openrouter/deepseek/deepseek-v4.1-flash-1017`. The controller defect occurs before provider work. Runtime artifacts came from immutable build `e7ea2e2961`, whose Runner package source is identical to the candidate. The current-source manifest, artifact, distribution, asset and all 110 archive-hash checks passed. Each attempt had $10 company/agent model budgets and zero automatic workflow/task retries. Accounting was incomplete; zero recorded spend is not proof of zero cost. Earlier setup failures are also retained: host PostgreSQL bootstrap denial, cached-pack compatibility admission before provider work, and an [official artifact build](https://github.com/paperclipai/paperclip/actions/runs/40696838122) that stopped during an unrelated Pi npm install at its 300-second bound. No manifest or integrity guard was weakened. Owned local processes and databases are stopped, and the temporary fixture patch is removed. Standard fixture cleanup had left one replacement sandbox started; it was stopped and deleted through the SDK. Every owned sandbox was then verified absent with a final 404 readback. Resource cleanup does not establish successful task finalization. An earlier head retained one [voice-recovery timing failure](https://github.com/paperclipai/paperclip/actions/runs/40695892339/job/122843005098): a deadline test observed two failed-provider inspections instead of one. A single isolated shard rerun passed all 1,579 tests without a source change. Its timing explanation remains an inference. The current reconciled head passed without a retry. ## Risks Restoration writes a local projection before the existing admission checks. The encrypted checkpoint loader validates company/task/run and normalized session before writing. A later ordinary continuation refreshes a prior-run projection so rejected admission cannot freeze an active owner's older checkpoint. Active, foreign or unsuspended prior owners still cannot start a successor. Missing or malformed evidence remains blocked. The change adds no retry, credential, provider binary or lifecycle-policy change; it preserves the existing legacy fallback. This PR does not deploy the repair or recover the original hosted tasks. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use, code execution and collaborating agents. The session exposes GPT-6 as its model version; a more specific model ID and context-window size are unavailable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/execution-semantics.md | 12 ++ maps/agent-access-map/runs-adapters.md | 10 + maps/feature-map/runs-adapters.md | 11 ++ .../native-session-executor.test.ts | 171 +++++++++++++++++- .../native-runtime/native-session-executor.ts | 19 ++ 5 files changed, 219 insertions(+), 4 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 2ceb6be9c74..61bffc56443 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -2531,3 +2531,15 @@ restored checkpoint must match the normalized session before writing its local projection. This adds a scoped checkpoint read and local reconstruction only when warm admission encounters an external marker or an older projection; it adds no streaming checkpoint writes or recurring idle work. + +Ordinary remote `per_turn` continuations restore a valid external controller +marker from the referenced run's company/task-bound checkpoint before reading +its durable identity. The restored checkpoint must match the normalized session. +An existing prior-run projection also refreshes from that checkpoint, so a +rejected attempt cannot retain state that its active owner later advances. +Current-run full state stays owned by its controller. Legacy file-backed state +retains its existing admission path when no database checkpoint exists. +The existing prior-owner, suspended-state and backup checks still decide whether +the next turn may start. Missing checkpoints and ambiguous identity remain +blocked; restoration does not authorize a new provider session or change the +configured lifecycle policy. diff --git a/maps/agent-access-map/runs-adapters.md b/maps/agent-access-map/runs-adapters.md index 1d0cfb31992..d5892c37ec5 100644 --- a/maps/agent-access-map/runs-adapters.md +++ b/maps/agent-access-map/runs-adapters.md @@ -299,6 +299,16 @@ Policy: Task-session listing and execution-context reset require board agent management at their HTTP routes. These controls are separate from delegated managed AI-account authentication attempts under `model-accounts`. +Ordinary remote `per_turn` continuation restores an external controller marker +through the existing company/task-bound checkpoint store, with the expected +normalized session. Prior-owner, lifecycle and backup admission checks remain +authoritative. Prior-run projections refresh when that checkpoint exists; legacy +file-backed admission remains available when it does not. This changes no agent +grant or session-control endpoint. The +[executor controls](../../server/src/services/native-runtime/native-session-executor.test.ts) +provide source-level restoration and denial evidence; live provider continuity +is a separate verification step. + Evidence: `source-only` — [implementation](../../server/src/routes/agents.ts#L6608) at the reviewed commit above. diff --git a/maps/feature-map/runs-adapters.md b/maps/feature-map/runs-adapters.md index e6bade34f1d..5d86501621a 100644 --- a/maps/feature-map/runs-adapters.md +++ b/maps/feature-map/runs-adapters.md @@ -74,6 +74,17 @@ mutations. Qualify a useful model task separately from saving the account. ### `session` +For a remote OpenCode agent configured with `per_turn`, complete a task, then +send two follow-ups without resetting its session. Verify that each continuation +restores its external controller checkpoint, retains provider context and saved +files, and finishes with no stranded task lock. A missing checkpoint, an active +prior owner or a mismatched session must still stop admission. The +rejected active-owner attempt must not prevent a later continuation from reading +the owner's newer completed checkpoint. The +[executor controls](../../server/src/services/native-runtime/native-session-executor.test.ts) +cover checkpoint restoration and its guards; real-provider acceptance remains +separate evidence. + After a completed warm turn, change an agent setting that invalidates its instruction copy, then send a follow-up. Verify the old runner suspends during preparation, saved files remain available, and the follow-up completes once. diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 97dd4685495..add9cfac21e 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -11262,10 +11262,13 @@ describe("runnerd provider runtime wiring", () => { function priorNativeRunDb(priorExecution: NativeExecutionInput, status = "succeeded"): Db { return { - select: () => ({ from: () => ({ where: () => ({ limit: async () => [{ - status, - runnerProfileJson: { nativeExecutionInput: priorExecution }, - }] }) }) }), + select: () => ({ from: (table: unknown) => { + const result = Promise.resolve(table === heartbeatRuns ? [{ + status, runnerProfileJson: { nativeExecutionInput: priorExecution }, + }] : []); + const query = { then: result.then.bind(result), where: () => query, limit: () => query }; + return query; + } }), } as unknown as Db; } @@ -13118,6 +13121,166 @@ describe("runnerd provider runtime wiring", () => { expect(state.createTransport).not.toHaveBeenCalled(); }); + it("refreshes a per-turn controller projection after rejecting an active prior owner", async () => { + const prior = parseNativeExecutionInput({ + ...scopedNativeExecution("per-turn-active-projection"), + schema: "paperclip.native-execution-input.v5", + executionMode: "default", planningContext: null, runtimeContext: nativeRuntimeContextFixture(), + provider: { kind: "opencode", model: "openrouter/deepseek/deepseek-v4-flash-0731", permissionMode: "allow" }, + session: { ...execution.session, normalizedSessionId: "session-per-turn-active-projection", + driverKind: "opencode_server", lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } }, + }); + const current = { ...prior, binding: { ...prior.binding, runId: `${prior.binding.runId}-next` } }; + const identity = { runId: prior.binding.runId, normalizedSessionId: prior.session.normalizedSessionId!, + runnerInstanceId: "runner-per-turn-active-projection", environmentLeaseId: "lease-per-turn-active-projection", + turnId: "turn-prior", itemId: "item-prior" }; + const root = await initializeScopedRunnerRoot(prior, identity.runnerInstanceId); + const controller = new DurablePrpControlPlane({ stateDirectory: join(root, "control-plane"), identity, + expectedRunnerVersion: "per-turn-fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` }); + let saved: import("../../vendor/paperclip-runner/index.js").DurablePrpCheckpoint | undefined; + controller.setCheckpointPersistence(async checkpoint => { saved = checkpoint; }); + await controller.flushCheckpoint(); + await mkdir(join(root, "runner"), { recursive: true }); + await writeFile(join(root, "runner/runner-state.json"), JSON.stringify(durableRunnerState(identity, "ready"))); + const restore = vi.spyOn(runnerCheckpoint, "restoreNativeRunnerCheckpoint").mockImplementation(async (_db, binding, directory, sessionId) => { + expect(binding).toEqual({ companyId: prior.binding.companyId, issueId: prior.binding.issueId, runId: prior.binding.runId }); + expect(sessionId).toBe(prior.session.normalizedSessionId); + saved!.restoreSessionDirectory(directory); + return saved!; + }); + const remote = { kind: "remote", transport: "sandbox", remoteCwd: "/workspace", runner: { execute: vi.fn() } } as const; + state.createBackend.mockClear(); + state.createTransport.mockClear(); + try { + await expect(createRunnerdBackend({ db: priorNativeRunDb(prior, "running"), execution: current, + runnerInstanceId: "runner-next", runnerExecutionTarget: remote as never })) + .rejects.toThrow("runner_state_identity_mismatch: prior_owner_active"); + const projected = JSON.parse(await readFile(join(root, "control-plane/control-plane-state.json"), "utf8")); + expect(projected).toMatchObject({ schema: "paperclip.runner.durable.control-plane-state.v1", commands: [] }); + expect(projected.runAttachTemplate).toBeNull(); + const template = { provider: prior.provider, normalizedSessionId: identity.normalizedSessionId }; + controller.persistRunAttachTemplate(template); + controller.queueCommand("run.prepare", template); + await controller.flushCheckpoint(); + await writeFile(join(root, "runner/runner-state.json"), JSON.stringify(durableRunnerState(identity, "suspended"))); + const admitted = new Error("per-turn resume reached tool authority"); + state.toolAuthorityDefinitions.mockRejectedValueOnce(admitted); + await expect(createRunnerdBackend({ db: priorNativeRunDb(prior), execution: current, + runnerInstanceId: "runner-next", runnerExecutionTarget: remote as never })).rejects.toBe(admitted); + const refreshed = JSON.parse(await readFile(join(root, "control-plane/control-plane-state.json"), "utf8")); + expect(refreshed.runAttachTemplate).toEqual(template); + expect(refreshed.commands).toEqual([expect.objectContaining({ type: "run.prepare", payload: template })]); + expect(restore).toHaveBeenCalledTimes(2); + expect(state.createBackend).not.toHaveBeenCalled(); + expect(state.createTransport).not.toHaveBeenCalled(); + expect(existsSync(join(isolatedStateDirectory, "quarantine"))).toBe(false); + } finally { + state.toolAuthorityDefinitions.mockReset().mockResolvedValue([]); + restore.mockRestore(); + await controller.stop(); + } + }); + + it.each(["suspended", "backup", "codex", "file-projection", "legacy-file-no-checkpoint", "file-checkpoint-binding-error", + "missing-checkpoint", "checkpoint-binding-error", "foreign-task", "foreign-marker-session", + "missing-marker", "malformed-marker", "unknown-marker-schema", "active-owner", "foreign-prior-scope", "unsuspended-runner"] as const)( + "restores only an authorized externally persisted per-turn controller before ordinary resume: %s", async scenario => { + const prior = parseNativeExecutionInput({ + ...scopedNativeExecution("per-turn-external-resume"), + schema: "paperclip.native-execution-input.v5", + executionMode: "default", planningContext: null, runtimeContext: nativeRuntimeContextFixture(), + provider: scenario === "codex" ? { kind: "codex", model: null, approvalPolicy: "never" } + : { kind: "opencode", model: "openrouter/deepseek/deepseek-v4-flash-0731", permissionMode: "allow" }, + session: { ...execution.session, normalizedSessionId: "session-per-turn-external-resume", + driverKind: scenario === "codex" ? "codex_app_server" : "opencode_server", + lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } }, + }); + const current = { ...prior, binding: { ...prior.binding, runId: `${prior.binding.runId}-next`, + issueId: scenario === "foreign-task" ? "another-task" : prior.binding.issueId } }; + const identity = { runId: prior.binding.runId, normalizedSessionId: prior.session.normalizedSessionId!, + runnerInstanceId: "runner-per-turn-external-resume", environmentLeaseId: "lease-per-turn-external-resume", + turnId: "turn-prior", itemId: "item-prior" }; + const root = await initializeScopedRunnerRoot(prior, identity.runnerInstanceId); + const controller = new DurablePrpControlPlane({ stateDirectory: join(root, "control-plane"), identity, + expectedRunnerVersion: "per-turn-fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` }); + let saved: import("../../vendor/paperclip-runner/index.js").DurablePrpCheckpoint | undefined; + controller.setCheckpointPersistence(async checkpoint => { saved = checkpoint; }); + await controller.flushCheckpoint(); + expect(JSON.parse(await readFile(join(root, "control-plane/control-plane-state.json"), "utf8")).schema) + .toBe("paperclip.runner.external-control-plane-state.v1"); + if (["file-projection", "legacy-file-no-checkpoint", "file-checkpoint-binding-error"].includes(scenario)) + saved!.restoreSessionDirectory(root); + if (scenario === "foreign-marker-session" || scenario === "unknown-marker-schema") { + const path = join(root, "control-plane/control-plane-state.json"); + const marker = JSON.parse(await readFile(path, "utf8")); + if (scenario === "foreign-marker-session") marker.identity.normalizedSessionId = "another-session"; + else marker.schema = "paperclip.runner.external-control-plane-state.v999"; + await writeFile(path, JSON.stringify(marker)); + } + if (scenario === "missing-marker") await rm(join(root, "control-plane/control-plane-state.json")); + if (scenario === "malformed-marker") await writeFile(join(root, "control-plane/control-plane-state.json"), "{"); + if (scenario === "backup") { + const backupRoot = join(root, "failover-backups/current"); + for (const directory of resolveNativeHarnessPersistenceProfile(prior).directories) + await mkdir(join(backupRoot, directory.name), { recursive: true }); + await writeFile(join(backupRoot, "runner/runner-state.json"), JSON.stringify(durableRunnerState(identity, "suspended"))); + await writeFile(join(backupRoot, "manifest.json"), JSON.stringify(buildNativeHarnessBackupManifest({ + backupRoot, execution: prior, runnerInstanceId: identity.runnerInstanceId, + providerSessionIdentity: { providerSessionId: "provider-per-turn-resume", providerBackendSessionId: null, + providerSessionIdentity: null }, sourceProviderLeaseId: "sandbox-per-turn-resume", + }))); + } else { + await mkdir(join(root, "runner"), { recursive: true }); + await writeFile(join(root, "runner/runner-state.json"), JSON.stringify(durableRunnerState(identity, + scenario === "unsuspended-runner" ? "ready" : "suspended"))); + } + const markerBefore = scenario === "missing-marker" ? null + : await readFile(join(root, "control-plane/control-plane-state.json"), "utf8"); + const restore = vi.spyOn(runnerCheckpoint, "restoreNativeRunnerCheckpoint").mockImplementation(async (_db, binding, directory, sessionId) => { + expect(binding).toEqual({ companyId: prior.binding.companyId, issueId: current.binding.issueId, runId: prior.binding.runId }); + expect(sessionId).toBe(prior.session.normalizedSessionId); + if (["missing-checkpoint", "legacy-file-no-checkpoint"].includes(scenario) || binding.issueId !== prior.binding.issueId) return null; + if (["checkpoint-binding-error", "file-checkpoint-binding-error"].includes(scenario)) throw new Error("native_runner_checkpoint_binding_mismatch"); + saved!.restoreSessionDirectory(directory); + return saved!; + }); + state.createBackend.mockClear(); + state.createTransport.mockClear(); + const admitted = new Error("per-turn resume reached tool authority"); + state.toolAuthorityDefinitions.mockRejectedValueOnce(admitted); + try { + const persistedPrior = scenario === "foreign-prior-scope" + ? { ...prior, binding: { ...prior.binding, agentId: "another-agent" } } : prior; + const resume = createRunnerdBackend({ db: priorNativeRunDb(persistedPrior, scenario === "active-owner" ? "running" : "succeeded"), execution: current, + runnerInstanceId: "runner-next", runnerExecutionTarget: { kind: "remote", transport: "sandbox", + remoteCwd: "/workspace", runner: { execute: vi.fn() } } as never }); + if (["suspended", "backup", "codex", "file-projection", "legacy-file-no-checkpoint"].includes(scenario)) { + await expect(resume).rejects.toBe(admitted); + const restored = JSON.parse(await readFile(join(root, "control-plane/control-plane-state.json"), "utf8")); + expect(restored).toMatchObject({ schema: "paperclip.runner.durable.control-plane-state.v1", identity }); + } else { + const reason = ["missing-checkpoint", "foreign-task"].includes(scenario) ? "native_runner_checkpoint_missing" + : ["checkpoint-binding-error", "file-checkpoint-binding-error"].includes(scenario) ? "native_runner_checkpoint_binding_mismatch" + : scenario === "active-owner" ? "prior_owner_active" + : scenario === "foreign-prior-scope" ? "prior_owner_scope_mismatch" + : scenario === "unsuspended-runner" ? "prior_owner_terminal_state_indeterminate" : "durable_identity_unreadable"; + await expect(resume).rejects.toThrow(reason); + if (["missing-checkpoint", "checkpoint-binding-error", "file-checkpoint-binding-error", "foreign-task"].includes(scenario)) + expect(await readFile(join(root, "control-plane/control-plane-state.json"), "utf8")).toBe(markerBefore); + } + if (["foreign-marker-session", "missing-marker", "malformed-marker", "unknown-marker-schema"].includes(scenario)) expect(restore).not.toHaveBeenCalled(); + else expect(restore).toHaveBeenCalledOnce(); + expect(state.createBackend).not.toHaveBeenCalled(); + expect(state.createTransport).not.toHaveBeenCalled(); + expect(existsSync(join(isolatedStateDirectory, "quarantine"))).toBe( + ["foreign-marker-session", "missing-marker", "malformed-marker", "unknown-marker-schema", "foreign-prior-scope", "unsuspended-runner"].includes(scenario)); + } finally { + state.toolAuthorityDefinitions.mockReset().mockResolvedValue([]); + restore.mockRestore(); + await controller.stop(); + } + }); + it.each([ "quarantined", "large control plane", diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 5deac9f947c..e19a19689b0 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -4997,6 +4997,7 @@ async function migrateRunnerdStateRootForExecution(input: { }): Promise { const scoped = scopedRunnerdStateRoot(input.execution); await restoreWarmRunnerdController(input); + await restorePerTurnRunnerdController(input); // A crash can leave a source archive intent before/after its rename. Until // that exact maintenance is settled, absence of a canonical root is not // permission to bootstrap a replacement provider session. @@ -5684,6 +5685,24 @@ function idleWarmNativeSessionOwnerRunId(input: { : undefined; } +async function restorePerTurnRunnerdController(input: { + db: Db; execution: NativeExecutionInput; + runnerExecutionTarget?: AdapterExecutionTarget | null; + restartRecovery?: NativeRestartRecoveryClaim; +}): Promise { + if (input.restartRecovery || input.runnerExecutionTarget?.kind !== "remote" || + input.execution.session.lifecyclePolicy.mode !== "per_turn") return; + const root = scopedRunnerdStateRoot(input.execution); + const identity = readRunnerdDurableIdentity(root); + const reference = identity ?? readRunnerdDurableIdentity(root, true); + if (!durableIdentityMatchesSession(reference, input.execution)) return; + if (identity?.runId === input.execution.binding.runId) return; + const restored = await restoreNativeRunnerCheckpoint(input.db, + { companyId: input.execution.binding.companyId, issueId: input.execution.binding.issueId, runId: reference.runId }, + root, nativeSessionKey(input.execution)); + if (!restored && !identity) throw new Error("native_runner_checkpoint_missing"); +} + /** Materialize opaque controller state before ordinary warm admission reads it. * A retained owner or server checkpoint identifies the last completed run; * the original external marker can precede several in-process warm turns. */