From 08ff0828d6e3b3ca7cee734e65976331d6e40562 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 11 Nov 2026 00:12:41 -0600 Subject: [PATCH] fix(native): offer Retry at once when the Runner refuses a turn start on a closed session (#20729) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native ACPX run (for example Claude) starts each turn with `turn.start`. When a start is cut off, the Runner reads the sidecar's dispatch journal (#20675). > - If the prompt never left, the Runner keeps the session reopenable, and the same-run resume sends the prompt once. If the prompt may have left, the Runner closes the session, because a resend could duplicate it. > - A closed session cannot reopen for the same run. Every same-run resume got `ACPX session is not open`. The server treated that as a generic interruption and spent three attempts (about 95 s) before the board offered Retry. > - This pull request gives that refusal a closed class in the Runner and a permanent, named failure code in the server. > - The benefit is that the board offers a working Retry on the first refusal, while every recoverable case keeps recovering by itself. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T011704-s9107`, task t05 (local, claude, workload C1, fault TM2). A freeze cut off a turn start, and the Runner saved the session `closed`. Three same-run resumes each failed with `PRP command turn.start failed: … "message":"ACPX session is not open"`. Recovery stopped with `native_session_retry_exhausted` at 01:25:14, and the board then offered Retry. The Retry run succeeded. **Expected behavior** When the Runner proves the session is closed for this run, the run stops at the first refusal with a named failure, and the board offers a Retry that works. A start that the Runner proved never reached the provider still recovers by itself. **Steps to reproduce** 1. `cd server && npx vitest run src/services/native-runtime/native-closed-session-retry.integration.test.ts` 2. On master, "offers Retry on the first refusal, and Retry starts a new run" fails: the coordinator is `retryable_failure` / `native_session_interrupted` (a same-run retry in 30 s). Refs #20675, #20720 ## What Changed - Runner (`durable/mod.rs`, `acpx_provider_backend.rs`): new closed class `provider_session_closed`. The Runner sets it on these refusals: - a turn start that saves the session `closed` (`ACPX turn start failed closed: …`), but not the rejected-checkpoint case, which stays `prepared` and reopens; - `ACPX session is not open` when the saved lifecycle is `closed` for this run; - the saved turn-end-restart refusal for this run (asserted in `acpx_turn_end_restart`). A session that waits for provider-exit proof can reopen after a restart. Its refusal gets no class. - Server (`native-session-executor.ts`): a `turn.start` failure with that class maps to the new permanent code `native_provider_session_closed`. The server reads the class from the authenticated command result, and from the command-failure text if the error was rewrapped. `nativeSessionFailureDisposition` treats the code as permanent, so the first refusal ends the run as `terminal_failure` with a board owner and a plain next action. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` states the class and the older-Runner behavior. - Tests: - `native-closed-session-retry.integration.test.ts` (new, embedded Postgres) drives `executePaperclipNativeSession` with the Runner refusal. On the first refusal, the board shows `canRetry: true`. `requestTaskRetry` is accepted, and `resumeTaskRetry` starts exactly one new run with a fresh provider session. A start proved undispatched stays `retryable_failure`. - Unit tests in `native-session-executor.test.ts`: both closed refusals, a rewrapped error, an undispatched start, an unclassified older-Runner refusal, and a class on another command. - Rust `acpx_cut_turn_start`: a start cut off before dispatch has no class. A start cut off after dispatch, and the later refusal, carry `provider_session_closed`. ## Verification - Red on master, green here: step 1 above. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts`: 1358/1358 pass. - `cargo test -p paperclip-runner-core --test acpx_park_preparation` (26 pass), `--lib acpx` (94 pass), `--test acpx_provider_turns --test acpx_provider_session --test acpx_warm_attachment` (all pass), `cargo fmt --check` clean. - `npx tsc --noEmit -p server/tsconfig.json`: clean. - Not run: a chaos wave. ## Risks - This changes the Rust Runner. Daytona needs a new runner image before remote runs get the class. Until then, an older Runner sends the refusal without the class, and the server keeps today's bounded same-run retries. That path is unchanged and ends with a working Retry. The server does not trust the text alone, because an older Runner sends the same text for a session that can still reopen. - The class is set only where the Runner's saved state proves that no later start of this run can reopen the session. A false class would cost one automatic resume and show Retry earlier. It never replays provider work. - No persisted shape changes. The class is an optional field of a failed command result. Old saved state resumes unchanged. Nothing new runs before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude --- .../paperclip-runner/docs/durable-recovery.md | 8 ++ .../runner-core/src/acpx_provider_backend.rs | 31 ++++- .../crates/runner-core/src/durable/mod.rs | 6 + .../tests/support/acpx_cut_turn_start.rs | 31 +++-- .../tests/support/acpx_turn_end_restart.rs | 10 +- ...e-closed-session-retry.integration.test.ts | 106 ++++++++++++++++++ .../native-session-executor.test.ts | 47 ++++++++ .../native-runtime/native-session-executor.ts | 36 +++++- 8 files changed, 255 insertions(+), 20 deletions(-) create mode 100644 server/src/services/native-runtime/native-closed-session-retry.integration.test.ts diff --git a/packages/paperclip-runner/docs/durable-recovery.md b/packages/paperclip-runner/docs/durable-recovery.md index 9b43063fd5c..4de69c15a29 100644 --- a/packages/paperclip-runner/docs/durable-recovery.md +++ b/packages/paperclip-runner/docs/durable-recovery.md @@ -143,6 +143,14 @@ which proves that the sidecar and the provider are gone. no file (an older sidecar) proves nothing. The earlier behavior applies unchanged: a failed start closes the session, and a killed Runner uses lost-turn recovery. +- A session this run closed cannot reopen for the same run. The failed start, + and every later `turn.start` of the run, fail with the closed class + `provider_session_closed` in the command result. The controller records + `native_provider_session_closed`, stops same-run retries, and the board offers + Retry at once. Retry starts a new run with a fresh provider conversation and + the saved task context. An older Runner sends the same text without the class; the + controller then keeps its bounded same-run retries, because the same text + also covers a session that can reopen after a restart. Until a proof settles it, the saved session stays `turn_starting`, so a Runner killed during the proof restores through the same check. The cut-off command is diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs index f15147c4664..7b610e1572f 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs @@ -28,7 +28,8 @@ use crate::durable::QualifiedLaunchArtifact; use crate::durable::{ create_private_temporary_file, open_private_regular_file, verify_private_directory, AcpxLaunchProfile, Command, CommandExecution, CommandExecutor, DurableRunnerConfig, - DurableRunnerError, EventPriority, PolledEvent, PROVIDER_STATE_INCOMPATIBLE, + DurableRunnerError, EventPriority, PolledEvent, PROVIDER_SESSION_CLOSED, + PROVIDER_STATE_INCOMPATIBLE, }; use crate::generated_acpx_sidecar_contract::GeneratedAcpxSidecarCommand; use crate::process_supervisor::{VerifiedProcessArgument, VerifiedProcessLaunch}; @@ -2061,7 +2062,8 @@ impl AcpxCommandExecutor { DurableRunnerError::invalid(format!("ACPX turn start failed closed: {error}")) } None => DurableRunnerError::invalid("ACPX session is not open"), - }); + } + .with_classification(PROVIDER_SESSION_CLOSED)); } let reopenable = self.session.is_none() && self.state.as_ref().is_some_and(|state| { @@ -2128,7 +2130,17 @@ impl AcpxCommandExecutor { self.retain_buffered_provider_events()?; } if self.session.is_none() { - return Err(DurableRunnerError::invalid("ACPX session is not open")); + let error = DurableRunnerError::invalid("ACPX session is not open"); + // Only a session this run closed is permanently closed for it. A + // session awaiting provider-exit proof can reopen after a restart. + let closed_for_run = self.state.as_ref().is_some_and(|state| { + state.lifecycle == "closed" && state.descriptor.run_id == self.context.run_id + }); + return Err(if closed_for_run { + error.with_classification(PROVIDER_SESSION_CLOSED) + } else { + error + }); } let (provider_process_will_be_replaced, previous_process_id) = self .session @@ -2206,6 +2218,7 @@ impl AcpxCommandExecutor { } .to_owned(); state.provider_exit_unconfirmed = rejected_checkpoint.is_some(); + let rejected_checkpoint_reopenable = rejected_checkpoint.is_some(); state.active_turn_id = None; self.context.provider_turn_id = None; self.session = None; @@ -2225,9 +2238,15 @@ impl AcpxCommandExecutor { return Err(error); } } - return Err(DurableRunnerError::invalid(format!( - "ACPX turn start failed closed: {error}" - ))); + let failure = + DurableRunnerError::invalid(format!("ACPX turn start failed closed: {error}")); + // A rejected checkpoint stays `prepared` and reopens on the next + // start. A `closed` session cannot reopen for this run. + return Err(if rejected_checkpoint_reopenable { + failure + } else { + failure.with_classification(PROVIDER_SESSION_CLOSED) + }); } // Copy numeric timing fields only. Sidecar contents never enter the receipt. let timing = if measure { diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/durable/mod.rs b/packages/paperclip-runner/runner/crates/runner-core/src/durable/mod.rs index b58e953a2a4..adaca2067c3 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/durable/mod.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/durable/mod.rs @@ -51,6 +51,12 @@ pub const PROVIDER_ATTACH_REFUSED: &str = "provider_attach_refused"; /// recorded result. Replacing the session could repeat or lose that action. pub const PROVIDER_SIDE_EFFECT_PENDING: &str = "provider_side_effect_pending"; +/// The runner closed this run's provider session after a turn start whose +/// prompt may have reached the provider. No later `turn.start` of the same run +/// can reopen it; only a new run's attachment can. Resending could duplicate +/// the prompt, so the controller stops same-run retries and offers Retry. +pub const PROVIDER_SESSION_CLOSED: &str = "provider_session_closed"; + #[derive(Clone, Debug, PartialEq, Eq)] pub struct DurableRunnerError(String, Option, Option<&'static str>); diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/support/acpx_cut_turn_start.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/support/acpx_cut_turn_start.rs index 98de3ed7735..5c0f5dccb8f 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/support/acpx_cut_turn_start.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/support/acpx_cut_turn_start.rs @@ -3,6 +3,7 @@ //! journal decides whether the prompt could have reached the provider. use super::*; +use paperclip_runner_core::durable::PROVIDER_SESSION_CLOSED; const JOURNAL: &str = "paperclip-turn-dispatch.jsonl"; @@ -27,8 +28,14 @@ fn kill(pid: &Value) { } /// Starts `turn` against a sidecar that freezes at `fault`, then kills the -/// sidecar once the journal shows `phases`, and returns the start's error. -fn cut_turn_start(harness: &mut Harness, turn: &str, fault: &str, phases: &[&str]) -> String { +/// sidecar once the journal shows `phases`, and returns the start's error and +/// its closed classification. +fn cut_turn_start( + harness: &mut Harness, + turn: &str, + fault: &str, + phases: &[&str], +) -> (String, Option<&'static str>) { harness.fixture.fault(fault); let sidecar = harness.fixture.process_starts().last().unwrap()["processStart"].clone(); let root = harness.fixture.0.clone(); @@ -68,21 +75,23 @@ fn cut_turn_start(harness: &mut Harness, turn: &str, fault: &str, phases: &[&str "turn.start", json!({"turnId": turn, "text": "Write the brief."}), )) - .unwrap_err() - .to_string(); + .unwrap_err(); killer.join().unwrap(); harness.fixture.fault(""); - error + (error.to_string(), error.classification()) } #[test] fn a_start_cut_off_before_the_prompt_left_reopens_and_sends_the_prompt_once() { let mut harness = Harness::new(); - let error = cut_turn_start(&mut harness, "cut", "hang-before-dispatch", &["received"]); + let (error, classification) = + cut_turn_start(&mut harness, "cut", "hang-before-dispatch", &["received"]); assert!( error.contains("before its prompt reached the provider"), "{error}" ); + // The run recovers by itself: the controller must keep retrying it. + assert_eq!(classification, None); let saved = harness.fixture.state(); assert_eq!(saved["lifecycle"], "prepared", "{saved}"); assert_eq!(saved["activeTurnId"], Value::Null); @@ -107,13 +116,15 @@ fn a_start_cut_off_before_the_prompt_left_reopens_and_sends_the_prompt_once() { #[test] fn a_start_cut_off_after_the_prompt_left_stays_closed_and_never_resends() { let mut harness = Harness::new(); - let error = cut_turn_start( + let (error, classification) = cut_turn_start( &mut harness, "cut", "hang-after-dispatch", &["received", "dispatched"], ); assert!(error.contains("failed closed"), "{error}"); + // No later start of this run can reopen it: the controller offers Retry at once. + assert_eq!(classification, Some(PROVIDER_SESSION_CLOSED)); assert_eq!(harness.fixture.state()["lifecycle"], "closed"); let starts = harness.fixture.process_starts().len(); harness.sequence += 1; @@ -124,9 +135,9 @@ fn a_start_cut_off_after_the_prompt_left_stays_closed_and_never_resends() { "turn.start", json!({"turnId": "resent", "text": "Write the brief."}), )) - .unwrap_err() - .to_string(); - assert_eq!(refused, "ACPX session is not open"); + .unwrap_err(); + assert_eq!(refused.to_string(), "ACPX session is not open"); + assert_eq!(refused.classification(), Some(PROVIDER_SESSION_CLOSED)); assert_eq!(harness.fixture.process_starts().len(), starts); } diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/support/acpx_turn_end_restart.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/support/acpx_turn_end_restart.rs index 6671c6b3106..d0aaa7df97f 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/support/acpx_turn_end_restart.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/support/acpx_turn_end_restart.rs @@ -437,9 +437,13 @@ fn a_runner_restart_in_the_same_run_keeps_the_failed_turn_end_restart_closed() { "turn.start", json!({"turnId": "second", "text": "Complete the work."}), )) - .unwrap_err() - .to_string(); - assert_eq!(failed, "ACPX session is not open"); + .unwrap_err(); + assert_eq!(failed.to_string(), "ACPX session is not open"); + // No same-run start can reopen it: the controller offers Retry at once. + assert_eq!( + failed.classification(), + Some(paperclip_runner_core::durable::PROVIDER_SESSION_CLOSED) + ); assert_eq!(harness.fixture.process_starts().len(), starts); // Neither session.open nor a same-run attach reopens it. let reopened = harness diff --git a/server/src/services/native-runtime/native-closed-session-retry.integration.test.ts b/server/src/services/native-runtime/native-closed-session-retry.integration.test.ts new file mode 100644 index 00000000000..07dc92bdde8 --- /dev/null +++ b/server/src/services/native-runtime/native-closed-session-retry.integration.test.ts @@ -0,0 +1,106 @@ +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { agents, companies, companyMemberships, completionContracts, createDb, heartbeatRuns, issues, nativeRunFinalizations } from "@paperclipai/db"; +import type { NativeExecutionInputV1 } from "@paperclipai/paperclip-runner"; +import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; + +const provider = vi.hoisted(() => ({ execute: vi.fn() })); +vi.mock("../../vendor/paperclip-runner/index.js", async (importOriginal) => ({ + ...(await importOriginal()), + executeNativeSession: provider.execute, +})); + +import { executePaperclipNativeSession, NATIVE_PROVIDER_SESSION_CLOSED } from "./native-session-executor.js"; +import { buildNativeCompletionContract } from "./completion-contracts.js"; +import { prepareNativeHeartbeatRun } from "./prepare-native-run.js"; +import { heartbeatService } from "../heartbeat.js"; +import { requestTaskRetry } from "../task-retry.js"; +import { taskInterruptionPresentation } from "../task-interruption-presentation.js"; + +// Chaos w-20261111T011704-s9107/t05 (claude, TM2): a cut-off ACPX turn start +// left this run's session closed with the prompt possibly sent. Every same-run +// resume got "ACPX session is not open"; the board offered Retry only after +// three attempts (about 95 s). The Runner now classifies that refusal, and +// Retry starts a new run with a fresh provider session. +describe("native turn start refused on a closed provider session", () => { + let temporary: Awaited>; + let db: ReturnType; + const companyId = randomUUID(); + const agentId = randomUUID(); + + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("native-closed-session-"); + db = createDb(temporary.connectionString); + await db.insert(companies).values({ id: companyId, name: "Closed session", issuePrefix: "NCS" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Claude agent", status: "active", adapterType: "paperclip_runner", + adapterConfig: { provider: "acpx", agent: "claude", model: "claude-test" } }); + await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: "board-user", status: "active", membershipRole: "member" }); + }); + afterAll(async () => { await temporary?.cleanup(); }); + + const refusal = (result: Record) => { + const commandResult = { commandId: "command_lab_1", commandType: "turn.start", controllerSeq: 10, result, status: "failed" }; + return Object.defineProperty(new Error(`PRP command turn.start failed: ${JSON.stringify(commandResult)}`), + "runnerCommandResult", { value: commandResult, enumerable: false }); + }; + + async function refuseTurnStart(failure: Error) { + const issueId = randomUUID(); + await db.insert(issues).values({ id: issueId, companyId, title: "Write the brief", status: "in_progress", workMode: "standard", + assigneeAgentId: agentId, createdByUserId: "board-user", responsibleUserId: "board-user" }); + const [issue] = await db.select().from(issues).where(eq(issues.id, issueId)); + const runId = randomUUID(); + const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native", + nativeIssueId: issueId, responsibleUserId: "board-user", invocationSource: "automation", triggerDetail: "system", + contextSnapshot: { issueId } }).returning(); + await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)); + const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId: randomUUID() }); + const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issueId)); + await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed" }); + const execution: NativeExecutionInputV1 = { + schema: "paperclip.native-execution-input.v1", + binding: { companyId, runId, issueId, agentId, executionWorkspaceId: runId }, + provider: { kind: "opencode", model: "openrouter/test-model" }, + task: { identifier: issueId, title: issue.title, description: null, prompt: "Write the brief.", workMode: "standard" }, + workspace: { cwd: "/tmp", repoUrl: null, repoRef: null, branchName: null }, + session: { normalizedSessionId: native.normalizedSessionId, driverKind: "opencode_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } }, + completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1", + contract: buildNativeCompletionContract(issue, { revision: contract.revision }) }, + interactionResponses: [], credentialBindings: [], + }; + // An established provider session: same-run resume is the ordinary recovery. + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: execution, + sessionCheckpoint: { backendKind: "mock", providerSessionId: "claude-session" } } }).where(eq(heartbeatRuns.id, runId)); + provider.execute.mockReset().mockRejectedValue(failure); + await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, backend: { + descriptor: async () => { throw new Error("Unexpected provider call"); }, + openSession: async () => { throw new Error("Unexpected provider call"); }, + } }).catch(() => undefined); + const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)); + return { issueId, runId, coordinator }; + } + + it("offers Retry on the first refusal, and Retry starts a new run", async () => { + const { issueId, runId, coordinator } = await refuseTurnStart(refusal({ code: "command_execution_failed", + message: "ACPX session is not open", classification: "provider_session_closed" })); + expect(coordinator).toMatchObject({ phase: "terminal_failure", failureCode: NATIVE_PROVIDER_SESSION_CLOSED, nextAttemptAt: null }); + const [issue] = await db.select().from(issues).where(eq(issues.id, issueId)); + await expect(taskInterruptionPresentation(db, issue)).resolves.toMatchObject({ state: "failed", canRetry: true }); + const receipt = await requestTaskRetry({ db, companyId, issueId, actorId: "board-user", actorSource: "session", + idempotencyKey: randomUUID(), observedRunId: runId }); + expect(receipt).toMatchObject({ accepted: true, stale: false }); + const heartbeat = heartbeatService(db); + await heartbeat.resumeTaskRetry(companyId, receipt.requestId!); + const retried = (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, agentId))) + .filter(r => r.contextSnapshot?.issueId === issueId && r.contextSnapshot?.taskRetryRequestId === receipt.requestId); + expect(retried).toHaveLength(1); + await heartbeat.drainActiveRunExecutions(); + }, 120_000); + + it("keeps a same-run retry when the Runner proved the prompt never left", async () => { + const { coordinator } = await refuseTurnStart(refusal({ code: "command_execution_failed", + message: "ACPX turn start failed before its prompt reached the provider: ACPX sidecar request timed out at turn.start" })); + expect(coordinator).toMatchObject({ phase: "retryable_failure", failureCode: "native_session_interrupted" }); + }, 120_000); +}); diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 29669b434ca..e37262c5b72 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -429,6 +429,7 @@ import { retainedNativeCleanupJournalMatches, nativeProviderUsageLimitFromEvent, nativeSessionFailureSourceCode, + NATIVE_PROVIDER_SESSION_CLOSED, nativeSessionRecoveryProjection, nativeGovernedWaitResult, nativeConversationReplyResult, @@ -13976,6 +13977,52 @@ describe("native session bounded recovery", () => { expect(nativeSessionFailureSourceCode(new Error(failure.message))).toBe("native_session_interrupted"); }); + // Chaos w-20261111T011704-s9107/t05: after a cut-off ACPX turn start the + // Runner saved the session closed; three same-run resumes each got the same + // refusal before the board offered Retry. + describe("a turn start the Runner refused because this run's session is closed", () => { + const turnStartFailure = (result: Record) => { + const commandResult = { commandId: "command_lab_1", commandType: "turn.start", controllerSeq: 10, result, status: "failed" }; + return Object.defineProperty(new Error(`PRP command turn.start failed: ${JSON.stringify(commandResult)}`), + "runnerCommandResult", { value: commandResult, enumerable: false }); + }; + const closed = (message: string) => turnStartFailure({ code: "command_execution_failed", message, + classification: "provider_session_closed" }); + + it.each(["ACPX session is not open", "ACPX turn start failed closed: ACPX sidecar request timed out at turn.start"])( + "stops on the first refusal and hands the run to the board: %s", (message) => { + const code = nativeSessionFailureSourceCode(closed(message)); + expect(code).toBe(NATIVE_PROVIDER_SESSION_CLOSED); + const disposition = nativeSessionFailureDisposition(1, new Date(), code); + expect(disposition).toEqual({ phase: "terminal_failure", failureCode: NATIVE_PROVIDER_SESSION_CLOSED, nextAttemptAt: null }); + expect(nativeSessionRecoveryProjection({ ...disposition, agentId: "agent" })).toMatchObject({ + issueStatus: "blocked", recoveryOwner: { kind: "board" }, + }); + // A rewrapped failure keeps the class through its authenticated text. + expect(nativeSessionFailureSourceCode(new Error(closed(message).message))).toBe(NATIVE_PROVIDER_SESSION_CLOSED); + expect(nativeSessionFailureSourceCode(Object.assign(new Error("native session failed"), { cause: closed(message) }))) + .toBe(NATIVE_PROVIDER_SESSION_CLOSED); + }); + + it("keeps retrying a start the Runner proved never reached the provider", () => { + const failure = turnStartFailure({ code: "command_execution_failed", + message: "ACPX turn start failed before its prompt reached the provider: ACPX sidecar request timed out at turn.start" }); + expect(nativeSessionFailureSourceCode(failure)).toBe("native_session_interrupted"); + expect(nativeSessionFailureDisposition(1, new Date(), "native_session_interrupted").phase).toBe("retryable_failure"); + }); + + it("keeps today's same-run retries for an unclassified refusal from an older Runner", () => { + // An older Runner sends the same text for a session that can reopen after + // a restart (provider exit not yet proven), so the text alone proves nothing. + const failure = turnStartFailure({ code: "command_execution_failed", message: "ACPX session is not open" }); + expect(nativeSessionFailureSourceCode(failure)).toBe("native_session_interrupted"); + // Another command's class is not a turn-start refusal. + const other = Object.defineProperty(new Error("PRP command session.open failed"), "runnerCommandResult", + { value: { commandType: "session.open", result: { classification: "provider_session_closed" } }, enumerable: false }); + expect(nativeSessionFailureSourceCode(other)).not.toBe(NATIVE_PROVIDER_SESSION_CLOSED); + }); + }); + it("stops same-run retries for an operator while a saved side-effecting call has no result", () => { const failure = new NativeProviderSideEffectPendingError( "Error: native_runner_source_semantic_reconciliation_required", diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index ac9a832446a..fff7e2b6e6f 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -8312,7 +8312,8 @@ export function nativeSessionFailureDisposition( sourceFailureCode === NATIVE_CURRENT_WAKE_COMMENTS_CHANGED_AFTER_READ || sourceFailureCode === "native_session_cleanup_quarantined" || sourceFailureCode === "native_adopted_runner_authentication_timeout" || - sourceFailureCode === "native_provider_usage_limit"; + sourceFailureCode === "native_provider_usage_limit" || + sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED; const permissionDeclined = sourceFailureCode === "native_permission_declined"; const exhausted = permanentFailure || permissionDeclined || attempt >= 3; return { @@ -8353,6 +8354,33 @@ export function nativeSessionRecoveryProjection(input: { }; } +/** + * The Runner closed this run's provider session after a turn start whose prompt + * may have reached the provider. Every later same-run `turn.start` is refused + * the same way, and resending could duplicate the prompt, so the run stops and + * the board offers Retry (a new run with a fresh provider session). A start the + * Runner proved never dispatched is not classified and still retries. + */ +export const NATIVE_PROVIDER_SESSION_CLOSED = "native_provider_session_closed"; +export const NATIVE_PROVIDER_SESSION_CLOSED_ACTION = + "The provider session closed after a turn start that may have reached the provider, and this run cannot reopen it. Retry starts a new run with a fresh provider conversation and the saved task context."; +const RUNNER_PROVIDER_SESSION_CLOSED = "provider_session_closed"; + +/** Reads the Runner's closed class from an authenticated turn.start failure. */ +function runnerClosedTurnStartSession(error: unknown, depth = 0): boolean { + if (!error || typeof error !== "object" || depth > 5) return false; + const value = error as Record; + const result = record(value.runnerCommandResult); + if (result.commandType === "turn.start" && record(result.result).classification === RUNNER_PROVIDER_SESSION_CLOSED) { + return true; + } + // The structured result can be lost when the failure is rewrapped; the + // authenticated command failure text carries the same closed class. + const message = error instanceof Error ? error.message : ""; + if (/\bPRP command turn\.start failed: \{.*"classification":"provider_session_closed"/.test(message)) return true; + return runnerClosedTurnStartSession(value.cause, depth + 1); +} + export function nativeSessionFailureSourceCode( error: unknown, ): @@ -8363,6 +8391,7 @@ export function nativeSessionFailureSourceCode( | typeof NATIVE_PROVIDER_SIDE_EFFECT_PENDING | "native_provider_approval_required" | "native_provider_usage_limit" + | typeof NATIVE_PROVIDER_SESSION_CLOSED | "native_session_cleanup_quarantined" | "native_adopted_runner_authentication_timeout" | "runner_remote_provider_artifact_incompatible" @@ -8424,6 +8453,7 @@ export function nativeSessionFailureSourceCode( return error.failure.httpStatus === 401 || error.failure.httpStatus === 403 ? "runner_ingress_access_rejected" : "runner_ingress_acquisition_failed"; } + if (runnerClosedTurnStartSession(error)) return NATIVE_PROVIDER_SESSION_CLOSED; const message = error instanceof Error ? error.message : String(error); if (/classification=runtime_package_missing\)/.test(message)) return "native_provider_runtime_missing"; @@ -12261,6 +12291,8 @@ async function executePaperclipNativeSessionAttempt( ? "Provider permission was declined. Inspect the task and give explicit direction before starting further provider work. Automatic recovery is stopped." : sourceFailureCode === "native_provider_approval_required" ? "Approval required. Review the operation and update the agent's permission setting before retrying. This runner has no interactive approval handler." + : sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED + ? NATIVE_PROVIDER_SESSION_CLOSED_ACTION : sourceFailureCode === "native_session_cleanup_quarantined" ? nativeCleanupRefusalAction(completedTurnCleanup) : sourceFailureCode === "runner_remote_provider_verification_unavailable" @@ -12530,6 +12562,8 @@ async function executePaperclipNativeSessionAttempt( ? "Provider permission was declined. Inspect the task and give explicit direction before starting further provider work. Automatic recovery is stopped." : sourceFailureCode === "native_provider_approval_required" ? "Approval required. Review the operation and update the agent's permission setting before retrying. This runner has no interactive approval handler." + : sourceFailureCode === NATIVE_PROVIDER_SESSION_CLOSED + ? NATIVE_PROVIDER_SESSION_CLOSED_ACTION : sourceFailureCode === "native_session_cleanup_quarantined" ? nativeCleanupRefusalAction(completedTurnCleanup) : sourceFailureCode === "runner_ingress_access_rejected"