From f1516e8009fd1dd7436ff6eab471cde83ea2f0a1 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 4 Nov 2026 23:24:34 -0600 Subject: [PATCH] fix(native): stop a session.open model rejection instead of a doomed bootstrap retry (#19190) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Runner runs start a provider session with the `session.open` command before any prompt > - A definite `session.open` rejection is treated as a pre-session failure and gets a bounded same-run bootstrap retry > - The runner also reports a closed classification. Two classifications say that the provider cannot run the selected model: `requested_model_unsupported` and `grok_model_mismatch` > - Paperclip ignored that classification. It scheduled a retry that must fail the same way. On a remote sandbox with a retained runner PID, the due retry entered restart recovery. Restart recovery has no checkpoint for a bootstrap retry, so it failed closed as `native_restart_recovery_blocked` / `ambiguous_remote_recovery_evidence` > - The task then stayed `blocked`. The recovery actions settled as no-replay holds, so no active action told anyone to change the model, and the parent task stalled too > - This pull request classifies those two definite rejections as the existing `native_provider_model_rejected` failure > - The benefit is a terminal failure on the first attempt, an accurate error, and an active board recovery action that names the next step: choose a supported model ## Linked Issues or Issue Description **What happened?** On a hosted staging instance, a Grok agent run failed at `session.open`. The runner log said `ACPX sidecar command session.open was rejected (retryable=false, classification=grok_model_mismatch)`. Paperclip recorded `recoveryMode: "bootstrap_retry"` and scheduled a same-run retry 30 seconds later. The run had a remote sandbox and a recorded sandbox PID, so the due retry went to restart recovery. Restart recovery found no checkpoint and no provider events and failed closed with `ambiguous_remote_recovery_evidence`. The run ended as `native_restart_recovery_blocked`. The task stayed `blocked` with no active recovery action for more than one hour. Its parent task stayed blocked on it. **What did you expect to happen?** The selected model cannot start, so a retry cannot succeed. The run should fail once with the model-rejection error. The task should show the existing "choose a model" action, owned by the board, until someone changes the model and retries. **Steps to reproduce** 1. Start a native run whose `session.open` command fails with `classification=grok_model_mismatch` (or `requested_model_unsupported`). 2. Read the `native_run_finalizations` row. Before this change it is `retryable_failure` with `recoveryMode: "bootstrap_retry"` and a `nextAttemptAt`. 3. On a remote sandbox run with a recorded PID, the due retry enters restart recovery and fails as `native_restart_recovery_blocked`. **Paperclip version or commit** `master` at `f5f5184f23`. Related: #18910 routes due remote retries with a retained PID to restart recovery. #18908 continues verified failed native preparation. Open PRs #19177, #19180, #19182 and #19184 change no-replay hold settlement and workspace hold release. They do not change this classification. This PR does not duplicate them. ## What Changed - `nativeSessionFailureSourceCode` (`server/src/services/native-runtime/native-session-executor.ts`) returns `native_provider_model_rejected` for a definite `session.open` failure or rejection whose runner classification is `requested_model_unsupported` or `grok_model_mismatch`. Other definite `session.open` rejections still return `provider_session_open_rejected` and keep the bounded startup retry. An indeterminate `session.open` outcome stays `native_session_interrupted`. - `native_provider_model_rejected` is already a permanent failure. The run stops on the first attempt with no `nextAttemptAt`, so it never enters the restart-recovery retry path. - When no provider session exists, the coordinator `nextAction` and the board recovery action `nextAction` say: choose a supported model for this agent, then retry the task. - The existing product paths for `native_provider_model_rejected` apply: the task presentation shows `choose_model`, and the recovery service treats it as `configuration_incomplete`. - `doc/execution-semantics.md` describes the rule under "Native provider model capacity". ## Verification Red before the fix (new test, unchanged production code): ```sh cd server && npx vitest run src/services/native-runtime/native-startup-retry.integration.test.ts # Tests 2 failed | 21 passed # failureCode: native_session_interrupted, phase: retryable_failure, recoveryMode: bootstrap_retry, # nextAction "Retry provider bootstrap on this same run ..." - the same record seen on staging ``` Green after the fix: ```sh cd server && npx vitest run src/services/native-runtime/native-startup-retry.integration.test.ts # Tests 23 passed cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts \ src/services/native-runtime/native-startup-retry.integration.test.ts src/services/task-interruption-presentation.test.ts # Test Files 3 passed, Tests 1186 passed pnpm --filter @paperclipai/server typecheck # exit 0 ``` The new integration cases use embedded PostgreSQL and the real executor failure path. For both classifications they check: - coordinator `terminal_failure`, `failureCode: native_provider_model_rejected`, `nextAttemptAt: null`, attempt 1 - run `failed` with `errorCode: native_provider_model_rejected` - task `blocked`, execution lock released - an `active` board recovery action with cause `native_provider_model_rejected`, return owner set to the agent, and the choose-model next action - `settleUnrecoverableExecutions` leaves the action `active`, so it does not become a silent no-replay hold - `claimNativeSessionResumptions` claims nothing, so there is no second executor The unit test also checks that `session_ensure_failed` keeps its bounded retry and that an indeterminate model rejection stays unclassified. A live Grok run with a mismatched model was not run. The failure record in the test matches the staging record field by field. ## Risks - Low risk. The change adds two closed runner classifications to an existing permanent failure code. Other `session.open` rejections keep their current retry. - A model mismatch that a retry could fix would now need an explicit retry. The runner already marks these rejections `retryable=false`. - This PR does not change restart recovery. A remote bootstrap retry with a retained sandbox PID for other `session.open` causes can still fail closed as `ambiguous_remote_recovery_evidence`. That needs a separate change with remote stop evidence. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code with tool use (shell, file edits, browser reads of the staging API). It did the investigation, code change, tests and PR text. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- doc/execution-semantics.md | 8 +++ .../native-session-executor.test.ts | 9 ++++ .../native-runtime/native-session-executor.ts | 13 +++++ .../native-startup-retry.integration.test.ts | 51 ++++++++++++++++++- 4 files changed, 80 insertions(+), 1 deletion(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 5eb3248d65e..4c4601dabc8 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -959,6 +959,14 @@ workspace finalization, and environment cleanup to settle. Model incompatibility usage-limit exhaustion, unknown failures, and unbound diagnostic text do not qualify as capacity failures. +A definite `session.open` rejection whose runner classification says the +provider cannot run the selected model (`requested_model_unsupported`, +`grok_model_mismatch`) is a model rejection, not a startup interruption. It +stops on its first attempt as `native_provider_model_rejected`, with no +same-run bootstrap retry, and leaves an active board recovery action that asks +for a supported model. Other `session.open` rejections keep the bounded +startup retry. An indeterminate `session.open` outcome stays unclassified. + ### Workspace scan failures before provider startup Workspace source preparation keeps its dedicated advisory-lock connection active diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index d0bb518a579..39a403fe553 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -17979,6 +17979,15 @@ it("classifies only a definite session.open outcome as a bounded pre-session fai .toBe("native_session_interrupted"); expect(nativeSessionFailureSourceCode(new Error(`PRP command turn.start failed: ${detail}`))) .toBe("native_session_interrupted"); + // A definite model rejection needs a model choice; an indeterminate one stays unknown. + const modelDetail = detail.replace("classification=session_ensure_failed", "classification=grok_model_mismatch"); + const modelCode = nativeSessionFailureSourceCode(new Error(`PRP command session.open failed: ${modelDetail}`)); + expect(modelCode).toBe("native_provider_model_rejected"); + expect(nativeSessionFailureDisposition(1, new Date(), modelCode)).toEqual({ + phase: "terminal_failure", failureCode: "native_provider_model_rejected", nextAttemptAt: null, + }); + expect(nativeSessionFailureSourceCode(new Error(`PRP command session.open indeterminate: ${modelDetail}`))) + .toBe("native_session_interrupted"); }); describe("HTTP Stop during remote runner staging", () => { it("does not dispatch after Stop and bounded bootstrap cleanup", async () => { diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index f55117d6b26..da9b82fbb41 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -7305,6 +7305,12 @@ export function nativeSessionFailureSourceCode( // before any prompt, so provider work cannot have started. An indeterminate // outcome stays generic. if (/\bPRP command session\.open (?:failed|rejected): /.test(message)) { + // The sidecar's closed classification says the provider cannot run the + // selected model. A same-run bootstrap retry would open the same session + // and fail the same way, so this needs a model choice, not a retry. + if (/classification=(?:requested_model_unsupported|grok_model_mismatch)\)/.test(message)) { + return "native_provider_model_rejected"; + } return "provider_session_open_rejected"; } return "native_session_interrupted"; @@ -7313,6 +7319,9 @@ export function nativeSessionFailureSourceCode( const NATIVE_PROVIDER_STATE_INCOMPATIBLE_ACTION = "The current runner cannot restore this task's saved provider session, so retrying this run cannot succeed. Retry the task to continue in a fresh provider session with the same workspace. The earlier provider conversation is not carried over."; +const NATIVE_STARTUP_MODEL_REJECTED_ACTION = + "The provider could not start the selected model before any provider work began. Choose a supported model for this agent, then retry the task. Automatic retries are stopped."; + const NATIVE_CLEANUP_OPERATOR_RECOVERY_MESSAGE = "Send a new message to continue after Paperclip verifies that the previous provider and its tools have stopped. If cleanup cannot be verified, inspect the run and its environment. Clearing a task session does not resolve this quarantine. Automatic retries are stopped."; @@ -10746,6 +10755,8 @@ async function executePaperclipNativeSessionWithinScope( ? "The computer provider rejected access. Restore the environment's credentials or permissions before retrying. Automatic retries are stopped." : sourceFailureCode === "native_provider_terminal_failed" ? "The provider session is permanently unusable. Verify stopped execution, completed actions, and task context before starting a linked continuation." + : sourceFailureCode === "native_provider_model_rejected" && !recoveryEvidence.providerSessionEstablished + ? NATIVE_STARTUP_MODEL_REJECTED_ACTION : sourceFailureCode === NATIVE_PROVIDER_STATE_INCOMPATIBLE ? NATIVE_PROVIDER_STATE_INCOMPATIBLE_ACTION : recoveryEvidence.recoveryMode === "ambiguous_state" @@ -10997,6 +11008,8 @@ async function executePaperclipNativeSessionWithinScope( ? "The computer provider rejected access. Restore the environment's credentials or permissions before retrying. Automatic retries are stopped." : sourceFailureCode === "native_provider_terminal_failed" ? "Verify that the failed provider stopped and reconcile its action outcomes. A linked continuation can proceed only after these checks succeed." + : sourceFailureCode === "native_provider_model_rejected" && !recoveryEvidence.providerSessionEstablished + ? NATIVE_STARTUP_MODEL_REJECTED_ACTION : sourceFailureCode === NATIVE_PROVIDER_STATE_INCOMPATIBLE ? NATIVE_PROVIDER_STATE_INCOMPATIBLE_ACTION : recoveryEvidence.recoveryMode === "ambiguous_state" diff --git a/server/src/services/native-runtime/native-startup-retry.integration.test.ts b/server/src/services/native-runtime/native-startup-retry.integration.test.ts index 6892000df69..df2eece61f7 100644 --- a/server/src/services/native-runtime/native-startup-retry.integration.test.ts +++ b/server/src/services/native-runtime/native-startup-retry.integration.test.ts @@ -38,6 +38,7 @@ import { currentNativeControllerIdentity } from "./native-restart-recovery.js"; import { loadNativeRunnerCheckpoint } from "./native-runner-checkpoint.js"; import { environmentRunnerIngressErrorData, JsonRpcCallError, PluginEnvironmentRunnerIngressError } from "@paperclipai/plugin-sdk"; import { accountRunCost } from "../run-cost-accounting.js"; +import { settleUnrecoverableExecutions } from "../execution-recovery-resolution.js"; type SessionOpenStatus = "failed" | "rejected" | "indeterminate"; @@ -45,6 +46,7 @@ type SessionOpenStatus = "failed" | "rejected" | "indeterminate"; function sessionOpenFailure(options: { status?: SessionOpenStatus; providerFailure?: Record; + classification?: string; } = {}) { const status = options.status ?? "failed"; const result = { @@ -52,7 +54,7 @@ function sessionOpenFailure(options: { code: "command_execution_failed", message: options.providerFailure ? "Provider request failed" - : "failed to start ACPX provider: ACPX sidecar command session.open was rejected (retryable=false, classification=session_ensure_failed)", + : `failed to start ACPX provider: ACPX sidecar command session.open was rejected (retryable=false, classification=${options.classification ?? "session_ensure_failed"})`, ...(options.providerFailure ? { providerFailure: options.providerFailure } : {}), }, status, @@ -476,6 +478,53 @@ describe("native provider startup retry", () => { expect(state.issue.status).toBe("in_progress"); }); + it.each(["grok_model_mismatch", "requested_model_unsupported"])( + "stops at a session.open %s rejection with a choose-model action instead of a doomed retry", + async (classification) => { + const seeded = await seedRun(await createAgent()); + await failStartup(seeded, sessionOpenFailure({ classification })); + + const state = await read(seeded); + // No automatic same-run bootstrap retry: the selected model cannot start. + expect(state.coordinator).toMatchObject({ + phase: "terminal_failure", + recoveryState: "blocked", + failureCode: "native_provider_model_rejected", + attempt: 1, + nextAttemptAt: null, + failureDetail: expect.objectContaining({ + originalFailureCode: "native_provider_model_rejected", + providerSessionEstablished: false, + providerEventsExist: false, + checkpointExists: false, + }), + }); + expect(String(state.coordinator.failureDetail?.nextAction)).toContain("Choose a supported model"); + expect(state.run).toMatchObject({ status: "failed", errorCode: "native_provider_model_rejected" }); + expect(state.issue.status).toBe("blocked"); + expect(state.issue.executionRunId).toBeNull(); + // A named, durable board action owns the next step. + expect(state.action).toMatchObject({ + status: "active", + ownerType: "board", + cause: "native_provider_model_rejected", + returnOwnerAgentId: seeded.agentId, + wakePolicy: null, + }); + expect(state.action!.nextAction).toContain("Choose a supported model"); + // The no-replay settler must not fold this owned action into a silent hold. + await settleUnrecoverableExecutions(db); + expect((await read(seeded)).action).toMatchObject({ id: state.action!.id, status: "active" }); + expect(await claimNativeSessionResumptions({ + db, + runnerInstanceId: "startup-retry-test", + runIds: [seeded.runId], + now: new Date(Date.now() + 60 * 60_000), + deferIneligible: false, + })).toEqual([]); + }, + ); + it("blocks with a clear reason once the bounded startup budget is exhausted", async () => { const seeded = await seedRun(await createAgent()); await failStartup(seeded, sessionOpenFailure());