From 82abb3ea0458c8e784e5ab793f83ed3eb8f9f808 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 10 Nov 2026 03:11:10 -0600 Subject: [PATCH] fix(native): classify a Codex API-key unknown model as a model rejection (#20488) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runner agents run on a provider (here Codex) with a model the user saves on the agent > - When the provider rejects that model, Paperclip should classify the run as `native_provider_model_rejected`, ask the user to choose a model, and let Retry finish the task > - Paperclip recognized only the ChatGPT-account rejection (a 400 JSON envelope). With API-key auth, Codex returns a different terminal error for an unknown model > - So the run failed as a generic `adapter_failed` ("Native session failed"), and recovery opened a reconciliation incident ("Recovery needed") for a turn that never did any work > - This pull request recognizes the real API-key shape, bound to the configured model, and names the rejected model on the task > - The benefit is that the task asks for a model choice instead of a reconciliation, and Retry completes the task after the user picks a valid model ## Linked Issues or Issue Description **What happened?** The restart-chaos config fault CF3 `model_id` saves a Codex agent with the model `chaos-missing-model-`. The run stays "running" for about 60 seconds, then fails with `errorCode: adapter_failed`, "Native session failed". The task shows "Recovery needed", cause `native_continuation_requires_reconciliation`. The grader fails as `CF3/model_id:actionable_message` (waves `w-20261110T061914-s206-cf` t01 and `w-20261110T053831-s205-cf` t02). Root cause: the wave runs Codex with an OpenAI API key. I drove the bundled `codex app-server` (0.160.0) directly with an unknown model: - With API-key auth, Codex retries the Responses API 404 ten times (about 59 s), then fails the turn with `{"message":"unexpected status 404 Not Found: The model \`\` does not exist or you do not have access to it., url: https://api.openai.com/v1/responses, cf-ray: …, request id: req_…","codexErrorInfo":{"httpConnectionFailed":{"httpStatusCode":404}}}`. - With ChatGPT auth, the turn fails in about 3 s with `message` = `{"type":"error","status":400,"error":{"type":"invalid_request_error","message":"The '' model is not supported when using Codex with a ChatGPT account.","code":"model_not_found"}}`. Paperclip already classified this one. `createNativeProviderFailureObservation` parsed only the JSON envelope, and the runner's provider-failure category for an HTTP 404 is `unknown`. The terminal therefore stayed unclassified. **Expected behavior** The run is `native_provider_model_rejected`, the task shows `userAction: "choose_model"` and names the model, no reconciliation incident opens, and Retry completes the task after the user picks a valid model. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/config-wave.ts --faults CF3 --adapters codex --references model_id --tasks 1 --concurrency 1 --restarts K2 --seed 206` on `master` (with `OPENAI_API_KEY` set). **Paperclip version or commit** `master` at 5ab98b0. ## What Changed - `server/src/services/native-runtime/native-provider-failure.ts`: the Codex observation recognizes the API-key terminal shape (`codexErrorInfo` `httpConnectionFailed` or `responseStreamDisconnected` with `httpStatusCode: 404`, and the message prefix for the configured model only). It classifies it as `native_provider_model_rejected` with a new closed diagnostic `NATIVE_MODEL_NOT_FOUND_DIAGNOSTIC` (`codex`, `model_not_found`, `404`, `api_key`). No provider text or model name is stored in the diagnostic. A recoverable turn, another model's 404, or another status is not classified. - `readNativeModelRejectionDiagnostic` accepts both closed diagnostics, so run failure diagnostics show it. - `TaskInterruptionPresentation.rejectedModel` (shared type): with `userAction: "choose_model"`, the task query carries the model id from the run's native execution input. Only a `native_provider_model_rejected` run discloses it. - `TaskChatThread` names the model in the "Model unavailable" marker: `The provider rejected the model "".` - The restart-chaos config wave reads `interruption.rejectedModel` as task-level text. For `choose_model`, its repair step uses the Retry that the "Model unavailable" marker offers (`POST /agents/{id}/wakeup` with `reason: "retry_failed_run"`, the same request as `agentsApi.retryFailedRun`), because a model rejection does not offer the task-level Retry. ## Verification - Red before, green after (the classification line disabled makes both fail): - `native-provider-failure.test.ts` — "classifies the API-key 404 for the configured model as a model rejection (codex 0.160.0 shape)", with the exact terminal error captured from `codex app-server` 0.160.0, plus negative cases. - `native-provider-failure.integration.test.ts` — new `api-key` scenario through `executePaperclipNativeSession`, the persisted evidence reader and `finalizeNativeRun`: the run ends `native_provider_model_rejected`, the issue is `blocked` with that decision, and no issue recovery action is created. - `task-interruption-query.test.ts` — the real task query carries `rejectedModel` only for a model rejection. `TaskChatThread.test.tsx` — the marker names the model. - `pnpm --filter @paperclipai/shared typecheck`, `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter @paperclipai/ui typecheck` pass. - Live proof: config wave `w-20261110T090801-s206-cf` (CF3 `model_id`, codex with an OpenAI API key, K2, seed 206) on commit 05e8938: **1/1 pass**. The run ends `native_provider_model_rejected`; the task rests as `waiting_for_user` / `choose_model` with `rejectedModel: "chaos-missing-model-472e1851"` and execution cause `native_provider_model_rejected` (no reconciliation incident), rests again after the K2 restart, and finishes 27 s after the model is restored and the marker's Retry is used. On `master` the same wave failed as `actionable_message` with "Recovery needed" / `native_continuation_requires_reconciliation` (`w-20261110T061914-s206-cf`). ## Risks - Low risk. The match is bound to the configured model, a 404 status and a non-recoverable turn, so other provider failures keep their current behavior. - The match depends on the Codex terminal text. If a later Codex release changes it, the run falls back to the old generic failure, not to a wrong classification. - Claude (ACPX) unknown models are not changed here. Claude Code reports an unknown model as a synthetic assistant message ("There's an issue with the selected model …") rather than a provider error; this PR does not change or live-verify that path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs, a direct `codex app-server` probe). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude --- .../shared/src/types/execution-projection.ts | 2 + .../__tests__/task-interruption-query.test.ts | 15 +++++ ...ative-provider-failure.integration.test.ts | 17 ++++-- .../native-provider-failure.test.ts | 28 ++++++++- .../native-runtime/native-provider-failure.ts | 57 ++++++++++++------- .../task-interruption-presentation.ts | 8 ++- tests/restart-chaos/config-faults.ts | 5 +- tests/restart-chaos/config-wave.ts | 11 +++- ui/src/components/TaskChatThread.test.tsx | 22 +++++++ ui/src/components/TaskChatThread.tsx | 4 +- 10 files changed, 137 insertions(+), 32 deletions(-) diff --git a/packages/shared/src/types/execution-projection.ts b/packages/shared/src/types/execution-projection.ts index bb6d31c3cf6..e72e641602b 100644 --- a/packages/shared/src/types/execution-projection.ts +++ b/packages/shared/src/types/execution-projection.ts @@ -136,6 +136,8 @@ export interface TaskInterruptionPresentation { output: string | null; message: string; }; + /** With `userAction: "choose_model"`: the configured model id the provider rejected. */ + rejectedModel?: string; /** A persisted Retry admission wait, normalized to public text by the server. */ retryWait?: { reason: string; message: string }; intentId: string | null; diff --git a/server/src/__tests__/task-interruption-query.test.ts b/server/src/__tests__/task-interruption-query.test.ts index 89368dceef8..fcdcf2dc530 100644 --- a/server/src/__tests__/task-interruption-query.test.ts +++ b/server/src/__tests__/task-interruption-query.test.ts @@ -403,6 +403,21 @@ describe.skipIf(!support.supported)("task interruption latest run query", () => } finally { vi.unstubAllEnvs(); } }); + it("names the configured model a provider rejected on task detail and list reads (CF3 model_id)", async () => { + const f = await seedInterruptionFixture(db, { selected: 1, history: 1, unrelated: 0, payloadBytes: 0 }); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId)); + await db.update(heartbeatRuns).set({ status: "failed", errorCode: "native_provider_model_rejected", + runnerProfileJson: { nativeExecutionInput: { provider: { kind: "codex", model: "chaos-missing-model-1234" } } } }).where(eq(heartbeatRuns.id, run!.id)); + for (const presentation of [(await taskInterruptionPresentations(db, f.companyId, f.input)).get(f.input[0]!.id), + await taskInterruptionPresentation(db, f.input[0]!)]) { + expect(presentation).toMatchObject({ responsibility: "user", state: "waiting_for_user", userAction: "choose_model", + rejectedModel: "chaos-missing-model-1234" }); + } + // Only a model rejection discloses the configured model. + await db.update(heartbeatRuns).set({ errorCode: "adapter_failed" }).where(eq(heartbeatRuns.id, run!.id)); + expect(await taskInterruptionPresentation(db, f.input[0]!)).not.toHaveProperty("rejectedModel"); + }); + it.each(["decryption failure", "expanded message"])("keeps task reads available after provider display %s", async scenario => { vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "a".repeat(64)); try { diff --git a/server/src/services/native-runtime/native-provider-failure.integration.test.ts b/server/src/services/native-runtime/native-provider-failure.integration.test.ts index 36a288d8a08..efe8b057285 100644 --- a/server/src/services/native-runtime/native-provider-failure.integration.test.ts +++ b/server/src/services/native-runtime/native-provider-failure.integration.test.ts @@ -16,7 +16,7 @@ import { executePaperclipNativeSession } from "./native-session-executor.js"; import { buildNativeCompletionContract } from "./completion-contracts.js"; import { PaperclipControlPlanePort } from "./paperclip-control-plane-port.js"; import { prepareNativeHeartbeatRun } from "./prepare-native-run.js"; -import { NATIVE_MODEL_REJECTION_DIAGNOSTIC, NATIVE_MODEL_REJECTION_MESSAGE } from "./native-provider-failure.js"; +import { NATIVE_MODEL_NOT_FOUND_DIAGNOSTIC, NATIVE_MODEL_REJECTION_DIAGNOSTIC, NATIVE_MODEL_REJECTION_MESSAGE } from "./native-provider-failure.js"; import { collectRunFailureDiagnostics } from "../run-failure-diagnostics.js"; import { finalizeNativeRun } from "./native-run-finalizer.js"; import { readPersistedNativeFailureEvidence, readPersistedNativeModelRejection } from "./native-provider-failure-evidence.js"; @@ -50,7 +50,7 @@ describe("native completed failure diagnostics", () => { }); afterAll(async () => { await temporary?.cleanup(); }); - it.each(["committed", "duplicate", "pending-review", "superseded-review", "resolved-review", "reassigned", "successor-race", "different-turn", "unknown-error", "opencode-lost-server", "untrusted-source", "untrusted-run", "untrusted-session", "succeeded", "cancelled"] as const)( + it.each(["committed", "api-key", "duplicate", "pending-review", "superseded-review", "resolved-review", "reassigned", "successor-race", "different-turn", "unknown-error", "opencode-lost-server", "untrusted-source", "untrusted-run", "untrusted-session", "succeeded", "cancelled"] as const)( "preserves accepted results and only labels the matching provider rejection (%s)", async (scenario) => { const issueId = randomUUID(); const runId = randomUUID(); @@ -103,6 +103,10 @@ describe("native completed failure diagnostics", () => { payload: opencode ? { status: "failed", providerTerminalObserved: false, error: { code: "opencode_server_exited", message: "Private stream failure", opencodeFailure: { kind: "server_exited", signal: "SIGKILL", connectionError: "ECONNREFUSED" }, + } } : scenario === "api-key" ? { status: "failed", error: { + // The terminal error codex 0.160.0 app-server sends with API-key auth, after its own retries. + message: `unexpected status 404 Not Found: The model \`${model}\` does not exist or you do not have access to it., url: https://api.openai.com/v1/responses, cf-ray: a5833f01da2716ce-IAH, request id: req_584e320775c34b2c9c6d98af40e0ea3d`, + codexErrorInfo: { httpConnectionFailed: { httpStatusCode: 404 } }, additionalDetails: null, misalignment: null, } } : { status: "failed", error: { codexErrorInfo: "other", message: JSON.stringify({ type: "error", status: 400, error: { type: "invalid_request_error", message: scenario === "unknown-error" ? "Private unknown failure" : `The '${model}' model is not supported when using Codex with a ChatGPT account.` }, }) } }, @@ -139,12 +143,13 @@ describe("native completed failure diagnostics", () => { descriptor: async () => { throw new Error("Unexpected provider call"); }, openSession: async () => { throw new Error("Unexpected provider call"); }, } }); - const classified = scenario === "committed" || scenario === "duplicate" || scenario.endsWith("-review") || scenario === "reassigned" || scenario === "successor-race"; + const expectedDiagnostic = scenario === "api-key" ? NATIVE_MODEL_NOT_FOUND_DIAGNOSTIC : NATIVE_MODEL_REJECTION_DIAGNOSTIC; + const classified = scenario === "committed" || scenario === "api-key" || scenario === "duplicate" || scenario.endsWith("-review") || scenario === "reassigned" || scenario === "successor-race"; const reviewAuthorityEnded = scenario === "superseded-review" || scenario === "resolved-review"; const ownerChanged = scenario === "reassigned" || scenario === "successor-race"; expect(adapter.errorCode).toBe(classified ? "native_provider_model_rejected" : undefined); expect(adapter.errorMessage).toBe(classified ? NATIVE_MODEL_REJECTION_MESSAGE : scenario === "succeeded" ? null : `Native session ${terminal.runTerminalState}`); - expect(adapter.resultJson?.nativeProviderFailure).toEqual(classified ? NATIVE_MODEL_REJECTION_DIAGNOSTIC : undefined); + expect(adapter.resultJson?.nativeProviderFailure).toEqual(classified ? expectedDiagnostic : undefined); // Only a committed, unclassified failure of the terminal turn explains the generic message. const turnFailure = scenario === "unknown-error" ? { provider: "codex", category: "turn_failed", codexErrorInfo: "other" } : opencode ? { provider: "opencode", category: "turn_failed", runnerFailureCode: "opencode_server_exited", streamFailure: "server_exited", signal: "SIGKILL", connectionError: "ECONNREFUSED" } @@ -159,13 +164,13 @@ describe("native completed failure diagnostics", () => { expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(0); expect(await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, runId))).toHaveLength(scenario.startsWith("untrusted-") ? 0 : 1); const diagnostics = collectRunFailureDiagnostics({ ...run, resultJson: adapter.resultJson ?? null }, {}); - expect(diagnostics.provider).toEqual(classified ? NATIVE_MODEL_REJECTION_DIAGNOSTIC : turnFailure ?? {}); + expect(diagnostics.provider).toEqual(classified ? expectedDiagnostic : turnFailure ?? {}); expect(JSON.stringify(diagnostics)).not.toContain(model); expect(JSON.stringify(diagnostics)).not.toContain("Private"); const [persistedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); expect(persistedRun.errorCode).toBeNull(); // No adapter metadata has been saved yet. expect(await readPersistedNativeModelRejection(db, persistedRun, turnId, terminal)).toEqual(classified ? { - errorCode: "native_provider_model_rejected", errorMessage: NATIVE_MODEL_REJECTION_MESSAGE, diagnostic: NATIVE_MODEL_REJECTION_DIAGNOSTIC, + errorCode: "native_provider_model_rejected", errorMessage: NATIVE_MODEL_REJECTION_MESSAGE, diagnostic: expectedDiagnostic, } : null); expect((await readPersistedNativeFailureEvidence(db, persistedRun, turnId, terminal)).turnFailure).toEqual(turnFailure ?? null); if (classified) { diff --git a/server/src/services/native-runtime/native-provider-failure.test.ts b/server/src/services/native-runtime/native-provider-failure.test.ts index dbab03ed300..a777eef55d7 100644 --- a/server/src/services/native-runtime/native-provider-failure.test.ts +++ b/server/src/services/native-runtime/native-provider-failure.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it } from "vitest"; import type { PrpEvent, PrpTerminalState } from "../../vendor/paperclip-runner/index.js"; -import { createNativeProviderFailureObservation, NATIVE_MODEL_REJECTION_DIAGNOSTIC, NATIVE_MODEL_REJECTION_MESSAGE, NATIVE_PROVIDER_OVERLOADED_MESSAGE, readNativeTurnFailureDiagnostic } from "./native-provider-failure.js"; +import { createNativeProviderFailureObservation, NATIVE_MODEL_NOT_FOUND_DIAGNOSTIC, NATIVE_MODEL_REJECTION_DIAGNOSTIC, NATIVE_MODEL_REJECTION_MESSAGE, NATIVE_PROVIDER_OVERLOADED_MESSAGE, readNativeTurnFailureDiagnostic } from "./native-provider-failure.js"; const model = "test-model"; const response = { @@ -52,6 +52,32 @@ describe("native provider failure observation", () => { observed.observe({ ...event, payload: { status: "failed", error: { providerFailure: { schema: "paperclip.provider-failure.v1", provider: "openrouter", phase: "turn", category: "model_unavailable", message: "busy", retryability } } } }); expect(observed.forTerminal("turn", terminal)).toBeNull(); }); + it("classifies the API-key 404 for the configured model as a model rejection (codex 0.160.0 shape)", () => { + const apiKeyFailure = (message: string, info: unknown = { httpConnectionFailed: { httpStatusCode: 404 } }) => ({ ...event, payload: { status: "failed", error: { + message, codexErrorInfo: info, additionalDetails: null, misalignment: null } } }); + const real = `unexpected status 404 Not Found: The model \`${model}\` does not exist or you do not have access to it., url: https://api.openai.com/v1/responses, cf-ray: a5833f01da2716ce-IAH, request id: req_584e320775c34b2c9c6d98af40e0ea3d`; + const observed = observation(); + observed.observe(apiKeyFailure(real)); + const result = observed.forTerminal("turn", terminal); + expect(result).toEqual({ errorCode: "native_provider_model_rejected", errorMessage: NATIVE_MODEL_REJECTION_MESSAGE, diagnostic: NATIVE_MODEL_NOT_FOUND_DIAGNOSTIC }); + expect(JSON.stringify(result)).not.toContain(model); + expect(observed.unclassifiedTurnFailure("turn", terminal)).toBeNull(); + expect(observed.forTerminal("other-turn", terminal)).toBeNull(); + // Another model's 404, another status, or a retryable turn is not this rejection. + for (const [message, info, recoverable] of [ + [real.replace(model, "other-model"), undefined, false], + [real, { httpConnectionFailed: { httpStatusCode: 500 } }, false], + [real, "other", false], + // Codex reports its retried 404s with this kind and willRetry; they never end the turn. + [real, { responseStreamDisconnected: { httpStatusCode: 404 } }, false], + [real, undefined, true], + ] as const) { + const other = observation(); + const failed = apiKeyFailure(message, info); + other.observe(recoverable ? { ...failed, payload: { ...failed.payload, recoverable: true } } : failed); + expect(other.forTerminal("turn", terminal)).toBeNull(); + } + }); it("surfaces a structured capacity failure without copying arbitrary provider text", () => { const observed = createNativeProviderFailureObservation({ kind: "codex" }); observed.observe({ ...event, payload: { status: "failed", error: { codexErrorInfo: "serverOverloaded", message: "Private arbitrary response" } } }); diff --git a/server/src/services/native-runtime/native-provider-failure.ts b/server/src/services/native-runtime/native-provider-failure.ts index ecbd4ed837a..ace90fe4401 100644 --- a/server/src/services/native-runtime/native-provider-failure.ts +++ b/server/src/services/native-runtime/native-provider-failure.ts @@ -27,13 +27,22 @@ function record(value: unknown): Record { : {}; } +/** An API-key Codex turn whose configured model the Responses API answers with 404. */ +export const NATIVE_MODEL_NOT_FOUND_DIAGNOSTIC = { + provider: "codex", + category: "model_not_found", + status: 404, + authMode: "api_key", +} as const; + /** Reconstruct only the closed diagnostic vocabulary, never provider text. */ export function readNativeModelRejectionDiagnostic(value: unknown) { try { const entry = record(value); - return Object.entries(NATIVE_MODEL_REJECTION_DIAGNOSTIC).every(([key, expected]) => entry[key] === expected) - ? { ...NATIVE_MODEL_REJECTION_DIAGNOSTIC } - : null; + for (const diagnostic of [NATIVE_MODEL_REJECTION_DIAGNOSTIC, NATIVE_MODEL_NOT_FOUND_DIAGNOSTIC]) { + if (Object.entries(diagnostic).every(([key, expected]) => entry[key] === expected)) return { ...diagnostic }; + } + return null; } catch { return null; } } @@ -213,6 +222,8 @@ export function readNativeTurnFailureDiagnostic(value: unknown): NativeTurnFailu /** Observe only events accepted by the bound control-plane persistence port. */ export function createNativeProviderFailureObservation(provider: { kind: string; model?: string | null }) { let rejectedTurnId: string | null = null; + /** A Codex API-key turn whose configured model the Responses API does not serve (404). */ + let missingModelTurnId: string | null = null; let overloadedTurnId: string | null = null; const failedTurns = new Map(); const providerFailures = new Map(); @@ -251,6 +262,16 @@ export function createNativeProviderFailureObservation(provider: { kind: string; } if (!provider.model || payload.status !== "failed" || payload.recoverable === true || failure.recoverable === true || typeof failure.message !== "string" || failure.message.length > 4_096) return; + // With an API key, Codex retries the Responses API 404 and then fails the turn with this + // terminal text (codex 0.160.0). Bind it to the configured model; store neither name nor text. + const info = record(failure.codexErrorInfo); + const infoKinds = Object.keys(info); + // Its retried 404s arrive as responseStreamDisconnected with willRetry; only the terminal kind counts. + if (infoKinds.length === 1 && infoKinds[0] === "httpConnectionFailed" && record(info.httpConnectionFailed).httpStatusCode === 404 && + failure.message.startsWith(`unexpected status 404 Not Found: The model \`${provider.model}\` does not exist or you do not have access to it.`)) { + missingModelTurnId = event.turnId; + return; + } let response: Record; try { response = record(JSON.parse(failure.message)); } catch { return; } const error = record(response.error); @@ -272,27 +293,23 @@ export function createNativeProviderFailureObservation(provider: { kind: string; errorMessage: NATIVE_PROVIDER_OVERLOADED_MESSAGE, diagnostic: { ...NATIVE_PROVIDER_OVERLOADED_DIAGNOSTIC }, }; - if (turnId !== rejectedTurnId) { - // A driver's closed category marks a rejected model for any provider. Only an - // action-required rejection qualifies; a transient one may succeed on retry. - if (!modelUnavailableTurn(turnId)) return null; - const failure = providerFailures.get(turnId)!; - return { - errorCode: "native_provider_model_rejected" as const, - errorMessage: NATIVE_MODEL_REJECTION_MESSAGE, - diagnostic: failedTurns.get(turnId) ?? { provider: failure.provider, category: failure.category }, - }; - } - return { - errorCode: "native_provider_model_rejected" as const, - errorMessage: NATIVE_MODEL_REJECTION_MESSAGE, - diagnostic: { ...NATIVE_MODEL_REJECTION_DIAGNOSTIC }, - }; + // Every model rejection shares one result; only its closed diagnostic differs. + const modelRejection = (diagnostic: T) => ({ + errorCode: "native_provider_model_rejected" as const, errorMessage: NATIVE_MODEL_REJECTION_MESSAGE, diagnostic, + }); + if (turnId === missingModelTurnId) return modelRejection({ ...NATIVE_MODEL_NOT_FOUND_DIAGNOSTIC }); + if (turnId === rejectedTurnId) return modelRejection({ ...NATIVE_MODEL_REJECTION_DIAGNOSTIC }); + // A driver's closed category marks a rejected model for any provider. Only an + // action-required rejection qualifies; a transient one may succeed on retry. + if (!modelUnavailableTurn(turnId)) return null; + const failure = providerFailures.get(turnId)!; + return modelRejection(failedTurns.get(turnId) ?? { provider: failure.provider, category: failure.category }); }, /** Diagnostics only: the failed turn that `forTerminal` did not classify. */ unclassifiedTurnFailure(turnId: string | null | undefined, terminal: PrpTerminalState) { if (!turnId || terminal.turnTerminalState !== "failed" || terminal.runTerminalState !== "failed" || - turnId === overloadedTurnId || turnId === rejectedTurnId || modelUnavailableTurn(turnId)) return null; + turnId === overloadedTurnId || turnId === rejectedTurnId || turnId === missingModelTurnId || + modelUnavailableTurn(turnId)) return null; const diagnostic = failedTurns.get(turnId); return diagnostic ? { ...diagnostic } : null; }, diff --git a/server/src/services/task-interruption-presentation.ts b/server/src/services/task-interruption-presentation.ts index 5009f22e5f6..7d8fc3b82dd 100644 --- a/server/src/services/task-interruption-presentation.ts +++ b/server/src/services/task-interruption-presentation.ts @@ -44,6 +44,8 @@ type ProjectionRun = Pick reason === "budget" || reason === "budget_exceeded") ? "budget" : input.paused || project?.pausedAt || agent?.status === "paused" || company?.status === "paused" ? "resume" : null; - if (userAction) return { ...base, responsibility: "user", state: "waiting_for_user", userAction }; + if (userAction) return { ...base, responsibility: "user", state: "waiting_for_user", userAction, + ...(userAction === "choose_model" && run.rejectedModel ? { rejectedModel: run.rejectedModel } : {}) }; // The previous controller cannot confirm its own stop. Name the one board // action instead of showing the fenced run as working (RC-47). if (input.ownerStopRequired) return { ...base, responsibility: "user", state: "waiting_for_user", userAction: "confirm_owner_stop" }; @@ -343,6 +346,9 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i and ${heartbeatRuns.error} ~ '^(Execution workspace provision|Workspace seed|Runtime provision) command "' then ${heartbeatRuns.error} else null end`, errorCode: heartbeatRuns.errorCode, livenessState: heartbeatRuns.livenessState, scheduledRetryAt: heartbeatRuns.scheduledRetryAt, + // The model this run asked the provider for; configuration, not provider text. + rejectedModel: sql`case when ${heartbeatRuns.errorCode} = 'native_provider_model_rejected' + then ${boundedJsonString(sql`${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput'->'provider'->'model'`, 200)} end`, scheduledRetryReason: heartbeatRuns.scheduledRetryReason, // Candidate only: the coordinator also pushes nextAttemptAt forward while // it defers an ineligible run. The scheduler's own admission check below diff --git a/tests/restart-chaos/config-faults.ts b/tests/restart-chaos/config-faults.ts index ea5fcc34cbf..732b9042b56 100644 --- a/tests/restart-chaos/config-faults.ts +++ b/tests/restart-chaos/config-faults.ts @@ -170,7 +170,9 @@ export interface ConfigSnapshot { runs: Array<{ id: string; status: string; createdAt: number | null; startedAt: number | null; errorCode: string | null; error: string | null }>; interruption: { state: string | null; canRetry: boolean; userAction: string | null; workspaceFailure: string | null; /** The named setup command failure (`setup_failed`), as the task page shows it. */ - setupFailure?: string | null; observedRunId: string | null } | null; + setupFailure?: string | null; + /** The model id the provider rejected, as the task page names it. */ + rejectedModel?: string | null; observedRunId: string | null } | null; execution: { label: string | null; cause: string | null; nextAction: string | null } | null; /** Pending thread interactions on the task (connection cards and similar). */ cards: Array<{ id: string; kind: string; status: string; serviceSlug: string | null; agentSetup: boolean }>; @@ -186,6 +188,7 @@ export function taskLevelTexts(snapshot: ConfigSnapshot): string[] { if (interruption?.workspaceFailure) texts.push(interruption.workspaceFailure); if (interruption?.setupFailure) texts.push(interruption.setupFailure); if (interruption?.userAction) texts.push(`userAction:${interruption.userAction}`); + if (interruption?.rejectedModel) texts.push(`rejected model ${interruption.rejectedModel}`); if (snapshot.execution?.label) texts.push(snapshot.execution.label); if (snapshot.execution?.nextAction) texts.push(snapshot.execution.nextAction); for (const card of snapshot.cards) { diff --git a/tests/restart-chaos/config-wave.ts b/tests/restart-chaos/config-wave.ts index 117ea641530..43f22651d46 100644 --- a/tests/restart-chaos/config-wave.ts +++ b/tests/restart-chaos/config-wave.ts @@ -233,7 +233,8 @@ async function snapshot(api: ChaosApi, issueId: string, agentId: string, errors: return { at, reachable: true, issueStatus: String(issue.status), runs: runSamples, interruption: interruption ? { state: interruption.state ?? null, canRetry: interruption.canRetry === true, userAction: interruption.userAction ?? null, - workspaceFailure: interruption.workspaceFailure?.message ?? null, setupFailure: interruption.setupFailure?.message ?? null, observedRunId: interruption.observedRunId ?? null } : null, + workspaceFailure: interruption.workspaceFailure?.message ?? null, setupFailure: interruption.setupFailure?.message ?? null, + rejectedModel: interruption.rejectedModel ?? null, observedRunId: interruption.observedRunId ?? null } : null, execution: projection ? { label: projection.label ?? null, cause: projection.cause ?? null, nextAction: projection.nextAction ?? null } : null, cards: (Array.isArray(interactions) ? interactions : []).map(card => ({ id: String(card.id), kind: String(card.kind), status: String(card.status), serviceSlug: card.payload?.serviceSlug ?? null, agentSetup: card.payload?.agentSetup === true })), @@ -253,7 +254,7 @@ function time(value: unknown): number | null { return Number.isFinite(at) ? at : null; } -/** Use what the task offers: complete a pending connection card, then Retry, then the agent's setup Retry. */ +/** Use what the task offers: complete a pending connection card, then Retry (the task's, or a rejected model's marker), then the agent's setup Retry. */ async function offeredAction(api: ChaosApi, record: ConfigTaskRecord, resources: TaskResources, current: ConfigSnapshot): Promise { const card = current.cards.find(candidate => candidate.status === "pending" && candidate.kind === "connection_intent"); if (card && resources.connectionId) { @@ -264,6 +265,12 @@ async function offeredAction(api: ChaosApi, record: ConfigTaskRecord, resources: const response = await api.post(`/api/issues/${record.issueId}/retry`, { idempotencyKey: randomUUID(), observedRunId: current.interruption.observedRunId }); return `clicked Retry (accepted=${String(response?.accepted)})`; } + // A rejected model offers Retry on the run's "Model unavailable" marker, which retries that run on the agent. + if (current.interruption?.userAction === "choose_model" && current.interruption.observedRunId) { + const response = await api.post(`/api/agents/${resources.agentId}/wakeup`, { source: "on_demand", triggerDetail: "manual", + reason: "retry_failed_run", failedRunId: current.interruption.observedRunId }); + return `clicked the model marker's Retry (run ${String(response?.id ?? "none")})`; + } if (current.agent && current.agent.lifecycleState !== "ready") { await api.post(`/api/agents/${resources.agentId}/lifecycle/retry`, {}); return `agent setup Retry (lifecycle ${current.agent.lifecycleState})`; diff --git a/ui/src/components/TaskChatThread.test.tsx b/ui/src/components/TaskChatThread.test.tsx index 1bf3b9c9096..17d62058637 100644 --- a/ui/src/components/TaskChatThread.test.tsx +++ b/ui/src/components/TaskChatThread.test.tsx @@ -2665,6 +2665,28 @@ describe("TaskChatThread runtime transcript selection", () => { expect(container.textContent).not.toContain("after returning a final response"); }); + it("names the rejected model id that the task interruption carries", () => { + renderThread({ + comments: [], + onAdd: async () => {}, + interruption: { intentId: null, responsibility: "user", state: "waiting_for_user", nextRetryAt: null, + canRetry: false, userAction: "choose_model", rejectedModel: "chaos-missing-model-1234", sourceRunIds: ["model-missing"], observedRunId: "model-missing" }, + linkedRuns: [{ + runId: "model-missing", runtimeMode: "native", status: "failed", + errorCode: "native_provider_model_rejected", + error: "The selected model is not available through the current AI connection. Choose a supported model or a compatible AI connection.", + agentId: "agent-1", agentName: "Runner", adapterType: "paperclip_runner", + createdAt: "2026-08-25T18:00:00.000Z", startedAt: "2026-08-25T18:00:00.000Z", + finishedAt: "2026-08-25T18:00:02.000Z", + }], + }); + const marker = container.querySelector('[data-testid="task-chat-collapsible-marker"]'); + expect(marker?.textContent).toContain("Model unavailable"); + flushSync(() => marker!.querySelector('button[aria-expanded="false"]')!.click()); + expect(container.textContent).toContain('The provider rejected the model "chaos-missing-model-1234".'); + expect(container.textContent).toContain("Choose a supported model"); + }); + it("keeps workspace contention out of the conversation's cancellation markers", () => { renderThread({ comments: [], diff --git a/ui/src/components/TaskChatThread.tsx b/ui/src/components/TaskChatThread.tsx index a862cb4af10..1a73d555d15 100644 --- a/ui/src/components/TaskChatThread.tsx +++ b/ui/src/components/TaskChatThread.tsx @@ -1868,7 +1868,9 @@ export function TaskChatThread(props: TaskChatThreadProps) { : code === "native_provider_approval_required" ? "This operation requires approval, but this runner has no interactive approval handler. Review the operation and update the agent's permission setting before retrying." : code === "native_provider_model_rejected" - ? `${meta?.error || "The provider rejected the selected model."} Choose a supported model or clear the task's model override, then retry.` + ? `${props.interruption?.observedRunId === source.id && props.interruption.rejectedModel + ? `The provider rejected the model "${props.interruption.rejectedModel}".` + : meta?.error || "The provider rejected the selected model."} Choose a supported model or clear the task's model override, then retry.` : code === "native_provider_usage_limit" && source.status === "failed" ? "The model provider has reached its current usage limit. Try again after the limit resets."