fix(native): classify a Codex API-key unknown model as a model rejection (#20488)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runner agents run on a provider (here Codex) with a model the
user saves on the agent
> - When the provider rejects that model, Paperclip should classify the
run as `native_provider_model_rejected`, ask the user to choose a model,
and let Retry finish the task
> - Paperclip recognized only the ChatGPT-account rejection (a 400 JSON
envelope). With API-key auth, Codex returns a different terminal error
for an unknown model
> - So the run failed as a generic `adapter_failed` ("Native session
failed"), and recovery opened a reconciliation incident ("Recovery
needed") for a turn that never did any work
> - This pull request recognizes the real API-key shape, bound to the
configured model, and names the rejected model on the task
> - The benefit is that the task asks for a model choice instead of a
reconciliation, and Retry completes the task after the user picks a
valid model

## Linked Issues or Issue Description

**What happened?**
The restart-chaos config fault CF3 `model_id` saves a Codex agent with
the model `chaos-missing-model-<nonce>`. The run stays "running" for
about 60 seconds, then fails with `errorCode: adapter_failed`, "Native
session failed". The task shows "Recovery needed", cause
`native_continuation_requires_reconciliation`. The grader fails as
`CF3/model_id:actionable_message` (waves `w-20261110T061914-s206-cf` t01
and `w-20261110T053831-s205-cf` t02).

Root cause: the wave runs Codex with an OpenAI API key. I drove the
bundled `codex app-server` (0.160.0) directly with an unknown model:
- With API-key auth, Codex retries the Responses API 404 ten times
(about 59 s), then fails the turn with `{"message":"unexpected status
404 Not Found: The model \`<id>\` does not exist or you do not have
access to it., url: https://api.openai.com/v1/responses, cf-ray: …,
request id:
req_…","codexErrorInfo":{"httpConnectionFailed":{"httpStatusCode":404}}}`.
- With ChatGPT auth, the turn fails in about 3 s with `message` =
`{"type":"error","status":400,"error":{"type":"invalid_request_error","message":"The
'<id>' model is not supported when using Codex with a ChatGPT
account.","code":"model_not_found"}}`. Paperclip already classified this
one.

`createNativeProviderFailureObservation` parsed only the JSON envelope,
and the runner's provider-failure category for an HTTP 404 is `unknown`.
The terminal therefore stayed unclassified.

**Expected behavior**
The run is `native_provider_model_rejected`, the task shows `userAction:
"choose_model"` and names the model, no reconciliation incident opens,
and Retry completes the task after the user picks a valid model.

**Steps to reproduce**
`node cli/node_modules/tsx/dist/cli.mjs
tests/restart-chaos/config-wave.ts --faults CF3 --adapters codex
--references model_id --tasks 1 --concurrency 1 --restarts K2 --seed
206` on `master` (with `OPENAI_API_KEY` set).

**Paperclip version or commit**
`master` at 5ab98b0.

## What Changed

- `server/src/services/native-runtime/native-provider-failure.ts`: the
Codex observation recognizes the API-key terminal shape
(`codexErrorInfo` `httpConnectionFailed` or `responseStreamDisconnected`
with `httpStatusCode: 404`, and the message prefix for the configured
model only). It classifies it as `native_provider_model_rejected` with a
new closed diagnostic `NATIVE_MODEL_NOT_FOUND_DIAGNOSTIC` (`codex`,
`model_not_found`, `404`, `api_key`). No provider text or model name is
stored in the diagnostic. A recoverable turn, another model's 404, or
another status is not classified.
- `readNativeModelRejectionDiagnostic` accepts both closed diagnostics,
so run failure diagnostics show it.
- `TaskInterruptionPresentation.rejectedModel` (shared type): with
`userAction: "choose_model"`, the task query carries the model id from
the run's native execution input. Only a
`native_provider_model_rejected` run discloses it.
- `TaskChatThread` names the model in the "Model unavailable" marker:
`The provider rejected the model "<id>".`
- The restart-chaos config wave reads `interruption.rejectedModel` as
task-level text. For `choose_model`, its repair step uses the Retry that
the "Model unavailable" marker offers (`POST /agents/{id}/wakeup` with
`reason: "retry_failed_run"`, the same request as
`agentsApi.retryFailedRun`), because a model rejection does not offer
the task-level Retry.

## Verification

- Red before, green after (the classification line disabled makes both
fail):
- `native-provider-failure.test.ts` — "classifies the API-key 404 for
the configured model as a model rejection (codex 0.160.0 shape)", with
the exact terminal error captured from `codex app-server` 0.160.0, plus
negative cases.
- `native-provider-failure.integration.test.ts` — new `api-key` scenario
through `executePaperclipNativeSession`, the persisted evidence reader
and `finalizeNativeRun`: the run ends `native_provider_model_rejected`,
the issue is `blocked` with that decision, and no issue recovery action
is created.
- `task-interruption-query.test.ts` — the real task query carries
`rejectedModel` only for a model rejection. `TaskChatThread.test.tsx` —
the marker names the model.
- `pnpm --filter @paperclipai/shared typecheck`, `pnpm --filter
@paperclipai/server typecheck`, `pnpm --filter @paperclipai/ui
typecheck` pass.
- Live proof: config wave `w-20261110T090801-s206-cf` (CF3 `model_id`,
codex with an OpenAI API key, K2, seed 206) on commit 05e8938: **1/1
pass**. The run ends `native_provider_model_rejected`; the task rests as
`waiting_for_user` / `choose_model` with `rejectedModel:
"chaos-missing-model-472e1851"` and execution cause
`native_provider_model_rejected` (no reconciliation incident), rests
again after the K2 restart, and finishes 27 s after the model is
restored and the marker's Retry is used. On `master` the same wave
failed as `actionable_message` with "Recovery needed" /
`native_continuation_requires_reconciliation`
(`w-20261110T061914-s206-cf`).

## Risks

- Low risk. The match is bound to the configured model, a 404 status and
a non-recoverable turn, so other provider failures keep their current
behavior.
- The match depends on the Codex terminal text. If a later Codex release
changes it, the run falls back to the old generic failure, not to a
wrong classification.
- Claude (ACPX) unknown models are not changed here. Claude Code reports
an unknown model as a synthetic assistant message ("There's an issue
with the selected model …") rather than a provider error; this PR does
not change or live-verify that path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs, a direct `codex
app-server` probe).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
DottaandClaude authored and GitHub committed 2026-11-10 03:11:10 -06:00
1 parent 8a8abdb7b9
commit 82abb3ea04
10 files changed
+133 -28

No files matched your search

@@ -136,6 +136,8 @@ export interface TaskInterruptionPresentation {
output: string | null; output: string | null;
message: string; message: string;
}; };
/** With `userAction: "choose_model"`: the configured model id the provider rejected. */
rejectedModel?: string;
/** A persisted Retry admission wait, normalized to public text by the server. */ /** A persisted Retry admission wait, normalized to public text by the server. */
retryWait?: { reason: string; message: string }; retryWait?: { reason: string; message: string };
intentId: string | null; intentId: string | null;
@@ -403,6 +403,21 @@ describe.skipIf(!support.supported)("task interruption latest run query", () =>
} finally { vi.unstubAllEnvs(); } } finally { vi.unstubAllEnvs(); }
}); });
it("names the configured model a provider rejected on task detail and list reads (CF3 model_id)", async () => {
const f = await seedInterruptionFixture(db, { selected: 1, history: 1, unrelated: 0, payloadBytes: 0 });
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId));
await db.update(heartbeatRuns).set({ status: "failed", errorCode: "native_provider_model_rejected",
runnerProfileJson: { nativeExecutionInput: { provider: { kind: "codex", model: "chaos-missing-model-1234" } } } }).where(eq(heartbeatRuns.id, run!.id));
for (const presentation of [(await taskInterruptionPresentations(db, f.companyId, f.input)).get(f.input[0]!.id),
await taskInterruptionPresentation(db, f.input[0]!)]) {
expect(presentation).toMatchObject({ responsibility: "user", state: "waiting_for_user", userAction: "choose_model",
rejectedModel: "chaos-missing-model-1234" });
}
// Only a model rejection discloses the configured model.
await db.update(heartbeatRuns).set({ errorCode: "adapter_failed" }).where(eq(heartbeatRuns.id, run!.id));
expect(await taskInterruptionPresentation(db, f.input[0]!)).not.toHaveProperty("rejectedModel");
});
it.each(["decryption failure", "expanded message"])("keeps task reads available after provider display %s", async scenario => { it.each(["decryption failure", "expanded message"])("keeps task reads available after provider display %s", async scenario => {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "a".repeat(64)); vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "a".repeat(64));
try { try {
@@ -16,7 +16,7 @@ import { executePaperclipNativeSession } from "./native-session-executor.js";
import { buildNativeCompletionContract } from "./completion-contracts.js"; import { buildNativeCompletionContract } from "./completion-contracts.js";
import { PaperclipControlPlanePort } from "./paperclip-control-plane-port.js"; import { PaperclipControlPlanePort } from "./paperclip-control-plane-port.js";
import { prepareNativeHeartbeatRun } from "./prepare-native-run.js"; import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
import { NATIVE_MODEL_REJECTION_DIAGNOSTIC, NATIVE_MODEL_REJECTION_MESSAGE } from "./native-provider-failure.js"; import { NATIVE_MODEL_NOT_FOUND_DIAGNOSTIC, NATIVE_MODEL_REJECTION_DIAGNOSTIC, NATIVE_MODEL_REJECTION_MESSAGE } from "./native-provider-failure.js";
import { collectRunFailureDiagnostics } from "../run-failure-diagnostics.js"; import { collectRunFailureDiagnostics } from "../run-failure-diagnostics.js";
import { finalizeNativeRun } from "./native-run-finalizer.js"; import { finalizeNativeRun } from "./native-run-finalizer.js";
import { readPersistedNativeFailureEvidence, readPersistedNativeModelRejection } from "./native-provider-failure-evidence.js"; import { readPersistedNativeFailureEvidence, readPersistedNativeModelRejection } from "./native-provider-failure-evidence.js";
@@ -50,7 +50,7 @@ describe("native completed failure diagnostics", () => {
}); });
afterAll(async () => { await temporary?.cleanup(); }); afterAll(async () => { await temporary?.cleanup(); });
it.each(["committed", "duplicate", "pending-review", "superseded-review", "resolved-review", "reassigned", "successor-race", "different-turn", "unknown-error", "opencode-lost-server", "untrusted-source", "untrusted-run", "untrusted-session", "succeeded", "cancelled"] as const)( it.each(["committed", "api-key", "duplicate", "pending-review", "superseded-review", "resolved-review", "reassigned", "successor-race", "different-turn", "unknown-error", "opencode-lost-server", "untrusted-source", "untrusted-run", "untrusted-session", "succeeded", "cancelled"] as const)(
"preserves accepted results and only labels the matching provider rejection (%s)", async (scenario) => { "preserves accepted results and only labels the matching provider rejection (%s)", async (scenario) => {
const issueId = randomUUID(); const issueId = randomUUID();
const runId = randomUUID(); const runId = randomUUID();
@@ -103,6 +103,10 @@ describe("native completed failure diagnostics", () => {
payload: opencode ? { status: "failed", providerTerminalObserved: false, error: { payload: opencode ? { status: "failed", providerTerminalObserved: false, error: {
code: "opencode_server_exited", message: "Private stream failure", code: "opencode_server_exited", message: "Private stream failure",
opencodeFailure: { kind: "server_exited", signal: "SIGKILL", connectionError: "ECONNREFUSED" }, opencodeFailure: { kind: "server_exited", signal: "SIGKILL", connectionError: "ECONNREFUSED" },
} } : scenario === "api-key" ? { status: "failed", error: {
// The terminal error codex 0.160.0 app-server sends with API-key auth, after its own retries.
message: `unexpected status 404 Not Found: The model \`${model}\` does not exist or you do not have access to it., url: https://api.openai.com/v1/responses, cf-ray: a5833f01da2716ce-IAH, request id: req_584e320775c34b2c9c6d98af40e0ea3d`,
codexErrorInfo: { httpConnectionFailed: { httpStatusCode: 404 } }, additionalDetails: null, misalignment: null,
} } : { status: "failed", error: { codexErrorInfo: "other", message: JSON.stringify({ } } : { status: "failed", error: { codexErrorInfo: "other", message: JSON.stringify({
type: "error", status: 400, error: { type: "invalid_request_error", message: scenario === "unknown-error" ? "Private unknown failure" : `The '${model}' model is not supported when using Codex with a ChatGPT account.` }, type: "error", status: 400, error: { type: "invalid_request_error", message: scenario === "unknown-error" ? "Private unknown failure" : `The '${model}' model is not supported when using Codex with a ChatGPT account.` },
}) } }, }) } },
@@ -139,12 +143,13 @@ describe("native completed failure diagnostics", () => {
descriptor: async () => { throw new Error("Unexpected provider call"); }, descriptor: async () => { throw new Error("Unexpected provider call"); },
openSession: async () => { throw new Error("Unexpected provider call"); }, openSession: async () => { throw new Error("Unexpected provider call"); },
} }); } });
const classified = scenario === "committed" || scenario === "duplicate" || scenario.endsWith("-review") || scenario === "reassigned" || scenario === "successor-race"; const expectedDiagnostic = scenario === "api-key" ? NATIVE_MODEL_NOT_FOUND_DIAGNOSTIC : NATIVE_MODEL_REJECTION_DIAGNOSTIC;
const classified = scenario === "committed" || scenario === "api-key" || scenario === "duplicate" || scenario.endsWith("-review") || scenario === "reassigned" || scenario === "successor-race";
const reviewAuthorityEnded = scenario === "superseded-review" || scenario === "resolved-review"; const reviewAuthorityEnded = scenario === "superseded-review" || scenario === "resolved-review";
const ownerChanged = scenario === "reassigned" || scenario === "successor-race"; const ownerChanged = scenario === "reassigned" || scenario === "successor-race";
expect(adapter.errorCode).toBe(classified ? "native_provider_model_rejected" : undefined); expect(adapter.errorCode).toBe(classified ? "native_provider_model_rejected" : undefined);
expect(adapter.errorMessage).toBe(classified ? NATIVE_MODEL_REJECTION_MESSAGE : scenario === "succeeded" ? null : `Native session ${terminal.runTerminalState}`); expect(adapter.errorMessage).toBe(classified ? NATIVE_MODEL_REJECTION_MESSAGE : scenario === "succeeded" ? null : `Native session ${terminal.runTerminalState}`);
expect(adapter.resultJson?.nativeProviderFailure).toEqual(classified ? NATIVE_MODEL_REJECTION_DIAGNOSTIC : undefined); expect(adapter.resultJson?.nativeProviderFailure).toEqual(classified ? expectedDiagnostic : undefined);
// Only a committed, unclassified failure of the terminal turn explains the generic message. // Only a committed, unclassified failure of the terminal turn explains the generic message.
const turnFailure = scenario === "unknown-error" ? { provider: "codex", category: "turn_failed", codexErrorInfo: "other" } const turnFailure = scenario === "unknown-error" ? { provider: "codex", category: "turn_failed", codexErrorInfo: "other" }
: opencode ? { provider: "opencode", category: "turn_failed", runnerFailureCode: "opencode_server_exited", streamFailure: "server_exited", signal: "SIGKILL", connectionError: "ECONNREFUSED" } : opencode ? { provider: "opencode", category: "turn_failed", runnerFailureCode: "opencode_server_exited", streamFailure: "server_exited", signal: "SIGKILL", connectionError: "ECONNREFUSED" }
@@ -159,13 +164,13 @@ describe("native completed failure diagnostics", () => {
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(0); expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(0);
expect(await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, runId))).toHaveLength(scenario.startsWith("untrusted-") ? 0 : 1); expect(await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, runId))).toHaveLength(scenario.startsWith("untrusted-") ? 0 : 1);
const diagnostics = collectRunFailureDiagnostics({ ...run, resultJson: adapter.resultJson ?? null }, {}); const diagnostics = collectRunFailureDiagnostics({ ...run, resultJson: adapter.resultJson ?? null }, {});
expect(diagnostics.provider).toEqual(classified ? NATIVE_MODEL_REJECTION_DIAGNOSTIC : turnFailure ?? {}); expect(diagnostics.provider).toEqual(classified ? expectedDiagnostic : turnFailure ?? {});
expect(JSON.stringify(diagnostics)).not.toContain(model); expect(JSON.stringify(diagnostics)).not.toContain(model);
expect(JSON.stringify(diagnostics)).not.toContain("Private"); expect(JSON.stringify(diagnostics)).not.toContain("Private");
const [persistedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); const [persistedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
expect(persistedRun.errorCode).toBeNull(); // No adapter metadata has been saved yet. expect(persistedRun.errorCode).toBeNull(); // No adapter metadata has been saved yet.
expect(await readPersistedNativeModelRejection(db, persistedRun, turnId, terminal)).toEqual(classified ? { expect(await readPersistedNativeModelRejection(db, persistedRun, turnId, terminal)).toEqual(classified ? {
errorCode: "native_provider_model_rejected", errorMessage: NATIVE_MODEL_REJECTION_MESSAGE, diagnostic: NATIVE_MODEL_REJECTION_DIAGNOSTIC, errorCode: "native_provider_model_rejected", errorMessage: NATIVE_MODEL_REJECTION_MESSAGE, diagnostic: expectedDiagnostic,
} : null); } : null);
expect((await readPersistedNativeFailureEvidence(db, persistedRun, turnId, terminal)).turnFailure).toEqual(turnFailure ?? null); expect((await readPersistedNativeFailureEvidence(db, persistedRun, turnId, terminal)).turnFailure).toEqual(turnFailure ?? null);
if (classified) { if (classified) {
@@ -1,6 +1,6 @@
import { describe, expect, it } from "vitest"; import { describe, expect, it } from "vitest";
import type { PrpEvent, PrpTerminalState } from "../../vendor/paperclip-runner/index.js"; import type { PrpEvent, PrpTerminalState } from "../../vendor/paperclip-runner/index.js";
import { createNativeProviderFailureObservation, NATIVE_MODEL_REJECTION_DIAGNOSTIC, NATIVE_MODEL_REJECTION_MESSAGE, NATIVE_PROVIDER_OVERLOADED_MESSAGE, readNativeTurnFailureDiagnostic } from "./native-provider-failure.js"; import { createNativeProviderFailureObservation, NATIVE_MODEL_NOT_FOUND_DIAGNOSTIC, NATIVE_MODEL_REJECTION_DIAGNOSTIC, NATIVE_MODEL_REJECTION_MESSAGE, NATIVE_PROVIDER_OVERLOADED_MESSAGE, readNativeTurnFailureDiagnostic } from "./native-provider-failure.js";
const model = "test-model"; const model = "test-model";
const response = { const response = {
@@ -52,6 +52,32 @@ describe("native provider failure observation", () => {
observed.observe({ ...event, payload: { status: "failed", error: { providerFailure: { schema: "paperclip.provider-failure.v1", provider: "openrouter", phase: "turn", category: "model_unavailable", message: "busy", retryability } } } }); observed.observe({ ...event, payload: { status: "failed", error: { providerFailure: { schema: "paperclip.provider-failure.v1", provider: "openrouter", phase: "turn", category: "model_unavailable", message: "busy", retryability } } } });
expect(observed.forTerminal("turn", terminal)).toBeNull(); expect(observed.forTerminal("turn", terminal)).toBeNull();
}); });
it("classifies the API-key 404 for the configured model as a model rejection (codex 0.160.0 shape)", () => {
const apiKeyFailure = (message: string, info: unknown = { httpConnectionFailed: { httpStatusCode: 404 } }) => ({ ...event, payload: { status: "failed", error: {
message, codexErrorInfo: info, additionalDetails: null, misalignment: null } } });
const real = `unexpected status 404 Not Found: The model \`${model}\` does not exist or you do not have access to it., url: https://api.openai.com/v1/responses, cf-ray: a5833f01da2716ce-IAH, request id: req_584e320775c34b2c9c6d98af40e0ea3d`;
const observed = observation();
observed.observe(apiKeyFailure(real));
const result = observed.forTerminal("turn", terminal);
expect(result).toEqual({ errorCode: "native_provider_model_rejected", errorMessage: NATIVE_MODEL_REJECTION_MESSAGE, diagnostic: NATIVE_MODEL_NOT_FOUND_DIAGNOSTIC });
expect(JSON.stringify(result)).not.toContain(model);
expect(observed.unclassifiedTurnFailure("turn", terminal)).toBeNull();
expect(observed.forTerminal("other-turn", terminal)).toBeNull();
// Another model's 404, another status, or a retryable turn is not this rejection.
for (const [message, info, recoverable] of [
[real.replace(model, "other-model"), undefined, false],
[real, { httpConnectionFailed: { httpStatusCode: 500 } }, false],
[real, "other", false],
// Codex reports its retried 404s with this kind and willRetry; they never end the turn.
[real, { responseStreamDisconnected: { httpStatusCode: 404 } }, false],
[real, undefined, true],
] as const) {
const other = observation();
const failed = apiKeyFailure(message, info);
other.observe(recoverable ? { ...failed, payload: { ...failed.payload, recoverable: true } } : failed);
expect(other.forTerminal("turn", terminal)).toBeNull();
}
});
it("surfaces a structured capacity failure without copying arbitrary provider text", () => { it("surfaces a structured capacity failure without copying arbitrary provider text", () => {
const observed = createNativeProviderFailureObservation({ kind: "codex" }); const observed = createNativeProviderFailureObservation({ kind: "codex" });
observed.observe({ ...event, payload: { status: "failed", error: { codexErrorInfo: "serverOverloaded", message: "Private arbitrary response" } } }); observed.observe({ ...event, payload: { status: "failed", error: { codexErrorInfo: "serverOverloaded", message: "Private arbitrary response" } } });
@@ -27,13 +27,22 @@ function record(value: unknown): Record<string, unknown> {
: {}; : {};
} }
/** An API-key Codex turn whose configured model the Responses API answers with 404. */
export const NATIVE_MODEL_NOT_FOUND_DIAGNOSTIC = {
provider: "codex",
category: "model_not_found",
status: 404,
authMode: "api_key",
} as const;
/** Reconstruct only the closed diagnostic vocabulary, never provider text. */ /** Reconstruct only the closed diagnostic vocabulary, never provider text. */
export function readNativeModelRejectionDiagnostic(value: unknown) { export function readNativeModelRejectionDiagnostic(value: unknown) {
try { try {
const entry = record(value); const entry = record(value);
return Object.entries(NATIVE_MODEL_REJECTION_DIAGNOSTIC).every(([key, expected]) => entry[key] === expected) for (const diagnostic of [NATIVE_MODEL_REJECTION_DIAGNOSTIC, NATIVE_MODEL_NOT_FOUND_DIAGNOSTIC]) {
? { ...NATIVE_MODEL_REJECTION_DIAGNOSTIC } if (Object.entries(diagnostic).every(([key, expected]) => entry[key] === expected)) return { ...diagnostic };
: null; }
return null;
} catch { return null; } } catch { return null; }
} }
@@ -213,6 +222,8 @@ export function readNativeTurnFailureDiagnostic(value: unknown): NativeTurnFailu
/** Observe only events accepted by the bound control-plane persistence port. */ /** Observe only events accepted by the bound control-plane persistence port. */
export function createNativeProviderFailureObservation(provider: { kind: string; model?: string | null }) { export function createNativeProviderFailureObservation(provider: { kind: string; model?: string | null }) {
let rejectedTurnId: string | null = null; let rejectedTurnId: string | null = null;
/** A Codex API-key turn whose configured model the Responses API does not serve (404). */
let missingModelTurnId: string | null = null;
let overloadedTurnId: string | null = null; let overloadedTurnId: string | null = null;
const failedTurns = new Map<string, NativeTurnFailureDiagnostic>(); const failedTurns = new Map<string, NativeTurnFailureDiagnostic>();
const providerFailures = new Map<string, ProviderFailure>(); const providerFailures = new Map<string, ProviderFailure>();
@@ -251,6 +262,16 @@ export function createNativeProviderFailureObservation(provider: { kind: string;
} }
if (!provider.model || payload.status !== "failed" || payload.recoverable === true || failure.recoverable === true || if (!provider.model || payload.status !== "failed" || payload.recoverable === true || failure.recoverable === true ||
typeof failure.message !== "string" || failure.message.length > 4_096) return; typeof failure.message !== "string" || failure.message.length > 4_096) return;
// With an API key, Codex retries the Responses API 404 and then fails the turn with this
// terminal text (codex 0.160.0). Bind it to the configured model; store neither name nor text.
const info = record(failure.codexErrorInfo);
const infoKinds = Object.keys(info);
// Its retried 404s arrive as responseStreamDisconnected with willRetry; only the terminal kind counts.
if (infoKinds.length === 1 && infoKinds[0] === "httpConnectionFailed" && record(info.httpConnectionFailed).httpStatusCode === 404 &&
failure.message.startsWith(`unexpected status 404 Not Found: The model \`${provider.model}\` does not exist or you do not have access to it.`)) {
missingModelTurnId = event.turnId;
return;
}
let response: Record<string, unknown>; let response: Record<string, unknown>;
try { response = record(JSON.parse(failure.message)); } catch { return; } try { response = record(JSON.parse(failure.message)); } catch { return; }
const error = record(response.error); const error = record(response.error);
@@ -272,27 +293,23 @@ export function createNativeProviderFailureObservation(provider: { kind: string;
errorMessage: NATIVE_PROVIDER_OVERLOADED_MESSAGE, errorMessage: NATIVE_PROVIDER_OVERLOADED_MESSAGE,
diagnostic: { ...NATIVE_PROVIDER_OVERLOADED_DIAGNOSTIC }, diagnostic: { ...NATIVE_PROVIDER_OVERLOADED_DIAGNOSTIC },
}; };
if (turnId !== rejectedTurnId) { // Every model rejection shares one result; only its closed diagnostic differs.
const modelRejection = <T>(diagnostic: T) => ({
errorCode: "native_provider_model_rejected" as const, errorMessage: NATIVE_MODEL_REJECTION_MESSAGE, diagnostic,
});
if (turnId === missingModelTurnId) return modelRejection({ ...NATIVE_MODEL_NOT_FOUND_DIAGNOSTIC });
if (turnId === rejectedTurnId) return modelRejection({ ...NATIVE_MODEL_REJECTION_DIAGNOSTIC });
// A driver's closed category marks a rejected model for any provider. Only an // A driver's closed category marks a rejected model for any provider. Only an
// action-required rejection qualifies; a transient one may succeed on retry. // action-required rejection qualifies; a transient one may succeed on retry.
if (!modelUnavailableTurn(turnId)) return null; if (!modelUnavailableTurn(turnId)) return null;
const failure = providerFailures.get(turnId)!; const failure = providerFailures.get(turnId)!;
return { return modelRejection(failedTurns.get(turnId) ?? { provider: failure.provider, category: failure.category });
errorCode: "native_provider_model_rejected" as const,
errorMessage: NATIVE_MODEL_REJECTION_MESSAGE,
diagnostic: failedTurns.get(turnId) ?? { provider: failure.provider, category: failure.category },
};
}
return {
errorCode: "native_provider_model_rejected" as const,
errorMessage: NATIVE_MODEL_REJECTION_MESSAGE,
diagnostic: { ...NATIVE_MODEL_REJECTION_DIAGNOSTIC },
};
}, },
/** Diagnostics only: the failed turn that `forTerminal` did not classify. */ /** Diagnostics only: the failed turn that `forTerminal` did not classify. */
unclassifiedTurnFailure(turnId: string | null | undefined, terminal: PrpTerminalState) { unclassifiedTurnFailure(turnId: string | null | undefined, terminal: PrpTerminalState) {
if (!turnId || terminal.turnTerminalState !== "failed" || terminal.runTerminalState !== "failed" || if (!turnId || terminal.turnTerminalState !== "failed" || terminal.runTerminalState !== "failed" ||
turnId === overloadedTurnId || turnId === rejectedTurnId || modelUnavailableTurn(turnId)) return null; turnId === overloadedTurnId || turnId === rejectedTurnId || turnId === missingModelTurnId ||
modelUnavailableTurn(turnId)) return null;
const diagnostic = failedTurns.get(turnId); const diagnostic = failedTurns.get(turnId);
return diagnostic ? { ...diagnostic } : null; return diagnostic ? { ...diagnostic } : null;
}, },
@@ -44,6 +44,8 @@ type ProjectionRun = Pick<typeof heartbeatRuns.$inferSelect,
nativeResumeAt?: Date | null; nativeResumeAt?: Date | null;
/** When the native coordinator saved this run's terminal failure, if it did. */ /** When the native coordinator saved this run's terminal failure, if it did. */
nativeFailedAt?: Date | null; nativeFailedAt?: Date | null;
/** The configured model id the provider rejected (native_provider_model_rejected only). */
rejectedModel?: string | null;
}; };
type ProjectionWait = { id: string; requestedAt: Date; sourceRunId?: string | null; executionWaitReason?: string | null; type ProjectionWait = { id: string; requestedAt: Date; sourceRunId?: string | null; executionWaitReason?: string | null;
preparationStatus?: string | null; preparationRetryAt?: number | string | null; preparationUpdatedAt?: Date | null; preparationStatus?: string | null; preparationRetryAt?: number | string | null; preparationUpdatedAt?: Date | null;
@@ -207,7 +209,8 @@ export function projectTaskInterruption(issue: ProjectionIssue, input: Projectio
: run.errorCode === "native_provider_model_rejected" ? "choose_model" : run.errorCode === "native_provider_model_rejected" ? "choose_model"
: [agent?.pauseReason, project?.pauseReason, company?.pauseReason].some(reason => reason === "budget" || reason === "budget_exceeded") : [agent?.pauseReason, project?.pauseReason, company?.pauseReason].some(reason => reason === "budget" || reason === "budget_exceeded")
? "budget" : input.paused || project?.pausedAt || agent?.status === "paused" || company?.status === "paused" ? "resume" : null; ? "budget" : input.paused || project?.pausedAt || agent?.status === "paused" || company?.status === "paused" ? "resume" : null;
if (userAction) return { ...base, responsibility: "user", state: "waiting_for_user", userAction }; if (userAction) return { ...base, responsibility: "user", state: "waiting_for_user", userAction,
...(userAction === "choose_model" && run.rejectedModel ? { rejectedModel: run.rejectedModel } : {}) };
// The previous controller cannot confirm its own stop. Name the one board // The previous controller cannot confirm its own stop. Name the one board
// action instead of showing the fenced run as working (RC-47). // action instead of showing the fenced run as working (RC-47).
if (input.ownerStopRequired) return { ...base, responsibility: "user", state: "waiting_for_user", userAction: "confirm_owner_stop" }; if (input.ownerStopRequired) return { ...base, responsibility: "user", state: "waiting_for_user", userAction: "confirm_owner_stop" };
@@ -343,6 +346,9 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
and ${heartbeatRuns.error} ~ '^(Execution workspace provision|Workspace seed|Runtime provision) command "' and ${heartbeatRuns.error} ~ '^(Execution workspace provision|Workspace seed|Runtime provision) command "'
then ${heartbeatRuns.error} else null end`, then ${heartbeatRuns.error} else null end`,
errorCode: heartbeatRuns.errorCode, livenessState: heartbeatRuns.livenessState, scheduledRetryAt: heartbeatRuns.scheduledRetryAt, errorCode: heartbeatRuns.errorCode, livenessState: heartbeatRuns.livenessState, scheduledRetryAt: heartbeatRuns.scheduledRetryAt,
// The model this run asked the provider for; configuration, not provider text.
rejectedModel: sql<string | null>`case when ${heartbeatRuns.errorCode} = 'native_provider_model_rejected'
then ${boundedJsonString(sql`${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput'->'provider'->'model'`, 200)} end`,
scheduledRetryReason: heartbeatRuns.scheduledRetryReason, scheduledRetryReason: heartbeatRuns.scheduledRetryReason,
// Candidate only: the coordinator also pushes nextAttemptAt forward while // Candidate only: the coordinator also pushes nextAttemptAt forward while
// it defers an ineligible run. The scheduler's own admission check below // it defers an ineligible run. The scheduler's own admission check below
+4 -1
View File
@@ -170,7 +170,9 @@ export interface ConfigSnapshot {
runs: Array<{ id: string; status: string; createdAt: number | null; startedAt: number | null; errorCode: string | null; error: string | null }>; runs: Array<{ id: string; status: string; createdAt: number | null; startedAt: number | null; errorCode: string | null; error: string | null }>;
interruption: { state: string | null; canRetry: boolean; userAction: string | null; workspaceFailure: string | null; interruption: { state: string | null; canRetry: boolean; userAction: string | null; workspaceFailure: string | null;
/** The named setup command failure (`setup_failed`), as the task page shows it. */ /** The named setup command failure (`setup_failed`), as the task page shows it. */
setupFailure?: string | null; observedRunId: string | null } | null; setupFailure?: string | null;
/** The model id the provider rejected, as the task page names it. */
rejectedModel?: string | null; observedRunId: string | null } | null;
execution: { label: string | null; cause: string | null; nextAction: string | null } | null; execution: { label: string | null; cause: string | null; nextAction: string | null } | null;
/** Pending thread interactions on the task (connection cards and similar). */ /** Pending thread interactions on the task (connection cards and similar). */
cards: Array<{ id: string; kind: string; status: string; serviceSlug: string | null; agentSetup: boolean }>; cards: Array<{ id: string; kind: string; status: string; serviceSlug: string | null; agentSetup: boolean }>;
@@ -186,6 +188,7 @@ export function taskLevelTexts(snapshot: ConfigSnapshot): string[] {
if (interruption?.workspaceFailure) texts.push(interruption.workspaceFailure); if (interruption?.workspaceFailure) texts.push(interruption.workspaceFailure);
if (interruption?.setupFailure) texts.push(interruption.setupFailure); if (interruption?.setupFailure) texts.push(interruption.setupFailure);
if (interruption?.userAction) texts.push(`userAction:${interruption.userAction}`); if (interruption?.userAction) texts.push(`userAction:${interruption.userAction}`);
if (interruption?.rejectedModel) texts.push(`rejected model ${interruption.rejectedModel}`);
if (snapshot.execution?.label) texts.push(snapshot.execution.label); if (snapshot.execution?.label) texts.push(snapshot.execution.label);
if (snapshot.execution?.nextAction) texts.push(snapshot.execution.nextAction); if (snapshot.execution?.nextAction) texts.push(snapshot.execution.nextAction);
for (const card of snapshot.cards) { for (const card of snapshot.cards) {
+9 -2
View File
@@ -233,7 +233,8 @@ async function snapshot(api: ChaosApi, issueId: string, agentId: string, errors:
return { return {
at, reachable: true, issueStatus: String(issue.status), runs: runSamples, at, reachable: true, issueStatus: String(issue.status), runs: runSamples,
interruption: interruption ? { state: interruption.state ?? null, canRetry: interruption.canRetry === true, userAction: interruption.userAction ?? null, interruption: interruption ? { state: interruption.state ?? null, canRetry: interruption.canRetry === true, userAction: interruption.userAction ?? null,
workspaceFailure: interruption.workspaceFailure?.message ?? null, setupFailure: interruption.setupFailure?.message ?? null, observedRunId: interruption.observedRunId ?? null } : null, workspaceFailure: interruption.workspaceFailure?.message ?? null, setupFailure: interruption.setupFailure?.message ?? null,
rejectedModel: interruption.rejectedModel ?? null, observedRunId: interruption.observedRunId ?? null } : null,
execution: projection ? { label: projection.label ?? null, cause: projection.cause ?? null, nextAction: projection.nextAction ?? null } : null, execution: projection ? { label: projection.label ?? null, cause: projection.cause ?? null, nextAction: projection.nextAction ?? null } : null,
cards: (Array.isArray(interactions) ? interactions : []).map(card => ({ id: String(card.id), kind: String(card.kind), status: String(card.status), cards: (Array.isArray(interactions) ? interactions : []).map(card => ({ id: String(card.id), kind: String(card.kind), status: String(card.status),
serviceSlug: card.payload?.serviceSlug ?? null, agentSetup: card.payload?.agentSetup === true })), serviceSlug: card.payload?.serviceSlug ?? null, agentSetup: card.payload?.agentSetup === true })),
@@ -253,7 +254,7 @@ function time(value: unknown): number | null {
return Number.isFinite(at) ? at : null; return Number.isFinite(at) ? at : null;
} }
/** Use what the task offers: complete a pending connection card, then Retry, then the agent's setup Retry. */ /** Use what the task offers: complete a pending connection card, then Retry (the task's, or a rejected model's marker), then the agent's setup Retry. */
async function offeredAction(api: ChaosApi, record: ConfigTaskRecord, resources: TaskResources, current: ConfigSnapshot): Promise<string> { async function offeredAction(api: ChaosApi, record: ConfigTaskRecord, resources: TaskResources, current: ConfigSnapshot): Promise<string> {
const card = current.cards.find(candidate => candidate.status === "pending" && candidate.kind === "connection_intent"); const card = current.cards.find(candidate => candidate.status === "pending" && candidate.kind === "connection_intent");
if (card && resources.connectionId) { if (card && resources.connectionId) {
@@ -264,6 +265,12 @@ async function offeredAction(api: ChaosApi, record: ConfigTaskRecord, resources:
const response = await api.post<Json>(`/api/issues/${record.issueId}/retry`, { idempotencyKey: randomUUID(), observedRunId: current.interruption.observedRunId }); const response = await api.post<Json>(`/api/issues/${record.issueId}/retry`, { idempotencyKey: randomUUID(), observedRunId: current.interruption.observedRunId });
return `clicked Retry (accepted=${String(response?.accepted)})`; return `clicked Retry (accepted=${String(response?.accepted)})`;
} }
// A rejected model offers Retry on the run's "Model unavailable" marker, which retries that run on the agent.
if (current.interruption?.userAction === "choose_model" && current.interruption.observedRunId) {
const response = await api.post<Json>(`/api/agents/${resources.agentId}/wakeup`, { source: "on_demand", triggerDetail: "manual",
reason: "retry_failed_run", failedRunId: current.interruption.observedRunId });
return `clicked the model marker's Retry (run ${String(response?.id ?? "none")})`;
}
if (current.agent && current.agent.lifecycleState !== "ready") { if (current.agent && current.agent.lifecycleState !== "ready") {
await api.post(`/api/agents/${resources.agentId}/lifecycle/retry`, {}); await api.post(`/api/agents/${resources.agentId}/lifecycle/retry`, {});
return `agent setup Retry (lifecycle ${current.agent.lifecycleState})`; return `agent setup Retry (lifecycle ${current.agent.lifecycleState})`;
+22
View File
@@ -2665,6 +2665,28 @@ describe("TaskChatThread runtime transcript selection", () => {
expect(container.textContent).not.toContain("after returning a final response"); expect(container.textContent).not.toContain("after returning a final response");
}); });
it("names the rejected model id that the task interruption carries", () => {
renderThread({
comments: [],
onAdd: async () => {},
interruption: { intentId: null, responsibility: "user", state: "waiting_for_user", nextRetryAt: null,
canRetry: false, userAction: "choose_model", rejectedModel: "chaos-missing-model-1234", sourceRunIds: ["model-missing"], observedRunId: "model-missing" },
linkedRuns: [{
runId: "model-missing", runtimeMode: "native", status: "failed",
errorCode: "native_provider_model_rejected",
error: "The selected model is not available through the current AI connection. Choose a supported model or a compatible AI connection.",
agentId: "agent-1", agentName: "Runner", adapterType: "paperclip_runner",
createdAt: "2026-08-25T18:00:00.000Z", startedAt: "2026-08-25T18:00:00.000Z",
finishedAt: "2026-08-25T18:00:02.000Z",
}],
});
const marker = container.querySelector('[data-testid="task-chat-collapsible-marker"]');
expect(marker?.textContent).toContain("Model unavailable");
flushSync(() => marker!.querySelector<HTMLButtonElement>('button[aria-expanded="false"]')!.click());
expect(container.textContent).toContain('The provider rejected the model "chaos-missing-model-1234".');
expect(container.textContent).toContain("Choose a supported model");
});
it("keeps workspace contention out of the conversation's cancellation markers", () => { it("keeps workspace contention out of the conversation's cancellation markers", () => {
renderThread({ renderThread({
comments: [], comments: [],
+3 -1
View File
@@ -1868,7 +1868,9 @@ export function TaskChatThread(props: TaskChatThreadProps) {
: code === "native_provider_approval_required" : code === "native_provider_approval_required"
? "This operation requires approval, but this runner has no interactive approval handler. Review the operation and update the agent's permission setting before retrying." ? "This operation requires approval, but this runner has no interactive approval handler. Review the operation and update the agent's permission setting before retrying."
: code === "native_provider_model_rejected" : code === "native_provider_model_rejected"
? `${meta?.error || "The provider rejected the selected model."} Choose a supported model or clear the task's model override, then retry.` ? `${props.interruption?.observedRunId === source.id && props.interruption.rejectedModel
? `The provider rejected the model "${props.interruption.rejectedModel}".`
: meta?.error || "The provider rejected the selected model."} Choose a supported model or clear the task's model override, then retry.`
: code === "native_provider_usage_limit" && : code === "native_provider_usage_limit" &&
source.status === "failed" source.status === "failed"
? "The model provider has reached its current usage limit. Try again after the limit resets." ? "The model provider has reached its current usage limit. Try again after the limit resets."