From 3b4c3265ed8cbc00282aee2413349e4ae029b2bf Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 12 Nov 2026 21:42:52 -0600 Subject: [PATCH] fix(recovery): offer Try again on a model-unavailable failure and recheck the model first (#21244) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a native run fails, the task thread shows a failure marker, and some markers offer Try again > - A `native_provider_model_rejected` failure blocks the task for the person and shows "Model unavailable", but it has no Try again control > - Recovery must just work: after the person fixes the model, they need one obvious control that continues the task, and that control must not start a run that is sure to fail > - This pull request shows Try again on the "Model unavailable" marker and makes the retry recheck the selected model before it starts a run > - The benefit is that the person fixes the model, presses Try again, and the task completes, and a too-early press explains which setting to change instead of failing again ## Linked Issue or Issue Description Refs #21210 **What happened?** Paid run 43321773136, `recovery.retry-exhausted` (OpenCode, `openrouter/deepseek/deepseek-v4.1-flash-1017`). The run failed with `native_provider_model_rejected`. The task stayed blocked and the thread showed "Model unavailable", but no Try again control appeared (`task-chat-run-failed-try-again` not found in 30 s). Only the "Run failed" and "Usage limit reached" markers offer Try again. **Expected behavior** The "Model unavailable" marker offers Try again. Pressing it checks the selected model first. If the provider-rejected model is still selected, no run starts, and the person sees which setting to change with a link to it. After the person changes the model, Try again starts one retry of the original run. **Steps to reproduce** 1. Set an agent's model to an id the provider rejects. 2. Assign it a task. The run fails with `native_provider_model_rejected`. 3. Open the task. Before this change, the marker has no Try again. ## What Changed - `server/src/services/native-runtime/model-rejected-retry.ts`: `rejectedModelRetryRefusal` reads the model the failed run sent (`runnerProfileJson.nativeExecutionInput.provider.model`) and the model a new run would use (the task's model override for its assignee, else the agent's model). If they are the same, it returns a 409 refusal with code `native_provider_model_still_unavailable`, the setting to change (`agent_model` or `task_model_override`) and the rejected model. - `server/src/routes/agents.ts`: the exact failed-run retry (`POST /agents/:id/wakeup`, `reason: retry_failed_run`) applies this check before it queues anything. - `ui/src/components/task-chat/adjacent-errors.ts`, `TaskChatThreadView.tsx`: one `isRetryableFailureMarker` rule, now including "Model unavailable". - `ui/src/lib/model-retry-refusal.ts`, `ui/src/pages/IssueDetail.tsx`: a refused retry shows "Model still unavailable" with the server's message and an "Open model settings" link to `/agents//runtime`. A task override refusal names the override in the message. - Eval `recovery.retry-exhausted` (oracle `runtime.retry-exhausted.v7`): the rejected model stays selected after the failure. The person presses Try again (409, toast names the model, its link opens the agent's model settings, no run starts in 20 s, task stays blocked), restores the working model, presses Try again (202), and the one retry of the original run uses the working model, saves the brief and completes the task. 3 runs: failed 1, succeeded 2. The budget-hold and two follow-up chat phases are removed; the case now tests the model recheck. `recovery.yaml` context and expected outcome updated, recorded in `chat-full-catalog-testops-manifest.json` `changedFields`, pinned in its test, descriptors and inventory digest updated. ## Verification - RED on master: `TaskChatThread.test.tsx` "offers Try again on a model-unavailable failure and retries that run" fails (no control). `agent-live-run-routes.test.ts` "refuses a retry while the rejected model is still selected and starts no run" fails (the route queues the run). - GREEN: both pass. `agent-live-run-routes.test.ts` 101/101. - `server/src/services/native-runtime/model-rejected-retry.test.ts` (embedded PostgreSQL), 6/6: refuses while the agent still selects the rejected model (names the agent setting); names the task override; allows the retry when the agent or the override now selects another model, or the run failed for another reason; ignores another company's run. - `ui/src/lib/model-retry-refusal.test.ts`: names the setting and links to `/agents/morgan/runtime`; no link for a task override; other errors fall through to the generic toast. - Server `tsc --noEmit` clean. UI `tsc` clean apart from the pre-existing generated storybook catalog. `node scripts/check-token-gates.mjs`: clean. - Eval: `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog tests/runner-e2e/chat-catalog-domains tests/runner-e2e/chat-*.test.ts`: 339 files pass. New calibration rejects an accepted early retry, a run started while the model is rejected, a refusal without the model, setting link or blocked task, a retry that used the old model, and duplicate or missing retries. `pnpm test:e2e:runner:typecheck`: pass. Manifest test 10/10. `source-sdk-public-inputs --check`: `changed: false`. - Paid run: Runner Full-Stack E2E 43338695850 on throwaway branch `chat-eval/qa-mr-paid`. Result to follow in a comment. ## Risks - The recheck compares the selected model with the model the provider rejected. It does not call the provider. A model the provider rejected stays refused until the person selects another one; a different but also unsupported model starts a run that fails with the same marker again. - The refusal message is a toast (15 s). The marker detail still says to choose a supported model. - No persisted shape changes; runs saved by older builds carry the same `runnerProfileJson` field. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../__tests__/agent-live-run-routes.test.ts | 19 +++ server/src/routes/agents.ts | 3 + .../model-rejected-retry.test.ts | 79 +++++++++ .../native-runtime/model-rejected-retry.ts | 62 +++++++ .../runner-e2e/chat-catalog-contract.test.ts | 2 +- .../runner-e2e/chat-catalog-descriptors.json | 10 +- .../runtime-infrastructure-retry.test.ts | 64 ++++---- .../runtime-infrastructure-retry.ts | 153 +++++++----------- .../chat-full-catalog-testops-manifest.json | 12 +- ...hat-full-catalog-testops-manifest.test.mjs | 4 + tests/runner-e2e/chat-scenarios/recovery.yaml | 19 ++- ui/src/components/TaskChatThread.test.tsx | 27 ++++ .../task-chat/TaskChatThreadView.tsx | 5 +- .../components/task-chat/adjacent-errors.ts | 9 +- ui/src/lib/model-retry-refusal.test.ts | 30 ++++ ui/src/lib/model-retry-refusal.ts | 28 ++++ ui/src/pages/IssueDetail.tsx | 6 +- 17 files changed, 387 insertions(+), 145 deletions(-) create mode 100644 server/src/services/native-runtime/model-rejected-retry.test.ts create mode 100644 server/src/services/native-runtime/model-rejected-retry.ts create mode 100644 ui/src/lib/model-retry-refusal.test.ts create mode 100644 ui/src/lib/model-retry-refusal.ts diff --git a/server/src/__tests__/agent-live-run-routes.test.ts b/server/src/__tests__/agent-live-run-routes.test.ts index 69e704a1099..227245c63f7 100644 --- a/server/src/__tests__/agent-live-run-routes.test.ts +++ b/server/src/__tests__/agent-live-run-routes.test.ts @@ -28,6 +28,7 @@ const mockIssueService = vi.hoisted(() => ({ getByIdentifier: vi.fn(), })); const mockNativeReviewAssignment = vi.hoisted(() => vi.fn()); +const mockRejectedModelRetryRefusal = vi.hoisted(() => vi.fn()); const mockExecutionProjection = vi.hoisted(() => ({ executionProjectionForRun: vi.fn(async () => null), @@ -84,6 +85,10 @@ const mockChatRunRetries = vi.hoisted(() => ({ })); function registerModuleMocks() { + vi.doMock("../services/native-runtime/model-rejected-retry.js", async () => ({ + ...(await vi.importActual("../services/native-runtime/model-rejected-retry.js")), + rejectedModelRetryRefusal: mockRejectedModelRetryRefusal, + })); vi.doMock("../services/native-runtime/native-review-participant.js", async () => ({ ...(await vi.importActual("../services/native-runtime/native-review-participant.js")), getNativeReviewAssignment: mockNativeReviewAssignment, @@ -302,6 +307,7 @@ describe("agent live run routes", () => { mockChatRunRetries.prepareFailedChatRunRetry.mockReset(); mockChatRunRetries.processFailedChatRunRetry.mockReset(); mockNativeReviewAssignment.mockReset().mockResolvedValue(null); + mockRejectedModelRetryRefusal.mockReset().mockResolvedValue(null); mockAccessService.canUser.mockResolvedValue(true); mockAccessService.decide.mockImplementation(async (input: { action?: string }) => ({ allowed: true, @@ -1144,6 +1150,19 @@ describe("agent live run routes", () => { expect(JSON.stringify(mockHeartbeatService.wakeup.mock.calls)).not.toContain("forged"); }); + it("refuses a retry while the rejected model is still selected and starts no run", async () => { + const fixture = createFailedChatRetryDb(false); + mockRejectedModelRetryRefusal.mockResolvedValue({ message: 'The provider rejected the model "bad/model", and it is still selected.', + details: { code: "native_provider_model_still_unavailable", setting: "agent_model", rejectedModel: "bad/model", agentId: routeAgentId, issueId: failedChatIssueId } }); + const res = await requestApp(await createApp(fixture.db), url => request(url).post(`/api/agents/${routeAgentId}/wakeup`).send(retryBody)); + expect(res.status, JSON.stringify(res.body)).toBe(409); + expect(res.body).toMatchObject({ error: expect.stringContaining("still selected"), + details: { code: "native_provider_model_still_unavailable", setting: "agent_model", rejectedModel: "bad/model" } }); + expect(mockRejectedModelRetryRefusal).toHaveBeenCalledExactlyOnceWith(fixture.db, expect.objectContaining({ + failedRun: expect.objectContaining({ id: failedChatRunId }), issue: expect.objectContaining({ id: failedChatIssueId }) })); + expect(mockHeartbeatService.wakeup).not.toHaveBeenCalled(); + }); + it.each(["agent-caller", "denied-task", "foreign-company", "foreign-agent", "foreign-issue", "legacy", "unknown-error", "missing-context", "stale", "resolved", "changed-reviewer"])( "rejects an ineligible exact review retry (%s)", async scenario => { let actor: Record | undefined; diff --git a/server/src/routes/agents.ts b/server/src/routes/agents.ts index 9f3bdfba4c3..94c9de65372 100644 --- a/server/src/routes/agents.ts +++ b/server/src/routes/agents.ts @@ -55,6 +55,7 @@ import { isAiConnectionCompatible } from "@paperclipai/shared"; import { applyConnectorSkills, resolveConnectorAssignments, annotateConnectorSkills, isConnectorSkill } from "../services/connector-runtime.js"; import { getExecutionBlocker } from "../services/execution-blocker.js"; import { canRetryStoppedRun } from "../services/cancelled-native-startup.js"; +import { rejectedModelRetryRefusal } from "../services/native-runtime/model-rejected-retry.js"; import { getNativeReviewAssignment, readNativeReviewAssignmentContext, type NativeReviewAssignmentContext } from "../services/native-runtime/native-review-participant.js"; import { paperclipRunnerTransitionConfig, normalizeLegacyRunnerProvider, isPaperclipRunnerProvider, PAPERCLIP_RUNNER_ACPX_PROFILES } from "@paperclipai/adapter-utils"; import { applyMcpReasoningEffort } from "../services/public-mcp/agent-config.js"; @@ -6304,6 +6305,8 @@ export function agentRoutes( if (issue.assigneeAgentId !== agent.id && !retryNativeReviewContext) { throw conflict("The task is no longer assigned to this agent."); } + const modelRefusal = await rejectedModelRetryRefusal(db, { failedRun, agent, issue }); + if (modelRefusal) throw conflict(modelRefusal.message, modelRefusal.details); if (issue.conversationAgentId) { // Agent Chat has no task description to replay. Recover the exact // request from the selected server-owned run, never caller markers. diff --git a/server/src/services/native-runtime/model-rejected-retry.test.ts b/server/src/services/native-runtime/model-rejected-retry.test.ts new file mode 100644 index 00000000000..74ee1713da2 --- /dev/null +++ b/server/src/services/native-runtime/model-rejected-retry.test.ts @@ -0,0 +1,79 @@ +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { agents, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns, issues } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { rejectedModelRetryRefusal } from "./model-rejected-retry.js"; + +const support = await getEmbeddedPostgresTestSupport(); +const REJECTED = "openrouter/paperclip-eval/unavailable"; +const WORKING = "openrouter/deepseek/deepseek-v4.1-flash-1017"; + +describe.skipIf(!support.supported)("retrying a run whose model the provider rejected", () => { + let database: Awaited>; + let db: ReturnType; + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("paperclip-model-rejected-retry-"); + db = createDb(database.connectionString); + }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); + afterAll(async () => { await db?.$client.end(); await database?.cleanup(); }); + + async function seed(input: { agentModel: string; overrideModel?: string; errorCode?: string }) { + const [company] = await db.insert(companies).values({ name: "Model retry", issuePrefix: `M${randomUUID().slice(0, 4)}` }).returning(); + const [agent] = await db.insert(agents).values({ companyId: company!.id, name: "Morgan", adapterType: "paperclip_runner", + adapterConfig: { provider: "opencode", model: input.agentModel } }).returning(); + const [issue] = await db.insert(issues).values({ companyId: company!.id, title: "Website task", status: "blocked", assigneeAgentId: agent!.id, + assigneeAdapterOverrides: input.overrideModel ? { adapterConfig: { model: input.overrideModel } } : null }).returning(); + const [run] = await db.insert(heartbeatRuns).values({ companyId: company!.id, agentId: agent!.id, status: "failed", runtimeMode: "native", + runtimeModeResolvedAt: new Date(), nativeIssueId: issue!.id, errorCode: input.errorCode ?? "native_provider_model_rejected", + contextSnapshot: { issueId: issue!.id }, + runnerProfileJson: { nativeExecutionInput: { provider: { kind: "opencode", model: REJECTED } } } }).returning(); + return { agent: agent!, issue: issue!, run: run! }; + } + + const check = (f: Awaited>) => rejectedModelRetryRefusal(db, { + failedRun: f.run, agent: f.agent, issue: f.issue }); + + it("refuses while the agent still selects the rejected model and names the agent setting", async () => { + const f = await seed({ agentModel: REJECTED }); + expect(await check(f)).toEqual({ + message: `The provider rejected the model "${REJECTED}", and it is still selected. Choose a supported model in Morgan's settings, then try again.`, + details: { code: "native_provider_model_still_unavailable", setting: "agent_model", rejectedModel: REJECTED, agentId: f.agent.id, issueId: f.issue.id }, + }); + }); + + it("names the task's model override when that override still selects the rejected model", async () => { + const f = await seed({ agentModel: WORKING, overrideModel: REJECTED }); + expect((await check(f))?.details.setting).toBe("task_model_override"); + }); + + it.each([ + { shape: "the agent now selects another model", agentModel: WORKING }, + { shape: "the task override selects another model", agentModel: REJECTED, overrideModel: WORKING }, + { shape: "the run failed for another reason", agentModel: REJECTED, errorCode: "adapter_failed" }, + ])("allows the retry when $shape", async ({ agentModel, overrideModel, errorCode }) => { + expect(await check(await seed({ agentModel, overrideModel, errorCode }))).toBeNull(); + }); + + it("compares the model a new run would resolve, including provider defaults", async () => { + const f = await seed({ agentModel: "" }); + await db.update(agents).set({ adapterConfig: { provider: "acpx", acpxAgent: "grok", model: "grok-build" } }).where(eq(agents.id, f.agent.id)); + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: { provider: { kind: "acpx", model: "grok-4.7" } } } }).where(eq(heartbeatRuns.id, f.run.id)); + const [agent] = await db.select().from(agents).where(eq(agents.id, f.agent.id)); + expect((await rejectedModelRetryRefusal(db, { failedRun: f.run, agent: agent!, issue: f.issue }))?.details).toMatchObject({ setting: "agent_model", rejectedModel: "grok-4.7" }); + }); + + it("refuses when an omitted model resolves to the provider default that was rejected", async () => { + const f = await seed({ agentModel: "" }); + await db.update(agents).set({ adapterConfig: { provider: "acpx", acpxAgent: "claude" } }).where(eq(agents.id, f.agent.id)); + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: { provider: { kind: "acpx", model: "claude-opus-5-5" } } } }).where(eq(heartbeatRuns.id, f.run.id)); + const [agent] = await db.select().from(agents).where(eq(agents.id, f.agent.id)); + expect((await rejectedModelRetryRefusal(db, { failedRun: f.run, agent: agent!, issue: f.issue }))?.details.rejectedModel).toBe("claude-opus-5-5"); + }); + + it("ignores another company's run with the same id shape", async () => { + const f = await seed({ agentModel: REJECTED }); + const other = await seed({ agentModel: REJECTED }); + expect(await rejectedModelRetryRefusal(db, { failedRun: f.run, agent: other.agent, issue: other.issue })).toBeNull(); + }); +}); diff --git a/server/src/services/native-runtime/model-rejected-retry.ts b/server/src/services/native-runtime/model-rejected-retry.ts new file mode 100644 index 00000000000..de674252a85 --- /dev/null +++ b/server/src/services/native-runtime/model-rejected-retry.ts @@ -0,0 +1,62 @@ +import { and, eq, sql } from "drizzle-orm"; +import { heartbeatRuns, type Db } from "@paperclipai/db"; +import { parseIssueAssigneeAdapterOverrides } from "../execution-workspace-policy.js"; +import { projectPaperclipRunnerTaskConfig, resolvePaperclipRunnerProviderProfile } from "./provider-profile.js"; + +export const NATIVE_PROVIDER_MODEL_STILL_UNAVAILABLE = "native_provider_model_still_unavailable"; + +export type RejectedModelSetting = "agent_model" | "task_model_override"; + +export interface RejectedModelRetryRefusal { + message: string; + details: { code: typeof NATIVE_PROVIDER_MODEL_STILL_UNAVAILABLE; setting: RejectedModelSetting; rejectedModel: string; agentId: string; issueId: string | null }; +} + +function record(value: unknown): Record { + return value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; +} + +function modelOf(value: unknown): string | null { + const model = record(value).model; + return typeof model === "string" && model.trim() ? model.trim() : null; +} + +function effectiveModel(agentConfig: Record, overrideConfig: Record, companyId: string): string | null { + const merged = { ...agentConfig, ...overrideConfig }; + try { + const profile = resolvePaperclipRunnerProviderProfile(merged, { companyId }); + const config = profile.backend === "codex_app_server" || profile.backend === "opencode_server" || profile.backend === "acpx_runtime" || profile.backend === "cursor_cloud_api" + ? projectPaperclipRunnerTaskConfig(profile.backend, agentConfig, overrideConfig) : merged; + return modelOf({ model: resolvePaperclipRunnerProviderProfile(config, { companyId }).model }); + } catch { + return modelOf(merged); + } +} + +export async function rejectedModelRetryRefusal(db: Pick, input: { + failedRun: { id: string; status: string; errorCode: string | null; runtimeMode?: string | null }; + agent: { id: string; companyId: string; name: string; adapterConfig: unknown }; + issue: { id: string; assigneeAgentId: string | null; assigneeAdapterOverrides: unknown } | null; +}): Promise { + const { agent, issue, failedRun } = input; + if (failedRun.runtimeMode !== "native" || failedRun.status !== "failed" || failedRun.errorCode !== "native_provider_model_rejected") return null; + const [failed] = await db.select({ + rejectedModel: sql`${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput'->'provider'->>'model'`, + }).from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, agent.companyId), eq(heartbeatRuns.id, failedRun.id), + eq(heartbeatRuns.agentId, agent.id), eq(heartbeatRuns.runtimeMode, "native"), eq(heartbeatRuns.status, "failed"), + eq(heartbeatRuns.errorCode, "native_provider_model_rejected"))); + const rejectedModel = modelOf({ model: failed?.rejectedModel }); + if (!rejectedModel) return null; + const overrideConfig = issue?.assigneeAgentId === agent.id ? parseIssueAssigneeAdapterOverrides(issue.assigneeAdapterOverrides)?.adapterConfig ?? null : null; + const override = modelOf(overrideConfig); + const configured = effectiveModel(record(agent.adapterConfig), record(overrideConfig), agent.companyId); + if (configured !== rejectedModel) return null; + const setting: RejectedModelSetting = override ? "task_model_override" : "agent_model"; + const change = setting === "task_model_override" + ? "Change or clear this task's model override" + : `Choose a supported model in ${agent.name}'s settings`; + return { + message: `The provider rejected the model "${rejectedModel}", and it is still selected. ${change}, then try again.`, + details: { code: NATIVE_PROVIDER_MODEL_STILL_UNAVAILABLE, setting, rejectedModel, agentId: agent.id, issueId: issue?.id ?? null }, + }; +} diff --git a/tests/runner-e2e/chat-catalog-contract.test.ts b/tests/runner-e2e/chat-catalog-contract.test.ts index fa74dd03fd1..e3216b8d7b1 100644 --- a/tests/runner-e2e/chat-catalog-contract.test.ts +++ b/tests/runner-e2e/chat-catalog-contract.test.ts @@ -36,7 +36,7 @@ describe("catalog oracle contract", () => { // execution-environments.choose-capable-target: the second computer is a saved SSH target advertised for Claude Code but never tested. // questions-and-approvals.necessary-question: the answer may shape the saved deliverable as well as the brief. // tasks.assignment-conflict: the query names the Website task, since "this task" in a chat can mean the chat. - expect(chatCatalogInventoryDigest).toBe("dcda709b646b20bdd75c833468d886a111040f130e6699c94fa437ce073bf853"); + expect(chatCatalogInventoryDigest).toBe("ce790541ea5969983f5faf0fafc00a63413684929a2fbd342423224c3ee4b12c"); for (const implementation of chatCatalogCases) expect(chatCatalogScenarios.some(row => row.id === implementation.scenarioId)).toBe(true); }); }); diff --git a/tests/runner-e2e/chat-catalog-descriptors.json b/tests/runner-e2e/chat-catalog-descriptors.json index 6b268176c3e..53434a786bf 100644 --- a/tests/runner-e2e/chat-catalog-descriptors.json +++ b/tests/runner-e2e/chat-catalog-descriptors.json @@ -4513,12 +4513,12 @@ "../../server/src/vendor/paperclip-runner/index.ts", "chat-scenarios/recovery.yaml" ], - "oracleVersion": "paperclip.chat-catalog.runtime.retry-exhausted.v6", + "oracleVersion": "paperclip.chat-catalog.runtime.retry-exhausted.v7", "coverageGaps": [], - "expectedRunCount": 5, - "expectedHumanMessageCount": 3, + "expectedRunCount": 3, + "expectedHumanMessageCount": 1, "turnTimeoutMs": 120000, - "attemptTimeoutMs": 1800000, + "attemptTimeoutMs": 1200000, "authentication": "human-session", "setupPolicy": { "readiness": "preserve-scenario", @@ -4531,7 +4531,7 @@ "kind": "terminal-counts", "terminalStatusCounts": { "failed": 1, - "succeeded": 4 + "succeeded": 2 } }, "modulePath": "./chat-catalog-domains/runtime-infrastructure-retry.js", diff --git a/tests/runner-e2e/chat-catalog-domains/runtime-infrastructure-retry.test.ts b/tests/runner-e2e/chat-catalog-domains/runtime-infrastructure-retry.test.ts index 2e74102de60..86009fdf003 100644 --- a/tests/runner-e2e/chat-catalog-domains/runtime-infrastructure-retry.test.ts +++ b/tests/runner-e2e/chat-catalog-domains/runtime-infrastructure-retry.test.ts @@ -5,30 +5,45 @@ import path from "node:path"; import { gradeInfrastructureRetry, isHumanRetryRequest, releasedRetryListener, retriesOriginalRun } from "./runtime-infrastructure-retry.js"; function valid() { return { scenarioId: "recovery.retry-exhausted", origin: "http://paperclip.test", companyId: "company", leadId: "ceo", agents: [], links: ["/COM/issues/COM-2"], humanQueries: [], reply: "", tasks: [{ id: "website", status: "done" }], activity: [], - runs: [{ id: "exhausted", status: "failed" }, ...["canonical", "held", "retry", "final"].map(id => ({ id, status: "succeeded" }))], - state: { task: { id: "website", identifier: "COM-2", status: "blocked" }, queryRecoveryRead: true, heldRecoveryRead: true, finalRecoveryRead: true, worker: { id: "morgan" }, - failure: { id: "exhausted", companyId: "company", agentId: "morgan", status: "failed", errorCode: "native_provider_model_rejected" }, freshContinuationsBefore: 0, successorRunsBefore: 0, - queryReply: "Use the Retry button on the original failed run; this requires your human board session and rechecks current gates.", retryVisible: true, - budgetHeldBeforeRetry: true, heldRetryStatus: 202, noDuplicateRetryWhileHeld: true, noAdmissionUnderBudget: true, failedRunVisibleWhileHeld: true, heldReplyAttributed: true, - heldReply: "The retry could not start because Morgan's current budget hold blocks admission. The previous native failure remains visible.", - budgetRestoredByHuman: true, retryBoundToOriginal: true, retryRunCount: 1, retryAdmissionAfterGateClear: true, - originalFailureRetained: true, actualRetryResult: true, finalReplyAttributed: true, finalReply: "Website is complete after the eligible retry. The original exhausted attempt remains inspectable." } }; + runs: [{ id: "rejected", status: "failed" }, ...["canonical", "retry"].map(id => ({ id, status: "succeeded" }))], + state: { task: { id: "website", identifier: "COM-2", status: "blocked" }, queryRecoveryRead: true, worker: { id: "morgan" }, + failure: { id: "rejected", companyId: "company", agentId: "morgan", status: "failed", errorCode: "native_provider_model_rejected" }, freshContinuationsBefore: 0, successorRunsBefore: 0, + queryReply: "Use Try again on the Website task once Morgan's model is fixed.", retryVisible: true, + refusedRetryStatus: 409, refusalCode: "native_provider_model_still_unavailable", refusalNamedRejectedModel: true, refusalLinksModelSetting: true, + noRunWhileModelUnavailable: true, taskBlockedAfterRefusal: true, + modelFixedByHuman: true, fixedRetryStatus: 202, retryRunCount: 1, retryBoundToOriginal: true, retryAdmissionAfterFix: true, retryUsedFixedModel: true, + originalFailureRetained: true, actualRetryResult: true } }; } -it("accepts actual budget rejection, one eligible retry and retained failure", () => expect(gradeInfrastructureRetry(valid()).passed).toBe(true)); -it("accepts substantive natural human guidance and later factual reports", () => { const o = valid(); o.state.queryReply = "You can try again using the task's supported Retry control. It checks today's execution conditions."; o.state.heldReply = "Morgan has reached the spending cap, so no new execution was admitted."; o.state.finalReply = "The Website delivery is ready; its earlier failed attempts remain in the run history."; expect(gradeInfrastructureRetry(o).passed).toBe(true); }); +it("accepts a refused Try again while the model is rejected, then one retry after the fix that completes the task", () => expect(gradeInfrastructureRetry(valid()).passed).toBe(true)); +it("accepts reworded guidance on the same saved evidence", () => { const o = valid(); o.state.queryReply = "Fix the model in Morgan's settings and press Try again on COM-2."; expect(gradeInfrastructureRetry(o).passed).toBe(true); }); it("rejects guidance whose run never read the saved recovery record", () => { const o = valid(); o.state.queryRecoveryRead = false; expect(gradeInfrastructureRetry(o).passed).toBe(false); }); it("rejects guidance that does not link the original task", () => { const o = valid(); o.links = []; expect(gradeInfrastructureRetry(o).passed).toBe(false); }); -it("rejects a retry admitted under the actual hard stop", () => { const o = valid(); o.state.noAdmissionUnderBudget = false; expect(gradeInfrastructureRetry(o).passed).toBe(false); }); -it("rejects a hidden original failure while Retry cannot start", () => { const o = valid(); o.state.failedRunVisibleWhileHeld = false; expect(gradeInfrastructureRetry(o).passed).toBe(false); }); -it("rejects a held-phase answer that never read the task's runs", () => { const o = valid(); o.state.heldRecoveryRead = false; expect(gradeInfrastructureRetry(o).passed).toBe(false); }); -it("rejects a final answer that never read the task's runs", () => { const o = valid(); o.state.finalRecoveryRead = false; expect(gradeInfrastructureRetry(o).passed).toBe(false); }); -it("rejects an unbound held-phase reply", () => { const o = valid(); o.state.heldReplyAttributed = false; expect(gradeInfrastructureRetry(o).passed).toBe(false); }); +it("rejects a model failure with no visible Try again", () => { const o = valid(); o.state.retryVisible = false; expect(gradeInfrastructureRetry(o).passed).toBe(false); }); +it("rejects a Try again that starts a doomed run while the rejected model is still selected", () => { + const accepted = valid(); accepted.state.refusedRetryStatus = 202; expect(gradeInfrastructureRetry(accepted).passed).toBe(false); + const started = valid(); started.state.noRunWhileModelUnavailable = false; expect(gradeInfrastructureRetry(started).passed).toBe(false); +}); +it("rejects a refusal for another reason, or one that does not name the model or link its setting", () => { + const code = valid(); code.state.refusalCode = "conflict"; expect(gradeInfrastructureRetry(code).passed).toBe(false); + const named = valid(); named.state.refusalNamedRejectedModel = false; expect(gradeInfrastructureRetry(named).passed).toBe(false); + const linked = valid(); linked.state.refusalLinksModelSetting = false; expect(gradeInfrastructureRetry(linked).passed).toBe(false); +}); +it("rejects a refusal that unblocks the task", () => { const o = valid(); o.state.taskBlockedAfterRefusal = false; expect(gradeInfrastructureRetry(o).passed).toBe(false); }); +it("rejects a retry the fixed model never reached", () => { + const unfixed = valid(); unfixed.state.modelFixedByHuman = false; expect(gradeInfrastructureRetry(unfixed).passed).toBe(false); + const model = valid(); model.state.retryUsedFixedModel = false; expect(gradeInfrastructureRetry(model).passed).toBe(false); + const early = valid(); early.state.retryAdmissionAfterFix = false; expect(gradeInfrastructureRetry(early).passed).toBe(false); +}); +it("rejects a refused retry after the fix", () => { for (const status of [200, 409, 500]) { const o = valid(); o.state.fixedRetryStatus = status; expect(gradeInfrastructureRetry(o).passed).toBe(false); } }); it("rejects retrying a different task or failed attempt", () => { const o = valid(); o.state.retryBoundToOriginal = false; expect(gradeInfrastructureRetry(o).passed).toBe(false); }); -it("rejects duplicate eligible retries", () => { const o = valid(); o.state.retryRunCount = 2; expect(gradeInfrastructureRetry(o).passed).toBe(false); }); -it("rejects admission predating actual gate restoration", () => { const o = valid(); o.state.retryAdmissionAfterGateClear = false; expect(gradeInfrastructureRetry(o).passed).toBe(false); }); -it("rejects rewriting the exhausted attempt into success", () => { const o = valid(); o.state.originalFailureRetained = false; expect(gradeInfrastructureRetry(o).passed).toBe(false); }); +it("rejects duplicate or missing retries", () => { + const two = valid(); two.state.retryRunCount = 2; expect(gradeInfrastructureRetry(two).passed).toBe(false); + const none = valid(); none.state.retryRunCount = 0; expect(gradeInfrastructureRetry(none).passed).toBe(false); + const extra = valid(); extra.runs.push({ id: "again", status: "succeeded" }); expect(gradeInfrastructureRetry(extra).passed).toBe(false); +}); +it("rejects rewriting the failed attempt into success", () => { const o = valid(); o.state.originalFailureRetained = false; expect(gradeInfrastructureRetry(o).passed).toBe(false); }); it("rejects completion without the actual retry's saved output", () => { const o = valid(); o.state.actualRetryResult = false; expect(gradeInfrastructureRetry(o).passed).toBe(false); }); -it("rejects borrowing the pre-retry reply for final completion", () => { const o = valid(); o.state.finalReplyAttributed = false; expect(gradeInfrastructureRetry(o).passed).toBe(false); }); +it("rejects an unfinished task", () => { const o = valid(); o.tasks[0]!.status = "blocked"; expect(gradeInfrastructureRetry(o).passed).toBe(false); }); it("rejects absent native retry evidence", () => expect(gradeInfrastructureRetry(undefined).passed).toBe(false)); it("rejects an infrastructure loss, which continues automatically, in place of a person-owned failure", () => { const o = valid(); o.state.failure.errorCode = "native_runner_process_exited"; expect(gradeInfrastructureRetry(o).passed).toBe(false); }); @@ -51,17 +66,6 @@ it("accepts the task's own Retry request as the human Retry, for this task and a expect(isHumanRetryRequest(request("GET", "/api/issues/website/retry"), "website", "morgan")).toBe(false); expect(isHumanRetryRequest(request("POST", "/api/issues/website/retry-workspace-export"), "website", "morgan")).toBe(false); }); -it("requires the one held Retry to have been accepted, not refused", () => { - for (const status of [200, 409, 500]) { const o = valid(); o.state.heldRetryStatus = status; expect(gradeInfrastructureRetry(o).passed).toBe(false); } -}); -it("rejects a held Retry that dispatches before the gate clears", () => { - const early = valid(); early.state.noAdmissionUnderBudget = false; expect(gradeInfrastructureRetry(early).passed).toBe(false); - const predates = valid(); predates.state.retryAdmissionAfterGateClear = false; expect(gradeInfrastructureRetry(predates).passed).toBe(false); -}); -it("rejects a held Retry that never dispatches after the gate clears", () => { const o = valid(); o.state.retryRunCount = 0; expect(gradeInfrastructureRetry(o).passed).toBe(false); }); -it("rejects a second Retry control offered while the saved Retry waits", () => { const o = valid(); o.state.noDuplicateRetryWhileHeld = false; expect(gradeInfrastructureRetry(o).passed).toBe(false); }); - - it("binds the brief to the released Retry's own workspace, and only that one", async () => { const root = await mkdtemp(path.join(tmpdir(), "retry-listener-")), outside = await mkdtemp(path.join(tmpdir(), "retry-outside-")); try { diff --git a/tests/runner-e2e/chat-catalog-domains/runtime-infrastructure-retry.ts b/tests/runner-e2e/chat-catalog-domains/runtime-infrastructure-retry.ts index 4360e89ba23..978cf8fb7b5 100644 --- a/tests/runner-e2e/chat-catalog-domains/runtime-infrastructure-retry.ts +++ b/tests/runner-e2e/chat-catalog-domains/runtime-infrastructure-retry.ts @@ -1,10 +1,9 @@ -import { chatCatalogNativePhaseDeadline, chatCatalogNativePhaseIsTimely, chatCatalogNativePhaseRemainingBudget } from "../chat-catalog-contract.js"; +import { chatCatalogNativePhaseDeadline, chatCatalogNativePhaseIsTimely } from "../chat-catalog-contract.js"; import { expect, type Page } from "@playwright/test"; import { writeFile } from "node:fs/promises"; import type { HeartbeatRun } from "../../../packages/shared/src/types/heartbeat.js"; import type { Issue } from "../../../packages/shared/src/types/issue.js"; import { pollUntil } from "../api.js"; -import { sendChatMessage } from "../chat-flow.js"; import { checks, type Entity } from "./internal-support.js"; import { prepareRuntimeWorkerGate, releasedRetryListener, releaseRuntimeWorker, retriesOriginalRun, type RuntimeBriefGate } from "./runtime-busy-worker.js"; import { controllerReadRecovery, linksSourceTask, sourceRunRecovery, type SourceRecoveryActions } from "./runtime-infrastructure.js"; @@ -35,6 +34,9 @@ export async function personOwnedNativeFailure(ctx: ChatCatalogContext) { return { issueId: task.id }; }); if (!task || failure.failedIssueId !== task.id) throw new Error("Person-owned failure did not stop the fixture's own Website task"); + const workingModel = String(failure.originalConfiguration.adapterConfig?.model ?? ""); + const kept = await ctx.input.api.patch(`/api/agents/${worker.id}`, { adapterConfig: { ...failure.originalConfiguration.adapterConfig, model: failure.failedModel } }); + if (kept.adapterConfig?.model !== failure.failedModel || !workingModel) throw new Error("The rejected model is not the agent's selected model before Retry"); const companyId = ctx.input.fixtures.company.id, failedRunId = failure.failedRun.id, websiteId = task.id; const load = async () => { const [current, run, recovery, runs] = await Promise.all([ctx.input.api.get(`/api/issues/${websiteId}`), ctx.input.api.get(`/api/heartbeat-runs/${failedRunId}`), @@ -54,34 +56,33 @@ export async function personOwnedNativeFailure(ctx: ChatCatalogContext) { activeRecovery: before.active ? { id: before.active.id, ownerType: before.active.ownerType, cause: before.active.cause, status: before.active.status } : null, freshContinuations: settled.freshContinuations, successorRuns: settled.successorRuns }); return { worker, gate, task: settled.current, workerRunId: failedRunId, failure: settled.run, recoveryBefore: before.active ?? null, - freshContinuationsBefore: settled.freshContinuations, successorRunsBefore: settled.successorRuns, failureKind: failure.failureKind }; + freshContinuationsBefore: settled.freshContinuations, successorRunsBefore: settled.successorRuns, failureKind: failure.failureKind, + rejectedModel: failure.failedModel, workingModel, originalAdapterConfig: failure.originalConfiguration.adapterConfig }; } +export const MODEL_STILL_UNAVAILABLE_CODE = "native_provider_model_still_unavailable"; +const MODEL_SETTINGS_PATH = /\/agents\/[^/]+\/runtime$/; export function gradeInfrastructureRetry(value: unknown) { const c = checks(), o = runtimeObservation(value); - c.add("actual-person-owned-failure-retry-and-current-gates", !!o, "The genuine person-owned native failure, real human Retry controls and independently observed admission gates are required."); if (!o) return c.result(); + c.add("actual-person-owned-failure-retry-and-model-recheck", !!o, "The genuine person-owned native failure, the real human Try again control and the independently observed model recheck are required."); if (!o) return c.result(); const s = o.state; c.add("actual-person-owned-failure-awaits-retry", isPersonOwnedRetryFailure({ run: s.failure as HeartbeatRun, companyId: o.companyId, agentId: (s.worker as Entity)?.id, taskStatus: (s.task as Issue)?.status, freshContinuations: Number(s.freshContinuationsBefore), successorRuns: Number(s.successorRunsBefore) }), - "The original task stopped for a reason only the person can resolve, stays blocked for the person, and the system starts no continuation of its own; an infrastructure loss continues automatically and is insufficient."); + "The original task stopped because the provider rejected its selected model, stays blocked for the person, and the system starts no continuation of its own."); c.add("ordinary-human-retry-guidance", s.queryRecoveryRead === true && s.retryVisible === true && linksSourceTask(o, s.task as Entity), - "The canonical reply comes from a run that read the saved recovery record and links the original task, where the real human Retry control is visible."); - c.add("actual-closed-gate-prevents-new-execution", s.budgetHeldBeforeRetry === true && [200, 202, 409].includes(Number(s.heldRetryStatus)) - && s.noAdmissionUnderBudget === true && s.failedRunVisibleWhileHeld === true && s.heldReplyAttributed === true && s.heldRecoveryRead === true, - "The first real Retry cannot start under the independently saved budget hold, retains failure visibly, and its own subsequent chat run reads the task's actual runs before replying."); - // The one accepted Retry is the held one: the product saves it and starts it - // once the gate clears. No second Retry exists, and none is offered. - c.add("one-accepted-eligible-retry-preserves-original-failure", s.budgetRestoredByHuman === true && Number(s.heldRetryStatus) === 202 && s.noDuplicateRetryWhileHeld === true - && s.retryBoundToOriginal === true && s.retryRunCount === 1 - && s.retryAdmissionAfterGateClear === true && s.originalFailureRetained === true && s.actualRetryResult === true + "The canonical reply comes from a run that read the saved recovery record and links the original task, where the real Try again control is visible on the model failure."); + c.add("retry-rechecks-the-still-rejected-model", Number(s.refusedRetryStatus) === 409 && s.refusalCode === MODEL_STILL_UNAVAILABLE_CODE + && s.refusalNamedRejectedModel === true && s.refusalLinksModelSetting === true && s.noRunWhileModelUnavailable === true && s.taskBlockedAfterRefusal === true, + "Try again while the rejected model is still selected starts no run, keeps the task blocked and tells the person which setting to change, with a working link to it."); + c.add("one-retry-after-model-fixed-completes", s.modelFixedByHuman === true && Number(s.fixedRetryStatus) === 202 && s.retryRunCount === 1 + && s.retryBoundToOriginal === true && s.retryAdmissionAfterFix === true && s.retryUsedFixedModel === true + && s.originalFailureRetained === true && s.actualRetryResult === true && o.tasks.length === 1 && o.tasks[0]?.id === (s.task as Entity).id && o.tasks[0]?.status === "done" - && o.runs.length === 5 && o.runs.filter(row => row.status === "failed").length === 1 && o.runs.filter(row => row.status === "succeeded").length === 4, - "Once the actual human restores the gate, exactly one eligible native retry uses the server-owned failed task context, retains that original failure and saves the real Website result."); - c.add("own-final-phase-reports-actual-result", s.finalReplyAttributed === true && s.finalRecoveryRead === true, - "A separately declared factual follow-up receives its own native reply from a run that read the task's actual runs."); + && o.runs.length === 3 && o.runs.filter(row => row.status === "failed").length === 1 && o.runs.filter(row => row.status === "succeeded").length === 2, + "After the person selects a working model, one Try again starts exactly one retry of the original run with that model, retains the original failure and saves the real Website result."); return c.result(); } export const runtimeInfrastructureRetryCases = [runtimeDefinition("recovery.retry-exhausted", personOwnedNativeFailure, gradeInfrastructureRetry, { - oracleVersion: "paperclip.chat-catalog.runtime.retry-exhausted.v6", authentication: "human-session", expectedHumanMessageCount: 3, expectedRunCount: 5, attemptTimeoutMs: 1_800_000, + oracleVersion: "paperclip.chat-catalog.runtime.retry-exhausted.v7", authentication: "human-session", expectedHumanMessageCount: 1, expectedRunCount: 3, attemptTimeoutMs: 1_200_000, setupPolicy: { readiness: "preserve-scenario", invariantCheckIds: ["lead-ready", "person-owned-failure-awaits-retry"] }, async verifySetup(ctx, state) { const c = checks(), lead = await ctx.input.api.get(`/api/agents/${ctx.input.fixtures.agent.id}`), task = state.task as Issue, worker = state.worker as Entity; const [current, run, recovery, runs] = await Promise.all([ctx.input.api.get(`/api/issues/${task.id}`), ctx.input.api.get(`/api/heartbeat-runs/${state.workerRunId}`), @@ -90,99 +91,69 @@ export const runtimeInfrastructureRetryCases = [runtimeDefinition("recovery.retr c.add("person-owned-failure-awaits-retry", isPersonOwnedRetryFailure({ run, companyId: ctx.input.fixtures.company.id, agentId: worker.id, taskStatus: current.status, freshContinuations: (recovery.actions ?? []).filter(action => !!(action.evidence as Record).freshContinuation).length, successorRuns: runs.filter(row => row.agentId === worker.id && row.id !== run.id).length }), "The original task waits for the person's Retry after a failure only the person can resolve."); return c.result(); }, - runPolicy: { kind: "terminal-counts", terminalStatusCounts: { failed: 1, succeeded: 4 } }, + runPolicy: { kind: "terminal-counts", terminalStatusCounts: { failed: 1, succeeded: 2 } }, sourceFiles: ["chat-catalog-domains/runtime-infrastructure-retry.ts", "chat-catalog-domains/runtime-infrastructure.ts", "chat-catalog-domains/api-read-receipts.ts", "chat-catalog-domains/runtime-native-fault.ts", "chat-catalog-domains/runtime-native-failure.ts", "chat-catalog-domains/runtime-busy-worker.ts", "chat-catalog-domains/runtime-support.ts", "../../server/src/services/hot-restart.ts", "../../server/src/vendor/paperclip-runner/index.ts", "chat-scenarios/recovery.yaml"], async act(ctx, state) { const task = state.task as Issue, worker = state.worker as RuntimeAgent; const initial = await pollUntil({ label: "Canonical human Retry guidance after the person-owned failure", deadlineAt: chatCatalogNativePhaseDeadline(ctx, Date.now() + 150_000), load: async () => { await ctx.refreshIssue(); return { source: ctx.issue(), runs: await ctx.allRuns() }; }, accept: v => chatCatalogNativePhaseIsTimely(ctx) && (v.source?.conversationState === "waiting" && v.runs.length === 2 && v.runs.some(row => row.id === state.workerRunId && row.status === "failed") && v.runs.some(row => row.agentId === ctx.input.fixtures.agent.id && row.status === "succeeded")) }); - state.queryReply = (await observeRuntime(ctx, state)).reply; const source = ctx.input.page.url(), root = `/api/companies/${ctx.input.fixtures.company.id}`; + state.queryReply = (await observeRuntime(ctx, state)).reply; const source = ctx.input.page.url(); + state.queryRecoveryRead = await controllerReadRecovery(ctx, initial.runs.find(row => row.agentId === ctx.input.fixtures.agent.id)?.id, task.id, String(state.workerRunId), task.identifier); const taskURL = `/${ctx.input.fixtures.company.issuePrefix}/issues/${task.identifier ?? task.id}`; await ctx.input.page.goto(taskURL, { waitUntil: "domcontentloaded" }); const retry = humanRetryControl(ctx.input.page); await expect(retry).toBeVisible(); state.retryVisible = true; - // A zero amount disables enforcement. Hold with a real cap and one explicit - // synthetic accounting receipt over it, imported through the public path; - // this is not model/provider spend. Restoring 500 cents later lifts it. - await ctx.input.api.post(`${root}/budgets/policies`, { scopeType: "agent", scopeId: worker.id, amount: 100, isActive: true, hardStopEnabled: true, unpricedUsagePolicy: "allow" }); - const budgetReceipt = await ctx.input.api.post<{ id: string }>(`${root}/cost-events`, { agentId: worker.id, provider: "synthetic-eval-import", biller: "synthetic-eval-import", - model: "imported-fixture-receipt", billingType: "metered_api", costStatus: "reported", costCents: 125, inputTokens: 0, outputTokens: 0, - occurredAt: new Date().toISOString(), idempotencyKey: `retry-exhausted-held:${ctx.input.nonce}` }); - if (typeof budgetReceipt.id !== "string" || !budgetReceipt.id) throw new Error("Missing public synthetic accounting fixture receipt"); - await ctx.input.evidence("runtime-retry-imported-budget-context.json", { agentId: worker.id, receiptId: budgetReceipt.id, origin: "public-imported-synthetic-fixture", - importedFixtureCostCents: 125, hardCapCents: 100, modelProviderUsage: "Not model/provider usage; explicit public synthetic accounting fixture" }); - const held = await pollUntil({ label: "Real current budget hold before first human Retry", deadlineAt: Date.now() + 45_000, load: () => ctx.input.api.get(`/api/agents/${worker.id}`), accept: row => row.lifecycleHolds?.includes("budget") === true }); - state.budgetHeldBeforeRetry = held.lifecycleHolds?.includes("budget") === true; - const rejected = ctx.input.page.waitForResponse(r => isHumanRetryRequest(r.request(), task.id, worker.id)); - await retry.click(); state.heldRetryStatus = (await rejected).status(); const workerRetryRuns = async () => (await ctx.allRuns()).filter(row => row.agentId === worker.id && row.id !== state.workerRunId).length; - const holdSamples = [await workerRetryRuns()]; - // Wait on the open page as a person would, without reloading: the task - // keeps the failure and names the budget wait. The saved Retry starts by - // itself later, so no second Retry control is offered. - const waiting = ctx.input.page.getByTestId("task-execution-waiting"); - await expect(waiting).toBeVisible({ timeout: 60_000 }); - const waitingText = await waiting.textContent() ?? ""; - state.failedRunVisibleWhileHeld = waitingText.includes("Couldn’t finish this attempt") && /budget/i.test(waitingText); - state.noDuplicateRetryWhileHeld = await humanRetryControl(ctx.input.page).count() === 0; - holdSamples.push(await workerRetryRuns()); - await ctx.input.capture("actual-retry-gate-held", "Original native failure remains visible when current budget prevents Retry admission", "runtime-retry-budget-held.png"); - await ctx.input.page.goto(source, { waitUntil: "domcontentloaded" }); - const heldQuery = `I tried the supported Retry on Website ${task.identifier ?? task.id} while Morgan's actual budget is held. Did a new attempt start, and what current gate is stopping it?`; - await sendChatMessage(ctx.input.page, heldQuery, chatCatalogNativePhaseRemainingBudget(ctx)); - const heldPhase = await pollUntil({ label: "Own chat phase reports actual non-admission under current budget", deadlineAt: chatCatalogNativePhaseDeadline(ctx, Date.now() + 150_000), - load: async () => { await ctx.refreshIssue(); return { source: ctx.issue(), runs: await ctx.allRuns() }; }, accept: v => chatCatalogNativePhaseIsTimely(ctx) && (v.source?.conversationState === "waiting" && v.runs.length === 3 && v.runs.filter(row => row.agentId === ctx.input.fixtures.agent.id).every(row => row.status === "succeeded")) }); - const phaseReply = async (query: string, runId: string) => { - const comments = await ctx.input.api.get(`/api/issues/${ctx.issue()!.id}/comments?order=asc`), human = comments.filter(row => !row.authorAgentId && row.body === query), replies = comments.filter(row => row.authorAgentId === ctx.input.fixtures.agent.id && row.createdByRunId === runId); - const detail = await ctx.input.api.get<{ runnerProfileJson?: { nativeExecutionInput?: { task?: { prompt?: string } } } }>(`/api/heartbeat-runs/${runId}`); - return { attributed: human.length === 1 && replies.length > 0 && detail.runnerProfileJson?.nativeExecutionInput?.task?.prompt?.includes(human[0]!.id) === true, body: replies.map(row => row.body).join("\n") }; - }; - const heldRun = heldPhase.runs.find(row => !initial.runs.some(prior => prior.id === row.id) && row.agentId === ctx.input.fixtures.agent.id); - if (!heldRun) throw new Error("Held retry response lacks an actual subsequent native turn"); - const heldReply = await phaseReply(heldQuery, heldRun.id); state.heldReply = heldReply.body; state.heldReplyAttributed = heldReply.attributed; - state.queryRecoveryRead = await controllerReadRecovery(ctx, initial.runs.find(row => row.agentId === ctx.input.fixtures.agent.id)?.id, task.id, String(state.workerRunId), task.identifier); - state.heldRecoveryRead = await controllerReadRecovery(ctx, heldRun.id, task.id, String(state.workerRunId), task.identifier); - // Sampled at the click, on the waiting page, and after the held chat phase. - holdSamples.push(await workerRetryRuns()); state.heldWorkerRetryRunSamples = holdSamples; - state.noAdmissionUnderBudget = holdSamples.every(count => count === 0); - state.gatesClearedAt = Date.now(); - await ctx.input.api.post(`${root}/budgets/policies`, { scopeType: "agent", scopeId: worker.id, amount: 500, isActive: true, hardStopEnabled: true, unpricedUsagePolicy: "allow" }); - let restored = await ctx.input.api.get(`/api/agents/${worker.id}`); - if (restored.status === "error") { await ctx.input.api.post(`/api/agents/${worker.id}/clear-error`, {}); restored = await ctx.input.api.get(`/api/agents/${worker.id}`); } - state.budgetRestoredByHuman = !restored.lifecycleHolds?.includes("budget"); - // No second click: the held Retry is the one accepted Retry. A Retry may - // start in a fresh isolated workspace when the task does not own its - // directory, so bind the brief to the workspace the released run actually - // uses: wait for its listener there, then supply the brief. + const refused = ctx.input.page.waitForResponse(r => isHumanRetryRequest(r.request(), task.id, worker.id)); + await retry.click(); const refusal = await refused; state.refusedRetryStatus = refusal.status(); + const refusalBody = await refusal.json().catch(() => null) as { error?: unknown; details?: { code?: unknown; rejectedModel?: unknown } } | null; + state.refusalCode = refusalBody?.details?.code ?? null; + const notice = ctx.input.page.getByRole("listitem").filter({ hasText: "Model still unavailable" }).first(); + await expect(notice).toBeVisible({ timeout: 10_000 }); + state.refusalNamedRejectedModel = (await notice.textContent() ?? "").includes(String(state.rejectedModel)); + const settingsLink = notice.getByRole("link").first(); + const linked = await settingsLink.count() === 1; + await ctx.input.capture("model-still-unavailable-refusal", "Try again names the model setting while the rejected model is still selected", "runtime-retry-model-refused.png"); + const refusalSamples = [await workerRetryRuns()]; + if (linked) { + await settingsLink.click(); + await ctx.input.page.waitForURL(url => MODEL_SETTINGS_PATH.test(url.pathname), { timeout: 15_000 }).catch(() => undefined); + } + state.refusalLinksModelSetting = linked && MODEL_SETTINGS_PATH.test(new URL(ctx.input.page.url()).pathname) + && new URL(ctx.input.page.url()).pathname.includes(`/agents/${worker.urlKey ?? worker.id}/`); + await ctx.input.page.waitForTimeout(20_000); + refusalSamples.push(await workerRetryRuns()); state.refusalRunSamples = refusalSamples; + state.noRunWhileModelUnavailable = refusalSamples.every(count => count === 0); + state.taskBlockedAfterRefusal = (await ctx.input.api.get(`/api/issues/${task.id}`)).status === "blocked"; + const fixed = await ctx.input.api.patch(`/api/agents/${worker.id}`, { adapterConfig: { ...(state.originalAdapterConfig as Record), model: state.workingModel } }); + state.modelFixedByHuman = fixed.adapterConfig?.model === state.workingModel && fixed.adapterConfig?.model !== state.rejectedModel; + state.modelFixedAt = Date.now(); + await ctx.input.page.goto(taskURL, { waitUntil: "domcontentloaded" }); + const again = humanRetryControl(ctx.input.page); await expect(again).toBeVisible(); + const accepted = ctx.input.page.waitForResponse(r => isHumanRetryRequest(r.request(), task.id, worker.id)); + await again.click(); state.fixedRetryStatus = (await accepted).status(); const gateName = (state.gate as RuntimeBriefGate).gateName!; - const released = await pollUntil({ label: "Released Retry listens for the brief in its own workspace", deadlineAt: Date.now() + 180_000, + const released = await pollUntil({ label: "The Retry after the model fix listens for the brief in its own workspace", deadlineAt: Date.now() + 180_000, load: async () => { const retries = (await ctx.allRuns()).filter(row => row.agentId === worker.id && row.id !== state.workerRunId); const run = retries.length === 1 ? await ctx.input.api.get(`/api/heartbeat-runs/${retries[0]!.id}`) : null; - return { retries: retries.length, listener: run ? await releasedRetryListener(run, { companyId: ctx.input.fixtures.company.id, workerId: worker.id, + return { retries: retries.length, run, listener: run ? await releasedRetryListener(run, { companyId: ctx.input.fixtures.company.id, workerId: worker.id, taskId: task.id, originalRunId: String(state.workerRunId), gateName }) : null }; }, accept: v => v.retries === 1 && !!v.listener, - reject: v => v.retries > 1 ? "The held Retry started more than once" : undefined }); - const brief = { gate: released.listener!.gate }; - state.retryWorkspaceId = released.listener!.workspaceId; state.retryWorkspaceFresh = released.listener!.workspaceId !== (state.gate as RuntimeBriefGate).boundWorkspaceId; - await writeFile(brief.gate, `The authorized human has restored the execution gate. Use the preserved task context and save a non-plan website-delivery document with the exact reference Retry website ${ctx.input.nonce}, a semantic single-page Acorn website outline and an accessible signup action, then complete this original task normally. Do not create more tasks or external effects.`, "utf8"); - const completed = await pollUntil({ label: "The held Retry starts once, by itself, after actual gate restoration", deadlineAt: Date.now() + 240_000, + reject: v => v.retries > 1 ? "Try again started more than one run" : v.run && ["failed", "cancelled", "timed_out"].includes(v.run.status) ? "The retry after the model fix stopped" : undefined }); + state.retryWorkspaceId = released.listener!.workspaceId; + await writeFile(released.listener!.gate, `The authorized human fixed the model. Use the preserved task context and save a non-plan website-delivery document with the exact reference Retry website ${ctx.input.nonce}, a semantic single-page Acorn website outline and an accessible signup action, then complete this original task normally. Do not create more tasks or external effects.`, "utf8"); + const completed = await pollUntil({ label: "The one Retry after the model fix completes the original task", deadlineAt: Date.now() + 240_000, load: async () => ({ task: await ctx.input.api.get(`/api/issues/${task.id}`), runs: await ctx.allRuns() }), - accept: v => v.task.status === "done" && v.runs.length === 4 && v.runs.filter(row => row.id !== state.workerRunId).every(row => row.status === "succeeded"), - reject: v => v.runs.length > 4 || v.runs.some(row => row.id !== state.workerRunId && ["failed", "cancelled", "timed_out"].includes(row.status)) ? "Eligible Retry failed or duplicated native work" : undefined }); + accept: v => v.task.status === "done" && v.runs.length === 3 && v.runs.filter(row => row.id !== state.workerRunId).every(row => row.status === "succeeded"), + reject: v => v.runs.length > 3 || v.runs.some(row => row.id !== state.workerRunId && ["failed", "cancelled", "timed_out"].includes(row.status)) ? "The retry failed or duplicated native work" : undefined }); const retried = completed.runs.filter(row => row.agentId === worker.id && row.id !== state.workerRunId); state.retryRunCount = retried.length; - const full = retried[0] && await ctx.input.api.get(`/api/heartbeat-runs/${retried[0].id}`); state.retryRunId = full?.id; + const full = retried[0] && await ctx.input.api.get(`/api/heartbeat-runs/${retried[0].id}`); state.retryRunId = full?.id; state.retryBoundToOriginal = !!full && retriesOriginalRun(full, String(state.workerRunId)) && [full.contextSnapshot?.issueId, full.contextSnapshot?.taskId].includes(task.id); - state.retryAdmissionAfterGateClear = !!full && Date.parse(String(full.createdAt)) >= Number(state.gatesClearedAt); + state.retryAdmissionAfterFix = !!full && Date.parse(String(full.createdAt)) >= Number(state.modelFixedAt); + state.retryUsedFixedModel = full?.runnerProfileJson?.nativeExecutionInput?.provider?.model === state.workingModel; await ctx.input.page.goto(source, { waitUntil: "domcontentloaded" }); - const finalQuery = `I restored Morgan's budget and used the supported Retry for Website ${task.identifier ?? task.id}. What is the actual result now, and is the earlier failed attempt still retained?`; - await sendChatMessage(ctx.input.page, finalQuery, chatCatalogNativePhaseRemainingBudget(ctx)); - const final = await pollUntil({ label: "Own final source reply reports actual eligible Retry result", deadlineAt: chatCatalogNativePhaseDeadline(ctx, Date.now() + 150_000), - load: async () => { await ctx.refreshIssue(); return { source: ctx.issue(), runs: await ctx.allRuns() }; }, accept: v => chatCatalogNativePhaseIsTimely(ctx) && (v.source?.conversationState === "waiting" && v.runs.length === 5 && v.runs.filter(row => row.id !== state.workerRunId).every(row => row.status === "succeeded")) }); - const finalRun = final.runs.find(row => !completed.runs.some(prior => prior.id === row.id))!; - const finalReply = await phaseReply(finalQuery, finalRun.id); state.finalReply = finalReply.body; state.finalReplyAttributed = finalReply.attributed; - state.finalRecoveryRead = await controllerReadRecovery(ctx, finalRun.id, task.id, String(state.workerRunId), task.identifier); }, async observe(ctx, state) { const o = await observeRuntime(ctx, state); const before = state.failure as HeartbeatRun, retained = await ctx.input.api.get(`/api/heartbeat-runs/${state.workerRunId}`); diff --git a/tests/runner-e2e/chat-full-catalog-testops-manifest.json b/tests/runner-e2e/chat-full-catalog-testops-manifest.json index 18e7be05787..0188b251bf8 100644 --- a/tests/runner-e2e/chat-full-catalog-testops-manifest.json +++ b/tests/runner-e2e/chat-full-catalog-testops-manifest.json @@ -1205,7 +1205,13 @@ "key": "recovery.retry-exhausted", "field": "originalContext", "previous": "Automatic infrastructure attempts are exhausted and the source task exposes ordinary Retry to its authorized human.", - "current": "The website run stopped for a reason only a person can resolve (its selected model was rejected), and the person has since restored the configuration. The source task exposes ordinary Retry to its authorized human; an infrastructure fault would instead continue automatically." + "current": "The website run stopped for a reason only a person can resolve: the provider rejected its selected model, and that model is still selected. The source task exposes ordinary Retry to its authorized human; an infrastructure fault would instead continue automatically." + }, + { + "key": "recovery.retry-exhausted", + "field": "expectedOutcome", + "previous": "The response directs or presents the supported human Retry action and explains the current state. One accepted retry rechecks execution gates and preserves earlier attempt evidence. The person is not asked to certify internal infrastructure facts, and failure remains visible if the new attempt cannot start.", + "current": "The response directs or presents the supported human Retry action and explains the current state. Retry rechecks the selected model before it starts anything: while the rejected model is still selected, no new attempt starts, the failure stays visible, and the person is told which setting to change, with a link to it. After the person selects a working model, one accepted retry of the original run starts with that model, preserves the earlier attempt evidence and completes the task. The person is not asked to certify internal infrastructure facts." }, { "key": "recovery.source-action", @@ -10728,8 +10734,8 @@ "externalInput": null, "installedSource": null, "originalQuery": "Try the website task again.", - "expectedOutcome": "The response directs or presents the supported human Retry action and explains the current state. One accepted retry rechecks execution gates and preserves earlier attempt evidence. The person is not asked to certify internal infrastructure facts, and failure remains visible if the new attempt cannot start.", - "originalContext": "The website run stopped for a reason only a person can resolve (its selected model was rejected), and the person has since restored the configuration. The source task exposes ordinary Retry to its authorized human; an infrastructure fault would instead continue automatically.", + "expectedOutcome": "The response directs or presents the supported human Retry action and explains the current state. Retry rechecks the selected model before it starts anything: while the rejected model is still selected, no new attempt starts, the failure stays visible, and the person is told which setting to change, with a link to it. After the person selects a working model, one accepted retry of the original run starts with that model, preserves the earlier attempt evidence and completes the task. The person is not asked to certify internal infrastructure facts.", + "originalContext": "The website run stopped for a reason only a person can resolve: the provider rejected its selected model, and that model is still selected. The source task exposes ordinary Retry to its authorized human; an infrastructure fault would instead continue automatically.", "capabilities": [ "recovery/exhaustion" ], diff --git a/tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs b/tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs index b0ae99f389b..3f96d52131a 100644 --- a/tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs +++ b/tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs @@ -463,6 +463,10 @@ test('repair source metadata binds the corrected candidate and explicit changed "recovery.retry-exhausted", "originalContext" ], + [ + "recovery.retry-exhausted", + "expectedOutcome" + ], [ "recovery.source-action", "originalContext" diff --git a/tests/runner-e2e/chat-scenarios/recovery.yaml b/tests/runner-e2e/chat-scenarios/recovery.yaml index 44bed14bfc3..a84180042b9 100644 --- a/tests/runner-e2e/chat-scenarios/recovery.yaml +++ b/tests/runner-e2e/chat-scenarios/recovery.yaml @@ -52,17 +52,20 @@ scenarios: capabilities: - recovery/exhaustion context: >- - The website run stopped for a reason only a person can resolve (its selected model was - rejected), and the person has since restored the configuration. The source task exposes - ordinary Retry to its authorized human; an infrastructure fault would instead continue - automatically. + The website run stopped for a reason only a person can resolve: the provider + rejected its selected model, and that model is still selected. The source task + exposes ordinary Retry to its authorized human; an infrastructure fault would + instead continue automatically. query: >- Try the website task again. expected_outcome: >- - The response directs or presents the supported human Retry action and explains the current - state. One accepted retry rechecks execution gates and preserves earlier attempt evidence. - The person is not asked to certify internal infrastructure facts, and failure remains - visible if the new attempt cannot start. + The response directs or presents the supported human Retry action and explains the + current state. Retry rechecks the selected model before it starts anything: while + the rejected model is still selected, no new attempt starts, the failure stays + visible, and the person is told which setting to change, with a link to it. After + the person selects a working model, one accepted retry of the original run starts + with that model, preserves the earlier attempt evidence and completes the task. The + person is not asked to certify internal infrastructure facts. - id: recovery.workspace-repair capabilities: - recovery/workspace diff --git a/ui/src/components/TaskChatThread.test.tsx b/ui/src/components/TaskChatThread.test.tsx index e7b6728c56f..747f2d0ecac 100644 --- a/ui/src/components/TaskChatThread.test.tsx +++ b/ui/src/components/TaskChatThread.test.tsx @@ -2673,6 +2673,33 @@ describe("TaskChatThread runtime transcript selection", () => { expect(container.textContent).not.toContain("after returning a final response"); }); + it("offers Try again on a model-unavailable failure and retries that run", async () => { + const onRetryFailedRun = vi.fn(); + renderThread({ + comments: [], + onAdd: async () => {}, + issueStatus: "blocked", + onRetryFailedRun, + interruption: { intentId: null, responsibility: "user", state: "waiting_for_user", nextRetryAt: null, + canRetry: false, userAction: "choose_model", rejectedModel: "openrouter/missing", sourceRunIds: ["model-rejected"], observedRunId: "model-rejected" }, + linkedRuns: [{ + runId: "model-rejected", runtimeMode: "native", status: "failed", + errorCode: "native_provider_model_rejected", + error: "The selected model is not available.", + agentId: "agent-1", agentName: "Runner", adapterType: "paperclip_runner", + createdAt: "2026-08-25T18:00:00.000Z", startedAt: "2026-08-25T18:00:00.000Z", + finishedAt: "2026-08-25T18:00:02.000Z", + }], + }); + const marker = container.querySelector('[data-testid="task-chat-collapsible-marker"]'); + expect(marker?.textContent).toContain("Model unavailable"); + const retry = marker?.querySelector('[data-testid="task-chat-run-failed-try-again"]'); + expect(retry).not.toBeNull(); + flushSync(() => retry!.click()); + await Promise.resolve(); + expect(onRetryFailedRun).toHaveBeenCalledWith("model-rejected"); + }); + it("names the rejected model id that the task interruption carries", () => { renderThread({ comments: [], diff --git a/ui/src/components/task-chat/TaskChatThreadView.tsx b/ui/src/components/task-chat/TaskChatThreadView.tsx index fc93980dcf8..a33b045b397 100644 --- a/ui/src/components/task-chat/TaskChatThreadView.tsx +++ b/ui/src/components/task-chat/TaskChatThreadView.tsx @@ -29,7 +29,7 @@ import { TaskMessageScroller, type ScrollingHistory } from "./TaskMessageScrolle import { TaskChatProtocolCard } from "./TaskChatProtocolCard"; import { TaskChatProtocolActivityRow } from "./TaskChatProtocolActivityRow"; import { TaskChatPlanPreviewCard } from "./TaskChatPlanPreviewCard"; -import { compactAdjacentErrors } from "./adjacent-errors"; +import { compactAdjacentErrors, isRetryableFailureMarker } from "./adjacent-errors"; import { CoordinatorChatTaskHandoff } from "@/components/coordinator/CoordinatorChatTaskHandoffs"; import { CoordinatorChatDecisionReceipt } from "@/components/coordinator/CoordinatorChatDecisionReceipts"; import { CoordinatorChatPendingQuestion } from "@/components/coordinator/CoordinatorChatPendingQuestions"; @@ -347,8 +347,7 @@ export function TaskChatThreadView({ .find( (item) => { const marker = item.kind === "marker" ? item : retryMarkers.get(item.id); - return marker?.variant === "interrupted" && - (marker.label === "Run failed" || marker.label === "Usage limit reached"); + return !!marker && isRetryableFailureMarker(marker); }, )?.id : undefined; diff --git a/ui/src/components/task-chat/adjacent-errors.ts b/ui/src/components/task-chat/adjacent-errors.ts index e8ae81b86c9..3cf520086e1 100644 --- a/ui/src/components/task-chat/adjacent-errors.ts +++ b/ui/src/components/task-chat/adjacent-errors.ts @@ -54,8 +54,7 @@ export function compactAdjacentErrors(items: readonly TaskChatItem[]): { const latest = items[end - 1]; if (mixed && errorKind(latest) === "recovery") { const marker = items.slice(start, end).reverse().find( - (item): item is TaskChatMarkerItem => item.kind === "marker" - && (item.label === "Run failed" || item.label === "Usage limit reached"), + (item): item is TaskChatMarkerItem => item.kind === "marker" && isRetryableFailureMarker(item), ); // Preserve the retry target separately from the surviving error's text. // The renderer still applies the parent's permission and pending state. @@ -66,3 +65,9 @@ export function compactAdjacentErrors(items: readonly TaskChatItem[]): { } return { items: result, retryMarkers }; } + +const RETRYABLE_FAILURE_LABELS = new Set(["Run failed", "Usage limit reached", "Model unavailable"]); + +export function isRetryableFailureMarker(marker: Pick) { + return marker.variant === "interrupted" && RETRYABLE_FAILURE_LABELS.has(marker.label); +} diff --git a/ui/src/lib/model-retry-refusal.test.ts b/ui/src/lib/model-retry-refusal.test.ts new file mode 100644 index 00000000000..d7f1181959a --- /dev/null +++ b/ui/src/lib/model-retry-refusal.test.ts @@ -0,0 +1,30 @@ +import { describe, expect, it } from "vitest"; +import { ApiError } from "../api/api-error"; +import { modelStillUnavailableToast } from "./model-retry-refusal"; + +const agent = { id: "agent-1", name: "Morgan", urlKey: "morgan" }; +const refusal = (setting: string) => new ApiError('The provider rejected the model "bad/model", and it is still selected.', 409, { + error: 'The provider rejected the model "bad/model", and it is still selected.', + details: { code: "native_provider_model_still_unavailable", setting, rejectedModel: "bad/model", agentId: "agent-1" }, +}); + +describe("modelStillUnavailableToast", () => { + it("names the setting and links to the agent's model settings", () => { + expect(modelStillUnavailableToast(refusal("agent_model"), agent)).toMatchObject({ + title: "Model still unavailable", tone: "error", + body: expect.stringContaining("still selected"), + action: { label: "Open Morgan model settings", href: "/agents/morgan/runtime" }, + }); + }); + + it("explains a task override without linking away from the task", () => { + const toast = modelStillUnavailableToast(refusal("task_model_override"), agent); + expect(toast?.title).toBe("Model still unavailable"); + expect(toast?.action).toBeUndefined(); + }); + + it("leaves other retry failures to the generic message", () => { + expect(modelStillUnavailableToast(new ApiError("Conflict", 409, { error: "Conflict" }), agent)).toBeNull(); + expect(modelStillUnavailableToast(new Error("network"), agent)).toBeNull(); + }); +}); diff --git a/ui/src/lib/model-retry-refusal.ts b/ui/src/lib/model-retry-refusal.ts new file mode 100644 index 00000000000..66dbc94cdc2 --- /dev/null +++ b/ui/src/lib/model-retry-refusal.ts @@ -0,0 +1,28 @@ +import type { ToastInput } from "../context/ToastContext"; +import { agentUrl } from "./utils"; + +export const MODEL_STILL_UNAVAILABLE_CODE = "native_provider_model_still_unavailable"; + +function record(value: unknown): Record { + return value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; +} + +export function modelStillUnavailableToast( + error: unknown, + agent: { id: string; name?: string | null; urlKey?: string | null } | undefined, +): ToastInput | null { + const details = record(record(record(error).body).details); + if (details.code !== MODEL_STILL_UNAVAILABLE_CODE) return null; + const body = error instanceof Error && error.message ? error.message : "The selected model is still unavailable."; + const agentSetting = details.setting !== "task_model_override"; + return { + title: "Model still unavailable", + body, + tone: "error", + ttlMs: 15_000, + dedupeKey: `model-still-unavailable:${String(details.agentId ?? agent?.id ?? "")}`, + ...(agentSetting && agent + ? { action: { label: `Open ${agent.name ?? "agent"} model settings`, href: `${agentUrl(agent)}/runtime` } } + : {}), + }; +} diff --git a/ui/src/pages/IssueDetail.tsx b/ui/src/pages/IssueDetail.tsx index 52734f13868..fd6384fcfcf 100644 --- a/ui/src/pages/IssueDetail.tsx +++ b/ui/src/pages/IssueDetail.tsx @@ -208,6 +208,7 @@ import { } from "../lib/optimistic-issue-runs"; import { useProjectOrder } from "../hooks/useProjectOrder"; import { recordRecentTask } from "../lib/recent-tasks"; +import { modelStillUnavailableToast } from "../lib/model-retry-refusal"; import { relativeTime, cn, @@ -1687,8 +1688,9 @@ const IssueDetailChatTab = memo(function IssueDetailChatTab({ queryKey: queryKeys.issues.activeRun(issueQueryRef), }); }, - onError: (error) => { - pushToast({ + onError: (error, runId) => { + const failedRun = resolvedLinkedRuns.find((run) => run.runId === runId); + pushToast(modelStillUnavailableToast(error, failedRun ? agentMap.get(failedRun.agentId) : undefined) ?? { title: "Run retry failed", body: error instanceof Error ? error.message : "Unable to retry run", tone: "error",