mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 12:37:55 +01:00
fix(native): record OpenCode's rejected model as a model rejection (#19865)
## Thinking Path - Paperclip runs agents through native providers; when a provider rejects the selected model, the person needs to see "Model unavailable" and choose another model. - #19548 made the OpenCode driver report OpenRouter's invalid model ID as a closed `model_unavailable` provider failure. The task banner now shows it. - The server's terminal classifier, however, recognized a model rejection only from Codex's exact ChatGPT error envelope. OpenCode's structured failure was stored for the banner but the run was recorded as `adapter_failed`. - So chat showed a generic failure, and the task was not blocked for a model choice. The `runs-adapters.inspect-failed-run` chat eval caught this. - This change maps an action-required `model_unavailable` failure to `native_provider_model_rejected` for any provider. ## Linked Issues or Issue Description Refs #19548. An OpenCode run whose model OpenRouter rejects (`... is not a valid model ID`, HTTP 400) finishes with `errorCode: adapter_failed`. Expected: `native_provider_model_rejected`, as for Codex. ## What Changed - `native-provider-failure.ts`: `forTerminal` returns `native_provider_model_rejected` when the failed turn's provider failure has category `model_unavailable` and retryability `requires_action`. Transient or unknown retryability still returns null. The diagnostic is the closed turn diagnostic; no provider text or model name is copied. - `NATIVE_MODEL_REJECTION_MESSAGE` no longer names a ChatGPT connection, since it is now the blocker text for any provider. - Tests for both providers, plus negative cases for transient/unknown retryability. - `wake-queue/adapters/postgres.ts`: a native run that failed for a rejected model or incomplete configuration, on a task the arbiter already blocked, is released without opening a `native_continuation_requires_reconciliation` incident. That incident put the run in `recovery_needed`, which hid the run page's Retry, though the provider refused the request and nothing ran to reconcile. Queued messages stay deferred. ## Verification - `pnpm exec vitest run src/services/native-runtime/native-provider-failure.test.ts src/services/native-runtime/status-arbiter.test.ts src/services/native-runtime/native-provider-failure.integration.test.ts` (server): 98 passed. - `pnpm exec vitest run src/modules/wake-queue` (server): passes, including a new test that fails on the old code (it opened the incident). - Server `tsc --noEmit`: clean. - End to end: the `runs-adapters.inspect-failed-run` chat eval will be rerun on the chat-eval candidate with this change. ## Risks - Codex model-rejection blocker wording changes from "current ChatGPT connection" to "current AI connection". - Any provider driver that reports `model_unavailable` with `requires_action` now blocks the task for a model choice instead of the generic failure path, which disables automatic adapter-failure retries for that run. That is the intended behavior for a rejected model. - Configuration-failure native runs on a blocked task no longer get a board reconciliation incident. If such a run had already produced side effects in earlier turns, they are not flagged for reconciliation; a provider model rejection happens before the turn does work. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use (shell, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
ba3fe09954
commit
2d89486c57
4 files changed
+58
-3
No files matched your search
@@ -780,6 +780,18 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => {
|
||||
expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0].statusVersion).toBe(blockedIssue.statusVersion);
|
||||
});
|
||||
|
||||
it("does not open a reconciliation incident for a native run whose model the provider rejected", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const agentId = await seedAgent({ companyId });
|
||||
const issueId = await seedIssue({ companyId, assigneeAgentId: agentId, status: "blocked" });
|
||||
const runId = await seedRun({ companyId, agentId, status: "failed", contextSnapshot: { issueId } });
|
||||
await db.update(heartbeatRuns).set({ runtimeMode: "native", errorCode: "native_provider_model_rejected" }).where(eq(heartbeatRuns.id, runId));
|
||||
const adapter = createPostgresWakeQueueAdapter(db, stubDeps);
|
||||
await adapter.withIssueExecutionLock({ companyId, runId, now: new Date() }, async () => { throw new Error("must not restart a rejected model"); });
|
||||
expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0].status).toBe("blocked");
|
||||
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(0);
|
||||
});
|
||||
|
||||
it.each(["active", "escalated"])("repairs a failed native task with an existing %s recovery action", async (status) => {
|
||||
const companyId = await seedCompany();
|
||||
const agentId = await seedAgent({ companyId });
|
||||
|
||||
@@ -1369,6 +1369,13 @@ export function createPostgresWakeQueueAdapter(db: Db, deps: WakeQueuePostgresAd
|
||||
};
|
||||
}
|
||||
|
||||
// A native run blocked for a rejected model or incomplete configuration waits for that
|
||||
// change and a retry. The provider refused the request, so there is no execution to
|
||||
// reconcile; a reconciliation incident would also hide the run's Retry.
|
||||
if (!handoffAssigneeAgentId && run.runtimeMode === "native" && isConfigurationIncompleteFailedRun(run)
|
||||
&& issueRow.status === "blocked" && issueRow.assigneeAgentId === run.agentId) {
|
||||
return { outcome: { kind: "released" }, postCommitEffects: [], run: runSnapshot };
|
||||
}
|
||||
if (!handoffAssigneeAgentId && await recordNativeTerminalRecoveryIfNeeded(tx, run, issueRow, input.now)) {
|
||||
return { outcome: { kind: "released" }, postCommitEffects: [], run: runSnapshot };
|
||||
}
|
||||
|
||||
@@ -32,6 +32,26 @@ describe("native provider failure observation", () => {
|
||||
expect(observed.providerFailureForTerminal("other-turn", terminal)).toBeNull();
|
||||
expect(observed.providerFailureForTerminal("turn", { ...terminal, runTerminalState: "succeeded" })).toBeNull();
|
||||
});
|
||||
it.each(["opencode", "codex"])("classifies an action-required model_unavailable failure from %s as a model rejection", kind => {
|
||||
const observed = createNativeProviderFailureObservation({ kind });
|
||||
const providerFailure = { schema: "paperclip.provider-failure.v1", provider: "openrouter", phase: "turn",
|
||||
category: "model_unavailable", message: "vendor/private-model is not a valid model ID", retryability: "requires_action", httpStatus: 400 };
|
||||
observed.observe({ ...event, payload: { status: "failed", error: { providerFailure } } });
|
||||
const result = observed.forTerminal("turn", terminal);
|
||||
expect(result).toMatchObject({ errorCode: "native_provider_model_rejected", errorMessage: NATIVE_MODEL_REJECTION_MESSAGE });
|
||||
expect(JSON.stringify(result)).not.toContain("private-model");
|
||||
// A closed diagnostic only, and the classified turn is not also reported as unclassified.
|
||||
expect(result?.diagnostic).toEqual(expect.objectContaining({ provider: kind }));
|
||||
expect(JSON.stringify(result?.diagnostic)).not.toContain("not a valid model");
|
||||
expect(observed.unclassifiedTurnFailure("turn", terminal)).toBeNull();
|
||||
expect(observed.forTerminal("other-turn", terminal)).toBeNull();
|
||||
expect(observed.forTerminal("turn", { ...terminal, runTerminalState: "succeeded" })).toBeNull();
|
||||
});
|
||||
it.each(["transient", "unknown"])("does not treat a %s model_unavailable failure as a model rejection", retryability => {
|
||||
const observed = createNativeProviderFailureObservation({ kind: "opencode" });
|
||||
observed.observe({ ...event, payload: { status: "failed", error: { providerFailure: { schema: "paperclip.provider-failure.v1", provider: "openrouter", phase: "turn", category: "model_unavailable", message: "busy", retryability } } } });
|
||||
expect(observed.forTerminal("turn", terminal)).toBeNull();
|
||||
});
|
||||
it("surfaces a structured capacity failure without copying arbitrary provider text", () => {
|
||||
const observed = createNativeProviderFailureObservation({ kind: "codex" });
|
||||
observed.observe({ ...event, payload: { status: "failed", error: { codexErrorInfo: "serverOverloaded", message: "Private arbitrary response" } } });
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
import type { PrpEvent, PrpTerminalState } from "../../vendor/paperclip-runner/index.js";
|
||||
import { readProviderFailure, type ProviderFailure } from "@paperclipai/shared";
|
||||
|
||||
// Shared by every provider whose selected model is rejected, so it names no specific connection.
|
||||
export const NATIVE_MODEL_REJECTION_MESSAGE =
|
||||
"The selected model is not supported by the current ChatGPT connection. Choose a supported model or a compatible AI connection.";
|
||||
"The selected model is not available through the current AI connection. Choose a supported model or a compatible AI connection.";
|
||||
|
||||
export const NATIVE_PROVIDER_OVERLOADED_MESSAGE =
|
||||
"Selected model is at capacity. Please try a different model.";
|
||||
@@ -215,6 +216,11 @@ export function createNativeProviderFailureObservation(provider: { kind: string;
|
||||
let overloadedTurnId: string | null = null;
|
||||
const failedTurns = new Map<string, NativeTurnFailureDiagnostic>();
|
||||
const providerFailures = new Map<string, ProviderFailure>();
|
||||
/** A driver-classified, action-required model rejection; `forTerminal` classifies it, so it is not "unclassified". */
|
||||
const modelUnavailableTurn = (turnId: string) => {
|
||||
const failure = providerFailures.get(turnId);
|
||||
return failure?.category === "model_unavailable" && failure.retryability === "requires_action";
|
||||
};
|
||||
return {
|
||||
observe(event: PrpEvent) {
|
||||
if (event.sourceKind === "runner" && event.eventType === "turn.failed" && event.turnId
|
||||
@@ -266,7 +272,17 @@ export function createNativeProviderFailureObservation(provider: { kind: string;
|
||||
errorMessage: NATIVE_PROVIDER_OVERLOADED_MESSAGE,
|
||||
diagnostic: { ...NATIVE_PROVIDER_OVERLOADED_DIAGNOSTIC },
|
||||
};
|
||||
if (turnId !== rejectedTurnId) return null;
|
||||
if (turnId !== rejectedTurnId) {
|
||||
// A driver's closed category marks a rejected model for any provider. Only an
|
||||
// action-required rejection qualifies; a transient one may succeed on retry.
|
||||
if (!modelUnavailableTurn(turnId)) return null;
|
||||
const failure = providerFailures.get(turnId)!;
|
||||
return {
|
||||
errorCode: "native_provider_model_rejected" as const,
|
||||
errorMessage: NATIVE_MODEL_REJECTION_MESSAGE,
|
||||
diagnostic: failedTurns.get(turnId) ?? { provider: failure.provider, category: failure.category },
|
||||
};
|
||||
}
|
||||
return {
|
||||
errorCode: "native_provider_model_rejected" as const,
|
||||
errorMessage: NATIVE_MODEL_REJECTION_MESSAGE,
|
||||
@@ -276,7 +292,7 @@ export function createNativeProviderFailureObservation(provider: { kind: string;
|
||||
/** Diagnostics only: the failed turn that `forTerminal` did not classify. */
|
||||
unclassifiedTurnFailure(turnId: string | null | undefined, terminal: PrpTerminalState) {
|
||||
if (!turnId || terminal.turnTerminalState !== "failed" || terminal.runTerminalState !== "failed" ||
|
||||
turnId === overloadedTurnId || turnId === rejectedTurnId) return null;
|
||||
turnId === overloadedTurnId || turnId === rejectedTurnId || modelUnavailableTurn(turnId)) return null;
|
||||
const diagnostic = failedTurns.get(turnId);
|
||||
return diagnostic ? { ...diagnostic } : null;
|
||||
},
|
||||
|
||||
Reference in new issue
Block a user