Files
PaperClipAI/server/src/services/native-runtime/codex-model-fallback.test.ts
T
eaac8ed0f7 feat(adapters): add Claude Fable 5.2 and GPT-6.2 with CLI floors, and refresh sandbox CLI pins (Nov 18 audit) (#22222)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents run on coding harnesses. Each adapter has a model catalog,
and sandboxes contain pinned CLIs.
> - Anthropic released Claude Fable 5.2 and OpenAI released GPT-6.2 Sol
and GPT-6.2 Astra for Codex. Both need newer CLIs than the Paperclip
Runner provider pack ships.
> - A stale catalog hides valid models. A model that needs a newer CLI
fails with an opaque backend error.
> - This pull request adds the verified model IDs with their CLI floors,
names the version gap in Paperclip Runner, and refreshes the sandbox CLI
pins.
> - The benefit is that operators can select current models and get a
named version error instead of a backend failure.

## Linked Issues or Issue Description

Refs #20774 and #21636 (both add parts of Fable 5.2 support; see
"Overlap" below).
Refs #18486 (adds GPT-6.1 Terra; this PR does not duplicate it).
Refs #16708 (adds Claude Haiku 5.5 to `claude_local`; this PR does not
duplicate it).

**What happened?**

Claude Fable 5.2 (`claude-fable-5-2`) and GPT-6.2 Sol / Astra
(`gpt-6.2-sol`, `gpt-6.2-astra`) are generally available. The
`claude_local`, `codex_local`, and OpenCode pickers do not list them.
Claude Code 2.1.321 is the first release with Fable 5.2. Codex 0.169.1
is the first release whose bundled catalog lists GPT-6.2. The Paperclip
Runner provider pack bundles Claude Code 2.1.286 and Codex 0.160.0.

**Expected behavior**

The pickers list the new models. A run on a CLI below the model minimum
stops with the detected and required versions, or falls back to a
compatible model with a visible message.

**Proposed change**

Add the models and their per-model CLI floors. Gate ACPX Claude in
Paperclip Runner on the Claude Code version that its Agent SDK bundles.
Refresh the sandbox CLI pins with verified digests. Record the audit.

## What Changed

- Claude Code (`claude_local`):
- Add `claude-fable-5-2` (label "Claude Fable 5.2") at the top of the
picker. `claude-fable-5-1` stays selectable after the current-generation
models.
- Add Bedrock `global.anthropic.claude-fable-5-2`. Anthropic documents
only this global ID for Bedrock.
  - Fable 5.2 offers all five efforts (`low` to `max`).
- `minimumClaudeCliVersionForModel("claude-fable-5-2")` returns
`2.1.321`. The existing CLI-lane Test and launch checks use this floor.
On a managed sandbox, the CLI lane installs a private Claude Code at
this version, as it does for the other floors.
- Paperclip Runner (ACPX Codex): `resolvePaperclipRunnerProviderProfile`
rejects a model whose Codex floor is above the qualified ACPX Codex
runtime (`0.160.0`). The error code is
`paperclip_runner_codex_cli_version_incompatible`, and the message gives
both versions.
- Paperclip Runner (ACPX Claude):
`resolvePaperclipRunnerProviderProfile` rejects a model whose Claude
Code floor is above the version that the qualified Agent SDK bundles.
The error code is `paperclip_runner_claude_cli_version_incompatible`.
The message gives both versions. A new helper,
`claudeCodeVersionForAgentSdk`, maps SDK `0.3.N` to Claude Code `2.1.N`
(the SDK's `claudeCodeVersion` field) and returns `null` for any other
line.
- Codex (`codex_local`):
- Add `gpt-6.2-sol` and `gpt-6.2-astra` at the top of the picker, in the
bundled catalog order.
  - Both offer Fast mode and the `low` to `ultra` efforts.
- Both have a CLI floor of `0.169.1`. On the native Codex provider with
the pinned Codex 0.160.0, the existing fallback moves `gpt-6.2-sol` to
`gpt-6.1-sol` and `gpt-6.2-astra` to `gpt-6-astra` with a visible
message.
  - The default stays `gpt-6.1-sol`.
- OpenCode fallback catalog: add `openai/gpt-6.2-sol`,
`openai/gpt-6.2-astra`, `anthropic/claude-fable-5-2`,
`google/gemini-4-carbon`, and `google/gemini-4-hydrogen`.
- Evaluation image sandbox layer (`docker/daytona-runner`):

  | CLI | Before | After |
  | --- | --- | --- |
  | Grok CLI | 1.0.70 | 1.0.89 |
  | Gemini CLI | 0.65.0 | 0.68.0 |
  | Cursor CLI | 2026.10.22-27dcf5f | 2026.11.17-f532c45 |
  | GitHub CLI | 2.103.3 | 2.105.0 |

- Add `doc/adapter-model-audit-2026-11-18.md` with sources,
compatibility findings, and deferred items. Add a note on above-window
Codex floors to `doc/architecture/paperclip-runner-compatibility.md`.

No saved agent changes model, effort, or spending settings. No picker
entry is removed.

### Managed sandboxes

On Paperclip Runner, Fable 5.2 and GPT-6.2 stay unusable until the
provider pack moves to Claude Code 2.1.321 or later and Codex 0.169.1 or
later. A sandbox image that carries that provider pack must also be
promoted. This PR does not change the provider pack (see "Deferred
items" in the audit). Native remote Codex accepts only `>=0.149.0
<0.161.0`, so GPT-6.2 also needs that window to move.

### Overlap

- #21636 sets the Fable 5.2 floor to `2.1.290`. The [Claude Code
changelog](https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md)
adds Fable 5.2 in 2.1.321. `git merge-tree` merges the two without a
text conflict, but #21636's line becomes unreachable after this PR's
line, and its test then fails. Rule: keep `2.1.321`. Merge this PR
first. Then close #21636 or change its floor and test to `2.1.321`.
- #20774 (draft) lists `claude-fable-5-2`. It conflicts in
`packages/adapters/claude-local/src/index.ts` and
`server/src/__tests__/adapter-models.test.ts`. Rule: keep this PR's
catalog order and Bedrock ID.
- #18486 adds `gpt-6.1-terra`. It conflicts in
`packages/adapters/codex-local/src/index.test.ts` and
`src/server/codex-args.test.ts` (the ordered ID list and the Fast-mode
message). Rule: keep both sets of models in the bundled catalog order
(`gpt-6.2-sol`, `gpt-6.2-astra`, `gpt-6-astra`, `gpt-6.1-sol`,
`gpt-6.1-terra`, ...). Either PR can merge first; the second rebases.
- #16708 adds Claude Haiku 5.5. It conflicts in
`packages/adapters/claude-local/src/index.test.ts` (adjacent test cases)
and `doc/DEVELOPING.md`. Rule: keep both test blocks.

## Verification

Research date: November 18, 2026. Sources are listed in the audit doc.

- `vitest run src/index.test.ts src/server/model-order.test.ts
src/server/cli-capabilities.test.ts src/server/test.remote.test.ts` in
`packages/adapters/claude-local`: 40 passed. `execute.remote.test.ts`
(29), `test.probe.test.ts` (21), `sandbox-cli-version.test.ts` (15),
`acp.auth.test.ts` (25), and `acp.test.ts` (47, with
`--testTimeout=60000`) also passed when run one file at a time.
- `vitest run src/index.test.ts src/server/codex-args.test.ts
src/ui/build-config.test.ts src/server/test.remote.test.ts` in
`packages/adapters/codex-local`: all passed (155). `acp.test.ts` (49),
`acp.cli-version.test.ts` (4), and `execute.test.ts` (10) also passed
one file at a time.
- `vitest run` in `packages/adapters/opencode-local`: 94 passed.
- `vitest run src/services/native-runtime/acpx-qualification.test.ts
src/services/native-runtime/codex-model-fallback.test.ts
src/__tests__/adapter-models.test.ts
src/__tests__/heartbeat-model-fallback-warning.test.ts` in `server`: all
passed. The new cases cover the ACPX Claude and ACPX Codex version
errors and the native GPT-6.2 fallback on Codex 0.160.0.
- `vitest run src/lib/agent-setup-fields.test.ts
src/lib/codex-reasoning-effort.test.ts` in `ui`: 11 passed.
- `tsc --noEmit` in the three adapters and `pnpm typecheck` in `server`:
passed.
- Cursor archive SHA-256
`189ed41d8963cd22dcdafbbd62f072cc55ac08374c922dbd04aee525f4018ab6` is
computed from the versioned Linux x64 tarball. The extracted binary
reports `2026.11.17-f532c45`, the version that
`https://cursor.com/install` resolves.
- GitHub CLI SHA-256
`79e7ab5db738c48777b696e374f8b428f0647debfb424f44d2f426370407fe2f`
matches `gh_2.105.0_checksums.txt`.
- Grok CLI 1.0.89 and Gemini CLI 0.68.0 installed from npm report those
versions. The flags that Paperclip uses are in `--help` (`-m`,
`--effort`, `grok models`; `-m`, `-p`, `--output-format`,
`--approval-mode`, `--yolo`).
- Host baseline: tests that create an AI provider need the host's empty
`ANTHROPIC_BASE_URL` unset. Under full-package parallel runs, SSH and
sandbox fixture tests time out at 5 seconds on this host; each file
passes alone. The audit doc records both.
- Not run locally: the full `pnpm -r typecheck`, `pnpm build`, browser
suites, and the image build. CI runs these. No live model inference ran.

## Risks

- Low risk for saved agents: no default, saved model, effort, or
spending setting changes.
- The new Paperclip Runner checks reject `claude-fable-5-2` on ACPX
Claude and `gpt-6.2-*` on ACPX Codex (and any later model whose floor is
above the bundled runtime) before launch. Without this PR, the turn
would start and fail at the backend.
- The `0.3.N` → `2.1.N` mapping covers only the current SDK line. Any
other line returns `null`, and the check is skipped.
- The sandbox CLI pins change only the evaluation image. The fleet image
change is separate.
- Rollback: revert this PR. No data migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, web fetch, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:27:00 -08:00

96 lines
4.9 KiB
TypeScript

import { describe, expect, it, vi } from "vitest";
import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
import {
codexModelFallbackCandidates,
compatibleCodexModel,
readRemoteCodexModelCliVersion,
selectCodexModelFallback,
} from "./codex-model-fallback.js";
describe("bounded Codex model fallback", () => {
it("keeps capability classes separate and appends the stable default once", () => {
expect(codexModelFallbackCandidates("gpt-6.1-sol")).toEqual(["gpt-6-sol", "gpt-5.6-sol"]);
expect(codexModelFallbackCandidates("gpt-6-luna")).toEqual(["gpt-5.6-luna", "gpt-5.6-sol"]);
expect(codexModelFallbackCandidates("gpt-6-astra")).toEqual(["gpt-5.6-sol"]);
expect(codexModelFallbackCandidates("gpt-5.6-sol")).toEqual([]);
});
it("uses the default after same-class candidates are unavailable, then stops", () => {
const compatible = vi.fn((model: string) => model === "gpt-5.6-sol");
expect(selectCodexModelFallback("gpt-6-luna", compatible)).toBe("gpt-5.6-sol");
expect(compatible.mock.calls).toEqual([["gpt-5.6-luna"], ["gpt-5.6-sol"]]);
const unavailable = vi.fn(() => false);
expect(selectCodexModelFallback("gpt-6.1-sol", unavailable)).toBeNull();
expect(unavailable.mock.calls).toEqual([["gpt-6-sol"], ["gpt-5.6-sol"]]);
});
it("moves GPT-6.2 to the newest same-class model the pinned 0.160.0 CLI accepts", () => {
expect(codexModelFallbackCandidates("gpt-6.2-sol")).toEqual(["gpt-6.1-sol", "gpt-6-sol", "gpt-5.6-sol"]);
expect(codexModelFallbackCandidates("gpt-6.2-astra")).toEqual(["gpt-6-astra", "gpt-5.6-sol"]);
expect(compatibleCodexModel("gpt-6.2-sol", "0.160.0")).toBe("gpt-6.1-sol");
expect(compatibleCodexModel("gpt-6.2-astra", "0.160.0")).toBe("gpt-6-astra");
});
it.each([
["gpt-6.1-sol", "0.159.0"], ["gpt-6-sol", "0.157.0"], ["gpt-5.6-sol", "0.156.0"],
["team-model", "0.156.0"], [null, "0.156.0"], ["gpt-6.1-sol", null],
["gpt-6.1-sol", "0.148.9"], ["gpt-6.1-sol", "0.161.0"], ["gpt-6.1-sol", "0.160.0-alpha.1"],
])("preserves %s on %s when no safe startup substitution applies", (model, version) => {
expect(compatibleCodexModel(model, version)).toBe(model);
});
});
function remoteTarget(stdout = "codex-cli 0.156.0", overrides = {}) {
const execute = vi.fn(async (command: { command: string; args?: string[] }) => ({
exitCode: 0, signal: null, timedOut: false, stderr: "",
stdout: command.command === "sh" ? "/opt/paperclip-runner/bin/codex\n" : stdout,
...(command.command !== "sh" ? overrides : {}),
}));
const target = { kind: "remote", transport: "sandbox", remoteCwd: "/workspace",
runner: { execute } } as unknown as AdapterExecutionTarget;
return { target, execute };
}
describe("remote model compatibility preflight", () => {
it.each(["discovery", "version"])("defers to launch verification when the %s probe rejects", async (phase) => {
const { target, execute } = remoteTarget();
if (phase === "version") {
execute.mockResolvedValueOnce({ exitCode: 0, signal: null, timedOut: false,
stderr: "", stdout: "/opt/paperclip-runner/bin/codex\n" });
}
execute.mockRejectedValueOnce(new Error("temporary sandbox command failure"));
await expect(readRemoteCodexModelCliVersion({ model: "gpt-6.1-sol", target })).resolves.toBeNull();
expect(execute).toHaveBeenCalledTimes(phase === "version" ? 2 : 1);
});
it("checks the image CLI without invoking a provider turn or installing software", async () => {
const { target, execute } = remoteTarget();
const version = await readRemoteCodexModelCliVersion({ model: "gpt-6.1-sol", target });
expect(version).toBe("0.156.0");
expect(compatibleCodexModel("gpt-6.1-sol", version)).toBe("gpt-5.6-sol");
expect(execute).toHaveBeenCalledTimes(2);
expect(execute).toHaveBeenLastCalledWith({ command: "/opt/paperclip-runner/bin/codex",
args: ["--version"], cwd: "/workspace", bypassSession: true, timeoutMs: 30_000 });
});
it.each([
{ remoteCodexPath: "/controller/codex" },
{ remoteCodexNpmSpec: "@openai/codex@0.160.0" },
{ model: "gpt-5.6-sol" },
{ model: "custom-model" },
{ target: { kind: "local" } as AdapterExecutionTarget },
])("preserves explicit artifacts, install pins, and unqualified models: %j", async (overrides) => {
const { target, execute } = remoteTarget();
expect(await readRemoteCodexModelCliVersion({ model: "gpt-6.1-sol", target, ...overrides })).toBeNull();
expect(execute).not.toHaveBeenCalled();
});
it.each([
["codex-cli 0.148.9", {}], ["codex-cli 0.161.0", {}], ["codex-cli 0.160.0-alpha.1", {}],
["codex-cli 0.156.0\ncodex-cli 0.160.0", {}], ["codex-cli 0.156.0", { timedOut: true }],
["codex-cli 0.156.0", { exitCode: 1 }],
])("keeps the launch verifier authoritative for invalid probes: %s %j", async (stdout, overrides) => {
expect(await readRemoteCodexModelCliVersion({ model: "gpt-6.1-sol", target: remoteTarget(stdout, overrides).target })).toBeNull();
});
});