mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 14:19:00 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Codex agents run through Paperclip Runner or the legacy
`codex_local` harness, and the runner E2E suites run paid Codex cells
> - The shared Codex default and many selections preferred `gpt-5.6-sol`
> - The reviewed OpenAI rate card in
`server/src/services/codex-pricing.ts` (`openai-standard-2026-09-30`)
prices `gpt-6.1-sol` at $2 input / $0.10 cached / $10 output per 1M
tokens, and `gpt-5.6-sol` at $4 / $0.40 / $20
> - So every default run and every paid Codex eval cell cost twice as
much as necessary
> - This pull request moves every place that prefers or selects
`gpt-5.6-sol` to `gpt-6.1-sol`, and keeps historical facts, rate cards
and compatibility fallbacks on `gpt-5.6-sol`
> - The benefit is half the token price for new default Codex agents and
for the Codex E2E and TestOps cells
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The default Codex model for new agents and the Codex model that the
runner E2E suites and TestOps plans select.
**Subsystem affected**
Adapters (`adapter-utils`, `codex-local`), server native runtime
fallback, UI AI connection pools, Paperclip Runner devtools and live
evals, runner E2E catalog and TestOps mappings.
**Current behavior**
New Codex agents without an explicit model store `gpt-5.6-sol`. The
runner E2E `runner-codex` profile, the Reliability `queue-steer` plan
cell, the Product regression TestOps mapping and the ACPX Codex
qualification selection all use `gpt-5.6-sol`.
**Proposed behavior**
They use `gpt-6.1-sol`. Existing agents that store a model keep it.
`gpt-5.6-sol` stays selectable.
**Reason and benefit**
`gpt-6.1-sol` costs half as much per token on the in-repo reviewed rate
card. The current OpenAI pricing page (retrieved 2026-11-06) shows the
same short-context rates and also lists long-context rates of $4 / $0.20
/ $15 for `gpt-6.1-sol` against $8 / $0.80 / $30 for `gpt-5.6-sol`.
**Breaking changes**
None for stored configurations. See Risks.
## What Changed
Changed defaults and selections (all now `gpt-6.1-sol`):
- `PAPERCLIP_RUNNER_DEFAULT_MODELS.codex` in
`packages/adapter-utils/src/paperclip-runner-permissions.ts`. This also
changes `DEFAULT_CODEX_LOCAL_MODEL`,
`resolvePaperclipRunnerModel("codex", …)`, runner config normalization,
`codex_local` → Paperclip Runner conversion, the UI new-agent and
agent-config defaults, the composer default model and the public MCP
effort lookup.
- `codex-local` `models`: the 5.6 entry is now the literal
`gpt-5.6-sol`, because it used to be `DEFAULT_CODEX_LOCAL_MODEL`. The
list order and contents do not change.
- AI connection pool: a new OpenAI member profile uses
`PAPERCLIP_RUNNER_DEFAULT_MODELS.codex` instead of a hard-coded
`gpt-5.6-sol`.
- Paperclip Runner devtools ACPX Codex default and preset, the
`runner-codex-sol` local provider smoke, and the `acpx-codex-sol` live
workflow candidate.
- `tests/acpx-qualification-models.ts` Codex selection (runner E2E and
runner acceptance ACPX Codex profiles).
- Runner E2E: the `runner-codex` profile follows the default, so all
native Codex cells move. This includes the `runner-codex` cells among
the 52 standard `everyday-workflows` cells and the seven Codex
reliability cells. `runner-codex-mini` stays on `gpt-5.4-mini`. The
Reliability plan `queue-steer` `steeringProfile` is now
`['runner-codex', 'gpt-6.1-sol']`. Its executionId
(`everyday-workflows.runner-codex.daytona.queue-steer`), its fullName
and its TestOps case identity do not change. No separate profile is
necessary.
- `product-regression-testops.mjs`: the declared model for every
`runner-codex` case is now `gpt-6.1-sol` (it was already that for
`configured-repository`).
- `tests/runner-e2e/installed-cli.ts`: the credential-free installed
startup proof opens a session on the default model.
- `tests/runner-e2e/scheduled-prices.json`: add a `gpt-6.1-sol` price
(`openai-standard-2026-11-06`, including 30m/12h cache writes and
long-context rates from the OpenAI pricing page). The weekly scheduled
Codex cells need an exact price for their model, or admission fails
closed.
- Storybook: the composer model picker fixtures, the connection pool
stories and host, and the agent settings preview show `gpt-6.1-sol` as
the default.
- Docs: `doc/DEVELOPING.md` (the Codex conversion default) and
`tests/runner-e2e/EVERYDAY-WORKFLOWS.md` (local matrix, reliability
extras, `queue-steer` row).
- Tests that assert the default now expect `gpt-6.1-sol`. New
assertions: the default has the `0.159.0` CLI floor, the compatibility
fallback is floor-free, the `queue-steer` plan cell runs `runner-codex`
on `gpt-6.1-sol`, and all native Codex everyday cells (standard and
reliability) use `gpt-6.1-sol`.
Deliberately kept on `gpt-5.6-sol`:
- Rate cards: `server/src/services/codex-pricing.ts` and
`packages/paperclip-runner/src/evals/model-pricing.ts`. Old receipts
need them. (The runner eval rate card has no `gpt-6.1-sol` entry. An
unpriced candidate is reported as unpriced, not as zero. I did not add a
rate there because that card has its own version and source.)
- The remote Codex CLI compatibility fallback. A new
`CODEX_LOCAL_COMPATIBILITY_FALLBACK_MODEL = "gpt-5.6-sol"` replaces the
default in `codexModelFallbackCandidates`. `gpt-6.1-sol` needs Codex
0.159.0 or newer, so it cannot be the floor-free last resort. Fallback
behavior is unchanged.
- `tests/runner-e2e/provider-usage.ts`: the implicit request
cache-policy attribution stays qualified only for `gpt-5.6-sol`. I did
not verify that policy for `gpt-6.1-sol`. A `gpt-6.1-sol` response that
omits the write-TTL split keeps its cache-write cost unknown, which is
the existing behavior for unqualified models.
- `tests/runner-e2e/scheduled-measurements.json`: historical
`gpt-5.6-sol` measurements. They no longer match the current suite
definitions, so the forecast reports them as historical only. The
forecast tests now build their synthetic cohorts on the current
definition hash.
- Model catalogs that list `gpt-5.6-sol` as available (`codex-local`,
`cursor-local`, `opencode-local`, Speko voice diagnostics, Pi picker
fixture).
- The `gpt-5.6` → `gpt-5.6-sol` alias for old stored configs.
- Dated or evidence docs: `doc/plans/*`, `doc/adapter-model-audit-*`,
`QUALIFICATION-*`, `doc/connections/MEMORY.md`,
`doc/connections/TASK-REVIEWS-VERIFICATION.md`,
`tests/runner-e2e/CHAT-FEATURES.md`, the Slack eval evidence, and the
S49 soak signature text (the soak failure happened on `gpt-5.6-sol`).
- Test fixtures that parse, price, migrate or round-trip an existing
`gpt-5.6-sol` value (Rust runner tests, driver tests, pricing tests,
stored-config tests, stories that show an existing agent).
## Verification
- `npx vitest run` on every server, UI, adapter-utils and codex-local
test file that mentions `gpt-5.6-sol` or the default, plus all of
`packages/adapter-utils` and `packages/adapters/codex-local`: 5,483
passed. Three timeouts under load (`sandbox-managed-runtime`,
`acp.quota`) passed when rerun.
- `packages/paperclip-runner`: `vitest run` for `src/eval`, `src/evals`,
`src/drivers/acpx`, `src/drivers/codex`, `src/cli`, `src/contracts`,
`src/backends`: 2,641 passed.
- `node --test tests/runner-e2e/reliability-testops.test.mjs
tests/runner-e2e/product-regression-testops.test.mjs`: 28 passed.
- `node tests/runner-e2e/run-unit.mjs`: all model, catalog, reliability,
scheduled and TestOps tests pass. Failures remain only in
`copilot-protection-flow`, `denied-target-isolation-flow`,
`pi-controls-flow`, `workflow-target` and `unit-launcher`. These five
also fail on unmodified master in this local environment (filesystem
watch and git environment), and they do not touch models.
- `vitest run --config tests/runner-acceptance/vitest.config.ts`: 19
passed.
- `pnpm test:e2e:runner -- --list --suite everyday-workflows --group
reliability`: the seven `everyday-workflows.runner-codex.daytona.<case>`
cells show `gpt-6.1-sol` with `OPENAI_API_KEY,DAYTONA_API_KEY`.
`--matrix-json` gives 16 reliability cells with the same credentials as
before. `--list --all` shows no `gpt-5.6-sol` cell.
- Typecheck: `adapter-utils`, `adapter-codex-local`, `server`, `ui`,
`paperclip-runner` (TypeScript), and `pnpm test:e2e:runner:typecheck`.
- `pnpm check:token-gates`: clean.
- Not run: paid cells, Rust tests (no Rust change), the browser suites,
and the full `pnpm -r typecheck` / `pnpm build` (CI runs them).
## Risks
- Agents without an explicit model change model on their next save or
normalization, and Paperclip Runner rows with a blank model use
`gpt-6.1-sol` at execution. Stored models, including `gpt-5.6-sol`, are
never rewritten. No migration is necessary.
- `gpt-6.1-sol` needs Codex CLI 0.159.0 or newer with ChatGPT sign-in.
The runner pins `@openai/codex@0.160.0`. On a remote target with an
older image CLI, the existing compatibility check falls back to
`gpt-6-sol` and then `gpt-5.6-sol`, and that probe now runs for
default-model agents too. A legacy `codex_local` agent created with the
new default on a host with an older Codex CLI gets the existing
actionable "upgrade Codex" Test result.
- Qualification: the native Codex profile uses `adapter_constant` model
qualification, so it follows the default. No qualification gate is
dropped. The ACPX Codex qualification selection moves to `gpt-6.1-sol`.
Its next qualification run uses the new model. The provider-usage
cache-policy attribution stays limited to `gpt-5.6-sol`.
- Runner E2E history: the Codex cells change their suite definition
hashes, so earlier `gpt-5.6-sol` results are a separate cohort and do
not count as current.
- The `gpt-6.1-sol` scheduled price uses the file's existing `expiresAt`
(2026-11-19). It needs the same review as the other entries before then.
- No `.github/` change. The Reliability queue-steer cell keeps the
`runner-codex` profile id, so the existing workflow credential and setup
steps still match.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, running in
Claude Code with tool use (shell, file edits, web fetch of the OpenAI
pricing page).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Related: #19604 (commits the Reliability TestOps identities; this PR
changes only `steeringProfile` in the same file).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
96 lines
4.5 KiB
TypeScript
96 lines
4.5 KiB
TypeScript
import { posix } from "node:path";
|
|
import {
|
|
codexCliVersionAtLeast,
|
|
CODEX_LOCAL_COMPATIBILITY_FALLBACK_MODEL,
|
|
minimumCodexCliVersionForModel,
|
|
models,
|
|
normalizeCodexModel,
|
|
} from "@paperclipai/adapter-codex-local";
|
|
import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
|
|
import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
|
|
import { createNativeSshCommandRunner } from "./native-ssh-command-runner.js";
|
|
import { isSupportedRemoteCodexVersion, parseCodexCliVersion } from "./codex-runtime-compatibility.js";
|
|
|
|
/** Newest older model in the same class first, then the floor-free compatibility model. */
|
|
export function codexModelFallbackCandidates(requestedModel: string): string[] {
|
|
const requested = normalizeCodexModel(requestedModel);
|
|
const identity = /^gpt-(\d+(?:\.\d+)?)-(astra|sol|terra|luna)$/.exec(requested);
|
|
const sameClass = identity ? models.flatMap(({ id }) => {
|
|
const candidate = /^gpt-(\d+(?:\.\d+)?)-(astra|sol|terra|luna)$/.exec(id);
|
|
return candidate && candidate[2] === identity[2] && Number(candidate[1]) < Number(identity[1])
|
|
? [id] : [];
|
|
}) : [];
|
|
return [...new Set([...sameClass, CODEX_LOCAL_COMPATIBILITY_FALLBACK_MODEL])].filter((model) => model !== requested);
|
|
}
|
|
|
|
export function compatibleCodexModel(model: string | null, version: string | null): string | null {
|
|
if (!version || !isSupportedRemoteCodexVersion(version)) return model;
|
|
const minimum = minimumCodexCliVersionForModel(model);
|
|
if (!minimum || codexCliVersionAtLeast(version, minimum)) return model;
|
|
return selectCodexModelFallback(model!, (candidate) => {
|
|
const floor = minimumCodexCliVersionForModel(candidate);
|
|
return !floor || codexCliVersionAtLeast(version, floor);
|
|
}) ?? model;
|
|
}
|
|
|
|
export function selectCodexModelFallback(requestedModel: string, compatible: (model: string) => boolean): string | null {
|
|
for (const candidate of codexModelFallbackCandidates(requestedModel)) {
|
|
if (compatible(candidate)) return candidate;
|
|
}
|
|
// Keep the verifier's existing actionable failure when no candidate is safe.
|
|
return null;
|
|
}
|
|
|
|
export function parseRemoteExecutableCandidate(stdout: string): string | null {
|
|
const lines = stdout.split(/\r?\n/).map((line) => line.trim()).filter(Boolean);
|
|
if (lines.length !== 1) return null;
|
|
const candidate = lines[0]!;
|
|
if (!candidate.startsWith("/") || candidate.length > 4_096 || !/^\/[A-Za-z0-9_./+@-]+$/.test(candidate)) return null;
|
|
return posix.normalize(candidate);
|
|
}
|
|
|
|
/** Shared with the launch verifier so preparation checks the same image CLI. */
|
|
export async function discoverRemoteExecutable(
|
|
runner: CommandManagedRuntimeRunner,
|
|
cwd: string,
|
|
name: "paperclip-runnerd" | "codex",
|
|
): Promise<string | null> {
|
|
const result = await runner.execute({
|
|
command: "sh",
|
|
args: ["-c", `for candidate in /opt/paperclip-runner/bin/${name} "$HOME/.local/bin/${name}"; do ` +
|
|
`if [ -x "$candidate" ]; then printf '%s\\n' "$candidate"; exit 0; fi; done; ` +
|
|
`command -v ${name} 2>/dev/null || true`],
|
|
cwd, bypassSession: true, timeoutMs: 10_000,
|
|
});
|
|
return result.exitCode === 0 && !result.timedOut ? parseRemoteExecutableCandidate(result.stdout) : null;
|
|
}
|
|
|
|
export async function readRemoteCodexModelCliVersion(input: {
|
|
model: string | null;
|
|
target?: AdapterExecutionTarget | null;
|
|
remoteCodexPath?: string | null;
|
|
remoteCodexNpmSpec?: string | null;
|
|
}): Promise<string | null> {
|
|
const target = input.target;
|
|
// An explicit artifact or install pin takes precedence over the image CLI.
|
|
if (target?.kind !== "remote" || !minimumCodexCliVersionForModel(input.model)
|
|
|| input.remoteCodexPath?.trim() || input.remoteCodexNpmSpec?.trim()) return null;
|
|
try {
|
|
const runner = target.transport === "ssh"
|
|
? createNativeSshCommandRunner({ spec: target.spec, defaultCwd: target.remoteCwd })
|
|
: target.runner;
|
|
if (!runner) return null;
|
|
const executable = await discoverRemoteExecutable(runner, target.remoteCwd, "codex");
|
|
if (!executable) return null;
|
|
const result = await runner.execute({ command: executable, args: ["--version"],
|
|
cwd: target.remoteCwd, bypassSession: true, timeoutMs: 30_000 });
|
|
if (result.exitCode !== 0 || result.timedOut) return null;
|
|
const version = parseCodexCliVersion(`${result.stdout}\n${result.stderr}`);
|
|
return version && isSupportedRemoteCodexVersion(version) ? version : null;
|
|
} catch {
|
|
// Optional preparation probes must not bypass launch verification or its
|
|
// existing recovery policy when a remote command rejects or times out.
|
|
return null;
|
|
}
|
|
}
|