mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-03 09:37:19 +01:00
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/26/changes/paperclip/26 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents run on coding harnesses. Each adapter has a model catalog, and sandboxes contain pinned CLIs. > - Upstream providers released new CLI versions and new models. OpenAI retired some Codex models for ChatGPT sign-in. > - A stale catalog hides valid models. A misleading error makes operators upgrade a CLI that is already current. > - This pull request adds verified OpenCode model IDs, names Codex ChatGPT retirements in the environment Test, and refreshes the sandbox CLI pins. > - The benefit is that operators can select current models and get the correct reason when a retired model fails. ## Linked Issues or Issue Description Refs #16708 (adds Claude Haiku 5.5 to `claude_local`; this PR does not duplicate it). Refs #17245 (moves the Codex runtime to 0.163.0; this PR does not change Codex pins). **What happened?** OpenAI retired `gpt-5.5` (October 14, 2026) and `gpt-5.4` / `gpt-5.4-mini` (August 31, 2026) from Codex with ChatGPT sign-in. A saved agent with one of these models and ChatGPT sign-in now fails. The legacy environment Test says that the plan, the rollout, or an old CLI can be the cause. The Paperclip Runner Test gives generic account and runtime advice. The real cause is the retirement. **Expected behavior** Both Tests give the retirement date and the documented replacement. API-key users keep the models. **Proposed change** Record the verified retirements. Add the new verified OpenCode IDs. Refresh the sandbox CLI pins with verified digests. Record the audit. ## What Changed - Codex: add a table of models retired from Codex with ChatGPT sign-in (`gpt-5.5` → `gpt-6.1-sol`, or `gpt-6-luna` on Free and Go; `gpt-5.4` → `gpt-6-sol`; `gpt-5.4-mini` → `gpt-6-luna`). The shared `codexChatGptRetirementHint` builds the hint. The legacy CLI Test (`codexHelloProbeModelRejectionCheck`) and the Paperclip Runner Test (`testNativeRunnerAuthentication`) both use it when the backend rejects one of these models. For other failures, the Paperclip Runner Test hint stays the same. The picker keeps the models for API-key users. - OpenCode: add `anthropic/claude-haiku-5-5` and `google/gemini-4-argon` to the remote fallback catalog. The Paperclip Runner OpenCode picker also uses this catalog. - Evaluation image sandbox layer (`docker/daytona-runner`): | CLI | Before | After | | --- | --- | --- | | Grok CLI | 1.0.46 | 1.0.70 | | Gemini CLI | 0.62.0 | 0.65.0 | | Kimi Code | 2.1.1 | 2.3.3 | | Cursor CLI | 2026.10.01-e373342 | 2026.10.22-27dcf5f | | GitHub CLI | 2.102.0 | 2.103.3 | - Add `doc/adapter-model-audit-2026-10-23.md` with sources, compatibility findings, and deferred items. No model in this PR needs a newer Codex, Claude, or OpenCode runtime than the current pins. Claude Haiku 5.5 needs Claude Code 2.1.293 or later; #16708 records that floor, and this PR does not add it. ## Verification - `vitest run src/index.test.ts src/server/test.remote.test.ts src/server/acp.cli-version.test.ts` in `packages/adapters/codex-local`: 41 passed. - `vitest run src/services/native-runtime/setup-readiness.test.ts` in `server`: 98 passed. These include the two new Paperclip Runner Test cases: a retired model shows the retirement hint, and a model that is not retired keeps the generic hint. The retirement case fails on the earlier code. Run with the host's empty `ANTHROPIC_BASE_URL` unset; see the baseline note in the audit doc. - `vitest run` in `packages/adapters/opencode-local`: 87 passed. - `vitest run src/__tests__/adapter-models.test.ts` in `server`: 27 passed. - `tsc --noEmit` in `packages/adapters/codex-local` and `packages/adapters/opencode-local`: passed. - Cursor archive SHA-256 `85296944e4600a83c4b764f578d1502bf9c416db200d33a593e8f3ecabfdc294` is computed from the versioned Linux x64 tarball. The extracted binary reports `2026.10.22-27dcf5f`. On October 26, 2026, `https://cursor.com/install` still resolves this version. - GitHub CLI SHA-256 `4c806978ffce1cec70853302a56d70ac6da4f19bae7adca4e1a8ab00633dd91d` matches `gh_2.103.3_checksums.txt`. - Grok CLI 1.0.70 and Kimi Code 2.3.3 installed from npm report those versions. The flags that Paperclip uses (`-m`, `--effort`, `grok models`; `-m`, `-p`, `--output-format`, `acp`) are present in `--help`. - `git merge-tree` with #16708 (head `3a06740e9`) and #17245 (head `41f9c611f`): this PR shares no files with either PR. The two PRs can merge in any order. - Not run in the authoring environment: the full `pnpm -r typecheck` and `pnpm build` (no Rust toolchain), the full server `tsc` (memory limit), and the image build. CI runs these. ## Risks - Low risk for catalogs: the IDs are additive suggestions. No saved selection, default, or spending setting changes. - The Paperclip Runner Test changes only the hint text, and only for a ChatGPT rejection of a retired model. Check codes and pass/fail results do not change. - Sandbox CLI pins: - Kimi Code 2.2.0 and 2.3.0 remove the `kaos` MCP executor and require qualified plugin skill names. Paperclip uses neither. Kimi Code 2.3.2 stops automatic swarms in `kimi -p` and ACP sessions. - Gemini CLI 0.65.0 makes workspace settings read-only in untrusted folders. Paperclip writes home settings and sets `GEMINI_CLI_TRUST_WORKSPACE` for remote runs. - xAI publishes no Grok CLI changelog. Grok 1.0.70 behavior changes are verified only against the CLI reference and `--help`. - The pins apply only after a new image is built and promoted. The fleet sandbox image definition must get the same pins. - Rollback: revert this PR. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, web research, repository search). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing>
614 lines
25 KiB
TypeScript
614 lines
25 KiB
TypeScript
import fs from "node:fs/promises";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
|
import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
|
|
|
|
const {
|
|
ensureAdapterExecutionTargetDirectory,
|
|
ensureAdapterExecutionTargetCommandResolvable,
|
|
maybeRunSandboxInstallCommand,
|
|
runAdapterExecutionTargetProcess,
|
|
describeAdapterExecutionTarget,
|
|
resolveAdapterExecutionTargetCwd,
|
|
prepareAdapterExecutionTargetRuntime,
|
|
prepareManagedCodexHome,
|
|
restoreWorkspace,
|
|
capturedHomeAssetFiles,
|
|
capturedHomeAssetAuthJson,
|
|
} = vi.hoisted(() => {
|
|
const restoreWorkspace = vi.fn(async () => {});
|
|
// Records the files staged in the uploaded "home" asset at call time, before
|
|
// the probe's cleanup deletes the temp dir. Lets tests assert the upload is a
|
|
// minimal credentials-only home and not the full managed CODEX_HOME.
|
|
const capturedHomeAssetFiles: { value: string[] | null } = { value: null };
|
|
// Records the staged auth.json content, so tests can assert WHICH home's
|
|
// credential the probe uploaded (the effective home a run would use).
|
|
const capturedHomeAssetAuthJson: { value: string | null } = { value: null };
|
|
return {
|
|
capturedHomeAssetFiles,
|
|
capturedHomeAssetAuthJson,
|
|
ensureAdapterExecutionTargetDirectory: vi.fn(async () => {}),
|
|
ensureAdapterExecutionTargetCommandResolvable: vi.fn(async () => {}),
|
|
maybeRunSandboxInstallCommand: vi.fn(async () => null),
|
|
runAdapterExecutionTargetProcess: vi.fn(async () => ({
|
|
exitCode: 0,
|
|
signal: null,
|
|
timedOut: false,
|
|
stdout: [
|
|
"{\"type\":\"thread.started\",\"thread_id\":\"thread-1\"}",
|
|
"{\"type\":\"item.completed\",\"item\":{\"type\":\"agent_message\",\"text\":\"hello\"}}",
|
|
"{\"type\":\"turn.completed\",\"usage\":{\"input_tokens\":1,\"cached_input_tokens\":0,\"output_tokens\":1}}",
|
|
].join("\n"),
|
|
stderr: "",
|
|
pid: 123,
|
|
startedAt: new Date().toISOString(),
|
|
})),
|
|
describeAdapterExecutionTarget: vi.fn(() => "QA SSH"),
|
|
resolveAdapterExecutionTargetCwd: vi.fn((target, configuredCwd, fallbackCwd) => {
|
|
if (typeof configuredCwd === "string" && configuredCwd.trim().length > 0) return configuredCwd;
|
|
if (target && typeof target === "object" && "remoteCwd" in target && typeof target.remoteCwd === "string") {
|
|
return target.remoteCwd;
|
|
}
|
|
return fallbackCwd;
|
|
}),
|
|
prepareAdapterExecutionTargetRuntime: vi.fn(async (input: { assets?: Array<{ key: string; localDir: string }> }) => {
|
|
const homeAsset = input?.assets?.find((asset) => asset.key === "home");
|
|
if (homeAsset) {
|
|
capturedHomeAssetFiles.value = (await fs.readdir(homeAsset.localDir)).sort();
|
|
capturedHomeAssetAuthJson.value = await fs
|
|
.readFile(`${homeAsset.localDir}/auth.json`, "utf8")
|
|
.catch(() => null);
|
|
}
|
|
return {
|
|
target: null,
|
|
workspaceRemoteDir: "/remote/workspace/.paperclip-runtime/runs/test/workspace",
|
|
runtimeRootDir: "/remote/workspace/.paperclip-runtime/runs/test/workspace/.paperclip-runtime/codex",
|
|
assetDirs: {
|
|
home: "/remote/workspace/.paperclip-runtime/runs/test/workspace/.paperclip-runtime/codex/home",
|
|
},
|
|
restoreWorkspace,
|
|
};
|
|
}),
|
|
prepareManagedCodexHome: vi.fn(async () => {
|
|
// Return a real managed home seeded with credentials so the probe's
|
|
// minimal-home copy step (auth.json/config.toml) has something to read.
|
|
const dir = await fs.mkdtemp(`${os.tmpdir()}/paperclip-managed-codex-home-`);
|
|
await fs.writeFile(`${dir}/auth.json`, JSON.stringify({ OPENAI_API_KEY: "sk-managed" }));
|
|
await fs.writeFile(`${dir}/config.toml`, "model = \"gpt-5\"\n");
|
|
return dir;
|
|
}),
|
|
restoreWorkspace,
|
|
};
|
|
});
|
|
|
|
vi.mock("@paperclipai/adapter-utils/execution-target", async () => {
|
|
const actual = await vi.importActual<typeof import("@paperclipai/adapter-utils/execution-target")>(
|
|
"@paperclipai/adapter-utils/execution-target",
|
|
);
|
|
return {
|
|
...actual,
|
|
ensureAdapterExecutionTargetDirectory,
|
|
ensureAdapterExecutionTargetCommandResolvable,
|
|
maybeRunSandboxInstallCommand,
|
|
runAdapterExecutionTargetProcess,
|
|
describeAdapterExecutionTarget,
|
|
resolveAdapterExecutionTargetCwd,
|
|
prepareAdapterExecutionTargetRuntime,
|
|
};
|
|
});
|
|
|
|
vi.mock("./codex-home.js", async () => {
|
|
const actual = await vi.importActual<typeof import("./codex-home.js")>("./codex-home.js");
|
|
return {
|
|
...actual,
|
|
prepareManagedCodexHome,
|
|
};
|
|
});
|
|
|
|
import { testEnvironment } from "./test.js";
|
|
|
|
describe("codex remote environment diagnostics", () => {
|
|
const scratchDirs: string[] = [];
|
|
|
|
async function makeScratchDir(prefix: string): Promise<string> {
|
|
const dir = await fs.mkdtemp(path.join(os.tmpdir(), prefix));
|
|
scratchDirs.push(dir);
|
|
return dir;
|
|
}
|
|
|
|
beforeEach(async () => {
|
|
// The probe mirrors execute's home preparation, which reads the shared
|
|
// source home and the auth cache from `process.env`. Pin both to empty
|
|
// scratch locations so no test ever reads or writes the real ~/.codex or
|
|
// the real instance tree.
|
|
vi.stubEnv("CODEX_HOME", await makeScratchDir("paperclip-test-shared-codex-"));
|
|
vi.stubEnv("PAPERCLIP_HOME", await makeScratchDir("paperclip-test-instance-"));
|
|
vi.stubEnv("PAPERCLIP_INSTANCE_ID", "default");
|
|
});
|
|
|
|
afterEach(async () => {
|
|
vi.clearAllMocks();
|
|
vi.unstubAllEnvs();
|
|
delete process.env.OPENAI_API_KEY;
|
|
capturedHomeAssetFiles.value = null;
|
|
capturedHomeAssetAuthJson.value = null;
|
|
while (scratchDirs.length > 0) {
|
|
const dir = scratchDirs.pop();
|
|
if (dir) await fs.rm(dir, { recursive: true, force: true }).catch(() => undefined);
|
|
}
|
|
});
|
|
|
|
it("stages managed CODEX_HOME in an isolated runtime dir and keeps the probe cwd on the original remote workspace", async () => {
|
|
const remoteTarget: AdapterExecutionTarget = {
|
|
kind: "remote",
|
|
transport: "ssh",
|
|
remoteCwd: "/remote/workspace",
|
|
spec: {
|
|
host: "127.0.0.1",
|
|
port: 22,
|
|
username: "agent",
|
|
privateKey: "PRIVATE KEY",
|
|
knownHosts: "KNOWN HOSTS",
|
|
remoteCwd: "/remote/workspace",
|
|
remoteWorkspacePath: "/remote/workspace",
|
|
strictHostKeyChecking: false,
|
|
},
|
|
};
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "codex_local",
|
|
config: {
|
|
engine: "cli",
|
|
command: "codex",
|
|
},
|
|
executionTarget: remoteTarget,
|
|
environmentName: "QA SSH",
|
|
});
|
|
|
|
expect(result.status).toBe("pass");
|
|
expect(result.checks.some((check) => check.code === "codex_hello_probe_passed")).toBe(true);
|
|
expect(prepareManagedCodexHome).toHaveBeenCalledTimes(1);
|
|
expect(prepareAdapterExecutionTargetRuntime).toHaveBeenCalledTimes(1);
|
|
const runtimeCalls = prepareAdapterExecutionTargetRuntime.mock.calls as unknown as Array<[
|
|
{
|
|
workspaceLocalDir: string;
|
|
target?: { remoteCwd?: string };
|
|
workspaceRemoteDir?: string;
|
|
assets?: Array<{ key: string; localDir: string }>;
|
|
},
|
|
]>;
|
|
const runtimeInput = runtimeCalls[0]?.[0];
|
|
// The probe must upload only a minimal credentials-only home, never the
|
|
// full managed CODEX_HOME (which can be hundreds of MB of session history).
|
|
const homeAsset = runtimeInput?.assets?.find((asset) => asset.key === "home");
|
|
expect(homeAsset?.localDir).toContain(`${os.tmpdir()}/paperclip-codex-probe-home-`);
|
|
expect(capturedHomeAssetFiles.value).toEqual(["auth.json", "config.toml"]);
|
|
expect(runtimeInput?.workspaceLocalDir).toContain(`${os.tmpdir()}/paperclip-codex-envtest-`);
|
|
expect(runtimeInput?.workspaceLocalDir).not.toBe("/remote/workspace");
|
|
expect(await fs.stat(runtimeInput!.workspaceLocalDir).catch(() => null)).toBeNull();
|
|
expect(runtimeInput?.target?.remoteCwd).toBe("/remote/workspace");
|
|
// `workspaceRemoteDir` is the base path passed to the runtime; the
|
|
// helper's per-run subdirectory is appended internally inside
|
|
// `prepareRemoteManagedRuntime`. Pre-building a per-run prefix here
|
|
// would double-nest the run id in the final path.
|
|
expect(runtimeInput?.workspaceRemoteDir).toBe("/remote/workspace");
|
|
expect(runAdapterExecutionTargetProcess).toHaveBeenCalledTimes(1);
|
|
const probeCall = runAdapterExecutionTargetProcess.mock.calls[0] as unknown as
|
|
| [string, { kind: string; remoteCwd: string }, string, string[], { cwd: string; env: Record<string, string> }]
|
|
| undefined;
|
|
expect(probeCall?.[1]).toMatchObject({
|
|
kind: "remote",
|
|
remoteCwd: "/remote/workspace",
|
|
});
|
|
expect(probeCall?.[4]).toMatchObject({
|
|
cwd: "/remote/workspace",
|
|
env: expect.objectContaining({
|
|
CODEX_HOME: "/remote/workspace/.paperclip-runtime/runs/test/workspace/.paperclip-runtime/codex/home",
|
|
}),
|
|
});
|
|
expect(restoreWorkspace).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it("avoids /tmp CODEX_HOME for remote API-key hello probes", async () => {
|
|
const remoteTarget: AdapterExecutionTarget = {
|
|
kind: "remote",
|
|
transport: "sandbox",
|
|
providerKey: "cloudflare",
|
|
remoteCwd: "/remote/workspace",
|
|
runner: {
|
|
execute: async () => ({
|
|
exitCode: 0,
|
|
signal: null,
|
|
timedOut: false,
|
|
stdout: "",
|
|
stderr: "",
|
|
pid: null,
|
|
startedAt: new Date().toISOString(),
|
|
}),
|
|
},
|
|
};
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "codex_local",
|
|
config: {
|
|
engine: "cli",
|
|
command: "codex",
|
|
env: {
|
|
OPENAI_API_KEY: "sk-test",
|
|
},
|
|
},
|
|
executionTarget: remoteTarget,
|
|
environmentName: "QA Cloudflare",
|
|
});
|
|
|
|
expect(result.status).toBe("pass");
|
|
const probeCall = runAdapterExecutionTargetProcess.mock.calls[0] as unknown as
|
|
| [string, AdapterExecutionTarget, string, string[], { cwd: string; env: Record<string, string> }]
|
|
| undefined;
|
|
expect(probeCall?.[4].env.CODEX_HOME).toContain("/remote/workspace/.paperclip-runtime/codex/probe-home-codex-envtest-");
|
|
expect(probeCall?.[4].env.CODEX_HOME?.startsWith("/tmp/")).toBe(false);
|
|
expect(probeCall?.[3]).toContain("--skip-git-repo-check");
|
|
});
|
|
|
|
// `sandboxTarget()` is declared later in this describe block; function
|
|
// declarations hoist, so the version-floor cases below reuse it.
|
|
function versionProbeResult(stdout: string) {
|
|
return {
|
|
exitCode: 0,
|
|
signal: null,
|
|
timedOut: false,
|
|
stdout,
|
|
stderr: "",
|
|
pid: 111,
|
|
startedAt: new Date().toISOString(),
|
|
};
|
|
}
|
|
|
|
it("fails before the hello probe when the sandbox Codex CLI is older than the model's floor", async () => {
|
|
// A sandbox image baked before gpt-6.1-sol reached the catalog still runs
|
|
// Codex 0.156.0. The backend would reject every turn with "not supported
|
|
// when using Codex with a ChatGPT account"; the Test must name the real
|
|
// gap (the CLI in the image) instead of running that doomed probe.
|
|
runAdapterExecutionTargetProcess.mockResolvedValueOnce(versionProbeResult("codex-cli 0.156.0\n"));
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "codex_local",
|
|
config: { engine: "cli", command: "codex", model: "gpt-6.1-sol" },
|
|
executionTarget: sandboxTarget(),
|
|
environmentName: "QA Daytona",
|
|
});
|
|
|
|
expect(result.status).toBe("fail");
|
|
const incompatible = result.checks.find((check) => check.code === "codex_cli_version_incompatible");
|
|
expect(incompatible).toMatchObject({
|
|
level: "error",
|
|
message: "gpt-6.1-sol requires Codex CLI 0.159.0 or newer with ChatGPT sign-in.",
|
|
detail: "Detected Codex CLI 0.156.0.",
|
|
});
|
|
expect(incompatible?.hint).toContain("sandbox image");
|
|
expect(result.checks.some((check) => check.code === "codex_hello_probe_skipped_cli_version")).toBe(true);
|
|
expect(result.checks.some((check) => check.code.startsWith("codex_hello_probe_") && !check.code.includes("skipped"))).toBe(false);
|
|
expect(runAdapterExecutionTargetProcess).toHaveBeenCalledTimes(1);
|
|
const versionCall = runAdapterExecutionTargetProcess.mock.calls[0] as unknown as [string, AdapterExecutionTarget, string, string[]];
|
|
expect(versionCall[2]).toBe("codex");
|
|
expect(versionCall[3]).toEqual(["--version"]);
|
|
expect(prepareAdapterExecutionTargetRuntime).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("runs the hello probe when the sandbox Codex CLI satisfies the model's floor", async () => {
|
|
runAdapterExecutionTargetProcess.mockResolvedValueOnce(versionProbeResult("codex-cli 0.160.0\n"));
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "codex_local",
|
|
config: { engine: "cli", command: "codex", model: "gpt-6.1-sol" },
|
|
executionTarget: sandboxTarget(),
|
|
environmentName: "QA Daytona",
|
|
});
|
|
|
|
expect(result.status).toBe("pass");
|
|
expect(result.checks.find((check) => check.code === "codex_cli_version_compatible")).toMatchObject({
|
|
level: "info",
|
|
message: "Codex CLI 0.160.0 satisfies the 0.159.0 minimum for gpt-6.1-sol.",
|
|
});
|
|
expect(result.checks.some((check) => check.code === "codex_hello_probe_passed")).toBe(true);
|
|
expect(runAdapterExecutionTargetProcess).toHaveBeenCalledTimes(2);
|
|
});
|
|
|
|
it("skips the version probe for models without a verified floor", async () => {
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "codex_local",
|
|
config: { engine: "cli", command: "codex", model: "gpt-5.6-sol" },
|
|
executionTarget: sandboxTarget(),
|
|
environmentName: "QA Daytona",
|
|
});
|
|
|
|
expect(result.status).toBe("pass");
|
|
expect(result.checks.some((check) => check.code.startsWith("codex_cli_version_"))).toBe(false);
|
|
expect(runAdapterExecutionTargetProcess).toHaveBeenCalledTimes(1);
|
|
const helloCall = runAdapterExecutionTargetProcess.mock.calls[0] as unknown as [string, AdapterExecutionTarget, string, string[]];
|
|
expect(helloCall[3]).not.toEqual(["--version"]);
|
|
});
|
|
|
|
it("names the backend's ChatGPT model rejection instead of a generic probe failure", async () => {
|
|
// The CLI is new enough, but the account's plan or rollout does not include
|
|
// the model. The backend answers with one fixed sentence; surface it with
|
|
// the detected CLI version so the operator can tell the two causes apart.
|
|
runAdapterExecutionTargetProcess.mockResolvedValueOnce(versionProbeResult("codex-cli 0.160.0\n"));
|
|
runAdapterExecutionTargetProcess.mockResolvedValueOnce({
|
|
exitCode: 1,
|
|
signal: null,
|
|
timedOut: false,
|
|
stdout: "",
|
|
stderr: "ERROR: {\"type\":\"error\",\"status\":400,\"error\":{\"type\":\"invalid_request_error\",\"message\":\"The 'gpt-6.1-sol' model is not supported when using Codex with a ChatGPT account.\"}}",
|
|
pid: 222,
|
|
startedAt: new Date().toISOString(),
|
|
});
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "codex_local",
|
|
config: { engine: "cli", command: "codex", model: "gpt-6.1-sol" },
|
|
executionTarget: sandboxTarget(),
|
|
environmentName: "QA Daytona",
|
|
});
|
|
|
|
expect(result.status).toBe("fail");
|
|
const rejected = result.checks.find((check) => check.code === "codex_hello_probe_model_rejected");
|
|
expect(rejected).toMatchObject({
|
|
level: "error",
|
|
message: "The Codex backend rejected gpt-6.1-sol for this ChatGPT account.",
|
|
detail: "The 'gpt-6.1-sol' model is not supported when using Codex with a ChatGPT account. Detected Codex CLI 0.160.0.",
|
|
});
|
|
expect(rejected?.hint).toContain("0.159.0");
|
|
expect(rejected?.hint).toContain("plan");
|
|
expect(result.checks.some((check) => check.code === "codex_hello_probe_failed")).toBe(false);
|
|
// Not an auth failure: the managed-connection flow must not invalidate the credential.
|
|
expect(result.checks.some((check) => check.code === "codex_hello_probe_auth_required")).toBe(false);
|
|
});
|
|
|
|
it("names the ChatGPT retirement when the backend rejects a retired model", async () => {
|
|
runAdapterExecutionTargetProcess.mockResolvedValueOnce({
|
|
exitCode: 1,
|
|
signal: null,
|
|
timedOut: false,
|
|
stdout: "",
|
|
stderr: "ERROR: {\"type\":\"error\",\"status\":400,\"error\":{\"type\":\"invalid_request_error\",\"message\":\"The 'gpt-5.5' model is not supported when using Codex with a ChatGPT account.\"}}",
|
|
pid: 223,
|
|
startedAt: new Date().toISOString(),
|
|
});
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "codex_local",
|
|
config: { engine: "cli", command: "codex", model: "gpt-5.5" },
|
|
executionTarget: sandboxTarget(),
|
|
environmentName: "QA Daytona",
|
|
});
|
|
|
|
expect(result.status).toBe("fail");
|
|
const rejected = result.checks.find((check) => check.code === "codex_hello_probe_model_rejected");
|
|
expect(rejected).toMatchObject({
|
|
level: "error",
|
|
message: "The Codex backend rejected gpt-5.5 for this ChatGPT account.",
|
|
hint: "OpenAI retired gpt-5.5 from Codex with ChatGPT sign-in on 2026-10-14. Choose gpt-6.1-sol (gpt-6-luna on Free and Go plans), or use an OpenAI API key; API-key sessions are not affected.",
|
|
});
|
|
expect(rejected?.hint).not.toContain("Upgrade Codex");
|
|
});
|
|
|
|
it("emits the canonical adapter_auth_missing check when a sandbox hello probe reports missing auth", async () => {
|
|
// The sandbox has no seedable credentials, so the hello probe returns an
|
|
// authentication-required error. The Test must emit the neutral canonical
|
|
// check code. The user interface reads this code to decide login
|
|
// eligibility; it does not parse the message text or the top-level status.
|
|
prepareManagedCodexHome.mockImplementationOnce(async () => {
|
|
const dir = await fs.mkdtemp(`${os.tmpdir()}/paperclip-managed-codex-home-noauth-`);
|
|
await fs.writeFile(`${dir}/config.toml`, "model = \"gpt-5\"\n");
|
|
return dir;
|
|
});
|
|
runAdapterExecutionTargetProcess.mockResolvedValueOnce({
|
|
exitCode: 1,
|
|
signal: null,
|
|
timedOut: false,
|
|
stdout: "",
|
|
stderr: "Not logged in. Please run `codex login` to authenticate.",
|
|
pid: 321,
|
|
startedAt: new Date().toISOString(),
|
|
});
|
|
|
|
const remoteTarget: AdapterExecutionTarget = {
|
|
kind: "remote",
|
|
transport: "sandbox",
|
|
providerKey: "daytona",
|
|
remoteCwd: "/remote/workspace",
|
|
runner: {
|
|
execute: async () => ({
|
|
exitCode: 0,
|
|
signal: null,
|
|
timedOut: false,
|
|
stdout: "",
|
|
stderr: "",
|
|
pid: null,
|
|
startedAt: new Date().toISOString(),
|
|
}),
|
|
},
|
|
};
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "codex_local",
|
|
config: { engine: "cli", command: "codex" },
|
|
executionTarget: remoteTarget,
|
|
environmentName: "QA Daytona",
|
|
});
|
|
|
|
// A missing-auth probe is a warning, not a failure, so the environment stays
|
|
// testable and the user interface can offer login.
|
|
expect(result.status).toBe("warn");
|
|
expect(result.checks.some((check) => check.code === "adapter_auth_missing")).toBe(true);
|
|
// The descriptive probe check stays, so existing diagnostics keep working.
|
|
expect(result.checks.some((check) => check.code === "codex_hello_probe_auth_required")).toBe(true);
|
|
});
|
|
|
|
it("does not override CODEX_HOME when the host has no credentials to seed", async () => {
|
|
// Custom-image flow: the login lives inside the captured snapshot, and the
|
|
// host has no Codex auth.json. The probe must not upload an empty home or
|
|
// set CODEX_HOME, so Codex falls back to the sandbox's baked-in login.
|
|
prepareManagedCodexHome.mockImplementationOnce(async () => {
|
|
const dir = await fs.mkdtemp(`${os.tmpdir()}/paperclip-managed-codex-home-noauth-`);
|
|
// No auth.json — only a config file.
|
|
await fs.writeFile(`${dir}/config.toml`, "model = \"gpt-5\"\n");
|
|
return dir;
|
|
});
|
|
|
|
const remoteTarget: AdapterExecutionTarget = {
|
|
kind: "remote",
|
|
transport: "sandbox",
|
|
providerKey: "daytona",
|
|
remoteCwd: "/remote/workspace",
|
|
runner: {
|
|
execute: async () => ({
|
|
exitCode: 0,
|
|
signal: null,
|
|
timedOut: false,
|
|
stdout: "",
|
|
stderr: "",
|
|
pid: null,
|
|
startedAt: new Date().toISOString(),
|
|
}),
|
|
},
|
|
};
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "codex_local",
|
|
config: { engine: "cli", command: "codex" },
|
|
executionTarget: remoteTarget,
|
|
environmentName: "QA Daytona",
|
|
});
|
|
|
|
expect(result.status).toBe("pass");
|
|
// No managed-home upload, so the full-runtime staging is skipped entirely.
|
|
expect(prepareAdapterExecutionTargetRuntime).not.toHaveBeenCalled();
|
|
const probeCall = runAdapterExecutionTargetProcess.mock.calls[0] as unknown as
|
|
| [string, AdapterExecutionTarget, string, string[], { cwd: string; env: Record<string, string> }]
|
|
| undefined;
|
|
expect(probeCall?.[4].env.CODEX_HOME).toBeUndefined();
|
|
});
|
|
|
|
const subscriptionAuth = (accountId: string, marker: string, lastRefresh: string) =>
|
|
JSON.stringify({
|
|
tokens: {
|
|
id_token: `synthetic-id-token-${marker}`,
|
|
access_token: `synthetic-access-token-${marker}`,
|
|
refresh_token: `synthetic-refresh-token-${marker}`,
|
|
account_id: accountId,
|
|
},
|
|
last_refresh: lastRefresh,
|
|
});
|
|
|
|
function sandboxTarget(): AdapterExecutionTarget {
|
|
return {
|
|
kind: "remote",
|
|
transport: "sandbox",
|
|
providerKey: "daytona",
|
|
remoteCwd: "/remote/workspace",
|
|
runner: {
|
|
execute: async () => ({
|
|
exitCode: 0,
|
|
signal: null,
|
|
timedOut: false,
|
|
stdout: "",
|
|
stderr: "",
|
|
pid: null,
|
|
startedAt: new Date().toISOString(),
|
|
}),
|
|
},
|
|
};
|
|
}
|
|
|
|
it("stages a configured managed per-agent CODEX_HOME instead of the company default home", async () => {
|
|
// Execute honors env.CODEX_HOME, so the probe must exercise that same home
|
|
// — otherwise the Test and real runs authenticate with different
|
|
// credentials and can disagree in both directions.
|
|
const perAgentHome = path.join(
|
|
process.env.PAPERCLIP_HOME!,
|
|
"instances",
|
|
"default",
|
|
"companies",
|
|
"company-1",
|
|
"agents",
|
|
"agent-x",
|
|
"codex-home",
|
|
);
|
|
const promoted = subscriptionAuth("acct-agent", "promoted", "2026-07-09T02:00:00Z");
|
|
await fs.mkdir(perAgentHome, { recursive: true });
|
|
await fs.writeFile(path.join(perAgentHome, "auth.json"), promoted, "utf8");
|
|
await fs.writeFile(path.join(perAgentHome, "config.toml"), 'model = "gpt-5"\n', "utf8");
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "codex_local",
|
|
config: {
|
|
engine: "cli",
|
|
command: "codex",
|
|
env: { CODEX_HOME: perAgentHome },
|
|
},
|
|
executionTarget: sandboxTarget(),
|
|
environmentName: "QA Daytona",
|
|
});
|
|
|
|
expect(result.status).toBe("pass");
|
|
// The company default home preparation never ran; the configured managed
|
|
// home was seeded in place and its credential is what got staged.
|
|
expect(prepareManagedCodexHome).not.toHaveBeenCalled();
|
|
expect(capturedHomeAssetAuthJson.value).toBe(promoted);
|
|
// The real seeding pass ran on the per-agent home and kept the promoted
|
|
// regular-file credential (the shared source scratch home is empty).
|
|
const stat = await fs.lstat(path.join(perAgentHome, "auth.json"));
|
|
expect(stat.isSymbolicLink()).toBe(false);
|
|
expect(await fs.readFile(path.join(perAgentHome, "auth.json"), "utf8")).toBe(promoted);
|
|
});
|
|
|
|
it("stages an external CODEX_HOME's credentials as-is and never seeds or mutates it", async () => {
|
|
const externalHome = await makeScratchDir("paperclip-test-external-codex-");
|
|
const external = subscriptionAuth("acct-ext", "external", "2026-07-09T01:00:00Z");
|
|
await fs.writeFile(path.join(externalHome, "auth.json"), external, "utf8");
|
|
// Plant a same-identity, strictly-fresher credential in the shared source
|
|
// home: if the probe wrongly ran the managed seeding pass on the external
|
|
// home, the heal would swap its auth.json for a symlink to this file. The
|
|
// regular-file assertion below is therefore proof no seeding happened.
|
|
await fs.writeFile(
|
|
path.join(process.env.CODEX_HOME!, "auth.json"),
|
|
subscriptionAuth("acct-ext", "shared", "2026-07-09T02:00:00Z"),
|
|
"utf8",
|
|
);
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "codex_local",
|
|
config: {
|
|
engine: "cli",
|
|
command: "codex",
|
|
env: { CODEX_HOME: externalHome },
|
|
},
|
|
executionTarget: sandboxTarget(),
|
|
environmentName: "QA Daytona",
|
|
});
|
|
|
|
expect(result.status).toBe("pass");
|
|
expect(prepareManagedCodexHome).not.toHaveBeenCalled();
|
|
// The external home's own credential is what got staged — not the shared
|
|
// source's fresher copy, because an external override manages its own auth.
|
|
expect(capturedHomeAssetAuthJson.value).toBe(external);
|
|
const stat = await fs.lstat(path.join(externalHome, "auth.json"));
|
|
expect(stat.isSymbolicLink()).toBe(false);
|
|
expect(await fs.readFile(path.join(externalHome, "auth.json"), "utf8")).toBe(external);
|
|
});
|
|
});
|