Files
PaperClipAI/packages/adapters/opencode-local/src/server/test.ts
T
DottaandClaude Opus 5.5 854b4bcac4 fix(opencode): report a provider 401 from the hello probe as missing authentication (#20878)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each agent has a runtime readiness test ("test environment") that
runs a short hello probe through its harness
> - The OpenCode probe sorts failures into model, timeout,
authentication and generic errors
> - OpenRouter rejects an empty or unknown key with HTTP 401 and the
messages "Missing Authentication header" or "User not found."
> - The probe matched authentication only by wording, so these 401
replies became a generic `opencode_hello_probe_failed` error
> - The server maps `*_hello_probe_auth_required` to the missing-account
state, so the user saw a generic failure instead of "connect an account"
> - This pull request classifies the structured APIError status 401 as
`opencode_hello_probe_auth_required`
> - The benefit is that a member without a working account shows the
missing-account state and its fix

## Linked Issues or Issue Description

**What happened**
An OpenCode (native runner) agent with an empty OpenRouter key failed
its readiness test with the generic error `opencode_hello_probe_failed`.
The chat eval `teams.configure-team` could not form its "one member is
missing an account, one is ready" fixture because of this.

**Expected behavior**
The readiness test reports `opencode_hello_probe_auth_required` (warn)
when the provider rejects authentication with HTTP 401, whatever the
provider's wording.

**Steps to reproduce**
1. Configure a native runner OpenCode agent with model
`openrouter/deepseek/deepseek-v4.1-flash-1017` and `OPENROUTER_API_KEY`
set to an empty string.
2. Run the agent's environment test.
3. OpenCode 1.18.34 prints
`{"type":"error","error":{"name":"APIError","data":{"message":"Missing
Authentication header","statusCode":401}}}`.
4. The result is `opencode_hello_probe_failed` (error), not the
missing-authentication check.

## What Changed

- `parse.ts`: add `openCodeErrorStatusCodes`, which reads the HTTP
status from OpenCode error events.
- `test.ts`: treat a 401 status on the hello probe as
`opencode_hello_probe_auth_required`, in addition to the existing
wording match.
- `test.remote.test.ts`: regression tests with the two recorded
OpenRouter 401 outputs, and a negative control (500 and 429 stay generic
failures).

## Verification

- `pnpm exec vitest run
packages/adapters/opencode-local/src/server/test.remote.test.ts
packages/adapters/opencode-local/src/server/parse.test.ts` passes (28
tests).
- Red check: without the classifier change, the two new 401 tests fail.
- `npx tsc --noEmit -p packages/adapters/opencode-local` passes.
- I recorded the probe output from the pinned OpenCode 1.18.34 binary
against OpenRouter with an empty key and with an unknown key.

## Risks

- Low risk. Only a 401 status changes classification. A 401 means the
provider rejected the credential. Other statuses keep their current
result.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:44:09 -06:00

501 lines
20 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import type {
AdapterEnvironmentCheck,
AdapterEnvironmentTestContext,
AdapterEnvironmentTestResult,
} from "@paperclipai/adapter-utils";
import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
import {
asBoolean,
asNumber,
asString,
asStringArray,
parseObject,
ensurePathInEnv,
} from "@paperclipai/adapter-utils/server-utils";
import {
ensureAdapterExecutionTargetCommandResolvable,
maybeRunSandboxInstallCommand,
ensureAdapterExecutionTargetDirectory,
isAdapterExecutionTargetUnavailableError,
runAdapterExecutionTargetProcess,
describeAdapterExecutionTarget,
resolveAdapterExecutionTargetCwd,
prepareAdapterExecutionTargetRuntime,
overrideAdapterExecutionTargetRemoteCwd,
} from "@paperclipai/adapter-utils/execution-target";
import { discoverOpenCodeModels, ensureOpenCodeModelConfiguredAndAvailable, requireOpenCodeModelId } from "./models.js";
import { openCodeErrorStatusCodes, parseOpenCodeJsonl } from "./parse.js";
import { SANDBOX_INSTALL_COMMAND } from "../index.js";
import { prepareOpenCodeRuntimeConfig, prepareManagedOpenCodeRemoteHomes } from "./runtime-config.js";
function summarizeStatus(checks: AdapterEnvironmentCheck[]): AdapterEnvironmentTestResult["status"] {
if (checks.some((check) => check.level === "error")) return "fail";
if (checks.some((check) => check.level === "warn")) return "warn";
return "pass";
}
function firstNonEmptyLine(text: string): string {
return (
text
.split(/\r?\n/)
.map((line) => line.trim())
.find(Boolean) ?? ""
);
}
function summarizeProbeDetail(stdout: string, stderr: string, parsedError: string | null): string | null {
const raw = parsedError?.trim() || firstNonEmptyLine(stderr) || firstNonEmptyLine(stdout);
if (!raw) return null;
const clean = raw.replace(/\s+/g, " ").trim();
const max = 240;
return clean.length > max ? `${clean.slice(0, max - 1)}...` : clean;
}
function normalizeEnv(input: unknown): Record<string, string> {
if (typeof input !== "object" || input === null || Array.isArray(input)) return {};
const env: Record<string, string> = {};
for (const [key, value] of Object.entries(input as Record<string, unknown>)) {
if (typeof value === "string") env[key] = value;
}
return env;
}
const OPENCODE_AUTH_REQUIRED_RE =
/(?:auth(?:entication)?\s+required|api\s*key|invalid\s*api\s*key|not\s+logged\s+in|opencode\s+auth\s+login|free\s+usage\s+exceeded)/i;
export async function testEnvironment(
ctx: AdapterEnvironmentTestContext,
): Promise<AdapterEnvironmentTestResult> {
const checks: AdapterEnvironmentCheck[] = [];
const config = parseObject(ctx.config);
const command = asString(config.command, "opencode");
const target = ctx.executionTarget ?? null;
const targetIsRemote = target?.kind === "remote";
const targetIsSandbox = target?.kind === "remote" && target.transport === "sandbox";
const cwd = resolveAdapterExecutionTargetCwd(target, asString(config.cwd, ""), process.cwd());
const targetLabel = targetIsRemote
? ctx.environmentName ?? describeAdapterExecutionTarget(target)
: null;
const runId = `opencode-envtest-${Date.now()}-${Math.random().toString(16).slice(2)}`;
if (targetLabel) {
checks.push({
code: "opencode_environment_target",
level: "info",
message: `Probing inside environment: ${targetLabel}`,
});
}
try {
await ensureAdapterExecutionTargetDirectory(runId, target, cwd, {
cwd,
env: {},
createIfMissing: false,
});
checks.push({
code: "opencode_cwd_valid",
level: "info",
message: `Working directory is valid: ${cwd}`,
});
} catch (err) {
checks.push({
code: "opencode_cwd_invalid",
...(isAdapterExecutionTargetUnavailableError(err) ? { retryable: true } : {}),
level: "error",
message: err instanceof Error ? err.message : "Invalid working directory",
detail: cwd,
});
}
const envConfig = parseObject(config.env);
const env: Record<string, string> = {};
for (const [key, value] of Object.entries(envConfig)) {
if (typeof value === "string") env[key] = value;
}
const openaiKeyOverride = "OPENAI_API_KEY" in envConfig ? asString(envConfig.OPENAI_API_KEY, "") : null;
if (!config.managedAiConnection && openaiKeyOverride !== null && openaiKeyOverride.trim() === "") {
checks.push({
code: "opencode_openai_api_key_missing",
level: "warn",
message: "OPENAI_API_KEY override is empty.",
hint: "The OPENAI_API_KEY override is empty. Set a valid key or remove the override.",
});
}
// Prevent OpenCode from writing an opencode.json into the working directory.
env.OPENCODE_DISABLE_PROJECT_CONFIG = "true";
const preparedRuntimeConfig = await prepareOpenCodeRuntimeConfig({ env, config });
const localRuntimeConfigHome =
preparedRuntimeConfig.notes.length > 0 ? preparedRuntimeConfig.env.XDG_CONFIG_HOME : "";
if (asBoolean(config.dangerouslySkipPermissions, true)) {
checks.push({
code: "opencode_headless_permissions_enabled",
level: "info",
message: "Headless OpenCode permissions are auto-approved for all tools and connections.",
});
}
let restoreWorkspace: (() => Promise<void>) | null = null;
// Declared outside `try` so a failure inside `prepareAdapterExecutionTargetRuntime`
// still has the path available for cleanup in `finally` — otherwise the
// `fs.mkdtemp` directory leaks on the early-throw path.
let preparedRuntimeWorkspaceLocalDir: string | null = null;
let nativeProbeWorkspaceLocalDir: string | null = null;
let nativeProbeHomeLocalDir: string | null = null;
try {
let runtimeTarget: AdapterExecutionTarget | null = target ?? null;
let runtimeCwd = cwd;
// Native workspaces are assigned at execution time. A configuration probe
// without an explicit workspace must not run in the server's checkout.
if (!targetIsRemote && ctx.adapterType === "paperclip_runner" && !asString(config.cwd, "").trim()) {
nativeProbeWorkspaceLocalDir = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-opencode-native-probe-"));
runtimeCwd = nativeProbeWorkspaceLocalDir;
}
// The native runner gives every OpenCode process its own home and data
// directory. The probe must too: OpenCode keeps one SQLite database per
// data directory, and concurrent processes sharing the host user's
// database fail with "database is locked".
if (!targetIsRemote && ctx.adapterType === "paperclip_runner") {
nativeProbeHomeLocalDir = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-opencode-native-probe-home-"));
Object.assign(preparedRuntimeConfig.env, {
HOME: nativeProbeHomeLocalDir,
XDG_DATA_HOME: path.join(nativeProbeHomeLocalDir, "data"),
XDG_CACHE_HOME: path.join(nativeProbeHomeLocalDir, "cache"),
XDG_STATE_HOME: path.join(nativeProbeHomeLocalDir, "state"),
// Match the native driver: a fresh cache must not fetch a catalog or
// install default plugins before the pinned model answers.
OPENCODE_DISABLE_MODELS_FETCH: "true",
OPENCODE_DISABLE_DEFAULT_PLUGINS: "true",
});
}
if (targetIsRemote) {
preparedRuntimeWorkspaceLocalDir = await fs.mkdtemp(path.join(os.tmpdir(), `paperclip-opencode-envtest-${runId}-`));
const preparedExecutionTargetRuntime = await prepareAdapterExecutionTargetRuntime({
runId,
target,
adapterKey: "opencode",
workspaceLocalDir: preparedRuntimeWorkspaceLocalDir,
workspaceRemoteDir: cwd,
installCommand: SANDBOX_INSTALL_COMMAND,
detectCommand: command,
// The probe workspace is a throwaway temp dir; never wait on copy-back retries.
workspaceRestoreRetryDelaysMs: [],
assets: localRuntimeConfigHome
? [{
key: "xdgConfig",
localDir: localRuntimeConfigHome,
}]
: [],
});
restoreWorkspace = async () => {
await preparedExecutionTargetRuntime.restoreWorkspace().catch(() => {});
if (preparedRuntimeWorkspaceLocalDir) {
await fs.rm(preparedRuntimeWorkspaceLocalDir, { recursive: true, force: true }).catch(() => {});
}
};
runtimeCwd = preparedExecutionTargetRuntime.workspaceRemoteDir ?? runtimeCwd;
runtimeTarget = overrideAdapterExecutionTargetRemoteCwd(target ?? null, runtimeCwd) ?? null;
if (localRuntimeConfigHome && preparedExecutionTargetRuntime.assetDirs.xdgConfig) {
preparedRuntimeConfig.env.XDG_CONFIG_HOME = preparedExecutionTargetRuntime.assetDirs.xdgConfig;
}
prepareManagedOpenCodeRemoteHomes({
env: preparedRuntimeConfig.env,
config,
runtimeRootDir: preparedExecutionTargetRuntime.runtimeRootDir,
runId,
configDir: preparedExecutionTargetRuntime.assetDirs.xdgConfig,
});
}
const runtimeEnv = normalizeEnv(ensurePathInEnv({ ...process.env, ...preparedRuntimeConfig.env }));
const cwdInvalid = checks.some((check) => check.code === "opencode_cwd_invalid");
if (cwdInvalid) {
checks.push({
code: "opencode_command_skipped",
level: "warn",
message: "Skipped command check because working directory validation failed.",
detail: command,
});
} else {
const installCheck = await maybeRunSandboxInstallCommand({
runId,
target,
adapterKey: "opencode",
installCommand: SANDBOX_INSTALL_COMMAND,
detectCommand: command,
env,
});
if (installCheck) checks.push(installCheck);
try {
await ensureAdapterExecutionTargetCommandResolvable(command, runtimeTarget, runtimeCwd, runtimeEnv);
checks.push({
code: "opencode_command_resolvable",
level: "info",
message: `Command is executable: ${command}`,
});
} catch (err) {
checks.push({
code: "opencode_command_unresolvable",
...(isAdapterExecutionTargetUnavailableError(err) ? { retryable: true } : {}),
level: "error",
message: err instanceof Error ? err.message : "Command is not executable",
detail: command,
});
}
}
const canRunProbe =
checks.every((check) => check.code !== "opencode_cwd_invalid" && check.code !== "opencode_command_unresolvable");
let modelValidationPassed = false;
const configuredModel = asString(config.model, "").trim();
// The native runner already selects a concrete model. Its real hello probe
// verifies model access and credentials; a full catalog scan adds another
// cold CLI launch (and discovery retries) without verifying that request.
const nativeModelProbe = ctx.adapterType === "paperclip_runner" && Boolean(configuredModel);
// Model discovery and validation use local child processes against
// OpenCode's `models` subcommand and JSON config; these are not yet
// wired through the execution target. When probing a remote env, skip
// discovery/validation and rely on the remote hello probe to surface
// model/auth issues directly.
if (nativeModelProbe) {
try {
requireOpenCodeModelId(configuredModel);
modelValidationPassed = true;
checks.push({
code: "opencode_model_validation_via_probe",
level: "info",
message: "The configured native-runner model will be validated by the hello probe.",
});
} catch (error) {
checks.push({ code: "opencode_model_invalid", level: "error",
message: error instanceof Error ? error.message : "Configured model is invalid." });
}
} else if (targetIsRemote && configuredModel) {
checks.push({
code: "opencode_model_validation_skipped_remote",
level: "info",
message: `Skipped local model validation; will be validated by the hello probe inside ${targetLabel}.`,
});
modelValidationPassed = true;
} else if (canRunProbe && configuredModel) {
try {
const discovered = await discoverOpenCodeModels({ command, cwd, env: runtimeEnv });
if (discovered.length > 0) {
checks.push({
code: "opencode_models_discovered",
level: "info",
message: `Discovered ${discovered.length} model(s) from OpenCode providers.`,
});
} else {
checks.push({
code: "opencode_models_empty",
level: "error",
message: "OpenCode returned no models.",
hint: "Run `opencode models` and verify provider authentication.",
});
}
} catch (err) {
const errMsg = err instanceof Error ? err.message : String(err);
if (/ProviderModelNotFoundError/i.test(errMsg)) {
checks.push({
code: "opencode_hello_probe_model_unavailable",
level: "warn",
message: "The configured model was not found by the provider.",
detail: errMsg,
hint: "Run `opencode models` and choose an available provider/model ID.",
});
} else {
checks.push({
code: "opencode_models_discovery_failed",
level: "error",
message: errMsg || "OpenCode model discovery failed.",
hint: "Run `opencode models` manually to verify provider auth and config.",
});
}
}
} else if (!targetIsRemote && canRunProbe && !configuredModel) {
try {
const discovered = await discoverOpenCodeModels({ command, cwd, env: runtimeEnv });
if (discovered.length > 0) {
checks.push({
code: "opencode_models_discovered",
level: "info",
message: `Discovered ${discovered.length} model(s) from OpenCode providers.`,
});
}
} catch (err) {
const errMsg = err instanceof Error ? err.message : String(err);
if (/ProviderModelNotFoundError/i.test(errMsg)) {
checks.push({
code: "opencode_hello_probe_model_unavailable",
level: "warn",
message: "The configured model was not found by the provider.",
detail: errMsg,
hint: "Run `opencode models` and choose an available provider/model ID.",
});
} else {
checks.push({
code: "opencode_models_discovery_failed",
level: "warn",
message: errMsg || "OpenCode model discovery failed (best-effort, no model configured).",
hint: "Run `opencode models` manually to verify provider auth and config.",
});
}
}
}
const modelUnavailable = checks.some((check) => check.code === "opencode_hello_probe_model_unavailable");
if (!configuredModel && !modelUnavailable) {
// No model configured – skip model requirement if no model-related checks exist
} else if (!nativeModelProbe && !targetIsRemote && configuredModel && canRunProbe) {
try {
await ensureOpenCodeModelConfiguredAndAvailable({
model: configuredModel,
command,
cwd,
env: runtimeEnv,
});
checks.push({
code: "opencode_model_configured",
level: "info",
message: `Configured model: ${configuredModel}`,
});
modelValidationPassed = true;
} catch (err) {
checks.push({
code: "opencode_model_invalid",
level: "error",
message: err instanceof Error ? err.message : "Configured model is unavailable.",
hint: "Run `opencode models` and choose a currently available provider/model ID.",
});
}
}
if (canRunProbe && modelValidationPassed) {
const extraArgs = (() => {
const fromExtraArgs = asStringArray(config.extraArgs);
if (fromExtraArgs.length > 0) return fromExtraArgs;
return asStringArray(config.args);
})();
const variant = asString(config.variant, "").trim();
const probeModel = configuredModel;
const args = ["run", "--format", "json"];
args.push("--model", probeModel);
if (variant) args.push("--variant", variant);
if (extraArgs.length > 0) args.push(...extraArgs);
// Sandbox bridges still add cold-start and transport overhead, but the
// standard-2 Cloudflare tier now probes quickly enough that 90s keeps
// useful headroom without letting slow hangs linger.
const helloProbeTimeoutSec = Math.max(
1,
asNumber(config.helloProbeTimeoutSec, targetIsSandbox ? 90 : 60),
);
try {
const probe = await runAdapterExecutionTargetProcess(
runId,
runtimeTarget,
command,
args,
{
cwd: runtimeCwd,
env: runtimeEnv,
timeoutSec: helloProbeTimeoutSec,
graceSec: 5,
stdin: "Respond with hello.",
onLog: async () => {},
},
);
const parsed = parseOpenCodeJsonl(probe.stdout);
const detail = summarizeProbeDetail(probe.stdout, probe.stderr, parsed.errorMessage);
const authEvidence = `${parsed.errorMessage ?? ""}\n${probe.stdout}\n${probe.stderr}`.trim();
const providerRejectedAuth = openCodeErrorStatusCodes(probe.stdout).includes(401);
if (probe.timedOut) {
checks.push({
code: "opencode_hello_probe_timed_out",
level: "warn",
message: "OpenCode hello probe timed out.",
hint: "Retry the probe. If this persists, run OpenCode manually in this working directory.",
});
} else if ((probe.exitCode ?? 1) === 0 && !parsed.errorMessage) {
const summary = parsed.summary.trim();
const hasHello = /\bhello\b/i.test(summary);
checks.push({
code: hasHello ? "opencode_hello_probe_passed" : "opencode_hello_probe_unexpected_output",
level: hasHello ? "info" : "warn",
message: hasHello
? "OpenCode hello probe succeeded."
: "OpenCode probe ran but did not return `hello` as expected.",
...(summary ? { detail: summary.replace(/\s+/g, " ").trim().slice(0, 240) } : {}),
...(hasHello
? {}
: {
hint: "Run `opencode run --format json` manually and prompt `Respond with hello` to inspect output.",
}),
});
} else if (/ProviderModelNotFoundError/i.test(authEvidence)) {
checks.push({
code: "opencode_hello_probe_model_unavailable",
level: "warn",
message: "The configured model was not found by the provider.",
...(detail ? { detail } : {}),
hint: "Run `opencode models` and choose an available provider/model ID.",
});
} else if (providerRejectedAuth || OPENCODE_AUTH_REQUIRED_RE.test(authEvidence)) {
checks.push({
code: "opencode_hello_probe_auth_required",
level: "warn",
message: "OpenCode is installed, but provider authentication is not ready.",
...(detail ? { detail } : {}),
hint: "Run `opencode auth login` or set provider credentials, then retry the probe.",
});
} else {
checks.push({
code: "opencode_hello_probe_failed",
level: "error",
message: "OpenCode hello probe failed.",
...(detail ? { detail } : {}),
hint: "Run `opencode run --format json` manually in this working directory to debug.",
});
}
} catch (err) {
checks.push({
code: "opencode_hello_probe_failed",
level: "error",
message: "OpenCode hello probe failed.",
detail: err instanceof Error ? err.message : String(err),
hint: "Run `opencode run --format json` manually in this working directory to debug.",
});
}
}
} finally {
for (const dir of [nativeProbeWorkspaceLocalDir, nativeProbeHomeLocalDir]) {
if (dir) await fs.rm(dir, { recursive: true, force: true }).catch(() => {});
}
await restoreWorkspace?.();
if (!restoreWorkspace && preparedRuntimeWorkspaceLocalDir) {
// Reached when `prepareAdapterExecutionTargetRuntime` threw before
// assigning `restoreWorkspace`: clean up the temp dir directly.
await fs.rm(preparedRuntimeWorkspaceLocalDir, { recursive: true, force: true }).catch(() => {});
}
await preparedRuntimeConfig.cleanup();
}
return {
adapterType: ctx.adapterType,
status: summarizeStatus(checks),
checks,
testedAt: new Date().toISOString(),
};
}