mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 01:57:31 +01:00
## Thinking Path > - Paperclip lets users choose an agent harness and account. > - The server resolves the runner for new agents. > - Claude still needs the same ordinary native-default journey as Codex. > - Existing agents must retain their runner, model and account. > - This PR defaults new Apple Silicon Claude agents to Paperclip Runner with Opus 5.5. > - Users can connect and work without a runner override. ## Linked Issues or Issue Description Refs #15683. This PR depends on #15994 and precedes #15997. Related scope: #15422, #13978 and #15224. ## What Changed - Resolve new Apple Silicon Claude agents to `paperclip_runner`, `provider: acpx`, `acpxAgent: claude`. Keep explicit Legacy, models, provider routing and saved execution. - Use `claude-opus-5-5` for ordinary Claude defaults and retain explicit overrides. - Reuse existing API-key/subscription connections, credential storage and checks. Generalize the shared Advanced runner control with Claude branding. - Carry configured effort into the shared v9 contract. Preserve compatible permissions, instructions, skills, timeouts and session behavior. Require Legacy for unsupported settings. - Qualify a new immutable managed Claude target at explicit Connect through the existing permission-gated probe. Use fresh scoped discovery for test and hire. Discard stale results and require an explicit Connect after settings change. - Retain merged Codex onboarding recovery, auth preparation, import/effort corrections and Grok Bot gates. - Check the Claude first-task wizard's saved default directly. Update configuration/runtime documentation and exact Opus 5.5 accounting. - Align hire fixtures with the merged host-environment restriction. Native agent-authenticated hires cannot inject host env settings; both creation endpoints verify that no extra agent is saved. ## Verification - Updated onto master `9179d8dcc3e908c217fde4f63c9d08ccf8132eeb` through #15994. Native input v9 carries effort while preserving v7 home modes and v8 Muse. Codex defaults remain unchanged. - Focused local verification passed 285 server tests, 323 UI tests, 124 setup/registry tests, and two v9 permission-pin transport cases. Direct server/UI TypeScript checks passed. An initial 30-second hook timeout was resolved by serial verification with a longer hook budget: all 122 credential/permission cases and 232 snapshot/environment cases pass. Direct server TypeScript checking also passes. - Ordinary web creation with the experiment disabled saved native Claude, Opus 5.5 and High effort without runner/model overrides. The actual product task and follow-up both succeeded on integrated source `b2c9fe43f9cd3bc5dd9d22552da0148845ba71a4` with the cloud-built daemon from `04e0202ca81387b17cbe7fd094a8338648356fe9`. The final master reconciliation also changes shared runtime source. Historical proof retains its source identity; a final-source cloud qualification is running before refreshing the preview daemon. - Both actual browser downloads were checked: v1 has three sections/six checkboxes (451 bytes); v2 preserves the original body and adds two Rollback checks (612 bytes). The original attachment stayed byte-identical. The follow-up records session resume and the same provider session. Total estimated API usage: $0.518432800, with no pending/unpriced usage in the successful fixture. - Fixed the live-discovered setup incompatibility: native API-key validation no longer depends on the global Legacy Claude CLI. The existing provider endpoint verifies the key without a paid prompt; subscription/custom routing probes remain intact. - Fixed security review's native-hire credential redirection: agent-supplied URLs, proxy/launch settings and other host environment controls are rejected before inheritance. Native credential references remain supported and partial edits use the saved harness. Added create/hire/update regression coverage. - Current-head cloud CI is green. All three stack PRs have fresh Greptile 5/5 reviews and zero unresolved review threads. Required typecheck, test and build gates ran in cloud CI; Docker work stayed remote. Earlier failed checks remain in their original runs. - Docker and native release builds ran in cloud infrastructure. An inadvertent local Rust prerequisite was stopped; no local Docker build was run. - The final master merge preserves task-loading, permission, skill-retention and OpenRouter changes. Pi uses a fresh combined profile 27, preserving both historical profile-24 declarations; Copilot remains profile 22. Focused protocol checks passed 155 cases (one skipped), the updated Pi attestation check passed 18 (one skipped), and 91 evaluation/catalog checks passed. - Latest-application manual continuation passed in 37.942 seconds on `695e5c005a85bddc8711d3fcddcaf6b7e1fb64d0`, with High effort, native Opus 5.5, Anthropic API attribution, the same resumed Claude session, an independently checked 635-byte file and persisted Done status. It uses the earlier verified Mac daemon; the preceding-source package qualification passed at `dca62c7aeb4b45f0dd91a8e18d386a16a2658ef8`. That package run predates the small TypeScript startup fix and is not relabeled as qualification for it. - Upgrade limitation: changed compiled sidecar identity rejected old durable launch state and recovery hit session-open bounds before eventually resuming. That attempt was operator-cancelled and retained; the successful manual continuation does not qualify seamless upgrades. - Local latency profile, Opus 5.5/High unchanged: three alternating real-ACP loopback samples per variant reduce median provider-ready time from 5.081s to 4.034s. The matched real API task startup improves from 5.533s to 4.348s, but total reply time changes from 17.933s to 20.221s because agent work varies. This establishes a startup saving of about one second, not a consistent total-reply speedup. A real candidate follow-up passed in 20.146s and resumed the same saved session. The loopback fixture makes no external model requests; failed initial probes remain retained. - The startup change passes 107 focused runtime/effort/lease tests, 53 profile checks (one skipped), TypeScript checking and the TypeScript-only runtime build. Native hire fixture checks pass 70 tests; their first local run retained a socket-hangup failure and stale OpenCode expectation, both accounted for before the passing rerun. Broader required checks run in cloud CI. ## Risks - Missing packages, authentication rejection, unsupported settings and unavailable models must fail explicitly. Existing agents retain saved execution; explicit runner changes use existing revision/session invalidation. - Subscription follow-up remains unverified; remaining live acceptance uses API keys per human steering. Historical accounting holds and the separate $50 ceiling remain intact. - The subscription acceptance limit remains explicit for review. No Claude database migration is introduced. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, code execution and browser testing. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] Focused local tests and final cloud verification pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
696 lines
28 KiB
TypeScript
696 lines
28 KiB
TypeScript
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
|
import { chmod, mkdtemp, rm, writeFile } from "node:fs/promises";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
|
|
|
|
const {
|
|
ensureAdapterExecutionTargetDirectory,
|
|
ensureAdapterExecutionTargetCommandResolvable,
|
|
maybeRunSandboxInstallCommand,
|
|
runAdapterExecutionTargetProcess,
|
|
describeAdapterExecutionTarget,
|
|
resolveAdapterExecutionTargetCwd,
|
|
probeResult,
|
|
} = vi.hoisted(() => {
|
|
const probeResult: {
|
|
value: { exitCode: number; stdout: string; stderr: string };
|
|
throwError: Error | null;
|
|
} = {
|
|
value: { exitCode: 1, stdout: "", stderr: "" },
|
|
throwError: null,
|
|
};
|
|
return {
|
|
probeResult,
|
|
ensureAdapterExecutionTargetDirectory: vi.fn(async () => {}),
|
|
ensureAdapterExecutionTargetCommandResolvable: vi.fn(async () => {}),
|
|
maybeRunSandboxInstallCommand: vi.fn(async () => null),
|
|
runAdapterExecutionTargetProcess: vi.fn(async () => {
|
|
if (probeResult.throwError) throw probeResult.throwError;
|
|
return {
|
|
exitCode: probeResult.value.exitCode,
|
|
signal: null,
|
|
timedOut: false,
|
|
stdout: probeResult.value.stdout,
|
|
stderr: probeResult.value.stderr,
|
|
pid: 123,
|
|
startedAt: new Date().toISOString(),
|
|
};
|
|
}),
|
|
describeAdapterExecutionTarget: vi.fn(() => "Daytona"),
|
|
resolveAdapterExecutionTargetCwd: vi.fn(() => "/home/daytona/paperclip-workspace"),
|
|
};
|
|
});
|
|
|
|
vi.mock("@paperclipai/adapter-utils/execution-target", async () => {
|
|
const actual = await vi.importActual<typeof import("@paperclipai/adapter-utils/execution-target")>(
|
|
"@paperclipai/adapter-utils/execution-target",
|
|
);
|
|
return {
|
|
...actual,
|
|
ensureAdapterExecutionTargetDirectory,
|
|
ensureAdapterExecutionTargetCommandResolvable,
|
|
maybeRunSandboxInstallCommand,
|
|
runAdapterExecutionTargetProcess,
|
|
describeAdapterExecutionTarget,
|
|
resolveAdapterExecutionTargetCwd,
|
|
};
|
|
});
|
|
|
|
import { testEnvironment } from "./test.js";
|
|
|
|
const sandboxTarget: AdapterExecutionTarget = {
|
|
kind: "remote",
|
|
transport: "sandbox",
|
|
providerKey: "daytona",
|
|
remoteCwd: "/home/daytona/paperclip-workspace",
|
|
runner: {
|
|
execute: async () => ({
|
|
exitCode: 0,
|
|
signal: null,
|
|
timedOut: false,
|
|
stdout: "",
|
|
stderr: "",
|
|
pid: null,
|
|
startedAt: new Date().toISOString(),
|
|
}),
|
|
},
|
|
};
|
|
|
|
const initLine =
|
|
'{"type":"system","subtype":"init","cwd":"/home/daytona/paperclip-workspace","session_id":"abc","tools":["Bash","Read"]}';
|
|
|
|
afterEach(() => {
|
|
vi.clearAllMocks();
|
|
probeResult.throwError = null;
|
|
});
|
|
|
|
describe("claude sandbox hello probe diagnostics", () => {
|
|
it("keeps the raw failure result out of every check and out of the log", async () => {
|
|
// The non-zero result event carries a marker. The check must not repeat the
|
|
// marker, and the log must carry only the allowlisted classification.
|
|
const marker = "NONPATTERNMARKERfailure";
|
|
probeResult.value = {
|
|
exitCode: 1,
|
|
stdout: [
|
|
initLine,
|
|
`{"type":"result","subtype":"error_during_execution","is_error":true,"result":"API Error: 404 model not found ${marker}","session_id":"abc"}`,
|
|
].join("\n"),
|
|
stderr: "",
|
|
};
|
|
const warnSpy = vi.spyOn(console, "warn").mockImplementation(() => {});
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "claude_local",
|
|
config: { engine: "cli", command: "claude", model: "claude-opus-4-8" },
|
|
executionTarget: sandboxTarget,
|
|
environmentName: "Daytona",
|
|
});
|
|
|
|
expect(result.status).toBe("fail");
|
|
const failed = result.checks.find((check) => check.code === "claude_hello_probe_failed");
|
|
expect(failed).toBeTruthy();
|
|
// The public check carries only a fixed message and hint, no raw detail.
|
|
expect(failed?.detail).toBeUndefined();
|
|
const checkText = JSON.stringify(result.checks);
|
|
expect(checkText).not.toContain(marker);
|
|
// The unhelpful init line must never reach a check either.
|
|
expect(checkText).not.toContain('"subtype":"init"');
|
|
// The raw diagnostic never reaches the server log. The log carries only the
|
|
// fixed context and an allowlisted classification.
|
|
const loggedText = JSON.stringify(warnSpy.mock.calls);
|
|
expect(loggedText).not.toContain(marker);
|
|
expect(loggedText).toContain("nonzero_exit");
|
|
warnSpy.mockRestore();
|
|
});
|
|
|
|
it("keeps a stdout-fallback failure line out of every check and out of the log", async () => {
|
|
// The CLI dies before a result event, so the last non-init stdout line is
|
|
// the diagnostic. The check and the log must not repeat its marker.
|
|
const marker = "NONPATTERNMARKERstdout";
|
|
probeResult.value = {
|
|
exitCode: 1,
|
|
stdout: [initLine, `fatal: claude crashed ${marker}`].join("\n"),
|
|
stderr: "",
|
|
};
|
|
const warnSpy = vi.spyOn(console, "warn").mockImplementation(() => {});
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "claude_local",
|
|
config: { model: "claude-opus-5", engine: "cli", command: "claude" },
|
|
executionTarget: sandboxTarget,
|
|
environmentName: "Daytona",
|
|
});
|
|
|
|
const failed = result.checks.find((check) => check.code === "claude_hello_probe_failed");
|
|
expect(failed).toBeTruthy();
|
|
expect(failed?.detail).toBeUndefined();
|
|
const checkText = JSON.stringify(result.checks);
|
|
expect(checkText).not.toContain(marker);
|
|
expect(checkText).not.toContain('"subtype":"init"');
|
|
const loggedText = JSON.stringify(warnSpy.mock.calls);
|
|
expect(loggedText).not.toContain(marker);
|
|
expect(loggedText).toContain("nonzero_exit");
|
|
warnSpy.mockRestore();
|
|
});
|
|
|
|
it("never copies a credential-bearing stderr failure line into a check or the log", async () => {
|
|
// A verbose CLI can print a credential to stderr on failure. The check and
|
|
// the log must not repeat it.
|
|
const secret = "sk-ant-STDERRLEAK0123456789abcdef";
|
|
probeResult.value = {
|
|
exitCode: 1,
|
|
stdout: initLine,
|
|
stderr: `fatal: request failed with token ${secret}`,
|
|
};
|
|
const warnSpy = vi.spyOn(console, "warn").mockImplementation(() => {});
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "claude_local",
|
|
config: { model: "claude-opus-5", engine: "cli", command: "claude" },
|
|
executionTarget: sandboxTarget,
|
|
environmentName: "Daytona",
|
|
});
|
|
|
|
const checkText = JSON.stringify(result.checks);
|
|
expect(checkText).not.toContain(secret);
|
|
expect(checkText).not.toContain("STDERRLEAK");
|
|
const loggedText = JSON.stringify(warnSpy.mock.calls);
|
|
expect(loggedText).not.toContain(secret);
|
|
expect(loggedText).not.toContain("STDERRLEAK");
|
|
expect(loggedText).toContain("nonzero_exit");
|
|
warnSpy.mockRestore();
|
|
});
|
|
|
|
it("keeps an auth-required probe marker out of every check", async () => {
|
|
// The auth-required stdout and stderr carry a marker. The login-required
|
|
// checks must not repeat it, and the login gate code must stay stable.
|
|
const marker = "NONPATTERNMARKERauthcli";
|
|
probeResult.value = {
|
|
exitCode: 1,
|
|
stdout: [
|
|
initLine,
|
|
`{"type":"result","subtype":"error_during_execution","is_error":true,"result":"Please run \`claude login\` ${marker}","session_id":"abc"}`,
|
|
].join("\n"),
|
|
stderr: `Please run claude login ${marker}`,
|
|
};
|
|
const warnSpy = vi.spyOn(console, "warn").mockImplementation(() => {});
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "claude_local",
|
|
config: { model: "claude-opus-5", engine: "cli", command: "claude" },
|
|
executionTarget: sandboxTarget,
|
|
environmentName: "Daytona",
|
|
});
|
|
|
|
expect(result.checks.some((check) => check.code === "claude_hello_probe_auth_required")).toBe(true);
|
|
// The login gate code stays stable so the user interface can offer login.
|
|
expect(result.checks.some((check) => check.code === "adapter_auth_missing")).toBe(true);
|
|
const checkText = JSON.stringify(result.checks);
|
|
expect(checkText).not.toContain(marker);
|
|
const loggedText = JSON.stringify(warnSpy.mock.calls);
|
|
expect(loggedText).not.toContain(marker);
|
|
expect(loggedText).toContain("auth_required");
|
|
warnSpy.mockRestore();
|
|
});
|
|
|
|
it("classifies an invalid or expired token as adapter_auth_missing without leaking the token", async () => {
|
|
// Grounded on the real Claude CLI output for CLAUDE_CODE_OAUTH_TOKEN=invalid.
|
|
// The probe exits non-zero and the result event reports a 401 authentication
|
|
// failure with an "Invalid bearer token" message. A synthetic bearer marker
|
|
// rides along on a retry line, so the test proves the raw text never reaches
|
|
// a check.
|
|
const marker = "SUPERSECRETbearerMARKERcli";
|
|
probeResult.value = {
|
|
exitCode: 1,
|
|
stdout: [
|
|
initLine,
|
|
`{"type":"system","subtype":"api_retry","attempt":1,"error_status":401,"error":"authentication_failed: bearer ${marker} is invalid","session_id":"abc"}`,
|
|
'{"type":"result","subtype":"success","is_error":true,"api_error_status":401,"error":"authentication_failed","result":"Failed to authenticate. API Error: 401 Invalid bearer token","session_id":"abc"}',
|
|
].join("\n"),
|
|
stderr: "",
|
|
};
|
|
const warnSpy = vi.spyOn(console, "warn").mockImplementation(() => {});
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "claude_local",
|
|
config: { model: "claude-opus-5", engine: "cli", command: "claude" },
|
|
executionTarget: sandboxTarget,
|
|
environmentName: "Daytona",
|
|
});
|
|
|
|
// An auth failure returns the canonical login gate code, so the user
|
|
// interface can offer login.
|
|
expect(result.checks.some((check) => check.code === "claude_hello_probe_auth_required")).toBe(true);
|
|
expect(result.checks.some((check) => check.code === "adapter_auth_missing")).toBe(true);
|
|
// The raw probe text, including the bearer marker, never reaches a check.
|
|
expect(JSON.stringify(result.checks)).not.toContain(marker);
|
|
warnSpy.mockRestore();
|
|
});
|
|
|
|
it("does not flag a healthy probe whose assistant text repeats a token phrase", async () => {
|
|
// A healthy run prints an auth phrase in its answer text. The parsed result
|
|
// is a success, so the probe stays healthy and offers no login gate.
|
|
probeResult.value = {
|
|
exitCode: 0,
|
|
stdout: [
|
|
initLine,
|
|
'{"type":"assistant","message":{"role":"assistant","content":[{"type":"text","text":"hello — authentication_failed means an invalid bearer token"}]},"session_id":"abc"}',
|
|
'{"type":"result","subtype":"success","is_error":false,"result":"hello","session_id":"abc"}',
|
|
].join("\n"),
|
|
stderr: "",
|
|
};
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "claude_local",
|
|
config: { model: "claude-opus-5", engine: "cli", command: "claude" },
|
|
executionTarget: sandboxTarget,
|
|
environmentName: "Daytona",
|
|
});
|
|
|
|
expect(result.checks.some((check) => check.code === "adapter_auth_missing")).toBe(false);
|
|
expect(result.checks.some((check) => check.code === "claude_hello_probe_auth_required")).toBe(false);
|
|
expect(result.checks.some((check) => check.code === "claude_hello_probe_passed")).toBe(true);
|
|
});
|
|
|
|
it("keeps a transient failure with an assistant token phrase off the login gate", async () => {
|
|
// The probe fails on a 529 overload. The auth phrase appears only in the raw
|
|
// stdout assistant event, not the parsed result, so the run stays transient
|
|
// and never surfaces the login gate.
|
|
probeResult.value = {
|
|
exitCode: 1,
|
|
stdout: [
|
|
initLine,
|
|
'{"type":"assistant","message":{"role":"assistant","content":[{"type":"text","text":"authentication_failed: the bearer token is invalid"}]},"session_id":"abc"}',
|
|
'{"type":"result","subtype":"error_during_execution","is_error":true,"result":"API Error: 529 overloaded_error","session_id":"abc"}',
|
|
].join("\n"),
|
|
stderr: "",
|
|
};
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "claude_local",
|
|
config: { model: "claude-opus-5", engine: "cli", command: "claude" },
|
|
executionTarget: sandboxTarget,
|
|
environmentName: "Daytona",
|
|
});
|
|
|
|
expect(result.checks.some((check) => check.code === "adapter_auth_missing")).toBe(false);
|
|
expect(result.checks.some((check) => check.code === "claude_hello_probe_auth_required")).toBe(false);
|
|
expect(result.checks.some((check) => check.code === "claude_hello_probe_transient_upstream")).toBe(true);
|
|
});
|
|
|
|
it("keeps an unexpected successful summary out of every check", async () => {
|
|
// The probe exits 0 but does not return `hello`. The unexpected summary is
|
|
// untrusted output. The check must not repeat its marker.
|
|
const marker = "NONPATTERNMARKERunexpected";
|
|
probeResult.value = {
|
|
exitCode: 0,
|
|
stdout: [
|
|
initLine,
|
|
`{"type":"result","subtype":"success","is_error":false,"result":"Here is something else ${marker}","session_id":"abc"}`,
|
|
].join("\n"),
|
|
stderr: "",
|
|
};
|
|
const warnSpy = vi.spyOn(console, "warn").mockImplementation(() => {});
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "claude_local",
|
|
config: { model: "claude-opus-5", engine: "cli", command: "claude" },
|
|
executionTarget: sandboxTarget,
|
|
environmentName: "Daytona",
|
|
});
|
|
|
|
const unexpected = result.checks.find(
|
|
(check) => check.code === "claude_hello_probe_unexpected_output",
|
|
);
|
|
expect(unexpected).toBeTruthy();
|
|
expect(unexpected?.detail).toBeUndefined();
|
|
const checkText = JSON.stringify(result.checks);
|
|
expect(checkText).not.toContain(marker);
|
|
const loggedText = JSON.stringify(warnSpy.mock.calls);
|
|
expect(loggedText).not.toContain(marker);
|
|
expect(loggedText).toContain("unexpected_output");
|
|
warnSpy.mockRestore();
|
|
});
|
|
|
|
it("keeps an opaque credential marker and a proxy marker out of every check and the log", async () => {
|
|
// The failure output carries two untrusted values that the pattern
|
|
// sanitizer did not recognize: an opaque credential with no token shape and
|
|
// a proxy URL. One rides in stdout, the other in stderr. Neither may reach a
|
|
// check or the server log.
|
|
const opaqueCredMarker = "OPAQUECREDMARKERnoshape";
|
|
const proxyMarker = "http://user:pass@proxy.corp.internal:3128";
|
|
probeResult.value = {
|
|
exitCode: 7,
|
|
stdout: [
|
|
initLine,
|
|
`{"type":"result","subtype":"error_during_execution","is_error":true,"result":"probe failed with ${opaqueCredMarker}","session_id":"abc"}`,
|
|
].join("\n"),
|
|
stderr: `proxy connect failed: ${proxyMarker}`,
|
|
};
|
|
const warnSpy = vi.spyOn(console, "warn").mockImplementation(() => {});
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "claude_local",
|
|
config: { model: "claude-opus-5", engine: "cli", command: "claude" },
|
|
executionTarget: sandboxTarget,
|
|
environmentName: "Daytona",
|
|
});
|
|
|
|
const checkText = JSON.stringify(result.checks);
|
|
expect(checkText).not.toContain(opaqueCredMarker);
|
|
expect(checkText).not.toContain("proxy.corp.internal");
|
|
const loggedText = JSON.stringify(warnSpy.mock.calls);
|
|
expect(loggedText).not.toContain(opaqueCredMarker);
|
|
expect(loggedText).not.toContain("proxy.corp.internal");
|
|
// The log still carries the allowlisted classification and the safe exit
|
|
// code, so the diagnostic stays useful.
|
|
expect(loggedText).toContain("nonzero_exit");
|
|
expect(warnSpy.mock.calls[0]?.[1]).toMatchObject({
|
|
classification: "nonzero_exit",
|
|
exitCode: 7,
|
|
});
|
|
warnSpy.mockRestore();
|
|
});
|
|
|
|
it("never copies a thrown CLI probe error into a check or the log", async () => {
|
|
// A spawn or transport failure can throw an error whose text carries a
|
|
// credential. Inject an opaque credential marker and a proxy marker through
|
|
// the thrown error. The CLI lane has no catch around the hello probe call,
|
|
// so the thrown error propagates to the caller, which owns it. No check and
|
|
// no console.warn call inside this lane repeats either marker.
|
|
const opaqueCredMarker = "OPAQUECREDMARKERnoshape";
|
|
const proxyMarker = "http://user:pass@proxy.corp.internal:3128";
|
|
probeResult.throwError = new Error(
|
|
`probe spawn failed with ${opaqueCredMarker} via ${proxyMarker}`,
|
|
);
|
|
const warnSpy = vi.spyOn(console, "warn").mockImplementation(() => {});
|
|
|
|
// The current contract propagates the thrown error. The lane builds no
|
|
// Test-result check from the error, so the raw text cannot reach a check.
|
|
await expect(
|
|
testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "claude_local",
|
|
config: { model: "claude-opus-5", engine: "cli", command: "claude" },
|
|
executionTarget: sandboxTarget,
|
|
environmentName: "Daytona",
|
|
}),
|
|
).rejects.toThrow(opaqueCredMarker);
|
|
|
|
// The lane never routes the raw error text to the server log. No
|
|
// console.warn call repeats either marker.
|
|
const loggedText = JSON.stringify(warnSpy.mock.calls);
|
|
expect(loggedText).not.toContain(opaqueCredMarker);
|
|
expect(loggedText).not.toContain("proxy.corp.internal");
|
|
warnSpy.mockRestore();
|
|
});
|
|
|
|
it("classifies subscription usage-limit failures as a usage-limited warning, not a hard fail", async () => {
|
|
probeResult.value = {
|
|
exitCode: 1,
|
|
stdout: [
|
|
initLine,
|
|
'{"type":"result","subtype":"error_during_execution","is_error":true,"result":"Claude usage limit reached. Please try again later.","session_id":"abc"}',
|
|
].join("\n"),
|
|
stderr: "",
|
|
};
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "claude_local",
|
|
config: { model: "claude-opus-5", engine: "cli", command: "claude" },
|
|
executionTarget: sandboxTarget,
|
|
environmentName: "Daytona",
|
|
});
|
|
|
|
expect(result.checks.some((check) => check.code === "claude_hello_probe_usage_limited")).toBe(true);
|
|
expect(result.checks.some((check) => check.code === "claude_hello_probe_transient_upstream")).toBe(false);
|
|
expect(result.checks.some((check) => check.code === "claude_hello_probe_failed")).toBe(false);
|
|
});
|
|
|
|
it("classifies overload failures as a transient warning, not a hard fail", async () => {
|
|
probeResult.value = {
|
|
exitCode: 1,
|
|
stdout: [
|
|
initLine,
|
|
'{"type":"result","subtype":"error_during_execution","is_error":true,"result":"API Error: 529 overloaded_error","session_id":"abc"}',
|
|
].join("\n"),
|
|
stderr: "",
|
|
};
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "claude_local",
|
|
config: { model: "claude-opus-5", engine: "cli", command: "claude" },
|
|
executionTarget: sandboxTarget,
|
|
environmentName: "Daytona",
|
|
});
|
|
|
|
expect(result.checks.some((check) => check.code === "claude_hello_probe_transient_upstream")).toBe(true);
|
|
expect(result.checks.some((check) => check.code === "claude_hello_probe_usage_limited")).toBe(false);
|
|
expect(result.checks.some((check) => check.code === "claude_hello_probe_failed")).toBe(false);
|
|
});
|
|
|
|
it("keeps the failed check free of a detail when only the system/init line is present", async () => {
|
|
probeResult.value = {
|
|
exitCode: 1,
|
|
stdout: initLine,
|
|
stderr: "",
|
|
};
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "claude_local",
|
|
config: { model: "claude-opus-5", engine: "cli", command: "claude" },
|
|
executionTarget: sandboxTarget,
|
|
environmentName: "Daytona",
|
|
});
|
|
|
|
const failed = result.checks.find((check) => check.code === "claude_hello_probe_failed");
|
|
expect(failed?.detail).toBeUndefined();
|
|
expect(JSON.stringify(result.checks)).not.toContain('"subtype":"init"');
|
|
});
|
|
});
|
|
|
|
describe("claude auth mode hints", () => {
|
|
const successStdout = [
|
|
initLine,
|
|
'{"type":"result","subtype":"success","is_error":false,"result":"hello","session_id":"abc"}',
|
|
].join("\n");
|
|
|
|
it("reports the configured subscription token for remote targets", async () => {
|
|
probeResult.value = { exitCode: 0, stdout: successStdout, stderr: "" };
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "claude_local",
|
|
config: { model: "claude-opus-5",
|
|
engine: "cli",
|
|
command: "claude",
|
|
env: { CLAUDE_CODE_OAUTH_TOKEN: "oauth-test-token" },
|
|
},
|
|
executionTarget: sandboxTarget,
|
|
environmentName: "Daytona",
|
|
});
|
|
|
|
const hint = result.checks.find((check) => check.code === "claude_oauth_token_configured");
|
|
expect(hint).toBeTruthy();
|
|
expect(hint?.level).toBe("info");
|
|
expect(hint?.detail).toContain("configured environment variables");
|
|
expect(
|
|
result.checks.some((check) => check.code === "claude_anthropic_api_key_overrides_subscription"),
|
|
).toBe(false);
|
|
});
|
|
|
|
it("reports an intentionally selected managed API account without a subscription warning", async () => {
|
|
probeResult.value = { exitCode: 0, stdout: successStdout, stderr: "" };
|
|
const result = await testEnvironment({ companyId: "company-1", adapterType: "claude_local",
|
|
config: { model: "claude-opus-5", engine: "cli", command: "claude", managedAiConnection: { provider: "anthropic", method: "api_key" }, env: { ANTHROPIC_API_KEY: "api-test-key" } },
|
|
executionTarget: sandboxTarget, environmentName: "Daytona",
|
|
});
|
|
expect(result.checks.find(check => check.code === "claude_anthropic_api_key_overrides_subscription")).toMatchObject({ level: "info", message: "Using the selected Claude API connection." });
|
|
expect(JSON.stringify(result.checks)).not.toContain("Unset ANTHROPIC_API_KEY");
|
|
});
|
|
|
|
it("keeps the API-key warning authoritative when both ANTHROPIC_API_KEY and the token are set", async () => {
|
|
probeResult.value = { exitCode: 0, stdout: successStdout, stderr: "" };
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "claude_local",
|
|
config: { model: "claude-opus-5",
|
|
engine: "cli",
|
|
command: "claude",
|
|
env: {
|
|
ANTHROPIC_API_KEY: "api-test-key",
|
|
CLAUDE_CODE_OAUTH_TOKEN: "oauth-test-token",
|
|
},
|
|
},
|
|
executionTarget: sandboxTarget,
|
|
environmentName: "Daytona",
|
|
});
|
|
|
|
expect(
|
|
result.checks.some((check) => check.code === "claude_anthropic_api_key_overrides_subscription"),
|
|
).toBe(true);
|
|
expect(result.checks.some((check) => check.code === "claude_oauth_token_configured")).toBe(false);
|
|
});
|
|
});
|
|
|
|
describe("claude CLI local hello probe hardening", () => {
|
|
// Clear the host proxy and host auth variables so a local probe reads a
|
|
// deterministic env regardless of the machine that runs the suite.
|
|
const CLEARED_HOST_ENV_KEYS = [
|
|
"HTTP_PROXY",
|
|
"HTTPS_PROXY",
|
|
"NO_PROXY",
|
|
"http_proxy",
|
|
"https_proxy",
|
|
"no_proxy",
|
|
"ANTHROPIC_API_KEY",
|
|
"CLAUDE_CODE_OAUTH_TOKEN",
|
|
"CLAUDE_CODE_USE_BEDROCK",
|
|
"ANTHROPIC_BEDROCK_BASE_URL",
|
|
];
|
|
const successStdout = [
|
|
initLine,
|
|
'{"type":"result","subtype":"success","is_error":false,"result":"hello","session_id":"abc"}',
|
|
].join("\n");
|
|
|
|
let tempDir: string | null = null;
|
|
let claudePath = "";
|
|
let savedPath: string | undefined;
|
|
let savedEnv: Record<string, string | undefined> = {};
|
|
|
|
beforeEach(async () => {
|
|
tempDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-cli-localprobe-"));
|
|
claudePath = path.join(tempDir, "claude");
|
|
await writeFile(claudePath, "#!/bin/sh\nexit 0\n");
|
|
await chmod(claudePath, 0o755);
|
|
savedPath = process.env.PATH;
|
|
process.env.PATH = tempDir;
|
|
savedEnv = {};
|
|
for (const key of CLEARED_HOST_ENV_KEYS) {
|
|
savedEnv[key] = process.env[key];
|
|
delete process.env[key];
|
|
}
|
|
// The mocked cwd resolver returns a sandbox path; the local probe reads it
|
|
// as the cwd, so no host directory is touched.
|
|
resolveAdapterExecutionTargetCwd.mockReturnValue("/home/daytona/paperclip-workspace");
|
|
});
|
|
|
|
afterEach(async () => {
|
|
process.env.PATH = savedPath;
|
|
for (const [key, value] of Object.entries(savedEnv)) {
|
|
if (value === undefined) delete process.env[key];
|
|
else process.env[key] = value;
|
|
}
|
|
if (tempDir) await rm(tempDir, { recursive: true, force: true }).catch(() => undefined);
|
|
tempDir = null;
|
|
});
|
|
|
|
it("spawns the trusted resolved claude and drops hostile caller env for a local probe", async () => {
|
|
probeResult.value = { exitCode: 0, stdout: successStdout, stderr: "" };
|
|
process.env.HTTPS_PROXY = "http://trusted-proxy:8443";
|
|
|
|
await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "claude_local",
|
|
config: { model: "claude-opus-5",
|
|
engine: "cli",
|
|
command: "/tmp/evil/claude",
|
|
env: {
|
|
ANTHROPIC_API_KEY: "keep-this-key",
|
|
NODE_OPTIONS: "--require /hostile/evil.js",
|
|
PATH: "/hostile/bin",
|
|
LD_PRELOAD: "/hostile/evil.so",
|
|
HTTP_PROXY: "http://caller-proxy:8080",
|
|
},
|
|
},
|
|
executionTarget: null,
|
|
environmentName: null,
|
|
});
|
|
|
|
expect(runAdapterExecutionTargetProcess).toHaveBeenCalledTimes(1);
|
|
const call = runAdapterExecutionTargetProcess.mock.calls[0] as unknown as unknown[];
|
|
const spawnedCommand = call[2] as string;
|
|
const spawnedEnv = (call[4] as { env: Record<string, string> }).env;
|
|
// The trusted resolved claude executable, never the caller command path.
|
|
expect(spawnedCommand).toBe(claudePath);
|
|
expect(spawnedCommand).not.toContain("/tmp/evil");
|
|
// The approved key reaches the child; the hostile keys never do.
|
|
expect(spawnedEnv.ANTHROPIC_API_KEY).toBe("keep-this-key");
|
|
expect(spawnedEnv.NODE_OPTIONS).toBeUndefined();
|
|
expect(spawnedEnv.PATH).toBeUndefined();
|
|
expect(spawnedEnv.LD_PRELOAD).toBeUndefined();
|
|
expect(spawnedEnv.HTTP_PROXY).toBeUndefined();
|
|
// The trusted proxy reaches the child; the caller proxy never does.
|
|
expect(spawnedEnv.HTTPS_PROXY).toBe("http://trusted-proxy:8443");
|
|
expect(JSON.stringify(spawnedEnv)).not.toContain("caller-proxy");
|
|
});
|
|
|
|
it.each([
|
|
["claude-fable-5-1", "2.1.251"],
|
|
["claude-opus-5-5", "2.1.280"],
|
|
["claude-sonnet-5-5", "2.1.284"],
|
|
])("warns without executing %s when runtime PATH selects a different executable", async (model, minimumVersion) => {
|
|
const runtimeDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-cli-runtime-path-"));
|
|
const runtimeClaudePath = path.join(runtimeDir, "claude");
|
|
await writeFile(runtimeClaudePath, "#!/bin/sh\nexit 0\n");
|
|
await chmod(runtimeClaudePath, 0o755);
|
|
|
|
try {
|
|
probeResult.value = { exitCode: 0, stdout: `${minimumVersion} (Claude Code)\n`, stderr: "" };
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "claude_local",
|
|
config: {
|
|
engine: "cli",
|
|
command: "claude",
|
|
model,
|
|
env: { PATH: runtimeDir },
|
|
},
|
|
executionTarget: null,
|
|
environmentName: null,
|
|
});
|
|
|
|
expect(result.status).toBe("warn");
|
|
expect(result.checks).toContainEqual(expect.objectContaining({
|
|
code: "claude_cli_version_probe_mismatch",
|
|
hint: `Ensure the runtime-selected Claude Code is ${minimumVersion} or newer. Execution will verify that exact executable before launch.`,
|
|
level: "warn",
|
|
}));
|
|
expect(runAdapterExecutionTargetProcess).not.toHaveBeenCalled();
|
|
} finally {
|
|
await rm(runtimeDir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it("names the local host target on every result", async () => {
|
|
probeResult.value = { exitCode: 0, stdout: successStdout, stderr: "" };
|
|
|
|
const result = await testEnvironment({
|
|
companyId: "company-1",
|
|
adapterType: "claude_local",
|
|
config: { model: "claude-opus-5", engine: "cli", command: "claude" },
|
|
executionTarget: null,
|
|
environmentName: null,
|
|
});
|
|
|
|
const targetCheck = result.checks.find((check) => check.code === "claude_environment_target");
|
|
expect(targetCheck).toBeTruthy();
|
|
expect(targetCheck?.message).toContain("Paperclip host");
|
|
});
|
|
});
|