mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 10:40:33 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The cost ledger records usage and controls agent budgets. > - Some runs lose the selected model or API billing category before their usage reaches the ledger. > - The dashboard reports missing costs but does not identify the affected runs or the available actions. > - This pull request preserves usage attribution and adds a compact view of cost gaps grouped by agent. > - Operators can inspect runs, open agent settings, and record saved costs without starting new agent work. ## Linked Issues or Issue Description Refs #15634, #11938, #15879. This follows the merged subscription reporting in #15634. The older proposal in #11938 addresses part of ACP model attribution. This change also rejects conflicting models across a turn, fixes credential-backed billing capture, and explains cost gaps in the existing dashboard. The Pi receipt recovery work in #15879 covers a separate adapter recovery path. This PR preserves unresolved receipts and does not recover provider usage or reprice past entries. **What happened?** ACP results use the requested model even when the runtime selects a model. Native and OpenCode API runs can report unknown billing despite a matching API credential. The dashboard shows notices without links to the affected runs. These gaps make API and subscription totals harder to interpret. **Expected behavior** Record the model selected by the runtime. Identify API billing only from a matching credential. Keep costs unresolved when reliable usage or pricing is absent. Show the affected agents and runs. Offer an accounting retry only when saved evidence is ready. **Steps to reproduce** 1. Run an ACP agent with its model left unset, or use a matching API credential with a native or OpenCode agent. 2. Inspect the recorded model and billing category for the run. 3. Open Costs Overview for a period with unpriced usage or terminal runs awaiting receipts. 4. Inspect the notice and try to find the affected run or the relevant agent settings. **Paperclip version or commit** Current master at `657e71584a0`. The patch is rebased on that commit. **Deployment mode** Self-hosted server and local development. The cost detail tests use a disposable PostgreSQL database. ## What Changed - Preserve the runtime-selected ACP model in checkpoints and final results. Conflicting models across one turn remain unpriced. - Share credential-backed API billing checks across native and OpenCode results. Keep unknown billing when the provider and credential do not match. - Add company-scoped cost detail and accounting retry endpoints. Board viewers can read details but cannot retry. Agents and foreign companies are denied. - Limit each detail category to 25 entries. Use one consistent database snapshot and sparse indexes matching the filters and sort order. Add the unknown-billing count to the existing summary query. - Replace the notice with an expandable count line. Group entries alphabetically by agent. Show the saved blob icon, link each name to configuration, and put run and retry links beside the date. - Preserve estimate and unavailable-price labels on Providers and Billers; keep retry confirmation after the resolved notice disappears. - Fetch details only while expanded. Retain the last loaded details after a refresh error. Retry saved accounting only; do not rerun the agent. - Add capture, database, authorization, UI, and Storybook coverage. Update the implementation contract and cost reporting guide. ## Verification - `pnpm -r typecheck` passed after rebase. - `pnpm build` and `pnpm build-storybook` passed after rebase. - `pnpm check:token-gates` and `git diff --check` passed. - 1,086 focused tests passed across 17 files. These cover ACP, native and OpenCode capture, PostgreSQL reports, route authorization, OpenAPI, Costs UI, and existing accounting regressions. - Desktop and mobile browser checks passed during UI review. They cover lazy expansion, alphabetical groups, loaded blob icons, configuration and run links, keyboard retry, and layout overflow. - `pnpm test:run` was attempted with command-scoped `init.defaultBranch=master` for Git fixtures. It was stopped after a native session recovery failure reproduced on unchanged base `657e71584a0` (`archives a damaged prior epoch before completing guarded same-task replacement with real runnerd`). An MCP upload case failed in the broad run but passed in isolation on both the base and PR branch. These local limitations do not affect the 1,086 passing focused tests. - Current-head CI is green, including all unit test lanes, serialized server suites, runner checks, typecheck, build, policy, and eight browser shards. Apex re-review is 5/5 on `56de60effdb`; all four original findings are addressed and every review thread is resolved. - PostgreSQL applies migration `0333` in the integration fixtures. EXPLAIN ANALYZE on the production detail statements uses the new sparse indexes behind 5,000 healthy records in each table. - Ten provider/biller card cases cover missing and estimated prices; the retry test covers row removal and notice unmount. A live Storybook browser check verifies the confirmation and updated count. - Run `pnpm storybook`. Open **Costs / Subscriptions / Cost Data Quality Details** to inspect the production page with representative fixtures. ## Risks - Model and billing metadata change for future runs. Estimates still require complete usage and a supported price. No past cost amount is rewritten. - A missing final receipt remains unresolved for budget checks. The display changes after fifteen minutes; that time limit never authorizes settlement. - Accounting retry is a write action. It retains the existing company checks, receipt fences, idempotency, and audit logging. - Detail reads are bounded and run only after expansion. They poll every 30 seconds while open. No provider probe or new summary query is added. - Migration `0333` adds two partial indexes; it changes no stored cost amounts or table columns. Index creation scans the existing tables and may briefly block writes during deployment. No lockfile or workflow change is included. ## Model Used OpenAI Codex, based on GPT-6, with repository inspection, code editing, browser checks, and test execution. The exact serving model ID, reasoning setting, and context window were not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Paperclip <noreply@paperclip.ing>
359 lines
14 KiB
TypeScript
359 lines
14 KiB
TypeScript
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
|
import fs from "node:fs/promises";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
|
|
vi.mock("@paperclipai/adapter-utils/execution-target", async (importOriginal) => {
|
|
const actual = (await importOriginal()) as Record<string, unknown>;
|
|
return { ...actual, runAdapterExecutionTargetProcess: vi.fn() };
|
|
});
|
|
|
|
import { ensureRemoteOpenCodeModelConfiguredAndAvailable, execute } from "./execute.js";
|
|
import { runAdapterExecutionTargetProcess } from "@paperclipai/adapter-utils/execution-target";
|
|
import { createPromptContextFixture } from "@paperclipai/adapter-utils/test-fixtures/prompt-context";
|
|
|
|
const runProcessMock = vi.mocked(runAdapterExecutionTargetProcess);
|
|
|
|
async function createSkillDir(root: string, name: string): Promise<string> {
|
|
const skillDir = path.join(root, name);
|
|
await fs.mkdir(skillDir, { recursive: true });
|
|
await fs.writeFile(path.join(skillDir, "SKILL.md"), `# ${name}\n`, "utf8");
|
|
return skillDir;
|
|
}
|
|
|
|
function probeResult(overrides: Record<string, unknown>) {
|
|
return {
|
|
exitCode: 0,
|
|
signal: null,
|
|
timedOut: false,
|
|
stdout: "",
|
|
stderr: "",
|
|
pid: 123,
|
|
startedAt: new Date().toISOString(),
|
|
...overrides,
|
|
} as never;
|
|
}
|
|
|
|
describe("OpenCode local skill injection", () => {
|
|
let configHome: string;
|
|
|
|
beforeEach(async () => {
|
|
configHome = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-opencode-test-config-"));
|
|
vi.stubEnv("XDG_CONFIG_HOME", configHome);
|
|
});
|
|
|
|
afterEach(async () => {
|
|
vi.unstubAllEnvs();
|
|
await fs.rm(configHome, { recursive: true, force: true });
|
|
});
|
|
|
|
it.each([false, true])("keeps chat policy with a legacy OpenCode prompt (custom=%s)", async (custom) => {
|
|
const commandPath = path.join(configHome, "fake-opencode");
|
|
await fs.writeFile(commandPath, "#!/bin/sh\nexit 0\n", { mode: 0o755 });
|
|
runProcessMock.mockReset();
|
|
runProcessMock.mockResolvedValue(probeResult({ stdout: JSON.stringify({
|
|
type: "text", sessionID: "chat-session", part: { text: "Reply" },
|
|
}) }));
|
|
const directive = "Chat directive: clarify goals and hand plans off to project tasks.";
|
|
let prompt = "";
|
|
const result = await execute({
|
|
runId: "chat-run",
|
|
agent: { id: "agent-1", companyId: "company-1", name: "OpenCode", adapterType: "opencode_local", adapterConfig: {} },
|
|
runtime: { sessionId: null, sessionParams: null, sessionDisplayId: null, taskKey: null },
|
|
config: {
|
|
command: commandPath, cwd: configHome, model: "openai/gpt-5", env: { OPENCODE_ALLOW_ALL_MODELS: "1" },
|
|
...(custom ? { promptTemplate: "Custom agent instruction." } : {}),
|
|
},
|
|
context: {
|
|
conversationMode: true,
|
|
paperclipTaskMarkdown: directive,
|
|
paperclipWake: {
|
|
reason: "issue_commented", issue: { id: "chat-1", status: "in_progress", workMode: "planning" },
|
|
interactionKind: "request_confirmation", interactionStatus: "accepted",
|
|
},
|
|
},
|
|
onLog: async () => {},
|
|
onMeta: async (meta) => { prompt = String(meta.prompt ?? ""); },
|
|
});
|
|
expect(result.exitCode).toBe(0);
|
|
expect(prompt).toContain(directive);
|
|
expect(prompt).toContain(custom ? "Custom agent instruction." : "You are agent agent-1");
|
|
expect(prompt).not.toContain("Execution contract:");
|
|
expect(prompt).not.toContain("Create child issues");
|
|
});
|
|
|
|
it("delivers assignment context on an ordinary task turn and rebuilds it after resume fallback", async () => {
|
|
const commandPath = path.join(configHome, "fake-opencode-context");
|
|
await fs.writeFile(commandPath, "#!/bin/sh\nexit 0\n", { mode: 0o755 });
|
|
const prompts: string[] = [];
|
|
runProcessMock
|
|
.mockReset()
|
|
.mockResolvedValueOnce(probeResult({ stdout: JSON.stringify({ type: "error", error: "unknown session" }) }))
|
|
.mockResolvedValueOnce(probeResult({ stdout: JSON.stringify({ type: "text", sessionID: "fresh", part: { text: "done" } }) }));
|
|
await execute({
|
|
runId: "run-context-fallback",
|
|
agent: { id: "agent-1", companyId: "company-1", name: "OpenCode", adapterType: "opencode_local", adapterConfig: {} },
|
|
runtime: { sessionId: "previous", sessionParams: null, sessionDisplayId: null, taskKey: null },
|
|
config: { command: commandPath, cwd: configHome, model: "openai/gpt-5", env: { OPENCODE_ALLOW_ALL_MODELS: "1" } },
|
|
context: createPromptContextFixture(),
|
|
onLog: async () => {},
|
|
onMeta: async (meta) => { prompts.push(String(meta.prompt ?? "")); },
|
|
});
|
|
expect(prompts).toHaveLength(2);
|
|
expect(prompts[0]).toContain("## Compact assignment");
|
|
expect(prompts[1]).toContain("## Owned assignment");
|
|
for (const prompt of prompts) expect(prompt).not.toContain("Execution contract:");
|
|
expect(prompts[1]).toContain("You are agent agent-1 (OpenCode).");
|
|
expect(prompts[1]).toContain("Connection tools:");
|
|
});
|
|
|
|
it("injects runtime skills into the configured child HOME", async () => {
|
|
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-opencode-configured-home-"));
|
|
const processHome = path.join(root, "process-home");
|
|
const configuredHome = path.join(root, "configured-home");
|
|
const workspace = path.join(root, "workspace");
|
|
const commandPath = path.join(root, "opencode");
|
|
const skillSource = await createSkillDir(path.join(root, "runtime-skills"), "paperclip");
|
|
await fs.mkdir(workspace, { recursive: true });
|
|
await fs.writeFile(commandPath, "#!/bin/sh\nexit 0\n", "utf8");
|
|
await fs.chmod(commandPath, 0o755);
|
|
|
|
const previousHome = process.env.HOME;
|
|
process.env.HOME = processHome;
|
|
runProcessMock.mockReset();
|
|
runProcessMock.mockResolvedValueOnce(probeResult({
|
|
stdout: JSON.stringify({
|
|
type: "text",
|
|
sessionID: "session-configured-home",
|
|
part: { text: "done" },
|
|
}),
|
|
}));
|
|
|
|
try {
|
|
const result = await execute({
|
|
runId: "run-configured-home",
|
|
agent: {
|
|
id: "agent-1",
|
|
companyId: "company-1",
|
|
name: "OpenCode Coder",
|
|
adapterType: "opencode_local",
|
|
adapterConfig: {},
|
|
},
|
|
runtime: {
|
|
sessionId: null,
|
|
sessionParams: null,
|
|
sessionDisplayId: null,
|
|
taskKey: null,
|
|
},
|
|
config: {
|
|
command: commandPath,
|
|
cwd: workspace,
|
|
model: "openai/gpt-5",
|
|
env: {
|
|
HOME: configuredHome,
|
|
OPENCODE_ALLOW_ALL_MODELS: "1",
|
|
},
|
|
paperclipRuntimeSkills: [{
|
|
key: "paperclipai/paperclip/paperclip",
|
|
runtimeName: "paperclip",
|
|
source: skillSource,
|
|
}],
|
|
promptTemplate: "Follow the paperclip heartbeat.",
|
|
},
|
|
context: {},
|
|
authToken: "run-jwt-token",
|
|
onLog: async () => {},
|
|
});
|
|
|
|
expect(result.exitCode).toBe(0);
|
|
const installedSkill = path.join(configuredHome, ".claude", "skills", "paperclip");
|
|
expect((await fs.lstat(installedSkill)).isSymbolicLink()).toBe(true);
|
|
expect(await fs.realpath(installedSkill)).toBe(await fs.realpath(skillSource));
|
|
await expect(fs.lstat(path.join(processHome, ".claude", "skills", "paperclip"))).rejects.toThrow();
|
|
} finally {
|
|
if (previousHome === undefined) delete process.env.HOME;
|
|
else process.env.HOME = previousHome;
|
|
await fs.rm(root, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it.each([false, true])("preserves API billing in streamed usage and final results without leaking keys (timedOut=%s)", async (timedOut) => {
|
|
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-opencode-openrouter-"));
|
|
const workspace = path.join(root, "workspace");
|
|
const commandPath = path.join(root, "opencode");
|
|
const apiKey = "openrouter-test-secret";
|
|
const model = "openrouter/anthropic/claude-sonnet-4.5";
|
|
await fs.mkdir(workspace, { recursive: true });
|
|
await fs.writeFile(commandPath, "#!/bin/sh\nexit 0\n", "utf8");
|
|
await fs.chmod(commandPath, 0o755);
|
|
runProcessMock.mockReset();
|
|
const usageLine = JSON.stringify({ type: "step_finish", sessionID: "session-openrouter",
|
|
part: { cost: 0.0025, tokens: { input: 10, output: 2, cache: { read: 100, write: 0 } } } });
|
|
runProcessMock.mockImplementationOnce(async (...args) => {
|
|
await args[4].onLog("stdout", `${usageLine}\n`);
|
|
return probeResult({ stdout: usageLine, timedOut, exitCode: timedOut ? 143 : 0, signal: timedOut ? "SIGTERM" : null });
|
|
});
|
|
const logs: string[] = [];
|
|
const metadata: unknown[] = [];
|
|
const onUsage = vi.fn();
|
|
|
|
try {
|
|
const result = await execute({
|
|
runId: "run-openrouter",
|
|
agent: {
|
|
id: "agent-openrouter",
|
|
companyId: "company-1",
|
|
name: "OpenRouter Coder",
|
|
adapterType: "opencode_local",
|
|
adapterConfig: {},
|
|
},
|
|
runtime: {
|
|
sessionId: null,
|
|
sessionParams: null,
|
|
sessionDisplayId: null,
|
|
taskKey: null,
|
|
},
|
|
config: {
|
|
command: commandPath,
|
|
cwd: workspace,
|
|
model,
|
|
env: {
|
|
OPENROUTER_API_KEY: apiKey,
|
|
OPENCODE_ALLOW_ALL_MODELS: "1",
|
|
},
|
|
promptTemplate: "Run the task.",
|
|
},
|
|
context: {},
|
|
authToken: "run-jwt-token",
|
|
onUsage,
|
|
onLog: async (_stream, chunk) => {
|
|
logs.push(chunk);
|
|
},
|
|
onMeta: async (value) => {
|
|
metadata.push(value);
|
|
},
|
|
});
|
|
|
|
expect(result.exitCode).toBe(timedOut ? 143 : 0);
|
|
expect(result.usageComplete).toBe(!timedOut);
|
|
expect(result.model).toBe(model);
|
|
expect(result.billingType).toBe("metered_api");
|
|
expect(result.biller).toBe("openrouter");
|
|
expect(onUsage).toHaveBeenCalledWith(expect.objectContaining({ billingType: "metered_api", biller: "openrouter", model, complete: false }));
|
|
expect(onUsage).toHaveBeenLastCalledWith(expect.objectContaining({ billingType: "metered_api", biller: "openrouter", model, complete: !timedOut }));
|
|
const executionCall = runProcessMock.mock.calls.at(-1)!;
|
|
expect(executionCall[3]).toContain("--model");
|
|
expect(executionCall[3]).toContain(model);
|
|
expect((executionCall[4] as { env: Record<string, string> }).env.OPENROUTER_API_KEY).toBe(apiKey);
|
|
expect(JSON.stringify({ logs, metadata, result })).not.toContain(apiKey);
|
|
expect(JSON.stringify(metadata)).toContain('"OPENROUTER_API_KEY":"***REDACTED***"');
|
|
} finally {
|
|
await fs.rm(root, { recursive: true, force: true });
|
|
}
|
|
});
|
|
});
|
|
|
|
describe("ensureRemoteOpenCodeModelConfiguredAndAvailable", () => {
|
|
afterEach(() => {
|
|
delete process.env.OPENCODE_ALLOW_ALL_MODELS;
|
|
});
|
|
|
|
// The remote/sandbox execution path must honour OPENCODE_ALLOW_ALL_MODELS just
|
|
// like the local path: gateway-routed models (e.g. anthropic/<gateway>/<model>
|
|
// via Bifrost) never appear in `opencode models`, so the availability probe
|
|
// must be skipped. The early return happens before the executionTarget is ever
|
|
// touched, so a bogus target proves the probe was not run.
|
|
const bogusTarget = {} as never;
|
|
|
|
it("skips the remote availability probe when OPENCODE_ALLOW_ALL_MODELS is set in the run env", async () => {
|
|
await expect(
|
|
ensureRemoteOpenCodeModelConfiguredAndAvailable({
|
|
runId: "run-1",
|
|
executionTarget: bogusTarget,
|
|
command: "opencode",
|
|
model: "anthropic/tensorix/deepseek/deepseek-chat-v3.1",
|
|
cwd: "/tmp",
|
|
env: { OPENCODE_ALLOW_ALL_MODELS: "true" },
|
|
timeoutSec: 30,
|
|
graceSec: 5,
|
|
}),
|
|
).resolves.toBeUndefined();
|
|
});
|
|
|
|
it("honours OPENCODE_ALLOW_ALL_MODELS from the process env", async () => {
|
|
process.env.OPENCODE_ALLOW_ALL_MODELS = "1";
|
|
await expect(
|
|
ensureRemoteOpenCodeModelConfiguredAndAvailable({
|
|
runId: "run-2",
|
|
executionTarget: bogusTarget,
|
|
command: "opencode",
|
|
model: "anthropic/tensorix/deepseek/deepseek-chat-v3.1",
|
|
cwd: "/tmp",
|
|
env: {},
|
|
timeoutSec: 30,
|
|
graceSec: 5,
|
|
}),
|
|
).resolves.toBeUndefined();
|
|
});
|
|
|
|
it("still enforces provider/model format even when the bypass flag is set", async () => {
|
|
await expect(
|
|
ensureRemoteOpenCodeModelConfiguredAndAvailable({
|
|
runId: "run-3",
|
|
executionTarget: bogusTarget,
|
|
command: "opencode",
|
|
model: "",
|
|
cwd: "/tmp",
|
|
env: { OPENCODE_ALLOW_ALL_MODELS: "true" },
|
|
timeoutSec: 30,
|
|
graceSec: 5,
|
|
}),
|
|
).rejects.toThrow();
|
|
});
|
|
});
|
|
|
|
describe("ensureRemoteOpenCodeModelConfiguredAndAvailable — probe is non-fatal when it cannot run", () => {
|
|
const target = { kind: "remote", transport: "ssh" } as never;
|
|
const base = {
|
|
runId: "run-probe",
|
|
executionTarget: target,
|
|
command: "opencode",
|
|
cwd: "/tmp",
|
|
env: {} as Record<string, string>,
|
|
timeoutSec: 30,
|
|
graceSec: 5,
|
|
};
|
|
|
|
beforeEach(() => {
|
|
runProcessMock.mockReset();
|
|
});
|
|
|
|
it("proceeds when the remote probe exits non-zero (e.g. a transient `Unexpected error`)", async () => {
|
|
runProcessMock.mockResolvedValueOnce(probeResult({ exitCode: 1, stderr: "Unexpected error" }));
|
|
await expect(
|
|
ensureRemoteOpenCodeModelConfiguredAndAvailable({ ...base, model: "openai/gpt-5" }),
|
|
).resolves.toBeUndefined();
|
|
});
|
|
|
|
it("proceeds when the remote probe times out", async () => {
|
|
runProcessMock.mockResolvedValueOnce(probeResult({ timedOut: true, exitCode: null }));
|
|
await expect(
|
|
ensureRemoteOpenCodeModelConfiguredAndAvailable({ ...base, model: "openai/gpt-5" }),
|
|
).resolves.toBeUndefined();
|
|
});
|
|
|
|
it("proceeds when the remote probe returns no models", async () => {
|
|
runProcessMock.mockResolvedValueOnce(probeResult({ exitCode: 0, stdout: "" }));
|
|
await expect(
|
|
ensureRemoteOpenCodeModelConfiguredAndAvailable({ ...base, model: "openai/gpt-5" }),
|
|
).resolves.toBeUndefined();
|
|
});
|
|
|
|
it("still rejects when the probe succeeds but the configured model is absent (guard retained)", async () => {
|
|
runProcessMock.mockResolvedValueOnce(probeResult({ exitCode: 0, stdout: "openai/gpt-4.1\n" }));
|
|
await expect(
|
|
ensureRemoteOpenCodeModelConfiguredAndAvailable({ ...base, model: "openai/gpt-5" }),
|
|
).rejects.toThrow("Configured OpenCode model is unavailable on the remote execution target");
|
|
});
|
|
});
|