mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person can configure a secret that Paperclip delivers to an agent's commands as an environment variable > - The value must stay out of what the agent runtime stores and what it sends to the model > - With the OpenCode runtime, a command that printed the variable put the value in OpenCode's tool output. OpenCode stored that output in its session database and sent it to the model. Paperclip's own run log already redacted it > - Paperclip did not tell the runner which configured values are secrets, so the runner could not protect them without also hiding ordinary settings > - This pull request sends secret names (never values) to the runner, filters exact secret values before OpenCode captures any tool output, redacts them from model requests, and tells the agent how to check a secret without printing it > - The benefit is that a configured secret stays out of model context and out of runtime storage, even when an agent prints it ## Linked Issues or Issue Description **What happened?** The `chat-features.runner-opencode.local.secrets.personal-secret` eval failed because the persisted-state scan found a configured secret value in OpenCode's session database. The person entered the value through the normal secret entry form. Nobody pasted it in chat. The agent ran a shell command that printed the environment variable it was given. Paperclip's run log redacted the value. OpenCode stored the raw output in its database and included it in the next model request. **Expected behavior** A configured secret must never reach model context or runtime persistence. Ordinary configured values must not be redacted. **Steps to reproduce** Configure a secret environment variable for an OpenCode agent. Have the agent run a command that prints the variable. Then search the OpenCode session data and the model requests for the value. ## What Changed - **Secret names cross to the runner.** `configuredEnvironmentProjection` takes the controller's secret key names and adds `PAPERCLIP_SECRET_ENV_KEYS`. This marker holds only the names of configured entries that came from secrets. It travels with the configured projection across the TypeScript and Rust runner boundaries (`configured-environment.ts`, `configured_environment.rs`). The server passes `secretKeys` in `runtime-execution.ts`. - **Command output is filtered before OpenCode reads it.** For a launch that has configured secrets, the runner sets OpenCode's `shell` to a small POSIX wrapper (`drivers/opencode/output-custody.ts`). The wrapper runs the shell OpenCode would use anyway. It sends stdout and stderr through an `awk` filter that replaces exact secret values. OpenCode never receives the raw bytes, so its streamed previews, its database and its model requests stay clean. The wrapper keeps the exit status. It does not wait for background processes. The values come from the environment that OpenCode already holds and are never written to disk. - **Other tool results are rewritten before they are saved.** A local OpenCode plugin rewrites completed non-shell results (file reads, searches, MCP results) in `tool.execute.after`. OpenCode runs that hook before it stores or returns the result. The wrapper refuses to run any command until the plugin has started for the current launch (per-launch nonce). So no command can write a secret to a file that a later read returns without filtering. - **Model requests are redacted as a second layer.** The OpenCode provider proxy removes exact secret values from the string values of each parsed model request. Field names and other literals are never changed. A quick byte scan for each value, raw and JSON-escaped, sends the original bytes unchanged when no value is present. - **The agent gets guidance.** Native task prompts list the configured secret names. They say to check a secret with `[ -n "$VAR" ]` and never to print, echo, log or write a secret value. - **No added startup delay.** The wrapper is content-addressed under the runtime directory and reused across launches, so it is not rewritten on every launch. OpenCode's plugin-dependency install is marked as already done, so loading the plugin does not add wait time to the first tool call. - **Only exact values are redacted.** Command output is filtered line by line, so each line of a multi-line value that has at least 4 characters is redacted too. If a configured secret has no line of 4 or more characters, the launch is refused with `opencode_output_custody_unavailable` instead of running unprotected. ## Verification - `packages/paperclip-runner/src/drivers/opencode/output-custody.test.ts` (10 tests): the wrapper filters stdout, stderr and a partial last line; keeps exit codes (7, and 143 for SIGTERM); filters each line of a multi-line value; refuses to run before plugin readiness or with an old nonce; returns at once when a background process keeps running; leaves ordinary values unchanged. The plugin rewrites nested results. Model-request redaction covers raw and JSON-escaped values and never changes field names or literals (a secret equal to `true` leaves `"stream": true` intact). Secrets the filter cannot match are refused. No value is written to disk. - `opencode-server-driver.test.ts`: one saved session launches three times. First with no secret marker (older controller), then with the marker, then with no marker again (rollback). The first launch is unchanged, the second applies custody, and the third launch still starts. This test fails without the driver change. - `configured-environment.test.ts`: the marker names only secret entries, crosses the runner spawn and OpenCode boundaries, reads as "no secrets" when absent, and is refused when malformed. Rust: `secret_name_marker_crosses_only_with_its_projection`. - `server/src/__tests__/configured-secret-guidance.test.ts`: guidance content, and no guidance when there are no secrets. - Real OpenCode 1.18.34 qualification (`PAPERCLIP_OPENCODE_QUALIFY=1 node --test scripts/qualify-opencode-runtime.test.mjs`). A fake model asks for a command that prints the secret and writes it to a file, then a read of that file. Unprotected baseline: the value is in the session database and in model requests (red). With custody: it is in neither, and the exit status is kept (green). - **Latency** (local, macOS, real OpenCode, 8 runs each): boot 629 ms vs 697 ms baseline; first model request after the prompt 678 ms vs 698 ms (no regression); first command 139 ms vs 191 ms; each later command +9 ms (36 vs 27 ms). Proxy cost per request when no secret is present (the normal case): 33 µs at 50 KiB, 164 µs at 300 KiB, 0.55 ms at 1 MiB, 2.3 ms at 4 MiB. When a secret is present: 91 µs, 0.48 ms, 1.6 ms and 6.8 ms. The scan runs only for launches that have configured secrets, and it adds nothing to OpenCode's own startup. - A narrow paid run of `chat-features.runner-opencode.local.secrets.personal-secret` on this branch is linked in a comment. ## Risks - **Previous-release story.** The marker is new and optional. A runner that receives no marker (older controller) launches exactly as before. An older runner drops the marker at its environment boundary. The change saves no new run or session state. OpenCode configuration is regenerated on every launch, so sessions saved before this change resume under custody (covered by the cutover test). Warm-process reuse keys include the marker, so a warm process from before the deploy is not reused for a run that has secrets. - **Commands go through the wrapper.** Output arrives line by line. Output written after the shell exits is dropped, which matches how OpenCode stops reading at exit. If `awk`, `mkfifo` or `rm` is missing, the launch is refused with `opencode_output_custody_unavailable` and does not run unprotected. - **Scope.** Only exact values are redacted. An agent that encodes a value (for example base64) is out of scope. The model-request layer applies when Paperclip routes the model through its proxy (OpenRouter and Paperclip connections). ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
257 lines
14 KiB
JavaScript
257 lines
14 KiB
JavaScript
// Opt-in qualification of the installed CLI, not the fake protocol server.
|
|
// PAPERCLIP_OPENCODE_QUALIFY=1 node --test scripts/qualify-opencode-runtime.test.mjs
|
|
import assert from "node:assert/strict";
|
|
import { spawn, execFileSync } from "node:child_process";
|
|
import { once } from "node:events";
|
|
import { mkdir, mkdtemp, readdir, readFile, rm, writeFile } from "node:fs/promises";
|
|
import { createServer } from "node:http";
|
|
import { tmpdir } from "node:os";
|
|
import { dirname, join, resolve } from "node:path";
|
|
import { setTimeout as delay } from "node:timers/promises";
|
|
import test from "node:test";
|
|
|
|
const enabled = process.env.PAPERCLIP_OPENCODE_QUALIFY === "1";
|
|
const packageRoot = resolve(import.meta.dirname, "..");
|
|
const manifest = JSON.parse(await readFile(join(packageRoot, "package.json"), "utf8"));
|
|
|
|
test("the pinned OpenCode executable serves health, sessions, SSE, and a local-provider prompt", {
|
|
skip: !enabled,
|
|
timeout: 60_000,
|
|
}, async (t) => {
|
|
const root = await mkdtemp(join(tmpdir(), "paperclip-opencode-qualification-"));
|
|
let provider;
|
|
let child;
|
|
let exited;
|
|
const streamAbort = new AbortController();
|
|
const killGroup = (signal) => {
|
|
if (!child?.pid) return;
|
|
try { process.kill(-child.pid, signal); } catch (error) { if (error.code !== "ESRCH") throw error; }
|
|
};
|
|
t.after(async () => {
|
|
streamAbort.abort();
|
|
if (child?.pid) {
|
|
killGroup("SIGTERM");
|
|
const kill = setTimeout(() => killGroup("SIGKILL"), 3000);
|
|
try { await exited; } finally { clearTimeout(kill); killGroup("SIGKILL"); }
|
|
}
|
|
if (provider) { provider.closeAllConnections(); await new Promise((done) => provider.close(done)); }
|
|
await rm(root, { recursive: true, force: true });
|
|
});
|
|
const command = resolve(process.env.PAPERCLIP_TEST_OPENCODE_BINARY ?? join(packageRoot, "node_modules/opencode-ai/bin/opencode.exe"));
|
|
assert.equal(execFileSync(command, ["--version"], { encoding: "utf8", timeout: 10_000 }).trim(), manifest.dependencies["opencode-ai"]);
|
|
const requests = [];
|
|
provider = createServer(async (request, response) => {
|
|
const chunks = [];
|
|
for await (const chunk of request) chunks.push(chunk);
|
|
const body = JSON.parse(Buffer.concat(chunks).toString("utf8") || "{}");
|
|
requests.push({ path: request.url, body });
|
|
if (request.url !== "/v1/chat/completions") {
|
|
response.writeHead(404).end();
|
|
return;
|
|
}
|
|
response.writeHead(200, { "Content-Type": "text/event-stream" });
|
|
for (const [delta, finish_reason] of [
|
|
[{ role: "assistant", content: "paperclip-opencode-qualified" }, null],
|
|
[{}, "stop"],
|
|
]) {
|
|
response.write(`data: ${JSON.stringify({ id: "chatcmpl-qualification", object: "chat.completion.chunk", created: 1, model: "qualification", choices: [{ index: 0, delta, finish_reason }] })}\n\n`);
|
|
}
|
|
response.end("data: [DONE]\n\n");
|
|
});
|
|
provider.listen(0, "127.0.0.1");
|
|
await once(provider, "listening");
|
|
const providerUrl = `http://127.0.0.1:${provider.address().port}/v1`;
|
|
const config = join(root, "opencode.json");
|
|
await writeFile(config, JSON.stringify({
|
|
model: "openrouter/qualification",
|
|
small_model: "openrouter/qualification",
|
|
share: "disabled", autoupdate: false, plugin: [],
|
|
enabled_providers: ["openrouter"],
|
|
provider: { openrouter: { options: { baseURL: providerUrl, apiKey: "local-fixture-only" }, models: { qualification: { name: "qualification", limit: { context: 10000, output: 1000 } } } } },
|
|
permission: { "*": "deny" },
|
|
}));
|
|
const workspace = join(root, "workspace");
|
|
await mkdir(workspace);
|
|
child = spawn(command, ["serve", "--hostname", "127.0.0.1", "--port", "0"], {
|
|
cwd: workspace,
|
|
env: {
|
|
PATH: `${dirname(process.execPath)}:${process.env.PATH ?? "/usr/bin:/bin"}`,
|
|
HOME: root, XDG_CONFIG_HOME: join(root, "config"), XDG_DATA_HOME: join(root, "data"), XDG_CACHE_HOME: join(root, "cache"),
|
|
OPENCODE_CONFIG: config,
|
|
OPENCODE_DISABLE_PROJECT_CONFIG: "true", OPENCODE_DISABLE_MODELS_FETCH: "true", OPENCODE_DISABLE_DEFAULT_PLUGINS: "true",
|
|
OPENCODE_SERVER_USERNAME: "paperclip", OPENCODE_SERVER_PASSWORD: "local-fixture-only",
|
|
},
|
|
stdio: ["ignore", "pipe", "pipe"],
|
|
detached: true,
|
|
});
|
|
let output = "";
|
|
for (const stream of [child.stdout, child.stderr]) stream.on("data", (chunk) => { output = `${output}${chunk}`.slice(-16384); });
|
|
exited = once(child, "exit");
|
|
let baseUrl;
|
|
for (let attempt = 0; attempt < 150; attempt++) {
|
|
baseUrl = output.match(/http:\/\/127\.0\.0\.1:\d+/)?.[0];
|
|
if (baseUrl) break;
|
|
assert.equal(child.exitCode, null, output);
|
|
await delay(100);
|
|
}
|
|
assert.ok(baseUrl, `OpenCode did not start: ${output}`);
|
|
const headers = { Authorization: `Basic ${Buffer.from("paperclip:local-fixture-only").toString("base64")}`, "Content-Type": "application/json" };
|
|
const api = async (path, options = {}) => {
|
|
const response = await fetch(`${baseUrl}${path}`, { headers, signal: AbortSignal.timeout(15_000), ...options });
|
|
assert.ok(response.ok, `${path}: ${response.status} ${await response.clone().text()}`);
|
|
return response;
|
|
};
|
|
const health = await (await api("/global/health")).json();
|
|
assert.deepEqual(health, { healthy: true, version: manifest.dependencies["opencode-ai"] });
|
|
const session = await (await api("/session", { method: "POST", body: JSON.stringify({ title: "Runtime qualification" }) })).json();
|
|
assert.equal((await (await api(`/session/${session.id}`)).json()).id, session.id);
|
|
const stream = await fetch(`${baseUrl}/event`, { headers, signal: streamAbort.signal });
|
|
assert.equal(stream.status, 200);
|
|
assert.match(stream.headers.get("content-type"), /text\/event-stream/);
|
|
const reader = stream.body.getReader();
|
|
const events = [];
|
|
const readEvents = (async () => {
|
|
let buffer = "";
|
|
const decoder = new TextDecoder();
|
|
try {
|
|
while (true) {
|
|
const { done, value } = await reader.read();
|
|
if (done) break;
|
|
buffer += decoder.decode(value, { stream: true });
|
|
let boundary;
|
|
while ((boundary = buffer.indexOf("\n\n")) !== -1) {
|
|
const frame = buffer.slice(0, boundary); buffer = buffer.slice(boundary + 2);
|
|
for (const line of frame.split("\n")) if (line.startsWith("data: ")) events.push(JSON.parse(line.slice(6)));
|
|
}
|
|
}
|
|
} catch (error) { if (!streamAbort.signal.aborted) throw error; }
|
|
})();
|
|
const prompt = await api(`/session/${session.id}/prompt_async`, {
|
|
method: "POST",
|
|
body: JSON.stringify({ model: { providerID: "openrouter", modelID: "qualification" }, parts: [{ type: "text", text: "Reply with the qualification marker." }] }),
|
|
});
|
|
assert.equal(prompt.status, 204);
|
|
let messages;
|
|
for (let attempt = 0; attempt < 150; attempt++) {
|
|
messages = await (await api(`/session/${session.id}/message`)).json();
|
|
if (messages.some((message) => message.info.role === "assistant" && message.parts.some((part) => part.type === "text" && part.text === "paperclip-opencode-qualified"))) break;
|
|
await delay(100);
|
|
}
|
|
assert.ok(messages.some((message) => message.info.role === "assistant" && message.parts.some((part) => part.type === "text" && part.text === "paperclip-opencode-qualified")), JSON.stringify(messages));
|
|
assert.ok(requests.some((request) => request.path === "/v1/chat/completions" && request.body.model === "qualification"));
|
|
assert.ok(events.some((event) => event.type === "message.part.updated" || event.type === "message.part.delta"));
|
|
streamAbort.abort();
|
|
await readEvents;
|
|
assert.equal((await api(`/session/${session.id}`, { method: "DELETE" })).status, 200);
|
|
});
|
|
|
|
// Output custody against the real executable: a command that prints a
|
|
// configured secret and a file read of that secret must reach neither the
|
|
// model request nor OpenCode's session store. Requires `pnpm build:typescript`.
|
|
for (const custody of [false, true]) {
|
|
test(`configured secrets ${custody ? "stay out of" : "reach (unprotected baseline)"} OpenCode's store and model requests`, {
|
|
skip: !enabled,
|
|
timeout: 90_000,
|
|
}, async (t) => {
|
|
const { prepareOpenCodeOutputCustody } = await import("../dist/drivers/opencode/output-custody.js");
|
|
const secret = `synthetic-qualification-secret-${Date.now().toString(36)}`;
|
|
const root = await mkdtemp(join(tmpdir(), "paperclip-opencode-custody-"));
|
|
const workspace = join(root, "workspace");
|
|
await mkdir(workspace);
|
|
let child;
|
|
let step = 0;
|
|
const bodies = [];
|
|
const provider = createServer(async (request, response) => {
|
|
const chunks = [];
|
|
for await (const chunk of request) chunks.push(chunk);
|
|
const raw = Buffer.concat(chunks).toString("utf8");
|
|
bodies.push(raw);
|
|
const body = JSON.parse(raw || "{}");
|
|
response.writeHead(200, { "Content-Type": "text/event-stream" });
|
|
const call = (name, args) => [{ role: "assistant", tool_calls: [{ index: 0, id: `call_${step}`, type: "function", function: { name, arguments: JSON.stringify(args) } }] }, null];
|
|
let frames = [[{ role: "assistant", content: "title" }, null], [{}, "stop"]];
|
|
if (Array.isArray(body.tools) && body.tools.length > 0) {
|
|
step += 1;
|
|
frames = step === 1 ? [call("bash", { command: "printenv QUALIFY_SECRET; printenv QUALIFY_SECRET > leak.txt; exit 3", description: "print" }), [{}, "tool_calls"]]
|
|
: step === 2 ? [call("read", { filePath: join(workspace, "leak.txt") }), [{}, "tool_calls"]]
|
|
: [[{ role: "assistant", content: "custody-done" }, null], [{}, "stop"]];
|
|
}
|
|
for (const [delta, finish_reason] of frames) response.write(`data: ${JSON.stringify({ id: "q", object: "chat.completion.chunk", created: 1, model: "q", choices: [{ index: 0, delta, finish_reason }] })}\n\n`);
|
|
response.end("data: [DONE]\n\n");
|
|
});
|
|
t.after(async () => {
|
|
if (child?.pid) { try { process.kill(-child.pid, "SIGKILL"); } catch {} }
|
|
provider.closeAllConnections();
|
|
await new Promise((done) => provider.close(done));
|
|
await rm(root, { recursive: true, force: true });
|
|
});
|
|
provider.listen(0, "127.0.0.1");
|
|
await once(provider, "listening");
|
|
const configHome = join(root, "config");
|
|
await mkdir(join(configHome, "opencode"), { recursive: true });
|
|
const env = {
|
|
PATH: `${dirname(process.execPath)}:${process.env.PATH ?? "/usr/bin:/bin"}`,
|
|
HOME: root, XDG_CONFIG_HOME: configHome, XDG_DATA_HOME: join(root, "data"), XDG_CACHE_HOME: join(root, "cache"),
|
|
OPENCODE_DISABLE_PROJECT_CONFIG: "true", OPENCODE_DISABLE_MODELS_FETCH: "true", OPENCODE_DISABLE_DEFAULT_PLUGINS: "true",
|
|
OPENCODE_SERVER_USERNAME: "paperclip", OPENCODE_SERVER_PASSWORD: "local-fixture-only", QUALIFY_SECRET: secret,
|
|
};
|
|
const config = {
|
|
model: "openrouter/q", small_model: "openrouter/q", share: "disabled", autoupdate: false, plugin: [], instructions: [],
|
|
enabled_providers: ["openrouter"],
|
|
provider: { openrouter: { options: { baseURL: `http://127.0.0.1:${provider.address().port}/v1`, apiKey: "local-fixture-only" }, models: { q: { name: "q", limit: { context: 100000, output: 1000 } } } } },
|
|
permission: { "*": "allow", external_directory: { "*": "allow" } },
|
|
};
|
|
if (custody) {
|
|
const prepared = await prepareOpenCodeOutputCustody({ directory: join(root, "custody"), binDirectory: join(root, "bin"),
|
|
configDirectory: join(configHome, "opencode"), environment: env, secrets: [{ name: "QUALIFY_SECRET", value: secret }] });
|
|
Object.assign(config, prepared.config);
|
|
Object.assign(env, prepared.environment);
|
|
}
|
|
await writeFile(join(configHome, "opencode", "opencode.json"), JSON.stringify(config));
|
|
const command = resolve(process.env.PAPERCLIP_TEST_OPENCODE_BINARY ?? join(packageRoot, "node_modules/opencode-ai/bin/opencode.exe"));
|
|
child = spawn(command, ["serve", "--hostname", "127.0.0.1", "--port", "0"], { cwd: workspace, env, stdio: ["ignore", "pipe", "pipe"], detached: true });
|
|
let output = "";
|
|
for (const stream of [child.stdout, child.stderr]) stream.on("data", (chunk) => { output = `${output}${chunk}`.slice(-16384); });
|
|
let baseUrl;
|
|
for (let attempt = 0; attempt < 150 && !baseUrl; attempt++) { baseUrl = output.match(/http:\/\/127\.0\.0\.1:\d+/)?.[0]; if (!baseUrl) await delay(100); }
|
|
assert.ok(baseUrl, `OpenCode did not start: ${output}`);
|
|
const headers = { Authorization: `Basic ${Buffer.from("paperclip:local-fixture-only").toString("base64")}`, "Content-Type": "application/json" };
|
|
const api = async (path, options = {}) => {
|
|
const response = await fetch(`${baseUrl}${path}`, { headers, signal: AbortSignal.timeout(15_000), ...options });
|
|
assert.ok(response.ok, `${path}: ${response.status}`);
|
|
return response;
|
|
};
|
|
const session = await (await api("/session", { method: "POST", body: "{}" })).json();
|
|
await api(`/session/${session.id}/prompt_async`, { method: "POST", body: JSON.stringify({ model: { providerID: "openrouter", modelID: "q" }, parts: [{ type: "text", text: "go" }] }) });
|
|
let messages = [];
|
|
for (let attempt = 0; attempt < 300; attempt++) {
|
|
messages = await (await api(`/session/${session.id}/message`)).json();
|
|
if (messages.some((message) => message.parts.some((part) => part.type === "text" && part.text === "custody-done"))) break;
|
|
await delay(100);
|
|
}
|
|
const tools = messages.flatMap((message) => message.parts.filter((part) => part.type === "tool"));
|
|
assert.equal(tools.length, 2, JSON.stringify(tools));
|
|
assert.equal(tools[0].state.metadata?.exit, 3);
|
|
process.kill(-child.pid, "SIGTERM");
|
|
await once(child, "exit");
|
|
const persisted = [];
|
|
const scan = async (directory) => {
|
|
for (const entry of await readdir(directory, { withFileTypes: true })) {
|
|
const file = join(directory, entry.name);
|
|
if (entry.isDirectory()) await scan(file);
|
|
else if (entry.isFile() && (await readFile(file)).includes(secret)) persisted.push(file);
|
|
}
|
|
};
|
|
await scan(join(root, "data"));
|
|
const sent = bodies.filter((body) => body.includes(secret)).length;
|
|
if (custody) {
|
|
assert.deepEqual(persisted, []);
|
|
assert.equal(sent, 0);
|
|
assert.match(String(tools[0].state.output), /\[REDACTED\]/);
|
|
assert.match(String(tools[1].state.output), /\[REDACTED\]/);
|
|
} else {
|
|
assert.ok(persisted.length > 0 && sent > 0, "the unprotected baseline must reproduce the leak");
|
|
}
|
|
});
|
|
}
|