Files
PaperClipAI/packages/paperclip-runner/scripts/qualify-opencode-runtime.test.mjs
T
DottaandClaude Opus 5.5 43aabb7080 fix(runner): keep configured secret values out of OpenCode output, storage and model requests (#21330)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can configure a secret that Paperclip delivers to an
agent's commands as an environment variable
> - The value must stay out of what the agent runtime stores and what it
sends to the model
> - With the OpenCode runtime, a command that printed the variable put
the value in OpenCode's tool output. OpenCode stored that output in its
session database and sent it to the model. Paperclip's own run log
already redacted it
> - Paperclip did not tell the runner which configured values are
secrets, so the runner could not protect them without also hiding
ordinary settings
> - This pull request sends secret names (never values) to the runner,
filters exact secret values before OpenCode captures any tool output,
redacts them from model requests, and tells the agent how to check a
secret without printing it
> - The benefit is that a configured secret stays out of model context
and out of runtime storage, even when an agent prints it

## Linked Issues or Issue Description

**What happened?**
The `chat-features.runner-opencode.local.secrets.personal-secret` eval
failed because the persisted-state scan found a configured secret value
in OpenCode's session database. The person entered the value through the
normal secret entry form. Nobody pasted it in chat. The agent ran a
shell command that printed the environment variable it was given.
Paperclip's run log redacted the value. OpenCode stored the raw output
in its database and included it in the next model request.

**Expected behavior**
A configured secret must never reach model context or runtime
persistence. Ordinary configured values must not be redacted.

**Steps to reproduce**
Configure a secret environment variable for an OpenCode agent. Have the
agent run a command that prints the variable. Then search the OpenCode
session data and the model requests for the value.

## What Changed

- **Secret names cross to the runner.**
`configuredEnvironmentProjection` takes the controller's secret key
names and adds `PAPERCLIP_SECRET_ENV_KEYS`. This marker holds only the
names of configured entries that came from secrets. It travels with the
configured projection across the TypeScript and Rust runner boundaries
(`configured-environment.ts`, `configured_environment.rs`). The server
passes `secretKeys` in `runtime-execution.ts`.
- **Command output is filtered before OpenCode reads it.** For a launch
that has configured secrets, the runner sets OpenCode's `shell` to a
small POSIX wrapper (`drivers/opencode/output-custody.ts`). The wrapper
runs the shell OpenCode would use anyway. It sends stdout and stderr
through an `awk` filter that replaces exact secret values. OpenCode
never receives the raw bytes, so its streamed previews, its database and
its model requests stay clean. The wrapper keeps the exit status. It
does not wait for background processes. The values come from the
environment that OpenCode already holds and are never written to disk.
- **Other tool results are rewritten before they are saved.** A local
OpenCode plugin rewrites completed non-shell results (file reads,
searches, MCP results) in `tool.execute.after`. OpenCode runs that hook
before it stores or returns the result. The wrapper refuses to run any
command until the plugin has started for the current launch (per-launch
nonce). So no command can write a secret to a file that a later read
returns without filtering.
- **Model requests are redacted as a second layer.** The OpenCode
provider proxy removes exact secret values from the string values of
each parsed model request. Field names and other literals are never
changed. A quick byte scan for each value, raw and JSON-escaped, sends
the original bytes unchanged when no value is present.
- **The agent gets guidance.** Native task prompts list the configured
secret names. They say to check a secret with `[ -n "$VAR" ]` and never
to print, echo, log or write a secret value.
- **No added startup delay.** The wrapper is content-addressed under the
runtime directory and reused across launches, so it is not rewritten on
every launch. OpenCode's plugin-dependency install is marked as already
done, so loading the plugin does not add wait time to the first tool
call.
- **Only exact values are redacted.** Command output is filtered line by
line, so each line of a multi-line value that has at least 4 characters
is redacted too. If a configured secret has no line of 4 or more
characters, the launch is refused with
`opencode_output_custody_unavailable` instead of running unprotected.

## Verification

-
`packages/paperclip-runner/src/drivers/opencode/output-custody.test.ts`
(10 tests): the wrapper filters stdout, stderr and a partial last line;
keeps exit codes (7, and 143 for SIGTERM); filters each line of a
multi-line value; refuses to run before plugin readiness or with an old
nonce; returns at once when a background process keeps running; leaves
ordinary values unchanged. The plugin rewrites nested results.
Model-request redaction covers raw and JSON-escaped values and never
changes field names or literals (a secret equal to `true` leaves
`"stream": true` intact). Secrets the filter cannot match are refused.
No value is written to disk.
- `opencode-server-driver.test.ts`: one saved session launches three
times. First with no secret marker (older controller), then with the
marker, then with no marker again (rollback). The first launch is
unchanged, the second applies custody, and the third launch still
starts. This test fails without the driver change.
- `configured-environment.test.ts`: the marker names only secret
entries, crosses the runner spawn and OpenCode boundaries, reads as "no
secrets" when absent, and is refused when malformed. Rust:
`secret_name_marker_crosses_only_with_its_projection`.
- `server/src/__tests__/configured-secret-guidance.test.ts`: guidance
content, and no guidance when there are no secrets.
- Real OpenCode 1.18.34 qualification (`PAPERCLIP_OPENCODE_QUALIFY=1
node --test scripts/qualify-opencode-runtime.test.mjs`). A fake model
asks for a command that prints the secret and writes it to a file, then
a read of that file. Unprotected baseline: the value is in the session
database and in model requests (red). With custody: it is in neither,
and the exit status is kept (green).
- **Latency** (local, macOS, real OpenCode, 8 runs each): boot 629 ms vs
697 ms baseline; first model request after the prompt 678 ms vs 698 ms
(no regression); first command 139 ms vs 191 ms; each later command +9
ms (36 vs 27 ms). Proxy cost per request when no secret is present (the
normal case): 33 µs at 50 KiB, 164 µs at 300 KiB, 0.55 ms at 1 MiB, 2.3
ms at 4 MiB. When a secret is present: 91 µs, 0.48 ms, 1.6 ms and 6.8
ms. The scan runs only for launches that have configured secrets, and it
adds nothing to OpenCode's own startup.
- A narrow paid run of
`chat-features.runner-opencode.local.secrets.personal-secret` on this
branch is linked in a comment.

## Risks

- **Previous-release story.** The marker is new and optional. A runner
that receives no marker (older controller) launches exactly as before.
An older runner drops the marker at its environment boundary. The change
saves no new run or session state. OpenCode configuration is regenerated
on every launch, so sessions saved before this change resume under
custody (covered by the cutover test). Warm-process reuse keys include
the marker, so a warm process from before the deploy is not reused for a
run that has secrets.
- **Commands go through the wrapper.** Output arrives line by line.
Output written after the shell exits is dropped, which matches how
OpenCode stops reading at exit. If `awk`, `mkfifo` or `rm` is missing,
the launch is refused with `opencode_output_custody_unavailable` and
does not run unprotected.
- **Scope.** Only exact values are redacted. An agent that encodes a
value (for example base64) is out of scope. The model-request layer
applies when Paperclip routes the model through its proxy (OpenRouter
and Paperclip connections).

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 03:23:56 -06:00

257 lines
14 KiB
JavaScript

// Opt-in qualification of the installed CLI, not the fake protocol server.
// PAPERCLIP_OPENCODE_QUALIFY=1 node --test scripts/qualify-opencode-runtime.test.mjs
import assert from "node:assert/strict";
import { spawn, execFileSync } from "node:child_process";
import { once } from "node:events";
import { mkdir, mkdtemp, readdir, readFile, rm, writeFile } from "node:fs/promises";
import { createServer } from "node:http";
import { tmpdir } from "node:os";
import { dirname, join, resolve } from "node:path";
import { setTimeout as delay } from "node:timers/promises";
import test from "node:test";
const enabled = process.env.PAPERCLIP_OPENCODE_QUALIFY === "1";
const packageRoot = resolve(import.meta.dirname, "..");
const manifest = JSON.parse(await readFile(join(packageRoot, "package.json"), "utf8"));
test("the pinned OpenCode executable serves health, sessions, SSE, and a local-provider prompt", {
skip: !enabled,
timeout: 60_000,
}, async (t) => {
const root = await mkdtemp(join(tmpdir(), "paperclip-opencode-qualification-"));
let provider;
let child;
let exited;
const streamAbort = new AbortController();
const killGroup = (signal) => {
if (!child?.pid) return;
try { process.kill(-child.pid, signal); } catch (error) { if (error.code !== "ESRCH") throw error; }
};
t.after(async () => {
streamAbort.abort();
if (child?.pid) {
killGroup("SIGTERM");
const kill = setTimeout(() => killGroup("SIGKILL"), 3000);
try { await exited; } finally { clearTimeout(kill); killGroup("SIGKILL"); }
}
if (provider) { provider.closeAllConnections(); await new Promise((done) => provider.close(done)); }
await rm(root, { recursive: true, force: true });
});
const command = resolve(process.env.PAPERCLIP_TEST_OPENCODE_BINARY ?? join(packageRoot, "node_modules/opencode-ai/bin/opencode.exe"));
assert.equal(execFileSync(command, ["--version"], { encoding: "utf8", timeout: 10_000 }).trim(), manifest.dependencies["opencode-ai"]);
const requests = [];
provider = createServer(async (request, response) => {
const chunks = [];
for await (const chunk of request) chunks.push(chunk);
const body = JSON.parse(Buffer.concat(chunks).toString("utf8") || "{}");
requests.push({ path: request.url, body });
if (request.url !== "/v1/chat/completions") {
response.writeHead(404).end();
return;
}
response.writeHead(200, { "Content-Type": "text/event-stream" });
for (const [delta, finish_reason] of [
[{ role: "assistant", content: "paperclip-opencode-qualified" }, null],
[{}, "stop"],
]) {
response.write(`data: ${JSON.stringify({ id: "chatcmpl-qualification", object: "chat.completion.chunk", created: 1, model: "qualification", choices: [{ index: 0, delta, finish_reason }] })}\n\n`);
}
response.end("data: [DONE]\n\n");
});
provider.listen(0, "127.0.0.1");
await once(provider, "listening");
const providerUrl = `http://127.0.0.1:${provider.address().port}/v1`;
const config = join(root, "opencode.json");
await writeFile(config, JSON.stringify({
model: "openrouter/qualification",
small_model: "openrouter/qualification",
share: "disabled", autoupdate: false, plugin: [],
enabled_providers: ["openrouter"],
provider: { openrouter: { options: { baseURL: providerUrl, apiKey: "local-fixture-only" }, models: { qualification: { name: "qualification", limit: { context: 10000, output: 1000 } } } } },
permission: { "*": "deny" },
}));
const workspace = join(root, "workspace");
await mkdir(workspace);
child = spawn(command, ["serve", "--hostname", "127.0.0.1", "--port", "0"], {
cwd: workspace,
env: {
PATH: `${dirname(process.execPath)}:${process.env.PATH ?? "/usr/bin:/bin"}`,
HOME: root, XDG_CONFIG_HOME: join(root, "config"), XDG_DATA_HOME: join(root, "data"), XDG_CACHE_HOME: join(root, "cache"),
OPENCODE_CONFIG: config,
OPENCODE_DISABLE_PROJECT_CONFIG: "true", OPENCODE_DISABLE_MODELS_FETCH: "true", OPENCODE_DISABLE_DEFAULT_PLUGINS: "true",
OPENCODE_SERVER_USERNAME: "paperclip", OPENCODE_SERVER_PASSWORD: "local-fixture-only",
},
stdio: ["ignore", "pipe", "pipe"],
detached: true,
});
let output = "";
for (const stream of [child.stdout, child.stderr]) stream.on("data", (chunk) => { output = `${output}${chunk}`.slice(-16384); });
exited = once(child, "exit");
let baseUrl;
for (let attempt = 0; attempt < 150; attempt++) {
baseUrl = output.match(/http:\/\/127\.0\.0\.1:\d+/)?.[0];
if (baseUrl) break;
assert.equal(child.exitCode, null, output);
await delay(100);
}
assert.ok(baseUrl, `OpenCode did not start: ${output}`);
const headers = { Authorization: `Basic ${Buffer.from("paperclip:local-fixture-only").toString("base64")}`, "Content-Type": "application/json" };
const api = async (path, options = {}) => {
const response = await fetch(`${baseUrl}${path}`, { headers, signal: AbortSignal.timeout(15_000), ...options });
assert.ok(response.ok, `${path}: ${response.status} ${await response.clone().text()}`);
return response;
};
const health = await (await api("/global/health")).json();
assert.deepEqual(health, { healthy: true, version: manifest.dependencies["opencode-ai"] });
const session = await (await api("/session", { method: "POST", body: JSON.stringify({ title: "Runtime qualification" }) })).json();
assert.equal((await (await api(`/session/${session.id}`)).json()).id, session.id);
const stream = await fetch(`${baseUrl}/event`, { headers, signal: streamAbort.signal });
assert.equal(stream.status, 200);
assert.match(stream.headers.get("content-type"), /text\/event-stream/);
const reader = stream.body.getReader();
const events = [];
const readEvents = (async () => {
let buffer = "";
const decoder = new TextDecoder();
try {
while (true) {
const { done, value } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
let boundary;
while ((boundary = buffer.indexOf("\n\n")) !== -1) {
const frame = buffer.slice(0, boundary); buffer = buffer.slice(boundary + 2);
for (const line of frame.split("\n")) if (line.startsWith("data: ")) events.push(JSON.parse(line.slice(6)));
}
}
} catch (error) { if (!streamAbort.signal.aborted) throw error; }
})();
const prompt = await api(`/session/${session.id}/prompt_async`, {
method: "POST",
body: JSON.stringify({ model: { providerID: "openrouter", modelID: "qualification" }, parts: [{ type: "text", text: "Reply with the qualification marker." }] }),
});
assert.equal(prompt.status, 204);
let messages;
for (let attempt = 0; attempt < 150; attempt++) {
messages = await (await api(`/session/${session.id}/message`)).json();
if (messages.some((message) => message.info.role === "assistant" && message.parts.some((part) => part.type === "text" && part.text === "paperclip-opencode-qualified"))) break;
await delay(100);
}
assert.ok(messages.some((message) => message.info.role === "assistant" && message.parts.some((part) => part.type === "text" && part.text === "paperclip-opencode-qualified")), JSON.stringify(messages));
assert.ok(requests.some((request) => request.path === "/v1/chat/completions" && request.body.model === "qualification"));
assert.ok(events.some((event) => event.type === "message.part.updated" || event.type === "message.part.delta"));
streamAbort.abort();
await readEvents;
assert.equal((await api(`/session/${session.id}`, { method: "DELETE" })).status, 200);
});
// Output custody against the real executable: a command that prints a
// configured secret and a file read of that secret must reach neither the
// model request nor OpenCode's session store. Requires `pnpm build:typescript`.
for (const custody of [false, true]) {
test(`configured secrets ${custody ? "stay out of" : "reach (unprotected baseline)"} OpenCode's store and model requests`, {
skip: !enabled,
timeout: 90_000,
}, async (t) => {
const { prepareOpenCodeOutputCustody } = await import("../dist/drivers/opencode/output-custody.js");
const secret = `synthetic-qualification-secret-${Date.now().toString(36)}`;
const root = await mkdtemp(join(tmpdir(), "paperclip-opencode-custody-"));
const workspace = join(root, "workspace");
await mkdir(workspace);
let child;
let step = 0;
const bodies = [];
const provider = createServer(async (request, response) => {
const chunks = [];
for await (const chunk of request) chunks.push(chunk);
const raw = Buffer.concat(chunks).toString("utf8");
bodies.push(raw);
const body = JSON.parse(raw || "{}");
response.writeHead(200, { "Content-Type": "text/event-stream" });
const call = (name, args) => [{ role: "assistant", tool_calls: [{ index: 0, id: `call_${step}`, type: "function", function: { name, arguments: JSON.stringify(args) } }] }, null];
let frames = [[{ role: "assistant", content: "title" }, null], [{}, "stop"]];
if (Array.isArray(body.tools) && body.tools.length > 0) {
step += 1;
frames = step === 1 ? [call("bash", { command: "printenv QUALIFY_SECRET; printenv QUALIFY_SECRET > leak.txt; exit 3", description: "print" }), [{}, "tool_calls"]]
: step === 2 ? [call("read", { filePath: join(workspace, "leak.txt") }), [{}, "tool_calls"]]
: [[{ role: "assistant", content: "custody-done" }, null], [{}, "stop"]];
}
for (const [delta, finish_reason] of frames) response.write(`data: ${JSON.stringify({ id: "q", object: "chat.completion.chunk", created: 1, model: "q", choices: [{ index: 0, delta, finish_reason }] })}\n\n`);
response.end("data: [DONE]\n\n");
});
t.after(async () => {
if (child?.pid) { try { process.kill(-child.pid, "SIGKILL"); } catch {} }
provider.closeAllConnections();
await new Promise((done) => provider.close(done));
await rm(root, { recursive: true, force: true });
});
provider.listen(0, "127.0.0.1");
await once(provider, "listening");
const configHome = join(root, "config");
await mkdir(join(configHome, "opencode"), { recursive: true });
const env = {
PATH: `${dirname(process.execPath)}:${process.env.PATH ?? "/usr/bin:/bin"}`,
HOME: root, XDG_CONFIG_HOME: configHome, XDG_DATA_HOME: join(root, "data"), XDG_CACHE_HOME: join(root, "cache"),
OPENCODE_DISABLE_PROJECT_CONFIG: "true", OPENCODE_DISABLE_MODELS_FETCH: "true", OPENCODE_DISABLE_DEFAULT_PLUGINS: "true",
OPENCODE_SERVER_USERNAME: "paperclip", OPENCODE_SERVER_PASSWORD: "local-fixture-only", QUALIFY_SECRET: secret,
};
const config = {
model: "openrouter/q", small_model: "openrouter/q", share: "disabled", autoupdate: false, plugin: [], instructions: [],
enabled_providers: ["openrouter"],
provider: { openrouter: { options: { baseURL: `http://127.0.0.1:${provider.address().port}/v1`, apiKey: "local-fixture-only" }, models: { q: { name: "q", limit: { context: 100000, output: 1000 } } } } },
permission: { "*": "allow", external_directory: { "*": "allow" } },
};
if (custody) {
const prepared = await prepareOpenCodeOutputCustody({ directory: join(root, "custody"), binDirectory: join(root, "bin"),
configDirectory: join(configHome, "opencode"), environment: env, secrets: [{ name: "QUALIFY_SECRET", value: secret }] });
Object.assign(config, prepared.config);
Object.assign(env, prepared.environment);
}
await writeFile(join(configHome, "opencode", "opencode.json"), JSON.stringify(config));
const command = resolve(process.env.PAPERCLIP_TEST_OPENCODE_BINARY ?? join(packageRoot, "node_modules/opencode-ai/bin/opencode.exe"));
child = spawn(command, ["serve", "--hostname", "127.0.0.1", "--port", "0"], { cwd: workspace, env, stdio: ["ignore", "pipe", "pipe"], detached: true });
let output = "";
for (const stream of [child.stdout, child.stderr]) stream.on("data", (chunk) => { output = `${output}${chunk}`.slice(-16384); });
let baseUrl;
for (let attempt = 0; attempt < 150 && !baseUrl; attempt++) { baseUrl = output.match(/http:\/\/127\.0\.0\.1:\d+/)?.[0]; if (!baseUrl) await delay(100); }
assert.ok(baseUrl, `OpenCode did not start: ${output}`);
const headers = { Authorization: `Basic ${Buffer.from("paperclip:local-fixture-only").toString("base64")}`, "Content-Type": "application/json" };
const api = async (path, options = {}) => {
const response = await fetch(`${baseUrl}${path}`, { headers, signal: AbortSignal.timeout(15_000), ...options });
assert.ok(response.ok, `${path}: ${response.status}`);
return response;
};
const session = await (await api("/session", { method: "POST", body: "{}" })).json();
await api(`/session/${session.id}/prompt_async`, { method: "POST", body: JSON.stringify({ model: { providerID: "openrouter", modelID: "q" }, parts: [{ type: "text", text: "go" }] }) });
let messages = [];
for (let attempt = 0; attempt < 300; attempt++) {
messages = await (await api(`/session/${session.id}/message`)).json();
if (messages.some((message) => message.parts.some((part) => part.type === "text" && part.text === "custody-done"))) break;
await delay(100);
}
const tools = messages.flatMap((message) => message.parts.filter((part) => part.type === "tool"));
assert.equal(tools.length, 2, JSON.stringify(tools));
assert.equal(tools[0].state.metadata?.exit, 3);
process.kill(-child.pid, "SIGTERM");
await once(child, "exit");
const persisted = [];
const scan = async (directory) => {
for (const entry of await readdir(directory, { withFileTypes: true })) {
const file = join(directory, entry.name);
if (entry.isDirectory()) await scan(file);
else if (entry.isFile() && (await readFile(file)).includes(secret)) persisted.push(file);
}
};
await scan(join(root, "data"));
const sent = bodies.filter((body) => body.includes(secret)).length;
if (custody) {
assert.deepEqual(persisted, []);
assert.equal(sent, 0);
assert.match(String(tools[0].state.output), /\[REDACTED\]/);
assert.match(String(tools[1].state.output), /\[REDACTED\]/);
} else {
assert.ok(persisted.length > 0 && sent > 0, "the unprotected baseline must reproduce the leak");
}
});
}