test(runner-e2e): scope each agent's sandbox to its own run's saved state (#20253)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The runner E2E chat evals run real agents against a local Paperclip
instance on a CI host
> - PR #20044 hides the eval harness from local native agents with a
Linux Landlock domain around each runnerd launch
> - That domain is the same for every run in an attempt, so an eval
agent can still read another run's saved state
> - Recent chat-eval runs show agents that read other runs' OpenCode
session databases, another run's run log, the instance `pg_wal`,
`control-plane-state.json`, and `/proc/<pid>/cmdline` of harness
processes
> - This pull request scopes each runnerd launch to its own run and
moves the selected case id off the command line
> - The benefit is that an eval agent cannot learn about the case or
other runs from files outside its own run

## Linked Issues or Issue Description

Refs #20044

**What happened**

In local native OpenCode chat-eval cells, worker agents read files that
belong to other runs or to the harness:

- other runs' OpenCode sqlite session databases under
`runtime/paperclip-runner/durable-sessions/` (a status-cards run)
- `/proc/<pid>/cmdline` of harness processes, which shows harness paths
and the case id (`routines.pause-owned`)
- another run's run-log ndjson, and a listing of the instance
`db/pg_wal` (`routines.pause-owned`)
- a listing of `control-plane-state.json`
(`runs-adapters.list-model-options`)

The Landlock sandbox from #20044 hides only the harness (`tests/`, the
Git store, private attempt directories). All runnerd launches share one
hidden set, so the instance's per-run directories stay readable.

**Expected behavior**

An agent process sees only its own run's durable session, OpenCode
session and provider trace. Other runs' sessions, run logs and traces,
the instance database files, secrets and the shared control-plane state
in the attempt instance are denied. The case id is not on any harness
command line.

**Steps to reproduce**

1. Dispatch a local native OpenCode chat-features cell that has two
agents (for example a status-cards case).
2. In the worker's shell, run `ls
<instance>/runtime/paperclip-runner/durable-sessions/*/` and `ls
<instance>/db/pg_wal`.
3. Run `cat /proc/<launcher pid>/cmdline`.
4. Before this change, all three succeed and show other runs' files and
`--id chat-features...`.

## What Changed

- `agent-filesystem-sandbox.py`: new `--expose PATH` option. It grants a
path beneath a hidden directory. Landlock does not check traversal, so
the run uses its own entry while the parent stays unlistable. The helper
refuses a missing exposure, an exposure outside every hidden path, or
one that would reveal a hidden path. It fails the launch unless every
exposed path is readable.
- `agent-filesystem-sandbox.ts`: the launcher passes the instance root
in the sandbox configuration. `runnerdLaunchSandbox` adds per-launch
hidden paths: `durable-sessions/`, `runner/`, `control-plane/`,
`opencode/` under `runtime/paperclip-runner/`, and `data/run-logs/`,
`data/provider-traces/`, `data/backups/`, `db/`, `secrets/`. It exposes
only the launch's own durable-session directory (from `--state-dir`),
Codex runner state directory, OpenCode session directory (from
`--session-id`, named as the OpenCode driver names it) and provider
trace file.
- The wrapper creates the exposed entries, and the shared instance
directories that the product creates on first use, before runnerd
starts. It logs the run-scoped hidden and exposed paths for each launch.
- New `probeRunScopedSandbox`: at server start, a sandboxed shell runs
against a fixture instance. Another run's session, OpenCode database,
runner state, run log, trace, `pg_wal`, secrets, control-plane state and
the server's `/proc/<pid>/environ` must be denied. The run's own
session, OpenCode directory, ACPX directory and trace must be writable.
In CI any failure stops the attempt. The result goes to `server.log`.
- `selectors.ts` and `launch.ts`: the launcher reads the selected cell
from `PAPERCLIP_RUNNER_E2E_SELECTED_ID` and removes it from its
environment.
- `runner-full-stack-e2e.yml` (Run paid cell) and
`chat-catalog-testops-case.yml`: pass the cell in that variable instead
of `--id`. A target branch that does not read the variable still gets
`--id`.
- `SECURITY.md`: documents the run scope, the `/proc` limit and what
stays readable.

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/agent-filesystem-sandbox.test.ts
tests/runner-e2e/workflow-security.test.ts
tests/runner-e2e/manual-private-workflow.test.ts
tests/runner-e2e/private-trust-workflow-security.test.ts` passes on
macOS. The Landlock host tests (including the new fixture-instance probe
test) run only on Linux CI.
- `npx tsc --noEmit -p tests/runner-e2e` shows no errors in the changed
files.
- Paid run
[42611358428](https://github.com/paperclipai/paperclip/actions/runs/42611358428):
`chat-features.runner-opencode.local.agents.change-manager` on the same
commit applied to `chat-eval/master-candidate` (fca4bb7ce3). Result:
passed. In `server.log` the setup probe reports `failures: []` and
`run:otherDurableSession`, `run:durableSessionListing`,
`run:otherRunnerState`, `run:controlPlaneState`,
`run:otherOpencodeSession`, `run:otherRunLog`, `run:databaseWal`,
`run:secrets`, `run:otherProviderTrace` and `run:harnessEnviron` all
`denied`; `run:ownDurableSession`, `run:ownOpencodeSession`,
`run:acpxRuntime` and `run:ownProviderTrace` `allowed`;
`run:harnessCmdline` `allowed` (recorded, Landlock limit). The runnerd
launch line lists the nine run-scoped hidden instance paths and exposes
only its own durable-session and OpenCode session directories. The agent
finished the case inside that sandbox.

## Risks

- A sandboxed runnerd cannot create new direct entries in the instance
root, `runtime/`, `runtime/paperclip-runner/` or `data/`. The wrapper
creates the known ones first (`acpx`, `companies`, `projects`,
`workspaces`, `runtime-context-assets`, `skills`,
`runtime/provider-packs`). A provider that creates a different new
directory there fails in a sandboxed local cell. The paid run covers
OpenCode only. ACPX cells keep their runtime directory readable.
- A run still reads its own session's control-plane state, and other
agents' homes and task workspaces in the same instance.
- Landlock cannot hide `/proc/<pid>/cmdline`. Harness paths stay visible
there. Only the case id moves to the environment. Scheduled multi-cell
campaigns still pass `--id`.
- Harness-only change. No product code changes.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), extended thinking, tool
use (shell, file edits, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-09 07:28:12 -06:00
1 parent ad440d4e91
commit 347d21fcbd
8 files changed
+486 -25

No files matched your search

@@ -165,7 +165,12 @@ jobs:
PAPERCLIP_RUNNER_E2E_SOURCE_SHA: ${{ inputs.target_sha }}
PAPERCLIP_RUNNER_E2E_SOURCE_REF: refs/heads/codex/chat-catalog-master-integration
PAPERCLIP_PLAYWRIGHT_CHANNEL: chrome
run: pnpm test:e2e:runner -- --id "${{ fromJSON(inputs.cell).executionId }}" --max-parallel 1 --max-automatic-retries 0 > "$PRIVATE_EVIDENCE_ROOT/launcher-output.log" 2>&1
PAPERCLIP_RUNNER_E2E_SELECTED_ID: ${{ fromJSON(inputs.cell).executionId }}
run: |
# Agents can read any process's argv in /proc; keep the case id off the command line.
selector=(--id "$PAPERCLIP_RUNNER_E2E_SELECTED_ID")
if grep -q PAPERCLIP_RUNNER_E2E_SELECTED_ID tests/runner-e2e/selectors.ts 2>/dev/null; then selector=(); fi
pnpm test:e2e:runner -- "${selector[@]}" --max-parallel 1 --max-automatic-retries 0 > "$PRIVATE_EVIDENCE_ROOT/launcher-output.log" 2>&1
- name: Retain original evidence after success or failure
if: always() && steps.cell_credentials.outcome == 'success'
env: *cell_session
+7 -2
View File
@@ -1435,11 +1435,16 @@ jobs:
PAPERCLIP_RUNNER_E2E_SOURCE_SHA: ${{ needs.authorize.outputs.target_sha }}
PAPERCLIP_RUNNER_E2E_SOURCE_REF: ${{ needs.authorize.outputs.target_ref }}
PAPERCLIP_PLAYWRIGHT_CHANNEL: ${{ needs.authorize.outputs.playwright_channel }}
PAPERCLIP_RUNNER_E2E_SELECTED_ID: ${{ matrix.executionId }}
run: |
# Agents can read any process's argv in /proc. A target that reads the
# selected cell from the environment keeps the case id off the command line.
selector=(-- --id "$PAPERCLIP_RUNNER_E2E_SELECTED_ID")
if grep -q PAPERCLIP_RUNNER_E2E_SELECTED_ID tests/runner-e2e/selectors.ts 2>/dev/null; then selector=(); fi
if [ -n "$PAPERCLIP_RUNNER_E2E_PRIVATE_PREFLIGHT" ]; then
pnpm test:e2e:runner -- --id "${{ matrix.executionId }}" > "$PRIVATE_EVIDENCE_ROOT/launcher-output.log" 2>&1
pnpm test:e2e:runner "${selector[@]}" > "$PRIVATE_EVIDENCE_ROOT/launcher-output.log" 2>&1
else
pnpm test:e2e:runner -- --id "${{ matrix.executionId }}"
pnpm test:e2e:runner "${selector[@]}"
fi
- name: Stage bounded campaign evidence
+32 -2
View File
@@ -363,11 +363,41 @@ each wrapped runnerd launch. In CI a missing or ineffective sandbox stops the
attempt. Landlock does not hide that a path exists (`stat` still works); it
denies reading and listing.
Each runnerd launch is also scoped to its own run, so an eval agent learns
nothing about the case from another run's saved session, run log or provider
trace, or from the attempt instance's database files and secrets. The launcher passes the instance root to the server entry, and each
runnerd launch additionally hides the instance's `durable-sessions/`,
`runner/`, `control-plane/` and `opencode/` directories under
`runtime/paperclip-runner/`, and `data/run-logs/`, `data/provider-traces/`,
`data/backups/`, `db/` and `secrets/`. It then exposes only the launch's own
entries, found from its `--state-dir`, `--session-id` and provider trace path:
its durable session directory (or Codex runner state directory), its OpenCode
session directory and its trace file. Landlock does not check path traversal,
so a rule on an exposed path grants it although the directory above it has
none; listing that directory stays denied, and entries created there later
(another run's new session) stay unreadable. The server creates the exposed
entries, and the shared directories the product creates beside them on first
use, before the launch, because their hidden or ungranted parents accept no new
entries from inside the sandbox. The helper fails the launch unless every
hidden path is denied and every exposed path is readable. The setup probe
repeats this on a fixture instance (another run's session, OpenCode database,
run log, trace, `pg_wal`, secrets and control-plane state denied; the run's own
session, OpenCode directory and trace writable) and stops the attempt in CI on
any failure. A run still sees its own session's control-plane state.
Landlock cannot hide `/proc/<pid>/cmdline`. Same-user `hidepid` does not help
either: it hides only other users' processes. Landlock does deny reading a
process's environment, memory and file descriptors outside the sandbox, so CI
passes the selected cell in `PAPERCLIP_RUNNER_E2E_SELECTED_ID` instead of
`--id`; the launcher reads and removes it. The workflow keeps `--id` for target
branches that predate this.
Limits: Codex cells keep Codex's own bubblewrap sandbox, which cannot mount
inside Landlock. Legacy adapters, installed-release launches and macOS run
without this sandbox. A process that can reach the Docker daemon can still
mount the checkout into a container, and `ps` shows the launcher's command
line. Case fixtures that create their own private directories directly in the
mount the checkout into a container, and `ps` shows every harness process's
command line (paths, but no longer the selected cell in CI). Other agents' homes
and task workspaces in the same instance stay readable. Case fixtures that create their own private directories directly in the
shared temporary directory stay readable. The boundary is filesystem only: the
scenarios and graders are public on GitHub, and agents have network access for
their model provider. Daytona cells run the agent in a remote sandbox that holds only the
+35 -9
View File
@@ -8,7 +8,10 @@ descendant inherits the domain, including an agent's shell commands.
Usage:
agent-filesystem-sandbox.py [--required] [--unset NAME]... [--set NAME=VALUE]...
--hide PATH [--hide PATH]... -- COMMAND [ARG]...
--hide PATH [--hide PATH]... [--expose PATH]... -- COMMAND [ARG]...
An exposed path lies beneath a hidden directory and stays usable: a run sees its
own saved session while the directory holding every run's sessions stays hidden.
Without Landlock, --required exits 125 instead of running the command.
"""
@@ -46,6 +49,7 @@ def fail(message, code=125):
def parse(argv):
required = False
hidden = []
exposed = []
unset = []
assignments = []
index = 0
@@ -55,16 +59,16 @@ def parse(argv):
command = argv[index + 1:]
if not command:
fail("missing command", 2)
return required, hidden, unset, assignments, command
return required, hidden, exposed, unset, assignments, command
if token == "--required":
required = True
elif token in ("--hide", "--unset", "--set") and index + 1 < len(argv):
elif token in ("--hide", "--expose", "--unset", "--set") and index + 1 < len(argv):
index += 1
value = argv[index]
if token == "--hide":
if token in ("--hide", "--expose"):
if not os.path.isabs(value):
fail("hidden path must be absolute: %s" % value, 2)
hidden.append(value)
fail("%s path must be absolute: %s" % (token[2:], value), 2)
(hidden if token == "--hide" else exposed).append(value)
elif token == "--unset":
unset.append(value)
else:
@@ -95,7 +99,7 @@ def landlock():
return libc, syscall
def restrict(hidden):
def restrict(hidden, exposed=()):
libc, syscall = landlock()
try:
abi = syscall(
@@ -165,6 +169,10 @@ def restrict(hidden):
allow(child)
grant("/")
# Landlock does not check path traversal, so a rule on an exposed path
# grants it even though the hidden directory above it has no rule.
for path in exposed:
allow(path)
if libc.prctl(ctypes.c_int(PR_SET_NO_NEW_PRIVS), ctypes.c_ulong(1), ctypes.c_ulong(0), ctypes.c_ulong(0), ctypes.c_ulong(0)) != 0:
raise RuntimeError("prctl(PR_SET_NO_NEW_PRIVS) failed")
syscall(ctypes.c_long(SYS_LANDLOCK_RESTRICT_SELF), ctypes.c_long(ruleset), ctypes.c_long(0))
@@ -181,11 +189,19 @@ def restrict(hidden):
except OSError:
continue
raise RuntimeError("sandbox ineffective: %s is still readable" % path)
for path in exposed:
try:
if os.path.isdir(path):
os.listdir(path)
else:
os.close(os.open(path, os.O_RDONLY | os.O_CLOEXEC))
except OSError as error:
raise RuntimeError("sandbox unusable: %s is not readable (%s)" % (path, error))
return abi
def main():
required, hidden_inputs, unset, assignments, command = parse(sys.argv[1:])
required, hidden_inputs, exposed_inputs, unset, assignments, command = parse(sys.argv[1:])
hidden = []
for path in hidden_inputs:
# Keep paths that do not exist yet: their ancestors stay ungranted, so
@@ -194,6 +210,16 @@ def main():
if real == "/":
fail("refusing to hide the filesystem root", 2)
hidden.append(real)
exposed = []
for path in exposed_inputs:
real = os.path.realpath(path)
if not os.path.exists(real):
fail("exposed path must exist before the sandbox starts: %s" % real, 2)
if not any(within(real, root) and real != root for root in hidden):
fail("exposed path must lie beneath a hidden directory: %s" % real, 2)
if any(within(root, real) for root in hidden):
fail("exposed path would reveal a hidden path: %s" % real, 2)
exposed.append(real)
for name in unset:
os.environ.pop(name, None)
for name, value in assignments:
@@ -204,7 +230,7 @@ def main():
fail("Landlock requires Linux")
else:
try:
restrict(hidden)
restrict(hidden, exposed)
except (OSError, RuntimeError) as error:
if required:
fail(str(error))
@@ -1,6 +1,6 @@
import { execFileSync } from "node:child_process";
import { mkdtempSync, realpathSync, rmSync } from "node:fs";
import { mkdir, mkdtemp, readFile, realpath, rm, writeFile } from "node:fs/promises";
import { existsSync, mkdtempSync, readFileSync, realpathSync, rmSync, statSync } from "node:fs";
import { mkdir, mkdtemp, readdir, readFile, realpath, rm, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it } from "vitest";
@@ -20,7 +20,13 @@ import {
wrapRunnerdLaunch,
gitStorePaths,
RUNNERD_LAUNCH_APIS,
instanceRunStatePaths,
opencodeSessionDirectoryName,
prepareRunnerdLaunchSandbox,
probeRunScopedSandbox,
runnerdLaunchSandbox,
} from "./agent-filesystem-sandbox.js";
import { RUNNER_SELECTED_ID_ENV, parseRunnerSelectors, selectorArgumentsWithEnvironment } from "./selectors.js";
import { runnerMatrix } from "./catalog.js";
const repositoryRoot = "/work/paperclip";
@@ -113,6 +119,140 @@ describe("agent filesystem sandbox selection", () => {
});
});
describe("run-scoped runnerd sandbox", () => {
const instanceRoot = `${temporaryRoot}/paperclip-home/instances/runner-e2e-1`;
const runner = `${instanceRoot}/runtime/paperclip-runner`;
const config = { hidden: ["/work/paperclip/tests"], required: true, instanceRoot };
it("keeps the instance root in the attempt's sandbox configuration", () => {
expect(agentFilesystemSandboxConfig({ ...base, instanceRoot })?.instanceRoot).toBe(instanceRoot);
expect(agentFilesystemSandboxConfig({ ...base, instanceRoot: "/tmp/home/instances/x" })?.instanceRoot).toBeUndefined();
const parsed = parseAgentFilesystemSandboxConfig(JSON.stringify(config));
expect(parsed).toEqual(config);
expect(() => parseAgentFilesystemSandboxConfig(JSON.stringify({ ...config, instanceRoot: "relative" }))).toThrow(/malformed/);
});
it("hides every run's state and keeps only the launch's own durable session, OpenCode session and trace", () => {
const trace = `${instanceRoot}/data/provider-traces/4b1c.ndjson`;
const launch = runnerdLaunchSandbox(config, [
"--connect-url", "ws://127.0.0.1:1/durableRecovery/connect",
"--state-dir", `${runner}/durable-sessions/abc123/runner`,
"--run-id", "run-1", "--session-id", "ses:own/1",
], { PAPERCLIP_OPENCODE_RUNTIME_DIR: `${runner}/opencode`, PAPERCLIP_PROVIDER_TRACE_PATH: trace });
expect(launch.hidden).toEqual([
"/work/paperclip/tests",
`${runner}/durable-sessions`,
`${runner}/runner`,
`${runner}/control-plane`,
`${runner}/opencode`,
`${instanceRoot}/data/run-logs`,
`${instanceRoot}/data/provider-traces`,
`${instanceRoot}/data/backups`,
`${instanceRoot}/db`,
`${instanceRoot}/secrets`,
]);
expect(launch.exposed).toEqual([`${runner}/durable-sessions/abc123`, `${runner}/opencode/ses_own_1`, trace]);
expect(launch.exposedFiles).toEqual([trace]);
expect(launch.precreated).toEqual([`${runner}/acpx`, ...["companies", "projects", "workspaces", "runtime-context-assets", "skills", "runtime/provider-packs"].map((name) => `${instanceRoot}/${name}`)]);
expect(launch.required).toBe(true);
});
it("keeps a Codex runner's own state directory and exposes nothing it was not given", () => {
const launch = runnerdLaunchSandbox(config, ["--state-dir", `${runner}/runner/run-9`, "--run-id", "run-9"], {
PAPERCLIP_PROVIDER_TRACE_PATH: `${instanceRoot}/elsewhere/trace.ndjson`,
});
expect(launch.exposed).toEqual([`${runner}/runner/run-9`]);
// A state directory naming the shared root itself, or escaping it, exposes nothing.
expect(runnerdLaunchSandbox(config, ["--state-dir", `${runner}/durable-sessions`], {}).exposed).toEqual([]);
expect(runnerdLaunchSandbox(config, ["--state-dir", `${runner}/durable-sessions/../control-plane`], {}).exposed).toEqual([]);
expect(runnerdLaunchSandbox(config, ["--state-dir", "relative/runner"], {}).exposed).toEqual([]);
});
it("adds nothing without an instance root", () => {
const launch = runnerdLaunchSandbox({ hidden: ["/h"], required: false }, ["--state-dir", `${runner}/durable-sessions/a/runner`], {});
expect(launch).toEqual({ hidden: ["/h"], exposed: [], exposedFiles: [], precreated: [], required: false });
});
it("names the OpenCode session directory as the driver does", () => {
expect(opencodeSessionDirectoryName("ses_1:x/y")).toBe("ses_1_x_y");
expect(opencodeSessionDirectoryName("a".repeat(200))).toBe("a".repeat(120));
expect(opencodeSessionDirectoryName("..")).toBeNull();
const driver = readFileSync(path.join(import.meta.dirname, "../../packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts"), "utf8");
const body = /export function opencodeRuntimeSessionDirectoryName\([^)]*\)[^{]*\{([\s\S]*?)\n\}/.exec(driver)?.[1] ?? "";
expect(body.replace(/\s+/g, "")).toContain('.replace(/[^a-zA-Z0-9._-]/g,"_").slice(0,120)');
});
it("creates the launch's own entries before runnerd starts and passes them as exposures", async () => {
const root = await realpath(await mkdtemp(path.join(os.tmpdir(), "agent-sandbox-run-")));
try {
const instance = path.join(root, "home", "instances", "i");
const paths = instanceRunStatePaths(instance);
const trace = path.join(paths.providerTraces, "t.ndjson");
const calls: unknown[][] = [];
const fake = ((...args: unknown[]) => { calls.push(args); return { pid: 3 }; }) as unknown as typeof import("node:child_process").spawn;
const spawn = wrapRunnerdSpawn(fake, { hidden: ["/work/paperclip/tests"], required: true, instanceRoot: instance }, "/usr/bin/python3");
const state = path.join(paths.durableSessions, "scope", "runner");
spawn("/r/paperclip-runnerd", ["--state-dir", state, "--session-id", "own"], { env: { PAPERCLIP_PROVIDER_TRACE_PATH: trace } });
expect(statSync(path.join(paths.durableSessions, "scope")).isDirectory()).toBe(true);
expect(statSync(path.join(paths.opencode, "own")).mode & 0o777).toBe(0o700);
expect(statSync(trace).isFile()).toBe(true);
expect(existsSync(paths.acpx)).toBe(true);
const args = calls[0]![1] as string[];
expect(args).toEqual(expect.arrayContaining(["--hide", paths.durableSessions, "--expose", path.join(paths.durableSessions, "scope"), "--expose", trace]));
expect(args.slice(args.indexOf("--"))).toEqual(["--", "/r/paperclip-runnerd", "--state-dir", state, "--session-id", "own"]);
// Creating an exposed trace never truncates one the run already wrote.
const launch = runnerdLaunchSandbox({ hidden: [], required: true, instanceRoot: instance }, [], { PAPERCLIP_PROVIDER_TRACE_PATH: trace });
await writeFile(trace, "frame\n");
prepareRunnerdLaunchSandbox(launch);
expect(await readFile(trace, "utf8")).toBe("frame\n");
} finally {
await rm(root, { recursive: true, force: true });
}
});
it("rejects exposures that are missing, outside every hidden path, or that would reveal one", async () => {
const root = await realpath(await mkdtemp(path.join(os.tmpdir(), "agent-sandbox-expose-")));
try {
const hidden = path.join(root, "sessions");
await mkdir(path.join(hidden, "own", "inner"), { recursive: true });
const run = (...args: string[]) => {
try {
execFileSync(pythonExecutable(), ["-I", "-S", agentFilesystemSandboxHelper, ...args, "--", "/bin/true"], { stdio: "pipe" });
return 0;
} catch (error) {
return (error as { status: number }).status;
}
};
expect(run("--hide", hidden, "--expose", path.join(hidden, "missing"))).toBe(2);
expect(run("--hide", hidden, "--expose", root)).toBe(2);
expect(run("--hide", hidden, "--expose", hidden)).toBe(2);
expect(run("--hide", path.join(hidden, "own", "inner"), "--hide", hidden, "--expose", path.join(hidden, "own"))).toBe(2);
expect(run("--hide", hidden, "--expose", "relative")).toBe(2);
} finally {
await rm(root, { recursive: true, force: true });
}
});
});
describe("selected cell outside the command line", () => {
it("reads the selected cell from the environment and removes it", () => {
const env: NodeJS.ProcessEnv = { [RUNNER_SELECTED_ID_ENV]: " chat-features.runner-opencode.local.agents.change-manager " };
const args = selectorArgumentsWithEnvironment([], env);
expect(args).toEqual(["--id", "chat-features.runner-opencode.local.agents.change-manager"]);
expect(env[RUNNER_SELECTED_ID_ENV]).toBeUndefined();
expect(parseRunnerSelectors(args).ids).toEqual(["chat-features.runner-opencode.local.agents.change-manager"]);
expect(selectorArgumentsWithEnvironment(["--", "--id", "x"], {})).toEqual(["--", "--id", "x"]);
});
it("is what CI passes instead of --id", async () => {
for (const workflow of ["runner-full-stack-e2e.yml", "chat-catalog-testops-case.yml"]) {
const source = await readFile(path.join(import.meta.dirname, "../../.github/workflows", workflow), "utf8");
expect(source).toContain(`${RUNNER_SELECTED_ID_ENV}: \${{`);
expect(source).not.toMatch(/test:e2e:runner -- --id "\$\{\{/);
}
});
});
describe("runnerd launch rewriting", () => {
it("wraps only the absolute runnerd executable and execs it with the original argv", () => {
const runnerd = "/work/paperclip/packages/paperclip-runner/runner/target/debug/paperclip-runnerd";
@@ -231,4 +371,23 @@ describe.skipIf(!landlock)("Landlock agent sandbox on this host", () => {
expect(attemptOutput).toContain("workspace=ok");
expect(attemptOutput).toContain("control=blocked");
});
it("keeps a run's own session and trace and denies every other run's state, the database and secrets", async () => {
const root = await realpath(await mkdtemp(path.join(os.tmpdir(), "agent-sandbox-run-scope-")));
roots.push(root);
const home = path.join(root, "attempt", "paperclip-home");
await mkdir(path.join(home, "instances", "i"), { recursive: true });
await mkdir(path.join(root, "attempt", "artifacts-private"));
const config = { hidden: [path.join(root, "attempt", "artifacts-private")], required: true, instanceRoot: path.join(home, "instances", "i") };
const probe = probeRunScopedSandbox(config, pythonExecutable(), process.env);
expect(probe.failures).toEqual([]);
expect(probe.result).toMatchObject({
"run:otherDurableSession": "denied", "run:durableSessionListing": "denied", "run:ownDurableSession": "allowed",
"run:otherOpencodeSession": "denied", "run:ownOpencodeSession": "allowed", "run:otherRunLog": "denied",
"run:databaseWal": "denied", "run:secrets": "denied", "run:controlPlaneState": "denied",
"run:otherProviderTrace": "denied", "run:ownProviderTrace": "allowed", "run:harnessEnviron": "denied",
});
// The fixture instance is removed after the probe.
expect((await readdir(home)).filter((name) => name.startsWith(".agent-sandbox-probe-"))).toEqual([]);
});
});
+228 -7
View File
@@ -1,5 +1,5 @@
import childProcess, { execFileSync, type SpawnOptions } from "node:child_process";
import { accessSync, constants, existsSync, readFileSync, realpathSync, statSync } from "node:fs";
import { accessSync, closeSync, constants, existsSync, mkdirSync, mkdtempSync, openSync, readFileSync, realpathSync, rmSync, statSync, writeFileSync } from "node:fs";
import { syncBuiltinESMExports } from "node:module";
import os from "node:os";
import { promisify } from "node:util";
@@ -23,8 +23,24 @@ export const RUNNERD_EXECUTABLE_NAME = "paperclip-runnerd";
export interface AgentFilesystemSandboxConfig {
hidden: string[];
required: boolean;
/**
* The product instance root. Each runnerd launch then also hides every run's
* saved sessions, run logs, database files and secrets, except its own session.
*/
instanceRoot?: string;
}
/** One runnerd launch's sandbox: the attempt's hidden set plus its run scope. */
export interface RunnerdLaunchSandbox {
hidden: string[];
/** The launch's own entries beneath hidden directories, created before launch. */
exposed: string[];
/** Exposed paths that are files rather than directories. */
exposedFiles: string[];
/** Directories created before launch because their parent is hidden. */
precreated: string[];
required: boolean;
}
function within(candidate: string, root: string) {
return candidate === root || candidate.startsWith(`${root}${path.sep}`);
}
@@ -90,6 +106,7 @@ export function agentFilesystemSandboxConfig(input: {
installedRelease: boolean;
sharedTemporary?: string;
gitStores?: readonly string[];
instanceRoot?: string;
}): AgentFilesystemSandboxConfig | null {
if (!agentFilesystemSandboxApplies(input)) return null;
const sharedTemporary = path.resolve(input.sharedTemporary ?? (input.env.TMPDIR?.trim() || os.tmpdir()));
@@ -125,23 +142,119 @@ export function agentFilesystemSandboxConfig(input: {
if (required) throw new Error(message);
console.warn(`[agent-filesystem-sandbox] ${message}`);
}
return hidden.length ? { hidden, required } : null;
if (!hidden.length) return null;
const instanceRoot = input.instanceRoot?.trim();
// The instance root lives in the attempt root; never in the shared directory.
if (instanceRoot && path.isAbsolute(instanceRoot) && !within(path.resolve(instanceRoot), sharedTemporary)) {
return { hidden, required, instanceRoot: path.resolve(instanceRoot) };
}
return { hidden, required };
}
function flagValue(args: readonly string[], flag: string) {
const index = args.indexOf(flag);
const value = index >= 0 ? args[index + 1] : undefined;
return typeof value === "string" && value.trim() ? value : undefined;
}
/** Mirrors opencodeRuntimeSessionDirectoryName in the OpenCode driver. */
export function opencodeSessionDirectoryName(normalizedSessionId: string) {
const safe = normalizedSessionId.replace(/[^a-zA-Z0-9._-]/g, "_").slice(0, 120);
return safe && safe !== "." && safe !== ".." ? safe : null;
}
/** The attempt instance's directories that hold every run's state. */
export function instanceRunStatePaths(instanceRoot: string) {
const runner = path.join(instanceRoot, "runtime", "paperclip-runner");
return {
durableSessions: path.join(runner, "durable-sessions"),
runnerStates: path.join(runner, "runner"),
controlPlane: path.join(runner, "control-plane"),
opencode: path.join(runner, "opencode"),
acpx: path.join(runner, "acpx"),
runLogs: path.join(instanceRoot, "data", "run-logs"),
providerTraces: path.join(instanceRoot, "data", "provider-traces"),
backups: path.join(instanceRoot, "data", "backups"),
database: path.join(instanceRoot, "db"),
secrets: path.join(instanceRoot, "secrets"),
};
}
/** Instance directories an agent may use, created on first use beside hidden ones. */
const INSTANCE_SHARED_DIRECTORIES = ["companies", "projects", "workspaces", "runtime-context-assets", "skills", "runtime/provider-packs"] as const;
/**
* Selects one runnerd launch's sandbox. Runnerd is launched per session with
* `--state-dir` inside that session's directory, so the launch keeps exactly
* its own durable session, OpenCode session and provider trace and loses every
* other run's.
*/
export function runnerdLaunchSandbox(config: AgentFilesystemSandboxConfig, args: readonly string[], env: NodeJS.ProcessEnv): RunnerdLaunchSandbox {
const launch: RunnerdLaunchSandbox = { hidden: [...config.hidden], exposed: [], exposedFiles: [], precreated: [], required: config.required };
if (!config.instanceRoot) return launch;
const paths = instanceRunStatePaths(config.instanceRoot);
const hide = (value: string) => { if (!launch.hidden.includes(value)) launch.hidden.push(value); };
const expose = (value: string, file = false) => {
if (!launch.exposed.includes(value)) launch.exposed.push(value);
if (file && !launch.exposedFiles.includes(value)) launch.exposedFiles.push(value);
};
for (const value of [paths.durableSessions, paths.runnerStates, paths.controlPlane, paths.opencode, paths.runLogs, paths.providerTraces, paths.backups, paths.database, paths.secrets]) hide(value);
// Hiding those leaves their parents (the instance root, its runtime and data
// directories) without new entries. Create the directories the product makes
// there on first use, so a long-lived runnerd still sees later additions.
launch.precreated.push(paths.acpx, ...INSTANCE_SHARED_DIRECTORIES.map((name) => path.join(config.instanceRoot!, ...name.split("/"))));
const stateDirectory = flagValue(args, "--state-dir");
if (stateDirectory && path.isAbsolute(stateDirectory)) {
const state = path.resolve(stateDirectory);
for (const root of [paths.durableSessions, paths.runnerStates]) {
if (!within(state, root) || state === root) continue;
// The session's own directory: `<root>/<scope>`, whatever lies below it.
expose(path.join(root, path.relative(root, state).split(path.sep)[0]!));
}
}
const sessionId = flagValue(args, "--session-id");
const configuredOpencode = env.PAPERCLIP_OPENCODE_RUNTIME_DIR?.trim();
const opencodeRoots = [paths.opencode, ...(configuredOpencode && path.isAbsolute(configuredOpencode) ? [path.resolve(configuredOpencode)] : [])];
for (const root of opencodeRoots) {
if (root !== paths.opencode && (within(root, paths.durableSessions) || within(root, paths.runnerStates))) continue;
hide(root);
const name = sessionId ? opencodeSessionDirectoryName(sessionId) : null;
if (name) expose(path.join(root, name));
}
const trace = env.PAPERCLIP_PROVIDER_TRACE_PATH?.trim();
if (trace && path.isAbsolute(trace) && path.dirname(path.resolve(trace)) === paths.providerTraces) expose(path.resolve(trace), true);
return launch;
}
/** Creates the launch's own entries, which a process inside the sandbox could not create. */
export function prepareRunnerdLaunchSandbox(launch: RunnerdLaunchSandbox) {
for (const directory of launch.precreated) mkdirSync(directory, { recursive: true, mode: 0o700 });
for (const exposed of launch.exposed) {
if (launch.exposedFiles.includes(exposed)) {
mkdirSync(path.dirname(exposed), { recursive: true, mode: 0o700 });
// Append mode never truncates a trace the run already started.
closeSync(openSync(exposed, "a", 0o600));
} else {
mkdirSync(exposed, { recursive: true, mode: 0o700 });
}
}
}
export function parseAgentFilesystemSandboxConfig(value: string | undefined): AgentFilesystemSandboxConfig | null {
if (!value?.trim()) return null;
const parsed = JSON.parse(value) as Partial<AgentFilesystemSandboxConfig>;
if (!Array.isArray(parsed.hidden) || parsed.hidden.some((entry) => typeof entry !== "string" || !path.isAbsolute(entry)) || typeof parsed.required !== "boolean") {
if (!Array.isArray(parsed.hidden) || parsed.hidden.some((entry) => typeof entry !== "string" || !path.isAbsolute(entry)) || typeof parsed.required !== "boolean"
|| (parsed.instanceRoot !== undefined && (typeof parsed.instanceRoot !== "string" || !path.isAbsolute(parsed.instanceRoot)))) {
throw new Error(`${AGENT_FILESYSTEM_SANDBOX_ENV} is malformed`);
}
return { hidden: parsed.hidden, required: parsed.required };
return { hidden: parsed.hidden, required: parsed.required, ...(parsed.instanceRoot ? { instanceRoot: parsed.instanceRoot } : {}) };
}
export const agentFilesystemSandboxHelper = path.join(import.meta.dirname, "agent-filesystem-sandbox.py");
/** Rewrites one launch so the helper applies the domain and then execs it unchanged. */
export function sandboxedInvocation(
config: AgentFilesystemSandboxConfig,
config: Pick<AgentFilesystemSandboxConfig, "hidden" | "required"> & { exposed?: readonly string[] },
python: string,
command: string,
args: readonly string[],
@@ -156,6 +269,7 @@ export function sandboxedInvocation(
...(config.required ? ["--required"] : []),
...locale,
...config.hidden.flatMap((hidden) => ["--hide", hidden]),
...(config.exposed ?? []).flatMap((exposed) => ["--expose", exposed]),
"--", command, ...args,
],
};
@@ -239,6 +353,105 @@ export function probeAgentFilesystemSandbox(
return { result, failures };
}
/** What a sandboxed run must be able to do with each probe fixture path. */
const RUN_SCOPE_EXPECTATIONS = {
otherDurableSession: "denied",
durableSessionListing: "denied",
ownDurableSession: "allowed",
otherRunnerState: "denied",
controlPlaneState: "denied",
otherOpencodeSession: "denied",
ownOpencodeSession: "allowed",
acpxRuntime: "allowed",
otherRunLog: "denied",
databaseWal: "denied",
secrets: "denied",
otherProviderTrace: "denied",
ownProviderTrace: "allowed",
harnessEnviron: "denied",
} as const;
/**
* Proves the run scope on a fixture instance laid out like the real one: a
* sandboxed run keeps its own session and trace and loses every other run's
* state, the instance database and its secrets.
*/
export function probeRunScopedSandbox(
config: AgentFilesystemSandboxConfig,
python = pythonExecutable(),
env: NodeJS.ProcessEnv = process.env,
harnessPid = process.pid,
) {
if (!config.instanceRoot) return { result: {} as Record<string, string>, failures: [] as string[] };
// A sibling of the instances directory: inside the attempt root, never in /tmp.
const fixture = mkdtempSync(path.join(path.dirname(path.dirname(config.instanceRoot)), ".agent-sandbox-probe-"));
try {
const instance = path.join(fixture, "instance");
const paths = instanceRunStatePaths(instance);
const write = (file: string) => { mkdirSync(path.dirname(file), { recursive: true, mode: 0o700 }); writeFileSync(file, "other run\n", { mode: 0o600 }); };
const ownState = path.join(paths.durableSessions, "own-scope", "runner");
mkdirSync(ownState, { recursive: true, mode: 0o700 });
const fixtures = {
otherDurableSession: path.join(paths.durableSessions, "other-scope", "opencode", "data", "opencode.db"),
otherRunnerState: path.join(paths.runnerStates, "other-run", "state.json"),
controlPlaneState: path.join(paths.controlPlane, "control-plane-state.json"),
otherOpencodeSession: path.join(paths.opencode, "other-session", "data", "opencode", "opencode.db"),
otherRunLog: path.join(paths.runLogs, "company", "agent", "other-run.ndjson"),
databaseWal: path.join(paths.database, "pg_wal", "000000010000000000000001"),
secrets: path.join(paths.secrets, "master.key"),
otherProviderTrace: path.join(paths.providerTraces, "other-run.ndjson"),
};
for (const file of Object.values(fixtures)) write(file);
const ownTrace = path.join(paths.providerTraces, "own-run.ndjson");
const launch = runnerdLaunchSandbox({ ...config, instanceRoot: instance },
["--state-dir", ownState, "--session-id", "own-session"], { PAPERCLIP_PROVIDER_TRACE_PATH: ownTrace });
prepareRunnerdLaunchSandbox(launch);
const checks = [
["otherDurableSession", "read", fixtures.otherDurableSession],
["durableSessionListing", "list", paths.durableSessions],
["ownDurableSession", "write", ownState],
["otherRunnerState", "read", fixtures.otherRunnerState],
["controlPlaneState", "read", fixtures.controlPlaneState],
["otherOpencodeSession", "read", fixtures.otherOpencodeSession],
["ownOpencodeSession", "write", path.join(paths.opencode, "own-session")],
["acpxRuntime", "write", paths.acpx],
["otherRunLog", "read", fixtures.otherRunLog],
["databaseWal", "list", path.dirname(fixtures.databaseWal)],
["secrets", "read", fixtures.secrets],
["otherProviderTrace", "read", fixtures.otherProviderTrace],
["ownProviderTrace", "append", ownTrace],
["harnessEnviron", "read", `/proc/${harnessPid}/environ`],
// Recorded, not required: Landlock cannot hide another process's argv.
["harnessCmdline", "read", `/proc/${harnessPid}/cmdline`],
] as const;
const script = [
'out=""',
'for spec in "$@"; do name=${spec%%|*}; rest=${spec#*|}; mode=${rest%%|*}; target=${rest#*|}',
' case "$mode" in',
' read) cat "$target" >/dev/null 2>&1 ;;',
' list) ls "$target" >/dev/null 2>&1 ;;',
' write) f="$target/.agent-sandbox-probe" && : > "$f" 2>/dev/null && cat "$f" >/dev/null 2>&1 && rm -f "$f" ;;',
' append) printf "" >> "$target" 2>/dev/null && cat "$target" >/dev/null 2>&1 ;;',
' esac',
' if [ $? -eq 0 ]; then out="$out$name=allowed;"; else out="$out$name=denied;"; fi',
'done',
'printf "%s\\n" "$out"',
].join("\n");
const invocation = sandboxedInvocation(launch, python, "/bin/sh", ["-c", script, "sh", ...checks.map(([name, mode, target]) => `${name}|${mode}|${target}`)], env);
const output = execFileSync(invocation.command, invocation.args, { encoding: "utf8", cwd: "/", env, timeout: 30_000 });
const result = Object.fromEntries(output.trim().split(";").filter(Boolean).map((entry) => {
const separator = entry.lastIndexOf("=");
return [`run:${entry.slice(0, separator)}`, entry.slice(separator + 1)];
})) as Record<string, string>;
const failures = Object.entries(RUN_SCOPE_EXPECTATIONS)
.filter(([name, expected]) => result[`run:${name}`] !== expected)
.map(([name, expected]) => `run scope: ${name} is ${result[`run:${name}`] ?? "unprobed"}, expected ${expected}`);
return { result, failures };
} finally {
rmSync(fixture, { recursive: true, force: true });
}
}
/**
* Wraps runnerd launches made by this process. Runs only in the E2E server
* entry; the wrapper execs runnerd, so its PID, argv, cwd and environment are
@@ -252,6 +465,8 @@ export function installAgentFilesystemSandbox(repositoryRoot: string, env: NodeJ
let probe: ReturnType<typeof probeAgentFilesystemSandbox>;
try {
probe = probeAgentFilesystemSandbox(config, repositoryRoot, python);
const runScope = probeRunScopedSandbox(config, python);
probe = { result: { ...probe.result, ...runScope.result }, failures: [...probe.failures, ...runScope.failures] };
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
if (config.required) throw new Error(`agent_filesystem_sandbox_unavailable: ${message}`);
@@ -286,10 +501,16 @@ export function wrapRunnerdLaunch<T extends (...args: any[]) => unknown>(origina
const afterArgs = hasArgs ? rest.slice(1) : rest;
const options = afterArgs[0] && typeof afterArgs[0] === "object" ? afterArgs[0] as SpawnOptions : undefined;
if (options?.shell) throw new Error("agent_filesystem_sandbox: refusing to start runnerd through a shell, outside the sandbox");
const invocation = sandboxedInvocation(config, python, command as string, args, options?.env ?? process.env);
const env = options?.env ?? process.env;
const launch = runnerdLaunchSandbox(config, args, env);
// The server creates the launch's own entries; inside the sandbox their
// hidden parents accept no new entries. A failure here fails the launch.
prepareRunnerdLaunchSandbox(launch);
const invocation = sandboxedInvocation(launch, python, command as string, args, env);
const result = Reflect.apply(original, this, [invocation.command, invocation.args, ...afterArgs]);
const pid = (result as { pid?: number } | undefined)?.pid;
console.log(`[agent-filesystem-sandbox] runnerd launched inside the sandbox via ${api}${pid ? ` (pid ${pid})` : ""}`);
const scope = config.instanceRoot ? ` ${JSON.stringify({ runScopedHidden: launch.hidden.filter((value) => !config.hidden.includes(value)), exposed: launch.exposed })}` : "";
console.log(`[agent-filesystem-sandbox] runnerd launched inside the sandbox via ${api}${pid ? ` (pid ${pid})` : ""}${scope}`);
return result;
} as T;
// execFile's promisified form resolves { stdout, stderr } through this hook;
+4 -2
View File
@@ -70,6 +70,7 @@ import {
buildMatrixJobs,
parseRunnerSelectors,
RunnerSelectorError,
selectorArgumentsWithEnvironment,
selectRunnerExecutions,
} from "./selectors.js";
import { resolveRunnerE2ESource } from "./source.js";
@@ -562,6 +563,7 @@ async function runAttempt(input: {
const agentFilesystemSandbox = agentFilesystemSandboxConfig({
platform: process.platform, env: process.env, executions, repositoryRoot, temporaryRoot, privateDir,
evidenceRoot: manualPrivateHandoff?.root, installedRelease: Boolean(installedCli),
instanceRoot: path.join(paperclipHome, "instances", instanceId),
});
// Playwright's default cache holds compiled graders in the shared /tmp.
if (sandboxedAgents) await rm(defaultPlaywrightCacheDirectory(), { recursive: true, force: true });
@@ -1069,14 +1071,14 @@ async function runWithConcurrency<T, R>(
}
async function main() {
const args = process.argv.slice(2);
const args = selectorArgumentsWithEnvironment(process.argv.slice(2));
if (args.includes("--target-config") || args.includes("product-journeys")) {
const { launchProductJourneys } = await import("./product-journeys/launch.js");
return launchProductJourneys(args, repositoryRoot);
}
let options: ReturnType<typeof parseRunnerSelectors>;
try {
options = parseRunnerSelectors(process.argv.slice(2));
options = parseRunnerSelectors(args);
} catch (error) {
if (error instanceof RunnerSelectorError)
throw new Error(`${error.message}\nUse --list to inspect valid cells.`);
+13
View File
@@ -27,6 +27,19 @@ export interface RunnerSelectorOptions {
export class RunnerSelectorError extends Error {}
/**
* CI passes the selected cell here rather than as `--id`. Every agent can read
* another process's argv in /proc, and the cell id names the case under test.
*/
export const RUNNER_SELECTED_ID_ENV = "PAPERCLIP_RUNNER_E2E_SELECTED_ID";
/** Adds the environment's selected cell to the arguments and removes it from the environment. */
export function selectorArgumentsWithEnvironment(args: readonly string[], env: NodeJS.ProcessEnv = process.env): string[] {
const selected = env[RUNNER_SELECTED_ID_ENV]?.trim();
delete env[RUNNER_SELECTED_ID_ENV];
return selected ? [...args, "--id", selected] : [...args];
}
function valueFor(args: string[], index: number, flag: string): string {
const value = args[index + 1];
if (!value || value.startsWith("--"))