From 79f71ae90875be690ad15c689f4eb4c48c58e92b Mon Sep 17 00:00:00 2001 From: Dotta Date: Sun, 8 Nov 2026 06:27:52 -0600 Subject: [PATCH] docs(runner-e2e): describe the agent view of the eval harness Co-Authored-By: Claude Opus 5.5 --- tests/runner-e2e/SECURITY.md | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/tests/runner-e2e/SECURITY.md b/tests/runner-e2e/SECURITY.md index bc53ef3fc36..151a024ed47 100644 --- a/tests/runner-e2e/SECURITY.md +++ b/tests/runner-e2e/SECURITY.md @@ -323,6 +323,41 @@ checks it out only after authorization. Target-controlled code cannot replace the report sanitizer or the AWS history publisher. Fork refs and target-controlled workflow definitions do not enter this path. +## Agent view of the eval harness + +The agent under test must not read the eval. A local cell runs the product from +the checkout that also holds `tests/`, as the same OS user as the harness, so +file modes cannot separate them. OpenCode's `external_directory` policy guards +its own read and grep tools, but not arbitrary shell commands. + +`agent-filesystem-sandbox.ts` closes that gap for local native cells on Linux. +The launcher selects the hidden paths: `tests/`, the trusted +`.manual-private-trusted/` copy, the private evidence directory and +`RUNNER_TEMP`. The E2E server entry starts each `paperclip-runnerd` through +`agent-filesystem-sandbox.py`, which applies a Landlock domain and then execs +runnerd with its original PID, argv, cwd and environment. Runnerd, the provider, +its tools and every shell command inherit the domain. The domain also sets +`no_new_privs`, so `sudo` cannot widen it. The product server itself stays +unrestricted because it loads its E2E entry from the harness. + +Before the product starts an agent, a probe runs a sandboxed shell. The probe +must be denied the harness and every hidden path, and it must read the product +package and write a temporary file. The server log records the probe result and +each wrapped runnerd launch. In CI a missing or ineffective sandbox stops the +attempt. Landlock does not hide that a path exists (`stat` still works); it +denies reading and listing. + +Limits: Codex cells keep Codex's own bubblewrap sandbox, which cannot mount +inside Landlock. Legacy adapters, installed-release launches and macOS run +without this sandbox. A process that can reach the Docker daemon can still +mount the checkout into a container, and `ps` shows the launcher's command +line. Daytona cells run the agent in a remote sandbox that holds only the +staged runnerd and provider pack, not the checkout. + +Fixture names that an agent can read (company, agent, environment, secret, +connection and project) use a neutral word and a digest suffix. They do not +contain case ids, suite names, "E2E" or "test". + ## AWS OIDC and S3 The AWS role trust policy should accept only GitHub's OIDC audience and the