docs(runner-e2e): describe the agent view of the eval harness

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 committed 2026-11-08 06:41:43 -06:00
1 parent b8ccd434ac
commit 79f71ae908
1 file changed
+35
+35
View File
@@ -323,6 +323,41 @@ checks it out only after authorization. Target-controlled code cannot replace
the report sanitizer or the AWS history publisher. Fork refs and
target-controlled workflow definitions do not enter this path.
## Agent view of the eval harness
The agent under test must not read the eval. A local cell runs the product from
the checkout that also holds `tests/`, as the same OS user as the harness, so
file modes cannot separate them. OpenCode's `external_directory` policy guards
its own read and grep tools, but not arbitrary shell commands.
`agent-filesystem-sandbox.ts` closes that gap for local native cells on Linux.
The launcher selects the hidden paths: `tests/`, the trusted
`.manual-private-trusted/` copy, the private evidence directory and
`RUNNER_TEMP`. The E2E server entry starts each `paperclip-runnerd` through
`agent-filesystem-sandbox.py`, which applies a Landlock domain and then execs
runnerd with its original PID, argv, cwd and environment. Runnerd, the provider,
its tools and every shell command inherit the domain. The domain also sets
`no_new_privs`, so `sudo` cannot widen it. The product server itself stays
unrestricted because it loads its E2E entry from the harness.
Before the product starts an agent, a probe runs a sandboxed shell. The probe
must be denied the harness and every hidden path, and it must read the product
package and write a temporary file. The server log records the probe result and
each wrapped runnerd launch. In CI a missing or ineffective sandbox stops the
attempt. Landlock does not hide that a path exists (`stat` still works); it
denies reading and listing.
Limits: Codex cells keep Codex's own bubblewrap sandbox, which cannot mount
inside Landlock. Legacy adapters, installed-release launches and macOS run
without this sandbox. A process that can reach the Docker daemon can still
mount the checkout into a container, and `ps` shows the launcher's command
line. Daytona cells run the agent in a remote sandbox that holds only the
staged runnerd and provider pack, not the checkout.
Fixture names that an agent can read (company, agent, environment, secret,
connection and project) use a neutral word and a digest suffix. They do not
contain case ids, suite names, "E2E" or "test".
## AWS OIDC and S3
The AWS role trust policy should accept only GitHub's OIDC audience and the