Files
PaperClipAI/tests/runner-e2e/manual-dispatch-authority.test.ts
T
DottaandClaude Opus 5.5 b7e28eb2f7 fix(evals): let an allowlisted manual dispatch authorize its private evidence (#19599)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat-feature evals run in the trusted `runner-full-stack-e2e`
workflow and keep their evidence in private history
> - A manual chat run needs authority to write that private evidence
> - Today that authority is a stored approval for the exact run ID in
`RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON`
> - The run ID exists only after dispatch, so every manual run stops
until a person writes the record
> - The dispatcher is already checked against
`RUNNER_E2E_ALLOWED_ACTOR_IDS`, so the second step adds friction without
a new decision
> - This pull request lets an allowlisted manual dispatch authorize its
own private evidence, as the TestOps admission already does
> - The benefit is that an allowlisted operator can start a chat eval
and it runs, while scheduled runs still never touch chat evidence

## Linked Issues or Issue Description

Refs #18134, Refs #18170

**What existing behavior does this improve?**
A manual `runner-full-stack-e2e` run with chat-feature IDs fails at
private evidence transfer unless a person first adds an approval row for
that run ID to a repository variable.

**Subsystem affected**
Runner E2E manual workflow private evidence authorization
(`manual-private-transfer.ts`, `runner-full-stack-e2e.yml`).

**Proposed change**
Derive the evidence authority from the dispatch itself, with the same
checks the TestOps chat admission uses.

## What Changed

- New `tests/runner-e2e/manual-dispatch-authority.ts`:
`authorizeManualDispatch` admits private evidence only when all of these
are true:
- The event is `workflow_dispatch` of `runner-full-stack-e2e.yml` on the
default branch, with the `manual` cadence.
  - The run attempt is `1`.
- The actor is in `RUNNER_E2E_ALLOWED_ACTOR_IDS`, the event sender is
that actor, and the triggering actor is the same person.
  - The trusted cell selection equals the dispatch's own `id` input.
- The authority binds the workflow SHA, the target SHA from the
authorize job, the run ID and the cells. Every job derives the same
identity hash.
- `manual-private-transfer.ts`: uses the TestOps authority first, then
the dispatch authority, then the stored approval. With the dispatch
authority, `prepare` records the target launcher digest instead of
comparing it with an approved digest.
- `runner-full-stack-e2e.yml`: the 14 steps that read
`RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON` now set
`PAPERCLIP_MANUAL_DISPATCH_AUTHORITY: v1` and pass
`RUNNER_E2E_ALLOWED_ACTOR_IDS`.
- `doc/manual-e2e-private-evidence.md`: describes the dispatch
authority. The stored-approval shape stays documented for the TestOps
catalog case workflow.
- The new file is separate on purpose. `manual-private-retention.ts` and
the other protocol files must match each target byte for byte, so
existing candidate branches keep working.

## Verification

- `vitest run tests/runner-e2e/manual-dispatch-authority.test.ts
tests/runner-e2e/manual-private-transfer.test.ts
tests/runner-e2e/manual-private-report.test.ts
tests/runner-e2e/manual-private-report-integration.test.ts
tests/runner-e2e/manual-private-policy.test.ts`: 57 tests passed.
- New tests cover one valid dispatch and rejection of: a scheduled run,
a bounded cadence, a non-allowlisted actor, an empty allowlist, a rerun
by another person, a second attempt, a non-default or different workflow
ref, a sender who is not the actor, a missing target SHA, a missing
authority switch, and cells that differ from the dispatch input. The
eight-cell limit still applies.
- A `prepare` test runs the real transfer script: a non-allowlisted
actor produces no outputs, and an allowlisted dispatch produces the
session policy and an identity that records the target launcher digest.
- Negative control: with the previous `manual-private-transfer.ts`, the
new `prepare` test fails.
- Workflow tests that read this workflow: 111 of 116 passed. The 5
failures in `workflow-target.test.ts` (`IMAGE_ONLY: unbound variable`)
also fail on unchanged `master`. `paid-runner-routing.test.mjs` passed
(7 tests).
- I did not run a live dispatch. The first live proof is a manual
dispatch of chat IDs after merge.

## Risks

- The dispatcher now vouches for the target branch. A target branch with
a changed launcher can write private evidence if an allowlisted operator
dispatches it. The protocol files must still match the trusted copies
byte for byte.
- The eight-cell limit is unchanged. The report job has eight fixed read
slots, and each read policy must fit the 2,048-character session policy
limit. Lifting the limit needs a separate change to the report sessions.
- The `runner-e2e-paid` environment still has required reviewers. That
is a repository setting and this change does not touch it.
- The authority uses `RUNNER_E2E_ALLOWED_ACTOR_IDS`, not `CODEOWNERS`.
Keep that variable aligned with the people who may start paid runs.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 14:26:31 -06:00

91 lines
6.4 KiB
TypeScript

import { spawnSync } from "node:child_process";
import { createRequire } from "node:module";
import { copyFile, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it } from "vitest";
import { authorizeManualDispatch } from "./manual-dispatch-authority.js";
const ids = ["activity.agent-feedback", "agents.choose-primary"].map(id => `chat-features.runner-opencode.local.${id}`);
const workflowSha = "a".repeat(40), targetSha = "b".repeat(40);
const env = (overrides: Record<string, string | undefined> = {}): NodeJS.ProcessEnv => ({
PAPERCLIP_MANUAL_DISPATCH_AUTHORITY: "v1", GITHUB_ACTIONS: "true", GITHUB_EVENT_NAME: "workflow_dispatch",
GITHUB_REPOSITORY: "paperclipai/paperclip", GITHUB_REF: "refs/heads/master",
GITHUB_WORKFLOW_REF: "paperclipai/paperclip/.github/workflows/runner-full-stack-e2e.yml@refs/heads/master",
GITHUB_SHA: workflowSha, TARGET_SHA: targetSha, GITHUB_RUN_ID: "123", GITHUB_RUN_ATTEMPT: "1",
GITHUB_ACTOR: "owner", GITHUB_TRIGGERING_ACTOR: "owner", GITHUB_ACTOR_ID: "42",
MANUAL_DISPATCH_ALLOWED_ACTOR_IDS: "[42,7]", MANUAL_PRIVATE_IDS: JSON.stringify(ids), ...overrides,
});
const event = (overrides: Record<string, unknown> = {}, inputs: Record<string, unknown> = {}) => ({
ref: "refs/heads/master", sender: { id: 42 }, repository: { full_name: "paperclipai/paperclip", default_branch: "master" },
inputs: { cadence: "manual", target_branch: "candidate", id: ids.join(","), ...inputs }, ...overrides,
});
describe("authorizeManualDispatch", () => {
it("authorizes an allowlisted operator's manual dispatch for exactly its own cells", () => {
const authority = authorizeManualDispatch(env(), event());
expect(authority).toMatchObject({ workflowSha, targetSha, runId: "123", attempt: "1", launchBinding: "dispatcher", private: true });
expect(authority.executionIds).toHaveLength(2);
expect(authority.approvalSha256).toMatch(/^[a-f0-9]{64}$/);
// Every job of the run derives the same identity.
expect(authorizeManualDispatch(env(), event()).approvalSha256).toBe(authority.approvalSha256);
});
it.each([
["a scheduled run", env({ GITHUB_EVENT_NAME: "schedule" }), event()],
["a bounded cadence", env(), event({}, { cadence: "daily" })],
["an operator outside the allowlist", env({ GITHUB_ACTOR_ID: "99" }), event({ sender: { id: 99 } })],
["an empty allowlist", env({ MANUAL_DISPATCH_ALLOWED_ACTOR_IDS: "[]" }), event()],
["a rerun by another person", env({ GITHUB_TRIGGERING_ACTOR: "someone-else" }), event()],
["a second attempt", env({ GITHUB_RUN_ATTEMPT: "2" }), event()],
["a non-default workflow ref", env({ GITHUB_WORKFLOW_REF: "paperclipai/paperclip/.github/workflows/runner-full-stack-e2e.yml@refs/heads/feature" }), event()],
["a different workflow", env({ GITHUB_WORKFLOW_REF: "paperclipai/paperclip/.github/workflows/other.yml@refs/heads/master" }), event()],
["a sender who is not the actor", env(), event({ sender: { id: 7 } })],
["a missing target commit", env({ TARGET_SHA: "" }), event()],
["the authority switch unset", env({ PAPERCLIP_MANUAL_DISPATCH_AUTHORITY: undefined }), event()],
])("rejects %s", (_label, environment, payload) => {
expect(() => authorizeManualDispatch(environment, payload)).toThrow();
});
it("rejects cells that differ from the dispatch's own selection", () => {
expect(() => authorizeManualDispatch(env(), event({}, { id: ids[0] }))).toThrow(/differ/);
expect(() => authorizeManualDispatch(env({ MANUAL_PRIVATE_IDS: JSON.stringify([ids[0]]) }), event())).toThrow(/differ/);
});
it("keeps the per-run cell limit", () => {
const many = ["activity.agent-feedback", "activity.work-timeline", "companies.import-preview", "navigation-preferences.dismiss-announcement",
"tasks.completion-with-gate", "tasks.private-task", "secrets.personal-secret", "agents.choose-primary", "access.cli-authorization"]
.map(id => `chat-features.runner-opencode.local.${id}`);
expect(() => authorizeManualDispatch(env({ MANUAL_PRIVATE_IDS: JSON.stringify(many) }), event({}, { id: many.join(",") }))).toThrow(/limit/);
});
});
const roots: string[] = [];
afterEach(async () => { await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); });
it("actual prepare admits a dispatched cell without a stored approval and records the target launcher", async () => {
const root = await mkdtemp(path.join(os.tmpdir(), "synthetic-dispatch-prepare-")); roots.push(root);
const target = path.join(root, "target"), modules = path.join(target, "tests/runner-e2e"); await mkdir(modules, { recursive: true });
await writeFile(path.join(modules, "launch.ts"), "synthetic candidate launcher; never executed");
for (const name of ["manual-private-handoff.ts", "manual-private-retention.ts", "manual-private-report-assets.ts"]) await copyFile(path.join(import.meta.dirname, name), path.join(modules, name));
const eventPath = path.join(root, "event.json"), output = path.join(root, "outputs.txt");
await writeFile(eventPath, JSON.stringify(event()));
const trusted = path.resolve(import.meta.dirname, "../.."), require = createRequire(import.meta.url), tsx = require.resolve("tsx", { paths: [path.join(trusted, "server")] });
const run = (extra: Record<string, string>) => spawnSync(process.execPath, ["--import", tsx, path.join(import.meta.dirname, "manual-private-transfer.ts"), "prepare"], {
encoding: "utf8", timeout: 10000,
env: { PATH: process.env.PATH, NODE_OPTIONS: "", NODE_PATH: "", ...env(), SELECTED_EXECUTION_ID: ids[0]!, GITHUB_EVENT_PATH: eventPath,
TARGET_ROOT: target, RUNNER_TEMP: root, RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: "private-bucket", GITHUB_OUTPUT: output, ...extra } as NodeJS.ProcessEnv,
});
const denied = run({ GITHUB_ACTOR_ID: "99" });
expect(denied.status).toBe(1);
await expect(readFile(output)).rejects.toMatchObject({ code: "ENOENT" });
const admitted = run({});
expect(admitted.error).toBeUndefined(); expect(admitted.status).toBe(0);
const outputs = await readFile(output, "utf8");
expect(outputs).toContain("session_policy=");
const identityRoot = outputs.split("\n").find(line => line.startsWith("root="))!.slice(5);
const identity = JSON.parse(await readFile(path.join(identityRoot, "identity.json"), "utf8"));
expect(identity).toMatchObject({ launchBinding: "dispatcher", runId: "123", targetSha, executionIds: [ids[0]] });
expect(identity.launchSha256).toMatch(/^[a-f0-9]{64}$/);
});