mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 07:10:34 +01:00
fix(evals): let an allowlisted manual dispatch authorize its private evidence (#19599)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Chat-feature evals run in the trusted `runner-full-stack-e2e` workflow and keep their evidence in private history > - A manual chat run needs authority to write that private evidence > - Today that authority is a stored approval for the exact run ID in `RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON` > - The run ID exists only after dispatch, so every manual run stops until a person writes the record > - The dispatcher is already checked against `RUNNER_E2E_ALLOWED_ACTOR_IDS`, so the second step adds friction without a new decision > - This pull request lets an allowlisted manual dispatch authorize its own private evidence, as the TestOps admission already does > - The benefit is that an allowlisted operator can start a chat eval and it runs, while scheduled runs still never touch chat evidence ## Linked Issues or Issue Description Refs #18134, Refs #18170 **What existing behavior does this improve?** A manual `runner-full-stack-e2e` run with chat-feature IDs fails at private evidence transfer unless a person first adds an approval row for that run ID to a repository variable. **Subsystem affected** Runner E2E manual workflow private evidence authorization (`manual-private-transfer.ts`, `runner-full-stack-e2e.yml`). **Proposed change** Derive the evidence authority from the dispatch itself, with the same checks the TestOps chat admission uses. ## What Changed - New `tests/runner-e2e/manual-dispatch-authority.ts`: `authorizeManualDispatch` admits private evidence only when all of these are true: - The event is `workflow_dispatch` of `runner-full-stack-e2e.yml` on the default branch, with the `manual` cadence. - The run attempt is `1`. - The actor is in `RUNNER_E2E_ALLOWED_ACTOR_IDS`, the event sender is that actor, and the triggering actor is the same person. - The trusted cell selection equals the dispatch's own `id` input. - The authority binds the workflow SHA, the target SHA from the authorize job, the run ID and the cells. Every job derives the same identity hash. - `manual-private-transfer.ts`: uses the TestOps authority first, then the dispatch authority, then the stored approval. With the dispatch authority, `prepare` records the target launcher digest instead of comparing it with an approved digest. - `runner-full-stack-e2e.yml`: the 14 steps that read `RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON` now set `PAPERCLIP_MANUAL_DISPATCH_AUTHORITY: v1` and pass `RUNNER_E2E_ALLOWED_ACTOR_IDS`. - `doc/manual-e2e-private-evidence.md`: describes the dispatch authority. The stored-approval shape stays documented for the TestOps catalog case workflow. - The new file is separate on purpose. `manual-private-retention.ts` and the other protocol files must match each target byte for byte, so existing candidate branches keep working. ## Verification - `vitest run tests/runner-e2e/manual-dispatch-authority.test.ts tests/runner-e2e/manual-private-transfer.test.ts tests/runner-e2e/manual-private-report.test.ts tests/runner-e2e/manual-private-report-integration.test.ts tests/runner-e2e/manual-private-policy.test.ts`: 57 tests passed. - New tests cover one valid dispatch and rejection of: a scheduled run, a bounded cadence, a non-allowlisted actor, an empty allowlist, a rerun by another person, a second attempt, a non-default or different workflow ref, a sender who is not the actor, a missing target SHA, a missing authority switch, and cells that differ from the dispatch input. The eight-cell limit still applies. - A `prepare` test runs the real transfer script: a non-allowlisted actor produces no outputs, and an allowlisted dispatch produces the session policy and an identity that records the target launcher digest. - Negative control: with the previous `manual-private-transfer.ts`, the new `prepare` test fails. - Workflow tests that read this workflow: 111 of 116 passed. The 5 failures in `workflow-target.test.ts` (`IMAGE_ONLY: unbound variable`) also fail on unchanged `master`. `paid-runner-routing.test.mjs` passed (7 tests). - I did not run a live dispatch. The first live proof is a manual dispatch of chat IDs after merge. ## Risks - The dispatcher now vouches for the target branch. A target branch with a changed launcher can write private evidence if an allowlisted operator dispatches it. The protocol files must still match the trusted copies byte for byte. - The eight-cell limit is unchanged. The report job has eight fixed read slots, and each read policy must fit the 2,048-character session policy limit. Lifting the limit needs a separate change to the report sessions. - The `runner-e2e-paid` environment still has required reviewers. That is a repository setting and this change does not touch it. - The authority uses `RUNNER_E2E_ALLOWED_ACTOR_IDS`, not `CODEOWNERS`. Keep that variable aligned with the people who may start paid runs. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
b55d8a9201
commit
b7e28eb2f7
5 files changed
+217
-41
No files matched your search
@@ -1101,7 +1101,8 @@ jobs:
|
||||
working-directory: .manual-private-trusted
|
||||
env:
|
||||
MANUAL_PRIVATE_IDS: ${{ needs.authorize.outputs.manual_private_ids }}
|
||||
MANUAL_PRIVATE_APPROVALS_JSON: ${{ vars.RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON }}
|
||||
PAPERCLIP_MANUAL_DISPATCH_AUTHORITY: v1
|
||||
MANUAL_DISPATCH_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
|
||||
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
|
||||
TARGET_ROOT: ${{ github.workspace }}
|
||||
SELECTED_EXECUTION_ID: ${{ matrix.executionId }}
|
||||
@@ -1130,7 +1131,8 @@ jobs:
|
||||
working-directory: .manual-private-trusted
|
||||
env:
|
||||
MANUAL_PRIVATE_IDS: ${{ needs.authorize.outputs.manual_private_ids }}
|
||||
MANUAL_PRIVATE_APPROVALS_JSON: ${{ vars.RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON }}
|
||||
PAPERCLIP_MANUAL_DISPATCH_AUTHORITY: v1
|
||||
MANUAL_DISPATCH_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
|
||||
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
|
||||
TARGET_ROOT: ${{ github.workspace }}
|
||||
SELECTED_EXECUTION_ID: ${{ matrix.executionId }}
|
||||
@@ -1456,7 +1458,8 @@ jobs:
|
||||
working-directory: .manual-private-trusted
|
||||
env:
|
||||
MANUAL_PRIVATE_IDS: ${{ needs.authorize.outputs.manual_private_ids }}
|
||||
MANUAL_PRIVATE_APPROVALS_JSON: ${{ vars.RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON }}
|
||||
PAPERCLIP_MANUAL_DISPATCH_AUTHORITY: v1
|
||||
MANUAL_DISPATCH_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
|
||||
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
|
||||
TARGET_ROOT: ${{ github.workspace }}
|
||||
SELECTED_EXECUTION_ID: ${{ matrix.executionId }}
|
||||
@@ -1557,7 +1560,8 @@ jobs:
|
||||
if: needs.authorize.outputs.evidence_retention == 'private'
|
||||
env:
|
||||
MANUAL_PRIVATE_IDS: ${{ needs.authorize.outputs.manual_private_ids }}
|
||||
MANUAL_PRIVATE_APPROVALS_JSON: ${{ vars.RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON }}
|
||||
PAPERCLIP_MANUAL_DISPATCH_AUTHORITY: v1
|
||||
MANUAL_DISPATCH_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
|
||||
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
|
||||
RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET }}
|
||||
NODE_OPTIONS: ''
|
||||
@@ -1595,7 +1599,8 @@ jobs:
|
||||
if: steps.manual_read_credentials_0.outcome == 'success'
|
||||
env:
|
||||
MANUAL_PRIVATE_IDS: ${{ needs.authorize.outputs.manual_private_ids }}
|
||||
MANUAL_PRIVATE_APPROVALS_JSON: ${{ vars.RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON }}
|
||||
PAPERCLIP_MANUAL_DISPATCH_AUTHORITY: v1
|
||||
MANUAL_DISPATCH_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
|
||||
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
|
||||
RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET }}
|
||||
NODE_OPTIONS: ''
|
||||
@@ -1626,7 +1631,8 @@ jobs:
|
||||
if: steps.manual_read_credentials_1.outcome == 'success'
|
||||
env:
|
||||
MANUAL_PRIVATE_IDS: ${{ needs.authorize.outputs.manual_private_ids }}
|
||||
MANUAL_PRIVATE_APPROVALS_JSON: ${{ vars.RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON }}
|
||||
PAPERCLIP_MANUAL_DISPATCH_AUTHORITY: v1
|
||||
MANUAL_DISPATCH_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
|
||||
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
|
||||
RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET }}
|
||||
NODE_OPTIONS: ''
|
||||
@@ -1657,7 +1663,8 @@ jobs:
|
||||
if: steps.manual_read_credentials_2.outcome == 'success'
|
||||
env:
|
||||
MANUAL_PRIVATE_IDS: ${{ needs.authorize.outputs.manual_private_ids }}
|
||||
MANUAL_PRIVATE_APPROVALS_JSON: ${{ vars.RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON }}
|
||||
PAPERCLIP_MANUAL_DISPATCH_AUTHORITY: v1
|
||||
MANUAL_DISPATCH_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
|
||||
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
|
||||
RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET }}
|
||||
NODE_OPTIONS: ''
|
||||
@@ -1688,7 +1695,8 @@ jobs:
|
||||
if: steps.manual_read_credentials_3.outcome == 'success'
|
||||
env:
|
||||
MANUAL_PRIVATE_IDS: ${{ needs.authorize.outputs.manual_private_ids }}
|
||||
MANUAL_PRIVATE_APPROVALS_JSON: ${{ vars.RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON }}
|
||||
PAPERCLIP_MANUAL_DISPATCH_AUTHORITY: v1
|
||||
MANUAL_DISPATCH_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
|
||||
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
|
||||
RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET }}
|
||||
NODE_OPTIONS: ''
|
||||
@@ -1719,7 +1727,8 @@ jobs:
|
||||
if: steps.manual_read_credentials_4.outcome == 'success'
|
||||
env:
|
||||
MANUAL_PRIVATE_IDS: ${{ needs.authorize.outputs.manual_private_ids }}
|
||||
MANUAL_PRIVATE_APPROVALS_JSON: ${{ vars.RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON }}
|
||||
PAPERCLIP_MANUAL_DISPATCH_AUTHORITY: v1
|
||||
MANUAL_DISPATCH_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
|
||||
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
|
||||
RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET }}
|
||||
NODE_OPTIONS: ''
|
||||
@@ -1750,7 +1759,8 @@ jobs:
|
||||
if: steps.manual_read_credentials_5.outcome == 'success'
|
||||
env:
|
||||
MANUAL_PRIVATE_IDS: ${{ needs.authorize.outputs.manual_private_ids }}
|
||||
MANUAL_PRIVATE_APPROVALS_JSON: ${{ vars.RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON }}
|
||||
PAPERCLIP_MANUAL_DISPATCH_AUTHORITY: v1
|
||||
MANUAL_DISPATCH_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
|
||||
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
|
||||
RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET }}
|
||||
NODE_OPTIONS: ''
|
||||
@@ -1781,7 +1791,8 @@ jobs:
|
||||
if: steps.manual_read_credentials_6.outcome == 'success'
|
||||
env:
|
||||
MANUAL_PRIVATE_IDS: ${{ needs.authorize.outputs.manual_private_ids }}
|
||||
MANUAL_PRIVATE_APPROVALS_JSON: ${{ vars.RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON }}
|
||||
PAPERCLIP_MANUAL_DISPATCH_AUTHORITY: v1
|
||||
MANUAL_DISPATCH_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
|
||||
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
|
||||
RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET }}
|
||||
NODE_OPTIONS: ''
|
||||
@@ -1812,7 +1823,8 @@ jobs:
|
||||
if: steps.manual_read_credentials_7.outcome == 'success'
|
||||
env:
|
||||
MANUAL_PRIVATE_IDS: ${{ needs.authorize.outputs.manual_private_ids }}
|
||||
MANUAL_PRIVATE_APPROVALS_JSON: ${{ vars.RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON }}
|
||||
PAPERCLIP_MANUAL_DISPATCH_AUTHORITY: v1
|
||||
MANUAL_DISPATCH_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
|
||||
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
|
||||
RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET }}
|
||||
NODE_OPTIONS: ''
|
||||
@@ -1831,7 +1843,8 @@ jobs:
|
||||
if: steps.manual_report.outcome == 'success'
|
||||
env:
|
||||
MANUAL_PRIVATE_IDS: ${{ needs.authorize.outputs.manual_private_ids }}
|
||||
MANUAL_PRIVATE_APPROVALS_JSON: ${{ vars.RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON }}
|
||||
PAPERCLIP_MANUAL_DISPATCH_AUTHORITY: v1
|
||||
MANUAL_DISPATCH_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
|
||||
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
|
||||
PRIVATE_REPORT_ROOT: ${{ github.workspace }}
|
||||
NODE_OPTIONS: ''
|
||||
@@ -2025,7 +2038,8 @@ jobs:
|
||||
if: always() && steps.manual_report_credentials.outcome == 'success' && steps.manual_cells_downloaded.outcome == 'success'
|
||||
env:
|
||||
MANUAL_PRIVATE_IDS: ${{ needs.authorize.outputs.manual_private_ids }}
|
||||
MANUAL_PRIVATE_APPROVALS_JSON: ${{ vars.RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON }}
|
||||
PAPERCLIP_MANUAL_DISPATCH_AUTHORITY: v1
|
||||
MANUAL_DISPATCH_ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
|
||||
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
|
||||
RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET }}
|
||||
NODE_OPTIONS: ''
|
||||
|
||||
@@ -24,7 +24,7 @@ aliases are not registered. The exact registry is in
|
||||
`tests/runner-e2e/manual-private-retention.ts`.
|
||||
|
||||
These IDs require the reviewed AWS Fleet route. Broad chat suites, mixed
|
||||
unsupported IDs, reruns and missing approvals fail closed. All chat-feature IDs
|
||||
unsupported IDs, reruns and runs without dispatch authority fail closed. All chat-feature IDs
|
||||
are classified as restricted by trusted code. A target descriptor cannot opt out.
|
||||
Other manual selections keep the existing Actions route only after a trusted
|
||||
per-cell check confirms that the actual selected execution is not restricted.
|
||||
@@ -38,8 +38,8 @@ credentials. Generic nonprivate catalog selection retains its existing semantics
|
||||
A registered ID is not a claim of feature readiness or permission to run it.
|
||||
Each target still needs the exact reviewed producer contracts, matching helper
|
||||
bytes, declared private inputs, fresh source fingerprints, an available original
|
||||
attempt and exact operation approval. Earlier accepted target branches do not
|
||||
acquire these bytes or approval automatically. Running jobs keep their pinned
|
||||
attempt and an allowlisted manual dispatch. Earlier accepted target branches do not
|
||||
acquire these bytes automatically. Running jobs keep their pinned
|
||||
sources. No retained grade or historical qualification transfers to a new target.
|
||||
|
||||
The private cell report contract binds the installed pinned Playwright producer
|
||||
@@ -58,30 +58,41 @@ or run. Before a supported run, the operator must:
|
||||
1. Deploy this workflow and its helpers together on the protected default branch.
|
||||
2. Keep the existing numeric actor allowlist and protected default-branch checks.
|
||||
3. Keep `runner-e2e-private-paid` and `runner-e2e-private-history` restricted to
|
||||
the default branch. Establish administrator-controlled approval for the exact
|
||||
source and original run. A reviewer UI may supply this control but is not
|
||||
mandatory. Prove the actual control and runtime contract before admission.
|
||||
the default branch.
|
||||
4. Provide the existing private-history role ARN, AWS region and bucket variables
|
||||
in those environments. The existing role must support the existing bounded
|
||||
prefix and bucket-policy, ownership, public-access and retention reads.
|
||||
5. Provide `RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON` through administrator-controlled
|
||||
configuration available to both protected jobs. Approve the exact resolved
|
||||
workflow SHA, target SHA, original run ID,
|
||||
attempt `1`, selected execution IDs and launcher SHA-256.
|
||||
6. Use a reviewed compatible target. Its private handoff and policy modules must
|
||||
match the workflow implementation byte for byte. Its launcher digest must
|
||||
match the protected approval. Earlier target branches do not gain this
|
||||
protocol automatically.
|
||||
5. Use a compatible target. Its private handoff and policy modules must match
|
||||
the workflow implementation byte for byte. Earlier target branches do not
|
||||
gain this protocol automatically.
|
||||
|
||||
Classification happens before the protected job. It grants no archive authority.
|
||||
The administrator-controlled record must bind the actual original run ID before
|
||||
target dependency setup or archive credentials. Prove who can change that record
|
||||
and that each job consumes the exact approved record. An environment reviewer
|
||||
gate is one way to govern authorization; its UI is not required. A missing or
|
||||
stale row still fails closed before target dependency setup, archive credentials,
|
||||
model credentials or fixture secrets. There is no wildcard approval.
|
||||
### Dispatch authority
|
||||
|
||||
The variable has this closed shape (all values below are placeholders):
|
||||
In `runner-full-stack-e2e.yml`, the manual run itself authorizes its private
|
||||
evidence. There is no separate per-run approval record. Trusted code in
|
||||
`tests/runner-e2e/manual-dispatch-authority.ts` admits a cell only when all of
|
||||
these are true:
|
||||
|
||||
- The event is `workflow_dispatch` of this workflow on the default branch, with
|
||||
the `manual` cadence. Scheduled and bounded-cadence runs never qualify.
|
||||
- The run is on attempt `1`. A rerun does not inherit authority.
|
||||
- The dispatching actor is in `RUNNER_E2E_ALLOWED_ACTOR_IDS`, the event sender is
|
||||
that actor, and the triggering actor is the same person.
|
||||
- The trusted cell selection equals the dispatch's own `id` input.
|
||||
|
||||
The authority binds the workflow SHA, the target SHA resolved by the authorize
|
||||
job, the run ID and the cells. The dispatcher chooses and vouches for the target
|
||||
branch, so its launcher digest is recorded in the private identity instead of
|
||||
compared with a separately approved digest. Classification still happens before
|
||||
the protected job, and a failed check still stops before target dependency
|
||||
setup, archive credentials, model credentials or fixture secrets. The per-run
|
||||
limit of eight cells is unchanged.
|
||||
|
||||
### Stored approval records
|
||||
|
||||
The TestOps catalog case workflow still passes a protected approval record as
|
||||
`MANUAL_PRIVATE_APPROVALS_JSON`. That record has this closed shape (all values
|
||||
below are placeholders):
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -105,7 +116,7 @@ It prepares pinned publication dependencies with `--ignore-scripts` and a
|
||||
separate store before archive credentials. Transfers use that checkout's
|
||||
absolute helper, tsx and archive-script paths. They do not load the target's
|
||||
node_modules or an ambient Node loader. The target executes only after the
|
||||
protected approval and private-storage preflight pass.
|
||||
evidence authority and private-storage preflight pass.
|
||||
|
||||
The AWS action returns session credentials as step outputs, not job-wide
|
||||
environment exports. Only trusted preflight and transfer steps receive those
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { createRequire } from "node:module";
|
||||
import { copyFile, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { authorizeManualDispatch } from "./manual-dispatch-authority.js";
|
||||
|
||||
const ids = ["activity.agent-feedback", "agents.choose-primary"].map(id => `chat-features.runner-opencode.local.${id}`);
|
||||
const workflowSha = "a".repeat(40), targetSha = "b".repeat(40);
|
||||
const env = (overrides: Record<string, string | undefined> = {}): NodeJS.ProcessEnv => ({
|
||||
PAPERCLIP_MANUAL_DISPATCH_AUTHORITY: "v1", GITHUB_ACTIONS: "true", GITHUB_EVENT_NAME: "workflow_dispatch",
|
||||
GITHUB_REPOSITORY: "paperclipai/paperclip", GITHUB_REF: "refs/heads/master",
|
||||
GITHUB_WORKFLOW_REF: "paperclipai/paperclip/.github/workflows/runner-full-stack-e2e.yml@refs/heads/master",
|
||||
GITHUB_SHA: workflowSha, TARGET_SHA: targetSha, GITHUB_RUN_ID: "123", GITHUB_RUN_ATTEMPT: "1",
|
||||
GITHUB_ACTOR: "owner", GITHUB_TRIGGERING_ACTOR: "owner", GITHUB_ACTOR_ID: "42",
|
||||
MANUAL_DISPATCH_ALLOWED_ACTOR_IDS: "[42,7]", MANUAL_PRIVATE_IDS: JSON.stringify(ids), ...overrides,
|
||||
});
|
||||
const event = (overrides: Record<string, unknown> = {}, inputs: Record<string, unknown> = {}) => ({
|
||||
ref: "refs/heads/master", sender: { id: 42 }, repository: { full_name: "paperclipai/paperclip", default_branch: "master" },
|
||||
inputs: { cadence: "manual", target_branch: "candidate", id: ids.join(","), ...inputs }, ...overrides,
|
||||
});
|
||||
|
||||
describe("authorizeManualDispatch", () => {
|
||||
it("authorizes an allowlisted operator's manual dispatch for exactly its own cells", () => {
|
||||
const authority = authorizeManualDispatch(env(), event());
|
||||
expect(authority).toMatchObject({ workflowSha, targetSha, runId: "123", attempt: "1", launchBinding: "dispatcher", private: true });
|
||||
expect(authority.executionIds).toHaveLength(2);
|
||||
expect(authority.approvalSha256).toMatch(/^[a-f0-9]{64}$/);
|
||||
// Every job of the run derives the same identity.
|
||||
expect(authorizeManualDispatch(env(), event()).approvalSha256).toBe(authority.approvalSha256);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["a scheduled run", env({ GITHUB_EVENT_NAME: "schedule" }), event()],
|
||||
["a bounded cadence", env(), event({}, { cadence: "daily" })],
|
||||
["an operator outside the allowlist", env({ GITHUB_ACTOR_ID: "99" }), event({ sender: { id: 99 } })],
|
||||
["an empty allowlist", env({ MANUAL_DISPATCH_ALLOWED_ACTOR_IDS: "[]" }), event()],
|
||||
["a rerun by another person", env({ GITHUB_TRIGGERING_ACTOR: "someone-else" }), event()],
|
||||
["a second attempt", env({ GITHUB_RUN_ATTEMPT: "2" }), event()],
|
||||
["a non-default workflow ref", env({ GITHUB_WORKFLOW_REF: "paperclipai/paperclip/.github/workflows/runner-full-stack-e2e.yml@refs/heads/feature" }), event()],
|
||||
["a different workflow", env({ GITHUB_WORKFLOW_REF: "paperclipai/paperclip/.github/workflows/other.yml@refs/heads/master" }), event()],
|
||||
["a sender who is not the actor", env(), event({ sender: { id: 7 } })],
|
||||
["a missing target commit", env({ TARGET_SHA: "" }), event()],
|
||||
["the authority switch unset", env({ PAPERCLIP_MANUAL_DISPATCH_AUTHORITY: undefined }), event()],
|
||||
])("rejects %s", (_label, environment, payload) => {
|
||||
expect(() => authorizeManualDispatch(environment, payload)).toThrow();
|
||||
});
|
||||
|
||||
it("rejects cells that differ from the dispatch's own selection", () => {
|
||||
expect(() => authorizeManualDispatch(env(), event({}, { id: ids[0] }))).toThrow(/differ/);
|
||||
expect(() => authorizeManualDispatch(env({ MANUAL_PRIVATE_IDS: JSON.stringify([ids[0]]) }), event())).toThrow(/differ/);
|
||||
});
|
||||
|
||||
it("keeps the per-run cell limit", () => {
|
||||
const many = ["activity.agent-feedback", "activity.work-timeline", "companies.import-preview", "navigation-preferences.dismiss-announcement",
|
||||
"tasks.completion-with-gate", "tasks.private-task", "secrets.personal-secret", "agents.choose-primary", "access.cli-authorization"]
|
||||
.map(id => `chat-features.runner-opencode.local.${id}`);
|
||||
expect(() => authorizeManualDispatch(env({ MANUAL_PRIVATE_IDS: JSON.stringify(many) }), event({}, { id: many.join(",") }))).toThrow(/limit/);
|
||||
});
|
||||
});
|
||||
|
||||
const roots: string[] = [];
|
||||
afterEach(async () => { await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); });
|
||||
|
||||
it("actual prepare admits a dispatched cell without a stored approval and records the target launcher", async () => {
|
||||
const root = await mkdtemp(path.join(os.tmpdir(), "synthetic-dispatch-prepare-")); roots.push(root);
|
||||
const target = path.join(root, "target"), modules = path.join(target, "tests/runner-e2e"); await mkdir(modules, { recursive: true });
|
||||
await writeFile(path.join(modules, "launch.ts"), "synthetic candidate launcher; never executed");
|
||||
for (const name of ["manual-private-handoff.ts", "manual-private-retention.ts", "manual-private-report-assets.ts"]) await copyFile(path.join(import.meta.dirname, name), path.join(modules, name));
|
||||
const eventPath = path.join(root, "event.json"), output = path.join(root, "outputs.txt");
|
||||
await writeFile(eventPath, JSON.stringify(event()));
|
||||
const trusted = path.resolve(import.meta.dirname, "../.."), require = createRequire(import.meta.url), tsx = require.resolve("tsx", { paths: [path.join(trusted, "server")] });
|
||||
const run = (extra: Record<string, string>) => spawnSync(process.execPath, ["--import", tsx, path.join(import.meta.dirname, "manual-private-transfer.ts"), "prepare"], {
|
||||
encoding: "utf8", timeout: 10000,
|
||||
env: { PATH: process.env.PATH, NODE_OPTIONS: "", NODE_PATH: "", ...env(), SELECTED_EXECUTION_ID: ids[0]!, GITHUB_EVENT_PATH: eventPath,
|
||||
TARGET_ROOT: target, RUNNER_TEMP: root, RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: "private-bucket", GITHUB_OUTPUT: output, ...extra } as NodeJS.ProcessEnv,
|
||||
});
|
||||
const denied = run({ GITHUB_ACTOR_ID: "99" });
|
||||
expect(denied.status).toBe(1);
|
||||
await expect(readFile(output)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
const admitted = run({});
|
||||
expect(admitted.error).toBeUndefined(); expect(admitted.status).toBe(0);
|
||||
const outputs = await readFile(output, "utf8");
|
||||
expect(outputs).toContain("session_policy=");
|
||||
const identityRoot = outputs.split("\n").find(line => line.startsWith("root="))!.slice(5);
|
||||
const identity = JSON.parse(await readFile(path.join(identityRoot, "identity.json"), "utf8"));
|
||||
expect(identity).toMatchObject({ launchBinding: "dispatcher", runId: "123", targetSha, executionIds: [ids[0]] });
|
||||
expect(identity.launchSha256).toMatch(/^[a-f0-9]{64}$/);
|
||||
});
|
||||
@@ -0,0 +1,54 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { lstatSync, readFileSync } from "node:fs";
|
||||
import { canonicalPrivateManualIds, MANUAL_PRIVATE_EXECUTIONS, MANUAL_PRIVATE_PROTOCOL } from "./manual-private-retention.js";
|
||||
|
||||
const authority = "paperclip.manual-dispatch-evidence/v1";
|
||||
const repository = "paperclipai/paperclip";
|
||||
const workflow = ".github/workflows/runner-full-stack-e2e.yml";
|
||||
const numericId = /^[1-9][0-9]{0,19}$/;
|
||||
const values = (value: unknown) => String(value ?? "").split(",").map(item => item.trim()).filter(Boolean);
|
||||
|
||||
/**
|
||||
* Private chat evidence authority for a manual run of the trusted workflow.
|
||||
*
|
||||
* The run itself is the approval: a `workflow_dispatch` of the default-branch
|
||||
* workflow, on its first attempt, started and triggered by an allowlisted
|
||||
* operator, for the exact cells named in its own `id` input. Scheduled and
|
||||
* bounded-cadence runs never qualify. The evidence stays bound to this run,
|
||||
* its workflow commit, its resolved target commit and those cells.
|
||||
*/
|
||||
export function authorizeManualDispatch(env: NodeJS.ProcessEnv, event: unknown) {
|
||||
const payload = event as { ref?: unknown; sender?: { id?: unknown }; repository?: { full_name?: unknown; default_branch?: unknown }; inputs?: Record<string, unknown> };
|
||||
const branch = payload.repository?.default_branch;
|
||||
const actor = env.GITHUB_ACTOR_ID;
|
||||
let allowed: unknown;
|
||||
try { allowed = JSON.parse(env.MANUAL_DISPATCH_ALLOWED_ACTOR_IDS ?? "null"); } catch { allowed = null; }
|
||||
if (env.PAPERCLIP_MANUAL_DISPATCH_AUTHORITY !== "v1" || env.GITHUB_ACTIONS !== "true" || env.GITHUB_EVENT_NAME !== "workflow_dispatch"
|
||||
|| env.GITHUB_REPOSITORY !== repository || payload.repository?.full_name !== repository
|
||||
|| typeof branch !== "string" || !/^[A-Za-z0-9._/-]+$/.test(branch) || env.GITHUB_REF !== `refs/heads/${branch}` || payload.ref !== env.GITHUB_REF
|
||||
|| env.GITHUB_WORKFLOW_REF !== `${repository}/${workflow}@refs/heads/${branch}`
|
||||
|| !/^[a-f0-9]{40}$/.test(env.GITHUB_SHA ?? "") || !/^[a-f0-9]{40}$/.test(env.TARGET_SHA ?? "")
|
||||
|| !numericId.test(env.GITHUB_RUN_ID ?? "") || env.GITHUB_RUN_ATTEMPT !== "1"
|
||||
|| !numericId.test(actor ?? "") || payload.sender?.id !== Number(actor)
|
||||
|| !env.GITHUB_ACTOR || env.GITHUB_TRIGGERING_ACTOR !== env.GITHUB_ACTOR
|
||||
|| !Array.isArray(allowed) || !allowed.length || allowed.some(v => !Number.isSafeInteger(v) || v <= 0) || !allowed.includes(Number(actor))
|
||||
|| (payload.inputs?.cadence ?? "manual") !== "manual") throw new Error("Invalid manual dispatch evidence authority");
|
||||
const requested = canonicalPrivateManualIds(values(payload.inputs?.id));
|
||||
const selected = canonicalPrivateManualIds(JSON.parse(env.MANUAL_PRIVATE_IDS ?? "null"));
|
||||
if (!selected.length || selected.join() !== requested.join() || selected.some(id => !MANUAL_PRIVATE_EXECUTIONS.includes(id)))
|
||||
throw new Error("Manual dispatch cells differ from the trusted selection");
|
||||
const identity = { workflowSha: env.GITHUB_SHA!, targetSha: env.TARGET_SHA!, runId: env.GITHUB_RUN_ID!, attempt: "1", executionIds: Object.freeze(selected) };
|
||||
const authoritySha256 = createHash("sha256").update(JSON.stringify({ authority, ...identity, actor })).digest("hex");
|
||||
// The dispatcher vouches for the target branch, so its launcher is recorded
|
||||
// rather than compared with a separately approved hash.
|
||||
return Object.freeze({ ...identity, authority, protocol: MANUAL_PRIVATE_PROTOCOL, launchBinding: "dispatcher" as const,
|
||||
authoritySha256, approvalSha256: authoritySha256, private: true as const });
|
||||
}
|
||||
|
||||
export function manualDispatchEvidenceAuthority(env: NodeJS.ProcessEnv) {
|
||||
const eventPath = env.GITHUB_EVENT_PATH;
|
||||
if (!eventPath) throw new Error("Missing manual dispatch event");
|
||||
const stat = lstatSync(eventPath);
|
||||
if (!stat.isFile() || stat.isSymbolicLink() || stat.size > 65536) throw new Error("Invalid manual dispatch event custody");
|
||||
return authorizeManualDispatch(env, JSON.parse(readFileSync(eventPath, "utf8")));
|
||||
}
|
||||
@@ -12,12 +12,17 @@ import { BoundedHistoryPublisher } from "./history-budget.js";
|
||||
import { privateArtifactKey, retainPrivateTransfer, transferPrivateArtifact, verifyPrivateHistory, type PrivateEvidenceCompleteness } from "./private-history.js";
|
||||
import { authorizeManualPrivateEvidence, MANUAL_PRIVATE_PROTOCOL, manualCellKind, manualPrivateSessionPolicy } from "./manual-private-retention.js";
|
||||
import { shippedChatEvidenceAuthority } from "./testops-chat-admission.js";
|
||||
import { manualDispatchEvidenceAuthority } from "./manual-dispatch-authority.js";
|
||||
|
||||
const hash = (bytes: string | Buffer) => createHash("sha256").update(bytes).digest("hex");
|
||||
function request(env: NodeJS.ProcessEnv) {
|
||||
const ids = JSON.parse(env.MANUAL_PRIVATE_IDS ?? "null") as string[];
|
||||
const approval = env.PAPERCLIP_TESTOPS_CHAT_MAIN_ADMISSION !== undefined ? shippedChatEvidenceAuthority(env) : authorizeManualPrivateEvidence({ workflowSha: env.GITHUB_SHA ?? "", targetSha: env.TARGET_SHA ?? "",
|
||||
runId: env.GITHUB_RUN_ID ?? "", attempt: env.GITHUB_RUN_ATTEMPT ?? "", executionIds: ids }, env.MANUAL_PRIVATE_APPROVALS_JSON ?? "");
|
||||
const approval: { workflowSha: string; targetSha: string; runId: string; attempt: string; executionIds: readonly string[]; approvalSha256: string;
|
||||
launchSha256?: string; launchBinding?: "dispatcher" } =
|
||||
env.PAPERCLIP_TESTOPS_CHAT_MAIN_ADMISSION !== undefined ? shippedChatEvidenceAuthority(env)
|
||||
: env.PAPERCLIP_MANUAL_DISPATCH_AUTHORITY !== undefined ? manualDispatchEvidenceAuthority(env)
|
||||
: authorizeManualPrivateEvidence({ workflowSha: env.GITHUB_SHA ?? "", targetSha: env.TARGET_SHA ?? "",
|
||||
runId: env.GITHUB_RUN_ID ?? "", attempt: env.GITHUB_RUN_ATTEMPT ?? "", executionIds: ids }, env.MANUAL_PRIVATE_APPROVALS_JSON ?? "");
|
||||
return { ids, approval, bucket: env.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET ?? "" };
|
||||
}
|
||||
async function output(name: string, value: string) {
|
||||
@@ -30,13 +35,15 @@ async function prepare(env: NodeJS.ProcessEnv) {
|
||||
const target = await realpath(env.TARGET_ROOT ?? ""), trusted = path.resolve(import.meta.dirname, "../..");
|
||||
if (target === trusted) throw new Error("Trusted transfer code must be separate from target checkout");
|
||||
const launch = await readFile(path.join(target, "tests/runner-e2e/launch.ts"));
|
||||
if (hash(launch) !== approval.launchSha256) throw new Error("Target evidence launcher differs from protected approval");
|
||||
const targetLaunchSha256 = hash(launch);
|
||||
if (approval.launchBinding !== "dispatcher" && targetLaunchSha256 !== approval.launchSha256)
|
||||
throw new Error("Target evidence launcher differs from protected approval");
|
||||
for (const file of ["manual-private-handoff.ts", "manual-private-retention.ts", "manual-private-report-assets.ts"]) {
|
||||
if (!(await readFile(path.join(target, "tests/runner-e2e", file))).equals(await readFile(path.join(trusted, "tests/runner-e2e", file))))
|
||||
throw new Error("Target private evidence protocol differs from trusted implementation");
|
||||
}
|
||||
const temporary = await realpath(env.RUNNER_TEMP ?? ""), root = await mkdtemp(path.join(temporary, "manual-private-evidence-"));
|
||||
await writeFile(path.join(root, "identity.json"), JSON.stringify({ ...approval, executionIds: [id], root, status: "pending" }), { mode: 0o600, flag: "wx" });
|
||||
await writeFile(path.join(root, "identity.json"), JSON.stringify({ ...approval, launchSha256: targetLaunchSha256, executionIds: [id], root, status: "pending" }), { mode: 0o600, flag: "wx" });
|
||||
await output("root", root); await output("session_policy", JSON.stringify(manualPrivateSessionPolicy({ bucket, runId: approval.runId, executionIds: [id], mode: "cell-write" })));
|
||||
}
|
||||
function requireSession(env: NodeJS.ProcessEnv) {
|
||||
|
||||
Reference in new issue
Block a user