PaperClipAI/tests/runner-e2e/SECURITY.md

Runner E2E security for a public repository

This suite can spend provider money, expose four API credentials to isolated test processes, publish a container, retain private visual evidence, and write public structured evidence. Treat changes to the workflow, harness, fixture prompts, evidence packager, and publisher as security-sensitive production changes.

GitHub authorization

Set RUNNER_E2E_ALLOWED_ACTOR_IDS to a non-empty JSON array of numeric GitHub user IDs, for example [123456,789012]. Resolve each ID from the authenticated CLI and verify the login before adding it:

gh api users/LOGIN --jq '{login,id}'

The paid workflows reject manual dispatches outside the default branch before checkout. They verify both the original actor and triggering actor for every scheduled or manual attempt, including human reruns. Every secret-bearing job repeats this check as its first step so GitHub's partial-job rerun feature cannot bypass a successful predecessor authorization job. The legacy manually dispatched E2E workflow uses the same gate. Numeric IDs are stable across username changes and prevent lookalike-name authorization.

The full-stack and live campaigns have one Sunday UTC schedule each and also support explicit manual dispatch. Their legacy-named nightly repository variables remain independent kill switches. Neither paid workflow accepts pull-request, push, workflow-run, or reusable-workflow triggers.

Protect the default branch, require review for workflow/harness paths, restrict workflow dispatch permission, and restrict repository variable/environment administration to the same trusted maintainers. Configure the organization to allow only approved GitHub Actions. A malicious change merged into the default branch executes with the same authority as the suite.

Every external action in the paid workflow is pinned to a full commit SHA. Keep the adjacent major-version comment for update tooling, and resolve and review a new immutable SHA before upgrading an action. The credential-free security test rejects mutable tag or branch references.

Secrets and protected environments

Create runner-e2e-paid, restrict deployments to the default branch, and put only OPENAI_API_KEY, ANTHROPIC_API_KEY, OPENROUTER_API_KEY, and DAYTONA_API_KEY in it. Do not duplicate these credentials as repository- or organization-level Actions secrets: environment scoping is the boundary that prevents branch or pull-request jobs from requesting them. Require approval from an account in RUNNER_E2E_ALLOWED_ACTOR_IDS for this environment and disable administrator bypass. The authorize, catalog, image, report, history, and Pages jobs receive none of these secrets. Each full-stack matrix cell receives only its selected profile credential, plus Daytona only for Daytona cells. Secret-bearing and OIDC jobs use frozen installs without a shared dependency cache. The Paperclip server process also receives none; the browser posts each value once to the encrypted company secret API and agents/environments retain only secret references.

Create runner-e2e-history, also default-branch-only, for the OIDC publishing job. It contains no long-lived AWS key. Required reviewers may be added when a human approval on every nightly publication is acceptable; otherwise rely on the actor gate, environment branch restriction, and protected default branch.

Runner fleet isolation

When RUNNER_E2E_AWS_ENABLED=true, paid matrix cells use the exact RunsOn fleet selector runs-on/fleet=paperclip-public-pr-x64/env=public-ci, matching the AWS fleet selected by pr-trusted.yml only after its stable numeric-ID trust gate. Any other or missing toggle value falls back to the existing ubuntu-latest-m paid runner and its lower concurrency ceiling. The workflow chooses between those two reviewed literal labels; it never evaluates a configured runner label.

Keep both runner targets restricted to paperclipai/paperclip and workflows that independently authorize trusted source revisions. Never let a fork or untrusted pull-request workflow target them. The RunsOn fleet must launch a fresh ephemeral instance for every job, prohibit persistent runner reuse, and disable interactive SSH/debug access unless a separate incident procedure explicitly authorizes it.

Changing the runner does not widen secret access. The paid workflow still has only schedule and manual triggers, requires the protected default branch and allowlisted stable actor IDs before checkout, repeats that authorization as the first matrix step, and receives provider credentials only from the protected runner-e2e-paid environment. The fleet selector is an exact workflow literal; the only repository-controlled input is its boolean rollout switch, so configuration cannot redirect a secret-bearing job to an arbitrary runner.

AWS OIDC and S3

The AWS role trust policy should accept only GitHub's OIDC audience and the publishing environment subject:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
          "token.actions.githubusercontent.com:sub": "repo:paperclipai/paperclip:environment:runner-e2e-history"
        }
      }
    }
  ]
}

Grant only List on the bucket prefix and Get/Put on its objects. Do not grant Delete, ACL, bucket-policy, or wildcard-resource permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::BUCKET",
      "Condition": {
        "StringLike": { "s3:prefix": ["runner-e2e", "runner-e2e/*"] }
      }
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::BUCKET/runner-e2e/*"
    }
  ]
}

Enable S3 versioning, default encryption, and Block Public Access. Disable object ACLs. CloudFront receives read-only access through Origin Access Control; the bucket itself stays private. Log S3 data writes and alert on attempts to write outside the prefix or assume the role with a different subject.

Campaign prefixes are content-digested and immutable. The publisher refuses a different digest at an existing campaign key. Only the compact history and latest pointers are mutable, and S3 versioning makes those updates recoverable.

Public evidence boundary

CloudFront and GitHub Pages are public. Fixture identifiers, timing, token usage, costs, normalized results, and allowlisted inert structured per-attempt evidence are expected public data. Screenshots, video, archives, generated Playwright/blob/HTML report trees, credentials, Paperclip homes, databases, workspaces, master keys, raw/unredacted logs, and unallowlisted files are not. Only allowlisted .log copies that passed exact-value/key-shape scanning and redaction may cross the public boundary.

The packaged evidence uploaded as a 30-day GitHub Actions artifact has a different, access-controlled boundary. Text is exact-value and key-shape scanned and redacted. PNG and WebM are raw-byte scanned but cannot be inspected for credentials rendered as pixels, so they remain only in local evidence and the access-controlled artifact. SVG is rejected during packaging because it is active content.

Before permanent publication, the campaign publisher prunes raster/video files, archives, and generated report trees. It then regenerates the dashboard from the remaining allowlisted .json, .log, .md, and .txt evidence and accepts only that dashboard, normalized JSON/JUnit/summary, fixed branding assets, and the inert structured evidence paths. Per-attempt XML is excluded because browsers can process XML/XSLT; the only public XML is the root junit.xml, which the report aggregator constructs from fixed markup and XML-escaped fields. The same pruned tree feeds both S3/CloudFront history and the optional GitHub Pages artifact. A leak fails the cell and withholds the unsafe file.

Rotate the affected credential immediately if a secret-scanning failure or unexpected public object is observed. Preserve the access-controlled Actions artifact and S3 object versions for incident analysis; do not weaken scanning to make a campaign publish.