Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-29 12:59:13 +01:00
Code Issues Packages Projects Releases Wiki Activity
Files
063cfd43813cabf18417605fb9eefb9aacf95c8d
PaperClipAI/.github/workflows/runner-full-stack-e2e.yml
T
DottaandPaperclip 287b04d72f fix(evals): keep report dependency proof with lockfile refresh (#18063) 
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/192/changes/paperclip/192
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip manages AI agents and runs paid Product E2E evaluations.
> - Paid work needs a report that can install its trusted dependencies.
> - The report uses a reviewed offline lock correction.
> - New UI dependencies can make that correction stale while its lock
hash still matches.
> - The lockfile bot must refresh the correction with its lock changes.
> - This PR also checks manifest consistency offline before model
admission.

## Linked Issues or Issue Description

**What happened?**

Current master declares new CodeMirror and YAML UI dependencies. Its
committed lock omits those UI importer entries. The existing report
correction accepts that lock, but a frozen install then fails with
`ERR_PNPM_OUTDATED_LOCKFILE`. This can waste completed paid work before
the report fails.

Refs #18059. That bot PR supplies the matching lock. This PR keeps the
root lock bot-owned and supplies its exact offline report correction.
The correction changes 21 importer lines and leaves all package records
unchanged.

**Expected behavior**

The existing bot includes matching report proof in its dependency PR.
Paid authorization rejects stale manifests before model work. Report
installation still accepts only reviewed hashes and applies no registry
resolution.

**Steps to reproduce**

1. Run a frozen pnpm 9.15.4 install on master at
`7f10cc058568e3c1ac133cf24f3b4000cbdfca0e`.
2. Observe the missing UI importer entries.
3. Apply the reviewed correction in this PR. The frozen install passes.

## What Changed

- Add an explicit refresh mode that derives the offline patch and hashes
from a known committed lock and the bot's resolved lock.
- Keep the existing unknown-lock, patch hash and output hash guards.
- Include the proof files in the existing lockfile bot PR. New bot PRs
use complete descriptions; existing descriptions stay intact.
- Validate the frozen lock against current manifests offline before paid
admission, using the existing pinned pnpm version.
- Refresh the current correction and document this process. The root
lockfile is not committed here.

## Verification

- 24 focused report-lock and workflow-security tests pass. They cover
offline replay, tampered input/patch/output rejection, real Git refresh,
unknown-base refusal and unchanged proof.
- A full frozen install passes after the reviewed correction, with
lifecycle scripts disabled.
- A fresh source archive validates offline with an empty package store.
- A deliberate manifest change fails that same offline check with
`ERR_PNPM_OUTDATED_LOCKFILE`.
- Both workflow files parse as YAML. All 95 shell steps pass Bash syntax
checks.
- `git diff --check` passes. These checks make no model requests.
- Full repository checks and fresh reviews must pass on the final PR
head before merge.

## Risks

- Dependency changes must still pass review. Refresh refuses an unknown
committed baseline.
- The bot now commits two additional proof files. Its write permissions
and target branch stay unchanged. Existing PR descriptions are
preserved.
- The paid authorization job adds an offline manifest check. It runs no
package lifecycle scripts.
- No budget, provider credential, environment permission, schedule or
paid enable gate changes.

## Model Used

- OpenAI Codex, GPT-6, with reasoning and code execution tools. The
exact serving variant and context window are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 15:30:58 -05:00

1590 lines
75 KiB
YAML
Raw Blame History

name: Runner Full-Stack E2E
run-name: Runner Full-Stack E2E / ${{ github.event.schedule || inputs.cadence || 'manual' }}
on:
schedule:
- cron: "47 8 * * *"
- cron: "47 9 * * 0"
workflow_dispatch:
inputs:
cadence:
description: "Bounded daily ($20) or weekly ($100) selection; manual keeps explicit selectors"
type: choice
options: [manual, daily, weekly]
default: manual
target_branch:
description: "Branch in paperclipai/paperclip to test; the trusted workflow still runs from master"
type: string
required: false
max_parallel:
description: "Optional lower concurrency for this campaign (cannot exceed the configured limit)"
type: string
required: false
all:
description: "Run the complete paid matrix when no narrower selector is supplied"
type: boolean
default: true
group:
description: "Comma-separated groups (AND semantics: legacy,native,local,daytona,warm,core,breadth)"
type: string
required: false
suite:
description: "Comma-separated suite IDs"
type: string
required: false
profile:
description: "Comma-separated runner profile fixture IDs"
type: string
required: false
environment:
description: "Comma-separated environment fixture IDs"
type: string
required: false
case:
description: "Comma-separated task case fixture IDs"
type: string
required: false
id:
description: "Comma-separated full suite.profile.environment.case IDs; exclusive with other selectors"
type: string
required: false
permissions:
contents: read
concurrency:
group: runner-full-stack-e2e-${{ github.event_name == 'workflow_dispatch' && inputs.target_branch != '' && inputs.target_branch != github.event.repository.default_branch && format('development-{0}', inputs.target_branch) || format('protected-{0}', github.run_id) }}
# Development branch campaigns supersede older runs for the same target.
# Give protected/default-branch campaigns unique groups because GitHub also
# replaces pending runs when cancel-in-progress is false.
cancel-in-progress: ${{ github.event_name == 'workflow_dispatch' && inputs.target_branch != '' && inputs.target_branch != github.event.repository.default_branch }}
jobs:
authorize:
name: Authorize paid campaign
if: github.event_name != 'schedule' || (vars.RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED == 'true' && vars.RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED == 'true')
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read
outputs:
test_runner: ${{ steps.runner.outputs.runner }}
max_parallel_default: ${{ steps.runner.outputs.max_parallel_default }}
max_parallel_limit: ${{ steps.runner.outputs.max_parallel_limit }}
playwright_channel: ${{ steps.runner.outputs.playwright_channel }}
target_sha: ${{ steps.target.outputs.sha }}
target_ref: ${{ steps.target.outputs.ref }}
steps:
- name: Require default branch and allowlisted numeric actor IDs
env:
GH_TOKEN: ${{ github.token }}
REPOSITORY: ${{ github.repository }}
REF: ${{ github.ref }}
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
ACTOR: ${{ github.actor }}
ACTOR_ID: ${{ github.actor_id }}
TRIGGERING_ACTOR: ${{ github.triggering_actor }}
ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
run: |
set -euo pipefail
if [ "$REF" != "refs/heads/$DEFAULT_BRANCH" ]; then
echo "Paid runner E2E campaigns may run only from the default branch." >&2
exit 1
fi
if ! jq -e 'type == "array" and length > 0 and all(.[]; type == "number" and . > 0 and floor == .)' <<< "${ALLOWED_ACTOR_IDS:-}" >/dev/null; then
echo "RUNNER_E2E_ALLOWED_ACTOR_IDS must be a non-empty JSON array of numeric GitHub user IDs." >&2
exit 1
fi
# Retry transient transport/server failures, but never proceed without
# a successfully resolved identity and the allowlist checks below.
for attempt in 1 2 3; do
if triggering_actor_id="$(timeout 30s gh api "users/$TRIGGERING_ACTOR" --jq .id)"; then
break
fi
if [ "$attempt" = 3 ]; then exit 1; fi
sleep "$((attempt * 2))"
done
if [ "$triggering_actor_id" != "$ACTOR_ID" ] && [ "$TRIGGERING_ACTOR" = "$ACTOR" ]; then
echo "GitHub actor identity contexts disagree; refusing the paid run." >&2
exit 1
fi
candidates=("$triggering_actor_id" "$ACTOR_ID")
for candidate in "${candidates[@]}"; do
if ! jq -e --argjson candidate "$candidate" 'index($candidate) != null' <<< "$ALLOWED_ACTOR_IDS" >/dev/null; then
echo "The initiating GitHub account is not authorized to run paid runner E2E campaigns." >&2
exit 1
fi
done
- name: Resolve requested repository branch to an immutable commit
id: target
env:
GH_TOKEN: ${{ github.token }}
REPOSITORY: ${{ github.repository }}
TARGET_BRANCH: ${{ inputs.target_branch || github.event.repository.default_branch }}
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
EVENT_NAME: ${{ github.event_name }}
EVAL_CADENCE: ${{ inputs.cadence || 'manual' }}
run: |
set -euo pipefail
if [ -z "$TARGET_BRANCH" ] || [[ "$TARGET_BRANCH" == refs/* ]]; then
echo "target_branch must name a branch in this repository without a refs/ prefix." >&2
exit 1
fi
if [ "$EVENT_NAME" = schedule ] || [ "$EVAL_CADENCE" != manual ]; then
if [ "$TARGET_BRANCH" != "$DEFAULT_BRANCH" ]; then
echo "Bounded campaigns require the workflow's default-branch commit." >&2
exit 1
fi
target_sha="$GITHUB_SHA"
else
encoded_branch="$(jq -rn --arg branch "$TARGET_BRANCH" '$branch | @uri')"
target_sha="$(gh api -X GET "repos/$REPOSITORY/branches/$encoded_branch" --jq .commit.sha)"
fi
if ! [[ "$target_sha" =~ ^[0-9a-f]{40}$ ]]; then
echo "The requested repository branch did not resolve to a commit." >&2
exit 1
fi
echo "sha=$target_sha" >> "$GITHUB_OUTPUT"
echo "ref=refs/heads/$TARGET_BRANCH" >> "$GITHUB_OUTPUT"
echo "Resolved the requested repository branch to $target_sha."
- name: Select paid test runner
id: runner
env:
AWS_PAID_RUNNER_ENABLED: ${{ vars.RUNNER_E2E_AWS_ENABLED }}
EVENT_NAME: ${{ github.event_name }}
EVAL_CADENCE: ${{ inputs.cadence || 'manual' }}
REPOSITORY_PRIVATE: ${{ github.event.repository.private }}
run: |
set -euo pipefail
github_runner='ubuntu-latest'
aws_runner='runs-on/fleet=paperclip-public-pr-x64/env=public-ci'
bounded=false
if [ "$EVENT_NAME" = schedule ] || [ "$EVAL_CADENCE" != manual ]; then
if [ "$REPOSITORY_PRIVATE" != false ]; then
echo 'Bounded campaigns require a public repository and standard GitHub-hosted compute.' >&2
exit 1
fi
bounded=true
fi
if [ "$AWS_PAID_RUNNER_ENABLED" = true ] && [ "$bounded" = false ]; then
{
echo "runner=$aws_runner"
echo "max_parallel_default=100"
echo "max_parallel_limit=100"
echo "playwright_channel=chrome"
} >> "$GITHUB_OUTPUT"
echo '::notice title=Paid runner routing::Using an ephemeral RunsOn Fleet runner'
else
{
echo "runner=$github_runner"
echo "max_parallel_default=32"
echo "max_parallel_limit=57"
echo "playwright_channel="
} >> "$GITHUB_OUTPUT"
echo '::notice title=Paid runner routing::Using a standard GitHub-hosted runner'
fi
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ github.sha }}
persist-credentials: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
- uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
env:
NPM_CONFIG_AUDIT: "false"
NPM_CONFIG_FUND: "false"
NPM_CONFIG_UPDATE_NOTIFIER: "false"
with:
version: 9.15.4
run_install: false
- name: Preflight reviewed report dependencies before paid admission
run: node tests/runner-e2e/prepare-report-lock.mjs
- name: Validate report dependency manifests offline
run: pnpm install --lockfile-only --frozen-lockfile --ignore-scripts --offline
target_lock:
name: Resolve target pnpm lockfile
needs: authorize
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
outputs:
artifact_id: ${{ steps.upload.outputs.artifact-id }}
lock_sha256: ${{ steps.lock.outputs.sha256 }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ needs.authorize.outputs.target_sha }}
persist-credentials: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
- uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
env:
NPM_CONFIG_AUDIT: "false"
NPM_CONFIG_FUND: "false"
NPM_CONFIG_UPDATE_NOTIFIER: "false"
with:
version: 9.15.4
- name: Resolve target lockfile without lifecycle scripts
id: lock
run: |
set -euo pipefail
pnpm install --ignore-scripts --no-frozen-lockfile --lockfile-only
test -s pnpm-lock.yaml
unexpected="$(git status --short | awk '$2 != "pnpm-lock.yaml" { print }')"
if [ -n "$unexpected" ]; then
echo "Lockfile resolution changed files other than pnpm-lock.yaml:" >&2
echo "$unexpected" >&2
exit 1
fi
echo "sha256=$(sha256sum pnpm-lock.yaml | cut -d ' ' -f 1)" >> "$GITHUB_OUTPUT"
- name: Bound scheduled lockfile upload
if: github.event_name == 'schedule' || (inputs.cadence != '' && inputs.cadence != 'manual')
run: node tests/runner-e2e/artifact-budget.mjs check lock pnpm-lock.yaml
- name: Upload resolved target lockfile
id: upload
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: runner-e2e-target-pnpm-lock-${{ github.run_id }}-${{ github.run_attempt }}
path: pnpm-lock.yaml
retention-days: 30
if-no-files-found: error
catalog:
name: Validate catalog and select cells
needs: [authorize, target_lock]
if: github.event_name != 'schedule' || (vars.RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED == 'true' && vars.RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED == 'true')
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
outputs:
matrix: ${{ steps.catalog.outputs.matrix }}
operations_cadence: ${{ steps.catalog.outputs.operations_cadence }}
operating_overhead: ${{ steps.catalog.outputs.operating_overhead }}
needs_daytona: ${{ steps.catalog.outputs.needs_daytona }}
needs_runner_typescript: ${{ steps.catalog.outputs.needs_runner_typescript }}
needs_native_binaries: ${{ steps.catalog.outputs.needs_native_binaries }}
needs_remote_provider_pack: ${{ steps.catalog.outputs.needs_remote_provider_pack }}
execution_ids: ${{ steps.catalog.outputs.execution_ids }}
max_parallel: ${{ steps.catalog.outputs.max_parallel }}
daytona_image_content_id: ${{ steps.daytona_image_content.outputs.content_id }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ needs.authorize.outputs.target_sha }}
persist-credentials: false
- name: Download resolved target lockfile
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
artifact-ids: ${{ needs.target_lock.outputs.artifact_id }}
path: ${{ runner.temp }}/runner-e2e-target-lock
- name: Restore resolved target lockfile
env:
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
EXPECTED_LOCK_SHA256: ${{ needs.target_lock.outputs.lock_sha256 }}
run: |
set -euo pipefail
test "$(git rev-parse HEAD)" = "$TARGET_SHA"
lock="$RUNNER_TEMP/runner-e2e-target-lock/pnpm-lock.yaml"
test -f "$lock"
test "$(find "$(dirname "$lock")" -type f | wc -l | tr -d ' ')" = 1
test "$(sha256sum "$lock" | cut -d ' ' -f 1)" = "$EXPECTED_LOCK_SHA256"
cp "$lock" pnpm-lock.yaml
test "$(sha256sum pnpm-lock.yaml | cut -d ' ' -f 1)" = "$EXPECTED_LOCK_SHA256"
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
- uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
env:
NPM_CONFIG_AUDIT: "false"
NPM_CONFIG_FUND: "false"
NPM_CONFIG_UPDATE_NOTIFIER: "false"
with:
version: 9.15.4
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
cache: pnpm
- run: pnpm install --frozen-lockfile
# The v2 contract fails closed unless every Docker FROM is digest-pinned,
# and hashes those exact base references into the immutable image tag.
- name: Compute Daytona image content ID with pinned bases
id: daytona_image_content
run: echo "content_id=$(pnpm --silent test:e2e:runner:image-id)" >> "$GITHUB_OUTPUT"
- name: Validate selectors and emit matrix
id: catalog
env:
EVENT_NAME: ${{ github.event_name }}
EVAL_EVENT: ${{ github.event_name }}
EVAL_SCHEDULE: ${{ github.event.schedule }}
EVAL_CADENCE: ${{ inputs.cadence || 'manual' }}
EVAL_OPERATIONS_ENABLED: ${{ vars.RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED }}
EVAL_OPERATING_OVERHEAD_JSON: ${{ vars.RUNNER_E2E_OPERATING_OVERHEAD_JSON }}
EVAL_TEST_RUNNER: ${{ needs.authorize.outputs.test_runner }}
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
SELECT_ALL: ${{ inputs.all }}
SELECT_SUITE: ${{ inputs.suite }}
SELECT_GROUP: ${{ inputs.group }}
SELECT_PROFILE: ${{ inputs.profile }}
SELECT_ENVIRONMENT: ${{ inputs.environment }}
SELECT_CASE: ${{ inputs.case }}
SELECT_ID: ${{ inputs.id }}
MAX_PARALLEL: ${{ vars.RUNNER_E2E_MAX_PARALLEL || needs.authorize.outputs.max_parallel_default }}
MAX_PARALLEL_LIMIT: ${{ needs.authorize.outputs.max_parallel_limit }}
REQUESTED_MAX_PARALLEL: ${{ inputs.max_parallel }}
run: |
set -euo pipefail
if [ "$EVAL_EVENT" = schedule ] || [ "$EVAL_CADENCE" != manual ]; then
test "$TARGET_SHA" = "$GITHUB_SHA"
test -z "${SELECT_SUITE:-}${SELECT_GROUP:-}${SELECT_PROFILE:-}${SELECT_ENVIRONMENT:-}${SELECT_CASE:-}${SELECT_ID:-}"
catalog_json="$(node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/scheduled-operations.ts matrix)"
cadence="$EVAL_CADENCE"
if [ "$EVAL_EVENT" = schedule ]; then
case "$EVAL_SCHEDULE" in
'47 8 * * *') cadence=daily ;;
'47 9 * * 0') cadence=weekly ;;
*) exit 1 ;;
esac
fi
{
echo "operations_cadence=$cadence"
echo "operating_overhead=$(jq -c '.overhead' <<< "$catalog_json")"
echo "matrix=$(jq -c '{include: .include}' <<< "$catalog_json")"
echo "execution_ids=$(jq -c '.executionIds' <<< "$catalog_json")"
echo "needs_daytona=false"
echo "needs_runner_typescript=true"
echo "needs_native_binaries=true"
echo "needs_remote_provider_pack=false"
echo "max_parallel=1"
} >> "$GITHUB_OUTPUT"
exit 0
fi
args=(--matrix-json)
add_values() {
local flag="$1"
local values="$2"
local value
IFS=',' read -ra entries <<< "$values"
for value in "${entries[@]}"; do
value="${value#"${value%%[![:space:]]*}"}"
value="${value%"${value##*[![:space:]]}"}"
if [ -n "$value" ]; then
args+=("$flag" "$value")
fi
done
}
explicit=false
if [ -n "${SELECT_ID:-}" ]; then
if [ -n "${SELECT_SUITE:-}${SELECT_GROUP:-}${SELECT_PROFILE:-}${SELECT_ENVIRONMENT:-}${SELECT_CASE:-}" ]; then
echo "The id selector is exclusive with suite/group/profile/environment/case" >&2
exit 1
fi
add_values --id "$SELECT_ID"
explicit=true
else
for pair in \
"--suite:${SELECT_SUITE:-}" \
"--group:${SELECT_GROUP:-}" \
"--profile:${SELECT_PROFILE:-}" \
"--environment:${SELECT_ENVIRONMENT:-}" \
"--case:${SELECT_CASE:-}"
do
flag="${pair%%:*}"
values="${pair#*:}"
if [ -n "$values" ]; then
add_values "$flag" "$values"
explicit=true
fi
done
fi
if [ "$explicit" = false ] && { [ "$EVENT_NAME" = schedule ] || [ "${SELECT_ALL:-false}" = true ]; }; then
args+=(--all)
fi
catalog_json="$(pnpm --silent test:e2e:runner -- "${args[@]}")"
{
echo "matrix=$(jq -c '{include: .include}' <<< "$catalog_json")"
echo "needs_daytona=$(jq -r '.needsDaytona' <<< "$catalog_json")"
echo "needs_runner_typescript=$(jq -r '[.include[] | select((.profileId == "runner-opencode") or (.profileId | startswith("runner-acpx-")) or (.suiteId == "openrouter-model-breadth"))] | length > 0' <<< "$catalog_json")"
echo "needs_native_binaries=$(jq -r '[.include[] | select((.profileId | startswith("runner-")) or (.suiteId == "openrouter-model-breadth"))] | length > 0' <<< "$catalog_json")"
echo "needs_remote_provider_pack=$(jq -r '[.include[] | select((.environmentId == "daytona") and ((.profileId == "runner-opencode") or (.profileId | startswith("runner-acpx-"))))] | length > 0' <<< "$catalog_json")"
echo "execution_ids=$(jq -c '.executionIds' <<< "$catalog_json")"
} >> "$GITHUB_OUTPUT"
if ! [[ "$MAX_PARALLEL_LIMIT" =~ ^[1-9][0-9]*$ ]] || [ "$MAX_PARALLEL_LIMIT" -gt 100 ]; then
echo "Runner selection emitted an invalid max-parallel limit." >&2
exit 1
fi
if ! [[ "$MAX_PARALLEL" =~ ^[1-9][0-9]*$ ]] || [ "$MAX_PARALLEL" -gt "$MAX_PARALLEL_LIMIT" ]; then
echo "RUNNER_E2E_MAX_PARALLEL must be an integer from 1 through $MAX_PARALLEL_LIMIT for the selected runner." >&2
exit 1
fi
if [ -n "${REQUESTED_MAX_PARALLEL:-}" ]; then
if ! [[ "$REQUESTED_MAX_PARALLEL" =~ ^[1-9][0-9]{0,2}$ ]] || [ "$REQUESTED_MAX_PARALLEL" -gt "$MAX_PARALLEL" ]; then
echo "max_parallel must be an integer from 1 through the configured campaign limit." >&2
exit 1
fi
MAX_PARALLEL="$REQUESTED_MAX_PARALLEL"
fi
echo "max_parallel=$MAX_PARALLEL" >> "$GITHUB_OUTPUT"
daytona_image:
name: Publish verified Daytona image
needs: [authorize, target_lock, catalog]
# Keep image builds on the same authorized fleet as campaign execution.
runs-on: ${{ needs.authorize.outputs.test_runner }}
timeout-minutes: 45
permissions:
contents: read
packages: write
id-token: write
outputs:
image: ${{ steps.image.outputs.image }}
source_revision: ${{ steps.image.outputs.source_revision }}
content_id: ${{ steps.image.outputs.content_id }}
steps:
- if: needs.catalog.outputs.needs_daytona == 'true'
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ needs.authorize.outputs.target_sha }}
persist-credentials: false
- name: Download resolved target lockfile
if: needs.catalog.outputs.needs_daytona == 'true'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
artifact-ids: ${{ needs.target_lock.outputs.artifact_id }}
path: ${{ runner.temp }}/runner-e2e-target-lock
- name: Restore resolved target lockfile
if: needs.catalog.outputs.needs_daytona == 'true'
env:
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
EXPECTED_LOCK_SHA256: ${{ needs.target_lock.outputs.lock_sha256 }}
run: |
set -euo pipefail
test "$(git rev-parse HEAD)" = "$TARGET_SHA"
lock="$RUNNER_TEMP/runner-e2e-target-lock/pnpm-lock.yaml"
test -f "$lock"
test "$(find "$(dirname "$lock")" -type f | wc -l | tr -d ' ')" = 1
test "$(sha256sum "$lock" | cut -d ' ' -f 1)" = "$EXPECTED_LOCK_SHA256"
cp "$lock" pnpm-lock.yaml
test "$(sha256sum pnpm-lock.yaml | cut -d ' ' -f 1)" = "$EXPECTED_LOCK_SHA256"
- name: No Daytona image needed
id: local_only
if: needs.catalog.outputs.needs_daytona != 'true'
run: echo "image=" >> "$GITHUB_OUTPUT"
- name: Set up Docker Buildx
if: needs.catalog.outputs.needs_daytona == 'true'
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4
- name: Log into GHCR
if: needs.catalog.outputs.needs_daytona == 'true'
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Install cosign
if: needs.catalog.outputs.needs_daytona == 'true'
uses: sigstore/cosign-installer@398d4b0eeef1380460a10c8013a76f728fb906ac # v3
- name: Reuse or publish immutable image
id: image
env:
NEEDS_DAYTONA: ${{ needs.catalog.outputs.needs_daytona }}
IMAGE_CONTENT_ID: ${{ needs.catalog.outputs.daytona_image_content_id }}
IMAGE_TAG: ghcr.io/paperclipai/paperclip-daytona-runner:e2e-content-${{ needs.catalog.outputs.daytona_image_content_id }}
IMAGE_CACHE: ghcr.io/paperclipai/paperclip-daytona-runner:e2e-buildcache-amd64
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
TARGET_LOCK_SHA256: ${{ needs.target_lock.outputs.lock_sha256 }}
TARGET_REF: ${{ needs.authorize.outputs.target_ref }}
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
run: |
set -euo pipefail
if [ "$NEEDS_DAYTONA" != true ]; then
{
echo "image="
echo "source_revision="
echo "content_id="
} >> "$GITHUB_OUTPUT"
exit 0
fi
[[ "$IMAGE_CONTENT_ID" =~ ^[0-9a-f]{64}$ ]]
[[ "$TARGET_LOCK_SHA256" =~ ^[0-9a-f]{64}$ ]]
identity="^https://github.com/${GITHUB_REPOSITORY}/.github/workflows/runner-full-stack-e2e.yml@"
if docker buildx imagetools inspect "$IMAGE_TAG" >/dev/null 2>&1; then
digest="$(docker buildx imagetools inspect "$IMAGE_TAG" --format '{{json .Manifest.Digest}}' | tr -d '"')"
else
cache_args=(
--cache-from "type=registry,ref=${IMAGE_CACHE}"
)
if [ "$TARGET_REF" = "refs/heads/$DEFAULT_BRANCH" ]; then
cache_args+=(
--cache-to "type=registry,ref=${IMAGE_CACHE},mode=max"
)
echo '::notice title=Daytona image cache::Publishing cache from the trusted default-branch target'
else
echo '::notice title=Daytona image cache::Using the default-branch cache without publishing development-branch layers'
fi
docker buildx build \
--platform linux/amd64 \
--build-arg "PAPERCLIP_RUNNER_CONTENT_ID=${IMAGE_CONTENT_ID}" \
--build-arg "PAPERCLIP_RUNNER_SOURCE_REVISION=${TARGET_SHA}" \
--build-arg "PAPERCLIP_RUNNER_LOCK_SHA256=${TARGET_LOCK_SHA256}" \
--file docker/daytona-runner/Dockerfile \
--tag "$IMAGE_TAG" \
"${cache_args[@]}" \
--push \
.
digest="$(docker buildx imagetools inspect "$IMAGE_TAG" --format '{{json .Manifest.Digest}}' | tr -d '"')"
cosign sign --yes "$IMAGE_TAG@$digest"
fi
cosign verify \
--certificate-identity-regexp "$identity" \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
"$IMAGE_TAG@$digest" >/dev/null
immutable="${IMAGE_TAG%:*}@$digest"
# The Dockerfile's final two RUN steps execute the runner metadata,
# transport-mode, provider-pack JSON, and pinned ACP binary checks as
# root and as the unprivileged Daytona user. Buildx reads the signed
# digest's OCI config directly from GHCR, so verification does not
# download the image's large filesystem layers. Logging out first
# preserves the proof that Daytona can retrieve this public image
# without the workflow's package credentials.
docker logout ghcr.io >/dev/null
image_config="$(docker buildx imagetools inspect "$immutable" \
--format '{{json .Image}}')"
published_content_id="$(jq -r '.config.Labels["io.paperclip.runner.content-id"] // empty' <<< "$image_config")"
source_revision="$(jq -r '.config.Labels["org.opencontainers.image.revision"] // empty' <<< "$image_config")"
test "$published_content_id" = "$IMAGE_CONTENT_ID"
[[ "$source_revision" =~ ^[0-9a-f]{40}$ ]]
jq -e \
'.architecture == "amd64" and
.os == "linux" and
.config.User == "daytona" and
(.config.Env | any(startswith("PAPERCLIP_RUNNER_PROVIDER_PACK_ROOT=")))' \
<<< "$image_config" >/dev/null
{
echo "image=$immutable"
echo "source_revision=$source_revision"
echo "content_id=$published_content_id"
} >> "$GITHUB_OUTPUT"
build_runner_artifacts:
name: Build reusable runner campaign artifacts
needs: [authorize, target_lock, catalog]
if: github.event_name != 'schedule' || (vars.RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED == 'true' && vars.RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED == 'true')
# Compile native binaries on the same reviewed image used to execute them,
# avoiding libc/architecture drift between GitHub-hosted and AWS lanes.
runs-on: ${{ needs.authorize.outputs.test_runner }}
timeout-minutes: 20
permissions:
contents: read
outputs:
build_artifact_name: ${{ steps.build_artifact_name.outputs.name }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ needs.authorize.outputs.target_sha }}
persist-credentials: false
- name: Download resolved target lockfile
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
artifact-ids: ${{ needs.target_lock.outputs.artifact_id }}
path: ${{ runner.temp }}/runner-e2e-target-lock
- name: Restore resolved target lockfile
env:
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
EXPECTED_LOCK_SHA256: ${{ needs.target_lock.outputs.lock_sha256 }}
run: |
set -euo pipefail
test "$(git rev-parse HEAD)" = "$TARGET_SHA"
lock="$RUNNER_TEMP/runner-e2e-target-lock/pnpm-lock.yaml"
test -f "$lock"
test "$(find "$(dirname "$lock")" -type f | wc -l | tr -d ' ')" = 1
test "$(sha256sum "$lock" | cut -d ' ' -f 1)" = "$EXPECTED_LOCK_SHA256"
cp "$lock" pnpm-lock.yaml
test "$(sha256sum pnpm-lock.yaml | cut -d ' ' -f 1)" = "$EXPECTED_LOCK_SHA256"
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
- uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
env:
NPM_CONFIG_AUDIT: "false"
NPM_CONFIG_FUND: "false"
NPM_CONFIG_UPDATE_NOTIFIER: "false"
with:
version: 9.15.4
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
cache: pnpm
- run: pnpm install --frozen-lockfile --ignore-scripts
# build:typescript also builds the eval-kernel dependency, so the two
# TypeScript trees are compiled at most once in this campaign.
- name: Build shared TypeScript and native runner outputs
env:
NEEDS_RUNNER_TYPESCRIPT: ${{ needs.catalog.outputs.needs_runner_typescript }}
NEEDS_NATIVE_BINARIES: ${{ needs.catalog.outputs.needs_native_binaries }}
run: |
set -euo pipefail
if [ "$NEEDS_RUNNER_TYPESCRIPT" = true ]; then
pnpm --filter @paperclipai/paperclip-runner build:typescript
else
pnpm --filter @paperclipai/paperclip-eval-kernel build
fi
if [ "$NEEDS_NATIVE_BINARIES" = true ]; then
pnpm --filter @paperclipai/paperclip-runner build:runner-binaries
fi
- name: Package immutable campaign outputs
env:
NEEDS_RUNNER_TYPESCRIPT: ${{ needs.catalog.outputs.needs_runner_typescript }}
NEEDS_NATIVE_BINARIES: ${{ needs.catalog.outputs.needs_native_binaries }}
run: |
set -euo pipefail
binary_root="packages/paperclip-runner/runner/target/debug"
binaries=(
conformance-tracer
paperclip-runnerd
fake-harness
fake-codex-app-server
fake-acpx-sidecar
)
archive_paths=(
packages/paperclip-eval-kernel/dist
)
if [ "$NEEDS_RUNNER_TYPESCRIPT" = true ]; then
test -d packages/paperclip-runner/dist
archive_paths+=(packages/paperclip-runner/dist)
fi
if [ "$NEEDS_NATIVE_BINARIES" = true ]; then
for binary in "${binaries[@]}"; do
test -x "$binary_root/$binary"
archive_paths+=("$binary_root/$binary")
done
fi
tar --create --gzip \
--file runner-e2e-build-bundle.tar.gz \
"${archive_paths[@]}"
sha256sum runner-e2e-build-bundle.tar.gz > runner-e2e-build-bundle.tar.gz.sha256
- name: Name immutable shared campaign outputs
id: build_artifact_name
env:
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
run: echo "name=runner-e2e-build-${TARGET_SHA}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" >> "$GITHUB_OUTPUT"
- name: Bound scheduled build upload
if: needs.catalog.outputs.operations_cadence != ''
run: node tests/runner-e2e/artifact-budget.mjs check build runner-e2e-build-bundle.tar.gz runner-e2e-build-bundle.tar.gz.sha256
- name: Upload immutable shared campaign outputs
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: ${{ steps.build_artifact_name.outputs.name }}
path: |
runner-e2e-build-bundle.tar.gz
runner-e2e-build-bundle.tar.gz.sha256
retention-days: 1
compression-level: 0
if-no-files-found: error
build_remote_provider_pack:
name: Build reusable remote provider pack
needs:
[authorize, target_lock, catalog, daytona_image, build_runner_artifacts]
if: github.event_name != 'schedule' || (vars.RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED == 'true' && vars.RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED == 'true')
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
contents: read
outputs:
provider_pack_artifact_name: ${{ steps.provider_pack_artifact_name.outputs.name }}
steps:
- name: No remote provider pack needed
if: needs.catalog.outputs.needs_remote_provider_pack != 'true'
run: echo "Selected cells do not require a remote provider pack."
- if: needs.catalog.outputs.needs_remote_provider_pack == 'true'
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ needs.authorize.outputs.target_sha }}
persist-credentials: false
- name: Download resolved target lockfile
if: needs.catalog.outputs.needs_remote_provider_pack == 'true'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
artifact-ids: ${{ needs.target_lock.outputs.artifact_id }}
path: ${{ runner.temp }}/runner-e2e-target-lock
- name: Restore resolved target lockfile
if: needs.catalog.outputs.needs_remote_provider_pack == 'true'
env:
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
EXPECTED_LOCK_SHA256: ${{ needs.target_lock.outputs.lock_sha256 }}
run: |
set -euo pipefail
test "$(git rev-parse HEAD)" = "$TARGET_SHA"
lock="$RUNNER_TEMP/runner-e2e-target-lock/pnpm-lock.yaml"
test -f "$lock"
test "$(find "$(dirname "$lock")" -type f | wc -l | tr -d ' ')" = 1
test "$(sha256sum "$lock" | cut -d ' ' -f 1)" = "$EXPECTED_LOCK_SHA256"
cp "$lock" pnpm-lock.yaml
test "$(sha256sum pnpm-lock.yaml | cut -d ' ' -f 1)" = "$EXPECTED_LOCK_SHA256"
- if: needs.catalog.outputs.needs_remote_provider_pack == 'true'
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
- if: needs.catalog.outputs.needs_remote_provider_pack == 'true'
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
env:
NPM_CONFIG_AUDIT: "false"
NPM_CONFIG_FUND: "false"
NPM_CONFIG_UPDATE_NOTIFIER: "false"
with:
version: 9.15.4
- if: needs.catalog.outputs.needs_remote_provider_pack == 'true'
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
cache: pnpm
- if: needs.catalog.outputs.needs_remote_provider_pack == 'true'
run: pnpm install --frozen-lockfile --ignore-scripts
- name: Materialize verified pinned OpenCode executable
if: needs.catalog.outputs.needs_remote_provider_pack == 'true'
run: node packages/paperclip-runner/scripts/materialize-opencode-binary.mjs
- name: Download immutable shared campaign outputs
if: needs.catalog.outputs.needs_remote_provider_pack == 'true'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: ${{ needs.build_runner_artifacts.outputs.build_artifact_name }}
path: runner-e2e-build
- name: Verify and restore shared TypeScript outputs
if: needs.catalog.outputs.needs_remote_provider_pack == 'true'
run: |
set -euo pipefail
(
cd runner-e2e-build
sha256sum --check runner-e2e-build-bundle.tar.gz.sha256
)
tar --extract --gzip \
--file runner-e2e-build/runner-e2e-build-bundle.tar.gz \
--directory "$GITHUB_WORKSPACE"
test -d packages/paperclip-eval-kernel/dist
test -d packages/paperclip-runner/dist
- name: Assemble native remote provider pack
if: needs.catalog.outputs.needs_remote_provider_pack == 'true'
env:
# A reused image can have an older source revision with the same
# content ID. Matching that revision lets remote execution reuse the
# verified pack already installed in the immutable image.
PAPERCLIP_RUNNER_SOURCE_REVISION: ${{ needs.daytona_image.outputs.source_revision }}
run: node packages/paperclip-runner/scripts/build-provider-pack.mjs packages/paperclip-runner/provider-pack
- name: Package verified remote provider pack
if: needs.catalog.outputs.needs_remote_provider_pack == 'true'
env:
IMAGE_SOURCE_REVISION: ${{ needs.daytona_image.outputs.source_revision }}
run: |
set -euo pipefail
test -f packages/paperclip-runner/provider-pack/provider-pack.json
jq -e \
--arg revision "$IMAGE_SOURCE_REVISION" \
'.schema == "paperclip-runner/remote-provider-pack/v1" and
.payload.runnerSourceRevision == $revision and
(.digest | test("^sha256:[0-9a-f]{64}$"))' \
packages/paperclip-runner/provider-pack/provider-pack.json >/dev/null
tar --create --gzip \
--file runner-e2e-provider-pack.tar.gz \
packages/paperclip-runner/provider-pack
sha256sum runner-e2e-provider-pack.tar.gz > runner-e2e-provider-pack.tar.gz.sha256
- name: Name immutable remote provider pack
id: provider_pack_artifact_name
if: needs.catalog.outputs.needs_remote_provider_pack == 'true'
env:
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
run: echo "name=runner-e2e-provider-pack-${TARGET_SHA}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" >> "$GITHUB_OUTPUT"
- name: Upload immutable remote provider pack
if: needs.catalog.outputs.needs_remote_provider_pack == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: ${{ steps.provider_pack_artifact_name.outputs.name }}
path: |
runner-e2e-provider-pack.tar.gz
runner-e2e-provider-pack.tar.gz.sha256
retention-days: 1
compression-level: 0
if-no-files-found: error
test:
name: ${{ matrix.executionId }}
if: needs.catalog.outputs.operations_cadence == ''
needs:
[
authorize,
target_lock,
catalog,
daytona_image,
build_runner_artifacts,
build_remote_provider_pack,
]
# The authorize job selects only one of two literal, reviewed runner labels;
# no dispatch input or repository variable can inject an arbitrary label.
runs-on: ${{ needs.authorize.outputs.test_runner }}
timeout-minutes: ${{ matrix.timeoutMinutes }}
permissions:
contents: read
environment:
name: runner-e2e-paid
strategy:
fail-fast: false
max-parallel: ${{ fromJSON(needs.catalog.outputs.max_parallel) }}
matrix: ${{ fromJSON(needs.catalog.outputs.matrix) }}
steps: &paid_test_steps
- name: Reauthorize paid execution before provider access
env:
GH_TOKEN: ${{ github.token }}
REF: ${{ github.ref }}
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
ACTOR_ID: ${{ github.actor_id }}
TRIGGERING_ACTOR: ${{ github.triggering_actor }}
ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
run: |
set -euo pipefail
test "$REF" = "refs/heads/$DEFAULT_BRANCH"
jq -e 'type == "array" and length > 0 and all(.[]; type == "number" and . > 0 and floor == .)' <<< "${ALLOWED_ACTOR_IDS:-}" >/dev/null
# Retry transient transport/server failures, but never proceed without
# a successfully resolved identity and the allowlist checks below.
for attempt in 1 2 3; do
if triggering_actor_id="$(timeout 30s gh api "users/$TRIGGERING_ACTOR" --jq .id)"; then
break
fi
if [ "$attempt" = 3 ]; then exit 1; fi
sleep "$((attempt * 2))"
done
jq -e --argjson candidate "$triggering_actor_id" 'index($candidate) != null' <<< "$ALLOWED_ACTOR_IDS" >/dev/null
jq -e --argjson candidate "$ACTOR_ID" 'index($candidate) != null' <<< "$ALLOWED_ACTOR_IDS" >/dev/null
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ needs.authorize.outputs.target_sha }}
persist-credentials: false
- name: Download resolved target lockfile
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
artifact-ids: ${{ needs.target_lock.outputs.artifact_id }}
path: ${{ runner.temp }}/runner-e2e-target-lock
- name: Restore resolved target lockfile
env:
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
EXPECTED_LOCK_SHA256: ${{ needs.target_lock.outputs.lock_sha256 }}
run: |
set -euo pipefail
test "$(git rev-parse HEAD)" = "$TARGET_SHA"
lock="$RUNNER_TEMP/runner-e2e-target-lock/pnpm-lock.yaml"
test -f "$lock"
test "$(find "$(dirname "$lock")" -type f | wc -l | tr -d ' ')" = 1
test "$(sha256sum "$lock" | cut -d ' ' -f 1)" = "$EXPECTED_LOCK_SHA256"
cp "$lock" pnpm-lock.yaml
test "$(sha256sum pnpm-lock.yaml | cut -d ' ' -f 1)" = "$EXPECTED_LOCK_SHA256"
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
- uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
env:
NPM_CONFIG_AUDIT: "false"
NPM_CONFIG_FUND: "false"
NPM_CONFIG_UPDATE_NOTIFIER: "false"
with:
version: 9.15.4
# This job receives provider credentials only in the final paid-test
# step. Keep target-selected dependency lifecycle code from running in
# the protected environment during setup.
- run: pnpm install --frozen-lockfile --ignore-scripts
# Sandbox-provider plugins are intentionally excluded from the root
# workspace. The ordinary root postinstall links the in-repo plugin SDK,
# but that lifecycle hook is deliberately disabled above. Prepare the
# one host plugin needed by Daytona explicitly, before this job receives
# provider credentials, and keep dependency lifecycle scripts disabled.
- name: Prepare bundled Daytona plugin without dependency lifecycle scripts
if: matrix.environmentId == 'daytona'
run: |
set -euo pipefail
daytona_root="packages/plugins/sandbox-providers/daytona"
sdk_root="packages/plugins/sdk"
test -d "$daytona_root"
test -d "$sdk_root"
test ! -L "$daytona_root"
test ! -L "$sdk_root"
test -f "$daytona_root/pnpm-lock.yaml"
test "$(jq -r .name "$daytona_root/package.json")" = "@paperclipai/plugin-daytona"
test "$(jq -r .name "$sdk_root/package.json")" = "@paperclipai/plugin-sdk"
(
cd "$daytona_root"
pnpm install --ignore-workspace --frozen-lockfile --ignore-scripts
)
node scripts/link-plugin-dev-sdk.mjs
test "$(realpath "$daytona_root/node_modules/@paperclipai/plugin-sdk")" = "$(realpath "$sdk_root")"
pnpm --dir "$daytona_root" build
test -f "$daytona_root/dist/manifest.js"
test -f "$daytona_root/dist/worker.js"
test -e "$daytona_root/node_modules/@daytonaio/sdk"
- name: Materialize verified pinned OpenCode executable
if: matrix.environmentId == 'local' && (matrix.profileId == 'legacy-opencode' || matrix.profileId == 'runner-opencode' || matrix.suiteId == 'openrouter-model-breadth')
run: node packages/paperclip-runner/scripts/materialize-opencode-binary.mjs
- name: Install checksum-verified Grok executable
if: matrix.environmentId == 'local' && (matrix.profileId == 'runner-acpx-grok' || matrix.profileId == 'runner-acpx-grok-subscription')
run: sudo node packages/paperclip-runner/scripts/provision-grok.mjs /opt/paperclip/providers/grok/1.0.13/grok
- name: Download immutable campaign outputs
if: startsWith(matrix.profileId, 'runner-') || matrix.suiteId == 'openrouter-model-breadth'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: ${{ needs.build_runner_artifacts.outputs.build_artifact_name }}
path: runner-e2e-build
- name: Download immutable remote provider pack
if: matrix.environmentId == 'daytona' && (matrix.profileId == 'runner-opencode' || startsWith(matrix.profileId, 'runner-acpx-'))
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: ${{ needs.build_remote_provider_pack.outputs.provider_pack_artifact_name }}
path: runner-e2e-provider-pack
- name: Verify and restore campaign outputs
if: startsWith(matrix.profileId, 'runner-') || matrix.suiteId == 'openrouter-model-breadth'
env:
NEEDS_RUNNER_TYPESCRIPT: ${{ matrix.profileId == 'runner-opencode' || startsWith(matrix.profileId, 'runner-acpx-') || matrix.suiteId == 'openrouter-model-breadth' }}
NEEDS_NATIVE_BINARY: ${{ startsWith(matrix.profileId, 'runner-') || matrix.suiteId == 'openrouter-model-breadth' }}
run: |
set -euo pipefail
(
cd runner-e2e-build
sha256sum --check runner-e2e-build-bundle.tar.gz.sha256
)
tar --extract --gzip \
--file runner-e2e-build/runner-e2e-build-bundle.tar.gz \
--directory "$GITHUB_WORKSPACE"
test -d packages/paperclip-eval-kernel/dist
if [ "$NEEDS_RUNNER_TYPESCRIPT" = true ]; then
test -d packages/paperclip-runner/dist
fi
if [ "$NEEDS_NATIVE_BINARY" = true ]; then
test -x packages/paperclip-runner/runner/target/debug/paperclip-runnerd
fi
- name: Verify and restore remote provider pack
if: matrix.environmentId == 'daytona' && (matrix.profileId == 'runner-opencode' || startsWith(matrix.profileId, 'runner-acpx-'))
env:
IMAGE_SOURCE_REVISION: ${{ needs.daytona_image.outputs.source_revision }}
run: |
set -euo pipefail
(
cd runner-e2e-provider-pack
sha256sum --check runner-e2e-provider-pack.tar.gz.sha256
)
tar --extract --gzip \
--file runner-e2e-provider-pack/runner-e2e-provider-pack.tar.gz \
--directory "$GITHUB_WORKSPACE"
jq -e \
--arg revision "$IMAGE_SOURCE_REVISION" \
'.schema == "paperclip-runner/remote-provider-pack/v1" and
.payload.runnerSourceRevision == $revision and
(.digest | test("^sha256:[0-9a-f]{64}$"))' \
packages/paperclip-runner/provider-pack/provider-pack.json >/dev/null
- name: Qualify local provider Node interpreter
if: matrix.environmentId == 'local' && (matrix.profileId == 'runner-opencode' || startsWith(matrix.profileId, 'runner-acpx-') || matrix.suiteId == 'openrouter-model-breadth')
run: |
node <<'NODE'
const fs = require("node:fs");
const mode = fs.statSync(process.execPath).mode & 0o777;
fs.chmodSync(process.execPath, mode & ~0o022);
if ((fs.statSync(process.execPath).mode & 0o022) !== 0) {
throw new Error("provider Node interpreter remains group- or world-writable");
}
NODE
- name: Install pinned Claude CLI for setup verification
if: matrix.profileId == 'legacy-claude' || matrix.profileId == 'runner-acpx-claude'
run: |
npm install --global --omit=dev --ignore-scripts @anthropic-ai/claude-code@2.1.277
launcher="$(npm root --global)/@anthropic-ai/claude-code/cli-wrapper.cjs"
chmod +x "$launcher"
ln -sf "$launcher" "$(npm prefix --global)/bin/claude"
claude --version
- name: Qualify preinstalled Chrome
if: needs.authorize.outputs.playwright_channel == 'chrome'
run: |
set -euo pipefail
chrome_path="$(command -v google-chrome)"
test -x "$chrome_path"
google-chrome --version
- name: Install Playwright FFmpeg on AWS runner
if: needs.authorize.outputs.playwright_channel == 'chrome'
run: |
set -euo pipefail
for attempt in 1 2 3; do
if pnpm exec playwright install ffmpeg; then
exit 0
fi
if [ "$attempt" -eq 3 ]; then
echo "Playwright FFmpeg installation failed after $attempt attempts." >&2
exit 1
fi
sleep "$((attempt * 10))"
done
- name: Install Chromium headless shell on GitHub-hosted fallback
if: needs.authorize.outputs.playwright_channel != 'chrome'
run: |
set -euo pipefail
for attempt in 1 2 3; do
if pnpm exec playwright install --with-deps --only-shell chromium; then
exit 0
fi
if [ "$attempt" -eq 3 ]; then
echo "Chromium headless shell installation failed after $attempt attempts." >&2
exit 1
fi
sleep "$((attempt * 10))"
done
# This definition executes only from the authorized default-branch workflow.
# Provision host policy before credentials reach target-controlled tests.
- name: Provision Codex sandbox on the disposable trusted runner
if: matrix.environmentId == 'local' && (needs.catalog.outputs.operations_cadence == 'weekly' || matrix.profileId == 'runner-codex' || matrix.profileId == 'runner-codex-mini' || matrix.profileId == 'runner-acpx-codex' || matrix.profileId == 'runner-codex-subscription')
run: |
node --input-type=module <<'NODE'
import { execFileSync } from "node:child_process";
import { createHash } from "node:crypto";
import { readFileSync, realpathSync, writeFileSync } from "node:fs";
import { createRequire } from "node:module";
import path from "node:path";
if (process.platform !== "linux") process.exit(0);
let restricted = "0";
try { restricted = readFileSync("/proc/sys/kernel/apparmor_restrict_unprivileged_userns", "utf8").trim(); } catch {}
if (restricted !== "1") process.exit(0);
const root = realpathSync(process.env.GITHUB_WORKSPACE);
const runnerRequire = createRequire(path.join(root, "packages/paperclip-runner/package.json"));
const acpRequire = createRequire(runnerRequire.resolve("@agentclientprotocol/codex-acp/package.json"));
const codexRequire = createRequire(acpRequire.resolve("@openai/codex/package.json"));
const arch = process.arch === "x64" ? "x64" : process.arch === "arm64" ? "arm64" : null;
if (!arch) throw new Error("Unsupported Codex CI architecture");
const platformPackage = codexRequire.resolve(`@openai/codex-linux-${arch}/package.json`);
const triple = arch === "x64" ? "x86_64-unknown-linux-musl" : "aarch64-unknown-linux-musl";
const suffix = `/vendor/${triple}/bin/codex`;
const binary = realpathSync(path.join(path.dirname(platformPackage), suffix));
if (!binary.startsWith(root + "/node_modules/.pnpm/") || !binary.endsWith(suffix) || !/^[/A-Za-z0-9_.@+\-]+$/.test(binary)) {
throw new Error("Codex executable is outside the resolved dependency tree");
}
const name = `paperclip-e2e-codex-${createHash("sha256").update(binary).digest("hex").slice(0,16)}`;
const profilePath = path.join(process.env.RUNNER_TEMP, "paperclip-codex-userns.apparmor");
writeFileSync(profilePath, `abi <abi/4.0>,\ninclude <tunables/global>\nprofile ${name} "${binary}" flags=(unconfined) {\n userns,\n}\n`, {mode:0o600, flag:"wx"});
execFileSync("sudo", ["-n", "apparmor_parser", "-r", profilePath], {timeout:15000, stdio:"pipe"});
NODE
- name: Prepare pinned Python artifact oracle image
if: (matrix.suiteId == 'everyday-workflows' || matrix.suiteId == 'grok-qualification' || matrix.suiteId == 'grok-subscription-qualification') && (matrix.caseId == 'build-revise' || matrix.caseId == 'delegate-feedback' || matrix.caseId == 'agent-review-handoff' || matrix.caseId == 'hire-reuse' || matrix.caseId == 'recover-controller' || matrix.caseId == 'stop-redirect')
run: |
set -euo pipefail
oracle_image='python@sha256:9d2e5553305c7c7b0097999bb17187c69b921ccd6bc9d40e4bb5ebe652c00285'
timeout 30s docker version --format '{{.Server.Version}}'
timeout 120s docker pull "$oracle_image"
timeout 30s docker image inspect "$oracle_image" --format '{{.Id}}'
- name: Obtain scoped private history credentials
id: private_history_credentials
if: needs.catalog.outputs.operations_cadence != ''
uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6
with:
role-to-assume: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_ROLE_ARN }}
aws-region: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
output-env-credentials: false
output-credentials: true
disable-retry: true
action-timeout-s: 60
- name: Verify private evidence retention before paid execution
if: needs.catalog.outputs.operations_cadence != ''
env:
AWS_ACCESS_KEY_ID: ${{ steps.private_history_credentials.outputs.aws-access-key-id }}
AWS_SECRET_ACCESS_KEY: ${{ steps.private_history_credentials.outputs.aws-secret-access-key }}
AWS_SESSION_TOKEN: ${{ steps.private_history_credentials.outputs.aws-session-token }}
AWS_DEFAULT_REGION: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET }}
run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/private-history.ts verify evidence "$GITHUB_RUN_ID" bounded-runner-e2e-evidence
- name: Prepare bounded campaign before credentials
if: needs.catalog.outputs.operations_cadence != ''
env:
EVAL_EVENT: ${{ github.event_name }}
EVAL_SCHEDULE: ${{ github.event.schedule }}
EVAL_CADENCE: ${{ needs.catalog.outputs.operations_cadence }}
EVAL_OPERATIONS_ENABLED: ${{ vars.RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED }}
EVAL_OPERATING_OVERHEAD_JSON: ${{ needs.catalog.outputs.operating_overhead }}
EVAL_TEST_RUNNER: ${{ needs.authorize.outputs.test_runner }}
PAPERCLIP_E2E_BRAINTRUST_PROJECT_ID: ${{ vars.RUNNER_E2E_BRAINTRUST_PROJECT_ID }}
run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/scheduled-operations.ts prepare
- name: Run bounded campaign and retain feedback review
if: needs.catalog.outputs.operations_cadence != ''
env:
BRAINTRUST_API_KEY: ${{ secrets.BRAINTRUST_API_KEY }}
PAPERCLIP_E2E_BRAINTRUST_PROJECT_ID: ${{ vars.RUNNER_E2E_BRAINTRUST_PROJECT_ID }}
PAPERCLIP_RUNNER_E2E_SOURCE_SHA: ${{ needs.authorize.outputs.target_sha }}
PAPERCLIP_RUNNER_E2E_SOURCE_REF: ${{ needs.authorize.outputs.target_ref }}
PAPERCLIP_PLAYWRIGHT_CHANNEL: ${{ needs.authorize.outputs.playwright_channel }}
run: |
set +e
args=(--max-parallel 1 --max-automatic-retries 0)
while IFS= read -r id; do args+=(--id "$id"); done < <(jq -r '.executionIds[]' "$PAPERCLIP_E2E_OPERATIONS_MANIFEST")
pnpm test:e2e:runner -- "${args[@]}"
run_status=$?
node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/scheduled-finalization.ts
finalization_status=$?
test "$run_status" = 0 && test "$finalization_status" = 0
- name: Run paid cell
if: needs.catalog.outputs.operations_cadence == ''
env:
OPENAI_API_KEY: ${{ matrix.credentialName == 'OPENAI_API_KEY' && secrets.OPENAI_API_KEY || '' }}
ANTHROPIC_API_KEY: ${{ matrix.credentialName == 'ANTHROPIC_API_KEY' && secrets.ANTHROPIC_API_KEY || '' }}
OPENROUTER_API_KEY: ${{ matrix.credentialName == 'OPENROUTER_API_KEY' && secrets.OPENROUTER_API_KEY || '' }}
XAI_API_KEY: ${{ matrix.credentialName == 'XAI_API_KEY' && secrets.XAI_API_KEY || '' }}
GROK_AUTH_JSON: ${{ matrix.credentialName == 'GROK_AUTH_JSON' && secrets.GROK_AUTH_JSON || '' }}
COPILOT_GITHUB_TOKEN: ${{ matrix.credentialName == 'COPILOT_GITHUB_TOKEN' && secrets.COPILOT_GITHUB_TOKEN || '' }}
DAYTONA_API_KEY: ${{ matrix.environmentId == 'daytona' && secrets.DAYTONA_API_KEY || '' }}
PAPERCLIP_E2E_DAYTONA_IMAGE: ${{ needs.daytona_image.outputs.image }}
PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH: ${{ github.workspace }}/packages/paperclip-runner/provider-pack
PAPERCLIP_E2E_CAMPAIGN_ID: gha-${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.executionId }}
PAPERCLIP_E2E_ACCOUNTING_CAMPAIGN_ID: gha-${{ github.run_id }}-${{ github.run_attempt }}
PAPERCLIP_RUNNER_E2E_SOURCE_SHA: ${{ needs.authorize.outputs.target_sha }}
PAPERCLIP_RUNNER_E2E_SOURCE_REF: ${{ needs.authorize.outputs.target_ref }}
PAPERCLIP_PLAYWRIGHT_CHANNEL: ${{ needs.authorize.outputs.playwright_channel }}
run: pnpm test:e2e:runner -- --id "${{ matrix.executionId }}"
- name: Stage bounded campaign evidence
id: stage_evidence
if: always() && needs.catalog.outputs.operations_cadence != ''
run: node tests/runner-e2e/artifact-budget.mjs stage evidence tests/runner-e2e/results bounded-runner-e2e-evidence
- name: Upload access-controlled packaged cell evidence
if: always() && needs.catalog.outputs.operations_cadence == ''
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: runner-e2e-${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.executionId }}
path: tests/runner-e2e/results/
retention-days: 30
if-no-files-found: error
- name: Retain bounded campaign evidence in private AWS storage
if: always() && needs.catalog.outputs.operations_cadence != '' && steps.private_history_credentials.outcome == 'success'
env:
AWS_ACCESS_KEY_ID: ${{ steps.private_history_credentials.outputs.aws-access-key-id }}
AWS_SECRET_ACCESS_KEY: ${{ steps.private_history_credentials.outputs.aws-secret-access-key }}
AWS_SESSION_TOKEN: ${{ steps.private_history_credentials.outputs.aws-session-token }}
AWS_DEFAULT_REGION: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET }}
run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/private-history.ts upload evidence "$GITHUB_RUN_ID" bounded-runner-e2e-evidence
- name: Enforce complete bounded evidence retention
if: always() && needs.catalog.outputs.operations_cadence != '' && steps.stage_evidence.outputs.complete != 'true'
run: exit 1
test_recurring:
name: ${{ matrix.executionId }}
if: needs.catalog.outputs.operations_cadence != '' && needs.authorize.outputs.target_sha == github.sha && github.ref == format('refs/heads/{0}', github.event.repository.default_branch)
needs: [authorize, target_lock, catalog, daytona_image, build_runner_artifacts, build_remote_provider_pack]
runs-on: ${{ needs.authorize.outputs.test_runner }}
timeout-minutes: ${{ matrix.timeoutMinutes }}
permissions:
contents: read
id-token: write
environment:
name: runner-e2e-private-paid
strategy:
fail-fast: false
max-parallel: ${{ fromJSON(needs.catalog.outputs.max_parallel) }}
matrix: ${{ fromJSON(needs.catalog.outputs.matrix) }}
steps: *paid_test_steps
report:
name: Merge and enforce campaign result
if: always() && !cancelled() && needs.catalog.result == 'success' && github.ref == format('refs/heads/{0}', github.event.repository.default_branch)
needs: [authorize, catalog, daytona_image, test, test_recurring]
outputs:
history_source_ready: ${{ steps.history_source_ready.outputs.ready }}
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
actions: read
contents: read
id-token: write
environment:
name: runner-e2e-private-history
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
# Reporting and sanitization are part of the trusted workflow boundary.
ref: ${{ github.sha }}
persist-credentials: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
- uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
env:
NPM_CONFIG_AUDIT: "false"
NPM_CONFIG_FUND: "false"
NPM_CONFIG_UPDATE_NOTIFIER: "false"
with:
version: 9.15.4
- name: Apply reviewed reporting lock corrections
run: node tests/runner-e2e/prepare-report-lock.mjs
- name: Install pinned reporting dependencies
run: pnpm install --frozen-lockfile --ignore-scripts --store-dir "${{ runner.temp }}/eval-report-store"
- name: Resolve workflow job attempts
env:
GH_TOKEN: ${{ github.token }}
REPOSITORY: ${{ github.repository }}
RUN_ID: ${{ github.run_id }}
run: |
set -euo pipefail
gh api --paginate --slurp \
"repos/$REPOSITORY/actions/runs/$RUN_ID/jobs?filter=all&per_page=100" \
> runner-e2e-job-pages.json
for attempt in $(seq 1 "${{ github.run_attempt }}"); do
gh api "repos/$REPOSITORY/actions/runs/$RUN_ID/attempts/$attempt" \
--jq '{run_attempt, run_started_at}'
done > runner-e2e-attempts.jsonl
jq -s '.' runner-e2e-attempts.jsonl > runner-e2e-attempts.json
jq --slurpfile attempts runner-e2e-attempts.json \
'{jobs: [.[].jobs[]], attempts: $attempts[0]}' \
runner-e2e-job-pages.json > runner-e2e-jobs.json
- name: Download cell evidence
id: download_evidence
if: needs.catalog.outputs.operations_cadence == ''
continue-on-error: true
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
pattern: runner-e2e-${{ github.run_id }}-*-*
path: downloaded-runner-e2e
merge-multiple: false
- name: Retry cell evidence download after transport failure
if: needs.catalog.outputs.operations_cadence == '' && steps.download_evidence.outcome == 'failure'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
pattern: runner-e2e-${{ github.run_id }}-*-*
path: downloaded-runner-e2e
merge-multiple: false
- name: Obtain scoped private history credentials
id: private_history_credentials
if: needs.catalog.outputs.operations_cadence != ''
uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6
with:
role-to-assume: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_ROLE_ARN }}
aws-region: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
output-env-credentials: false
output-credentials: true
disable-retry: true
action-timeout-s: 60
- name: Download private bounded campaign evidence
if: needs.catalog.outputs.operations_cadence != ''
env:
AWS_ACCESS_KEY_ID: ${{ steps.private_history_credentials.outputs.aws-access-key-id }}
AWS_SECRET_ACCESS_KEY: ${{ steps.private_history_credentials.outputs.aws-secret-access-key }}
AWS_SESSION_TOKEN: ${{ steps.private_history_credentials.outputs.aws-session-token }}
AWS_DEFAULT_REGION: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET }}
run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/private-history.ts download evidence "$GITHUB_RUN_ID" "downloaded-runner-e2e/runner-e2e-$GITHUB_RUN_ID-1-scheduled-campaign"
- name: Select latest workflow attempt per cell
if: always() && needs.catalog.outputs.operations_cadence == ''
env:
PAPERCLIP_RUNNER_E2E_ARTIFACT_ROOT: ${{ github.workspace }}/downloaded-runner-e2e
PAPERCLIP_RUNNER_E2E_SELECTED_ROOT: ${{ github.workspace }}/selected-runner-e2e
PAPERCLIP_RUNNER_E2E_JOBS_JSON: ${{ github.workspace }}/runner-e2e-jobs.json
PAPERCLIP_RUNNER_E2E_EXPECTED_IDS: ${{ needs.catalog.outputs.execution_ids }}
PAPERCLIP_RUNNER_E2E_SOURCE_SHA: ${{ needs.authorize.outputs.target_sha }}
PAPERCLIP_RUNNER_E2E_SOURCE_REF: ${{ needs.authorize.outputs.target_ref }}
run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/select-rerun-artifacts.ts
- name: Preserve bounded campaign evidence and journal
if: always() && needs.catalog.outputs.operations_cadence != ''
env:
RUN_ID: ${{ github.run_id }}
run: |
set -euo pipefail
artifact="downloaded-runner-e2e/runner-e2e-$RUN_ID-1-scheduled-campaign"
directory="$GITHUB_WORKSPACE/selected-runner-e2e/runner-e2e-$RUN_ID-1-scheduled-campaign/scheduled"
echo "PAPERCLIP_E2E_OPERATIONS_LEDGER=$directory/ledger.jsonl" >> "$GITHUB_ENV"
test -d "$artifact"
mkdir -p selected-runner-e2e
cp -R "$artifact" selected-runner-e2e/
if [ -f "$directory/feedback-review.json" ]; then
echo "PAPERCLIP_E2E_FEEDBACK_REVIEW=$directory/feedback-review.json" >> "$GITHUB_ENV"
fi
if [ -f "$directory/experiment-receipt.json" ]; then
echo "PAPERCLIP_E2E_BRAINTRUST_EXPERIMENT=$directory/experiment-receipt.json" >> "$GITHUB_ENV"
fi
- name: Collect blob reports
run: |
set -euo pipefail
mkdir -p merged-blob-reports
while IFS= read -r -d '' report; do
digest="$(sha256sum "$report" | cut -d ' ' -f 1)"
target="merged-blob-reports/report-${digest}.zip"
if [ ! -e "$target" ]; then
cp "$report" "$target"
fi
done < <(find selected-runner-e2e -path '*/blob-report/*.zip' -print0)
- name: Merge Playwright HTML and JUnit
if: always()
env:
PAPERCLIP_RUNNER_E2E_MERGED_REPORT_DIR: ${{ github.workspace }}/runner-e2e-merged-report
run: pnpm exec playwright merge-reports --config tests/runner-e2e/merge.config.ts merged-blob-reports
- name: Aggregate normalized campaign results
if: always()
env:
PAPERCLIP_RUNNER_E2E_REPORT_ROOT: ${{ github.workspace }}/selected-runner-e2e
PAPERCLIP_RUNNER_E2E_REPORT_OUT: ${{ github.workspace }}/runner-e2e-merged-report/normalized
PAPERCLIP_RUNNER_E2E_EXPECTED_IDS: ${{ needs.catalog.outputs.execution_ids }}
PAPERCLIP_E2E_CAMPAIGN_ID: ${{ needs.catalog.outputs.operations_cadence != '' && format('gha-{0}-1-scheduled-campaign', github.run_id) || format('gha-{0}-{1}', github.run_id, github.run_attempt) }}
PAPERCLIP_RUNNER_E2E_SOURCE_SHA: ${{ needs.authorize.outputs.target_sha }}
PAPERCLIP_RUNNER_E2E_SOURCE_REF: ${{ needs.authorize.outputs.target_ref }}
PAPERCLIP_RUNNER_E2E_HISTORY_PUBLIC_BASE_URL: ${{ needs.catalog.outputs.operations_cadence == '' && vars.RUNNER_E2E_HISTORY_PUBLIC_BASE_URL || '' }}
PAPERCLIP_RUNNER_E2E_HISTORY_PREFIX: ${{ vars.RUNNER_E2E_HISTORY_PREFIX || 'runner-e2e' }}
EVAL_OPERATIONS_CADENCE: ${{ needs.catalog.outputs.operations_cadence }}
run: |
set +e
if [ -n "$EVAL_OPERATIONS_CADENCE" ]; then
mkdir -p runner-e2e-merged-report
pnpm test:e2e:runner:report > runner-e2e-merged-report/report.log 2>&1
else
pnpm test:e2e:runner:report
fi
report_status=$?
set -e
if [ -z "$EVAL_OPERATIONS_CADENCE" ]; then
cat runner-e2e-merged-report/normalized/summary.md >> "$GITHUB_STEP_SUMMARY"
fi
exit "$report_status"
- name: Stage bounded merged report
id: stage_report
if: always() && needs.catalog.outputs.operations_cadence != ''
run: node tests/runner-e2e/artifact-budget.mjs stage report runner-e2e-merged-report bounded-runner-e2e-report
- name: Upload access-controlled merged report
if: always() && needs.catalog.outputs.operations_cadence == ''
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: runner-e2e-report-${{ github.run_id }}-${{ github.run_attempt }}
path: runner-e2e-merged-report/
retention-days: 30
if-no-files-found: error
- name: Retain bounded report in private AWS storage
id: private_report
if: always() && needs.catalog.outputs.operations_cadence != '' && steps.private_history_credentials.outcome == 'success'
env:
AWS_ACCESS_KEY_ID: ${{ steps.private_history_credentials.outputs.aws-access-key-id }}
AWS_SECRET_ACCESS_KEY: ${{ steps.private_history_credentials.outputs.aws-secret-access-key }}
AWS_SESSION_TOKEN: ${{ steps.private_history_credentials.outputs.aws-session-token }}
AWS_DEFAULT_REGION: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET }}
run: |
node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/private-history.ts upload report "$GITHUB_RUN_ID" bounded-runner-e2e-report
echo '{"version":1,"privateReportRetained":true}' > private-report-status.json
echo 'The recurring campaign report is retained in private AWS storage. See the eval operations runbook for authenticated retrieval.' >> "$GITHUB_STEP_SUMMARY"
- name: Retain report availability for the schedule monitor
if: always() && needs.catalog.outputs.operations_cadence != '' && steps.private_report.outcome == 'success'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: runner-e2e-report-${{ github.run_id }}-${{ github.run_attempt }}
path: private-report-status.json
retention-days: 30
if-no-files-found: error
- name: Enforce complete bounded report retention
if: always() && needs.catalog.outputs.operations_cadence != '' && steps.stage_report.outputs.complete != 'true'
run: exit 1
- name: Verify normalized history source report
id: history_source_ready
if: always()
run: |
set -euo pipefail
dashboard_root="runner-e2e-merged-report/normalized"
if [ -f "$dashboard_root/index.html" ] && [ -f "$dashboard_root/normalized-results.json" ]; then
echo "ready=true" >> "$GITHUB_OUTPUT"
else
echo "ready=false" >> "$GITHUB_OUTPUT"
fi
publish_history:
name: Publish S3 history and Pages bundle with declared screenshots
needs: [authorize, catalog, report]
if: always() && needs.catalog.result == 'success' && needs.catalog.outputs.operations_cadence == '' && needs.report.outputs.history_source_ready == 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
pages_artifact_name: ${{ steps.pages_artifact_name.outputs.name }}
concurrency:
group: runner-e2e-history-publish
cancel-in-progress: false
permissions:
contents: read
id-token: write
environment:
name: runner-e2e-history
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
# Never execute target-controlled publication code with AWS credentials.
ref: ${{ github.sha }}
persist-credentials: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
- uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
env:
NPM_CONFIG_AUDIT: "false"
NPM_CONFIG_FUND: "false"
NPM_CONFIG_UPDATE_NOTIFIER: "false"
with:
version: 9.15.4
# Use a job-local store and disable lifecycle scripts throughout install.
# OIDC is available to every step, even before the AWS credential exchange.
- name: Apply reviewed publication lock corrections
run: node tests/runner-e2e/prepare-report-lock.mjs
- name: Install pinned publication dependencies
run: pnpm install --frozen-lockfile --ignore-scripts --store-dir "${{ runner.temp }}/eval-publisher-store"
- name: Install publisher-only Chromium
run: pnpm exec playwright install --with-deps --only-shell chromium
- name: Download access-controlled normalized campaign
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: runner-e2e-report-${{ github.run_id }}-${{ github.run_attempt }}
path: runner-e2e-merged-report
- name: Exchange GitHub OIDC identity for scoped AWS credentials
uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6
with:
role-to-assume: ${{ vars.RUNNER_E2E_HISTORY_AWS_ROLE_ARN }}
aws-region: ${{ vars.RUNNER_E2E_HISTORY_AWS_REGION }}
- name: Publish trusted summary and declared screenshots to public bundles
env:
PAPERCLIP_RUNNER_E2E_REPORT_DIR: ${{ github.workspace }}/runner-e2e-merged-report/normalized
RUNNER_E2E_HISTORY_S3_BUCKET: ${{ vars.RUNNER_E2E_HISTORY_S3_BUCKET }}
RUNNER_E2E_HISTORY_PREFIX: ${{ vars.RUNNER_E2E_HISTORY_PREFIX || 'runner-e2e' }}
RUNNER_E2E_HISTORY_PUBLIC_BASE_URL: ${{ vars.RUNNER_E2E_HISTORY_PUBLIC_BASE_URL }}
run: pnpm test:e2e:runner:history:publish
- name: Resolve Pages artifact name
id: pages_artifact_name
if: vars.RUNNER_FULL_STACK_E2E_PUBLISH_PAGES == 'true'
run: echo "name=github-pages-${{ github.run_id }}-${{ github.run_attempt }}" >> "$GITHUB_OUTPUT"
- name: Package pruned dashboard with declared screenshots for GitHub Pages
if: vars.RUNNER_FULL_STACK_E2E_PUBLISH_PAGES == 'true'
uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0
with:
name: ${{ steps.pages_artifact_name.outputs.name }}
path: runner-e2e-merged-report/pages
pages:
name: Publish latest dashboard with declared screenshots
needs: [report, publish_history]
if: always() && needs.report.outputs.history_source_ready == 'true' && needs.publish_history.result == 'success' && vars.RUNNER_FULL_STACK_E2E_PUBLISH_PAGES == 'true'
runs-on: ubuntu-latest
permissions:
pages: write
id-token: write
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@368f82528645a54fb793d4d04e342629a3f51346 # v5.0.1
with:
# If only this failed job is rerun, GitHub retains the successful
# publisher job's output from the earlier workflow attempt.
artifact_name: ${{ needs.publish_history.outputs.pages_artifact_name }}
Reference in new issue View Git Blame Copy Permalink