mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
feat(evals): run the staging-soak reliability cases from a TestOps plan (#19549)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The runner full-stack E2E lane runs real Paperclip work against real agents, and TestOps launches its standing plans > - PR #19455 added seven staging-soak reliability cases, but no TestOps plan or workflow lane selects them, so they run only by hand with personal keys > - These cases catch restart, interrupt, queue and lost-runner regressions, so they must run from TestOps before a release with the standard reports > - This pull request adds a sibling "Reliability — staging soak" TestOps plan on the existing plan-2 lane, with native OpenCode and DeepSeek 1017 through OpenRouter as the plan profile for six of seven cases > - The benefit is a cheap, repeatable pre-release reliability launch with the same Allure/TestOps results, retained report and dashboard as plan 2 ## Linked Issues or Issue Description Refs #19455 **What existing behavior does this improve?** The staging-soak reliability cases from #19455 run only by a manual local command. **Subsystem affected** Runner full-stack E2E (`tests/runner-e2e/`) and its TestOps lane in `.github/workflows/runner-full-stack-e2e.yml`. **Current behavior** No TestOps plan includes the cases. The workflow cannot select them from TestOps. Every cell uses native Codex (`gpt-5.6-sol`) or an ACPX facade on Daytona, which is expensive. **Proposed behavior** A TestOps launch of the Reliability plan runs seven cells (six DeepSeek 1017, one native Codex for steering) through the standing plan-2 path and reports them like plan 2. **Reason and benefit** The cases become a first-class, low-cost, pre-release check with retained evidence. Plan 2 keeps its 20-case qualification semantics. **Breaking changes** None. Plan 2 selection, reporting and history identities stay the same. ## What Changed - **Plan choice: a sibling plan, not an extension of plan 2.** Plan 2 is bounded at 20 standing IDs. Its `full-run` / `test-plan-2` tag needs a complete 20-case attempt, and subset runs do not qualify it. Adding seven cases would change that denominator and the qualification history. The sibling plan shares the same TestOps GitHub job, selection, admission, projection, upload and verification code. A launch selects cases from exactly one plan (`plan_mixed` fails), and each plan has its own full-plan size (20 or 7). - `tests/runner-e2e/reliability-testops.mjs` (new): the mapping module. It has seven cases, full names `paperclip.Product E2E.everyday-workflows.<case>`, execution IDs, soak signatures and an empty `reliabilityRegistration`. A partial, duplicate, plan-2 or malformed registration is rejected. An empty registration makes selection fail closed before any spend (`product_regression_testops_reliability_unregistered` for a bootstrap, `case_mismatch` for a TestOps launch). No TestOps IDs are invented. - `product-regression-testops.mjs` / `-cli.mjs`: select a plan by its case identities or by the new bootstrap input. The projection binds each cell's own environment (`local` or `daytona`) instead of a fixed `local`. It adds a `soak-<signature>` tag and reports `plan` and a per-plan `fullPlan`. Plan-2 history IDs, parameters and results do not change. - Catalog: `runner-opencode-deepseek-1017` is now one shared fixture with `supportedEnvironments: ["local", "daytona"]`. The OpenCode server driver already runs on Daytona for `runner-opencode` (task-workspaces, core-compatibility, rolling deployment); only the model differs. The chat-feature suite stays local, and its definition hash does not change. `everyday-workflows` gains exactly six DeepSeek plan cells (definition version 12, 52 + 13 + 3 cells). - Cell matrix (plan), DeepSeek 1017 on native OpenCode: `interrupt-redirect`, `reassign-midrun` and `tree-pause-release` run on `local`. `three-turn-revise` runs on Daytona with the per-turn lease (S49). `long-silent-command` (S52/S52b) and `queued-across-restart` (S55/U9) run on the Daytona warm lease. - `queue-steer` is the one exception. It must steer an active turn, and the OpenCode server driver declares `steering: false`, so the route answers `steering_unsupported` and the case cannot pass. The plan uses the existing steering-capable `everyday-workflows.runner-codex.daytona.queue-steer` cell for it (Greptile finding). No DeepSeek `queue-steer` cell exists. The other native Codex Daytona cells and the ACPX facades stay selectable through `--group reliability`, but they are outside the plan. - `runner-full-stack-e2e.yml`: adds the `testops_plan` bootstrap input (`product-regression` by default), sets the plan-specific bootstrap launch name and tags, and renames two jobs to plan-neutral names. The OpenCode steps now match `startsWith(matrix.profileId, 'runner-opencode')`, so the DeepSeek profile gets the pinned OpenCode binary, runner TypeScript, the local Node check and the remote provider pack. Before this change, the DeepSeek profile could not run in this workflow. `OPENROUTER_API_KEY`, `OPENAI_API_KEY` (only for `queue-steer`) and `DAYTONA_API_KEY` come from the same `matrix.credentialName` and `matrix.environmentId` gates that other cells use. - `runner-e2e-testops-registration.yml` + `reliability-testops-registration.mjs` (new): a trusted, master-only, allowlisted-actor, idempotent `workflow_dispatch`. `inspect` only reads. `register` reuses cases with the same full name, creates missing cases and the plan, and assigns the plan to plan 2's GitHub job with plan 2's job parameters. It writes a receipt with a ready-to-commit `reliabilityRegistration` snippet. Only this job gets `ALLURE_TOKEN`, and it never runs a provider cell. - **No schedule and not per PR.** The plan runs only from a TestOps launch or a manual dispatch. It is not in the daily or weekly cron (`scheduledSelection` excludes it). A weekly run would add a paid 50-minute Daytona cell every week, but these signatures move with release-risky changes, not with time. - Docs: `EVERYDAY-WORKFLOWS.md` (matrix, environment reasons, local coverage limits, registration steps), `README.md`, `doc/evals.md`. ### Registration still needed (requires TestOps credentials) 1. Dispatch **Runner E2E TestOps registration** from master with `operation=inspect`. 2. Dispatch it with `operation=register`. 3. Commit the `reliabilityRegistration` snippet from the run summary into `tests/runner-e2e/reliability-testops.mjs` in one reviewed PR. 4. Launch the plan from TestOps. Tag only a complete seven-case attempt `full-run`. ### What local plan cells cannot reproduce The local cells do not reproduce the Daytona-only variants: the remote `workspace_busy` save after Stop (S47), remote stop settlement during reassignment (S48), and a lost or replaced remote runner during a tree hold (S50). S51 (lost runner) has no case yet. The explicit-only native Codex Daytona extras still cover S47, S48 and S50 on a sandbox. ## Verification - `node --test tests/runner-e2e/product-regression-testops.test.mjs tests/runner-e2e/reliability-testops.test.mjs`: 26 pass. These tests cover fail-closed selection, partial or duplicate registration, plan mixing, environment binding, per-plan denominators, preserved failure classes, the CLI select → project → verify flow for a seven-case launch, registration idempotency and job checks, and the workflow inputs and credential gates. - `vitest` for `reliability-testops-catalog`, `product-regression-testops-catalog`, `everyday-reliability` and `catalog.test`: 98 pass. These tests bind the mapping to real catalog cells (profile, model, environment, credentials, single attempt). They also check that the cells stay out of `--all`, the suite, profile and environment selectors and the daily/weekly cron. Real grader output must survive projection. - `pnpm test:e2e:runner:typecheck`: pass. - `pnpm test:e2e:runner -- --list --suite everyday-workflows --group reliability` lists 16 cells: 6 DeepSeek plan cells, 7 native Codex cells (one is the `queue-steer` plan cell) and 3 ACPX extras. `--matrix-json` for the DeepSeek plan IDs emits `needsDaytona=true` and three Daytona OpenCode cells with `credentialName=OPENROUTER_API_KEY`. - Full `pnpm test:e2e:runner:unit`: the remaining failures happen the same way on an unchanged master checkout. They are in `workflow-security` (3), `controller-outage`, `operating-receipts`, `pi-*`, `private-framework-report` and `remote-native-bootstrap`. #19529 tracks that drift. This PR adds no failures. - Not run: paid cells, a TestOps registration and a TestOps launch. They need provider and TestOps credentials, which exist only in protected CI environments. Catalog and projection tests are not live qualification. ## Risks - The registration script calls TestOps APIs (`POST /api/testcase`, `/api/testplan`, `/assign`, `/jobparameter`) through the same endpoints as the chat catalog reconcile. Tests use fakes, not the live API. `inspect` comes first, and `register` is idempotent. - DeepSeek 1017 on Daytona has not run before. The first launch is the acceptance step for those three cells. - If a later case needs another capability that OpenCode lacks, it must use a capable profile in the same way as `queue-steer`. The catalog test pins the single exception. - The OpenCode workflow conditions now match any `runner-opencode*` profile. Today this covers `runner-opencode` and `runner-opencode-deepseek-1017` only. - If someone commits a bad `reliabilityRegistration`, module load fails for both plans. A unit test rejects any committed registration that is not empty or complete. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed
1 parent
0e111c5075
commit
744056ecc8
21 files changed
+1110
-81
No files matched your search
@@ -0,0 +1,103 @@
|
||||
name: Runner E2E TestOps registration
|
||||
run-name: Runner E2E TestOps registration / ${{ inputs.plan }} / ${{ inputs.operation }}
|
||||
|
||||
# One-shot, idempotent registration of a sibling runner E2E TestOps plan. It
|
||||
# never runs a provider cell. `inspect` only reads; `register` creates missing
|
||||
# cases and the plan and assigns the plan to the standing plan's GitHub job.
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
plan:
|
||||
description: "Sibling TestOps plan to register"
|
||||
type: choice
|
||||
options: [reliability]
|
||||
default: reliability
|
||||
operation:
|
||||
description: "inspect reads TestOps only; register creates any missing case, the plan and its job assignment"
|
||||
type: choice
|
||||
options: [inspect, register]
|
||||
default: inspect
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: runner-e2e-testops-registration
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
authorize:
|
||||
name: Authorize registration
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
contents: read
|
||||
steps:
|
||||
- name: Require default branch and allowlisted numeric actor IDs
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
REF: ${{ github.ref }}
|
||||
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
|
||||
ACTOR: ${{ github.actor }}
|
||||
ACTOR_ID: ${{ github.actor_id }}
|
||||
TRIGGERING_ACTOR: ${{ github.triggering_actor }}
|
||||
ALLOWED_ACTOR_IDS: ${{ vars.RUNNER_E2E_ALLOWED_ACTOR_IDS }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ "$REF" != "refs/heads/$DEFAULT_BRANCH" ]; then
|
||||
echo "TestOps registration may run only from the default branch." >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! jq -e 'type == "array" and length > 0 and all(.[]; type == "number" and . > 0 and floor == .)' <<< "${ALLOWED_ACTOR_IDS:-}" >/dev/null; then
|
||||
echo "RUNNER_E2E_ALLOWED_ACTOR_IDS must be a non-empty JSON array of numeric GitHub user IDs." >&2
|
||||
exit 1
|
||||
fi
|
||||
for attempt in 1 2 3; do
|
||||
if triggering_actor_id="$(timeout 30s gh api "users/$TRIGGERING_ACTOR" --jq .id)"; then
|
||||
break
|
||||
fi
|
||||
if [ "$attempt" = 3 ]; then exit 1; fi
|
||||
sleep "$((attempt * 2))"
|
||||
done
|
||||
if [ "$triggering_actor_id" != "$ACTOR_ID" ] && [ "$TRIGGERING_ACTOR" = "$ACTOR" ]; then
|
||||
echo "GitHub actor identity contexts disagree; refusing registration." >&2
|
||||
exit 1
|
||||
fi
|
||||
for candidate in "$triggering_actor_id" "$ACTOR_ID"; do
|
||||
if ! jq -e --argjson candidate "$candidate" 'index($candidate) != null' <<< "$ALLOWED_ACTOR_IDS" >/dev/null; then
|
||||
echo "The initiating GitHub account is not authorized to register TestOps plans." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
register:
|
||||
name: Register the Reliability plan in TestOps
|
||||
needs: authorize
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
environment: runner-e2e-testops-reporting
|
||||
permissions:
|
||||
contents: read
|
||||
steps:
|
||||
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
||||
with:
|
||||
ref: ${{ github.sha }}
|
||||
persist-credentials: false
|
||||
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
|
||||
with:
|
||||
node-version: 24
|
||||
- name: Inspect or register the plan, its cases and its job assignment
|
||||
env:
|
||||
ALLURE_TOKEN: ${{ secrets.ALLURE_TOKEN }}
|
||||
ALLURE_ENDPOINT: https://paperclip.testops.cloud
|
||||
ALLURE_PROJECT_ID: '1'
|
||||
REGISTRATION_PLAN: ${{ inputs.plan }}
|
||||
REGISTRATION_OPERATION: ${{ inputs.operation }}
|
||||
run: node tests/runner-e2e/reliability-testops-registration.mjs testops-registration/receipt.json
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
||||
if: always()
|
||||
with:
|
||||
name: reliability-testops-registration-${{ github.run_id }}-${{ github.run_attempt }}
|
||||
path: testops-registration/receipt.json
|
||||
if-no-files-found: ignore
|
||||
retention-days: 90
|
||||
@@ -49,9 +49,14 @@ on:
|
||||
type: string
|
||||
required: false
|
||||
testops_case:
|
||||
description: "Product regression TestOps bootstrap: all, case name or case ID; leave empty for normal campaigns"
|
||||
description: "TestOps plan bootstrap: all, case name or case ID; leave empty for normal campaigns"
|
||||
type: string
|
||||
required: false
|
||||
testops_plan:
|
||||
description: "TestOps plan for a testops_case bootstrap; a TestOps launch names its own plan"
|
||||
type: choice
|
||||
options: [product-regression, reliability]
|
||||
default: product-regression
|
||||
ALLURE_JOB_RUN_ID:
|
||||
description: "Allure TestOps service input; its requested selection takes precedence"
|
||||
type: string
|
||||
@@ -307,7 +312,7 @@ jobs:
|
||||
if-no-files-found: error
|
||||
|
||||
testops_selection:
|
||||
name: Bind Product regression TestOps selection
|
||||
name: Bind TestOps plan selection
|
||||
needs: authorize
|
||||
if: github.event_name == 'workflow_dispatch' && (inputs.ALLURE_JOB_RUN_ID != '' || inputs.testops_case != '')
|
||||
runs-on: ubuntu-latest
|
||||
@@ -348,10 +353,12 @@ jobs:
|
||||
SELECT_ENVIRONMENT: ${{ inputs.environment }}
|
||||
SELECT_CASE: ${{ inputs.case }}
|
||||
TESTOPS_CASE: ${{ inputs.testops_case }}
|
||||
TESTOPS_PLAN: ${{ inputs.testops_plan || 'product-regression' }}
|
||||
ALLURE_JOB_RUN_ID: ${{ inputs.ALLURE_JOB_RUN_ID }}
|
||||
ALLURE_USERNAME: ${{ inputs.ALLURE_USERNAME }}
|
||||
ALLURE_LAUNCH_NAME: Product regression · ${{ github.run_number }}
|
||||
ALLURE_LAUNCH_TAGS: product-regression,github-actions
|
||||
# Names only a bootstrap launch; a TestOps launch already has its plan's name.
|
||||
ALLURE_LAUNCH_NAME: ${{ inputs.testops_plan == 'reliability' && 'Reliability' || 'Product regression' }} · ${{ github.run_number }}
|
||||
ALLURE_LAUNCH_TAGS: ${{ inputs.testops_plan == 'reliability' && 'reliability' || 'product-regression' }},github-actions
|
||||
run: node tests/runner-e2e/product-regression-testops-cli.mjs select product-testops/selection.json
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
||||
with:
|
||||
@@ -526,9 +533,9 @@ jobs:
|
||||
{
|
||||
echo "matrix=$(jq -c '{include: .include}' <<< "$catalog_json")"
|
||||
echo "needs_daytona=$(jq -r '.needsDaytona' <<< "$catalog_json")"
|
||||
echo "needs_runner_typescript=$(jq -r '[.include[] | select((.profileId == "runner-opencode") or (.profileId | startswith("runner-acpx-")) or (.suiteId == "openrouter-model-breadth"))] | length > 0' <<< "$catalog_json")"
|
||||
echo "needs_runner_typescript=$(jq -r '[.include[] | select((.profileId | startswith("runner-opencode")) or (.profileId | startswith("runner-acpx-")) or (.suiteId == "openrouter-model-breadth"))] | length > 0' <<< "$catalog_json")"
|
||||
echo "needs_native_binaries=$(jq -r '[.include[] | select((.profileId | startswith("runner-")) or (.suiteId == "openrouter-model-breadth"))] | length > 0' <<< "$catalog_json")"
|
||||
echo "needs_remote_provider_pack=$(jq -r '[.include[] | select((.environmentId == "daytona") and ((.profileId == "runner-opencode") or (.profileId | startswith("runner-acpx-"))))] | length > 0' <<< "$catalog_json")"
|
||||
echo "needs_remote_provider_pack=$(jq -r '[.include[] | select((.environmentId == "daytona") and ((.profileId | startswith("runner-opencode")) or (.profileId | startswith("runner-acpx-"))))] | length > 0' <<< "$catalog_json")"
|
||||
echo "execution_ids=$(jq -c '.executionIds' <<< "$catalog_json")"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
if ! [[ "$MAX_PARALLEL_LIMIT" =~ ^[1-9][0-9]*$ ]] || [ "$MAX_PARALLEL_LIMIT" -gt 100 ]; then
|
||||
@@ -1173,7 +1180,7 @@ jobs:
|
||||
test -e "$daytona_root/node_modules/@daytonaio/sdk"
|
||||
|
||||
- name: Materialize verified pinned OpenCode executable
|
||||
if: matrix.environmentId == 'local' && (matrix.profileId == 'legacy-opencode' || matrix.profileId == 'runner-opencode' || matrix.suiteId == 'openrouter-model-breadth')
|
||||
if: matrix.environmentId == 'local' && (matrix.profileId == 'legacy-opencode' || startsWith(matrix.profileId, 'runner-opencode') || matrix.suiteId == 'openrouter-model-breadth')
|
||||
run: node packages/paperclip-runner/scripts/materialize-opencode-binary.mjs
|
||||
|
||||
- name: Install checksum-verified Grok executable
|
||||
@@ -1188,7 +1195,7 @@ jobs:
|
||||
path: runner-e2e-build
|
||||
|
||||
- name: Download immutable remote provider pack
|
||||
if: matrix.environmentId == 'daytona' && (matrix.profileId == 'runner-opencode' || startsWith(matrix.profileId, 'runner-acpx-'))
|
||||
if: matrix.environmentId == 'daytona' && (startsWith(matrix.profileId, 'runner-opencode') || startsWith(matrix.profileId, 'runner-acpx-'))
|
||||
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
|
||||
with:
|
||||
name: ${{ needs.build_remote_provider_pack.outputs.provider_pack_artifact_name }}
|
||||
@@ -1197,7 +1204,7 @@ jobs:
|
||||
- name: Verify and restore campaign outputs
|
||||
if: startsWith(matrix.profileId, 'runner-') || matrix.suiteId == 'openrouter-model-breadth'
|
||||
env:
|
||||
NEEDS_RUNNER_TYPESCRIPT: ${{ matrix.profileId == 'runner-opencode' || startsWith(matrix.profileId, 'runner-acpx-') || matrix.suiteId == 'openrouter-model-breadth' }}
|
||||
NEEDS_RUNNER_TYPESCRIPT: ${{ startsWith(matrix.profileId, 'runner-opencode') || startsWith(matrix.profileId, 'runner-acpx-') || matrix.suiteId == 'openrouter-model-breadth' }}
|
||||
NEEDS_NATIVE_BINARY: ${{ startsWith(matrix.profileId, 'runner-') || matrix.suiteId == 'openrouter-model-breadth' }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
@@ -1217,7 +1224,7 @@ jobs:
|
||||
fi
|
||||
|
||||
- name: Verify and restore remote provider pack
|
||||
if: matrix.environmentId == 'daytona' && (matrix.profileId == 'runner-opencode' || startsWith(matrix.profileId, 'runner-acpx-'))
|
||||
if: matrix.environmentId == 'daytona' && (startsWith(matrix.profileId, 'runner-opencode') || startsWith(matrix.profileId, 'runner-acpx-'))
|
||||
env:
|
||||
IMAGE_SOURCE_REVISION: ${{ needs.daytona_image.outputs.source_revision }}
|
||||
run: |
|
||||
@@ -1237,7 +1244,7 @@ jobs:
|
||||
packages/paperclip-runner/provider-pack/provider-pack.json >/dev/null
|
||||
|
||||
- name: Qualify local provider Node interpreter
|
||||
if: matrix.environmentId == 'local' && (matrix.profileId == 'runner-opencode' || startsWith(matrix.profileId, 'runner-acpx-') || matrix.suiteId == 'openrouter-model-breadth')
|
||||
if: matrix.environmentId == 'local' && (startsWith(matrix.profileId, 'runner-opencode') || startsWith(matrix.profileId, 'runner-acpx-') || matrix.suiteId == 'openrouter-model-breadth')
|
||||
run: |
|
||||
node <<'NODE'
|
||||
const fs = require("node:fs");
|
||||
@@ -2047,7 +2054,7 @@ jobs:
|
||||
fi
|
||||
|
||||
publish_testops:
|
||||
name: Publish bound Product regression results to TestOps
|
||||
name: Publish bound TestOps plan results
|
||||
needs: [authorize, testops_selection, catalog, report]
|
||||
if: always() && needs.testops_selection.outputs.job_run_started == 'true' && inputs.image_only != true
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
Reference in new issue
Block a user