Files
PaperClipAI/tests/runner-e2e/reliability-testops.d.mts
T
DottaandClaude Opus 5.5 744056ecc8 feat(evals): run the staging-soak reliability cases from a TestOps plan (#19549)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The runner full-stack E2E lane runs real Paperclip work against real
agents, and TestOps launches its standing plans
> - PR #19455 added seven staging-soak reliability cases, but no TestOps
plan or workflow lane selects them, so they run only by hand with
personal keys
> - These cases catch restart, interrupt, queue and lost-runner
regressions, so they must run from TestOps before a release with the
standard reports
> - This pull request adds a sibling "Reliability — staging soak"
TestOps plan on the existing plan-2 lane, with native OpenCode and
DeepSeek 1017 through OpenRouter as the plan profile for six of seven
cases
> - The benefit is a cheap, repeatable pre-release reliability launch
with the same Allure/TestOps results, retained report and dashboard as
plan 2

## Linked Issues or Issue Description

Refs #19455

**What existing behavior does this improve?**
The staging-soak reliability cases from #19455 run only by a manual
local command.

**Subsystem affected**
Runner full-stack E2E (`tests/runner-e2e/`) and its TestOps lane in
`.github/workflows/runner-full-stack-e2e.yml`.

**Current behavior**
No TestOps plan includes the cases. The workflow cannot select them from
TestOps. Every cell uses native Codex (`gpt-5.6-sol`) or an ACPX facade
on Daytona, which is expensive.

**Proposed behavior**
A TestOps launch of the Reliability plan runs seven cells (six DeepSeek
1017, one native Codex for steering) through the standing plan-2 path
and reports them like plan 2.

**Reason and benefit**
The cases become a first-class, low-cost, pre-release check with
retained evidence. Plan 2 keeps its 20-case qualification semantics.

**Breaking changes**
None. Plan 2 selection, reporting and history identities stay the same.

## What Changed

- **Plan choice: a sibling plan, not an extension of plan 2.** Plan 2 is
bounded at 20 standing IDs. Its `full-run` / `test-plan-2` tag needs a
complete 20-case attempt, and subset runs do not qualify it. Adding
seven cases would change that denominator and the qualification history.
The sibling plan shares the same TestOps GitHub job, selection,
admission, projection, upload and verification code. A launch selects
cases from exactly one plan (`plan_mixed` fails), and each plan has its
own full-plan size (20 or 7).
- `tests/runner-e2e/reliability-testops.mjs` (new): the mapping module.
It has seven cases, full names `paperclip.Product
E2E.everyday-workflows.<case>`, execution IDs, soak signatures and an
empty `reliabilityRegistration`. A partial, duplicate, plan-2 or
malformed registration is rejected. An empty registration makes
selection fail closed before any spend
(`product_regression_testops_reliability_unregistered` for a bootstrap,
`case_mismatch` for a TestOps launch). No TestOps IDs are invented.
- `product-regression-testops.mjs` / `-cli.mjs`: select a plan by its
case identities or by the new bootstrap input. The projection binds each
cell's own environment (`local` or `daytona`) instead of a fixed
`local`. It adds a `soak-<signature>` tag and reports `plan` and a
per-plan `fullPlan`. Plan-2 history IDs, parameters and results do not
change.
- Catalog: `runner-opencode-deepseek-1017` is now one shared fixture
with `supportedEnvironments: ["local", "daytona"]`. The OpenCode server
driver already runs on Daytona for `runner-opencode` (task-workspaces,
core-compatibility, rolling deployment); only the model differs. The
chat-feature suite stays local, and its definition hash does not change.
`everyday-workflows` gains exactly six DeepSeek plan cells (definition
version 12, 52 + 13 + 3 cells).
- Cell matrix (plan), DeepSeek 1017 on native OpenCode:
`interrupt-redirect`, `reassign-midrun` and `tree-pause-release` run on
`local`. `three-turn-revise` runs on Daytona with the per-turn lease
(S49). `long-silent-command` (S52/S52b) and `queued-across-restart`
(S55/U9) run on the Daytona warm lease.
- `queue-steer` is the one exception. It must steer an active turn, and
the OpenCode server driver declares `steering: false`, so the route
answers `steering_unsupported` and the case cannot pass. The plan uses
the existing steering-capable
`everyday-workflows.runner-codex.daytona.queue-steer` cell for it
(Greptile finding). No DeepSeek `queue-steer` cell exists. The other
native Codex Daytona cells and the ACPX facades stay selectable through
`--group reliability`, but they are outside the plan.
- `runner-full-stack-e2e.yml`: adds the `testops_plan` bootstrap input
(`product-regression` by default), sets the plan-specific bootstrap
launch name and tags, and renames two jobs to plan-neutral names. The
OpenCode steps now match `startsWith(matrix.profileId,
'runner-opencode')`, so the DeepSeek profile gets the pinned OpenCode
binary, runner TypeScript, the local Node check and the remote provider
pack. Before this change, the DeepSeek profile could not run in this
workflow. `OPENROUTER_API_KEY`, `OPENAI_API_KEY` (only for
`queue-steer`) and `DAYTONA_API_KEY` come from the same
`matrix.credentialName` and `matrix.environmentId` gates that other
cells use.
- `runner-e2e-testops-registration.yml` +
`reliability-testops-registration.mjs` (new): a trusted, master-only,
allowlisted-actor, idempotent `workflow_dispatch`. `inspect` only reads.
`register` reuses cases with the same full name, creates missing cases
and the plan, and assigns the plan to plan 2's GitHub job with plan 2's
job parameters. It writes a receipt with a ready-to-commit
`reliabilityRegistration` snippet. Only this job gets `ALLURE_TOKEN`,
and it never runs a provider cell.
- **No schedule and not per PR.** The plan runs only from a TestOps
launch or a manual dispatch. It is not in the daily or weekly cron
(`scheduledSelection` excludes it). A weekly run would add a paid
50-minute Daytona cell every week, but these signatures move with
release-risky changes, not with time.
- Docs: `EVERYDAY-WORKFLOWS.md` (matrix, environment reasons, local
coverage limits, registration steps), `README.md`, `doc/evals.md`.

### Registration still needed (requires TestOps credentials)

1. Dispatch **Runner E2E TestOps registration** from master with
`operation=inspect`.
2. Dispatch it with `operation=register`.
3. Commit the `reliabilityRegistration` snippet from the run summary
into `tests/runner-e2e/reliability-testops.mjs` in one reviewed PR.
4. Launch the plan from TestOps. Tag only a complete seven-case attempt
`full-run`.

### What local plan cells cannot reproduce

The local cells do not reproduce the Daytona-only variants: the remote
`workspace_busy` save after Stop (S47), remote stop settlement during
reassignment (S48), and a lost or replaced remote runner during a tree
hold (S50). S51 (lost runner) has no case yet. The explicit-only native
Codex Daytona extras still cover S47, S48 and S50 on a sandbox.

## Verification

- `node --test tests/runner-e2e/product-regression-testops.test.mjs
tests/runner-e2e/reliability-testops.test.mjs`: 26 pass. These tests
cover fail-closed selection, partial or duplicate registration, plan
mixing, environment binding, per-plan denominators, preserved failure
classes, the CLI select → project → verify flow for a seven-case launch,
registration idempotency and job checks, and the workflow inputs and
credential gates.
- `vitest` for `reliability-testops-catalog`,
`product-regression-testops-catalog`, `everyday-reliability` and
`catalog.test`: 98 pass. These tests bind the mapping to real catalog
cells (profile, model, environment, credentials, single attempt). They
also check that the cells stay out of `--all`, the suite, profile and
environment selectors and the daily/weekly cron. Real grader output must
survive projection.
- `pnpm test:e2e:runner:typecheck`: pass.
- `pnpm test:e2e:runner -- --list --suite everyday-workflows --group
reliability` lists 16 cells: 6 DeepSeek plan cells, 7 native Codex cells
(one is the `queue-steer` plan cell) and 3 ACPX extras. `--matrix-json`
for the DeepSeek plan IDs emits `needsDaytona=true` and three Daytona
OpenCode cells with `credentialName=OPENROUTER_API_KEY`.
- Full `pnpm test:e2e:runner:unit`: the remaining failures happen the
same way on an unchanged master checkout. They are in
`workflow-security` (3), `controller-outage`, `operating-receipts`,
`pi-*`, `private-framework-report` and `remote-native-bootstrap`. #19529
tracks that drift. This PR adds no failures.
- Not run: paid cells, a TestOps registration and a TestOps launch. They
need provider and TestOps credentials, which exist only in protected CI
environments. Catalog and projection tests are not live qualification.

## Risks

- The registration script calls TestOps APIs (`POST /api/testcase`,
`/api/testplan`, `/assign`, `/jobparameter`) through the same endpoints
as the chat catalog reconcile. Tests use fakes, not the live API.
`inspect` comes first, and `register` is idempotent.
- DeepSeek 1017 on Daytona has not run before. The first launch is the
acceptance step for those three cells.
- If a later case needs another capability that OpenCode lacks, it must
use a capable profile in the same way as `queue-steer`. The catalog test
pins the single exception.
- The OpenCode workflow conditions now match any `runner-opencode*`
profile. Today this covers `runner-opencode` and
`runner-opencode-deepseek-1017` only.
- If someone commits a bad `reliabilityRegistration`, module load fails
for both plans. A unit test rejects any committed registration that is
not empty or complete.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use
(shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 14:19:58 -06:00

30 lines
962 B
TypeScript

export interface ReliabilityDefinition {
plan: 'reliability';
suite: 'everyday-workflows';
name: string;
environment: 'local' | 'daytona';
signature: string;
profile: string;
model: string;
fullName: string;
executionId: string;
}
export interface ReliabilityRegistration {
planId: number | null;
caseIds: Readonly<Record<string, number>>;
}
export const reliabilityPlanKey: 'reliability';
export const reliabilityPlanName: string;
export const reliabilitySuite: 'everyday-workflows';
export const reliabilityProfile: string;
export const reliabilityModel: string;
export const reliabilityDefinitions: readonly ReliabilityDefinition[];
export const reliabilityRegistration: Readonly<ReliabilityRegistration>;
export function registeredReliabilityCases(
registration?: ReliabilityRegistration,
reservedCaseIds?: readonly number[],
reservedPlanIds?: readonly number[],
): ReadonlyArray<ReliabilityDefinition & { id: number }> | null;