mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
cd8c9cd07c192832fa180a845577d36fb0bc41c8
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
744056ecc8 |
feat(evals): run the staging-soak reliability cases from a TestOps plan (#19549)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The runner full-stack E2E lane runs real Paperclip work against real agents, and TestOps launches its standing plans > - PR #19455 added seven staging-soak reliability cases, but no TestOps plan or workflow lane selects them, so they run only by hand with personal keys > - These cases catch restart, interrupt, queue and lost-runner regressions, so they must run from TestOps before a release with the standard reports > - This pull request adds a sibling "Reliability — staging soak" TestOps plan on the existing plan-2 lane, with native OpenCode and DeepSeek 1017 through OpenRouter as the plan profile for six of seven cases > - The benefit is a cheap, repeatable pre-release reliability launch with the same Allure/TestOps results, retained report and dashboard as plan 2 ## Linked Issues or Issue Description Refs #19455 **What existing behavior does this improve?** The staging-soak reliability cases from #19455 run only by a manual local command. **Subsystem affected** Runner full-stack E2E (`tests/runner-e2e/`) and its TestOps lane in `.github/workflows/runner-full-stack-e2e.yml`. **Current behavior** No TestOps plan includes the cases. The workflow cannot select them from TestOps. Every cell uses native Codex (`gpt-5.6-sol`) or an ACPX facade on Daytona, which is expensive. **Proposed behavior** A TestOps launch of the Reliability plan runs seven cells (six DeepSeek 1017, one native Codex for steering) through the standing plan-2 path and reports them like plan 2. **Reason and benefit** The cases become a first-class, low-cost, pre-release check with retained evidence. Plan 2 keeps its 20-case qualification semantics. **Breaking changes** None. Plan 2 selection, reporting and history identities stay the same. ## What Changed - **Plan choice: a sibling plan, not an extension of plan 2.** Plan 2 is bounded at 20 standing IDs. Its `full-run` / `test-plan-2` tag needs a complete 20-case attempt, and subset runs do not qualify it. Adding seven cases would change that denominator and the qualification history. The sibling plan shares the same TestOps GitHub job, selection, admission, projection, upload and verification code. A launch selects cases from exactly one plan (`plan_mixed` fails), and each plan has its own full-plan size (20 or 7). - `tests/runner-e2e/reliability-testops.mjs` (new): the mapping module. It has seven cases, full names `paperclip.Product E2E.everyday-workflows.<case>`, execution IDs, soak signatures and an empty `reliabilityRegistration`. A partial, duplicate, plan-2 or malformed registration is rejected. An empty registration makes selection fail closed before any spend (`product_regression_testops_reliability_unregistered` for a bootstrap, `case_mismatch` for a TestOps launch). No TestOps IDs are invented. - `product-regression-testops.mjs` / `-cli.mjs`: select a plan by its case identities or by the new bootstrap input. The projection binds each cell's own environment (`local` or `daytona`) instead of a fixed `local`. It adds a `soak-<signature>` tag and reports `plan` and a per-plan `fullPlan`. Plan-2 history IDs, parameters and results do not change. - Catalog: `runner-opencode-deepseek-1017` is now one shared fixture with `supportedEnvironments: ["local", "daytona"]`. The OpenCode server driver already runs on Daytona for `runner-opencode` (task-workspaces, core-compatibility, rolling deployment); only the model differs. The chat-feature suite stays local, and its definition hash does not change. `everyday-workflows` gains exactly six DeepSeek plan cells (definition version 12, 52 + 13 + 3 cells). - Cell matrix (plan), DeepSeek 1017 on native OpenCode: `interrupt-redirect`, `reassign-midrun` and `tree-pause-release` run on `local`. `three-turn-revise` runs on Daytona with the per-turn lease (S49). `long-silent-command` (S52/S52b) and `queued-across-restart` (S55/U9) run on the Daytona warm lease. - `queue-steer` is the one exception. It must steer an active turn, and the OpenCode server driver declares `steering: false`, so the route answers `steering_unsupported` and the case cannot pass. The plan uses the existing steering-capable `everyday-workflows.runner-codex.daytona.queue-steer` cell for it (Greptile finding). No DeepSeek `queue-steer` cell exists. The other native Codex Daytona cells and the ACPX facades stay selectable through `--group reliability`, but they are outside the plan. - `runner-full-stack-e2e.yml`: adds the `testops_plan` bootstrap input (`product-regression` by default), sets the plan-specific bootstrap launch name and tags, and renames two jobs to plan-neutral names. The OpenCode steps now match `startsWith(matrix.profileId, 'runner-opencode')`, so the DeepSeek profile gets the pinned OpenCode binary, runner TypeScript, the local Node check and the remote provider pack. Before this change, the DeepSeek profile could not run in this workflow. `OPENROUTER_API_KEY`, `OPENAI_API_KEY` (only for `queue-steer`) and `DAYTONA_API_KEY` come from the same `matrix.credentialName` and `matrix.environmentId` gates that other cells use. - `runner-e2e-testops-registration.yml` + `reliability-testops-registration.mjs` (new): a trusted, master-only, allowlisted-actor, idempotent `workflow_dispatch`. `inspect` only reads. `register` reuses cases with the same full name, creates missing cases and the plan, and assigns the plan to plan 2's GitHub job with plan 2's job parameters. It writes a receipt with a ready-to-commit `reliabilityRegistration` snippet. Only this job gets `ALLURE_TOKEN`, and it never runs a provider cell. - **No schedule and not per PR.** The plan runs only from a TestOps launch or a manual dispatch. It is not in the daily or weekly cron (`scheduledSelection` excludes it). A weekly run would add a paid 50-minute Daytona cell every week, but these signatures move with release-risky changes, not with time. - Docs: `EVERYDAY-WORKFLOWS.md` (matrix, environment reasons, local coverage limits, registration steps), `README.md`, `doc/evals.md`. ### Registration still needed (requires TestOps credentials) 1. Dispatch **Runner E2E TestOps registration** from master with `operation=inspect`. 2. Dispatch it with `operation=register`. 3. Commit the `reliabilityRegistration` snippet from the run summary into `tests/runner-e2e/reliability-testops.mjs` in one reviewed PR. 4. Launch the plan from TestOps. Tag only a complete seven-case attempt `full-run`. ### What local plan cells cannot reproduce The local cells do not reproduce the Daytona-only variants: the remote `workspace_busy` save after Stop (S47), remote stop settlement during reassignment (S48), and a lost or replaced remote runner during a tree hold (S50). S51 (lost runner) has no case yet. The explicit-only native Codex Daytona extras still cover S47, S48 and S50 on a sandbox. ## Verification - `node --test tests/runner-e2e/product-regression-testops.test.mjs tests/runner-e2e/reliability-testops.test.mjs`: 26 pass. These tests cover fail-closed selection, partial or duplicate registration, plan mixing, environment binding, per-plan denominators, preserved failure classes, the CLI select → project → verify flow for a seven-case launch, registration idempotency and job checks, and the workflow inputs and credential gates. - `vitest` for `reliability-testops-catalog`, `product-regression-testops-catalog`, `everyday-reliability` and `catalog.test`: 98 pass. These tests bind the mapping to real catalog cells (profile, model, environment, credentials, single attempt). They also check that the cells stay out of `--all`, the suite, profile and environment selectors and the daily/weekly cron. Real grader output must survive projection. - `pnpm test:e2e:runner:typecheck`: pass. - `pnpm test:e2e:runner -- --list --suite everyday-workflows --group reliability` lists 16 cells: 6 DeepSeek plan cells, 7 native Codex cells (one is the `queue-steer` plan cell) and 3 ACPX extras. `--matrix-json` for the DeepSeek plan IDs emits `needsDaytona=true` and three Daytona OpenCode cells with `credentialName=OPENROUTER_API_KEY`. - Full `pnpm test:e2e:runner:unit`: the remaining failures happen the same way on an unchanged master checkout. They are in `workflow-security` (3), `controller-outage`, `operating-receipts`, `pi-*`, `private-framework-report` and `remote-native-bootstrap`. #19529 tracks that drift. This PR adds no failures. - Not run: paid cells, a TestOps registration and a TestOps launch. They need provider and TestOps credentials, which exist only in protected CI environments. Catalog and projection tests are not live qualification. ## Risks - The registration script calls TestOps APIs (`POST /api/testcase`, `/api/testplan`, `/assign`, `/jobparameter`) through the same endpoints as the chat catalog reconcile. Tests use fakes, not the live API. `inspect` comes first, and `register` is idempotent. - DeepSeek 1017 on Daytona has not run before. The first launch is the acceptance step for those three cells. - If a later case needs another capability that OpenCode lacks, it must use a capable profile in the same way as `queue-steer`. The catalog test pins the single exception. - The OpenCode workflow conditions now match any `runner-opencode*` profile. Today this covers `runner-opencode` and `runner-opencode-deepseek-1017` only. - If someone commits a bad `reliabilityRegistration`, module load fails for both plans. A unit test rejects any committed registration that is not empty or complete. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
c99bcd89fe |
fix: preserve Product TestOps evidence and CI readiness (#19000)
## Thinking Path > - Paperclip manages agent work through governed product workflows. > - Product E2E tests check those workflows through the app and Runner. > - A trusted reporter sends retained results to TestOps. > - Free-form metadata and a short upload observer can lose safe, valid evidence. > - UUID navigation can also discard task metadata before the canonical query returns. > - Hermetic CI files start child processes that need CPU capacity within their readiness limits. > - This change closes the reporting contract, retains authenticated core aliases, and reserves CI CPU capacity. > - Original case identities, conditions, assertions, verdicts and deadlines remain intact. ## Linked Issues or Issue Description Refs #18983. This addresses its late [metadata disclosure finding](https://github.com/paperclipai/paperclip/pull/18983#discussion_r4470789114) and [upload readiness finding](https://github.com/paperclipai/paperclip/pull/18983#discussion_r4470790366). **What happened?** The reporter accepted model IDs and matcher names through syntax filters. A secret-shaped string could pass those filters. First-task results also use their top-level model field for an observed alias. Treating it as configured metadata can reject valid retained evidence. The upload observer stopped after 22 seconds of waits. [Required CI on the earlier head](https://github.com/paperclipai/paperclip/actions/runs/41886885193) also found two real browser title failures and five native package readiness failures. A deterministic control reproduced metadata loss during UUID-to-identifier navigation. A separate fixed-resource control reproduced startup deadline failures with four concurrent hermetic fixtures, while one and two fixtures passed. **Expected behavior** Reject unknown transmitted metadata before creating upload files. Preserve every original matcher verdict and attempt receipt. Observe server processing without rerunning the product. Keep authenticated task metadata visible through canonical navigation. Reserve capacity for hermetic child processes without changing the original readiness deadlines. **Steps to reproduce** Use a secret-shaped model ID or cleanup state in retained evidence, return an observed first-task model alias, or let upload processing settle after 40 seconds of virtual time. Hold the canonical core query pending after an authenticated UUID read. For the CPU control, run the same providerless startup/recovery fixture alone, four times concurrently, and twice concurrently under a fixed 4-vCPU quota and its original 8-second readiness limit. ## What Changed - Bind configured model receipts to each exact declared case model. Four first-task cases use captured agent configuration; their observed aliases and fallback text remain digest-only. A missing capture stays null / Not recorded. The other 16 cells use their configured profile model. Reject wrong or unknown configuration before any upload file exists. - Keep the existing model distribution: 18 Codex cells on `gpt-5.6-sol`, configured-repository case 9 on its existing `gpt-6.1-sol` override, and task-workspaces case 54 on `openrouter/deepseek/deepseek-v4-flash-0731`. The repository override exists at the [original canary source](https://github.com/paperclipai/paperclip/blob/86814c6550f52e99ede150a883bdc773cbb540e3/tests/runner-e2e/catalog.ts#L1148). No model configuration changes. - Allow only existing Matcher kinds, closed failure classes, and cleanup states `passed`, `failed`, or `not_started`. Publish each matcher's kind, ordinal, original boolean verdict and SHA256 path receipt. Full conditions remain in the linked original workflow artifact; raw dynamic path bytes do not enter TestOps. - Observe the same upload for at most 55 seconds of poll waits. Product scenarios, provider retries and scenario deadlines do not change. - Retain task core data under only the authenticated response's own UUID and identifier core keys. Preserve existing canonical data. Keep arbitrary references and full-detail keys empty until their own reads. - Retain a first-failure trace for the existing Cloud recovery test. Keep its assertions, timeout, retries, mock data and artifact policy unchanged. This adds diagnosis evidence; it does not claim to fix that separate Cloud title timeout. - Reserve half the available CPUs for Runner package CI workers, with the existing upper cap of four. A 4-vCPU host uses two workers; an 8-vCPU host still uses four. Local defaults and test selection do not change. ## Verification - Reporting controls first reproduced the two original gaps. All 14 Node contract tests now pass. Unknown configuration creates neither a result directory nor an export summary. A model valid for a different selected case is also rejected. Original functional failures and earlier accounting receipts survive. - The catalog compatibility test passes for all 20 selected cells, exact model tuples, real pure grader outputs and actual `projectFirstTaskResult` shapes. It covers observed aliases, fallback text, absent evidence and wrong or secret-shaped configured metadata. These are contract controls, not live journey passes. - Virtual-clock controls accept a correct result that settles at 40 seconds and reject a persistently wrong verdict after 55 seconds. They make no product or provider calls. - Two new core-cache controls failed before repair. The query/cache suites now pass all 8 tests. They hold canonical refresh pending and verify title/company retention, newer canonical data, full-detail enrichment and unrelated keys. UI typecheck and token gates pass. - The unchanged exact UUID cross-company link/upload browser test passed once after repair: 5.8-second scenario, 32.0-second complete command. The unchanged exact Cloud task-entry diagnostic passed once: 5.4-second scenario, 23.3-second complete command. Its original canonical-only timeout remains unexplained. All five Cloud tests still collect after the trace setting. - The providerless Linux control used the retained Runner and compiled controller from source `3a2249dd`, with a fixed 4-vCPU quota and original 8-second readiness limit. Solo passed 1/1; four concurrent fixtures failed 4/4; two concurrent fixtures passed 2/2. At each failure, Rust had accepted command 2 but startup was still pending; checkpoint and published status agreed, one connection was authenticated, no replay backlog existed, and the sidecar had not launched. Passing controls preserved the expected startup failure through controller replacement. No network or credentials were available. - That control uses Linux amd64 emulation and Node 24.21; hosted CI used Node 24.22.1. It supports the worker-cap change, but does not prove that the original `m5n.xlarge` host caused every CI failure. The original red attempts remain recorded. - Actual config-load controls verify 1 CPU → 1 worker, 2 → 1, 4 → 2, 8 → 4 and 16 → 4, with no local, timeout or retry override. The isolated policy change preserves the same 297 test-file inventory; integration of current master `89703c3ff7` retains all 298 current native package files. The original PR package/native command plans remain unchanged. - Final head `877e28da32f2d2f0d12906c89aff085e2c6ea943` passes both required checks, `ci / verify` and `ci / e2e`, in [CI run 41897677652, attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41897677652/attempts/2). The native package passes 4,950 tests with 10 existing skips. All eight browser shards pass; the Cloud shard passes all 41 tests, including all four unchanged recovery cases. - Attempt 1 preserved 46 successful jobs but its one-file heartbeat recovery shard hit the original 20-minute job limit without a named assertion or completion verdict. Its cause remains unidentified. One explicitly bounded debug replay retained those 46 successes and ran only that leaf and its dependent rollup. The unchanged file passed all 412 tests (346.91-second file, 371.34-second Vitest command, 7m11s job). Source, cases and deadlines did not change. This establishes intermittency, not a causal repair. The original timeout and debug pass both remain recorded. - Fresh [Greptile 5/5](https://github.com/paperclipai/paperclip/pull/19000#issuecomment-6637604918) and [Ponytail 5/5](https://github.com/paperclipai/paperclip/pull/19000#issuecomment-6637489454) reviews name exact head `877e28`. The single review thread is resolved. Live TestOps bootstrap and complete 20-case acceptance remain held until reviewed delivery. CI and focused controls do not count as product plan passes. ## Risks - Future model or Matcher kind changes require a reviewed reporting-contract update. - TestOps shows a closed Matcher kind and ordinal. The path digest binds the original condition in retained evidence. - Core alias data retains the existing authenticated query-client and server company boundaries. It never becomes full-detail evidence. - Fewer CI workers can increase one job's duration. The full suite must verify the final policy. The paid evaluation matrix and its parallelism do not change. - The original Cloud title timeout and the first-attempt heartbeat CI hang do not have proven causes. Current unchanged assertions pass, but this does not establish repeatability. The Cloud test now retains its first-failure trace. A successful upload or focused control alone cannot establish a complete product pass. ## Model Used OpenAI Codex with reasoning, tool use and code execution. The host does not expose the exact model ID or context window to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b99dd57c82 |
test(evals): run Product regression from TestOps (#18983)
## Thinking Path > - Paperclip manages agent work through governed product workflows. > - Product E2E tests check those workflows through the real app and runner. > - The saved Product regression plan has 20 imported case identities. > - The plan has no live executor, so historical baseline rows do not qualify its current behavior. > - The existing paid full-stack workflow already owns all 20 required journeys. > - This change binds TestOps selection and reporting to that workflow without changing the product oracle or paid admission. > - Maintainers can run the plan and inspect current case verdicts and retained evidence. ## Linked Issues or Issue Description **What happened?** The standing Product regression plan has no executor. Its dashboard points at a historical 355-result snapshot across several suites and revisions. **Expected behavior** A TestOps plan run must select the intended existing full-stack cells, execute them on GitHub, and report their original case IDs, assertions and source revision. Missing evidence must remain a visible failure. **Steps to reproduce** Open the Product regression plan and inspect Executors. There is no live job. The existing full-stack workflow accepts exact cell IDs but has no TestOps selection or reporting path. **Paperclip version or commit** The branch includes reviewed master correction |