mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
c66b4a55efdc5675b5a5146197078b94de6407da
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
02a2299173 |
feat(codex): default Codex to gpt-6.1-sol, half the gpt-5.6-sol price (#19623)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Codex agents run through Paperclip Runner or the legacy
`codex_local` harness, and the runner E2E suites run paid Codex cells
> - The shared Codex default and many selections preferred `gpt-5.6-sol`
> - The reviewed OpenAI rate card in
`server/src/services/codex-pricing.ts` (`openai-standard-2026-09-30`)
prices `gpt-6.1-sol` at $2 input / $0.10 cached / $10 output per 1M
tokens, and `gpt-5.6-sol` at $4 / $0.40 / $20
> - So every default run and every paid Codex eval cell cost twice as
much as necessary
> - This pull request moves every place that prefers or selects
`gpt-5.6-sol` to `gpt-6.1-sol`, and keeps historical facts, rate cards
and compatibility fallbacks on `gpt-5.6-sol`
> - The benefit is half the token price for new default Codex agents and
for the Codex E2E and TestOps cells
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The default Codex model for new agents and the Codex model that the
runner E2E suites and TestOps plans select.
**Subsystem affected**
Adapters (`adapter-utils`, `codex-local`), server native runtime
fallback, UI AI connection pools, Paperclip Runner devtools and live
evals, runner E2E catalog and TestOps mappings.
**Current behavior**
New Codex agents without an explicit model store `gpt-5.6-sol`. The
runner E2E `runner-codex` profile, the Reliability `queue-steer` plan
cell, the Product regression TestOps mapping and the ACPX Codex
qualification selection all use `gpt-5.6-sol`.
**Proposed behavior**
They use `gpt-6.1-sol`. Existing agents that store a model keep it.
`gpt-5.6-sol` stays selectable.
**Reason and benefit**
`gpt-6.1-sol` costs half as much per token on the in-repo reviewed rate
card. The current OpenAI pricing page (retrieved 2026-11-06) shows the
same short-context rates and also lists long-context rates of $4 / $0.20
/ $15 for `gpt-6.1-sol` against $8 / $0.80 / $30 for `gpt-5.6-sol`.
**Breaking changes**
None for stored configurations. See Risks.
## What Changed
Changed defaults and selections (all now `gpt-6.1-sol`):
- `PAPERCLIP_RUNNER_DEFAULT_MODELS.codex` in
`packages/adapter-utils/src/paperclip-runner-permissions.ts`. This also
changes `DEFAULT_CODEX_LOCAL_MODEL`,
`resolvePaperclipRunnerModel("codex", …)`, runner config normalization,
`codex_local` → Paperclip Runner conversion, the UI new-agent and
agent-config defaults, the composer default model and the public MCP
effort lookup.
- `codex-local` `models`: the 5.6 entry is now the literal
`gpt-5.6-sol`, because it used to be `DEFAULT_CODEX_LOCAL_MODEL`. The
list order and contents do not change.
- AI connection pool: a new OpenAI member profile uses
`PAPERCLIP_RUNNER_DEFAULT_MODELS.codex` instead of a hard-coded
`gpt-5.6-sol`.
- Paperclip Runner devtools ACPX Codex default and preset, the
`runner-codex-sol` local provider smoke, and the `acpx-codex-sol` live
workflow candidate.
- `tests/acpx-qualification-models.ts` Codex selection (runner E2E and
runner acceptance ACPX Codex profiles).
- Runner E2E: the `runner-codex` profile follows the default, so all
native Codex cells move. This includes the `runner-codex` cells among
the 52 standard `everyday-workflows` cells and the seven Codex
reliability cells. `runner-codex-mini` stays on `gpt-5.4-mini`. The
Reliability plan `queue-steer` `steeringProfile` is now
`['runner-codex', 'gpt-6.1-sol']`. Its executionId
(`everyday-workflows.runner-codex.daytona.queue-steer`), its fullName
and its TestOps case identity do not change. No separate profile is
necessary.
- `product-regression-testops.mjs`: the declared model for every
`runner-codex` case is now `gpt-6.1-sol` (it was already that for
`configured-repository`).
- `tests/runner-e2e/installed-cli.ts`: the credential-free installed
startup proof opens a session on the default model.
- `tests/runner-e2e/scheduled-prices.json`: add a `gpt-6.1-sol` price
(`openai-standard-2026-11-06`, including 30m/12h cache writes and
long-context rates from the OpenAI pricing page). The weekly scheduled
Codex cells need an exact price for their model, or admission fails
closed.
- Storybook: the composer model picker fixtures, the connection pool
stories and host, and the agent settings preview show `gpt-6.1-sol` as
the default.
- Docs: `doc/DEVELOPING.md` (the Codex conversion default) and
`tests/runner-e2e/EVERYDAY-WORKFLOWS.md` (local matrix, reliability
extras, `queue-steer` row).
- Tests that assert the default now expect `gpt-6.1-sol`. New
assertions: the default has the `0.159.0` CLI floor, the compatibility
fallback is floor-free, the `queue-steer` plan cell runs `runner-codex`
on `gpt-6.1-sol`, and all native Codex everyday cells (standard and
reliability) use `gpt-6.1-sol`.
Deliberately kept on `gpt-5.6-sol`:
- Rate cards: `server/src/services/codex-pricing.ts` and
`packages/paperclip-runner/src/evals/model-pricing.ts`. Old receipts
need them. (The runner eval rate card has no `gpt-6.1-sol` entry. An
unpriced candidate is reported as unpriced, not as zero. I did not add a
rate there because that card has its own version and source.)
- The remote Codex CLI compatibility fallback. A new
`CODEX_LOCAL_COMPATIBILITY_FALLBACK_MODEL = "gpt-5.6-sol"` replaces the
default in `codexModelFallbackCandidates`. `gpt-6.1-sol` needs Codex
0.159.0 or newer, so it cannot be the floor-free last resort. Fallback
behavior is unchanged.
- `tests/runner-e2e/provider-usage.ts`: the implicit request
cache-policy attribution stays qualified only for `gpt-5.6-sol`. I did
not verify that policy for `gpt-6.1-sol`. A `gpt-6.1-sol` response that
omits the write-TTL split keeps its cache-write cost unknown, which is
the existing behavior for unqualified models.
- `tests/runner-e2e/scheduled-measurements.json`: historical
`gpt-5.6-sol` measurements. They no longer match the current suite
definitions, so the forecast reports them as historical only. The
forecast tests now build their synthetic cohorts on the current
definition hash.
- Model catalogs that list `gpt-5.6-sol` as available (`codex-local`,
`cursor-local`, `opencode-local`, Speko voice diagnostics, Pi picker
fixture).
- The `gpt-5.6` → `gpt-5.6-sol` alias for old stored configs.
- Dated or evidence docs: `doc/plans/*`, `doc/adapter-model-audit-*`,
`QUALIFICATION-*`, `doc/connections/MEMORY.md`,
`doc/connections/TASK-REVIEWS-VERIFICATION.md`,
`tests/runner-e2e/CHAT-FEATURES.md`, the Slack eval evidence, and the
S49 soak signature text (the soak failure happened on `gpt-5.6-sol`).
- Test fixtures that parse, price, migrate or round-trip an existing
`gpt-5.6-sol` value (Rust runner tests, driver tests, pricing tests,
stored-config tests, stories that show an existing agent).
## Verification
- `npx vitest run` on every server, UI, adapter-utils and codex-local
test file that mentions `gpt-5.6-sol` or the default, plus all of
`packages/adapter-utils` and `packages/adapters/codex-local`: 5,483
passed. Three timeouts under load (`sandbox-managed-runtime`,
`acp.quota`) passed when rerun.
- `packages/paperclip-runner`: `vitest run` for `src/eval`, `src/evals`,
`src/drivers/acpx`, `src/drivers/codex`, `src/cli`, `src/contracts`,
`src/backends`: 2,641 passed.
- `node --test tests/runner-e2e/reliability-testops.test.mjs
tests/runner-e2e/product-regression-testops.test.mjs`: 28 passed.
- `node tests/runner-e2e/run-unit.mjs`: all model, catalog, reliability,
scheduled and TestOps tests pass. Failures remain only in
`copilot-protection-flow`, `denied-target-isolation-flow`,
`pi-controls-flow`, `workflow-target` and `unit-launcher`. These five
also fail on unmodified master in this local environment (filesystem
watch and git environment), and they do not touch models.
- `vitest run --config tests/runner-acceptance/vitest.config.ts`: 19
passed.
- `pnpm test:e2e:runner -- --list --suite everyday-workflows --group
reliability`: the seven `everyday-workflows.runner-codex.daytona.<case>`
cells show `gpt-6.1-sol` with `OPENAI_API_KEY,DAYTONA_API_KEY`.
`--matrix-json` gives 16 reliability cells with the same credentials as
before. `--list --all` shows no `gpt-5.6-sol` cell.
- Typecheck: `adapter-utils`, `adapter-codex-local`, `server`, `ui`,
`paperclip-runner` (TypeScript), and `pnpm test:e2e:runner:typecheck`.
- `pnpm check:token-gates`: clean.
- Not run: paid cells, Rust tests (no Rust change), the browser suites,
and the full `pnpm -r typecheck` / `pnpm build` (CI runs them).
## Risks
- Agents without an explicit model change model on their next save or
normalization, and Paperclip Runner rows with a blank model use
`gpt-6.1-sol` at execution. Stored models, including `gpt-5.6-sol`, are
never rewritten. No migration is necessary.
- `gpt-6.1-sol` needs Codex CLI 0.159.0 or newer with ChatGPT sign-in.
The runner pins `@openai/codex@0.160.0`. On a remote target with an
older image CLI, the existing compatibility check falls back to
`gpt-6-sol` and then `gpt-5.6-sol`, and that probe now runs for
default-model agents too. A legacy `codex_local` agent created with the
new default on a host with an older Codex CLI gets the existing
actionable "upgrade Codex" Test result.
- Qualification: the native Codex profile uses `adapter_constant` model
qualification, so it follows the default. No qualification gate is
dropped. The ACPX Codex qualification selection moves to `gpt-6.1-sol`.
Its next qualification run uses the new model. The provider-usage
cache-policy attribution stays limited to `gpt-5.6-sol`.
- Runner E2E history: the Codex cells change their suite definition
hashes, so earlier `gpt-5.6-sol` results are a separate cohort and do
not count as current.
- The `gpt-6.1-sol` scheduled price uses the file's existing `expiresAt`
(2026-11-19). It needs the same review as the other entries before then.
- No `.github/` change. The Reliability queue-steer cell keeps the
`runner-codex` profile id, so the existing workflow credential and setup
steps still match.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, running in
Claude Code with tool use (shell, file edits, web fetch of the OpenAI
pricing page).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Related: #19604 (commits the Reliability TestOps identities; this PR
changes only `steeringProfile` in the same file).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
744056ecc8 |
feat(evals): run the staging-soak reliability cases from a TestOps plan (#19549)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The runner full-stack E2E lane runs real Paperclip work against real agents, and TestOps launches its standing plans > - PR #19455 added seven staging-soak reliability cases, but no TestOps plan or workflow lane selects them, so they run only by hand with personal keys > - These cases catch restart, interrupt, queue and lost-runner regressions, so they must run from TestOps before a release with the standard reports > - This pull request adds a sibling "Reliability — staging soak" TestOps plan on the existing plan-2 lane, with native OpenCode and DeepSeek 1017 through OpenRouter as the plan profile for six of seven cases > - The benefit is a cheap, repeatable pre-release reliability launch with the same Allure/TestOps results, retained report and dashboard as plan 2 ## Linked Issues or Issue Description Refs #19455 **What existing behavior does this improve?** The staging-soak reliability cases from #19455 run only by a manual local command. **Subsystem affected** Runner full-stack E2E (`tests/runner-e2e/`) and its TestOps lane in `.github/workflows/runner-full-stack-e2e.yml`. **Current behavior** No TestOps plan includes the cases. The workflow cannot select them from TestOps. Every cell uses native Codex (`gpt-5.6-sol`) or an ACPX facade on Daytona, which is expensive. **Proposed behavior** A TestOps launch of the Reliability plan runs seven cells (six DeepSeek 1017, one native Codex for steering) through the standing plan-2 path and reports them like plan 2. **Reason and benefit** The cases become a first-class, low-cost, pre-release check with retained evidence. Plan 2 keeps its 20-case qualification semantics. **Breaking changes** None. Plan 2 selection, reporting and history identities stay the same. ## What Changed - **Plan choice: a sibling plan, not an extension of plan 2.** Plan 2 is bounded at 20 standing IDs. Its `full-run` / `test-plan-2` tag needs a complete 20-case attempt, and subset runs do not qualify it. Adding seven cases would change that denominator and the qualification history. The sibling plan shares the same TestOps GitHub job, selection, admission, projection, upload and verification code. A launch selects cases from exactly one plan (`plan_mixed` fails), and each plan has its own full-plan size (20 or 7). - `tests/runner-e2e/reliability-testops.mjs` (new): the mapping module. It has seven cases, full names `paperclip.Product E2E.everyday-workflows.<case>`, execution IDs, soak signatures and an empty `reliabilityRegistration`. A partial, duplicate, plan-2 or malformed registration is rejected. An empty registration makes selection fail closed before any spend (`product_regression_testops_reliability_unregistered` for a bootstrap, `case_mismatch` for a TestOps launch). No TestOps IDs are invented. - `product-regression-testops.mjs` / `-cli.mjs`: select a plan by its case identities or by the new bootstrap input. The projection binds each cell's own environment (`local` or `daytona`) instead of a fixed `local`. It adds a `soak-<signature>` tag and reports `plan` and a per-plan `fullPlan`. Plan-2 history IDs, parameters and results do not change. - Catalog: `runner-opencode-deepseek-1017` is now one shared fixture with `supportedEnvironments: ["local", "daytona"]`. The OpenCode server driver already runs on Daytona for `runner-opencode` (task-workspaces, core-compatibility, rolling deployment); only the model differs. The chat-feature suite stays local, and its definition hash does not change. `everyday-workflows` gains exactly six DeepSeek plan cells (definition version 12, 52 + 13 + 3 cells). - Cell matrix (plan), DeepSeek 1017 on native OpenCode: `interrupt-redirect`, `reassign-midrun` and `tree-pause-release` run on `local`. `three-turn-revise` runs on Daytona with the per-turn lease (S49). `long-silent-command` (S52/S52b) and `queued-across-restart` (S55/U9) run on the Daytona warm lease. - `queue-steer` is the one exception. It must steer an active turn, and the OpenCode server driver declares `steering: false`, so the route answers `steering_unsupported` and the case cannot pass. The plan uses the existing steering-capable `everyday-workflows.runner-codex.daytona.queue-steer` cell for it (Greptile finding). No DeepSeek `queue-steer` cell exists. The other native Codex Daytona cells and the ACPX facades stay selectable through `--group reliability`, but they are outside the plan. - `runner-full-stack-e2e.yml`: adds the `testops_plan` bootstrap input (`product-regression` by default), sets the plan-specific bootstrap launch name and tags, and renames two jobs to plan-neutral names. The OpenCode steps now match `startsWith(matrix.profileId, 'runner-opencode')`, so the DeepSeek profile gets the pinned OpenCode binary, runner TypeScript, the local Node check and the remote provider pack. Before this change, the DeepSeek profile could not run in this workflow. `OPENROUTER_API_KEY`, `OPENAI_API_KEY` (only for `queue-steer`) and `DAYTONA_API_KEY` come from the same `matrix.credentialName` and `matrix.environmentId` gates that other cells use. - `runner-e2e-testops-registration.yml` + `reliability-testops-registration.mjs` (new): a trusted, master-only, allowlisted-actor, idempotent `workflow_dispatch`. `inspect` only reads. `register` reuses cases with the same full name, creates missing cases and the plan, and assigns the plan to plan 2's GitHub job with plan 2's job parameters. It writes a receipt with a ready-to-commit `reliabilityRegistration` snippet. Only this job gets `ALLURE_TOKEN`, and it never runs a provider cell. - **No schedule and not per PR.** The plan runs only from a TestOps launch or a manual dispatch. It is not in the daily or weekly cron (`scheduledSelection` excludes it). A weekly run would add a paid 50-minute Daytona cell every week, but these signatures move with release-risky changes, not with time. - Docs: `EVERYDAY-WORKFLOWS.md` (matrix, environment reasons, local coverage limits, registration steps), `README.md`, `doc/evals.md`. ### Registration still needed (requires TestOps credentials) 1. Dispatch **Runner E2E TestOps registration** from master with `operation=inspect`. 2. Dispatch it with `operation=register`. 3. Commit the `reliabilityRegistration` snippet from the run summary into `tests/runner-e2e/reliability-testops.mjs` in one reviewed PR. 4. Launch the plan from TestOps. Tag only a complete seven-case attempt `full-run`. ### What local plan cells cannot reproduce The local cells do not reproduce the Daytona-only variants: the remote `workspace_busy` save after Stop (S47), remote stop settlement during reassignment (S48), and a lost or replaced remote runner during a tree hold (S50). S51 (lost runner) has no case yet. The explicit-only native Codex Daytona extras still cover S47, S48 and S50 on a sandbox. ## Verification - `node --test tests/runner-e2e/product-regression-testops.test.mjs tests/runner-e2e/reliability-testops.test.mjs`: 26 pass. These tests cover fail-closed selection, partial or duplicate registration, plan mixing, environment binding, per-plan denominators, preserved failure classes, the CLI select → project → verify flow for a seven-case launch, registration idempotency and job checks, and the workflow inputs and credential gates. - `vitest` for `reliability-testops-catalog`, `product-regression-testops-catalog`, `everyday-reliability` and `catalog.test`: 98 pass. These tests bind the mapping to real catalog cells (profile, model, environment, credentials, single attempt). They also check that the cells stay out of `--all`, the suite, profile and environment selectors and the daily/weekly cron. Real grader output must survive projection. - `pnpm test:e2e:runner:typecheck`: pass. - `pnpm test:e2e:runner -- --list --suite everyday-workflows --group reliability` lists 16 cells: 6 DeepSeek plan cells, 7 native Codex cells (one is the `queue-steer` plan cell) and 3 ACPX extras. `--matrix-json` for the DeepSeek plan IDs emits `needsDaytona=true` and three Daytona OpenCode cells with `credentialName=OPENROUTER_API_KEY`. - Full `pnpm test:e2e:runner:unit`: the remaining failures happen the same way on an unchanged master checkout. They are in `workflow-security` (3), `controller-outage`, `operating-receipts`, `pi-*`, `private-framework-report` and `remote-native-bootstrap`. #19529 tracks that drift. This PR adds no failures. - Not run: paid cells, a TestOps registration and a TestOps launch. They need provider and TestOps credentials, which exist only in protected CI environments. Catalog and projection tests are not live qualification. ## Risks - The registration script calls TestOps APIs (`POST /api/testcase`, `/api/testplan`, `/assign`, `/jobparameter`) through the same endpoints as the chat catalog reconcile. Tests use fakes, not the live API. `inspect` comes first, and `register` is idempotent. - DeepSeek 1017 on Daytona has not run before. The first launch is the acceptance step for those three cells. - If a later case needs another capability that OpenCode lacks, it must use a capable profile in the same way as `queue-steer`. The catalog test pins the single exception. - The OpenCode workflow conditions now match any `runner-opencode*` profile. Today this covers `runner-opencode` and `runner-opencode-deepseek-1017` only. - If someone commits a bad `reliabilityRegistration`, module load fails for both plans. A unit test rejects any committed registration that is not empty or complete. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
ac57b183f0 |
fix(evals): run campaign cells when TestOps selection is skipped (#19053)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner Full-Stack E2E workflow runs the paid Product E2E cells, including the Product regression TestOps plan > - #18983 added an optional `testops_selection` job and made `catalog` depend on it > - That job is skipped in every ordinary manual or scheduled campaign > - GitHub's implicit `success()` also fails when an ancestor job was skipped, so every job below `catalog` was skipped > - Since #18983 merged, ordinary campaigns run no cells: the merge reports every selected cell as "No result artifact was uploaded" > - This pull request gives each job below `catalog` an explicit status condition > - The benefit is that ordinary, nightly and branch-targeted campaigns run their cells again, and TestOps campaigns are unchanged ## Linked Issues or Issue Description Refs #18983, #19000 **What happened?** Run https://github.com/paperclipai/paperclip/actions/runs/41914430034 selected 20 Product regression cells with the `id` input. `catalog` passed. `daytona_image`, `build_runner_artifacts`, `build_remote_provider_pack` and every matrix job were skipped. The merge failed with 20 "No result artifact was uploaded" rows. **Expected behavior** An ordinary manual campaign runs its selected cells when `testops_selection` is skipped, as it did before #18983. **Steps to reproduce** Dispatch `runner-full-stack-e2e.yml` on `master` without `testops_case` or `ALLURE_JOB_RUN_ID`. The scheduled daily campaign has the same path. ## What Changed - `daytona_image`, `build_runner_artifacts`, `build_remote_provider_pack`, `test`, `test_recurring` and `test_manual_private` now start with `!cancelled()`. - Each of those jobs now requires `needs.<job>.result == 'success'` for each of its real dependencies. A failed or skipped build still skips the cells, as before. The existing schedule, cadence and evidence-route conditions are unchanged. - A new regression test parses the workflow. It requires every job below `testops_selection` to use a status function and to require each real dependency result. ## Verification - `node --test tests/runner-e2e/product-regression-testops.test.mjs`: 15/15 pass. - Negative control: the new test fails against the current `master` workflow. - `daytona-image.test.ts`, `workflow-target.test.ts` and `workflow-timeout.test.ts` pass. `workflow-security.test.ts` has 5 failures. The same 5 fail on unchanged `master` (they concern `pr-trusted.yml` and `chat-catalog-testops-case.yml`). - A workflow change takes effect only from `master`, because paid campaigns run only from the default branch. After merge, dispatch an ordinary `id=` campaign and confirm the cells run. ## Risks - Low risk. The conditions keep the previous failure propagation: a job runs only when each direct dependency succeeded and the run is not cancelled. - `catalog` success already implies `authorize` and `target_lock` success, so those are not repeated. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
c99bcd89fe |
fix: preserve Product TestOps evidence and CI readiness (#19000)
## Thinking Path > - Paperclip manages agent work through governed product workflows. > - Product E2E tests check those workflows through the app and Runner. > - A trusted reporter sends retained results to TestOps. > - Free-form metadata and a short upload observer can lose safe, valid evidence. > - UUID navigation can also discard task metadata before the canonical query returns. > - Hermetic CI files start child processes that need CPU capacity within their readiness limits. > - This change closes the reporting contract, retains authenticated core aliases, and reserves CI CPU capacity. > - Original case identities, conditions, assertions, verdicts and deadlines remain intact. ## Linked Issues or Issue Description Refs #18983. This addresses its late [metadata disclosure finding](https://github.com/paperclipai/paperclip/pull/18983#discussion_r4470789114) and [upload readiness finding](https://github.com/paperclipai/paperclip/pull/18983#discussion_r4470790366). **What happened?** The reporter accepted model IDs and matcher names through syntax filters. A secret-shaped string could pass those filters. First-task results also use their top-level model field for an observed alias. Treating it as configured metadata can reject valid retained evidence. The upload observer stopped after 22 seconds of waits. [Required CI on the earlier head](https://github.com/paperclipai/paperclip/actions/runs/41886885193) also found two real browser title failures and five native package readiness failures. A deterministic control reproduced metadata loss during UUID-to-identifier navigation. A separate fixed-resource control reproduced startup deadline failures with four concurrent hermetic fixtures, while one and two fixtures passed. **Expected behavior** Reject unknown transmitted metadata before creating upload files. Preserve every original matcher verdict and attempt receipt. Observe server processing without rerunning the product. Keep authenticated task metadata visible through canonical navigation. Reserve capacity for hermetic child processes without changing the original readiness deadlines. **Steps to reproduce** Use a secret-shaped model ID or cleanup state in retained evidence, return an observed first-task model alias, or let upload processing settle after 40 seconds of virtual time. Hold the canonical core query pending after an authenticated UUID read. For the CPU control, run the same providerless startup/recovery fixture alone, four times concurrently, and twice concurrently under a fixed 4-vCPU quota and its original 8-second readiness limit. ## What Changed - Bind configured model receipts to each exact declared case model. Four first-task cases use captured agent configuration; their observed aliases and fallback text remain digest-only. A missing capture stays null / Not recorded. The other 16 cells use their configured profile model. Reject wrong or unknown configuration before any upload file exists. - Keep the existing model distribution: 18 Codex cells on `gpt-5.6-sol`, configured-repository case 9 on its existing `gpt-6.1-sol` override, and task-workspaces case 54 on `openrouter/deepseek/deepseek-v4-flash-0731`. The repository override exists at the [original canary source](https://github.com/paperclipai/paperclip/blob/86814c6550f52e99ede150a883bdc773cbb540e3/tests/runner-e2e/catalog.ts#L1148). No model configuration changes. - Allow only existing Matcher kinds, closed failure classes, and cleanup states `passed`, `failed`, or `not_started`. Publish each matcher's kind, ordinal, original boolean verdict and SHA256 path receipt. Full conditions remain in the linked original workflow artifact; raw dynamic path bytes do not enter TestOps. - Observe the same upload for at most 55 seconds of poll waits. Product scenarios, provider retries and scenario deadlines do not change. - Retain task core data under only the authenticated response's own UUID and identifier core keys. Preserve existing canonical data. Keep arbitrary references and full-detail keys empty until their own reads. - Retain a first-failure trace for the existing Cloud recovery test. Keep its assertions, timeout, retries, mock data and artifact policy unchanged. This adds diagnosis evidence; it does not claim to fix that separate Cloud title timeout. - Reserve half the available CPUs for Runner package CI workers, with the existing upper cap of four. A 4-vCPU host uses two workers; an 8-vCPU host still uses four. Local defaults and test selection do not change. ## Verification - Reporting controls first reproduced the two original gaps. All 14 Node contract tests now pass. Unknown configuration creates neither a result directory nor an export summary. A model valid for a different selected case is also rejected. Original functional failures and earlier accounting receipts survive. - The catalog compatibility test passes for all 20 selected cells, exact model tuples, real pure grader outputs and actual `projectFirstTaskResult` shapes. It covers observed aliases, fallback text, absent evidence and wrong or secret-shaped configured metadata. These are contract controls, not live journey passes. - Virtual-clock controls accept a correct result that settles at 40 seconds and reject a persistently wrong verdict after 55 seconds. They make no product or provider calls. - Two new core-cache controls failed before repair. The query/cache suites now pass all 8 tests. They hold canonical refresh pending and verify title/company retention, newer canonical data, full-detail enrichment and unrelated keys. UI typecheck and token gates pass. - The unchanged exact UUID cross-company link/upload browser test passed once after repair: 5.8-second scenario, 32.0-second complete command. The unchanged exact Cloud task-entry diagnostic passed once: 5.4-second scenario, 23.3-second complete command. Its original canonical-only timeout remains unexplained. All five Cloud tests still collect after the trace setting. - The providerless Linux control used the retained Runner and compiled controller from source `3a2249dd`, with a fixed 4-vCPU quota and original 8-second readiness limit. Solo passed 1/1; four concurrent fixtures failed 4/4; two concurrent fixtures passed 2/2. At each failure, Rust had accepted command 2 but startup was still pending; checkpoint and published status agreed, one connection was authenticated, no replay backlog existed, and the sidecar had not launched. Passing controls preserved the expected startup failure through controller replacement. No network or credentials were available. - That control uses Linux amd64 emulation and Node 24.21; hosted CI used Node 24.22.1. It supports the worker-cap change, but does not prove that the original `m5n.xlarge` host caused every CI failure. The original red attempts remain recorded. - Actual config-load controls verify 1 CPU → 1 worker, 2 → 1, 4 → 2, 8 → 4 and 16 → 4, with no local, timeout or retry override. The isolated policy change preserves the same 297 test-file inventory; integration of current master `89703c3ff7` retains all 298 current native package files. The original PR package/native command plans remain unchanged. - Final head `877e28da32f2d2f0d12906c89aff085e2c6ea943` passes both required checks, `ci / verify` and `ci / e2e`, in [CI run 41897677652, attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41897677652/attempts/2). The native package passes 4,950 tests with 10 existing skips. All eight browser shards pass; the Cloud shard passes all 41 tests, including all four unchanged recovery cases. - Attempt 1 preserved 46 successful jobs but its one-file heartbeat recovery shard hit the original 20-minute job limit without a named assertion or completion verdict. Its cause remains unidentified. One explicitly bounded debug replay retained those 46 successes and ran only that leaf and its dependent rollup. The unchanged file passed all 412 tests (346.91-second file, 371.34-second Vitest command, 7m11s job). Source, cases and deadlines did not change. This establishes intermittency, not a causal repair. The original timeout and debug pass both remain recorded. - Fresh [Greptile 5/5](https://github.com/paperclipai/paperclip/pull/19000#issuecomment-6637604918) and [Ponytail 5/5](https://github.com/paperclipai/paperclip/pull/19000#issuecomment-6637489454) reviews name exact head `877e28`. The single review thread is resolved. Live TestOps bootstrap and complete 20-case acceptance remain held until reviewed delivery. CI and focused controls do not count as product plan passes. ## Risks - Future model or Matcher kind changes require a reviewed reporting-contract update. - TestOps shows a closed Matcher kind and ordinal. The path digest binds the original condition in retained evidence. - Core alias data retains the existing authenticated query-client and server company boundaries. It never becomes full-detail evidence. - Fewer CI workers can increase one job's duration. The full suite must verify the final policy. The paid evaluation matrix and its parallelism do not change. - The original Cloud title timeout and the first-attempt heartbeat CI hang do not have proven causes. Current unchanged assertions pass, but this does not establish repeatability. The Cloud test now retains its first-failure trace. A successful upload or focused control alone cannot establish a complete product pass. ## Model Used OpenAI Codex with reasoning, tool use and code execution. The host does not expose the exact model ID or context window to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b99dd57c82 |
test(evals): run Product regression from TestOps (#18983)
## Thinking Path > - Paperclip manages agent work through governed product workflows. > - Product E2E tests check those workflows through the real app and runner. > - The saved Product regression plan has 20 imported case identities. > - The plan has no live executor, so historical baseline rows do not qualify its current behavior. > - The existing paid full-stack workflow already owns all 20 required journeys. > - This change binds TestOps selection and reporting to that workflow without changing the product oracle or paid admission. > - Maintainers can run the plan and inspect current case verdicts and retained evidence. ## Linked Issues or Issue Description **What happened?** The standing Product regression plan has no executor. Its dashboard points at a historical 355-result snapshot across several suites and revisions. **Expected behavior** A TestOps plan run must select the intended existing full-stack cells, execute them on GitHub, and report their original case IDs, assertions and source revision. Missing evidence must remain a visible failure. **Steps to reproduce** Open the Product regression plan and inspect Executors. There is no live job. The existing full-stack workflow accepts exact cell IDs but has no TestOps selection or reporting path. **Paperclip version or commit** The branch includes reviewed master correction |