mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 21:35:36 +01:00
DottaandPaperclip
b99dd57c82
## Thinking Path > - Paperclip manages agent work through governed product workflows. > - Product E2E tests check those workflows through the real app and runner. > - The saved Product regression plan has 20 imported case identities. > - The plan has no live executor, so historical baseline rows do not qualify its current behavior. > - The existing paid full-stack workflow already owns all 20 required journeys. > - This change binds TestOps selection and reporting to that workflow without changing the product oracle or paid admission. > - Maintainers can run the plan and inspect current case verdicts and retained evidence. ## Linked Issues or Issue Description **What happened?** The standing Product regression plan has no executor. Its dashboard points at a historical 355-result snapshot across several suites and revisions. **Expected behavior** A TestOps plan run must select the intended existing full-stack cells, execute them on GitHub, and report their original case IDs, assertions and source revision. Missing evidence must remain a visible failure. **Steps to reproduce** Open the Product regression plan and inspect Executors. There is no live job. The existing full-stack workflow accepts exact cell IDs but has no TestOps selection or reporting path. **Paperclip version or commit** The branch includes reviewed master correction04a24f4e06. This change does not alter scenario prompts, models, deadlines or assertions. Related public work: #18542 establishes the separate chat-catalog lane. #18666 repairs its workflow URL binding. These changes do not implement this Product plan. ## What Changed - Add optional TestOps service inputs and a manual bootstrap selector to the existing full-stack workflow. - Map the original 20 case IDs and full names to existing native Codex/local cells, with the existing native OpenCode/local task-workspaces case. - Keep reporting tokens in trusted default-branch selection and publication jobs. Keep the existing paid provider authorization and protected environments. - Bind retained results to the workflow, job run, launch, source revision and selection hash. Preserve missing-result failures and earlier attempt verdict receipts. - Verify uploaded identities, coverage and verdicts. Document full-plan and subset reporting limits. - Keep the outstanding-decision reservation live while its existing deletion-fence test asserts the original 409 guard. Release the test provider in finally. Production deletion and accounting code stay unchanged. ## Verification - `node --test tests/runner-e2e/product-regression-testops.test.mjs`: 13 positive and negative controls pass, including the separated CLI handoff. - Existing paid-workflow security regression passes. - `actionlint -shellcheck= .github/workflows/runner-full-stack-e2e.yml`: passes. - The Product E2E catalog compatibility test confirms all 20 exact cells exist: passes. - `pnpm test:e2e:runner:typecheck`: passes. - Fresh Greptile review of final head61ef7a6eis 5/5 (11 files, no new findings); both initial reporting findings are repaired and resolved. - `server/src/__tests__/decision-models.test.ts`: all 37 controls pass on a disposable official PostgreSQL 18 container in 22.02 seconds. The unchanged focused case passed once and all five local samples; the original Linux CI failure remains retained. This is not a claimed deterministic local red reproduction. The new gate explicitly preserves the outstanding-reservation precondition. - The existing master readiness-fixture correction passes all three retained-workspace controls on the same database in 25.79 seconds. It is incorporated from master, not duplicated in this PR's diff. - Final-head Linux CI also passed all 37 decision-model controls in 7.047 seconds; the server shard passed 120 files and 2,004 tests. See [the retained successful shard](https://github.com/paperclipai/paperclip/actions/runs/41870214517/job/126715358759). - Previous CI failures are retained: [three stale readiness-fixture cases](https://github.com/paperclipai/paperclip/actions/runs/41864061890/job/126694812885) and [the reservation fixture race](https://github.com/paperclipai/paperclip/actions/runs/41864061890/job/126694812894). The separate [Canary Dry Run](https://github.com/paperclipai/paperclip/actions/runs/41864061890/job/126694811605) recorded a lost self-hosted runner connection before any step. The fixture repairs are verified on changed source; the separate Spot-interrupted CI attempt and its bounded retry are recorded below. - Actual one-cell GitHub canary on the unchanged product source: https://github.com/paperclipai/paperclip/actions/runs/41860189801 . The native Codex/local message-marker case passed all six assertions and cleanup in 35.24 seconds. The persisted issue is Done and its one run succeeded. This is supporting evidence; it is not TestOps acceptance. - Final-head CI attempt 1 was interrupted: three failed jobs have EC2 `Server.SpotInstanceTermination` receipts; six cancelled jobs show the same simultaneous shutdown. Original logs are retained. One [failed-job retry](https://github.com/paperclipai/paperclip/actions/runs/41870214517/attempts/2) passed all nine interrupted checks while preserving prior successful jobs. - Final-head full repository CI passes, including both required `ci / verify` and `ci / e2e` gates. GitHub still requires a non-author code-owner approval for the workflow change; normal auto-merge is queued. A fresh TestOps-triggered subset/full-plan acceptance remains pending. Local projection checks do not establish a live Product pass. ## Risks - The first bootstrap upload must establish the official TestOps job before the saved plan can use it. - This lane tests one declared local native configuration per case. It does not qualify the historical 355-row matrix. - Upload failures remain visible. A missing retained report produces broken case results and cannot pass. - The existing non-TestOps campaign path must keep its selection and scheduling behavior; paid-workflow regression coverage remains in place. ## Model Used OpenAI Codex with reasoning, code execution and tool use. The host does not expose the exact model ID or context window to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
14 lines
835 B
TypeScript
14 lines
835 B
TypeScript
import { configDefaults, defineConfig } from "vitest/config";
|
|
|
|
export default defineConfig({
|
|
root: import.meta.dirname,
|
|
test: {
|
|
name: "runner-e2e-support",
|
|
globalSetup: ["./verified-provider-bundles.setup.mjs"],
|
|
include: ["**/*.test.ts", "**/*.test.mjs"],
|
|
// These use node:test and run separately in test:e2e:runner:unit.
|
|
exclude: [...configDefaults.exclude, "product-regression-testops.test.mjs", "native-completion-{checks,git-source,source-contract}.test.mjs", "artifact-budget.test.mjs", "chat-feature-testops-api.test.mjs", "chat-feature-testops-cli.test.mjs", "chat-feature-testops-results.test.mjs", "testops-chat-reporter.test.mjs", "chat-completion-testops-{api,contract}.test.mjs", "chat-full-catalog-testops-*.test.mjs", "chat-catalog-testops-streaming.test.mjs"],
|
|
testTimeout: 30_000,
|
|
},
|
|
});
|