mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-03 17:51:58 +01:00
test(evals): retire duplicate chat TestOps plans (#18625)
## Thinking Path > - Paperclip manages AI agents and preserves their work history. > - Product E2E cases use TestOps plans to select real AWS evaluations. > - The full chat catalog now includes the former smoke and completion-pilot cases. > - The two old bootstrap workflows can recreate their smaller plans after deletion. > - This pull request removes those workflows and stops their plan-creation functions before API access. > - The full-catalog workflow keeps its shared clients, evidence rules, case identities, and measured results. ## Linked Issues or Issue Description Refs #18542, #18516. The user requested removal of the duplicate smoke and completion-pilot plans and investigation of their recreation paths. Both old workflows accepted manual dispatch only. Their bootstrap operations created plans by name when the plans were absent. They had no schedules in their workflow definitions. The old workflow records were disabled before plan removal. An existing pilot execution was allowed to finish successfully. This change does not cancel executions or delete cases, results, launches, or private evidence. ## What Changed - Remove `chat-feature-testops.yml` and `chat-completion-testops.yml`. - Make the former singleton plan creator reject every call before API reads or mutation markers. - Reject pilot bootstrap before API access. Keep the full-catalog branch and its integration validation. - Remove tests that require the retired workflow files. Keep shared helper and full-catalog controls. Run the trusted-checkout boundary test against the active catalog workflow. - Replace old setup instructions with retirement guidance and the active full-catalog route. The master sync preserves the incoming typed launch-search fix from #18620, including both regression tests. The only conflict was the retired pilot documentation. Its search-format note remains as a retained-helper contract. ## Verification - Two new retirement controls fail against the parent code because it reaches an API or mutation marker. Both pass after the change. - Six owning Node suites after the master sync: 146 passed (144 before the sync). Shared contract: 22 passed. Catalog and retained export controls: 12 passed. Focused workflow and release controls: 7 passed. - Two modified API modules pass Node syntax checks. `git diff --check` passes. - The pure workflow-security file has the same four failures and 13 passes on both the untouched parent and this branch. Two assertions expect eight trusted bootstrap steps where master now has nine. Two lists omit the already-shipped catalog workflow. This change does not repair or relax those unrelated assertions. - A broader release-test invocation had 73 passes and 20 sandbox failures from blocked mock HTTP listeners or npm permissions. Its focused affected workflow control passes. A standard Vitest setup attempt stopped on a missing `acpx/runtime` bundle dependency before tests. No local app, database, provider, or native runtime was qualified. - Current-head PR CI passes: 48 successful jobs and one skipped job, including both required checks. Greptile is 5/5. Independent source review passes. No new model run is needed for this retirement. ## Risks - Existing integrations that point at either retired workflow will stop dispatching. This is the requested behavior; use the full catalog instead. - Historical evidence readers still use shared helpers. Those helpers remain, including the transport and official-context parser used by the active catalog. - Workflow removal is a repository change. TestOps plan deletion and disabling obsolete executors are separate service actions. Neither action should delete shared cases or measurements. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with source editing, shell tools, and independent source review. The context-window size is not asserted. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
3f7811eef0
commit
97c4dc969e
14 files changed
+46
-1491
No files matched your search
@@ -579,8 +579,6 @@ test("Runner eval workflows pin actions and gate paid live execution", () => {
|
||||
readWorkflow("runner-full-stack-e2e.yml"),
|
||||
readWorkflow("e2e.yml"),
|
||||
readWorkflow("runner-protocol-live-evals.yml"),
|
||||
readWorkflow("chat-feature-testops.yml"),
|
||||
readWorkflow("chat-completion-testops.yml"),
|
||||
readWorkflow("chat-catalog-testops.yml"),
|
||||
];
|
||||
|
||||
@@ -621,8 +619,6 @@ test("Runner eval workflows pin actions and gate paid live execution", () => {
|
||||
"runner-full-stack-e2e.yml",
|
||||
"runner-live-evals.yml",
|
||||
"runner-protocol-live-evals.yml",
|
||||
"chat-feature-testops.yml",
|
||||
"chat-completion-testops.yml",
|
||||
"chat-catalog-testops.yml",
|
||||
];
|
||||
const paidWorkflowNameSet = new Set(paidWorkflowNames);
|
||||
@@ -705,38 +701,7 @@ test("Runner eval workflows pin actions and gate paid live execution", () => {
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (name === "chat-feature-testops.yml") {
|
||||
assert.equal(providerJobs.length, 1);
|
||||
const assertBoundary = (block) => {
|
||||
assert.match(block, /^ chat:\n/);
|
||||
assert.match(block, /\n needs: \[authorize, select, build\]\n/);
|
||||
assert.match(block, /\n if: needs\.authorize\.outputs\.operation == 'evaluate'\n/);
|
||||
assert.match(block, /\n environment: runner-e2e-private-paid\n/);
|
||||
assert.match(block, /\n - \*checkout\n - \*node\n - \*guard\n/);
|
||||
assert.match(block, /testops-chat-reporter\.mjs verify-selection chat-testops-selection\/testops-selection\.json/);
|
||||
assert.match(block, /OPENROUTER_API_KEY: \$\{\{ secrets\.OPENROUTER_API_KEY \}\}/);
|
||||
assert.match(block, /--id chat-features\.runner-opencode-deepseek-1017\.local\.create-backlog --max-parallel 1 --max-automatic-retries 0/);
|
||||
assert.doesNotMatch(block, /ALLURE_TOKEN/);
|
||||
assert.equal([...block.matchAll(providerSecretReference)].length, 1);
|
||||
assert.ok(block.indexOf(" - *guard") < block.search(providerSecretReference));
|
||||
assert.ok(block.indexOf("testops-chat-reporter.mjs verify-selection") < block.search(providerSecretReference));
|
||||
};
|
||||
const block = providerJobs[0];
|
||||
assertBoundary(block);
|
||||
assert.equal([...workflow.matchAll(providerSecretReference)].length, 1);
|
||||
assert.match(workflow, /ref: \$\{\{ github\.sha \}\}/);
|
||||
assert.match(workflow, /run: node tests\/runner-e2e\/testops-chat-dispatch\.ts/);
|
||||
assert.match(workflow, /TESTOPS_ALLOWED_ACTOR_IDS: \$\{\{ vars\.RUNNER_E2E_ALLOWED_ACTOR_IDS \}\}/);
|
||||
for (const mutant of [
|
||||
block.replace("needs: [authorize, select, build]", "needs: [authorize, build]"),
|
||||
block.replace("environment: runner-e2e-private-paid", "environment: unprotected"),
|
||||
block.replace(" - *guard\n", ""),
|
||||
block.replace("testops-chat-reporter.mjs verify-selection", "testops-chat-reporter.mjs unverified-selection"),
|
||||
block.replace("OPENROUTER_API_KEY:", "ALLURE_TOKEN:"),
|
||||
block.replace("--id chat-features.runner-opencode-deepseek-1017.local.create-backlog", "--id chat-features.runner-opencode.local.create-backlog"),
|
||||
]) assert.throws(() => assertBoundary(mutant));
|
||||
continue;
|
||||
}
|
||||
|
||||
for (const block of providerJobs) {
|
||||
if (name === "runner-protocol-live-evals.yml") {
|
||||
assert.ok(block.includes("name: ${{ needs.authorize.outputs.accounting_mode == 'controlled' && 'runner-e2e-private-paid' || 'runner-e2e-paid' }}"));
|
||||
|
||||
Reference in new issue
Block a user