From e5e22148e4a7b1209c49d2b11593fd59caee8218 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 2 Nov 2026 20:28:06 -0600 Subject: [PATCH] test(evals): pin the reviewed chat catalog fixture repairs (#18710) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The chat catalog uses a trusted TestOps workflow and an immutable candidate source. > - The retained first pass found fixture failures before real feature work could start. > - Candidate repairs must use the same source checks as the original lane. > - This pull request advances only the fixed catalog target and its source note. > - New AWS measurements keep the original case queries, limits, graders and reporting path. ## Linked Issues or Issue Description Refs: #18542, #18647, #18680, #18693. **What happened?** The full-catalog lane remains pinned to the original candidate. A repair candidate cannot run through the trusted TestOps selection without a reviewed pin change. **Expected behavior** TestOps must select the reviewed immutable repair source. It must keep the existing case identities, model, clocks, private evidence controls and original grading. **Steps to reproduce** Inspect the full-catalog target in the manifest. The trusted contract uses that exact commit for selection, source checks, build and export. ## What Changed - Advance the immutable full-catalog target to [`9ab0f951a5b5990c266ef4456610e0a0775aa334`](https://github.com/paperclipai/paperclip/commit/9ab0f951a5b5990c266ef4456610e0a0775aa334). [Candidate-only delta](https://github.com/paperclipai/paperclip/compare/3b66c3c144360ddf32714a0742fd774b8ceafb16...9ab0f951a5b5990c266ef4456610e0a0775aa334). - Preserve all original queries and limits. Record company observer source ownership and the secret-entry v2 oracle as explicit metadata overrides against the original definition source. - Explain SDK staging, scoped human membership, original-clock observation and exact controller-secret baseline conditions. Preserve the original failed measurements. - Leave the application and held core fixes off master. This change does not copy the candidate app or catalog implementation to master. ## Verification - Candidate integration: 167 focused pure tests passed for SDK custody, membership, feature and privacy controls. - Trusted target consumers: 46 manifest, full-catalog adapter and dispatch contract tests passed. - The candidate has no diff to the original YAML queries or descriptor bytes. - `git diff --check` passed. - Local full typecheck, application build, DB and model execution were not run for this two-file pin change. Worker typecheck attempts hit existing missing workspace dependencies and are not recorded as passes. The trusted AWS source check and build must pass before any native query. It imports the real candidate catalog and checks every descriptor before provider access. It does not typecheck all harness TypeScript files. - Initial CI41563164012 passed48 jobs with1 skip. A later review requested the exact target assertion; the valid-SHA mutant is now rejected. Corrected-head CI41564795118 passed48 jobs with1 skip on2226f093. Both required checks passed. Greptile and the follow-up source review are5/5 with no unresolved threads. No new candidate case result is claimed here. ## Risks A new fixture condition can change a measured outcome. Each result records the actual candidate and trusted workflow revision. Original failed measurements remain unchanged. The model, case bounds, independent feature requirements, private custody, reporting identities and automatic retry count remain fixed. This pin applies to future full-catalog runs, so operators must not infer that every catalog case has been qualified. The inherited source-ref label is not source authority; the exact checked commit is authoritative. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with reasoning, source editing and tool use. The exact context window is not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes:` / `Closes:` / `Refs:` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable; the existing target consumers cover this data-only change - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- tests/runner-e2e/CHAT-FEATURES.md | 18 ++- .../chat-full-catalog-testops-manifest.json | 103 +++++++++++++++++- ...hat-full-catalog-testops-manifest.test.mjs | 22 ++++ 3 files changed, 136 insertions(+), 7 deletions(-) diff --git a/tests/runner-e2e/CHAT-FEATURES.md b/tests/runner-e2e/CHAT-FEATURES.md index a2267a42242..c1ce3eca7e7 100644 --- a/tests/runner-e2e/CHAT-FEATURES.md +++ b/tests/runner-e2e/CHAT-FEATURES.md @@ -217,10 +217,20 @@ overlap. Exact requirements and limitations appear in each case's preconditions. A declared credential does not establish whether its runtime reference exists. This lane runs the app and fixtures from candidate -`3b66c3c144360ddf32714a0742fd774b8ceafb16`. Its app and domain definitions are -unchanged from `de0941835cc8f0d01c982574ee258e110babdbeb`; the successor adds only -finite TestOps host admission. Trusted selection, evidence transfer and publishing -use the workflow's immutable master revision. Each result records both revisions. +`9ab0f951a5b5990c266ef4456610e0a0775aa334`. It retains the evaluated app from +`3b66c3c144360ddf32714a0742fd774b8ceafb16` with four bounded fixture repairs: +owned SDK staging, public membership for three incidental human fixtures, +company settings observation inside its original case deadline, and exact +controller-secret baseline handling for secure human entry. The last repair +uses the explicit `company-secret-controller-baseline.v2` oracle. Its deployment +secret, Morgan-only binding and privacy checks remain required. + +The manifest retains the original `de0941835cc8f0d01c982574ee258e110babdbeb` +definition provenance and records each changed row field in `source.repairMetadata`. +Queries, case limits, models and the other feature predicates remain unchanged. +Original failed measurements stay historical. Trusted selection, evidence transfer +and publishing use the workflow's immutable master revision. Each result records +both revisions. A pass qualifies that candidate measurement, not the current master app. Other lanes sharing a case retain their own sources. This integration does not merge the broad catalog branch or its core fixes, reset earlier campaign ordinals, or diff --git a/tests/runner-e2e/chat-full-catalog-testops-manifest.json b/tests/runner-e2e/chat-full-catalog-testops-manifest.json index f5da21569f1..ff93faaee37 100644 --- a/tests/runner-e2e/chat-full-catalog-testops-manifest.json +++ b/tests/runner-e2e/chat-full-catalog-testops-manifest.json @@ -1,6 +1,6 @@ { "schema": "paperclip.chat-full-catalog-testops-manifest/v1", - "targetSha": "3b66c3c144360ddf32714a0742fd774b8ceafb16", + "targetSha": "9ab0f951a5b5990c266ef4456610e0a0775aa334", "metadataSourceSha": "de0941835cc8f0d01c982574ee258e110babdbeb", "profile": "runner-opencode", "model": "openrouter/deepseek/deepseek-v4.1-flash-1017", @@ -167,7 +167,100 @@ "snapshots/fixtures.json", "snapshots/api-state.json", "blob-report/*.zip" - ] + ], + "repairMetadata": { + "baseTargetSha": "3b66c3c144360ddf32714a0742fd774b8ceafb16", + "definitionSourceSha": "de0941835cc8f0d01c982574ee258e110babdbeb", + "changedFields": [ + { + "key": "companies.agent-observability", + "field": "sourceFiles", + "previous": [ + "chat-catalog-domains/identity-observability.ts", + "chat-catalog-domains/identity-private-browser.ts", + "chat-catalog-domains/identity-private-browser-worker.ts", + "chat-catalog-domains/identity-wizard.ts", + "chat-catalog-domains/identity-support.ts", + "chat-catalog-domains/internal-support.ts", + "chat-scenarios/companies.yaml" + ], + "current": [ + "chat-catalog-domains/identity-observability.ts", + "chat-catalog-domains/identity-company-observer.ts", + "chat-catalog-domains/identity-private-browser.ts", + "chat-catalog-domains/identity-private-browser-worker.ts", + "chat-catalog-domains/identity-wizard.ts", + "chat-catalog-domains/identity-support.ts", + "chat-catalog-domains/internal-support.ts", + "chat-scenarios/companies.yaml" + ] + }, + { + "key": "companies.configure-decision-model", + "field": "sourceFiles", + "previous": [ + "chat-catalog-domains/identity-company-models.ts", + "chat-catalog-domains/identity-support.ts", + "chat-catalog-domains/internal-support.ts", + "chat-scenarios/companies.yaml" + ], + "current": [ + "chat-catalog-domains/identity-company-models.ts", + "chat-catalog-domains/identity-company-observer.ts", + "chat-catalog-domains/identity-support.ts", + "chat-catalog-domains/internal-support.ts", + "chat-scenarios/companies.yaml" + ] + }, + { + "key": "companies.fast-responses", + "field": "sourceFiles", + "previous": [ + "chat-catalog-domains/identity-company-models.ts", + "chat-catalog-domains/identity-support.ts", + "chat-catalog-domains/internal-support.ts", + "chat-scenarios/companies.yaml" + ], + "current": [ + "chat-catalog-domains/identity-company-models.ts", + "chat-catalog-domains/identity-company-observer.ts", + "chat-catalog-domains/identity-support.ts", + "chat-catalog-domains/internal-support.ts", + "chat-scenarios/companies.yaml" + ] + }, + { + "key": "companies.prompt-injection-guard", + "field": "sourceFiles", + "previous": [ + "chat-catalog-domains/identity-company-models.ts", + "chat-catalog-domains/identity-support.ts", + "chat-catalog-domains/internal-support.ts", + "chat-scenarios/companies.yaml" + ], + "current": [ + "chat-catalog-domains/identity-company-models.ts", + "chat-catalog-domains/identity-company-observer.ts", + "chat-catalog-domains/identity-support.ts", + "chat-catalog-domains/internal-support.ts", + "chat-scenarios/companies.yaml" + ] + }, + { + "key": "secrets.configure-company-secret", + "field": "oracleVersion", + "previous": "paperclip.chat-catalog.external.v1", + "current": "paperclip.chat-catalog.company-secret-controller-baseline.v2" + } + ], + "conditions": [ + "Initialize owned SDK evidence staging before retention.", + "Use real public scoped membership only for three incidental human fixtures.", + "Company settings observation uses the original setup-inclusive deadline; no larger case cap.", + "Company secret setup and v2 oracle permit only exact immutable controller-bootstrap secret references." + ], + "originalMeasurements": "Preserved. These are new candidate conditions, not retroactive passes." + } }, "cases": [ { @@ -3828,6 +3921,7 @@ "oracleVersion": "paperclip.chat-catalog.identity.v1", "sourceFiles": [ "chat-catalog-domains/identity-observability.ts", + "chat-catalog-domains/identity-company-observer.ts", "chat-catalog-domains/identity-private-browser.ts", "chat-catalog-domains/identity-private-browser-worker.ts", "chat-catalog-domains/identity-wizard.ts", @@ -3917,6 +4011,7 @@ "oracleVersion": "paperclip.chat-catalog.identity.v1", "sourceFiles": [ "chat-catalog-domains/identity-company-models.ts", + "chat-catalog-domains/identity-company-observer.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-support.ts", "chat-scenarios/companies.yaml" @@ -4003,6 +4098,7 @@ "oracleVersion": "paperclip.chat-catalog.identity.v1", "sourceFiles": [ "chat-catalog-domains/identity-company-models.ts", + "chat-catalog-domains/identity-company-observer.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-support.ts", "chat-scenarios/companies.yaml" @@ -4090,6 +4186,7 @@ "oracleVersion": "paperclip.chat-catalog.identity.v1", "sourceFiles": [ "chat-catalog-domains/identity-company-models.ts", + "chat-catalog-domains/identity-company-observer.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-support.ts", "chat-scenarios/companies.yaml" @@ -9469,7 +9566,7 @@ "connection-setup/credentials" ], "checks": null, - "oracleVersion": "paperclip.chat-catalog.external.v1", + "oracleVersion": "paperclip.chat-catalog.company-secret-controller-baseline.v2", "sourceFiles": [ "chat-catalog-domains/external-company-secret.ts", "chat-catalog-domains/external-private-browser.ts", diff --git a/tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs b/tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs index a5108f97094..afcd3ec2e09 100644 --- a/tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs +++ b/tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs @@ -94,3 +94,25 @@ test('foreign identity, duplicated case and changed run-bound metadata are rejec const bounds = copy(); bounds.cases[0].minimumRuns = bounds.cases[0].maximumRuns + 1; assert.throws(() => validateFullCatalogManifest(bounds), /row_invalid/); }); + + +test('repair source metadata preserves original definitions and declares scoped observer and oracle changes', () => { + const repair = manifest.source.repairMetadata; + assert.equal(repair.baseTargetSha, '3b66c3c144360ddf32714a0742fd774b8ceafb16'); + assert.equal(manifest.targetSha, '9ab0f951a5b5990c266ef4456610e0a0775aa334'); + assert.equal(repair.definitionSourceSha, manifest.metadataSourceSha); + assert.deepEqual(repair.changedFields.map(value => [value.key, value.field]), [ + ['companies.agent-observability', 'sourceFiles'], ['companies.configure-decision-model', 'sourceFiles'], + ['companies.fast-responses', 'sourceFiles'], ['companies.prompt-injection-guard', 'sourceFiles'], + ['secrets.configure-company-secret', 'oracleVersion'], + ]); + for (const change of repair.changedFields) { + assert.deepEqual(row(change.key)[change.field], change.current); + if (change.field === 'sourceFiles') { + assert.deepEqual(change.current.filter(value => value !== 'chat-catalog-domains/identity-company-observer.ts'), change.previous); + } else { + assert.equal(change.previous, 'paperclip.chat-catalog.external.v1'); + assert.equal(change.current, 'paperclip.chat-catalog.company-secret-controller-baseline.v2'); + } + } +});