From e968131efa7e63d321f3bcca8d47a62513c2efb4 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 4 Nov 2026 10:31:23 -0600 Subject: [PATCH] test(evals): pin reviewed outcome and native handoff repairs (#18976) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - The catalog must measure the original human requests through ordinary product controls. > - The latest four-case run exposed two oracle false negatives and a real suppressed native handoff. > - Preserve those original failures and fix only their demonstrated causes. > - Keep the reviewed company-selector successor and add the accepted timeline, approved-replay and personal-completion corrections. > - Select native standing-chat terminal answers by authority while preserving governed waits and ordinary-task behavior. > - Bind the resulting immutable candidate and its three changed oracle versions through the existing manifest and descriptor snapshot. > - The next personal-secret attempt must prove the complete secure-entry and worker journey before company selection or a final ten-case regression. ## Linked Issues or Issue Description Refs #18971, #18967, #18968, #18972, #18977. Follows #18974. The latest four-case run retained original failures for timeline, approved tool replay and personal-secret setup. Team installation independently passed. The personal native run had actionable final directions, but presentation selected an earlier ordinary comment and suppressed that answer. This pin makes the reviewed corrections available; it does not regrade those attempts or claim a new native pass. ## What Changed - Pin immutable candidate `5f5e2ca8f18063d051c3d948105f0049562fe4ce` on `codex/chat-catalog-four-outcome-repairs-20261104`. - Refresh the existing descriptor snapshot and matching manifest metadata: timeline v5, app-review v5 and personal-secret v4. Company v3 is retained. Preserve the prior reviewed company recipient correction. - Include the reviewed structural standing-chat response selection and finished-turn recovery, preserving earlier comment history, governed interaction ownership and no-final fallback. The superseded prose classifier is absent. - Preserve all 244 identities, original queries, provider/model, human and native counts, clocks, launcher bytes, private evidence limits and qualification holds. The existing SDK registry/compiler proof is unchanged. - The infrastructure change is three metadata files. No product fix is merged by this PR; its owning core PR remains separate. ## Verification - Exact final candidate: 510 owning tests passed across 19 suites, including catalog contract/consistency and lazy declaration loading. The candidate declares 242 cases plus two existing legacy tasks; no fixture callbacks execute during admission. - Candidate response summary suite: 96 tests passed. The owner separately reports 103 summary/notice controls and 14 recovery PostgreSQL controls; these are distinct from candidate results. - Candidate PostgreSQL scope/recovery controls: four passed (three native/ordinary/legacy caller scopes and the new server-finished yielded unknown-channel recovery). The focused filter left 52 unrelated controls unselected. Earlier setup attempts failed before assertions due missing native library links and plugin-sdk output; package-cwd shipped hydration and one owning SDK compiler build enabled this final actual run. No source workaround, timeout change or acceptance weakening was used. - Two deliberate baseline joins preserve the frozen candidate: the caller test adds only its required `issueComments` import; recovery adds only current-conversation matching yielded result/terminal with no owned interaction, retaining all existing succeeded/completed/session/ownership/authorization guards. The test adds only `statusDecisions` and needed fixture options. Fresh-master ordinary `needs_review` behavior and unrelated imports were not copied. - All 131 existing pin, TestOps contract/API, adapter, reconciliation and streaming controls passed. The descriptor snapshot exactly matches the candidate declaration bytes. Diff whitespace checks passed. - SDK source guard: 506 inputs, `changed: false`, `runtimePinsChanged: false`, registry `efa70621e9fb42aa185f7ee616a1a726aa3eccb367e67e28817cedf26ef2ade8`. The local compiler invocation supplied missing ignored test output; it is not a renewed SDK qualification or runtime certificate. - Existing reporting-lock guard returned `already-reviewed`. Fresh protected master `86814c6550f52e99ede150a883bdc773cbb540e3` has admitted lock `60f4c06bca47af3ab9d74281f2ef2e1ff559d566b862ba51c2fbe6c8e5bac35b`. Repeat the existing serving-master guard after merge and before a new Submit. - The earlier pin run exposed a pre-existing remote-failure-retention fixture failure. Normal protected merge #18984 repaired that baseline on master `04a24f4e06da68043378bdf3646f50292efd5316`; its 47 real jobs and aggregates succeeded despite delayed GitHub UI/API status projection. This branch normally merges that actual master. Its three pin metadata files are byte-identical to prior reviewed head `dec78737cc520ce6d489fb8bfccafb460b79e870`, with the same immutable candidate, descriptor bytes and model/count/clock/provenance fields. No baseline fixture patch was duplicated in this PR. - On updated head `bd5402d4688e690ffd9f1f80081577733ceeda25`, the existing 131 controls pass, reporting lock is already reviewed, and existing `selectedCases` admits exactly case 580 without execution. Fresh actual master lock remains admitted. - Exact-head normal CI run 41865757115 has completed its full actual job set: 48 succeeded and one legitimate selector skip, including `ci / e2e` and `ci / verify`. RunsOn completion records and exact GitHub aggregate logs corroborate the results. Windows run 41865757064 completed SUCCESS; its exact log reports all executed Rust test results as passing. GitHub UI/API/check-rollup projections still contain stale queued/in-progress rows, as independently observed during #18984; this is not a claim that those displayed rows are green. A normal protected merge must be accepted by GitHub without override. - Fresh exact-head Greptile is 5/5, the current Paperclip assessment explicitly says 5/5 and Ship, all review threads are resolved (zero threads), and the security scan succeeded. Earlier company-only checks are not used to qualify this head. Full repository build/typecheck/test were not repeated locally for this metadata PR; owning source checks and ordinary hosted CI provide the relevant gates. - No model run, dispatch, retry, cancellation, secret mutation or merge was performed by this update. ## Risks The structural correction changes final-answer selection only for admitted native standing conversations; it still needs a new real secure-entry journey. Oracle corrections can admit previously rejected wording, so retained positive and negative controls remain required. Original failed and unknown attempts are immutable. Company selection remains gated on actual personal-secret secure-entry/worker proof, and all 244 cases remain unqualified as a whole. ## Model Used OpenAI `gpt-6.1-sol` through Codex. Capabilities used: source analysis, isolated integration and local checks. The host does not expose the context-window limit. The evaluated model remains OpenCode/OpenRouter `deepseek/deepseek-v4.1-flash-1017`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes — manifest and expected-pin assertion agree - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green — actual job/aggregate results passed; stale UI/API projection is disclosed above - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../chat-full-catalog-testops-manifest.json | 24 +++++++++---------- ...hat-full-catalog-testops-manifest.test.mjs | 2 +- .../chat-repair-candidate-descriptors.json | 6 ++--- 3 files changed, 16 insertions(+), 16 deletions(-) diff --git a/tests/runner-e2e/chat-full-catalog-testops-manifest.json b/tests/runner-e2e/chat-full-catalog-testops-manifest.json index 1b4adcba600..1cf3ddcf7c0 100644 --- a/tests/runner-e2e/chat-full-catalog-testops-manifest.json +++ b/tests/runner-e2e/chat-full-catalog-testops-manifest.json @@ -1,6 +1,6 @@ { "schema": "paperclip.chat-full-catalog-testops-manifest/v1", - "targetSha": "e757e447371c4781a3eff3a0ea9dc9c4eceb060a", + "targetSha": "5f5e2ca8f18063d051c3d948105f0049562fe4ce", "metadataSourceSha": "de0941835cc8f0d01c982574ee258e110babdbeb", "profile": "runner-opencode", "model": "openrouter/deepseek/deepseek-v4.1-flash-1017", @@ -172,7 +172,7 @@ "baseTargetSha": "3b66c3c144360ddf32714a0742fd774b8ceafb16", "previousTargetSha": "ab2019e976076ec0eb60b026844f3d9211cb0208", "definitionSourceSha": "de0941835cc8f0d01c982574ee258e110babdbeb", - "candidateDescriptorSha256": "e7681aa6bb41a8b71f6e30ae3279e0579c72f55dabdaeedbf984e22268b070db", + "candidateDescriptorSha256": "640743a65be417330144b66858704e1e378fe2edf26a7977910d1756fde9b692", "changedDefinitionFiles": [ { "path": "tests/runner-e2e/chat-scenarios/companies.yaml", @@ -232,7 +232,7 @@ "key": "activity.work-timeline", "field": "oracleVersion", "previous": "paperclip.chat-catalog.internal.v1", - "current": "paperclip.chat-catalog.work-timeline.v4" + "current": "paperclip.chat-catalog.work-timeline.v5" }, { "key": "activity.work-timeline", @@ -308,7 +308,7 @@ "key": "app-permissions.ask-before-action", "field": "oracleVersion", "previous": "paperclip.chat-catalog.external.v1", - "current": "paperclip.chat-catalog.app-review.v4" + "current": "paperclip.chat-catalog.app-review.v5" }, { "key": "app-permissions.test-as-agent", @@ -694,7 +694,7 @@ "key": "secrets.personal-secret", "field": "oracleVersion", "previous": "paperclip.chat-catalog.external.v1", - "current": "paperclip.chat-catalog.personal-secret.v3" + "current": "paperclip.chat-catalog.personal-secret.v4" }, { "key": "skills.author-skill", @@ -938,7 +938,7 @@ "key": "activity.work-timeline", "field": "oracleVersion", "previous": "paperclip.chat-catalog.internal.v1", - "current": "paperclip.chat-catalog.work-timeline.v4" + "current": "paperclip.chat-catalog.work-timeline.v5" }, { "key": "agents.choose-primary", @@ -1008,7 +1008,7 @@ "key": "app-permissions.ask-before-action", "field": "oracleVersion", "previous": "paperclip.chat-catalog.external.v1", - "current": "paperclip.chat-catalog.app-review.v4" + "current": "paperclip.chat-catalog.app-review.v5" }, { "key": "app-permissions.test-as-agent", @@ -1381,7 +1381,7 @@ "key": "secrets.personal-secret", "field": "oracleVersion", "previous": "paperclip.chat-catalog.external.v1", - "current": "paperclip.chat-catalog.personal-secret.v3" + "current": "paperclip.chat-catalog.personal-secret.v4" }, { "key": "skills.author-skill", @@ -1745,7 +1745,7 @@ }, "candidateLauncherSnapshot": { "path": "fixtures/chat-repair-candidate-launcher.txt", - "targetSha": "e757e447371c4781a3eff3a0ea9dc9c4eceb060a", + "targetSha": "5f5e2ca8f18063d051c3d948105f0049562fe4ce", "sourcePath": "tests/runner-e2e/launch.ts", "gitBlobOid": "dba1467e211611f37937e9e1fc4c0c5be4329cbc" } @@ -2288,7 +2288,7 @@ "activity/timeline" ], "checks": null, - "oracleVersion": "paperclip.chat-catalog.work-timeline.v4", + "oracleVersion": "paperclip.chat-catalog.work-timeline.v5", "sourceFiles": [ "chat-catalog-domains/internal-definition.ts", "chat-catalog-domains/internal-support.ts", @@ -3366,7 +3366,7 @@ "questions-and-approvals/tool-review" ], "checks": null, - "oracleVersion": "paperclip.chat-catalog.app-review.v4", + "oracleVersion": "paperclip.chat-catalog.app-review.v5", "sourceFiles": [ "chat-catalog-domains/external-app-review.ts", "chat-catalog-domains/external-app-review-outcomes.ts", @@ -11196,7 +11196,7 @@ "secrets/user-secrets" ], "checks": null, - "oracleVersion": "paperclip.chat-catalog.personal-secret.v3", + "oracleVersion": "paperclip.chat-catalog.personal-secret.v4", "sourceFiles": [ "chat-catalog-domains/external-personal-secret.ts", "chat-catalog-domains/external-secret-question.ts", diff --git a/tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs b/tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs index e7bf1bbb496..4f5a2615d6d 100644 --- a/tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs +++ b/tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs @@ -103,7 +103,7 @@ test('repair source metadata binds the corrected candidate and explicit changed const repair = manifest.source.repairMetadata; assert.equal(repair.baseTargetSha, '3b66c3c144360ddf32714a0742fd774b8ceafb16'); assert.equal(repair.previousTargetSha, 'ab2019e976076ec0eb60b026844f3d9211cb0208'); - assert.equal(manifest.targetSha, 'e757e447371c4781a3eff3a0ea9dc9c4eceb060a'); + assert.equal(manifest.targetSha, '5f5e2ca8f18063d051c3d948105f0049562fe4ce'); assert.equal(repair.definitionSourceSha, manifest.metadataSourceSha); assert.equal(repair.candidateDescriptorSnapshot, 'fixtures/chat-repair-candidate-descriptors.json'); const descriptorBytes = readFileSync(new URL(repair.candidateDescriptorSnapshot, import.meta.url)); diff --git a/tests/runner-e2e/fixtures/chat-repair-candidate-descriptors.json b/tests/runner-e2e/fixtures/chat-repair-candidate-descriptors.json index 88d201c16bb..114c4bcc91e 100644 --- a/tests/runner-e2e/fixtures/chat-repair-candidate-descriptors.json +++ b/tests/runner-e2e/fixtures/chat-repair-candidate-descriptors.json @@ -258,7 +258,7 @@ "../../server/src/services/native-runtime/status-decision-committer.ts", "../../server/src/services/work-timeline.ts" ], - "oracleVersion": "paperclip.chat-catalog.work-timeline.v4", + "oracleVersion": "paperclip.chat-catalog.work-timeline.v5", "coverageGaps": [], "expectedRunCount": 3, "minimumRunCount": 3, @@ -830,7 +830,7 @@ "../../ui/src/components/task-chat/TaskChatComposer.tsx", "chat-scenarios/app-permissions.yaml" ], - "oracleVersion": "paperclip.chat-catalog.app-review.v4", + "oracleVersion": "paperclip.chat-catalog.app-review.v5", "expectedRunCount": 6, "expectedHumanMessageCount": 3, "minimumRunCount": 6, @@ -4981,7 +4981,7 @@ "../../ui/src/pages/secrets/SetMyUserSecretDialog.tsx", "chat-scenarios/secrets.yaml" ], - "oracleVersion": "paperclip.chat-catalog.personal-secret.v3", + "oracleVersion": "paperclip.chat-catalog.personal-secret.v4", "expectedRunCount": 3, "expectedHumanMessageCount": 2, "minimumRunCount": 3,