diff --git a/tests/runner-e2e/CHAT-FEATURES.md b/tests/runner-e2e/CHAT-FEATURES.md index f2f40c3481f..7432163eeba 100644 --- a/tests/runner-e2e/CHAT-FEATURES.md +++ b/tests/runner-e2e/CHAT-FEATURES.md @@ -217,7 +217,7 @@ overlap. Exact requirements and limitations appear in each case's preconditions. A declared credential does not establish whether its runtime reference exists. This lane runs the app and fixtures from candidate -`43c44ccc823a8c972b247fa7159281887e6f5255`. It retains the evaluated app from +`51ef450754370c8a8f2cbe4bd18e1536bbc93966`. It retains the evaluated app from `3b66c3c144360ddf32714a0742fd774b8ceafb16`. The first repair source added: owned SDK staging, public membership for three incidental human fixtures, company settings observation inside its original case deadline, and exact @@ -239,8 +239,8 @@ change the previous 196-of-244 acceptance gate. Native execution uses OpenCode and `openrouter/deepseek/deepseek-v4.1-flash-1017` on the existing AWS fleet. Each case -keeps its original query, oracle, run counts, clocks, budgets and zero automatic -retries. Browser boundaries and the external assistant retain their original +uses the manifest-declared query, oracle, counts, clocks and budgets with zero +automatic retries; explicit reviewed condition changes are recorded below. Browser boundaries and the external assistant retain their original surfaces and may have zero native work runs. The external companion keeps its own declared provider requirement. Setup inference and unpriced usage are not free. @@ -304,12 +304,21 @@ The manifest keeps the original definition source and values, with explicit scop Workflow, model, case clocks, native-run bounds, credentials, protected reporting and case identities do not change. The full harness TypeScript graph has not been qualified by local pure tests; source import/admission and the existing credential-free remote Runner build remain required before native queries. Static TestOps descriptions may still show the original definition until a separate idempotent metadata reconciliation at a safe source boundary. -### Repair source 3 (2026-11-03; candidate only) +### Historical repair source 3 (2026-11-03; candidate only) -The current target is the canonical revision above. It adds the reviewed skill import guidance, version-aware Studio sample checks, scoped app review/test and team-auth prerequisites, and the synthetic announcement-dismissal input. The core guidance PR remains unmerged; candidate-owned catalogs carry that explicit runtime/tool-description change. App testing declares its changed setup context and outcome. The exact human query is preserved. +Historical target `43c44ccc823a8c972b247fa7159281887e6f5255` added the reviewed skill import guidance, version-aware Studio sample checks, scoped app review/test and team-auth prerequisites, and the synthetic announcement-dismissal input. The core guidance PR remains unmerged; candidate-owned catalogs carry that explicit runtime/tool-description change. App testing declares its changed setup context and outcome. The exact human query is preserved. Studio test 144 alone changes native accounting from minimum/maximum 2/2 to 2/3. Acceptance requires all three genuine settled turns: initial CEO, actual Studio worker, and exactly one source-bound completion delivery. The original 540-second case clock, controller, one human request and all other case limits remain fixed. The manifest records each previous/current field against the original definition source; no old grade or catalog pass gate is rewritten. The SDK public-input registry is generated from the committed candidate tree. Compiler, parser, Zod, lock runtime identities, scanner and canary build certificates are unchanged. This does not establish the original SDK detector or a plugin pass. Plugin 558 remains held until genuine canary build evidence is reauthenticated. The public announcement fixture is one reviewed synthetic 443-byte JSON under an isolated staging prefix; production announcements and the global index are untouched. Each original worker process still selects one case, so only the dismissal case enables that input. The full-catalog private evidence policy binds the launcher digest from the same reviewed manifest as the immutable candidate target. Every final GO reads `launch.ts` from that exact Git object and validates its digest against every selected singleton policy before submission. The legacy pilot launcher digest remains separately pinned. + + +### Repair source 4 (2026-11-03; candidate only) + +The current target is the canonical revision above. This source adds remaining-original-deadline observation for primary guidance; independently expected normalized MCP results; fixed owned timeline-file reads through fixture board authority; finite supported CLI profile guidance with no identifier substitutions and fresh board/API/company/human readback; real completion-turn observation for plan review; and the reviewed true-price hire counterpart. Original failed attempts remain unchanged. + +The app-review journey now requires inspection of the exact full redacted arguments and audit hash in the normal shared review card, then verifies that authorized provider dispatch equals that proposal. The 17-line product disclosure is an explicit candidate-only overlay from unmerged core PR #18734; no backend permission or approval rule changes. The independent core guidance PR #18717 also remains unmerged. + +SDK public-source eligibility explicitly changes its expected restored lock from stale c7f230 to the authenticated credential-free build-input SHA256 `54a81b2ed3f1f85eaa085c034560ac248e384db34fbc7aba8c5a3846c29b3ce8`. Compiler/parser/Zod identities and all privacy/AST/custody guards remain enforced. This correction is not a plugin pass: case 115 must qualify with its original 300-second/one-native limits, and the other five SDK cases, including plugin 558's canary certificates, remain held. All case clocks and the controller model remain unchanged; Studio144's earlier explicit 2..3 accounting revision remains the only native-bound change. diff --git a/tests/runner-e2e/chat-full-catalog-testops-manifest.json b/tests/runner-e2e/chat-full-catalog-testops-manifest.json index 62ba66c3de5..1c03270a466 100644 --- a/tests/runner-e2e/chat-full-catalog-testops-manifest.json +++ b/tests/runner-e2e/chat-full-catalog-testops-manifest.json @@ -1,6 +1,6 @@ { "schema": "paperclip.chat-full-catalog-testops-manifest/v1", - "targetSha": "43c44ccc823a8c972b247fa7159281887e6f5255", + "targetSha": "51ef450754370c8a8f2cbe4bd18e1536bbc93966", "metadataSourceSha": "de0941835cc8f0d01c982574ee258e110babdbeb", "profile": "runner-opencode", "model": "openrouter/deepseek/deepseek-v4.1-flash-1017", @@ -170,9 +170,9 @@ ], "repairMetadata": { "baseTargetSha": "3b66c3c144360ddf32714a0742fd774b8ceafb16", - "previousTargetSha": "36dff2eab3dfb6de6efb4fd2ce6c1a58f8a5f383", + "previousTargetSha": "43c44ccc823a8c972b247fa7159281887e6f5255", "definitionSourceSha": "de0941835cc8f0d01c982574ee258e110babdbeb", - "candidateDescriptorSha256": "04205b930fc0f8c93dee5d99f9a9531ae9799067e9c9b67e28ed8f694a43d403", + "candidateDescriptorSha256": "54760e1dca998c2ac2b6546f6daf5a13f399d2e9b125a465810b8fadf66c35d8", "changedDefinitionFiles": [ { "path": "tests/runner-e2e/chat-scenarios/companies.yaml", @@ -191,7 +191,7 @@ "sha256": "18fbb9419bbdacb46eacdaa62dfa99feb9024faf2d1eef829aa70d43952575a9" } ], - "conditions": "New candidate measurements only. Original 244 definitions, attempt history and old pass gate remain immutable. SDK audit custody remains diagnostic; scan rejection is unchanged. Explicit field overrides below retain original definition values. 144 alone declares reviewedmin2/max3 accounting and requires actual3 settled source-bound turns. All original clocks/models/human counts and all other nativebounds remain. No workflow/credential/reporting authority expansion.", + "conditions": "New candidate measurements only; original definitions and all failed attempts remain historical. Source4 repairs remaining original observation, normalized-result and owned fixture authorization/profile boundaries, exact full-proposal approval evidence and optional real completion observation. Product487 disclosure is a reviewed candidate-only overlay from UNMERGED18734. SDK privacy/AST guards are unchanged; expected restored lock now binds authenticated BUILD-input54a81 rather than stale c7f230, explicitly a new runtime eligibility condition. Only144 retains its earlier reviewed2..3 native-accounting revision with actual3 accepted; all original clocks/model/human counts and all other native bounds stay fixed. No core merge, workflow/credential/permission expansion or regrade", "changedFields": [ { "key": "agents.choose-primary", @@ -252,6 +252,12 @@ "chat-scenarios/app-permissions.yaml" ] }, + { + "key": "app-permissions.ask-before-action", + "field": "oracleVersion", + "previous": "paperclip.chat-catalog.external.v1", + "current": "paperclip.chat-catalog.app-review.v2" + }, { "key": "app-permissions.test-as-agent", "field": "sourceFiles", @@ -286,7 +292,7 @@ "key": "app-permissions.test-as-agent", "field": "oracleVersion", "previous": "paperclip.chat-catalog.external.v1", - "current": "paperclip.chat-catalog.app-test.v2" + "current": "paperclip.chat-catalog.app-test.v3" }, { "key": "app-permissions.test-as-agent", @@ -300,6 +306,34 @@ "previous": "The agent routes to the supported test-as-agent control with the intended actor and action identified. The result reports actual effective access and provider outcome. A successful board test is described as that test, not proof that Morgan's next runtime has the tool admitted or that the CEO can impersonate Morgan.", "current": "The canonical native request performs one supported, authorized test of the selected connection's sample-page read as Morgan and reports the actual result. Persisted source and target provenance plus the independent provider read prove the outcome. Existing caller and target grants remain unchanged; the test does not start Morgan's runtime, widen permissions, use another credential or create unrelated work. The successful test is not proof that Morgan's next runtime has the tool admitted." }, + { + "key": "cli-operations.choose-cli-context", + "field": "sourceFiles", + "previous": [ + "chat-catalog-domains/privileged-support.ts", + "chat-catalog-domains/privileged-cli.ts", + "chat-scenarios/cli-operations.yaml", + "chat-catalog-domains/privileged-cli-support.ts", + "../../cli/src/index.ts" + ], + "current": [ + "chat-catalog-domains/privileged-support.ts", + "chat-catalog-domains/privileged-cli.ts", + "chat-scenarios/cli-operations.yaml", + "chat-catalog-domains/privileged-cli-support.ts", + "../../cli/src/index.ts", + "../../cli/src/commands/client/context.ts", + "../../cli/src/commands/client/company.ts", + "../../cli/src/commands/client/auth.ts", + "../../cli/src/client/context.ts" + ] + }, + { + "key": "cli-operations.choose-cli-context", + "field": "oracleVersion", + "previous": "paperclip.chat-catalog.privileged.v1", + "current": "paperclip.chat-catalog.cli-context.v2" + }, { "key": "companies.agent-observability", "field": "sourceFiles", @@ -433,12 +467,64 @@ "previous": "paperclip.chat-catalog.identity.v1", "current": "paperclip.chat-catalog.answer-delivery.v2" }, + { + "key": "questions-and-approvals.formal-hire-revision", + "field": "oracleVersion", + "previous": "paperclip.chat-catalog.hire-revision.native1017.v2", + "current": "paperclip.chat-catalog.hire-revision.native1017.v3" + }, { "key": "questions-and-approvals.multi-part-question", "field": "oracleVersion", "previous": "paperclip.chat-catalog.identity.v1", "current": "paperclip.chat-catalog.required-question.v2" }, + { + "key": "questions-and-approvals.reject-and-revise", + "field": "sourceFiles", + "previous": [ + "chat-catalog-domains/identity-plan-review.ts", + "chat-catalog-domains/identity-support.ts", + "chat-catalog-domains/internal-support.ts", + "chat-scenarios/questions-and-approvals.yaml" + ], + "current": [ + "chat-catalog-domains/identity-plan-review.ts", + "chat-catalog-domains/identity-support.ts", + "chat-catalog-domains/internal-support.ts", + "chat-scenarios/questions-and-approvals.yaml", + "../../server/src/services/chat-completion-delivery.ts" + ] + }, + { + "key": "questions-and-approvals.reject-and-revise", + "field": "oracleVersion", + "previous": "paperclip.chat-catalog.identity.v1", + "current": "paperclip.chat-catalog.plan-review.v2" + }, + { + "key": "questions-and-approvals.review-plan", + "field": "sourceFiles", + "previous": [ + "chat-catalog-domains/identity-plan-review.ts", + "chat-catalog-domains/identity-support.ts", + "chat-catalog-domains/internal-support.ts", + "chat-scenarios/questions-and-approvals.yaml" + ], + "current": [ + "chat-catalog-domains/identity-plan-review.ts", + "chat-catalog-domains/identity-support.ts", + "chat-catalog-domains/internal-support.ts", + "chat-scenarios/questions-and-approvals.yaml", + "../../server/src/services/chat-completion-delivery.ts" + ] + }, + { + "key": "questions-and-approvals.review-plan", + "field": "oracleVersion", + "previous": "paperclip.chat-catalog.identity.v1", + "current": "paperclip.chat-catalog.plan-review.v2" + }, { "key": "secrets.configure-company-secret", "field": "oracleVersion", @@ -710,10 +796,10 @@ ] }, { - "key": "app-permissions.test-as-agent", + "key": "app-permissions.ask-before-action", "field": "oracleVersion", "previous": "paperclip.chat-catalog.external.v1", - "current": "paperclip.chat-catalog.app-test.v2" + "current": "paperclip.chat-catalog.app-review.v2" }, { "key": "app-permissions.test-as-agent", @@ -745,6 +831,40 @@ "chat-scenarios/app-permissions.yaml" ] }, + { + "key": "app-permissions.test-as-agent", + "field": "oracleVersion", + "previous": "paperclip.chat-catalog.external.v1", + "current": "paperclip.chat-catalog.app-test.v3" + }, + { + "key": "cli-operations.choose-cli-context", + "field": "sourceFiles", + "previous": [ + "chat-catalog-domains/privileged-support.ts", + "chat-catalog-domains/privileged-cli.ts", + "chat-scenarios/cli-operations.yaml", + "chat-catalog-domains/privileged-cli-support.ts", + "../../cli/src/index.ts" + ], + "current": [ + "chat-catalog-domains/privileged-support.ts", + "chat-catalog-domains/privileged-cli.ts", + "chat-scenarios/cli-operations.yaml", + "chat-catalog-domains/privileged-cli-support.ts", + "../../cli/src/index.ts", + "../../cli/src/commands/client/context.ts", + "../../cli/src/commands/client/company.ts", + "../../cli/src/commands/client/auth.ts", + "../../cli/src/client/context.ts" + ] + }, + { + "key": "cli-operations.choose-cli-context", + "field": "oracleVersion", + "previous": "paperclip.chat-catalog.privileged.v1", + "current": "paperclip.chat-catalog.cli-context.v2" + }, { "key": "companies.agent-observability", "field": "sourceFiles", @@ -825,12 +945,6 @@ "previous": "paperclip.chat-catalog.internal.v1", "current": "paperclip.chat-catalog.goal-hierarchy.v2" }, - { - "key": "navigation-preferences.dismiss-announcement", - "field": "oracleVersion", - "previous": "paperclip.chat-catalog.identity.v1", - "current": "paperclip.chat-catalog.announcement-dismissal.v2" - }, { "key": "navigation-preferences.dismiss-announcement", "field": "sourceFiles", @@ -860,18 +974,76 @@ "../../server/src/routes/announcements.ts" ] }, + { + "key": "navigation-preferences.dismiss-announcement", + "field": "oracleVersion", + "previous": "paperclip.chat-catalog.identity.v1", + "current": "paperclip.chat-catalog.announcement-dismissal.v2" + }, { "key": "questions-and-approvals.duplicate-answer", "field": "oracleVersion", "previous": "paperclip.chat-catalog.identity.v1", "current": "paperclip.chat-catalog.answer-delivery.v2" }, + { + "key": "questions-and-approvals.formal-hire-revision", + "field": "oracleVersion", + "previous": "paperclip.chat-catalog.hire-revision.native1017.v2", + "current": "paperclip.chat-catalog.hire-revision.native1017.v3" + }, { "key": "questions-and-approvals.multi-part-question", "field": "oracleVersion", "previous": "paperclip.chat-catalog.identity.v1", "current": "paperclip.chat-catalog.required-question.v2" }, + { + "key": "questions-and-approvals.reject-and-revise", + "field": "sourceFiles", + "previous": [ + "chat-catalog-domains/identity-plan-review.ts", + "chat-catalog-domains/identity-support.ts", + "chat-catalog-domains/internal-support.ts", + "chat-scenarios/questions-and-approvals.yaml" + ], + "current": [ + "chat-catalog-domains/identity-plan-review.ts", + "chat-catalog-domains/identity-support.ts", + "chat-catalog-domains/internal-support.ts", + "chat-scenarios/questions-and-approvals.yaml", + "../../server/src/services/chat-completion-delivery.ts" + ] + }, + { + "key": "questions-and-approvals.reject-and-revise", + "field": "oracleVersion", + "previous": "paperclip.chat-catalog.identity.v1", + "current": "paperclip.chat-catalog.plan-review.v2" + }, + { + "key": "questions-and-approvals.review-plan", + "field": "sourceFiles", + "previous": [ + "chat-catalog-domains/identity-plan-review.ts", + "chat-catalog-domains/identity-support.ts", + "chat-catalog-domains/internal-support.ts", + "chat-scenarios/questions-and-approvals.yaml" + ], + "current": [ + "chat-catalog-domains/identity-plan-review.ts", + "chat-catalog-domains/identity-support.ts", + "chat-catalog-domains/internal-support.ts", + "chat-scenarios/questions-and-approvals.yaml", + "../../server/src/services/chat-completion-delivery.ts" + ] + }, + { + "key": "questions-and-approvals.review-plan", + "field": "oracleVersion", + "previous": "paperclip.chat-catalog.identity.v1", + "current": "paperclip.chat-catalog.plan-review.v2" + }, { "key": "secrets.configure-company-secret", "field": "oracleVersion", @@ -889,12 +1061,6 @@ ] } }, - { - "key": "skills.author-skill", - "field": "oracleVersion", - "previous": "paperclip.chat-catalog.integrator.v1", - "current": "paperclip.chat-catalog.skill-authoring.v2" - }, { "key": "skills.author-skill", "field": "sourceFiles", @@ -913,6 +1079,12 @@ "chat-catalog-domains/identity-support.ts" ] }, + { + "key": "skills.author-skill", + "field": "oracleVersion", + "previous": "paperclip.chat-catalog.integrator.v1", + "current": "paperclip.chat-catalog.skill-authoring.v2" + }, { "key": "skills.enable-for-agent", "field": "sourceFiles", @@ -1021,24 +1193,6 @@ "chat-scenarios/skills.yaml" ] }, - { - "key": "skills.studio-test", - "field": "expectedRunCount", - "previous": 2, - "current": 3 - }, - { - "key": "skills.studio-test", - "field": "minimumRunCount", - "previous": null, - "current": 2 - }, - { - "key": "skills.studio-test", - "field": "oracleVersion", - "previous": "paperclip.chat-catalog.integrator.v1", - "current": "paperclip.chat-catalog.skill-studio.v2" - }, { "key": "skills.studio-test", "field": "sourceFiles", @@ -1059,6 +1213,24 @@ "chat-scenarios/skills.yaml" ] }, + { + "key": "skills.studio-test", + "field": "oracleVersion", + "previous": "paperclip.chat-catalog.integrator.v1", + "current": "paperclip.chat-catalog.skill-studio.v2" + }, + { + "key": "skills.studio-test", + "field": "expectedRunCount", + "previous": 2, + "current": 3 + }, + { + "key": "skills.studio-test", + "field": "minimumRunCount", + "previous": null, + "current": 2 + }, { "key": "steering.stop-versus-pause", "field": "setupPolicy", @@ -1103,12 +1275,6 @@ ] } }, - { - "key": "teams.configure-team", - "field": "oracleVersion", - "previous": "paperclip.chat-catalog.external.v1", - "current": "paperclip.chat-catalog.team-configure.v2" - }, { "key": "teams.configure-team", "field": "sourceFiles", @@ -1134,6 +1300,12 @@ "chat-flow.ts", "chat-scenarios/teams.yaml" ] + }, + { + "key": "teams.configure-team", + "field": "oracleVersion", + "previous": "paperclip.chat-catalog.external.v1", + "current": "paperclip.chat-catalog.team-configure.v2" } ], "runtimeConditions": { @@ -1163,18 +1335,27 @@ "condition": "Reviewed synthetic staging input only; per-worker singleton550 enables it; production feed and global index unchanged." }, "sdkPublicProof": { - "previousRegistrySha256": "4330369fd34812273db11762c82ce4c427bae2353991eae354e7923e229b4a4a", - "registrySha256": "018b5b15450e84f63920575de3b5903f7838678ef66a1e17aa050e0b21b40319", + "previousRegistrySha256": "018b5b15450e84f63920575de3b5903f7838678ef66a1e17aa050e0b21b40319", + "registrySha256": "32c7971e1d9c9d31cd6791045b0265f0475c8f10932ff6dd34681681b6a89342", "inputCount": 506, - "runtimeCompilerParserZodLockIdentitiesUnchanged": true, "scannerAndCanaryCertificatesUnchanged": true, - "actual115DetectorUnknown": true, - "plugin558StillHeld": true + "plugin558StillHeld": true, + "runtimeCompilerParserZodIdentitiesUnchanged": true, + "previousExpectedLockSha256": "c7f23021335b60da0cff1dd4c75cf8b948333ef3165d7d1703eb79a589edb798", + "expectedLockSha256": "54a81b2ed3f1f85eaa085c034560ac248e384db34fbc7aba8c5a3846c29b3ce8", + "historical43Finding": "detector3 unresolved_context/publicSource null; owning proof rejected stale expected lock", + "newCandidate115Unqualified": true + }, + "toolProposal487": { + "corePr": 18734, + "coreHead": "17236e1736bbecdf04eeeee87ccb794c72d03da1", + "corePRUnmerged": true, + "condition": "Candidate-only exact shared-card bounded redacted full-argument/audit disclosure; no backend authority change. Actual human review/provider dispatch joined by oracle v2." } }, "candidateLauncherSnapshot": { "path": "fixtures/chat-repair-candidate-launcher.txt", - "targetSha": "43c44ccc823a8c972b247fa7159281887e6f5255", + "targetSha": "51ef450754370c8a8f2cbe4bd18e1536bbc93966", "sourcePath": "tests/runner-e2e/launch.ts", "gitBlobOid": "dba1467e211611f37937e9e1fc4c0c5be4329cbc" } @@ -2792,7 +2973,7 @@ "questions-and-approvals/tool-review" ], "checks": null, - "oracleVersion": "paperclip.chat-catalog.external.v1", + "oracleVersion": "paperclip.chat-catalog.app-review.v2", "sourceFiles": [ "chat-catalog-domains/external-app-review.ts", "chat-catalog-domains/external-provider.ts", @@ -3057,7 +3238,7 @@ "app-permissions/test-action" ], "checks": null, - "oracleVersion": "paperclip.chat-catalog.app-test.v2", + "oracleVersion": "paperclip.chat-catalog.app-test.v3", "sourceFiles": [ "chat-catalog-domains/external-app-test.ts", "chat-catalog-domains/external-provider.ts", @@ -4569,13 +4750,17 @@ "cli-operations/context" ], "checks": null, - "oracleVersion": "paperclip.chat-catalog.privileged.v1", + "oracleVersion": "paperclip.chat-catalog.cli-context.v2", "sourceFiles": [ "chat-catalog-domains/privileged-support.ts", "chat-catalog-domains/privileged-cli.ts", "chat-scenarios/cli-operations.yaml", "chat-catalog-domains/privileged-cli-support.ts", - "../../cli/src/index.ts" + "../../cli/src/index.ts", + "../../cli/src/commands/client/context.ts", + "../../cli/src/commands/client/company.ts", + "../../cli/src/commands/client/auth.ts", + "../../cli/src/client/context.ts" ], "screenshot": "final-state.png", "evidenceKind": "private-final-state-png", @@ -9230,7 +9415,7 @@ "agents/hire" ], "checks": null, - "oracleVersion": "paperclip.chat-catalog.hire-revision.native1017.v2", + "oracleVersion": "paperclip.chat-catalog.hire-revision.native1017.v3", "sourceFiles": [ "chat-catalog-domains/identity-hire-revision.ts", "chat-catalog-domains/identity-hire-model-choice.ts", @@ -9358,12 +9543,13 @@ "questions-and-approvals/confirmations" ], "checks": null, - "oracleVersion": "paperclip.chat-catalog.identity.v1", + "oracleVersion": "paperclip.chat-catalog.plan-review.v2", "sourceFiles": [ "chat-catalog-domains/identity-plan-review.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-support.ts", - "chat-scenarios/questions-and-approvals.yaml" + "chat-scenarios/questions-and-approvals.yaml", + "../../server/src/services/chat-completion-delivery.ts" ], "screenshot": "final-state.png", "evidenceKind": "private-final-state-png", @@ -9402,12 +9588,13 @@ "agent-chat/handoff" ], "checks": null, - "oracleVersion": "paperclip.chat-catalog.identity.v1", + "oracleVersion": "paperclip.chat-catalog.plan-review.v2", "sourceFiles": [ "chat-catalog-domains/identity-plan-review.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-support.ts", - "chat-scenarios/questions-and-approvals.yaml" + "chat-scenarios/questions-and-approvals.yaml", + "../../server/src/services/chat-completion-delivery.ts" ], "screenshot": "final-state.png", "evidenceKind": "private-final-state-png", diff --git a/tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs b/tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs index e484af13dd5..f8601a1762e 100644 --- a/tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs +++ b/tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs @@ -102,8 +102,8 @@ test('foreign identity, duplicated case and changed run-bound metadata are rejec test('repair source metadata binds the corrected candidate and explicit changed definitions', () => { const repair = manifest.source.repairMetadata; assert.equal(repair.baseTargetSha, '3b66c3c144360ddf32714a0742fd774b8ceafb16'); - assert.equal(repair.previousTargetSha, '36dff2eab3dfb6de6efb4fd2ce6c1a58f8a5f383'); - assert.equal(manifest.targetSha, '43c44ccc823a8c972b247fa7159281887e6f5255'); + assert.equal(repair.previousTargetSha, '43c44ccc823a8c972b247fa7159281887e6f5255'); + assert.equal(manifest.targetSha, '51ef450754370c8a8f2cbe4bd18e1536bbc93966'); assert.equal(repair.definitionSourceSha, manifest.metadataSourceSha); assert.equal(repair.candidateDescriptorSnapshot, 'fixtures/chat-repair-candidate-descriptors.json'); const descriptorBytes = readFileSync(new URL(repair.candidateDescriptorSnapshot, import.meta.url)); @@ -120,6 +120,10 @@ test('repair source metadata binds the corrected candidate and explicit changed "app-permissions.ask-before-action", "sourceFiles" ], + [ + "app-permissions.ask-before-action", + "oracleVersion" + ], [ "app-permissions.test-as-agent", "sourceFiles" @@ -136,6 +140,14 @@ test('repair source metadata binds the corrected candidate and explicit changed "app-permissions.test-as-agent", "expectedOutcome" ], + [ + "cli-operations.choose-cli-context", + "sourceFiles" + ], + [ + "cli-operations.choose-cli-context", + "oracleVersion" + ], [ "companies.agent-observability", "sourceFiles" @@ -176,10 +188,30 @@ test('repair source metadata binds the corrected candidate and explicit changed "questions-and-approvals.duplicate-answer", "oracleVersion" ], + [ + "questions-and-approvals.formal-hire-revision", + "oracleVersion" + ], [ "questions-and-approvals.multi-part-question", "oracleVersion" ], + [ + "questions-and-approvals.reject-and-revise", + "sourceFiles" + ], + [ + "questions-and-approvals.reject-and-revise", + "oracleVersion" + ], + [ + "questions-and-approvals.review-plan", + "sourceFiles" + ], + [ + "questions-and-approvals.review-plan", + "oracleVersion" + ], [ "secrets.configure-company-secret", "oracleVersion" @@ -241,8 +273,8 @@ test('repair source metadata binds the corrected candidate and explicit changed assert.deepEqual(row(change.key)[change.field], change.current); assert.notDeepEqual(change.current, change.previous); } - assert.equal(repair.descriptorChanges.length, 30); - assert.equal(new Set(repair.descriptorChanges.map(change => change.key)).size, 23); + assert.equal(repair.descriptorChanges.length, 38); + assert.equal(new Set(repair.descriptorChanges.map(change => change.key)).size, 27); assert.deepEqual([...new Set(repair.descriptorChanges.map(change => change.field))].sort(), ['expectedRunCount', 'minimumRunCount', 'oracleVersion', 'setupPolicy', 'sourceFiles']); for (const change of repair.descriptorChanges) { assert.notDeepEqual(change.current, change.previous); @@ -269,4 +301,7 @@ test('repair source metadata binds the corrected candidate and explicit changed assert.equal(repair.runtimeConditions.toolGuidance106.corePRUnmerged, true); assert.equal(repair.runtimeConditions.sdkPublicProof.scannerAndCanaryCertificatesUnchanged, true); assert.equal(repair.runtimeConditions.sdkPublicProof.plugin558StillHeld, true); + assert.equal(repair.runtimeConditions.sdkPublicProof.expectedLockSha256, "54a81b2ed3f1f85eaa085c034560ac248e384db34fbc7aba8c5a3846c29b3ce8"); + assert.equal(repair.runtimeConditions.sdkPublicProof.runtimeCompilerParserZodIdentitiesUnchanged, true); + assert.equal(repair.runtimeConditions.toolProposal487.corePRUnmerged, true); }); diff --git a/tests/runner-e2e/fixtures/chat-repair-candidate-descriptors.json b/tests/runner-e2e/fixtures/chat-repair-candidate-descriptors.json index bb0282c7a5a..ca33fc8ec86 100644 --- a/tests/runner-e2e/fixtures/chat-repair-candidate-descriptors.json +++ b/tests/runner-e2e/fixtures/chat-repair-candidate-descriptors.json @@ -822,7 +822,7 @@ "../../ui/src/components/task-chat/TaskChatComposer.tsx", "chat-scenarios/app-permissions.yaml" ], - "oracleVersion": "paperclip.chat-catalog.external.v1", + "oracleVersion": "paperclip.chat-catalog.app-review.v2", "expectedRunCount": 6, "expectedHumanMessageCount": 3, "minimumRunCount": 6, @@ -956,7 +956,7 @@ "../../server/src/services/tool-approval-execution.ts", "chat-scenarios/app-permissions.yaml" ], - "oracleVersion": "paperclip.chat-catalog.app-test.v2", + "oracleVersion": "paperclip.chat-catalog.app-test.v3", "expectedRunCount": 1, "turnTimeoutMs": 120000, "attemptTimeoutMs": 300000, @@ -1823,9 +1823,13 @@ "chat-catalog-domains/privileged-cli.ts", "chat-scenarios/cli-operations.yaml", "chat-catalog-domains/privileged-cli-support.ts", - "../../cli/src/index.ts" + "../../cli/src/index.ts", + "../../cli/src/commands/client/context.ts", + "../../cli/src/commands/client/company.ts", + "../../cli/src/commands/client/auth.ts", + "../../cli/src/client/context.ts" ], - "oracleVersion": "paperclip.chat-catalog.privileged.v1", + "oracleVersion": "paperclip.chat-catalog.cli-context.v2", "coverageGaps": [], "expectedRunCount": 1, "turnTimeoutMs": 120000, @@ -4194,7 +4198,7 @@ "chat-catalog-domains/internal-support.ts", "chat-scenarios/questions-and-approvals.yaml" ], - "oracleVersion": "paperclip.chat-catalog.hire-revision.native1017.v2", + "oracleVersion": "paperclip.chat-catalog.hire-revision.native1017.v3", "expectedRunCount": 1, "turnTimeoutMs": 120000, "attemptTimeoutMs": 300000, @@ -4239,9 +4243,10 @@ "chat-catalog-domains/identity-plan-review.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-support.ts", - "chat-scenarios/questions-and-approvals.yaml" + "chat-scenarios/questions-and-approvals.yaml", + "../../server/src/services/chat-completion-delivery.ts" ], - "oracleVersion": "paperclip.chat-catalog.identity.v1", + "oracleVersion": "paperclip.chat-catalog.plan-review.v2", "expectedRunCount": 3, "minimumRunCount": 1, "turnTimeoutMs": 120000, @@ -4260,9 +4265,10 @@ "chat-catalog-domains/identity-plan-review.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-support.ts", - "chat-scenarios/questions-and-approvals.yaml" + "chat-scenarios/questions-and-approvals.yaml", + "../../server/src/services/chat-completion-delivery.ts" ], - "oracleVersion": "paperclip.chat-catalog.identity.v1", + "oracleVersion": "paperclip.chat-catalog.plan-review.v2", "expectedRunCount": 3, "minimumRunCount": 2, "turnTimeoutMs": 120000,