mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 01:50:13 +01:00
test(evals): pin coherent chat feature follow-ups (#19411)
## Thinking Path > - Paperclip manages agent work and its permission boundaries. > - The chat catalog runs feature checks from an immutable source. > - The last full-ten attempt had five reported passes, four failures and one missing original. > - Local evidence identified narrow assertion, setup and evidence-retention repairs. > - This PR updates the trusted source pin and its exact descriptor snapshot. > - The next five-case measurement can test the affected features under the same model, clocks and native cap. ## Linked Issues or Issue Description Refs: #19395, #19405, #19093, #19084, #19112. The previous source remains preserved. This PR selects reviewed source `f5692c57dafaf9afbef479c25fcd277815932516` on `codex/chat-full-ten-coherent-qualified-f5692c57-20261106`, tree `b9e302d2ea7b605a60125dec8c81672eac7a3c57`. The full catalog and all core PRs remain unmerged. Core #19405 is included only as a source overlay. ## What Changed - Update the source target, exact descriptor snapshot and the matching pin checks. Preserve historical conditions, frozen source anchors and the prior query clarification. - Record Company ACK handling and Company/Personal negation corrections, plus the skill-update preview negation correction. A denied or unapplied action still fails when the agent claims it completed. - Disclose the Team setup change: a disposable catalog package selects the approved OpenCode runtime and same-company account. Other file bytes and non-runtime sidecar fields stay unchanged. Original and effective package identities and hashes are recorded. A real PostgreSQL install without caller overrides proves the saved runtime. This gives no end-to-end runtime-migration or override-handling credit. - Include defensive failed-turn evidence retirement and generic, schema-valid human-input examples. The old case 53 missing-original ENOENT cause is not diagnosed. The examples are a usability hypothesis, not a schema repair or native-success claim. All 244 YAML definition fields (query, assumed context and expected outcome) remain byte-identical versus source 936c. The Team derived runtime fixture and task setup context change is disclosed above. Native/human bounds, clocks, DeepSeek 1017, credentials, SDK506/efa, launcher, private interfaces, native cap two and zero automatic retries remain unchanged. Only descriptor `sourceFiles` and oracle versions change. The 196/244 gate stays unchanged. ## Verification Pin-local checks: ```sh node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs tests/runner-e2e/chat-full-catalog-testops-adapter.test.mjs tests/runner-e2e/chat-completion-testops-contract.test.mjs ``` - All 49 pin/admission controls pass locally. They join all 244 identities, historical query overrides, native bounds and the exact 242-row descriptor snapshot. - Source qualification: 447 controls across 32 suites, full E2E TypeScript checks and normal contract generators pass. The separate 21 core contract controls are carried-forward component evidence; they are not added to 447. Local source checks used Node 26.4.0. AWS runtime settings are unchanged. - The selected source descriptor SHA256 is `24fe88af167724c6fb120d3984296aab47c993a65782158e03d80a936fbd94ed`. - The actual source launcher is byte-identical to 936c: SHA256 `2b9a6953ee51f0ea0dbbd475b0491d9a6c39a1d6835eeaa912e0dbb7cf987724`, Git blob `c88df15f509b2ac8638af826647a97d45f16bd1f`. - All 506 registered SDK inputs and the ten trusted helper bytes are unchanged. Five required source-helper joins pass. No SDK renewal or native build is part of this PR. - The current controller's 67 dependency inputs match the qualified frozen reporting graph. No reporting-lock, package, permission or workflow change is included. - Exact head `b833036fb477b55b05cbdf1f4feb2e2c5387b420` is green: CI run 42166216108 attempt 2 succeeded, Windows succeeded, and both current reviews are 5/5 with zero unresolved threads. The check rollup has 56 successes, three configured skips and one neutral result. New representative cases 53, 578, 580, 585 and 610 are unrun. A later uniform-ten regression is separate. - CI attempt 1 remains retained: one cloud-auth browser fixture read a trailing health response after context teardown began, after all task-recovery assertions had passed. One authorized rerun of only that failed job and its dependent checks succeeded on the unchanged head. This does not fix or qualify fixture reliability; no eval or model run was retried. The previous full-ten outcome remains five reported passes, four failures and one missing-original/reporting gap. Case 585 reached only its first preview before the false rejection; its update phase is unqualified. Case 610's old noncompletion and runtime-override journey remain unresolved. All original failures and evidence remain retained. No old result is regraded or replaced. ## Risks The local repairs may not produce a native pass. The new Team setup tests installation with an already selected runtime package; it does not qualify runtime migration. Failed-turn retirement has local regression coverage but does not establish the cause of the old case 53 reporting gap. The core guidance overlay remains unmerged and its native benefit is unproved. Normal protected merge and a new authenticated measurement are still required. ## Model Used OpenAI `gpt-6.1-sol` through Codex. It used code inspection, local test execution and GitHub tooling. The context window size is not exposed. No model inference or provider call was made for this PR's local checks. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
66d3fa97a9
commit
b918ee165d
3 files changed
+210
-41
No files matched your search
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"schema": "paperclip.chat-full-catalog-testops-manifest/v1",
|
||||
"targetSha": "936c3bbe0e83ca15616254f23d9023b2065fc5fb",
|
||||
"targetSha": "f5692c57dafaf9afbef479c25fcd277815932516",
|
||||
"metadataSourceSha": "de0941835cc8f0d01c982574ee258e110babdbeb",
|
||||
"profile": "runner-opencode",
|
||||
"model": "openrouter/deepseek/deepseek-v4.1-flash-1017",
|
||||
@@ -172,7 +172,7 @@
|
||||
"baseTargetSha": "3b66c3c144360ddf32714a0742fd774b8ceafb16",
|
||||
"previousTargetSha": "ab2019e976076ec0eb60b026844f3d9211cb0208",
|
||||
"definitionSourceSha": "de0941835cc8f0d01c982574ee258e110babdbeb",
|
||||
"candidateDescriptorSha256": "19185aa5298c93d159c3d8547661fd6ef22185e74be1c18b8497bae2ef8051c7",
|
||||
"candidateDescriptorSha256": "24fe88af167724c6fb120d3984296aab47c993a65782158e03d80a936fbd94ed",
|
||||
"changedDefinitionFiles": [
|
||||
{
|
||||
"path": "tests/runner-e2e/chat-scenarios/companies.yaml",
|
||||
@@ -199,7 +199,7 @@
|
||||
"sha256": "91a9ddc4f5357f19a2cdc73ff8a090c741d14780e4ed502c856ebd557986f7e1"
|
||||
}
|
||||
],
|
||||
"conditions": "Historical qualification conditions (retained; statements that all requests or sourceFiles remain fixed refer to those earlier successors): New candidate measurements only. Original definitions, prior failed attempts and source6 ab201 remain historical. This target retains that catalog and integrates merged runtime fixes #18711, #18713 and #18718, with the prior-run restoration condition adapted to its existing owner. It adds the reviewed previous-question locator, attributed human-only denial and conditional-outcome oracle, and actor-sensitive timeline narrative/recorded-reassignment fixes. Closed finish diagnostics retain an effective schema digest, hashed call correlation and bounded type/category enums only. Historical MCP timeout and continuation-shape causes remain unresolved; no new all-ten PASS is claimed. Queries, model, human/native counts and original clocks remain fixed; the earlier Studio144 explicit 2..3 accounting stays. Core PRs18717/18734/18742 are now merged. SDK115 completion remains UNKNOWN and the other five SDK cases remain held. Full244 runtime catalog is not merged to master and the196/244 gate is unchanged. This successor binds the seven reviewed scenario repairs plus framework evidence: canonical skill navigation, original announcement phase clock, attributed timeline context, bound hire revision history, credential warning and configuration-question admission, completed team visibility, and canonical private framework evidence. Twelve declaration fields across eleven cases change only oracle versions and skill source-file provenance. All original requests, setup, human/native bounds, model and clocks remain fixed. Historical failed attempts and missing-original UNKNOWN grades remain unchanged; no uniform-ten PASS is claimed. The existing SDK506/efa qualification is reused only for byte-identical registered source inputs. This next successor adds reviewed question-surface observation, grounded scenario grading and safe credential-guidance boundaries after the retained uniform attempt. Eleven existing oracle-version leaves advance; all sourceFiles declarations, requests, setup, counts, clocks, model, launcher and private protocol helpers remain fixed. Component qualification does not attest a new uniform-ten pass.\n\nPrevious successor conditions (retained): This successor measures five reviewed repairs at the retained original model and bounds. Exactly one request is clarified: multipart53 now explicitly asks to walk through the visible form; its original broad request and failures4623/4717 remain historical. The previous inventory0cf530 is retained and the disclosed clarified inventory is814bb363. Other243 catalog requests are unchanged. The company-secret second human message now references the actual saved non-secret metadata instead of repeating the initial request; native continuation must remain attributed to the actual agent. Five oracle-version leaves and four source-file additions change; launcher, private helpers, SDK506/efa, identities, counts, clocks and the196/244 gate remain fixed. All five successor journeys remain unrun; component proof is not a PASS. For Team610, independently saved canonical skill/version/member proof establishes recovery; no required recovery-success phrase is imposed. A contradictory current denial still fails. If the required skill remains missing, truthful disclosure of that condition remains required.\n\nPrevious retained successor conditions: This target adds the reviewed personal secure-entry confirmation handoff, the typographic team partial-warning observer correction, and the company-secret authorization fix from unmerged core PR #19325. Personal v10 retains two declared logical human actions: one canonical chat request plus one independently verified same-human secure-save confirmation handoff. Raw chat comments remain one. Team v5 accepts the actual named-skill warning with a typographic apostrophe; saved-state, repeat, runtime, link, and false-success checks remain required. The company route returns scoped 403 guidance before malformed-body validation and preserves board-only creation. Its effect on the unfinished native turn is unproven. The three affected journeys are unrun on this target. Existing failed attempts remain failed. All queries, expected outcomes, native and human bounds, clocks, profile, model, launcher, private protocol, SDK506/efa, and the 196/244 admission gate remain fixed, including the recorded case53 query clarification. Local component proof is not native feature qualification or main/core merge credit.\n\nPrevious retained successor conditions: This target repairs personal secure-entry observation and truthful human acknowledgment, company secure-entry language and ordinary visible menu navigation, team current-skill claim attribution, and native completion feedback that asks for usable product links. Personal v11 expands visible Details before entry when needed, binds the exact value-enLine truncated
|
||||
"conditions": "Historical qualification conditions (retained; statements that all requests or sourceFiles remain fixed refer to those earlier successors): New candidate measurements only. Original definitions, prior failed attempts and source6 ab201 remain historical. This target retains that catalog and integrates merged runtime fixes #18711, #18713 and #18718, with the prior-run restoration condition adapted to its existing owner. It adds the reviewed previous-question locator, attributed human-only denial and conditional-outcome oracle, and actor-sensitive timeline narrative/recorded-reassignment fixes. Closed finish diagnostics retain an effective schema digest, hashed call correlation and bounded type/category enums only. Historical MCP timeout and continuation-shape causes remain unresolved; no new all-ten PASS is claimed. Queries, model, human/native counts and original clocks remain fixed; the earlier Studio144 explicit 2..3 accounting stays. Core PRs18717/18734/18742 are now merged. SDK115 completion remains UNKNOWN and the other five SDK cases remain held. Full244 runtime catalog is not merged to master and the196/244 gate is unchanged. This successor binds the seven reviewed scenario repairs plus framework evidence: canonical skill navigation, original announcement phase clock, attributed timeline context, bound hire revision history, credential warning and configuration-question admission, completed team visibility, and canonical private framework evidence. Twelve declaration fields across eleven cases change only oracle versions and skill source-file provenance. All original requests, setup, human/native bounds, model and clocks remain fixed. Historical failed attempts and missing-original UNKNOWN grades remain unchanged; no uniform-ten PASS is claimed. The existing SDK506/efa qualification is reused only for byte-identical registered source inputs. This next successor adds reviewed question-surface observation, grounded scenario grading and safe credential-guidance boundaries after the retained uniform attempt. Eleven existing oracle-version leaves advance; all sourceFiles declarations, requests, setup, counts, clocks, model, launcher and private protocol helpers remain fixed. Component qualification does not attest a new uniform-ten pass.\n\nPrevious successor conditions (retained): This successor measures five reviewed repairs at the retained original model and bounds. Exactly one request is clarified: multipart53 now explicitly asks to walk through the visible form; its original broad request and failures4623/4717 remain historical. The previous inventory0cf530 is retained and the disclosed clarified inventory is814bb363. Other243 catalog requests are unchanged. The company-secret second human message now references the actual saved non-secret metadata instead of repeating the initial request; native continuation must remain attributed to the actual agent. Five oracle-version leaves and four source-file additions change; launcher, private helpers, SDK506/efa, identities, counts, clocks and the196/244 gate remain fixed. All five successor journeys remain unrun; component proof is not a PASS. For Team610, independently saved canonical skill/version/member proof establishes recovery; no required recovery-success phrase is imposed. A contradictory current denial still fails. If the required skill remains missing, truthful disclosure of that condition remains required.\n\nPrevious retained successor conditions: This target adds the reviewed personal secure-entry confirmation handoff, the typographic team partial-warning observer correction, and the company-secret authorization fix from unmerged core PR #19325. Personal v10 retains two declared logical human actions: one canonical chat request plus one independently verified same-human secure-save confirmation handoff. Raw chat comments remain one. Team v5 accepts the actual named-skill warning with a typographic apostrophe; saved-state, repeat, runtime, link, and false-success checks remain required. The company route returns scoped 403 guidance before malformed-body validation and preserves board-only creation. Its effect on the unfinished native turn is unproven. The three affected journeys are unrun on this target. Existing failed attempts remain failed. All queries, expected outcomes, native and human bounds, clocks, profile, model, launcher, private protocol, SDK506/efa, and the 196/244 admission gate remain fixed, including the recorded case53 query clarification. Local component proof is not native feature qualification or main/core merge credit.\n\nPrevious retained successor conditions: This target repairs personal secure-entry observation and truthful human acknowledgment, company secure-entry language and ordinary visible menu navigation, team current-skill claim attribution, and native completion feedback that asks for usable product links. Personal v11 expands visible Details before entry when needed, binds the exact value-enLine truncated
|
||||
"changedFields": [
|
||||
{
|
||||
"key": "activity.work-timeline",
|
||||
@@ -620,7 +620,9 @@
|
||||
"chat-scenarios/secrets.yaml"
|
||||
],
|
||||
"current": [
|
||||
"../../packages/paperclip-runner/src/protocol-actions/request-human-input.ts",
|
||||
"chat-catalog-domains/external-company-secret.ts",
|
||||
"chat-catalog-domains/external-company-secret-confirmation.ts",
|
||||
"chat-catalog-domains/external-secret-proposal.ts",
|
||||
"chat-catalog-domains/external-secret-question.ts",
|
||||
"chat-catalog-domains/external-secret-route.ts",
|
||||
@@ -647,7 +649,7 @@
|
||||
"key": "secrets.configure-company-secret",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.external.v1",
|
||||
"current": "paperclip.chat-catalog.company-secret-controller-baseline.v13"
|
||||
"current": "paperclip.chat-catalog.company-secret-controller-baseline.v14"
|
||||
},
|
||||
{
|
||||
"key": "secrets.personal-secret",
|
||||
@@ -677,6 +679,7 @@
|
||||
"chat-scenarios/secrets.yaml"
|
||||
],
|
||||
"current": [
|
||||
"../../packages/paperclip-runner/src/protocol-actions/request-human-input.ts",
|
||||
"chat-catalog-domains/external-personal-secret.ts",
|
||||
"chat-catalog-domains/external-personal-secret-confirmation.ts",
|
||||
"chat-catalog-contract.ts",
|
||||
@@ -711,7 +714,7 @@
|
||||
"key": "secrets.personal-secret",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.external.v1",
|
||||
"current": "paperclip.chat-catalog.personal-secret.v12"
|
||||
"current": "paperclip.chat-catalog.personal-secret.v13"
|
||||
},
|
||||
{
|
||||
"key": "skills.author-skill",
|
||||
@@ -923,31 +926,31 @@
|
||||
"key": "connection-setup.secure-api-key",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.external.v1",
|
||||
"current": "paperclip.chat-catalog.external.v6"
|
||||
"current": "paperclip.chat-catalog.external.v7"
|
||||
},
|
||||
{
|
||||
"key": "secrets.least-scope-delivery",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.external.v1",
|
||||
"current": "paperclip.chat-catalog.external.v6"
|
||||
"current": "paperclip.chat-catalog.external.v7"
|
||||
},
|
||||
{
|
||||
"key": "secrets.secret-proposal",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.external.v1",
|
||||
"current": "paperclip.chat-catalog.external.v6"
|
||||
"current": "paperclip.chat-catalog.external.v7"
|
||||
},
|
||||
{
|
||||
"key": "secrets.vault-reference",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.external.v1",
|
||||
"current": "paperclip.chat-catalog.external.v6"
|
||||
"current": "paperclip.chat-catalog.external.v7"
|
||||
},
|
||||
{
|
||||
"key": "teams.install-team",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.external.v1",
|
||||
"current": "paperclip.chat-catalog.team-install.v6"
|
||||
"current": "paperclip.chat-catalog.team-install.v7"
|
||||
},
|
||||
{
|
||||
"key": "questions-and-approvals.multi-part-question",
|
||||
@@ -972,6 +975,66 @@
|
||||
"field": "expectedHumanMessageCount",
|
||||
"previous": null,
|
||||
"current": 2
|
||||
},
|
||||
{
|
||||
"key": "questions-and-approvals.multi-part-question",
|
||||
"field": "sourceFiles",
|
||||
"previous": [
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/identity-questions-and-approvals.ts",
|
||||
"chat-scenarios/questions-and-approvals.yaml"
|
||||
],
|
||||
"current": [
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/identity-questions-and-approvals.ts",
|
||||
"chat-scenarios/questions-and-approvals.yaml",
|
||||
"../../packages/paperclip-runner/src/protocol-actions/request-human-input.ts",
|
||||
"chat-catalog-flow.ts",
|
||||
"chat-flow.ts",
|
||||
"cleanup-verification.ts",
|
||||
"runner.spec.ts"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "skills.review-update",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.integrator.v1",
|
||||
"current": "paperclip.chat-catalog.integrator.v2"
|
||||
},
|
||||
{
|
||||
"key": "teams.install-team",
|
||||
"field": "sourceFiles",
|
||||
"previous": [
|
||||
"chat-catalog-domains/external-team-install.ts",
|
||||
"chat-catalog-domains/external-team-conditional.ts",
|
||||
"../../server/src/services/teams-catalog.ts",
|
||||
"../../server/src/services/company-skills.ts",
|
||||
"../../server/src/services/company-portability.ts",
|
||||
"../../packages/teams-catalog/generated/catalog.json",
|
||||
"chat-catalog-domains/external-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-flow.ts",
|
||||
"chat-scenarios/teams.yaml"
|
||||
],
|
||||
"current": [
|
||||
"chat-catalog-domains/external-team-install.ts",
|
||||
"chat-catalog-domains/external-team-conditional.ts",
|
||||
"chat-catalog-domains/external-team-runtime-fixture.ts",
|
||||
"server.ts",
|
||||
"../../packages/teams-catalog/src/catalog-builder.ts",
|
||||
"../../server/src/services/teams-catalog.ts",
|
||||
"../../server/src/services/company-skills.ts",
|
||||
"../../server/src/services/company-portability.ts",
|
||||
"../../packages/teams-catalog/generated/catalog.json",
|
||||
"chat-catalog-domains/external-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-flow.ts",
|
||||
"chat-scenarios/teams.yaml"
|
||||
]
|
||||
}
|
||||
],
|
||||
"candidateDescriptorSnapshot": "fixtures/chat-repair-candidate-descriptors.json",
|
||||
@@ -1365,7 +1428,9 @@
|
||||
"chat-scenarios/secrets.yaml"
|
||||
],
|
||||
"current": [
|
||||
"../../packages/paperclip-runner/src/protocol-actions/request-human-input.ts",
|
||||
"chat-catalog-domains/external-company-secret.ts",
|
||||
"chat-catalog-domains/external-company-secret-confirmation.ts",
|
||||
"chat-catalog-domains/external-secret-proposal.ts",
|
||||
"chat-catalog-domains/external-secret-question.ts",
|
||||
"chat-catalog-domains/external-secret-route.ts",
|
||||
@@ -1392,7 +1457,7 @@
|
||||
"key": "secrets.configure-company-secret",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.external.v1",
|
||||
"current": "paperclip.chat-catalog.company-secret-controller-baseline.v13"
|
||||
"current": "paperclip.chat-catalog.company-secret-controller-baseline.v14"
|
||||
},
|
||||
{
|
||||
"key": "secrets.personal-secret",
|
||||
@@ -1422,6 +1487,7 @@
|
||||
"chat-scenarios/secrets.yaml"
|
||||
],
|
||||
"current": [
|
||||
"../../packages/paperclip-runner/src/protocol-actions/request-human-input.ts",
|
||||
"chat-catalog-domains/external-personal-secret.ts",
|
||||
"chat-catalog-domains/external-personal-secret-confirmation.ts",
|
||||
"chat-catalog-contract.ts",
|
||||
@@ -1467,7 +1533,7 @@
|
||||
"key": "secrets.personal-secret",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.external.v1",
|
||||
"current": "paperclip.chat-catalog.personal-secret.v12"
|
||||
"current": "paperclip.chat-catalog.personal-secret.v13"
|
||||
},
|
||||
{
|
||||
"key": "skills.author-skill",
|
||||
@@ -1723,31 +1789,31 @@
|
||||
"key": "connection-setup.secure-api-key",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.external.v1",
|
||||
"current": "paperclip.chat-catalog.external.v6"
|
||||
"current": "paperclip.chat-catalog.external.v7"
|
||||
},
|
||||
{
|
||||
"key": "secrets.least-scope-delivery",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.external.v1",
|
||||
"current": "paperclip.chat-catalog.external.v6"
|
||||
"current": "paperclip.chat-catalog.external.v7"
|
||||
},
|
||||
{
|
||||
"key": "secrets.secret-proposal",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.external.v1",
|
||||
"current": "paperclip.chat-catalog.external.v6"
|
||||
"current": "paperclip.chat-catalog.external.v7"
|
||||
},
|
||||
{
|
||||
"key": "secrets.vault-reference",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.external.v1",
|
||||
"current": "paperclip.chat-catalog.external.v6"
|
||||
"current": "paperclip.chat-catalog.external.v7"
|
||||
},
|
||||
{
|
||||
"key": "teams.install-team",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.external.v1",
|
||||
"current": "paperclip.chat-catalog.team-install.v6"
|
||||
"current": "paperclip.chat-catalog.team-install.v7"
|
||||
},
|
||||
{
|
||||
"key": "secrets.configure-company-secret",
|
||||
@@ -1766,6 +1832,66 @@
|
||||
"field": "minimumRunCount",
|
||||
"previous": null,
|
||||
"current": 2
|
||||
},
|
||||
{
|
||||
"key": "questions-and-approvals.multi-part-question",
|
||||
"field": "sourceFiles",
|
||||
"previous": [
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/identity-questions-and-approvals.ts",
|
||||
"chat-scenarios/questions-and-approvals.yaml"
|
||||
],
|
||||
"current": [
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/identity-questions-and-approvals.ts",
|
||||
"chat-scenarios/questions-and-approvals.yaml",
|
||||
"../../packages/paperclip-runner/src/protocol-actions/request-human-input.ts",
|
||||
"chat-catalog-flow.ts",
|
||||
"chat-flow.ts",
|
||||
"cleanup-verification.ts",
|
||||
"runner.spec.ts"
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "skills.review-update",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.integrator.v1",
|
||||
"current": "paperclip.chat-catalog.integrator.v2"
|
||||
},
|
||||
{
|
||||
"key": "teams.install-team",
|
||||
"field": "sourceFiles",
|
||||
"previous": [
|
||||
"chat-catalog-domains/external-team-install.ts",
|
||||
"chat-catalog-domains/external-team-conditional.ts",
|
||||
"../../server/src/services/teams-catalog.ts",
|
||||
"../../server/src/services/company-skills.ts",
|
||||
"../../server/src/services/company-portability.ts",
|
||||
"../../packages/teams-catalog/generated/catalog.json",
|
||||
"chat-catalog-domains/external-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-flow.ts",
|
||||
"chat-scenarios/teams.yaml"
|
||||
],
|
||||
"current": [
|
||||
"chat-catalog-domains/external-team-install.ts",
|
||||
"chat-catalog-domains/external-team-conditional.ts",
|
||||
"chat-catalog-domains/external-team-runtime-fixture.ts",
|
||||
"server.ts",
|
||||
"../../packages/teams-catalog/src/catalog-builder.ts",
|
||||
"../../server/src/services/teams-catalog.ts",
|
||||
"../../server/src/services/company-skills.ts",
|
||||
"../../server/src/services/company-portability.ts",
|
||||
"../../packages/teams-catalog/generated/catalog.json",
|
||||
"chat-catalog-domains/external-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-flow.ts",
|
||||
"chat-scenarios/teams.yaml"
|
||||
]
|
||||
}
|
||||
],
|
||||
"runtimeConditions": {
|
||||
@@ -1899,7 +2025,7 @@
|
||||
},
|
||||
"candidateLauncherSnapshot": {
|
||||
"path": "fixtures/chat-repair-candidate-launcher.txt",
|
||||
"targetSha": "936c3bbe0e83ca15616254f23d9023b2065fc5fb",
|
||||
"targetSha": "f5692c57dafaf9afbef479c25fcd277815932516",
|
||||
"sourcePath": "tests/runner-e2e/launch.ts",
|
||||
"gitBlobOid": "c88df15f509b2ac8638af826647a97d45f16bd1f"
|
||||
}
|
||||
@@ -6665,7 +6791,7 @@
|
||||
"connection-setup/credentials"
|
||||
],
|
||||
"checks": null,
|
||||
"oracleVersion": "paperclip.chat-catalog.external.v6",
|
||||
"oracleVersion": "paperclip.chat-catalog.external.v7",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/external-secure-key.ts",
|
||||
"chat-query-attachments.ts",
|
||||
@@ -10019,7 +10145,12 @@
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/identity-questions-and-approvals.ts",
|
||||
"chat-scenarios/questions-and-approvals.yaml"
|
||||
"chat-scenarios/questions-and-approvals.yaml",
|
||||
"../../packages/paperclip-runner/src/protocol-actions/request-human-input.ts",
|
||||
"chat-catalog-flow.ts",
|
||||
"chat-flow.ts",
|
||||
"cleanup-verification.ts",
|
||||
"runner.spec.ts"
|
||||
],
|
||||
"screenshot": "final-state.png",
|
||||
"evidenceKind": "private-final-state-png",
|
||||
@@ -11235,9 +11366,11 @@
|
||||
"connection-setup/credentials"
|
||||
],
|
||||
"checks": null,
|
||||
"oracleVersion": "paperclip.chat-catalog.company-secret-controller-baseline.v13",
|
||||
"oracleVersion": "paperclip.chat-catalog.company-secret-controller-baseline.v14",
|
||||
"sourceFiles": [
|
||||
"../../packages/paperclip-runner/src/protocol-actions/request-human-input.ts",
|
||||
"chat-catalog-domains/external-company-secret.ts",
|
||||
"chat-catalog-domains/external-company-secret-confirmation.ts",
|
||||
"chat-catalog-domains/external-secret-proposal.ts",
|
||||
"chat-catalog-domains/external-secret-question.ts",
|
||||
"chat-catalog-domains/external-secret-route.ts",
|
||||
@@ -11296,7 +11429,7 @@
|
||||
"secrets/runtime-delivery"
|
||||
],
|
||||
"checks": null,
|
||||
"oracleVersion": "paperclip.chat-catalog.external.v6",
|
||||
"oracleVersion": "paperclip.chat-catalog.external.v7",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/external-secret-delivery.ts",
|
||||
"chat-catalog-domains/external-secret-endpoint.ts",
|
||||
@@ -11360,8 +11493,9 @@
|
||||
"secrets/user-secrets"
|
||||
],
|
||||
"checks": null,
|
||||
"oracleVersion": "paperclip.chat-catalog.personal-secret.v12",
|
||||
"oracleVersion": "paperclip.chat-catalog.personal-secret.v13",
|
||||
"sourceFiles": [
|
||||
"../../packages/paperclip-runner/src/protocol-actions/request-human-input.ts",
|
||||
"chat-catalog-domains/external-personal-secret.ts",
|
||||
"chat-catalog-domains/external-personal-secret-confirmation.ts",
|
||||
"chat-catalog-contract.ts",
|
||||
@@ -11426,7 +11560,7 @@
|
||||
"secrets/proposals"
|
||||
],
|
||||
"checks": null,
|
||||
"oracleVersion": "paperclip.chat-catalog.external.v6",
|
||||
"oracleVersion": "paperclip.chat-catalog.external.v7",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/external-secret-proposal.ts",
|
||||
"chat-catalog-domains/external-secret-scan.ts",
|
||||
@@ -11489,7 +11623,7 @@
|
||||
"secrets/vault-import"
|
||||
],
|
||||
"checks": null,
|
||||
"oracleVersion": "paperclip.chat-catalog.external.v6",
|
||||
"oracleVersion": "paperclip.chat-catalog.external.v7",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/external-vault-reference.ts",
|
||||
"chat-catalog-domains/external-fixtures.ts",
|
||||
@@ -11780,7 +11914,7 @@
|
||||
"skills/versions"
|
||||
],
|
||||
"checks": null,
|
||||
"oracleVersion": "paperclip.chat-catalog.integrator.v1",
|
||||
"oracleVersion": "paperclip.chat-catalog.integrator.v2",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-catalog-domains/integrator-skill-versions.ts",
|
||||
@@ -13345,10 +13479,13 @@
|
||||
"agents/hire"
|
||||
],
|
||||
"checks": null,
|
||||
"oracleVersion": "paperclip.chat-catalog.team-install.v6",
|
||||
"oracleVersion": "paperclip.chat-catalog.team-install.v7",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/external-team-install.ts",
|
||||
"chat-catalog-domains/external-team-conditional.ts",
|
||||
"chat-catalog-domains/external-team-runtime-fixture.ts",
|
||||
"server.ts",
|
||||
"../../packages/teams-catalog/src/catalog-builder.ts",
|
||||
"../../server/src/services/teams-catalog.ts",
|
||||
"../../server/src/services/company-skills.ts",
|
||||
"../../server/src/services/company-portability.ts",
|
||||
|
||||
@@ -103,7 +103,7 @@ test('repair source metadata binds the corrected candidate and explicit changed
|
||||
const repair = manifest.source.repairMetadata;
|
||||
assert.equal(repair.baseTargetSha, '3b66c3c144360ddf32714a0742fd774b8ceafb16');
|
||||
assert.equal(repair.previousTargetSha, 'ab2019e976076ec0eb60b026844f3d9211cb0208');
|
||||
assert.equal(manifest.targetSha, '936c3bbe0e83ca15616254f23d9023b2065fc5fb');
|
||||
assert.equal(manifest.targetSha, 'f5692c57dafaf9afbef479c25fcd277815932516');
|
||||
assert.equal(repair.definitionSourceSha, manifest.metadataSourceSha);
|
||||
assert.equal(repair.candidateDescriptorSnapshot, 'fixtures/chat-repair-candidate-descriptors.json');
|
||||
const descriptorBytes = readFileSync(new URL(repair.candidateDescriptorSnapshot, import.meta.url));
|
||||
@@ -320,9 +320,30 @@ test('repair source metadata binds the corrected candidate and explicit changed
|
||||
"questions-and-approvals.multi-part-question",
|
||||
"originalQuery"
|
||||
],
|
||||
['secrets.configure-company-secret', 'maximumRuns'],
|
||||
['secrets.configure-company-secret', 'optionalCompanySecretBindingApproval'],
|
||||
['secrets.configure-company-secret', 'expectedHumanMessageCount']
|
||||
[
|
||||
"secrets.configure-company-secret",
|
||||
"maximumRuns"
|
||||
],
|
||||
[
|
||||
"secrets.configure-company-secret",
|
||||
"optionalCompanySecretBindingApproval"
|
||||
],
|
||||
[
|
||||
"secrets.configure-company-secret",
|
||||
"expectedHumanMessageCount"
|
||||
],
|
||||
[
|
||||
"questions-and-approvals.multi-part-question",
|
||||
"sourceFiles"
|
||||
],
|
||||
[
|
||||
"skills.review-update",
|
||||
"oracleVersion"
|
||||
],
|
||||
[
|
||||
"teams.install-team",
|
||||
"sourceFiles"
|
||||
]
|
||||
]);
|
||||
for (const change of repair.changedFields) {
|
||||
assert.deepEqual(row(change.key)[change.field], change.current);
|
||||
@@ -333,7 +354,7 @@ test('repair source metadata binds the corrected candidate and explicit changed
|
||||
assert.equal(repair.runtimeConditions.multipart53QuestionClarification.unchangedOtherCatalogQueries, 243);
|
||||
assert.equal(repair.runtimeConditions.multipart53QuestionClarification.previousInventoryDigest, '0cf530c11449508e91b5ce664d9375fc8e76466fa908255a1af0ff55169d0220');
|
||||
assert.equal(repair.runtimeConditions.multipart53QuestionClarification.inventoryDigest, '814bb3633268cb0488563aed51fce2f987bd47650ddfb5a3302d0354292c2323');
|
||||
assert.equal(repair.descriptorChanges.length, 52);
|
||||
assert.equal(repair.descriptorChanges.length, 55);
|
||||
assert.equal(new Set(repair.descriptorChanges.map(change => change.key)).size, 33);
|
||||
assert.deepEqual([...new Set(repair.descriptorChanges.map(change => change.field))].sort(), ['expectedRunCount', 'minimumRunCount', 'optionalCompanySecretBindingApproval', 'oracleVersion', 'setupPolicy', 'sourceFiles']);
|
||||
for (const change of repair.descriptorChanges) {
|
||||
|
||||
@@ -2600,7 +2600,7 @@
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-scenarios/connection-setup.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.external.v6",
|
||||
"oracleVersion": "paperclip.chat-catalog.external.v7",
|
||||
"expectedRunCount": 1,
|
||||
"turnTimeoutMs": 120000,
|
||||
"attemptTimeoutMs": 420000,
|
||||
@@ -4224,7 +4224,12 @@
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/identity-questions-and-approvals.ts",
|
||||
"chat-scenarios/questions-and-approvals.yaml"
|
||||
"chat-scenarios/questions-and-approvals.yaml",
|
||||
"../../packages/paperclip-runner/src/protocol-actions/request-human-input.ts",
|
||||
"chat-catalog-flow.ts",
|
||||
"chat-flow.ts",
|
||||
"cleanup-verification.ts",
|
||||
"runner.spec.ts"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.required-question.v4",
|
||||
"expectedRunCount": 2,
|
||||
@@ -4891,7 +4896,9 @@
|
||||
{
|
||||
"scenarioId": "secrets.configure-company-secret",
|
||||
"sourceFiles": [
|
||||
"../../packages/paperclip-runner/src/protocol-actions/request-human-input.ts",
|
||||
"chat-catalog-domains/external-company-secret.ts",
|
||||
"chat-catalog-domains/external-company-secret-confirmation.ts",
|
||||
"chat-catalog-domains/external-secret-proposal.ts",
|
||||
"chat-catalog-domains/external-secret-question.ts",
|
||||
"chat-catalog-domains/external-secret-route.ts",
|
||||
@@ -4913,7 +4920,7 @@
|
||||
"chat-flow.ts",
|
||||
"chat-scenarios/secrets.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.company-secret-controller-baseline.v13",
|
||||
"oracleVersion": "paperclip.chat-catalog.company-secret-controller-baseline.v14",
|
||||
"expectedRunCount": 3,
|
||||
"minimumRunCount": 2,
|
||||
"expectedHumanMessageCount": 2,
|
||||
@@ -4954,7 +4961,7 @@
|
||||
"../../packages/plugins/sdk/dist/index.js",
|
||||
"chat-scenarios/secrets.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.external.v6",
|
||||
"oracleVersion": "paperclip.chat-catalog.external.v7",
|
||||
"expectedRunCount": 3,
|
||||
"expectedHumanMessageCount": 1,
|
||||
"minimumRunCount": 3,
|
||||
@@ -4967,6 +4974,7 @@
|
||||
{
|
||||
"scenarioId": "secrets.personal-secret",
|
||||
"sourceFiles": [
|
||||
"../../packages/paperclip-runner/src/protocol-actions/request-human-input.ts",
|
||||
"chat-catalog-domains/external-personal-secret.ts",
|
||||
"chat-catalog-domains/external-personal-secret-confirmation.ts",
|
||||
"chat-catalog-contract.ts",
|
||||
@@ -4996,7 +5004,7 @@
|
||||
"../../ui/src/pages/secrets/SetMyUserSecretDialog.tsx",
|
||||
"chat-scenarios/secrets.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.personal-secret.v12",
|
||||
"oracleVersion": "paperclip.chat-catalog.personal-secret.v13",
|
||||
"expectedRunCount": 3,
|
||||
"expectedHumanMessageCount": 2,
|
||||
"minimumRunCount": 3,
|
||||
@@ -5028,7 +5036,7 @@
|
||||
"../../server/src/services/secret-proposal-notifications.ts",
|
||||
"chat-scenarios/secrets.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.external.v6",
|
||||
"oracleVersion": "paperclip.chat-catalog.external.v7",
|
||||
"expectedRunCount": 4,
|
||||
"expectedHumanMessageCount": 4,
|
||||
"minimumRunCount": 4,
|
||||
@@ -5076,7 +5084,7 @@
|
||||
"../../ui/src/pages/secrets/ImportFromVaultDialog.tsx",
|
||||
"chat-scenarios/secrets.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.external.v6",
|
||||
"oracleVersion": "paperclip.chat-catalog.external.v7",
|
||||
"expectedRunCount": 3,
|
||||
"expectedHumanMessageCount": 2,
|
||||
"minimumRunCount": 2,
|
||||
@@ -5218,7 +5226,7 @@
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/skills.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.integrator.v1",
|
||||
"oracleVersion": "paperclip.chat-catalog.integrator.v2",
|
||||
"expectedRunCount": 2,
|
||||
"expectedHumanMessageCount": 2,
|
||||
"turnTimeoutMs": 120000,
|
||||
@@ -6020,6 +6028,9 @@
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/external-team-install.ts",
|
||||
"chat-catalog-domains/external-team-conditional.ts",
|
||||
"chat-catalog-domains/external-team-runtime-fixture.ts",
|
||||
"server.ts",
|
||||
"../../packages/teams-catalog/src/catalog-builder.ts",
|
||||
"../../server/src/services/teams-catalog.ts",
|
||||
"../../server/src/services/company-skills.ts",
|
||||
"../../server/src/services/company-portability.ts",
|
||||
@@ -6030,7 +6041,7 @@
|
||||
"chat-flow.ts",
|
||||
"chat-scenarios/teams.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.team-install.v6",
|
||||
"oracleVersion": "paperclip.chat-catalog.team-install.v7",
|
||||
"expectedRunCount": 2,
|
||||
"expectedHumanMessageCount": 2,
|
||||
"turnTimeoutMs": 120000,
|
||||
|
||||
Reference in new issue
Block a user