mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
test(evals): pin company environment and API search repairs (#19392)
## Thinking Path > - Paperclip manages AI agents and their governed actions. > - The TestOps controller runs an immutable product-eval source. > - The last company-secret attempt rejected a safe metadata question before private entry. > - A governed approval also had a redundant server notification and wake path. > - Natural-language API search could hide the resource needed for a timeline answer. > - This pull request pins the reviewed source and its exact descriptor snapshot. > - New native evidence must still prove the affected journeys. ## Linked Issues or Issue Description Refs #19381, #19093, #19387, #19388. This is the next controller pin for those reviewed changes. The core pull requests remain unmerged. ## What Changed - Pin source `2c99c248e839f41b520fc2d7e3702169ac6cb0a2` and its exact descriptor snapshot. - Advance company oracle v12 to v13. Admit bounded environment-name questions without a required authored pattern or exact prompt template. Preserve independent human-answer validation. - Include the reviewed interaction-accept notification and API-search ranking overlays. Record their route and catalog dependencies in the company and timeline provenance. - Retain the existing cap of two, zero retries, DeepSeek1017 model, and all 244 current YAML definitions. Company keeps 480000ms, native2..3, base human2 plus one independently proved approval. Timeline keeps720000ms, native3/human1. ## Verification - Source:396 controls in9 suites,2046 API catalog controls,145 route controls, full runner TypeScript check, and unchanged SDK506 input guard passed. Fresh-master API PR has2059 controls; its other baseline tests are absent from this catalog source. - Controller:133 contract, API, adapter, reconcile, streaming and manifest controls passed. Existing report-lock preparation returned `already-reviewed`. - All current YAML definitions and launcher bytes remain unchanged from the prior source. No model or native eval ran for this pin. - The original company5059 and timeline5060 remain failed. Company private entry, approval and final binding remain unproved on this successor. - Exact-head normal checks are terminal:56 success,3 intentional skips, and one neutral Paperclip review. Greptile and Paperclip both report5/5 with no findings; review threads are empty. The next TestOps/AWS run follows the protected controller merge. ## Risks The new source is not a native feature pass. Its effect on model completion is unproved. The original480000ms company budget may still be too short for three native turns. No budget change or count waiver is included. The full catalog remains outside master and its existing admission gate remains unchanged. The controller reuses byte-identical SDK506 evidence; no compiler or registry renewal is claimed. Actual post-merge helper imports still need qualification against the merged controller lock. The CI virtual merge includes a newer upstream lock than the local frozen dependency proof. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with repository tools and code execution. The host did not report its context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is5/5 with no open findings - [x] I will address reviewer comments before requesting merge Co-authored-by: Paperclip <paperclip@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
4033f7f17f
commit
7c04adb2d6
3 files changed
+25
-13
No files matched your search
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"schema": "paperclip.chat-full-catalog-testops-manifest/v1",
|
||||
"targetSha": "fe547be84826b4611d3ba9ee1a7d7aa0bd0a2843",
|
||||
"targetSha": "2c99c248e839f41b520fc2d7e3702169ac6cb0a2",
|
||||
"metadataSourceSha": "de0941835cc8f0d01c982574ee258e110babdbeb",
|
||||
"profile": "runner-opencode",
|
||||
"model": "openrouter/deepseek/deepseek-v4.1-flash-1017",
|
||||
@@ -172,7 +172,7 @@
|
||||
"baseTargetSha": "3b66c3c144360ddf32714a0742fd774b8ceafb16",
|
||||
"previousTargetSha": "ab2019e976076ec0eb60b026844f3d9211cb0208",
|
||||
"definitionSourceSha": "de0941835cc8f0d01c982574ee258e110babdbeb",
|
||||
"candidateDescriptorSha256": "50e90d8c02ad57f444e27b522efe694db9734d47bdbc3a058475af77bac8b2ca",
|
||||
"candidateDescriptorSha256": "e12d1600a7c99ec94ca0d90cdedee85ad89f6a1d637eeb7cf939dfa36c27f015",
|
||||
"changedDefinitionFiles": [
|
||||
{
|
||||
"path": "tests/runner-e2e/chat-scenarios/companies.yaml",
|
||||
@@ -199,7 +199,7 @@
|
||||
"sha256": "91a9ddc4f5357f19a2cdc73ff8a090c741d14780e4ed502c856ebd557986f7e1"
|
||||
}
|
||||
],
|
||||
"conditions": "Historical qualification conditions (retained; statements that all requests or sourceFiles remain fixed refer to those earlier successors): New candidate measurements only. Original definitions, prior failed attempts and source6 ab201 remain historical. This target retains that catalog and integrates merged runtime fixes #18711, #18713 and #18718, with the prior-run restoration condition adapted to its existing owner. It adds the reviewed previous-question locator, attributed human-only denial and conditional-outcome oracle, and actor-sensitive timeline narrative/recorded-reassignment fixes. Closed finish diagnostics retain an effective schema digest, hashed call correlation and bounded type/category enums only. Historical MCP timeout and continuation-shape causes remain unresolved; no new all-ten PASS is claimed. Queries, model, human/native counts and original clocks remain fixed; the earlier Studio144 explicit 2..3 accounting stays. Core PRs18717/18734/18742 are now merged. SDK115 completion remains UNKNOWN and the other five SDK cases remain held. Full244 runtime catalog is not merged to master and the196/244 gate is unchanged. This successor binds the seven reviewed scenario repairs plus framework evidence: canonical skill navigation, original announcement phase clock, attributed timeline context, bound hire revision history, credential warning and configuration-question admission, completed team visibility, and canonical private framework evidence. Twelve declaration fields across eleven cases change only oracle versions and skill source-file provenance. All original requests, setup, human/native bounds, model and clocks remain fixed. Historical failed attempts and missing-original UNKNOWN grades remain unchanged; no uniform-ten PASS is claimed. The existing SDK506/efa qualification is reused only for byte-identical registered source inputs. This next successor adds reviewed question-surface observation, grounded scenario grading and safe credential-guidance boundaries after the retained uniform attempt. Eleven existing oracle-version leaves advance; all sourceFiles declarations, requests, setup, counts, clocks, model, launcher and private protocol helpers remain fixed. Component qualification does not attest a new uniform-ten pass.\n\nPrevious successor conditions (retained): This successor measures five reviewed repairs at the retained original model and bounds. Exactly one request is clarified: multipart53 now explicitly asks to walk through the visible form; its original broad request and failures4623/4717 remain historical. The previous inventory0cf530 is retained and the disclosed clarified inventory is814bb363. Other243 catalog requests are unchanged. The company-secret second human message now references the actual saved non-secret metadata instead of repeating the initial request; native continuation must remain attributed to the actual agent. Five oracle-version leaves and four source-file additions change; launcher, private helpers, SDK506/efa, identities, counts, clocks and the196/244 gate remain fixed. All five successor journeys remain unrun; component proof is not a PASS. For Team610, independently saved canonical skill/version/member proof establishes recovery; no required recovery-success phrase is imposed. A contradictory current denial still fails. If the required skill remains missing, truthful disclosure of that condition remains required.\n\nPrevious retained successor conditions: This target adds the reviewed personal secure-entry confirmation handoff, the typographic team partial-warning observer correction, and the company-secret authorization fix from unmerged core PR #19325. Personal v10 retains two declared logical human actions: one canonical chat request plus one independently verified same-human secure-save confirmation handoff. Raw chat comments remain one. Team v5 accepts the actual named-skill warning with a typographic apostrophe; saved-state, repeat, runtime, link, and false-success checks remain required. The company route returns scoped 403 guidance before malformed-body validation and preserves board-only creation. Its effect on the unfinished native turn is unproven. The three affected journeys are unrun on this target. Existing failed attempts remain failed. All queries, expected outcomes, native and human bounds, clocks, profile, model, launcher, private protocol, SDK506/efa, and the 196/244 admission gate remain fixed, including the recorded case53 query clarification. Local component proof is not native feature qualification or main/core merge credit.\n\nPrevious retained successor conditions: This target repairs personal secure-entry observation and truthful human acknowledgment, company secure-entry language and ordinary visible menu navigation, team current-skill claim attribution, and native completion feedback that asks for usable product links. Personal v11 expands visible Details before entry when needed, binds the exact value-enLine truncated
|
||||
"conditions": "Historical qualification conditions (retained; statements that all requests or sourceFiles remain fixed refer to those earlier successors): New candidate measurements only. Original definitions, prior failed attempts and source6 ab201 remain historical. This target retains that catalog and integrates merged runtime fixes #18711, #18713 and #18718, with the prior-run restoration condition adapted to its existing owner. It adds the reviewed previous-question locator, attributed human-only denial and conditional-outcome oracle, and actor-sensitive timeline narrative/recorded-reassignment fixes. Closed finish diagnostics retain an effective schema digest, hashed call correlation and bounded type/category enums only. Historical MCP timeout and continuation-shape causes remain unresolved; no new all-ten PASS is claimed. Queries, model, human/native counts and original clocks remain fixed; the earlier Studio144 explicit 2..3 accounting stays. Core PRs18717/18734/18742 are now merged. SDK115 completion remains UNKNOWN and the other five SDK cases remain held. Full244 runtime catalog is not merged to master and the196/244 gate is unchanged. This successor binds the seven reviewed scenario repairs plus framework evidence: canonical skill navigation, original announcement phase clock, attributed timeline context, bound hire revision history, credential warning and configuration-question admission, completed team visibility, and canonical private framework evidence. Twelve declaration fields across eleven cases change only oracle versions and skill source-file provenance. All original requests, setup, human/native bounds, model and clocks remain fixed. Historical failed attempts and missing-original UNKNOWN grades remain unchanged; no uniform-ten PASS is claimed. The existing SDK506/efa qualification is reused only for byte-identical registered source inputs. This next successor adds reviewed question-surface observation, grounded scenario grading and safe credential-guidance boundaries after the retained uniform attempt. Eleven existing oracle-version leaves advance; all sourceFiles declarations, requests, setup, counts, clocks, model, launcher and private protocol helpers remain fixed. Component qualification does not attest a new uniform-ten pass.\n\nPrevious successor conditions (retained): This successor measures five reviewed repairs at the retained original model and bounds. Exactly one request is clarified: multipart53 now explicitly asks to walk through the visible form; its original broad request and failures4623/4717 remain historical. The previous inventory0cf530 is retained and the disclosed clarified inventory is814bb363. Other243 catalog requests are unchanged. The company-secret second human message now references the actual saved non-secret metadata instead of repeating the initial request; native continuation must remain attributed to the actual agent. Five oracle-version leaves and four source-file additions change; launcher, private helpers, SDK506/efa, identities, counts, clocks and the196/244 gate remain fixed. All five successor journeys remain unrun; component proof is not a PASS. For Team610, independently saved canonical skill/version/member proof establishes recovery; no required recovery-success phrase is imposed. A contradictory current denial still fails. If the required skill remains missing, truthful disclosure of that condition remains required.\n\nPrevious retained successor conditions: This target adds the reviewed personal secure-entry confirmation handoff, the typographic team partial-warning observer correction, and the company-secret authorization fix from unmerged core PR #19325. Personal v10 retains two declared logical human actions: one canonical chat request plus one independently verified same-human secure-save confirmation handoff. Raw chat comments remain one. Team v5 accepts the actual named-skill warning with a typographic apostrophe; saved-state, repeat, runtime, link, and false-success checks remain required. The company route returns scoped 403 guidance before malformed-body validation and preserves board-only creation. Its effect on the unfinished native turn is unproven. The three affected journeys are unrun on this target. Existing failed attempts remain failed. All queries, expected outcomes, native and human bounds, clocks, profile, model, launcher, private protocol, SDK506/efa, and the 196/244 admission gate remain fixed, including the recorded case53 query clarification. Local component proof is not native feature qualification or main/core merge credit.\n\nPrevious retained successor conditions: This target repairs personal secure-entry observation and truthful human acknowledgment, company secure-entry language and ordinary visible menu navigation, team current-skill claim attribution, and native completion feedback that asks for usable product links. Personal v11 expands visible Details before entry when needed, binds the exact value-enLine truncated
|
||||
"changedFields": [
|
||||
{
|
||||
"key": "activity.work-timeline",
|
||||
@@ -229,7 +229,8 @@
|
||||
"chat-catalog-domains/internal-routine-audit.ts",
|
||||
"../e2e/fetch-allowed-port.ts",
|
||||
"../../server/src/services/native-runtime/status-decision-committer.ts",
|
||||
"../../server/src/services/work-timeline.ts"
|
||||
"../../server/src/services/work-timeline.ts",
|
||||
"../../server/src/services/native-runtime/runner-api-catalog.ts"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -635,6 +636,8 @@
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"live-fixtures.ts",
|
||||
"../../server/src/services/secrets.ts",
|
||||
"../../server/src/services/native-runtime/runner-api-catalog.ts",
|
||||
"../../server/src/routes/issues.ts",
|
||||
"run-observations.ts",
|
||||
"chat-flow.ts",
|
||||
"chat-scenarios/secrets.yaml"
|
||||
@@ -644,7 +647,7 @@
|
||||
"key": "secrets.configure-company-secret",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.external.v1",
|
||||
"current": "paperclip.chat-catalog.company-secret-controller-baseline.v12"
|
||||
"current": "paperclip.chat-catalog.company-secret-controller-baseline.v13"
|
||||
},
|
||||
{
|
||||
"key": "secrets.personal-secret",
|
||||
@@ -1001,7 +1004,8 @@
|
||||
"chat-catalog-domains/internal-routine-audit.ts",
|
||||
"../e2e/fetch-allowed-port.ts",
|
||||
"../../server/src/services/native-runtime/status-decision-committer.ts",
|
||||
"../../server/src/services/work-timeline.ts"
|
||||
"../../server/src/services/work-timeline.ts",
|
||||
"../../server/src/services/native-runtime/runner-api-catalog.ts"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1377,6 +1381,8 @@
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"live-fixtures.ts",
|
||||
"../../server/src/services/secrets.ts",
|
||||
"../../server/src/services/native-runtime/runner-api-catalog.ts",
|
||||
"../../server/src/routes/issues.ts",
|
||||
"run-observations.ts",
|
||||
"chat-flow.ts",
|
||||
"chat-scenarios/secrets.yaml"
|
||||
@@ -1386,7 +1392,7 @@
|
||||
"key": "secrets.configure-company-secret",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.external.v1",
|
||||
"current": "paperclip.chat-catalog.company-secret-controller-baseline.v12"
|
||||
"current": "paperclip.chat-catalog.company-secret-controller-baseline.v13"
|
||||
},
|
||||
{
|
||||
"key": "secrets.personal-secret",
|
||||
@@ -1893,7 +1899,7 @@
|
||||
},
|
||||
"candidateLauncherSnapshot": {
|
||||
"path": "fixtures/chat-repair-candidate-launcher.txt",
|
||||
"targetSha": "fe547be84826b4611d3ba9ee1a7d7aa0bd0a2843",
|
||||
"targetSha": "2c99c248e839f41b520fc2d7e3702169ac6cb0a2",
|
||||
"sourcePath": "tests/runner-e2e/launch.ts",
|
||||
"gitBlobOid": "c88df15f509b2ac8638af826647a97d45f16bd1f"
|
||||
}
|
||||
@@ -2450,7 +2456,8 @@
|
||||
"chat-catalog-domains/internal-routine-audit.ts",
|
||||
"../e2e/fetch-allowed-port.ts",
|
||||
"../../server/src/services/native-runtime/status-decision-committer.ts",
|
||||
"../../server/src/services/work-timeline.ts"
|
||||
"../../server/src/services/work-timeline.ts",
|
||||
"../../server/src/services/native-runtime/runner-api-catalog.ts"
|
||||
],
|
||||
"screenshot": "final-state.png",
|
||||
"evidenceKind": "private-final-state-png",
|
||||
@@ -11228,7 +11235,7 @@
|
||||
"connection-setup/credentials"
|
||||
],
|
||||
"checks": null,
|
||||
"oracleVersion": "paperclip.chat-catalog.company-secret-controller-baseline.v12",
|
||||
"oracleVersion": "paperclip.chat-catalog.company-secret-controller-baseline.v13",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/external-company-secret.ts",
|
||||
"chat-catalog-domains/external-secret-proposal.ts",
|
||||
@@ -11246,6 +11253,8 @@
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"live-fixtures.ts",
|
||||
"../../server/src/services/secrets.ts",
|
||||
"../../server/src/services/native-runtime/runner-api-catalog.ts",
|
||||
"../../server/src/routes/issues.ts",
|
||||
"run-observations.ts",
|
||||
"chat-flow.ts",
|
||||
"chat-scenarios/secrets.yaml"
|
||||
|
||||
@@ -103,7 +103,7 @@ test('repair source metadata binds the corrected candidate and explicit changed
|
||||
const repair = manifest.source.repairMetadata;
|
||||
assert.equal(repair.baseTargetSha, '3b66c3c144360ddf32714a0742fd774b8ceafb16');
|
||||
assert.equal(repair.previousTargetSha, 'ab2019e976076ec0eb60b026844f3d9211cb0208');
|
||||
assert.equal(manifest.targetSha, 'fe547be84826b4611d3ba9ee1a7d7aa0bd0a2843');
|
||||
assert.equal(manifest.targetSha, '2c99c248e839f41b520fc2d7e3702169ac6cb0a2');
|
||||
assert.equal(repair.definitionSourceSha, manifest.metadataSourceSha);
|
||||
assert.equal(repair.candidateDescriptorSnapshot, 'fixtures/chat-repair-candidate-descriptors.json');
|
||||
const descriptorBytes = readFileSync(new URL(repair.candidateDescriptorSnapshot, import.meta.url));
|
||||
|
||||
@@ -256,7 +256,8 @@
|
||||
"chat-catalog-domains/internal-routine-audit.ts",
|
||||
"../e2e/fetch-allowed-port.ts",
|
||||
"../../server/src/services/native-runtime/status-decision-committer.ts",
|
||||
"../../server/src/services/work-timeline.ts"
|
||||
"../../server/src/services/work-timeline.ts",
|
||||
"../../server/src/services/native-runtime/runner-api-catalog.ts"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.work-timeline.v11",
|
||||
"coverageGaps": [],
|
||||
@@ -4906,11 +4907,13 @@
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"live-fixtures.ts",
|
||||
"../../server/src/services/secrets.ts",
|
||||
"../../server/src/services/native-runtime/runner-api-catalog.ts",
|
||||
"../../server/src/routes/issues.ts",
|
||||
"run-observations.ts",
|
||||
"chat-flow.ts",
|
||||
"chat-scenarios/secrets.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.company-secret-controller-baseline.v12",
|
||||
"oracleVersion": "paperclip.chat-catalog.company-secret-controller-baseline.v13",
|
||||
"expectedRunCount": 3,
|
||||
"minimumRunCount": 2,
|
||||
"expectedHumanMessageCount": 2,
|
||||
|
||||
Reference in new issue
Block a user