mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 07:10:34 +01:00
test(evals): pin completed timeline command account for full-ten measurement (#19395)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The chat catalog needs fixed source and explicit conditions for each measurement. > - The latest timeline run delivered a complete account within its original deadline. > - Its assertion missed ordinary exitCode notation and a command with an omitted subject inside a named worker bullet. > - The reviewed source corrects those bounded textual checks while retaining independent history and command proof. > - This pull request pins that immutable source for one full-ten measurement. > - Original failed grades and unresolved Company timing remain visible. ## Linked Issues or Issue Description Refs: #19392, #19086, #19388. This follows the prior source pin. The latest timeline response correctly described requirements, a real failed command, saved pause, human reassignment, repair and successful verification. Its original grade still failed one narrative assertion. This pin carries the reviewed correction without merging the full catalog or its production overlays. ## What Changed - Pin source `936c3bbe0e83ca15616254f23d9023b2065fc5fb`, tree `e5c64d67935497164b098e31e995f87ba7875d06`. - Advance only the timeline oracle from v11 to v12 in the candidate snapshot and manifest. - Retain all 244 current queries, contexts, expected outcomes, model, credentials, clocks and human/native accounting. - Record one planned full-ten bounded measurement at native cap 2 and zero automatic retries. - Change only the manifest, its target assertion and the exact candidate descriptor snapshot. ## Verification - I verified the immutable source branch readback before I published this pin. - The new narrative control fails on the parent. The fixed source passes 300 timeline, native-proof and declaration controls, including all 242 lazy declarations with zero fixture callbacks. - The response from result 5094 is accepted by the local diagnostic. The original grade is retained as FAILED; this is not a new native PASS. - Fresh full runner E2E types and the SDK guard over 506 inputs pass in the qualified source graph. An earlier isolated worktree type attempt had dependency-alias errors; the clean source qualification supersedes that setup gap. - Controller: `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs tests/runner-e2e/chat-full-catalog-testops-adapter.test.mjs tests/runner-e2e/chat-full-catalog-testops-reconcile.test.mjs` — 40 controls pass. - The snapshot equals the committed source descriptor bytes. All 244 rows compare equal to the prior pin except the timeline oracle version. - Source descriptor SHA256: `19185aa5298c93d159c3d8547661fd6ef22185e74be1c18b8497bae2ef8051c7`. - Controller manifest SHA256: `822494b170139a8204ff36b33891e2f4960ac0aba6800e9856f2d25dd24de7c6`. - Launcher, protected evidence wire and the accepted 506 SDK inputs remain unchanged. The source SDK registry stays `efa70621e9fb42aa185f7ee616a1a726aa3eccb367e67e28817cedf26ef2ade8`. - Exact pin head `31f47fcdfb600e6b15b35ed50c2a447d03681f06` has 56 successful checks and three configured skips. Both protected checks pass. Greptile is 5/5 for this head with zero open threads. Paperclip Review completed neutral at 5/5. GitHub reports CLEAN and MERGEABLE. These checks do not qualify the unrun native experiment. ## Risks - Original timeline result 5094 remains FAILED. All three native runs succeeded and the account was complete in 248,876 ms, but the original narrative check failed. Local calibration does not rewrite that grade. - Original Company result 5093 ended without first-CEO completion. Its model/runtime timing cause remains unresolved. This source makes no Company runtime or latency fix claim. - The full-ten measurement is unrun. It is a bounded experiment, not proof of deterministic timing, reliability or an all-ten pass. - Company retains two base human inputs plus at most one independently verified UI approval, minimum two and maximum three native runs, and 480,000 ms. Timeline retains three native runs, one human input and 720,000 ms. No condition is waived. - Secret journeys still require independent semantic safety review. Metadata shape alone does not establish that a proposed name is non-sensitive. - The interaction-accept and API-search overlays remain separate, reviewed, unmerged core PRs. This pin does not merge them or the full catalog. The 196/244 admission gate stays fixed. ## Model Used OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository tools and code execution. The host does not expose an exact context-window size. No model or paid provider was called for local controls. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
This commit is contained in:
Dotta authored and GitHub committed
1 parent
f8cb4957dc
commit
d8b0eae687
3 files changed
+9
-9
No files matched your search
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"schema": "paperclip.chat-full-catalog-testops-manifest/v1",
|
||||
"targetSha": "2c99c248e839f41b520fc2d7e3702169ac6cb0a2",
|
||||
"targetSha": "936c3bbe0e83ca15616254f23d9023b2065fc5fb",
|
||||
"metadataSourceSha": "de0941835cc8f0d01c982574ee258e110babdbeb",
|
||||
"profile": "runner-opencode",
|
||||
"model": "openrouter/deepseek/deepseek-v4.1-flash-1017",
|
||||
@@ -172,7 +172,7 @@
|
||||
"baseTargetSha": "3b66c3c144360ddf32714a0742fd774b8ceafb16",
|
||||
"previousTargetSha": "ab2019e976076ec0eb60b026844f3d9211cb0208",
|
||||
"definitionSourceSha": "de0941835cc8f0d01c982574ee258e110babdbeb",
|
||||
"candidateDescriptorSha256": "e12d1600a7c99ec94ca0d90cdedee85ad89f6a1d637eeb7cf939dfa36c27f015",
|
||||
"candidateDescriptorSha256": "19185aa5298c93d159c3d8547661fd6ef22185e74be1c18b8497bae2ef8051c7",
|
||||
"changedDefinitionFiles": [
|
||||
{
|
||||
"path": "tests/runner-e2e/chat-scenarios/companies.yaml",
|
||||
@@ -199,7 +199,7 @@
|
||||
"sha256": "91a9ddc4f5357f19a2cdc73ff8a090c741d14780e4ed502c856ebd557986f7e1"
|
||||
}
|
||||
],
|
||||
"conditions": "Historical qualification conditions (retained; statements that all requests or sourceFiles remain fixed refer to those earlier successors): New candidate measurements only. Original definitions, prior failed attempts and source6 ab201 remain historical. This target retains that catalog and integrates merged runtime fixes #18711, #18713 and #18718, with the prior-run restoration condition adapted to its existing owner. It adds the reviewed previous-question locator, attributed human-only denial and conditional-outcome oracle, and actor-sensitive timeline narrative/recorded-reassignment fixes. Closed finish diagnostics retain an effective schema digest, hashed call correlation and bounded type/category enums only. Historical MCP timeout and continuation-shape causes remain unresolved; no new all-ten PASS is claimed. Queries, model, human/native counts and original clocks remain fixed; the earlier Studio144 explicit 2..3 accounting stays. Core PRs18717/18734/18742 are now merged. SDK115 completion remains UNKNOWN and the other five SDK cases remain held. Full244 runtime catalog is not merged to master and the196/244 gate is unchanged. This successor binds the seven reviewed scenario repairs plus framework evidence: canonical skill navigation, original announcement phase clock, attributed timeline context, bound hire revision history, credential warning and configuration-question admission, completed team visibility, and canonical private framework evidence. Twelve declaration fields across eleven cases change only oracle versions and skill source-file provenance. All original requests, setup, human/native bounds, model and clocks remain fixed. Historical failed attempts and missing-original UNKNOWN grades remain unchanged; no uniform-ten PASS is claimed. The existing SDK506/efa qualification is reused only for byte-identical registered source inputs. This next successor adds reviewed question-surface observation, grounded scenario grading and safe credential-guidance boundaries after the retained uniform attempt. Eleven existing oracle-version leaves advance; all sourceFiles declarations, requests, setup, counts, clocks, model, launcher and private protocol helpers remain fixed. Component qualification does not attest a new uniform-ten pass.\n\nPrevious successor conditions (retained): This successor measures five reviewed repairs at the retained original model and bounds. Exactly one request is clarified: multipart53 now explicitly asks to walk through the visible form; its original broad request and failures4623/4717 remain historical. The previous inventory0cf530 is retained and the disclosed clarified inventory is814bb363. Other243 catalog requests are unchanged. The company-secret second human message now references the actual saved non-secret metadata instead of repeating the initial request; native continuation must remain attributed to the actual agent. Five oracle-version leaves and four source-file additions change; launcher, private helpers, SDK506/efa, identities, counts, clocks and the196/244 gate remain fixed. All five successor journeys remain unrun; component proof is not a PASS. For Team610, independently saved canonical skill/version/member proof establishes recovery; no required recovery-success phrase is imposed. A contradictory current denial still fails. If the required skill remains missing, truthful disclosure of that condition remains required.\n\nPrevious retained successor conditions: This target adds the reviewed personal secure-entry confirmation handoff, the typographic team partial-warning observer correction, and the company-secret authorization fix from unmerged core PR #19325. Personal v10 retains two declared logical human actions: one canonical chat request plus one independently verified same-human secure-save confirmation handoff. Raw chat comments remain one. Team v5 accepts the actual named-skill warning with a typographic apostrophe; saved-state, repeat, runtime, link, and false-success checks remain required. The company route returns scoped 403 guidance before malformed-body validation and preserves board-only creation. Its effect on the unfinished native turn is unproven. The three affected journeys are unrun on this target. Existing failed attempts remain failed. All queries, expected outcomes, native and human bounds, clocks, profile, model, launcher, private protocol, SDK506/efa, and the 196/244 admission gate remain fixed, including the recorded case53 query clarification. Local component proof is not native feature qualification or main/core merge credit.\n\nPrevious retained successor conditions: This target repairs personal secure-entry observation and truthful human acknowledgment, company secure-entry language and ordinary visible menu navigation, team current-skill claim attribution, and native completion feedback that asks for usable product links. Personal v11 expands visible Details before entry when needed, binds the exact value-enLine truncated
|
||||
"conditions": "Historical qualification conditions (retained; statements that all requests or sourceFiles remain fixed refer to those earlier successors): New candidate measurements only. Original definitions, prior failed attempts and source6 ab201 remain historical. This target retains that catalog and integrates merged runtime fixes #18711, #18713 and #18718, with the prior-run restoration condition adapted to its existing owner. It adds the reviewed previous-question locator, attributed human-only denial and conditional-outcome oracle, and actor-sensitive timeline narrative/recorded-reassignment fixes. Closed finish diagnostics retain an effective schema digest, hashed call correlation and bounded type/category enums only. Historical MCP timeout and continuation-shape causes remain unresolved; no new all-ten PASS is claimed. Queries, model, human/native counts and original clocks remain fixed; the earlier Studio144 explicit 2..3 accounting stays. Core PRs18717/18734/18742 are now merged. SDK115 completion remains UNKNOWN and the other five SDK cases remain held. Full244 runtime catalog is not merged to master and the196/244 gate is unchanged. This successor binds the seven reviewed scenario repairs plus framework evidence: canonical skill navigation, original announcement phase clock, attributed timeline context, bound hire revision history, credential warning and configuration-question admission, completed team visibility, and canonical private framework evidence. Twelve declaration fields across eleven cases change only oracle versions and skill source-file provenance. All original requests, setup, human/native bounds, model and clocks remain fixed. Historical failed attempts and missing-original UNKNOWN grades remain unchanged; no uniform-ten PASS is claimed. The existing SDK506/efa qualification is reused only for byte-identical registered source inputs. This next successor adds reviewed question-surface observation, grounded scenario grading and safe credential-guidance boundaries after the retained uniform attempt. Eleven existing oracle-version leaves advance; all sourceFiles declarations, requests, setup, counts, clocks, model, launcher and private protocol helpers remain fixed. Component qualification does not attest a new uniform-ten pass.\n\nPrevious successor conditions (retained): This successor measures five reviewed repairs at the retained original model and bounds. Exactly one request is clarified: multipart53 now explicitly asks to walk through the visible form; its original broad request and failures4623/4717 remain historical. The previous inventory0cf530 is retained and the disclosed clarified inventory is814bb363. Other243 catalog requests are unchanged. The company-secret second human message now references the actual saved non-secret metadata instead of repeating the initial request; native continuation must remain attributed to the actual agent. Five oracle-version leaves and four source-file additions change; launcher, private helpers, SDK506/efa, identities, counts, clocks and the196/244 gate remain fixed. All five successor journeys remain unrun; component proof is not a PASS. For Team610, independently saved canonical skill/version/member proof establishes recovery; no required recovery-success phrase is imposed. A contradictory current denial still fails. If the required skill remains missing, truthful disclosure of that condition remains required.\n\nPrevious retained successor conditions: This target adds the reviewed personal secure-entry confirmation handoff, the typographic team partial-warning observer correction, and the company-secret authorization fix from unmerged core PR #19325. Personal v10 retains two declared logical human actions: one canonical chat request plus one independently verified same-human secure-save confirmation handoff. Raw chat comments remain one. Team v5 accepts the actual named-skill warning with a typographic apostrophe; saved-state, repeat, runtime, link, and false-success checks remain required. The company route returns scoped 403 guidance before malformed-body validation and preserves board-only creation. Its effect on the unfinished native turn is unproven. The three affected journeys are unrun on this target. Existing failed attempts remain failed. All queries, expected outcomes, native and human bounds, clocks, profile, model, launcher, private protocol, SDK506/efa, and the 196/244 admission gate remain fixed, including the recorded case53 query clarification. Local component proof is not native feature qualification or main/core merge credit.\n\nPrevious retained successor conditions: This target repairs personal secure-entry observation and truthful human acknowledgment, company secure-entry language and ordinary visible menu navigation, team current-skill claim attribution, and native completion feedback that asks for usable product links. Personal v11 expands visible Details before entry when needed, binds the exact value-enLine truncated
|
||||
"changedFields": [
|
||||
{
|
||||
"key": "activity.work-timeline",
|
||||
@@ -237,7 +237,7 @@
|
||||
"key": "activity.work-timeline",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.internal.v1",
|
||||
"current": "paperclip.chat-catalog.work-timeline.v11"
|
||||
"current": "paperclip.chat-catalog.work-timeline.v12"
|
||||
},
|
||||
{
|
||||
"key": "activity.work-timeline",
|
||||
@@ -1012,7 +1012,7 @@
|
||||
"key": "activity.work-timeline",
|
||||
"field": "oracleVersion",
|
||||
"previous": "paperclip.chat-catalog.internal.v1",
|
||||
"current": "paperclip.chat-catalog.work-timeline.v11"
|
||||
"current": "paperclip.chat-catalog.work-timeline.v12"
|
||||
},
|
||||
{
|
||||
"key": "agents.choose-primary",
|
||||
@@ -1899,7 +1899,7 @@
|
||||
},
|
||||
"candidateLauncherSnapshot": {
|
||||
"path": "fixtures/chat-repair-candidate-launcher.txt",
|
||||
"targetSha": "2c99c248e839f41b520fc2d7e3702169ac6cb0a2",
|
||||
"targetSha": "936c3bbe0e83ca15616254f23d9023b2065fc5fb",
|
||||
"sourcePath": "tests/runner-e2e/launch.ts",
|
||||
"gitBlobOid": "c88df15f509b2ac8638af826647a97d45f16bd1f"
|
||||
}
|
||||
@@ -2442,7 +2442,7 @@
|
||||
"activity/timeline"
|
||||
],
|
||||
"checks": null,
|
||||
"oracleVersion": "paperclip.chat-catalog.work-timeline.v11",
|
||||
"oracleVersion": "paperclip.chat-catalog.work-timeline.v12",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/internal-definition.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
|
||||
@@ -103,7 +103,7 @@ test('repair source metadata binds the corrected candidate and explicit changed
|
||||
const repair = manifest.source.repairMetadata;
|
||||
assert.equal(repair.baseTargetSha, '3b66c3c144360ddf32714a0742fd774b8ceafb16');
|
||||
assert.equal(repair.previousTargetSha, 'ab2019e976076ec0eb60b026844f3d9211cb0208');
|
||||
assert.equal(manifest.targetSha, '2c99c248e839f41b520fc2d7e3702169ac6cb0a2');
|
||||
assert.equal(manifest.targetSha, '936c3bbe0e83ca15616254f23d9023b2065fc5fb');
|
||||
assert.equal(repair.definitionSourceSha, manifest.metadataSourceSha);
|
||||
assert.equal(repair.candidateDescriptorSnapshot, 'fixtures/chat-repair-candidate-descriptors.json');
|
||||
const descriptorBytes = readFileSync(new URL(repair.candidateDescriptorSnapshot, import.meta.url));
|
||||
|
||||
@@ -259,7 +259,7 @@
|
||||
"../../server/src/services/work-timeline.ts",
|
||||
"../../server/src/services/native-runtime/runner-api-catalog.ts"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.work-timeline.v11",
|
||||
"oracleVersion": "paperclip.chat-catalog.work-timeline.v12",
|
||||
"coverageGaps": [],
|
||||
"expectedRunCount": 3,
|
||||
"minimumRunCount": 3,
|
||||
|
||||
Reference in new issue
Block a user