mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-04 19:28:46 +01:00
test(e2e): reconcile finite catalog authoring checkpoint
This commit is contained in:
Dotta committed
1 parent
50775edc48
commit
a1859e0e67
4 files changed
+1061
-161
No files matched your search
@@ -5,7 +5,7 @@ Updated: 2026-10-26. This is the single progress record for [the saved goal](202
|
||||
## Current state
|
||||
|
||||
- **244 scenarios / 35 areas / 191 capabilities**, frozen inventory digest `9dc9cf6a5426cf1b56bc30a3571851f233f9dd5e93a1292eaee414d33b52df77`. All original queries, contexts and expected outcomes remain unchanged. **196 passes required**; blocked, partial, unsupported and unrun remain in the denominator.
|
||||
- **67 authored: 57 complete, 10 partial, 177 unimplemented** at committed-source metadata checkpoint `9920727210c7193ca1baf58a260a48ac6dd32a82`. Newer authored changes wait for the next finite checkpoint. There are 65 concrete lazy descriptors plus two legacy task cases. Counts come from actual exports, bounds, source files and executable callbacks, not registration-only stubs. Paused inventory owners now have explicit saved readiness-policy evidence in `b53c596bc2`.
|
||||
- **110 authored: 103 without declared coverage gaps, 7 partial, 134 unimplemented** at committed-source metadata checkpoint `b33f3b8b649cc597ffe9a4a66fc834e47fd93e0e`. There are 108 concrete lazy descriptors plus two legacy task cases. Counts come from actual committed exports, bounds, source files and executable callbacks; they are authoring readiness, not acceptance. Newer area commits, including the task execution-ownership closure, wait for the next finite checkpoint.
|
||||
- **Three scenarios attempted; four actual measurements; zero first-attempt passes and zero final passes (0/244 for both rates).** Goal-tree is exhausted after two failures. Alignment and purpose each retain one failed initial attempt and one allowed correction/rerun remaining.
|
||||
- **No live launcher, browser, database or provider process is running.** Manager held the two final retries before reservation, so remaining shared authoring contracts cannot stale a pass after its final allowed attempt. No unchanged rerun, extra nudge, deadline extension, automatic retry or relabeled canonical health probe was used.
|
||||
- Production baseline `8373d7867963674ff960d2345d0d28086f799bc7`; branch `codex/chat-catalog-evals`. PR #17794 merged via `2ee36a4`. No eval/core PR has been created. No merge is eligible.
|
||||
@@ -47,12 +47,33 @@ The finite assignments cover the full denominator; future-wave assignments remai
|
||||
| `eval_external` | 52 | connection-setup, app-permissions, chat-channels, assistant-connections, gateways-profiles, plugins, teams, secrets |
|
||||
| `eval_campaign` | 16 | instance-operations, cli-operations, developer-labs; also sole live coordinator and ledger/report owner |
|
||||
|
||||
Next: author the 16 privileged cases against actual owned operator targets with explicit developer/presentation scope, then consume the two remaining goal retries only after shared dependencies settle. Do not rerun exhausted goal-tree. No live test suites run concurrently with campaign instances. Preflight rechecks IPC capacity and exact source/binary/dependency identity; no arbitrary IPC cleanup, kernel tuning or live database changes.
|
||||
Next: finish remaining shared/area authoring contracts before immutable freeze A, then run remaining initial attempts sequentially with retries zero. Accumulate one coherent correction set, freeze B, and review the exact eligible final retry set before execution. Alignment and purpose final retries stay deferred into B; goal-tree is exhausted and must not receive a third run. The coordinator pauses after this finite checkpoint for the external authoring slot and retains sole live/ledger ownership. No live test suites run concurrently with campaign instances. Preflight rechecks IPC capacity and exact source/binary/dependency identity; no arbitrary IPC cleanup, kernel tuning or live database changes.
|
||||
|
||||
Local tools/dependencies/private subscription auth file are present; provider-key variables, `.env.runner-e2e.local`, Daytona credentials and immutable image were absent at inspection. Presence does not establish validity. External provider/configuration blockers remain explicit and never passing coverage; simulations cannot establish live provider acceptance. No private artifacts have been published.
|
||||
|
||||
Next-wave review items: bound identity probe credentials privately across repeated fresh reads and revoke at owned cleanup, rather than create/revoke on every observation; allow harmless secret-related artifacts only after actual private byte scanning, and compare secret version/identity semantics rather than JSON property order. These are implementation corrections, not excuses to weaken outcomes.
|
||||
Identity and domain probe helpers now retain one attempt-private credential lifetime with fresh reads and explicit revocation. Remaining external review: allow harmless secret-related artifacts only after actual private byte scanning, and compare secret version/identity semantics rather than JSON property order. Named ChatGPT setup is explicitly partial: real generic OAuth/MCP client consent is retained but cannot prove the named ChatGPT human setup path.
|
||||
|
||||
Core behavior fixes require separate focused PRs that remain unmerged; their dependent passes cannot count toward the eval threshold. Eval merge requires every scenario implemented/processed, at least 196 final eligible passes, latest-head checks/reviews and actual protected merge. Per-case fingerprints, global suite hash and production/source revisions remain distinct; stale passes cannot be carried across changed execution/grader dependencies.
|
||||
|
||||
Credential discovery: relevant source metadata is readable through the existing 1Password service account outside the sandbox. Notion, QA AgentMail, GitHub, Daytona and Anthropic references are available; no matching Slack reference was found. Credential validity, a disposable repository allowlist, designated mail recipient and pinned cloud image remain unverified. Automatic approval review rejected credentialed provider identity/resource discovery and metadata persistence; explicit approval for the narrow read-only AgentMail/GitHub endpoints is pending. No provider discovery request or metadata write occurred. Private item IDs and account metadata remain outside tracked documents.
|
||||
|
||||
## Finite authoring handoff at b33f3b8b64
|
||||
|
||||
| Owner | Without declared gaps | Partial | Authored |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| Domains | 37 | 2 | 39 |
|
||||
| Identity | 19 | 0 | 19 |
|
||||
| Runtime | 23 | 1 | 24 |
|
||||
| External | 8 | 2 | 10 |
|
||||
| Integrator legacy | 2 | 0 | 2 |
|
||||
| Coordinator privileged | 14 | 2 | 16 |
|
||||
|
||||
All 16 privileged IDs have executable paths. The health case submits the exact canonical body to the same owned backend only after genuine owned PostgreSQL shutdown and actual HTTP/doctor failure; route timing and early evidence survive ordinary admission failure. Cleanup restores the verified owned cluster before generic cleanup. No database fault was performed during authoring.
|
||||
|
||||
Backup restoration now includes a declared factual operator receipt and second, phase-attributed native restore-result report. Performance similarly supplies actual documented fixture measurements to a declared follow-up; generic no-measurement interpretation cannot pass. Feedback retains an actual board-cancelled, unfinished native worker with original comment/run/events, actual human CLI export, independent JSON/archive byte and scope checks, unchanged sharing, and a truthful local-location reply. It is labelled interrupted/cancelled work, never model failure or failed-native coverage. CLI returned commands execute their actual supported argv with an explicit source executable mapping and unresolved-identifier receipts; no JSON flags are silently added. Real state readback supplies independent identity checks.
|
||||
|
||||
The two privileged partials are `instance-operations.update-instance` and `instance-operations.provision-runtime`: `installed-source-activation-and-artifact-fingerprint-lane`. Ordinary chat currently rejects installed activation; this is a missing harness capability, not credential absence. Their area code requires an actual source installed target and private mode0600 `PAPERCLIP_CHAT_OPERATOR_INSTALL_RECEIPT` inside the owned attempt: exact CLI/server hashes, complete owned npm-prefix dependency-tree hash, source process PID/UID, source HTTP version and disposable ownership. Update additionally requires an exact supported published version/integrity and retains the concrete original-version reinstall option; it applies the public updater, restarts the actual source and independently reads its new version/readiness. Provision requires an explicitly named private OpenRouter environment reference, missing release-matched assets, the actual public provisioner and closure identity, then a separate genuinely running native Pi worker and exact saved file, followed by a phase-attributed result report. Its declared total is three native runs (initial controller, Pi worker, controller interpretation) and two human messages. Download alone never qualifies it.
|
||||
|
||||
Integrator owns the narrow allowlisted installed ordinary-chat selector/fingerprint/receipt lane. It must preserve source-mode guards and bind actual installed artifact closure rather than relabel a source pack as a published release. Coordinator resumes the operator cases after that capability; these partials cannot count for acceptance in the meantime. Runtime's fifth pre-chat `agent-chat.chat-disabled` boundary is allowlisted with real authenticated harness context and zero admissions; runtime still owns its case implementation.
|
||||
|
||||
Credential-free validation: 32 privileged controls across seven files passed; the later exact-argv change passed five CLI controls and E2E typecheck. Metadata was reconciled only from Git archive b33f3b8b64; dependency symlinks to the earlier immutable candidate were used for metadata imports only, never a live measurement. Newer committed/WIP area changes are pending the next checkpoint. No extra selected-case probe, reservation, live run, credentialed provider discovery or final retry occurred.
|
||||
@@ -25,8 +25,8 @@
|
||||
"sourceSha256": "14324aa63a93c0ac919e7b8650bed87d8ae24ca7ebdbcbc68f5a441b925bf9e8",
|
||||
"scenarioSha256": "41527a821ba793d6dbb1053a7ec68a5b9787b56cf856c2d6b7aa50f6653b107b",
|
||||
"owner": "eval_identity",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.access.cli-authorization",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -126,8 +126,8 @@
|
||||
"sourceSha256": "14324aa63a93c0ac919e7b8650bed87d8ae24ca7ebdbcbc68f5a441b925bf9e8",
|
||||
"scenarioSha256": "e33310e9624ea49cc21e83dae75e4cf08e571fb645bb8c66156b32a1f28ee964",
|
||||
"owner": "eval_identity",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.access.sign-in",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -146,8 +146,8 @@
|
||||
"sourceSha256": "a9b43a3c0500b8628031ed6178dec40c80b1dc0b6df116a32597ba74c9e0de6f",
|
||||
"scenarioSha256": "e0d80d373468c2487f2c29bdd9809bb2fbb98557c64c68b594f4104c0758f0b0",
|
||||
"owner": "eval_domains",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.activity.agent-feedback",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -206,8 +206,8 @@
|
||||
"sourceSha256": "a9b43a3c0500b8628031ed6178dec40c80b1dc0b6df116a32597ba74c9e0de6f",
|
||||
"scenarioSha256": "a340cf61cb2fda86d420ca576e98c26296f4131a59c3d978d1ecb164475b3d3c",
|
||||
"owner": "eval_domains",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.activity.find-runs",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -330,8 +330,8 @@
|
||||
"sourceSha256": "19defd25f25ebd32749e069a1db15af779ba08b482e136a15473ab6897053f4a",
|
||||
"scenarioSha256": "7bb7a05011d7e278bc3060ac9fd04ad37be3cdec7ab0e99679ee45ada4a40f4c",
|
||||
"owner": "eval_runtime",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.agent-chat.remember-and-follow-up",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -372,8 +372,8 @@
|
||||
"sourceSha256": "19defd25f25ebd32749e069a1db15af779ba08b482e136a15473ab6897053f4a",
|
||||
"scenarioSha256": "08d5cb833e45393a84cde5225b558c708546ef33be2273372a851331c6350ffd",
|
||||
"owner": "eval_runtime",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.agent-chat.track-delegated-work",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -600,8 +600,8 @@
|
||||
"sourceSha256": "a5aa89ff1a55f7a2f580230c72f20d05420d272dbad365c4d7dd963016332bf3",
|
||||
"scenarioSha256": "35244f0953539274eea8d2bf250d9eb0b1bc404c62d64875bae6e96f40833557",
|
||||
"owner": "eval_runtime",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.agents.hire-needs-approval",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -700,8 +700,8 @@
|
||||
"sourceSha256": "a5aa89ff1a55f7a2f580230c72f20d05420d272dbad365c4d7dd963016332bf3",
|
||||
"scenarioSha256": "38b4b159d89acf62c5257edc93045c91bc93e926e0366a6ddc5402df3fc7d31e",
|
||||
"owner": "eval_runtime",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.agents.retire-agent",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -842,9 +842,9 @@
|
||||
"sourceSha256": "b87c34edd99aa5cce96cf2196327b66366ef24bbcea96dc8c86d404030372901",
|
||||
"scenarioSha256": "8e1b5269d92d5b144054e7ea43f08aed14f516dbe1bff0db73e971bb3bdb5f96",
|
||||
"owner": "eval_external",
|
||||
"readiness": "implemented",
|
||||
"readiness": "partial",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.assistant-connections.connect-external-assistant",
|
||||
"blocker": null,
|
||||
"blocker": "named-chatgpt-human-setup",
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null
|
||||
@@ -1412,8 +1412,8 @@
|
||||
"sourceSha256": "d3d89e58aeb7e36198c9a1109c31ff501e927621ce68f36d78a10b5d00839565",
|
||||
"scenarioSha256": "1c29be9a9e8351e00602d37ca1cf670c0dc83cf8474739c7983209764065670c",
|
||||
"owner": "eval_campaign",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.cli-operations.choose-cli-context",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -1433,8 +1433,8 @@
|
||||
"sourceSha256": "d3d89e58aeb7e36198c9a1109c31ff501e927621ce68f36d78a10b5d00839565",
|
||||
"scenarioSha256": "6eb82a4a3f0dd7a7c197063dc685fdb7cdaa59e07532157e40983643e5bebfe9",
|
||||
"owner": "eval_campaign",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.cli-operations.create-from-cli",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -1454,8 +1454,8 @@
|
||||
"sourceSha256": "d3d89e58aeb7e36198c9a1109c31ff501e927621ce68f36d78a10b5d00839565",
|
||||
"scenarioSha256": "6c82c34c435e528504b136f65d46062011c0c1a094047592bdd76c197d99e9bd",
|
||||
"owner": "eval_campaign",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.cli-operations.export-document-cli",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -1475,8 +1475,8 @@
|
||||
"sourceSha256": "d3d89e58aeb7e36198c9a1109c31ff501e927621ce68f36d78a10b5d00839565",
|
||||
"scenarioSha256": "3496eafcb5cf5aebd86a1efaf1db636073e9ea3254d43421e3c09af8e45928e3",
|
||||
"owner": "eval_campaign",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.cli-operations.feedback-bundle",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -1496,8 +1496,8 @@
|
||||
"sourceSha256": "d3d89e58aeb7e36198c9a1109c31ff501e927621ce68f36d78a10b5d00839565",
|
||||
"scenarioSha256": "788d495f8d426b07844edb19a81d4d0cd8bb8ee1db611307f175708c9a6f09a7",
|
||||
"owner": "eval_campaign",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.cli-operations.inspect-run-cli",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -1517,8 +1517,8 @@
|
||||
"sourceSha256": "d3d89e58aeb7e36198c9a1109c31ff501e927621ce68f36d78a10b5d00839565",
|
||||
"scenarioSha256": "3036873f567ce80ea64c7ee3d0753afb04686314b02a6d775ea13bd0f2a7f6ee",
|
||||
"owner": "eval_campaign",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.cli-operations.owned-dev-instance",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -1557,8 +1557,8 @@
|
||||
"sourceSha256": "395472a5531e93259672c2c07f1819992f6f14ccf11513e1990aa3c576f3fa27",
|
||||
"scenarioSha256": "4b53ad7ca9885c939ff6b69d685078be0ff1756e79f7f132b7253d2959796abb",
|
||||
"owner": "eval_identity",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.companies.archive-company",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -1699,8 +1699,8 @@
|
||||
"sourceSha256": "395472a5531e93259672c2c07f1819992f6f14ccf11513e1990aa3c576f3fa27",
|
||||
"scenarioSha256": "04c42aee139857a9158fffdbc57f062a5ead44b06d438f9aea9926fff5e1ea66",
|
||||
"owner": "eval_identity",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.companies.switch-company",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -1990,8 +1990,8 @@
|
||||
"sourceSha256": "c09b9eeb0b76dd462dc2a42678d97859c4289e6265cf76097bde8dd7c8322cf1",
|
||||
"scenarioSha256": "1381c6828ab6059e280782897ac9cc1d7a47cc6b4cbb317ae3c4716040c04890",
|
||||
"owner": "eval_campaign",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.developer-labs.component-reference",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -2011,8 +2011,8 @@
|
||||
"sourceSha256": "c09b9eeb0b76dd462dc2a42678d97859c4289e6265cf76097bde8dd7c8322cf1",
|
||||
"scenarioSha256": "995508eb028d95beffdb28b2c546a1606ddb903911ba0d5b824c07b0260c73b3",
|
||||
"owner": "eval_campaign",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.developer-labs.gated-lab",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -2031,8 +2031,8 @@
|
||||
"sourceSha256": "c09b9eeb0b76dd462dc2a42678d97859c4289e6265cf76097bde8dd7c8322cf1",
|
||||
"scenarioSha256": "a586a03c729abbb3ab305fe789c3d0e00d93d13da2d0e5bc216ea3c43b6e0f29",
|
||||
"owner": "eval_campaign",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.developer-labs.interaction-fixture",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -2051,8 +2051,8 @@
|
||||
"sourceSha256": "c09b9eeb0b76dd462dc2a42678d97859c4289e6265cf76097bde8dd7c8322cf1",
|
||||
"scenarioSha256": "7a36242f134f6783deb659f0cc38f40598776c03e4ba90bd88e681a134892c88",
|
||||
"owner": "eval_campaign",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.developer-labs.long-thread-performance",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -2072,8 +2072,8 @@
|
||||
"sourceSha256": "766da628af9419eaf0b112ec635cddb8e9d1fca903d1dc4602f69f31b0c539ca",
|
||||
"scenarioSha256": "48f3a32f0e2a1aa4878b22c658ee0987056dfd798819e938e59210ba51678706",
|
||||
"owner": "eval_domains",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.documents-artifacts.find-artifact",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -2093,8 +2093,8 @@
|
||||
"sourceSha256": "766da628af9419eaf0b112ec635cddb8e9d1fca903d1dc4602f69f31b0c539ca",
|
||||
"scenarioSha256": "887b384828ded827e052a068f2d50513677eb82be1cf0dbe4ca204d0963519ce",
|
||||
"owner": "eval_domains",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.documents-artifacts.private-download",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -2114,8 +2114,8 @@
|
||||
"sourceSha256": "766da628af9419eaf0b112ec635cddb8e9d1fca903d1dc4602f69f31b0c539ca",
|
||||
"scenarioSha256": "400e48d0b70407f718712718f3252747339152473d2d7fc8078e26e88080996a",
|
||||
"owner": "eval_domains",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.documents-artifacts.produce-download",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -2175,8 +2175,8 @@
|
||||
"sourceSha256": "766da628af9419eaf0b112ec635cddb8e9d1fca903d1dc4602f69f31b0c539ca",
|
||||
"scenarioSha256": "b4d273782ebdb6ab66ab8c10e1af3a9fd7249a9b896a2a723dd8a01a6b2606ff",
|
||||
"owner": "eval_domains",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.documents-artifacts.revise-delivery",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -2621,8 +2621,8 @@
|
||||
"sourceSha256": "afc939f4eec821edd38b8e819274ede302c10550aca68b2ff72c5ebf870aed05",
|
||||
"scenarioSha256": "a02544496a0f5a71bd58a5c00f5df5ac4cbe3199b624c5324fa4defa5d53b856",
|
||||
"owner": "eval_domains",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.inbox-search.archive-done",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -2681,8 +2681,8 @@
|
||||
"sourceSha256": "afc939f4eec821edd38b8e819274ede302c10550aca68b2ff72c5ebf870aed05",
|
||||
"scenarioSha256": "703eb37ed81af1af5c2b3739409d85e4062125d8ea286bf8d93a7f15c056337d",
|
||||
"owner": "eval_domains",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.inbox-search.find-design",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -2702,8 +2702,8 @@
|
||||
"sourceSha256": "afc939f4eec821edd38b8e819274ede302c10550aca68b2ff72c5ebf870aed05",
|
||||
"scenarioSha256": "58051c51f23a3bc27e7333ae6cd0afa6f66de08513f50beae58eaf0cab0dcec9",
|
||||
"owner": "eval_domains",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.inbox-search.notification-audience",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -2722,8 +2722,8 @@
|
||||
"sourceSha256": "afc939f4eec821edd38b8e819274ede302c10550aca68b2ff72c5ebf870aed05",
|
||||
"scenarioSha256": "855fc33bc106cfb11caea791ccb7b52e9fb00dcd20f960bfcda90f4dd36bcc8f",
|
||||
"owner": "eval_domains",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.inbox-search.restore-archive",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -2743,8 +2743,8 @@
|
||||
"sourceSha256": "afc939f4eec821edd38b8e819274ede302c10550aca68b2ff72c5ebf870aed05",
|
||||
"scenarioSha256": "b27eaf08a3f7a9602b4d5d08569ada924c5c5bcf03c1d5d6399b2188177f328c",
|
||||
"owner": "eval_domains",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.inbox-search.show-decisions",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -2783,8 +2783,8 @@
|
||||
"sourceSha256": "7cf3dbf8bcddd40a9b5d7ca9430231c15dc2d10346eadad96ab81617c20c6aa4",
|
||||
"scenarioSha256": "211416b7d059a27afa2120548496a3bcdb1cac14b3a9623148e3a061a2841306",
|
||||
"owner": "eval_campaign",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.instance-operations.backup-and-restore",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -2803,8 +2803,8 @@
|
||||
"sourceSha256": "7cf3dbf8bcddd40a9b5d7ca9430231c15dc2d10346eadad96ab81617c20c6aa4",
|
||||
"scenarioSha256": "486dc1d42513db88f52e90a3b91cb6ba644e62ee739e126140c7cc3c243a4b94",
|
||||
"owner": "eval_campaign",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.instance-operations.diagnose-health",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -2823,8 +2823,8 @@
|
||||
"sourceSha256": "7cf3dbf8bcddd40a9b5d7ca9430231c15dc2d10346eadad96ab81617c20c6aa4",
|
||||
"scenarioSha256": "778883565814847de78d8c1cba156e9d659f4ed536f422e8435a852d35b464f6",
|
||||
"owner": "eval_campaign",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.instance-operations.instance-settings",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -2843,9 +2843,9 @@
|
||||
"sourceSha256": "7cf3dbf8bcddd40a9b5d7ca9430231c15dc2d10346eadad96ab81617c20c6aa4",
|
||||
"scenarioSha256": "286887bd9a36e5ba926b5420db411cab8ac05ddb0bffa41e6b014594f87d369b",
|
||||
"owner": "eval_campaign",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"blocker": null,
|
||||
"readiness": "partial",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.instance-operations.provision-runtime",
|
||||
"blocker": "installed-source-activation-and-artifact-fingerprint-lane",
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null
|
||||
@@ -2863,8 +2863,8 @@
|
||||
"sourceSha256": "7cf3dbf8bcddd40a9b5d7ca9430231c15dc2d10346eadad96ab81617c20c6aa4",
|
||||
"scenarioSha256": "fbfe0844b0b859ea1a2ee5d0368df09dcabe6bbbd04820f9f4eba9dfdf3e2448",
|
||||
"owner": "eval_campaign",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.instance-operations.restart-service",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -2883,9 +2883,9 @@
|
||||
"sourceSha256": "7cf3dbf8bcddd40a9b5d7ca9430231c15dc2d10346eadad96ab81617c20c6aa4",
|
||||
"scenarioSha256": "e1dae8554a765745a6e05392a5e42d3532716904b30c2c64c1f9c4fd789866d7",
|
||||
"owner": "eval_campaign",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"blocker": null,
|
||||
"readiness": "partial",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.instance-operations.update-instance",
|
||||
"blocker": "installed-source-activation-and-artifact-fingerprint-lane",
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null
|
||||
@@ -2944,8 +2944,8 @@
|
||||
"sourceSha256": "097af7c681a618cf3b1b543a6218ebb732f94b0873e388d9795b88d33894b3ab",
|
||||
"scenarioSha256": "49dbb46581e775b36016577113475f9555a996e7c1868c511f1d7c544260aec8",
|
||||
"owner": "eval_identity",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.navigation-preferences.edit-profile",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -2986,9 +2986,9 @@
|
||||
"sourceSha256": "097af7c681a618cf3b1b543a6218ebb732f94b0873e388d9795b88d33894b3ab",
|
||||
"scenarioSha256": "7bba464b3edbdb96b9181c6b4f89cdfdab38d94a5bcb4e3f5711ead767dd8c91",
|
||||
"owner": "eval_identity",
|
||||
"readiness": "partial",
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.navigation-preferences.quick-search",
|
||||
"blocker": "filter-preservation; authenticated-search-denial",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null
|
||||
@@ -3006,8 +3006,8 @@
|
||||
"sourceSha256": "097af7c681a618cf3b1b543a6218ebb732f94b0873e388d9795b88d33894b3ab",
|
||||
"scenarioSha256": "c9aa7fdd8e6be91bcf411c81cb6151419fd6d1580b1cabd70a527562cc5ad75e",
|
||||
"owner": "eval_identity",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.navigation-preferences.sidebar-order",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -3397,9 +3397,9 @@
|
||||
"sourceSha256": "f72b3d7e4ba0b4bb52b2912e3bce58c5f75fa0561411cb86330f2f5ba6d2df7f",
|
||||
"scenarioSha256": "d2647b32f3939c2b08738340d02cf5c71b82326f4e3299accbb0944cf0f0f1e1",
|
||||
"owner": "eval_identity",
|
||||
"readiness": "partial",
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.questions-and-approvals.ambiguous-yes",
|
||||
"blocker": "stale-and-expired-proposal-rejection",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null
|
||||
@@ -3418,9 +3418,9 @@
|
||||
"sourceSha256": "f72b3d7e4ba0b4bb52b2912e3bce58c5f75fa0561411cb86330f2f5ba6d2df7f",
|
||||
"scenarioSha256": "3db91caac72e32867b80c1006fc54864936053a285b52582722345ec6b60aea2",
|
||||
"owner": "eval_identity",
|
||||
"readiness": "partial",
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.questions-and-approvals.approve-tool-once",
|
||||
"blocker": "second-action-requires-distinct-approval; approved-service-failure-retained",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null
|
||||
@@ -3439,9 +3439,9 @@
|
||||
"sourceSha256": "f72b3d7e4ba0b4bb52b2912e3bce58c5f75fa0561411cb86330f2f5ba6d2df7f",
|
||||
"scenarioSha256": "2c4bda5093eb3f29bdd60b16351a4f98ebc985e41cd6fb763aa2637bacb357d6",
|
||||
"owner": "eval_identity",
|
||||
"readiness": "partial",
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.questions-and-approvals.decline-tool",
|
||||
"blocker": "decline-request-must-already-be-pending-before-original-query",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null
|
||||
@@ -3460,9 +3460,9 @@
|
||||
"sourceSha256": "f72b3d7e4ba0b4bb52b2912e3bce58c5f75fa0561411cb86330f2f5ba6d2df7f",
|
||||
"scenarioSha256": "67251ef988b8bd96ea0bf1b237d793495ed09370b35df82f7f67dae435d95a5d",
|
||||
"owner": "eval_identity",
|
||||
"readiness": "partial",
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.questions-and-approvals.defer-and-reopen",
|
||||
"blocker": "draft-state-retention; protected-gate-not-waived; later-answer-continuation",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null
|
||||
@@ -3521,9 +3521,9 @@
|
||||
"sourceSha256": "f72b3d7e4ba0b4bb52b2912e3bce58c5f75fa0561411cb86330f2f5ba6d2df7f",
|
||||
"scenarioSha256": "ff3b9bca8076608a758954d05b3a1c73d96f72e8be94c67720e11db55e03f9e2",
|
||||
"owner": "eval_identity",
|
||||
"readiness": "partial",
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.questions-and-approvals.multi-part-question",
|
||||
"blocker": "submitted-set-resumes-work; accurate-agent-summary",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null
|
||||
@@ -3542,9 +3542,9 @@
|
||||
"sourceSha256": "f72b3d7e4ba0b4bb52b2912e3bce58c5f75fa0561411cb86330f2f5ba6d2df7f",
|
||||
"scenarioSha256": "9ca36d3ec76ce296651c2cd403e41960125af1fbc7702802cc3b36549adf6855",
|
||||
"owner": "eval_identity",
|
||||
"readiness": "partial",
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.questions-and-approvals.necessary-question",
|
||||
"blocker": "provider-native-question-correlation; incidental-settings-exclusion; task-brief-version-provenance",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null
|
||||
@@ -3626,9 +3626,9 @@
|
||||
"sourceSha256": "f74fb387ab717d656e4ce11b8abc03960946f0cba83ccf76e83f4ac77c520a3a",
|
||||
"scenarioSha256": "1a24782c0adce5db07abd005849c5b7770936902d43d18ba51ff57e117e6a4cf",
|
||||
"owner": "eval_runtime",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"blocker": null,
|
||||
"readiness": "partial",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.recovery.browser-reconnect",
|
||||
"blocker": "Reconnect after a genuinely failed native run remains unexercised; successful and still-running native phases plus failed-evidence negative controls are authored.",
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null
|
||||
@@ -3976,8 +3976,8 @@
|
||||
"sourceSha256": "15d3741baab816afe14d24043d8229010fc204146ccb4bb2f525eb0f8d3037fa",
|
||||
"scenarioSha256": "dbcbce44978644b3a10b1e8ad2cea9449d23eeb0cdf7bf6511450a6f3d08a7ed",
|
||||
"owner": "eval_runtime",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.runs-adapters.reset-session",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -4695,8 +4695,8 @@
|
||||
"sourceSha256": "31fca6be61221bb970b7c67f1b9234aa0705162571b46c99592926a286b240f0",
|
||||
"scenarioSha256": "6255dc9a8c2fa505c544d6c46d07c6e92380025897095bfabb4def986b802746",
|
||||
"owner": "eval_domains",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.tasks.assignment-conflict",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -4863,8 +4863,8 @@
|
||||
"sourceSha256": "31fca6be61221bb970b7c67f1b9234aa0705162571b46c99592926a286b240f0",
|
||||
"scenarioSha256": "1b7a53fb395d169e0c40bb258b38ff658c39dfff4c9eede4810d912689f5917b",
|
||||
"owner": "eval_domains",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.tasks.private-task",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -5029,8 +5029,8 @@
|
||||
"sourceSha256": "6b4be6151f155981ae7a68ac69c3a31420d54fe33f1ac567de0db73b651b3ae7",
|
||||
"scenarioSha256": "71548aa07a1e910b82535691a66f9e8d4f9dafc911191b43e9460a32fce1bb98",
|
||||
"owner": "eval_runtime",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.workspaces.close-and-reopen",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -5049,8 +5049,8 @@
|
||||
"sourceSha256": "6b4be6151f155981ae7a68ac69c3a31420d54fe33f1ac567de0db73b651b3ae7",
|
||||
"scenarioSha256": "db6dd1bf35167ccc383cc913aef0110e41cca2c553d3ff9ff338fde2681bc33d",
|
||||
"owner": "eval_runtime",
|
||||
"readiness": "unimplemented",
|
||||
"executionId": null,
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.workspaces.inspect-git",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
@@ -5160,5 +5160,13 @@
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null
|
||||
}
|
||||
]
|
||||
],
|
||||
"implementationCheckpoint": {
|
||||
"sourceRevision": "b33f3b8b649cc597ffe9a4a66fc834e47fd93e0e",
|
||||
"descriptorSha256": "6273f9afea18717074abe13bda4bc3132485c2c73245a42b9bee7f25c07a3c73",
|
||||
"authored": 110,
|
||||
"complete": 103,
|
||||
"partial": 7,
|
||||
"unimplemented": 134
|
||||
}
|
||||
}
|
||||
@@ -48,6 +48,8 @@ export interface ChatCampaign {
|
||||
denominator: number; areas: number; capabilities: number;
|
||||
requiredPasses: number;
|
||||
corePrs: string[]; evalPr: string | null;
|
||||
/** Authored metadata only; never passing execution evidence. */
|
||||
implementationCheckpoint?: { sourceRevision: string; descriptorSha256: string; authored: number; complete: number; partial: number; unimplemented: number };
|
||||
scenarios: CampaignScenario[];
|
||||
}
|
||||
const digest = (value: string) => createHash("sha256").update(value).digest("hex");
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
Reference in new issue
Block a user