mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-06 05:56:19 +01:00
Close thirty initial attempts and reconcile all authored catalog callbacks
This commit is contained in:
1 parent
673586931f
commit
b971314319
3 files changed
+616
-81
No files matched your search
@@ -5,15 +5,15 @@ Updated: 2026-10-27. This is the single progress record for [the saved goal](202
|
||||
## Current state
|
||||
|
||||
- **Frozen inventory:244 scenarios /35areas /191capabilities;196 eligible passes required.** Inventory digest `9dc9cf6a5426cf1b56bc30a3571851f233f9dd5e93a1292eaee414d33b52df77`. Original queries, contexts and outcomes remain unchanged; blocked, partial and unrun cases stay in the denominator.
|
||||
- **All244 callbacks authored** at closed metadata source `406eb610dd95604079838220f29fb765977a8910`:235 without declared gaps,9 partial,0 absent (242 lazy definitions +2 legacy). **241 authoring-resolved:**235 complete and6 concrete executable blocker probes. **3 unfinished:**named ChatGPT setup, private GitHub after-grant completion and Grok operator closure. Grok's later area-only `f457fa84b0` completion is queued for the next metadata checkpoint, not silently mixed into this archive. Source-only controls are not live acceptance.
|
||||
- **Closed checkpoint:42 scenarios processed /43 completed measurements:8 initial passes and35 failures;0 final eligible passes.** Actual retained native usage:17,478,933 tokens; subscription dollars remain unpriced. The latest24-case batch finished5passed/19failed, with verified cleanup and exact Docker removal for all24. The authoritative closed summary is `tests/runner-e2e/results/chat-catalog-campaign/closed-initial-checkpoint-42-43.json`. Two older cleanup uncertainties remain unresolved; later partial PID/container absence evidence does not rewrite them.
|
||||
- **Single coherent final-B retry set remains held.** Goal-tree exhausted its two measurements. Every original failure/result/digest is preserved. Eight real initial passes retain their proof but need final current-closure eligibility review. First-failure triage (34 first failures; goal-tree's extra final failure excluded):setup/fixture11, locator/oracle9, harness/startup4, native execution2, unsupported context1, model output1, unknown/untriaged6. Setup-stage uncertainty and native timeouts are not assumed product/vendor defects.
|
||||
- **Next30 initial-only batch authorized and running sequentially**, immutable source `efbbb88a70dc6bcd0e6893702d8a482e46753fdf`, baseline8373, maxParallel1/retries0,38 declared native runs/32 source-chat human turns. Source archive typecheck and84 focused controls passed; actual list244, all30 fingerprints have narrow closures and replay after private-file cleanup is unnecessary/null identity verified. Five broad CLI/plugin closures are explicitly deferred because they include unrelated eval source. Conference-room ordinal1 was reserved and admitted at2026-10-27T09:20:35Z; every later case is reserved only at its actual admission. Live counts come from the manifest and `active-next30-efbb.json`, not this closed checkpoint. Known invitation-acceptance and same-title-alias fixtures remain quarantined/unreserved.
|
||||
- **All244 callbacks authored and authoring-resolved** at closed source `b65cce861b5667f2424bdfacbdfd610fa697ba60`:238 without declared gaps and6 concrete executable blocker probes,0 absent/unfinished (242 lazy definitions +2 legacy). Grok, named ChatGPT and private GitHub after-grant callbacks are integrated. This is authored coverage, not live acceptance.
|
||||
- **72 scenarios processed /73 completed measurements:10 initial passes and63 failures;0 running at the closed checkpoint;0 final eligible passes.** Actual retained native usage:28,018,825 tokens; subscription dollars remain unpriced. The latest30-case batch finished2passed/28failed; all30 have verified cleanup and exact Docker removal. Closed receipt:`tests/runner-e2e/results/chat-catalog-campaign/closed-initial-checkpoint-72-73.json`. Two older cleanup uncertainties remain unresolved; original measurement bytes are preserved.
|
||||
- **Single coherent final-B retry set remains held.** Goal-tree exhausted two measurements. All original outcomes, evidence and measured digests remain unchanged. Actual current selected identity review is retained in `closed-pass-identity-review-b65.json`; changed selected execution/grader closures require reviewed revalidation within the same max-two budget. Unrelated Git revisions and unchanged failed behavior do not justify reruns.
|
||||
- **The30-case efbb batch completed; a two-case concurrent initial qualification is authorized** from immutable `c167a54162fadffe665b4d35a7a1308c6233b75d`:companies.archive-company and connection-setup.choose-provider, each1native/1source-chat human turn/300s; two one-case children, each maxParallel1/retries0. Exact source/fingerprints, qualified Docker/native,40 archived helper/shared controls plus parent prepare/negative/atomic controls passed. Prior launcher/writer exited and exact last container absence was independently verified. Sole-writer qualifying metadata is prepared before the parent lock; stop after the pair for review. No B rerun is included.
|
||||
- **Infrastructure is qualified:**owned local Docker PostgreSQL18.6 at the pinned installed image, normal onboarding/migrations/API and final persisted-state scan passed with zero model/native/scenario work; exact resources removed. No further infrastructure health is needed for the unchanged lane. The proposed host SysV cleanup was never performed and is no longer needed. Source/build/native bytes remain bound to production baseline `8373d7867963674ff960d2345d0d28086f799bc7`; no core candidate is used as benchmark proof.
|
||||
- **Separate core PRs remain OPEN/unmerged, auto-merge off.** Manager's fresh checkpoint: [doctor #17847](https://github.com/paperclipai/paperclip/pull/17847), head `a042fb4a28f2e1574972798ed80de2e616daf9fc`; [invite URL logging #17848](https://github.com/paperclipai/paperclip/pull/17848), head `98a2432f88c33afacff2e014bd8c30630b26b339`. Both55checks successful,0 pending,0 failures; Greptile5/5 and0 unresolved threads at those heads. They do not change this campaign's measured baseline. The separate budget mutation/attribution candidate [#17857](https://github.com/paperclipai/paperclip/pull/17857), head `f970f6b01f0d6ec05c2e879d637fa5d0c44a0134`, is also OPEN/unmerged with CI pending; it does not replace the retained set-budget failure. No eval PR or merge is eligible.
|
||||
- **Separate core PRs remain OPEN/unmerged, auto-merge off.** Doctor [#17847](https://github.com/paperclipai/paperclip/pull/17847), head `a042fb4a28f2e1574972798ed80de2e616daf9fc`, and invite logging [#17848](https://github.com/paperclipai/paperclip/pull/17848), head `98a2432f88c33afacff2e014bd8c30630b26b339`, each have55 successful checks. Budget [#17857](https://github.com/paperclipai/paperclip/pull/17857), head `e5682d22fa8f7c0dd01ddb5e9cc42d941db6fe43`, and approval routing [#17858](https://github.com/paperclipai/paperclip/pull/17858), head `b60aaf19343389faf67580fb1ff7a561983bb4f5`, have CI/review pending at the latest manager checkpoint. None changes measured baseline8373. No eval PR or merge is eligible.
|
||||
- **Genuine missing inputs:**AgentMail/GitHub read-only discovery was authorized and completed; designated self-mail/recipient and repository choices remain pending. Standard provider-key source references exist, but validity and disposable remote/operator targets remain unqualified. No provider send, remote setup or account discovery is implied by authoring coverage.
|
||||
|
||||
Current live checkpoint:efbb batch20/30 completed,1passed/19failed, all completed cases cleanup passed and exact Docker removed. skills.find-and-install is running; authoritative ledger63 admitted scenarios/64 measurements (63 completed,1running),9 retained initial passes/54failed measurements. Exact sidecar:`initial-checkpoint-efbb-first20.json`.
|
||||
The efbb coordinator exited0 after30 retained measurements; no old launcher/writer remains. The authoritative ledger is the current manifest; immutable snapshot selection summaries are historical. The remaining initial-only draft covers172 IDs after this batch, including concrete missing-resource probes; its known two alias fixtures are now corrected in151488bf8c before future admission.
|
||||
|
||||
## Current measured continuation and held corrections
|
||||
|
||||
|
||||
@@ -11,7 +11,9 @@
|
||||
"requiredPasses": 196,
|
||||
"corePrs": [
|
||||
"https://github.com/paperclipai/paperclip/pull/17847",
|
||||
"https://github.com/paperclipai/paperclip/pull/17848"
|
||||
"https://github.com/paperclipai/paperclip/pull/17848",
|
||||
"https://github.com/paperclipai/paperclip/pull/17857",
|
||||
"https://github.com/paperclipai/paperclip/pull/17858"
|
||||
],
|
||||
"evalPr": null,
|
||||
"scenarios": [
|
||||
@@ -1155,9 +1157,9 @@
|
||||
"sourceSha256": "a5aa89ff1a55f7a2f580230c72f20d05420d272dbad365c4d7dd963016332bf3",
|
||||
"scenarioSha256": "f1aed5652250f3553f4089a6e1135cb3a67cdc055a87f9718ea037df516f5ae6",
|
||||
"owner": "eval_runtime",
|
||||
"readiness": "partial",
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.agents.existing-grok-bot",
|
||||
"blocker": "Disposable harness public/auth origin is pinned to loopback HTTP; actual owned HTTPS origin/tunnel binding is not yet supported. This is separate from the genuine existing Bot human OAuth/pairing prerequisite.",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null,
|
||||
@@ -1165,7 +1167,7 @@
|
||||
"lane": "source-build",
|
||||
"sourceBaselineRevision": "8373d7867963674ff960d2345d0d28086f799bc7"
|
||||
},
|
||||
"authoringStatus": "unfinished"
|
||||
"authoringStatus": "complete"
|
||||
},
|
||||
{
|
||||
"id": "agents.hire-claude-code",
|
||||
@@ -1871,9 +1873,9 @@
|
||||
"sourceSha256": "b87c34edd99aa5cce96cf2196327b66366ef24bbcea96dc8c86d404030372901",
|
||||
"scenarioSha256": "8e1b5269d92d5b144054e7ea43f08aed14f516dbe1bff0db73e971bb3bdb5f96",
|
||||
"owner": "eval_external",
|
||||
"readiness": "partial",
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.assistant-connections.connect-external-assistant",
|
||||
"blocker": "named-chatgpt-human-setup",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null,
|
||||
@@ -1881,7 +1883,7 @@
|
||||
"lane": "source-build",
|
||||
"sourceBaselineRevision": "8373d7867963674ff960d2345d0d28086f799bc7"
|
||||
},
|
||||
"authoringStatus": "unfinished"
|
||||
"authoringStatus": "complete"
|
||||
},
|
||||
{
|
||||
"id": "assistant-connections.consent-read-only",
|
||||
@@ -2793,8 +2795,8 @@
|
||||
}
|
||||
}
|
||||
],
|
||||
"qualifyingDefinitionDigest": "9bda9f51a289f716d370a9516f6a125960f397d4457e41b36253be52751e0324",
|
||||
"qualifyingGraderDigest": "9bda9f51a289f716d370a9516f6a125960f397d4457e41b36253be52751e0324",
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null,
|
||||
"productPolicy": {
|
||||
"lane": "source-build",
|
||||
"sourceBaselineRevision": "8373d7867963674ff960d2345d0d28086f799bc7"
|
||||
@@ -3074,8 +3076,8 @@
|
||||
"executionId": "chat-features.runner-codex-subscription.local.companies.archive-company",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null,
|
||||
"qualifyingDefinitionDigest": "e45461119cd11cbc7d9f3659a35bbd7a6c1ac72d5c5b2e4f187f97210ac8c7c7",
|
||||
"qualifyingGraderDigest": "e45461119cd11cbc7d9f3659a35bbd7a6c1ac72d5c5b2e4f187f97210ac8c7c7",
|
||||
"productPolicy": {
|
||||
"lane": "source-build",
|
||||
"sourceBaselineRevision": "8373d7867963674ff960d2345d0d28086f799bc7"
|
||||
@@ -3404,8 +3406,8 @@
|
||||
"executionId": "chat-features.runner-codex-subscription.local.connection-setup.choose-provider",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null,
|
||||
"qualifyingDefinitionDigest": "2650b5969af7a28ce555ba5ec92dfb6cf3eb3c21dfce0b8ab99be9c41e2ff772",
|
||||
"qualifyingGraderDigest": "2650b5969af7a28ce555ba5ec92dfb6cf3eb3c21dfce0b8ab99be9c41e2ff772",
|
||||
"productPolicy": {
|
||||
"lane": "source-build",
|
||||
"sourceBaselineRevision": "8373d7867963674ff960d2345d0d28086f799bc7"
|
||||
@@ -7942,10 +7944,10 @@
|
||||
"attempts": [
|
||||
{
|
||||
"ordinal": 1,
|
||||
"outcome": "running",
|
||||
"outcome": "failed",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.skills.find-and-install",
|
||||
"startedAt": "2026-10-27T09:55:56.981Z",
|
||||
"finishedAt": null,
|
||||
"finishedAt": "2026-10-27T09:58:55.045Z",
|
||||
"sourceRevision": "efbbb88a70dc6bcd0e6893702d8a482e46753fdf",
|
||||
"workingTreeDigest": "8b2855ee26357901bcbbfe2bb71e0f7440f6281c3e18a13a97c2b47d9502dd78",
|
||||
"definitionDigest": "564c8915f233ed3a0abe52a9220cdf7ff6bda7bb909fba1e16c3ca2cf4b88c31",
|
||||
@@ -7956,17 +7958,19 @@
|
||||
"environment": "local",
|
||||
"automaticRetries": 0,
|
||||
"correction": null,
|
||||
"classification": null,
|
||||
"observedFailure": null,
|
||||
"classification": "transient_infrastructure",
|
||||
"observedFailure": "Stopped waiting for skills.find-and-install: Observed provider run failed",
|
||||
"evidencePaths": [
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/next30-initial-batch-efbbb88a70.json"
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/next30-initial-batch-efbbb88a70.json",
|
||||
"tests/runner-e2e/results/catalog-initial-2026-10-27T09-55-56-981Z-skills-find-and-install/chat-features/runner-codex-subscription/local/skills.find-and-install/attempt-1/result.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/active-skills.find-and-install.json"
|
||||
],
|
||||
"usage": {
|
||||
"costUsd": null,
|
||||
"tokens": null,
|
||||
"coverage": "pending actual native usage receipts"
|
||||
"tokens": 749367,
|
||||
"coverage": "Actual heartbeat usage receipts, including cache; subscription cost unpriced unless complete billing; fixture costs excluded."
|
||||
},
|
||||
"cleanup": "pending",
|
||||
"cleanup": "passed",
|
||||
"corePrDependencies": [],
|
||||
"databaseLane": {
|
||||
"lane": "owned-docker-postgres",
|
||||
@@ -8014,9 +8018,60 @@
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.skills.fork-read-only",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null,
|
||||
"attempts": [
|
||||
{
|
||||
"ordinal": 1,
|
||||
"outcome": "failed",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.skills.fork-read-only",
|
||||
"startedAt": "2026-10-27T09:58:56.324Z",
|
||||
"finishedAt": "2026-10-27T09:59:23.910Z",
|
||||
"sourceRevision": "efbbb88a70dc6bcd0e6893702d8a482e46753fdf",
|
||||
"workingTreeDigest": "8b2855ee26357901bcbbfe2bb71e0f7440f6281c3e18a13a97c2b47d9502dd78",
|
||||
"definitionDigest": "7cd5ff26c8f3cb86e10937465c78c266c6e98f7cd9a4574b77447370fa4e13f3",
|
||||
"suiteDefinitionDigest": "3e4dd6bab0b717855ed15a758f00d9506ac29e53600b8c383304fa44e847e4a3",
|
||||
"graderDigest": "7cd5ff26c8f3cb86e10937465c78c266c6e98f7cd9a4574b77447370fa4e13f3",
|
||||
"model": "gpt-5.6-sol",
|
||||
"profile": "runner-codex-subscription",
|
||||
"environment": "local",
|
||||
"automaticRetries": 0,
|
||||
"correction": null,
|
||||
"classification": "candidate_failure",
|
||||
"observedFailure": "POST http://127.0.0.1:50130/api/companies/506d5d61-74f8-46d1-ba6c-e574e33fd168/skills/import returned 422: {\"error\":\"Remote skill sources must use HTTPS\",\"code\":\"skill_source_validation_failed\",\"details\":{\"code\":\"skill_source_validation_failed\"}}",
|
||||
"evidencePaths": [
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/next30-initial-batch-efbbb88a70.json",
|
||||
"tests/runner-e2e/results/catalog-initial-2026-10-27T09-58-56-324Z-skills-fork-read-only/chat-features/runner-codex-subscription/local/skills.fork-read-only/attempt-1/result.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/active-skills.fork-read-only.json"
|
||||
],
|
||||
"usage": {
|
||||
"costUsd": null,
|
||||
"tokens": 0,
|
||||
"coverage": "Actual heartbeat usage receipts, including cache; subscription cost unpriced unless complete billing; fixture costs excluded."
|
||||
},
|
||||
"cleanup": "passed",
|
||||
"corePrDependencies": [],
|
||||
"databaseLane": {
|
||||
"lane": "owned-docker-postgres",
|
||||
"image": "postgres@sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336",
|
||||
"imageId": "sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336",
|
||||
"version": "18.6-1.pgdg13+2"
|
||||
},
|
||||
"productIdentity": {
|
||||
"kind": "source-build",
|
||||
"productionSourceRevision": "8373d7867963674ff960d2345d0d28086f799bc7",
|
||||
"snapshotSourceRevision": "efbbb88a70dc6bcd0e6893702d8a482e46753fdf",
|
||||
"nativeBinarySha256": "9d8af6b85b836a03092ade2273be6f29f5c2f6ec67ee891d83ee9693ba39d3b4",
|
||||
"workspaceBuildDigest": "222e10a8a2a20d3015ec4470b952e2dd9b927dfe9390c4a5ed1e19d740c0e35b",
|
||||
"receiptPaths": [
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-efbbb88a70.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-10d0ce63610.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-0fdcb539a8.json"
|
||||
],
|
||||
"coreCandidate": false
|
||||
}
|
||||
}
|
||||
],
|
||||
"qualifyingDefinitionDigest": "7cd5ff26c8f3cb86e10937465c78c266c6e98f7cd9a4574b77447370fa4e13f3",
|
||||
"qualifyingGraderDigest": "7cd5ff26c8f3cb86e10937465c78c266c6e98f7cd9a4574b77447370fa4e13f3",
|
||||
"productPolicy": {
|
||||
"lane": "source-build",
|
||||
"sourceBaselineRevision": "8373d7867963674ff960d2345d0d28086f799bc7"
|
||||
@@ -8039,9 +8094,60 @@
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.skills.review-update",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null,
|
||||
"attempts": [
|
||||
{
|
||||
"ordinal": 1,
|
||||
"outcome": "failed",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.skills.review-update",
|
||||
"startedAt": "2026-10-27T09:59:24.725Z",
|
||||
"finishedAt": "2026-10-27T10:00:57.760Z",
|
||||
"sourceRevision": "efbbb88a70dc6bcd0e6893702d8a482e46753fdf",
|
||||
"workingTreeDigest": "8ebfbe92ac66c83a6d87907c2ccd9581ee9c32a13b42ba2fa0eb2f4ae36b9d47",
|
||||
"definitionDigest": "4ec6118528d2661a3f801f000c22757fc743282d47bf7bf6ee9a7fa71b38ab7f",
|
||||
"suiteDefinitionDigest": "3e4dd6bab0b717855ed15a758f00d9506ac29e53600b8c383304fa44e847e4a3",
|
||||
"graderDigest": "4ec6118528d2661a3f801f000c22757fc743282d47bf7bf6ee9a7fa71b38ab7f",
|
||||
"model": "gpt-5.6-sol",
|
||||
"profile": "runner-codex-subscription",
|
||||
"environment": "local",
|
||||
"automaticRetries": 0,
|
||||
"correction": null,
|
||||
"classification": "candidate_failure",
|
||||
"observedFailure": "The initial actual diff review was wrong or installed prematurely; later updates cannot repair it",
|
||||
"evidencePaths": [
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/next30-initial-batch-efbbb88a70.json",
|
||||
"tests/runner-e2e/results/catalog-initial-2026-10-27T09-59-24-725Z-skills-review-update/chat-features/runner-codex-subscription/local/skills.review-update/attempt-1/result.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/active-skills.review-update.json"
|
||||
],
|
||||
"usage": {
|
||||
"costUsd": null,
|
||||
"tokens": 416837,
|
||||
"coverage": "Actual heartbeat usage receipts, including cache; subscription cost unpriced unless complete billing; fixture costs excluded."
|
||||
},
|
||||
"cleanup": "passed",
|
||||
"corePrDependencies": [],
|
||||
"databaseLane": {
|
||||
"lane": "owned-docker-postgres",
|
||||
"image": "postgres@sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336",
|
||||
"imageId": "sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336",
|
||||
"version": "18.6-1.pgdg13+2"
|
||||
},
|
||||
"productIdentity": {
|
||||
"kind": "source-build",
|
||||
"productionSourceRevision": "8373d7867963674ff960d2345d0d28086f799bc7",
|
||||
"snapshotSourceRevision": "efbbb88a70dc6bcd0e6893702d8a482e46753fdf",
|
||||
"nativeBinarySha256": "9d8af6b85b836a03092ade2273be6f29f5c2f6ec67ee891d83ee9693ba39d3b4",
|
||||
"workspaceBuildDigest": "222e10a8a2a20d3015ec4470b952e2dd9b927dfe9390c4a5ed1e19d740c0e35b",
|
||||
"receiptPaths": [
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-efbbb88a70.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-10d0ce63610.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-0fdcb539a8.json"
|
||||
],
|
||||
"coreCandidate": false
|
||||
}
|
||||
}
|
||||
],
|
||||
"qualifyingDefinitionDigest": "4ec6118528d2661a3f801f000c22757fc743282d47bf7bf6ee9a7fa71b38ab7f",
|
||||
"qualifyingGraderDigest": "4ec6118528d2661a3f801f000c22757fc743282d47bf7bf6ee9a7fa71b38ab7f",
|
||||
"productPolicy": {
|
||||
"lane": "source-build",
|
||||
"sourceBaselineRevision": "8373d7867963674ff960d2345d0d28086f799bc7"
|
||||
@@ -8065,9 +8171,60 @@
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.skills.studio-test",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null,
|
||||
"attempts": [
|
||||
{
|
||||
"ordinal": 1,
|
||||
"outcome": "failed",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.skills.studio-test",
|
||||
"startedAt": "2026-10-27T10:00:58.667Z",
|
||||
"finishedAt": "2026-10-27T10:05:37.912Z",
|
||||
"sourceRevision": "efbbb88a70dc6bcd0e6893702d8a482e46753fdf",
|
||||
"workingTreeDigest": "4a159f02bea829e2468aac8b076efb6770e7a57b6f406ff273d8d1b49ca7189b",
|
||||
"definitionDigest": "8617ffdec76e0e9319f6ce4538a778ec1fca74fe63523e8515438d8b376a04b2",
|
||||
"suiteDefinitionDigest": "3e4dd6bab0b717855ed15a758f00d9506ac29e53600b8c383304fa44e847e4a3",
|
||||
"graderDigest": "8617ffdec76e0e9319f6ce4538a778ec1fca74fe63523e8515438d8b376a04b2",
|
||||
"model": "gpt-5.6-sol",
|
||||
"profile": "runner-codex-subscription",
|
||||
"environment": "local",
|
||||
"automaticRetries": 0,
|
||||
"correction": null,
|
||||
"classification": "candidate_failure",
|
||||
"observedFailure": "Timed out waiting for actual bounded skill Studio output; the observed state did not satisfy the condition. See the saved state evidence.",
|
||||
"evidencePaths": [
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/next30-initial-batch-efbbb88a70.json",
|
||||
"tests/runner-e2e/results/catalog-initial-2026-10-27T10-00-58-667Z-skills-studio-test/chat-features/runner-codex-subscription/local/skills.studio-test/attempt-1/result.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/active-skills.studio-test.json"
|
||||
],
|
||||
"usage": {
|
||||
"costUsd": null,
|
||||
"tokens": 395976,
|
||||
"coverage": "Actual heartbeat usage receipts, including cache; subscription cost unpriced unless complete billing; fixture costs excluded."
|
||||
},
|
||||
"cleanup": "passed",
|
||||
"corePrDependencies": [],
|
||||
"databaseLane": {
|
||||
"lane": "owned-docker-postgres",
|
||||
"image": "postgres@sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336",
|
||||
"imageId": "sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336",
|
||||
"version": "18.6-1.pgdg13+2"
|
||||
},
|
||||
"productIdentity": {
|
||||
"kind": "source-build",
|
||||
"productionSourceRevision": "8373d7867963674ff960d2345d0d28086f799bc7",
|
||||
"snapshotSourceRevision": "efbbb88a70dc6bcd0e6893702d8a482e46753fdf",
|
||||
"nativeBinarySha256": "9d8af6b85b836a03092ade2273be6f29f5c2f6ec67ee891d83ee9693ba39d3b4",
|
||||
"workspaceBuildDigest": "222e10a8a2a20d3015ec4470b952e2dd9b927dfe9390c4a5ed1e19d740c0e35b",
|
||||
"receiptPaths": [
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-efbbb88a70.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-10d0ce63610.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-0fdcb539a8.json"
|
||||
],
|
||||
"coreCandidate": false
|
||||
}
|
||||
}
|
||||
],
|
||||
"qualifyingDefinitionDigest": "8617ffdec76e0e9319f6ce4538a778ec1fca74fe63523e8515438d8b376a04b2",
|
||||
"qualifyingGraderDigest": "8617ffdec76e0e9319f6ce4538a778ec1fca74fe63523e8515438d8b376a04b2",
|
||||
"productPolicy": {
|
||||
"lane": "source-build",
|
||||
"sourceBaselineRevision": "8373d7867963674ff960d2345d0d28086f799bc7"
|
||||
@@ -8115,9 +8272,60 @@
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.status-cards.create-card",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null,
|
||||
"attempts": [
|
||||
{
|
||||
"ordinal": 1,
|
||||
"outcome": "failed",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.status-cards.create-card",
|
||||
"startedAt": "2026-10-27T10:05:38.807Z",
|
||||
"finishedAt": "2026-10-27T10:07:41.947Z",
|
||||
"sourceRevision": "efbbb88a70dc6bcd0e6893702d8a482e46753fdf",
|
||||
"workingTreeDigest": "cb16f7db62baead80c3efb724dcc597bebbca516ec20e36cd000e0f073d3184e",
|
||||
"definitionDigest": "140871b4be05e4e9a80dfe97c7d5a7a49b78abc2e6aea95f4b1753eab2b0862d",
|
||||
"suiteDefinitionDigest": "3e4dd6bab0b717855ed15a758f00d9506ac29e53600b8c383304fa44e847e4a3",
|
||||
"graderDigest": "140871b4be05e4e9a80dfe97c7d5a7a49b78abc2e6aea95f4b1753eab2b0862d",
|
||||
"model": "gpt-5.6-sol",
|
||||
"profile": "runner-codex-subscription",
|
||||
"environment": "local",
|
||||
"automaticRetries": 0,
|
||||
"correction": null,
|
||||
"classification": "candidate_failure",
|
||||
"observedFailure": "Stopped waiting for status-cards.create-card: Settled scenario failed: one-agent-authored-valid-card, compiled-launch-watch-scope, inspectable-card-link, configuration-not-false-summary, catalog-bounded-native-runs",
|
||||
"evidencePaths": [
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/next30-initial-batch-efbbb88a70.json",
|
||||
"tests/runner-e2e/results/catalog-initial-2026-10-27T10-05-38-807Z-status-cards-create-card/chat-features/runner-codex-subscription/local/status-cards.create-card/attempt-1/result.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/active-status-cards.create-card.json"
|
||||
],
|
||||
"usage": {
|
||||
"costUsd": null,
|
||||
"tokens": 331038,
|
||||
"coverage": "Actual heartbeat usage receipts, including cache; subscription cost unpriced unless complete billing; fixture costs excluded."
|
||||
},
|
||||
"cleanup": "passed",
|
||||
"corePrDependencies": [],
|
||||
"databaseLane": {
|
||||
"lane": "owned-docker-postgres",
|
||||
"image": "postgres@sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336",
|
||||
"imageId": "sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336",
|
||||
"version": "18.6-1.pgdg13+2"
|
||||
},
|
||||
"productIdentity": {
|
||||
"kind": "source-build",
|
||||
"productionSourceRevision": "8373d7867963674ff960d2345d0d28086f799bc7",
|
||||
"snapshotSourceRevision": "efbbb88a70dc6bcd0e6893702d8a482e46753fdf",
|
||||
"nativeBinarySha256": "9d8af6b85b836a03092ade2273be6f29f5c2f6ec67ee891d83ee9693ba39d3b4",
|
||||
"workspaceBuildDigest": "222e10a8a2a20d3015ec4470b952e2dd9b927dfe9390c4a5ed1e19d740c0e35b",
|
||||
"receiptPaths": [
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-efbbb88a70.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-10d0ce63610.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-0fdcb539a8.json"
|
||||
],
|
||||
"coreCandidate": false
|
||||
}
|
||||
}
|
||||
],
|
||||
"qualifyingDefinitionDigest": "140871b4be05e4e9a80dfe97c7d5a7a49b78abc2e6aea95f4b1753eab2b0862d",
|
||||
"qualifyingGraderDigest": "140871b4be05e4e9a80dfe97c7d5a7a49b78abc2e6aea95f4b1753eab2b0862d",
|
||||
"productPolicy": {
|
||||
"lane": "source-build",
|
||||
"sourceBaselineRevision": "8373d7867963674ff960d2345d0d28086f799bc7"
|
||||
@@ -8166,9 +8374,60 @@
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.status-cards.other-authors-card",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null,
|
||||
"attempts": [
|
||||
{
|
||||
"ordinal": 1,
|
||||
"outcome": "failed",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.status-cards.other-authors-card",
|
||||
"startedAt": "2026-10-27T10:07:42.857Z",
|
||||
"finishedAt": "2026-10-27T10:10:53.049Z",
|
||||
"sourceRevision": "efbbb88a70dc6bcd0e6893702d8a482e46753fdf",
|
||||
"workingTreeDigest": "cb16f7db62baead80c3efb724dcc597bebbca516ec20e36cd000e0f073d3184e",
|
||||
"definitionDigest": "cef88eb4af3ae728e2ad2dd920c8712874e40f13105183aee288123ea56f0f11",
|
||||
"suiteDefinitionDigest": "3e4dd6bab0b717855ed15a758f00d9506ac29e53600b8c383304fa44e847e4a3",
|
||||
"graderDigest": "cef88eb4af3ae728e2ad2dd920c8712874e40f13105183aee288123ea56f0f11",
|
||||
"model": "gpt-5.6-sol",
|
||||
"profile": "runner-codex-subscription",
|
||||
"environment": "local",
|
||||
"automaticRetries": 0,
|
||||
"correction": null,
|
||||
"classification": "candidate_failure",
|
||||
"observedFailure": "Stopped waiting for status-cards.other-authors-card: Scenario exceeded its declared run bound",
|
||||
"evidencePaths": [
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/next30-initial-batch-efbbb88a70.json",
|
||||
"tests/runner-e2e/results/catalog-initial-2026-10-27T10-07-42-857Z-status-cards-other-authors-card/chat-features/runner-codex-subscription/local/status-cards.other-authors-card/attempt-1/result.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/active-status-cards.other-authors-card.json"
|
||||
],
|
||||
"usage": {
|
||||
"costUsd": null,
|
||||
"tokens": 854348,
|
||||
"coverage": "Actual heartbeat usage receipts, including cache; subscription cost unpriced unless complete billing; fixture costs excluded."
|
||||
},
|
||||
"cleanup": "passed",
|
||||
"corePrDependencies": [],
|
||||
"databaseLane": {
|
||||
"lane": "owned-docker-postgres",
|
||||
"image": "postgres@sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336",
|
||||
"imageId": "sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336",
|
||||
"version": "18.6-1.pgdg13+2"
|
||||
},
|
||||
"productIdentity": {
|
||||
"kind": "source-build",
|
||||
"productionSourceRevision": "8373d7867963674ff960d2345d0d28086f799bc7",
|
||||
"snapshotSourceRevision": "efbbb88a70dc6bcd0e6893702d8a482e46753fdf",
|
||||
"nativeBinarySha256": "9d8af6b85b836a03092ade2273be6f29f5c2f6ec67ee891d83ee9693ba39d3b4",
|
||||
"workspaceBuildDigest": "222e10a8a2a20d3015ec4470b952e2dd9b927dfe9390c4a5ed1e19d740c0e35b",
|
||||
"receiptPaths": [
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-efbbb88a70.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-10d0ce63610.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-0fdcb539a8.json"
|
||||
],
|
||||
"coreCandidate": false
|
||||
}
|
||||
}
|
||||
],
|
||||
"qualifyingDefinitionDigest": "cef88eb4af3ae728e2ad2dd920c8712874e40f13105183aee288123ea56f0f11",
|
||||
"qualifyingGraderDigest": "cef88eb4af3ae728e2ad2dd920c8712874e40f13105183aee288123ea56f0f11",
|
||||
"productPolicy": {
|
||||
"lane": "source-build",
|
||||
"sourceBaselineRevision": "8373d7867963674ff960d2345d0d28086f799bc7"
|
||||
@@ -8523,7 +8782,58 @@
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.task-coordination.reject-dependency-cycle",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"attempts": [
|
||||
{
|
||||
"ordinal": 1,
|
||||
"outcome": "passed",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.task-coordination.reject-dependency-cycle",
|
||||
"startedAt": "2026-10-27T10:10:54.177Z",
|
||||
"finishedAt": "2026-10-27T10:12:31.491Z",
|
||||
"sourceRevision": "efbbb88a70dc6bcd0e6893702d8a482e46753fdf",
|
||||
"workingTreeDigest": "83a52bc5d8b24603ea6895f88ef8109a3e8ca8d0ada3721a3ce3f2dc0e19faed",
|
||||
"definitionDigest": "220181151301273b69fcfda0b47cb02f69206671c19daf56e3ca632075b95dce",
|
||||
"suiteDefinitionDigest": "3e4dd6bab0b717855ed15a758f00d9506ac29e53600b8c383304fa44e847e4a3",
|
||||
"graderDigest": "220181151301273b69fcfda0b47cb02f69206671c19daf56e3ca632075b95dce",
|
||||
"model": "gpt-5.6-sol",
|
||||
"profile": "runner-codex-subscription",
|
||||
"environment": "local",
|
||||
"automaticRetries": 0,
|
||||
"correction": null,
|
||||
"classification": "provisional_initial_pass_pending_final_closure",
|
||||
"observedFailure": null,
|
||||
"evidencePaths": [
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/next30-initial-batch-efbbb88a70.json",
|
||||
"tests/runner-e2e/results/catalog-initial-2026-10-27T10-10-54-177Z-task-coordination-reject-dependency-cycle/chat-features/runner-codex-subscription/local/task-coordination.reject-dependency-cycle/attempt-1/result.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/active-task-coordination.reject-dependency-cycle.json"
|
||||
],
|
||||
"usage": {
|
||||
"costUsd": null,
|
||||
"tokens": 454268,
|
||||
"coverage": "Actual heartbeat usage receipts, including cache; subscription cost unpriced unless complete billing; fixture costs excluded."
|
||||
},
|
||||
"cleanup": "passed",
|
||||
"corePrDependencies": [],
|
||||
"databaseLane": {
|
||||
"lane": "owned-docker-postgres",
|
||||
"image": "postgres@sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336",
|
||||
"imageId": "sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336",
|
||||
"version": "18.6-1.pgdg13+2"
|
||||
},
|
||||
"productIdentity": {
|
||||
"kind": "source-build",
|
||||
"productionSourceRevision": "8373d7867963674ff960d2345d0d28086f799bc7",
|
||||
"snapshotSourceRevision": "efbbb88a70dc6bcd0e6893702d8a482e46753fdf",
|
||||
"nativeBinarySha256": "9d8af6b85b836a03092ade2273be6f29f5c2f6ec67ee891d83ee9693ba39d3b4",
|
||||
"workspaceBuildDigest": "222e10a8a2a20d3015ec4470b952e2dd9b927dfe9390c4a5ed1e19d740c0e35b",
|
||||
"receiptPaths": [
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-efbbb88a70.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-10d0ce63610.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-0fdcb539a8.json"
|
||||
],
|
||||
"coreCandidate": false
|
||||
}
|
||||
}
|
||||
],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null,
|
||||
"productPolicy": {
|
||||
@@ -9220,9 +9530,60 @@
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.teams.find-team",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null,
|
||||
"attempts": [
|
||||
{
|
||||
"ordinal": 1,
|
||||
"outcome": "failed",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.teams.find-team",
|
||||
"startedAt": "2026-10-27T10:12:32.354Z",
|
||||
"finishedAt": "2026-10-27T10:13:50.001Z",
|
||||
"sourceRevision": "efbbb88a70dc6bcd0e6893702d8a482e46753fdf",
|
||||
"workingTreeDigest": "1b21e3c0fd1340747b39de72ce2ac0b3fdb01df4de11298c4281fe4892eb778d",
|
||||
"definitionDigest": "a08e94733ee8f8ac25dd879e03e8dac3900332d53107081d9c42b436bf0c9880",
|
||||
"suiteDefinitionDigest": "3e4dd6bab0b717855ed15a758f00d9506ac29e53600b8c383304fa44e847e4a3",
|
||||
"graderDigest": "a08e94733ee8f8ac25dd879e03e8dac3900332d53107081d9c42b436bf0c9880",
|
||||
"model": "gpt-5.6-sol",
|
||||
"profile": "runner-codex-subscription",
|
||||
"environment": "local",
|
||||
"automaticRetries": 0,
|
||||
"correction": null,
|
||||
"classification": "candidate_failure",
|
||||
"observedFailure": "Stopped waiting for teams.find-team: Settled scenario failed: existing-package-grounding, package-requirements-described",
|
||||
"evidencePaths": [
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/next30-initial-batch-efbbb88a70.json",
|
||||
"tests/runner-e2e/results/catalog-initial-2026-10-27T10-12-32-354Z-teams-find-team/chat-features/runner-codex-subscription/local/teams.find-team/attempt-1/result.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/active-teams.find-team.json"
|
||||
],
|
||||
"usage": {
|
||||
"costUsd": null,
|
||||
"tokens": 191246,
|
||||
"coverage": "Actual heartbeat usage receipts, including cache; subscription cost unpriced unless complete billing; fixture costs excluded."
|
||||
},
|
||||
"cleanup": "passed",
|
||||
"corePrDependencies": [],
|
||||
"databaseLane": {
|
||||
"lane": "owned-docker-postgres",
|
||||
"image": "postgres@sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336",
|
||||
"imageId": "sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336",
|
||||
"version": "18.6-1.pgdg13+2"
|
||||
},
|
||||
"productIdentity": {
|
||||
"kind": "source-build",
|
||||
"productionSourceRevision": "8373d7867963674ff960d2345d0d28086f799bc7",
|
||||
"snapshotSourceRevision": "efbbb88a70dc6bcd0e6893702d8a482e46753fdf",
|
||||
"nativeBinarySha256": "9d8af6b85b836a03092ade2273be6f29f5c2f6ec67ee891d83ee9693ba39d3b4",
|
||||
"workspaceBuildDigest": "222e10a8a2a20d3015ec4470b952e2dd9b927dfe9390c4a5ed1e19d740c0e35b",
|
||||
"receiptPaths": [
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-efbbb88a70.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-10d0ce63610.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-0fdcb539a8.json"
|
||||
],
|
||||
"coreCandidate": false
|
||||
}
|
||||
}
|
||||
],
|
||||
"qualifyingDefinitionDigest": "a08e94733ee8f8ac25dd879e03e8dac3900332d53107081d9c42b436bf0c9880",
|
||||
"qualifyingGraderDigest": "a08e94733ee8f8ac25dd879e03e8dac3900332d53107081d9c42b436bf0c9880",
|
||||
"productPolicy": {
|
||||
"lane": "source-build",
|
||||
"sourceBaselineRevision": "8373d7867963674ff960d2345d0d28086f799bc7"
|
||||
@@ -9271,9 +9632,60 @@
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.teams.preview-install",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null,
|
||||
"attempts": [
|
||||
{
|
||||
"ordinal": 1,
|
||||
"outcome": "failed",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.teams.preview-install",
|
||||
"startedAt": "2026-10-27T10:13:50.886Z",
|
||||
"finishedAt": "2026-10-27T10:15:55.859Z",
|
||||
"sourceRevision": "efbbb88a70dc6bcd0e6893702d8a482e46753fdf",
|
||||
"workingTreeDigest": "1b21e3c0fd1340747b39de72ce2ac0b3fdb01df4de11298c4281fe4892eb778d",
|
||||
"definitionDigest": "c72ce96d035526a28c3a8f8a6ffc12b4f65422f9582723356331a97b2dfa81c7",
|
||||
"suiteDefinitionDigest": "3e4dd6bab0b717855ed15a758f00d9506ac29e53600b8c383304fa44e847e4a3",
|
||||
"graderDigest": "c72ce96d035526a28c3a8f8a6ffc12b4f65422f9582723356331a97b2dfa81c7",
|
||||
"model": "gpt-5.6-sol",
|
||||
"profile": "runner-codex-subscription",
|
||||
"environment": "local",
|
||||
"automaticRetries": 0,
|
||||
"correction": null,
|
||||
"classification": "transient_infrastructure",
|
||||
"observedFailure": "Stopped waiting for teams.preview-install: Settled scenario failed: existing-package-grounding, package-requirements-described, preview-resources-described",
|
||||
"evidencePaths": [
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/next30-initial-batch-efbbb88a70.json",
|
||||
"tests/runner-e2e/results/catalog-initial-2026-10-27T10-13-50-886Z-teams-preview-install/chat-features/runner-codex-subscription/local/teams.preview-install/attempt-1/result.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/active-teams.preview-install.json"
|
||||
],
|
||||
"usage": {
|
||||
"costUsd": null,
|
||||
"tokens": 413504,
|
||||
"coverage": "Actual heartbeat usage receipts, including cache; subscription cost unpriced unless complete billing; fixture costs excluded."
|
||||
},
|
||||
"cleanup": "passed",
|
||||
"corePrDependencies": [],
|
||||
"databaseLane": {
|
||||
"lane": "owned-docker-postgres",
|
||||
"image": "postgres@sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336",
|
||||
"imageId": "sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336",
|
||||
"version": "18.6-1.pgdg13+2"
|
||||
},
|
||||
"productIdentity": {
|
||||
"kind": "source-build",
|
||||
"productionSourceRevision": "8373d7867963674ff960d2345d0d28086f799bc7",
|
||||
"snapshotSourceRevision": "efbbb88a70dc6bcd0e6893702d8a482e46753fdf",
|
||||
"nativeBinarySha256": "9d8af6b85b836a03092ade2273be6f29f5c2f6ec67ee891d83ee9693ba39d3b4",
|
||||
"workspaceBuildDigest": "222e10a8a2a20d3015ec4470b952e2dd9b927dfe9390c4a5ed1e19d740c0e35b",
|
||||
"receiptPaths": [
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-efbbb88a70.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-10d0ce63610.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-0fdcb539a8.json"
|
||||
],
|
||||
"coreCandidate": false
|
||||
}
|
||||
}
|
||||
],
|
||||
"qualifyingDefinitionDigest": "c72ce96d035526a28c3a8f8a6ffc12b4f65422f9582723356331a97b2dfa81c7",
|
||||
"qualifyingGraderDigest": "c72ce96d035526a28c3a8f8a6ffc12b4f65422f9582723356331a97b2dfa81c7",
|
||||
"productPolicy": {
|
||||
"lane": "source-build",
|
||||
"sourceBaselineRevision": "8373d7867963674ff960d2345d0d28086f799bc7"
|
||||
@@ -9321,9 +9733,60 @@
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.workspaces.inspect-git",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null,
|
||||
"attempts": [
|
||||
{
|
||||
"ordinal": 1,
|
||||
"outcome": "failed",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.workspaces.inspect-git",
|
||||
"startedAt": "2026-10-27T10:15:56.871Z",
|
||||
"finishedAt": "2026-10-27T10:19:30.489Z",
|
||||
"sourceRevision": "efbbb88a70dc6bcd0e6893702d8a482e46753fdf",
|
||||
"workingTreeDigest": "21b57838d47cc5455149fd1ac93ccccd470b9492fce0f32709f120728e176eb8",
|
||||
"definitionDigest": "eff110b7d8532253ec9e60ae23f799a2c0add2bf77122b33f0f947e83d93a5b3",
|
||||
"suiteDefinitionDigest": "3e4dd6bab0b717855ed15a758f00d9506ac29e53600b8c383304fa44e847e4a3",
|
||||
"graderDigest": "eff110b7d8532253ec9e60ae23f799a2c0add2bf77122b33f0f947e83d93a5b3",
|
||||
"model": "gpt-5.6-sol",
|
||||
"profile": "runner-codex-subscription",
|
||||
"environment": "local",
|
||||
"automaticRetries": 0,
|
||||
"correction": null,
|
||||
"classification": "candidate_failure",
|
||||
"observedFailure": "Stopped waiting for workspaces.inspect-git: Settled scenario failed: actual-diff-summary-with-file-references",
|
||||
"evidencePaths": [
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/next30-initial-batch-efbbb88a70.json",
|
||||
"tests/runner-e2e/results/catalog-initial-2026-10-27T10-15-56-871Z-workspaces-inspect-git/chat-features/runner-codex-subscription/local/workspaces.inspect-git/attempt-1/result.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/active-workspaces.inspect-git.json"
|
||||
],
|
||||
"usage": {
|
||||
"costUsd": null,
|
||||
"tokens": 465814,
|
||||
"coverage": "Actual heartbeat usage receipts, including cache; subscription cost unpriced unless complete billing; fixture costs excluded."
|
||||
},
|
||||
"cleanup": "passed",
|
||||
"corePrDependencies": [],
|
||||
"databaseLane": {
|
||||
"lane": "owned-docker-postgres",
|
||||
"image": "postgres@sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336",
|
||||
"imageId": "sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336",
|
||||
"version": "18.6-1.pgdg13+2"
|
||||
},
|
||||
"productIdentity": {
|
||||
"kind": "source-build",
|
||||
"productionSourceRevision": "8373d7867963674ff960d2345d0d28086f799bc7",
|
||||
"snapshotSourceRevision": "efbbb88a70dc6bcd0e6893702d8a482e46753fdf",
|
||||
"nativeBinarySha256": "9d8af6b85b836a03092ade2273be6f29f5c2f6ec67ee891d83ee9693ba39d3b4",
|
||||
"workspaceBuildDigest": "222e10a8a2a20d3015ec4470b952e2dd9b927dfe9390c4a5ed1e19d740c0e35b",
|
||||
"receiptPaths": [
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-efbbb88a70.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-10d0ce63610.json",
|
||||
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-0fdcb539a8.json"
|
||||
],
|
||||
"coreCandidate": false
|
||||
}
|
||||
}
|
||||
],
|
||||
"qualifyingDefinitionDigest": "eff110b7d8532253ec9e60ae23f799a2c0add2bf77122b33f0f947e83d93a5b3",
|
||||
"qualifyingGraderDigest": "eff110b7d8532253ec9e60ae23f799a2c0add2bf77122b33f0f947e83d93a5b3",
|
||||
"productPolicy": {
|
||||
"lane": "source-build",
|
||||
"sourceBaselineRevision": "8373d7867963674ff960d2345d0d28086f799bc7"
|
||||
@@ -9472,9 +9935,9 @@
|
||||
"sourceSha256": "6b4be6151f155981ae7a68ac69c3a31420d54fe33f1ac567de0db73b651b3ae7",
|
||||
"scenarioSha256": "e00dc78e1cc306e89ca407c275f2a06d9dbbde3cec2cfe176a37e4090c81a063",
|
||||
"owner": "eval_runtime",
|
||||
"readiness": "partial",
|
||||
"readiness": "implemented",
|
||||
"executionId": "chat-features.runner-codex-subscription.local.workspaces.private-repository-no-access",
|
||||
"blocker": "Unfinished required after-grant conditional: the selected metadata-only fixture has no executable protected human GitHub account/installation completion callback and therefore cannot yet resume preparation/fix the header after authorization. Reserved act preserves the actual initial request and probes real setup options, then stops instead of fabricating a grant. Missing designated disposable repo is a separate external prerequisite, not this authoring gap.",
|
||||
"blocker": null,
|
||||
"attempts": [],
|
||||
"qualifyingDefinitionDigest": null,
|
||||
"qualifyingGraderDigest": null,
|
||||
@@ -9482,7 +9945,7 @@
|
||||
"lane": "source-build",
|
||||
"sourceBaselineRevision": "8373d7867963674ff960d2345d0d28086f799bc7"
|
||||
},
|
||||
"authoringStatus": "unfinished"
|
||||
"authoringStatus": "complete"
|
||||
},
|
||||
{
|
||||
"id": "workspaces.recover-lost-sandbox",
|
||||
@@ -9512,11 +9975,11 @@
|
||||
}
|
||||
],
|
||||
"implementationCheckpoint": {
|
||||
"sourceRevision": "406eb610dd95604079838220f29fb765977a8910",
|
||||
"descriptorSha256": "abb024741de821bba2085762d4661ceeef0e5564bfc366e0d6fa59533f75e1b3",
|
||||
"sourceRevision": "b65cce861b5667f2424bdfacbdfd610fa697ba60",
|
||||
"descriptorSha256": "a906bdf1088967ab2579d19e477c825933e227f28ddea40cf63c90a2b985d631",
|
||||
"authored": 244,
|
||||
"complete": 235,
|
||||
"partial": 9,
|
||||
"complete": 238,
|
||||
"partial": 6,
|
||||
"unimplemented": 0
|
||||
}
|
||||
}
|
||||
@@ -535,6 +535,22 @@
|
||||
"scenarioId": "agents.existing-grok-bot",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/runtime-grok.ts",
|
||||
"chat-catalog-domains/runtime-grok-operator.ts",
|
||||
"chat-catalog-domains/runtime-grok-operator-worker.ts",
|
||||
"chat-catalog-domains/runtime-grok-operator-contract.ts",
|
||||
"chat-runtime-private-requirements.ts",
|
||||
"public-mcp-flow.ts",
|
||||
"chat-public-origin.ts",
|
||||
"chat-catalog-domains/external-public-signin.ts",
|
||||
"chat-catalog-domains/external-public-signin-worker.ts",
|
||||
"chat-catalog-domains/external-public-signin-contract.ts",
|
||||
"../../ui/src/pages/Auth.tsx",
|
||||
"../../ui/src/pages/McpConnect.tsx",
|
||||
"../../ui/src/components/AgentSetupPrompt.tsx",
|
||||
"../../ui/src/lib/grok-bot-setup-prompt.ts",
|
||||
"../../ui/src/components/new-agent/GrokBotSetup.tsx",
|
||||
"../../ui/src/components/NewAgentDialog.tsx",
|
||||
"../../ui/src/components/new-agent/AgentBasicsDialog.tsx",
|
||||
"chat-catalog-domains/runtime-private.ts",
|
||||
"chat-catalog-domains/runtime-support.ts",
|
||||
"chat-catalog-domains/external-private-browser.ts",
|
||||
@@ -543,14 +559,18 @@
|
||||
"../../server/src/routes/grok-bot.ts",
|
||||
"../../server/src/services/grok-bot/routine.ts",
|
||||
"../../server/src/services/dot-runner-broker.ts",
|
||||
"../../server/src/modules/external-agents/dot.ts",
|
||||
"../../server/src/services/public-mcp/oauth.ts",
|
||||
"../../ui/src/components/AgentsToolbar.tsx",
|
||||
"../../ui/src/pages/Agents.tsx",
|
||||
"../../ui/src/components/new-agent/ExternalAgentInviteDialog.tsx",
|
||||
"../../ui/src/components/new-agent/ExternalAgentInviteContent.tsx",
|
||||
"../../ui/src/components/GrokBotConnection.tsx",
|
||||
"../../ui/src/components/new-agent/GrokBotSetupContent.tsx",
|
||||
"chat-scenarios/agents.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.runtime.v2",
|
||||
"coverageGaps": [
|
||||
"Disposable harness public/auth origin is pinned to loopback HTTP; actual owned HTTPS origin/tunnel binding is not yet supported. This is separate from the genuine existing Bot human OAuth/pairing prerequisite."
|
||||
],
|
||||
"coverageGaps": [],
|
||||
"expectedRunCount": 2,
|
||||
"expectedHumanMessageCount": 2,
|
||||
"turnTimeoutMs": 120000,
|
||||
@@ -838,6 +858,10 @@
|
||||
{
|
||||
"scenarioId": "app-permissions.new-quarantined-action",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/external-app-fixture.ts",
|
||||
"../../packages/shared/src/connection-search.ts",
|
||||
"../../server/src/routes/tool-access.ts",
|
||||
"../../server/src/services/tool-access/catalog/publication.ts",
|
||||
"chat-catalog-domains/external-app-catalog.ts",
|
||||
"chat-catalog-domains/external-provider.ts",
|
||||
"chat-catalog-domains/external-support.ts",
|
||||
@@ -856,6 +880,10 @@
|
||||
"scenarioId": "app-permissions.read-allowed-write-off",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/external-app-permissions.ts",
|
||||
"chat-catalog-domains/external-app-fixture.ts",
|
||||
"../../packages/shared/src/connection-search.ts",
|
||||
"../../server/src/services/connection-intents.ts",
|
||||
"../../server/src/routes/tool-access.ts",
|
||||
"chat-catalog-domains/external-provider.ts",
|
||||
"chat-catalog-domains/external-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
@@ -919,22 +947,44 @@
|
||||
"exportName": "appTestExternalCases"
|
||||
},
|
||||
{
|
||||
"requiredCredentials": [],
|
||||
"externalInput": {
|
||||
"provider": "chatgpt",
|
||||
"fields": [
|
||||
"operatorRef",
|
||||
"chatgptAuthorizationRef",
|
||||
"operatorMode"
|
||||
]
|
||||
},
|
||||
"scenarioId": "assistant-connections.connect-external-assistant",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/external-assistant-connections.ts",
|
||||
"chat-catalog-domains/external-chatgpt-setup.ts",
|
||||
"chat-catalog-domains/external-chatgpt.ts",
|
||||
"chat-catalog-domains/external-chatgpt-worker.ts",
|
||||
"chat-catalog-domains/external-chatgpt-contract.ts",
|
||||
"chat-catalog-domains/runtime-private.ts",
|
||||
"chat-runtime-private-requirements.ts",
|
||||
"chat-catalog-domains/external-public-signin.ts",
|
||||
"chat-catalog-domains/external-public-signin-worker.ts",
|
||||
"chat-catalog-domains/external-public-signin-contract.ts",
|
||||
"public-mcp-flow.ts",
|
||||
"chat-public-origin.ts",
|
||||
"chat-catalog-domains/external-support.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"public-mcp-client.ts",
|
||||
"chat-scenarios/assistant-connections.yaml"
|
||||
"chat-scenarios/assistant-connections.yaml",
|
||||
"../../ui/src/pages/apps/AssistantConnection.tsx",
|
||||
"../../ui/src/pages/McpConnect.tsx",
|
||||
"../../server/src/routes/public-mcp.ts",
|
||||
"../../server/src/services/public-mcp/capabilities.ts",
|
||||
"../../server/src/services/public-mcp/oauth.ts",
|
||||
"../../packages/shared/src/mcp-setup.ts"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.external.v1",
|
||||
"coverageGaps": [
|
||||
"named-chatgpt-human-setup"
|
||||
],
|
||||
"expectedRunCount": 1,
|
||||
"expectedHumanMessageCount": 1,
|
||||
"turnTimeoutMs": 120000,
|
||||
"attemptTimeoutMs": 300000,
|
||||
"attemptTimeoutMs": 1500000,
|
||||
"authentication": "human-session",
|
||||
"modulePath": "./chat-catalog-domains/external-assistant-connections.js",
|
||||
"exportName": "assistantConnectionExternalCases"
|
||||
@@ -3831,6 +3881,14 @@
|
||||
"minimumRunCount": 3,
|
||||
"turnTimeoutMs": 120000,
|
||||
"attemptTimeoutMs": 660000,
|
||||
"setupPolicy": {
|
||||
"readiness": "preserve-scenario",
|
||||
"invariantCheckIds": [
|
||||
"lead-ready",
|
||||
"future-worker-ready",
|
||||
"existing-work-actively-owned"
|
||||
]
|
||||
},
|
||||
"runPolicy": {
|
||||
"kind": "source-settled-with-active-work",
|
||||
"maximumActiveWorkerRuns": 1
|
||||
@@ -5437,7 +5495,8 @@
|
||||
"chat-catalog-domains/internal-definition.ts",
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-catalog-domains/internal-tasks.ts",
|
||||
"chat-scenarios/tasks.yaml"
|
||||
"chat-scenarios/tasks.yaml",
|
||||
"chat-catalog-domains/internal-distinct-fixtures.ts"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.internal.v1",
|
||||
"coverageGaps": [],
|
||||
@@ -5551,18 +5610,20 @@
|
||||
"chat-scenarios/tasks.yaml",
|
||||
"chat-catalog-domains/internal-edit-ownership.ts",
|
||||
"chat-catalog-domains/internal-run-brief.ts",
|
||||
"../e2e/fetch-allowed-port.ts"
|
||||
"../e2e/fetch-allowed-port.ts",
|
||||
"run-observations.ts"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.internal.v1",
|
||||
"coverageGaps": [],
|
||||
"expectedRunCount": 3,
|
||||
"expectedRunCount": 4,
|
||||
"expectedHumanMessageCount": 2,
|
||||
"minimumRunCount": 3,
|
||||
"turnTimeoutMs": 120000,
|
||||
"attemptTimeoutMs": 600000,
|
||||
"runPolicy": {
|
||||
"kind": "source-settled-with-active-work",
|
||||
"maximumActiveWorkerRuns": 1
|
||||
"maximumActiveWorkerRuns": 1,
|
||||
"expectedWorkerCancellation": "owned-task-reassignment"
|
||||
},
|
||||
"modulePath": "./chat-catalog-domains/internal-tasks.js",
|
||||
"exportName": "tasksCases"
|
||||
@@ -5803,9 +5864,11 @@
|
||||
{
|
||||
"scenarioId": "workspaces.locate-files",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/runtime-support.ts",
|
||||
"chat-catalog-domains/runtime-workspaces.ts",
|
||||
"chat-scenarios/workspaces.yaml"
|
||||
"chat-catalog-domains/runtime-support.ts",
|
||||
"chat-scenarios/workspaces.yaml",
|
||||
"../../ui/src/components/IsolatedWorkspacesRouteGate.tsx",
|
||||
"../../ui/src/pages/ExecutionWorkspaceDetail.tsx"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.runtime.v2",
|
||||
"coverageGaps": [],
|
||||
@@ -5828,6 +5891,8 @@
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/runtime-workspace-repositories.ts",
|
||||
"chat-catalog-domains/runtime-workspace-repositories-support.ts",
|
||||
"chat-catalog-domains/runtime-workspace-repositories-auth.ts",
|
||||
"chat-catalog-domains/runtime-workspace-repositories-auth-worker.ts",
|
||||
"chat-catalog-domains/runtime-workspaces.ts",
|
||||
"chat-catalog-domains/runtime-support.ts",
|
||||
"chat-catalog-domains/internal-repositories.ts",
|
||||
@@ -5838,7 +5903,10 @@
|
||||
"../../server/src/services/git-credentials.ts",
|
||||
"../../server/src/services/github-operation-credentials.ts",
|
||||
"../../packages/adapter-utils/src/github-launcher.ts",
|
||||
"../../server/src/routes/secrets.ts"
|
||||
"../../server/src/routes/secrets.ts",
|
||||
"../../ui/src/features/connections/ConnectionIntentInteractionBody.tsx",
|
||||
"../../ui/src/features/connections/ConnectionSetupFlow.tsx",
|
||||
"../../server/src/routes/tool-access.ts"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.runtime.v2",
|
||||
"coverageGaps": [],
|
||||
@@ -5882,13 +5950,16 @@
|
||||
"fields": [
|
||||
"siteRepositoryId",
|
||||
"siteRepositoryFullName",
|
||||
"tokenEnv"
|
||||
"tokenEnv",
|
||||
"humanAuthEnv"
|
||||
]
|
||||
},
|
||||
"scenarioId": "workspaces.private-repository-no-access",
|
||||
"sourceFiles": [
|
||||
"chat-catalog-domains/runtime-workspace-repositories.ts",
|
||||
"chat-catalog-domains/runtime-workspace-repositories-support.ts",
|
||||
"chat-catalog-domains/runtime-workspace-repositories-auth.ts",
|
||||
"chat-catalog-domains/runtime-workspace-repositories-auth-worker.ts",
|
||||
"chat-catalog-domains/runtime-workspaces.ts",
|
||||
"chat-catalog-domains/runtime-support.ts",
|
||||
"chat-catalog-domains/internal-repositories.ts",
|
||||
@@ -5899,12 +5970,13 @@
|
||||
"../../server/src/services/git-credentials.ts",
|
||||
"../../server/src/services/github-operation-credentials.ts",
|
||||
"../../packages/adapter-utils/src/github-launcher.ts",
|
||||
"../../server/src/routes/secrets.ts"
|
||||
"../../server/src/routes/secrets.ts",
|
||||
"../../ui/src/features/connections/ConnectionIntentInteractionBody.tsx",
|
||||
"../../ui/src/features/connections/ConnectionSetupFlow.tsx",
|
||||
"../../server/src/routes/tool-access.ts"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.runtime.v2",
|
||||
"coverageGaps": [
|
||||
"Unfinished required after-grant conditional: the selected metadata-only fixture has no executable protected human GitHub account/installation completion callback and therefore cannot yet resume preparation/fix the header after authorization. Reserved act preserves the actual initial request and probes real setup options, then stops instead of fabricating a grant. Missing designated disposable repo is a separate external prerequisite, not this authoring gap."
|
||||
],
|
||||
"coverageGaps": [],
|
||||
"expectedRunCount": 10,
|
||||
"expectedHumanMessageCount": 3,
|
||||
"minimumRunCount": 7,
|
||||
|
||||
Reference in new issue
Block a user