From 6d9865068aa67a58f0c089329ae119de6a515602 Mon Sep 17 00:00:00 2001 From: Dotta Date: Mon, 26 Oct 2026 23:58:56 -0500 Subject: [PATCH] Reconcile archive transport and 210 authored catalog cases --- .../2026-10-26-chat-catalog-eval-progress.md | 15 +- tests/runner-e2e/chat-campaign-manifest.json | 36 ++--- .../runner-e2e/chat-catalog-descriptors.json | 129 +++++++++++++++++- 3 files changed, 156 insertions(+), 24 deletions(-) diff --git a/doc/plans/2026-10-26-chat-catalog-eval-progress.md b/doc/plans/2026-10-26-chat-catalog-eval-progress.md index 97d4f11e53f..ba53c4b4222 100644 --- a/doc/plans/2026-10-26-chat-catalog-eval-progress.md +++ b/doc/plans/2026-10-26-chat-catalog-eval-progress.md @@ -5,8 +5,8 @@ Updated: 2026-10-26. This is the single progress record for [the saved goal](202 ## Current state - **244 scenarios / 35 areas / 191 capabilities**, frozen inventory digest `9dc9cf6a5426cf1b56bc30a3571851f233f9dd5e93a1292eaee414d33b52df77`. All original queries, contexts and expected outcomes remain unchanged. **196 passes required**; blocked, partial, unsupported and unrun remain in the denominator. -- **207 authored: 198 without declared coverage gaps, 9 partial, 37 unimplemented** at exact committed-source metadata checkpoint `e4b62f48aacef0bef16fd1f5a28433a8192af8ba`. There are 205 concrete lazy descriptors plus two legacy task cases. All actual callbacks, bounds, policies and source dependencies were reconciled from this one Git archive; later external/runtime work remains pending. These are authoring counts, not acceptance. -- Five partial definitions contain concrete executable supported-surface/context blocker probes: two access grants, subscription provenance, skill-use authority/context and genuine prior-day routine history. Four have unfinished required branches: ask-before-action, named ChatGPT setup, team installer failures and native archive-fault launcher transport. **203 authoring-resolved; 4 unfinished; 37 absent.** Only actual reserved execution can process a blocker; offline probes/calibrations do not establish processing. Incomplete conditions cannot pass. +- **210 authored: 202 without declared coverage gaps, 8 partial, 34 unimplemented** at exact committed-source metadata checkpoint `4043cf898485d8a4cff9789d6b2708ab581b195c`. There are 208 concrete lazy descriptors plus two legacy task cases. Actual callbacks, bounds, policies and source dependencies were reconciled from this one Git archive; newer area work remains pending. These are authoring counts, not acceptance. +- Five partial definitions contain concrete executable supported-surface/context blocker probes: two access grants, subscription provenance, skill-use authority/context and genuine prior-day routine history. Three have unfinished required branches: ask-before-action, named ChatGPT setup and team installer failures. **207 authoring-resolved; 3 unfinished; 34 absent.** The native archive-fault transport is now fully authored and offline-qualified; it has not run against a live product instance. Only actual reserved execution can process a blocker; offline probes/calibrations do not establish processing. Incomplete conditions cannot pass. - **Four scenarios attempted; five actual measurements; zero first-attempt passes and zero final passes (0/244 for both rates).** Goal-tree is exhausted after two failures. Alignment and purpose each retain one failed initial attempt and one allowed correction/rerun remaining. - **No live launcher or provider process is running.** Admissions stopped after the authorized sign-in initial failed PostgreSQL bootstrap and finalization; its recovery state remains private with cleanup unconfirmed. The other four authorized initial cases remain unreserved. Final retries remain held for coherent B. No unchanged rerun, extra nudge, deadline extension, automatic retry or relabeled canonical health probe was used. - Production baseline `8373d7867963674ff960d2345d0d28086f799bc7`; branch `codex/chat-catalog-evals`. PR #17794 merged via `2ee36a4`. No eval/core PR has been created. No merge is eligible. @@ -179,3 +179,14 @@ Privacy review supersedes the first root-inference correction in `025bf9a420`: t Exact archive imports verified 205 lazy definitions plus two legacy tasks: domains 43+2 legacy (41 no-gap/two partial), identity 44 (41/three), runtime 43 (43/zero), integrator 27 (26/one), external 32 (29/three), privileged 16 (16/zero). Total 207 authored/198 no-gap/nine partial/37 absent. No newer WIP is included. The canonical 244/35/191 inventory and original four-scenario/five-measurement history are preserved. Five concrete blockers and four unfinished branches remain distinct. Registered native Slack metadata is supported, but future Slack callback authoring remains outside this source checkpoint. Diagnostics/reporting integration preserves isolated source refs `0fdcb539a8` and `025bf9a420` and adopts their coordinator-owned changes without modifying the executed snapshot. Current raw-root expansion uses explicit process-local provenance and preserves independently registered overlapping credentials; seven narrow controls and 400 related evidence/recovery controls passed. Six Python diagnostic guards passed. No production code was edited. No second bootstrap, canonical admission, final retry or IPC release occurred while authorization for the bounded dry-run recovery proposal remained pending. + + +## Integrated finite archive transport checkpoint 4043cf8984 + +Exact committed-source archive reconciliation found 208 lazy definitions plus two legacy cases: domains 43+2 legacy (42 no-gap/one concrete blocker), identity 44 (41/three blockers), runtime 44 (44/zero), integrator 27 (26/one blocker), external 34 (31/three unfinished) and privileged 16 (16/zero). This supersedes e4b62f48aa metadata while preserving its history. Total **210 authored/202 no-gap/eight partial/34 absent**, with 207 authoring-resolved. Newly reconciled actual paths are Slack reply-in-thread, wrong-person connection resolution and recovery post-stop-message; their declared transport, source-chat human and native-run bounds match their actual exports. + +The archive partial-failure case now has its required pre-server lifecycle and authenticated cross-process control handoff. Only its exact single local execution can start the owned fixed-loopback proxy. The product inherits the ordinary API-origin override only; control receipt, URL and bearer remain in the test host. Actual public agent/run/responsible-person reads restrict the three-task fault. Two real backend completions precede a third pre-admission drop; repeated failed-item calls remain dropped until the explicit human retry. Independent saved-state/audit oracles and first-phase checkpoint remain required. Teardown closes owned connections/control and removes its private receipt after backend shutdown; cleanup errors remain failures. This closes unfinished authoring, **not live coverage**. + +Fifty offline controls in six files passed from the pinned archive, including a real separate Node control client, HTTP/upgrade byte preservation, malformed/oversized passthrough, exact actor/person/run restrictions, duplicate fault/retry behavior, private product-environment exclusion, cleanup and fingerprint/accounting checks. Archived E2E typecheck passed, all 208 runtime descriptors matched, and actual `--list` returned 210 cells. Generated CLI/SDK artifacts were adopted only from the previously audited immutable candidate after exact owning-source/lock equality and byte hashes; no mutable worktree dist was imported. Production source diff from 8373 remains empty. Receipt: `results/chat-catalog-campaign/metadata-4043cf8984.json`. + +All original four-scenario/five-measurement records remain byte-for-byte unchanged. Zero live/DB/provider/browser/native runs, reservations, second bootstraps or IPC removals occurred during this closure. The eight-set recovery proposal still awaits user approval. Original sign-in stderr was not captured: the later same-binary one-shot diagnostic directly proves current semaphore exhaustion and strongly explains the earlier startup failure, but does not replace missing original stderr or unknown original details. Original cleanup uncertainty remains preserved. diff --git a/tests/runner-e2e/chat-campaign-manifest.json b/tests/runner-e2e/chat-campaign-manifest.json index a983b4dc942..551479c1659 100644 --- a/tests/runner-e2e/chat-campaign-manifest.json +++ b/tests/runner-e2e/chat-campaign-manifest.json @@ -1763,8 +1763,8 @@ "sourceSha256": "e9eacfaf22ab445de8ecd131439a2bab37708b6bd9ea1edcf72c55a2948e6c46", "scenarioSha256": "b44908f447ae54ab82edef01bd417e0f82b3e97808f571d7b2caf08fd9df3026", "owner": "eval_external", - "readiness": "unimplemented", - "executionId": null, + "readiness": "implemented", + "executionId": "chat-features.runner-codex-subscription.local.chat-channels.reply-in-thread", "blocker": null, "attempts": [], "qualifyingDefinitionDigest": null, @@ -1773,7 +1773,7 @@ "lane": "source-build", "sourceBaselineRevision": "8373d7867963674ff960d2345d0d28086f799bc7" }, - "authoringStatus": "absent" + "authoringStatus": "complete" }, { "id": "chat-channels.reply-to-email", @@ -2506,8 +2506,8 @@ "sourceSha256": "9e71e1ea7d9216d80396df117d14f7629aace3dbcfdcddc14dfb1edf38e992c4", "scenarioSha256": "18aaf7e343d9a1a81eebc09e4cd2733ce9764997fec0b4b23ba58f6b69e55ec0", "owner": "eval_external", - "readiness": "unimplemented", - "executionId": null, + "readiness": "implemented", + "executionId": "chat-features.runner-codex-subscription.local.connection-setup.wrong-person-resolves", "blocker": null, "attempts": [], "qualifyingDefinitionDigest": null, @@ -2516,7 +2516,7 @@ "lane": "source-build", "sourceBaselineRevision": "8373d7867963674ff960d2345d0d28086f799bc7" }, - "authoringStatus": "absent" + "authoringStatus": "complete" }, { "id": "developer-labs.component-reference", @@ -3351,9 +3351,9 @@ "sourceSha256": "afc939f4eec821edd38b8e819274ede302c10550aca68b2ff72c5ebf870aed05", "scenarioSha256": "327f96d910bf6645fab9dfcd49b38e26eec57e44d31764bbd2fea13fdb08409d", "owner": "eval_domains", - "readiness": "partial", + "readiness": "implemented", "executionId": "chat-features.runner-codex-subscription.local.inbox-search.archive-partial-failure", - "blocker": "native-archive-transport-launcher-wiring", + "blocker": null, "attempts": [], "qualifyingDefinitionDigest": null, "qualifyingGraderDigest": null, @@ -3361,7 +3361,7 @@ "lane": "source-build", "sourceBaselineRevision": "8373d7867963674ff960d2345d0d28086f799bc7" }, - "authoringStatus": "unfinished" + "authoringStatus": "complete" }, { "id": "inbox-search.archive-without-grant", @@ -4654,8 +4654,8 @@ "sourceSha256": "f74fb387ab717d656e4ce11b8abc03960946f0cba83ccf76e83f4ac77c520a3a", "scenarioSha256": "6d063076ce3d4b3763610a4bec43c5a8685970d88f98e2f22e4a69bcdd1def3c", "owner": "eval_runtime", - "readiness": "unimplemented", - "executionId": null, + "readiness": "implemented", + "executionId": "chat-features.runner-codex-subscription.local.recovery.post-stop-message", "blocker": null, "attempts": [], "qualifyingDefinitionDigest": null, @@ -4664,7 +4664,7 @@ "lane": "source-build", "sourceBaselineRevision": "8373d7867963674ff960d2345d0d28086f799bc7" }, - "authoringStatus": "absent" + "authoringStatus": "complete" }, { "id": "recovery.retry-exhausted", @@ -6490,11 +6490,11 @@ } ], "implementationCheckpoint": { - "sourceRevision": "e4b62f48aacef0bef16fd1f5a28433a8192af8ba", - "descriptorSha256": "dc3ef609f877a88a8546e6e9c2c29d9462dcfe693b85e15d398b56ccb4c2c069", - "authored": 207, - "complete": 198, - "partial": 9, - "unimplemented": 37 + "sourceRevision": "4043cf898485d8a4cff9789d6b2708ab581b195c", + "descriptorSha256": "ca0146b40da076fd21a5a1e4670c317b40b613638a794f921c3ebc0e795fdbb7", + "authored": 210, + "complete": 202, + "partial": 8, + "unimplemented": 34 } } diff --git a/tests/runner-e2e/chat-catalog-descriptors.json b/tests/runner-e2e/chat-catalog-descriptors.json index a5077d212cd..6f11a7a4679 100644 --- a/tests/runner-e2e/chat-catalog-descriptors.json +++ b/tests/runner-e2e/chat-catalog-descriptors.json @@ -1195,6 +1195,60 @@ "modulePath": "./chat-catalog-domains/external-chat-channels.js", "exportName": "chatChannelExternalCases" }, + { + "slackIngress": { + "kind": "native-slack-thread", + "maximumProviderCalls": 20, + "ingressTimeoutMs": 120000, + "setupThread": "same-agent-distractor" + }, + "externalInput": { + "provider": "slack", + "fields": [ + "workspaceId", + "channelId", + "senderUserId", + "botUserId", + "appId", + "registeredWebhookOrigin", + "webhookRelayPort", + "botTokenEnv", + "senderTokenEnv", + "signingSecretEnv", + "configurationTokenEnv" + ] + }, + "scenarioId": "chat-channels.reply-in-thread", + "sourceFiles": [ + "chat-catalog-domains/external-slack-thread.ts", + "chat-catalog-domains/external-slack-fixture.ts", + "chat-catalog-domains/external-slack-provider.ts", + "chat-catalog-domains/external-slack-relay.ts", + "chat-catalog-domains/external-slack-render.ts", + "chat-catalog-domains/runtime-rendered-artifact.ts", + "chat-catalog-domains/external-fixtures.ts", + "chat-catalog-domains/external-support.ts", + "chat-catalog-domains/internal-support.ts", + "chat-catalog-domains/identity-support.ts", + "chat-slack-ingress.ts", + "../../packages/shared/src/types/chat-channels.ts", + "../../server/src/routes/chat-channels.ts", + "../../server/src/services/chat-channels.ts", + "../../server/src/services/chat-channels/inbound/authorization.ts", + "../../server/src/services/chat-channels/inbound/wakeups.ts", + "chat-scenarios/chat-channels.yaml" + ], + "oracleVersion": "paperclip.chat-catalog.external.v1", + "expectedRunCount": 4, + "expectedHumanMessageCount": 1, + "minimumRunCount": 2, + "turnTimeoutMs": 120000, + "attemptTimeoutMs": 600000, + "journeyKind": "native-slack-thread", + "authentication": "human-session", + "modulePath": "./chat-catalog-domains/external-slack-thread.js", + "exportName": "slackThreadExternalCases" + }, { "scenarioId": "cli-operations.choose-cli-context", "sourceFiles": [ @@ -1640,6 +1694,31 @@ "modulePath": "./chat-catalog-domains/external-mcp-failure.js", "exportName": "mcpFailureExternalCases" }, + { + "scenarioId": "connection-setup.wrong-person-resolves", + "sourceFiles": [ + "chat-catalog-domains/external-wrong-resolver.ts", + "chat-catalog-domains/identity-auth.ts", + "chat-catalog-domains/identity-private-browser.ts", + "chat-catalog-domains/identity-private-browser-worker.ts", + "chat-catalog-domains/identity-wizard.ts", + "chat-catalog-domains/identity-support.ts", + "chat-catalog-domains/external-support.ts", + "chat-catalog-domains/internal-support.ts", + "../../server/src/routes/connection-intents.ts", + "../../server/src/services/connection-intents.ts", + "chat-scenarios/connection-setup.yaml" + ], + "oracleVersion": "paperclip.chat-catalog.external.v1", + "expectedRunCount": 2, + "expectedHumanMessageCount": 1, + "minimumRunCount": 2, + "turnTimeoutMs": 120000, + "attemptTimeoutMs": 540000, + "authentication": "human-session", + "modulePath": "./chat-catalog-domains/external-wrong-resolver.js", + "exportName": "wrongResolverExternalCases" + }, { "scenarioId": "developer-labs.component-reference", "sourceFiles": [ @@ -2017,12 +2096,11 @@ "chat-scenarios/inbox-search.yaml", "chat-catalog-domains/internal-archive-partial.ts", "chat-catalog-domains/internal-archive-transport.ts", - "../e2e/fetch-allowed-port.ts" + "../e2e/fetch-allowed-port.ts", + "chat-archive-fault-control.ts" ], "oracleVersion": "paperclip.chat-catalog.internal.v1", - "coverageGaps": [ - "native-archive-transport-launcher-wiring" - ], + "coverageGaps": [], "expectedRunCount": 2, "expectedHumanMessageCount": 2, "minimumRunCount": 2, @@ -3087,6 +3165,49 @@ "modulePath": "./chat-catalog-domains/runtime-infrastructure.js", "exportName": "runtimeInfrastructureCases" }, + { + "canonicalBoundary": "native-stop-cleanup", + "scenarioId": "recovery.post-stop-message", + "sourceFiles": [ + "chat-catalog-domains/runtime-post-stop.ts", + "chat-catalog-domains/runtime-provider-cleanup.ts", + "chat-catalog-domains/runtime-native-fault.ts", + "chat-catalog-domains/runtime-busy-worker.ts", + "chat-catalog-domains/runtime-rendered-artifact.ts", + "chat-catalog-domains/runtime-support.ts", + "chat-native-stop-boundary.ts", + "../../server/src/services/native-runtime/native-session-executor.ts", + "../../server/src/services/explicit-native-continuation.ts", + "../../packages/paperclip-runner/src/live/runnerd-codex-transport.ts", + "../../packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs", + "../../packages/paperclip-runner/runner/crates/runner-core/src/process_supervisor.rs", + "registered-artifact.ts", + "chat-scenarios/recovery.yaml" + ], + "oracleVersion": "paperclip.chat-catalog.runtime.v2", + "coverageGaps": [], + "expectedRunCount": 2, + "setupHumanMessageCount": 1, + "expectedHumanMessageCount": 2, + "turnTimeoutMs": 120000, + "attemptTimeoutMs": 720000, + "authentication": "human-session", + "setupPolicy": { + "readiness": "preserve-scenario", + "invariantCheckIds": [ + "actual-source-native-owner" + ] + }, + "runPolicy": { + "kind": "terminal-counts", + "terminalStatusCounts": { + "cancelled": 1, + "succeeded": 1 + } + }, + "modulePath": "./chat-catalog-domains/runtime-post-stop.js", + "exportName": "runtimePostStopCases" + }, { "scenarioId": "recovery.retry-exhausted", "sourceFiles": [