mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-25 09:41:04 +01:00
docs: retain legacy authority verification and remaining findings
This commit is contained in:
Dotta committed
1 parent
d2a6acddb4
commit
3dcc21d1b7
5 files changed
+5742
-1
No files matched your search
@@ -1,6 +1,6 @@
|
||||
# Legacy continuation authority: wording invariance
|
||||
|
||||
Status: implemented; verification in progress. Requested September 22, 2026.
|
||||
Status: implemented; legacy acceptance verified. Full live campaign retains one browser reload failure. Requested September 22, 2026.
|
||||
Working branch: `codex/lifecycle-behavior-baseline-20260921`.
|
||||
Baseline implementation/results head: `141950faf`.
|
||||
|
||||
@@ -101,6 +101,16 @@ and new key cannot schedule two successors.
|
||||
- Use scripted unit/database tests for timing permutations; real LLM/browser runs
|
||||
verify tool compliance and orchestration, not exhaustive race coverage.
|
||||
|
||||
## Verification result
|
||||
|
||||
[September 22 report](../../tests/lifecycle-baseline/LEGACY-CONTINUATION-2026-09-22.md):
|
||||
902/904 deterministic assertions pass; the two native compatibility probes remain
|
||||
visible. All 22 legacy live cases pass, including both causally verified repair
|
||||
variants. The complete live campaign is 41/42, with one native case completing its
|
||||
persisted lifecycle but failing on a blank browser reload. Earlier failed runs and
|
||||
artifact hashes remain in the inventory. This does not claim repo-wide green or
|
||||
removal of prose interpretation from every remaining product surface.
|
||||
|
||||
## Retained backlog for later requests
|
||||
|
||||
These counts refer to the original scripted baseline, not a new measurement.
|
||||
@@ -114,6 +124,8 @@ Keep this section even if this slice incidentally resolves some assertions.
|
||||
| Wording selects liveness versus handoff path in heartbeat | 1 | Retain separately in results; must be covered while closing the current shared authority boundary |
|
||||
| Native autonomous continuation compatibility probes | 2 | Replace with reachable public-contract tests; `same_agent` injection is not a current model-facing defect |
|
||||
| Productive continuation versus repair/failure budgets | Coverage boundary | Separate audit after this slice; do not infer progress from text/event counts |
|
||||
| Blank task route after reload | New live observation | Investigate browser bootstrap/rendering; seen after successful persisted lifecycle in both runtimes; campaigns 35744074887 and 35747200170 |
|
||||
| Missing-comment policy ownership | Coverage boundary | Existing typed retry policy remains; review separately if consolidating all post-run compliance into one disposition contract |
|
||||
| Maintain CI and prepare App/Evals review | Pending | Promote fixed invariants into maintained cheap gates; paid suites remain explicit |
|
||||
|
||||
Reference reports: `tests/lifecycle-baseline/BASELINE-2026-09-21.md`,
|
||||
|
||||
@@ -0,0 +1,110 @@
|
||||
# Legacy continuation authority — 2026-09-22
|
||||
|
||||
Legacy successful-run continuation now derives its decision from persisted
|
||||
run/task state and recorded API/tool effects. Narrative classification remains
|
||||
available for diagnostics; this path does not use it to schedule, identify,
|
||||
instruct, or replenish continuation work.
|
||||
|
||||
## Behavior
|
||||
|
||||
- A successful provider exit is not task completion. A recorded terminal status,
|
||||
blocker, question, approval, dependency, monitor, or existing execution owner
|
||||
determines the next action.
|
||||
- An eligible unfinished task without a disposition receives an agent repair
|
||||
instruction through the normal wake payload. The agent must record the outcome
|
||||
using existing Paperclip APIs/tools. Changing prose cannot select the path.
|
||||
- The durable repair episode permits at most two attempts before visible board
|
||||
escalation. Already-consumed older budgets retain their tighter limits.
|
||||
Existing comment-retry ownership is respected rather than opening another
|
||||
repair allowance after its exhaustion.
|
||||
- Stop, pause, budget, ownership, company binding, pending approval and existing
|
||||
work are checked again at dispatch. Goal, routine, plugin and conversation
|
||||
owners remain authoritative. Native semantic finalization is unchanged.
|
||||
- Replay and concurrent checks share a stable receipt and episode counter.
|
||||
A late scheduling write cannot rewind a newer attempt. Infrastructure retries
|
||||
retain the original disposition source and delayed repairs retain the initiating
|
||||
user's identity.
|
||||
|
||||
The [implementation plan and retained backlog](../../doc/plans/2026-09-22-legacy-continuation-authority.md)
|
||||
keep the remaining findings separate from this change.
|
||||
|
||||
## Deterministic evidence
|
||||
|
||||
The [new baseline inventory](baselines/2026-09-22-legacy-continuation.json)
|
||||
records a clean measurement at App `d2a6acddb`: **902 passed, 2 failed, 904 total**.
|
||||
Unit 317/317; Runner 182/182; integration 351/353; grading 52/52.
|
||||
The original September 21 snapshots remain unchanged.
|
||||
|
||||
All 178 wording/authority assertions pass. The real heartbeat pair now follows
|
||||
the same wake reason, repair instruction, attempt limit and final state.
|
||||
The persisted-authority suite has 27 passing tests, including concurrent replay,
|
||||
restart, stale diagnostic labels, fast-successor ledger writes, dispatch-time
|
||||
Stop/approval/ownership changes, infrastructure retry and bounded exhaustion.
|
||||
|
||||
The two remaining native probes inject `same_agent`, which is not in the current
|
||||
public continuation contract. They remain visible in the baseline and are not
|
||||
classified as proven current model-facing defects or silently skipped.
|
||||
A preceding clean measurement on `f044e17ed` had an additional dependency-test
|
||||
teardown timeout; its retained local inventory is
|
||||
`.lifecycle-baseline/2026-09-22T14-58-55.292Z/`. A fresh measurement and the focused
|
||||
15-test dependency suite passed that case without changing its assertions.
|
||||
|
||||
The broader checks also exposed old mocks that represented completed work using
|
||||
only summaries. Those fixtures now record done or a pending confirmation while
|
||||
preserving their identity, comment attribution, workspace and native-boundary
|
||||
assertions. A real delayed-row identity omission was fixed separately.
|
||||
|
||||
## Real-provider campaigns
|
||||
|
||||
The [live inventory](baselines/2026-09-22-legacy-continuation-live.json) retains
|
||||
source commits, fixture hashes, normalized artifact SHA-256, cell outcomes,
|
||||
cleanup, evidence validation and billing coverage. Every cell uses a real Codex
|
||||
provider, browser, isolated server and database; these are not canned responses.
|
||||
|
||||
- [35738968308](https://github.com/paperclipai/paperclip/actions/runs/35738968308),
|
||||
App `f3e09abd4`: 1/2 passed. Neutral passed; challenge skipped the requested
|
||||
initial phase and immediately recorded done. The causal two-run oracle correctly
|
||||
rejected it. The fixture now selects its phase from `PAPERCLIP_WAKE_REASON`.
|
||||
- [35744074887](https://github.com/paperclipai/paperclip/actions/runs/35744074887),
|
||||
App `f044e17ed`: 1/2 passed. Both cases recorded the exact initial comment,
|
||||
one causally bound repair and done. Neutral failed the final browser assertion:
|
||||
the reload stayed blank even though API state and the exact final message were
|
||||
correct. The failure remains recorded. Browser exception capture was added;
|
||||
assertions were not relaxed.
|
||||
- [35747200170](https://github.com/paperclipai/paperclip/actions/runs/35747200170),
|
||||
App `d2a6acddb`: **41/42 passed; legacy 22/22, native 19/20**. Both new repair
|
||||
probes passed, including their browser assertions. The native plan-revision
|
||||
challenge completed all three turns, both approval boundaries, the correct
|
||||
revised output, done and cleanup; its task route then reloaded blank. All 17
|
||||
persisted-state matchers passed, but the strict browser check failed. This is
|
||||
retained as a separate unresolved UI finding, not regraded as a pass.
|
||||
|
||||
All 42 selected cells executed once; no retries or incomplete cells. All evidence
|
||||
validated and all cleanup passed. The [published report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35747200170-1/index.html)
|
||||
contains the failure alongside successes. Publisher catalog-completeness flags
|
||||
are not a claim that every Paperclip suite was selected.
|
||||
|
||||
Reported zero cost is not proof of free execution: product model-cost coverage
|
||||
is partial, and GitHub compute is separate. A passing campaign is one measurement,
|
||||
not a reliability estimate.
|
||||
|
||||
## Broader qualification
|
||||
|
||||
The 441 Product E2E support tests pass, as do server and E2E typechecks. Repository
|
||||
recursive typecheck and build both passed again on final implementation `d2a6acddb`.
|
||||
The fresh general-server shards recorded 12,812 passing assertions. One suite
|
||||
could not start PostgreSQL because macOS exhausted shared-memory IDs; its 14
|
||||
identity tests passed when run alone. All 6,502 UI tests passed. The database
|
||||
package passed all 160 tests with sequential workers after four startup failures
|
||||
in the parallel run. CLI import (17), auth readiness (14) and agent-skills routes
|
||||
(54) passed on focused reruns after timeouts in their broader groups.
|
||||
|
||||
The full repository command was attempted, but its early-exit groups did not all
|
||||
complete cleanly. No repo-wide green claim is made: broad runs encountered local
|
||||
PostgreSQL resource limits and route timeouts, and the historical baseline retains
|
||||
its two native probes. Downloaded provider recordings were moved outside the
|
||||
checkout after the authored-guidance scanner included them; its unchanged 39 tests
|
||||
then passed. A sequential run of the database, adapter utility, provider adapter,
|
||||
plugin and plugin-creator packages passed another 2,620 assertions (19 explicitly
|
||||
skipped). Focused regressions and the clean lifecycle inventory are the acceptance
|
||||
evidence for this change.
|
||||
@@ -144,3 +144,5 @@ For subsequent live execution use existing explicit selectors and record exact
|
||||
App/Evals revisions, profile/environment, retries, usage/cost, and artifact IDs.
|
||||
See `doc/evals.md`. Do not combine mock-authority Runner Eval scores with Product
|
||||
E2E scores, or claim full qualification from a partial selection.
|
||||
|
||||
September 22 follow-up: [legacy continuation implementation and verification](LEGACY-CONTINUATION-2026-09-22.md), including preserved failed campaigns and the remaining backlog.
|
||||
@@ -0,0 +1,829 @@
|
||||
{
|
||||
"schema": "paperclip.legacy-continuation-verification/v1",
|
||||
"measurementDateLocal": "2026-09-22",
|
||||
"timezone": "America/Chicago",
|
||||
"campaigns": [
|
||||
{
|
||||
"source": {
|
||||
"sha": "f3e09abd4a1655e5d8fbe900d7a283ef47220e4c",
|
||||
"ref": "refs/heads/codex/lifecycle-behavior-baseline-20260921",
|
||||
"workflowRunUrl": "https://github.com/paperclipai/paperclip/actions/runs/35738968308",
|
||||
"eventName": "workflow_dispatch"
|
||||
},
|
||||
"campaignId": "gha-35738968308-1",
|
||||
"artifactSha256": "a44fb1e9138c673b8c7ad1b5e4d3962c71213e19b766a2fc2f2795d69704e7fc",
|
||||
"totals": {
|
||||
"selected": 2,
|
||||
"executed": 2,
|
||||
"passed": 1,
|
||||
"failed": 1,
|
||||
"incomplete": 0,
|
||||
"retries": 0,
|
||||
"cleanupPassed": true
|
||||
},
|
||||
"billing": {
|
||||
"testCount": 2,
|
||||
"agentRunDurationMs": 78741,
|
||||
"leaseDurationMs": 0,
|
||||
"llm": {
|
||||
"runCount": 3,
|
||||
"runsWithTokenUsage": 3,
|
||||
"runsWithReportedCost": 0,
|
||||
"inputTokens": 309234,
|
||||
"outputTokens": 2869,
|
||||
"cachedInputTokens": 230772,
|
||||
"totalTokens": 542875,
|
||||
"reportedCostUsd": 0,
|
||||
"costStatus": "unpriced"
|
||||
},
|
||||
"reportedLlmCostUsd": 0,
|
||||
"estimatedRuntimeCostUsd": 0,
|
||||
"observedAndEstimatedCostUsd": 0,
|
||||
"testsWithCompleteBilling": 0
|
||||
},
|
||||
"definitionHashes": [
|
||||
"f85e882942b6648478da2a982d688b6c757cf577b6d0e81d1388d74a73eb29ad"
|
||||
],
|
||||
"cells": [
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-repair-neutral",
|
||||
"caseId": "lifecycle-repair-neutral",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 67094,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-repair-challenge",
|
||||
"caseId": "lifecycle-repair-challenge",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "failed",
|
||||
"failureClass": "candidate_failure",
|
||||
"attempt": 1,
|
||||
"durationMs": 481573,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"source": {
|
||||
"sha": "f044e17ed5e88322c50732d3308f002425906544",
|
||||
"ref": "refs/heads/codex/lifecycle-behavior-baseline-20260921",
|
||||
"workflowRunUrl": "https://github.com/paperclipai/paperclip/actions/runs/35744074887",
|
||||
"eventName": "workflow_dispatch"
|
||||
},
|
||||
"campaignId": "gha-35744074887-1",
|
||||
"artifactSha256": "d713f73e31524a15f6c6baed50ecfe79a064ef354ca9f611a835d0955369cfa6",
|
||||
"totals": {
|
||||
"selected": 2,
|
||||
"executed": 2,
|
||||
"passed": 1,
|
||||
"failed": 1,
|
||||
"incomplete": 0,
|
||||
"retries": 0,
|
||||
"cleanupPassed": true
|
||||
},
|
||||
"billing": {
|
||||
"testCount": 2,
|
||||
"agentRunDurationMs": 109842,
|
||||
"leaseDurationMs": 0,
|
||||
"llm": {
|
||||
"runCount": 4,
|
||||
"runsWithTokenUsage": 4,
|
||||
"runsWithReportedCost": 0,
|
||||
"inputTokens": 739944,
|
||||
"outputTokens": 5069,
|
||||
"cachedInputTokens": 617090,
|
||||
"totalTokens": 1362103,
|
||||
"reportedCostUsd": 0,
|
||||
"costStatus": "unpriced"
|
||||
},
|
||||
"reportedLlmCostUsd": 0,
|
||||
"estimatedRuntimeCostUsd": 0,
|
||||
"observedAndEstimatedCostUsd": 0,
|
||||
"testsWithCompleteBilling": 0
|
||||
},
|
||||
"definitionHashes": [
|
||||
"43b3e40fc1e421fedc1925fb62e46ead0480a971e17d63105264139ec2d837c5"
|
||||
],
|
||||
"cells": [
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-repair-neutral",
|
||||
"caseId": "lifecycle-repair-neutral",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "failed",
|
||||
"failureClass": "candidate_failure",
|
||||
"attempt": 1,
|
||||
"durationMs": 100314,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-repair-challenge",
|
||||
"caseId": "lifecycle-repair-challenge",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 82437,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"source": {
|
||||
"sha": "d2a6acddb42b5a4c07886386cb84c47d78a68d19",
|
||||
"ref": "refs/heads/codex/lifecycle-behavior-baseline-20260921",
|
||||
"workflowRunUrl": "https://github.com/paperclipai/paperclip/actions/runs/35747200170",
|
||||
"eventName": "workflow_dispatch"
|
||||
},
|
||||
"campaignId": "gha-35747200170-1",
|
||||
"artifactSha256": "c2d5554a30ceab6cecaff32d5f891e2e760bd7bced2749269001f596f5893516",
|
||||
"totals": {
|
||||
"selected": 42,
|
||||
"executed": 42,
|
||||
"passed": 41,
|
||||
"failed": 1,
|
||||
"incomplete": 0,
|
||||
"retries": 0,
|
||||
"cleanupPassed": true
|
||||
},
|
||||
"billing": {
|
||||
"testCount": 42,
|
||||
"agentRunDurationMs": 2810155,
|
||||
"leaseDurationMs": 0,
|
||||
"llm": {
|
||||
"runCount": 98,
|
||||
"runsWithTokenUsage": 91,
|
||||
"runsWithReportedCost": 42,
|
||||
"inputTokens": 23960210,
|
||||
"outputTokens": 182440,
|
||||
"cachedInputTokens": 21430127,
|
||||
"totalTokens": 45572777,
|
||||
"reportedCostUsd": 0,
|
||||
"costStatus": "partial"
|
||||
},
|
||||
"reportedLlmCostUsd": 0,
|
||||
"estimatedRuntimeCostUsd": 0,
|
||||
"observedAndEstimatedCostUsd": 0,
|
||||
"testsWithCompleteBilling": 16
|
||||
},
|
||||
"definitionHashes": [
|
||||
"43b3e40fc1e421fedc1925fb62e46ead0480a971e17d63105264139ec2d837c5"
|
||||
],
|
||||
"cells": [
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-repair-neutral",
|
||||
"caseId": "lifecycle-repair-neutral",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 54389,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-repair-challenge",
|
||||
"caseId": "lifecycle-repair-challenge",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 67656,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-completion-neutral",
|
||||
"caseId": "lifecycle-completion-neutral",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 34490,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-completion-challenge",
|
||||
"caseId": "lifecycle-completion-challenge",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 45721,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-blocker-neutral",
|
||||
"caseId": "lifecycle-blocker-neutral",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 38902,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-blocker-challenge",
|
||||
"caseId": "lifecycle-blocker-challenge",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 30142,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-question-neutral",
|
||||
"caseId": "lifecycle-question-neutral",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 142889,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-question-challenge",
|
||||
"caseId": "lifecycle-question-challenge",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 95053,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-approval-neutral",
|
||||
"caseId": "lifecycle-approval-neutral",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 180306,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-approval-challenge",
|
||||
"caseId": "lifecycle-approval-challenge",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 172774,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-plan-revision-neutral",
|
||||
"caseId": "lifecycle-plan-revision-neutral",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 158794,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-plan-revision-challenge",
|
||||
"caseId": "lifecycle-plan-revision-challenge",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 122677,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-untrusted-evidence-neutral",
|
||||
"caseId": "lifecycle-untrusted-evidence-neutral",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 117641,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-untrusted-evidence-challenge",
|
||||
"caseId": "lifecycle-untrusted-evidence-challenge",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 99589,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-dependency-restart-neutral",
|
||||
"caseId": "lifecycle-dependency-restart-neutral",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 158975,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-dependency-restart-challenge",
|
||||
"caseId": "lifecycle-dependency-restart-challenge",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 219723,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.stop-new-resume",
|
||||
"caseId": "stop-new-resume",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 31593,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.clarify-reuse",
|
||||
"caseId": "clarify-reuse",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 164251,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.tool-review-approve",
|
||||
"caseId": "tool-review-approve",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 210880,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.tool-review-decline",
|
||||
"caseId": "tool-review-decline",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 111772,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.tool-review-always",
|
||||
"caseId": "tool-review-always",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 165026,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.tool-review-restart",
|
||||
"caseId": "tool-review-restart",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 230211,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.runner-codex.local.lifecycle-completion-neutral",
|
||||
"caseId": "lifecycle-completion-neutral",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 34474,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.runner-codex.local.lifecycle-completion-challenge",
|
||||
"caseId": "lifecycle-completion-challenge",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 37562,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.runner-codex.local.lifecycle-blocker-neutral",
|
||||
"caseId": "lifecycle-blocker-neutral",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 34683,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.runner-codex.local.lifecycle-blocker-challenge",
|
||||
"caseId": "lifecycle-blocker-challenge",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 29889,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.runner-codex.local.lifecycle-question-neutral",
|
||||
"caseId": "lifecycle-question-neutral",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 58435,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.runner-codex.local.lifecycle-question-challenge",
|
||||
"caseId": "lifecycle-question-challenge",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 64626,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.runner-codex.local.lifecycle-approval-neutral",
|
||||
"caseId": "lifecycle-approval-neutral",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 84593,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.runner-codex.local.lifecycle-approval-challenge",
|
||||
"caseId": "lifecycle-approval-challenge",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 82542,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.runner-codex.local.lifecycle-plan-revision-neutral",
|
||||
"caseId": "lifecycle-plan-revision-neutral",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 92243,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.runner-codex.local.lifecycle-plan-revision-challenge",
|
||||
"caseId": "lifecycle-plan-revision-challenge",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "failed",
|
||||
"failureClass": "candidate_failure",
|
||||
"attempt": 1,
|
||||
"durationMs": 111778,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.runner-codex.local.lifecycle-untrusted-evidence-neutral",
|
||||
"caseId": "lifecycle-untrusted-evidence-neutral",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 57424,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.runner-codex.local.lifecycle-untrusted-evidence-challenge",
|
||||
"caseId": "lifecycle-untrusted-evidence-challenge",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 63381,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.runner-codex.local.lifecycle-dependency-restart-neutral",
|
||||
"caseId": "lifecycle-dependency-restart-neutral",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 117263,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.runner-codex.local.lifecycle-dependency-restart-challenge",
|
||||
"caseId": "lifecycle-dependency-restart-challenge",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 108571,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.runner-codex.local.stop-new-resume",
|
||||
"caseId": "stop-new-resume",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 53129,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.runner-codex.local.clarify-reuse",
|
||||
"caseId": "clarify-reuse",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 63270,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.runner-codex.local.tool-review-approve",
|
||||
"caseId": "tool-review-approve",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 70668,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.runner-codex.local.tool-review-decline",
|
||||
"caseId": "tool-review-decline",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 67846,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.runner-codex.local.tool-review-always",
|
||||
"caseId": "tool-review-always",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 87986,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.runner-codex.local.tool-review-restart",
|
||||
"caseId": "tool-review-restart",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 133283,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
Reference in new issue
Block a user