diff --git a/tests/lifecycle-baseline/LIVE-BASELINE-2026-09-21.md b/tests/lifecycle-baseline/LIVE-BASELINE-2026-09-21.md index 98009a4075d..48cae22133b 100644 --- a/tests/lifecycle-baseline/LIVE-BASELINE-2026-09-21.md +++ b/tests/lifecycle-baseline/LIVE-BASELINE-2026-09-21.md @@ -2,7 +2,8 @@ Measured on GitHub Actions with real LLM calls on September 21 (America/Chicago; September 22 UTC). This supplements the immutable [scripted baseline](BASELINE-2026-09-21.md). -No production lifecycle behavior was changed. +No production lifecycle behavior was changed for the initial measurement. +Subsequent changes and fresh runs are recorded in the [fix report](LIVE-FIXES-2026-09-21.md). ## Initial measurements @@ -148,4 +149,4 @@ a turn. Earlier cancellation rejects the late session; later cancellation can reach the published handle. Deterministic tests inject Stop in that exact gap. The existing execution-owned cleanup closes rejected sessions. -Fresh live verification is recorded separately when complete. +Fresh live verification is recorded in the [fix report](LIVE-FIXES-2026-09-21.md). diff --git a/tests/lifecycle-baseline/LIVE-FIXES-2026-09-21.md b/tests/lifecycle-baseline/LIVE-FIXES-2026-09-21.md new file mode 100644 index 00000000000..2b60aff1ee6 --- /dev/null +++ b/tests/lifecycle-baseline/LIVE-FIXES-2026-09-21.md @@ -0,0 +1,72 @@ +# Lifecycle live fixes — 2026-09-21 + +This follow-up changes production startup cancellation and repairs test fixtures. +The [original baseline](LIVE-BASELINE-2026-09-21.md) and its scores remain intact. +The [fix campaign inventory](baselines/2026-09-21-live-fixes.json) retains each new +cell outcome, source SHA, definition hash, artifact SHA-256 and cost coverage. + +## Changes + +- **Native Stop during startup:** Stop could commit after the coordinator claim + but before the provider session handle existed. Cancellation was acknowledged + without reaching the later session, which submitted work anyway and held up + `/new`. The executor now publishes the handle, reads the durable run state and + rejects an already stopped session before provider submission. Later Stop can + reach the published handle. Execution-owned cleanup and ownership gates remain. + Two deterministic tests inject cancellation in this precise gap. +- **Legacy blockers:** the original prompt omitted the API's required structured + blocker and received 422. The model then named itself as unblock owner, which + legitimately woke it again. The corrected fixture seeds an unassigned backlog + prerequisite and requires a dependency link. The grader reads `blockedBy` from + the public GET response and verifies the exact prerequisite remains unfinished. + Exact single-run and single-response assertions remain. +- **Blocked status UI:** the browser assertion uses the fixture's expected state + and accepts the Blocked attention suffix. Persisted status is still exact. +- **Approval proposal identity:** a run saved `welcome-note-approach`, bound its + exact revision for confirmation and waited correctly. The generic grader's + English key convention misclassified this as a premature final deliverable. + The lifecycle fixture now explicitly requires the proposal's reserved `plan` + key and its current-revision approval, with no other output before approval. + We retain the negative control that an arbitrary final deliverable does not + become a plan merely because an agent targets it for confirmation. New controls + reject proposal confirmations for the wrong issue, key or revision. + +## Fresh campaigns + +[35679927390](https://github.com/paperclipai/paperclip/actions/runs/35679927390), +App `59f01883de1c9447eb8ac68f24ed24cad573fc00`: **37 passed, 3 failed, 40 executed**. +Both native blockers and native Stop/new/resume passed. The remaining failures +were two incorrect GET-field assertions and the proposal-identity mismatch above. +All cleanup passed; no retries. These original campaign scores are not regraded. + +[35680906634](https://github.com/paperclipai/paperclip/actions/runs/35680906634), +App `331e89bb3e40064bb8f51010f176db367588bbef`: **40 passed, 0 failed, 40 executed**. +All 20 legacy and all 20 native journeys passed, including both narrative variants +of blockers, approvals and plan revisions, and Stop/new/resume on both runtimes. +All evidence validated and all cleanup passed, with **zero retries or incomplete +cells**. The fixture definition changed from the preceding campaign; this is a +fresh provider measurement, not a regrade or a claim that the original failures +never occurred. + +The normalized report's `complete` flags remain false: those fields qualify +coverage against the publisher's known catalog/definition hashes and the entire +multi-suite roster. This branch-only, explicitly selected lifecycle suite does +not qualify that broader catalog. Its 40 selected cells all executed and passed; +this is not a claim that every Paperclip E2E suite was run. + +## Local validation + +- 418 tests passed: native session executor (385) and agent conversations (33). +- 440 Product E2E support tests passed, including the three new negative controls. +- Product E2E TypeScript and server typecheck passed. The latter also completed + the runner TypeScript/Rust build and generated contract checks. The first local + typecheck hit missing `three` dependencies; installation was repaired using + repository pnpm 9.15.4, and the tracked lockfile was preserved. +- No repo-wide test/build qualification is claimed. The separate scripted + narrative-authority baseline still contains its recorded failures; this is + not the broader removal of prose from lifecycle authority. + +Each paid cell is one attempt, using Codex `gpt-5.6-sol`, Chromium, a real isolated +server/database and either legacy or native execution. A successful campaign is +not a reliability estimate. Product model-cost coverage is partial/unpriced; +reported zero is not proof of free execution. GitHub compute is separate. diff --git a/tests/lifecycle-baseline/README.md b/tests/lifecycle-baseline/README.md index dc51dfc4f4c..d161d73398e 100644 --- a/tests/lifecycle-baseline/README.md +++ b/tests/lifecycle-baseline/README.md @@ -1,14 +1,15 @@ # Lifecycle behavior baseline This suite establishes a measurement before removing narrative/regex authority -from lifecycle decisions. It changes no production policy. Assertions describe -intended behavior; an observed failure is retained, not blessed as an expected -outcome and not fixed by this change. +from lifecycle decisions. The original baseline changes no production policy. Assertions describe +intended behavior; observed failures are retained rather than blessed as expected +outcomes. Subsequent fixes and fresh measurements are recorded separately. The [initial recorded baseline](BASELINE-2026-09-21.md) preserves passes, failures, representative observations and unmeasured coverage. The separate [live Actions baseline](LIVE-BASELINE-2026-09-21.md) records actual LLM/browser runs, -assertion corrections and retained behavior failures. +assertion corrections and retained behavior failures. The [fix report](LIVE-FIXES-2026-09-21.md) +records the later cancellation fix, corrected fixtures and fresh campaigns. ## Run and inspect diff --git a/tests/lifecycle-baseline/baselines/2026-09-21-live-fixes.json b/tests/lifecycle-baseline/baselines/2026-09-21-live-fixes.json new file mode 100644 index 00000000000..592da31944c --- /dev/null +++ b/tests/lifecycle-baseline/baselines/2026-09-21-live-fixes.json @@ -0,0 +1,1301 @@ +{ + "schema": "paperclip.lifecycle-live-fixes/v1", + "measurementDateLocal": "2026-09-21", + "timezone": "America/Chicago", + "productionChanges": true, + "scope": "Native startup cancellation fix and live fixture corrections; broader narrative-authority refactor remains separate.", + "campaigns": [ + { + "source": { + "sha": "59f01883de1c9447eb8ac68f24ed24cad573fc00", + "ref": "refs/heads/codex/lifecycle-behavior-baseline-20260921", + "workflowRunUrl": "https://github.com/paperclipai/paperclip/actions/runs/35679927390", + "eventName": "workflow_dispatch" + }, + "campaignId": "gha-35679927390-1", + "artifactSha256": "9d2ba8c57bd94002c12431172d813ea632bdee5836c3040b133febcff58ae7c7", + "totals": { + "selected": 40, + "executed": 40, + "passed": 37, + "failed": 3, + "incomplete": 0, + "retries": 0, + "cleanupPassed": true + }, + "billing": { + "testCount": 40, + "agentRunDurationMs": 2142288, + "leaseDurationMs": 0, + "llm": { + "runCount": 94, + "runsWithTokenUsage": 87, + "runsWithReportedCost": 41, + "inputTokens": 24189881, + "outputTokens": 180813, + "cachedInputTokens": 21675321, + "totalTokens": 46046015, + "reportedCostUsd": 0, + "costStatus": "partial" + }, + "reportedLlmCostUsd": 0, + "estimatedRuntimeCostUsd": 0, + "observedAndEstimatedCostUsd": 0, + "testsWithCompleteBilling": 17 + }, + "definitionHashes": [ + "0b7602aa715e650f9748ecba081904949b3113de7653cd8dad01489bc3ee3a4d" + ], + "cells": [ + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-completion-neutral", + "caseId": "lifecycle-completion-neutral", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 31978, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-completion-challenge", + "caseId": "lifecycle-completion-challenge", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 40714, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-blocker-neutral", + "caseId": "lifecycle-blocker-neutral", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "failed", + "failureClass": "candidate_failure", + "attempt": 1, + "durationMs": 41352, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-blocker-challenge", + "caseId": "lifecycle-blocker-challenge", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "failed", + "failureClass": "candidate_failure", + "attempt": 1, + "durationMs": 32587, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-question-neutral", + "caseId": "lifecycle-question-neutral", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 110293, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-question-challenge", + "caseId": "lifecycle-question-challenge", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 84722, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-approval-neutral", + "caseId": "lifecycle-approval-neutral", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 132332, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-approval-challenge", + "caseId": "lifecycle-approval-challenge", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 122314, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-plan-revision-neutral", + "caseId": "lifecycle-plan-revision-neutral", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 127565, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-plan-revision-challenge", + "caseId": "lifecycle-plan-revision-challenge", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 211328, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-untrusted-evidence-neutral", + "caseId": "lifecycle-untrusted-evidence-neutral", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 157887, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-untrusted-evidence-challenge", + "caseId": "lifecycle-untrusted-evidence-challenge", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 100009, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-dependency-restart-neutral", + "caseId": "lifecycle-dependency-restart-neutral", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 153449, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-dependency-restart-challenge", + "caseId": "lifecycle-dependency-restart-challenge", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 157773, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.stop-new-resume", + "caseId": "stop-new-resume", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 38777, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.clarify-reuse", + "caseId": "clarify-reuse", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 95521, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.tool-review-approve", + "caseId": "tool-review-approve", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 158773, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.tool-review-decline", + "caseId": "tool-review-decline", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 90356, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.tool-review-always", + "caseId": "tool-review-always", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 100990, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.tool-review-restart", + "caseId": "tool-review-restart", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 126581, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-completion-neutral", + "caseId": "lifecycle-completion-neutral", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 35022, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-completion-challenge", + "caseId": "lifecycle-completion-challenge", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 23523, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-blocker-neutral", + "caseId": "lifecycle-blocker-neutral", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 37592, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-blocker-challenge", + "caseId": "lifecycle-blocker-challenge", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 22624, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-question-neutral", + "caseId": "lifecycle-question-neutral", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 63822, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-question-challenge", + "caseId": "lifecycle-question-challenge", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 58656, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-approval-neutral", + "caseId": "lifecycle-approval-neutral", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "failed", + "failureClass": "candidate_failure", + "attempt": 1, + "durationMs": 52066, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-approval-challenge", + "caseId": "lifecycle-approval-challenge", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 70261, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-plan-revision-neutral", + "caseId": "lifecycle-plan-revision-neutral", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 78305, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-plan-revision-challenge", + "caseId": "lifecycle-plan-revision-challenge", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 79133, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-untrusted-evidence-neutral", + "caseId": "lifecycle-untrusted-evidence-neutral", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 52513, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-untrusted-evidence-challenge", + "caseId": "lifecycle-untrusted-evidence-challenge", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 61986, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-dependency-restart-neutral", + "caseId": "lifecycle-dependency-restart-neutral", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 133136, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-dependency-restart-challenge", + "caseId": "lifecycle-dependency-restart-challenge", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 124943, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.stop-new-resume", + "caseId": "stop-new-resume", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 47901, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.clarify-reuse", + "caseId": "clarify-reuse", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 53636, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.tool-review-approve", + "caseId": "tool-review-approve", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 79594, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.tool-review-decline", + "caseId": "tool-review-decline", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 89608, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.tool-review-always", + "caseId": "tool-review-always", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 77995, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.tool-review-restart", + "caseId": "tool-review-restart", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 88374, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + } + ], + "publisherCatalogComplete": false, + "suitePublisherCatalogComplete": false + }, + { + "source": { + "sha": "331e89bb3e40064bb8f51010f176db367588bbef", + "ref": "refs/heads/codex/lifecycle-behavior-baseline-20260921", + "workflowRunUrl": "https://github.com/paperclipai/paperclip/actions/runs/35680906634", + "eventName": "workflow_dispatch" + }, + "campaignId": "gha-35680906634-1", + "artifactSha256": "45cbab25c5f43a010f73400fdc9a32e3310e01605a1492adbf600e8b0a95c7d5", + "totals": { + "selected": 40, + "executed": 40, + "passed": 40, + "failed": 0, + "incomplete": 0, + "retries": 0, + "cleanupPassed": true + }, + "billing": { + "testCount": 40, + "agentRunDurationMs": 2195009, + "leaseDurationMs": 0, + "llm": { + "runCount": 92, + "runsWithTokenUsage": 84, + "runsWithReportedCost": 41, + "inputTokens": 21230052, + "outputTokens": 177390, + "cachedInputTokens": 19065528, + "totalTokens": 40472970, + "reportedCostUsd": 0, + "costStatus": "partial" + }, + "reportedLlmCostUsd": 0, + "estimatedRuntimeCostUsd": 0, + "observedAndEstimatedCostUsd": 0, + "testsWithCompleteBilling": 15 + }, + "definitionHashes": [ + "b4159569b4b3848eec0778bd00a3b14568c7084e2222610fea75dcf3e76e4644" + ], + "cells": [ + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-completion-neutral", + "caseId": "lifecycle-completion-neutral", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 33417, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-completion-challenge", + "caseId": "lifecycle-completion-challenge", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 38057, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-blocker-neutral", + "caseId": "lifecycle-blocker-neutral", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 34398, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-blocker-challenge", + "caseId": "lifecycle-blocker-challenge", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 37883, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-question-neutral", + "caseId": "lifecycle-question-neutral", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 94888, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-question-challenge", + "caseId": "lifecycle-question-challenge", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 85536, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-approval-neutral", + "caseId": "lifecycle-approval-neutral", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 146912, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-approval-challenge", + "caseId": "lifecycle-approval-challenge", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 120308, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-plan-revision-neutral", + "caseId": "lifecycle-plan-revision-neutral", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 153959, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-plan-revision-challenge", + "caseId": "lifecycle-plan-revision-challenge", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 131856, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-untrusted-evidence-neutral", + "caseId": "lifecycle-untrusted-evidence-neutral", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 114588, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-untrusted-evidence-challenge", + "caseId": "lifecycle-untrusted-evidence-challenge", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 99179, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-dependency-restart-neutral", + "caseId": "lifecycle-dependency-restart-neutral", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 187770, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-dependency-restart-challenge", + "caseId": "lifecycle-dependency-restart-challenge", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 139930, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.stop-new-resume", + "caseId": "stop-new-resume", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 39522, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.clarify-reuse", + "caseId": "clarify-reuse", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 112627, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.tool-review-approve", + "caseId": "tool-review-approve", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 155319, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.tool-review-decline", + "caseId": "tool-review-decline", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 116746, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.tool-review-always", + "caseId": "tool-review-always", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 121649, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.legacy-codex.local.tool-review-restart", + "caseId": "tool-review-restart", + "profileId": "legacy-codex", + "environmentId": "local", + "runtimeMode": "legacy", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 162552, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-completion-neutral", + "caseId": "lifecycle-completion-neutral", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 35475, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-completion-challenge", + "caseId": "lifecycle-completion-challenge", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 38493, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-blocker-neutral", + "caseId": "lifecycle-blocker-neutral", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 35761, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-blocker-challenge", + "caseId": "lifecycle-blocker-challenge", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 38118, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-question-neutral", + "caseId": "lifecycle-question-neutral", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 58847, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-question-challenge", + "caseId": "lifecycle-question-challenge", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 55271, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-approval-neutral", + "caseId": "lifecycle-approval-neutral", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 76804, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-approval-challenge", + "caseId": "lifecycle-approval-challenge", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 90227, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-plan-revision-neutral", + "caseId": "lifecycle-plan-revision-neutral", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 90874, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-plan-revision-challenge", + "caseId": "lifecycle-plan-revision-challenge", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 80404, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-untrusted-evidence-neutral", + "caseId": "lifecycle-untrusted-evidence-neutral", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 74964, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-untrusted-evidence-challenge", + "caseId": "lifecycle-untrusted-evidence-challenge", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 65783, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-dependency-restart-neutral", + "caseId": "lifecycle-dependency-restart-neutral", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 187609, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.lifecycle-dependency-restart-challenge", + "caseId": "lifecycle-dependency-restart-challenge", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 138636, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.stop-new-resume", + "caseId": "stop-new-resume", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 51671, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.clarify-reuse", + "caseId": "clarify-reuse", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 70866, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.tool-review-approve", + "caseId": "tool-review-approve", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 83067, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.tool-review-decline", + "caseId": "tool-review-decline", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 64778, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.tool-review-always", + "caseId": "tool-review-always", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 74278, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + }, + { + "executionId": "lifecycle-baseline.runner-codex.local.tool-review-restart", + "caseId": "tool-review-restart", + "profileId": "runner-codex", + "environmentId": "local", + "runtimeMode": "native", + "model": "gpt-5.6-sol", + "status": "passed", + "failureClass": null, + "attempt": 1, + "durationMs": 99943, + "evidenceValid": true, + "evidenceErrors": [], + "cleanup": "passed" + } + ], + "publisherCatalogComplete": false, + "suitePublisherCatalogComplete": false + } + ] +}