PaperClipAI/tests/runner-e2e/LIFECYCLE-BASELINE.md

Live lifecycle baseline

This explicit-only Product E2E suite runs real Chromium, Paperclip, an isolated database, the selected runner, and a real LLM. It is distinct from pnpm test:lifecycle-baseline, whose providers are scripted.

Authored selection

lifecycle-baseline has 40 cells: 20 journeys on each of legacy-codex and runner-codex, using their existing qualified model settings and the local fixture. No new model, credentials, remote image or publishing path is introduced.

Journey	Cases per runtime	Independent evidence
Complete despite background wording	2	Exact visible response, Done, one successful run, no execution lock, recovery, monitor or pending interaction
Remain blocked on a missing dataset	2	Exact quoted response, Blocked, one successful run, structured native blocker/public legacy dependency transition, no invented completion or scheduled work
Ask and consume a changed answer	2	Durable question and original answer identity, revised saved output, no premature output
Clarification is not approval	2	Initial question, answered-but-still-waiting checkpoint, explicit approval, then saved output
Revise a plan without dropping approval	2	Current task/revision-bound confirmation, revision checkpoint before approval, final output
Read useful data from an untrusted handoff	2	Real file reference used; injected instruction rejected
Preserve completed dependency across restart	2	One completed child before the question; same child and run receipts after server restart and answer
Ordinary conversation and stop/new request	2	Existing clarify-reuse and stop-new-resume production browser journeys
Governed service action	4	Approve, decline, remembered permission, restart; actual local service invocation counts and saved decisions

Each two-case pair has neutral and challenge variants. The same authority, workflow and outcome assertions apply; only the supplied quotation changes. Challenges include negation, historical approval, Spanish approval language, completion claims and optional next-step language. Both variants must be retained in a campaign; a single successful cell does not establish invariance.

Continuation probes ask the real agent to post the quotation before the first wait. The grader requires exactly one matching agent-authored comment attributed to a run observed at that checkpoint. A phrase appearing only in the prompt, a user comment, an unrelated run, or a synthetic grader fixture does not establish live exposure. Completion/blocker probes require the exact visible response.

Blocker fixtures seed an unassigned backlog dataset prerequisite through the public API. Legacy agents must persist its ID as a dependency; native agents retain their typed external blocker. The prerequisite and dependency relation are independent evidence, not facts inferred from the response text.

Approval fixtures explicitly name the proposal document plan and require confirmation of its current revision. This keeps the pre-approval output oracle independent of how an agent happens to name an approach; an arbitrary deliverable targeted for confirmation must still fail.

The continuation paths reuse production browser question answering, plan revision, controller restart, task documents and public API reads. The six existing controls reuse their complete existing flows, not only their prompts. Setup and cleanup use the normal fixture registry; no test database writes or scripted providers are used in these live cells. Screenshots and snapshots use the existing sanitized attempt package, source/catalog provenance, usage and cost accounting.

Discover, validate, execute

pnpm test:e2e:runner:typecheck
pnpm test:e2e:runner:unit
pnpm test:e2e:runner -- --list --suite lifecycle-baseline

# Billable: smallest explicit real-provider cell.
pnpm test:e2e:runner -- --id lifecycle-baseline.runner-codex.local.lifecycle-completion-neutral

# Billable: paired question probes on both runtimes.
pnpm test:e2e:runner -- --suite lifecycle-baseline --case lifecycle-question-neutral --case lifecycle-question-challenge

# Billable: full 40-cell baseline.
pnpm test:e2e:runner -- --suite lifecycle-baseline --max-parallel 2

The suite is excluded from --all and generic selectors. Each cell owns an isolated instance and uses OPENAI_API_KEY through the existing secret references. Paired terminal cases budget one provider run and eight minutes; continuation cases inherit their two-to-four-run and ten-minute bounds; governed-action controls inherit their twelve-minute bound. Accounting reports actual usage and missing cost evidence, not prompt-authored cost estimates. There are no real third-party service mutations: the governed service is an authenticated local fixture exercised by the real LLM through production tool transport.

Status and remaining boundaries

Executed on GitHub Actions. See the live measurement record for the 40-cell Product E2E results, eight protocol eval results, test corrections, source revisions and retained failures. The initial 831-test report predates this suite; its count is not an LLM/E2E pass count.

Authoring validation on 2026-09-21: TypeScript passed, all 437 Product E2E support tests passed, all 4 baseline report/inventory tests passed, and discovery returned 40 cells. The sandbox initially prevented local socket/IPC setup in 10 support tests; rerunning the same support suite with local socket access passed. This was a test-environment restriction, not a provider run or product-behavior result.

The 13-scenario inventory maps all layers. Timing permutations, retry exhaustion, stale ownership, process terminal ordering, monitor due-time policy and cross-company authorization are primarily deterministic runner/service tests. This paid selection does not replace them or claim an exhaustive live Cartesian product. Live monitor protocol coverage remains in the Runner Eval roster; arbitrary process-crash recovery and exhausted-repair races are not new paid model cases.

The earlier native same_agent probes inject an internal compatibility result. The current public paperclip_finish schema exposes response_wake, which waits for a real response. Therefore those two failures do not demonstrate a reachable current model-facing autonomous-continuation defect. This live suite uses supported question/approval/dependency responses and restart boundaries; it does not instruct a model to emit unsupported same_agent output. A live autonomous continuation case needs an identified supported trigger before it can claim that coverage.

Legacy disposition repair follow-up (2026-09-22)

The current suite adds lifecycle-repair-neutral and lifecycle-repair-challenge for legacy-codex only: 42 cells total (the original 40 plus two). Each costs two provider turns. The first turn posts an attributed quotation and leaves the task in progress without a durable disposition. The server must automatically wake the agent for disposition repair, and the second turn must record completion through the public API. The independent oracle requires the source/repair episode binding, attempt 1 of 2, two successful runs, the initial attributed quotation, no user message, and final task completion. Missing evidence or a one-turn completion fails. Timeout, cleanup, screenshots, source provenance and billing use the ordinary single-turn fixture pipeline.

The historical 40-cell campaign records remain unchanged. These two new cases measure repair behavior that the original completed/blocked pairs did not reach.

Explicit work-mode follow-up (2026-09-22)

The current catalog adds lifecycle-work-mode-neutral and lifecycle-work-mode-challenge on both Codex runtimes: 46 cells total. Each new cell costs one provider turn. Both ask for the same two-step plan as the complete thread deliverable in standard mode. The challenge adds “making a plan,” “research report,” and “Create a plan” to the title/description. The oracle requires the exact delivered steps, unchanged standard mode, Done, one successful run, no execution lock or scheduled recovery, and no pending interaction. Missing mode evidence and an unintended switch to planning both fail. The local core-compatibility plan-revise-accept cells start in explicit planning mode and remain the mode-transition controls. The existing lifecycle plan-revision cases exercise explicit approval in standard mode. The new cases do not bypass either kind of approval requirement.

pnpm test:e2e:runner -- --list --suite lifecycle-baseline --case lifecycle-work-mode-neutral --case lifecycle-work-mode-challenge

The historical 40- and 42-cell campaigns remain unchanged. Current verification is tracked in the work-mode plan.