PaperClipAI/tests/lifecycle-baseline/README.md

Lifecycle behavior baseline

This suite establishes a measurement before removing narrative/regex authority from lifecycle decisions. It changes no production policy. Assertions describe intended behavior; an observed failure is retained, not blessed as an expected outcome and not fixed by this change.

The initial recorded baseline preserves passes, failures, representative observations and unmeasured coverage. The separate live Actions baseline records actual LLM/browser runs, assertion corrections and retained behavior failures.

Run and inspect

From an installed Paperclip checkout:

pnpm test:lifecycle-baseline --list
pnpm test:lifecycle-baseline unit
pnpm test:lifecycle-baseline runner
pnpm test:lifecycle-baseline integration
pnpm test:lifecycle-baseline grading
pnpm test:lifecycle-baseline
pnpm test:lifecycle-baseline:support
pnpm exec tsc -p tests/lifecycle-baseline/tsconfig.json

All four lanes are credential-free. The integration lane uses disposable embedded Postgres and scripted providers. No command above invokes a model, starts a paid campaign, or changes an existing Paperclip instance. Tests are outside default server/workspace discovery; Product E2E matcher calibration remains in its normal opt-in support suite. The baseline command returns nonzero on failed assertions, missing evidence, or unavailable selected coverage. Ordinary CI is unaffected.

Each invocation retains an independent directory under .lifecycle-baseline/:

• baseline.md and baseline.json: scenario inventory joined to actual assertions;
• per-lane Vitest JSON: complete assertion results, durations, and failures;
• observation JSONL: fixture inputs/classifications and actual authority effects, including both sides of narrative pairs and persisted heartbeat outcomes;
• source.diff: tracked implementation delta from the recorded commit.

The commit plus fingerprint covers tracked differences and untracked authored files. Keep the worktree or commit its tests with retained measurements when comparing revisions. The scripted report marks live coverage not_run; use the separate live Actions record for provider measurements. Authored definitions and passing grader calibration are not proof of real provider behavior. A test suite that cannot load is an evidence/harness failure, not a product finding. Skips are unavailable coverage, never a pass. Failed assertions require triage; raw runner/provider text is not uploaded by this command.

Scenario inventory

inventory.mjs is the executable mapping. References to existing suites reuse their actual assertions rather than duplicating test bodies or counting catalog entries as executed tests. The report records which matching assertions ran.

ID	Scenario	Contract
LCA-01	Ordinary completion	Valid completion, delivered response, no extra execution
LCA-02	Productive multiple turns	Continue appropriate work without repeating completed operations
LCA-03	Human question	Durable question, matching response, one causal continuation
LCA-04	Approval/decline	Correct actor and approval; other admission gates still apply
LCA-05	Planning/revision	Work mode and exact accepted revision authorize execution
LCA-06	Dependencies	Satisfied dependency condition wakes the parent once
LCA-07	External monitor	Durable eligible wait, one-shot wake, bounded expiry/retry
LCA-08	Conversation	Deliver the reply without forcing task completion or repair loops
LCA-09	Missing disposition	Bounded explicit repair; comments/restarts do not reset attempts
LCA-10	Stop/pause/budget	Preserve distinct semantics; no unauthorized continuation
LCA-11	Terminal ordering	Completion report, provider terminal, and cleanup remain distinct
LCA-12	Replay/restart/ownership	Preserve receipts and budgets; fence stale finalizers
LCA-13	Review	Concrete reviewer/owner and authoritative review outcome

Troublesome combinations included in the inventory and reused suites:

• Completion followed by failure/cancellation or stream closure without terminal.
• Approval while paused/over budget; response arriving during input handoff/cleanup.
• Reassignment/closure before late finalization; restart between commit and delivery.
• Exhaustion followed by commentary versus an authorized new user request.
• Stale plan revision approval; duplicate question/dependency/reconciler events.
• Wrong company/task or unauthorized resolution; productive versus failure retries.

Narrative pairs and positive controls

authority.test.ts composes the real legacy classifier with its actual continuation consumer, so assertions compare scheduling decisions rather than requiring diagnostic labels to be identical. Native pairs exercise the actual status arbiter. The heartbeat cases cross the real persistence/finalization boundary for both runtimes, including explicit native continuation and the legacy missing-disposition path. Observations precede cleanup; a second queue/drain pass checks for additional dispatch. This is not proof about arbitrary future timers: the existing monitor/retry/reconciler suites separately exercise due-time policy.

Vary summaries/results, existing comment bodies, continuation summaries, stdout, stderr, titles, and descriptions. Variants cover negation, historical quotation, Spanish, optional next steps, unsupported completion claims, encouraging prose, and empty narrative. Commentary-only evidence must not manufacture progress. Keep authority identical for each pair. Positive controls change real status, approval, continuation, budget, or ownership with identical prose. New authenticated user messages are separate causal events, not interchangeable text.

Scripted runner tests validate actual session/event behavior and structured-result contracts. They do not prove OS termination or real provider compliance. Native replacement tests inject verifier evidence at their documented boundary. Route unit tests mock services; database suites test persisted application behavior. These proof boundaries must remain visible when interpreting the baseline.

Live evals

The dedicated live lifecycle suite adds 40 explicitly selected real-LLM/browser cells on both runtime generations. It was authored after the initial scripted baseline and has now run on GitHub Actions; see the separate live record for measured outcomes. Use pnpm test:e2e:runner -- --list --suite lifecycle-baseline to inspect it.

Product E2E uses the existing continuation, agent-chat, and everyday-workflows catalogs. Continuation now retains an explicit lifecycle snapshot at each browser checkpoint and grades pending question identity, plan revision binding, original run receipts, and the absence of execution/recovery paths after completion. The grader has valid/wrong/missing-evidence calibration. The continuation definition version advances so measurements are distinguishable.

Validate/discover without spending:

pnpm test:e2e:runner:typecheck
pnpm test:e2e:runner:unit
pnpm test:e2e:runner -- --list --suite continuation
pnpm test:e2e:runner -- --list --suite agent-chat
pnpm test:e2e:runner -- --list --suite everyday-workflows

The sibling paperclip-evals change adds the opt-in rosters/live-lifecycle-narrative-baseline.json: ordinary finish/block controls, two misleading-summary variants, and question/review/dependency/wake cases. It does not expand the maintained paid campaign. Both cases and the roster are validated without providers; successful, wrong-state, missing-tool, and extra-wake evidence calibrate their actual grader.

For subsequent live execution use existing explicit selectors and record exact App/Evals revisions, profile/environment, retries, usage/cost, and artifact IDs. See doc/evals.md. Do not combine mock-authority Runner Eval scores with Product E2E scores, or claim full qualification from a partial selection.