Record continuation accounting baseline and retained live failures

This commit is contained in:
Dotta committed 2026-09-22 20:26:23 -07:00
1 parent 379a70315e
commit c0b5ee8b38
8 files changed
+14367 -3

No files matched your search

@@ -29,7 +29,7 @@ new paid productive cases use public questions and responses instead.
| Repeated missing disposition | Existing repair episode unit/DB tests; complete sweep after exhaustion | Initial turn plus exactly two legacy repairs; visible board recovery, blocked task |
| Comments/confident prose/tool calls without state change | Quiet/noisy exhausted repair and failure variants; native event replay | Quiet versus three distinct attributed misleading comments per run; no extra allowance |
| Late approval/pause/spending/ownership gate | Second repair delayed, new gate, service restart, same debit; existing admission tests cover both runtimes | Approval created by first repair owns wait; acceptance causes one completion; Stop before second repair dispatch |
| Restart while a repair is scheduled | Durable episode/dispatch tests | Controller restart before delayed repair; identical run IDs and episode counters |
| Restart while a repair is scheduled | Actual delayed-retry promotion as well as the repair gate; retain episode and debit | Controller restart before delayed repair; identical run IDs and episode counters; fail with the persisted cancellation reason if promotion cancels it |
| Duplicate/out-of-order handling | Concurrent retry/recovery tests and native consumed-repair replay with commentary/tool events | Exact run counts, document revisions and retained IDs; exhaustive races stay deterministic |
| No reset after exhaustion; legitimate new request | Existing heartbeat exhausted failure versus actual new user request | No user message during exhaustion; a real approval resumes its owned path |
| Missing or misleading evidence | Calibrated oracle rejects missing state, wrong documents, overwritten steps, fabricated progress, extra runs, lost receipts, decline, executed Stop | Every live result uses that oracle and existing screenshot/evidence/billing pipeline |
@@ -64,3 +64,11 @@ spending gates and timing races use deterministic fixtures rather than artificia
spending money or hoping for a real outage. The original native compatibility
probes, blank-route reload finding and duplicate-response finding remain in the
[retained backlog](2026-09-22-legacy-continuation-authority.md).
## Recorded baseline
The [measurement report](../../tests/lifecycle-baseline/CONTINUATION-ACCOUNTING-2026-09-22.md)
retains the deterministic inventory and each paid campaign without regrading.
The new suite exposes shared allowance accounting in both directions and a
delayed-repair promotion mismatch. These are production follow-ups; the test
slice leaves their intended-behavior assertions enabled and failing.
@@ -123,8 +123,8 @@ Keep this section even if this slice incidentally resolves some assertions.
| Commentary counts as progress | 1 | Retain for progress-evidence audit; no comment-count budget resets in this slice |
| Wording selects liveness versus handoff path in heartbeat | 1 | Retain separately in results; must be covered while closing the current shared authority boundary |
| Native autonomous continuation compatibility probes | 2 | Replace with reachable public-contract tests; `same_agent` injection is not a current model-facing defect |
| Productive continuation versus repair/failure budgets | Coverage boundary | Separate audit after this slice; do not infer progress from text/event counts |
| Blank task route after reload | New live observation | Investigate browser bootstrap/rendering; seen after successful persisted lifecycle in both runtimes; campaigns 35744074887 and 35747200170 |
| Productive continuation versus repair/failure budgets | Three new findings | [Accounting baseline](2026-09-22-continuation-accounting-baseline.md) exposes delayed repair spending infrastructure allowance, infrastructure retries spending productive allowance, and delayed repair cancellation during promotion. Test-only slice; production fixes remain open |
| Blank task route after reload | Repeated live observation | Investigate browser bootstrap/rendering; campaigns 35744074887 and 35747200170, plus accounting campaign 35813099816 after two correct legacy steps with a pending question. Empty screenshot retained; full quiet productive case remains unverified |
| Duplicate legacy response | One later live observation | Campaign 35805477715 legacy work-mode neutral: provider issued two successful PATCH calls, second escaping an underscore. Retain as model behavior evidence; exact-once matcher rejected it |
| Missing-comment policy ownership | Coverage boundary | Existing typed retry policy remains; review separately if consolidating all post-run compliance into one disposition contract |
| Maintain CI and prepare App/Evals review | Pending | Promote fixed invariants into maintained cheap gates; paid suites remain explicit |
@@ -0,0 +1,138 @@
# Continuation accounting baseline — September 22, 2026
This is a test-first slice. Production scheduling/authority code is unchanged.
The [scenario matrix](../../doc/plans/2026-09-22-continuation-accounting-baseline.md)
and executable inventory cover separate allowances, false progress, late gates,
restart and duplicate handling. There are 67 additional deterministic tests
and eight new explicit-only real-provider Product E2E cells.
## Deterministic measurement
Source: `379a70315e47de1ac2c8e4eb452eb3121401c5c7`, no tracked source changes at measurement.
Raw evidence: `.lifecycle-baseline/2026-09-23T03-04-36.107Z/`.
[Saved inventory and results](baselines/2026-09-22-continuation-accounting.json).
| Layer | Passed / total | Failed |
|---|---:|---:|
| Unit | 413 / 414 | 1 |
| Runner | 184 / 184 | 0 |
| Database/heartbeat integration | 371 / 376 | 5 |
| Grading | 85 / 85 | 0 |
| Total | 1,053 / 1,059 | 6 |
No skipped tests or missing inventory evidence. Four newly failing tests
identify three production problems:
1. **Delayed repair spends infrastructure allowance.** Immediate first repair
has no scheduled retry debit and passes. Delayed repair 2 has
`scheduledRetryAttempt: 2`, `scheduledRetryReason: issue_disposition_repair`.
`executionFailureRetryCount` interprets it as two prior failures, so a first
safe infrastructure failure is already exhausted. One pure assertion and one
actual scheduler assertion expose this.
2. **Infrastructure retries spend productive allowance.** A run resumed after
two infrastructure retries reaches its first typed max-turn continuation.
The scheduler reads the same counter as two consumed productive attempts and
returns exhausted. The test expects productive attempt 1.
3. **The delayed second repair is cancelled during promotion.** The legacy
repair dispatch gate accepts the persisted episode, but the shared dispatch
path compares its fingerprint against a different disposition fingerprint.
It cancels the unstarted repair with
`issue_disposition_repair_superseded`. The new database test reproduces this
through actual due-retry promotion after recreating the controller service.
The first live exhaustion run also retained this exact cancellation, with
only two provider executions and no final exhaustion checkpoint.
The other two failures are the previously retained native `same_agent`
compatibility probes. They inject an unsupported current public continuation
shape and do not establish a current model-facing defect. The new paid native
productive tests use reachable question/response contracts.
The tests also prove that twenty comments or a large raw tool count do not
replenish exhausted legacy repairs, including through the full recovery sweep
while the issue is still active. A new approval, company budget hard-stop,
agent pause or reassignment during the second repair delay blocks dispatch
without another debit. Native replay with twenty commentary/tool events does
not resubmit consumed disposition repair. Existing gates and duplicate/restart
regressions remain linked in the inventory.
## Other verification
- Product E2E support: 474/474 passing, including 31 new accounting support tests.
- Inventory/report support: 4/4 passing.
- E2E and lifecycle baseline typechecks pass.
- Server and runner package typechecks pass; server preparation also completes
the runner build and generated-contract checks.
- Full general repository tests, recursive typecheck and root build were not
rerun: this slice changes only tests, harness and documentation. This is not
a claim of repository-wide green or PR readiness. The new intended-behavior
failures remain enabled and will fail their normal test lane until fixed.
The [initial deterministic measurement](baselines/2026-09-22-continuation-accounting-initial.json)
at `ed53efb1e0e363b382b5101340259ad3cab34307` is retained separately: 1,051/1,056.
The later total adds the promotion regression and two harness calibration tests.
## Live measurement
All cells use qualified Codex `gpt-5.6-sol`, real Chromium, and isolated local
server/database fixtures in parallel GitHub Actions jobs. Original machine
grades, source/definition fingerprints, cleanup and billing are retained for
each campaign; unavailable cost is not zero spending.
First campaign: [35811412940](https://github.com/paperclipai/paperclip/actions/runs/35811412940),
source `9cd7a585f5c69dc327b7d90534ad51be89ae5bb7`. All eight cells failed.
This measurement is not regraded. Initial fixtures requested three identical
comments, which the product deduplicates per run; rich-text serialization also
escaped underscores in literal test tokens. Screenshot names did not meet the
existing evidence allowlist, so the public report publisher rejected the bundle.
Legacy quiet productive behavior passed every causal check before packaging
failed. Approval and Stop lifecycle checks also passed while their comment
perturbation checks failed. Both exhaustion cells failed to reach the expected
exhaustion checkpoint; retained API state identifies the cancelled delayed repair.
Corrected fixtures use three distinct numbered comments, plain alphanumeric
tokens and existing allowed screenshot names. A packaging test verifies those
screenshots survive. Cancellation fails promptly with its typed reason and the
last observed state, instead of waiting for the overall deadline. These are
test-harness changes, not changes to production behavior.
Corrected campaign: [35813099816](https://github.com/paperclipai/paperclip/actions/runs/35813099816),
source `379a70315e47de1ac2c8e4eb452eb3121401c5c7`:
**5/8 passed, 3 failed**, no missing cells or retries, all eight evidence packages
valid and cleanup successful. The corrected public report published successfully.
| Cell | Result | Observed behavior |
|---|---|---|
| Legacy productive, quiet | Fail | After two successful runs, two exact revision-one records, and a pending next question, a task-route reload renders a blank page. `task-chat-thread` never appears within 30 seconds. Remaining three steps are unverified |
| Legacy productive, noisy | Pass | Five productive runs, five exact revision-one documents, four real answered questions, no repair/failure debits |
| Native productive, quiet | Pass | Same five-step causal assertions |
| Native productive, noisy | Pass | Same five-step causal assertions despite misleading attributed comments |
| Legacy exhaustion, quiet | Fail | Second repair cancelled before dispatch with `issue_disposition_repair_superseded` |
| Legacy exhaustion, noisy | Fail | Identical typed cancellation despite the additional comments |
| Legacy repair then Stop | Pass | Delayed second repair cancelled by the operator before dispatch; remains stopped after controller restart and scheduled due time |
| Legacy repair then approval | Pass | Pending approval owns the wait; real browser acceptance produces exactly one successful completion without repair debt |
The blank-page screenshot is empty. This matches the symptom of the
retained task-route reload finding; it does not establish the underlying UI
cause, and it is not counted as a completed productive case. The successful
native noisy final screenshot shows Done and all five document cards at revision
one. Both exhaustion checkpoints retain the scheduled and restarted run IDs;
their last observations include the cancelled unstarted repair and a board-owned
`active_run_watchdog` action instead of completed disposition-repair exhaustion.
- [Published corrected report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35813099816-1/index.html)
- [Both immutable campaign measurements, fingerprints, grades and billing](baselines/2026-09-22-continuation-accounting-live.json)
Billing is partial in both campaigns. Provider-reported zero totals do **not**
establish free execution: the first campaign records token usage for 29/32 run
records and reported cost for 10; the corrected campaign records usage for 26/29
and reported cost for 10. Each includes three cancelled, unstarted records.
The saved billing fields preserve the original coverage and cost status.
## Next production slice
Fix allowance separation and the delayed-repair promotion contract against the
enabled deterministic failures, then rerun the same eight-cell suite. Keep the
blank-route finding, unsupported native compatibility probes and earlier backlog
distinct. This baseline establishes assertions and observed behavior; it does
not claim the product is already robust or that every supported provider is
qualified.
+2
View File
@@ -150,3 +150,5 @@ E2E scores, or claim full qualification from a partial selection.
September 22 follow-up: [legacy continuation implementation and verification](LEGACY-CONTINUATION-2026-09-22.md), including preserved failed campaigns and the remaining backlog.
The [continuation accounting matrix](../../doc/plans/2026-09-22-continuation-accounting-baseline.md) adds ACCT-01 through ACCT-04 for separate allowances, false progress, late gates and restart/replay. Its real-provider companion is the explicit-only `continuation-accounting` Product E2E suite.
The [September 22 measurement](CONTINUATION-ACCOUNTING-2026-09-22.md) records the
enabled failures and preserves both initial and corrected live campaigns.
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
@@ -36,3 +36,7 @@ See the [scenario matrix](../../doc/plans/2026-09-22-continuation-accounting-bas
for deterministic fault, replay, spending and ownership coverage. Infrastructure
failures are not induced in paid cells; their cross-lane allowance semantics are
covered by actual scheduler tests with controlled failures.
Results and follow-ups are retained in the [accounting measurement report](../lifecycle-baseline/CONTINUATION-ACCOUNTING-2026-09-22.md),
including the original failed fixture campaign. A new measurement never replaces
or regrades an earlier campaign.