mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-25 02:26:12 +02:00
Record continuation accounting baseline and retained live failures
This commit is contained in:
1 parent
379a70315e
commit
c0b5ee8b38
8 files changed
+14367
-3
No files matched your search
@@ -29,7 +29,7 @@ new paid productive cases use public questions and responses instead.
|
||||
| Repeated missing disposition | Existing repair episode unit/DB tests; complete sweep after exhaustion | Initial turn plus exactly two legacy repairs; visible board recovery, blocked task |
|
||||
| Comments/confident prose/tool calls without state change | Quiet/noisy exhausted repair and failure variants; native event replay | Quiet versus three distinct attributed misleading comments per run; no extra allowance |
|
||||
| Late approval/pause/spending/ownership gate | Second repair delayed, new gate, service restart, same debit; existing admission tests cover both runtimes | Approval created by first repair owns wait; acceptance causes one completion; Stop before second repair dispatch |
|
||||
| Restart while a repair is scheduled | Durable episode/dispatch tests | Controller restart before delayed repair; identical run IDs and episode counters |
|
||||
| Restart while a repair is scheduled | Actual delayed-retry promotion as well as the repair gate; retain episode and debit | Controller restart before delayed repair; identical run IDs and episode counters; fail with the persisted cancellation reason if promotion cancels it |
|
||||
| Duplicate/out-of-order handling | Concurrent retry/recovery tests and native consumed-repair replay with commentary/tool events | Exact run counts, document revisions and retained IDs; exhaustive races stay deterministic |
|
||||
| No reset after exhaustion; legitimate new request | Existing heartbeat exhausted failure versus actual new user request | No user message during exhaustion; a real approval resumes its owned path |
|
||||
| Missing or misleading evidence | Calibrated oracle rejects missing state, wrong documents, overwritten steps, fabricated progress, extra runs, lost receipts, decline, executed Stop | Every live result uses that oracle and existing screenshot/evidence/billing pipeline |
|
||||
@@ -64,3 +64,11 @@ spending gates and timing races use deterministic fixtures rather than artificia
|
||||
spending money or hoping for a real outage. The original native compatibility
|
||||
probes, blank-route reload finding and duplicate-response finding remain in the
|
||||
[retained backlog](2026-09-22-legacy-continuation-authority.md).
|
||||
|
||||
## Recorded baseline
|
||||
|
||||
The [measurement report](../../tests/lifecycle-baseline/CONTINUATION-ACCOUNTING-2026-09-22.md)
|
||||
retains the deterministic inventory and each paid campaign without regrading.
|
||||
The new suite exposes shared allowance accounting in both directions and a
|
||||
delayed-repair promotion mismatch. These are production follow-ups; the test
|
||||
slice leaves their intended-behavior assertions enabled and failing.
|
||||
@@ -123,8 +123,8 @@ Keep this section even if this slice incidentally resolves some assertions.
|
||||
| Commentary counts as progress | 1 | Retain for progress-evidence audit; no comment-count budget resets in this slice |
|
||||
| Wording selects liveness versus handoff path in heartbeat | 1 | Retain separately in results; must be covered while closing the current shared authority boundary |
|
||||
| Native autonomous continuation compatibility probes | 2 | Replace with reachable public-contract tests; `same_agent` injection is not a current model-facing defect |
|
||||
| Productive continuation versus repair/failure budgets | Coverage boundary | Separate audit after this slice; do not infer progress from text/event counts |
|
||||
| Blank task route after reload | New live observation | Investigate browser bootstrap/rendering; seen after successful persisted lifecycle in both runtimes; campaigns 35744074887 and 35747200170 |
|
||||
| Productive continuation versus repair/failure budgets | Three new findings | [Accounting baseline](2026-09-22-continuation-accounting-baseline.md) exposes delayed repair spending infrastructure allowance, infrastructure retries spending productive allowance, and delayed repair cancellation during promotion. Test-only slice; production fixes remain open |
|
||||
| Blank task route after reload | Repeated live observation | Investigate browser bootstrap/rendering; campaigns 35744074887 and 35747200170, plus accounting campaign 35813099816 after two correct legacy steps with a pending question. Empty screenshot retained; full quiet productive case remains unverified |
|
||||
| Duplicate legacy response | One later live observation | Campaign 35805477715 legacy work-mode neutral: provider issued two successful PATCH calls, second escaping an underscore. Retain as model behavior evidence; exact-once matcher rejected it |
|
||||
| Missing-comment policy ownership | Coverage boundary | Existing typed retry policy remains; review separately if consolidating all post-run compliance into one disposition contract |
|
||||
| Maintain CI and prepare App/Evals review | Pending | Promote fixed invariants into maintained cheap gates; paid suites remain explicit |
|
||||
|
||||
@@ -0,0 +1,138 @@
|
||||
# Continuation accounting baseline — September 22, 2026
|
||||
|
||||
This is a test-first slice. Production scheduling/authority code is unchanged.
|
||||
The [scenario matrix](../../doc/plans/2026-09-22-continuation-accounting-baseline.md)
|
||||
and executable inventory cover separate allowances, false progress, late gates,
|
||||
restart and duplicate handling. There are 67 additional deterministic tests
|
||||
and eight new explicit-only real-provider Product E2E cells.
|
||||
|
||||
## Deterministic measurement
|
||||
|
||||
Source: `379a70315e47de1ac2c8e4eb452eb3121401c5c7`, no tracked source changes at measurement.
|
||||
Raw evidence: `.lifecycle-baseline/2026-09-23T03-04-36.107Z/`.
|
||||
[Saved inventory and results](baselines/2026-09-22-continuation-accounting.json).
|
||||
|
||||
| Layer | Passed / total | Failed |
|
||||
|---|---:|---:|
|
||||
| Unit | 413 / 414 | 1 |
|
||||
| Runner | 184 / 184 | 0 |
|
||||
| Database/heartbeat integration | 371 / 376 | 5 |
|
||||
| Grading | 85 / 85 | 0 |
|
||||
| Total | 1,053 / 1,059 | 6 |
|
||||
|
||||
No skipped tests or missing inventory evidence. Four newly failing tests
|
||||
identify three production problems:
|
||||
|
||||
1. **Delayed repair spends infrastructure allowance.** Immediate first repair
|
||||
has no scheduled retry debit and passes. Delayed repair 2 has
|
||||
`scheduledRetryAttempt: 2`, `scheduledRetryReason: issue_disposition_repair`.
|
||||
`executionFailureRetryCount` interprets it as two prior failures, so a first
|
||||
safe infrastructure failure is already exhausted. One pure assertion and one
|
||||
actual scheduler assertion expose this.
|
||||
2. **Infrastructure retries spend productive allowance.** A run resumed after
|
||||
two infrastructure retries reaches its first typed max-turn continuation.
|
||||
The scheduler reads the same counter as two consumed productive attempts and
|
||||
returns exhausted. The test expects productive attempt 1.
|
||||
3. **The delayed second repair is cancelled during promotion.** The legacy
|
||||
repair dispatch gate accepts the persisted episode, but the shared dispatch
|
||||
path compares its fingerprint against a different disposition fingerprint.
|
||||
It cancels the unstarted repair with
|
||||
`issue_disposition_repair_superseded`. The new database test reproduces this
|
||||
through actual due-retry promotion after recreating the controller service.
|
||||
The first live exhaustion run also retained this exact cancellation, with
|
||||
only two provider executions and no final exhaustion checkpoint.
|
||||
|
||||
The other two failures are the previously retained native `same_agent`
|
||||
compatibility probes. They inject an unsupported current public continuation
|
||||
shape and do not establish a current model-facing defect. The new paid native
|
||||
productive tests use reachable question/response contracts.
|
||||
|
||||
The tests also prove that twenty comments or a large raw tool count do not
|
||||
replenish exhausted legacy repairs, including through the full recovery sweep
|
||||
while the issue is still active. A new approval, company budget hard-stop,
|
||||
agent pause or reassignment during the second repair delay blocks dispatch
|
||||
without another debit. Native replay with twenty commentary/tool events does
|
||||
not resubmit consumed disposition repair. Existing gates and duplicate/restart
|
||||
regressions remain linked in the inventory.
|
||||
|
||||
## Other verification
|
||||
|
||||
- Product E2E support: 474/474 passing, including 31 new accounting support tests.
|
||||
- Inventory/report support: 4/4 passing.
|
||||
- E2E and lifecycle baseline typechecks pass.
|
||||
- Server and runner package typechecks pass; server preparation also completes
|
||||
the runner build and generated-contract checks.
|
||||
- Full general repository tests, recursive typecheck and root build were not
|
||||
rerun: this slice changes only tests, harness and documentation. This is not
|
||||
a claim of repository-wide green or PR readiness. The new intended-behavior
|
||||
failures remain enabled and will fail their normal test lane until fixed.
|
||||
|
||||
The [initial deterministic measurement](baselines/2026-09-22-continuation-accounting-initial.json)
|
||||
at `ed53efb1e0e363b382b5101340259ad3cab34307` is retained separately: 1,051/1,056.
|
||||
The later total adds the promotion regression and two harness calibration tests.
|
||||
|
||||
## Live measurement
|
||||
|
||||
All cells use qualified Codex `gpt-5.6-sol`, real Chromium, and isolated local
|
||||
server/database fixtures in parallel GitHub Actions jobs. Original machine
|
||||
grades, source/definition fingerprints, cleanup and billing are retained for
|
||||
each campaign; unavailable cost is not zero spending.
|
||||
|
||||
First campaign: [35811412940](https://github.com/paperclipai/paperclip/actions/runs/35811412940),
|
||||
source `9cd7a585f5c69dc327b7d90534ad51be89ae5bb7`. All eight cells failed.
|
||||
This measurement is not regraded. Initial fixtures requested three identical
|
||||
comments, which the product deduplicates per run; rich-text serialization also
|
||||
escaped underscores in literal test tokens. Screenshot names did not meet the
|
||||
existing evidence allowlist, so the public report publisher rejected the bundle.
|
||||
Legacy quiet productive behavior passed every causal check before packaging
|
||||
failed. Approval and Stop lifecycle checks also passed while their comment
|
||||
perturbation checks failed. Both exhaustion cells failed to reach the expected
|
||||
exhaustion checkpoint; retained API state identifies the cancelled delayed repair.
|
||||
|
||||
Corrected fixtures use three distinct numbered comments, plain alphanumeric
|
||||
tokens and existing allowed screenshot names. A packaging test verifies those
|
||||
screenshots survive. Cancellation fails promptly with its typed reason and the
|
||||
last observed state, instead of waiting for the overall deadline. These are
|
||||
test-harness changes, not changes to production behavior.
|
||||
|
||||
Corrected campaign: [35813099816](https://github.com/paperclipai/paperclip/actions/runs/35813099816),
|
||||
source `379a70315e47de1ac2c8e4eb452eb3121401c5c7`:
|
||||
**5/8 passed, 3 failed**, no missing cells or retries, all eight evidence packages
|
||||
valid and cleanup successful. The corrected public report published successfully.
|
||||
|
||||
| Cell | Result | Observed behavior |
|
||||
|---|---|---|
|
||||
| Legacy productive, quiet | Fail | After two successful runs, two exact revision-one records, and a pending next question, a task-route reload renders a blank page. `task-chat-thread` never appears within 30 seconds. Remaining three steps are unverified |
|
||||
| Legacy productive, noisy | Pass | Five productive runs, five exact revision-one documents, four real answered questions, no repair/failure debits |
|
||||
| Native productive, quiet | Pass | Same five-step causal assertions |
|
||||
| Native productive, noisy | Pass | Same five-step causal assertions despite misleading attributed comments |
|
||||
| Legacy exhaustion, quiet | Fail | Second repair cancelled before dispatch with `issue_disposition_repair_superseded` |
|
||||
| Legacy exhaustion, noisy | Fail | Identical typed cancellation despite the additional comments |
|
||||
| Legacy repair then Stop | Pass | Delayed second repair cancelled by the operator before dispatch; remains stopped after controller restart and scheduled due time |
|
||||
| Legacy repair then approval | Pass | Pending approval owns the wait; real browser acceptance produces exactly one successful completion without repair debt |
|
||||
|
||||
The blank-page screenshot is empty. This matches the symptom of the
|
||||
retained task-route reload finding; it does not establish the underlying UI
|
||||
cause, and it is not counted as a completed productive case. The successful
|
||||
native noisy final screenshot shows Done and all five document cards at revision
|
||||
one. Both exhaustion checkpoints retain the scheduled and restarted run IDs;
|
||||
their last observations include the cancelled unstarted repair and a board-owned
|
||||
`active_run_watchdog` action instead of completed disposition-repair exhaustion.
|
||||
|
||||
- [Published corrected report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35813099816-1/index.html)
|
||||
- [Both immutable campaign measurements, fingerprints, grades and billing](baselines/2026-09-22-continuation-accounting-live.json)
|
||||
|
||||
Billing is partial in both campaigns. Provider-reported zero totals do **not**
|
||||
establish free execution: the first campaign records token usage for 29/32 run
|
||||
records and reported cost for 10; the corrected campaign records usage for 26/29
|
||||
and reported cost for 10. Each includes three cancelled, unstarted records.
|
||||
The saved billing fields preserve the original coverage and cost status.
|
||||
|
||||
## Next production slice
|
||||
|
||||
Fix allowance separation and the delayed-repair promotion contract against the
|
||||
enabled deterministic failures, then rerun the same eight-cell suite. Keep the
|
||||
blank-route finding, unsupported native compatibility probes and earlier backlog
|
||||
distinct. This baseline establishes assertions and observed behavior; it does
|
||||
not claim the product is already robust or that every supported provider is
|
||||
qualified.
|
||||
@@ -150,3 +150,5 @@ E2E scores, or claim full qualification from a partial selection.
|
||||
September 22 follow-up: [legacy continuation implementation and verification](LEGACY-CONTINUATION-2026-09-22.md), including preserved failed campaigns and the remaining backlog.
|
||||
|
||||
The [continuation accounting matrix](../../doc/plans/2026-09-22-continuation-accounting-baseline.md) adds ACCT-01 through ACCT-04 for separate allowances, false progress, late gates and restart/replay. Its real-provider companion is the explicit-only `continuation-accounting` Product E2E suite.
|
||||
The [September 22 measurement](CONTINUATION-ACCOUNTING-2026-09-22.md) records the
|
||||
enabled failures and preserves both initial and corrected live campaigns.
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -36,3 +36,7 @@ See the [scenario matrix](../../doc/plans/2026-09-22-continuation-accounting-bas
|
||||
for deterministic fault, replay, spending and ownership coverage. Infrastructure
|
||||
failures are not induced in paid cells; their cross-lane allowance semantics are
|
||||
covered by actual scheduler tests with controlled failures.
|
||||
|
||||
Results and follow-ups are retained in the [accounting measurement report](../lifecycle-baseline/CONTINUATION-ACCOUNTING-2026-09-22.md),
|
||||
including the original failed fixture campaign. A new measurement never replaces
|
||||
or regrades an earlier campaign.
|
||||
Reference in new issue
Block a user