Record work-mode baseline and real-provider verification

This commit is contained in:
Dotta committed 2026-09-22 18:38:40 -07:00
1 parent 2e9ee31937
commit e58857534b
6 files changed
+5321 -3

No files matched your search

@@ -32,5 +32,13 @@ attempts and final results before declaring verification complete.
## Verification
In progress. The historical remaining-findings inventory is retained in
Deterministic baseline: 990/992 pass, with only the two previously recorded native
compatibility probes failing. All work-mode assertions pass; E2E support,
typechecks and build pass. Both explicit planning controls passed in campaign
35805477715; its four new wording cases exposed a Markdown/browser fixture
mismatch, with one additional duplicate model response. Those failures remain
recorded. After a fixture-only plain-text correction, both wording pairs passed
in campaign 35806360797 (4/4). See the
[measurement report](../../tests/lifecycle-baseline/EXPLICIT-WORK-MODE-2026-09-22.md).
The historical remaining-findings inventory is retained in
[the continuation plan](2026-09-22-legacy-continuation-authority.md).
@@ -118,13 +118,14 @@ Keep this section even if this slice incidentally resolves some assertions.
| Finding | Recorded count | Disposition |
|---|---:|---|
| Narrative changes continuation | 36 | Current focus |
| Title/description words select work mode | 4 | Retain for explicit work-mode follow-up; no title/body field in the new authority contract |
| Narrative changes continuation | 36 | Fixed by the persisted-disposition authority slice; see verification above |
| Title/description words select work mode | 4 | Heuristic removed in [explicit work-mode follow-up](2026-09-22-explicit-work-mode-authority.md). Clarification: this was a liveness diagnostic exemption, not a stored mode mutation. Deterministic checks and 4/4 corrected live wording cases pass; both explicit planning controls pass |
| Commentary counts as progress | 1 | Retain for progress-evidence audit; no comment-count budget resets in this slice |
| Wording selects liveness versus handoff path in heartbeat | 1 | Retain separately in results; must be covered while closing the current shared authority boundary |
| Native autonomous continuation compatibility probes | 2 | Replace with reachable public-contract tests; `same_agent` injection is not a current model-facing defect |
| Productive continuation versus repair/failure budgets | Coverage boundary | Separate audit after this slice; do not infer progress from text/event counts |
| Blank task route after reload | New live observation | Investigate browser bootstrap/rendering; seen after successful persisted lifecycle in both runtimes; campaigns 35744074887 and 35747200170 |
| Duplicate legacy response | One later live observation | Campaign 35805477715 legacy work-mode neutral: provider issued two successful PATCH calls, second escaping an underscore. Retain as model behavior evidence; exact-once matcher rejected it |
| Missing-comment policy ownership | Coverage boundary | Existing typed retry policy remains; review separately if consolidating all post-run compliance into one disposition contract |
| Maintain CI and prepare App/Evals review | Pending | Promote fixed invariants into maintained cheap gates; paid suites remain explicit |
@@ -0,0 +1,91 @@
# Explicit work-mode authority — 2026-09-22
Product source: `d17c92eae10e949643a3b9d1be829061c30e77d6`.
Fixture correction: `2e9ee319379ef263a511ef31717ee1185b53abdf`.
Isolated worktree: `/private/tmp/paperclip-lifecycle-baseline-20260921`.
## Behavior
The task creation/update and native/legacy runner mode paths already use explicit
fields. The remaining title/description regexes were a legacy liveness diagnostic
exemption, not a write to `issues.work_mode`. They are now removed. Heartbeat and
activity-ledger backfill pass the persisted work mode to the classifier.
A standard task can deliver a requested plan in its thread or canonical plan
document and complete. It does not become a planning-mode task. Explicit planning
mode and revision-bound approval transitions retain their existing contract.
No schema migration or inferred rewrite of existing tasks is needed.
This is a bounded work-mode change. Other prose diagnostics, productive-progress
accounting, the native compatibility probes and the browser reload finding remain
tracked in [the backlog](../../doc/plans/2026-09-22-legacy-continuation-authority.md).
## Deterministic verification
The [retained baseline](baselines/2026-09-22-explicit-work-mode.json) records
**990 passed / 992 total**: unit 393/393, runner 182/182, database integration
361/363, grading 54/54. The same two unsupported native `same_agent` probes
remain failed and visible; no new failure was accepted or skipped. The original
baseline reports remain unchanged. Measurement finished September 22 in
America/Los_Angeles (September 23 UTC).
Five direct comparison probes also reject the previous classifier: the four
historical title/body triggers incorrectly received the planning exemption,
while a neutral-title task explicitly in planning mode did not. The corrected
classifier passes all five. These comparisons are supplemental to the counted
Vitest baseline.
Coverage includes all former title/description trigger words, missing/default
mode, explicit standard/planning/ask/skill-test modes, full/resumed prompt
directives, native execution input, database create/edit/explicit-mode updates,
canonical plan creation in standard mode, activity backfill, and real heartbeat
repair pairs. The heartbeat comparison uses durable effects because successor
completion can precede a predecessor's diagnostic projection.
All 443 Product E2E support tests and all four inventory/report checks pass.
Repository recursive typecheck, Product E2E typecheck, lifecycle-baseline typecheck,
and repository build pass. The full general repository test command was not
rerun for this small classifier change; the relevant suites are included above.
## Real-provider verification
Campaign [35805477715](https://github.com/paperclipai/paperclip/actions/runs/35805477715)
finished **2/6 passed** on the product source SHA above. It selected six local Codex cells, using
the existing `gpt-5.6-sol` profiles:
- `lifecycle-work-mode-neutral` and `lifecycle-work-mode-challenge` on both
legacy and native: one provider turn per cell, a requested two-step plan,
exact visible output, standard mode before/after execution, Done, no extra
recovery or pending interaction.
- `core-compatibility` / `plan-revise-accept` on both runtimes: explicit planning
mode, initial and revised plan, exact revision-bound approval, then completion.
Both explicit planning controls passed. All four wording cases preserved
standard mode and completed, but failed the browser's literal full-text match:
Markdown renders ordered-list numbers separately from the DOM text. The retained
legacy challenge screenshot shows the correct plan and Done. Three cases passed
all persisted-state/output matchers. Legacy-neutral additionally posted the
response twice: the provider log records two successful PATCH calls, the second
escaping an underscore. That is a model behavior failure, not a duplicate server
projection. The original machine grades remain unchanged.
The fixture correction uses an equivalent plain-text two-step plan so the raw
exact-output and browser exact-text contracts agree. All 443 support tests and
the E2E typecheck pass again. Campaign
[35806360797](https://github.com/paperclipai/paperclip/actions/runs/35806360797)
is a fresh four-cell measurement of both wording pairs on that corrected source;
**all four live cells passed**. Each runtime delivered the exact two-step plan
once, preserved standard mode, completed in one run and left no extra work or
pending interaction. All evidence validated and cleanup passed, with no retries
or incomplete cells. The two explicit planning/revision/acceptance controls
passed in the first campaign on identical product code. The corrected campaign
does not regrade the first campaign or establish a statistical reliability rate.
These are real browser/server/database/provider journeys. The full 46-cell
lifecycle catalog is not selected by this follow-up. Both campaigns and their
catalog hashes, source revisions, cleanup and billing coverage are retained in
the [live inventory](baselines/2026-09-22-explicit-work-mode-live.json).
Missing cost evidence must not be interpreted as free.
Published reports: [original six-cell attempt](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35805477715-1/index.html),
[corrected four-cell campaign](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35806360797-1/index.html).
+2
View File
@@ -10,6 +10,8 @@ failures, representative observations and unmeasured coverage. The separate
[live Actions baseline](LIVE-BASELINE-2026-09-21.md) records actual LLM/browser runs,
assertion corrections and retained behavior failures. The [fix report](LIVE-FIXES-2026-09-21.md)
records the later cancellation fix, corrected fixtures and fresh campaigns.
The [explicit work-mode report](EXPLICIT-WORK-MODE-2026-09-22.md) records the
title/description heuristic removal and its layered verification.
## Run and inspect
@@ -0,0 +1,286 @@
{
"schema": "paperclip.explicit-work-mode-verification/v1",
"measurementDateLocal": "2026-09-22",
"timezone": "America/Los_Angeles",
"campaigns": [
{
"source": {
"sha": "d17c92eae10e949643a3b9d1be829061c30e77d6",
"ref": "refs/heads/codex/lifecycle-behavior-baseline-20260921",
"workflowRunUrl": "https://github.com/paperclipai/paperclip/actions/runs/35805477715",
"eventName": "workflow_dispatch"
},
"campaignId": "gha-35805477715-1",
"artifactSha256": "921c157c55769bd3febe3a29d1aaa64c6c8c2c5e7b349c4e865964425c2b0c2f",
"totals": {
"selected": 6,
"executed": 6,
"passed": 2,
"failed": 4,
"incomplete": 0,
"retries": 0,
"cleanupPassed": true
},
"billing": {
"testCount": 6,
"agentRunDurationMs": 119026,
"leaseDurationMs": 0,
"llm": {
"runCount": 10,
"runsWithTokenUsage": 10,
"runsWithReportedCost": 5,
"inputTokens": 641012,
"outputTokens": 7375,
"cachedInputTokens": 410079,
"totalTokens": 1058466,
"reportedCostUsd": 0,
"costStatus": "partial"
},
"reportedLlmCostUsd": 0,
"estimatedRuntimeCostUsd": 0,
"observedAndEstimatedCostUsd": 0,
"testsWithCompleteBilling": 3
},
"definitionHashes": [
"b38be17e21fd6b4f81ae4a1ffc0488559940c65dec016ffbcf41107195775249",
"f82ddc73601921ce113ace60db788053b1c93ed11cc5add75fa237ea4c3e0393"
],
"cells": [
{
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-work-mode-neutral",
"caseId": "lifecycle-work-mode-neutral",
"profileId": "legacy-codex",
"environmentId": "local",
"runtimeMode": "legacy",
"model": "gpt-5.6-sol",
"status": "failed",
"failureClass": "candidate_failure",
"attempt": 1,
"durationMs": 62305,
"evidenceValid": true,
"evidenceErrors": [],
"cleanup": "passed",
"runCount": 1,
"failedMatchers": [
{
"matcher": {
"kind": "message_exact",
"expected": "ROLLOUT_7ddf65563aed-1\n1. Inspect the exporter.\n2. Implement and verify the change."
},
"passed": false,
"detail": "expected {\"kind\":\"message_exact\",\"expected\":\"ROLLOUT_7ddf65563aed-1\\n1. Inspect the exporter.\\n2. Implement and verify the change.\"}; observed \"ROLLOUT_7ddf65563aed-1\\n1. Inspect the exporter.\\n2. Implement and verify the change.\\nROLLOUT_7ddf65563aed-1\\n1. Inspect the exporter.\\n2. Implement and verify the change.\""
},
{
"matcher": {
"kind": "message_occurrences",
"expected": "ROLLOUT_7ddf65563aed-1\n1. Inspect the exporter.\n2. Implement and verify the change.",
"count": 1
},
"passed": false,
"detail": "expected {\"kind\":\"message_occurrences\",\"expected\":\"ROLLOUT_7ddf65563aed-1\\n1. Inspect the exporter.\\n2. Implement and verify the change.\",\"count\":1}; observed 2"
}
]
},
{
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-work-mode-challenge",
"caseId": "lifecycle-work-mode-challenge",
"profileId": "legacy-codex",
"environmentId": "local",
"runtimeMode": "legacy",
"model": "gpt-5.6-sol",
"status": "failed",
"failureClass": "candidate_failure",
"attempt": 1,
"durationMs": 56471,
"evidenceValid": true,
"evidenceErrors": [],
"cleanup": "passed",
"runCount": 1,
"failedMatchers": []
},
{
"executionId": "lifecycle-baseline.runner-codex.local.lifecycle-work-mode-neutral",
"caseId": "lifecycle-work-mode-neutral",
"profileId": "runner-codex",
"environmentId": "local",
"runtimeMode": "native",
"model": "gpt-5.6-sol",
"status": "failed",
"failureClass": "candidate_failure",
"attempt": 1,
"durationMs": 57976,
"evidenceValid": true,
"evidenceErrors": [],
"cleanup": "passed",
"runCount": 1,
"failedMatchers": []
},
{
"executionId": "lifecycle-baseline.runner-codex.local.lifecycle-work-mode-challenge",
"caseId": "lifecycle-work-mode-challenge",
"profileId": "runner-codex",
"environmentId": "local",
"runtimeMode": "native",
"model": "gpt-5.6-sol",
"status": "failed",
"failureClass": "candidate_failure",
"attempt": 1,
"durationMs": 51867,
"evidenceValid": true,
"evidenceErrors": [],
"cleanup": "passed",
"runCount": 1,
"failedMatchers": []
},
{
"executionId": "core-compatibility.legacy-codex.local.plan-revise-accept",
"caseId": "plan-revise-accept",
"profileId": "legacy-codex",
"environmentId": "local",
"runtimeMode": "legacy",
"model": "gpt-5.6-sol",
"status": "passed",
"failureClass": null,
"attempt": 1,
"durationMs": 62650,
"evidenceValid": true,
"evidenceErrors": [],
"cleanup": "passed",
"runCount": 3,
"failedMatchers": []
},
{
"executionId": "core-compatibility.runner-codex.local.plan-revise-accept",
"caseId": "plan-revise-accept",
"profileId": "runner-codex",
"environmentId": "local",
"runtimeMode": "native",
"model": "gpt-5.6-sol",
"status": "passed",
"failureClass": null,
"attempt": 1,
"durationMs": 60071,
"evidenceValid": true,
"evidenceErrors": [],
"cleanup": "passed",
"runCount": 3,
"failedMatchers": []
}
],
"interpretation": "Both planning controls passed. Four new wording cases failed literal browser matching of Markdown lists. Legacy-neutral also issued two successful comment PATCH calls; model behavior evidence is retained. No regrading."
},
{
"source": {
"sha": "2e9ee319379ef263a511ef31717ee1185b53abdf",
"ref": "refs/heads/codex/lifecycle-behavior-baseline-20260921",
"workflowRunUrl": "https://github.com/paperclipai/paperclip/actions/runs/35806360797",
"eventName": "workflow_dispatch"
},
"campaignId": "gha-35806360797-1",
"artifactSha256": "6927541ae792c0f4e7013672de4e56135eeada52d5976b78bae467ba35b61647",
"totals": {
"selected": 4,
"executed": 4,
"passed": 4,
"failed": 0,
"incomplete": 0,
"retries": 0,
"cleanupPassed": true
},
"billing": {
"testCount": 4,
"agentRunDurationMs": 53034,
"leaseDurationMs": 0,
"llm": {
"runCount": 4,
"runsWithTokenUsage": 4,
"runsWithReportedCost": 2,
"inputTokens": 202266,
"outputTokens": 1767,
"cachedInputTokens": 124958,
"totalTokens": 328991,
"reportedCostUsd": 0,
"costStatus": "partial"
},
"reportedLlmCostUsd": 0,
"estimatedRuntimeCostUsd": 0,
"observedAndEstimatedCostUsd": 0,
"testsWithCompleteBilling": 2
},
"definitionHashes": [
"6c1a13b14f9f15e6eac6e557f39337ce4396b40a732f37a6fa1096b6b5ab2868"
],
"cells": [
{
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-work-mode-neutral",
"caseId": "lifecycle-work-mode-neutral",
"profileId": "legacy-codex",
"environmentId": "local",
"runtimeMode": "legacy",
"model": "gpt-5.6-sol",
"status": "passed",
"failureClass": null,
"attempt": 1,
"durationMs": 37511,
"evidenceValid": true,
"evidenceErrors": [],
"cleanup": "passed",
"runCount": 1,
"failedMatchers": []
},
{
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-work-mode-challenge",
"caseId": "lifecycle-work-mode-challenge",
"profileId": "legacy-codex",
"environmentId": "local",
"runtimeMode": "legacy",
"model": "gpt-5.6-sol",
"status": "passed",
"failureClass": null,
"attempt": 1,
"durationMs": 34480,
"evidenceValid": true,
"evidenceErrors": [],
"cleanup": "passed",
"runCount": 1,
"failedMatchers": []
},
{
"executionId": "lifecycle-baseline.runner-codex.local.lifecycle-work-mode-neutral",
"caseId": "lifecycle-work-mode-neutral",
"profileId": "runner-codex",
"environmentId": "local",
"runtimeMode": "native",
"model": "gpt-5.6-sol",
"status": "passed",
"failureClass": null,
"attempt": 1,
"durationMs": 39179,
"evidenceValid": true,
"evidenceErrors": [],
"cleanup": "passed",
"runCount": 1,
"failedMatchers": []
},
{
"executionId": "lifecycle-baseline.runner-codex.local.lifecycle-work-mode-challenge",
"caseId": "lifecycle-work-mode-challenge",
"profileId": "runner-codex",
"environmentId": "local",
"runtimeMode": "native",
"model": "gpt-5.6-sol",
"status": "passed",
"failureClass": null,
"attempt": 1,
"durationMs": 33067,
"evidenceValid": true,
"evidenceErrors": [],
"cleanup": "passed",
"runCount": 1,
"failedMatchers": []
}
],
"interpretation": "All four corrected standard-mode wording cells passed exact output, visible browser state, persisted mode and completion. One provider run per cell, no retries, valid evidence and successful cleanup."
}
]
}
File diff suppressed because it is too large. Load diff