mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
Record work-mode baseline and real-provider verification
This commit is contained in:
1 parent
2e9ee31937
commit
e58857534b
6 files changed
+5321
-3
No files matched your search
@@ -32,5 +32,13 @@ attempts and final results before declaring verification complete.
|
||||
|
||||
## Verification
|
||||
|
||||
In progress. The historical remaining-findings inventory is retained in
|
||||
Deterministic baseline: 990/992 pass, with only the two previously recorded native
|
||||
compatibility probes failing. All work-mode assertions pass; E2E support,
|
||||
typechecks and build pass. Both explicit planning controls passed in campaign
|
||||
35805477715; its four new wording cases exposed a Markdown/browser fixture
|
||||
mismatch, with one additional duplicate model response. Those failures remain
|
||||
recorded. After a fixture-only plain-text correction, both wording pairs passed
|
||||
in campaign 35806360797 (4/4). See the
|
||||
[measurement report](../../tests/lifecycle-baseline/EXPLICIT-WORK-MODE-2026-09-22.md).
|
||||
The historical remaining-findings inventory is retained in
|
||||
[the continuation plan](2026-09-22-legacy-continuation-authority.md).
|
||||
@@ -118,13 +118,14 @@ Keep this section even if this slice incidentally resolves some assertions.
|
||||
|
||||
| Finding | Recorded count | Disposition |
|
||||
|---|---:|---|
|
||||
| Narrative changes continuation | 36 | Current focus |
|
||||
| Title/description words select work mode | 4 | Retain for explicit work-mode follow-up; no title/body field in the new authority contract |
|
||||
| Narrative changes continuation | 36 | Fixed by the persisted-disposition authority slice; see verification above |
|
||||
| Title/description words select work mode | 4 | Heuristic removed in [explicit work-mode follow-up](2026-09-22-explicit-work-mode-authority.md). Clarification: this was a liveness diagnostic exemption, not a stored mode mutation. Deterministic checks and 4/4 corrected live wording cases pass; both explicit planning controls pass |
|
||||
| Commentary counts as progress | 1 | Retain for progress-evidence audit; no comment-count budget resets in this slice |
|
||||
| Wording selects liveness versus handoff path in heartbeat | 1 | Retain separately in results; must be covered while closing the current shared authority boundary |
|
||||
| Native autonomous continuation compatibility probes | 2 | Replace with reachable public-contract tests; `same_agent` injection is not a current model-facing defect |
|
||||
| Productive continuation versus repair/failure budgets | Coverage boundary | Separate audit after this slice; do not infer progress from text/event counts |
|
||||
| Blank task route after reload | New live observation | Investigate browser bootstrap/rendering; seen after successful persisted lifecycle in both runtimes; campaigns 35744074887 and 35747200170 |
|
||||
| Duplicate legacy response | One later live observation | Campaign 35805477715 legacy work-mode neutral: provider issued two successful PATCH calls, second escaping an underscore. Retain as model behavior evidence; exact-once matcher rejected it |
|
||||
| Missing-comment policy ownership | Coverage boundary | Existing typed retry policy remains; review separately if consolidating all post-run compliance into one disposition contract |
|
||||
| Maintain CI and prepare App/Evals review | Pending | Promote fixed invariants into maintained cheap gates; paid suites remain explicit |
|
||||
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
# Explicit work-mode authority — 2026-09-22
|
||||
|
||||
Product source: `d17c92eae10e949643a3b9d1be829061c30e77d6`.
|
||||
Fixture correction: `2e9ee319379ef263a511ef31717ee1185b53abdf`.
|
||||
Isolated worktree: `/private/tmp/paperclip-lifecycle-baseline-20260921`.
|
||||
|
||||
## Behavior
|
||||
|
||||
The task creation/update and native/legacy runner mode paths already use explicit
|
||||
fields. The remaining title/description regexes were a legacy liveness diagnostic
|
||||
exemption, not a write to `issues.work_mode`. They are now removed. Heartbeat and
|
||||
activity-ledger backfill pass the persisted work mode to the classifier.
|
||||
|
||||
A standard task can deliver a requested plan in its thread or canonical plan
|
||||
document and complete. It does not become a planning-mode task. Explicit planning
|
||||
mode and revision-bound approval transitions retain their existing contract.
|
||||
No schema migration or inferred rewrite of existing tasks is needed.
|
||||
|
||||
This is a bounded work-mode change. Other prose diagnostics, productive-progress
|
||||
accounting, the native compatibility probes and the browser reload finding remain
|
||||
tracked in [the backlog](../../doc/plans/2026-09-22-legacy-continuation-authority.md).
|
||||
|
||||
## Deterministic verification
|
||||
|
||||
The [retained baseline](baselines/2026-09-22-explicit-work-mode.json) records
|
||||
**990 passed / 992 total**: unit 393/393, runner 182/182, database integration
|
||||
361/363, grading 54/54. The same two unsupported native `same_agent` probes
|
||||
remain failed and visible; no new failure was accepted or skipped. The original
|
||||
baseline reports remain unchanged. Measurement finished September 22 in
|
||||
America/Los_Angeles (September 23 UTC).
|
||||
|
||||
Five direct comparison probes also reject the previous classifier: the four
|
||||
historical title/body triggers incorrectly received the planning exemption,
|
||||
while a neutral-title task explicitly in planning mode did not. The corrected
|
||||
classifier passes all five. These comparisons are supplemental to the counted
|
||||
Vitest baseline.
|
||||
|
||||
Coverage includes all former title/description trigger words, missing/default
|
||||
mode, explicit standard/planning/ask/skill-test modes, full/resumed prompt
|
||||
directives, native execution input, database create/edit/explicit-mode updates,
|
||||
canonical plan creation in standard mode, activity backfill, and real heartbeat
|
||||
repair pairs. The heartbeat comparison uses durable effects because successor
|
||||
completion can precede a predecessor's diagnostic projection.
|
||||
|
||||
All 443 Product E2E support tests and all four inventory/report checks pass.
|
||||
Repository recursive typecheck, Product E2E typecheck, lifecycle-baseline typecheck,
|
||||
and repository build pass. The full general repository test command was not
|
||||
rerun for this small classifier change; the relevant suites are included above.
|
||||
|
||||
## Real-provider verification
|
||||
|
||||
Campaign [35805477715](https://github.com/paperclipai/paperclip/actions/runs/35805477715)
|
||||
finished **2/6 passed** on the product source SHA above. It selected six local Codex cells, using
|
||||
the existing `gpt-5.6-sol` profiles:
|
||||
|
||||
- `lifecycle-work-mode-neutral` and `lifecycle-work-mode-challenge` on both
|
||||
legacy and native: one provider turn per cell, a requested two-step plan,
|
||||
exact visible output, standard mode before/after execution, Done, no extra
|
||||
recovery or pending interaction.
|
||||
- `core-compatibility` / `plan-revise-accept` on both runtimes: explicit planning
|
||||
mode, initial and revised plan, exact revision-bound approval, then completion.
|
||||
|
||||
Both explicit planning controls passed. All four wording cases preserved
|
||||
standard mode and completed, but failed the browser's literal full-text match:
|
||||
Markdown renders ordered-list numbers separately from the DOM text. The retained
|
||||
legacy challenge screenshot shows the correct plan and Done. Three cases passed
|
||||
all persisted-state/output matchers. Legacy-neutral additionally posted the
|
||||
response twice: the provider log records two successful PATCH calls, the second
|
||||
escaping an underscore. That is a model behavior failure, not a duplicate server
|
||||
projection. The original machine grades remain unchanged.
|
||||
|
||||
The fixture correction uses an equivalent plain-text two-step plan so the raw
|
||||
exact-output and browser exact-text contracts agree. All 443 support tests and
|
||||
the E2E typecheck pass again. Campaign
|
||||
[35806360797](https://github.com/paperclipai/paperclip/actions/runs/35806360797)
|
||||
is a fresh four-cell measurement of both wording pairs on that corrected source;
|
||||
**all four live cells passed**. Each runtime delivered the exact two-step plan
|
||||
once, preserved standard mode, completed in one run and left no extra work or
|
||||
pending interaction. All evidence validated and cleanup passed, with no retries
|
||||
or incomplete cells. The two explicit planning/revision/acceptance controls
|
||||
passed in the first campaign on identical product code. The corrected campaign
|
||||
does not regrade the first campaign or establish a statistical reliability rate.
|
||||
|
||||
These are real browser/server/database/provider journeys. The full 46-cell
|
||||
lifecycle catalog is not selected by this follow-up. Both campaigns and their
|
||||
catalog hashes, source revisions, cleanup and billing coverage are retained in
|
||||
the [live inventory](baselines/2026-09-22-explicit-work-mode-live.json).
|
||||
Missing cost evidence must not be interpreted as free.
|
||||
|
||||
Published reports: [original six-cell attempt](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35805477715-1/index.html),
|
||||
[corrected four-cell campaign](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35806360797-1/index.html).
|
||||
@@ -10,6 +10,8 @@ failures, representative observations and unmeasured coverage. The separate
|
||||
[live Actions baseline](LIVE-BASELINE-2026-09-21.md) records actual LLM/browser runs,
|
||||
assertion corrections and retained behavior failures. The [fix report](LIVE-FIXES-2026-09-21.md)
|
||||
records the later cancellation fix, corrected fixtures and fresh campaigns.
|
||||
The [explicit work-mode report](EXPLICIT-WORK-MODE-2026-09-22.md) records the
|
||||
title/description heuristic removal and its layered verification.
|
||||
|
||||
## Run and inspect
|
||||
|
||||
|
||||
@@ -0,0 +1,286 @@
|
||||
{
|
||||
"schema": "paperclip.explicit-work-mode-verification/v1",
|
||||
"measurementDateLocal": "2026-09-22",
|
||||
"timezone": "America/Los_Angeles",
|
||||
"campaigns": [
|
||||
{
|
||||
"source": {
|
||||
"sha": "d17c92eae10e949643a3b9d1be829061c30e77d6",
|
||||
"ref": "refs/heads/codex/lifecycle-behavior-baseline-20260921",
|
||||
"workflowRunUrl": "https://github.com/paperclipai/paperclip/actions/runs/35805477715",
|
||||
"eventName": "workflow_dispatch"
|
||||
},
|
||||
"campaignId": "gha-35805477715-1",
|
||||
"artifactSha256": "921c157c55769bd3febe3a29d1aaa64c6c8c2c5e7b349c4e865964425c2b0c2f",
|
||||
"totals": {
|
||||
"selected": 6,
|
||||
"executed": 6,
|
||||
"passed": 2,
|
||||
"failed": 4,
|
||||
"incomplete": 0,
|
||||
"retries": 0,
|
||||
"cleanupPassed": true
|
||||
},
|
||||
"billing": {
|
||||
"testCount": 6,
|
||||
"agentRunDurationMs": 119026,
|
||||
"leaseDurationMs": 0,
|
||||
"llm": {
|
||||
"runCount": 10,
|
||||
"runsWithTokenUsage": 10,
|
||||
"runsWithReportedCost": 5,
|
||||
"inputTokens": 641012,
|
||||
"outputTokens": 7375,
|
||||
"cachedInputTokens": 410079,
|
||||
"totalTokens": 1058466,
|
||||
"reportedCostUsd": 0,
|
||||
"costStatus": "partial"
|
||||
},
|
||||
"reportedLlmCostUsd": 0,
|
||||
"estimatedRuntimeCostUsd": 0,
|
||||
"observedAndEstimatedCostUsd": 0,
|
||||
"testsWithCompleteBilling": 3
|
||||
},
|
||||
"definitionHashes": [
|
||||
"b38be17e21fd6b4f81ae4a1ffc0488559940c65dec016ffbcf41107195775249",
|
||||
"f82ddc73601921ce113ace60db788053b1c93ed11cc5add75fa237ea4c3e0393"
|
||||
],
|
||||
"cells": [
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-work-mode-neutral",
|
||||
"caseId": "lifecycle-work-mode-neutral",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "failed",
|
||||
"failureClass": "candidate_failure",
|
||||
"attempt": 1,
|
||||
"durationMs": 62305,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed",
|
||||
"runCount": 1,
|
||||
"failedMatchers": [
|
||||
{
|
||||
"matcher": {
|
||||
"kind": "message_exact",
|
||||
"expected": "ROLLOUT_7ddf65563aed-1\n1. Inspect the exporter.\n2. Implement and verify the change."
|
||||
},
|
||||
"passed": false,
|
||||
"detail": "expected {\"kind\":\"message_exact\",\"expected\":\"ROLLOUT_7ddf65563aed-1\\n1. Inspect the exporter.\\n2. Implement and verify the change.\"}; observed \"ROLLOUT_7ddf65563aed-1\\n1. Inspect the exporter.\\n2. Implement and verify the change.\\nROLLOUT_7ddf65563aed-1\\n1. Inspect the exporter.\\n2. Implement and verify the change.\""
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"kind": "message_occurrences",
|
||||
"expected": "ROLLOUT_7ddf65563aed-1\n1. Inspect the exporter.\n2. Implement and verify the change.",
|
||||
"count": 1
|
||||
},
|
||||
"passed": false,
|
||||
"detail": "expected {\"kind\":\"message_occurrences\",\"expected\":\"ROLLOUT_7ddf65563aed-1\\n1. Inspect the exporter.\\n2. Implement and verify the change.\",\"count\":1}; observed 2"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-work-mode-challenge",
|
||||
"caseId": "lifecycle-work-mode-challenge",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "failed",
|
||||
"failureClass": "candidate_failure",
|
||||
"attempt": 1,
|
||||
"durationMs": 56471,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed",
|
||||
"runCount": 1,
|
||||
"failedMatchers": []
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.runner-codex.local.lifecycle-work-mode-neutral",
|
||||
"caseId": "lifecycle-work-mode-neutral",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "failed",
|
||||
"failureClass": "candidate_failure",
|
||||
"attempt": 1,
|
||||
"durationMs": 57976,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed",
|
||||
"runCount": 1,
|
||||
"failedMatchers": []
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.runner-codex.local.lifecycle-work-mode-challenge",
|
||||
"caseId": "lifecycle-work-mode-challenge",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "failed",
|
||||
"failureClass": "candidate_failure",
|
||||
"attempt": 1,
|
||||
"durationMs": 51867,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed",
|
||||
"runCount": 1,
|
||||
"failedMatchers": []
|
||||
},
|
||||
{
|
||||
"executionId": "core-compatibility.legacy-codex.local.plan-revise-accept",
|
||||
"caseId": "plan-revise-accept",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 62650,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed",
|
||||
"runCount": 3,
|
||||
"failedMatchers": []
|
||||
},
|
||||
{
|
||||
"executionId": "core-compatibility.runner-codex.local.plan-revise-accept",
|
||||
"caseId": "plan-revise-accept",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 60071,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed",
|
||||
"runCount": 3,
|
||||
"failedMatchers": []
|
||||
}
|
||||
],
|
||||
"interpretation": "Both planning controls passed. Four new wording cases failed literal browser matching of Markdown lists. Legacy-neutral also issued two successful comment PATCH calls; model behavior evidence is retained. No regrading."
|
||||
},
|
||||
{
|
||||
"source": {
|
||||
"sha": "2e9ee319379ef263a511ef31717ee1185b53abdf",
|
||||
"ref": "refs/heads/codex/lifecycle-behavior-baseline-20260921",
|
||||
"workflowRunUrl": "https://github.com/paperclipai/paperclip/actions/runs/35806360797",
|
||||
"eventName": "workflow_dispatch"
|
||||
},
|
||||
"campaignId": "gha-35806360797-1",
|
||||
"artifactSha256": "6927541ae792c0f4e7013672de4e56135eeada52d5976b78bae467ba35b61647",
|
||||
"totals": {
|
||||
"selected": 4,
|
||||
"executed": 4,
|
||||
"passed": 4,
|
||||
"failed": 0,
|
||||
"incomplete": 0,
|
||||
"retries": 0,
|
||||
"cleanupPassed": true
|
||||
},
|
||||
"billing": {
|
||||
"testCount": 4,
|
||||
"agentRunDurationMs": 53034,
|
||||
"leaseDurationMs": 0,
|
||||
"llm": {
|
||||
"runCount": 4,
|
||||
"runsWithTokenUsage": 4,
|
||||
"runsWithReportedCost": 2,
|
||||
"inputTokens": 202266,
|
||||
"outputTokens": 1767,
|
||||
"cachedInputTokens": 124958,
|
||||
"totalTokens": 328991,
|
||||
"reportedCostUsd": 0,
|
||||
"costStatus": "partial"
|
||||
},
|
||||
"reportedLlmCostUsd": 0,
|
||||
"estimatedRuntimeCostUsd": 0,
|
||||
"observedAndEstimatedCostUsd": 0,
|
||||
"testsWithCompleteBilling": 2
|
||||
},
|
||||
"definitionHashes": [
|
||||
"6c1a13b14f9f15e6eac6e557f39337ce4396b40a732f37a6fa1096b6b5ab2868"
|
||||
],
|
||||
"cells": [
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-work-mode-neutral",
|
||||
"caseId": "lifecycle-work-mode-neutral",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 37511,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed",
|
||||
"runCount": 1,
|
||||
"failedMatchers": []
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.legacy-codex.local.lifecycle-work-mode-challenge",
|
||||
"caseId": "lifecycle-work-mode-challenge",
|
||||
"profileId": "legacy-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "legacy",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 34480,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed",
|
||||
"runCount": 1,
|
||||
"failedMatchers": []
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.runner-codex.local.lifecycle-work-mode-neutral",
|
||||
"caseId": "lifecycle-work-mode-neutral",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 39179,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed",
|
||||
"runCount": 1,
|
||||
"failedMatchers": []
|
||||
},
|
||||
{
|
||||
"executionId": "lifecycle-baseline.runner-codex.local.lifecycle-work-mode-challenge",
|
||||
"caseId": "lifecycle-work-mode-challenge",
|
||||
"profileId": "runner-codex",
|
||||
"environmentId": "local",
|
||||
"runtimeMode": "native",
|
||||
"model": "gpt-5.6-sol",
|
||||
"status": "passed",
|
||||
"failureClass": null,
|
||||
"attempt": 1,
|
||||
"durationMs": 33067,
|
||||
"evidenceValid": true,
|
||||
"evidenceErrors": [],
|
||||
"cleanup": "passed",
|
||||
"runCount": 1,
|
||||
"failedMatchers": []
|
||||
}
|
||||
],
|
||||
"interpretation": "All four corrected standard-mode wording cells passed exact output, visible browser state, persisted mode and completion. One provider run per cell, no retries, valid evidence and successful cleanup."
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
Reference in new issue
Block a user