diff --git a/doc/plans/2026-10-02-stock-harness-live-comparison.md b/doc/plans/2026-10-02-stock-harness-live-comparison.md index 42c7ae3fa33..4a4918e139c 100644 --- a/doc/plans/2026-10-02-stock-harness-live-comparison.md +++ b/doc/plans/2026-10-02-stock-harness-live-comparison.md @@ -1,11 +1,17 @@ # Stock-harness live comparison — 2026-10-02 -**TL;DR:** Two newly failing paired cases: classic Claude/OpenCode document delivery. Two newly passing cases: classic and native OpenCode ordered continuation. Seven unchanged failures and 13 unchanged passes. The extra ACP Claude storage symptom occurs beneath an existing credential-guard failure. All 48 results are available; no general performance equivalence is established. +**TL;DR:** Two newly failing paired cases: classic Claude/OpenCode document delivery. Two newly passing cases: classic and native OpenCode ordered continuation. Seven unchanged failures and 13 unchanged passes. Legacy ACP Claude additionally loses its document in the reduced variant beneath an existing credential-guard failure; no later matched run closes that behavioral defect. All 48 results are available; no general performance equivalence is established. The reduced instructions are **not yet qualified for merge**. Classic Claude and classic OpenCode pass the historical skill-output case but fail with the reduced instructions: they finish without saving a Paperclip task document. Native Codex and native Claude pass all three journeys in both variants. Single trials and ambiguous fixture storage wording limit causal attribution; the failed oracle remains unchanged. The reductions and evaluation setup are in draft [PR #14948](https://github.com/paperclipai/paperclip/pull/14948). This report and its [safe evidence projection](2026-10-02-stock-harness-live-comparison.json) record the measured revisions rather than claiming the final documentation head was run through the full matrix. +## Current merge assessment + +PR #14948 is CI/review-ready but **does not meet the no-extra-failing-behavior merge criterion**. The legacy ACP Claude skill cell saves a Paperclip document in the historical cohort and none in the reduced cohort, while both already fail the credential guard. That additional delivery defect must not disappear inside the unchanged overall Fail → Fail. Later repair campaigns selected classic Claude/OpenCode, not legacy ACP Claude; its behavioral difference remains unclosed. + +The latest [OpenCode comparison](2026-10-02-opencode-skill-routing-link-qualification.md) is a **skill-only** experiment: frozen `fe9dc1e3c518825242ed889ab9c8352986f8c2ed` and `0d7ecfa96d72fba79b7f0a25052b42c0686c0488` both use reduced manuals/shared prompts and vary only two operational skill sources plus the baseline provenance receipt. Its two Pass → Pass cases are not a repeat of historical-manual versus reduced-manual qualification and do not close the ACP Claude defect. Original-case handoff imperfections and the later provider-free-only link correction remain separately recorded. No result or oracle is changed by this assessment, and no new provider runs are authorized by it. + ## Revisions and matched inputs - Candidate: `f02d8d0df327abb43b20c7e7beb86798239abbf5`, eight-word hire manual plus reduced shared startup/resume instructions. diff --git a/doc/plans/2026-10-02-stock-harness-paperclip-checklist.md b/doc/plans/2026-10-02-stock-harness-paperclip-checklist.md index 7737e53c9e9..b709c03f289 100644 --- a/doc/plans/2026-10-02-stock-harness-paperclip-checklist.md +++ b/doc/plans/2026-10-02-stock-harness-paperclip-checklist.md @@ -6,7 +6,7 @@ hire manual is reduced to identity only, and common legacy startup/resume instructions are in [PR #14948](https://github.com/paperclipai/paperclip/pull/14948). GitHub live qualification retains earlier document-delivery failures; the latest focused OpenCode comparison passes both cases in both variants but still exposes -deficient original-case handoff links. PR #14948 remains draft. The original +deficient original-case handoff links. That later trial varies only skill sources while holding reduced manuals/shared prompts fixed. Legacy ACP Claude still loses the document only in the original reduced cohort beneath its unchanged credential-guard failure, and no later matched run closes that extra behavioral defect. PR #14948 is CI/review-ready but fails the no-extra-failing-behavior merge criterion and remains draft. The original [live report](2026-10-02-stock-harness-live-comparison.md), completed [skill repair](2026-10-02-legacy-document-skill-repair.md) and latest [stock-selection/link comparison](2026-10-02-opencode-skill-routing-link-qualification.md) @@ -444,7 +444,7 @@ Confirmed mechanics below do not by themselves establish an effect on task quali | F11 | Task Markdown and the wake renderer prescribe different accepted-plan behavior for planning-mode accepted-confirmation payloads; tests currently expect both. | 2; unify the directive owner; production reachability still to trace | | F12 | Native fixed instructions and full-turn constraints repeat completion and uncommon procedures, while reserved finish/block tool descriptions are only one sentence each. | 2; improve tool documentation before removing needed native protocol guidance | | F13 | Existing context-integrity/chat fixtures injected a QA manual, so their green results did not qualify the production tiny hire default. | Dedicated stock-harness suite measured real default hires; retained failures prevent blanket qualification. | -| F14 | Historical classic Claude/OpenCode skill runs save one Paperclip document; reduced runs save none while completing the task. Legacy ACP Claude has the same behavior under a separate guard failure. Pinned skill storage wording is ambiguous. | Measured delivery failures; keep original oracle. Improve legacy skill/API delivery guidance separately, then compare both variants with preserved and storage-specific cases. | +| F14 | Historical classic Claude/OpenCode skill runs save one Paperclip document; reduced runs save none while completing the task. Legacy ACP Claude has the same behavior under a separate guard failure. Pinned skill storage wording is ambiguous. | Measured delivery failures; keep original oracle. Classic Claude/OpenCode have later narrow skill-only repair trials. Legacy ACP Claude has no closing matched run and remains an additional merge-blocking behavioral defect beneath its unchanged credential-guard failure. | | F15 | Classic Claude restart chat fails its memory assertion in both variants. | Existing behavior, not attributable to this reduction from these trials. | | F16 | All six legacy ACP cells per variant fail the persisted-provider-credential guard. Three historical ACP cells also have clipped public prompt retrieval, and candidate ACP Codex chat lacks a complete invocation receipt. | Security/receipt qualification follow-ups; do not waive guard, expose raw sessions, or infer missing provider instructions from clipped receipts. | | F17 | Prerequisites beside the campaign root violate trusted artifact selection. A same-target pilot superseded 13 candidate cells; one historical AWS runner shut down without upload. | Packaging fixed at `1eb5ba420` and pilot passes. Directory-layout-only copies preserve every byte; missing cells alone recovered at unchanged source, completed failures never rerun. | @@ -475,6 +475,7 @@ will address them. Record intentional behavior explicitly rather than as a bug. | 2026-10-02 | Dotta approved minimal operational skill selection/link correction after retained OpenCode diagnosis. | Candidate `fe9dc1e3c` and baseline `0d7ecfa96d` have two matched Pass → Pass cases, zero new failures/passes and no pending pairs. All four exact-source gates pass 587 checks, all four provider runs and cleanup pass. Candidate original loads Paperclip/reference before delivery, but uses the wrong PAP prefix; baseline original saves publicly later within the same assignment and gives a bare path. Explicit clickable UI links are correct in both. [Complete report](2026-10-02-opencode-skill-routing-link-qualification.md); prior failures retained, no causal or broad quality claim. | | 2026-10-02 | Hiring #14985 merged with explicit user authorization at `862a5758ba0e88a33232c1f1fa645e85c38a3113`; its live measurement uses immutable unions. | Corrected production source `57dcee147` had all final checks and fresh 5/5 review. Candidate `9f5404ad3` and historical `296a4df85` share fixture/model/auth/common manual/operational/native context. All four original cells are retained; both pairs Fail → Fail only on exact-five-turn counting, six core checks pass both, source coverage uncomparable. No model reruns. [Complete report](2026-10-02-hiring-template-live-comparison.md). | | 2026-10-02 | Both remaining mutable PRs are green after merged-hiring rebase/restack. | #14948 `36aa4d81c`: 52 successful checks / two intentional skips, 599 credential-free prerequisites; #14961 `9138f570c`: 52 successful checks, 637 prerequisites. Both fresh Greptile 5/5, zero unresolved threads, mergeable, draft/unmerged. Prior browser/review failures and all measured refs remain preserved. | +| 2026-10-02 | Dotta requires no extra failing behavior before merging, beyond green CI. | PR #14948 does not meet that criterion: legacy ACP Claude loses its document only in the reduced cohort beneath an unchanged credential-guard failure, and subsequent repairs did not select it. Latest OpenCode Pass → Pass is skill-only with reduced manuals held constant, not a historical-manual repeat. Keep #14948 draft; native #14961 has no new failures within its six-pair scope but depends on resolving the base. No new sources, paid runs or merges. | | 2026-10-02 | Bounded hiring notification accounting is ported into both executable count guards on a separate master-based branch. | Draft [PR #15007](https://github.com/paperclipai/paperclip/pull/15007), `778308e032`: Initial 946 support checks and task-only replay are preserved. Current `fed172901` fixes both review findings with 977 support checks / 64 files, typecheck/discovery/contract checks; strict Codex action accounting verifies both variants, ACPX action attribution is unresolved/uncomparable. Original Fail → Fail, source-read gaps and historical body mismatch remain. Exact-head normal CI passes all required gates; fresh Greptile review pending; no models rerun. [Replay](https://github.com/paperclipai/paperclip/blob/fed1729018cc100f5f4bbfb692777e49009c423b/doc/plans/2026-10-02-hiring-executable-accounting-replay.md). | - F18: Native blocker browser assertion required a marker-only reply despite asking for owner/action/reason. Correct marker-plus-explanation checks symmetrically, calibrate contradictory and future-condition replies, retain original verdicts.