diff --git a/doc/plans/2026-10-02-hiring-template-live-comparison.md b/doc/plans/2026-10-02-hiring-template-live-comparison.md index 6eabbf503fa..5e973f1fa6d 100644 --- a/doc/plans/2026-10-02-hiring-template-live-comparison.md +++ b/doc/plans/2026-10-02-hiring-template-live-comparison.md @@ -1,6 +1,6 @@ # Matched hiring template qualification -TL;DR: **No new overall machine failures were observed in the two paired cases: Codex Fail → Fail and Claude Fail → Fail. There are 0 new passes, 2 unchanged failures, 0 pending pairs, and 2 uncomparable pairs.** All four cells hired/reused one coder, saved two correct independently checked JSON documents, and preserved the first. The same strict five-turn assertion fails all four because each has **7 successful runs**, including 2 automatic task-completion chat wakes. Instruction-read coverage remains **uncomparable**, so this trial cannot establish non-regression. **Separate provider-free accounting passes all four retained cells**, with 69/69 calibrations and strict known-task notification checks; original failed grades and uncomparable coverage remain unchanged. No models have been rerun. +TL;DR: **No new overall machine failures were observed in the two paired cases: Codex Fail → Fail and Claude Fail → Fail. There are 0 new passes, 2 unchanged failures, 0 pending pairs, and 2 uncomparable pairs.** All four cells hired/reused one coder, saved two correct independently checked JSON documents, and preserved the first. The same strict five-turn assertion fails all four because each has **7 successful runs**, including 2 automatic task-completion chat wakes. Instruction-read coverage remains **uncomparable**, so this trial cannot establish non-regression. **Limited sidecar-v1 accounting passes all four retained cells**, with 69/69 calibrations and known-task notification checks. It does not check unrelated document writes. A later stricter executable replay verifies Codex action accounting in both variants; ACPX action attribution remains unresolved/uncomparable. Original failed grades and source-read coverage remain unchanged. No models have been rerun. ## Frozen comparison @@ -31,7 +31,7 @@ The selected scope was 4 cells / 20 expected work turns, 15 minutes per cell, ze ## Original failures and retained-read diagnosis -All four original result files report `Hiring-template workflow outcome failed: five-successful-turns`; their machine failures remain unchanged. Exactly five requested work turns are accompanied by two automatic lead runs with `wakeReason: chat_task_completed`. All seven succeed in each cell. The separately versioned provider-free assessment below strictly validates every notification and passes all four retained cells. Original grades remain failed and source-read coverage is unchanged. +All four original result files report `Hiring-template workflow outcome failed: five-successful-turns`; their machine failures remain unchanged. Exactly five requested work turns are accompanied by two automatic lead runs with `wakeReason: chat_task_completed`. All seven succeed in each cell. The limited provider-free sidecar below validates task/lifecycle notification attribution and passes all four retained cells; it does not prove the absence of unrelated writes. The stricter assessment is linked below. Original grades remain failed and source-read coverage is unchanged. All four cells have zero recognized pre-hire skill/read receipts, making coverage uncomparable. Source fingerprints, actual production CEO bundle selection, assigned hiring skill, saved instructions and saved skills pass. Candidate coder content matches its source exactly in both profiles; historical Codex also matches. Historical Claude removes only six backticks wrapping three existing Markdown links in the saved coder body (3,990 expected versus 3,984 saved bytes). The role/policy text is otherwise identical. Its exact-body coverage failure is retained; this punctuation change does not establish a task-quality improvement. @@ -62,7 +62,7 @@ All eight declared screenshots were located, matched against their SHA-256 recei ## Separately versioned lifecycle accounting -This is **provider-free retained-evidence analysis**, not a replacement for the original machine results. [Sidecar v1 source](../../tests/runner-e2e/hiring-template-turn-accounting.sidecar.mjs) and [69/69 passing calibrations](../../tests/runner-e2e/hiring-template-turn-accounting.sidecar.test.mjs) strictly admit the five requested work turns plus at most two verified server completion turns. Positive calibrations cover five core runs, six with one batched notification, and seven with two distinct notifications. Negatives reject retries/failures, unknown or duplicate notifications, wrong task/account/user/company/generation, missing requests/replies, task creation by notification, extra work and inconsistent public ledgers. +This is **provider-free retained-evidence analysis**, not a replacement for the original machine results. [Sidecar v1 source](../../tests/runner-e2e/hiring-template-turn-accounting.sidecar.mjs) and [69/69 passing calibrations](../../tests/runner-e2e/hiring-template-turn-accounting.sidecar.test.mjs) strictly admit the five requested work turns plus at most two verified server completion turns. Positive calibrations cover five core runs, six with one batched notification, and seven with two distinct notifications. Negatives reject retries/failures, unknown or duplicate notifications, wrong task/account/user/company/generation, missing requests/replies, task creation by notification, extra task/run work and inconsistent public ledgers. Unrelated document writes were outside this v1 check; its passing verdict is limited accordingly. | Retained pair | Historical bounded accounting | Reduced bounded accounting | Original executable verdict | Source coverage | | --- | --- | --- | --- | --- | @@ -74,3 +74,11 @@ Every cell has exactly three distinct user-comment-attributed CEO turns, one cod The sidecar source SHA-256 is `693c639ab4ec159e30dfad98af75e43e91d75d5245ad035dfa66cd9f82a3eef8`; its calibration source hash is `36f5f03f0c315070aee0a5764fb8df7c32e8152b61df1a7a6d967f08c1ab0758`. Each receipt in the JSON report pins the exact original result, hiring snapshot, public API snapshot and grader bytes, along with the original measured source/schema/suite digest. It preserves original failed machine status and all coverage failures. No source-read grade is upgraded and no broad equivalence is claimed. Before future executable hiring runs, the normal scorer and the separate `chat-flow.ts` exact-five-run guard would both need this calibrated bounded rule. This report-only assessment changes neither frozen fixture nor executable verdict; no further model runs are planned. + +## Later strict action and settled-observation repair + +The separate draft [PR #15007](https://github.com/paperclipai/paperclip/pull/15007) ports bounded accounting into both executable guards and fixes two review findings: racing observation generations and completion-turn writes missed by task counts. The [current versioned replay](https://github.com/paperclipai/paperclip/blob/fed1729018cc100f5f4bbfb692777e49009c423b/doc/plans/2026-10-02-hiring-executable-accounting-replay.md) and its v2 JSON pin code `e4077ade1818d98b9862ae79ee1d49a007dcf9c1`, hiring grader v3, turn accounting v2 and unchanged retained inputs. All 977 support tests, typecheck, discovery and canonical metadata checks pass; fresh PR review/CI remains required. + +Strict action proof verifies Codex candidate/baseline (10/11 canonical executions, five/seven exactly matched successful GET receipts). ACPX's native host request IDs and provider execution IDs lack a proven mapping; both stricter action assessments remain unresolved/uncomparable. All observed native notification API receipts are successful GETs and no writes are observed, but name/order/count cannot substitute for exact attribution. This is missing evidence, not an observed new task failure. Original Fail → Fail, limited sidecar-v1 passes, all original source-read grades, historical Claude six-backtick mismatch, all 28 runs and unknown actual charges remain preserved. No models were rerun. + +The [independent carrier audit](2026-10-02-hiring-read-evidence-carrier-audit.md) separately explains source-read target loss and ACPX preview/full-result metadata corruption. It proposes bounded truthful receipts; no production carrier change or retrospective source-read upgrade is included. diff --git a/doc/plans/2026-10-02-stock-harness-paperclip-checklist.md b/doc/plans/2026-10-02-stock-harness-paperclip-checklist.md index b3675e6e673..3ff14014e31 100644 --- a/doc/plans/2026-10-02-stock-harness-paperclip-checklist.md +++ b/doc/plans/2026-10-02-stock-harness-paperclip-checklist.md @@ -475,7 +475,7 @@ will address them. Record intentional behavior explicitly rather than as a bug. | 2026-10-02 | Dotta approved minimal operational skill selection/link correction after retained OpenCode diagnosis. | Candidate `fe9dc1e3c` and baseline `0d7ecfa96d` have two matched Pass → Pass cases, zero new failures/passes and no pending pairs. All four exact-source gates pass 587 checks, all four provider runs and cleanup pass. Candidate original loads Paperclip/reference before delivery, but uses the wrong PAP prefix; baseline original saves publicly later within the same assignment and gives a bare path. Explicit clickable UI links are correct in both. [Complete report](2026-10-02-opencode-skill-routing-link-qualification.md); prior failures retained, no causal or broad quality claim. | | 2026-10-02 | Hiring #14985 merged with explicit user authorization at `862a5758ba0e88a33232c1f1fa645e85c38a3113`; its live measurement uses immutable unions. | Corrected production source `57dcee147` had all final checks and fresh 5/5 review. Candidate `9f5404ad3` and historical `296a4df85` share fixture/model/auth/common manual/operational/native context. All four original cells are retained; both pairs Fail → Fail only on exact-five-turn counting, six core checks pass both, source coverage uncomparable. No model reruns. [Complete report](2026-10-02-hiring-template-live-comparison.md). | | 2026-10-02 | Both remaining mutable PRs are green after merged-hiring rebase/restack. | #14948 `36aa4d81c`: 52 successful checks / two intentional skips, 599 credential-free prerequisites; #14961 `9138f570c`: 52 successful checks, 637 prerequisites. Both fresh Greptile 5/5, zero unresolved threads, mergeable, draft/unmerged. Prior browser/review failures and all measured refs remain preserved. | -| 2026-10-02 | Bounded hiring notification accounting is ported into both executable count guards on a separate master-based branch. | Draft [PR #15007](https://github.com/paperclipai/paperclip/pull/15007), `778308e032`: 946 support tests / 64 files, 99 core calibrations, typecheck, discovery and contract checks pass. Retained replay passes both corrected executable paths in all four attempts; original Fail → Fail, source-read uncomparable and historical Claude body mismatch remain. CI/review pending; no model reruns. [Replay](https://github.com/paperclipai/paperclip/blob/778308e032b7f76639ce73c0355ebe83142a94da/doc/plans/2026-10-02-hiring-executable-accounting-replay.md). | +| 2026-10-02 | Bounded hiring notification accounting is ported into both executable count guards on a separate master-based branch. | Draft [PR #15007](https://github.com/paperclipai/paperclip/pull/15007), `778308e032`: Initial 946 support checks and task-only replay are preserved. Current `fed172901` fixes both review findings with 977 support checks / 64 files, typecheck/discovery/contract checks; strict Codex action accounting verifies both variants, ACPX action attribution is unresolved/uncomparable. Original Fail → Fail, source-read gaps and historical body mismatch remain. Fresh CI/review pending; no models rerun. [Replay](https://github.com/paperclipai/paperclip/blob/fed1729018cc100f5f4bbfb692777e49009c423b/doc/plans/2026-10-02-hiring-executable-accounting-replay.md). | - F18: Native blocker browser assertion required a marker-only reply despite asking for owner/action/reason. Correct marker-plus-explanation checks symmetrically, calibrate contradictory and future-condition replies, retain original verdicts. - F19: Original prompt-removal cohorts loaded Paperclip; OpenCode skill truncation omitted the late API recipe. The early repair fixed Claude in one paired trial. In the repaired original OpenCode assignment, Paperclip was first loaded only during disposition recovery after a local-file write. Shared legacy operational-skill delivery/selection needs review before more recipe expansion. @@ -487,7 +487,9 @@ will address them. Record intentional behavior explicitly rather than as a bug. For each completed item, add the chosen behavior, changed paths, verification results, remaining exceptions, and follow-ups here before checking it off. -- F24: Hiring original oracle requires exactly five total runs, but production adds two legitimate `chat_task_completed` notification runs in every retained cell. All 28 actual runs remain counted. A separately versioned strict lifecycle accounting sidecar passes all four retained attempts with 69/69 calibrations and 12/12 predicates; no blanket run exclusion or original grade rewrite. Source-read coverage remains uncomparable. Both executable guards are corrected in separate draft PR #15007 with a separately versioned provider-free replay; current-head CI/review remains required. Original machine results and source-read grading remain unchanged. +- F24: Hiring original oracle requires exactly five total runs, but production adds two legitimate `chat_task_completed` notification runs in every retained cell. All 28 actual runs remain counted. A separately versioned strict lifecycle accounting sidecar passes all four retained attempts with 69/69 calibrations and 12/12 predicates; no blanket run exclusion or original grade rewrite. Source-read coverage remains uncomparable. Both executable guards are corrected in separate draft PR #15007. Its stricter v3 replay verifies Codex action accounting; ACPX host/provider ID mapping remains unresolved/uncomparable. The v1 sidecar task-only limitation and original passes are retained; current-head CI/review remains required. Original machine results and source-read grading remain unchanged. - F25: Hiring stock read records have null path targets; Codex compound process command/output previews and historical Claude file previews are clipped. Actual read operations are observed before hire, but source-consumption coverage remains uncomparable. Six-backtick historical Claude body mismatch is a distinct exact-template coverage failure. - F26: Read-only native hiring carrier audit confirms deliberate source-target loss in both frozen variants and an ACPX full-result hash/count recomputation from a roughly 4 KiB diagnostic preview. This does not establish provider-visible truncation or a prompt regression. Closed managed-source receipts and truthful full-result metadata are proposals only; no production carrier changes or retrospective source-read regrade. [Inspectable audit](2026-10-02-hiring-read-evidence-carrier-audit.md). + +- F27: Native hiring completion actions carry internally paired native API IDs, but ACPX host request IDs differ from canonical provider execution IDs with no established mapping. Strict v3 accounting does not join by name/order/count; both ACPX action assessments remain uncomparable despite only successful native GET receipts being observed. Codex exact IDs verify both variants. Keep this action-attribution gap separate from source-read F25/F26, and do not weaken the carrier/oracle or rerun providers.