mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 06:10:44 +01:00
docs(evals): qualify retained browser failure evidence
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
64986c5f25
commit
9d7c8a1b61
2 files changed
+7
-3
No files matched your search
@@ -77,8 +77,12 @@ Before future executable hiring runs, the normal scorer and the separate `chat-f
|
||||
|
||||
## Later strict action and settled-observation repair
|
||||
|
||||
The separate draft [PR #15007](https://github.com/paperclipai/paperclip/pull/15007) ports bounded accounting into both executable guards and fixes two review findings: racing observation generations and completion-turn writes missed by task counts. The [current versioned replay](https://github.com/paperclipai/paperclip/blob/fed1729018cc100f5f4bbfb692777e49009c423b/doc/plans/2026-10-02-hiring-executable-accounting-replay.md) and its v2 JSON pin code `e4077ade1818d98b9862ae79ee1d49a007dcf9c1`, hiring grader v3, turn accounting v2 and unchanged retained inputs. All 977 support tests, typecheck, discovery and canonical metadata checks pass; fresh PR review/CI remains required.
|
||||
The separate draft [PR #15007](https://github.com/paperclipai/paperclip/pull/15007) ports bounded accounting into both executable guards and fixes two review findings: racing observation generations and completion-turn writes missed by task counts. The [current versioned replay](https://github.com/paperclipai/paperclip/blob/fed1729018cc100f5f4bbfb692777e49009c423b/doc/plans/2026-10-02-hiring-executable-accounting-replay.md) and its v2 JSON pin code `e4077ade1818d98b9862ae79ee1d49a007dcf9c1`, hiring grader v3, turn accounting v2 and unchanged retained inputs. All 977 support tests, typecheck, discovery and canonical metadata checks pass. Normal CI on exact PR head `fed1729018cc100f5f4bbfb692777e49009c423b` passes all repository gates; fresh Greptile review remains pending. The PR stays draft and unmerged.
|
||||
|
||||
Strict action proof verifies Codex candidate/baseline (10/11 canonical executions, five/seven exactly matched successful GET receipts). ACPX's native host request IDs and provider execution IDs lack a proven mapping; both stricter action assessments remain unresolved/uncomparable. All observed native notification API receipts are successful GETs and no writes are observed, but name/order/count cannot substitute for exact attribution. This is missing evidence, not an observed new task failure. Original Fail → Fail, limited sidecar-v1 passes, all original source-read grades, historical Claude six-backtick mismatch, all 28 runs and unknown actual charges remain preserved. No models were rerun.
|
||||
|
||||
The [independent carrier audit](2026-10-02-hiring-read-evidence-carrier-audit.md) separately explains source-read target loss and ACPX preview/full-result metadata corruption. It proposes bounded truthful receipts; no production carrier change or retrospective source-read upgrade is included.
|
||||
|
||||
### Browser CI evidence clarification
|
||||
|
||||
The initial executable-replay report described its original-head browser failures as unrelated. The failures were in unchanged agent-run denial feedback and touch-picker scroll tests, and those paths/imports do not use the changed runner-eval helper, but their cause was not established. Both browser checks pass on the necessary review-fix head. The original failed jobs remain retained; no blind rerun was requested. This is a qualification of the earlier wording, not a causal diagnosis or a change to the live campaign verdicts.
|
||||
@@ -475,7 +475,7 @@ will address them. Record intentional behavior explicitly rather than as a bug.
|
||||
| 2026-10-02 | Dotta approved minimal operational skill selection/link correction after retained OpenCode diagnosis. | Candidate `fe9dc1e3c` and baseline `0d7ecfa96d` have two matched Pass → Pass cases, zero new failures/passes and no pending pairs. All four exact-source gates pass 587 checks, all four provider runs and cleanup pass. Candidate original loads Paperclip/reference before delivery, but uses the wrong PAP prefix; baseline original saves publicly later within the same assignment and gives a bare path. Explicit clickable UI links are correct in both. [Complete report](2026-10-02-opencode-skill-routing-link-qualification.md); prior failures retained, no causal or broad quality claim. |
|
||||
| 2026-10-02 | Hiring #14985 merged with explicit user authorization at `862a5758ba0e88a33232c1f1fa645e85c38a3113`; its live measurement uses immutable unions. | Corrected production source `57dcee147` had all final checks and fresh 5/5 review. Candidate `9f5404ad3` and historical `296a4df85` share fixture/model/auth/common manual/operational/native context. All four original cells are retained; both pairs Fail → Fail only on exact-five-turn counting, six core checks pass both, source coverage uncomparable. No model reruns. [Complete report](2026-10-02-hiring-template-live-comparison.md). |
|
||||
| 2026-10-02 | Both remaining mutable PRs are green after merged-hiring rebase/restack. | #14948 `36aa4d81c`: 52 successful checks / two intentional skips, 599 credential-free prerequisites; #14961 `9138f570c`: 52 successful checks, 637 prerequisites. Both fresh Greptile 5/5, zero unresolved threads, mergeable, draft/unmerged. Prior browser/review failures and all measured refs remain preserved. |
|
||||
| 2026-10-02 | Bounded hiring notification accounting is ported into both executable count guards on a separate master-based branch. | Draft [PR #15007](https://github.com/paperclipai/paperclip/pull/15007), `778308e032`: Initial 946 support checks and task-only replay are preserved. Current `fed172901` fixes both review findings with 977 support checks / 64 files, typecheck/discovery/contract checks; strict Codex action accounting verifies both variants, ACPX action attribution is unresolved/uncomparable. Original Fail → Fail, source-read gaps and historical body mismatch remain. Fresh CI/review pending; no models rerun. [Replay](https://github.com/paperclipai/paperclip/blob/fed1729018cc100f5f4bbfb692777e49009c423b/doc/plans/2026-10-02-hiring-executable-accounting-replay.md). |
|
||||
| 2026-10-02 | Bounded hiring notification accounting is ported into both executable count guards on a separate master-based branch. | Draft [PR #15007](https://github.com/paperclipai/paperclip/pull/15007), `778308e032`: Initial 946 support checks and task-only replay are preserved. Current `fed172901` fixes both review findings with 977 support checks / 64 files, typecheck/discovery/contract checks; strict Codex action accounting verifies both variants, ACPX action attribution is unresolved/uncomparable. Original Fail → Fail, source-read gaps and historical body mismatch remain. Exact-head normal CI passes all required gates; fresh Greptile review pending; no models rerun. [Replay](https://github.com/paperclipai/paperclip/blob/fed1729018cc100f5f4bbfb692777e49009c423b/doc/plans/2026-10-02-hiring-executable-accounting-replay.md). |
|
||||
|
||||
- F18: Native blocker browser assertion required a marker-only reply despite asking for owner/action/reason. Correct marker-plus-explanation checks symmetrically, calibrate contradictory and future-condition replies, retain original verdicts.
|
||||
- F19: Original prompt-removal cohorts loaded Paperclip; OpenCode skill truncation omitted the late API recipe. The early repair fixed Claude in one paired trial. In the repaired original OpenCode assignment, Paperclip was first loaded only during disposition recovery after a local-file write. Shared legacy operational-skill delivery/selection needs review before more recipe expansion.
|
||||
@@ -487,7 +487,7 @@ will address them. Record intentional behavior explicitly rather than as a bug.
|
||||
For each completed item, add the chosen behavior, changed paths, verification
|
||||
results, remaining exceptions, and follow-ups here before checking it off.
|
||||
|
||||
- F24: Hiring original oracle requires exactly five total runs, but production adds two legitimate `chat_task_completed` notification runs in every retained cell. All 28 actual runs remain counted. A separately versioned strict lifecycle accounting sidecar passes all four retained attempts with 69/69 calibrations and 12/12 predicates; no blanket run exclusion or original grade rewrite. Source-read coverage remains uncomparable. Both executable guards are corrected in separate draft PR #15007. Its stricter v3 replay verifies Codex action accounting; ACPX host/provider ID mapping remains unresolved/uncomparable. The v1 sidecar task-only limitation and original passes are retained; current-head CI/review remains required. Original machine results and source-read grading remain unchanged.
|
||||
- F24: Hiring original oracle requires exactly five total runs, but production adds two legitimate `chat_task_completed` notification runs in every retained cell. All 28 actual runs remain counted. A separately versioned strict lifecycle accounting sidecar passes all four retained attempts with 69/69 calibrations and 12/12 predicates; no blanket run exclusion or original grade rewrite. Source-read coverage remains uncomparable. Both executable guards are corrected in separate draft PR #15007. Its stricter v3 replay verifies Codex action accounting; ACPX host/provider ID mapping remains unresolved/uncomparable. The v1 sidecar task-only limitation and original passes are retained; Exact-head normal CI is green; fresh Greptile remains pending. Original-head browser failures occurred in unchanged paths, but their cause was not established; the review-fix head passes those checks without a blind rerun. Original machine results and source-read grading remain unchanged.
|
||||
- F25: Hiring stock read records have null path targets; Codex compound process command/output previews and historical Claude file previews are clipped. Actual read operations are observed before hire, but source-consumption coverage remains uncomparable. Six-backtick historical Claude body mismatch is a distinct exact-template coverage failure.
|
||||
|
||||
- F26: Read-only native hiring carrier audit confirms deliberate source-target loss in both frozen variants and an ACPX full-result hash/count recomputation from a roughly 4 KiB diagnostic preview. This does not establish provider-visible truncation or a prompt regression. Closed managed-source receipts and truthful full-result metadata are proposals only; no production carrier changes or retrospective source-read regrade. [Inspectable audit](2026-10-02-hiring-read-evidence-carrier-audit.md).
|
||||
|
||||
Reference in new issue
Block a user