Files
PaperClipAI/tests/runner-e2e/fixtures
DottaandPaperclip 3bd52a9fa0 test(evals): pin task-history discovery and governed handoff repairs (#18995)
## Thinking Path

> - Paperclip must complete work through ordinary human and agent
controls.
> - The paired timeline attempt omitted a real human handoff, although
the other ten checks passed.
> - The agent used the always-exposed active-task history tool as if its
comments were a status audit.
> - The tool now states its actual scope and directs chronology and
handoff questions to the existing discovery and API tools.
> - A separate approved-tool attempt preserved permission and write
boundaries, but paired Markdown emphasis caused a false outcome
rejection.
> - The reviewed oracle normalizes that formatting before the existing
outcome checks and retains balanced negatives.
> - This PR pins those corrections with the already reviewed personal
secure-handoff admission repair for one new representative batch.

## Linked Issues or Issue Description

Refs #18962, #18968, #18971, #18972, #18977. Follows merged #18976.

The original paired results remain FAILED. Result 4412 omitted the
actual human handoff; this is a real answer gap, not a date-oracle
excuse. Result 4411 correctly handled the first rejection with no writes
or Trust action and rejected agent permission, but the observer misread
paired inline emphasis in the truthful outcome. The personal-secret
result 4378 remains FAILED at admission; secure save and worker proof
were not reached. No prior grade is rewritten and no new native pass is
claimed.

## What Changed

- Pin immutable candidate `249b487a2d2451763913038cc009f7090580d1f1` on
`codex/chat-catalog-human-answer-admission-candidate-20261104`.
- Extend exact prior source `b1119630003785c9e26e14c763859125642ad756`
with only the accepted timeline-discovery and formatting corrections: 12
source, test and generated files, with 72 additions and ten removals.
- Use the canonical `get_task_history` description in the actual exposed
semantic catalog. It reads only bounded active-task comments; calling
`get_task` does not change that active scope. It is not a status or
assignment audit. Progress and handoff questions must discover the
company timeline or issue activity, then use the existing API call.
- Regenerate semantic catalog and protocol projections on this frozen
candidate. Preserve all non-description fields, schemas and authority
boundaries. Do not copy newer master catalog or document-schema changes.
- Normalize paired inline emphasis before approved-replay outcome
checks. Declare only that oracle as app-review v6. Keep timeline v5,
personal v5 and company v4.
- Retain the reviewed personal completion-context, addressed-human,
optional-note and rendered-placeholder safety controls. Actual private
save, owner isolation, audit, authorized worker delivery and native
continuation remain required.
- Update only the existing manifest, expected pin and descriptor
snapshot in this infrastructure PR. Preserve all 244 identities,
original requests, run/human counts, clocks, model, launcher bytes,
privacy limits and qualification holds.

## Verification

- The combined candidate passed 542 existing owning controls across 19
suites, including catalog contracts, consistency and 242 lazy
declarations with no fixture callbacks. The manifest retains two legacy
tasks, for 244 identities.
- Candidate Runner checks passed: 15 semantic consumer tests and 12
catalog tests. The owning Runner TypeScript compiler passed after the
existing eval-kernel prerequisite built. The first compile reported
missing ignored eval-kernel output; the repair used candidate-owned
ignored dependency links and output only. No repository type or
acceptance rule was changed.
- All three generated checks passed: semantic-action catalog, semantic
contracts and protocol manifest. Deep comparisons prove each generated
catalog differs only in `get_task_history.description`. The seeded
fixture changes only its catalog hash, and the protocol manifest changes
only the hash of that fixture. The compiled canonical catalog hash
matches the fixture. The candidate import conflict added only
`getTaskHistoryAction`; no unrelated newer document-schema import was
copied.
- Both owning fixes cleared fresh exact-head Greptile 5/5 with no
actionable threads before this publication: timeline
`7f6ba9686b5edba005a49cd15c0d0b52248178dc` and approved replay
`0015a79c1e0a3af5292edf4a61aa9bab8f381899`. The personal prerequisite
`789bec74a0f43011723fff45596ef9d084bfaba7` also cleared exact-head 5/5
and all 12 threads before its earlier integration.
- All 131 existing pin, TestOps contract/API, adapter, reconciliation
and streaming controls passed. The descriptor snapshot matches the
actual candidate bytes. The only new declaration change from the prior
candidate is app-review v5 to v6. Whitespace checks passed.
- The SDK source guard reports 506 unchanged inputs, runtime pins
unchanged, registry
`efa70621e9fb42aa185f7ee616a1a726aa3eccb367e67e28817cedf26ef2ade8`. No
SDK registry or runtime certificate renewal occurred. The existing
reporting-lock guard returned `already-reviewed` with lock
`60f4c06bca47af3ab9d74281f2ef2e1ff559d566b862ba51c2fbe6c8e5bac35b`.
- Historical CI results remain recorded. Head 73ea had a server-5
cancellation with exact AWS Spot-reclamation proof. Head 16134 had 44
successful jobs, three failed leaves and dependent verify failure, plus
one selector skip. Two failed hosts had exact Spot termination proof;
server 11 had two real deadline PostgreSQL failures. After an initial
libzstd setup block, the unchanged four-test suite passed locally on
exact merge `70e68bff5fb89251f769671ce9ab949ab5bf7fc9` in 33.69 seconds
with its original deadlines and assertions. This does not establish the
CI cause. One recovery POST returned HTTP403; zero replications were
accepted.
- Prior head c95 finished with 41 successful jobs, six failures, one
cancellation and one selector skip. Four leaves had exact
Spot-termination proof; three of their execution logs were unavailable.
Server 11 instead had a real `live-recovery-work` next-deadline
assertion failure, and server 4 had a real workspace-health
connection-refused failure. Dependent verify failed. Windows passed. No
replay or claimed test fix erased those results; their two assertion
causes remain unresolved.
- Exact head `1e89df928f7ccb43bf7af3b814f2c2207d75883a` now has
successful normal CI run 41888521175 (attempt 2: 48 successful jobs and
one selector skip) and successful Windows run 41888521015. The first
attempt retained 46 successes, the failed Typecheck leaf and failed
dependent verify, plus one selector skip. Typecheck reported runner
shutdown without a TypeScript diagnostic; its Spot request reported no
capacity, while the later EC2 termination reason was
Client.UserInitiatedShutdown. One authorized unchanged-head targeted
recovery reran only Typecheck and dependent verify; both passed. This
does not claim a code fix or establish the cleanup caller. All other
successful results were carried forward, not fresh remeasurements.
- Current exact-head Greptile and Paperclip assessments are both 5/5
with no actionable findings; Paperclip check conclusion is neutral, but
its final assessment completed and states Ship. Security checks passed
and no review thread is unresolved. Full repository build/typecheck/test
were not repeated locally for this three-file metadata change. A fresh
actual-serving source, reporting-lock and selection guard remains
required after a normal merge and before a new Submit.
- No model call, eval dispatch, secret mutation, private archive
download, fleet change or merge was performed by this update. The single
targeted CI recovery is recorded above. Separate core PRs remain
unmerged.

## Risks

The discovery description may change tool choice; its effect on the
missing handoff remains unproved until a new native attempt. Formatting
normalization can affect outcome admission, so existing positive and
negative semantics remain required. The personal secure-entry and worker
phases are still unproved. The next representative set is timeline,
approved-tool replay and personal secret. Company selection remains
gated on personal secure-entry and worker proof. The uniform ten-case
regression remains pending, and the full 244-case catalog remains
unqualified.

## Model Used

OpenAI `gpt-6.1-sol` through Codex. Capabilities used: source analysis,
isolated mechanical integration and local code execution. The host does
not expose the context-window limit. The evaluated model remains
OpenCode/OpenRouter `deepseek/deepseek-v4.1-flash-1017`.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 13:00:37 -06:00
..