Files
PaperClipAI/tests
DottaandPaperclip cb8df178c8 test(evals): pin the clarified five-case chat repairs (#19261)
## Thinking Path

> - Paperclip helps people manage AI agents and their work.
> - The TestOps catalog executes one reviewed immutable source with
declared model, permissions, counts, clocks, and private evidence gates.
> - Recent retained attempts exposed specific question, timeline,
secret-handoff, and team-reporting contract gaps.
> - Five focused repairs now have one qualified source; their component
checks do not establish a native pass.
> - This PR binds the trusted catalog lane to that source and records
its exact changed definitions.
> - One visible-form question is clarified and disclosed; all other
catalog requests stay fixed.
> - The controller, launcher, private evidence helpers, and execution
authority stay unchanged.

## Linked Issues or Issue Description

Refs: #19086, #19084, #19093, #19112, #19148, #19188.

**What happened?**
The retained native attempts include a timeline narrative rejected
despite the saved handoff, secure-entry admission failures before
downstream verification, and a truthful contextual team caveat rejected
after a correct repeat. Multipart attempts4623 and4717 did not complete
under the original broad request. All original statuses and evidence
remain unchanged.

**Expected behavior**
Measure the five reviewed repairs on one immutable source with the
original model, limits, and privacy gates. A component replay must not
rewrite an old result or count as a product pass. The catalog must
display and execute the same versioned request.

**Steps to reproduce**
After normal protected merge and fresh controller admission, use the
existing TestOps plan102 and job100 to select the intended five cases on
master. Root owns the actual Submit and original-evidence review. No
evaluation is dispatched by this PR.

## What Changed

- Pin only the catalog lane to
`6773b1c128a2e7efa7571ff04b38238f00d06117`, tree
`5669b199edc1aed3ab812403407b5930f5127da4`.
- Copy its exact descriptor snapshot, SHA256
`aaf15ca6755fcfe550daa98bca1ed263e3d625159b53cd1e1f52fefbf9474778`.
Advance only multipart53 v4, timeline466 v10, company-secret578 v8,
personal-secret580 v9, and team-install610 v4. Add four declared source
paths across the two secret cases.
- Record exactly one query clarification through the existing per-case
changed-field join. Old: “Help me choose the details for the launch page
before you build it.” New: “Walk me through the launch-page question
form so I can choose the audience and tone and enter my own headline
before you build anything.” Preserve the old query and failed results in
history. Retain inventory `0cf530…` and disclose clarified inventory
`814bb363…`. All other243 catalog requests are unchanged.
- Disclose that the company-secret second human message now uses actual
saved non-secret metadata instead of repeating the initial request. Its
continuation still requires the actual agent identity. No fabricated
agent comment or private value establishes the handoff.
- Team recovery uses independent saved canonical skill/version/member
proof. No recovery-success phrase is required. A contradictory current
denial still fails; a still-missing skill still requires truthful
disclosure.
- Keep all244 identities, the196/244 admission gate, model/profile,
human/native counts, clocks, retries0, cap8, launcher, private protocol,
and SDK506/efa qualification unchanged. No product code, full catalog,
workflow, permissions, secret, runtime, or helper change lands on
master.

## Verification

- Source owner qualification:680 affected controls on the accepted
parent and213 focused successor controls plus two explicit browser
canaries, fresh same-source server declarations, and full E2E TypeScript
pass on localNode26.4.0. Registered SDK506 Git inputs, production
source, private protocol, launcher, and original limits match the
accepted source. The source-only contract/flow additions are not copied
into this controller.
- Pin-local command: `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs
tests/runner-e2e/chat-full-catalog-testops-reconcile.test.mjs
tests/runner-e2e/chat-full-catalog-testops-adapter.test.mjs` —38 pass.
- Exact source descriptor and launcher bytes match the bound hashes. The
manifest keeps the original metadata source and source history, joins
the new descriptor, and permits only the one explicitly recorded53 query
clarification. `git diff --check` passes.
- Warm controller admission:10 committed helper byte checks, five
required source matches, and four trusted module imports passed with the
existing exact-main60f dependency graph. No SDK or native rebuild ran.
Actual post-merge master and imports still require a fresh check before
Submit.
- The53 clarification changes the evaluated question. It does not fix or
regrade the original300-second failures. All form validation, back
retention, custom answer, persisted-answer, continuation, and
visible-summary checks remain required. One targeted post-clarification
measurement is planned; any later whole-ten regression is separate, with
all attempts retained and no retry-until-green. The existing case53
description must be updated and read back after merge so TestOps
displays the clarified request.
- All five successor scenarios remain unrun. No new native pass,
uniform-ten pass, reliability claim, or master catalog qualification is
made. Focused source PRs remain separate and unmerged; their independent
CI does not replace this PR's required checks.
- Full workspace typecheck/build were not run locally for this
metadata-only PR. Current-head required CI, Windows, genuine review, and
zero unresolved material threads are required before normal protected
merge.

## Risks

A wrong source or descriptor would misstate the measured definition.
Strict target, snapshot, field, and identity joins cover that risk. The
clarified question cannot establish success for the old broad request.
Safe guidance admission still needs real human entry, private UI
persistence, and worker evidence in the next attempt. Component controls
do not prove those outcomes or completion within the original limits.

## Model Used

OpenAI `gpt-6.1-sol`, with reasoning, repository tools, and local code
execution. The host did not expose a context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-05 10:00:50 -06:00
..