Files
PaperClipAI/tests
DottaandPaperclip b3c2172d41 fix(evals): verify native protected-action outcomes (#18255)
## Thinking Path

> - Paperclip manages work assigned to agents.
> - Product E2E stories ask for useful user-facing results.
> - Classic runner and deployed QA reuse the same service
approve/decline task prompts.
> - The service returns a receipt marker that belongs in a saved
briefing document.
> - State that destination explicitly and ask for the document link when
complete.
> - The deployed decline path also needs the same durable final-reply
proof as Plan and Stop.

## Linked Issues or Issue Description

Refs #18222. Builds on the merged reply observer in #18248.

**What happened?**
The shared service tasks ask for titles and a “verification code”
without specifying that the code belongs in the document or asking for
its final link. A native QA attempt created its briefing but then
repeatedly failed the completion tool’s array validation. Those invalid
arguments remain a separate observed failure; the wording change does
not establish their cause or relax validation. Source inspection also
found that the declined-action oracle requires a projected comment even
when the settled native thread renders a durable final message.

**Expected behavior**
Ask for page titles and the service receipt code in the saved briefing,
then a document link. After a declined action, require the existing
explanation-only fallback without another service call.

**Steps to reproduce**
Run the shared service-approve/service-decline stories through the
classic runner or QA Product E2E adapter.

## What Changed

- Clarify the natural-language task for both shared service stories: put
page titles and service receipt code in the document and provide its
link.
- Use the reloaded run-bound final reply for QA service declines. Pass
an explicit native receipt to the existing decline oracle; do not
synthesize issue comments. Require the final native run to start after
the saved decline and keep exact decision, truthful explanation, no
unread marker, zero calls and unchanged connection checks.
- Keep classic callers on their existing attributed-comment path. No
internal tool instructions are added to the user prompt; ownership,
service data and model/runtime limits stay strict.

## Verification

- Decline regression observed RED: two cases reject a valid saved native
explanation without projected comments. After the repair, 79 focused
tests passed across native reply, decline guidance, agent evidence and
journey contracts. A two-successful-run control rejects the earlier
receipt and accepts the later one. Negatives cover stale/wrong run,
wrong agent/task, missing final proof, summary/unsettled/hidden output,
false explanations and unread service data.
- Product E2E TypeScript and `git diff --check` passed.
- Live protected-action qualification is required after merging this and
the companion private service/cleanup fixes. The previous failed result
remains failed; this is a bounded task-clarity experiment, not a claimed
model or completion-schema repair.
- Current-head Greptile and Paperclip reviews are 5/5 with zero open
threads. CI initially failed the unrelated agent-chat-projects cleanup
POST with a PostgreSQL lock timeout after its assertions completed. One
bounded retry passed on unchanged source; all current-head gates passed
at `64bc169e147290ed4dbaa18221e0268362ce81ee`. The original failing
attempt and its logs remain available. The task-clarity change is still
a bounded experiment; the observer repair addresses a demonstrated
storage-assumption defect.

## Risks

A clearer task can still fail with the selected model. All acceptance
checks and completion input validation remain strict; a saved briefing
alone cannot qualify the complete journey. Only test prompts and test
evidence code change. Missing or contradictory native evidence fails
closed.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model ID and context
window are not exposed by this session. Used reasoning, source editing,
shell execution and test inspection.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-29 22:14:37 -05:00
..