fix(evals): verify native QA replies on the reloaded task thread (#18248)

## Thinking Path

> - Paperclip manages AI agents that do work for users.
> - Product E2E journeys check user outcomes on real deployments.
> - Native task threads can render saved final replies from a settled
run.
> - The new QA Plan and Stop checks looked only for agent comments.
> - Both flows reached the requested terminal state and showed the
correct reply, but that storage assumption rejected them.
> - This PR checks the reloaded visible reply and binds it to the final
successful run. Explicit presentation decisions remain authoritative.

## Linked Issues or Issue Description

Refs #18222 and #18244.

**What happened?**

The deployed Plan journey completed all three native runs, including
revision and approval. Stop completed native cancellation, resource
release and a new direction. Their final-answer checks failed because
the comments endpoint did not contain the rendered native reply. The
original failed grades remain unchanged.

**Expected behavior**

Require exactly one correct visible final answer after reload,
attributed to the final successful owned run. Reject an earlier run,
progress text, hidden content, duplicate answers or a cancelled run. If
its selected completion comment is projected, require that comment and
matching author/run/text. Otherwise require a single durable final
assistant message and its settled native final-response anchor.

**Steps to reproduce**

Run the Plan or Stop journey on a native deployment whose task thread
renders the settled run before a completion comment is projected.

## What Changed

- Add a narrow reply observer over the existing task-thread hooks. It
reloads the page and refreshes saved run/comment metadata while waiting.
- Bind native replies to a validated durable final assistant event and
the successful run's settled final-response anchor. Require the selected
comment when projected; support pending or suppressed projection without
accepting a summary fallback. Accept payload and item-normalized finals,
rejecting contradictory fields.
- Use the observer for Plan completion and Stop's new direction. Keep
existing run counts, deployment binding, cancellation and artifact
checks.
- Add negative grading tests, include the helper in the definition hash,
and document its evidence contract. Classic protocol comment matchers
are unchanged.

## Verification

- 63 focused tests passed across native reply evidence, agent evidence
and journey contracts.
- Product E2E TypeScript passed. The reply observer uses the remaining
journey deadline and backs off polling.
- Live Plan and Stop attempts exposed this grading failure. Their
original outcomes remain failed; fresh qualification is required after
merge.
- Full current-head CI passed at
`40840ed742977853630d733292461ad7003fc7fb`; Greptile and Paperclip
Review both scored 5/5, with zero unresolved review threads.

## Risks

The observer depends on existing task-thread test hooks and run anchors.
Missing or changed hooks fail closed. It reads rendered user-facing
text; a finish summary alone cannot satisfy it. Native event provenance,
run ownership and exact reply equality are required. No backend, model,
provider, budget or cancellation policy changes are included.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model ID and context
window are not exposed by this session. Used reasoning, source editing,
shell execution and test inspection.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

tests/runner-e2e/native-task-reply.ts

@@ -0,0 +1,82 @@
+import { expect, type Page } from "@playwright/test";
+import { isValidNativePrpEnvelope } from "./native-event-envelope.js";
+
+type Row = Record<string, any>;
+export interface TaskReplyObservation {
+  body: string; anchor: string | null; settled: boolean; nativeFinal: boolean; visible: boolean;
+}
+
+/** Native task replies can render from a saved turn before a completion comment
+ * is projected. Respect an explicit presentation decision when one exists. */
+export function nativeFinalReplyEvents(events: Row[], run: Row) {
+  return events.flatMap(event => {
+    const envelope = event.payload?.prpEvent, payload = envelope?.payload, item = payload?.item;
+    if (!envelope || !Number.isSafeInteger(event.seq) || event.seq < 1 || !isValidNativePrpEnvelope(envelope, event.protocolSchemaVersion)
+      || event.runId !== run.id || event.companyId !== run.companyId || event.agentId !== run.agentId
+      || envelope.runId !== run.id || envelope.sourceKind !== "runner" || event.eventType !== "item.completed" || envelope.eventType !== event.eventType
+      || typeof envelope.sourceEventId !== "string" || !envelope.sourceEventId || event.sourceEventId !== envelope.sourceEventId
+      || typeof envelope.sourceInstanceId !== "string" || !envelope.sourceInstanceId || event.sourceInstanceId !== envelope.sourceInstanceId
+      || !Number.isSafeInteger(envelope.sourceSeq) || envelope.sourceSeq < 1 || event.sourceSeq !== envelope.sourceSeq
+      || !["agentMessage", "assistant_message"].includes(payload?.kind ?? item?.type) || (payload.channel ?? item?.channel) !== "final"
+      || payload.kind != null && item?.type != null && payload.kind !== item.type
+      || payload.channel != null && item?.channel != null && payload.channel !== item.channel) return [];
+    const text = typeof payload.text === "string" ? payload.text : item?.text;
+    if (typeof text !== "string" || !text.trim() || typeof payload.text === "string" && typeof item?.text === "string" && payload.text !== item.text) return [];
+    return [{ seq: event.seq, sourceEventId: envelope.sourceEventId, sourceInstanceId: envelope.sourceInstanceId,
+      sourceSeq: envelope.sourceSeq, protocolSchemaVersion: event.protocolSchemaVersion, body: text.trim() }];
+  });
+}
+
+export function taskReplyBelongsToRun(observation: TaskReplyObservation, run: Row, agentId: string, comments: Row[], events: Row[] = []) {
+  if (typeof run.id !== "string" || !run.id || !agentId || run.agentId !== agentId || run.status !== "succeeded" || run.runtimeMode !== "native" || !observation.body || !observation.visible) return false;
+  const decision = run.resultJson?.presentationDecision;
+  if (decision != null && (typeof decision !== "object" || Array.isArray(decision))) return false;
+  if (decision?.commentId != null && typeof decision.commentId !== "string") return false;
+  const projected = typeof decision?.commentId === "string" ? comments.find(comment => comment.id === decision.commentId) : null;
+  if (projected) {
+    return observation.anchor === projected.id && projected.authorAgentId === agentId
+      && projected.createdByRunId === run.id && typeof projected.body === "string" && projected.body.trim() === observation.body;
+  }
+  const finals = nativeFinalReplyEvents(events, run);
+  return finals.length === 1 && finals[0]!.body === observation.body && observation.settled && observation.nativeFinal
+    && (observation.anchor === `${run.id}:turn` || observation.anchor?.startsWith(`${run.id}:turn:`) === true);
+}
+
+export function uniqueBoundTaskReply(all: TaskReplyObservation[], run: Row, agentId: string, comments: Row[], expectedText?: string, events: Row[] = []) {
+  const matches = all.filter(row => taskReplyBelongsToRun(row, run, agentId, comments, events)
+    && (expectedText === undefined || row.body === expectedText));
+  return matches.length === 1 && (expectedText === undefined || all.filter(row => row.visible && row.body === expectedText).length === 1)
+    ? matches[0]! : null;
+}
+
+/** Observe the real task thread after a fresh load, rather than a finish summary
+ * or a provider's claimed result. The caller separately validates task/run scope. */
+export async function readBoundTaskFinalReply(input: {
+  page: Page; runId: string; agentId: string; expectedText?: string; deadlineAt: number;
+  load(): Promise<{ run: Row; comments: Row[]; events: Row[] }>;
+}) {
+  await input.page.reload();
+  const replies = input.page.getByTestId("task-chat-thread").getByTestId("task-chat-agent-bubble");
+  const observe = () => replies.evaluateAll(elements => elements.map(element => ({
+    body: (element as HTMLElement).innerText.trim(),
+    anchor: element.closest("[data-thread-anchor]")?.getAttribute("data-thread-anchor") ?? null,
+    settled: element.closest('[data-testid="task-chat-turn"]')?.getAttribute("data-settled") === "true",
+    nativeFinal: element.closest('[data-testid="task-chat-final-response"]') !== null,
+    visible: element.getClientRects().length > 0 && getComputedStyle(element).visibility !== "hidden" && getComputedStyle(element).display !== "none",
+  })));
+  const read = async () => {
+    const state = await input.load(), all = await observe();
+    const match = state.run.id === input.runId ? uniqueBoundTaskReply(all, state.run, input.agentId, state.comments, input.expectedText, state.events) : null;
+    return { ...state, match };
+  };
+  const timeout = input.deadlineAt - Date.now();
+  if (timeout <= 0) throw new Error("native_reply_deadline");
+  await expect.poll(async () => {
+    return Boolean((await read()).match);
+  }, { timeout, intervals: [1000, 2000, 5000] }).toBe(true);
+  const { match, run, events } = await read();
+  if (!match) throw new Error("native_reply_changed");
+  const source = match.anchor === run.resultJson?.presentationDecision?.commentId ? "posted-comment" : "native-turn";
+  return { runId: input.runId, agentId: input.agentId, observedAfterReload: true,
+    source, finalEvent: source === "native-turn" ? nativeFinalReplyEvents(events, run)[0] : null, ...match };
+}