Files
PaperClipAI/tests/runner-e2e/task-chat-report.ts
T
DottaandPaperclip 7f10cc0585 feat(evals): show full task conversations in E2E reports (#18055)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Product E2E tests verify tasks through the application and retain
reports.
> - A final screenshot cannot show the complete task conversation.
> - Debugging a result requires the earlier requests, replies, and
answers.
> - This pull request captures task conversations before fixture cleanup
and adds them to Test details.
> - Reviewers can read each task thread without a running test instance.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The task evidence shown in Product E2E campaign reports.

**Current behavior**

Most task results show screenshots and checks. Screenshots show the
final visible state. They omit earlier messages and long replies. Only
the first-task suite has a conversation view.

**Proposed behavior**

Every standard runner attempt captures the saved conversation before
cleanup. Test details show full task messages, requests, answers, and
native agent progress. Related tasks have separate threads. Failed
attempts also retain available history. Partial captures are labeled.

**Reason and benefit**

Readers can debug a task by reading its conversation. They do not need
to reconstruct it from a screenshot or private run logs. GitHub searches
found no duplicate PR or issue for this change. This is an improvement
to test reporting and does not overlap a planned core feature in
ROADMAP.md.

**Breaking changes**

None. The result adds an optional `taskChat` field. Old results still
render. A result without a task has no chat section. Older reports need
retained conversation evidence before they can gain this view.

## What Changed

- Capture comments through all pages, task briefs, visible request copy,
recorded answers, and native progress before fixture cleanup.
- Include delegated tasks and standing chats. Preserve available history
when a read fails. Report capture limits explicitly.
- Add a closed, credential-redacted conversation format to normalized
results. Exclude reasoning, raw tool payloads, and provider session
metadata.
- Render task conversations inside Test details. Retain them through
report aggregation and dashboard regeneration.
- Add capture, publication-format, regeneration, and browser tests.
Update the evidence and security contracts.

## Verification

- Report tests: `node node_modules/vitest/vitest.mjs run --config
tests/runner-e2e/vitest.config.ts task-chat.test.ts
first-task-report.test.ts report.test.ts history.test.ts` — 103 tests
passed after review fixes.
- Browser tests: `node node_modules/@playwright/test/cli.js test
--config tests/runner-e2e/report-playwright.config.ts` — 6 tests passed
after rebase.
- Targeted TypeScript check of the new capture, renderer, validator,
report, and browser-test files passed.
- Tests cover a 1,105-message thread, a long final reply, event
pagination, partial capture, company access, credential redaction, HTML
escaping, and report regeneration.
- Review regressions reproduced two failures before repair: the task-run
API uses `runId`, and item reviews save verdicts in `result.items`.
Capture now follows these contracts. Both regression tests pass.
- Greptile reviewed commit `471236438e03e54cc4ad85de6653adf45835063c` at
5/5 with no new findings. The separate Paperclip review execution
stopped without an assessment and ended neutral. Its cause is not
established; this is not a completed Paperclip assessment.
- Execution discovery passed for
`core-compatibility.runner-codex.local.message-marker`. No paid provider
run was started.
- Full local commands were attempted. `pnpm -r typecheck` and `pnpm
build` stop because the DB package cannot find `tsx`. `pnpm test:run`
stops because the general-server lane cannot find `vitest`. These
package dependency links are absent in this checkout.
- [Remote CI on the final
commit](https://github.com/paperclipai/paperclip/actions/runs/40728689517)
passed: typecheck and release registry, all general and serialized test
shards, Runner static/Rust/Vitest checks, build, canary dry run, all
eight browser E2E shards, and final verification. Security and
contributor checks also passed.
- Open a new task-based result and expand **Test details**. Read **Task
chat** and expand each related task thread. Results without tasks keep
their existing layout.

## Risks

- Published reports now include fixture conversation text. Capture
retains only reviewed display fields and redacts credentials. Fixture
authors must keep this text suitable for public reports.
- Long threads increase report size. Collection has a 30-second
request-admission budget and page limits. It labels incomplete evidence
and does not change the workflow grade.
- Old reports cannot recover conversation text from screenshots. New
attempts must retain the evidence.

## Model Used

OpenAI Codex (GPT-6). The exact serving snapshot and context-window size
are not exposed in this session. Used reasoning, code execution,
repository tools, and browser tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 11:57:06 -05:00

24 lines
2.0 KiB
TypeScript

import type { TaskChatEvidence } from "./task-chat.js";
const html = (value: unknown) => String(value ?? "").replace(/[&<>"']/g,
c => ({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;", "'": "&#39;" })[c]!);
export function renderTaskChat(evidence?: TaskChatEvidence) {
if (!evidence || (!evidence.threads.length && evidence.status === "complete")) return "";
return `<section class="task-chat-recording" aria-label="Task chat">
<h4>Task chat</h4>
${evidence.status !== "complete" ? `<p class="detail">Conversation capture ${html(evidence.status)}. Some task history could not be collected (${evidence.missing.map(html).join(", ")}).</p>` : ""}
${evidence.threads.map(thread => `<details class="case-context conversation-details"${evidence.threads.length === 1 ? " open" : ""}>
<summary>${html(thread.identifier ? `${thread.identifier}: ` : "")}${html(thread.title)} · ${thread.messages.length} entries${thread.status === "partial" ? " · partial capture" : ""}</summary>
<section class="transcript" aria-label="Conversation for ${html(thread.title)}">
<p class="transcript-about">Recorded task brief, messages, requests and answers, with native agent progress when available. This is a read-only recording of the saved task conversation.</p>
${thread.messages.length ? thread.messages.map(message => `<article class="transcript-entry ${message.role === "user" ? "transcript-human" : "transcript-agent"}${message.kind === "interaction" ? " transcript-card" : ""}">
<header><strong>${html(message.author)}</strong>${message.kind === "answer" ? "<span>Response</span>" : ""}</header>
<div class="transcript-bubble"><div class="transcript-text">${html(message.body)}</div></div>
${message.at ? `<footer><time datetime="${html(message.at)}">${html(message.at)}</time></footer>` : ""}
</article>`).join("") : '<p class="detail">No messages were saved on this task.</p>'}
</section>
</details>`).join("")}
</section>`;
}