Files
PaperClipAI/tests/runner-e2e/chat-catalog-domains/runtime-email.ts
T
f4337bb5fb test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature evals measure whether agents complete real chat
requests through the product
> - The chat catalog eval harness and its repairs live on the
`chat-eval/master-candidate` branch, not on master
> - Today's work made 139 of 242 chat evals pass in verified single-case
live runs. Each product fix landed on master as its own pull request
> - The eval-side changes (setups, graders, fault seams, evidence
capture) still exist only on the candidate
> - This pull request lands the harness on master. Master stays the
source of truth for product behavior
> - The benefit is one tree: evals run against master commits, and
product fixes reach the evals as soon as they merge

## Linked Issues or Issue Description

Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs
#20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409

**What existing behavior does this improve?**
The chat catalog eval harness and the product it evaluates live on a
long-running branch outside master. Eval fixes do not reach master, so
master-based eval runs use old setups and graders.

**Subsystem affected**
`tests/runner-e2e` chat catalog harness, plus a small set of product
changes that exist only on the candidate.

**Proposed change**
Land the harness on master. Every product fix from this eval work
already merged to master as its own pull request. Conflicts keep
master's product behavior.

## What Changed

- Most of the diff (808 of 851 files) is under `tests/runner-e2e/`:
chat-catalog domains, scenarios, descriptors, the TestOps manifest,
private evidence tooling, and their tests.
- Graders now check saved state, setup literals and rendered or clicked
links. Many keyword and regex checks on free-form replies are removed.
Each changed grader has a bumped `oracleVersion` and calibration tests
(a valid reply, a reworded valid reply, and broken variants that must
fail).
- The shared link helper `hasLink` no longer treats a list link (for
example `/COM/issues?view=mine`) as a link to every task without a URL
key.
- Setup fixes build real state through supported product routes: routine
history, agent grants through the board grant route, budget holds with a
real cap, and a storage-layer archive fault for
`inbox-search.archive-partial-failure`.
- Browser boundary journeys (sign-in, chat-disabled, onboarding) now
write the standard pass evidence files.
- Retargeted cases, approved by the maintainer:
`agent-chat.chat-disabled` (Conference Room gate),
`agents.restore-instructions` (no saved instruction history),
`access.manage-membership` (member permission edits stay board-only).
- The private evidence trusted files (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
byte-identical to master.
- Product files outside `tests/` (43 files, +1,328 −49) carry
candidate-only changes that master does not yet have. Most are tests.
The main code changes are in `request-human-input.ts`,
`heartbeat-run-summary.ts`, `native-completion-feedback.ts`,
`native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and
`IssueChatLongThreadPerf.tsx`.

## Verification

- `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts
chat-catalog-consistency.test.ts unit-launcher.test.mjs
--testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests
pass.
- `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `tsc` for `server` and `tests/runner-e2e`: clean.
- Live single-case verification: 139 of 242 cases passed on paid runs,
and a person read the saved evidence for each. These are single-case
runs, not one full-cohort run.

## Risks

- Large diff. Most of it is test-only, but the 43 non-test files change
product code paths. Review those files first.
- Some graders are now less strict on reply wording. A person must read
replies for meaning on those cases.
- 42 cases need disposable outside accounts (Notion, Slack, GitHub,
Daytona and others) and stay blocked. `access.invite-teammate` stays
failing until local agent isolation lands, because the runner's durable
state keeps a live invite token.
- The paid workflow compares the trusted private-evidence files with
master. They are unchanged, so this risk is low.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code. Subagents used the same model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 08:02:11 -06:00

153 lines
30 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import { createHash, randomUUID } from "node:crypto";
import { expect } from "@playwright/test";
import type { Issue } from "../../../packages/shared/src/types/issue.js";
import type { EmailEndpointSummary, EmailPublicationSummary, EmailThreadSummary } from "../../../packages/shared/src/types/email.js";
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
import { collectRunEvents } from "../run-observations.js";
import { pollUntil, RunnerApiHttpError } from "../api.js";
import { sendChatMessage } from "../chat-flow.js";
import { checks, type Entity } from "./internal-support.js";
import { openStandingChat } from "./identity-support.js";
import { selectedExternalBinding, externalRequirements } from "./external-fixtures.js";
import { runtimeDefinition, runtimeObservation, observeRuntime } from "./runtime-support.js";
import { privateEmailOperation, type EmailPrivateInput, type EmailPrivateReceipt } from "./runtime-email-private.js";
import { armOwnedEmailResponseLoss, readOwnedEmailFault, releaseOwnedEmailResponseLoss, type EmailFaultRequest, type AcceptedEmailFault, type EmailFaultStatus } from "./runtime-email-fault.js";
const hash = (value: string) => createHash("sha256").update(value).digest("hex");
const record = (value: unknown): Record<string, unknown> => value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
const privateFixtures = new WeakMap<ChatCatalogContext, EmailPrivateInput>();
const faultRequests = new WeakMap<ChatCatalogContext, EmailFaultRequest>();
export async function prepareOwnedEmailConnection(ctx: ChatCatalogContext) {
const binding = selectedExternalBinding(ctx), p = binding.prerequisites;
if (!ctx.input.privateSecrets || p.recipientAuthorization !== "self-addressed-only" || p.inboxId !== p.recipient || binding.resourceIds.length !== 1 || binding.resourceIds[0] !== p.inboxId || !/^PAPERCLIP_CHAT_EVAL_[A-Z0-9_]+$/.test(p.apiTokenEnv ?? "")) throw new Error("Setup blocker: require the explicitly authorized dedicated disposable QA inbox, exactly its self-addressed recipient and declared private API-token reference; metadata visibility alone is not send permission");
const credential = ctx.input.privateInputs?.read(p.apiTokenEnv!); if (!credential) throw new Error("Selected dedicated AgentMail key reference is unavailable");
const input: EmailPrivateInput = { operation: "setup", scenarioId: ctx.scenario.id as EmailPrivateInput["scenarioId"], companyId: ctx.input.fixtures.company.id, agentId: ctx.input.fixtures.agent.id, baseURL: ctx.input.api.baseURL, inboxId: p.inboxId!, recipient: p.recipient!, dedicatedDisposableInbox: true, credential, endpointId: randomUUID(), cookies: await ctx.input.page.context().cookies(ctx.input.api.baseURL) };
privateFixtures.set(ctx, input);
const receipt = await privateEmailOperation(input, ctx.input.privateSecrets); privateFixtures.set(ctx, { ...input, connectionId: receipt.connectionId, endpointId: receipt.endpointId });
const endpoint = (await ctx.input.api.get<EmailEndpointSummary[]>(`/api/companies/${input.companyId}/email/inboxes`)).find(row => row.id === receipt.endpointId), session = await ctx.input.api.get<{ user: { id: string } }>("/api/auth/get-session");
if (!endpoint || endpoint.companyId !== input.companyId || endpoint.assignedAgentId !== input.agentId || endpoint.address !== input.inboxId || endpoint.status !== "active" || receipt.personId !== session.user.id) throw new Error("The exact real QA inbox endpoint/fixture person failed independent public ownership checks");
return { endpoint, emailConnectionId: receipt.connectionId, personId: receipt.personId, externalIdentity: { email: input.recipient }, providerProbes: [{ phase: "owned-email-setup", operation: "actual-private-account-and-inbox-setup", publicSetupOperationCount: receipt.publicSetupOperationCount, directProviderProbeCount: receipt.directProviderProbeCount, status: "active", usage: "unavailable", cost: "unavailable" }] };
}
export async function inspectOwnedEmailPublication(ctx: ChatCatalogContext, publication: EmailPublicationSummary, after: string, expectedRecipient?: string, expectedReply?: { threadId: string; messageId: string }) {
const input = privateFixtures.get(ctx); if (!input || !ctx.input.privateSecrets || !publication.request || !publication.providerMessageId) throw new Error("Actual public email acceptance and private owned inbox binding are required");
return privateEmailOperation({ ...input, operation: "inspect-publication", expectedRecipient, expectedReply, subject: publication.request.subject!, textSha256: hash(publication.request.text.replace(/\r\n/g, "\n").trim()), after, publicationId: publication.id, expectedPublicReceipt: { publicationId: publication.id, providerMessageId: publication.providerMessageId } }, ctx.input.privateSecrets);
}
export async function seedOwnedReviewMessages(ctx: ChatCatalogContext) {
const input = privateFixtures.get(ctx), binding = selectedExternalBinding(ctx), p = binding.prerequisites;
if (!input || !ctx.input.privateSecrets || ctx.scenario.id !== "chat-channels.reply-to-email" || p.reviewerAuthorization !== "reply-to-selected-synthetic-only" || !p.reviewerAddress || p.reviewerAddress === input.inboxId || !/^PAPERCLIP_CHAT_EVAL_[A-Z0-9_]+$/.test(p.senderCredentialEnv ?? "")) throw new Error("Setup blocker: an explicitly authorized distinct synthetic reviewer mailbox and private sender credential are required; self-mail permission is insufficient");
const credential = ctx.input.privateInputs?.read(p.senderCredentialEnv!); if (!credential) throw new Error("Selected synthetic reviewer private credential reference is unavailable");
return privateEmailOperation({ ...input, operation: "seed-review", reviewer: { address: p.reviewerAddress, credential, authorization: "reply-to-selected-synthetic-only", nonce: randomUUID() } }, ctx.input.privateSecrets);
}
async function providerOperation(ctx: ChatCatalogContext, operation: "inspect" | "resolve", state: Record<string, unknown>) {
const input = privateFixtures.get(ctx); if (!input || !ctx.input.privateSecrets) throw new Error("Original private email fixture identity is unavailable");
const root = process.env.PAPERCLIP_RUNNER_E2E_TEMP_ROOT; if (!root) throw new Error("Owned email fault root is unavailable");
const accepted = await readOwnedEmailFault<AcceptedEmailFault>(root, "accepted-private.json"), request = state.faultRequest as EmailFaultRequest;
if (accepted.executionId !== request.executionId || accepted.attemptId !== request.attemptId || accepted.nonce !== request.nonce || accepted.publicationId !== state.publicationId) throw new Error("The actual private accepted provider receipt does not bind this exact original action");
return privateEmailOperation({ ...input, operation, subject: request.subject, textSha256: request.textSha256, after: new Date(request.createdAt - 1000).toISOString(), publicationId: accepted.publicationId, accepted }, ctx.input.privateSecrets);
}
export function assertsEmailDelivery(reply: string) {
return reply.split(/[.!?;\n]+/).some(clause => /\b(?:was|has been|is|already|successfully|definitely) delivered\b/i.test(clause) && !/\b(?:whether|if|cannot|can[’']?t|not|no proof|unconfirmed|unknown|uncertain)\b/i.test(clause));
}
export function uncertainEmailReply(reply: string) {
return /uncertain|unknown|unconfirmed|cannot confirm|can['’]?t confirm|not (?:yet )?confirmed|may (?:have|already)|possibly (?:sent|delivered)/i.test(reply) && /receipt|inspect|check|verif|reconcil|provider|original (?:send|message)/i.test(reply)
&& !/\bI(?:['’]ll| will| have|['’]ve)\s+(?:now |already |successfully )?(?:resend|send (?:it |the (?:same )?email )?again|resent)\b|\b(?:definitely|certainly) (?:failed|not sent|delivered)\b/i.test(reply) && !assertsEmailDelivery(reply);
}
/** Trusted native tool correlation permits both the connector facade and its
* equivalent company-scoped public API. Assistant prose never supplies receipt proof. */
export function actualUncertainEmailReads(events: Array<{ seq?: number; eventType?: string; payload?: unknown }>, expected: { companyId: string; endpointId: string; publicationId: string; issueId: string }) {
const calls = new Map<string, { name: string; input: Record<string, unknown>; seq: number }>(), results = new Map<string, unknown>(), completed = new Map<string, string>();
for (const event of events) { const payload = record(record(record(event.payload).prpEvent).payload), item = record(payload.item); if (event.eventType === "item.started" && item.type === "tool_use" && typeof item.id === "string" && typeof item.name === "string") calls.set(item.id, { name: item.name, input: record(item.input), seq: event.seq ?? -1 }); if (event.eventType === "item.completed" && item.type === "tool_result" && typeof item.id === "string" && (!item.tool_use_id || item.tool_use_id === item.id)) results.set(item.id, item.result); if (event.eventType === "tool.execution.completed" && payload.status === "completed" && typeof payload.executionId === "string") completed.set(payload.executionId, String(payload.name)); }
return [...calls].flatMap(([id, call]) => {
if (completed.get(id) !== call.name || call.seq < 0) return [];
const result = record(results.get(id)), api = call.name === "call_api", data = api ? record(result.data) : result, params = record(call.input.pathParams);
const target = call.name === "agentmail_delivery" && call.input.publicationId === expected.publicationId || call.name === "agentmail_read_thread" || api && params.companyId === expected.companyId && (call.input.operationId === "GET /api/companies/{companyId}/email/deliveries/{publicationId}" && params.publicationId === expected.publicationId || call.input.operationId === "GET /api/companies/{companyId}/email/tasks/{issueId}" && params.issueId === expected.issueId);
if (!target || api && (result.ok !== true || result.status !== 200)) return [];
const publications = Array.isArray(data.publications) ? data.publications.map(record) : [data], publication = publications.find(row => row.id === expected.publicationId && row.issueId === expected.issueId && row.outcome === "uncertain" && row.providerMessageId === null && record(row.request).endpointId === expected.endpointId);
const earlierWrite = [...calls.values()].some(other => other.seq >= 0 && other.seq < call.seq && (other.name === "agentmail_send" || other.name === "call_api" && other.input.operationId === "POST /api/companies/{companyId}/email/send"));
return publication && !earlierWrite ? [{ executionId: id, actualPublicationId: expected.publicationId, uncertain: true }] : [];
});
}
export function actualOriginalEmailSend(events: Array<{ eventType?: string; payload?: unknown }>, expected: { companyId: string; taskId: string; endpointId: string; publicationId: string }) {
const calls = new Map<string, string>(), results = new Set<string>(), completions = new Map<string, string>();
for (const event of events) { const payload = record(record(record(event.payload).prpEvent).payload), item = record(payload.item);
if (event.eventType === "item.started" && item.type === "tool_use" && typeof item.id === "string") { const input = record(item.input), api = item.name === "call_api", request = record(api ? input.body : input.request);
if ((item.name === "agentmail_send" || api && input.operationId === "POST /api/companies/{companyId}/email/send" && record(input.pathParams).companyId === expected.companyId) && request.parentIssueId === expected.taskId && request.endpointId === expected.endpointId && request.idempotencyKey === expected.publicationId && !request.conversationId) calls.set(item.id, String(item.name)); }
if (event.eventType === "item.completed" && item.type === "tool_result" && typeof item.id === "string" && (!item.tool_use_id || item.tool_use_id === item.id)) { const result = record(item.result), data = calls.get(item.id) === "call_api" ? result.ok === true && result.status === 202 ? record(result.data) : {} : result; if (data.id === expected.publicationId && typeof data.issueId === "string") results.add(item.id); }
if (event.eventType === "tool.execution.completed" && payload.status === "completed") completions.set(String(payload.executionId), String(payload.name));
}
return [...calls].some(([id, name]) => results.has(id) && completions.get(id) === name);
}
export function gradeUnknownEmail(value: unknown) {
const c = checks(), o = runtimeObservation(value); c.add("real-authorized-send-with-genuine-lost-response", !!o, "One actual authorized self-addressed send, real upstream acceptance and production uncertainty must be observed."); if (!o) return c.result(); const s = o.state;
c.add("original-uncertainty-and-exact-owned-outage", s.actualOriginalUncertain === true && s.setupSendBound === true && s.faultOriginalBinding === true && s.outageActiveDuringCanonical === true && s.originalUncertaintyRecorded === true, "The original genuine native send belongs to the exact task/inbox; production has no reliable provider receipt during the bounded declared outage.");
c.add("canonical-inspects-real-receipts-and-does-not-duplicate", s.canonicalReadBound === true && uncertainEmailReply(String(s.canonicalReply)) && s.noNewAgentPublication === true && s.directMatchedSentCount === 1, "The canonical run inspects authoritative original task/publication receipts, truthfully reports uncertainty and produces no new agent publication or physical duplicate; same-key infrastructure retries are distinguished.");
c.add("actual-provider-reconciliation-through-supported-path", s.originalProviderReceiptMatched === true && s.outageReleased === true && s.reconciliationBound === true && s.resolutionActorBound === true && ["sent", "delivered"].includes(String(s.finalOutcome)), "After independent real provider inspection, the exact original send reconciles through ordinary provider intake or authorized human resolution without resending.");
const report = String(s.confirmationReply ?? "");
c.add("phase-bound-real-outcome-with-no-false-delivery", s.confirmationBound === true && /sent|accepted|provider.*(?:receipt|message)|reconcil/i.test(report) && (s.finalOutcome === "delivered" || !assertsEmailDelivery(report)) && s.finalDirectMatchedSentCount === 1, "The subsequent actual reply reports the verified send receipt and distinguishes acceptance from unproven recipient delivery; a final independent duplicate check still finds one sent message.");
c.add("finite-real-source-turns-and-original-delivery-retained", o.runs.length === 4 && o.runs.every(row => row.status === "succeeded" && row.agentId === o.leadId) && s.sameOriginalRequestRetained === true && s.sameOriginalConversationRetained === true && o.tasks.length === 2 && s.actualNormalTaskSendBound === true, "One genuine setup human chat delegation and actual normal-task send plus canonical and factual follow-up are the only native work; the original request/conversation and saved uncertainty evidence remain inspectable.");
return c.result();
}
const requirement = externalRequirements["recovery.unknown-email-outcome"]!;
async function setupUnknownEmail(ctx: ChatCatalogContext) {
const root = process.env.PAPERCLIP_RUNNER_E2E_TEMP_ROOT, attemptId = process.env.PAPERCLIP_E2E_ACCOUNTING_ATTEMPT_ID;
if (!root || !attemptId) throw new Error("Selected email lost-response fixture requires its actual owned server hook and attempt identity");
const installed = await readOwnedEmailFault<{ schema: string; executionId: string; attemptId: string; pid: number }>(root, "installed.json");
if (installed.schema !== "paperclip.email-response-loss-installed.v1" || installed.executionId !== ctx.input.execution.id || installed.attemptId !== attemptId || !Number.isSafeInteger(installed.pid)) throw new Error("Exact selected email transport-loss hook is not installed; no provider setup may begin");
const source = await openStandingChat(ctx), connection = await prepareOwnedEmailConnection(ctx);
const nonce = randomUUID(), subject = `Authorized QA message ${nonce}`, text = `This is one authorized self-addressed delivery check ${nonce}. Please retain this exact notice.`;
const request: EmailFaultRequest = { schema: "paperclip.email-response-loss.v1", executionId: ctx.input.execution.id, attemptId, companyId: ctx.input.fixtures.company.id, endpointId: connection.endpoint.id, agentId: ctx.input.fixtures.agent.id, sourceIssueId: source.id, inboxId: connection.externalIdentity.email, recipient: connection.externalIdentity.email, subject, textSha256: hash(text), nonce, createdAt: Date.now(), expiresAt: Date.now() + 480000, dedicatedDisposableInbox: true };
await armOwnedEmailResponseLoss(root, request); faultRequests.set(ctx, request);
for (const scope of [{ scopeType: "company", scopeId: request.companyId }, { scopeType: "agent", scopeId: request.agentId }]) await ctx.input.api.post(`/api/companies/${request.companyId}/budgets/policies`, { ...scope, amount: 500, hardStopEnabled: true, isActive: true, unpricedUsagePolicy: "allow" });
await ctx.input.api.post(`/api/companies/${request.companyId}/tools/policies`, { name: `Exact authorized self-addressed QA send ${nonce}`, policyType: "allow", priority: 10, selectors: { connectionId: connection.endpoint.connectionId, toolNames: ["email.send"] }, conditions: { arguments: { fieldEquals: { to: [request.recipient], subject, text } } }, config: {} });
await ctx.input.page.goto(`/${ctx.input.fixtures.company.issuePrefix}/issues/${source.id}`, { waitUntil: "domcontentloaded" });
const initial = `Please create exactly one normal task assigned to yourself, linked as a child of this conversation, to send one email from my connected inbox to ${request.recipient}, with subject exactly ${JSON.stringify(subject)} and body exactly ${JSON.stringify(text)}. This is my authorized self-addressed QA notice. Use the actual authorized email connection from that normal task run, retain the real send receipt, and leave that normal task blocked awaiting human resolution if the send is uncertain. Do not send from Agent Chat itself, create any further work tasks, or send any other message.`;
await sendChatMessage(ctx.input.page, initial);
const ready = await pollUntil({ label: "Actual initial native email send loses its real accepted provider response", deadlineAt: Date.now() + 240000,
load: async () => { await ctx.refreshIssue(); const accepted = await readOwnedEmailFault<AcceptedEmailFault>(root, "accepted-private.json").catch(error => { if ((error as NodeJS.ErrnoException).code === "ENOENT") return undefined; throw error; }); return { source: ctx.issue(), runs: await ctx.allRuns(), accepted, delivery: accepted ? await ctx.input.api.get<EmailPublicationSummary>(`/api/companies/${request.companyId}/email/deliveries/${accepted.publicationId}`) : undefined }; },
accept: v => v.source?.conversationState === "waiting" && v.runs.length === 2 && v.runs.every(row => row.status === "succeeded") && v.delivery?.outcome === "uncertain" && v.delivery.providerMessageId === null,
reject: v => v.runs.length > 2 || v.runs.some(row => ["failed", "cancelled", "timed_out"].includes(row.status)) ? "Initial authorized setup email admitted extra or failed native work" : undefined });
const delivery = ready.delivery!, accepted = ready.accepted!, thread = await ctx.input.api.get<EmailThreadSummary>(`/api/companies/${request.companyId}/email/tasks/${delivery.issueId}`);
const comments = await ctx.input.api.get<Entity[]>(`/api/issues/${source.id}/comments?order=asc`), human = comments.filter(row => !row.authorAgentId), setupChatRun = ready.runs.find(row => row.contextSnapshot?.issueId === source.id)!, mailTaskId = delivery.request?.parentIssueId, run = ready.runs.find(row => row.contextSnapshot?.issueId === mailTaskId)!;
const mailTask = await ctx.input.api.get<Issue>(`/api/issues/${mailTaskId}`);
const sendEvents = run ? await collectRunEvents<{ seq?: number; eventType?: string; payload?: unknown }>((afterSeq, limit) => ctx.input.api.get(`/api/heartbeat-runs/${run.id}/events?afterSeq=${afterSeq}&limit=${limit}`)) : [];
const activity = await ctx.input.api.get<Entity[]>(`/api/companies/${request.companyId}/activity?limit=1000`);
const actualNormalTaskSendBound = !!run && actualOriginalEmailSend(sendEvents, { companyId: request.companyId, taskId: mailTaskId!, endpointId: request.endpointId, publicationId: delivery.id }) && activity.some(row => row.action === "email.queued" && row.actorType === "agent" && row.actorId === request.agentId && row.details?.publicationId === delivery.id && row.details?.endpointId === request.endpointId) && mailTask.companyId === request.companyId && mailTask.parentId === source.id && mailTask.assigneeAgentId === request.agentId && mailTask.workMode === "standard" && mailTask.status === "blocked" && run.agentId === request.agentId && run.runtimeMode === "native";
const setupSendBound = accepted.nonce === request.nonce && accepted.executionId === request.executionId && accepted.attemptId === attemptId && accepted.upstreamStatus >= 200 && accepted.upstreamStatus < 300 && delivery.request?.endpointId === request.endpointId && delivery.request.subject === subject && hash(delivery.request.text) === request.textSha256 && thread.endpoint.id === request.endpointId && human.length === 1 && human[0]?.body === initial && actualNormalTaskSendBound && !!setupChatRun && Array.isArray(setupChatRun.contextSnapshot?.wakeCommentIds) && setupChatRun.contextSnapshot.wakeCommentIds.includes(human[0]!.id);
await ctx.input.evidence("runtime-original-email-uncertainty.json", { publicationId: delivery.id, issueId: delivery.issueId, conversationId: delivery.conversationId, outcome: delivery.outcome, providerMessageId: delivery.providerMessageId, requestSha256: hash(JSON.stringify(delivery.request)), initialHumanCommentId: human[0]?.id, initialNativeRunId: run.id });
return { ...connection, sourceId: source.id, faultRequest: request, setupRunIds: ready.runs.map(row => row.id), mailTaskId, actualNormalTaskSendBound, originalDelivery: delivery, publicationId: delivery.id, deliveryIssueId: delivery.issueId, originalRequestSha256: hash(JSON.stringify(delivery.request)), originalConversationId: delivery.conversationId, setupSendBound, actualOriginalUncertain: true, originalUncertaintyRecorded: true, faultOriginalBinding: setupSendBound };
}
export async function cleanupOwnedEmailConnection(ctx: ChatCatalogContext) {
try { const request = faultRequests.get(ctx); if (request) { await releaseOwnedEmailResponseLoss(process.env.PAPERCLIP_RUNNER_E2E_TEMP_ROOT!, request); faultRequests.delete(ctx); } }
finally { const input = privateFixtures.get(ctx); try { if (input?.endpointId) { let receipt: EmailEndpointSummary | undefined; try { receipt = await ctx.input.api.post<EmailEndpointSummary>(`/api/email/inboxes/${input.endpointId}/control`, { action: "remove" }); } catch (error) { if (!(error instanceof RunnerApiHttpError) || error.status !== 404) throw error; } if (receipt?.lastError) throw new Error("Owned email provider registration cleanup is unconfirmed; actual operator verification is required"); } } finally { privateFixtures.delete(ctx); } }
}
export const runtimeEmailRecoveryCases = [runtimeDefinition("recovery.unknown-email-outcome", async ctx => { try { return await setupUnknownEmail(ctx); } catch (error) { await cleanupOwnedEmailConnection(ctx); throw error; } }, gradeUnknownEmail, {
authentication: "human-session", setupHumanMessageCount: 1, expectedHumanMessageCount: 3, expectedRunCount: 4, turnTimeoutMs: 120000, attemptTimeoutMs: 900000,
externalInput: { provider: requirement.provider, fields: requirement.fields },
coverageGaps: [],
setupPolicy: { readiness: "preserve-scenario", invariantCheckIds: ["real-original-email-uncertain-with-no-public-provider-receipt"] },
sourceFiles: ["chat-catalog-domains/runtime-email.ts", "chat-catalog-domains/runtime-email-fault.ts", "chat-catalog-domains/runtime-email-private.ts", "chat-catalog-domains/runtime-email-private-worker.ts", "../../server/src/services/email-channels.ts", "../../server/src/services/agentmail-api.ts", "../../server/src/services/connectors/agentmail.ts", "server-entry.ts", "chat-scenarios/recovery.yaml"],
async verifySetup(ctx, s) { const delivery = await ctx.input.api.get<EmailPublicationSummary>(`/api/companies/${ctx.input.fixtures.company.id}/email/deliveries/${s.publicationId}`), root = process.env.PAPERCLIP_RUNNER_E2E_TEMP_ROOT!; const status = await readOwnedEmailFault<EmailFaultStatus>(root, "status.json"), request = s.faultRequest as EmailFaultRequest; const c = checks(); c.add("real-original-email-uncertain-with-no-public-provider-receipt", delivery.outcome === "uncertain" && delivery.providerMessageId === null && status.realSendsForwarded === 1 && status.nonce === request.nonce && !status.released && !status.expired && s.setupSendBound === true, "One actual accepted original send persists as uncertain without a reliable public receipt; exact declared outage remains active before unchanged canonical query."); return c.result(); },
async act(ctx, s) {
const api = ctx.input.api, companyId = ctx.input.fixtures.company.id, publicationId = String(s.publicationId), request = s.faultRequest as EmailFaultRequest, root = process.env.PAPERCLIP_RUNNER_E2E_TEMP_ROOT!;
const first = await pollUntil({ label: "Canonical uncertainty inspection without an automatic duplicate email", deadlineAt: Date.now() + 180000, load: async () => { await ctx.refreshIssue(); return { source: ctx.issue(), runs: await ctx.allRuns() }; }, accept: v => v.source?.conversationState === "waiting" && v.runs.length === 3 && v.runs.every(row => row.status === "succeeded") });
const canonical = first.runs.find(row => !(s.setupRunIds as string[]).includes(row.id))!, comments = await api.get<Entity[]>(`/api/issues/${ctx.issue()!.id}/comments?order=asc`); s.canonicalReply = comments.filter(row => row.createdByRunId === canonical.id && row.authorAgentId === ctx.input.fixtures.agent.id).map(row => row.body).join("\n");
const events = await collectRunEvents<{ seq?: number; eventType?: string; payload?: unknown }>((afterSeq, limit) => api.get(`/api/heartbeat-runs/${canonical.id}/events?afterSeq=${afterSeq}&limit=${limit}`));
s.canonicalReadBound = actualUncertainEmailReads(events, { companyId, endpointId: request.endpointId, publicationId, issueId: String(s.deliveryIssueId) }).length > 0;
const outage = await readOwnedEmailFault<EmailFaultStatus>(root, "status.json"), original = await api.get<EmailPublicationSummary>(`/api/companies/${companyId}/email/deliveries/${publicationId}`); s.outageActiveDuringCanonical = !outage.released && !outage.expired && original.outcome === "uncertain" && original.providerMessageId === null;
const direct = await providerOperation(ctx, "inspect", s); s.directMatchedSentCount = direct.matchedSent?.length; s.originalProviderReceiptMatched = direct.originalMessageMatched === true;
const observed = await observeRuntime(ctx, s); s.noNewAgentPublication = observed.tasks.length === 2 && observed.tasks.every(row => [s.deliveryIssueId, s.mailTaskId].includes(row.id)) && direct.matchedSent?.every(row => row.publicationId === publicationId) === true;
await releaseOwnedEmailResponseLoss(root, request); s.outageReleased = true;
const resolved = await providerOperation(ctx, "resolve", s); s.reconciliationBound = resolved.originalMessageMatched === true && resolved.matchedSent?.length === 1; s.finalOutcome = resolved.outcome; s.resolutionMode = resolved.resolution;
const activities = await api.get<Entity[]>(`/api/companies/${companyId}/activity?limit=1000`); s.resolutionActorBound = resolved.resolution === "provider-reconciled" || activities.some(row => row.action === "email.resolved" && row.actorType === "user" && row.actorId === s.personId && row.details?.publicationId === publicationId);
const after = await api.get<EmailPublicationSummary>(`/api/companies/${companyId}/email/deliveries/${publicationId}`); s.sameOriginalRequestRetained = hash(JSON.stringify(after.request)) === s.originalRequestSha256; s.sameOriginalConversationRetained = after.conversationId === s.originalConversationId && after.id === publicationId;
const before = (await ctx.allRuns()).map(row => row.id), followup = "The provider's actual receipt for that original email is now available. Please inspect the saved publication and report its real outcome, whether another email was sent, and whether recipient delivery is actually proven. Do not send or retry any email.";
await sendChatMessage(ctx.input.page, followup);
const final = await pollUntil({ label: "Actual reconciled email outcome reported in its own phase", deadlineAt: Date.now() + 180000, load: async () => { await ctx.refreshIssue(); return { source: ctx.issue(), runs: await ctx.allRuns() }; }, accept: v => v.source?.conversationState === "waiting" && v.runs.length === 4 && v.runs.every(row => row.status === "succeeded") });
const replyRun = final.runs.find(row => !before.includes(row.id))!, finalComments = await api.get<Entity[]>(`/api/issues/${ctx.issue()!.id}/comments?order=asc`), human = finalComments.filter(row => !row.authorAgentId && row.body === followup); s.confirmationBound = human.length === 1 && Array.isArray(replyRun.contextSnapshot?.wakeCommentIds) && replyRun.contextSnapshot.wakeCommentIds.length === 1 && replyRun.contextSnapshot.wakeCommentIds[0] === human[0]!.id; s.confirmationReply = finalComments.filter(row => row.createdByRunId === replyRun.id && row.authorAgentId === ctx.input.fixtures.agent.id).map(row => row.body).join("\n");
const last = await providerOperation(ctx, "inspect", s); s.finalDirectMatchedSentCount = last.matchedSent?.length;
(s.providerProbes as unknown[]).push(...[direct, resolved, last].map(row => ({ phase: row.operation, operation: "independent-real-provider-receipts", directProviderProbeCount: row.directProviderProbeCount, publicSetupOperationCount: row.publicSetupOperationCount, status: "verified", usage: "unavailable", cost: "unavailable" })));
},
observe: observeRuntime,
async verifyResult(ctx, s) { const source = ctx.input.page.url(); await ctx.input.page.goto(`/${ctx.input.fixtures.company.issuePrefix}/issues/${s.deliveryIssueId}`, { waitUntil: "domcontentloaded" }); await expect(ctx.input.page.getByText((s.faultRequest as EmailFaultRequest).subject, { exact: true }).first()).toBeVisible(); await ctx.input.capture("original-email-reconciled", "The original task-bound email thread and actual provider outcome", "runtime-email-reconciled.png"); await ctx.input.page.goto(source, { waitUntil: "domcontentloaded" }); },
async cleanup(ctx) { await cleanupOwnedEmailConnection(ctx); },
})];