diff --git a/doc/evals.md b/doc/evals.md index 88fedec15ed..842422bf1c7 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -60,6 +60,12 @@ harness are under `tests/runner-e2e`; the package scripts are `test:e2e:runner`, the server and embedded database, creates the company/agent/task through the real APIs, drives Chromium, and invokes the selected local or Daytona runner. +The explicit-only `agent-chat-hardening` Product E2E suite covers native chat +recovery, hiring, status evidence, and review handoff on local and selected warm +Daytona paths. Its [fixture contract](../tests/runner-e2e/README.md) distinguishes +startup cancellation from active response cancellation and HTTP send replay +from ambiguous provider action recovery. Select it explicitly; `--all` excludes it. + ## Validation ladder Start with credential-free checks and a catalog listing. For Product E2E: diff --git a/tests/runner-e2e/FIXTURES.md b/tests/runner-e2e/FIXTURES.md index 636068a71c8..38680039c1f 100644 --- a/tests/runner-e2e/FIXTURES.md +++ b/tests/runner-e2e/FIXTURES.md @@ -158,9 +158,9 @@ manual or scheduled campaign, not a PR requirement. ## Persistent chat fixtures -`chat-cases.ts` defines the six-case `agent-chat` suite; `chat-flow.ts` drives the +`chat-cases.ts` defines the eight-case `agent-chat` suite; `chat-flow.ts` drives the production composer, plan revision/approval controls, questions, reset command, -and project cards. Keep its 24 local cells intentional. `expectedRunCount` +and project cards. Keep its 28 local cells intentional. `expectedRunCount` counts provider turns, including cancelled and handed-off task runs, but excludes synthetic `/new` runs. Assertions must inspect all company runs because ordinary issue lists exclude the source conversation. `assertChatHandoff` rejects missing @@ -172,3 +172,12 @@ workspaces, task documents, and ordering. They pass through the normal sanitizer Screenshots are allowlisted to the exact disposable agent chat. Cleanup cancels all active runs in the isolated company, including handed-off work; usage from failed and cancelled runs must not disappear from campaign totals. + +`chat-hardening.ts` adds the explicit-only `agent-chat-hardening` journeys. Use +the ordinary public APIs to seed source documents and blockers. Keep the answer +out of the user's status/review request. Grade the exact source values, latest +blocker, preserved task identities, worker-authored output, and real executions. +The lost-acknowledgement probe may interrupt only the fixture browser's own +comment request after the real server has committed it. Retain its request ID +and replay that same request through the public API after restarting the server. +Never fabricate tool results or repair task state after a failed assertion. diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index 05ea24825a5..9c9c1bf879b 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -131,15 +131,15 @@ runner instance, PID, and process-start identity. Each turn is bounded to ten minutes, the cell to thirty minutes, and cleanup explicitly deletes the sandbox rather than waiting for Daytona's idle timeout. -`agent-chat` (**Persistent Agent Chat**) adds six workflows on `legacy-codex`, -`legacy-claude`, `runner-codex`, and `runner-acpx-claude`: **26 local cells**. +`agent-chat` (**Persistent Agent Chat**) has eight workflows on `legacy-codex`, +`legacy-claude`, `runner-codex`, and `runner-acpx-claude`: **28 local cells**. They cover continuity across server restart, fresh context after `/new`, Stop/reset/resume, draft/revise/approve/plan handoff, clarification with existing project reuse, and a new project with two repository URLs. Each cell opens the production chat surface and resolves the backing issue through the chat API. The source conversation must settle to `in_review` / `waiting`; handed-off execution tasks must finish with their initial Plan and output documents. -Reset runs are retained separately from the 72 expected provider turns in this +Reset runs are retained separately from the 76 expected run attempts in this suite. Cancelled turns and execution-task runs remain included in billing and cleanup. The production chat directive is injected normally; fixtures do not replace it with completion instructions. Daytona is excluded. @@ -151,6 +151,40 @@ and backlog tasks. The oracle verifies stable task IDs, preserved descriptions, assignment audit evidence, exactly one successful successor run and its output document, no backlog execution, and a usable source conversation after reload. +`agent-chat-hardening` is an explicit-only native Codex/Claude suite with +**18 cells: 12 local and six warm Daytona**. It adds startup Stop/reset, +hire/delegate/reuse, grounded blocker reporting with source-document review, +and committed-send retry. It also runs active Stop/reset and restart continuity. +Daytona selects active Stop/reset, restart continuity, and committed-send retry. +The 56 expected run attempts include cancelled attempts; synthetic resets are +recorded separately. The suite uses production permission defaults and prompts. +Only hiring and cross-task status/review enable the opt-in native API tools. +Hiring uses a personal managed AI connection and verifies the hired worker's +actual execution account. This is not an onboarding-default qualification. + +Stop during startup and Stop during an active response are separate boundaries. +The native active-response case requires a recorded provider turn start; generic +lifecycle/performance events are insufficient. The startup case must stop after +a process launch request but before a provider turn starts. Missing the boundary +fails the case instead of silently testing another phase. + +The committed-send case withholds the browser's comment acknowledgement until +the agent has saved one backlog task, drops that response, restarts Paperclip, +and replays the exact public request with the original client request ID. It +requires the original comment, task, plan, and single consuming run. This proves +HTTP request idempotency across restart, not replay safety for an ambiguous +provider tool response. Existing native tool-receipt tests cover that boundary. + +```sh +pnpm test:e2e:runner -- --list --suite agent-chat-hardening +pnpm test:e2e:runner -- --id agent-chat-hardening.runner-codex.local.stop-startup-new-resume +``` + +Each hardening oracle has positive and plausible-negative calibration tests. +The review grader parses the worker's saved JSON and compares both source values +and the consistency verdict. Hiring requires one identity, correct reporting +line, managed credentials, and real task execution; chat claims cannot pass it. + ```bash # Run these after deterministic checks, with the required provider keys set. pnpm test:e2e:runner -- --id agent-chat.legacy-codex.local.continuity-restart @@ -176,8 +210,9 @@ throwaway instance; never point the authenticated suite at the running demo. Missing provider credentials fail paid preflight and are not passing coverage. The default `--all` selection is 171 cells (148 local and 23 Daytona) and 371 -expected paid agent turns. The explicit-only everyday suite adds 35 catalog cells -and is excluded from `--all`. Follow-up steps remain ordered within their cell; all other +expected paid agent turns. The explicit-only everyday suite adds 38 catalog cells +and chat hardening adds 18. Both are excluded from `--all`. The full catalog has 227 cells. +Follow-up steps remain ordered within their cell; all other cells are independent. Narrow selectors are strongly recommended while developing fixtures. diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index 955368c0ad6..f04f0a21d9d 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -57,10 +57,10 @@ describe("runner E2E catalog", () => { expect(localIntegrityTasks).toHaveLength(2); expect(openRouterBreadthTasks).toHaveLength(3); expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([ - 23, 38, 52, 28, 42, 14, 10, 2, + 23, 38, 52, 28, 18, 42, 14, 10, 2, ]); - expect(validateRunnerCatalog()).toHaveLength(209); - expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(209); + expect(validateRunnerCatalog()).toHaveLength(227); + expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(227); expect( runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"), ).toHaveLength(42); diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index c19dd0a8e37..3732cb22a9d 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -2,7 +2,7 @@ import { continuationTasks } from "./continuation-cases.js"; import { everydayTasks, productionStoryProfile } from "./everyday-cases.js"; import { firstTaskTasks } from "./first-task-cases.js"; -import { chatTasks } from "./chat-cases.js"; +import { chatTasks, chatHardeningTasks } from "./chat-cases.js"; import { createHash } from "node:crypto"; import { createAgentSchema } from "../../packages/shared/src/validators/agent.js"; import { createEnvironmentSchema } from "../../packages/shared/src/validators/environment.js"; @@ -944,7 +944,17 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ profiles: runnerProfiles.filter(profile => ["legacy-codex", "legacy-claude", "runner-codex", "runner-acpx-claude"].includes(profile.id)).map(defaultPermissionProfile), environments: [localEnvironment], tasks: chatTasks, expectedMatrixSize: 28, excludedExecutionIds: ["legacy-codex", "legacy-claude"].flatMap(profile => ["reassign-task", "create-backlog"].map(task => `agent-chat.${profile}.local.${task}`)), - definitionMetadata: { version: 4, resetRunsCountedSeparately: true, permissions: "production-defaults" }, + definitionMetadata: { version: 5, resetRunsCountedSeparately: true, permissions: "production-defaults", stopBoundary: "provider-turn-started" }, + }, + { + id: "agent-chat-hardening", label: "Agent Chat Recovery and Coordination", manualOnly: true, + description: "Native chat startup cancellation, committed sends, hiring, grounded status, and remote continuity.", + groups: ["chat", "native"], + profiles: runnerProfiles.filter(profile => ["runner-codex", "runner-acpx-claude"].includes(profile.id)).map(defaultPermissionProfile), + environments: [localEnvironment, daytonaWarmEnvironment], tasks: chatHardeningTasks, expectedMatrixSize: 18, + excludedExecutionIds: ["runner-codex", "runner-acpx-claude"].flatMap(profile => + ["stop-startup-new-resume", "hire-delegate-reuse", "blocked-status-review"].map(task => `agent-chat-hardening.${profile}.daytona.${task}`)), + definitionMetadata: { version: 1, permissions: "production-defaults", grading: "durable-state-and-source-evidence", scheduling: "explicit-only" }, }, ...(process.env.PAPERCLIP_RUNNER_E2E_CONNECTION_REVIEWS === "1" ? [connectionReviewSuite] : []), { diff --git a/tests/runner-e2e/chat-cases.ts b/tests/runner-e2e/chat-cases.ts index a8f2cd31d70..c0e59620e08 100644 --- a/tests/runner-e2e/chat-cases.ts +++ b/tests/runner-e2e/chat-cases.ts @@ -20,7 +20,15 @@ export const CHAT_CASES = [ ["multi-repository", "Create a project with multiple repository URLs", 2], ] as const; export type ChatCase = (typeof CHAT_CASES)[number][0]; -export const chatTasks: readonly RunnerTaskFixture[] = CHAT_CASES.map( +const HARDENING_CASES = [ + ["stop-startup-new-resume", "Stop during startup, reset, and resume", 3], + ["hire-delegate-reuse", "Hire through chat, delegate, and reuse the same teammate", 5], + ["blocked-status-review", "Read the actual blocker and hand source material to a reviewer", 4], + ["committed-send-retry", "Recover a lost send acknowledgement without repeating committed work", 2], + ...CHAT_CASES.filter(([id]) => ["stop-new-resume", "continuity-restart"].includes(id)), +] as const; +function buildChatTasks(definitions: readonly (readonly [string, string, number])[]): readonly RunnerTaskFixture[] { + return definitions.map( ([id, label, expectedRunCount]) => ({ id, label, @@ -35,4 +43,15 @@ export const chatTasks: readonly RunnerTaskFixture[] = CHAT_CASES.map( buildVisibleMarker: (nonce) => chatMarker("CHAT", nonce), buildMatchers: () => [{ kind: "issue_status", expected: "in_review" }], }), -); + ); +} +export const chatTasks = buildChatTasks(CHAT_CASES); +export const chatHardeningTasks = buildChatTasks(HARDENING_CASES); + +export function chatNeedsApiTools(suiteId: string, caseId: string): boolean { + return suiteId === "agent-chat-hardening" && ["hire-delegate-reuse", "blocked-status-review"].includes(caseId); +} +export function isManagedHiringCase(suiteId: string, caseId: string): boolean { + return (suiteId === "everyday-workflows" && caseId === "hire-reuse") || + (suiteId === "agent-chat-hardening" && caseId === "hire-delegate-reuse"); +} diff --git a/tests/runner-e2e/chat-flow.ts b/tests/runner-e2e/chat-flow.ts index 24d5aa9233c..a5544ad6f87 100644 --- a/tests/runner-e2e/chat-flow.ts +++ b/tests/runner-e2e/chat-flow.ts @@ -8,6 +8,7 @@ import type { LiveFixtureValues } from "./live-fixtures.js"; import type { MatrixExecution } from "./types.js"; import { isBlockedUnstartedWake } from "./non-execution-wake.js"; import { chatMarker } from "./chat-cases.js"; +import { isChatStopReady, runChatHardeningFlow } from "./chat-hardening.js"; // Public API observations only: this driver never fabricates provider results or writes DB state. export interface ChatIssue { @@ -285,7 +286,7 @@ export function assertChatReassignment(input: { expect(input.outputBody).toContain(input.marker); } -export async function runChatFlow(input: { +export interface ChatFlowInput { page: Page; api: RunnerApi; fixtures: LiveFixtureValues; @@ -296,13 +297,15 @@ export async function runChatFlow(input: { observe: (issue: ChatIssue, runs: ChatRun[]) => void; capture: (id: string, label: string, file: string) => Promise; evidence: (name: string, data: unknown) => Promise; -}) { +} +export async function runChatFlow(input: ChatFlowInput) { const { page, api, fixtures: f, execution, nonce } = input; const chatPath = `/api/companies/${f.company.id}/chats/${f.agent.id}`; const route = `/${f.company.issuePrefix}/chats/${f.agent.id}`; const marker = execution.task.buildVisibleMarker(nonce); const draftMarker = chatMarker("DRAFT", nonce); const caseId = execution.task.id; + const stopCase = ["stop-new-resume", "stop-startup-new-resume"].includes(caseId); let issue: ChatIssue; let runs: ChatRun[] = []; const settings = await api.get>( @@ -346,7 +349,7 @@ export async function runChatFlow(input: { activeRuns: runs .filter((run) => ["queued", "running"].includes(run.status)) .map((run) => run.id), - failure: chatRunFailure(runs, caseId === "stop-new-resume"), + failure: chatRunFailure(runs, stopCase), }; }, reject: (state) => state.failure ?? inconsistentIdle(state), @@ -378,7 +381,7 @@ export async function runChatFlow(input: { expect(await allRuns()).toHaveLength(0); if ( - ["continuity-restart", "new-session", "stop-new-resume"].includes(caseId) + ["continuity-restart", "new-session", "stop-new-resume", "stop-startup-new-resume"].includes(caseId) ) { const secret = chatMarker("OLDCONTEXT", nonce); await turn( @@ -411,7 +414,7 @@ export async function runChatFlow(input: { ); } else { let cancelledId: string | undefined; - if (caseId === "stop-new-resume") { + if (stopCase) { await sendChatMessage( page, "Explain the history of gardening at length here, in 100 numbered paragraphs. This is discussion only; do not create work.", @@ -427,9 +430,14 @@ export async function runChatFlow(input: { const events = await api.get>>( `/api/heartbeat-runs/${active.id}/events?limit=1000`, ); - const log = await readRunningChatLog(api, active.id); - if (!(events.length || log?.length)) return false; + if (execution.profile.generation === "native") { + if (!isChatStopReady(events, caseId === "stop-startup-new-resume" ? "startup" : "active")) return false; + } else if (!(events.length || (await readRunningChatLog(api, active.id))?.length)) return false; cancelledId = active.id; + await input.evidence("chat-stop-boundary.json", { + phase: caseId === "stop-startup-new-resume" ? "startup" : "active", + runId: active.id, events, + }); return true; }, { timeout: 120_000 }, @@ -441,6 +449,7 @@ export async function runChatFlow(input: { async () => (await api.get(`/api/heartbeat-runs/${cancelledId}`)) .status, + { timeout: 30_000 }, ) .toBe("cancelled"); } @@ -502,6 +511,8 @@ export async function runChatFlow(input: { } expect(issue!.id).toBe(initialId); await noTasks(); + } else if (["hire-delegate-reuse", "blocked-status-review", "committed-send-retry"].includes(caseId)) { + await runChatHardeningFlow({ input, marker, issue: () => issue!, turn, idle, tasks, allRuns, comments }); } else if (caseId === "create-backlog") { const project = await api.post<{ id: string }>(`/api/companies/${f.company.id}/projects`, { name: `Later planning ${nonce}`, description: "Repository-free plans to save for later.", @@ -894,7 +905,7 @@ export async function runChatFlow(input: { for (const run of runs.filter((run) => !isResetRun(run))) { expect(run.runtimeMode).toBe(execution.profile.expectedRuntimeMode); expect(run.status).toBe( - caseId === "stop-new-resume" && run.status === "cancelled" + stopCase && run.status === "cancelled" ? "cancelled" : "succeeded", ); diff --git a/tests/runner-e2e/chat-hardening.test.ts b/tests/runner-e2e/chat-hardening.test.ts new file mode 100644 index 00000000000..934e39efabe --- /dev/null +++ b/tests/runner-e2e/chat-hardening.test.ts @@ -0,0 +1,79 @@ +import { describe, expect, it } from "vitest"; +import { assertChatHire, assertChatSourceReview, assertCommittedSendRetry, assertGroundedChatStatus, isChatStopReady } from "./chat-hardening.js"; +import { runnerMatrix } from "./catalog.js"; +import { buildRunnerE2EProcessEnvironment } from "./harness-env.js"; + +const binding = { provider: "anthropic", method: "api_key", mode: "responsible_user" }; +const hire = { id: "hire", name: "Morgan", reportsTo: "lead", adapterType: "paperclip_runner", adapterConfig: { model: "model" }, runtimeConfig: { aiConnection: binding } }; +const task = { id: "task", companyId: "company", title: "Checklist", status: "done", assigneeAgentId: "hire", parentId: null, projectId: "project" }; +const run = { id: "run", companyId: "company", agentId: "hire", status: "succeeded", runtimeMode: "native", contextSnapshot: { issueId: "task", aiConnection: { connectionId: "account" } } }; +const hiring = { agents: [{ id: "lead", name: "Lead", adapterConfig: { model: "model" } }, hire], leadId: "lead", hireName: "Morgan", hiredId: "hire", connectionId: "account", binding, taskIds: ["task"], tasks: [task], runs: [run] }; + +describe("agent chat hardening oracles", () => { + it("grades actual execution by the original hired identity and managed account", () => { + expect(() => assertChatHire(hiring)).not.toThrow(); + for (const wrong of [ + { ...hiring, agents: [...hiring.agents, { ...hire, id: "duplicate" }] }, + { ...hiring, hiredId: "replaced" }, + { ...hiring, tasks: [{ ...task, parentId: "chat" }] }, + { ...hiring, tasks: [{ ...task, assigneeAgentId: "lead" }] }, + { ...hiring, runs: [] }, + { ...hiring, runs: [run, { ...run, id: "duplicate-run" }] }, + { ...hiring, runs: [{ ...run, agentId: "lead" }] }, + { ...hiring, connectionId: "wrong-account" }, + ]) expect(() => assertChatHire(wrong)).toThrow(); + }); + + it("requires the current recorded blocker without silently starting or changing work", () => { + const before = { ...task, status: "blocked" }; + const valid = { reply: "RUN-2 is blocked on VENUE123.", expectedIssueIdentifier: "RUN-2", blocker: "VENUE123", staleBlocker: "BUDGET123", + before, after: before, taskIdsBefore: ["task"], taskIdsAfter: ["task"], taskRuns: [] }; + expect(() => assertGroundedChatStatus(valid)).not.toThrow(); + expect(() => assertGroundedChatStatus({ ...valid, reply: "RUN-2 is blocked on BUDGET123." })).toThrow(); + expect(() => assertGroundedChatStatus({ ...valid, after: task })).toThrow(); + expect(() => assertGroundedChatStatus({ ...valid, taskRuns: [run] })).toThrow(); + expect(() => assertGroundedChatStatus({ ...valid, taskIdsAfter: ["task", "replacement"] })).toThrow(); + }); + + it("independently compares review values and verdict, rejecting copied sources and missing evidence", () => { + const expected = { planLaunchDay: "Tuesday", briefLaunchDay: "Wednesday" }; + expect(() => assertChatSourceReview(JSON.stringify({ ...expected, consistent: false }), expected)).not.toThrow(); + expect(() => assertChatSourceReview(JSON.stringify({ ...expected, consistent: true }), expected)).toThrow(); + expect(() => assertChatSourceReview("The plan says Tuesday and the brief says Wednesday.", expected)).toThrow(); + expect(() => assertChatSourceReview("{}", expected)).toThrow(); + expect(() => assertChatSourceReview(JSON.stringify({ ...expected, planLaunchDay: "Friday", consistent: false }), expected)).toThrow(); + expect(() => assertChatSourceReview('{"planLaunchDay":"Tuesday","briefLaunchDay":"Tuesday","consistent":true}', { planLaunchDay: "Tuesday", briefLaunchDay: "Tuesday" })).not.toThrow(); + }); + + it("does not confuse lifecycle logs, provider startup, and an active turn", () => { + expect(isChatStopReady([{ eventType: "lifecycle" }, { eventType: "run.performance.span" }], "active")).toBe(false); + const startup = [{ eventType: "native.process_start_requested" }]; + expect(isChatStopReady(startup, "startup")).toBe(true); + expect(isChatStopReady(startup, "active")).toBe(false); + const active = [...startup, { eventType: "turn.started" }]; + expect(isChatStopReady(active, "startup")).toBe(false); + expect(isChatStopReady(active, "active")).toBe(true); + }); + + it("requires one committed comment, one task, and one consuming run after request replay", () => { + const valid = { commentId: "comment", clientRequestId: "request", comments: [{ id: "comment", body: "create", clientRequestId: "request" }], taskId: "task", + tasks: [{ ...task, status: "backlog" }], chatId: "chat", runs: [{ ...run, contextSnapshot: { issueId: "chat", wakeCommentId: "comment" } }] }; + expect(() => assertCommittedSendRetry(valid)).not.toThrow(); + expect(() => assertCommittedSendRetry({ ...valid, comments: [...valid.comments, { ...valid.comments[0]!, id: "duplicate" }] })).toThrow(); + expect(() => assertCommittedSendRetry({ ...valid, tasks: [...valid.tasks, { ...task, id: "duplicate" }] })).toThrow(); + expect(() => assertCommittedSendRetry({ ...valid, runs: [...valid.runs, run] })).toThrow(); + expect(() => assertCommittedSendRetry({ ...valid, runs: [...valid.runs, { ...valid.runs[0]!, id: "duplicate" }] })).toThrow(); + expect(() => assertCommittedSendRetry({ ...valid, runs: [] })).toThrow(); + }); + + it("keeps the paid hardening matrix explicit and limits API-tool opt-in to coordination", () => { + const cells = runnerMatrix.filter(cell => cell.suite.id === "agent-chat-hardening"); + expect(cells).toHaveLength(18); + expect(cells.every(cell => cell.suite.manualOnly && cell.profile.generation === "native")).toBe(true); + expect(cells.filter(cell => cell.environment.id === "daytona")).toHaveLength(6); + for (const cell of cells) { + expect(buildRunnerE2EProcessEnvironment({}, [cell]).PAPERCLIP_RUNNER_API_TOOLS_ENABLED).toBe( + ["hire-delegate-reuse", "blocked-status-review"].includes(cell.task.id) ? "true" : undefined); + } + }); +}); diff --git a/tests/runner-e2e/chat-hardening.ts b/tests/runner-e2e/chat-hardening.ts new file mode 100644 index 00000000000..ff7b16e138a --- /dev/null +++ b/tests/runner-e2e/chat-hardening.ts @@ -0,0 +1,199 @@ +import { expect, type Route } from "@playwright/test"; +import type { RunnerApi } from "./api.js"; +import { readChatOutputDocument, sendChatMessage, type ChatFlowInput, type ChatIssue, type ChatRun } from "./chat-flow.js"; + +type Agent = { + id: string; name: string; reportsTo?: string; adapterType?: string; + adapterConfig?: Record; runtimeConfig?: Record; +}; +type Comment = { id: string; body: string; authorAgentId?: string; clientRequestId?: string }; +type Document = { id: string; body: string; latestRevisionId: string; createdByAgentId?: string }; + +export function assertChatHire(input: { + agents: Agent[]; leadId: string; hireName: string; hiredId?: string; + connectionId: string; binding: unknown; taskIds: string[]; tasks: ChatIssue[]; runs: ChatRun[]; +}) { + const hires = input.agents.filter(agent => agent.name === input.hireName); + expect(hires).toHaveLength(1); + const hired = hires[0]!; + const lead = input.agents.find(agent => agent.id === input.leadId)!; + expect(input.connectionId).toBeTruthy(); + expect(hired).toMatchObject({ reportsTo: input.leadId, adapterType: "paperclip_runner" }); + if (input.hiredId) expect(hired.id).toBe(input.hiredId); + expect(hired.adapterConfig?.model).toBe(lead.adapterConfig?.model); + expect(hired.runtimeConfig?.aiConnection).toEqual(input.binding); + expect(input.tasks.map(task => task.id).sort()).toEqual([...input.taskIds].sort()); + for (const id of input.taskIds) { + const task = input.tasks.find(task => task.id === id)!; + expect(task).toMatchObject({ status: "done", assigneeAgentId: hired.id, parentId: null }); + expect(task.projectId).toBeTruthy(); + const runs = input.runs.filter(run => run.contextSnapshot?.issueId === id); + expect(runs).toHaveLength(1); + expect(runs[0]).toMatchObject({ agentId: hired.id, status: "succeeded", runtimeMode: "native" }); + expect(runs[0]!.contextSnapshot?.aiConnection).toMatchObject({ connectionId: input.connectionId }); + } + return hired; +} + +export function assertGroundedChatStatus(input: { + reply: string; expectedIssueIdentifier: string; blocker: string; staleBlocker: string; + before: ChatIssue; after: ChatIssue; taskIdsBefore: string[]; taskIdsAfter: string[]; taskRuns: ChatRun[]; +}) { + expect(input.reply).toContain(input.expectedIssueIdentifier); + expect(input.reply).toMatch(/blocked/i); + expect(input.reply).toContain(input.blocker); + expect(input.reply).not.toContain(input.staleBlocker); + expect(input.after).toMatchObject({ id: input.before.id, status: "blocked", assigneeAgentId: input.before.assigneeAgentId }); + expect([...input.taskIdsAfter].sort()).toEqual([...input.taskIdsBefore].sort()); + expect(input.taskRuns).toHaveLength(0); +} + +export function assertChatSourceReview(body: string, expected: { planLaunchDay: string; briefLaunchDay: string }) { + // The user requests JSON, so an independent parser grades the actual saved + // deliverable. A claim in the chat or a copied source document cannot pass. + const json = body.trim().replace(/^```(?:json)?\s*/, "").replace(/\s*```$/, ""); + expect(JSON.parse(json)).toMatchObject({ ...expected, consistent: expected.planLaunchDay === expected.briefLaunchDay }); +} + +export function assertCommittedSendRetry(input: { + commentId: string; clientRequestId: string; comments: Comment[]; taskId: string; + tasks: ChatIssue[]; runs: ChatRun[]; chatId: string; +}) { + expect(input.comments.filter(comment => comment.clientRequestId === input.clientRequestId).map(comment => comment.id)).toEqual([input.commentId]); + expect(input.tasks.map(task => task.id)).toEqual([input.taskId]); + expect(input.tasks[0]).toMatchObject({ status: "backlog", parentId: null }); + expect(input.runs.filter(run => run.contextSnapshot?.issueId === input.taskId)).toHaveLength(0); + const original = input.runs.filter(run => run.contextSnapshot?.wakeCommentId === input.commentId || + (Array.isArray(run.contextSnapshot?.wakeCommentIds) && run.contextSnapshot.wakeCommentIds.includes(input.commentId))); + expect(original).toHaveLength(1); + expect(original[0]).toMatchObject({ status: "succeeded", runtimeMode: "native" }); + expect(original[0]!.contextSnapshot?.issueId).toBe(input.chatId); +} + +export function isChatStopReady(events: Array<{ eventType?: unknown }>, phase: "startup" | "active"): boolean { + const active = events.some(event => ["turn.started", "turn.accepted"].includes(String(event.eventType))); + return phase === "active" ? active : !active && events.some(event => event.eventType === "native.process_start_requested"); +} + +export async function runChatHardeningFlow(context: { + input: ChatFlowInput; marker: string; issue(): ChatIssue; + turn(message: string, count: number): Promise; idle(count: number): Promise; + tasks(): Promise; allRuns(): Promise; comments(): Promise; +}) { + const { input, marker, turn, idle, tasks, allRuns, comments } = context; + const { api, page, fixtures: f, execution, nonce } = input; + const companyPath = `/api/companies/${f.company.id}`; + const project = await api.post<{ id: string; name: string }>(`${companyPath}/projects`, { + name: `Chat coordination ${nonce}`, description: "Repository-free launch planning and review.", + }); + const agents = () => api.get(`${companyPath}/agents`); + const latestReply = async () => (await comments()).filter(comment => comment.authorAgentId === f.agent.id).at(-1)?.body ?? ""; + const output = (id: string, token: string) => readChatOutputDocument(api, id, token); + + if (execution.task.id === "hire-delegate-reuse") { + const hireName = `Morgan Reviewer ${nonce}`; + await turn(`Hire exactly one teammate named ${hireName}, reporting to you, using your native runner, model, and available AI connection. Have that teammate write a concise launch checklist as a saved Paperclip document containing ${marker}. Put the work in one assigned task in ${project.name}. Link it here and let the teammate complete it.`, 2); + const first = (await tasks())[0]!; + expect(first).toBeTruthy(); + const firstOutput = await output(first.id, marker); + const account = f.aiConnection!; + const hired = assertChatHire({ agents: await agents(), leadId: f.agent.id, hireName, + connectionId: account.connectionId, binding: account.binding, taskIds: [first.id], tasks: await tasks(), runs: await allRuns() }); + expect((firstOutput as Document).createdByAgentId).toBe(hired.id); + await turn(`Have the existing ${hireName} review the checklist on ${first.identifier} and write a separate saved review document containing REVIEW${marker}. Create one review task in ${project.name}, assigned to that same teammate. Include the actual checklist in the handoff so they can review it. Preserve the original checklist and task.`, 4); + const observed = await tasks(); + const second = observed.find(task => task.id !== first.id)!; + expect(second).toBeTruthy(); + const reviewOutput = await output(second.id, `REVIEW${marker}`); + expect((reviewOutput as Document).createdByAgentId).toBe(hired.id); + expect(await api.get(`/api/issues/${first.id}/documents/${encodeURIComponent(firstOutput.key)}`)).toEqual(firstOutput); + await turn(`Give me a brief status update on ${first.identifier} and ${second.identifier}: identify their owner and recorded status. Do not create or change work.`, 5); + assertChatHire({ agents: await agents(), leadId: f.agent.id, hireName, hiredId: hired.id, + connectionId: account.connectionId, binding: account.binding, taskIds: [first.id, second.id], tasks: await tasks(), runs: await allRuns() }); + const reply = await latestReply(); + expect(reply).toContain(first.identifier!); expect(reply).toContain(second.identifier!); + expect(reply).toContain(hireName); expect(reply).toMatch(/done|completed/i); + await input.evidence("chat-hire-reuse.json", { agents: await agents(), tasks: await tasks(), firstOutput, reviewOutput, runs: await allRuns() }); + } else if (execution.task.id === "blocked-status-review") { + const blocker = `VENUE${marker}`, staleBlocker = `BUDGET${marker}`; + const source = await api.post(`${companyPath}/issues`, { + title: `Launch brief ${nonce}`, status: "blocked", projectId: project.id, + description: "The launch brief is waiting for its recorded blocker to be resolved.", + initialPlan: "The approved launch day is Tuesday.", + }); + const docResponse = await api.request.put(`/api/issues/${source.id}/documents/brief`, { + data: { title: "Launch brief", format: "markdown", body: "The launch brief schedules the launch for Wednesday." }, + }); + expect(docResponse.ok()).toBe(true); + const sourcePlan = await api.get(`/api/issues/${source.id}/documents/plan`); + const sourceBrief = await api.get(`/api/issues/${source.id}/documents/brief`); + await api.post(`/api/issues/${source.id}/comments`, { body: `Earlier blocker: ${staleBlocker}. Waiting for the budget.` }); + await api.post(`/api/issues/${source.id}/comments`, { body: `Budget is resolved. Current blocker: ${blocker}. Waiting for the venue confirmation. Keep this task blocked; no execution is active.` }); + await turn(`What is the actual current blocker on ${source.identifier}? Read the latest recorded explanation, include its exact blocker label, and distinguish task status from active execution. Just report; do not change it or create work.`, 1); + assertGroundedChatStatus({ reply: await latestReply(), expectedIssueIdentifier: source.identifier!, blocker, staleBlocker, + before: source, after: await api.get(`/api/issues/${source.id}`), taskIdsBefore: [source.id], + taskIdsAfter: (await tasks()).map(task => task.id), taskRuns: (await allRuns()).filter(run => run.contextSnapshot?.issueId === source.id) }); + const config = execution.profile.buildAgent({ environmentId: f.environment.id, environmentFixtureId: execution.environment.id, + workspacePath: input.workspacePath, secretRefs: f.secretRefs, executionId: nonce }); + const reviewer = await api.post(`${companyPath}/agents`, { ...config, name: `Riley Reviewer ${nonce}`, role: "engineer", reportsTo: f.agent.id }); + await turn(`Have ${reviewer.name} review the saved brief on ${source.identifier} against that task's saved plan, in one separate task in ${project.name}. Give the reviewer both source documents. They should save a JSON review document with planLaunchDay, briefLaunchDay, and a boolean consistent, then complete the review task. Preserve the source task and both documents.`, 3); + const review = (await tasks()).find(task => task.id !== source.id)!; + expect(review).toMatchObject({ status: "done", assigneeAgentId: reviewer.id, parentId: null, projectId: project.id }); + const document = await output(review.id, "planLaunchDay"); + assertChatSourceReview(document.body, { planLaunchDay: "Tuesday", briefLaunchDay: "Wednesday" }); + expect((document as Document).createdByAgentId).toBe(reviewer.id); + const reviewRuns = (await allRuns()).filter(run => run.contextSnapshot?.issueId === review.id); + expect(reviewRuns).toHaveLength(1); + expect(reviewRuns[0]).toMatchObject({ agentId: reviewer.id, status: "succeeded", runtimeMode: "native" }); + await turn(`Summarize ${review.identifier}'s recorded review result and tell me whether ${source.identifier} is still blocked. Read the saved evidence; do not change or create work.`, 4); + const reply = await latestReply(); + expect(reply).toContain("Tuesday"); expect(reply).toContain("Wednesday"); expect(reply).toContain(source.identifier!); expect(reply).toMatch(/blocked/i); + expect((await tasks()).map(task => task.id).sort()).toEqual([source.id, review.id].sort()); + expect(await api.get(`/api/issues/${source.id}/documents/plan`)).toEqual(sourcePlan); + expect(await api.get(`/api/issues/${source.id}/documents/brief`)).toEqual(sourceBrief); + expect(await api.get(`/api/issues/${source.id}`)).toMatchObject({ status: "blocked" }); + await input.evidence("chat-status-review.json", { source, sourcePlan, sourceBrief, review, document, reviewRuns, reply }); + } else if (execution.task.id === "committed-send-retry") { + let sent: { path: string; data: Record; commentId: string } | undefined; + let abortResponse!: () => void; + const responseMayDrop = new Promise(resolve => { abortResponse = resolve; }); + const handler = async (route: Route) => { + if (route.request().method() !== "POST") return route.continue(); + const data = route.request().postDataJSON() as Record; + if (typeof data.body !== "string" || !data.body.includes(marker) || sent) return route.continue(); + const response = await route.fetch(); + expect(response.ok()).toBe(true); + const comment = await response.json() as { id: string }; + sent = { path: new URL(route.request().url()).pathname, data, commentId: comment.id }; + await responseMayDrop; + await route.abort("connectionreset"); + }; + await page.route("**/api/issues/*/comments", handler); + try { + await sendChatMessage(page, `Create exactly one task titled Later ${nonce} in ${project.name}, assigned to yourself, in backlog. Save an initial plan containing ${marker}. Do not start it. Link the task here.`); + await idle(1); // The server and real agent complete while the browser acknowledgement is withheld. + expect(sent).toBeTruthy(); + const saved = (await tasks())[0]!; + expect(saved).toBeTruthy(); + const plan = await api.get(`/api/issues/${saved.id}/documents/plan`); + expect(plan.body).toContain(marker); + abortResponse(); + await page.unroute("**/api/issues/*/comments", handler); + await input.restart(); + // Replay the exact interrupted public request, including its original key. + // This tests transport retries, not automatic replay of a failed provider turn. + const retried = await api.post(sent!.path, sent!.data); + expect(retried.id).toBe(sent!.commentId); + await page.goto(`/${f.company.issuePrefix}/chats/${f.agent.id}`, { waitUntil: "domcontentloaded" }); + await turn(`What is the saved status of ${saved.identifier}? Just report it; do not create or change work.`, 2); + assertCommittedSendRetry({ commentId: sent!.commentId, clientRequestId: String(sent!.data.clientRequestId), + comments: await comments(), taskId: saved.id, tasks: await tasks(), runs: await allRuns(), chatId: context.issue().id }); + expect(await api.get(`/api/issues/${saved.id}/documents/plan`)).toEqual(plan); + expect(await latestReply()).toMatch(/backlog/i); + await input.evidence("chat-committed-send-retry.json", { commentId: sent!.commentId, task: saved, plan, runs: await allRuns() }); + } finally { + abortResponse(); + await page.unroute("**/api/issues/*/comments", handler); + } + } else throw new Error(`Unsupported chat hardening case ${execution.task.id}`); +} diff --git a/tests/runner-e2e/harness-env.ts b/tests/runner-e2e/harness-env.ts index d1e9cede374..6539f905b81 100644 --- a/tests/runner-e2e/harness-env.ts +++ b/tests/runner-e2e/harness-env.ts @@ -1,4 +1,5 @@ import path from "node:path"; +import { chatNeedsApiTools, isManagedHiringCase } from "./chat-cases.js"; import { CREDENTIAL_NAMES } from "./types.js"; import type { MatrixExecution } from "./types.js"; @@ -109,7 +110,7 @@ export function buildRunnerE2EProcessEnvironment( delete result.OPENCODE_ALLOW_ALL_MODELS; // Hiring needs the opt-in native API surface. Scope this to the explicit // manual hiring story; production and other suites retain their defaults. - if (executions.some((e) => e.suite.id === "everyday-workflows" && e.task.id === "hire-reuse")) { + if (executions.some((e) => isManagedHiringCase(e.suite.id, e.task.id) || chatNeedsApiTools(e.suite.id, e.task.id))) { result.PAPERCLIP_RUNNER_API_TOOLS_ENABLED = "true"; } if ( diff --git a/tests/runner-e2e/live-fixtures.ts b/tests/runner-e2e/live-fixtures.ts index c7532a54a3c..edb960b851b 100644 --- a/tests/runner-e2e/live-fixtures.ts +++ b/tests/runner-e2e/live-fixtures.ts @@ -1,4 +1,5 @@ import path from "node:path"; +import { isManagedHiringCase } from "./chat-cases.js"; import { FixtureRegistry } from "./fixture-registry.js"; import type { RunnerApi } from "./api.js"; import type { @@ -212,9 +213,7 @@ export async function setupLiveFixtures(input: { }, }); - const managedHiring = - execution.suite.id === "everyday-workflows" && - execution.task.id === "hire-reuse"; + const managedHiring = isManagedHiringCase(execution.suite.id, execution.task.id); if (managedHiring) { registry.register({ id: "ai-connection",