mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-25 02:26:12 +02:00
test: add native agent chat coordination and recovery evals
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed
1 parent
29d6b35096
commit
3d6140ca5d
11 files changed
+394
-26
No files matched your search
@@ -60,6 +60,12 @@ harness are under `tests/runner-e2e`; the package scripts are `test:e2e:runner`,
|
||||
the server and embedded database, creates the company/agent/task through the
|
||||
real APIs, drives Chromium, and invokes the selected local or Daytona runner.
|
||||
|
||||
The explicit-only `agent-chat-hardening` Product E2E suite covers native chat
|
||||
recovery, hiring, status evidence, and review handoff on local and selected warm
|
||||
Daytona paths. Its [fixture contract](../tests/runner-e2e/README.md) distinguishes
|
||||
startup cancellation from active response cancellation and HTTP send replay
|
||||
from ambiguous provider action recovery. Select it explicitly; `--all` excludes it.
|
||||
|
||||
## Validation ladder
|
||||
|
||||
Start with credential-free checks and a catalog listing. For Product E2E:
|
||||
|
||||
@@ -158,9 +158,9 @@ manual or scheduled campaign, not a PR requirement.
|
||||
|
||||
## Persistent chat fixtures
|
||||
|
||||
`chat-cases.ts` defines the six-case `agent-chat` suite; `chat-flow.ts` drives the
|
||||
`chat-cases.ts` defines the eight-case `agent-chat` suite; `chat-flow.ts` drives the
|
||||
production composer, plan revision/approval controls, questions, reset command,
|
||||
and project cards. Keep its 24 local cells intentional. `expectedRunCount`
|
||||
and project cards. Keep its 28 local cells intentional. `expectedRunCount`
|
||||
counts provider turns, including cancelled and handed-off task runs, but excludes
|
||||
synthetic `/new` runs. Assertions must inspect all company runs because ordinary
|
||||
issue lists exclude the source conversation. `assertChatHandoff` rejects missing
|
||||
@@ -172,3 +172,12 @@ workspaces, task documents, and ordering. They pass through the normal sanitizer
|
||||
Screenshots are allowlisted to the exact disposable agent chat. Cleanup cancels
|
||||
all active runs in the isolated company, including handed-off work; usage from
|
||||
failed and cancelled runs must not disappear from campaign totals.
|
||||
|
||||
`chat-hardening.ts` adds the explicit-only `agent-chat-hardening` journeys. Use
|
||||
the ordinary public APIs to seed source documents and blockers. Keep the answer
|
||||
out of the user's status/review request. Grade the exact source values, latest
|
||||
blocker, preserved task identities, worker-authored output, and real executions.
|
||||
The lost-acknowledgement probe may interrupt only the fixture browser's own
|
||||
comment request after the real server has committed it. Retain its request ID
|
||||
and replay that same request through the public API after restarting the server.
|
||||
Never fabricate tool results or repair task state after a failed assertion.
|
||||
@@ -131,15 +131,15 @@ runner instance, PID, and process-start identity. Each turn is bounded to ten
|
||||
minutes, the cell to thirty minutes, and cleanup explicitly deletes the
|
||||
sandbox rather than waiting for Daytona's idle timeout.
|
||||
|
||||
`agent-chat` (**Persistent Agent Chat**) adds six workflows on `legacy-codex`,
|
||||
`legacy-claude`, `runner-codex`, and `runner-acpx-claude`: **26 local cells**.
|
||||
`agent-chat` (**Persistent Agent Chat**) has eight workflows on `legacy-codex`,
|
||||
`legacy-claude`, `runner-codex`, and `runner-acpx-claude`: **28 local cells**.
|
||||
They cover continuity across server restart, fresh context after `/new`,
|
||||
Stop/reset/resume, draft/revise/approve/plan handoff, clarification with existing
|
||||
project reuse, and a new project with two repository URLs. Each cell opens the
|
||||
production chat surface and resolves the backing issue through the chat API.
|
||||
The source conversation must settle to `in_review` / `waiting`; handed-off
|
||||
execution tasks must finish with their initial Plan and output documents.
|
||||
Reset runs are retained separately from the 72 expected provider turns in this
|
||||
Reset runs are retained separately from the 76 expected run attempts in this
|
||||
suite. Cancelled turns and execution-task runs remain included in billing and
|
||||
cleanup. The production chat directive is injected normally; fixtures do not
|
||||
replace it with completion instructions. Daytona is excluded.
|
||||
@@ -151,6 +151,40 @@ and backlog tasks. The oracle verifies stable task IDs, preserved descriptions,
|
||||
assignment audit evidence, exactly one successful successor run and its output
|
||||
document, no backlog execution, and a usable source conversation after reload.
|
||||
|
||||
`agent-chat-hardening` is an explicit-only native Codex/Claude suite with
|
||||
**18 cells: 12 local and six warm Daytona**. It adds startup Stop/reset,
|
||||
hire/delegate/reuse, grounded blocker reporting with source-document review,
|
||||
and committed-send retry. It also runs active Stop/reset and restart continuity.
|
||||
Daytona selects active Stop/reset, restart continuity, and committed-send retry.
|
||||
The 56 expected run attempts include cancelled attempts; synthetic resets are
|
||||
recorded separately. The suite uses production permission defaults and prompts.
|
||||
Only hiring and cross-task status/review enable the opt-in native API tools.
|
||||
Hiring uses a personal managed AI connection and verifies the hired worker's
|
||||
actual execution account. This is not an onboarding-default qualification.
|
||||
|
||||
Stop during startup and Stop during an active response are separate boundaries.
|
||||
The native active-response case requires a recorded provider turn start; generic
|
||||
lifecycle/performance events are insufficient. The startup case must stop after
|
||||
a process launch request but before a provider turn starts. Missing the boundary
|
||||
fails the case instead of silently testing another phase.
|
||||
|
||||
The committed-send case withholds the browser's comment acknowledgement until
|
||||
the agent has saved one backlog task, drops that response, restarts Paperclip,
|
||||
and replays the exact public request with the original client request ID. It
|
||||
requires the original comment, task, plan, and single consuming run. This proves
|
||||
HTTP request idempotency across restart, not replay safety for an ambiguous
|
||||
provider tool response. Existing native tool-receipt tests cover that boundary.
|
||||
|
||||
```sh
|
||||
pnpm test:e2e:runner -- --list --suite agent-chat-hardening
|
||||
pnpm test:e2e:runner -- --id agent-chat-hardening.runner-codex.local.stop-startup-new-resume
|
||||
```
|
||||
|
||||
Each hardening oracle has positive and plausible-negative calibration tests.
|
||||
The review grader parses the worker's saved JSON and compares both source values
|
||||
and the consistency verdict. Hiring requires one identity, correct reporting
|
||||
line, managed credentials, and real task execution; chat claims cannot pass it.
|
||||
|
||||
```bash
|
||||
# Run these after deterministic checks, with the required provider keys set.
|
||||
pnpm test:e2e:runner -- --id agent-chat.legacy-codex.local.continuity-restart
|
||||
@@ -176,8 +210,9 @@ throwaway instance; never point the authenticated suite at the running demo.
|
||||
Missing provider credentials fail paid preflight and are not passing coverage.
|
||||
|
||||
The default `--all` selection is 171 cells (148 local and 23 Daytona) and 371
|
||||
expected paid agent turns. The explicit-only everyday suite adds 35 catalog cells
|
||||
and is excluded from `--all`. Follow-up steps remain ordered within their cell; all other
|
||||
expected paid agent turns. The explicit-only everyday suite adds 38 catalog cells
|
||||
and chat hardening adds 18. Both are excluded from `--all`. The full catalog has 227 cells.
|
||||
Follow-up steps remain ordered within their cell; all other
|
||||
cells are independent. Narrow selectors are strongly recommended while
|
||||
developing fixtures.
|
||||
|
||||
|
||||
@@ -57,10 +57,10 @@ describe("runner E2E catalog", () => {
|
||||
expect(localIntegrityTasks).toHaveLength(2);
|
||||
expect(openRouterBreadthTasks).toHaveLength(3);
|
||||
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
|
||||
23, 38, 52, 28, 42, 14, 10, 2,
|
||||
23, 38, 52, 28, 18, 42, 14, 10, 2,
|
||||
]);
|
||||
expect(validateRunnerCatalog()).toHaveLength(209);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(209);
|
||||
expect(validateRunnerCatalog()).toHaveLength(227);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(227);
|
||||
expect(
|
||||
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
|
||||
).toHaveLength(42);
|
||||
|
||||
@@ -2,7 +2,7 @@ import { continuationTasks } from "./continuation-cases.js";
|
||||
import { everydayTasks, productionStoryProfile } from "./everyday-cases.js";
|
||||
|
||||
import { firstTaskTasks } from "./first-task-cases.js";
|
||||
import { chatTasks } from "./chat-cases.js";
|
||||
import { chatTasks, chatHardeningTasks } from "./chat-cases.js";
|
||||
import { createHash } from "node:crypto";
|
||||
import { createAgentSchema } from "../../packages/shared/src/validators/agent.js";
|
||||
import { createEnvironmentSchema } from "../../packages/shared/src/validators/environment.js";
|
||||
@@ -944,7 +944,17 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
|
||||
profiles: runnerProfiles.filter(profile => ["legacy-codex", "legacy-claude", "runner-codex", "runner-acpx-claude"].includes(profile.id)).map(defaultPermissionProfile),
|
||||
environments: [localEnvironment], tasks: chatTasks, expectedMatrixSize: 28,
|
||||
excludedExecutionIds: ["legacy-codex", "legacy-claude"].flatMap(profile => ["reassign-task", "create-backlog"].map(task => `agent-chat.${profile}.local.${task}`)),
|
||||
definitionMetadata: { version: 4, resetRunsCountedSeparately: true, permissions: "production-defaults" },
|
||||
definitionMetadata: { version: 5, resetRunsCountedSeparately: true, permissions: "production-defaults", stopBoundary: "provider-turn-started" },
|
||||
},
|
||||
{
|
||||
id: "agent-chat-hardening", label: "Agent Chat Recovery and Coordination", manualOnly: true,
|
||||
description: "Native chat startup cancellation, committed sends, hiring, grounded status, and remote continuity.",
|
||||
groups: ["chat", "native"],
|
||||
profiles: runnerProfiles.filter(profile => ["runner-codex", "runner-acpx-claude"].includes(profile.id)).map(defaultPermissionProfile),
|
||||
environments: [localEnvironment, daytonaWarmEnvironment], tasks: chatHardeningTasks, expectedMatrixSize: 18,
|
||||
excludedExecutionIds: ["runner-codex", "runner-acpx-claude"].flatMap(profile =>
|
||||
["stop-startup-new-resume", "hire-delegate-reuse", "blocked-status-review"].map(task => `agent-chat-hardening.${profile}.daytona.${task}`)),
|
||||
definitionMetadata: { version: 1, permissions: "production-defaults", grading: "durable-state-and-source-evidence", scheduling: "explicit-only" },
|
||||
},
|
||||
...(process.env.PAPERCLIP_RUNNER_E2E_CONNECTION_REVIEWS === "1" ? [connectionReviewSuite] : []),
|
||||
{
|
||||
|
||||
@@ -20,7 +20,15 @@ export const CHAT_CASES = [
|
||||
["multi-repository", "Create a project with multiple repository URLs", 2],
|
||||
] as const;
|
||||
export type ChatCase = (typeof CHAT_CASES)[number][0];
|
||||
export const chatTasks: readonly RunnerTaskFixture[] = CHAT_CASES.map(
|
||||
const HARDENING_CASES = [
|
||||
["stop-startup-new-resume", "Stop during startup, reset, and resume", 3],
|
||||
["hire-delegate-reuse", "Hire through chat, delegate, and reuse the same teammate", 5],
|
||||
["blocked-status-review", "Read the actual blocker and hand source material to a reviewer", 4],
|
||||
["committed-send-retry", "Recover a lost send acknowledgement without repeating committed work", 2],
|
||||
...CHAT_CASES.filter(([id]) => ["stop-new-resume", "continuity-restart"].includes(id)),
|
||||
] as const;
|
||||
function buildChatTasks(definitions: readonly (readonly [string, string, number])[]): readonly RunnerTaskFixture[] {
|
||||
return definitions.map(
|
||||
([id, label, expectedRunCount]) => ({
|
||||
id,
|
||||
label,
|
||||
@@ -35,4 +43,15 @@ export const chatTasks: readonly RunnerTaskFixture[] = CHAT_CASES.map(
|
||||
buildVisibleMarker: (nonce) => chatMarker("CHAT", nonce),
|
||||
buildMatchers: () => [{ kind: "issue_status", expected: "in_review" }],
|
||||
}),
|
||||
);
|
||||
);
|
||||
}
|
||||
export const chatTasks = buildChatTasks(CHAT_CASES);
|
||||
export const chatHardeningTasks = buildChatTasks(HARDENING_CASES);
|
||||
|
||||
export function chatNeedsApiTools(suiteId: string, caseId: string): boolean {
|
||||
return suiteId === "agent-chat-hardening" && ["hire-delegate-reuse", "blocked-status-review"].includes(caseId);
|
||||
}
|
||||
export function isManagedHiringCase(suiteId: string, caseId: string): boolean {
|
||||
return (suiteId === "everyday-workflows" && caseId === "hire-reuse") ||
|
||||
(suiteId === "agent-chat-hardening" && caseId === "hire-delegate-reuse");
|
||||
}
|
||||
@@ -8,6 +8,7 @@ import type { LiveFixtureValues } from "./live-fixtures.js";
|
||||
import type { MatrixExecution } from "./types.js";
|
||||
import { isBlockedUnstartedWake } from "./non-execution-wake.js";
|
||||
import { chatMarker } from "./chat-cases.js";
|
||||
import { isChatStopReady, runChatHardeningFlow } from "./chat-hardening.js";
|
||||
|
||||
// Public API observations only: this driver never fabricates provider results or writes DB state.
|
||||
export interface ChatIssue {
|
||||
@@ -285,7 +286,7 @@ export function assertChatReassignment(input: {
|
||||
expect(input.outputBody).toContain(input.marker);
|
||||
}
|
||||
|
||||
export async function runChatFlow(input: {
|
||||
export interface ChatFlowInput {
|
||||
page: Page;
|
||||
api: RunnerApi;
|
||||
fixtures: LiveFixtureValues;
|
||||
@@ -296,13 +297,15 @@ export async function runChatFlow(input: {
|
||||
observe: (issue: ChatIssue, runs: ChatRun[]) => void;
|
||||
capture: (id: string, label: string, file: string) => Promise<void>;
|
||||
evidence: (name: string, data: unknown) => Promise<void>;
|
||||
}) {
|
||||
}
|
||||
export async function runChatFlow(input: ChatFlowInput) {
|
||||
const { page, api, fixtures: f, execution, nonce } = input;
|
||||
const chatPath = `/api/companies/${f.company.id}/chats/${f.agent.id}`;
|
||||
const route = `/${f.company.issuePrefix}/chats/${f.agent.id}`;
|
||||
const marker = execution.task.buildVisibleMarker(nonce);
|
||||
const draftMarker = chatMarker("DRAFT", nonce);
|
||||
const caseId = execution.task.id;
|
||||
const stopCase = ["stop-new-resume", "stop-startup-new-resume"].includes(caseId);
|
||||
let issue: ChatIssue;
|
||||
let runs: ChatRun[] = [];
|
||||
const settings = await api.get<Record<string, unknown>>(
|
||||
@@ -346,7 +349,7 @@ export async function runChatFlow(input: {
|
||||
activeRuns: runs
|
||||
.filter((run) => ["queued", "running"].includes(run.status))
|
||||
.map((run) => run.id),
|
||||
failure: chatRunFailure(runs, caseId === "stop-new-resume"),
|
||||
failure: chatRunFailure(runs, stopCase),
|
||||
};
|
||||
},
|
||||
reject: (state) => state.failure ?? inconsistentIdle(state),
|
||||
@@ -378,7 +381,7 @@ export async function runChatFlow(input: {
|
||||
expect(await allRuns()).toHaveLength(0);
|
||||
|
||||
if (
|
||||
["continuity-restart", "new-session", "stop-new-resume"].includes(caseId)
|
||||
["continuity-restart", "new-session", "stop-new-resume", "stop-startup-new-resume"].includes(caseId)
|
||||
) {
|
||||
const secret = chatMarker("OLDCONTEXT", nonce);
|
||||
await turn(
|
||||
@@ -411,7 +414,7 @@ export async function runChatFlow(input: {
|
||||
);
|
||||
} else {
|
||||
let cancelledId: string | undefined;
|
||||
if (caseId === "stop-new-resume") {
|
||||
if (stopCase) {
|
||||
await sendChatMessage(
|
||||
page,
|
||||
"Explain the history of gardening at length here, in 100 numbered paragraphs. This is discussion only; do not create work.",
|
||||
@@ -427,9 +430,14 @@ export async function runChatFlow(input: {
|
||||
const events = await api.get<Array<Record<string, unknown>>>(
|
||||
`/api/heartbeat-runs/${active.id}/events?limit=1000`,
|
||||
);
|
||||
const log = await readRunningChatLog(api, active.id);
|
||||
if (!(events.length || log?.length)) return false;
|
||||
if (execution.profile.generation === "native") {
|
||||
if (!isChatStopReady(events, caseId === "stop-startup-new-resume" ? "startup" : "active")) return false;
|
||||
} else if (!(events.length || (await readRunningChatLog(api, active.id))?.length)) return false;
|
||||
cancelledId = active.id;
|
||||
await input.evidence("chat-stop-boundary.json", {
|
||||
phase: caseId === "stop-startup-new-resume" ? "startup" : "active",
|
||||
runId: active.id, events,
|
||||
});
|
||||
return true;
|
||||
},
|
||||
{ timeout: 120_000 },
|
||||
@@ -441,6 +449,7 @@ export async function runChatFlow(input: {
|
||||
async () =>
|
||||
(await api.get<ChatRun>(`/api/heartbeat-runs/${cancelledId}`))
|
||||
.status,
|
||||
{ timeout: 30_000 },
|
||||
)
|
||||
.toBe("cancelled");
|
||||
}
|
||||
@@ -502,6 +511,8 @@ export async function runChatFlow(input: {
|
||||
}
|
||||
expect(issue!.id).toBe(initialId);
|
||||
await noTasks();
|
||||
} else if (["hire-delegate-reuse", "blocked-status-review", "committed-send-retry"].includes(caseId)) {
|
||||
await runChatHardeningFlow({ input, marker, issue: () => issue!, turn, idle, tasks, allRuns, comments });
|
||||
} else if (caseId === "create-backlog") {
|
||||
const project = await api.post<{ id: string }>(`/api/companies/${f.company.id}/projects`, {
|
||||
name: `Later planning ${nonce}`, description: "Repository-free plans to save for later.",
|
||||
@@ -894,7 +905,7 @@ export async function runChatFlow(input: {
|
||||
for (const run of runs.filter((run) => !isResetRun(run))) {
|
||||
expect(run.runtimeMode).toBe(execution.profile.expectedRuntimeMode);
|
||||
expect(run.status).toBe(
|
||||
caseId === "stop-new-resume" && run.status === "cancelled"
|
||||
stopCase && run.status === "cancelled"
|
||||
? "cancelled"
|
||||
: "succeeded",
|
||||
);
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { assertChatHire, assertChatSourceReview, assertCommittedSendRetry, assertGroundedChatStatus, isChatStopReady } from "./chat-hardening.js";
|
||||
import { runnerMatrix } from "./catalog.js";
|
||||
import { buildRunnerE2EProcessEnvironment } from "./harness-env.js";
|
||||
|
||||
const binding = { provider: "anthropic", method: "api_key", mode: "responsible_user" };
|
||||
const hire = { id: "hire", name: "Morgan", reportsTo: "lead", adapterType: "paperclip_runner", adapterConfig: { model: "model" }, runtimeConfig: { aiConnection: binding } };
|
||||
const task = { id: "task", companyId: "company", title: "Checklist", status: "done", assigneeAgentId: "hire", parentId: null, projectId: "project" };
|
||||
const run = { id: "run", companyId: "company", agentId: "hire", status: "succeeded", runtimeMode: "native", contextSnapshot: { issueId: "task", aiConnection: { connectionId: "account" } } };
|
||||
const hiring = { agents: [{ id: "lead", name: "Lead", adapterConfig: { model: "model" } }, hire], leadId: "lead", hireName: "Morgan", hiredId: "hire", connectionId: "account", binding, taskIds: ["task"], tasks: [task], runs: [run] };
|
||||
|
||||
describe("agent chat hardening oracles", () => {
|
||||
it("grades actual execution by the original hired identity and managed account", () => {
|
||||
expect(() => assertChatHire(hiring)).not.toThrow();
|
||||
for (const wrong of [
|
||||
{ ...hiring, agents: [...hiring.agents, { ...hire, id: "duplicate" }] },
|
||||
{ ...hiring, hiredId: "replaced" },
|
||||
{ ...hiring, tasks: [{ ...task, parentId: "chat" }] },
|
||||
{ ...hiring, tasks: [{ ...task, assigneeAgentId: "lead" }] },
|
||||
{ ...hiring, runs: [] },
|
||||
{ ...hiring, runs: [run, { ...run, id: "duplicate-run" }] },
|
||||
{ ...hiring, runs: [{ ...run, agentId: "lead" }] },
|
||||
{ ...hiring, connectionId: "wrong-account" },
|
||||
]) expect(() => assertChatHire(wrong)).toThrow();
|
||||
});
|
||||
|
||||
it("requires the current recorded blocker without silently starting or changing work", () => {
|
||||
const before = { ...task, status: "blocked" };
|
||||
const valid = { reply: "RUN-2 is blocked on VENUE123.", expectedIssueIdentifier: "RUN-2", blocker: "VENUE123", staleBlocker: "BUDGET123",
|
||||
before, after: before, taskIdsBefore: ["task"], taskIdsAfter: ["task"], taskRuns: [] };
|
||||
expect(() => assertGroundedChatStatus(valid)).not.toThrow();
|
||||
expect(() => assertGroundedChatStatus({ ...valid, reply: "RUN-2 is blocked on BUDGET123." })).toThrow();
|
||||
expect(() => assertGroundedChatStatus({ ...valid, after: task })).toThrow();
|
||||
expect(() => assertGroundedChatStatus({ ...valid, taskRuns: [run] })).toThrow();
|
||||
expect(() => assertGroundedChatStatus({ ...valid, taskIdsAfter: ["task", "replacement"] })).toThrow();
|
||||
});
|
||||
|
||||
it("independently compares review values and verdict, rejecting copied sources and missing evidence", () => {
|
||||
const expected = { planLaunchDay: "Tuesday", briefLaunchDay: "Wednesday" };
|
||||
expect(() => assertChatSourceReview(JSON.stringify({ ...expected, consistent: false }), expected)).not.toThrow();
|
||||
expect(() => assertChatSourceReview(JSON.stringify({ ...expected, consistent: true }), expected)).toThrow();
|
||||
expect(() => assertChatSourceReview("The plan says Tuesday and the brief says Wednesday.", expected)).toThrow();
|
||||
expect(() => assertChatSourceReview("{}", expected)).toThrow();
|
||||
expect(() => assertChatSourceReview(JSON.stringify({ ...expected, planLaunchDay: "Friday", consistent: false }), expected)).toThrow();
|
||||
expect(() => assertChatSourceReview('{"planLaunchDay":"Tuesday","briefLaunchDay":"Tuesday","consistent":true}', { planLaunchDay: "Tuesday", briefLaunchDay: "Tuesday" })).not.toThrow();
|
||||
});
|
||||
|
||||
it("does not confuse lifecycle logs, provider startup, and an active turn", () => {
|
||||
expect(isChatStopReady([{ eventType: "lifecycle" }, { eventType: "run.performance.span" }], "active")).toBe(false);
|
||||
const startup = [{ eventType: "native.process_start_requested" }];
|
||||
expect(isChatStopReady(startup, "startup")).toBe(true);
|
||||
expect(isChatStopReady(startup, "active")).toBe(false);
|
||||
const active = [...startup, { eventType: "turn.started" }];
|
||||
expect(isChatStopReady(active, "startup")).toBe(false);
|
||||
expect(isChatStopReady(active, "active")).toBe(true);
|
||||
});
|
||||
|
||||
it("requires one committed comment, one task, and one consuming run after request replay", () => {
|
||||
const valid = { commentId: "comment", clientRequestId: "request", comments: [{ id: "comment", body: "create", clientRequestId: "request" }], taskId: "task",
|
||||
tasks: [{ ...task, status: "backlog" }], chatId: "chat", runs: [{ ...run, contextSnapshot: { issueId: "chat", wakeCommentId: "comment" } }] };
|
||||
expect(() => assertCommittedSendRetry(valid)).not.toThrow();
|
||||
expect(() => assertCommittedSendRetry({ ...valid, comments: [...valid.comments, { ...valid.comments[0]!, id: "duplicate" }] })).toThrow();
|
||||
expect(() => assertCommittedSendRetry({ ...valid, tasks: [...valid.tasks, { ...task, id: "duplicate" }] })).toThrow();
|
||||
expect(() => assertCommittedSendRetry({ ...valid, runs: [...valid.runs, run] })).toThrow();
|
||||
expect(() => assertCommittedSendRetry({ ...valid, runs: [...valid.runs, { ...valid.runs[0]!, id: "duplicate" }] })).toThrow();
|
||||
expect(() => assertCommittedSendRetry({ ...valid, runs: [] })).toThrow();
|
||||
});
|
||||
|
||||
it("keeps the paid hardening matrix explicit and limits API-tool opt-in to coordination", () => {
|
||||
const cells = runnerMatrix.filter(cell => cell.suite.id === "agent-chat-hardening");
|
||||
expect(cells).toHaveLength(18);
|
||||
expect(cells.every(cell => cell.suite.manualOnly && cell.profile.generation === "native")).toBe(true);
|
||||
expect(cells.filter(cell => cell.environment.id === "daytona")).toHaveLength(6);
|
||||
for (const cell of cells) {
|
||||
expect(buildRunnerE2EProcessEnvironment({}, [cell]).PAPERCLIP_RUNNER_API_TOOLS_ENABLED).toBe(
|
||||
["hire-delegate-reuse", "blocked-status-review"].includes(cell.task.id) ? "true" : undefined);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,199 @@
|
||||
import { expect, type Route } from "@playwright/test";
|
||||
import type { RunnerApi } from "./api.js";
|
||||
import { readChatOutputDocument, sendChatMessage, type ChatFlowInput, type ChatIssue, type ChatRun } from "./chat-flow.js";
|
||||
|
||||
type Agent = {
|
||||
id: string; name: string; reportsTo?: string; adapterType?: string;
|
||||
adapterConfig?: Record<string, unknown>; runtimeConfig?: Record<string, unknown>;
|
||||
};
|
||||
type Comment = { id: string; body: string; authorAgentId?: string; clientRequestId?: string };
|
||||
type Document = { id: string; body: string; latestRevisionId: string; createdByAgentId?: string };
|
||||
|
||||
export function assertChatHire(input: {
|
||||
agents: Agent[]; leadId: string; hireName: string; hiredId?: string;
|
||||
connectionId: string; binding: unknown; taskIds: string[]; tasks: ChatIssue[]; runs: ChatRun[];
|
||||
}) {
|
||||
const hires = input.agents.filter(agent => agent.name === input.hireName);
|
||||
expect(hires).toHaveLength(1);
|
||||
const hired = hires[0]!;
|
||||
const lead = input.agents.find(agent => agent.id === input.leadId)!;
|
||||
expect(input.connectionId).toBeTruthy();
|
||||
expect(hired).toMatchObject({ reportsTo: input.leadId, adapterType: "paperclip_runner" });
|
||||
if (input.hiredId) expect(hired.id).toBe(input.hiredId);
|
||||
expect(hired.adapterConfig?.model).toBe(lead.adapterConfig?.model);
|
||||
expect(hired.runtimeConfig?.aiConnection).toEqual(input.binding);
|
||||
expect(input.tasks.map(task => task.id).sort()).toEqual([...input.taskIds].sort());
|
||||
for (const id of input.taskIds) {
|
||||
const task = input.tasks.find(task => task.id === id)!;
|
||||
expect(task).toMatchObject({ status: "done", assigneeAgentId: hired.id, parentId: null });
|
||||
expect(task.projectId).toBeTruthy();
|
||||
const runs = input.runs.filter(run => run.contextSnapshot?.issueId === id);
|
||||
expect(runs).toHaveLength(1);
|
||||
expect(runs[0]).toMatchObject({ agentId: hired.id, status: "succeeded", runtimeMode: "native" });
|
||||
expect(runs[0]!.contextSnapshot?.aiConnection).toMatchObject({ connectionId: input.connectionId });
|
||||
}
|
||||
return hired;
|
||||
}
|
||||
|
||||
export function assertGroundedChatStatus(input: {
|
||||
reply: string; expectedIssueIdentifier: string; blocker: string; staleBlocker: string;
|
||||
before: ChatIssue; after: ChatIssue; taskIdsBefore: string[]; taskIdsAfter: string[]; taskRuns: ChatRun[];
|
||||
}) {
|
||||
expect(input.reply).toContain(input.expectedIssueIdentifier);
|
||||
expect(input.reply).toMatch(/blocked/i);
|
||||
expect(input.reply).toContain(input.blocker);
|
||||
expect(input.reply).not.toContain(input.staleBlocker);
|
||||
expect(input.after).toMatchObject({ id: input.before.id, status: "blocked", assigneeAgentId: input.before.assigneeAgentId });
|
||||
expect([...input.taskIdsAfter].sort()).toEqual([...input.taskIdsBefore].sort());
|
||||
expect(input.taskRuns).toHaveLength(0);
|
||||
}
|
||||
|
||||
export function assertChatSourceReview(body: string, expected: { planLaunchDay: string; briefLaunchDay: string }) {
|
||||
// The user requests JSON, so an independent parser grades the actual saved
|
||||
// deliverable. A claim in the chat or a copied source document cannot pass.
|
||||
const json = body.trim().replace(/^```(?:json)?\s*/, "").replace(/\s*```$/, "");
|
||||
expect(JSON.parse(json)).toMatchObject({ ...expected, consistent: expected.planLaunchDay === expected.briefLaunchDay });
|
||||
}
|
||||
|
||||
export function assertCommittedSendRetry(input: {
|
||||
commentId: string; clientRequestId: string; comments: Comment[]; taskId: string;
|
||||
tasks: ChatIssue[]; runs: ChatRun[]; chatId: string;
|
||||
}) {
|
||||
expect(input.comments.filter(comment => comment.clientRequestId === input.clientRequestId).map(comment => comment.id)).toEqual([input.commentId]);
|
||||
expect(input.tasks.map(task => task.id)).toEqual([input.taskId]);
|
||||
expect(input.tasks[0]).toMatchObject({ status: "backlog", parentId: null });
|
||||
expect(input.runs.filter(run => run.contextSnapshot?.issueId === input.taskId)).toHaveLength(0);
|
||||
const original = input.runs.filter(run => run.contextSnapshot?.wakeCommentId === input.commentId ||
|
||||
(Array.isArray(run.contextSnapshot?.wakeCommentIds) && run.contextSnapshot.wakeCommentIds.includes(input.commentId)));
|
||||
expect(original).toHaveLength(1);
|
||||
expect(original[0]).toMatchObject({ status: "succeeded", runtimeMode: "native" });
|
||||
expect(original[0]!.contextSnapshot?.issueId).toBe(input.chatId);
|
||||
}
|
||||
|
||||
export function isChatStopReady(events: Array<{ eventType?: unknown }>, phase: "startup" | "active"): boolean {
|
||||
const active = events.some(event => ["turn.started", "turn.accepted"].includes(String(event.eventType)));
|
||||
return phase === "active" ? active : !active && events.some(event => event.eventType === "native.process_start_requested");
|
||||
}
|
||||
|
||||
export async function runChatHardeningFlow(context: {
|
||||
input: ChatFlowInput; marker: string; issue(): ChatIssue;
|
||||
turn(message: string, count: number): Promise<void>; idle(count: number): Promise<void>;
|
||||
tasks(): Promise<ChatIssue[]>; allRuns(): Promise<ChatRun[]>; comments(): Promise<Comment[]>;
|
||||
}) {
|
||||
const { input, marker, turn, idle, tasks, allRuns, comments } = context;
|
||||
const { api, page, fixtures: f, execution, nonce } = input;
|
||||
const companyPath = `/api/companies/${f.company.id}`;
|
||||
const project = await api.post<{ id: string; name: string }>(`${companyPath}/projects`, {
|
||||
name: `Chat coordination ${nonce}`, description: "Repository-free launch planning and review.",
|
||||
});
|
||||
const agents = () => api.get<Agent[]>(`${companyPath}/agents`);
|
||||
const latestReply = async () => (await comments()).filter(comment => comment.authorAgentId === f.agent.id).at(-1)?.body ?? "";
|
||||
const output = (id: string, token: string) => readChatOutputDocument(api, id, token);
|
||||
|
||||
if (execution.task.id === "hire-delegate-reuse") {
|
||||
const hireName = `Morgan Reviewer ${nonce}`;
|
||||
await turn(`Hire exactly one teammate named ${hireName}, reporting to you, using your native runner, model, and available AI connection. Have that teammate write a concise launch checklist as a saved Paperclip document containing ${marker}. Put the work in one assigned task in ${project.name}. Link it here and let the teammate complete it.`, 2);
|
||||
const first = (await tasks())[0]!;
|
||||
expect(first).toBeTruthy();
|
||||
const firstOutput = await output(first.id, marker);
|
||||
const account = f.aiConnection!;
|
||||
const hired = assertChatHire({ agents: await agents(), leadId: f.agent.id, hireName,
|
||||
connectionId: account.connectionId, binding: account.binding, taskIds: [first.id], tasks: await tasks(), runs: await allRuns() });
|
||||
expect((firstOutput as Document).createdByAgentId).toBe(hired.id);
|
||||
await turn(`Have the existing ${hireName} review the checklist on ${first.identifier} and write a separate saved review document containing REVIEW${marker}. Create one review task in ${project.name}, assigned to that same teammate. Include the actual checklist in the handoff so they can review it. Preserve the original checklist and task.`, 4);
|
||||
const observed = await tasks();
|
||||
const second = observed.find(task => task.id !== first.id)!;
|
||||
expect(second).toBeTruthy();
|
||||
const reviewOutput = await output(second.id, `REVIEW${marker}`);
|
||||
expect((reviewOutput as Document).createdByAgentId).toBe(hired.id);
|
||||
expect(await api.get(`/api/issues/${first.id}/documents/${encodeURIComponent(firstOutput.key)}`)).toEqual(firstOutput);
|
||||
await turn(`Give me a brief status update on ${first.identifier} and ${second.identifier}: identify their owner and recorded status. Do not create or change work.`, 5);
|
||||
assertChatHire({ agents: await agents(), leadId: f.agent.id, hireName, hiredId: hired.id,
|
||||
connectionId: account.connectionId, binding: account.binding, taskIds: [first.id, second.id], tasks: await tasks(), runs: await allRuns() });
|
||||
const reply = await latestReply();
|
||||
expect(reply).toContain(first.identifier!); expect(reply).toContain(second.identifier!);
|
||||
expect(reply).toContain(hireName); expect(reply).toMatch(/done|completed/i);
|
||||
await input.evidence("chat-hire-reuse.json", { agents: await agents(), tasks: await tasks(), firstOutput, reviewOutput, runs: await allRuns() });
|
||||
} else if (execution.task.id === "blocked-status-review") {
|
||||
const blocker = `VENUE${marker}`, staleBlocker = `BUDGET${marker}`;
|
||||
const source = await api.post<ChatIssue>(`${companyPath}/issues`, {
|
||||
title: `Launch brief ${nonce}`, status: "blocked", projectId: project.id,
|
||||
description: "The launch brief is waiting for its recorded blocker to be resolved.",
|
||||
initialPlan: "The approved launch day is Tuesday.",
|
||||
});
|
||||
const docResponse = await api.request.put(`/api/issues/${source.id}/documents/brief`, {
|
||||
data: { title: "Launch brief", format: "markdown", body: "The launch brief schedules the launch for Wednesday." },
|
||||
});
|
||||
expect(docResponse.ok()).toBe(true);
|
||||
const sourcePlan = await api.get<Document>(`/api/issues/${source.id}/documents/plan`);
|
||||
const sourceBrief = await api.get<Document>(`/api/issues/${source.id}/documents/brief`);
|
||||
await api.post(`/api/issues/${source.id}/comments`, { body: `Earlier blocker: ${staleBlocker}. Waiting for the budget.` });
|
||||
await api.post(`/api/issues/${source.id}/comments`, { body: `Budget is resolved. Current blocker: ${blocker}. Waiting for the venue confirmation. Keep this task blocked; no execution is active.` });
|
||||
await turn(`What is the actual current blocker on ${source.identifier}? Read the latest recorded explanation, include its exact blocker label, and distinguish task status from active execution. Just report; do not change it or create work.`, 1);
|
||||
assertGroundedChatStatus({ reply: await latestReply(), expectedIssueIdentifier: source.identifier!, blocker, staleBlocker,
|
||||
before: source, after: await api.get(`/api/issues/${source.id}`), taskIdsBefore: [source.id],
|
||||
taskIdsAfter: (await tasks()).map(task => task.id), taskRuns: (await allRuns()).filter(run => run.contextSnapshot?.issueId === source.id) });
|
||||
const config = execution.profile.buildAgent({ environmentId: f.environment.id, environmentFixtureId: execution.environment.id,
|
||||
workspacePath: input.workspacePath, secretRefs: f.secretRefs, executionId: nonce });
|
||||
const reviewer = await api.post<Agent>(`${companyPath}/agents`, { ...config, name: `Riley Reviewer ${nonce}`, role: "engineer", reportsTo: f.agent.id });
|
||||
await turn(`Have ${reviewer.name} review the saved brief on ${source.identifier} against that task's saved plan, in one separate task in ${project.name}. Give the reviewer both source documents. They should save a JSON review document with planLaunchDay, briefLaunchDay, and a boolean consistent, then complete the review task. Preserve the source task and both documents.`, 3);
|
||||
const review = (await tasks()).find(task => task.id !== source.id)!;
|
||||
expect(review).toMatchObject({ status: "done", assigneeAgentId: reviewer.id, parentId: null, projectId: project.id });
|
||||
const document = await output(review.id, "planLaunchDay");
|
||||
assertChatSourceReview(document.body, { planLaunchDay: "Tuesday", briefLaunchDay: "Wednesday" });
|
||||
expect((document as Document).createdByAgentId).toBe(reviewer.id);
|
||||
const reviewRuns = (await allRuns()).filter(run => run.contextSnapshot?.issueId === review.id);
|
||||
expect(reviewRuns).toHaveLength(1);
|
||||
expect(reviewRuns[0]).toMatchObject({ agentId: reviewer.id, status: "succeeded", runtimeMode: "native" });
|
||||
await turn(`Summarize ${review.identifier}'s recorded review result and tell me whether ${source.identifier} is still blocked. Read the saved evidence; do not change or create work.`, 4);
|
||||
const reply = await latestReply();
|
||||
expect(reply).toContain("Tuesday"); expect(reply).toContain("Wednesday"); expect(reply).toContain(source.identifier!); expect(reply).toMatch(/blocked/i);
|
||||
expect((await tasks()).map(task => task.id).sort()).toEqual([source.id, review.id].sort());
|
||||
expect(await api.get(`/api/issues/${source.id}/documents/plan`)).toEqual(sourcePlan);
|
||||
expect(await api.get(`/api/issues/${source.id}/documents/brief`)).toEqual(sourceBrief);
|
||||
expect(await api.get(`/api/issues/${source.id}`)).toMatchObject({ status: "blocked" });
|
||||
await input.evidence("chat-status-review.json", { source, sourcePlan, sourceBrief, review, document, reviewRuns, reply });
|
||||
} else if (execution.task.id === "committed-send-retry") {
|
||||
let sent: { path: string; data: Record<string, unknown>; commentId: string } | undefined;
|
||||
let abortResponse!: () => void;
|
||||
const responseMayDrop = new Promise<void>(resolve => { abortResponse = resolve; });
|
||||
const handler = async (route: Route) => {
|
||||
if (route.request().method() !== "POST") return route.continue();
|
||||
const data = route.request().postDataJSON() as Record<string, unknown>;
|
||||
if (typeof data.body !== "string" || !data.body.includes(marker) || sent) return route.continue();
|
||||
const response = await route.fetch();
|
||||
expect(response.ok()).toBe(true);
|
||||
const comment = await response.json() as { id: string };
|
||||
sent = { path: new URL(route.request().url()).pathname, data, commentId: comment.id };
|
||||
await responseMayDrop;
|
||||
await route.abort("connectionreset");
|
||||
};
|
||||
await page.route("**/api/issues/*/comments", handler);
|
||||
try {
|
||||
await sendChatMessage(page, `Create exactly one task titled Later ${nonce} in ${project.name}, assigned to yourself, in backlog. Save an initial plan containing ${marker}. Do not start it. Link the task here.`);
|
||||
await idle(1); // The server and real agent complete while the browser acknowledgement is withheld.
|
||||
expect(sent).toBeTruthy();
|
||||
const saved = (await tasks())[0]!;
|
||||
expect(saved).toBeTruthy();
|
||||
const plan = await api.get<Document>(`/api/issues/${saved.id}/documents/plan`);
|
||||
expect(plan.body).toContain(marker);
|
||||
abortResponse();
|
||||
await page.unroute("**/api/issues/*/comments", handler);
|
||||
await input.restart();
|
||||
// Replay the exact interrupted public request, including its original key.
|
||||
// This tests transport retries, not automatic replay of a failed provider turn.
|
||||
const retried = await api.post<Comment>(sent!.path, sent!.data);
|
||||
expect(retried.id).toBe(sent!.commentId);
|
||||
await page.goto(`/${f.company.issuePrefix}/chats/${f.agent.id}`, { waitUntil: "domcontentloaded" });
|
||||
await turn(`What is the saved status of ${saved.identifier}? Just report it; do not create or change work.`, 2);
|
||||
assertCommittedSendRetry({ commentId: sent!.commentId, clientRequestId: String(sent!.data.clientRequestId),
|
||||
comments: await comments(), taskId: saved.id, tasks: await tasks(), runs: await allRuns(), chatId: context.issue().id });
|
||||
expect(await api.get(`/api/issues/${saved.id}/documents/plan`)).toEqual(plan);
|
||||
expect(await latestReply()).toMatch(/backlog/i);
|
||||
await input.evidence("chat-committed-send-retry.json", { commentId: sent!.commentId, task: saved, plan, runs: await allRuns() });
|
||||
} finally {
|
||||
abortResponse();
|
||||
await page.unroute("**/api/issues/*/comments", handler);
|
||||
}
|
||||
} else throw new Error(`Unsupported chat hardening case ${execution.task.id}`);
|
||||
}
|
||||
@@ -1,4 +1,5 @@
|
||||
import path from "node:path";
|
||||
import { chatNeedsApiTools, isManagedHiringCase } from "./chat-cases.js";
|
||||
import { CREDENTIAL_NAMES } from "./types.js";
|
||||
import type { MatrixExecution } from "./types.js";
|
||||
|
||||
@@ -109,7 +110,7 @@ export function buildRunnerE2EProcessEnvironment(
|
||||
delete result.OPENCODE_ALLOW_ALL_MODELS;
|
||||
// Hiring needs the opt-in native API surface. Scope this to the explicit
|
||||
// manual hiring story; production and other suites retain their defaults.
|
||||
if (executions.some((e) => e.suite.id === "everyday-workflows" && e.task.id === "hire-reuse")) {
|
||||
if (executions.some((e) => isManagedHiringCase(e.suite.id, e.task.id) || chatNeedsApiTools(e.suite.id, e.task.id))) {
|
||||
result.PAPERCLIP_RUNNER_API_TOOLS_ENABLED = "true";
|
||||
}
|
||||
if (
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import path from "node:path";
|
||||
import { isManagedHiringCase } from "./chat-cases.js";
|
||||
import { FixtureRegistry } from "./fixture-registry.js";
|
||||
import type { RunnerApi } from "./api.js";
|
||||
import type {
|
||||
@@ -212,9 +213,7 @@ export async function setupLiveFixtures(input: {
|
||||
},
|
||||
});
|
||||
|
||||
const managedHiring =
|
||||
execution.suite.id === "everyday-workflows" &&
|
||||
execution.task.id === "hire-reuse";
|
||||
const managedHiring = isManagedHiringCase(execution.suite.id, execution.task.id);
|
||||
if (managedHiring) {
|
||||
registry.register<ManagedAccountFixture>({
|
||||
id: "ai-connection",
|
||||
|
||||
Reference in new issue
Block a user