From 1511c92965325249ed6b52dd5be52eb6d736fbfa Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 30 Oct 2026 10:40:00 -0500 Subject: [PATCH] fix(evals): account for superseded delegation wakes (#18319) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip lets people ask agents to delegate work and return the result. > - The chat feature eval checks task creation, worker execution, saved output, and delivery to the source chat. > - A task can first belong to the CEO and then move to Morgan. > - Paperclip correctly cancels the old assignment wake before it starts. > - The eval treats that queue record as a failed provider run and stops before Morgan works. > - This change accounts for that record separately and preserves the complete outcome checks. > - The report also keeps partial check results when an attempt fails. ## Linked Issues or Issue Description **What happened?** The delegation journey stopped on an unstarted queue record with `issue_assignee_changed`. The CEO had created a task, assigned Morgan, saved the brief, and returned a task link. The test stopped before worker execution. Its report showed no individual checks even though the saved observation contained partial results. [Original public measurement](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/chat-progress-39e-112-latest-121-measurements-with-screenshots-40649033672/?view=case&case=chat-features.delegate-announcement). **Expected behavior** Keep the obsolete queue record and its events. Continue observing the intended worker. Pass only after three successful native work runs, a current Morgan-authored announcement, a completed task, automatic source-chat delivery, and browser readback. **Steps to reproduce** 1. Ask the CEO to have Morgan write and save a launch announcement, then return it to chat. 2. Create the task under the CEO, then assign Morgan before the queued CEO run starts. 3. Observe the cancelled old assignment. The previous eval aborts at that point. **Paperclip version or commit** Based on `6da112ab40d7b3cc0bcb79e08ddbb9b915a268a4` from `origin/master`. **Deployment mode** Disposable Product E2E fixtures. The live check uses the trusted default-branch workflow and AWS fleet. Related contribution: #17794 introduced these two journeys. The open-PR search found no separate fix for this queue-accounting defect. ## What Changed - Recognize only explicit dispatch-gate reassignment cancellations with a null start time and no native issue. Verify the company, prior CEO owner, and target task now assigned to Morgan before excluding that record from work-run counts. - Keep every queue record in the full observation and final ledger. Retain its events with an explicit reason for the absent provider log. - Let the feature flow own its complete saved-state and browser oracle. Return only actual work runs to the common provider checks. - Record saved-state checks on failures as well as successes. Capture the required `final-state.png` after the feature-specific browser readback. - Add two explicit local OpenCode cells alongside the two existing Codex subscription cells. Use the current production-qualified model. The earlier linked campaign used a different model, so this is a new measurement. - Advance the grader and suite versions. Keep the original request, output requirements, 120-second run cap, five-minute observation bound, budgets, and single-attempt policy. ## Verification - Red/green regressions cover queue accounting, missing-log collection, partial checks on evidence failure, and the required final screenshot. Flow tests invoke the actual polling callbacks: restoring the original callback reproduces both early-stop errors. Started cancellations still fail. - All 87 focused flow/grading/queue tests pass. Catalog/report checks and the 137 additional Node checks pass. Eval typechecking passes on final commit `0a6e11408c4470683417e6bf7965c73cec2a9a60`. - All current CI tests/builds pass. Greptile is 5/5 on the final commit, with no unresolved review threads. - The final commit adds only flow test coverage. Its registered definition digest equals live target `e4e0be875ee12c134fe8dd2ce15a88bfcbb100ef`: `ac6bba21d4b9cb86088be5d00df6442af90268186d6e94c9bea7d611d0488563`. - Replaying the original saved observation still fails for missing Morgan execution and delivery. Its stale queue record no longer counts as provider work. | Operation | Selected coverage | Observed result | | --- | --- | --- | | [41071710105](https://github.com/paperclipai/paperclip/actions/runs/41071710105), `b8dff816` | Backlog and delegation | Refused at the exact private-retention gate before fixtures or providers. Preserved without feature coverage. | | [41073708150](https://github.com/paperclipai/paperclip/actions/runs/41073708150), `39be9e45` | Backlog and delegation | Both passed their full product/browser oracles. Retained grades correctly fail missing `final-state.png`; the feature flow had saved the image under another name. Fixed in `e4e0be87`. Each original archive remains private and intact; later workflow concurrency superseded only the report aggregation. | | [41076854978](https://github.com/paperclipai/paperclip/actions/runs/41076854978), `e4e0be87` | Backlog and delegation | Backlog passed in 92.576 seconds, with a 47.388-second native run, complete required artifacts and passed cleanup. Delegation was interrupted by a runner-host shutdown at 15:19:23 UTC; all later evidence-upload steps were skipped. Its private archive is absent, so behavior, usage and cost remain unverified. | | [41080065077](https://github.com/paperclipai/paperclip/actions/runs/41080065077), `0a6e1140` | Delegation only | Passed on the final commit in 240.786 seconds. All 14 product checks plus browser readback passed; exact private identity, current worker output, Done state, correlated source-chat reply, required screenshot and cleanup verified. The three native runs took 42–60 seconds each. Passing control was not repeated. One-cell workflow and report enforcement passed. | The completed delegation in `41073708150` passed all 14 product checks plus browser readback in 222.593 seconds. Morgan saved the current two-sentence announcement, the task reached Done, and the CEO automatically delivered that saved text to the original chat. Its three successful native runs took 49.117, 40.060 and 39.107 seconds. The failed required screenshot remains a failed result in that measurement. All live operations use explicit IDs, concurrency one, no automatic retries, the unchanged 120-second per-run cap, and scoped private evidence retention bound to the exact workflow, target, original run, selector and launcher digest. These narrow measurements do not qualify the full catalog or establish repeated-run reliability. The broad local support suite passed 5,139 tests, with two skipped. Two database tests hit exhausted host semaphore capacity after the installed Postgres package's missing library symlinks were repaired. Their recheck reported `could not create semaphores: No space left on device`. No host IPC resources or kernel settings were changed. The corresponding full CI tests passed on a fresh host. ## Risks - A broad cancellation exception could hide failed work. This exception requires the dispatch gate's exact receipt, a null start time, the expected company and prior owner, and the actual reassigned task. Started and unproved cancellations still fail. Backlog creation permits no task wake. - New manual OpenCode cells can spend provider money only when explicitly selected. Default scheduled selection is unchanged. - This changes eval interpretation and evidence collection. It does not change product scheduling or task ownership. ## Model Used OpenAI Codex, GPT-6 family, with code execution and tool use. The exact model ID and context-window size are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/evals.md | 3 +- tests/runner-e2e/CHAT-FEATURES.md | 23 ++- tests/runner-e2e/SECURITY.md | 5 +- tests/runner-e2e/catalog.test.ts | 6 +- tests/runner-e2e/catalog.ts | 9 +- tests/runner-e2e/chat-feature-cases.ts | 4 +- tests/runner-e2e/chat-feature-flow.test.ts | 135 ++++++++++++++++++ tests/runner-e2e/chat-feature-flow.ts | 24 ++-- tests/runner-e2e/chat-feature-scoring.test.ts | 39 ++++- tests/runner-e2e/chat-feature-scoring.ts | 27 +++- tests/runner-e2e/chat-flow.test.ts | 14 ++ tests/runner-e2e/chat-flow.ts | 9 +- tests/runner-e2e/non-execution-wake.ts | 10 ++ 13 files changed, 272 insertions(+), 36 deletions(-) create mode 100644 tests/runner-e2e/chat-feature-flow.test.ts diff --git a/doc/evals.md b/doc/evals.md index 6cd938a5718..0e42d6a56df 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -39,7 +39,8 @@ only index HTML and history/pointer JSON; it does not upload evidence or change retained campaign bundles. The explicit-only [chat feature evals](../tests/runner-e2e/CHAT-FEATURES.md) promote -two natural-language task scenarios into local native Product E2E cells. They +two natural-language task scenarios into local native Codex subscription and +OpenCode Product E2E cells. They verify saving an unstarted task and delegating a saved announcement back to chat, using ordinary production instructions, real browser messages and independent state/provenance checks. The broader YAML catalog remains planning inventory. diff --git a/tests/runner-e2e/CHAT-FEATURES.md b/tests/runner-e2e/CHAT-FEATURES.md index 547633405a6..a70961c74d9 100644 --- a/tests/runner-e2e/CHAT-FEATURES.md +++ b/tests/runner-e2e/CHAT-FEATURES.md @@ -20,25 +20,30 @@ task, or sends a hidden follow-up. ## Run the first slice -Follow the installation steps in [README.md](README.md#local-commands). The current -profile is local native Codex using the catalog's recorded default model and an +Follow the installation steps in [README.md](README.md#local-commands). The Codex +subscription profile uses the catalog's recorded default model and an explicit `CODEX_AUTH_JSON` login. Load that login privately into the launcher environment; do not print it or save it in tracked configuration. The existing subscription fixture stages only `auth.json` in the disposable company login slot and removes it with the isolated instance. +The local native OpenCode profile uses the current production-qualified model +and `OPENROUTER_API_KEY` through the existing encrypted company-secret fixture. +Each of the two journeys has a cell for each profile. All four cells are manual. + ```sh pnpm test:e2e:runner:typecheck pnpm test:e2e:runner:unit pnpm test:e2e:runner -- --list --suite chat-features pnpm test:e2e:runner -- --id chat-features.runner-codex-subscription.local.create-backlog --max-automatic-retries 0 pnpm test:e2e:runner -- --id chat-features.runner-codex-subscription.local.delegate-announcement --max-automatic-retries 0 +pnpm test:e2e:runner -- --id chat-features.runner-opencode.local.delegate-announcement --max-automatic-retries 0 ``` Run creation first and inspect its packaged report before delegation. For initial acceptance, run each cell three times against fresh fixtures on the same definition revision. Keep every attempt, including failures; a retry does not erase one. A -fixture or product correction starts a new three-sample measurement. Both cells +fixture or product correction starts a new three-sample measurement. All cells are explicit-only and excluded from `--all`; this does not implement or qualify the rest of the YAML inventory. @@ -59,6 +64,16 @@ probes fall outside heartbeat usage and budget accounting; detailed setup tokens and cost are unavailable. These limits are not an absolute inference or cash cap. Subscription cost coverage must remain explicit in the report. +An obsolete CEO assignment can leave a cancelled queue entry when the CEO creates +a task and then assigns it to Morgan. Grader v2 distinguishes that entry from +provider work only when the dispatch gate records `issue_assignee_changed`, an +explicitly null start time, no native issue, and its unchanged stale-queue stop +receipt. The entry must belong to this company and the one task now assigned to +Morgan. It stays in the full observation and final ledger, with its events and an +explicit provider-log omission reason. Active, unrelated, or unproved cancellations +still fail. The three successful work runs, worker output, completion delivery, +and original deadlines remain required. Backlog creation still permits no task wake. + Because this subscription currently reports unpriced usage, setup explicitly sets `unpricedUsagePolicy: allow` on the disposable company and both agents through the public budget API and verifies all three policies. The 500-cent hard stops remain @@ -74,6 +89,8 @@ on failures. Setup evidence records fixture IDs, synthetic brief, readiness and saved budgets. Ordinary Product E2E evidence packaging, redaction, cleanup, failure classification and dashboards own the result. The definition digest includes the query, defaults, flow and grader sources. +Individual saved-state checks are also recorded when the journey fails, so the +report shows partial progress without turning it into a passing workflow. The grader rejects duplicate tasks, wrong owners/projects/companies, early execution, missing audit attribution, missing or stale deliverables, extra human nudges and diff --git a/tests/runner-e2e/SECURITY.md b/tests/runner-e2e/SECURITY.md index f3a4c8a4b22..906b6e1d1d5 100644 --- a/tests/runner-e2e/SECURITY.md +++ b/tests/runner-e2e/SECURITY.md @@ -1,8 +1,9 @@ # Runner E2E security for a public repository The explicit [chat feature evals](CHAT-FEATURES.md) use synthetic project briefs -and a privately supplied Codex subscription login through the existing isolated -company staging helper. Only source-chat screenshots are captured. Task, document +and either a privately supplied Codex subscription login through the existing +isolated company staging helper or the OpenCode profile's encrypted OpenRouter +secret reference. Only source-chat screenshots are captured. Task, document and run evidence stays behind the existing sanitizer and publication allowlist; the YAML catalog contains no credentials. The suite's accounted-work budgets do not cover production setup probes or imply zero subscription cost. diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index 6b6a37e1b6f..1bbde4a9cd4 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -219,10 +219,10 @@ describe("runner E2E catalog", () => { expect(localIntegrityTasks).toHaveLength(2); expect(openRouterBreadthTasks).toHaveLength(3); expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([ - 2, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 2, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2, + 2, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 4, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2, ]); - expect(validateRunnerCatalog()).toHaveLength(772); - expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(772); + expect(validateRunnerCatalog()).toHaveLength(774); + expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(774); expect( runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"), ).toHaveLength(48); diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index 566b2a218d4..c419d76d1bf 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -1609,10 +1609,11 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ id: CHAT_FEATURE_SUITE, label: "Chat feature journeys", manualOnly: true, description: "Natural catalog requests create backlog work or delegate an announcement and return its saved result.", groups: ["chat", "native"], - profiles: [chatFeatureProfile(nativeProfile({ id: "runner-codex-subscription", label: "Native Codex subscription", - provider: "codex", model: DEFAULT_CODEX_LOCAL_MODEL, credential: "CODEX_AUTH_JSON", supportedEnvironments: ["local"] }))], - environments: [localEnvironment], tasks: chatFeatureTasks, expectedMatrixSize: 2, - definitionMetadata: { version: 1, definitionDigest: chatFeatureDefinitionDigest, scheduling: "explicit-only", + profiles: [nativeProfile({ id: "runner-codex-subscription", label: "Native Codex subscription", + provider: "codex", model: DEFAULT_CODEX_LOCAL_MODEL, credential: "CODEX_AUTH_JSON", supportedEnvironments: ["local"] }), + runnerProfiles.find(profile => profile.id === "runner-opencode")!].map(chatFeatureProfile), + environments: [localEnvironment], tasks: chatFeatureTasks, expectedMatrixSize: 4, + definitionMetadata: { version: 2, definitionDigest: chatFeatureDefinitionDigest, scheduling: "explicit-only", instructions: "production-role-defaults", companyAndAgentBudgetCents: CHAT_FEATURE_BUDGET_CENTS, unpricedUsagePolicy: "allow-through-public-budget-policy", taskRunLimits: { ceo: 3, worker: 1 }, maximumAttemptsPerCell: 1, setupProbeUsage: "unavailable-outside-heartbeat-accounting", diff --git a/tests/runner-e2e/chat-feature-cases.ts b/tests/runner-e2e/chat-feature-cases.ts index 1458d94e9d9..fb2337bb447 100644 --- a/tests/runner-e2e/chat-feature-cases.ts +++ b/tests/runner-e2e/chat-feature-cases.ts @@ -5,7 +5,7 @@ import type { RunnerProfileFixture, RunnerTaskFixture } from "./types.js"; export const CHAT_FEATURE_SUITE = "chat-features"; export const CHAT_FEATURE_BUDGET_CENTS = 500; -export const CHAT_FEATURE_GRADER = "paperclip.chat-features.v1"; +export const CHAT_FEATURE_GRADER = "paperclip.chat-features.v2"; export const CHAT_FEATURE_CASES = ["create-backlog", "delegate-announcement"] as const; export type ChatFeatureCase = typeof CHAT_FEATURE_CASES[number]; export interface ChatScenario { @@ -36,7 +36,7 @@ export function readChatFeatureScenarios(text = readFileSync(new URL("./chat-sce export const chatFeatureScenarios = readChatFeatureScenarios(); export const chatFeatureDefinitionDigest = createHash("sha256").update([ "chat-feature-cases.ts", "chat-feature-flow.ts", "chat-feature-scoring.ts", "chat-flow.ts", - "completion-updates.ts", "redaction.ts", "launch.ts", "chat-scenarios/tasks.yaml", "chat-scenarios/defaults.yaml", + "completion-updates.ts", "non-execution-wake.ts", "redaction.ts", "launch.ts", "chat-scenarios/tasks.yaml", "chat-scenarios/defaults.yaml", "../../server/src/services/agent-conversations.ts", ].map(file => readFileSync(new URL(file, import.meta.url), "utf8")).join("\n")).digest("hex"); diff --git a/tests/runner-e2e/chat-feature-flow.test.ts b/tests/runner-e2e/chat-feature-flow.test.ts new file mode 100644 index 00000000000..9b294922342 --- /dev/null +++ b/tests/runner-e2e/chat-feature-flow.test.ts @@ -0,0 +1,135 @@ +import { afterEach, expect, it, vi } from "vitest"; +import * as chatFlow from "./chat-flow.js"; +import type { ChatFlowInput, ChatIssue, ChatRun } from "./chat-flow.js"; +import { runChatFeatureFlow } from "./chat-feature-flow.js"; +import { chatFeatureScenarios } from "./chat-feature-cases.js"; + +const polling = vi.hoisted(() => ({ failScenario: true })); +vi.mock("@playwright/test", async importOriginal => ({ + ...await importOriginal(), + expect: (actual: { isVisible?(): Promise }, message?: string) => Object.assign(expect(actual, message), { + toBeVisible: async () => expect(await actual.isVisible!()).toBe(true), + }), +})); +vi.mock("./api.js", async importOriginal => ({ + ...await importOriginal(), + pollUntil: async (input: { label: string; load(): Promise; accept?(value: unknown): boolean; reject?(value: unknown): string | undefined }) => { + const value = await input.load(); + if (input.accept?.(value)) return value; + const rejection = input.reject?.(value); + if (rejection) throw new Error(rejection); + if (polling.failScenario && input.label.startsWith("chat scenario")) throw new Error("Original scenario failure"); + return value; + }, +})); + +afterEach(() => { vi.restoreAllMocks(); polling.failScenario = true; }); + +function fixture(boundary?: string) { + vi.spyOn(chatFlow, "sendChatMessage").mockResolvedValue(); + const caseId = boundary ? "delegate-announcement" : "create-backlog"; + const query = chatFeatureScenarios.find(scenario => scenario.id === `tasks.${caseId}`)!.query; + const source: ChatIssue = { id: "chat", companyId: "company", title: "CEO", status: "in_review", + assigneeAgentId: "ceo", conversationState: "waiting" }; + const task = { id: "task", identifier: "EVA-2", companyId: "company", title: "Launch announcement", + assigneeAgentId: "morgan", projectId: "launch", status: boundary ? "todo" : "backlog", parentId: null }; + const runs = [{ id: "handoff", companyId: "company", agentId: "ceo", status: "succeeded", + runtimeMode: "native", contextSnapshot: { issueId: "chat" } }]; + const comments = [ + { id: "human", issueId: "chat", body: query, authorUserId: "person", createdAt: "2026-10-30T10:00:00Z" }, + { id: "reply", issueId: "chat", body: "Assigned Morgan: EVA-2, saved in backlog.", authorAgentId: "ceo", createdByRunId: "handoff", createdAt: "2026-10-30T10:00:10Z" }, + ]; + const scopes = [{ scopeType: "company", scopeId: "company" }, ...["ceo", "morgan"].map(scopeId => ({ scopeType: "agent", scopeId }))]; + const api = { + baseURL: "http://fixture.invalid", + get: vi.fn(async (path: string) => { + if (path.endsWith("issues?includeAgentCreatedTasks=true")) return []; + if (path.includes("issues?includeAgentCreatedTasks=true&limit=100")) return [task]; + if (path.endsWith("/comments?order=asc")) return comments; + if (path.endsWith("/documents")) return []; + if (path.endsWith("/activity")) return [{ action: "issue.created", agentId: "ceo", details: { status: task.status } }]; + if (path === "/api/issues/EVA-2") return task; + if (path.includes("/events?")) return []; + if (path.endsWith("/budgets/overview")) return { policies: scopes.map(scope => ({ ...scope, + amount: 500, hardStopEnabled: true, isActive: true, unpricedUsagePolicy: "allow" })) }; + return { budgetMonthlyCents: 500, lifecycleState: "ready", status: "idle" }; + }), + post: vi.fn(async (path: string) => ({ id: path.endsWith("/projects") ? "launch" : "morgan" })), + }; + const checks: Array<{ id: string; passed: boolean }> = []; + const evidence = vi.fn(async (name: string) => { + if (boundary === "save" && name === "chat-feature.json") throw new Error("Final evidence write failed"); + }); + const locator = { + isVisible: async () => true, + locator: () => locator, getByRole: () => locator, nth: () => locator, click: vi.fn(), + evaluateAll: async (_callback: unknown, target?: unknown) => target ? 0 : ["/EVA/issues/EVA-2"], + }; + const capture = vi.fn(); + const input = { + api, nonce: "fixture", workspacePath: "/fixture", + page: { locator: () => locator, getByTestId: () => locator, reload: vi.fn(), goto: vi.fn(), url: () => "http://fixture.invalid/chat" }, + fixtures: { company: { id: "company" }, agent: { id: "ceo" }, environment: { id: "local" }, secretRefs: {} }, + execution: { environment: { id: "local" }, task: { id: caseId, expectedRunCount: boundary ? 3 : 1, + buildVisibleMarker: () => "ACORN123", turnTimeoutMs: 120_000 }, + profile: { credential: "OPENROUTER_API_KEY", buildAgent: () => ({ adapterConfig: {} }) } }, + evidence, capture, check: (id: string, passed: boolean) => checks.push({ id, passed }), + } as unknown as ChatFlowInput; + let refreshes = 0; + const context: Parameters[0] = { input, issue: () => source, allRuns: async () => runs, + refreshIssue: async () => { + if (++refreshes > 1 && boundary === "refresh") throw new Error("Final observation read failed"); + }, + }; + return { context, checks, evidence, capture }; +} + +function oldAssignment(startedAt: string | null = null): ChatRun { + return { id: "old-assignment", companyId: "company", agentId: "ceo", status: "cancelled", runtimeMode: "legacy", + startedAt, nativeIssueId: null, errorCode: "issue_assignee_changed", contextSnapshot: { issueId: "task" }, + resultJson: { stopReason: "issue_assignee_changed", timeoutSource: "stale_queued_run_gate", timeoutFired: false } }; +} + +it.each(["refresh", "save"])("keeps the last observed checks when final %s fails", async boundary => { + const { context, checks, evidence } = fixture(boundary); + await expect(runChatFeatureFlow(context)).rejects.toThrow("Original scenario failure"); + expect(evidence).toHaveBeenCalledWith("chat-feature-evidence-error.json", expect.objectContaining({ incomplete: true })); + expect(checks).toEqual(expect.arrayContaining([ + { id: "one-assigned-task", passed: true }, { id: "agent-created-task", passed: true }, + { id: "chat-task-link", passed: true }, { id: "worker-executed", passed: false }, + { id: "completion-result-access", passed: false }, + ])); + expect(checks).toHaveLength(14); +}); + +it.each([1, 3])("continues observation past a superseded wake with %s actual work runs", async count => { + const { context, evidence } = fixture("superseded"); + const runs: ChatRun[] = await context.allRuns(); + if (count === 3) runs.push( + { id: "writing", companyId: "company", agentId: "morgan", status: "queued", runtimeMode: "native", contextSnapshot: { issueId: "task" } }, + { id: "delivery", companyId: "company", agentId: "ceo", status: "queued", runtimeMode: "native", contextSnapshot: { issueId: "chat" } }, + ); + runs.push(oldAssignment()); + context.allRuns = async () => runs; + await expect(runChatFeatureFlow(context)).rejects.toThrow("Original scenario failure"); + expect(evidence).toHaveBeenCalledWith("chat-feature.json", expect.objectContaining({ + observation: expect.objectContaining({ runs }), grade: expect.objectContaining({ passed: false }), + })); +}); + +it("still rejects a cancelled assignment that actually started", async () => { + const { context } = fixture("started"); + const runs = [...await context.allRuns(), oldAssignment("2026-10-30T10:00:01Z")]; + context.allRuns = async () => runs; + await expect(runChatFeatureFlow(context)).rejects.toThrow("An observed provider run failed"); +}); + +it("retains required final evidence after the feature flow verifies browser readback", async () => { + polling.failScenario = false; + const { context, checks, evidence, capture } = fixture(); + await expect(runChatFeatureFlow(context)).resolves.toMatchObject({ issue: { id: "chat" }, runs: [{ id: "handoff" }] }); + expect(checks.every(check => check.passed)).toBe(true); + expect(checks).toContainEqual({ id: "browser-result-access", passed: true }); + expect(capture).toHaveBeenCalledWith("final-state", expect.any(String), "final-state.png"); + expect(evidence).toHaveBeenCalledWith("api-state.json", expect.objectContaining({ issue: expect.objectContaining({ id: "chat" }) })); +}); diff --git a/tests/runner-e2e/chat-feature-flow.ts b/tests/runner-e2e/chat-feature-flow.ts index 539664e49b6..105374929b8 100644 --- a/tests/runner-e2e/chat-feature-flow.ts +++ b/tests/runner-e2e/chat-feature-flow.ts @@ -4,7 +4,7 @@ import { collectChatRunEvidence, sendChatMessage, type ChatFlowInput, type ChatI import { FixtureRegistry } from "./fixture-registry.js"; import { codexSubscriptionAgentConfig } from "./codex-subscription-fixture.js"; import { CHAT_FEATURE_BUDGET_CENTS, CHAT_FEATURE_GRADER, chatFeatureScenarios, type ChatFeatureCase } from "./chat-feature-cases.js"; -import { gradeChatFeature, linksFeatureTask, type ChatFeatureObservation, type FeatureDocument, type FeatureTask } from "./chat-feature-scoring.js"; +import { chatFeatureExecutionRuns, gradeChatFeature, linksFeatureTask, type ChatFeatureObservation, type FeatureDocument, type FeatureTask } from "./chat-feature-scoring.js"; export async function runChatFeatureFlow(context: { input: ChatFlowInput; issue(): ChatIssue | undefined; allRuns(): Promise; refreshIssue(): Promise; @@ -110,9 +110,10 @@ export async function runChatFeatureFlow(context: { return Date.now() - settledAt >= 5_000; }, reject: grade => { - if (observation.runs.length > execution.task.expectedRunCount) return "Chat exceeded its declared run bound"; - if (observation.runs.some(run => ["failed", "cancelled", "timed_out"].includes(run.status))) return "An observed provider run failed"; - const finished = observation.runs.length === execution.task.expectedRunCount && observation.runs.every(run => run.status === "succeeded") + const runs = chatFeatureExecutionRuns(observation); + if (runs.length > execution.task.expectedRunCount) return "Chat exceeded its declared run bound"; + if (runs.some(run => ["failed", "cancelled", "timed_out"].includes(run.status))) return "An observed provider run failed"; + const finished = runs.length === execution.task.expectedRunCount && runs.every(run => run.status === "succeeded") && observation.source?.conversationState === "waiting"; if (!finished) finishedWorkAt = undefined; else finishedWorkAt ??= Date.now(); @@ -143,9 +144,9 @@ export async function runChatFeatureFlow(context: { expect((await api.get(`/api/issues/${encodeURIComponent(savedTask.identifier ?? savedTask.id)}`)).id).toBe(savedTask.id); await page.goto(sourceUrl, { waitUntil: "domcontentloaded" }); await expect(page.getByTestId("task-chat-composer-input")).toBeVisible(); - await input.capture("chat-feature-result", "Created task and retained source conversation", "chat-feature-result.png"); - for (const check of gradeChatFeature(observation).checks) input.check?.(check.id, check.passed, check.detail); + await input.capture("final-state", "Created task and retained source conversation", "final-state.png"); input.check?.("browser-result-access", true, "The exact linked task loaded in the browser and the source conversation survived reload."); + return { issue: observation.source!, runs: chatFeatureExecutionRuns(observation) }; } catch (error) { failure = error; throw error; @@ -153,11 +154,18 @@ export async function runChatFeatureFlow(context: { try { await refresh(); await saveEvidence(); - await input.evidence("chat-feature-runs.json", await Promise.all(observation.runs.map(run => collectChatRunEvidence(api, run)))); + const runEvidence = await Promise.all(observation.runs.map(run => collectChatRunEvidence(api, run))); + await input.evidence("chat-feature-runs.json", runEvidence); + await input.evidence("api-state.json", { issue: observation.source, runs: observation.runs, + comments: observation.comments, activity: observation.activity, runEvidence }); } catch (error) { if (!failure) throw error; await input.evidence("chat-feature-evidence-error.json", { incomplete: true, message: error instanceof Error ? error.message : "Evidence collection failed" }); - } finally { await fixtures.teardown(); } + } finally { + try { + for (const check of gradeChatFeature(observation).checks) input.check?.(check.id, check.passed, check.detail); + } finally { await fixtures.teardown(); } + } } } diff --git a/tests/runner-e2e/chat-feature-scoring.test.ts b/tests/runner-e2e/chat-feature-scoring.test.ts index f1cbb10b900..f76bec3ca6b 100644 --- a/tests/runner-e2e/chat-feature-scoring.test.ts +++ b/tests/runner-e2e/chat-feature-scoring.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from "vitest"; -import { gradeChatFeature, type ChatFeatureObservation } from "./chat-feature-scoring.js"; +import { chatFeatureExecutionRuns, gradeChatFeature, type ChatFeatureObservation } from "./chat-feature-scoring.js"; import { chatFeatureScenarios, readChatFeatureScenarios } from "./chat-feature-cases.js"; import { runnerMatrix } from "./catalog.js"; import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js"; @@ -32,11 +32,42 @@ function outcome(delegated = false): ChatFeatureObservation { return input; } const failures = (input: ChatFeatureObservation) => gradeChatFeature(input).checks.filter(check => !check.passed).map(check => check.id); +function withSupersededWake(input = outcome(true)) { + input.runs.push({ id: "old-assignment", companyId: "company", agentId: "ceo", status: "cancelled", runtimeMode: "legacy", + startedAt: null, errorCode: "issue_assignee_changed", contextSnapshot: { issueId: "task" }, + resultJson: { stopReason: "issue_assignee_changed", timeoutSource: "stale_queued_run_gate", timeoutFired: false } }); + return input; +} describe("chat feature saved-state oracle", () => { it.each([false, true])("accepts a complete %s delegation outcome", delegated => { expect(failures(outcome(delegated))).toEqual([]); }); + it("accepts a completed delegation after an obsolete assignment wake was cancelled before execution", () => { + const input = withSupersededWake(); + expect(failures(input)).toEqual([]); + expect(chatFeatureExecutionRuns(input).map(run => run.id)).toEqual(["handoff", "writing", "delivery"]); + expect(input.runs).toHaveLength(4); + }); + it("keeps an unfinished delegation failed after a harmless queued cancellation", () => { + const input = withSupersededWake(); + input.runs.splice(1, 2); + input.tasks[0]!.status = "todo"; + input.documents = []; + input.comments.pop(); + expect(chatFeatureExecutionRuns(input).map(run => run.id)).toEqual(["handoff"]); + expect(failures(input)).toEqual(expect.arrayContaining(["worker-executed", "grounded-worker-document", "completion-source-response"])); + }); + it.each([ + { startedAt: "2026-10-26T10:00:15Z" }, { startedAt: undefined }, { errorCode: "cancelled" }, + { companyId: "foreign" }, { agentId: "unrelated" }, { contextSnapshot: { issueId: "other-task" } }, + { contextSnapshot: { issueId: "chat" } }, { resultJson: {} }, { nativeIssueId: "task" }, + ])("rejects cancelled work without proof of an obsolete unstarted handoff: %j", patch => { + const input = withSupersededWake(); + Object.assign(input.runs.at(-1)!, patch); + expect(chatFeatureExecutionRuns(input)).toHaveLength(4); + expect(failures(input)).toContain("bounded-native-work"); + }); it.each([ ["duplicate task", (o: ChatFeatureObservation) => o.tasks.push({ ...o.tasks[0]!, id: "duplicate" }), "one-assigned-task"], ["wrong company", (o: ChatFeatureObservation) => { o.tasks[0]!.companyId = "foreign"; }, "one-assigned-task"], @@ -76,12 +107,12 @@ describe("chat feature saved-state oracle", () => { }); describe("chat feature selection", () => { - it("registers only the two authored queries as explicit local subscription cells", () => { + it("registers the two authored queries as explicit local subscription and OpenCode cells", () => { const cells = runnerMatrix.filter(cell => cell.suite.id === "chat-features"); - expect(cells).toHaveLength(2); + expect(cells).toHaveLength(4); for (const cell of cells) { expect(cell.environment.id).toBe("local"); - expect(cell.requiredCredentials).toEqual(["CODEX_AUTH_JSON"]); + expect(cell.requiredCredentials).toEqual([cell.profile.id === "runner-opencode" ? "OPENROUTER_API_KEY" : "CODEX_AUTH_JSON"]); expect(cell.task.automaticRetryPolicy).toBe("single_attempt"); expect(cell.task.buildPrompt("nonce")).toBe(chatFeatureScenarios.find(row => row.id === `tasks.${cell.task.id}`)!.query); expect(selectRunnerExecutions(parseRunnerSelectors(["--id", cell.id])).map(row => row.id)).toEqual([cell.id]); diff --git a/tests/runner-e2e/chat-feature-scoring.ts b/tests/runner-e2e/chat-feature-scoring.ts index cc8a0de202e..fd362f4614d 100644 --- a/tests/runner-e2e/chat-feature-scoring.ts +++ b/tests/runner-e2e/chat-feature-scoring.ts @@ -1,5 +1,6 @@ import { completionDelivery } from "./completion-updates.js"; import type { ChatFeatureCase } from "./chat-feature-cases.js"; +import { isReassignedUnstartedWake } from "./non-execution-wake.js"; export interface FeatureTask { id: string; companyId: string; title: string; description?: string | null; identifier?: string; status: string; @@ -8,6 +9,7 @@ export interface FeatureTask { } export interface FeatureRun { id: string; companyId: string; agentId: string; status: string; runtimeMode?: string; + errorCode?: string | null; nativeIssueId?: string | null; startedAt?: string | null; finishedAt?: string | null; contextSnapshot?: Record; resultJson?: Record; } @@ -45,13 +47,26 @@ export function announcementMatchesBrief(body: string, marker: string): boolean && sentences.length === 2 && sentences.every(sentence => /[.!?]["'”’]*$/.test(sentence)); } +/** Keep obsolete queue entries in the observation, separate from actual work. + * Only the CEO's prior assignment on the one task now owned by Morgan qualifies. + * Backlog creation must never queue work, even if that wake is later cancelled. */ +export function chatFeatureExecutionRuns(input: ChatFeatureObservation): FeatureRun[] { + if (input.caseId !== "delegate-announcement" || input.tasks.length !== 1) return input.runs; + const task = input.tasks[0]!; + return input.runs.filter(run => !(isReassignedUnstartedWake(run) + && run.companyId === input.companyId && task.companyId === input.companyId + && run.agentId === input.leadId && task.assigneeAgentId === input.workerId + && task.id !== input.source?.id && run.contextSnapshot?.issueId === task.id)); +} + export function gradeChatFeature(input: ChatFeatureObservation) { const checks: Array<{ id: string; passed: boolean; detail: string }> = []; const add = (id: string, passed: boolean, detail: string) => checks.push({ id, passed, detail }); const task = input.tasks[0]; const source = input.source; - const taskRuns = task ? input.runs.filter(run => (run.nativeIssueId ?? run.contextSnapshot?.issueId) === task.id) : []; - const sourceRuns = input.runs.filter(run => source && (run.nativeIssueId ?? run.contextSnapshot?.issueId) === source.id); + const runs = chatFeatureExecutionRuns(input); + const taskRuns = task ? runs.filter(run => (run.nativeIssueId ?? run.contextSnapshot?.issueId) === task.id) : []; + const sourceRuns = runs.filter(run => source && (run.nativeIssueId ?? run.contextSnapshot?.issueId) === source.id); const savedQuery = input.comments.filter(comment => !comment.authorAgentId && comment.body.trim() === input.query.trim()); const replies = input.comments.filter(comment => comment.authorAgentId === input.leadId && sourceRuns.some(run => run.id === comment.createdByRunId)); add("one-human-request", savedQuery.length === 1 && input.comments.filter(comment => !comment.authorAgentId).length === 1, @@ -65,10 +80,10 @@ export function gradeChatFeature(input: ChatFeatureObservation) { const creations = input.activity.filter(event => event.action === "issue.created"); add("agent-created-task", creations.length === 1 && creations[0]!.agentId === input.leadId, "The CEO's actual creation audit proves the harness did not create the result."); - add("bounded-native-work", input.runs.length === (input.caseId === "create-backlog" ? 1 : 3) - && input.runs.every(run => run.companyId === input.companyId && run.runtimeMode === "native" && run.status === "succeeded") - && input.runs.length === sourceRuns.length + taskRuns.length, - "All company runs are accounted for, native, succeeded, and bound to the source or delegated task."); + add("bounded-native-work", runs.length === (input.caseId === "create-backlog" ? 1 : 3) + && runs.every(run => run.companyId === input.companyId && run.runtimeMode === "native" && run.status === "succeeded") + && runs.length === sourceRuns.length + taskRuns.length, + "All work runs are native, succeeded, and task-bound; proven superseded unstarted assignments remain in the ledger."); add("conversation-retained", !!source && source.companyId === input.companyId && source.assigneeAgentId === input.leadId && source.status === "in_review" && source.conversationState === "waiting" && !source.scheduledRetry && !source.activeRecoveryAction, "The source conversation remains available and settled without recovery."); diff --git a/tests/runner-e2e/chat-flow.test.ts b/tests/runner-e2e/chat-flow.test.ts index ceff2c91a87..bbe41379f84 100644 --- a/tests/runner-e2e/chat-flow.test.ts +++ b/tests/runner-e2e/chat-flow.test.ts @@ -293,6 +293,20 @@ If you only have the club name, audience, and tone, that is enough to begin; I c [`/api/heartbeat-runs/${run.id}/events?afterSeq=1000&limit=1000`], ]); }); + it("retains a superseded queued run's events without requesting a provider log that was never created", async () => { + const get = vi.fn().mockResolvedValue([{ seq: 1, eventType: "lifecycle" }]); + const suppressed = { ...run, status: "cancelled", errorCode: "issue_assignee_changed", startedAt: null, + resultJson: { stopReason: "issue_assignee_changed", timeoutSource: "stale_queued_run_gate", timeoutFired: false } }; + expect(await collectChatRunEvidence({ get }, suppressed)).toMatchObject({ + runId: run.id, log: null, logOmissionReason: "assignee_changed_before_start", events: [{ seq: 1, eventType: "lifecycle" }], + }); + expect(get.mock.calls).toEqual([[`/api/heartbeat-runs/${run.id}/events?afterSeq=0&limit=1000`]]); + get.mockRejectedValue(new Error("Run log not found")); + for (const patch of [{ startedAt: "2026-10-26T10:00:15Z" }, { startedAt: undefined }, + { errorCode: "cancelled" }, { resultJson: {} }, { nativeIssueId: "task" }]) { + await expect(collectChatRunEvidence({ get }, { ...suppressed, ...patch })).rejects.toThrow("Run log not found"); + } + }); it("records legitimate pre-provider log absence only with explicit bootstrap and storage evidence", async () => { const stopped = { ...run, status: "interrupted", runtimeMode: "legacy", runtimeModeResolvedAt: null, logStore: null, logRef: null, resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } }; diff --git a/tests/runner-e2e/chat-flow.ts b/tests/runner-e2e/chat-flow.ts index d06bd9c7663..f529ca2f781 100644 --- a/tests/runner-e2e/chat-flow.ts +++ b/tests/runner-e2e/chat-flow.ts @@ -10,7 +10,7 @@ import type { } from "../../packages/shared/src/types/issue.js"; import type { LiveFixtureValues } from "./live-fixtures.js"; import type { MatrixExecution } from "./types.js"; -import { isBlockedUnstartedWake } from "./non-execution-wake.js"; +import { isBlockedUnstartedWake, isReassignedUnstartedWake } from "./non-execution-wake.js"; import { collectRunEvents } from "./run-observations.js"; import { chatMarker } from "./chat-cases.js"; import { assertChatRememberedAfterRestart, assertChatStartupStopped, isChatStopReady, runChatHardeningFlow } from "./chat-hardening.js"; @@ -68,6 +68,7 @@ export interface ChatRun { sessionIdBefore?: string | null; sessionIdAfter?: string | null; startedAt?: string | null; + nativeIssueId?: string | null; } type Comment = { id: string; @@ -296,10 +297,12 @@ export async function collectChatRunEvidence( const providerNeverStarted = run.runtimeMode === "legacy" && !run.runtimeModeResolvedAt && run.logStore === null && run.logRef === null && ["failed", "interrupted"].includes(run.status) && recovery?.kind === "bootstrap" && recovery.providerWorkStarted === false; + const reassignedBeforeStart = isReassignedUnstartedWake(run); return { runId: run.id, ...(providerNeverStarted ? { logOmissionReason: "provider_not_started" } : {}), - log: isResetRun(run) || isBlockedUnstartedWake({ ...run }) || providerNeverStarted + ...(reassignedBeforeStart ? { logOmissionReason: "assignee_changed_before_start" } : {}), + log: isResetRun(run) || isBlockedUnstartedWake({ ...run }) || providerNeverStarted || reassignedBeforeStart ? null : await api.get(`/api/heartbeat-runs/${run.id}/log?limitBytes=1048576`), events: await collectRunEvents((afterSeq, limit) => @@ -455,7 +458,7 @@ export async function runChatFlow(input: ChatFlowInput) { expect(await allRuns()).toHaveLength(0); if (execution.suite.id === CHAT_FEATURE_SUITE) { - await runChatFeatureFlow({ input, issue: () => issue, allRuns, + return await runChatFeatureFlow({ input, issue: () => issue, allRuns, refreshIssue: async () => { issue = await api.get(chatPath); if (issue) input.observe(issue, await allRuns()); } }); } else if (caseId === "confirmation-ambiguous") { await runAmbiguousConfirmationReply({ input, issue: () => issue!, idle, comments }); diff --git a/tests/runner-e2e/non-execution-wake.ts b/tests/runner-e2e/non-execution-wake.ts index 4e3cd1b81d9..4518ff391ac 100644 --- a/tests/runner-e2e/non-execution-wake.ts +++ b/tests/runner-e2e/non-execution-wake.ts @@ -4,6 +4,16 @@ export function isBlockedUnstartedWake(run: Record) { return run.status === "cancelled" && run.errorCode === "issue_dependencies_blocked" && run.startedAt === null; } +/** The dispatch gate retired an obsolete assignment before any provider work. */ +export function isReassignedUnstartedWake(run: { + status?: unknown; errorCode?: unknown; startedAt?: unknown; nativeIssueId?: unknown; + resultJson?: Record; +}) { + return run.status === "cancelled" && run.errorCode === "issue_assignee_changed" && run.startedAt === null + && run.nativeIssueId == null && run.resultJson?.stopReason === "issue_assignee_changed" + && run.resultJson.timeoutSource === "stale_queued_run_gate" && run.resultJson.timeoutFired === false; +} + /** A terminal issue may make a queued wake obsolete before provider admission. */ export function isTerminalUnstartedWake( run: Record,