diff --git a/doc/evals.md b/doc/evals.md index f6004db4b7a..d67d4c3b66a 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -447,3 +447,13 @@ See the [suite contract](../tests/runner-e2e/README.md#conversational-confirmati Hiring notification accounting now also requires exact completed action attribution. Missing native/provider ID mapping is uncomparable evidence; it must not be reported as a model task regression or waived through name/order matching. The fixture waits for both known completion callbacks and settled bracketed observations, including the gap before pending outbox work becomes a wake. Strict action replay and original machine verdicts are retained separately. The explicit-only [planning guidance utility comparison](../tests/runner-e2e/PLAN-TASK-GUIDANCE.md) measures task decomposition and handoffs with current, short, and disabled skills. + +The native interaction suite also declares one local `native-question-batch-stop` +cell. It leaves a complete native callback unanswered, retains the pending +request, clicks the browser Stop control, and independently requires the exact +cancelled callback/turn and audited public Stop acknowledgement. The task must +remain In Progress, a late answer must receive HTTP 409, and the public per-turn +process owner and observed descendants must retire before cleanup and remain +retired through cleanup. Cancelled-run account attribution and complete reported +OpenRouter billing are required. This is a separate pending release gate; remote +Stop still requires its own retirement observer and actual Daytona proof. diff --git a/doc/plans/2026-10-06-hermes-native-runner.md b/doc/plans/2026-10-06-hermes-native-runner.md index 128d890025b..588602fdc02 100644 --- a/doc/plans/2026-10-06-hermes-native-runner.md +++ b/doc/plans/2026-10-06-hermes-native-runner.md @@ -1310,3 +1310,30 @@ prohibitions and the current completion claim shape. An admission test calls the actual production file/document guard. Product E2E typecheck, all 1,954 Vitest tests with one existing skip, and exact discovery pass for this correction. The production guard is unchanged. A fresh live attempt remains required. + +### 2026-10-08 — native question browser acceptance and Stop gate + +At source `6f08ecceda0afcf3e3d3ae0ba5043d82add96ac7`, corrected campaign +`hermes-macos-native-question-corrected-6f08ecceda-20261008-attempt1` passes all +25 checks and cleanup in 64.270 seconds. One original native run retains a +three-question callback across browser reload, receives the exact single, +multiple, custom and undisclosed text answers once, rejects a late duplicate +with HTTP 409 and completes with the returned text. Public provider-reported +cost is `$0.002484928`. Inspected screenshots and the canonical packaged +evidence have no missing entries or leaks. This is local native question proof. +The earlier failed attempt retains its original grade and unknown per-run cost; +aggregate provider-key usage is not a substitute settlement receipt. + +Both current-source cloud native jobs pass nine fixtures and 38 pinned Python +checks without credentials. The cloud merge tree equals the recorded PR head; +55 checks pass with two skips, fresh Greptile 5/5 and no open review threads. +These checks do not qualify subscriptions, Bedrock or actual Daytona. + +The next explicit local cell leaves the native question unanswered and clicks +the real browser Stop control. It must retain and bind the pending callback, +require one cancelled request and turn with an audited same-scope Stop receipt, +observe expiration and stale-answer refusal, preserve the unfinished task, +settle cancelled-run reported usage, and prove the public per-turn owner and +observed descendants retire before and through cleanup. This cell is pending +live qualification. Remote Stop still needs its own remote retirement observer. +No local Docker or Rust build is used. diff --git a/tests/runner-e2e/FIXTURES.md b/tests/runner-e2e/FIXTURES.md index bb37e504d6d..497c00d2765 100644 --- a/tests/runner-e2e/FIXTURES.md +++ b/tests/runner-e2e/FIXTURES.md @@ -385,3 +385,13 @@ as completed and end the native turn; Paperclip must retain a failed run with missing semantic finalization and an unfinished task. That is a denial outcome, not task success or operator cancellation. Stop during an unresolved permission remains a separate `native-active-stop/pending-permission-stop` gate. + +`native-question-batch-stop` adds a separate local native Stop case. It retains +one pending Hermes callback before the browser click, never submits its answers, +and grades exact callback cancellation, turn cancellation and same-scope audited +Stop acknowledgement. It requires the original card to expire, the task to stay +In Progress, a stale answer to receive HTTP 409, and no follow-up run. The public +run PID/group/start identity supplies ownership for a read-only descendant +journal; observed owners must retire before cleanup and stay retired through it. +Cancelled-run cost settlement remains required. Remote Stop is a separate pending +gate and cannot pass from this local observation. diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index a4708f7f3d6..3f5ada0e47b 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -1590,8 +1590,8 @@ pnpm test:e2e:runner -- --list --suite hermes-bedrock-connections pnpm test:e2e:runner -- --id hermes-bedrock-connections.runner-acpx-hermes-bedrock.local.hello-complete --max-automatic-retries 0 ``` -The explicit-only `hermes-native-interactions` suite declares two pending cells -for `native-question-batch-reconnect`, on local and Daytona. One native Hermes +The explicit-only `hermes-native-interactions` suite declares question/reconnect +cells on local and Daytona, plus a separate local native Stop cell. One native Hermes `clarify` call asks an ordered batch of single choice, multiple choice with a custom answer, and free text. The browser reloads while the original run remains active, checks the same durable request and complete form, then submits all @@ -1850,3 +1850,20 @@ Private traces and database files must not be published. Unconfirmed cleanup always preserves recovery state regardless of this optional diagnostic flag. The `file-edit-validate` fixture independently downloads the exact active artifact work product from the tested run. It requires the matching run-attributed attachment, filename, MIME type, recorded byte count and SHA-256, then compares the downloaded content to the expected bytes. A workspace file alone cannot satisfy this gate. Explicit failed-case diagnostic retention follows the final result after integrity, isolation and evidence checks, including incomplete publication. + +The suite now declares three explicit cells: question/reconnect on local and +Daytona, plus local `native-question-batch-stop`. The Stop cell uses the real +browser composer control while the complete native question remains unanswered. +It binds the cancellation response to the retained request and exact native turn, +requires an audited run-only cancellation and an expired unanswerable card, and +rejects a late answer with HTTP 409. One cancelled run must leave the task In +Progress. A read-only PID/start-time journal captures the public per-turn owner +and descendants before Stop and verifies retirement before and through cleanup. +Its selected account/model and reported wire usage must settle with healthy +200-cent company and agent budgets. One attempt, zero retries, one provider turn. +Remote Stop is deliberately excluded until its remote retirement observer exists; +the release requirement remains pending. + +```sh +pnpm test:e2e:runner -- --id hermes-native-interactions.runner-acpx-hermes.local.native-question-batch-stop --max-automatic-retries 0 +``` diff --git a/tests/runner-e2e/SECURITY.md b/tests/runner-e2e/SECURITY.md index f5257d9899b..4a54605ade7 100644 --- a/tests/runner-e2e/SECURITY.md +++ b/tests/runner-e2e/SECURITY.md @@ -341,3 +341,13 @@ bodies are withheld. Source admission precedes credential handoff, both public budgets are 200 cents, OpenRouter cost must settle, and automatic retries remain disabled. Registered local/Daytona cells are pending qualification, not permission to launch a paid campaign or a claim of remote coverage. + +The local native question Stop cell reads the OS process table only to observe +the isolated run's public PID/group/start identity and descendants. It records +PID/start-time evidence; command lines stay in memory. The fixture never signals +those PIDs. Cancellation goes through the normal browser Stop control and public +board API. No question answer is delivered; the only response attempt is a late +fixture answer that must be rejected. Retained evidence uses the existing +sanitizer and screenshot publication gates. The cell reuses the selected key +allocation and 200-cent budgets, with zero automatic retries. It neither creates +remote resources nor grants permission to use another secret. diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index 5caa8b6ce72..02ce437573b 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -146,10 +146,10 @@ describe("runner E2E catalog", () => { expect(localIntegrityTasks).toHaveLength(2); expect(openRouterBreadthTasks).toHaveLength(3); expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([ - 63, 12, 6, 8, 2, 2, 2, 40, 2, 10, 2, 3, 16, 16, 2, 6, 8, 46, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, 116, + 63, 12, 6, 8, 2, 2, 2, 40, 3, 10, 2, 3, 16, 16, 2, 6, 8, 46, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, 116, ]); - expect(validateRunnerCatalog()).toHaveLength(722); - expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(722); + expect(validateRunnerCatalog()).toHaveLength(723); + expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(723); expect( runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"), ).toHaveLength(48); diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index a89de106d04..acf54a5cc54 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -5,7 +5,7 @@ import { nativeCompletionProfile, NATIVE_COMPLETION_BUDGET_CENTS } from "./nativ import { chatConfirmationTasks } from "./chat-cases.js"; import { buildConnectionSuite } from "./connection-cases.js"; import { hiringTemplateTasks, hiringTemplateProfile, hiringTemplateDefinitionDigest } from "./hiring-template-cases.js"; -import { HERMES_API_CONNECTION_BUDGET_CENTS, HERMES_NATIVE_INTERACTION_SUITE, hermesApiConnectionChoices, hermesApiConnectionDefinitionDigest, hermesBedrockConnectionChoice, hermesNativeQuestionTask } from "./hermes-api-connections.js"; +import { HERMES_API_CONNECTION_BUDGET_CENTS, HERMES_NATIVE_INTERACTION_SUITE, hermesApiConnectionChoices, hermesApiConnectionDefinitionDigest, hermesBedrockConnectionChoice, hermesNativeQuestionTask, hermesNativeQuestionStopTask } from "./hermes-api-connections.js"; import { nativeActiveStopTasks } from "./native-active-stop-tasks.js"; import { cursorNativeTasks } from "./cursor-native-cases.js"; import { instructionPersistenceTask } from "./instruction-persistence.js"; @@ -1215,15 +1215,18 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ }, { id: HERMES_NATIVE_INTERACTION_SUITE, label: "Hermes native interactions", manualOnly: true, - description: "A native Hermes question batch retains its callback through browser reconnect and receives the exact reviewer's answers within one active turn.", + description: "A native Hermes question batch receives exact answers through reconnect; browser Stop cancels an unanswered local callback and retires its owned process tree.", groups: ["native"], profiles: [extendedHarnessProfiles.find(profile => profile.qualificationCandidate === "hermes")!], - environments: runnerEnvironments, tasks: [hermesNativeQuestionTask], expectedMatrixSize: 2, + environments: runnerEnvironments, tasks: [hermesNativeQuestionTask, hermesNativeQuestionStopTask], + excludedExecutionIds: [`${HERMES_NATIVE_INTERACTION_SUITE}.runner-acpx-hermes.daytona.native-question-batch-stop`], expectedMatrixSize: 3, definitionMetadata: { - version: 2, qualification: "pending", scheduling: "explicit-only", accountMethod: "api_key", accountMode: "responsible_user", + version: 3, qualification: "pending", scheduling: "explicit-only", accountMethod: "api_key", accountMode: "responsible_user", providerTurns: 1, lifecycle: "per-turn", nativeMethod: "_hermes/ask_questions", maximumAttemptsPerCell: 1, objectiveAdmission: "production-delivery-guard-question-only", budgetMonthlyCents: HERMES_API_CONNECTION_BUDGET_CENTS, billing: "reported-cost-and-budget-health", - coverage: "native-question-batch-browser-reconnect-exact-delivery", sourceDigest: hermesApiConnectionDefinitionDigest, + coverage: "native-question-batch-browser-reconnect-exact-delivery-and-local-stop", + stopBoundary: "retained-unanswered-native-question-before-browser-click", stopProcessEvidence: "local-public-per-turn-owner-and-descendants-through-cleanup", + remoteStopQualification: "pending-separate-remote-retirement-observer", sourceDigest: hermesApiConnectionDefinitionDigest, }, }, { diff --git a/tests/runner-e2e/hermes-api-connections.test.ts b/tests/runner-e2e/hermes-api-connections.test.ts index 8868f31f174..b09d530fc03 100644 --- a/tests/runner-e2e/hermes-api-connections.test.ts +++ b/tests/runner-e2e/hermes-api-connections.test.ts @@ -3,7 +3,7 @@ import { runnerMatrix, runnerSuites, suiteDefinitionHash } from "./catalog.js"; import { buildMatrixJobs, parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js"; import { buildRunnerE2EProcessEnvironment } from "./harness-env.js"; import { explicitlyRequestsFileOutput, explicitlyRequestsTaskDocumentOutput } from "../../server/src/services/native-runtime/native-deliverable-feedback.js"; -import { captureHermesApiAccountOwner, captureHermesApiBudgets, captureHermesOpenRouterSettlement, gradeHermesApiConnection, isHermesOpenRouterWorkflow, isHermesConnectionSuite, HERMES_NATIVE_INTERACTION_SUITE, hasExactHermesNativeQuestionResponse, hasHermesNativeQuestionBatch, hermesNativeAnswerText } from "./hermes-api-connections.js"; +import { captureHermesApiAccountOwner, captureHermesApiBudgets, captureHermesOpenRouterSettlement, gradeHermesApiConnection, isHermesOpenRouterWorkflow, isHermesConnectionSuite, HERMES_NATIVE_INTERACTION_SUITE, hasExactHermesNativeQuestionResponse, hasHermesNativeQuestionBatch, hermesNativeAnswerText, hasHermesNativeQuestionStop } from "./hermes-api-connections.js"; describe("Hermes native browser questions", () => { const suite = runnerSuites.find(s => s.id === HERMES_NATIVE_INTERACTION_SUITE)!; @@ -21,6 +21,7 @@ describe("Hermes native browser questions", () => { const event = (eventType: string, sourceSeq: number, payload: unknown) => ({ runId: "run", protocolSchemaVersion: 1, payload: { prpEvent: { schema: "paperclip.prp.event.v1", schemaVersion: 1, runId: "run", turnId: "turn", eventType, sourceSeq, payload, + sourceKind: "runner", sourceInstanceId: "instance", normalizedSessionId: "session", } }, }); const native = (adapter = "acpx-runtime-sidecar") => [ @@ -29,11 +30,12 @@ describe("Hermes native browser questions", () => { event("runtime_request.resolved", 3, { requestId: "request", turnId: "turn", action: "submit", response }), ]; const grade = (events: unknown[]) => hasExactHermesNativeQuestionResponse({ events, runId: "run", turnId: "turn", requestId: "request", questionSet, response }); - it("declares only two explicit bounded single-attempt managed-account cells", () => { - expect(cells).toHaveLength(2); + it("declares three explicit bounded single-attempt cells with local-only Stop ownership proof", () => { + expect(cells).toHaveLength(3); expect(suite.manualOnly).toBe(true); + expect(cells.filter(cell => cell.task.flow === "native_question_stop").map(cell => cell.environment.id)).toEqual(["local"]); expect(new Set(cells.map(cell => cell.environment.id))).toEqual(new Set(["local", "daytona"])); - expect(cells.every(cell => cell.profile.qualificationCandidate === "hermes" && cell.task.flow === "native_question_completion" + expect(cells.every(cell => cell.profile.qualificationCandidate === "hermes" && ["native_question_completion", "native_question_stop"].includes(cell.task.flow) && cell.task.expectedRunCount === 1 && cell.task.automaticRetryPolicy === "single_attempt")).toBe(true); expect(suite.definitionMetadata).toMatchObject({ qualification: "pending", providerTurns: 1, maximumAttemptsPerCell: 1, budgetMonthlyCents: 200, lifecycle: "per-turn", nativeMethod: "_hermes/ask_questions", billing: "reported-cost-and-budget-health" }); @@ -56,6 +58,76 @@ describe("Hermes native browser questions", () => { expect(explicitlyRequestsFileOutput("Write a JSON file with the returned answers.")).toBe(true); expect(explicitlyRequestsTaskDocumentOutput("Write a document on this task with the returned answers.")).toBe(true); }); + const stopped = () => { + const events = [native()[0]!, + event("runtime_request.cancelled", 3, { requestId: "request", turnId: "turn", itemId: "clarify-tool", requestKind: "runtime", action: "cancel", reason: "provider request aborted" }), + event("turn.cancelled", 4, { status: "cancelled", error: null }), + ]; + Object.assign((events[0]!.payload.prpEvent.payload as Record).request, { itemId: "clarify-tool", requestKind: "runtime" }); + const run = { id: "run", companyId: "company", issueId: "task", runtimeMode: "native", status: "cancelled", resultJson: { + cancelledByActorType: "user", cancelledByUserId: "caller", nativeCancellation: { + schema: "paperclip.native-cancellation.v1", companyId: "company", runId: "run", issueId: "task", scope: "run", + dispatched: true, dispatchState: "acknowledged", reasonCode: "cancellation_run_only", effects: ["release_run_resources"], + intentId: "intent", intentAuditId: "audit-intent", acknowledgementAuditId: "audit-ack", + }, + } }; + return { events, run, pendingEvent: structuredClone(events[0]!.payload.prpEvent), issue: { id: "task", companyId: "company", status: "in_progress" }, + runId: "run", turnId: "turn", requestId: "request", questionSet, companyId: "company", callerUserId: "caller", browserResponse: structuredClone(run) }; + }; + it("requires the native callback cancellation and exact browser-owned Stop acknowledgement", () => { + expect(hasHermesNativeQuestionStop(stopped())).toBe(true); + }); + it.each(["missing-created", "missing-closure", "missing-terminal", "duplicate-created", "duplicate-closure", "duplicate-terminal", "second-request", + "completed", "failed", "interrupted", "resolved", "expired", "answer", "replay", "wrong-action", "wrong-order", "wrong-schema", "wrong-version", "wrong-protocol", + "foreign-row", "foreign-event", "foreign-turn", "foreign-session", "foreign-source", "not-runner", "changed-pending", "changed-input", "wrong-provider", "semantic-tool", + "post-stop-tool", "post-stop-question", "task-completed", "foreign-company", "foreign-task", "foreign-caller", "wrong-actor", "run-succeeded", + "missing-ack", "wrong-scope", "wrong-effects", "no-audit", "same-audit", "wrong-browser-intent", "wrong-browser-run"])("rejects native Stop fault: %s", fault => { + const value = stopped() as Record; + const created = value.events[0].payload.prpEvent, closed = value.events[1].payload.prpEvent, terminal = value.events[2].payload.prpEvent; + const request = created.payload.request, stop = value.run.resultJson.nativeCancellation; + if (fault === "missing-created") value.events.splice(0,1); + if (fault === "missing-closure") value.events.splice(1,1); + if (fault === "missing-terminal") value.events.splice(2,1); + if (fault === "duplicate-created") value.events.push(structuredClone(value.events[0])); + if (fault === "duplicate-closure") value.events.push(structuredClone(value.events[1])); + if (fault === "duplicate-terminal") value.events.push(structuredClone(value.events[2])); + if (fault === "second-request") { const extra = structuredClone(value.events[0]); extra.payload.prpEvent.payload.request.requestId="other"; value.events.push(extra); } + if (["completed","failed","interrupted"].includes(fault)) terminal.eventType=`turn.${fault}`; + if (["resolved","expired"].includes(fault)) closed.eventType=`runtime_request.${fault}`; + if (fault === "answer") closed.payload.response=response; + if (fault === "replay") closed.payload.replayAllowed=true; + if (fault === "wrong-action") closed.payload.action="submit"; + if (fault === "wrong-order") terminal.sourceSeq=2; + if (fault === "wrong-schema") closed.schema="made-up"; + if (fault === "wrong-version") closed.schemaVersion=2; + if (fault === "wrong-protocol") value.events[1].protocolSchemaVersion=2; + if (fault === "foreign-row") value.events[1].runId="foreign"; + if (fault === "foreign-event") closed.runId="foreign"; + if (fault === "foreign-turn") closed.turnId="foreign"; + if (fault === "foreign-session") closed.normalizedSessionId="foreign"; + if (fault === "foreign-source") closed.sourceInstanceId="foreign"; + if (fault === "not-runner") closed.sourceKind="control_plane"; + if (fault === "changed-pending") value.pendingEvent.sourceSeq=1; + if (fault === "changed-input") request.input={}; + if (fault === "wrong-provider") request.origin.provider="cursor"; + if (fault === "semantic-tool") request.origin.method="request_human_input"; + if (fault === "post-stop-tool") value.events.push(event("tool.execution.started",5,{})); + if (fault === "post-stop-question") value.events.push(event("runtime_request.created",5,{request:{}})); + if (fault === "task-completed") value.issue.status="done"; + if (fault === "foreign-company") value.run.companyId="foreign"; + if (fault === "foreign-task") value.run.issueId="foreign"; + if (fault === "foreign-caller") value.run.resultJson.cancelledByUserId="foreign"; + if (fault === "wrong-actor") value.run.resultJson.cancelledByActorType="agent"; + if (fault === "run-succeeded") value.run.status="succeeded"; + if (fault === "missing-ack") stop.dispatchState="pending"; + if (fault === "wrong-scope") stop.scope="agent"; + if (fault === "wrong-effects") stop.effects=["cancel_task"]; + if (fault === "no-audit") delete stop.acknowledgementAuditId; + if (fault === "same-audit") stop.acknowledgementAuditId=stop.intentAuditId; + if (fault === "wrong-browser-intent") value.browserResponse.resultJson.nativeCancellation.intentId="foreign"; + if (fault === "wrong-browser-run") value.browserResponse.id="foreign"; + expect(hasHermesNativeQuestionStop(value as Parameters[0])).toBe(false); + }); it.each(["acpx-runtime", "acpx-runtime-sidecar"])("accepts a complete native delivery from %s", adapter => { expect(hasHermesNativeQuestionBatch(questionSet)).toBe(true); expect(grade(native(adapter))).toBe(true); @@ -161,7 +233,13 @@ describe("Hermes managed API connection qualification", () => { it("accepts independently observed account/model metadata", () => { expect(gradeHermesApiConnection(valid).every(check => check.passed)).toBe(true); }); - it.each(["valid", "reported-zero", "company-scope", "agent-scope", "run-scope", "task-scope", "paused-agent", "paused-company", "pause-reason", "budget-changed", + it("grades cancelled account metadata only when cancellation is explicitly expected", () => { + const value = { ...valid, runs: [{ ...valid.runs[0]!, status: "cancelled" }] }; + expect(gradeHermesApiConnection(value).every(check => check.passed)).toBe(false); + expect(gradeHermesApiConnection({ ...value, expectedRunStatus: "cancelled" }).every(check => check.passed)).toBe(true); + }); + + it.each(["valid", "reported-zero", "cancelled", "unexpected-cancelled", "cancelled-pending", "failed-cancellation", "company-scope", "agent-scope", "run-scope", "task-scope", "paused-agent", "paused-company", "pause-reason", "budget-changed", "pending", "missing-settlement", "missing-price", "estimated", "partial", "wrong-biller", "wrong-provenance", "mismatched-exact", "unknown-tokens"])("calibrates public OpenRouter settlement: %s", async fault => { const company: Record = { id: "company", status: "active", budgetMonthlyCents: 200 }; const agent: Record = { id: "agent", companyId: "company", status: "idle", pauseReason: null, budgetMonthlyCents: 200 }; @@ -170,6 +248,9 @@ describe("Hermes managed API connection qualification", () => { const run: Record = { id: "run", companyId: "company", agentId: "agent", issueId: "task", status: "succeeded", usageJson: usage, costAccountingPending: false, costAccountedAt: "2026-10-08T03:00:00Z" }; if (fault === "reported-zero") { usage.costUsd = 0; usage.costUsdExact = "0.000000000"; } + if (["cancelled", "unexpected-cancelled", "cancelled-pending"].includes(fault)) run.status="cancelled"; + if (fault === "cancelled-pending") run.costAccountingPending=true; + if (fault === "failed-cancellation") run.status="failed"; if (fault === "company-scope") company.id = "foreign"; if (fault === "agent-scope") agent.companyId = "foreign"; if (fault === "run-scope") run.agentId = "foreign"; @@ -188,11 +269,12 @@ describe("Hermes managed API connection qualification", () => { if (fault === "mismatched-exact") usage.costUsdExact = "0.040000000"; if (fault === "unknown-tokens") delete usage.inputTokens; const paths: string[] = []; - const receipt = await captureHermesOpenRouterSettlement({ companyId: "company", agentId: "agent", issueId: "task", runId: "run", api: { + const receipt = await captureHermesOpenRouterSettlement({ companyId: "company", agentId: "agent", issueId: "task", runId: "run", + ...(["cancelled", "cancelled-pending", "failed-cancellation"].includes(fault) ? { expectedRunStatus: "cancelled" as const } : {}), api: { async get(url: string) { paths.push(url); return (url === "/api/companies/company" ? company : url === "/api/agents/agent" ? agent : run) as T; }, } }); expect(new Set(paths)).toEqual(new Set(["/api/companies/company", "/api/agents/agent", "/api/heartbeat-runs/run"])); - expect(receipt.checks.every(check => check.passed)).toBe(["valid", "reported-zero"].includes(fault)); + expect(receipt.checks.every(check => check.passed)).toBe(["valid", "reported-zero", "cancelled"].includes(fault)); }); it.each(["valid", "missing", "duplicate", "company", "provider", "method", "ownership", "status", "owner", "caller"])("establishes the expected user from public account readback: %s", async fault => { const account = { id: "account", companyId: "company", provider: "xai", method: "api_key", ownership: "personal", ownerUserId: "user", status: "connected" }; diff --git a/tests/runner-e2e/hermes-api-connections.ts b/tests/runner-e2e/hermes-api-connections.ts index fe5b7017a3c..74f66fbf5bd 100644 --- a/tests/runner-e2e/hermes-api-connections.ts +++ b/tests/runner-e2e/hermes-api-connections.ts @@ -3,7 +3,13 @@ import { readFileSync } from "node:fs"; import { isDeepStrictEqual } from "node:util"; import type { AiProviderRouting } from "../../packages/shared/src/ai-provider-routing.js"; import { hasAcpxNativeOrigin } from "./acpx-native-origin.js"; -import type { RunnerProfileFixture, RunnerTaskFixture } from "./types.js"; +import type { Page } from "@playwright/test"; +import { pollUntil, type RunnerApi } from "./api.js"; +import { collectRunEvents } from "./run-observations.js"; +import { createTaskThroughUi } from "./user-actions.js"; +import { observeRunProcesses, type RunProcessAuthority } from "./native-local-fixtures.js"; +import type { LiveFixtureValues } from "./live-fixtures.js"; +import type { MatrixExecution, RunnerProfileFixture, RunnerTaskFixture } from "./types.js"; const record = (value: unknown): Record => value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; @@ -53,6 +59,17 @@ export const hermesNativeQuestionTask: RunnerTaskFixture = { ], }; +export const hermesNativeQuestionStopTask: RunnerTaskFixture = { + ...hermesNativeQuestionTask, + id: "native-question-batch-stop", label: "Stop an unanswered native question batch", + flow: "native_question_stop", expectedTerminalState: { issue: "in_progress", run: "cancelled" }, + buildTitle: nonce => `Stop Hermes native questions ${nonce}`, + buildMatchers: (_nonce, execution) => [ + { kind: "issue_status", expected: "in_progress" }, { kind: "run_status", expected: "cancelled" }, + { kind: "runtime_mode", expected: "native" }, { kind: "environment", expected: execution.environment.id }, + ], +}; + /** Preserve the pinned native form; semantic tools cannot satisfy this oracle. */ export function hasHermesNativeQuestionBatch(value: unknown): boolean { const set = record(value), questions = Array.isArray(set.questions) ? set.questions.map(record) : []; @@ -102,6 +119,192 @@ export function hasExactHermesNativeQuestionResponse(input: { && (created[0]!.event.sourceSeq as number) >= 0 && (outcome.sourceSeq as number) > (created[0]!.event.sourceSeq as number); } +/** A provider cancellation must close the retained unanswered callback, not complete it. */ +export function hasHermesNativeQuestionStop(input: { + events: readonly unknown[]; runId: string; turnId: string; requestId: string; questionSet: unknown; + pendingEvent: unknown; run: unknown; issue: unknown; companyId: string; callerUserId: string; + browserResponse: unknown; +}): boolean { + if (![input.runId, input.turnId, input.requestId, input.companyId, input.callerUserId].every(present) + || !hasHermesNativeQuestionBatch(input.questionSet)) return false; + const pending = record(input.pendingEvent), run = record(input.run), issue = record(input.issue); + const rows = input.events.map(record).map(row => ({ row, event: record(record(row.payload).prpEvent) })); + const created = rows.filter(({ event }) => event.eventType === "runtime_request.created" + && hasAcpxNativeOrigin(record(record(event.payload).request).origin, "hermes", "_hermes/ask_questions")); + const outcomes = rows.filter(({ event }) => ["runtime_request.resolved", "runtime_request.expired", "runtime_request.cancelled"].includes(String(event.eventType)) + && record(event.payload).requestId === input.requestId); + const terminals = rows.filter(({ event }) => ["turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted"].includes(String(event.eventType))); + if (created.length !== 1 || outcomes.length !== 1 || terminals.length !== 1) return false; + const start = created[0]!, closed = outcomes[0]!, terminal = terminals[0]!; + const request = record(record(start.event.payload).request), closure = record(closed.event.payload), end = record(terminal.event.payload); + if (!isDeepStrictEqual(start.event, pending) || !isDeepStrictEqual(request.input, input.questionSet) + || request.requestId !== input.requestId || request.turnId !== input.turnId || request.type !== "input" || request.status !== "pending" + || !present(request.itemId) || !present(request.requestKind) + || ![pending.sourceInstanceId, pending.normalizedSessionId].every(present)) return false; + if ([start, closed, terminal].some(({ row, event }) => row.runId !== input.runId || row.protocolSchemaVersion !== 1 + || event.runId !== input.runId || event.turnId !== input.turnId || event.schema !== "paperclip.prp.event.v1" || event.schemaVersion !== 1 + || event.sourceKind !== "runner" || event.sourceInstanceId !== pending.sourceInstanceId || event.normalizedSessionId !== pending.normalizedSessionId + || !Number.isSafeInteger(event.sourceSeq) || (event.sourceSeq as number) < 0)) return false; + if (closed.event.eventType !== "runtime_request.cancelled" || closure.turnId !== input.turnId || closure.itemId !== request.itemId + || closure.requestKind !== request.requestKind || closure.response !== undefined || closure.replayAllowed === true + || (closure.action !== undefined && closure.action !== "cancel") || !present(closure.reason) + || terminal.event.eventType !== "turn.cancelled" || end.status !== "cancelled" || end.error !== null + || (closed.event.sourceSeq as number) <= (start.event.sourceSeq as number) + || (terminal.event.sourceSeq as number) <= (closed.event.sourceSeq as number)) return false; + if (rows.some(({ event }) => event.sourceKind === "runner" && event.sourceInstanceId === pending.sourceInstanceId + && typeof event.sourceSeq === "number" && event.sourceSeq > (terminal.event.sourceSeq as number) + && (String(event.eventType).startsWith("tool.execution.") || event.eventType === "runtime_request.created"))) return false; + const result = record(run.resultJson), stop = record(result.nativeCancellation), browser = record(input.browserResponse); + return run.id === input.runId && run.companyId === input.companyId && run.issueId === issue.id && run.runtimeMode === "native" && run.status === "cancelled" + && issue.companyId === input.companyId && issue.status === "in_progress" + && result.cancelledByActorType === "user" && result.cancelledByUserId === input.callerUserId + && stop.schema === "paperclip.native-cancellation.v1" && stop.companyId === input.companyId && stop.runId === run.id && stop.issueId === issue.id + && stop.scope === "run" && stop.dispatched === true && stop.dispatchState === "acknowledged" && stop.reasonCode === "cancellation_run_only" + && isDeepStrictEqual(stop.effects, ["release_run_resources"]) && present(stop.intentId) + && present(stop.intentAuditId) && present(stop.acknowledgementAuditId) && stop.intentAuditId !== stop.acknowledgementAuditId + && browser.id === run.id && record(record(browser.resultJson).nativeCancellation).intentId === stop.intentId; +} + +/** The browser owns Stop; public receipts and read-only OS observations grade it. */ +export async function runHermesNativeQuestionStop(input: { + page: Page; api: RunnerApi; fixtures: LiveFixtureValues; execution: MatrixExecution; nonce: string; + workspacePath: string; deadlineAt: number; callerUserId: string; + observe(issue: Record, runs: Record[]): void; + capture(id: string, label: string, file: string): Promise; + evidence(name: string, value: unknown): Promise; + registerCleanupAssertion(callback: () => Promise>): void; +}) { + const { api, page, fixtures, execution, nonce, deadlineAt } = input; + if (execution.environment.id !== "local" || execution.suite.id !== HERMES_NATIVE_INTERACTION_SUITE + || execution.profile.qualificationCandidate !== "hermes" || execution.task.id !== hermesNativeQuestionStopTask.id + || !fixtures.aiConnection || !present(input.callerUserId)) throw new Error("Unadmitted Hermes native question Stop cell"); + const checks: Array<{ id: string; passed: boolean; detail: string }> = []; + const check = (id: string, passed: boolean, detail: string) => { + checks.push({ id, passed, detail }); if (!passed) throw new Error(`Hermes native question Stop: ${detail}`); + }; + type StopRun = Parameters[0]["runs"][number] & Record; + let issue: Record = {}, runs: StopRun[] = []; + const observer = observeRunProcesses(); + let authority: RunProcessAuthority | undefined, processIdentityChanged = false; + let processes = observer.sample(), complete = false; + const sample = () => { processes = observer.sample(authority); }; + const timer = setInterval(() => { try { sample(); } catch { processIdentityChanged = true; } }, 100); + const load = async () => { + if (issue.id) issue = await api.get(`/api/issues/${issue.id}`); + const list = await api.get>(`/api/companies/${fixtures.company.id}/heartbeat-runs?limit=100`); + runs = await Promise.all(list.map(run => api.get(`/api/heartbeat-runs/${run.id}`))); + if (runs.length > 1) throw new Error("Hermes native question Stop dispatched an extra run"); + const run: Record = runs[0] ?? {}; + if (run.processPid && run.processGroupId && run.processStartedAt) { + const current = { pid: run.processPid, groupId: run.processGroupId, startedAt: run.processStartedAt, runId: run.id }; + if (authority && !isDeepStrictEqual(authority, current)) processIdentityChanged = true; + authority ??= current; + } + sample(); input.observe(issue, runs); + const events = run.id ? await collectRunEvents>((afterSeq, limit) => api.get(`/api/heartbeat-runs/${run.id}/events?afterSeq=${afterSeq}&limit=${limit}`)) : []; + const interactions = issue.id ? await api.get[]>(`/api/issues/${issue.id}/interactions`) : []; + return { run, issue, events, interactions }; + }; + let identity: Omit[0], "events" | "run" | "issue"> | undefined; + let interactionId: string | undefined; + const retired = () => processes.captured && processes.journal.length >= 2 && processes.live.length === 0 && !processIdentityChanged; + input.registerCleanupAssertion(async () => { + try { + const state = await load(); + const passed = complete && retired() && runs.length === 1 && state.interactions.length === 1 + && state.interactions[0]?.id === interactionId && state.interactions[0]?.status === "expired" + && Boolean(identity && hasHermesNativeQuestionStop({ ...identity, ...state })); + await input.evidence("hermes-native-stop-cleanup.json", { complete, processes, processIdentityChanged, passed }); + return [{ id: "hermes-native-stop-retired-through-cleanup", passed, detail: "The exact per-turn owner and observed descendants remain retired; no late answer or follow-up appears." }]; + } finally { clearInterval(timer); } + }); + const project = await api.post>(`/api/companies/${fixtures.company.id}/projects`, { + name: `Hermes native Stop ${nonce}`, executionWorkspacePolicy: { enabled: true, defaultMode: "shared_workspace", sharedWorkspaceConcurrency: "serialize", + allowIssueOverride: false, environmentId: fixtures.environment.id, workspaceStrategy: { type: "project_primary" } }, + workspace: { name: "Primary", sourceType: "local_path", cwd: input.workspacePath, isPrimary: true }, + }); + const created = await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name, + title: execution.task.buildTitle(nonce), prompt: execution.task.buildPrompt(nonce), workMode: "standard", projectName: project.name, requireExplicitTitle: true }); + issue = await api.get(`/api/issues/${created.issueId}`); + const pending = await pollUntil({ label: "unanswered native Hermes question before browser Stop", deadlineAt, intervalMs: 200, load, + reject: state => ["succeeded", "failed", "timed_out", "cancelled"].includes(state.run.status) ? "Hermes ended before the pending native Stop boundary" : undefined, + accept: state => state.run.status === "running" && state.interactions.length === 1 && state.interactions[0]?.status === "pending" + && state.events.some(row => record(record(row.payload).prpEvent).eventType === "runtime_request.created" + && hasAcpxNativeOrigin(record(record(record(record(row.payload).prpEvent).payload).request).origin, "hermes", "_hermes/ask_questions")), + }); + const card = pending.interactions[0]!, questionSet = card.payload?.questionSet; + const pendingEvents = pending.events.map(row => record(record(row.payload).prpEvent)).filter(event => event.eventType === "runtime_request.created" + && hasAcpxNativeOrigin(record(record(event.payload).request).origin, "hermes", "_hermes/ask_questions")); + const pendingEvent = pendingEvents[0]!, request = record(record(pendingEvent.payload).request); + check("retained-unanswered-native-callback", pendingEvents.length === 1 && hasHermesNativeQuestionBatch(questionSet) + && isDeepStrictEqual(request.input, questionSet) && request.requestId === card.payload?.runtimeRequestId + && card.sourceRunId === pending.run.id && card.continuationPolicy === "none" && !card.result + && !record(pending.run.resultJson).nativeCancellation, "One complete native callback remains unanswered in its original active turn before Stop."); + interactionId = card.id; + await input.evidence("hermes-native-stop-pending.json", { card, event: pendingEvent, processes, checks }); + await page.getByRole("radio", { name: questionSet.questions[0].options[0].label, exact: true }).last().waitFor({ state: "visible" }); + await input.capture("hermes-native-stop-pending", "Unanswered native Hermes question before browser Stop", "hermes-native-stop-pending.png"); + check("owned-live-process-tree", processes.captured && processes.journal.length >= 2 && processes.live.length >= 2 && !processIdentityChanged, + "The public per-turn process identity binds a live owner and descendants before Stop."); + const fresh = await load(); + check("fresh-pending-boundary", fresh.run.status === "running" && fresh.interactions.length === 1 && fresh.interactions[0]?.id === card.id + && fresh.interactions[0]?.status === "pending" && !fresh.interactions[0]?.result + && isDeepStrictEqual(fresh.interactions[0]?.payload, card.payload), "The same unanswered callback is still pending immediately before the browser click."); + const cancelPath = `/api/heartbeat-runs/${pending.run.id}/cancel`; + const responsePromise = page.waitForResponse(response => new URL(response.url()).pathname === cancelPath && response.request().method() === "POST", + { timeout: Math.max(1, deadlineAt - Date.now()) }); + await page.getByRole("button", { name: "Stop", exact: true }).last().click(); + const response = await responsePromise; + check("browser-stop-response", response.status() === 200, "The browser Stop control calls the exact active run's public cancellation API."); + const browserResponse = await response.json(); + identity = { runId: pending.run.id, turnId: pendingEvent.turnId as string, requestId: request.requestId as string, + questionSet, pendingEvent, companyId: fixtures.company.id, callerUserId: input.callerUserId, browserResponse }; + const stopped = await pollUntil({ label: "Hermes native question cancellation and owned Stop acknowledgement", deadlineAt, intervalMs: 200, load, + reject: state => ["succeeded", "failed", "timed_out"].includes(state.run.status) ? "Stop did not produce a cancelled native turn" : undefined, + accept: state => state.interactions.length === 1 && state.interactions[0]?.id === card.id && state.interactions[0]?.status === "expired" + && !state.interactions[0]?.result && hasHermesNativeQuestionStop({ ...identity!, ...state }), + }); + check("exact-native-cancellation", hasHermesNativeQuestionStop({ ...identity, ...stopped }), + "One ordered cancellation closes the retained native request and turn, with the browser's exact audited run Stop acknowledgement."); + let staleRejected = false; + try { + await api.postSensitive(`/api/issues/${issue.id}/interactions/${card.id}/respond`, { answers: [ + { questionId: "q0", optionIds: ["o0"] }, { questionId: "q1", optionIds: ["o0"] }, { questionId: "q2", optionIds: [], otherText: "Late answer" }, + ] }); + } catch (error) { + staleRejected = error instanceof Error && error.message === `Sensitive POST /api/issues/${issue.id}/interactions/${card.id}/respond returned 409; response body withheld`; + if (!staleRejected) throw error; + } + check("stale-answer-refused", staleRejected, "The public response API rejects a later answer with HTTP 409."); + await pollUntil({ label: "Hermes Stop owner and descendant retirement", deadlineAt: Math.min(deadlineAt, Date.now() + 10_000), intervalMs: 100, + load: async () => { sample(); return processes; }, accept: retired }); + check("owned-process-tree-retired", retired(), "The exact owner and observed descendants retire before cleanup starts."); + await page.reload(); + await page.getByTestId("issue-detail-header").getByRole("button", { name: "Change status (current: In Progress)", exact: true }).waitFor({ state: "visible" }); + await page.getByTestId("task-chat-thread").getByTestId("task-chat-collapsible-marker").filter({ has: page.getByText("Run cancelled", { exact: true }) }).waitFor({ state: "visible" }); + await page.getByTestId("task-chat-history-loading").waitFor({ state: "hidden" }); + check("question-unanswerable-after-reload", await page.getByRole("radio", { name: questionSet.questions[0].options[0].label, exact: true }).count() === 0, + "The loaded cancelled turn offers no native question answer control."); + const final = await load(); + check("one-unfinished-cancelled-run", runs.length === 1 && final.run.status === "cancelled" && final.issue.status === "in_progress" + && final.interactions.length === 1 && final.interactions[0]?.status === "expired" && !final.interactions[0]?.result + && hasHermesNativeQuestionStop({ ...identity, ...final }), "Stop leaves one unfinished task and one cancelled run, without an answer or follow-up."); + const connectionChecks = gradeHermesApiConnection({ companyId: fixtures.company.id, agentId: fixtures.agent.id, + issueId: issue.id, connectionId: fixtures.aiConnection.connectionId, provider: fixtures.aiConnection.binding.provider, + expectedResponsibleUserId: input.callerUserId, model: execution.profile.model, runs, expectedRunStatus: "cancelled" }); + connectionChecks.forEach(value => check(value.id, value.passed, "Public native model and account attribution remain correct after Stop.")); + let billing: Awaited> | undefined; + await pollUntil({ label: "cancelled Hermes run billing settlement", deadlineAt: Math.min(deadlineAt, Date.now() + 30_000), intervalMs: 200, + load: async () => billing = await captureHermesOpenRouterSettlement({ api, companyId: fixtures.company.id, agentId: fixtures.agent.id, + issueId: issue.id, runId: final.run.id, expectedRunStatus: "cancelled" }), accept: receipt => receipt.checks.every(value => value.passed) }); + billing!.checks.forEach(value => check(value.id, value.passed, "Cancelled native usage settles reported cost while the company and agent retain healthy budgets.")); + await input.evidence("hermes-openrouter-settlement.json", billing); + await input.capture("final-state", "Cancelled native question remains unanswerable", "final-state.png"); + await input.evidence("api-state.json", { ...final, runs, run: final.run, checks, processes, nativeStop: identity }); + complete = true; + return { issue, runs, checks }; +} + /** Establish the intended personal-account user independently, before a paid task. */ export async function captureHermesApiAccountOwner(input: { api: { get(path: string): Promise }; companyId: string; connectionId: string; provider: string; @@ -141,6 +344,7 @@ export async function captureHermesApiBudgets(input: { /** Observe settled billing and budget health before fixture cleanup pauses the agent. */ export async function captureHermesOpenRouterSettlement(input: { api: { get(path: string): Promise }; companyId: string; agentId: string; issueId: string; runId: string; + expectedRunStatus?: "succeeded" | "cancelled"; }) { const [company, agent, run] = await Promise.all([ input.api.get>(`/api/companies/${input.companyId}`), @@ -158,7 +362,7 @@ export async function captureHermesOpenRouterSettlement(input: { }, checks: [ { id: "billing-observation-scope", passed: company.id === input.companyId && agent.id === input.agentId && agent.companyId === input.companyId && run.id === input.runId && run.companyId === input.companyId && run.agentId === input.agentId && run.issueId === input.issueId }, - { id: "settled-openrouter-reported-cost", passed: run.status === "succeeded" && run.costAccountingPending === false && present(run.costAccountedAt) + { id: "settled-openrouter-reported-cost", passed: run.status === (input.expectedRunStatus ?? "succeeded") && run.costAccountingPending === false && present(run.costAccountedAt) && usage.accountingReceiptReady === true && usage.biller === "openrouter" && usage.billingType === "metered_api" && usage.costStatus === "reported" && provenance.source === "provider_reported" && provenance.version === "hermes-openrouter-wire/v1" && typeof cost === "number" && Number.isFinite(cost) && cost >= 0 && typeof exact === "string" && /^(0|[1-9][0-9]{0,6})\.[0-9]{9}$/.test(exact) && Number(exact) === cost @@ -173,6 +377,7 @@ export async function captureHermesOpenRouterSettlement(input: { export function gradeHermesApiConnection(input: { companyId: string; agentId: string; issueId: string; connectionId: string; provider: string; model: string; expectedResponsibleUserId: string; accountMode?: "responsible_user" | "delegated"; expectedGrantId?: string; + expectedRunStatus?: "succeeded" | "cancelled"; runs: readonly { companyId: string; agentId: string; status: string; runtimeMode?: string; issueId?: string | null; responsibleUserId?: string | null; @@ -184,7 +389,8 @@ export function gradeHermesApiConnection(input: { const identity = record(record(record(run?.runnerProfileJson).sessionCheckpoint).providerIdentity); const provider = record(record(record(run?.runnerProfileJson).nativeExecutionInput).provider); return [ - { id: "one-successful-native-run", passed: input.runs.length === 1 && run?.status === "succeeded" && run.runtimeMode === "native" }, + { id: input.expectedRunStatus === "cancelled" ? "one-cancelled-native-run" : "one-successful-native-run", + passed: input.runs.length === 1 && run?.status === (input.expectedRunStatus ?? "succeeded") && run.runtimeMode === "native" }, { id: "company-agent-task-scope", passed: run?.companyId === input.companyId && run.agentId === input.agentId && run.issueId === input.issueId }, { id: "selected-managed-api-account", passed: account.connectionId === input.connectionId && account.provider === input.provider && account.method === "api_key" && account.mode === (input.accountMode ?? "responsible_user") }, ...(input.expectedGrantId !== undefined ? [{ id: "selected-managed-account-grant", passed: present(input.expectedGrantId) && account.grantId === input.expectedGrantId }] : []), @@ -205,6 +411,6 @@ export const hermesApiConnectionChoices = [ ] as const satisfies readonly { provider: string; credential: RunnerProfileFixture["credential"]; model: string }[]; export const hermesApiConnectionDefinitionDigest = createHash("sha256").update( - ["hermes-api-connections.ts", "acpx-native-origin.ts", "live-fixtures.ts", "harness-env.ts", "runner.spec.ts"] + ["hermes-api-connections.ts", "acpx-native-origin.ts", "native-local-fixtures.ts", "user-actions.ts", "live-fixtures.ts", "harness-env.ts", "runner.spec.ts"] .map(file => readFileSync(new URL(`./${file}`, import.meta.url), "utf8")).join("\n"), ).digest("hex"); diff --git a/tests/runner-e2e/runner.spec.ts b/tests/runner-e2e/runner.spec.ts index a7d58a6dfd7..e5e0bfafa69 100644 --- a/tests/runner-e2e/runner.spec.ts +++ b/tests/runner-e2e/runner.spec.ts @@ -1,5 +1,5 @@ import { runPlanTaskFlow } from "./plan-task-flow.js"; -import { captureHermesApiAccountOwner, captureHermesApiBudgets, captureHermesOpenRouterSettlement, gradeHermesApiConnection, isHermesConnectionSuite, isHermesOpenRouterWorkflow, hasExactHermesNativeQuestionResponse, hasHermesNativeQuestionBatch, hermesNativeAnswerText } from "./hermes-api-connections.js"; +import { captureHermesApiAccountOwner, captureHermesApiBudgets, captureHermesOpenRouterSettlement, gradeHermesApiConnection, isHermesConnectionSuite, isHermesOpenRouterWorkflow, hasExactHermesNativeQuestionResponse, hasHermesNativeQuestionBatch, hermesNativeAnswerText, runHermesNativeQuestionStop } from "./hermes-api-connections.js"; import { hasAcpxNativeOrigin } from "./acpx-native-origin.js"; import { isDeepStrictEqual } from "node:util"; import { assertNativeCompletionSelection, NATIVE_COMPLETION_PREFLIGHT_ENV, verifyNativeCompletionPreflight } from "./native-completion-admission.js"; @@ -592,7 +592,7 @@ for (const execution of executions) { const credentials = credentialValues(); const secrets = normalizedSecrets(Object.values(credentials)); const api = new RunnerApi(request); - const companyRunFlow = execution.suite.id === "task-titles" || ["plan_task_guidance", "blocker_guidance", "continuation_accounting", "continuation", "context_integrity", "agent_chat", "everyday_workflow", "first_task", "instruction_persistence", "cursor_native", "native_active_stop", "native_provider_loss", "public_mcp"].includes(execution.task.flow); + const companyRunFlow = execution.suite.id === "task-titles" || ["plan_task_guidance", "blocker_guidance", "continuation_accounting", "continuation", "context_integrity", "agent_chat", "everyday_workflow", "first_task", "instruction_persistence", "cursor_native", "native_active_stop", "native_provider_loss", "native_question_stop", "public_mcp"].includes(execution.task.flow); const publicMcpUsage = execution.task.flow === "public_mcp" ? assistantUsage(execution.profile.provider === "claude" ? "anthropic" : "openai", execution.profile.model) : undefined; let publicMcpUserId = ""; const consoleDiagnostics: Array> = []; @@ -1047,6 +1047,16 @@ for (const execution of executions) { }); issue = story.issue as IssueRecord; selectedRuns = story.runs as RunRecord[]; matcherResults = story.checks.map(check => ({ matcher: { kind: "json_path" as const, path: `nativeProviderLoss.${check.id}`, expected: true }, passed: check.passed, detail: check.detail })); + } else if (execution.task.flow === "native_question_stop") { + const story = await runHermesNativeQuestionStop({ + page, api, fixtures, execution, nonce, workspacePath, deadlineAt: startedAtMs + deadlineMs, + callerUserId: hermesApiAccountOwner?.expectedResponsibleUserId ?? "", + observe: (currentIssue, currentRuns) => { issue = currentIssue as IssueRecord; selectedRuns = currentRuns as RunRecord[]; }, + capture: captureScreenshot, evidence: (name, data) => writeSanitizedJson(snapshotsDir, name, data, secrets), + registerCleanupAssertion, + }); + issue = story.issue as IssueRecord; selectedRuns = story.runs as RunRecord[]; + matcherResults = story.checks.map(check => ({ matcher: { kind: "json_path" as const, path: `hermesNativeStop.${check.id}`, expected: true }, passed: check.passed, detail: check.detail })); } else if (execution.task.flow === "native_active_stop") { const story = await runNativeActiveStopFlow({ page, api, fixtures, execution, nonce, workspacePath, deadlineAt: startedAtMs + deadlineMs, diff --git a/tests/runner-e2e/types.ts b/tests/runner-e2e/types.ts index 2f8989d08ff..062d650214d 100644 --- a/tests/runner-e2e/types.ts +++ b/tests/runner-e2e/types.ts @@ -35,6 +35,7 @@ export type RunnerTaskFlow = | "plan_revision_acceptance" | "question_resume_completion" | "native_question_completion" + | "native_question_stop" | "plan_approval_completion" | "warm_three_turn" | "instruction_persistence"