From d87f6fa4c2f753979a98f8a2dff26397a733795a Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 08:37:22 -0600 Subject: [PATCH] test(runner-e2e): add staging-soak reliability cases to everyday workflows (#19455) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The paid runner E2E suite (`tests/runner-e2e`) drives the real browser, server, native runner, provider and Daytona > - A manual staging soak found about nine reliability bugs (S47–S55, H2, U9). Each fix got unit or integration tests > - The end-to-end journeys that exposed the bugs existed only as ad-hoc browser scripts, so a regression could return without a signal > - These journeys are expensive, so they must not run per PR or in `--all`. They must run on demand before a release > - This pull request adds the journeys to the existing `everyday-workflows` suite as an explicit `reliability` group > - The benefit is one command that runs the whole reliability set before a release or after a risky runner change ## Linked Issues or Issue Description **What is the problem?** The staging-soak scenarios that found S47–S55, H2 and U9 are not repeatable. They were ad-hoc browser scripts on staging. **Why does it matter?** Without a repeatable end-to-end case, a reliability regression (stalled follow-ups, duplicate execution, dead Retry, hangs after silent commands) can return without a signal. **What is the proposed solution?** Add the scenarios to the paid `everyday-workflows` suite as explicit-only cases with independent graders and a selectable `reliability` group. Refs #19370, #19355, #19354, #19407, #19419, #19394, #19421, #19431 (the related fixes). ## What Changed - `everyday-reliability-cases.ts`: seven new cases (`interrupt-redirect`, `reassign-midrun`, `three-turn-revise`, `long-silent-command`, `tree-pause-release`, `queue-steer`, `queued-across-restart`), their soak signatures, prompts, bounds and the explicit matrix. - `everyday-reliability-flow.ts`: browser and public-API drivers for each journey. A shared watcher records `executionBlocker`, `interruption` and execution projections while it waits. It uses an offered Retry at most once (never while an automatic retry is scheduled). - `everyday-reliability.ts`: pure graders. They encode the product rule: interrupted work resumes automatically or offers a Retry that works on first use. A send-a-message blocker, a dead or repeated Retry, a silent stall, or duplicate execution fails. - `types.ts` / `selectors.ts`: new optional `explicitGroup` on a task. Such a task is selected only by its group, `--case`, or an exact `--id`. `--all`, the plain suite and generic selectors skip it. `--list` still shows it. - `catalog.ts`: the suite gets the reliability tasks, the released Grok/Kimi/Pi facades (only for `long-silent-command`), a per-turn Daytona override for `three-turn-revise`, the `reliability` and `slow` groups, definition version 11, and a computed `expectedMatrixSize` (52 → 62). - `everyday-flow.ts`: a narrow hook calls the scenario driver after task creation. Riley is created for the reassign and tree cases. The reliability checks are added after settlement. - `everyday-observations.ts`: an allowed interrupted run may end `operator_interrupted` or `issue_reassigned`. Lifecycle checks accept a declared reassigned parent owner. - `live-fixtures.ts`: reliability cells get the existing 1,000-cent company and lead-agent hard stop. - Tests: new `everyday-reliability.test.ts` (selection exclusion, matrix, credentials, and calibrated pass/fail cases for every grader). Updated counts in `catalog.test.ts`, `everyday.test.ts` and `installed-cli.test.ts`. - Docs: new "Staging-soak reliability group" section in `EVERYDAY-WORKFLOWS.md` (cases, signatures, run commands, cadence, coverage boundary) and a link from `doc/evals.md`. `everyday-flow.ts` was already over 1,000 lines. This PR adds only about 20 lines of wiring there. The scenario logic is in the new modules. Matrix (10 cells, all Daytona, single attempt): one `runner-codex` (gpt-5.6-sol) cell per case, plus `runner-acpx-grok`, `runner-acpx-kimi` and `runner-acpx-pi` for `long-silent-command`. Run command: ```sh pnpm test:e2e:runner -- --suite everyday-workflows --group reliability --max-parallel 2 ``` ## Verification - `pnpm test:e2e:runner:typecheck` passes. - `node node_modules/vitest/vitest.mjs run --config tests/runner-e2e/vitest.config.ts everyday catalog installed-cli selectors support.test`: 15 files, 362 tests pass. - The full runner-e2e Vitest run has the same 14 failures with and without this change (`controller-outage`, `operating-receipts`, `pi-controls-evidence`, `pi-file-evidence`, `workflow-security`). These fail on the base commit in this local environment too. - `pnpm test:e2e:runner -- --list --suite everyday-workflows --group reliability` lists exactly the ten cells. - `--all`, `--suite everyday-workflows` (still 52 cells), profile/environment selectors and the daily/weekly scheduled selections contain no reliability cell. A test asserts this. - Not run: the paid cells. They need provider and Daytona credentials. The first paid run of the group is the acceptance step. ## Risks - The drivers have not run against a live stack. A first paid run can expose harness timing or selector issues (for example, how fast the queued Interrupt control appears). The failure then names the stalled step and keeps `reliability-observations.json`. - `long-silent-command` does not inject a runner connection loss. The named-cause contract is checked only if a loss happens naturally. - The Grok/Kimi/Pi facades run as ordinary released profiles. If one returns to pending qualification, it drops out of the group and the matrix size follows. - No product code changes. Existing everyday cells keep their selection and checks. The suite definition hash changes because the suite definition changed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, code execution). Standard context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/evals.md | 5 +- tests/runner-e2e/EVERYDAY-WORKFLOWS.md | 68 +++ tests/runner-e2e/catalog.test.ts | 6 +- tests/runner-e2e/catalog.ts | 51 +- tests/runner-e2e/everyday-flow.ts | 36 +- tests/runner-e2e/everyday-observations.ts | 17 +- .../runner-e2e/everyday-reliability-cases.ts | 125 +++++ tests/runner-e2e/everyday-reliability-flow.ts | 434 ++++++++++++++++++ tests/runner-e2e/everyday-reliability.test.ts | 253 ++++++++++ tests/runner-e2e/everyday-reliability.ts | 326 +++++++++++++ tests/runner-e2e/everyday.test.ts | 3 +- tests/runner-e2e/installed-cli.test.ts | 4 +- tests/runner-e2e/live-fixtures.ts | 7 +- tests/runner-e2e/selectors.ts | 6 + tests/runner-e2e/types.ts | 6 + 15 files changed, 1322 insertions(+), 25 deletions(-) create mode 100644 tests/runner-e2e/everyday-reliability-cases.ts create mode 100644 tests/runner-e2e/everyday-reliability-flow.ts create mode 100644 tests/runner-e2e/everyday-reliability.test.ts create mode 100644 tests/runner-e2e/everyday-reliability.ts diff --git a/doc/evals.md b/doc/evals.md index 37b160e28f1..2bd484459b0 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -378,7 +378,10 @@ Use the selectors documented in the [runner E2E README](../tests/runner-e2e/READ for a suite, profile, case, group, or environment. Daytona needs the immutable image digest and `DAYTONA_API_KEY`; follow the README and fixture security guide. `--all` excludes manual suites such as `everyday-workflows`. Select that suite -explicitly; use a narrow selector while developing a fixture. +explicitly; use a narrow selector while developing a fixture. Its staging-soak +reliability cases need `--group reliability` as well; see +[Everyday workflows](../tests/runner-e2e/EVERYDAY-WORKFLOWS.md#staging-soak-reliability-group) +for the command and the before-release cadence. For Runner Evals, the narrowest useful local validation is the report program's help/validation path and the deterministic Runner checks documented in diff --git a/tests/runner-e2e/EVERYDAY-WORKFLOWS.md b/tests/runner-e2e/EVERYDAY-WORKFLOWS.md index 1d7bf6f6fcb..2676d06357d 100644 --- a/tests/runner-e2e/EVERYDAY-WORKFLOWS.md +++ b/tests/runner-e2e/EVERYDAY-WORKFLOWS.md @@ -73,6 +73,74 @@ The first two did not establish a safe restart boundary, and the uncertainty case measures a deterministic safety rule. None supports ranking models. See [controlled recovery tests](../runner-recovery/README.md). +## Staging-soak reliability group + +Seven explicit-only cases turn the staging soak's ad-hoc browser scripts into +repeatable runs. Each case comes from a real staging failure. They live in the +`everyday-workflows` suite, but neither `--all`, `--suite everyday-workflows`, a +profile/environment selector, nor a scheduled campaign selects them. Only the +`reliability` group, an explicit `--case`, or an exact `--id` does +(`explicitGroup` on the task fixture enforces this in `selectors.ts`). + +```sh +pnpm test:e2e:runner -- --list --suite everyday-workflows --group reliability +pnpm test:e2e:runner -- --suite everyday-workflows --group reliability --max-parallel 2 +pnpm test:e2e:runner -- --suite everyday-workflows --case queue-steer +pnpm test:e2e:runner -- --id everyday-workflows.runner-codex.daytona.interrupt-redirect +``` + +**Cadence:** run the whole group before a release and after a risky runner, +recovery, queue, tree-control or restart change. Do not run it per PR. It is +paid: ten Daytona cells, single attempt each, with a 1,000-cent company and +lead-agent hard stop. Missing provider or Daytona credentials, or a mutable +Daytona image, stop the launch before any provider spend. + +**Product rule (every case):** work interrupted by a restart, Stop, +reassignment or lost runner must resume automatically, or offer a Retry that +works on first use. The harness watches each task's `executionBlocker`, +`interruption` and execution projection while it waits. It uses an offered +Retry at most once (never while an automatic retry is scheduled) and requires +a new run from it. These fail: a "send a message to continue" blocker +(`resume.no-send-message-blocker`), a dead or repeated Retry +(`resume.retry-works-first-use`), a silent stall past the case bound +(`resume.no-silent-stall`), and duplicate execution. Evidence is saved to +`reliability-observations.json`. + +Message delivery is attributed through each run's wake context +(`wakeCommentIds`), not its saved prompt: the prompt carries the whole +conversation history, so later turns legitimately repeat earlier messages. A +message delivered by more than one successful run is duplicate execution. +`--group slow` selects only `long-silent-command`. + +| Case | Soak signature | Journey | Key checks | +|---|---|---|---| +| `interrupt-redirect` | S47 (fixes #19370, #19355) | While a slow step loop runs, send a follow-up and click its queued **Interrupt**. Then reopen with a second slow loop, click **Stop**, and immediately send another message while the stopped run is still saving (the `workspace_busy` retry path). | Interrupted run ends `operator_interrupted`; exactly one continuation carries the message and starts within 180 s (the soak saw 13–90 s once fixed, 4 minutes or never before); no second user message; the Stop-settlement message is consumed once within 180 s; Done. | +| `reassign-midrun` | S48 (fix #19354) | While the lead runs a resumable step loop, reassign the task to Riley through the public issue API. | Request is 2xx within 10 s (no 409 or 20 s hang); old run ends `issue_reassigned`; Riley's run starts after the request and succeeds; no overlapping executions on the task; `steps.txt` holds step-1..step-8 exactly once, so work continued from saved files. | +| `three-turn-revise` | S49 | Create `items.md`, then two reopen-comment revisions on the same workspace: add two items, then convert to a table. Runs on the **per-turn** Daytona lease (the bug hit gpt-5.6-sol with an ephemeral lease on turn 3). | Each turn succeeds; no reconciliation or send-a-message blocker; the file is a five-row Number/Item table. | +| `long-silent-command` | S52/S52b (fixes #19407, #19419) | The agent runs one 13-minute foreground loop (`sleep 60` × 13, appending to `silent.txt`) that prints nothing, then reports. Tagged `slow`; 50-minute Daytona deadline. | Run succeeds after the command (not shortened) and is terminal within 8 minutes after it; `silent.txt` holds tick-1..tick-13; Stop stays available in API and browser samples throughout; any failure carries a named cause. | +| `tree-pause-release` | S50/S50b (fixes #19394, #19421) | The lead delegates two slow children to Riley. While a child is queued or has a parked `workspace_busy` retry, pause the parent's tree with `POST /api/issues/:id/tree-holds`. Hold past the retry's due time (at least 75 s), then release **without** `wakeAgents`. | The hold interrupted work; no child run starts during the hold; each interrupted task gets exactly one `issue_tree_resumed` run; all tasks reach Done. | +| `queue-steer` | H2 | During an active run, queue three follow-ups through the public comment API; edit one, discard one, and steer one (retrying only `steering_temporarily_unavailable`, for up to two minutes). | Edited text reaches the agent and the original never does; the discarded text never reaches the agent; the steered message is acknowledged by the active turn and never replayed; the remaining message is handled by exactly one run. | +| `queued-across-restart` | S55/U9 (fix #19431) | Queue a follow-up through the composer while a run is active, then restart the controller with the existing restart machinery. | The follow-up dispatches only after the restart, automatically within six minutes, exactly once; every observed `queued` wait names a cause (`server_restarting`, `task_drain`, …) rather than a bare "Queued"; any offered Retry works on first use. | + +**Matrix (10 cells).** Each case runs once on native Codex (`runner-codex`, +`gpt-5.6-sol`) on Daytona: the warm environment, except `three-turn-revise` on +the per-turn lease. `long-silent-command` also runs on the released ACPX +facades Grok (`runner-acpx-grok`), Kimi (`runner-acpx-kimi`) and OpenRouter/Pi +(`runner-acpx-pi`), which hung 3/3 before the fix. They run as ordinary released +profiles, as in the rolling-deployment matrix; a profile whose ACPX +qualification returns to pending drops out of the group automatically. Their +credentials (`XAI_API_KEY`, `KIMI_API_KEY`, `OPENROUTER_API_KEY`) are only +required when those cells are selected. + +**Coverage boundary.** The graders and selection are calibrated by +`everyday-reliability.test.ts` against passing and plausible failing +observations. No paid cell has run yet; a first run of the group is the +acceptance step. The original soak browser scripts are not in this repository, +so these cases encode the reported signatures, not a replay of those scripts. +`long-silent-command` checks the completion and Stop paths; it does not inject a +runner connection loss, so the "fails within a bounded time with a named +cause" contract is checked only if a loss happens naturally. + ## Matrix and running The local matrix has fourteen cases on native Codex `gpt-5.6-sol`, native ACPX Claude diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index d8706ffd9f1..5ea48690cb2 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -329,11 +329,11 @@ describe("runner E2E catalog", () => { expect(localIntegrityTasks).toHaveLength(2); expect(openRouterBreadthTasks).toHaveLength(3); expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([ - 1, 9, 2, 9, 2, 4, 1, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 20, 2, 1, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 2, 52, 5, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2, + 1, 9, 2, 9, 2, 4, 1, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 20, 2, 1, 6, 8, 46, 2, 23, 15, 62, 6, 6, 20, 26, 2, 52, 5, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2, ]); - expect(validateRunnerCatalog()).toHaveLength(824); - expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(824); + expect(validateRunnerCatalog()).toHaveLength(834); + expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(834); expect( runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"), ).toHaveLength(48); diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index 14889bb6006..d20a9377361 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -41,6 +41,7 @@ import { lifecycleLiveTasks, lifecycleLiveDefinitionDigest } from "./lifecycle-l import { publicMcpTasks, publicMcpExpandedDigest, publicMcpSetupDigest, publicMcpWorkflowDigest, publicMcpWorkerInstructions, publicMcpWorkerSkillDigest } from "./public-mcp-cases.js"; import { graderVersion as publicMcpGraderVersion } from "./public-mcp-grading.js"; import { everydayTasks, productionStoryProfile } from "./everyday-cases.js"; +import { EVERYDAY_RELIABILITY_GROUP, EVERYDAY_RELIABILITY_SLOW_GROUP, RELIABILITY_FACADE_PROFILE_IDS, everydayReliabilityExecutionIds, everydayReliabilitySignatures, everydayReliabilityTasks } from "./everyday-reliability-cases.js"; import { REMOTE_WORKSPACE_STORAGE_SUITE, remoteWorkspaceStoragePrompt } from "./remote-workspace-storage.js"; import { CONNECTION_GUIDANCE_SUITE, CONNECTION_GUIDANCE_BUDGET_CENTS, connectionGuidanceTasks, connectionGuidanceDefinitionDigest } from "./connection-guidance-cases.js"; @@ -90,6 +91,8 @@ const SELECTABLE_GROUPS = [ "onboarding", "context-integrity", "stock-harness", + EVERYDAY_RELIABILITY_GROUP, + EVERYDAY_RELIABILITY_SLOW_GROUP, ] as const; const SAMPLE_UUID = "11111111-1111-4111-8111-111111111111"; @@ -1253,6 +1256,33 @@ export const kimiRunnerProfile = nativeProfile({ modelQualification: { source: "qualified_runner_profile", qualificationId: "kimi:2.3.3:native-qualified-profile17" }, }); +/** + * ACPX facades admitted only for the reliability long-silent-command case. Like the + * rolling-deployment matrix, they run as ordinary released profiles: a pending + * candidate needs its own qualification suite and is rejected here. + */ +const everydayFacadeProfiles = RELIABILITY_FACADE_PROFILE_IDS + .map(id => [...runnerProfiles, ...extendedHarnessProfiles, kimiRunnerProfile].find(profile => profile.id === id)!) + .filter(original => !original.qualificationCandidate || QUALIFIED_ACPX_PROFILES[original.qualificationCandidate].qualificationStatus !== "pending") + .map(original => { + const { qualificationCandidate: _candidate, ...profile } = productionStoryProfile(original); + return profile; + }); +/** Released facades currently in the reliability group; a pending one drops out. */ +export const everydayReliabilityFacadeProfileIds = everydayFacadeProfiles.map(profile => profile.id); +const everydayWorkflowProfiles = [...everydayProfiles, + productionStoryProfile(runnerProfiles.find(profile => profile.id === "runner-opencode")!), + ...everydayFacadeProfiles]; + +/** Every reliability cell outside the deliberately small explicit matrix. */ +function everydayReliabilityExcludedIds(suiteId: string): string[] { + const allowed = new Set(everydayReliabilityExecutionIds(suiteId, everydayReliabilityFacadeProfileIds)); + return everydayWorkflowProfiles.flatMap(profile => ["local", "daytona"].flatMap(environmentId => + everydayReliabilityTasks.map(task => `${suiteId}.${profile.id}.${environmentId}.${task.id}`))) + .filter(id => !allowed.has(id)); +} + + const rollingDeploymentProfiles = [...runnerProfiles, ...extendedHarnessProfiles, kimiRunnerProfile] .filter(profile => profile.generation === "native" && profile.supportedEnvironments.includes("daytona") && (!profile.qualificationCandidate || QUALIFIED_ACPX_PROFILES[profile.qualificationCandidate].qualificationStatus !== "pending")) @@ -1704,11 +1734,13 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ { id: "everyday-workflows", label: "Everyday Paperclip Work", manualOnly: true, description: "Real user requests, useful downloaded work, and durable continuation using production instructions.", - groups: ["native"], profiles: [...everydayProfiles, - productionStoryProfile(runnerProfiles.find(profile => profile.id === "runner-opencode")!), - ], environments: [localEnvironment, daytonaWarmEnvironment], - tasks: everydayTasks.map(task => ["hire-reuse", "delegate-feedback", "recover-controller"].includes(task.id) - ? { ...task, automaticRetryPolicy: "single_attempt" as const } : task), expectedMatrixSize: 52, + groups: ["native"], profiles: everydayWorkflowProfiles, environments: [localEnvironment, daytonaWarmEnvironment], + // The S49 turn-3 failure needed an ephemeral per-turn lease, not the warm one. + environmentOverrides: { "three-turn-revise": { daytona: runnerEnvironments.find(environment => environment.id === "daytona")! } }, + tasks: [...everydayTasks.map(task => ["hire-reuse", "delegate-feedback", "recover-controller"].includes(task.id) + ? { ...task, automaticRetryPolicy: "single_attempt" as const } : task), ...everydayReliabilityTasks], + // 52 everyday cells, one native Daytona cell per reliability case, and released facades for the silent command. + expectedMatrixSize: 52 + everydayReliabilityTasks.length + everydayFacadeProfiles.length, excludedExecutionIds: [...everydayProfiles.flatMap(profile => everydayTasks .filter(task => !["build-revise", "delegate-feedback", "recover-controller", "create-skill-studio"].includes(task.id)) .map(task => `everyday-workflows.${profile.id}.daytona.${task.id}`)), @@ -1716,8 +1748,14 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ `everyday-workflows.runner-opencode.daytona.${task.id}`, ...(["hire-reuse", "delegate-feedback"].includes(task.id) ? [] : [`everyday-workflows.runner-opencode.local.${task.id}`]), ]), + // ACPX facades join only the reliability long-silent-command cells. + ...everydayFacadeProfiles.flatMap(profile => everydayTasks.flatMap(task => + ["local", "daytona"].map(environmentId => `everyday-workflows.${profile.id}.${environmentId}.${task.id}`))), + ...everydayReliabilityExcludedIds("everyday-workflows"), ], - definitionMetadata: { version: 10, instructions: "production", grading: "outcome-and-invariants", scheduling: "explicit-only", + definitionMetadata: { version: 11, instructions: "production", grading: "outcome-and-invariants", scheduling: "explicit-only", + reliability: { group: EVERYDAY_RELIABILITY_GROUP, selection: "explicit-group-only", signatures: everydayReliabilitySignatures, + productRule: "interrupted-work-resumes-automatically-or-offers-a-working-first-retry", companyAndLeadBudgetCents: 1_000 }, controllerDowntimeMs: 125_000, remoteCodexRestart: "bounded-main-server-outage-receipt-empty-cache-same-active-process-and-finalization", procedureCases: { ids: ["hire-reuse", "delegate-feedback"], maximumAttemptsPerCell: 1, companyAndLeadBudgetCents: 1_000 } }, @@ -2186,6 +2224,7 @@ export function validateRunnerCatalog(): MatrixExecution[] { ...continuationTasks, questionResumeTask, ...everydayTasks, + ...everydayReliabilityTasks, ...taskTitleTasks, ...runnerTasks, ...localIntegrityTasks, diff --git a/tests/runner-e2e/everyday-flow.ts b/tests/runner-e2e/everyday-flow.ts index f10f675db26..47cf65c97b5 100644 --- a/tests/runner-e2e/everyday-flow.ts +++ b/tests/runner-e2e/everyday-flow.ts @@ -40,6 +40,8 @@ import { type StoryInteraction, } from "./everyday-decisions.js"; import { LATE_REQUIREMENT, SLUGIFY_REVISION, SLUGIFY_DEPLOYMENT_SOAK, requiresEverydayArtifactOracle } from "./everyday-cases.js"; +import { everydayReliabilitySignatures, isEverydayReliabilityCase } from "./everyday-reliability-cases.js"; +import { driveReliabilityScenario, type ReliabilityOutcome } from "./everyday-reliability-flow.js"; import { isActiveStoryRun, isStoryWorkspaceDeferral, @@ -91,8 +93,11 @@ export interface EverydayEvidence { agents: Row[]; downloads: Row[]; allowedInterruptedRuns: string[]; + /** Reliability scenarios: failures a later run on the same task verifiably resumed. */ + recoveredRuns?: string[]; remoteRestart?: RemoteRestartEvidence; remoteWorkspaceStorage?: RemoteWorkspaceStorageEvidence; + reliability?: { signature: string; observed: string; fixes: readonly string[] }; } interface Input { page: Page; @@ -206,6 +211,8 @@ export async function runEverydayFlow(input: Input) { let settledAgentReply: Row | undefined; let postControllerApiFollowup = false; let retainedFollowupCommentId: string | undefined; + let reliability: ReliabilityOutcome | undefined; + let helperAgentId: string | undefined; let reviewHandoffBoundary: { childId: string; assigneeAgentId: string | null | undefined; @@ -333,6 +340,7 @@ export async function runEverydayFlow(input: Input) { issueId: target.id, commentIds: added.map((c) => c.id), }); + return added.map((c) => String(c.id)); } function declineInput(state: Pick): ConnectionGuidanceDeclineInput | undefined { if (execution.suite.id !== CONNECTION_GUIDANCE_SUITE || @@ -378,7 +386,7 @@ export async function runEverydayFlow(input: Input) { )), reject: (state) => { if (state.runs.length > 12) return "bounded execution count exceeded"; - const bad = storyUnexpectedRunFailure(state.runs, ev.allowedInterruptedRuns); + const bad = storyUnexpectedRunFailure(state.runs, ev.allowedInterruptedRuns, ev.recoveredRuns); if (bad) return `native execution failed ${bad.errorCode ?? ""}: ${bad.error ?? bad.status}`; if ( @@ -618,6 +626,9 @@ export async function runEverydayFlow(input: Input) { "everyday-delivery.ts", "everyday-observations.ts", "everyday-remote-restart.ts", + "everyday-reliability-cases.ts", + "everyday-reliability.ts", + "everyday-reliability-flow.ts", "retained-controller-continuation.ts", "controller-outage.ts", "controller-rolling-deployment.ts", @@ -716,7 +727,8 @@ export async function runEverydayFlow(input: Input) { canCreateAgents: true, canAssignTasks: true, }); - if (caseId === "delegate-feedback" || caseId === "agent-review-handoff") { + if (isEverydayReliabilityCase(caseId)) ev.reliability = everydayReliabilitySignatures[caseId]; + if (["delegate-feedback", "agent-review-handoff", "reassign-midrun", "tree-pause-release"].includes(caseId)) { const config = execution.profile.buildAgent({ environmentId: fixtures.environment.id, environmentFixtureId: execution.environment.id, @@ -724,7 +736,7 @@ export async function runEverydayFlow(input: Input) { secretRefs: fixtures.secretRefs, executionId: nonce, }); - await api.post(`/api/companies/${fixtures.company.id}/agents`, { + helperAgentId = (await api.post(`/api/companies/${fixtures.company.id}/agents`, { ...config, name: "Riley Builder", role: "engineer", @@ -737,7 +749,7 @@ export async function runEverydayFlow(input: Input) { "You implement small software projects and verify your work. Follow task feedback and deliver usable files.", }, }, - }); + })).id; } if (caseId.startsWith("service-")) review = await setupConnectionReview({ @@ -785,6 +797,14 @@ export async function runEverydayFlow(input: Input) { input.observe(parent!, []); note("task-submitted", { issueId: parent!.id }); await openParent(); + if (isEverydayReliabilityCase(caseId)) { + reliability = await driveReliabilityScenario(caseId, { page, api, execution, nonce, leadAgentId: fixtures.agent.id, + helperAgentId, workspacePath: input.workspacePath, deadlineAt: input.deadlineAt, parentId: parent!.id, + refresh, reply: message => reply(message), openParent, + restart: () => input.restart(), note, allowInterrupted: runId => { ev.allowedInterruptedRuns.push(runId); }, + allowRecovered: runIds => { ev.recoveredRuns = [...runIds]; }, + evidence: input.evidence, capture: input.capture }); + } if (caseId === "agent-review-handoff") { const boundary = await pollUntil({ label: "blocked parent with agent review wake", @@ -1210,8 +1230,9 @@ export async function runEverydayFlow(input: Input) { } await settled( caseId === "stop-redirect" ? `Reference ${nonce}` - : caseId === "provider-second" ? `CONTACTS_${nonce}` : undefined, + : caseId === "provider-second" ? `CONTACTS_${nonce}` : reliability?.expectedReply, ); + if (reliability) ev.checks.push(...await reliability.checks()); if (caseId === "create-skill-studio") { const createdSkills = await api.get( `/api/companies/${fixtures.company.id}/skills`, @@ -1620,7 +1641,8 @@ export async function runEverydayFlow(input: Input) { .filter( (r) => !isStoryWorkspaceDeferral(r) && - !isExpectedStoryInterruption(r, ev.allowedInterruptedRuns), + !isExpectedStoryInterruption(r, ev.allowedInterruptedRuns) && + !(ev.recoveredRuns ?? []).includes(r.id), ) .every( (r) => @@ -1667,6 +1689,8 @@ export async function runEverydayFlow(input: Input) { parentId: parent!.id, leadId: fixtures.agent.id, allowedInterruptedRuns: ev.allowedInterruptedRuns, + recoveredRuns: ev.recoveredRuns, + parentAgentIds: reliability?.parentAgentIds, requireRemoteRestart, remoteRestart: ev.remoteRestart, }), diff --git a/tests/runner-e2e/everyday-observations.ts b/tests/runner-e2e/everyday-observations.ts index 798b85e5899..bfb4dcc8666 100644 --- a/tests/runner-e2e/everyday-observations.ts +++ b/tests/runner-e2e/everyday-observations.ts @@ -220,16 +220,18 @@ export function isExpectedStoryInterruption( ): boolean { if (!allowedRunIds.includes(run.id)) return false; return ( - (run.status === "cancelled" && (!run.errorCode || run.errorCode === "cancelled")) || + // Operator interrupts and reassignment are deliberate reliability-scenario stops. + (run.status === "cancelled" && (!run.errorCode || ["cancelled", "operator_interrupted", "issue_reassigned"].includes(run.errorCode))) || (run.status === "interrupted" && run.errorCode === "server_shutdown_interrupted") || (run.status === "failed" && run.errorCode === "process_lost") ); } -export function storyUnexpectedRunFailure(runs: StoryRun[], allowedRunIds: readonly string[]) { +/** `recoveredRunIds`: failures whose work a later run on the same task verifiably resumed. */ +export function storyUnexpectedRunFailure(runs: StoryRun[], allowedRunIds: readonly string[], recoveredRunIds: readonly string[] = []) { return runs.find((run) => ["failed", "timed_out", "cancelled", "interrupted"].includes(run.status) && - !isStoryWorkspaceDeferral(run) && + !isStoryWorkspaceDeferral(run) && !recoveredRunIds.includes(run.id) && !isExpectedStoryInterruption(run, allowedRunIds), ); } @@ -274,6 +276,10 @@ export function storyLifecycleChecks(input: { parentId: string; leadId: string; allowedInterruptedRuns?: string[]; + /** Failed runs whose work a later run on the same task verifiably resumed (reliability scenarios). */ + recoveredRuns?: readonly string[]; + /** Agents that may execute on the parent; defaults to the lead (reassignment adds the new owner). */ + parentAgentIds?: readonly string[]; requireRemoteRestart?: boolean; remoteRestart?: RemoteRestartEvidence; }): StoryCheck[] { @@ -307,7 +313,8 @@ export function storyLifecycleChecks(input: { ), check( "successful-runs", - executed.every((r) => r.status === "succeeded" || isExpectedStoryInterruption(r, allowed)), + executed.every((r) => r.status === "succeeded" || isExpectedStoryInterruption(r, allowed) + || (input.recoveredRuns ?? []).includes(r.id)), "Only the expected stop or process-loss outcome of an injected interruption is exempt.", ), check( @@ -321,7 +328,7 @@ export function storyLifecycleChecks(input: { (r) => (r.contextSnapshot?.issueId === input.parentId || r.contextSnapshot?.taskId === input.parentId) && - r.agentId !== input.leadId, + !(input.parentAgentIds ?? [input.leadId]).includes(r.agentId), ), "A mentioned worker must not execute on the parent.", ), diff --git a/tests/runner-e2e/everyday-reliability-cases.ts b/tests/runner-e2e/everyday-reliability-cases.ts new file mode 100644 index 00000000000..db349d61d06 --- /dev/null +++ b/tests/runner-e2e/everyday-reliability-cases.ts @@ -0,0 +1,125 @@ +import type { RunnerTaskFixture } from "./types.js"; + +/** + * Staging-soak reliability scenarios. Every case reproduces a real staging + * failure end to end. They are explicit-only: `--all`, `--suite + * everyday-workflows` and generic profile/environment selectors never pick + * them. Select them with `--group reliability`, a `--case`, or an exact `--id`. + */ +export const EVERYDAY_RELIABILITY_GROUP = "reliability"; +export const EVERYDAY_RELIABILITY_SLOW_GROUP = "slow"; + +/** Seconds per silent iteration and iteration count for `long-silent-command`. */ +export const LONG_SILENT_ITERATION_SECONDS = 60; +export const LONG_SILENT_ITERATIONS = 13; + +/** Soak signature, the observed failure before its fix, and related fix PRs. */ +export const everydayReliabilitySignatures = { + "interrupt-redirect": { signature: "S47", observed: "Interrupted follow-up started after 4 minutes or never; needed a second message.", fixes: ["#19370", "#19355"] }, + "reassign-midrun": { signature: "S48", observed: "Reassigning an active task returned 409 or hung for 20 seconds; the old run kept executing.", fixes: ["#19354"] }, + "three-turn-revise": { signature: "S49", observed: "Turn 3 on an ephemeral Daytona lease (gpt-5.6-sol) stopped behind a reconciliation / send-a-message blocker.", fixes: [] }, + "long-silent-command": { signature: "S52/S52b", observed: "ACPX-facade runs (Grok, Kimi, OpenRouter/Pi) hung 3/3 after a long silent foreground command; Stop disappeared.", fixes: ["#19407", "#19419"] }, + "tree-pause-release": { signature: "S50/S50b", observed: "A child started during a tree pause, or did not resume after release without wakeAgents.", fixes: ["#19394", "#19421"] }, + "queue-steer": { signature: "H2", observed: "Edited, discarded and steered queue entries were lost, replayed, or delivered twice.", fixes: [] }, + "queued-across-restart": { signature: "S55/U9", observed: "Follow-up queued before a controller restart sat as a bare Queued until another message.", fixes: ["#19431"] }, +} as const; + +export type EverydayReliabilityCaseId = keyof typeof everydayReliabilitySignatures; +export const everydayReliabilityCaseIds = Object.keys(everydayReliabilitySignatures) as EverydayReliabilityCaseId[]; + +export function isEverydayReliabilityCase(caseId: string): caseId is EverydayReliabilityCaseId { + return Object.hasOwn(everydayReliabilitySignatures, caseId); +} + +/** Bounds encoded in the graders. Soak saw 13-90 s once fixed; 4 minutes or never before. */ +export const RELIABILITY_BOUNDS = { + continuationStartMs: 3 * 60_000, + reassignRequestMs: 10_000, + restartDispatchMs: 6 * 60_000, + silentCompletionAfterCommandMs: 8 * 60_000, + steeringRetryWindowMs: 2 * 60_000, + holdObservationMs: 75_000, +} as const; + +/** A resumable slow loop: an interrupted or reassigned run must continue from saved files. */ +export function slowStepsPrompt(file: string, steps: number, pauseSeconds: number): string { + return `Create ${file} in the project workspace and record progress one line at a time: for each step from 1 to ${steps}, append the line "step-N" (for example "step-1"), then wait ${pauseSeconds} seconds before the next step. Do the waiting in ordinary foreground shell commands. If ${file} already has some steps, continue from the next missing step; never write a step twice. When all ${steps} steps are recorded, reply with a one-line summary.`; +} + +export const interruptRedirectMessage = (nonce: string) => + `Change of plan: stop the step loop where it is. Create redirect.txt containing exactly REDIRECT_${nonce} and reply with "Redirected ${nonce}".`; +export const stopSettleMessage = (nonce: string) => + `After the stop: leave the step file as it is. Create settled.txt containing exactly SETTLED_${nonce} and reply with "Settled ${nonce}".`; +export const threeTurnRevisions = (nonce: string) => [ + `Add two more items to items.md: "delta ${nonce}" and "epsilon ${nonce}". Keep the existing items.`, + `Convert items.md into a Markdown table with the columns Number and Item, one row per existing item in the same order. Keep no bullet list.`, +] as const; +export const longSilentCommand = (iterations = LONG_SILENT_ITERATIONS, seconds = LONG_SILENT_ITERATION_SECONDS) => + `for i in $(seq 1 ${iterations}); do sleep ${seconds}; echo "tick-$i" >> silent.txt; done`; +export const queueSteerMessages = (nonce: string) => ({ + original: `Also create edited.txt containing exactly ORIGINAL_${nonce}.`, + edited: `Also create edited.txt containing exactly EDITED_${nonce}.`, + discarded: `Also create discarded.txt containing exactly DISCARDED_${nonce}.`, + steered: `While you work, also create steered.txt containing exactly STEERED_${nonce}.`, +}); +export const queuedRestartMessage = (nonce: string) => + `Next, create after-restart.txt containing exactly RESTARTED_${nonce} and reply with "Restarted ${nonce}".`; + +const definitions: ReadonlyArray<{ + id: EverydayReliabilityCaseId; + label: string; + prompt: (nonce: string) => string; + expectedRunCount: number; + daytonaTimeoutMs: number; + slow?: boolean; +}> = [ + { id: "interrupt-redirect", label: "Interrupt a run with a follow-up, then send during Stop settlement", + prompt: () => slowStepsPrompt("steps.txt", 8, 30), expectedRunCount: 6, daytonaTimeoutMs: 35 * 60_000 }, + { id: "reassign-midrun", label: "Reassign an active task to another agent", + prompt: () => slowStepsPrompt("steps.txt", 8, 30), expectedRunCount: 3, daytonaTimeoutMs: 30 * 60_000 }, + { id: "three-turn-revise", label: "Create and revise one workspace file across three turns", + prompt: nonce => `Create items.md in the project workspace with a bullet list of exactly three items: "alpha ${nonce}", "beta ${nonce}" and "gamma ${nonce}". Reply when it is saved.`, + expectedRunCount: 3, daytonaTimeoutMs: 35 * 60_000 }, + { id: "long-silent-command", label: "Finish after a long silent foreground command", + prompt: nonce => `Run exactly this shell command in the project workspace as one foreground command and wait for it to finish. It takes about ${LONG_SILENT_ITERATIONS} minutes and prints nothing; that is expected. Do not background it, shorten it, split it, or interrupt it:\n\n${longSilentCommand()}\n\nAfter it finishes, count the lines in silent.txt and reply with "SILENT_DONE_${nonce} ".`, + expectedRunCount: 1, daytonaTimeoutMs: 50 * 60_000, slow: true }, + { id: "tree-pause-release", label: "Pause and release a delegated task tree", + prompt: () => `Delegate two child tasks to Riley Builder, one per file. Child A: ${slowStepsPrompt("a.txt", 5, 20)} Child B: ${slowStepsPrompt("b.txt", 5, 20)} Keep the implementation with Riley. When both children are done, confirm both files are complete and finish.`, + expectedRunCount: 6, daytonaTimeoutMs: 40 * 60_000 }, + { id: "queue-steer", label: "Edit, discard and steer queued follow-ups", + prompt: () => slowStepsPrompt("steps.txt", 8, 30), expectedRunCount: 2, daytonaTimeoutMs: 30 * 60_000 }, + { id: "queued-across-restart", label: "Dispatch a follow-up queued before a controller restart", + prompt: () => slowStepsPrompt("steps.txt", 6, 20), expectedRunCount: 3, daytonaTimeoutMs: 30 * 60_000 }, +]; + +export const everydayReliabilityTasks: readonly RunnerTaskFixture[] = definitions.map(definition => ({ + id: definition.id, + label: definition.label, + groups: [EVERYDAY_RELIABILITY_GROUP, ...(definition.slow ? [EVERYDAY_RELIABILITY_SLOW_GROUP] : [])], + explicitGroup: EVERYDAY_RELIABILITY_GROUP, + workMode: "standard", + flow: "everyday_workflow", + // An upper bound for cost estimates; an offered Retry or workspace deferral may add a run. + expectedRunCount: definition.expectedRunCount, + minimumExpectedRunCount: 1, + automaticRetryPolicy: "single_attempt", + attemptTimeoutMs: { local: 20 * 60_000, daytona: definition.daytonaTimeoutMs }, + expectedTerminalState: { issue: "done", run: "succeeded" }, + buildTitle: nonce => `${definition.label} ${nonce}`, + buildPrompt: definition.prompt, + buildVisibleMarker: nonce => `STUDIO_${nonce}`, + buildMatchers: () => [], // The workflow records independent reliability checks. +})); + +/** The native profile used for every reliability case, plus ACPX facades for the silent command. */ +export const RELIABILITY_NATIVE_PROFILE_ID = "runner-codex"; +export const RELIABILITY_FACADE_PROFILE_IDS = ["runner-acpx-grok", "runner-acpx-kimi", "runner-acpx-pi"] as const; + +/** The deliberately small paid matrix: one native Daytona cell per case, plus facades for long-silent-command. */ +export function everydayReliabilityExecutionIds(suiteId: string, + facadeProfileIds: readonly string[] = RELIABILITY_FACADE_PROFILE_IDS): string[] { + return [ + ...everydayReliabilityCaseIds.map(caseId => `${suiteId}.${RELIABILITY_NATIVE_PROFILE_ID}.daytona.${caseId}`), + ...facadeProfileIds.map(profileId => `${suiteId}.${profileId}.daytona.long-silent-command`), + ]; +} diff --git a/tests/runner-e2e/everyday-reliability-flow.ts b/tests/runner-e2e/everyday-reliability-flow.ts new file mode 100644 index 00000000000..3c7cac46e7c --- /dev/null +++ b/tests/runner-e2e/everyday-reliability-flow.ts @@ -0,0 +1,434 @@ +import { expect, type Page } from "@playwright/test"; +import { randomUUID } from "node:crypto"; +import { readFile } from "node:fs/promises"; +import path from "node:path"; +import { pollUntil, type RunnerApi } from "./api.js"; +import { + LONG_SILENT_ITERATION_SECONDS, LONG_SILENT_ITERATIONS, RELIABILITY_BOUNDS, everydayReliabilitySignatures, + interruptRedirectMessage, queueSteerMessages, queuedRestartMessage, slowStepsPrompt, stopSettleMessage, + threeTurnRevisions, type EverydayReliabilityCaseId, +} from "./everyday-reliability-cases.js"; +import { + executedRuns, gradeInterruptRedirect, recoveredRunIds, gradeLongSilentCommand, gradeQueueSteer, gradeQueuedAcrossRestart, + gradeReassignMidrun, gradeResumptionRule, gradeThreeTurnRevise, gradeTreePauseRelease, runIssueId, runsCarrying, + type ResumptionObservation, type RetryClick, +} from "./everyday-reliability.js"; +import { isActiveStoryRun, type StoryCheck, type StoryIssue, type StoryReplyExpectation, type StoryRun } from "./everyday-observations.js"; +import { collectRunEvents } from "./run-observations.js"; +import type { MatrixExecution } from "./types.js"; + +type Row = Record; +interface StoryState { issues: Array; runs: StoryRun[] } + +/** Narrow capabilities lent by the everyday workflow; no direct database or runner hooks. */ +export interface ReliabilityFlowContext { + page: Page; + api: RunnerApi; + execution: MatrixExecution; + nonce: string; + leadAgentId: string; + helperAgentId?: string; + workspacePath: string; + deadlineAt: number; + parentId: string; + refresh(): Promise; + /** Composer reply; returns the stored user comment IDs. */ + reply(message: string): Promise; + openParent(): Promise; + restart(): Promise; + note(action: string, detail?: unknown): void; + allowInterrupted(runId: string): void; + /** Record failures whose work a later run verifiably resumed. */ + allowRecovered(runIds: readonly string[]): void; + evidence(name: string, value: unknown): Promise; + capture(id: string, label: string, file: string): Promise; +} + +export interface ReliabilityOutcome { + expectedReply?: StoryReplyExpectation; + /** Agents permitted to execute on the parent task (reassignment moves ownership). */ + parentAgentIds: string[]; + checks(): Promise; +} + +const sleep = (milliseconds: number) => new Promise(resolve => setTimeout(resolve, milliseconds)); + +/** Interrupted work resumes automatically, or a Retry works on first use; otherwise fail. */ +class ResumptionWatcher { + readonly observations: ResumptionObservation[] = []; + readonly retryClicks: RetryClick[] = []; + stalled?: string; + constructor(private readonly ctx: ReliabilityFlowContext) {} + + async observe(state: StoryState) { + const issues = state.issues.filter(issue => issue.id === this.ctx.parentId || issue.parentId === this.ctx.parentId); + for (const issue of issues) { + const [detail, execution] = await Promise.all([ + this.ctx.api.get(`/api/issues/${issue.id}`), + this.ctx.api.get(`/api/issues/${issue.id}/execution`).catch(() => null), + ]); + const observation: ResumptionObservation = { + at: new Date().toISOString(), issueId: issue.id, + blocker: detail.executionBlocker ?? null, interruption: detail.interruption ?? null, + execution: execution?.execution ? { phase: execution.execution.phase, label: execution.execution.label, cause: execution.execution.cause } : null, + }; + this.observations.push(observation); + // An automatic retry that is already scheduled is the harness doing its job; wait for it. + const automatic = Boolean(detail.scheduledRetry) || ["working", "retry_scheduled", "resolved"].includes(detail.interruption?.state); + const offered = detail.interruption?.canRetry === true || detail.executionBlocker?.canRetry === true; + if (offered && !automatic && detail.status !== "done" + && !state.runs.some(run => isActiveStoryRun(run) && runIssueId(run) === issue.id)) + await this.retry(issue.id, detail.interruption?.observedRunId ?? detail.executionBlocker?.runId ?? null); + } + } + + /** Use an offered Retry once and require a new run on that task. */ + private async retry(issueId: string, observedRunId: string | null) { + const click: RetryClick = { at: new Date().toISOString(), issueId }; + this.retryClicks.push(click); + if (this.retryClicks.filter(previous => previous.issueId === issueId).length > 1) return; + const before = new Set((await this.ctx.refresh()).runs.map(run => run.id)); + const response = await this.ctx.api.post(`/api/issues/${issueId}/retry`, { idempotencyKey: randomUUID(), observedRunId }); + this.ctx.note("reliability-retry-used", { issueId, observedRunId, accepted: response.accepted, stale: response.stale }); + const started = await pollUntil({ + label: "run started by Retry", deadlineAt: Math.min(this.ctx.deadlineAt, Date.now() + 120_000), intervalMs: 2_000, + load: () => this.ctx.refresh(), + accept: state => state.runs.some(run => !before.has(run.id) && runIssueId(run) === issueId), + }).catch(() => undefined); + click.startedRunId = started?.runs.find(run => !before.has(run.id) && runIssueId(run) === issueId)?.id; + } + + /** Poll until accepted, recording blockers and using any offered Retry. */ + async until(label: string, accept: (state: StoryState) => boolean, boundMs: number, + onPoll?: () => Promise): Promise { + const deadlineAt = Math.min(this.ctx.deadlineAt, Date.now() + boundMs); + let last: StoryState | undefined; + while (Date.now() < deadlineAt) { + await onPoll?.(); + last = await this.ctx.refresh(); + if (accept(last)) return last; + await this.observe(last); + await sleep(3_000); + } + this.stalled = `Stalled waiting for ${label} (bound ${Math.round(boundMs / 1000)}s).`; + throw new Error(this.stalled); + } + + checks() { + return gradeResumptionRule({ observations: this.observations, retryClicks: this.retryClicks, stalled: this.stalled }); + } +} + +const allDone = (state: StoryState) => state.issues.length > 0 + && state.issues.every(issue => issue.status === "done" && (issue.queuedComments?.entries?.length ?? 0) === 0) + && !state.runs.some(isActiveStoryRun); + +/** + * A claimed run is not a provider turn: require a completed native tool event, + * unless the first tool is itself the long command under test. + */ +async function waitForActiveTurn(ctx: ReliabilityFlowContext, issueId = ctx.parentId, label = "active provider turn", + requireTool = true): Promise { + const state = await pollUntil({ + label, deadlineAt: ctx.deadlineAt, intervalMs: 2_000, + load: async () => { + const current = await ctx.refresh(); + const run = current.runs.find(candidate => candidate.status === "running" && runIssueId(candidate) === issueId); + if (!run) return { current, run: undefined, tool: false }; + if (!requireTool) return { current, run, tool: true }; + const events = await collectRunEvents((afterSeq, limit) => + ctx.api.get(`/api/heartbeat-runs/${run.id}/events?afterSeq=${afterSeq}&limit=${limit}`)); + return { current, run, tool: events.some(row => row.payload?.prpEvent?.eventType === "tool.execution.completed") }; + }, + accept: value => Boolean(value.run && value.tool), + reject: value => value.current.issues.find(issue => issue.id === issueId)?.status === "done" + && !value.current.runs.some(isActiveStoryRun) ? "Boundary unexercised: the task finished before a turn could be interrupted" : undefined, + }); + ctx.note("reliability-active-turn", { issueId, runId: state.run!.id }); + return state.run!; +} + +/** Saved workspace bytes through the public file API, or the local project path. */ +async function readWorkspaceFile(ctx: ReliabilityFlowContext, file: string): Promise { + try { + const value = await ctx.api.get(`/api/issues/${ctx.parentId}/file-resources/content?workspace=execution&path=${encodeURIComponent(file)}`); + if (value.content?.encoding === "utf8" && typeof value.content.data === "string") return value.content.data; + } catch { /* Fall back to the local project checkout below. */ } + if (ctx.execution.environment.id !== "local") return undefined; + return readFile(path.join(ctx.workspacePath, file), "utf8").catch(() => undefined); +} + +async function queue(ctx: ReliabilityFlowContext): Promise { + return ctx.api.get(`/api/issues/${ctx.parentId}/queued-comments`); +} + +/** Issue an HTTP mutation without throwing, keeping its status and error code. */ +async function send(ctx: ReliabilityFlowContext, method: "post" | "patch" | "delete", url: string, data: unknown) { + const startedAt = Date.now(); + const response = await ctx.api.request[method](url, { data }); + const body = await response.json().catch(() => ({})) as Row; + return { status: response.status(), latencyMs: Date.now() - startedAt, body, code: typeof body.code === "string" ? body.code : undefined }; +} + +async function interruptRedirect(ctx: ReliabilityFlowContext, watcher: ResumptionWatcher): Promise { + const active = await waitForActiveTurn(ctx); + const [commentId] = await ctx.reply(interruptRedirectMessage(ctx.nonce)); + if (!commentId) throw new Error("The interrupt follow-up was not stored"); + const interrupt = ctx.page.getByTestId(`task-chat-queued-interrupt-${commentId}`); + await expect(interrupt).toBeVisible({ timeout: 30_000 }); + ctx.allowInterrupted(active.id); + const interruptedAt = Date.now(); + await interrupt.click(); + ctx.note("reliability-interrupt-clicked", { runId: active.id, commentId }); + await watcher.until("interrupt continuation finished", state => allDone(state) + && runsCarrying(state.runs, commentId).some(run => run.status === "succeeded"), RELIABILITY_BOUNDS.continuationStartMs + 10 * 60_000); + await ctx.capture("reliability-interrupt", "Interrupted follow-up delivered once", "reliability-interrupt.png"); + + // Stop, then send while the stopped run's save is still settling: the workspace_busy retry path. + await ctx.reply(slowStepsPrompt("more-steps.txt", 6, 30)); + const stopped = await waitForActiveTurn(ctx, ctx.parentId, "second active provider turn"); + ctx.allowInterrupted(stopped.id); + await ctx.page.getByTestId("task-chat-composer-stop").last().click(); + ctx.note("reliability-stop-clicked", { runId: stopped.id }); + const sentAt = Date.now(); + const [settleCommentId] = await ctx.reply(stopSettleMessage(ctx.nonce)); + if (!settleCommentId) throw new Error("The message sent during Stop settlement was not stored"); + await watcher.until("message sent during Stop settlement consumed", state => allDone(state) + && runsCarrying(state.runs, settleCommentId).some(run => run.status === "succeeded"), RELIABILITY_BOUNDS.continuationStartMs + 10 * 60_000); + return { + expectedReply: `Settled ${ctx.nonce}`, parentAgentIds: [ctx.leadAgentId], + checks: async () => { + const state = await ctx.refresh(); + const parent = state.issues.find(issue => issue.id === ctx.parentId); + return gradeInterruptRedirect({ runs: state.runs, comments: parent?.comments ?? [], interruptedRunId: active.id, + interruptCommentId: commentId, interruptedAt, stopSettle: { stoppedRunId: stopped.id, commentId: settleCommentId, sentAt } }); + }, + }; +} + +async function reassignMidrun(ctx: ReliabilityFlowContext, watcher: ResumptionWatcher): Promise { + if (!ctx.helperAgentId) throw new Error("Reassignment requires the second agent before task creation"); + const active = await waitForActiveTurn(ctx); + ctx.allowInterrupted(active.id); + const requestedAt = Date.now(); + const response = await send(ctx, "patch", `/api/issues/${ctx.parentId}`, { assigneeAgentId: ctx.helperAgentId }); + ctx.note("reliability-reassigned", { runId: active.id, status: response.status, latencyMs: response.latencyMs, code: response.code }); + if (response.status >= 300) throw new Error(`Reassignment returned ${response.status} ${response.code ?? ""}`); + await ctx.openParent(); + await watcher.until("reassigned task finished", state => allDone(state) + && executedRuns(state.runs).some(run => run.agentId === ctx.helperAgentId && run.status === "succeeded"), 20 * 60_000); + return { + parentAgentIds: [ctx.leadAgentId, ctx.helperAgentId], + checks: async () => gradeReassignMidrun({ runs: (await ctx.refresh()).runs, issueId: ctx.parentId, oldRunId: active.id, + newAgentId: ctx.helperAgentId!, request: { status: response.status, latencyMs: response.latencyMs, requestedAt }, + stepFile: await readWorkspaceFile(ctx, "steps.txt"), steps: 8 }), + }; +} + +async function threeTurnRevise(ctx: ReliabilityFlowContext, watcher: ResumptionWatcher): Promise { + const turnBound = 12 * 60_000; + // Turn 1 is the last successful parent run before the first revision (it may follow an automatic resume). + const turnOne = (await watcher.until("turn 1 finished", allDone, turnBound)).runs + .filter(run => runIssueId(run) === ctx.parentId && run.status === "succeeded") + .sort((a, b) => Date.parse(a.finishedAt ?? "") - Date.parse(b.finishedAt ?? "")).at(-1); + const commentIds: string[] = []; + for (const [index, revision] of threeTurnRevisions(ctx.nonce).entries()) { + const [commentId] = await ctx.reply(revision); + if (!commentId) throw new Error(`Revision ${index + 2} was not stored`); + commentIds.push(commentId); + await watcher.until(`turn ${index + 2} finished`, state => allDone(state) + && runsCarrying(state.runs, commentId).some(run => run.status === "succeeded"), turnBound); + ctx.note("reliability-turn-finished", { turn: index + 2, commentId }); + } + return { + parentAgentIds: [ctx.leadAgentId], + checks: async () => { + const state = await ctx.refresh(); + return gradeThreeTurnRevise({ nonce: ctx.nonce, content: await readWorkspaceFile(ctx, "items.md"), + turnRuns: [turnOne, ...commentIds.map(id => runsCarrying(state.runs, id).find(run => run.status === "succeeded") + ?? runsCarrying(state.runs, id).at(-1))] }); + }, + }; +} + +async function longSilentCommand(ctx: ReliabilityFlowContext, watcher: ResumptionWatcher): Promise { + const commandMs = LONG_SILENT_ITERATIONS * LONG_SILENT_ITERATION_SECONDS * 1000; + const run = await waitForActiveTurn(ctx, ctx.parentId, "silent command run", false); + const samples: Array<{ at: string; canStop: boolean; source: "api" | "browser" }> = []; + const browserAt = new Set([0, Math.floor(LONG_SILENT_ITERATIONS / 2), LONG_SILENT_ITERATIONS - 1]); + const bound = Date.now() + commandMs + RELIABILITY_BOUNDS.silentCompletionAfterCommandMs; + for (let minute = 0; Date.now() < Math.min(bound, ctx.deadlineAt); minute += 1) { + const current = await ctx.api.get(`/api/heartbeat-runs/${run.id}`); + if (current.status !== "running") break; + const projection = await ctx.api.get(`/api/issues/${ctx.parentId}/execution`).catch(() => null); + samples.push({ at: new Date().toISOString(), canStop: projection?.execution?.canStop === true, source: "api" }); + if (browserAt.has(minute)) { + await ctx.openParent(); + samples.push({ at: new Date().toISOString(), source: "browser", + canStop: await ctx.page.getByTestId("task-chat-composer-stop").last().isVisible().catch(() => false) }); + } + await sleep(60_000); + } + ctx.note("reliability-stop-samples", samples); + await watcher.until("silent command run finished", allDone, RELIABILITY_BOUNDS.silentCompletionAfterCommandMs); + return { + expectedReply: body => body.includes(`SILENT_DONE_${ctx.nonce}`), parentAgentIds: [ctx.leadAgentId], + checks: async () => { + const state = await ctx.refresh(); + const reply = state.issues.find(issue => issue.id === ctx.parentId)?.comments + ?.filter(comment => comment.authorAgentId).map(comment => String(comment.body ?? "")).join("\n"); + return gradeLongSilentCommand({ run: state.runs.find(candidate => candidate.id === run.id), content: await readWorkspaceFile(ctx, "silent.txt"), + iterations: LONG_SILENT_ITERATIONS, iterationSeconds: LONG_SILENT_ITERATION_SECONDS, stopSamples: samples, reply, nonce: ctx.nonce }); + }, + }; +} + +async function treePauseRelease(ctx: ReliabilityFlowContext, watcher: ResumptionWatcher): Promise { + const childrenOf = (state: StoryState) => state.issues.filter(issue => issue.parentId === ctx.parentId); + const waitingStatuses = ["queued", "scheduled_retry"]; + const boundary = await pollUntil({ + label: "delegated child running with a sibling queued", deadlineAt: ctx.deadlineAt, intervalMs: 2_000, + load: async () => { + const state = await ctx.refresh(); + const children = childrenOf(state); + const childRuns = state.runs.filter(run => children.some(child => child.id === runIssueId(run))); + return { state, children, running: childRuns.find(run => run.status === "running"), waiting: childRuns.find(run => waitingStatuses.includes(run.status)), + allActive: children.length >= 2 && children.every(child => childRuns.some(run => runIssueId(run) === child.id && isActiveStoryRun(run))) }; + }, + // A queued sibling or parked workspace_busy retry is the S50 boundary; two concurrently active children also qualify. + accept: value => value.children.length >= 2 && (Boolean(value.waiting) || value.allActive), + reject: value => value.children.length >= 2 && value.children.every(child => child.status === "done") + ? "Boundary unexercised: both children finished before the hold" : undefined, + }); + const childIds = boundary.children.map(child => child.id); + ctx.note("reliability-tree-boundary", { childIds, running: boundary.running?.id, waiting: boundary.waiting?.id, waitingStatus: boundary.waiting?.status }); + const holdCreatedAt = Date.now(); + const created = await ctx.api.post(`/api/issues/${ctx.parentId}/tree-holds`, { mode: "pause", reason: `Reliability eval pause ${ctx.nonce}` }); + const holdId = String(created.hold?.id ?? ""); + const interruptedRuns = ((created.preview?.activeRuns ?? []) as Row[]).map(run => ({ id: String(run.id), issueId: String(run.issueId) })); + for (const run of interruptedRuns) ctx.allowInterrupted(run.id); + ctx.note("reliability-tree-held", { holdId, interruptedRuns }); + if (!holdId) throw new Error("Tree hold was not created"); + // Hold past the parked retry's due time so a premature dispatch would be observed. + const dueAt = Math.max(0, ...boundary.state.runs.map(run => Date.parse(String((run as Row).scheduledRetryAt ?? ""))).filter(Number.isFinite)); + const holdUntil = Math.min(ctx.deadlineAt, Math.max(holdCreatedAt + RELIABILITY_BOUNDS.holdObservationMs, dueAt + 15_000), holdCreatedAt + 5 * 60_000); + while (Date.now() < holdUntil) { + const state = await ctx.refresh(); + // A run that started while the hold request was in flight is one the hold interrupted. + const early = executedRuns(state.runs).find(run => childIds.includes(runIssueId(run) ?? "") + && !interruptedRuns.some(interrupted => interrupted.id === run.id) && Date.parse(run.startedAt!) > holdCreatedAt); + if (early) throw new Error(`Child run ${early.id} started during the tree hold`); + await sleep(5_000); + } + const releasedAt = Date.now(); + await ctx.api.post(`/api/issues/${ctx.parentId}/tree-holds/${holdId}/release`, { reason: `Reliability eval release ${ctx.nonce}` }); + ctx.note("reliability-tree-released", { holdId, wakeAgents: false }); + await ctx.openParent(); + await watcher.until("released tree finished", allDone, 20 * 60_000); + return { + parentAgentIds: [ctx.leadAgentId], + checks: async () => gradeTreePauseRelease({ runs: (await ctx.refresh()).runs, childIds, holdCreatedAt, releasedAt, interruptedRuns }), + }; +} + +async function queueSteer(ctx: ReliabilityFlowContext, watcher: ResumptionWatcher): Promise { + const messages = queueSteerMessages(ctx.nonce); + const active = await waitForActiveTurn(ctx); + const post = async (body: string) => (await ctx.api.post(`/api/issues/${ctx.parentId}/comments`, { body })).id as string; + const editedId = await post(messages.original); + const discardedId = await post(messages.discarded); + const steeredId = await post(messages.steered); + ctx.note("reliability-queued", { editedId, discardedId, steeredId, runId: active.id }); + let current = await pollUntil({ label: "three queued follow-ups", deadlineAt: ctx.deadlineAt, intervalMs: 1_000, + load: () => queue(ctx), accept: value => [editedId, discardedId, steeredId] + .every(id => (value.entries ?? []).some((entry: Row) => entry.comment?.id === id)) }); + const edit = await send(ctx, "patch", `/api/issues/${ctx.parentId}/queued-comments/${editedId}`, + { queueId: current.queueId, revision: current.revision, body: messages.edited }); + if (edit.status >= 300) throw new Error(`Queued edit returned ${edit.status} ${edit.code ?? ""}`); + current = await queue(ctx); + const discard = await send(ctx, "delete", `/api/issues/${ctx.parentId}/queued-comments/${discardedId}`, + { queueId: current.queueId, revision: current.revision }); + if (discard.status >= 300) throw new Error(`Queued discard returned ${discard.status} ${discard.code ?? ""}`); + const steerAttempts: Array<{ status: number; code?: string }> = []; + const steerUntil = Date.now() + RELIABILITY_BOUNDS.steeringRetryWindowMs; + for (;;) { + current = await queue(ctx); + const attempt = await send(ctx, "post", `/api/issues/${ctx.parentId}/queued-comments/${steeredId}/steer`, + { queueId: current.queueId, revision: current.revision, targetRunId: current.targetRunId ?? active.id }); + steerAttempts.push({ status: attempt.status, code: attempt.code }); + if (attempt.status < 300 || attempt.code !== "steering_temporarily_unavailable" || Date.now() > steerUntil) break; + await sleep(5_000); + } + ctx.note("reliability-steer-attempts", steerAttempts); + await watcher.until("queued follow-ups finished", state => allDone(state) + && runsCarrying(state.runs, editedId).some(run => run.status === "succeeded"), 20 * 60_000); + return { + parentAgentIds: [ctx.leadAgentId], + checks: async () => gradeQueueSteer({ runs: (await ctx.refresh()).runs, activeRunId: active.id, steerAttempts, + edited: { commentId: editedId, originalMarker: `ORIGINAL_${ctx.nonce}`, editedMarker: `EDITED_${ctx.nonce}` }, + discarded: { commentId: discardedId, marker: `DISCARDED_${ctx.nonce}` }, + steered: { commentId: steeredId, marker: `STEERED_${ctx.nonce}` }, + files: { edited: await readWorkspaceFile(ctx, "edited.txt"), discarded: await readWorkspaceFile(ctx, "discarded.txt"), + steered: await readWorkspaceFile(ctx, "steered.txt") } }), + }; +} + +async function queuedAcrossRestart(ctx: ReliabilityFlowContext, watcher: ResumptionWatcher): Promise { + const active = await waitForActiveTurn(ctx); + const [commentId] = await ctx.reply(queuedRestartMessage(ctx.nonce)); + if (!commentId) throw new Error("The queued follow-up was not stored"); + // The original turn may be interrupted by shutdown; its recovery is graded by the resumption rule. + ctx.allowInterrupted(active.id); + const dispatchedBeforeRestart = runsCarrying((await ctx.refresh()).runs, commentId).length > 0; + // A healthy replacement may dispatch before the restart handshake returns; bound from the request. + const restartStartedAt = Date.now(); + await ctx.restart(); + ctx.note("reliability-controller-restarted", { runId: active.id, commentId, dispatchedBeforeRestart, readyAfterMs: Date.now() - restartStartedAt }); + const waitObservations: Array<{ at: string; phase?: string; label?: string; cause?: string | null }> = []; + await watcher.until("queued follow-up dispatched after restart", state => runsCarrying(state.runs, commentId).length > 0, + RELIABILITY_BOUNDS.restartDispatchMs + 60_000, async () => { + const value = await ctx.api.get(`/api/issues/${ctx.parentId}/execution`).catch(() => null); + if (value?.execution) waitObservations.push({ at: new Date().toISOString(), phase: value.execution.phase, + label: value.execution.label, cause: value.execution.cause }); + }); + await ctx.openParent(); + await watcher.until("queued follow-up finished", allDone, 15 * 60_000); + return { + expectedReply: `Restarted ${ctx.nonce}`, parentAgentIds: [ctx.leadAgentId], + checks: async () => gradeQueuedAcrossRestart({ runs: (await ctx.refresh()).runs, commentId, restartStartedAt, + dispatchedBeforeRestart, waitObservations }), + }; +} + +const scenarios: Record Promise> = { + "interrupt-redirect": interruptRedirect, + "reassign-midrun": reassignMidrun, + "three-turn-revise": threeTurnRevise, + "long-silent-command": longSilentCommand, + "tree-pause-release": treePauseRelease, + "queue-steer": queueSteer, + "queued-across-restart": queuedAcrossRestart, +}; + +/** Drive one reliability scenario after task creation; checks combine its grader and the resumption rule. */ +export async function driveReliabilityScenario(caseId: EverydayReliabilityCaseId, ctx: ReliabilityFlowContext): Promise { + ctx.note("reliability-scenario", { caseId, ...everydayReliabilitySignatures[caseId] }); + const watcher = new ResumptionWatcher(ctx); + try { + const outcome = await scenarios[caseId](ctx, watcher); + const recovered = recoveredRunIds((await ctx.refresh()).runs); + ctx.allowRecovered(recovered); + if (recovered.length) ctx.note("reliability-recovered-runs", recovered); + return { ...outcome, checks: async () => { + const checks = [...await outcome.checks(), ...watcher.checks()]; + await ctx.evidence("reliability-observations.json", { caseId, signature: everydayReliabilitySignatures[caseId], + observations: watcher.observations, retryClicks: watcher.retryClicks, checks }); + return checks; + } }; + } catch (error) { + await ctx.evidence("reliability-observations.json", { caseId, signature: everydayReliabilitySignatures[caseId], + observations: watcher.observations, retryClicks: watcher.retryClicks, stalled: watcher.stalled, + error: error instanceof Error ? error.message : String(error) }); + throw error; + } +} diff --git a/tests/runner-e2e/everyday-reliability.test.ts b/tests/runner-e2e/everyday-reliability.test.ts new file mode 100644 index 00000000000..584b5632ccc --- /dev/null +++ b/tests/runner-e2e/everyday-reliability.test.ts @@ -0,0 +1,253 @@ +import { describe, expect, it } from "vitest"; +import { buildMatrixJobs, parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js"; +import { everydayReliabilityFacadeProfileIds, runnerMatrix } from "./catalog.js"; +import { scheduledSelection } from "./scheduled-operations.js"; +import { + EVERYDAY_RELIABILITY_GROUP, everydayReliabilityCaseIds, everydayReliabilityExecutionIds, everydayReliabilitySignatures, + everydayReliabilityTasks, isEverydayReliabilityCase, +} from "./everyday-reliability-cases.js"; +import { + completeStepLines, concurrentExecutors, recoveredRunIds, gradeInterruptRedirect, gradeLongSilentCommand, gradeQueueSteer, + gradeQueuedAcrossRestart, gradeReassignMidrun, gradeResumptionRule, gradeThreeTurnRevise, gradeTreePauseRelease, + requiresUserMessage, +} from "./everyday-reliability.js"; +import { isExpectedStoryInterruption, storyLifecycleChecks, type StoryCheck, type StoryRun } from "./everyday-observations.js"; + +const T0 = Date.parse("2026-11-06T10:00:00.000Z"); +const at = (seconds: number) => new Date(T0 + seconds * 1000).toISOString(); +const run = (id: string, overrides: Partial = {}): StoryRun => ({ + id, companyId: "company", agentId: "lead", status: "succeeded", runtimeMode: "native", runnerInstanceId: "runner", + nativeIssueId: "parent", startedAt: at(0), finishedAt: at(60), ...overrides, +}); +/** A run woken to deliver `wake`; its prompt holds `history` (the whole conversation so far). */ +const carrying = (id: string, wake: string, overrides: Partial = {}, history = "") => + run(id, { contextSnapshot: { issueId: "parent", wakeCommentIds: [wake] }, + runnerProfileJson: { nativeExecutionInput: { task: { prompt: `${history} ${wake}` } } }, ...overrides }); +const reliabilityCellCount = everydayReliabilityCaseIds.length + everydayReliabilityFacadeProfileIds.length; +const failed = (checks: StoryCheck[]) => checks.filter(check => !check.passed).map(check => check.id); + +describe("reliability selection stays explicit", () => { + const reliable = (executions: { groups: readonly string[] }[]) => executions.filter(execution => execution.groups.includes(EVERYDAY_RELIABILITY_GROUP)); + + it("selects the whole reliability set only through its group", () => { + const selected = selectRunnerExecutions(parseRunnerSelectors(["--suite", "everyday-workflows", "--group", "reliability"])); + expect(selected.map(execution => execution.id).sort()) + .toEqual(everydayReliabilityExecutionIds("everyday-workflows", everydayReliabilityFacadeProfileIds).sort()); + expect(selected).toHaveLength(reliabilityCellCount); + expect(selected.every(execution => execution.environment.id === "daytona")).toBe(true); + }); + + it("is excluded from --all, the plain suite, generic selectors and scheduled campaigns", () => { + for (const args of [["--all"], ["--suite", "everyday-workflows"], ["--suite", "everyday-workflows", "--environment", "daytona"], + ["--suite", "everyday-workflows", "--profile", "runner-codex"], ["--suite", "everyday-workflows", "--group", "native"], + ["--profile", "runner-codex"], ["--environment", "daytona"]]) { + expect(reliable(selectRunnerExecutions(parseRunnerSelectors(args))), args.join(" ")).toEqual([]); + } + expect(selectRunnerExecutions(parseRunnerSelectors(["--suite", "everyday-workflows"]))).toHaveLength(52); + for (const cadence of ["daily", "weekly"] as const) + expect(scheduledSelection(cadence).filter(id => id.startsWith("everyday-workflows."))).toEqual([]); + }); + + it("allows a single case or exact ID and keeps the cells discoverable in --list", () => { + expect(selectRunnerExecutions(parseRunnerSelectors(["--suite", "everyday-workflows", "--case", "long-silent-command"])) + .map(execution => execution.profile.id).sort()).toEqual(["runner-codex", ...everydayReliabilityFacadeProfileIds].sort()); + expect(selectRunnerExecutions(parseRunnerSelectors(["--id", "everyday-workflows.runner-codex.daytona.queue-steer"]))).toHaveLength(1); + expect(reliable(selectRunnerExecutions(parseRunnerSelectors(["--list"])))).toHaveLength(reliabilityCellCount); + expect(selectRunnerExecutions(parseRunnerSelectors(["--suite", "everyday-workflows", "--group", "slow"])) + .every(execution => execution.task.id === "long-silent-command")).toBe(true); + }); + + it("uses one native Daytona profile per case and facades only for the silent command", () => { + const cells = runnerMatrix.filter(execution => isEverydayReliabilityCase(execution.task.id)); + expect(cells).toHaveLength(reliabilityCellCount); + for (const caseId of everydayReliabilityCaseIds) { + const profiles = cells.filter(execution => execution.task.id === caseId).map(execution => execution.profile.id); + expect(profiles).toContain("runner-codex"); + expect(profiles.length).toBe(caseId === "long-silent-command" ? 1 + everydayReliabilityFacadeProfileIds.length : 1); + } + expect(runnerMatrix.filter(execution => execution.suite.id === "everyday-workflows" + && ["runner-acpx-grok", "runner-acpx-kimi", "runner-acpx-pi"].includes(execution.profile.id) + && execution.task.id !== "long-silent-command")).toEqual([]); + const turnThree = cells.find(execution => execution.task.id === "three-turn-revise")!; + expect(turnThree.environment.configurationKey).toBeUndefined(); // Per-turn ephemeral lease, as in S49. + expect(cells.find(execution => execution.task.id === "long-silent-command")!.groups).toContain("slow"); + }); + + it("gives the slow case its own deadline and single attempts", () => { + const silent = everydayReliabilityTasks.find(task => task.id === "long-silent-command")!; + expect(silent.attemptTimeoutMs.daytona).toBeGreaterThanOrEqual(45 * 60_000); + expect(everydayReliabilityTasks.every(task => task.automaticRetryPolicy === "single_attempt")).toBe(true); + const [job] = buildMatrixJobs(runnerMatrix.filter(execution => execution.id === "everyday-workflows.runner-codex.daytona.long-silent-command")); + expect(job!.timeoutMinutes).toBeGreaterThan(100); + }); + + it("requires the facade credentials before provider spend and records each soak signature", () => { + const credentials = (profile: string) => runnerMatrix.find(execution => + execution.id === `everyday-workflows.${profile}.daytona.long-silent-command`)!.requiredCredentials; + const facadeCredentials: Record = { "runner-acpx-grok": "XAI_API_KEY", "runner-acpx-kimi": "KIMI_API_KEY", "runner-acpx-pi": "OPENROUTER_API_KEY" }; + for (const profile of everydayReliabilityFacadeProfileIds) + expect(credentials(profile)).toEqual([facadeCredentials[profile], "DAYTONA_API_KEY"]); + expect(Object.keys(everydayReliabilitySignatures).sort()).toEqual([...everydayReliabilityCaseIds].sort()); + for (const task of everydayReliabilityTasks) + expect(task.buildPrompt("sample")).not.toMatch(/finish_task|paperclip_finish|PATCH|mark .*done|api\/issues/i); + }); +}); + +describe("reliability interruption bookkeeping", () => { + it("exempts only deliberate interrupt and reassignment stops", () => { + expect(isExpectedStoryInterruption(run("a", { status: "cancelled", errorCode: "operator_interrupted" }), ["a"])).toBe(true); + expect(isExpectedStoryInterruption(run("a", { status: "cancelled", errorCode: "issue_reassigned" }), ["a"])).toBe(true); + expect(isExpectedStoryInterruption(run("a", { status: "cancelled", errorCode: "issue_reassigned" }), [])).toBe(false); + expect(isExpectedStoryInterruption(run("a", { status: "cancelled", errorCode: "workspace_lost" }), ["a"])).toBe(false); + }); + it("lets a reassigned owner execute on the parent only when declared", () => { + const input = { issues: [{ id: "parent", companyId: "company", title: "t", status: "done" }], parentId: "parent", leadId: "lead", + runs: [run("a", { agentId: "riley", contextSnapshot: { issueId: "parent" } })] }; + expect(failed(storyLifecycleChecks(input))).toContain("parent-owned-by-lead"); + expect(failed(storyLifecycleChecks({ ...input, parentAgentIds: ["lead", "riley"] }))).toEqual([]); + }); +}); + +describe("product rule: resume automatically or offer a Retry that works", () => { + it("treats a failure as recovered only when a later run on the same task succeeds", () => { + const lost = run("lost", { status: "failed", errorCode: "runner_lost", finishedAt: at(60) }); + expect(recoveredRunIds([lost, run("retry", { startedAt: at(70), finishedAt: at(120) })])).toEqual(["lost"]); + expect(recoveredRunIds([lost])).toEqual([]); + expect(recoveredRunIds([lost, run("other", { nativeIssueId: "child", startedAt: at(70) })])).toEqual([]); + const lifecycle = { issues: [{ id: "parent", companyId: "company", title: "t", status: "done" }], parentId: "parent", leadId: "lead", + runs: [lost, run("retry", { startedAt: at(70), finishedAt: at(120) })] }; + expect(failed(storyLifecycleChecks(lifecycle))).toEqual(["successful-runs"]); + expect(failed(storyLifecycleChecks({ ...lifecycle, recoveredRuns: ["lost"] }))).toEqual([]); + }); + it("passes automatic resumption", () => expect(failed(gradeResumptionRule({ observations: [], retryClicks: [] }))).toEqual([])); + it("fails a send-a-message blocker, a dead Retry, a repeated Retry, and a stall", () => { + const blocker = { at: at(1), issueId: "parent", blocker: { cause: "x", nextAction: "Saved work recovered. Send a message to continue.", canRetry: false } }; + expect(requiresUserMessage(blocker)).toBe(true); + expect(requiresUserMessage({ ...blocker, blocker: { ...blocker.blocker, canRetry: true } })).toBe(false); + expect(requiresUserMessage({ at: at(1), issueId: "parent", interruption: { state: "waiting_for_user", canRetry: false, userAction: null } })).toBe(true); + expect(failed(gradeResumptionRule({ observations: [blocker], retryClicks: [] }))).toEqual(["resume.no-send-message-blocker"]); + expect(failed(gradeResumptionRule({ observations: [], retryClicks: [{ at: at(1), issueId: "parent" }] }))).toEqual(["resume.retry-works-first-use"]); + expect(failed(gradeResumptionRule({ observations: [], retryClicks: [ + { at: at(1), issueId: "parent", startedRunId: "r1" }, { at: at(2), issueId: "parent", startedRunId: "r2" }] }))).toEqual(["resume.retry-works-first-use"]); + expect(failed(gradeResumptionRule({ observations: [], retryClicks: [], stalled: "Stalled" }))).toEqual(["resume.no-silent-stall"]); + }); +}); + +describe("scenario graders", () => { + it("detects concurrent executors and exact step files", () => { + expect(concurrentExecutors([run("a", { finishedAt: at(60) }), run("b", { startedAt: at(61), finishedAt: at(90) })], "parent")).toEqual([]); + expect(concurrentExecutors([run("a", { finishedAt: at(60) }), run("b", { startedAt: at(30) })], "parent")).toEqual([["a", "b"]]); + expect(completeStepLines("step-1\nstep-2\n", 2)).toBe(true); + expect(completeStepLines("step-1\nstep-1\nstep-2\n", 2)).toBe(false); + expect(completeStepLines(undefined, 2)).toBe(false); + }); + + it("interrupt-redirect: one bounded continuation, and Stop-settlement consumption", () => { + const interrupted = run("old", { status: "cancelled", errorCode: "operator_interrupted", finishedAt: at(10) }); + const good = [interrupted, carrying("next", "comment-1", { startedAt: at(40), finishedAt: at(90) }), + run("stopped", { status: "cancelled", errorCode: "cancelled", startedAt: at(100), finishedAt: at(130) }), + carrying("settle", "comment-2", { startedAt: at(150), finishedAt: at(200) })]; + const input = { runs: good, comments: [], interruptedRunId: "old", interruptCommentId: "comment-1", interruptedAt: T0 + 10_000, + stopSettle: { stoppedRunId: "stopped", commentId: "comment-2", sentAt: T0 + 120_000 } }; + expect(failed(gradeInterruptRedirect(input))).toEqual([]); + expect(failed(gradeInterruptRedirect({ ...input, runs: [...good, carrying("dup", "comment-1", { startedAt: at(300) })] }))) + .toContain("interrupt.exactly-one-continuation"); + // Later turns keep the interrupt message in their prompt history; that is not a second delivery. + expect(failed(gradeInterruptRedirect({ ...input, runs: [...good, run("later", { startedAt: at(300), + runnerProfileJson: { nativeExecutionInput: { task: { prompt: "comment-1 comment-2 history" } } } })] }))).toEqual([]); + expect(failed(gradeInterruptRedirect({ ...input, runs: [interrupted, carrying("late", "comment-1", { startedAt: at(250) }), ...good.slice(2)] }))) + .toEqual(["interrupt.continuation-bounded"]); + expect(failed(gradeInterruptRedirect({ ...input, comments: [{ id: "nudge", createdAt: at(20) }] }))).toEqual(["interrupt.no-second-message"]); + expect(failed(gradeInterruptRedirect({ ...input, runs: [{ ...interrupted, errorCode: "cancelled" }, ...good.slice(1)] }))) + .toEqual(["interrupt.operator-interrupt"]); + }); + + it("reassign-midrun: prompt request, old run reassigned, no overlap, saved files continued", () => { + const steps = Array.from({ length: 8 }, (_, index) => `step-${index + 1}`).join("\n"); + const input = { issueId: "parent", oldRunId: "old", newAgentId: "riley", steps: 8, stepFile: steps as string | undefined, + request: { status: 200, latencyMs: 800, requestedAt: T0 + 30_000 }, + runs: [run("old", { status: "cancelled", errorCode: "issue_reassigned", finishedAt: at(32) }), + run("new", { agentId: "riley", startedAt: at(40), finishedAt: at(200) })] }; + expect(failed(gradeReassignMidrun(input))).toEqual([]); + expect(failed(gradeReassignMidrun({ ...input, request: { status: 409, latencyMs: 20_000, requestedAt: T0 } }))).toContain("reassign.request-prompt"); + expect(failed(gradeReassignMidrun({ ...input, runs: [run("old", { status: "cancelled", errorCode: "issue_reassigned", finishedAt: at(120) }), input.runs[1]!] }))) + .toEqual(["reassign.no-concurrent-executors"]); + expect(failed(gradeReassignMidrun({ ...input, stepFile: `${steps}\nstep-3` }))).toEqual(["reassign.continued-from-saved-files"]); + }); + + it("three-turn-revise: every turn succeeds and the table is complete", () => { + const nonce = "N1"; + const table = `| Number | Item |\n| --- | --- |\n${["alpha", "beta", "gamma", "delta", "epsilon"].map((word, index) => `| ${index + 1} | ${word} ${nonce} |`).join("\n")}\n`; + const turns = [run("t1"), run("t2"), run("t3")]; + expect(failed(gradeThreeTurnRevise({ turnRuns: turns, content: table, nonce }))).toEqual([]); + expect(failed(gradeThreeTurnRevise({ turnRuns: [turns[0], turns[1], undefined], content: table, nonce }))).toEqual(["three-turn.turn-3-succeeded"]); + expect(failed(gradeThreeTurnRevise({ turnRuns: turns, content: `${table}- alpha ${nonce}\n`, nonce }))).toEqual(["three-turn.final-table"]); + expect(failed(gradeThreeTurnRevise({ turnRuns: turns, content: table.replace(`epsilon ${nonce}`, "missing"), nonce }))).toEqual(["three-turn.final-table"]); + }); + + it("long-silent-command: completes after the command, Stop stays available, no hang", () => { + const content = Array.from({ length: 13 }, (_, index) => `tick-${index + 1}`).join("\n"); + const samples = [{ at: at(60), canStop: true, source: "api" as const }, { at: at(61), canStop: true, source: "browser" as const }, + { at: at(400), canStop: true, source: "api" as const }]; + const input = { run: run("silent", { startedAt: at(0), finishedAt: at(13 * 60 + 90) }), content, iterations: 13, iterationSeconds: 60, + stopSamples: samples, reply: "SILENT_DONE_N1 13", nonce: "N1" }; + expect(failed(gradeLongSilentCommand(input))).toEqual([]); + expect(failed(gradeLongSilentCommand({ ...input, run: run("silent", { status: "failed", errorCode: null, startedAt: at(0), finishedAt: at(13 * 60 + 60 * 30) }) }))) + .toEqual(expect.arrayContaining(["silent.completed", "silent.no-hang", "silent.named-failure"])); + expect(failed(gradeLongSilentCommand({ ...input, run: run("silent", { startedAt: at(0), finishedAt: at(120) }) }))).toEqual(["silent.not-shortened"]); + expect(failed(gradeLongSilentCommand({ ...input, stopSamples: [...samples, { at: at(500), canStop: false, source: "api" }] }))).toEqual(["silent.stop-available"]); + }); + + it("tree-pause-release: no start during hold and exactly one resume per interrupted task", () => { + const input = { childIds: ["a", "b"], holdCreatedAt: T0 + 100_000, releasedAt: T0 + 200_000, + interruptedRuns: [{ id: "ra", issueId: "a" }, { id: "rb", issueId: "b" }], + runs: [run("ra", { nativeIssueId: "a", status: "cancelled", finishedAt: at(101) }), + run("rb", { nativeIssueId: "b", status: "cancelled", startedAt: null, finishedAt: at(101) }), + run("ra2", { nativeIssueId: "a", startedAt: at(210), contextSnapshot: { wakeReason: "issue_tree_resumed" } }), + run("rb2", { nativeIssueId: "b", startedAt: at(260), contextSnapshot: { wakeReason: "issue_tree_resumed" } })] }; + expect(failed(gradeTreePauseRelease(input))).toEqual([]); + expect(failed(gradeTreePauseRelease({ ...input, runs: [...input.runs, run("early", { nativeIssueId: "b", startedAt: at(150) })] }))) + .toEqual(["tree.no-start-during-hold"]); + expect(failed(gradeTreePauseRelease({ ...input, runs: [...input.runs, run("rb3", { nativeIssueId: "b", startedAt: at(300), contextSnapshot: { wakeReason: "issue_tree_resumed" } })] }))) + .toEqual(["tree.one-resume-per-interrupted"]); + expect(failed(gradeTreePauseRelease({ ...input, runs: input.runs.slice(0, 3) }))).toEqual(["tree.one-resume-per-interrupted"]); + expect(failed(gradeTreePauseRelease({ ...input, interruptedRuns: [] }))).toContain("tree.hold-interrupted-work"); + // A run that started while the hold request was in flight was interrupted by it, not started during it. + expect(failed(gradeTreePauseRelease({ ...input, runs: input.runs.map(entry => entry.id === "ra" ? { ...entry, startedAt: at(100.5) } : entry) }))).toEqual([]); + }); + + it("queue-steer: edit used, discard never delivered, steer consumed once in the active turn", () => { + const active = run("active", { resultJson: { queuedSteeringAcknowledgements: { steer: { status: "acknowledged" } } } }); + const input = { activeRunId: "active", runs: [active, carrying("next", "edit", {}, "EDITED_N1 history: steer drop")], + edited: { commentId: "edit", originalMarker: "ORIGINAL_N1", editedMarker: "EDITED_N1" }, + discarded: { commentId: "drop", marker: "DISCARDED_N1" }, steered: { commentId: "steer", marker: "STEERED_N1" }, + steerAttempts: [{ status: 409, code: "steering_temporarily_unavailable" }, { status: 200 }], + files: { edited: "EDITED_N1", steered: "STEERED_N1" } }; + expect(failed(gradeQueueSteer(input))).toEqual([]); + expect(failed(gradeQueueSteer({ ...input, runs: [active, carrying("next", "edit", {}, "EDITED_N1 DISCARDED_N1")] }))) + .toEqual(["queue.discard-never-delivered"]); + expect(failed(gradeQueueSteer({ ...input, runs: [active, carrying("next", "edit", {}, "ORIGINAL_N1")], files: { ...input.files, edited: "ORIGINAL_N1" } }))) + .toEqual(["queue.edit-used"]); + expect(failed(gradeQueueSteer({ ...input, runs: [...input.runs, carrying("replay", "steer", { startedAt: at(200) }, "EDITED_N1")] }))) + .toEqual(["queue.steer-consumed-in-active-run"]); + expect(failed(gradeQueueSteer({ ...input, steerAttempts: [{ status: 409, code: "queued_comment_stale_target" }, { status: 200 }] }))) + .toEqual(["queue.steer-retry-bounded"]); + expect(failed(gradeQueueSteer({ ...input, runs: [...input.runs, carrying("again", "edit", { startedAt: at(300) }, "EDITED_N1")] }))) + .toEqual(["queue.remaining-handled-once"]); + }); + + it("queued-across-restart: bounded automatic dispatch with a named wait", () => { + const input = { commentId: "c1", restartStartedAt: T0 + 60_000, dispatchedBeforeRestart: false, + runs: [carrying("next", "c1", { startedAt: at(120), finishedAt: at(200) })], + waitObservations: [{ at: at(65), phase: "queued", label: "Waiting for server restart", cause: "server_restarting" }] }; + expect(failed(gradeQueuedAcrossRestart(input))).toEqual([]); + expect(failed(gradeQueuedAcrossRestart({ ...input, waitObservations: [{ at: at(65), phase: "queued", label: "Queued", cause: null }] }))) + .toEqual(["restart.named-wait"]); + expect(failed(gradeQueuedAcrossRestart({ ...input, runs: [] }))).toEqual(["restart.dispatched-bounded", "restart.exactly-once"]); + expect(failed(gradeQueuedAcrossRestart({ ...input, runs: [carrying("next", "c1", { startedAt: at(60 + 7 * 60) })] }))) + .toEqual(["restart.dispatched-bounded"]); + expect(failed(gradeQueuedAcrossRestart({ ...input, runs: [carrying("early", "c1", { startedAt: at(30) })] }))) + .toEqual(["restart.boundary-exercised"]); + expect(failed(gradeQueuedAcrossRestart({ ...input, dispatchedBeforeRestart: true }))).toEqual(["restart.boundary-exercised"]); + }); +}); diff --git a/tests/runner-e2e/everyday-reliability.ts b/tests/runner-e2e/everyday-reliability.ts new file mode 100644 index 00000000000..0a93317a357 --- /dev/null +++ b/tests/runner-e2e/everyday-reliability.ts @@ -0,0 +1,326 @@ +import { RELIABILITY_BOUNDS } from "./everyday-reliability-cases.js"; +import { isStoryWorkspaceDeferral, type StoryCheck, type StoryComment, type StoryRun } from "./everyday-observations.js"; + +/** + * Pure graders for the staging-soak reliability scenarios. Product rule: + * work interrupted by a restart, Stop, reassignment or lost runner resumes + * automatically, or offers a Retry that works on first use. "Send a message + * to continue", a dead Retry, a silent stall and duplicate execution all fail. + */ + +const check = (id: string, passed: boolean, detail: string): StoryCheck => ({ id, passed, detail }); +const ms = (value: string | null | undefined) => Date.parse(value ?? ""); +/** Clock granularity between separately written run timestamps. */ +const OVERLAP_TOLERANCE_MS = 2_000; + +export function runIssueId(run: StoryRun): string | undefined { + const context = run.contextSnapshot ?? {}; + return run.nativeIssueId ?? (typeof context.issueId === "string" ? context.issueId + : typeof context.taskId === "string" ? context.taskId : undefined); +} + +/** Serialized native execution input; a delivered message appears here by ID and body. */ +export function runNativeInput(run: StoryRun): string { + return JSON.stringify(run.runnerProfileJson?.nativeExecutionInput ?? null); +} + +/** Runs that reached provider execution (workspace deferrals never did). */ +export function executedRuns(runs: readonly StoryRun[]): StoryRun[] { + return runs.filter(run => !isStoryWorkspaceDeferral(run) && Boolean(run.startedAt)); +} + +/** + * Comments this run was woken to deliver. The saved prompt also holds the whole + * conversation history, so it cannot attribute delivery; the wake context can. + */ +export function runWakeCommentIds(run: StoryRun): string[] { + const context = run.contextSnapshot ?? {}; + const ids = Array.isArray(context.wakeCommentIds) ? context.wakeCommentIds : []; + return [...new Set([...ids, context.wakeCommentId, context.commentId].filter((id): id is string => typeof id === "string"))]; +} + +/** Executed runs woken to deliver this comment (a Retry of a failed delivery inherits it). */ +export function runsCarrying(runs: readonly StoryRun[], commentId: string): StoryRun[] { + return executedRuns(runs).filter(run => runWakeCommentIds(run).includes(commentId)); +} + +/** Successful deliveries of one comment; more than one is duplicate execution. */ +export const successfulDeliveries = (runs: readonly StoryRun[], commentId: string) => + runsCarrying(runs, commentId).filter(run => run.status === "succeeded"); + +/** + * Failed or interrupted runs whose task a later run completed successfully: the + * product rule's "resumed automatically or through a working Retry". The + * resumption checks separately fail send-a-message blockers and dead Retries. + */ +export function recoveredRunIds(runs: readonly StoryRun[]): string[] { + return executedRuns(runs).filter(run => ["failed", "timed_out", "cancelled", "interrupted"].includes(run.status) + && runs.some(later => later.status === "succeeded" && runIssueId(later) === runIssueId(run) + && ms(later.startedAt) >= ms(run.finishedAt))).map(run => run.id); +} + +const earliest = (runs: readonly StoryRun[]) => [...runs].sort((a, b) => ms(a.startedAt) - ms(b.startedAt))[0]; + +/** Pairs of runs that executed on the same task at the same time. */ +export function concurrentExecutors(runs: readonly StoryRun[], issueId: string, now = Date.now()): Array<[string, string]> { + const spans = executedRuns(runs).filter(run => runIssueId(run) === issueId) + .map(run => ({ id: run.id, start: ms(run.startedAt), end: Number.isFinite(ms(run.finishedAt)) ? ms(run.finishedAt) : now })) + .sort((left, right) => left.start - right.start); + const pairs: Array<[string, string]> = []; + for (let index = 0; index < spans.length; index += 1) + for (let other = index + 1; other < spans.length; other += 1) + if (spans[other]!.start + OVERLAP_TOLERANCE_MS < spans[index]!.end) pairs.push([spans[index]!.id, spans[other]!.id]); + return pairs; +} + +/** Exact "step-1".."step-N" lines, each once and in order. */ +export function completeStepLines(content: string | undefined, steps: number, prefix = "step"): boolean { + if (content === undefined) return false; + const lines = content.split(/\r?\n/).map(line => line.trim()).filter(Boolean); + return lines.length === steps && lines.every((line, index) => line === `${prefix}-${index + 1}`); +} + +export interface ResumptionObservation { + at: string; + issueId: string; + blocker?: { cause?: string | null; nextAction?: string | null; canRetry?: boolean; canContinue?: boolean } | null; + interruption?: { state?: string; canRetry?: boolean; userAction?: string | null; responsibility?: string } | null; + execution?: { phase?: string; label?: string; cause?: string | null } | null; +} +export interface RetryClick { at: string; issueId: string; startedRunId?: string } + +/** The only way forward is another user message: the S47/S49 failure mode. */ +export function requiresUserMessage(observation: ResumptionObservation): boolean { + const blocker = observation.blocker; + if (blocker && !blocker.canRetry && /send (?:a |another |a new )?(?:new )?(?:chat )?message/i.test(blocker.nextAction ?? "")) + return true; + const interruption = observation.interruption; + return Boolean(interruption && interruption.state === "waiting_for_user" && !interruption.canRetry && !interruption.userAction); +} + +export function gradeResumptionRule(input: { + observations: readonly ResumptionObservation[]; + retryClicks: readonly RetryClick[]; + stalled?: string; +}): StoryCheck[] { + const blocked = input.observations.find(requiresUserMessage); + const dead = input.retryClicks.filter(click => !click.startedRunId); + const repeated = new Set(input.retryClicks.map(click => click.issueId)).size !== input.retryClicks.length; + return [ + check("resume.no-send-message-blocker", !blocked, + blocked ? `Blocked at ${blocked.at}: ${blocked.blocker?.nextAction ?? blocked.interruption?.state}` : "Interrupted work never required another user message to continue."), + check("resume.retry-works-first-use", dead.length === 0 && !repeated, + input.retryClicks.length === 0 ? "No Retry was needed; work resumed automatically." + : dead.length ? `Retry clicked without starting a run at ${dead.map(click => click.at).join(", ")}.` + : repeated ? "Retry had to be used more than once for the same task." : "Each offered Retry started work on first use."), + check("resume.no-silent-stall", !input.stalled, input.stalled ?? "Work resumed or finished within the scenario bound."), + ]; +} + +const userCommentsBetween = (comments: readonly StoryComment[], from: number, to: number, except: readonly string[]) => + comments.filter(comment => !comment.authorAgentId && !except.includes(comment.id ?? "") + && ms(comment.createdAt) > from && ms(comment.createdAt) < to); + +export function gradeInterruptRedirect(input: { + runs: readonly StoryRun[]; + comments: readonly StoryComment[]; + interruptedRunId: string; + interruptCommentId: string; + interruptedAt: number; + stopSettle?: { stoppedRunId: string; commentId: string; sentAt: number }; +}): StoryCheck[] { + const interrupted = input.runs.find(run => run.id === input.interruptedRunId); + const carriers = successfulDeliveries(input.runs, input.interruptCommentId); + const continuation = earliest(runsCarrying(input.runs, input.interruptCommentId)); + const delay = ms(continuation?.startedAt) - input.interruptedAt; + const extra = continuation ? userCommentsBetween(input.comments, input.interruptedAt, ms(continuation.startedAt), [input.interruptCommentId]) : []; + const checks = [ + check("interrupt.operator-interrupt", interrupted?.status === "cancelled" + && (interrupted.errorCode === "operator_interrupted" || interrupted.resultJson?.operatorInterrupted === true), + `The interrupted run ends as an operator interrupt (observed ${interrupted?.status ?? "missing"}/${interrupted?.errorCode ?? "none"}).`), + check("interrupt.exactly-one-continuation", carriers.length === 1, + `Exactly one successful continuation delivers the interrupt message (observed ${carriers.length}).`), + check("interrupt.continuation-bounded", Number.isFinite(delay) && delay <= RELIABILITY_BOUNDS.continuationStartMs, + `The continuation starts within ${RELIABILITY_BOUNDS.continuationStartMs / 1000}s of the interrupt (observed ${Number.isFinite(delay) ? Math.round(delay / 1000) + "s" : "never"}).`), + check("interrupt.no-second-message", extra.length === 0, "No second user message was needed before the continuation started."), + ]; + if (input.stopSettle) { + const stopped = input.runs.find(run => run.id === input.stopSettle!.stoppedRunId); + const settleCarriers = successfulDeliveries(input.runs, input.stopSettle.commentId); + const settleDelay = ms(earliest(runsCarrying(input.runs, input.stopSettle.commentId))?.startedAt) - input.stopSettle.sentAt; + checks.push( + check("stop-settle.stopped", stopped?.status === "cancelled", "The stopped run is terminal as cancelled."), + check("stop-settle.consumed-once", settleCarriers.length === 1, + `A message sent while Stop was settling is consumed by exactly one successful run (observed ${settleCarriers.length}).`), + check("stop-settle.bounded", Number.isFinite(settleDelay) && settleDelay <= RELIABILITY_BOUNDS.continuationStartMs, + `The workspace_busy retry dispatches within ${RELIABILITY_BOUNDS.continuationStartMs / 1000}s (observed ${Number.isFinite(settleDelay) ? Math.round(settleDelay / 1000) + "s" : "never"}).`), + ); + } + return checks; +} + +export function gradeReassignMidrun(input: { + runs: readonly StoryRun[]; + issueId: string; + oldRunId: string; + newAgentId: string; + request: { status: number; latencyMs: number; requestedAt: number }; + stepFile?: string; + steps: number; +}): StoryCheck[] { + const old = input.runs.find(run => run.id === input.oldRunId); + // A failed first attempt followed by a working Retry is allowed: require any success. + const successor = executedRuns(input.runs).find(run => run.agentId === input.newAgentId && runIssueId(run) === input.issueId + && ms(run.startedAt) >= input.request.requestedAt && run.status === "succeeded"); + const overlaps = concurrentExecutors(input.runs, input.issueId); + return [ + check("reassign.request-prompt", input.request.status >= 200 && input.request.status < 300 + && input.request.latencyMs <= RELIABILITY_BOUNDS.reassignRequestMs, + `Reassignment returns ${input.request.status} in ${input.request.latencyMs}ms (bound ${RELIABILITY_BOUNDS.reassignRequestMs}ms, no 409).`), + check("reassign.old-run-ended", old?.status === "cancelled" && old.errorCode === "issue_reassigned", + `The original run ends issue_reassigned (observed ${old?.status ?? "missing"}/${old?.errorCode ?? "none"}).`), + check("reassign.new-assignee-finished", successor?.status === "succeeded", "The new assignee's run starts after the request and succeeds."), + check("reassign.no-concurrent-executors", overlaps.length === 0, + overlaps.length ? `Concurrent executions: ${overlaps.map(pair => pair.join("+")).join(", ")}.` : "Never two executions on the task at once."), + check("reassign.continued-from-saved-files", completeStepLines(input.stepFile, input.steps), + `The saved step file holds step-1..step-${input.steps} exactly once, so work continued from saved files.`), + ]; +} + +export function gradeThreeTurnRevise(input: { + turnRuns: ReadonlyArray; + content?: string; + nonce: string; +}): StoryCheck[] { + const items = ["alpha", "beta", "gamma", "delta", "epsilon"].map(word => `${word} ${input.nonce}`); + const rows = (input.content ?? "").split(/\r?\n/).map(line => line.trim()).filter(line => line.startsWith("|")); + const header = rows[0]?.toLowerCase() ?? ""; + const body = rows.slice(2); + const table = /number/.test(header) && /item/.test(header) && /^\|[\s:|-]+\|$/.test(rows[1] ?? "") + && body.length === items.length && body.every((row, index) => row.includes(items[index]!)); + const bullets = /^\s*[-*]\s+/m.test(input.content ?? ""); + return [ + ...input.turnRuns.map((run, index) => check(`three-turn.turn-${index + 1}-succeeded`, run?.status === "succeeded", + `Turn ${index + 1} completes in a successful run without a reconciliation stop (observed ${run?.status ?? "no run"}).`)), + check("three-turn.final-table", table && !bullets, "items.md is a Number/Item table with all five items in order and no bullet list."), + ]; +} + +export function gradeLongSilentCommand(input: { + run?: StoryRun; + content?: string; + iterations: number; + iterationSeconds: number; + stopSamples: ReadonlyArray<{ at: string; canStop: boolean; source: "api" | "browser" }>; + reply?: string; + nonce: string; +}): StoryCheck[] { + const commandMs = input.iterations * input.iterationSeconds * 1000; + const elapsed = ms(input.run?.finishedAt) - ms(input.run?.startedAt); + const named = input.run?.status === "succeeded" || Boolean(input.run?.errorCode); + return [ + check("silent.completed", input.run?.status === "succeeded", + `The run completes after the silent command (observed ${input.run?.status ?? "no run"}${input.run?.errorCode ? `/${input.run.errorCode}` : ""}).`), + check("silent.not-shortened", Number.isFinite(elapsed) && elapsed >= commandMs * 0.95, + `The run lasted at least the ${input.iterations}-minute command (observed ${Number.isFinite(elapsed) ? Math.round(elapsed / 1000) + "s" : "unknown"}).`), + check("silent.no-hang", Number.isFinite(elapsed) && elapsed <= commandMs + RELIABILITY_BOUNDS.silentCompletionAfterCommandMs, + `The run is terminal within ${RELIABILITY_BOUNDS.silentCompletionAfterCommandMs / 60_000} minutes after the command ends.`), + check("silent.named-failure", named, named ? "Any failure carries a named cause." : "The run ended without a named cause."), + check("silent.file-complete", completeStepLines(input.content, input.iterations, "tick"), + `silent.txt holds tick-1..tick-${input.iterations} exactly once.`), + check("silent.stop-available", input.stopSamples.length >= 3 && input.stopSamples.every(sample => sample.canStop) + && input.stopSamples.some(sample => sample.source === "browser"), + `Stop stays available throughout the silent interval (${input.stopSamples.filter(sample => sample.canStop).length}/${input.stopSamples.length} samples).`), + check("silent.reported", new RegExp(`SILENT_DONE_${input.nonce}\\s+${input.iterations}\\b`).test(input.reply ?? ""), + "The agent reports the completed line count after the command."), + ]; +} + +export function gradeTreePauseRelease(input: { + runs: readonly StoryRun[]; + childIds: readonly string[]; + holdCreatedAt: number; + releasedAt: number; + interruptedRuns: ReadonlyArray<{ id: string; issueId: string }>; +}): StoryCheck[] { + const executed = executedRuns(input.runs); + // A run that started while the hold request was in flight is one the hold interrupted. + const interruptedIds = new Set(input.interruptedRuns.map(run => run.id)); + const startedDuringHold = executed.filter(run => input.childIds.includes(runIssueId(run) ?? "") && !interruptedIds.has(run.id) + && ms(run.startedAt) > input.holdCreatedAt && ms(run.startedAt) < input.releasedAt); + const interruptedIssues = [...new Set(input.interruptedRuns.map(run => run.issueId))]; + const resumed = (issueId: string) => input.runs.filter(run => runIssueId(run) === issueId + && run.contextSnapshot?.wakeReason === "issue_tree_resumed" && !isStoryWorkspaceDeferral(run)); + const resumeCounts = interruptedIssues.map(issueId => ({ issueId, count: resumed(issueId).length })); + const stray = input.runs.filter(run => run.contextSnapshot?.wakeReason === "issue_tree_resumed" + && !interruptedIssues.includes(runIssueId(run) ?? "")); + return [ + check("tree.hold-interrupted-work", input.interruptedRuns.length > 0, + input.interruptedRuns.length ? `The hold interrupted ${input.interruptedRuns.length} run(s).` : "Boundary unexercised: no child work was active or queued when the hold was created."), + check("tree.no-start-during-hold", startedDuringHold.length === 0, + startedDuringHold.length ? `Child runs started during the hold: ${startedDuringHold.map(run => run.id).join(", ")}.` : "No child run started while the tree was paused."), + check("tree.one-resume-per-interrupted", resumeCounts.every(entry => entry.count === 1) && stray.length === 0, + `Each interrupted task gets exactly one issue_tree_resumed run (observed ${resumeCounts.map(entry => entry.count).join(", ") || "none"}; stray ${stray.length}).`), + ]; +} + +export function gradeQueueSteer(input: { + runs: readonly StoryRun[]; + activeRunId: string; + edited: { commentId: string; originalMarker: string; editedMarker: string }; + discarded: { commentId: string; marker: string }; + steered: { commentId: string; marker: string }; + steerAttempts: ReadonlyArray<{ status: number; code?: string }>; + files: { edited?: string; discarded?: string; steered?: string }; +}): StoryCheck[] { + const executed = executedRuns(input.runs); + const anyInput = (needle: string) => executed.some(run => runNativeInput(run).includes(needle)); + const active = input.runs.find(run => run.id === input.activeRunId); + const acks = (active?.resultJson?.queuedSteeringAcknowledgements ?? {}) as Record; + const editedCarriers = successfulDeliveries(input.runs, input.edited.commentId); + const steerReplays = runsCarrying(input.runs, input.steered.commentId); + const finalAttempt = input.steerAttempts.at(-1); + const retriesValid = input.steerAttempts.slice(0, -1).every(attempt => attempt.code === "steering_temporarily_unavailable"); + return [ + check("queue.edit-used", anyInput(input.edited.editedMarker) && !anyInput(input.edited.originalMarker) + && (input.files.edited ?? "").includes(input.edited.editedMarker), + "The edited text reached the agent; the original text never did."), + check("queue.remaining-handled-once", editedCarriers.length === 1, + `The remaining queued message is handled by exactly one successful run (observed ${editedCarriers.length}).`), + // History may retain a deleted comment's ID, but never its text or a delivery wake. + check("queue.discard-never-delivered", !anyInput(input.discarded.marker) && runsCarrying(input.runs, input.discarded.commentId).length === 0 + && !acks[input.discarded.commentId] && input.files.discarded === undefined, + "The discarded message never reached the agent."), + check("queue.steer-consumed-in-active-run", acks[input.steered.commentId]?.status === "acknowledged" && steerReplays.length === 0 + && (input.files.steered ?? "").includes(input.steered.marker), + "The steered message is acknowledged by the active turn, acted on there, and never replayed as a new run."), + check("queue.steer-retry-bounded", Boolean(finalAttempt && finalAttempt.status >= 200 && finalAttempt.status < 300) && retriesValid, + `Steering succeeded after ${input.steerAttempts.length} attempt(s); only steering_temporarily_unavailable was retried.`), + ]; +} + +export function gradeQueuedAcrossRestart(input: { + runs: readonly StoryRun[]; + commentId: string; + /** When the controller stop was requested; the old controller cannot dispatch after it. */ + restartStartedAt: number; + /** Whether a delivery already existed when the restart was requested. */ + dispatchedBeforeRestart: boolean; + waitObservations: ReadonlyArray<{ at: string; phase?: string; label?: string; cause?: string | null }>; +}): StoryCheck[] { + const carriers = successfulDeliveries(input.runs, input.commentId); + const start = ms(earliest(runsCarrying(input.runs, input.commentId))?.startedAt); + const bare = input.waitObservations.filter(observation => observation.phase === "queued" + && (!observation.cause || observation.label === "Queued")); + return [ + check("restart.boundary-exercised", !input.dispatchedBeforeRestart && (!Number.isFinite(start) || start >= input.restartStartedAt), + "The queued follow-up had not dispatched when the controller restart was requested."), + check("restart.dispatched-bounded", Number.isFinite(start) && start - input.restartStartedAt <= RELIABILITY_BOUNDS.restartDispatchMs, + `The follow-up dispatches automatically within ${RELIABILITY_BOUNDS.restartDispatchMs / 60_000} minutes of the restart (observed ${Number.isFinite(start) ? Math.round((start - input.restartStartedAt) / 1000) + "s" : "never"}).`), + check("restart.exactly-once", carriers.length === 1, + `The follow-up is consumed by exactly one successful run (observed ${carriers.length}).`), + check("restart.named-wait", bare.length === 0, + bare.length ? `Bare Queued without a cause at ${bare.map(observation => observation.at).join(", ")}.` + : `Every observed wait names its cause (${input.waitObservations.length} observations).`), + ]; +} diff --git a/tests/runner-e2e/everyday.test.ts b/tests/runner-e2e/everyday.test.ts index c9880edaa23..a3efdc02f5b 100644 --- a/tests/runner-e2e/everyday.test.ts +++ b/tests/runner-e2e/everyday.test.ts @@ -268,9 +268,10 @@ describe("manual everyday workflow catalog", () => { ); }); it("does not claim unsupported remote crash/hiring coverage", () => { + // The explicit reliability group is covered by everyday-reliability.test.ts. const remote = runnerMatrix.filter( (e) => - e.suite.id === "everyday-workflows" && e.environment.id === "daytona", + e.suite.id === "everyday-workflows" && e.environment.id === "daytona" && !e.task.explicitGroup, ); expect(remote).toHaveLength(8); expect(new Set(remote.map((e) => e.task.id))).toEqual( diff --git a/tests/runner-e2e/installed-cli.test.ts b/tests/runner-e2e/installed-cli.test.ts index 17fe4787087..2d1ade82a8d 100644 --- a/tests/runner-e2e/installed-cli.test.ts +++ b/tests/runner-e2e/installed-cli.test.ts @@ -9,7 +9,9 @@ import { afterEach, beforeEach, expect, it, vi } from "vitest"; import { runnerMatrix } from "./catalog.js"; import { assertInstalledCliSelection, installedCliKeys, probeInstalledCodexStartup, verifyInstalledCli } from "./installed-cli.js"; import { buildPaperclipServerEnvironment, buildRunnerE2EProcessEnvironment } from "./harness-env.js"; -const cells = runnerMatrix.filter(e => !isRollingWebDeployment(e) && !isLegacyWorkspaceMigration(e) && e.profile.id === "runner-acpx-pi"); +const cells = runnerMatrix.filter(e => !isRollingWebDeployment(e) && !isLegacyWorkspaceMigration(e) && e.profile.id === "runner-acpx-pi" + // The explicit reliability facade cell is not an installed-CLI Pi qualification case. + && e.task.explicitGroup === undefined); const roots: string[] = []; // pnpm's source test wrapper sets NODE_PATH; the disposable public-install fixture must start closed. beforeEach(() => { vi.stubEnv("NODE_PATH", undefined); vi.stubEnv("NODE_OPTIONS", undefined); }); diff --git a/tests/runner-e2e/live-fixtures.ts b/tests/runner-e2e/live-fixtures.ts index f97c42e5dc6..d1a71cbe45f 100644 --- a/tests/runner-e2e/live-fixtures.ts +++ b/tests/runner-e2e/live-fixtures.ts @@ -1,4 +1,5 @@ import { isEvalRemoteOrigin } from "./eval-remote-gateway.js"; +import { isEverydayReliabilityCase } from "./everyday-reliability-cases.js"; import type { AiConnectionBinding } from "../../packages/shared/src/ai-connections.js"; import { verifyInstalledDaytonaPlugin } from "./installed-daytona-plugin.js"; import { NATIVE_COMPLETION_BUDGET_CENTS } from "./native-completion-defaults.js"; @@ -221,7 +222,8 @@ export async function setupLiveFixtures(input: { budgetMonthlyCents: execution.suite.id === CHAT_FEATURE_SUITE ? CHAT_FEATURE_BUDGET_CENTS : receiptSmoke ? CASE_RECEIPT_BUDGET_CENTS : ["native-completion", "native-instruction-consolidation", "native-connection-guidance"].includes(execution.suite.id) - || (execution.suite.id === "everyday-workflows" && ["hire-reuse", "delegate-feedback"].includes(execution.task.id)) ? NATIVE_COMPLETION_BUDGET_CENTS + || (execution.suite.id === "everyday-workflows" && (["hire-reuse", "delegate-feedback"].includes(execution.task.id) + || isEverydayReliabilityCase(execution.task.id))) ? NATIVE_COMPLETION_BUDGET_CENTS : execution.suite.id === "task-titles" ? TASK_TITLE_BUDGET_CENTS : ["cursor-cloud", "stock-harness", "task-workspaces", "case-handoffs", "feedback-persistence", "daytona-restart-continuity", "remote-workspace-storage", "rolling-web-deployment"].includes(execution.suite.id) ? 1_000 : 0, }); @@ -419,7 +421,8 @@ export async function setupLiveFixtures(input: { } } if (["stock-harness", "native-connection-guidance", "feedback-persistence", "daytona-restart-continuity", "remote-workspace-storage", "rolling-web-deployment"].includes(execution.suite.id) - || (execution.suite.id === "everyday-workflows" && ["hire-reuse", "delegate-feedback"].includes(execution.task.id))) { + || (execution.suite.id === "everyday-workflows" && (["hire-reuse", "delegate-feedback"].includes(execution.task.id) + || isEverydayReliabilityCase(execution.task.id)))) { agent.budgetMonthlyCents = 1_000; } if (receiptSmoke) { diff --git a/tests/runner-e2e/selectors.ts b/tests/runner-e2e/selectors.ts index 204be0d847a..6e8312ed1d4 100644 --- a/tests/runner-e2e/selectors.ts +++ b/tests/runner-e2e/selectors.ts @@ -195,6 +195,12 @@ export function selectRunnerExecutions( const selected = matrix.filter((execution) => { if (options.ids.length > 0) return options.ids.includes(execution.id); if (execution.suite.manualOnly && !options.suites.includes(execution.suite.id) && !options.list) return false; + const unfilteredList = options.list && options.groups.length + options.suites.length + options.profiles.length + + options.environments.length + options.cases.length === 0; + const explicitGroup = execution.task.explicitGroup; + // Naming any of the task's own groups (for example `slow`) also opts in. + if (explicitGroup && !unfilteredList && !options.groups.some(group => group === explicitGroup || execution.task.groups.includes(group)) + && !options.cases.includes(execution.task.id)) return false; if ( options.all || (options.list && diff --git a/tests/runner-e2e/types.ts b/tests/runner-e2e/types.ts index 5ab04bac2c4..0b6410397c9 100644 --- a/tests/runner-e2e/types.ts +++ b/tests/runner-e2e/types.ts @@ -160,6 +160,12 @@ export interface RunnerTaskFixture { id: string; label: string; groups: readonly string[]; + /** + * Opt-in group. When set, the task is selected only by naming this group + * (`--group`), the case (`--case`), or an exact execution ID. Suite, profile + * and environment selectors alone never select it. + */ + explicitGroup?: string; workMode: RunnerTaskWorkMode; flow: RunnerTaskFlow; expectedRunCount: number;