mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 07:10:34 +01:00
test(runner-e2e): add staging-soak reliability cases to everyday workflows (#19455)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The paid runner E2E suite (`tests/runner-e2e`) drives the real browser, server, native runner, provider and Daytona > - A manual staging soak found about nine reliability bugs (S47–S55, H2, U9). Each fix got unit or integration tests > - The end-to-end journeys that exposed the bugs existed only as ad-hoc browser scripts, so a regression could return without a signal > - These journeys are expensive, so they must not run per PR or in `--all`. They must run on demand before a release > - This pull request adds the journeys to the existing `everyday-workflows` suite as an explicit `reliability` group > - The benefit is one command that runs the whole reliability set before a release or after a risky runner change ## Linked Issues or Issue Description **What is the problem?** The staging-soak scenarios that found S47–S55, H2 and U9 are not repeatable. They were ad-hoc browser scripts on staging. **Why does it matter?** Without a repeatable end-to-end case, a reliability regression (stalled follow-ups, duplicate execution, dead Retry, hangs after silent commands) can return without a signal. **What is the proposed solution?** Add the scenarios to the paid `everyday-workflows` suite as explicit-only cases with independent graders and a selectable `reliability` group. Refs #19370, #19355, #19354, #19407, #19419, #19394, #19421, #19431 (the related fixes). ## What Changed - `everyday-reliability-cases.ts`: seven new cases (`interrupt-redirect`, `reassign-midrun`, `three-turn-revise`, `long-silent-command`, `tree-pause-release`, `queue-steer`, `queued-across-restart`), their soak signatures, prompts, bounds and the explicit matrix. - `everyday-reliability-flow.ts`: browser and public-API drivers for each journey. A shared watcher records `executionBlocker`, `interruption` and execution projections while it waits. It uses an offered Retry at most once (never while an automatic retry is scheduled). - `everyday-reliability.ts`: pure graders. They encode the product rule: interrupted work resumes automatically or offers a Retry that works on first use. A send-a-message blocker, a dead or repeated Retry, a silent stall, or duplicate execution fails. - `types.ts` / `selectors.ts`: new optional `explicitGroup` on a task. Such a task is selected only by its group, `--case`, or an exact `--id`. `--all`, the plain suite and generic selectors skip it. `--list` still shows it. - `catalog.ts`: the suite gets the reliability tasks, the released Grok/Kimi/Pi facades (only for `long-silent-command`), a per-turn Daytona override for `three-turn-revise`, the `reliability` and `slow` groups, definition version 11, and a computed `expectedMatrixSize` (52 → 62). - `everyday-flow.ts`: a narrow hook calls the scenario driver after task creation. Riley is created for the reassign and tree cases. The reliability checks are added after settlement. - `everyday-observations.ts`: an allowed interrupted run may end `operator_interrupted` or `issue_reassigned`. Lifecycle checks accept a declared reassigned parent owner. - `live-fixtures.ts`: reliability cells get the existing 1,000-cent company and lead-agent hard stop. - Tests: new `everyday-reliability.test.ts` (selection exclusion, matrix, credentials, and calibrated pass/fail cases for every grader). Updated counts in `catalog.test.ts`, `everyday.test.ts` and `installed-cli.test.ts`. - Docs: new "Staging-soak reliability group" section in `EVERYDAY-WORKFLOWS.md` (cases, signatures, run commands, cadence, coverage boundary) and a link from `doc/evals.md`. `everyday-flow.ts` was already over 1,000 lines. This PR adds only about 20 lines of wiring there. The scenario logic is in the new modules. Matrix (10 cells, all Daytona, single attempt): one `runner-codex` (gpt-5.6-sol) cell per case, plus `runner-acpx-grok`, `runner-acpx-kimi` and `runner-acpx-pi` for `long-silent-command`. Run command: ```sh pnpm test:e2e:runner -- --suite everyday-workflows --group reliability --max-parallel 2 ``` ## Verification - `pnpm test:e2e:runner:typecheck` passes. - `node node_modules/vitest/vitest.mjs run --config tests/runner-e2e/vitest.config.ts everyday catalog installed-cli selectors support.test`: 15 files, 362 tests pass. - The full runner-e2e Vitest run has the same 14 failures with and without this change (`controller-outage`, `operating-receipts`, `pi-controls-evidence`, `pi-file-evidence`, `workflow-security`). These fail on the base commit in this local environment too. - `pnpm test:e2e:runner -- --list --suite everyday-workflows --group reliability` lists exactly the ten cells. - `--all`, `--suite everyday-workflows` (still 52 cells), profile/environment selectors and the daily/weekly scheduled selections contain no reliability cell. A test asserts this. - Not run: the paid cells. They need provider and Daytona credentials. The first paid run of the group is the acceptance step. ## Risks - The drivers have not run against a live stack. A first paid run can expose harness timing or selector issues (for example, how fast the queued Interrupt control appears). The failure then names the stalled step and keeps `reliability-observations.json`. - `long-silent-command` does not inject a runner connection loss. The named-cause contract is checked only if a loss happens naturally. - The Grok/Kimi/Pi facades run as ordinary released profiles. If one returns to pending qualification, it drops out of the group and the matrix size follows. - No product code changes. Existing everyday cells keep their selection and checks. The suite definition hash changes because the suite definition changed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, code execution). Standard context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed
1 parent
b8e69c6dd0
commit
d87f6fa4c2
15 files changed
+1322
-25
No files matched your search
+4
-1
@@ -378,7 +378,10 @@ Use the selectors documented in the [runner E2E README](../tests/runner-e2e/READ
|
||||
for a suite, profile, case, group, or environment. Daytona needs the immutable
|
||||
image digest and `DAYTONA_API_KEY`; follow the README and fixture security guide.
|
||||
`--all` excludes manual suites such as `everyday-workflows`. Select that suite
|
||||
explicitly; use a narrow selector while developing a fixture.
|
||||
explicitly; use a narrow selector while developing a fixture. Its staging-soak
|
||||
reliability cases need `--group reliability` as well; see
|
||||
[Everyday workflows](../tests/runner-e2e/EVERYDAY-WORKFLOWS.md#staging-soak-reliability-group)
|
||||
for the command and the before-release cadence.
|
||||
|
||||
For Runner Evals, the narrowest useful local validation is the report program's
|
||||
help/validation path and the deterministic Runner checks documented in
|
||||
|
||||
@@ -73,6 +73,74 @@ The first two did not establish a safe restart boundary, and the uncertainty
|
||||
case measures a deterministic safety rule. None supports ranking models.
|
||||
See [controlled recovery tests](../runner-recovery/README.md).
|
||||
|
||||
## Staging-soak reliability group
|
||||
|
||||
Seven explicit-only cases turn the staging soak's ad-hoc browser scripts into
|
||||
repeatable runs. Each case comes from a real staging failure. They live in the
|
||||
`everyday-workflows` suite, but neither `--all`, `--suite everyday-workflows`, a
|
||||
profile/environment selector, nor a scheduled campaign selects them. Only the
|
||||
`reliability` group, an explicit `--case`, or an exact `--id` does
|
||||
(`explicitGroup` on the task fixture enforces this in `selectors.ts`).
|
||||
|
||||
```sh
|
||||
pnpm test:e2e:runner -- --list --suite everyday-workflows --group reliability
|
||||
pnpm test:e2e:runner -- --suite everyday-workflows --group reliability --max-parallel 2
|
||||
pnpm test:e2e:runner -- --suite everyday-workflows --case queue-steer
|
||||
pnpm test:e2e:runner -- --id everyday-workflows.runner-codex.daytona.interrupt-redirect
|
||||
```
|
||||
|
||||
**Cadence:** run the whole group before a release and after a risky runner,
|
||||
recovery, queue, tree-control or restart change. Do not run it per PR. It is
|
||||
paid: ten Daytona cells, single attempt each, with a 1,000-cent company and
|
||||
lead-agent hard stop. Missing provider or Daytona credentials, or a mutable
|
||||
Daytona image, stop the launch before any provider spend.
|
||||
|
||||
**Product rule (every case):** work interrupted by a restart, Stop,
|
||||
reassignment or lost runner must resume automatically, or offer a Retry that
|
||||
works on first use. The harness watches each task's `executionBlocker`,
|
||||
`interruption` and execution projection while it waits. It uses an offered
|
||||
Retry at most once (never while an automatic retry is scheduled) and requires
|
||||
a new run from it. These fail: a "send a message to continue" blocker
|
||||
(`resume.no-send-message-blocker`), a dead or repeated Retry
|
||||
(`resume.retry-works-first-use`), a silent stall past the case bound
|
||||
(`resume.no-silent-stall`), and duplicate execution. Evidence is saved to
|
||||
`reliability-observations.json`.
|
||||
|
||||
Message delivery is attributed through each run's wake context
|
||||
(`wakeCommentIds`), not its saved prompt: the prompt carries the whole
|
||||
conversation history, so later turns legitimately repeat earlier messages. A
|
||||
message delivered by more than one successful run is duplicate execution.
|
||||
`--group slow` selects only `long-silent-command`.
|
||||
|
||||
| Case | Soak signature | Journey | Key checks |
|
||||
|---|---|---|---|
|
||||
| `interrupt-redirect` | S47 (fixes #19370, #19355) | While a slow step loop runs, send a follow-up and click its queued **Interrupt**. Then reopen with a second slow loop, click **Stop**, and immediately send another message while the stopped run is still saving (the `workspace_busy` retry path). | Interrupted run ends `operator_interrupted`; exactly one continuation carries the message and starts within 180 s (the soak saw 13–90 s once fixed, 4 minutes or never before); no second user message; the Stop-settlement message is consumed once within 180 s; Done. |
|
||||
| `reassign-midrun` | S48 (fix #19354) | While the lead runs a resumable step loop, reassign the task to Riley through the public issue API. | Request is 2xx within 10 s (no 409 or 20 s hang); old run ends `issue_reassigned`; Riley's run starts after the request and succeeds; no overlapping executions on the task; `steps.txt` holds step-1..step-8 exactly once, so work continued from saved files. |
|
||||
| `three-turn-revise` | S49 | Create `items.md`, then two reopen-comment revisions on the same workspace: add two items, then convert to a table. Runs on the **per-turn** Daytona lease (the bug hit gpt-5.6-sol with an ephemeral lease on turn 3). | Each turn succeeds; no reconciliation or send-a-message blocker; the file is a five-row Number/Item table. |
|
||||
| `long-silent-command` | S52/S52b (fixes #19407, #19419) | The agent runs one 13-minute foreground loop (`sleep 60` × 13, appending to `silent.txt`) that prints nothing, then reports. Tagged `slow`; 50-minute Daytona deadline. | Run succeeds after the command (not shortened) and is terminal within 8 minutes after it; `silent.txt` holds tick-1..tick-13; Stop stays available in API and browser samples throughout; any failure carries a named cause. |
|
||||
| `tree-pause-release` | S50/S50b (fixes #19394, #19421) | The lead delegates two slow children to Riley. While a child is queued or has a parked `workspace_busy` retry, pause the parent's tree with `POST /api/issues/:id/tree-holds`. Hold past the retry's due time (at least 75 s), then release **without** `wakeAgents`. | The hold interrupted work; no child run starts during the hold; each interrupted task gets exactly one `issue_tree_resumed` run; all tasks reach Done. |
|
||||
| `queue-steer` | H2 | During an active run, queue three follow-ups through the public comment API; edit one, discard one, and steer one (retrying only `steering_temporarily_unavailable`, for up to two minutes). | Edited text reaches the agent and the original never does; the discarded text never reaches the agent; the steered message is acknowledged by the active turn and never replayed; the remaining message is handled by exactly one run. |
|
||||
| `queued-across-restart` | S55/U9 (fix #19431) | Queue a follow-up through the composer while a run is active, then restart the controller with the existing restart machinery. | The follow-up dispatches only after the restart, automatically within six minutes, exactly once; every observed `queued` wait names a cause (`server_restarting`, `task_drain`, …) rather than a bare "Queued"; any offered Retry works on first use. |
|
||||
|
||||
**Matrix (10 cells).** Each case runs once on native Codex (`runner-codex`,
|
||||
`gpt-5.6-sol`) on Daytona: the warm environment, except `three-turn-revise` on
|
||||
the per-turn lease. `long-silent-command` also runs on the released ACPX
|
||||
facades Grok (`runner-acpx-grok`), Kimi (`runner-acpx-kimi`) and OpenRouter/Pi
|
||||
(`runner-acpx-pi`), which hung 3/3 before the fix. They run as ordinary released
|
||||
profiles, as in the rolling-deployment matrix; a profile whose ACPX
|
||||
qualification returns to pending drops out of the group automatically. Their
|
||||
credentials (`XAI_API_KEY`, `KIMI_API_KEY`, `OPENROUTER_API_KEY`) are only
|
||||
required when those cells are selected.
|
||||
|
||||
**Coverage boundary.** The graders and selection are calibrated by
|
||||
`everyday-reliability.test.ts` against passing and plausible failing
|
||||
observations. No paid cell has run yet; a first run of the group is the
|
||||
acceptance step. The original soak browser scripts are not in this repository,
|
||||
so these cases encode the reported signatures, not a replay of those scripts.
|
||||
`long-silent-command` checks the completion and Stop paths; it does not inject a
|
||||
runner connection loss, so the "fails within a bounded time with a named
|
||||
cause" contract is checked only if a loss happens naturally.
|
||||
|
||||
## Matrix and running
|
||||
|
||||
The local matrix has fourteen cases on native Codex `gpt-5.6-sol`, native ACPX Claude
|
||||
|
||||
@@ -329,11 +329,11 @@ describe("runner E2E catalog", () => {
|
||||
expect(localIntegrityTasks).toHaveLength(2);
|
||||
expect(openRouterBreadthTasks).toHaveLength(3);
|
||||
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
|
||||
1, 9, 2, 9, 2, 4, 1, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 20, 2, 1, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 2, 52, 5, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2,
|
||||
1, 9, 2, 9, 2, 4, 1, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 20, 2, 1, 6, 8, 46, 2, 23, 15, 62, 6, 6, 20, 26, 2, 52, 5, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2,
|
||||
|
||||
]);
|
||||
expect(validateRunnerCatalog()).toHaveLength(824);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(824);
|
||||
expect(validateRunnerCatalog()).toHaveLength(834);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(834);
|
||||
expect(
|
||||
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
|
||||
).toHaveLength(48);
|
||||
|
||||
@@ -41,6 +41,7 @@ import { lifecycleLiveTasks, lifecycleLiveDefinitionDigest } from "./lifecycle-l
|
||||
import { publicMcpTasks, publicMcpExpandedDigest, publicMcpSetupDigest, publicMcpWorkflowDigest, publicMcpWorkerInstructions, publicMcpWorkerSkillDigest } from "./public-mcp-cases.js";
|
||||
import { graderVersion as publicMcpGraderVersion } from "./public-mcp-grading.js";
|
||||
import { everydayTasks, productionStoryProfile } from "./everyday-cases.js";
|
||||
import { EVERYDAY_RELIABILITY_GROUP, EVERYDAY_RELIABILITY_SLOW_GROUP, RELIABILITY_FACADE_PROFILE_IDS, everydayReliabilityExecutionIds, everydayReliabilitySignatures, everydayReliabilityTasks } from "./everyday-reliability-cases.js";
|
||||
import { REMOTE_WORKSPACE_STORAGE_SUITE, remoteWorkspaceStoragePrompt } from "./remote-workspace-storage.js";
|
||||
import { CONNECTION_GUIDANCE_SUITE, CONNECTION_GUIDANCE_BUDGET_CENTS, connectionGuidanceTasks, connectionGuidanceDefinitionDigest } from "./connection-guidance-cases.js";
|
||||
|
||||
@@ -90,6 +91,8 @@ const SELECTABLE_GROUPS = [
|
||||
"onboarding",
|
||||
"context-integrity",
|
||||
"stock-harness",
|
||||
EVERYDAY_RELIABILITY_GROUP,
|
||||
EVERYDAY_RELIABILITY_SLOW_GROUP,
|
||||
] as const;
|
||||
const SAMPLE_UUID = "11111111-1111-4111-8111-111111111111";
|
||||
|
||||
@@ -1253,6 +1256,33 @@ export const kimiRunnerProfile = nativeProfile({
|
||||
modelQualification: { source: "qualified_runner_profile", qualificationId: "kimi:2.3.3:native-qualified-profile17" },
|
||||
});
|
||||
|
||||
/**
|
||||
* ACPX facades admitted only for the reliability long-silent-command case. Like the
|
||||
* rolling-deployment matrix, they run as ordinary released profiles: a pending
|
||||
* candidate needs its own qualification suite and is rejected here.
|
||||
*/
|
||||
const everydayFacadeProfiles = RELIABILITY_FACADE_PROFILE_IDS
|
||||
.map(id => [...runnerProfiles, ...extendedHarnessProfiles, kimiRunnerProfile].find(profile => profile.id === id)!)
|
||||
.filter(original => !original.qualificationCandidate || QUALIFIED_ACPX_PROFILES[original.qualificationCandidate].qualificationStatus !== "pending")
|
||||
.map(original => {
|
||||
const { qualificationCandidate: _candidate, ...profile } = productionStoryProfile(original);
|
||||
return profile;
|
||||
});
|
||||
/** Released facades currently in the reliability group; a pending one drops out. */
|
||||
export const everydayReliabilityFacadeProfileIds = everydayFacadeProfiles.map(profile => profile.id);
|
||||
const everydayWorkflowProfiles = [...everydayProfiles,
|
||||
productionStoryProfile(runnerProfiles.find(profile => profile.id === "runner-opencode")!),
|
||||
...everydayFacadeProfiles];
|
||||
|
||||
/** Every reliability cell outside the deliberately small explicit matrix. */
|
||||
function everydayReliabilityExcludedIds(suiteId: string): string[] {
|
||||
const allowed = new Set(everydayReliabilityExecutionIds(suiteId, everydayReliabilityFacadeProfileIds));
|
||||
return everydayWorkflowProfiles.flatMap(profile => ["local", "daytona"].flatMap(environmentId =>
|
||||
everydayReliabilityTasks.map(task => `${suiteId}.${profile.id}.${environmentId}.${task.id}`)))
|
||||
.filter(id => !allowed.has(id));
|
||||
}
|
||||
|
||||
|
||||
const rollingDeploymentProfiles = [...runnerProfiles, ...extendedHarnessProfiles, kimiRunnerProfile]
|
||||
.filter(profile => profile.generation === "native" && profile.supportedEnvironments.includes("daytona")
|
||||
&& (!profile.qualificationCandidate || QUALIFIED_ACPX_PROFILES[profile.qualificationCandidate].qualificationStatus !== "pending"))
|
||||
@@ -1704,11 +1734,13 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
|
||||
{
|
||||
id: "everyday-workflows", label: "Everyday Paperclip Work", manualOnly: true,
|
||||
description: "Real user requests, useful downloaded work, and durable continuation using production instructions.",
|
||||
groups: ["native"], profiles: [...everydayProfiles,
|
||||
productionStoryProfile(runnerProfiles.find(profile => profile.id === "runner-opencode")!),
|
||||
], environments: [localEnvironment, daytonaWarmEnvironment],
|
||||
tasks: everydayTasks.map(task => ["hire-reuse", "delegate-feedback", "recover-controller"].includes(task.id)
|
||||
? { ...task, automaticRetryPolicy: "single_attempt" as const } : task), expectedMatrixSize: 52,
|
||||
groups: ["native"], profiles: everydayWorkflowProfiles, environments: [localEnvironment, daytonaWarmEnvironment],
|
||||
// The S49 turn-3 failure needed an ephemeral per-turn lease, not the warm one.
|
||||
environmentOverrides: { "three-turn-revise": { daytona: runnerEnvironments.find(environment => environment.id === "daytona")! } },
|
||||
tasks: [...everydayTasks.map(task => ["hire-reuse", "delegate-feedback", "recover-controller"].includes(task.id)
|
||||
? { ...task, automaticRetryPolicy: "single_attempt" as const } : task), ...everydayReliabilityTasks],
|
||||
// 52 everyday cells, one native Daytona cell per reliability case, and released facades for the silent command.
|
||||
expectedMatrixSize: 52 + everydayReliabilityTasks.length + everydayFacadeProfiles.length,
|
||||
excludedExecutionIds: [...everydayProfiles.flatMap(profile => everydayTasks
|
||||
.filter(task => !["build-revise", "delegate-feedback", "recover-controller", "create-skill-studio"].includes(task.id))
|
||||
.map(task => `everyday-workflows.${profile.id}.daytona.${task.id}`)),
|
||||
@@ -1716,8 +1748,14 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
|
||||
`everyday-workflows.runner-opencode.daytona.${task.id}`,
|
||||
...(["hire-reuse", "delegate-feedback"].includes(task.id) ? [] : [`everyday-workflows.runner-opencode.local.${task.id}`]),
|
||||
]),
|
||||
// ACPX facades join only the reliability long-silent-command cells.
|
||||
...everydayFacadeProfiles.flatMap(profile => everydayTasks.flatMap(task =>
|
||||
["local", "daytona"].map(environmentId => `everyday-workflows.${profile.id}.${environmentId}.${task.id}`))),
|
||||
...everydayReliabilityExcludedIds("everyday-workflows"),
|
||||
],
|
||||
definitionMetadata: { version: 10, instructions: "production", grading: "outcome-and-invariants", scheduling: "explicit-only",
|
||||
definitionMetadata: { version: 11, instructions: "production", grading: "outcome-and-invariants", scheduling: "explicit-only",
|
||||
reliability: { group: EVERYDAY_RELIABILITY_GROUP, selection: "explicit-group-only", signatures: everydayReliabilitySignatures,
|
||||
productRule: "interrupted-work-resumes-automatically-or-offers-a-working-first-retry", companyAndLeadBudgetCents: 1_000 },
|
||||
controllerDowntimeMs: 125_000,
|
||||
remoteCodexRestart: "bounded-main-server-outage-receipt-empty-cache-same-active-process-and-finalization",
|
||||
procedureCases: { ids: ["hire-reuse", "delegate-feedback"], maximumAttemptsPerCell: 1, companyAndLeadBudgetCents: 1_000 } },
|
||||
@@ -2186,6 +2224,7 @@ export function validateRunnerCatalog(): MatrixExecution[] {
|
||||
...continuationTasks,
|
||||
questionResumeTask,
|
||||
...everydayTasks,
|
||||
...everydayReliabilityTasks,
|
||||
...taskTitleTasks,
|
||||
...runnerTasks,
|
||||
...localIntegrityTasks,
|
||||
|
||||
@@ -40,6 +40,8 @@ import {
|
||||
type StoryInteraction,
|
||||
} from "./everyday-decisions.js";
|
||||
import { LATE_REQUIREMENT, SLUGIFY_REVISION, SLUGIFY_DEPLOYMENT_SOAK, requiresEverydayArtifactOracle } from "./everyday-cases.js";
|
||||
import { everydayReliabilitySignatures, isEverydayReliabilityCase } from "./everyday-reliability-cases.js";
|
||||
import { driveReliabilityScenario, type ReliabilityOutcome } from "./everyday-reliability-flow.js";
|
||||
import {
|
||||
isActiveStoryRun,
|
||||
isStoryWorkspaceDeferral,
|
||||
@@ -91,8 +93,11 @@ export interface EverydayEvidence {
|
||||
agents: Row[];
|
||||
downloads: Row[];
|
||||
allowedInterruptedRuns: string[];
|
||||
/** Reliability scenarios: failures a later run on the same task verifiably resumed. */
|
||||
recoveredRuns?: string[];
|
||||
remoteRestart?: RemoteRestartEvidence;
|
||||
remoteWorkspaceStorage?: RemoteWorkspaceStorageEvidence;
|
||||
reliability?: { signature: string; observed: string; fixes: readonly string[] };
|
||||
}
|
||||
interface Input {
|
||||
page: Page;
|
||||
@@ -206,6 +211,8 @@ export async function runEverydayFlow(input: Input) {
|
||||
let settledAgentReply: Row | undefined;
|
||||
let postControllerApiFollowup = false;
|
||||
let retainedFollowupCommentId: string | undefined;
|
||||
let reliability: ReliabilityOutcome | undefined;
|
||||
let helperAgentId: string | undefined;
|
||||
let reviewHandoffBoundary: {
|
||||
childId: string;
|
||||
assigneeAgentId: string | null | undefined;
|
||||
@@ -333,6 +340,7 @@ export async function runEverydayFlow(input: Input) {
|
||||
issueId: target.id,
|
||||
commentIds: added.map((c) => c.id),
|
||||
});
|
||||
return added.map((c) => String(c.id));
|
||||
}
|
||||
function declineInput(state: Pick<EverydayEvidence, "issues" | "runs">): ConnectionGuidanceDeclineInput | undefined {
|
||||
if (execution.suite.id !== CONNECTION_GUIDANCE_SUITE ||
|
||||
@@ -378,7 +386,7 @@ export async function runEverydayFlow(input: Input) {
|
||||
)),
|
||||
reject: (state) => {
|
||||
if (state.runs.length > 12) return "bounded execution count exceeded";
|
||||
const bad = storyUnexpectedRunFailure(state.runs, ev.allowedInterruptedRuns);
|
||||
const bad = storyUnexpectedRunFailure(state.runs, ev.allowedInterruptedRuns, ev.recoveredRuns);
|
||||
if (bad)
|
||||
return `native execution failed ${bad.errorCode ?? ""}: ${bad.error ?? bad.status}`;
|
||||
if (
|
||||
@@ -618,6 +626,9 @@ export async function runEverydayFlow(input: Input) {
|
||||
"everyday-delivery.ts",
|
||||
"everyday-observations.ts",
|
||||
"everyday-remote-restart.ts",
|
||||
"everyday-reliability-cases.ts",
|
||||
"everyday-reliability.ts",
|
||||
"everyday-reliability-flow.ts",
|
||||
"retained-controller-continuation.ts",
|
||||
"controller-outage.ts",
|
||||
"controller-rolling-deployment.ts",
|
||||
@@ -716,7 +727,8 @@ export async function runEverydayFlow(input: Input) {
|
||||
canCreateAgents: true,
|
||||
canAssignTasks: true,
|
||||
});
|
||||
if (caseId === "delegate-feedback" || caseId === "agent-review-handoff") {
|
||||
if (isEverydayReliabilityCase(caseId)) ev.reliability = everydayReliabilitySignatures[caseId];
|
||||
if (["delegate-feedback", "agent-review-handoff", "reassign-midrun", "tree-pause-release"].includes(caseId)) {
|
||||
const config = execution.profile.buildAgent({
|
||||
environmentId: fixtures.environment.id,
|
||||
environmentFixtureId: execution.environment.id,
|
||||
@@ -724,7 +736,7 @@ export async function runEverydayFlow(input: Input) {
|
||||
secretRefs: fixtures.secretRefs,
|
||||
executionId: nonce,
|
||||
});
|
||||
await api.post(`/api/companies/${fixtures.company.id}/agents`, {
|
||||
helperAgentId = (await api.post<Row>(`/api/companies/${fixtures.company.id}/agents`, {
|
||||
...config,
|
||||
name: "Riley Builder",
|
||||
role: "engineer",
|
||||
@@ -737,7 +749,7 @@ export async function runEverydayFlow(input: Input) {
|
||||
"You implement small software projects and verify your work. Follow task feedback and deliver usable files.",
|
||||
},
|
||||
},
|
||||
});
|
||||
})).id;
|
||||
}
|
||||
if (caseId.startsWith("service-"))
|
||||
review = await setupConnectionReview({
|
||||
@@ -785,6 +797,14 @@ export async function runEverydayFlow(input: Input) {
|
||||
input.observe(parent!, []);
|
||||
note("task-submitted", { issueId: parent!.id });
|
||||
await openParent();
|
||||
if (isEverydayReliabilityCase(caseId)) {
|
||||
reliability = await driveReliabilityScenario(caseId, { page, api, execution, nonce, leadAgentId: fixtures.agent.id,
|
||||
helperAgentId, workspacePath: input.workspacePath, deadlineAt: input.deadlineAt, parentId: parent!.id,
|
||||
refresh, reply: message => reply(message), openParent,
|
||||
restart: () => input.restart(), note, allowInterrupted: runId => { ev.allowedInterruptedRuns.push(runId); },
|
||||
allowRecovered: runIds => { ev.recoveredRuns = [...runIds]; },
|
||||
evidence: input.evidence, capture: input.capture });
|
||||
}
|
||||
if (caseId === "agent-review-handoff") {
|
||||
const boundary = await pollUntil({
|
||||
label: "blocked parent with agent review wake",
|
||||
@@ -1210,8 +1230,9 @@ export async function runEverydayFlow(input: Input) {
|
||||
}
|
||||
await settled(
|
||||
caseId === "stop-redirect" ? `Reference ${nonce}`
|
||||
: caseId === "provider-second" ? `CONTACTS_${nonce}` : undefined,
|
||||
: caseId === "provider-second" ? `CONTACTS_${nonce}` : reliability?.expectedReply,
|
||||
);
|
||||
if (reliability) ev.checks.push(...await reliability.checks());
|
||||
if (caseId === "create-skill-studio") {
|
||||
const createdSkills = await api.get<Row[]>(
|
||||
`/api/companies/${fixtures.company.id}/skills`,
|
||||
@@ -1620,7 +1641,8 @@ export async function runEverydayFlow(input: Input) {
|
||||
.filter(
|
||||
(r) =>
|
||||
!isStoryWorkspaceDeferral(r) &&
|
||||
!isExpectedStoryInterruption(r, ev.allowedInterruptedRuns),
|
||||
!isExpectedStoryInterruption(r, ev.allowedInterruptedRuns) &&
|
||||
!(ev.recoveredRuns ?? []).includes(r.id),
|
||||
)
|
||||
.every(
|
||||
(r) =>
|
||||
@@ -1667,6 +1689,8 @@ export async function runEverydayFlow(input: Input) {
|
||||
parentId: parent!.id,
|
||||
leadId: fixtures.agent.id,
|
||||
allowedInterruptedRuns: ev.allowedInterruptedRuns,
|
||||
recoveredRuns: ev.recoveredRuns,
|
||||
parentAgentIds: reliability?.parentAgentIds,
|
||||
requireRemoteRestart,
|
||||
remoteRestart: ev.remoteRestart,
|
||||
}),
|
||||
|
||||
@@ -220,16 +220,18 @@ export function isExpectedStoryInterruption(
|
||||
): boolean {
|
||||
if (!allowedRunIds.includes(run.id)) return false;
|
||||
return (
|
||||
(run.status === "cancelled" && (!run.errorCode || run.errorCode === "cancelled")) ||
|
||||
// Operator interrupts and reassignment are deliberate reliability-scenario stops.
|
||||
(run.status === "cancelled" && (!run.errorCode || ["cancelled", "operator_interrupted", "issue_reassigned"].includes(run.errorCode))) ||
|
||||
(run.status === "interrupted" && run.errorCode === "server_shutdown_interrupted") ||
|
||||
(run.status === "failed" && run.errorCode === "process_lost")
|
||||
);
|
||||
}
|
||||
|
||||
export function storyUnexpectedRunFailure(runs: StoryRun[], allowedRunIds: readonly string[]) {
|
||||
/** `recoveredRunIds`: failures whose work a later run on the same task verifiably resumed. */
|
||||
export function storyUnexpectedRunFailure(runs: StoryRun[], allowedRunIds: readonly string[], recoveredRunIds: readonly string[] = []) {
|
||||
return runs.find((run) =>
|
||||
["failed", "timed_out", "cancelled", "interrupted"].includes(run.status) &&
|
||||
!isStoryWorkspaceDeferral(run) &&
|
||||
!isStoryWorkspaceDeferral(run) && !recoveredRunIds.includes(run.id) &&
|
||||
!isExpectedStoryInterruption(run, allowedRunIds),
|
||||
);
|
||||
}
|
||||
@@ -274,6 +276,10 @@ export function storyLifecycleChecks(input: {
|
||||
parentId: string;
|
||||
leadId: string;
|
||||
allowedInterruptedRuns?: string[];
|
||||
/** Failed runs whose work a later run on the same task verifiably resumed (reliability scenarios). */
|
||||
recoveredRuns?: readonly string[];
|
||||
/** Agents that may execute on the parent; defaults to the lead (reassignment adds the new owner). */
|
||||
parentAgentIds?: readonly string[];
|
||||
requireRemoteRestart?: boolean;
|
||||
remoteRestart?: RemoteRestartEvidence;
|
||||
}): StoryCheck[] {
|
||||
@@ -307,7 +313,8 @@ export function storyLifecycleChecks(input: {
|
||||
),
|
||||
check(
|
||||
"successful-runs",
|
||||
executed.every((r) => r.status === "succeeded" || isExpectedStoryInterruption(r, allowed)),
|
||||
executed.every((r) => r.status === "succeeded" || isExpectedStoryInterruption(r, allowed)
|
||||
|| (input.recoveredRuns ?? []).includes(r.id)),
|
||||
"Only the expected stop or process-loss outcome of an injected interruption is exempt.",
|
||||
),
|
||||
check(
|
||||
@@ -321,7 +328,7 @@ export function storyLifecycleChecks(input: {
|
||||
(r) =>
|
||||
(r.contextSnapshot?.issueId === input.parentId ||
|
||||
r.contextSnapshot?.taskId === input.parentId) &&
|
||||
r.agentId !== input.leadId,
|
||||
!(input.parentAgentIds ?? [input.leadId]).includes(r.agentId),
|
||||
),
|
||||
"A mentioned worker must not execute on the parent.",
|
||||
),
|
||||
|
||||
@@ -0,0 +1,125 @@
|
||||
import type { RunnerTaskFixture } from "./types.js";
|
||||
|
||||
/**
|
||||
* Staging-soak reliability scenarios. Every case reproduces a real staging
|
||||
* failure end to end. They are explicit-only: `--all`, `--suite
|
||||
* everyday-workflows` and generic profile/environment selectors never pick
|
||||
* them. Select them with `--group reliability`, a `--case`, or an exact `--id`.
|
||||
*/
|
||||
export const EVERYDAY_RELIABILITY_GROUP = "reliability";
|
||||
export const EVERYDAY_RELIABILITY_SLOW_GROUP = "slow";
|
||||
|
||||
/** Seconds per silent iteration and iteration count for `long-silent-command`. */
|
||||
export const LONG_SILENT_ITERATION_SECONDS = 60;
|
||||
export const LONG_SILENT_ITERATIONS = 13;
|
||||
|
||||
/** Soak signature, the observed failure before its fix, and related fix PRs. */
|
||||
export const everydayReliabilitySignatures = {
|
||||
"interrupt-redirect": { signature: "S47", observed: "Interrupted follow-up started after 4 minutes or never; needed a second message.", fixes: ["#19370", "#19355"] },
|
||||
"reassign-midrun": { signature: "S48", observed: "Reassigning an active task returned 409 or hung for 20 seconds; the old run kept executing.", fixes: ["#19354"] },
|
||||
"three-turn-revise": { signature: "S49", observed: "Turn 3 on an ephemeral Daytona lease (gpt-5.6-sol) stopped behind a reconciliation / send-a-message blocker.", fixes: [] },
|
||||
"long-silent-command": { signature: "S52/S52b", observed: "ACPX-facade runs (Grok, Kimi, OpenRouter/Pi) hung 3/3 after a long silent foreground command; Stop disappeared.", fixes: ["#19407", "#19419"] },
|
||||
"tree-pause-release": { signature: "S50/S50b", observed: "A child started during a tree pause, or did not resume after release without wakeAgents.", fixes: ["#19394", "#19421"] },
|
||||
"queue-steer": { signature: "H2", observed: "Edited, discarded and steered queue entries were lost, replayed, or delivered twice.", fixes: [] },
|
||||
"queued-across-restart": { signature: "S55/U9", observed: "Follow-up queued before a controller restart sat as a bare Queued until another message.", fixes: ["#19431"] },
|
||||
} as const;
|
||||
|
||||
export type EverydayReliabilityCaseId = keyof typeof everydayReliabilitySignatures;
|
||||
export const everydayReliabilityCaseIds = Object.keys(everydayReliabilitySignatures) as EverydayReliabilityCaseId[];
|
||||
|
||||
export function isEverydayReliabilityCase(caseId: string): caseId is EverydayReliabilityCaseId {
|
||||
return Object.hasOwn(everydayReliabilitySignatures, caseId);
|
||||
}
|
||||
|
||||
/** Bounds encoded in the graders. Soak saw 13-90 s once fixed; 4 minutes or never before. */
|
||||
export const RELIABILITY_BOUNDS = {
|
||||
continuationStartMs: 3 * 60_000,
|
||||
reassignRequestMs: 10_000,
|
||||
restartDispatchMs: 6 * 60_000,
|
||||
silentCompletionAfterCommandMs: 8 * 60_000,
|
||||
steeringRetryWindowMs: 2 * 60_000,
|
||||
holdObservationMs: 75_000,
|
||||
} as const;
|
||||
|
||||
/** A resumable slow loop: an interrupted or reassigned run must continue from saved files. */
|
||||
export function slowStepsPrompt(file: string, steps: number, pauseSeconds: number): string {
|
||||
return `Create ${file} in the project workspace and record progress one line at a time: for each step from 1 to ${steps}, append the line "step-N" (for example "step-1"), then wait ${pauseSeconds} seconds before the next step. Do the waiting in ordinary foreground shell commands. If ${file} already has some steps, continue from the next missing step; never write a step twice. When all ${steps} steps are recorded, reply with a one-line summary.`;
|
||||
}
|
||||
|
||||
export const interruptRedirectMessage = (nonce: string) =>
|
||||
`Change of plan: stop the step loop where it is. Create redirect.txt containing exactly REDIRECT_${nonce} and reply with "Redirected ${nonce}".`;
|
||||
export const stopSettleMessage = (nonce: string) =>
|
||||
`After the stop: leave the step file as it is. Create settled.txt containing exactly SETTLED_${nonce} and reply with "Settled ${nonce}".`;
|
||||
export const threeTurnRevisions = (nonce: string) => [
|
||||
`Add two more items to items.md: "delta ${nonce}" and "epsilon ${nonce}". Keep the existing items.`,
|
||||
`Convert items.md into a Markdown table with the columns Number and Item, one row per existing item in the same order. Keep no bullet list.`,
|
||||
] as const;
|
||||
export const longSilentCommand = (iterations = LONG_SILENT_ITERATIONS, seconds = LONG_SILENT_ITERATION_SECONDS) =>
|
||||
`for i in $(seq 1 ${iterations}); do sleep ${seconds}; echo "tick-$i" >> silent.txt; done`;
|
||||
export const queueSteerMessages = (nonce: string) => ({
|
||||
original: `Also create edited.txt containing exactly ORIGINAL_${nonce}.`,
|
||||
edited: `Also create edited.txt containing exactly EDITED_${nonce}.`,
|
||||
discarded: `Also create discarded.txt containing exactly DISCARDED_${nonce}.`,
|
||||
steered: `While you work, also create steered.txt containing exactly STEERED_${nonce}.`,
|
||||
});
|
||||
export const queuedRestartMessage = (nonce: string) =>
|
||||
`Next, create after-restart.txt containing exactly RESTARTED_${nonce} and reply with "Restarted ${nonce}".`;
|
||||
|
||||
const definitions: ReadonlyArray<{
|
||||
id: EverydayReliabilityCaseId;
|
||||
label: string;
|
||||
prompt: (nonce: string) => string;
|
||||
expectedRunCount: number;
|
||||
daytonaTimeoutMs: number;
|
||||
slow?: boolean;
|
||||
}> = [
|
||||
{ id: "interrupt-redirect", label: "Interrupt a run with a follow-up, then send during Stop settlement",
|
||||
prompt: () => slowStepsPrompt("steps.txt", 8, 30), expectedRunCount: 6, daytonaTimeoutMs: 35 * 60_000 },
|
||||
{ id: "reassign-midrun", label: "Reassign an active task to another agent",
|
||||
prompt: () => slowStepsPrompt("steps.txt", 8, 30), expectedRunCount: 3, daytonaTimeoutMs: 30 * 60_000 },
|
||||
{ id: "three-turn-revise", label: "Create and revise one workspace file across three turns",
|
||||
prompt: nonce => `Create items.md in the project workspace with a bullet list of exactly three items: "alpha ${nonce}", "beta ${nonce}" and "gamma ${nonce}". Reply when it is saved.`,
|
||||
expectedRunCount: 3, daytonaTimeoutMs: 35 * 60_000 },
|
||||
{ id: "long-silent-command", label: "Finish after a long silent foreground command",
|
||||
prompt: nonce => `Run exactly this shell command in the project workspace as one foreground command and wait for it to finish. It takes about ${LONG_SILENT_ITERATIONS} minutes and prints nothing; that is expected. Do not background it, shorten it, split it, or interrupt it:\n\n${longSilentCommand()}\n\nAfter it finishes, count the lines in silent.txt and reply with "SILENT_DONE_${nonce} <line count>".`,
|
||||
expectedRunCount: 1, daytonaTimeoutMs: 50 * 60_000, slow: true },
|
||||
{ id: "tree-pause-release", label: "Pause and release a delegated task tree",
|
||||
prompt: () => `Delegate two child tasks to Riley Builder, one per file. Child A: ${slowStepsPrompt("a.txt", 5, 20)} Child B: ${slowStepsPrompt("b.txt", 5, 20)} Keep the implementation with Riley. When both children are done, confirm both files are complete and finish.`,
|
||||
expectedRunCount: 6, daytonaTimeoutMs: 40 * 60_000 },
|
||||
{ id: "queue-steer", label: "Edit, discard and steer queued follow-ups",
|
||||
prompt: () => slowStepsPrompt("steps.txt", 8, 30), expectedRunCount: 2, daytonaTimeoutMs: 30 * 60_000 },
|
||||
{ id: "queued-across-restart", label: "Dispatch a follow-up queued before a controller restart",
|
||||
prompt: () => slowStepsPrompt("steps.txt", 6, 20), expectedRunCount: 3, daytonaTimeoutMs: 30 * 60_000 },
|
||||
];
|
||||
|
||||
export const everydayReliabilityTasks: readonly RunnerTaskFixture[] = definitions.map(definition => ({
|
||||
id: definition.id,
|
||||
label: definition.label,
|
||||
groups: [EVERYDAY_RELIABILITY_GROUP, ...(definition.slow ? [EVERYDAY_RELIABILITY_SLOW_GROUP] : [])],
|
||||
explicitGroup: EVERYDAY_RELIABILITY_GROUP,
|
||||
workMode: "standard",
|
||||
flow: "everyday_workflow",
|
||||
// An upper bound for cost estimates; an offered Retry or workspace deferral may add a run.
|
||||
expectedRunCount: definition.expectedRunCount,
|
||||
minimumExpectedRunCount: 1,
|
||||
automaticRetryPolicy: "single_attempt",
|
||||
attemptTimeoutMs: { local: 20 * 60_000, daytona: definition.daytonaTimeoutMs },
|
||||
expectedTerminalState: { issue: "done", run: "succeeded" },
|
||||
buildTitle: nonce => `${definition.label} ${nonce}`,
|
||||
buildPrompt: definition.prompt,
|
||||
buildVisibleMarker: nonce => `STUDIO_${nonce}`,
|
||||
buildMatchers: () => [], // The workflow records independent reliability checks.
|
||||
}));
|
||||
|
||||
/** The native profile used for every reliability case, plus ACPX facades for the silent command. */
|
||||
export const RELIABILITY_NATIVE_PROFILE_ID = "runner-codex";
|
||||
export const RELIABILITY_FACADE_PROFILE_IDS = ["runner-acpx-grok", "runner-acpx-kimi", "runner-acpx-pi"] as const;
|
||||
|
||||
/** The deliberately small paid matrix: one native Daytona cell per case, plus facades for long-silent-command. */
|
||||
export function everydayReliabilityExecutionIds(suiteId: string,
|
||||
facadeProfileIds: readonly string[] = RELIABILITY_FACADE_PROFILE_IDS): string[] {
|
||||
return [
|
||||
...everydayReliabilityCaseIds.map(caseId => `${suiteId}.${RELIABILITY_NATIVE_PROFILE_ID}.daytona.${caseId}`),
|
||||
...facadeProfileIds.map(profileId => `${suiteId}.${profileId}.daytona.long-silent-command`),
|
||||
];
|
||||
}
|
||||
@@ -0,0 +1,434 @@
|
||||
import { expect, type Page } from "@playwright/test";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { readFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { pollUntil, type RunnerApi } from "./api.js";
|
||||
import {
|
||||
LONG_SILENT_ITERATION_SECONDS, LONG_SILENT_ITERATIONS, RELIABILITY_BOUNDS, everydayReliabilitySignatures,
|
||||
interruptRedirectMessage, queueSteerMessages, queuedRestartMessage, slowStepsPrompt, stopSettleMessage,
|
||||
threeTurnRevisions, type EverydayReliabilityCaseId,
|
||||
} from "./everyday-reliability-cases.js";
|
||||
import {
|
||||
executedRuns, gradeInterruptRedirect, recoveredRunIds, gradeLongSilentCommand, gradeQueueSteer, gradeQueuedAcrossRestart,
|
||||
gradeReassignMidrun, gradeResumptionRule, gradeThreeTurnRevise, gradeTreePauseRelease, runIssueId, runsCarrying,
|
||||
type ResumptionObservation, type RetryClick,
|
||||
} from "./everyday-reliability.js";
|
||||
import { isActiveStoryRun, type StoryCheck, type StoryIssue, type StoryReplyExpectation, type StoryRun } from "./everyday-observations.js";
|
||||
import { collectRunEvents } from "./run-observations.js";
|
||||
import type { MatrixExecution } from "./types.js";
|
||||
|
||||
type Row = Record<string, any>;
|
||||
interface StoryState { issues: Array<StoryIssue & Row>; runs: StoryRun[] }
|
||||
|
||||
/** Narrow capabilities lent by the everyday workflow; no direct database or runner hooks. */
|
||||
export interface ReliabilityFlowContext {
|
||||
page: Page;
|
||||
api: RunnerApi;
|
||||
execution: MatrixExecution;
|
||||
nonce: string;
|
||||
leadAgentId: string;
|
||||
helperAgentId?: string;
|
||||
workspacePath: string;
|
||||
deadlineAt: number;
|
||||
parentId: string;
|
||||
refresh(): Promise<StoryState>;
|
||||
/** Composer reply; returns the stored user comment IDs. */
|
||||
reply(message: string): Promise<string[]>;
|
||||
openParent(): Promise<void>;
|
||||
restart(): Promise<unknown>;
|
||||
note(action: string, detail?: unknown): void;
|
||||
allowInterrupted(runId: string): void;
|
||||
/** Record failures whose work a later run verifiably resumed. */
|
||||
allowRecovered(runIds: readonly string[]): void;
|
||||
evidence(name: string, value: unknown): Promise<void>;
|
||||
capture(id: string, label: string, file: string): Promise<void>;
|
||||
}
|
||||
|
||||
export interface ReliabilityOutcome {
|
||||
expectedReply?: StoryReplyExpectation;
|
||||
/** Agents permitted to execute on the parent task (reassignment moves ownership). */
|
||||
parentAgentIds: string[];
|
||||
checks(): Promise<StoryCheck[]>;
|
||||
}
|
||||
|
||||
const sleep = (milliseconds: number) => new Promise(resolve => setTimeout(resolve, milliseconds));
|
||||
|
||||
/** Interrupted work resumes automatically, or a Retry works on first use; otherwise fail. */
|
||||
class ResumptionWatcher {
|
||||
readonly observations: ResumptionObservation[] = [];
|
||||
readonly retryClicks: RetryClick[] = [];
|
||||
stalled?: string;
|
||||
constructor(private readonly ctx: ReliabilityFlowContext) {}
|
||||
|
||||
async observe(state: StoryState) {
|
||||
const issues = state.issues.filter(issue => issue.id === this.ctx.parentId || issue.parentId === this.ctx.parentId);
|
||||
for (const issue of issues) {
|
||||
const [detail, execution] = await Promise.all([
|
||||
this.ctx.api.get<Row>(`/api/issues/${issue.id}`),
|
||||
this.ctx.api.get<Row | null>(`/api/issues/${issue.id}/execution`).catch(() => null),
|
||||
]);
|
||||
const observation: ResumptionObservation = {
|
||||
at: new Date().toISOString(), issueId: issue.id,
|
||||
blocker: detail.executionBlocker ?? null, interruption: detail.interruption ?? null,
|
||||
execution: execution?.execution ? { phase: execution.execution.phase, label: execution.execution.label, cause: execution.execution.cause } : null,
|
||||
};
|
||||
this.observations.push(observation);
|
||||
// An automatic retry that is already scheduled is the harness doing its job; wait for it.
|
||||
const automatic = Boolean(detail.scheduledRetry) || ["working", "retry_scheduled", "resolved"].includes(detail.interruption?.state);
|
||||
const offered = detail.interruption?.canRetry === true || detail.executionBlocker?.canRetry === true;
|
||||
if (offered && !automatic && detail.status !== "done"
|
||||
&& !state.runs.some(run => isActiveStoryRun(run) && runIssueId(run) === issue.id))
|
||||
await this.retry(issue.id, detail.interruption?.observedRunId ?? detail.executionBlocker?.runId ?? null);
|
||||
}
|
||||
}
|
||||
|
||||
/** Use an offered Retry once and require a new run on that task. */
|
||||
private async retry(issueId: string, observedRunId: string | null) {
|
||||
const click: RetryClick = { at: new Date().toISOString(), issueId };
|
||||
this.retryClicks.push(click);
|
||||
if (this.retryClicks.filter(previous => previous.issueId === issueId).length > 1) return;
|
||||
const before = new Set((await this.ctx.refresh()).runs.map(run => run.id));
|
||||
const response = await this.ctx.api.post<Row>(`/api/issues/${issueId}/retry`, { idempotencyKey: randomUUID(), observedRunId });
|
||||
this.ctx.note("reliability-retry-used", { issueId, observedRunId, accepted: response.accepted, stale: response.stale });
|
||||
const started = await pollUntil({
|
||||
label: "run started by Retry", deadlineAt: Math.min(this.ctx.deadlineAt, Date.now() + 120_000), intervalMs: 2_000,
|
||||
load: () => this.ctx.refresh(),
|
||||
accept: state => state.runs.some(run => !before.has(run.id) && runIssueId(run) === issueId),
|
||||
}).catch(() => undefined);
|
||||
click.startedRunId = started?.runs.find(run => !before.has(run.id) && runIssueId(run) === issueId)?.id;
|
||||
}
|
||||
|
||||
/** Poll until accepted, recording blockers and using any offered Retry. */
|
||||
async until(label: string, accept: (state: StoryState) => boolean, boundMs: number,
|
||||
onPoll?: () => Promise<void>): Promise<StoryState> {
|
||||
const deadlineAt = Math.min(this.ctx.deadlineAt, Date.now() + boundMs);
|
||||
let last: StoryState | undefined;
|
||||
while (Date.now() < deadlineAt) {
|
||||
await onPoll?.();
|
||||
last = await this.ctx.refresh();
|
||||
if (accept(last)) return last;
|
||||
await this.observe(last);
|
||||
await sleep(3_000);
|
||||
}
|
||||
this.stalled = `Stalled waiting for ${label} (bound ${Math.round(boundMs / 1000)}s).`;
|
||||
throw new Error(this.stalled);
|
||||
}
|
||||
|
||||
checks() {
|
||||
return gradeResumptionRule({ observations: this.observations, retryClicks: this.retryClicks, stalled: this.stalled });
|
||||
}
|
||||
}
|
||||
|
||||
const allDone = (state: StoryState) => state.issues.length > 0
|
||||
&& state.issues.every(issue => issue.status === "done" && (issue.queuedComments?.entries?.length ?? 0) === 0)
|
||||
&& !state.runs.some(isActiveStoryRun);
|
||||
|
||||
/**
|
||||
* A claimed run is not a provider turn: require a completed native tool event,
|
||||
* unless the first tool is itself the long command under test.
|
||||
*/
|
||||
async function waitForActiveTurn(ctx: ReliabilityFlowContext, issueId = ctx.parentId, label = "active provider turn",
|
||||
requireTool = true): Promise<StoryRun> {
|
||||
const state = await pollUntil({
|
||||
label, deadlineAt: ctx.deadlineAt, intervalMs: 2_000,
|
||||
load: async () => {
|
||||
const current = await ctx.refresh();
|
||||
const run = current.runs.find(candidate => candidate.status === "running" && runIssueId(candidate) === issueId);
|
||||
if (!run) return { current, run: undefined, tool: false };
|
||||
if (!requireTool) return { current, run, tool: true };
|
||||
const events = await collectRunEvents<Row>((afterSeq, limit) =>
|
||||
ctx.api.get(`/api/heartbeat-runs/${run.id}/events?afterSeq=${afterSeq}&limit=${limit}`));
|
||||
return { current, run, tool: events.some(row => row.payload?.prpEvent?.eventType === "tool.execution.completed") };
|
||||
},
|
||||
accept: value => Boolean(value.run && value.tool),
|
||||
reject: value => value.current.issues.find(issue => issue.id === issueId)?.status === "done"
|
||||
&& !value.current.runs.some(isActiveStoryRun) ? "Boundary unexercised: the task finished before a turn could be interrupted" : undefined,
|
||||
});
|
||||
ctx.note("reliability-active-turn", { issueId, runId: state.run!.id });
|
||||
return state.run!;
|
||||
}
|
||||
|
||||
/** Saved workspace bytes through the public file API, or the local project path. */
|
||||
async function readWorkspaceFile(ctx: ReliabilityFlowContext, file: string): Promise<string | undefined> {
|
||||
try {
|
||||
const value = await ctx.api.get<Row>(`/api/issues/${ctx.parentId}/file-resources/content?workspace=execution&path=${encodeURIComponent(file)}`);
|
||||
if (value.content?.encoding === "utf8" && typeof value.content.data === "string") return value.content.data;
|
||||
} catch { /* Fall back to the local project checkout below. */ }
|
||||
if (ctx.execution.environment.id !== "local") return undefined;
|
||||
return readFile(path.join(ctx.workspacePath, file), "utf8").catch(() => undefined);
|
||||
}
|
||||
|
||||
async function queue(ctx: ReliabilityFlowContext): Promise<Row> {
|
||||
return ctx.api.get<Row>(`/api/issues/${ctx.parentId}/queued-comments`);
|
||||
}
|
||||
|
||||
/** Issue an HTTP mutation without throwing, keeping its status and error code. */
|
||||
async function send(ctx: ReliabilityFlowContext, method: "post" | "patch" | "delete", url: string, data: unknown) {
|
||||
const startedAt = Date.now();
|
||||
const response = await ctx.api.request[method](url, { data });
|
||||
const body = await response.json().catch(() => ({})) as Row;
|
||||
return { status: response.status(), latencyMs: Date.now() - startedAt, body, code: typeof body.code === "string" ? body.code : undefined };
|
||||
}
|
||||
|
||||
async function interruptRedirect(ctx: ReliabilityFlowContext, watcher: ResumptionWatcher): Promise<ReliabilityOutcome> {
|
||||
const active = await waitForActiveTurn(ctx);
|
||||
const [commentId] = await ctx.reply(interruptRedirectMessage(ctx.nonce));
|
||||
if (!commentId) throw new Error("The interrupt follow-up was not stored");
|
||||
const interrupt = ctx.page.getByTestId(`task-chat-queued-interrupt-${commentId}`);
|
||||
await expect(interrupt).toBeVisible({ timeout: 30_000 });
|
||||
ctx.allowInterrupted(active.id);
|
||||
const interruptedAt = Date.now();
|
||||
await interrupt.click();
|
||||
ctx.note("reliability-interrupt-clicked", { runId: active.id, commentId });
|
||||
await watcher.until("interrupt continuation finished", state => allDone(state)
|
||||
&& runsCarrying(state.runs, commentId).some(run => run.status === "succeeded"), RELIABILITY_BOUNDS.continuationStartMs + 10 * 60_000);
|
||||
await ctx.capture("reliability-interrupt", "Interrupted follow-up delivered once", "reliability-interrupt.png");
|
||||
|
||||
// Stop, then send while the stopped run's save is still settling: the workspace_busy retry path.
|
||||
await ctx.reply(slowStepsPrompt("more-steps.txt", 6, 30));
|
||||
const stopped = await waitForActiveTurn(ctx, ctx.parentId, "second active provider turn");
|
||||
ctx.allowInterrupted(stopped.id);
|
||||
await ctx.page.getByTestId("task-chat-composer-stop").last().click();
|
||||
ctx.note("reliability-stop-clicked", { runId: stopped.id });
|
||||
const sentAt = Date.now();
|
||||
const [settleCommentId] = await ctx.reply(stopSettleMessage(ctx.nonce));
|
||||
if (!settleCommentId) throw new Error("The message sent during Stop settlement was not stored");
|
||||
await watcher.until("message sent during Stop settlement consumed", state => allDone(state)
|
||||
&& runsCarrying(state.runs, settleCommentId).some(run => run.status === "succeeded"), RELIABILITY_BOUNDS.continuationStartMs + 10 * 60_000);
|
||||
return {
|
||||
expectedReply: `Settled ${ctx.nonce}`, parentAgentIds: [ctx.leadAgentId],
|
||||
checks: async () => {
|
||||
const state = await ctx.refresh();
|
||||
const parent = state.issues.find(issue => issue.id === ctx.parentId);
|
||||
return gradeInterruptRedirect({ runs: state.runs, comments: parent?.comments ?? [], interruptedRunId: active.id,
|
||||
interruptCommentId: commentId, interruptedAt, stopSettle: { stoppedRunId: stopped.id, commentId: settleCommentId, sentAt } });
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function reassignMidrun(ctx: ReliabilityFlowContext, watcher: ResumptionWatcher): Promise<ReliabilityOutcome> {
|
||||
if (!ctx.helperAgentId) throw new Error("Reassignment requires the second agent before task creation");
|
||||
const active = await waitForActiveTurn(ctx);
|
||||
ctx.allowInterrupted(active.id);
|
||||
const requestedAt = Date.now();
|
||||
const response = await send(ctx, "patch", `/api/issues/${ctx.parentId}`, { assigneeAgentId: ctx.helperAgentId });
|
||||
ctx.note("reliability-reassigned", { runId: active.id, status: response.status, latencyMs: response.latencyMs, code: response.code });
|
||||
if (response.status >= 300) throw new Error(`Reassignment returned ${response.status} ${response.code ?? ""}`);
|
||||
await ctx.openParent();
|
||||
await watcher.until("reassigned task finished", state => allDone(state)
|
||||
&& executedRuns(state.runs).some(run => run.agentId === ctx.helperAgentId && run.status === "succeeded"), 20 * 60_000);
|
||||
return {
|
||||
parentAgentIds: [ctx.leadAgentId, ctx.helperAgentId],
|
||||
checks: async () => gradeReassignMidrun({ runs: (await ctx.refresh()).runs, issueId: ctx.parentId, oldRunId: active.id,
|
||||
newAgentId: ctx.helperAgentId!, request: { status: response.status, latencyMs: response.latencyMs, requestedAt },
|
||||
stepFile: await readWorkspaceFile(ctx, "steps.txt"), steps: 8 }),
|
||||
};
|
||||
}
|
||||
|
||||
async function threeTurnRevise(ctx: ReliabilityFlowContext, watcher: ResumptionWatcher): Promise<ReliabilityOutcome> {
|
||||
const turnBound = 12 * 60_000;
|
||||
// Turn 1 is the last successful parent run before the first revision (it may follow an automatic resume).
|
||||
const turnOne = (await watcher.until("turn 1 finished", allDone, turnBound)).runs
|
||||
.filter(run => runIssueId(run) === ctx.parentId && run.status === "succeeded")
|
||||
.sort((a, b) => Date.parse(a.finishedAt ?? "") - Date.parse(b.finishedAt ?? "")).at(-1);
|
||||
const commentIds: string[] = [];
|
||||
for (const [index, revision] of threeTurnRevisions(ctx.nonce).entries()) {
|
||||
const [commentId] = await ctx.reply(revision);
|
||||
if (!commentId) throw new Error(`Revision ${index + 2} was not stored`);
|
||||
commentIds.push(commentId);
|
||||
await watcher.until(`turn ${index + 2} finished`, state => allDone(state)
|
||||
&& runsCarrying(state.runs, commentId).some(run => run.status === "succeeded"), turnBound);
|
||||
ctx.note("reliability-turn-finished", { turn: index + 2, commentId });
|
||||
}
|
||||
return {
|
||||
parentAgentIds: [ctx.leadAgentId],
|
||||
checks: async () => {
|
||||
const state = await ctx.refresh();
|
||||
return gradeThreeTurnRevise({ nonce: ctx.nonce, content: await readWorkspaceFile(ctx, "items.md"),
|
||||
turnRuns: [turnOne, ...commentIds.map(id => runsCarrying(state.runs, id).find(run => run.status === "succeeded")
|
||||
?? runsCarrying(state.runs, id).at(-1))] });
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function longSilentCommand(ctx: ReliabilityFlowContext, watcher: ResumptionWatcher): Promise<ReliabilityOutcome> {
|
||||
const commandMs = LONG_SILENT_ITERATIONS * LONG_SILENT_ITERATION_SECONDS * 1000;
|
||||
const run = await waitForActiveTurn(ctx, ctx.parentId, "silent command run", false);
|
||||
const samples: Array<{ at: string; canStop: boolean; source: "api" | "browser" }> = [];
|
||||
const browserAt = new Set([0, Math.floor(LONG_SILENT_ITERATIONS / 2), LONG_SILENT_ITERATIONS - 1]);
|
||||
const bound = Date.now() + commandMs + RELIABILITY_BOUNDS.silentCompletionAfterCommandMs;
|
||||
for (let minute = 0; Date.now() < Math.min(bound, ctx.deadlineAt); minute += 1) {
|
||||
const current = await ctx.api.get<StoryRun>(`/api/heartbeat-runs/${run.id}`);
|
||||
if (current.status !== "running") break;
|
||||
const projection = await ctx.api.get<Row | null>(`/api/issues/${ctx.parentId}/execution`).catch(() => null);
|
||||
samples.push({ at: new Date().toISOString(), canStop: projection?.execution?.canStop === true, source: "api" });
|
||||
if (browserAt.has(minute)) {
|
||||
await ctx.openParent();
|
||||
samples.push({ at: new Date().toISOString(), source: "browser",
|
||||
canStop: await ctx.page.getByTestId("task-chat-composer-stop").last().isVisible().catch(() => false) });
|
||||
}
|
||||
await sleep(60_000);
|
||||
}
|
||||
ctx.note("reliability-stop-samples", samples);
|
||||
await watcher.until("silent command run finished", allDone, RELIABILITY_BOUNDS.silentCompletionAfterCommandMs);
|
||||
return {
|
||||
expectedReply: body => body.includes(`SILENT_DONE_${ctx.nonce}`), parentAgentIds: [ctx.leadAgentId],
|
||||
checks: async () => {
|
||||
const state = await ctx.refresh();
|
||||
const reply = state.issues.find(issue => issue.id === ctx.parentId)?.comments
|
||||
?.filter(comment => comment.authorAgentId).map(comment => String(comment.body ?? "")).join("\n");
|
||||
return gradeLongSilentCommand({ run: state.runs.find(candidate => candidate.id === run.id), content: await readWorkspaceFile(ctx, "silent.txt"),
|
||||
iterations: LONG_SILENT_ITERATIONS, iterationSeconds: LONG_SILENT_ITERATION_SECONDS, stopSamples: samples, reply, nonce: ctx.nonce });
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function treePauseRelease(ctx: ReliabilityFlowContext, watcher: ResumptionWatcher): Promise<ReliabilityOutcome> {
|
||||
const childrenOf = (state: StoryState) => state.issues.filter(issue => issue.parentId === ctx.parentId);
|
||||
const waitingStatuses = ["queued", "scheduled_retry"];
|
||||
const boundary = await pollUntil({
|
||||
label: "delegated child running with a sibling queued", deadlineAt: ctx.deadlineAt, intervalMs: 2_000,
|
||||
load: async () => {
|
||||
const state = await ctx.refresh();
|
||||
const children = childrenOf(state);
|
||||
const childRuns = state.runs.filter(run => children.some(child => child.id === runIssueId(run)));
|
||||
return { state, children, running: childRuns.find(run => run.status === "running"), waiting: childRuns.find(run => waitingStatuses.includes(run.status)),
|
||||
allActive: children.length >= 2 && children.every(child => childRuns.some(run => runIssueId(run) === child.id && isActiveStoryRun(run))) };
|
||||
},
|
||||
// A queued sibling or parked workspace_busy retry is the S50 boundary; two concurrently active children also qualify.
|
||||
accept: value => value.children.length >= 2 && (Boolean(value.waiting) || value.allActive),
|
||||
reject: value => value.children.length >= 2 && value.children.every(child => child.status === "done")
|
||||
? "Boundary unexercised: both children finished before the hold" : undefined,
|
||||
});
|
||||
const childIds = boundary.children.map(child => child.id);
|
||||
ctx.note("reliability-tree-boundary", { childIds, running: boundary.running?.id, waiting: boundary.waiting?.id, waitingStatus: boundary.waiting?.status });
|
||||
const holdCreatedAt = Date.now();
|
||||
const created = await ctx.api.post<Row>(`/api/issues/${ctx.parentId}/tree-holds`, { mode: "pause", reason: `Reliability eval pause ${ctx.nonce}` });
|
||||
const holdId = String(created.hold?.id ?? "");
|
||||
const interruptedRuns = ((created.preview?.activeRuns ?? []) as Row[]).map(run => ({ id: String(run.id), issueId: String(run.issueId) }));
|
||||
for (const run of interruptedRuns) ctx.allowInterrupted(run.id);
|
||||
ctx.note("reliability-tree-held", { holdId, interruptedRuns });
|
||||
if (!holdId) throw new Error("Tree hold was not created");
|
||||
// Hold past the parked retry's due time so a premature dispatch would be observed.
|
||||
const dueAt = Math.max(0, ...boundary.state.runs.map(run => Date.parse(String((run as Row).scheduledRetryAt ?? ""))).filter(Number.isFinite));
|
||||
const holdUntil = Math.min(ctx.deadlineAt, Math.max(holdCreatedAt + RELIABILITY_BOUNDS.holdObservationMs, dueAt + 15_000), holdCreatedAt + 5 * 60_000);
|
||||
while (Date.now() < holdUntil) {
|
||||
const state = await ctx.refresh();
|
||||
// A run that started while the hold request was in flight is one the hold interrupted.
|
||||
const early = executedRuns(state.runs).find(run => childIds.includes(runIssueId(run) ?? "")
|
||||
&& !interruptedRuns.some(interrupted => interrupted.id === run.id) && Date.parse(run.startedAt!) > holdCreatedAt);
|
||||
if (early) throw new Error(`Child run ${early.id} started during the tree hold`);
|
||||
await sleep(5_000);
|
||||
}
|
||||
const releasedAt = Date.now();
|
||||
await ctx.api.post(`/api/issues/${ctx.parentId}/tree-holds/${holdId}/release`, { reason: `Reliability eval release ${ctx.nonce}` });
|
||||
ctx.note("reliability-tree-released", { holdId, wakeAgents: false });
|
||||
await ctx.openParent();
|
||||
await watcher.until("released tree finished", allDone, 20 * 60_000);
|
||||
return {
|
||||
parentAgentIds: [ctx.leadAgentId],
|
||||
checks: async () => gradeTreePauseRelease({ runs: (await ctx.refresh()).runs, childIds, holdCreatedAt, releasedAt, interruptedRuns }),
|
||||
};
|
||||
}
|
||||
|
||||
async function queueSteer(ctx: ReliabilityFlowContext, watcher: ResumptionWatcher): Promise<ReliabilityOutcome> {
|
||||
const messages = queueSteerMessages(ctx.nonce);
|
||||
const active = await waitForActiveTurn(ctx);
|
||||
const post = async (body: string) => (await ctx.api.post<Row>(`/api/issues/${ctx.parentId}/comments`, { body })).id as string;
|
||||
const editedId = await post(messages.original);
|
||||
const discardedId = await post(messages.discarded);
|
||||
const steeredId = await post(messages.steered);
|
||||
ctx.note("reliability-queued", { editedId, discardedId, steeredId, runId: active.id });
|
||||
let current = await pollUntil({ label: "three queued follow-ups", deadlineAt: ctx.deadlineAt, intervalMs: 1_000,
|
||||
load: () => queue(ctx), accept: value => [editedId, discardedId, steeredId]
|
||||
.every(id => (value.entries ?? []).some((entry: Row) => entry.comment?.id === id)) });
|
||||
const edit = await send(ctx, "patch", `/api/issues/${ctx.parentId}/queued-comments/${editedId}`,
|
||||
{ queueId: current.queueId, revision: current.revision, body: messages.edited });
|
||||
if (edit.status >= 300) throw new Error(`Queued edit returned ${edit.status} ${edit.code ?? ""}`);
|
||||
current = await queue(ctx);
|
||||
const discard = await send(ctx, "delete", `/api/issues/${ctx.parentId}/queued-comments/${discardedId}`,
|
||||
{ queueId: current.queueId, revision: current.revision });
|
||||
if (discard.status >= 300) throw new Error(`Queued discard returned ${discard.status} ${discard.code ?? ""}`);
|
||||
const steerAttempts: Array<{ status: number; code?: string }> = [];
|
||||
const steerUntil = Date.now() + RELIABILITY_BOUNDS.steeringRetryWindowMs;
|
||||
for (;;) {
|
||||
current = await queue(ctx);
|
||||
const attempt = await send(ctx, "post", `/api/issues/${ctx.parentId}/queued-comments/${steeredId}/steer`,
|
||||
{ queueId: current.queueId, revision: current.revision, targetRunId: current.targetRunId ?? active.id });
|
||||
steerAttempts.push({ status: attempt.status, code: attempt.code });
|
||||
if (attempt.status < 300 || attempt.code !== "steering_temporarily_unavailable" || Date.now() > steerUntil) break;
|
||||
await sleep(5_000);
|
||||
}
|
||||
ctx.note("reliability-steer-attempts", steerAttempts);
|
||||
await watcher.until("queued follow-ups finished", state => allDone(state)
|
||||
&& runsCarrying(state.runs, editedId).some(run => run.status === "succeeded"), 20 * 60_000);
|
||||
return {
|
||||
parentAgentIds: [ctx.leadAgentId],
|
||||
checks: async () => gradeQueueSteer({ runs: (await ctx.refresh()).runs, activeRunId: active.id, steerAttempts,
|
||||
edited: { commentId: editedId, originalMarker: `ORIGINAL_${ctx.nonce}`, editedMarker: `EDITED_${ctx.nonce}` },
|
||||
discarded: { commentId: discardedId, marker: `DISCARDED_${ctx.nonce}` },
|
||||
steered: { commentId: steeredId, marker: `STEERED_${ctx.nonce}` },
|
||||
files: { edited: await readWorkspaceFile(ctx, "edited.txt"), discarded: await readWorkspaceFile(ctx, "discarded.txt"),
|
||||
steered: await readWorkspaceFile(ctx, "steered.txt") } }),
|
||||
};
|
||||
}
|
||||
|
||||
async function queuedAcrossRestart(ctx: ReliabilityFlowContext, watcher: ResumptionWatcher): Promise<ReliabilityOutcome> {
|
||||
const active = await waitForActiveTurn(ctx);
|
||||
const [commentId] = await ctx.reply(queuedRestartMessage(ctx.nonce));
|
||||
if (!commentId) throw new Error("The queued follow-up was not stored");
|
||||
// The original turn may be interrupted by shutdown; its recovery is graded by the resumption rule.
|
||||
ctx.allowInterrupted(active.id);
|
||||
const dispatchedBeforeRestart = runsCarrying((await ctx.refresh()).runs, commentId).length > 0;
|
||||
// A healthy replacement may dispatch before the restart handshake returns; bound from the request.
|
||||
const restartStartedAt = Date.now();
|
||||
await ctx.restart();
|
||||
ctx.note("reliability-controller-restarted", { runId: active.id, commentId, dispatchedBeforeRestart, readyAfterMs: Date.now() - restartStartedAt });
|
||||
const waitObservations: Array<{ at: string; phase?: string; label?: string; cause?: string | null }> = [];
|
||||
await watcher.until("queued follow-up dispatched after restart", state => runsCarrying(state.runs, commentId).length > 0,
|
||||
RELIABILITY_BOUNDS.restartDispatchMs + 60_000, async () => {
|
||||
const value = await ctx.api.get<Row | null>(`/api/issues/${ctx.parentId}/execution`).catch(() => null);
|
||||
if (value?.execution) waitObservations.push({ at: new Date().toISOString(), phase: value.execution.phase,
|
||||
label: value.execution.label, cause: value.execution.cause });
|
||||
});
|
||||
await ctx.openParent();
|
||||
await watcher.until("queued follow-up finished", allDone, 15 * 60_000);
|
||||
return {
|
||||
expectedReply: `Restarted ${ctx.nonce}`, parentAgentIds: [ctx.leadAgentId],
|
||||
checks: async () => gradeQueuedAcrossRestart({ runs: (await ctx.refresh()).runs, commentId, restartStartedAt,
|
||||
dispatchedBeforeRestart, waitObservations }),
|
||||
};
|
||||
}
|
||||
|
||||
const scenarios: Record<EverydayReliabilityCaseId, (ctx: ReliabilityFlowContext, watcher: ResumptionWatcher) => Promise<ReliabilityOutcome>> = {
|
||||
"interrupt-redirect": interruptRedirect,
|
||||
"reassign-midrun": reassignMidrun,
|
||||
"three-turn-revise": threeTurnRevise,
|
||||
"long-silent-command": longSilentCommand,
|
||||
"tree-pause-release": treePauseRelease,
|
||||
"queue-steer": queueSteer,
|
||||
"queued-across-restart": queuedAcrossRestart,
|
||||
};
|
||||
|
||||
/** Drive one reliability scenario after task creation; checks combine its grader and the resumption rule. */
|
||||
export async function driveReliabilityScenario(caseId: EverydayReliabilityCaseId, ctx: ReliabilityFlowContext): Promise<ReliabilityOutcome> {
|
||||
ctx.note("reliability-scenario", { caseId, ...everydayReliabilitySignatures[caseId] });
|
||||
const watcher = new ResumptionWatcher(ctx);
|
||||
try {
|
||||
const outcome = await scenarios[caseId](ctx, watcher);
|
||||
const recovered = recoveredRunIds((await ctx.refresh()).runs);
|
||||
ctx.allowRecovered(recovered);
|
||||
if (recovered.length) ctx.note("reliability-recovered-runs", recovered);
|
||||
return { ...outcome, checks: async () => {
|
||||
const checks = [...await outcome.checks(), ...watcher.checks()];
|
||||
await ctx.evidence("reliability-observations.json", { caseId, signature: everydayReliabilitySignatures[caseId],
|
||||
observations: watcher.observations, retryClicks: watcher.retryClicks, checks });
|
||||
return checks;
|
||||
} };
|
||||
} catch (error) {
|
||||
await ctx.evidence("reliability-observations.json", { caseId, signature: everydayReliabilitySignatures[caseId],
|
||||
observations: watcher.observations, retryClicks: watcher.retryClicks, stalled: watcher.stalled,
|
||||
error: error instanceof Error ? error.message : String(error) });
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,253 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { buildMatrixJobs, parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js";
|
||||
import { everydayReliabilityFacadeProfileIds, runnerMatrix } from "./catalog.js";
|
||||
import { scheduledSelection } from "./scheduled-operations.js";
|
||||
import {
|
||||
EVERYDAY_RELIABILITY_GROUP, everydayReliabilityCaseIds, everydayReliabilityExecutionIds, everydayReliabilitySignatures,
|
||||
everydayReliabilityTasks, isEverydayReliabilityCase,
|
||||
} from "./everyday-reliability-cases.js";
|
||||
import {
|
||||
completeStepLines, concurrentExecutors, recoveredRunIds, gradeInterruptRedirect, gradeLongSilentCommand, gradeQueueSteer,
|
||||
gradeQueuedAcrossRestart, gradeReassignMidrun, gradeResumptionRule, gradeThreeTurnRevise, gradeTreePauseRelease,
|
||||
requiresUserMessage,
|
||||
} from "./everyday-reliability.js";
|
||||
import { isExpectedStoryInterruption, storyLifecycleChecks, type StoryCheck, type StoryRun } from "./everyday-observations.js";
|
||||
|
||||
const T0 = Date.parse("2026-11-06T10:00:00.000Z");
|
||||
const at = (seconds: number) => new Date(T0 + seconds * 1000).toISOString();
|
||||
const run = (id: string, overrides: Partial<StoryRun> = {}): StoryRun => ({
|
||||
id, companyId: "company", agentId: "lead", status: "succeeded", runtimeMode: "native", runnerInstanceId: "runner",
|
||||
nativeIssueId: "parent", startedAt: at(0), finishedAt: at(60), ...overrides,
|
||||
});
|
||||
/** A run woken to deliver `wake`; its prompt holds `history` (the whole conversation so far). */
|
||||
const carrying = (id: string, wake: string, overrides: Partial<StoryRun> = {}, history = "") =>
|
||||
run(id, { contextSnapshot: { issueId: "parent", wakeCommentIds: [wake] },
|
||||
runnerProfileJson: { nativeExecutionInput: { task: { prompt: `${history} ${wake}` } } }, ...overrides });
|
||||
const reliabilityCellCount = everydayReliabilityCaseIds.length + everydayReliabilityFacadeProfileIds.length;
|
||||
const failed = (checks: StoryCheck[]) => checks.filter(check => !check.passed).map(check => check.id);
|
||||
|
||||
describe("reliability selection stays explicit", () => {
|
||||
const reliable = (executions: { groups: readonly string[] }[]) => executions.filter(execution => execution.groups.includes(EVERYDAY_RELIABILITY_GROUP));
|
||||
|
||||
it("selects the whole reliability set only through its group", () => {
|
||||
const selected = selectRunnerExecutions(parseRunnerSelectors(["--suite", "everyday-workflows", "--group", "reliability"]));
|
||||
expect(selected.map(execution => execution.id).sort())
|
||||
.toEqual(everydayReliabilityExecutionIds("everyday-workflows", everydayReliabilityFacadeProfileIds).sort());
|
||||
expect(selected).toHaveLength(reliabilityCellCount);
|
||||
expect(selected.every(execution => execution.environment.id === "daytona")).toBe(true);
|
||||
});
|
||||
|
||||
it("is excluded from --all, the plain suite, generic selectors and scheduled campaigns", () => {
|
||||
for (const args of [["--all"], ["--suite", "everyday-workflows"], ["--suite", "everyday-workflows", "--environment", "daytona"],
|
||||
["--suite", "everyday-workflows", "--profile", "runner-codex"], ["--suite", "everyday-workflows", "--group", "native"],
|
||||
["--profile", "runner-codex"], ["--environment", "daytona"]]) {
|
||||
expect(reliable(selectRunnerExecutions(parseRunnerSelectors(args))), args.join(" ")).toEqual([]);
|
||||
}
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--suite", "everyday-workflows"]))).toHaveLength(52);
|
||||
for (const cadence of ["daily", "weekly"] as const)
|
||||
expect(scheduledSelection(cadence).filter(id => id.startsWith("everyday-workflows."))).toEqual([]);
|
||||
});
|
||||
|
||||
it("allows a single case or exact ID and keeps the cells discoverable in --list", () => {
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--suite", "everyday-workflows", "--case", "long-silent-command"]))
|
||||
.map(execution => execution.profile.id).sort()).toEqual(["runner-codex", ...everydayReliabilityFacadeProfileIds].sort());
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--id", "everyday-workflows.runner-codex.daytona.queue-steer"]))).toHaveLength(1);
|
||||
expect(reliable(selectRunnerExecutions(parseRunnerSelectors(["--list"])))).toHaveLength(reliabilityCellCount);
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--suite", "everyday-workflows", "--group", "slow"]))
|
||||
.every(execution => execution.task.id === "long-silent-command")).toBe(true);
|
||||
});
|
||||
|
||||
it("uses one native Daytona profile per case and facades only for the silent command", () => {
|
||||
const cells = runnerMatrix.filter(execution => isEverydayReliabilityCase(execution.task.id));
|
||||
expect(cells).toHaveLength(reliabilityCellCount);
|
||||
for (const caseId of everydayReliabilityCaseIds) {
|
||||
const profiles = cells.filter(execution => execution.task.id === caseId).map(execution => execution.profile.id);
|
||||
expect(profiles).toContain("runner-codex");
|
||||
expect(profiles.length).toBe(caseId === "long-silent-command" ? 1 + everydayReliabilityFacadeProfileIds.length : 1);
|
||||
}
|
||||
expect(runnerMatrix.filter(execution => execution.suite.id === "everyday-workflows"
|
||||
&& ["runner-acpx-grok", "runner-acpx-kimi", "runner-acpx-pi"].includes(execution.profile.id)
|
||||
&& execution.task.id !== "long-silent-command")).toEqual([]);
|
||||
const turnThree = cells.find(execution => execution.task.id === "three-turn-revise")!;
|
||||
expect(turnThree.environment.configurationKey).toBeUndefined(); // Per-turn ephemeral lease, as in S49.
|
||||
expect(cells.find(execution => execution.task.id === "long-silent-command")!.groups).toContain("slow");
|
||||
});
|
||||
|
||||
it("gives the slow case its own deadline and single attempts", () => {
|
||||
const silent = everydayReliabilityTasks.find(task => task.id === "long-silent-command")!;
|
||||
expect(silent.attemptTimeoutMs.daytona).toBeGreaterThanOrEqual(45 * 60_000);
|
||||
expect(everydayReliabilityTasks.every(task => task.automaticRetryPolicy === "single_attempt")).toBe(true);
|
||||
const [job] = buildMatrixJobs(runnerMatrix.filter(execution => execution.id === "everyday-workflows.runner-codex.daytona.long-silent-command"));
|
||||
expect(job!.timeoutMinutes).toBeGreaterThan(100);
|
||||
});
|
||||
|
||||
it("requires the facade credentials before provider spend and records each soak signature", () => {
|
||||
const credentials = (profile: string) => runnerMatrix.find(execution =>
|
||||
execution.id === `everyday-workflows.${profile}.daytona.long-silent-command`)!.requiredCredentials;
|
||||
const facadeCredentials: Record<string, string> = { "runner-acpx-grok": "XAI_API_KEY", "runner-acpx-kimi": "KIMI_API_KEY", "runner-acpx-pi": "OPENROUTER_API_KEY" };
|
||||
for (const profile of everydayReliabilityFacadeProfileIds)
|
||||
expect(credentials(profile)).toEqual([facadeCredentials[profile], "DAYTONA_API_KEY"]);
|
||||
expect(Object.keys(everydayReliabilitySignatures).sort()).toEqual([...everydayReliabilityCaseIds].sort());
|
||||
for (const task of everydayReliabilityTasks)
|
||||
expect(task.buildPrompt("sample")).not.toMatch(/finish_task|paperclip_finish|PATCH|mark .*done|api\/issues/i);
|
||||
});
|
||||
});
|
||||
|
||||
describe("reliability interruption bookkeeping", () => {
|
||||
it("exempts only deliberate interrupt and reassignment stops", () => {
|
||||
expect(isExpectedStoryInterruption(run("a", { status: "cancelled", errorCode: "operator_interrupted" }), ["a"])).toBe(true);
|
||||
expect(isExpectedStoryInterruption(run("a", { status: "cancelled", errorCode: "issue_reassigned" }), ["a"])).toBe(true);
|
||||
expect(isExpectedStoryInterruption(run("a", { status: "cancelled", errorCode: "issue_reassigned" }), [])).toBe(false);
|
||||
expect(isExpectedStoryInterruption(run("a", { status: "cancelled", errorCode: "workspace_lost" }), ["a"])).toBe(false);
|
||||
});
|
||||
it("lets a reassigned owner execute on the parent only when declared", () => {
|
||||
const input = { issues: [{ id: "parent", companyId: "company", title: "t", status: "done" }], parentId: "parent", leadId: "lead",
|
||||
runs: [run("a", { agentId: "riley", contextSnapshot: { issueId: "parent" } })] };
|
||||
expect(failed(storyLifecycleChecks(input))).toContain("parent-owned-by-lead");
|
||||
expect(failed(storyLifecycleChecks({ ...input, parentAgentIds: ["lead", "riley"] }))).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("product rule: resume automatically or offer a Retry that works", () => {
|
||||
it("treats a failure as recovered only when a later run on the same task succeeds", () => {
|
||||
const lost = run("lost", { status: "failed", errorCode: "runner_lost", finishedAt: at(60) });
|
||||
expect(recoveredRunIds([lost, run("retry", { startedAt: at(70), finishedAt: at(120) })])).toEqual(["lost"]);
|
||||
expect(recoveredRunIds([lost])).toEqual([]);
|
||||
expect(recoveredRunIds([lost, run("other", { nativeIssueId: "child", startedAt: at(70) })])).toEqual([]);
|
||||
const lifecycle = { issues: [{ id: "parent", companyId: "company", title: "t", status: "done" }], parentId: "parent", leadId: "lead",
|
||||
runs: [lost, run("retry", { startedAt: at(70), finishedAt: at(120) })] };
|
||||
expect(failed(storyLifecycleChecks(lifecycle))).toEqual(["successful-runs"]);
|
||||
expect(failed(storyLifecycleChecks({ ...lifecycle, recoveredRuns: ["lost"] }))).toEqual([]);
|
||||
});
|
||||
it("passes automatic resumption", () => expect(failed(gradeResumptionRule({ observations: [], retryClicks: [] }))).toEqual([]));
|
||||
it("fails a send-a-message blocker, a dead Retry, a repeated Retry, and a stall", () => {
|
||||
const blocker = { at: at(1), issueId: "parent", blocker: { cause: "x", nextAction: "Saved work recovered. Send a message to continue.", canRetry: false } };
|
||||
expect(requiresUserMessage(blocker)).toBe(true);
|
||||
expect(requiresUserMessage({ ...blocker, blocker: { ...blocker.blocker, canRetry: true } })).toBe(false);
|
||||
expect(requiresUserMessage({ at: at(1), issueId: "parent", interruption: { state: "waiting_for_user", canRetry: false, userAction: null } })).toBe(true);
|
||||
expect(failed(gradeResumptionRule({ observations: [blocker], retryClicks: [] }))).toEqual(["resume.no-send-message-blocker"]);
|
||||
expect(failed(gradeResumptionRule({ observations: [], retryClicks: [{ at: at(1), issueId: "parent" }] }))).toEqual(["resume.retry-works-first-use"]);
|
||||
expect(failed(gradeResumptionRule({ observations: [], retryClicks: [
|
||||
{ at: at(1), issueId: "parent", startedRunId: "r1" }, { at: at(2), issueId: "parent", startedRunId: "r2" }] }))).toEqual(["resume.retry-works-first-use"]);
|
||||
expect(failed(gradeResumptionRule({ observations: [], retryClicks: [], stalled: "Stalled" }))).toEqual(["resume.no-silent-stall"]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("scenario graders", () => {
|
||||
it("detects concurrent executors and exact step files", () => {
|
||||
expect(concurrentExecutors([run("a", { finishedAt: at(60) }), run("b", { startedAt: at(61), finishedAt: at(90) })], "parent")).toEqual([]);
|
||||
expect(concurrentExecutors([run("a", { finishedAt: at(60) }), run("b", { startedAt: at(30) })], "parent")).toEqual([["a", "b"]]);
|
||||
expect(completeStepLines("step-1\nstep-2\n", 2)).toBe(true);
|
||||
expect(completeStepLines("step-1\nstep-1\nstep-2\n", 2)).toBe(false);
|
||||
expect(completeStepLines(undefined, 2)).toBe(false);
|
||||
});
|
||||
|
||||
it("interrupt-redirect: one bounded continuation, and Stop-settlement consumption", () => {
|
||||
const interrupted = run("old", { status: "cancelled", errorCode: "operator_interrupted", finishedAt: at(10) });
|
||||
const good = [interrupted, carrying("next", "comment-1", { startedAt: at(40), finishedAt: at(90) }),
|
||||
run("stopped", { status: "cancelled", errorCode: "cancelled", startedAt: at(100), finishedAt: at(130) }),
|
||||
carrying("settle", "comment-2", { startedAt: at(150), finishedAt: at(200) })];
|
||||
const input = { runs: good, comments: [], interruptedRunId: "old", interruptCommentId: "comment-1", interruptedAt: T0 + 10_000,
|
||||
stopSettle: { stoppedRunId: "stopped", commentId: "comment-2", sentAt: T0 + 120_000 } };
|
||||
expect(failed(gradeInterruptRedirect(input))).toEqual([]);
|
||||
expect(failed(gradeInterruptRedirect({ ...input, runs: [...good, carrying("dup", "comment-1", { startedAt: at(300) })] })))
|
||||
.toContain("interrupt.exactly-one-continuation");
|
||||
// Later turns keep the interrupt message in their prompt history; that is not a second delivery.
|
||||
expect(failed(gradeInterruptRedirect({ ...input, runs: [...good, run("later", { startedAt: at(300),
|
||||
runnerProfileJson: { nativeExecutionInput: { task: { prompt: "comment-1 comment-2 history" } } } })] }))).toEqual([]);
|
||||
expect(failed(gradeInterruptRedirect({ ...input, runs: [interrupted, carrying("late", "comment-1", { startedAt: at(250) }), ...good.slice(2)] })))
|
||||
.toEqual(["interrupt.continuation-bounded"]);
|
||||
expect(failed(gradeInterruptRedirect({ ...input, comments: [{ id: "nudge", createdAt: at(20) }] }))).toEqual(["interrupt.no-second-message"]);
|
||||
expect(failed(gradeInterruptRedirect({ ...input, runs: [{ ...interrupted, errorCode: "cancelled" }, ...good.slice(1)] })))
|
||||
.toEqual(["interrupt.operator-interrupt"]);
|
||||
});
|
||||
|
||||
it("reassign-midrun: prompt request, old run reassigned, no overlap, saved files continued", () => {
|
||||
const steps = Array.from({ length: 8 }, (_, index) => `step-${index + 1}`).join("\n");
|
||||
const input = { issueId: "parent", oldRunId: "old", newAgentId: "riley", steps: 8, stepFile: steps as string | undefined,
|
||||
request: { status: 200, latencyMs: 800, requestedAt: T0 + 30_000 },
|
||||
runs: [run("old", { status: "cancelled", errorCode: "issue_reassigned", finishedAt: at(32) }),
|
||||
run("new", { agentId: "riley", startedAt: at(40), finishedAt: at(200) })] };
|
||||
expect(failed(gradeReassignMidrun(input))).toEqual([]);
|
||||
expect(failed(gradeReassignMidrun({ ...input, request: { status: 409, latencyMs: 20_000, requestedAt: T0 } }))).toContain("reassign.request-prompt");
|
||||
expect(failed(gradeReassignMidrun({ ...input, runs: [run("old", { status: "cancelled", errorCode: "issue_reassigned", finishedAt: at(120) }), input.runs[1]!] })))
|
||||
.toEqual(["reassign.no-concurrent-executors"]);
|
||||
expect(failed(gradeReassignMidrun({ ...input, stepFile: `${steps}\nstep-3` }))).toEqual(["reassign.continued-from-saved-files"]);
|
||||
});
|
||||
|
||||
it("three-turn-revise: every turn succeeds and the table is complete", () => {
|
||||
const nonce = "N1";
|
||||
const table = `| Number | Item |\n| --- | --- |\n${["alpha", "beta", "gamma", "delta", "epsilon"].map((word, index) => `| ${index + 1} | ${word} ${nonce} |`).join("\n")}\n`;
|
||||
const turns = [run("t1"), run("t2"), run("t3")];
|
||||
expect(failed(gradeThreeTurnRevise({ turnRuns: turns, content: table, nonce }))).toEqual([]);
|
||||
expect(failed(gradeThreeTurnRevise({ turnRuns: [turns[0], turns[1], undefined], content: table, nonce }))).toEqual(["three-turn.turn-3-succeeded"]);
|
||||
expect(failed(gradeThreeTurnRevise({ turnRuns: turns, content: `${table}- alpha ${nonce}\n`, nonce }))).toEqual(["three-turn.final-table"]);
|
||||
expect(failed(gradeThreeTurnRevise({ turnRuns: turns, content: table.replace(`epsilon ${nonce}`, "missing"), nonce }))).toEqual(["three-turn.final-table"]);
|
||||
});
|
||||
|
||||
it("long-silent-command: completes after the command, Stop stays available, no hang", () => {
|
||||
const content = Array.from({ length: 13 }, (_, index) => `tick-${index + 1}`).join("\n");
|
||||
const samples = [{ at: at(60), canStop: true, source: "api" as const }, { at: at(61), canStop: true, source: "browser" as const },
|
||||
{ at: at(400), canStop: true, source: "api" as const }];
|
||||
const input = { run: run("silent", { startedAt: at(0), finishedAt: at(13 * 60 + 90) }), content, iterations: 13, iterationSeconds: 60,
|
||||
stopSamples: samples, reply: "SILENT_DONE_N1 13", nonce: "N1" };
|
||||
expect(failed(gradeLongSilentCommand(input))).toEqual([]);
|
||||
expect(failed(gradeLongSilentCommand({ ...input, run: run("silent", { status: "failed", errorCode: null, startedAt: at(0), finishedAt: at(13 * 60 + 60 * 30) }) })))
|
||||
.toEqual(expect.arrayContaining(["silent.completed", "silent.no-hang", "silent.named-failure"]));
|
||||
expect(failed(gradeLongSilentCommand({ ...input, run: run("silent", { startedAt: at(0), finishedAt: at(120) }) }))).toEqual(["silent.not-shortened"]);
|
||||
expect(failed(gradeLongSilentCommand({ ...input, stopSamples: [...samples, { at: at(500), canStop: false, source: "api" }] }))).toEqual(["silent.stop-available"]);
|
||||
});
|
||||
|
||||
it("tree-pause-release: no start during hold and exactly one resume per interrupted task", () => {
|
||||
const input = { childIds: ["a", "b"], holdCreatedAt: T0 + 100_000, releasedAt: T0 + 200_000,
|
||||
interruptedRuns: [{ id: "ra", issueId: "a" }, { id: "rb", issueId: "b" }],
|
||||
runs: [run("ra", { nativeIssueId: "a", status: "cancelled", finishedAt: at(101) }),
|
||||
run("rb", { nativeIssueId: "b", status: "cancelled", startedAt: null, finishedAt: at(101) }),
|
||||
run("ra2", { nativeIssueId: "a", startedAt: at(210), contextSnapshot: { wakeReason: "issue_tree_resumed" } }),
|
||||
run("rb2", { nativeIssueId: "b", startedAt: at(260), contextSnapshot: { wakeReason: "issue_tree_resumed" } })] };
|
||||
expect(failed(gradeTreePauseRelease(input))).toEqual([]);
|
||||
expect(failed(gradeTreePauseRelease({ ...input, runs: [...input.runs, run("early", { nativeIssueId: "b", startedAt: at(150) })] })))
|
||||
.toEqual(["tree.no-start-during-hold"]);
|
||||
expect(failed(gradeTreePauseRelease({ ...input, runs: [...input.runs, run("rb3", { nativeIssueId: "b", startedAt: at(300), contextSnapshot: { wakeReason: "issue_tree_resumed" } })] })))
|
||||
.toEqual(["tree.one-resume-per-interrupted"]);
|
||||
expect(failed(gradeTreePauseRelease({ ...input, runs: input.runs.slice(0, 3) }))).toEqual(["tree.one-resume-per-interrupted"]);
|
||||
expect(failed(gradeTreePauseRelease({ ...input, interruptedRuns: [] }))).toContain("tree.hold-interrupted-work");
|
||||
// A run that started while the hold request was in flight was interrupted by it, not started during it.
|
||||
expect(failed(gradeTreePauseRelease({ ...input, runs: input.runs.map(entry => entry.id === "ra" ? { ...entry, startedAt: at(100.5) } : entry) }))).toEqual([]);
|
||||
});
|
||||
|
||||
it("queue-steer: edit used, discard never delivered, steer consumed once in the active turn", () => {
|
||||
const active = run("active", { resultJson: { queuedSteeringAcknowledgements: { steer: { status: "acknowledged" } } } });
|
||||
const input = { activeRunId: "active", runs: [active, carrying("next", "edit", {}, "EDITED_N1 history: steer drop")],
|
||||
edited: { commentId: "edit", originalMarker: "ORIGINAL_N1", editedMarker: "EDITED_N1" },
|
||||
discarded: { commentId: "drop", marker: "DISCARDED_N1" }, steered: { commentId: "steer", marker: "STEERED_N1" },
|
||||
steerAttempts: [{ status: 409, code: "steering_temporarily_unavailable" }, { status: 200 }],
|
||||
files: { edited: "EDITED_N1", steered: "STEERED_N1" } };
|
||||
expect(failed(gradeQueueSteer(input))).toEqual([]);
|
||||
expect(failed(gradeQueueSteer({ ...input, runs: [active, carrying("next", "edit", {}, "EDITED_N1 DISCARDED_N1")] })))
|
||||
.toEqual(["queue.discard-never-delivered"]);
|
||||
expect(failed(gradeQueueSteer({ ...input, runs: [active, carrying("next", "edit", {}, "ORIGINAL_N1")], files: { ...input.files, edited: "ORIGINAL_N1" } })))
|
||||
.toEqual(["queue.edit-used"]);
|
||||
expect(failed(gradeQueueSteer({ ...input, runs: [...input.runs, carrying("replay", "steer", { startedAt: at(200) }, "EDITED_N1")] })))
|
||||
.toEqual(["queue.steer-consumed-in-active-run"]);
|
||||
expect(failed(gradeQueueSteer({ ...input, steerAttempts: [{ status: 409, code: "queued_comment_stale_target" }, { status: 200 }] })))
|
||||
.toEqual(["queue.steer-retry-bounded"]);
|
||||
expect(failed(gradeQueueSteer({ ...input, runs: [...input.runs, carrying("again", "edit", { startedAt: at(300) }, "EDITED_N1")] })))
|
||||
.toEqual(["queue.remaining-handled-once"]);
|
||||
});
|
||||
|
||||
it("queued-across-restart: bounded automatic dispatch with a named wait", () => {
|
||||
const input = { commentId: "c1", restartStartedAt: T0 + 60_000, dispatchedBeforeRestart: false,
|
||||
runs: [carrying("next", "c1", { startedAt: at(120), finishedAt: at(200) })],
|
||||
waitObservations: [{ at: at(65), phase: "queued", label: "Waiting for server restart", cause: "server_restarting" }] };
|
||||
expect(failed(gradeQueuedAcrossRestart(input))).toEqual([]);
|
||||
expect(failed(gradeQueuedAcrossRestart({ ...input, waitObservations: [{ at: at(65), phase: "queued", label: "Queued", cause: null }] })))
|
||||
.toEqual(["restart.named-wait"]);
|
||||
expect(failed(gradeQueuedAcrossRestart({ ...input, runs: [] }))).toEqual(["restart.dispatched-bounded", "restart.exactly-once"]);
|
||||
expect(failed(gradeQueuedAcrossRestart({ ...input, runs: [carrying("next", "c1", { startedAt: at(60 + 7 * 60) })] })))
|
||||
.toEqual(["restart.dispatched-bounded"]);
|
||||
expect(failed(gradeQueuedAcrossRestart({ ...input, runs: [carrying("early", "c1", { startedAt: at(30) })] })))
|
||||
.toEqual(["restart.boundary-exercised"]);
|
||||
expect(failed(gradeQueuedAcrossRestart({ ...input, dispatchedBeforeRestart: true }))).toEqual(["restart.boundary-exercised"]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,326 @@
|
||||
import { RELIABILITY_BOUNDS } from "./everyday-reliability-cases.js";
|
||||
import { isStoryWorkspaceDeferral, type StoryCheck, type StoryComment, type StoryRun } from "./everyday-observations.js";
|
||||
|
||||
/**
|
||||
* Pure graders for the staging-soak reliability scenarios. Product rule:
|
||||
* work interrupted by a restart, Stop, reassignment or lost runner resumes
|
||||
* automatically, or offers a Retry that works on first use. "Send a message
|
||||
* to continue", a dead Retry, a silent stall and duplicate execution all fail.
|
||||
*/
|
||||
|
||||
const check = (id: string, passed: boolean, detail: string): StoryCheck => ({ id, passed, detail });
|
||||
const ms = (value: string | null | undefined) => Date.parse(value ?? "");
|
||||
/** Clock granularity between separately written run timestamps. */
|
||||
const OVERLAP_TOLERANCE_MS = 2_000;
|
||||
|
||||
export function runIssueId(run: StoryRun): string | undefined {
|
||||
const context = run.contextSnapshot ?? {};
|
||||
return run.nativeIssueId ?? (typeof context.issueId === "string" ? context.issueId
|
||||
: typeof context.taskId === "string" ? context.taskId : undefined);
|
||||
}
|
||||
|
||||
/** Serialized native execution input; a delivered message appears here by ID and body. */
|
||||
export function runNativeInput(run: StoryRun): string {
|
||||
return JSON.stringify(run.runnerProfileJson?.nativeExecutionInput ?? null);
|
||||
}
|
||||
|
||||
/** Runs that reached provider execution (workspace deferrals never did). */
|
||||
export function executedRuns(runs: readonly StoryRun[]): StoryRun[] {
|
||||
return runs.filter(run => !isStoryWorkspaceDeferral(run) && Boolean(run.startedAt));
|
||||
}
|
||||
|
||||
/**
|
||||
* Comments this run was woken to deliver. The saved prompt also holds the whole
|
||||
* conversation history, so it cannot attribute delivery; the wake context can.
|
||||
*/
|
||||
export function runWakeCommentIds(run: StoryRun): string[] {
|
||||
const context = run.contextSnapshot ?? {};
|
||||
const ids = Array.isArray(context.wakeCommentIds) ? context.wakeCommentIds : [];
|
||||
return [...new Set([...ids, context.wakeCommentId, context.commentId].filter((id): id is string => typeof id === "string"))];
|
||||
}
|
||||
|
||||
/** Executed runs woken to deliver this comment (a Retry of a failed delivery inherits it). */
|
||||
export function runsCarrying(runs: readonly StoryRun[], commentId: string): StoryRun[] {
|
||||
return executedRuns(runs).filter(run => runWakeCommentIds(run).includes(commentId));
|
||||
}
|
||||
|
||||
/** Successful deliveries of one comment; more than one is duplicate execution. */
|
||||
export const successfulDeliveries = (runs: readonly StoryRun[], commentId: string) =>
|
||||
runsCarrying(runs, commentId).filter(run => run.status === "succeeded");
|
||||
|
||||
/**
|
||||
* Failed or interrupted runs whose task a later run completed successfully: the
|
||||
* product rule's "resumed automatically or through a working Retry". The
|
||||
* resumption checks separately fail send-a-message blockers and dead Retries.
|
||||
*/
|
||||
export function recoveredRunIds(runs: readonly StoryRun[]): string[] {
|
||||
return executedRuns(runs).filter(run => ["failed", "timed_out", "cancelled", "interrupted"].includes(run.status)
|
||||
&& runs.some(later => later.status === "succeeded" && runIssueId(later) === runIssueId(run)
|
||||
&& ms(later.startedAt) >= ms(run.finishedAt))).map(run => run.id);
|
||||
}
|
||||
|
||||
const earliest = (runs: readonly StoryRun[]) => [...runs].sort((a, b) => ms(a.startedAt) - ms(b.startedAt))[0];
|
||||
|
||||
/** Pairs of runs that executed on the same task at the same time. */
|
||||
export function concurrentExecutors(runs: readonly StoryRun[], issueId: string, now = Date.now()): Array<[string, string]> {
|
||||
const spans = executedRuns(runs).filter(run => runIssueId(run) === issueId)
|
||||
.map(run => ({ id: run.id, start: ms(run.startedAt), end: Number.isFinite(ms(run.finishedAt)) ? ms(run.finishedAt) : now }))
|
||||
.sort((left, right) => left.start - right.start);
|
||||
const pairs: Array<[string, string]> = [];
|
||||
for (let index = 0; index < spans.length; index += 1)
|
||||
for (let other = index + 1; other < spans.length; other += 1)
|
||||
if (spans[other]!.start + OVERLAP_TOLERANCE_MS < spans[index]!.end) pairs.push([spans[index]!.id, spans[other]!.id]);
|
||||
return pairs;
|
||||
}
|
||||
|
||||
/** Exact "step-1".."step-N" lines, each once and in order. */
|
||||
export function completeStepLines(content: string | undefined, steps: number, prefix = "step"): boolean {
|
||||
if (content === undefined) return false;
|
||||
const lines = content.split(/\r?\n/).map(line => line.trim()).filter(Boolean);
|
||||
return lines.length === steps && lines.every((line, index) => line === `${prefix}-${index + 1}`);
|
||||
}
|
||||
|
||||
export interface ResumptionObservation {
|
||||
at: string;
|
||||
issueId: string;
|
||||
blocker?: { cause?: string | null; nextAction?: string | null; canRetry?: boolean; canContinue?: boolean } | null;
|
||||
interruption?: { state?: string; canRetry?: boolean; userAction?: string | null; responsibility?: string } | null;
|
||||
execution?: { phase?: string; label?: string; cause?: string | null } | null;
|
||||
}
|
||||
export interface RetryClick { at: string; issueId: string; startedRunId?: string }
|
||||
|
||||
/** The only way forward is another user message: the S47/S49 failure mode. */
|
||||
export function requiresUserMessage(observation: ResumptionObservation): boolean {
|
||||
const blocker = observation.blocker;
|
||||
if (blocker && !blocker.canRetry && /send (?:a |another |a new )?(?:new )?(?:chat )?message/i.test(blocker.nextAction ?? ""))
|
||||
return true;
|
||||
const interruption = observation.interruption;
|
||||
return Boolean(interruption && interruption.state === "waiting_for_user" && !interruption.canRetry && !interruption.userAction);
|
||||
}
|
||||
|
||||
export function gradeResumptionRule(input: {
|
||||
observations: readonly ResumptionObservation[];
|
||||
retryClicks: readonly RetryClick[];
|
||||
stalled?: string;
|
||||
}): StoryCheck[] {
|
||||
const blocked = input.observations.find(requiresUserMessage);
|
||||
const dead = input.retryClicks.filter(click => !click.startedRunId);
|
||||
const repeated = new Set(input.retryClicks.map(click => click.issueId)).size !== input.retryClicks.length;
|
||||
return [
|
||||
check("resume.no-send-message-blocker", !blocked,
|
||||
blocked ? `Blocked at ${blocked.at}: ${blocked.blocker?.nextAction ?? blocked.interruption?.state}` : "Interrupted work never required another user message to continue."),
|
||||
check("resume.retry-works-first-use", dead.length === 0 && !repeated,
|
||||
input.retryClicks.length === 0 ? "No Retry was needed; work resumed automatically."
|
||||
: dead.length ? `Retry clicked without starting a run at ${dead.map(click => click.at).join(", ")}.`
|
||||
: repeated ? "Retry had to be used more than once for the same task." : "Each offered Retry started work on first use."),
|
||||
check("resume.no-silent-stall", !input.stalled, input.stalled ?? "Work resumed or finished within the scenario bound."),
|
||||
];
|
||||
}
|
||||
|
||||
const userCommentsBetween = (comments: readonly StoryComment[], from: number, to: number, except: readonly string[]) =>
|
||||
comments.filter(comment => !comment.authorAgentId && !except.includes(comment.id ?? "")
|
||||
&& ms(comment.createdAt) > from && ms(comment.createdAt) < to);
|
||||
|
||||
export function gradeInterruptRedirect(input: {
|
||||
runs: readonly StoryRun[];
|
||||
comments: readonly StoryComment[];
|
||||
interruptedRunId: string;
|
||||
interruptCommentId: string;
|
||||
interruptedAt: number;
|
||||
stopSettle?: { stoppedRunId: string; commentId: string; sentAt: number };
|
||||
}): StoryCheck[] {
|
||||
const interrupted = input.runs.find(run => run.id === input.interruptedRunId);
|
||||
const carriers = successfulDeliveries(input.runs, input.interruptCommentId);
|
||||
const continuation = earliest(runsCarrying(input.runs, input.interruptCommentId));
|
||||
const delay = ms(continuation?.startedAt) - input.interruptedAt;
|
||||
const extra = continuation ? userCommentsBetween(input.comments, input.interruptedAt, ms(continuation.startedAt), [input.interruptCommentId]) : [];
|
||||
const checks = [
|
||||
check("interrupt.operator-interrupt", interrupted?.status === "cancelled"
|
||||
&& (interrupted.errorCode === "operator_interrupted" || interrupted.resultJson?.operatorInterrupted === true),
|
||||
`The interrupted run ends as an operator interrupt (observed ${interrupted?.status ?? "missing"}/${interrupted?.errorCode ?? "none"}).`),
|
||||
check("interrupt.exactly-one-continuation", carriers.length === 1,
|
||||
`Exactly one successful continuation delivers the interrupt message (observed ${carriers.length}).`),
|
||||
check("interrupt.continuation-bounded", Number.isFinite(delay) && delay <= RELIABILITY_BOUNDS.continuationStartMs,
|
||||
`The continuation starts within ${RELIABILITY_BOUNDS.continuationStartMs / 1000}s of the interrupt (observed ${Number.isFinite(delay) ? Math.round(delay / 1000) + "s" : "never"}).`),
|
||||
check("interrupt.no-second-message", extra.length === 0, "No second user message was needed before the continuation started."),
|
||||
];
|
||||
if (input.stopSettle) {
|
||||
const stopped = input.runs.find(run => run.id === input.stopSettle!.stoppedRunId);
|
||||
const settleCarriers = successfulDeliveries(input.runs, input.stopSettle.commentId);
|
||||
const settleDelay = ms(earliest(runsCarrying(input.runs, input.stopSettle.commentId))?.startedAt) - input.stopSettle.sentAt;
|
||||
checks.push(
|
||||
check("stop-settle.stopped", stopped?.status === "cancelled", "The stopped run is terminal as cancelled."),
|
||||
check("stop-settle.consumed-once", settleCarriers.length === 1,
|
||||
`A message sent while Stop was settling is consumed by exactly one successful run (observed ${settleCarriers.length}).`),
|
||||
check("stop-settle.bounded", Number.isFinite(settleDelay) && settleDelay <= RELIABILITY_BOUNDS.continuationStartMs,
|
||||
`The workspace_busy retry dispatches within ${RELIABILITY_BOUNDS.continuationStartMs / 1000}s (observed ${Number.isFinite(settleDelay) ? Math.round(settleDelay / 1000) + "s" : "never"}).`),
|
||||
);
|
||||
}
|
||||
return checks;
|
||||
}
|
||||
|
||||
export function gradeReassignMidrun(input: {
|
||||
runs: readonly StoryRun[];
|
||||
issueId: string;
|
||||
oldRunId: string;
|
||||
newAgentId: string;
|
||||
request: { status: number; latencyMs: number; requestedAt: number };
|
||||
stepFile?: string;
|
||||
steps: number;
|
||||
}): StoryCheck[] {
|
||||
const old = input.runs.find(run => run.id === input.oldRunId);
|
||||
// A failed first attempt followed by a working Retry is allowed: require any success.
|
||||
const successor = executedRuns(input.runs).find(run => run.agentId === input.newAgentId && runIssueId(run) === input.issueId
|
||||
&& ms(run.startedAt) >= input.request.requestedAt && run.status === "succeeded");
|
||||
const overlaps = concurrentExecutors(input.runs, input.issueId);
|
||||
return [
|
||||
check("reassign.request-prompt", input.request.status >= 200 && input.request.status < 300
|
||||
&& input.request.latencyMs <= RELIABILITY_BOUNDS.reassignRequestMs,
|
||||
`Reassignment returns ${input.request.status} in ${input.request.latencyMs}ms (bound ${RELIABILITY_BOUNDS.reassignRequestMs}ms, no 409).`),
|
||||
check("reassign.old-run-ended", old?.status === "cancelled" && old.errorCode === "issue_reassigned",
|
||||
`The original run ends issue_reassigned (observed ${old?.status ?? "missing"}/${old?.errorCode ?? "none"}).`),
|
||||
check("reassign.new-assignee-finished", successor?.status === "succeeded", "The new assignee's run starts after the request and succeeds."),
|
||||
check("reassign.no-concurrent-executors", overlaps.length === 0,
|
||||
overlaps.length ? `Concurrent executions: ${overlaps.map(pair => pair.join("+")).join(", ")}.` : "Never two executions on the task at once."),
|
||||
check("reassign.continued-from-saved-files", completeStepLines(input.stepFile, input.steps),
|
||||
`The saved step file holds step-1..step-${input.steps} exactly once, so work continued from saved files.`),
|
||||
];
|
||||
}
|
||||
|
||||
export function gradeThreeTurnRevise(input: {
|
||||
turnRuns: ReadonlyArray<StoryRun | undefined>;
|
||||
content?: string;
|
||||
nonce: string;
|
||||
}): StoryCheck[] {
|
||||
const items = ["alpha", "beta", "gamma", "delta", "epsilon"].map(word => `${word} ${input.nonce}`);
|
||||
const rows = (input.content ?? "").split(/\r?\n/).map(line => line.trim()).filter(line => line.startsWith("|"));
|
||||
const header = rows[0]?.toLowerCase() ?? "";
|
||||
const body = rows.slice(2);
|
||||
const table = /number/.test(header) && /item/.test(header) && /^\|[\s:|-]+\|$/.test(rows[1] ?? "")
|
||||
&& body.length === items.length && body.every((row, index) => row.includes(items[index]!));
|
||||
const bullets = /^\s*[-*]\s+/m.test(input.content ?? "");
|
||||
return [
|
||||
...input.turnRuns.map((run, index) => check(`three-turn.turn-${index + 1}-succeeded`, run?.status === "succeeded",
|
||||
`Turn ${index + 1} completes in a successful run without a reconciliation stop (observed ${run?.status ?? "no run"}).`)),
|
||||
check("three-turn.final-table", table && !bullets, "items.md is a Number/Item table with all five items in order and no bullet list."),
|
||||
];
|
||||
}
|
||||
|
||||
export function gradeLongSilentCommand(input: {
|
||||
run?: StoryRun;
|
||||
content?: string;
|
||||
iterations: number;
|
||||
iterationSeconds: number;
|
||||
stopSamples: ReadonlyArray<{ at: string; canStop: boolean; source: "api" | "browser" }>;
|
||||
reply?: string;
|
||||
nonce: string;
|
||||
}): StoryCheck[] {
|
||||
const commandMs = input.iterations * input.iterationSeconds * 1000;
|
||||
const elapsed = ms(input.run?.finishedAt) - ms(input.run?.startedAt);
|
||||
const named = input.run?.status === "succeeded" || Boolean(input.run?.errorCode);
|
||||
return [
|
||||
check("silent.completed", input.run?.status === "succeeded",
|
||||
`The run completes after the silent command (observed ${input.run?.status ?? "no run"}${input.run?.errorCode ? `/${input.run.errorCode}` : ""}).`),
|
||||
check("silent.not-shortened", Number.isFinite(elapsed) && elapsed >= commandMs * 0.95,
|
||||
`The run lasted at least the ${input.iterations}-minute command (observed ${Number.isFinite(elapsed) ? Math.round(elapsed / 1000) + "s" : "unknown"}).`),
|
||||
check("silent.no-hang", Number.isFinite(elapsed) && elapsed <= commandMs + RELIABILITY_BOUNDS.silentCompletionAfterCommandMs,
|
||||
`The run is terminal within ${RELIABILITY_BOUNDS.silentCompletionAfterCommandMs / 60_000} minutes after the command ends.`),
|
||||
check("silent.named-failure", named, named ? "Any failure carries a named cause." : "The run ended without a named cause."),
|
||||
check("silent.file-complete", completeStepLines(input.content, input.iterations, "tick"),
|
||||
`silent.txt holds tick-1..tick-${input.iterations} exactly once.`),
|
||||
check("silent.stop-available", input.stopSamples.length >= 3 && input.stopSamples.every(sample => sample.canStop)
|
||||
&& input.stopSamples.some(sample => sample.source === "browser"),
|
||||
`Stop stays available throughout the silent interval (${input.stopSamples.filter(sample => sample.canStop).length}/${input.stopSamples.length} samples).`),
|
||||
check("silent.reported", new RegExp(`SILENT_DONE_${input.nonce}\\s+${input.iterations}\\b`).test(input.reply ?? ""),
|
||||
"The agent reports the completed line count after the command."),
|
||||
];
|
||||
}
|
||||
|
||||
export function gradeTreePauseRelease(input: {
|
||||
runs: readonly StoryRun[];
|
||||
childIds: readonly string[];
|
||||
holdCreatedAt: number;
|
||||
releasedAt: number;
|
||||
interruptedRuns: ReadonlyArray<{ id: string; issueId: string }>;
|
||||
}): StoryCheck[] {
|
||||
const executed = executedRuns(input.runs);
|
||||
// A run that started while the hold request was in flight is one the hold interrupted.
|
||||
const interruptedIds = new Set(input.interruptedRuns.map(run => run.id));
|
||||
const startedDuringHold = executed.filter(run => input.childIds.includes(runIssueId(run) ?? "") && !interruptedIds.has(run.id)
|
||||
&& ms(run.startedAt) > input.holdCreatedAt && ms(run.startedAt) < input.releasedAt);
|
||||
const interruptedIssues = [...new Set(input.interruptedRuns.map(run => run.issueId))];
|
||||
const resumed = (issueId: string) => input.runs.filter(run => runIssueId(run) === issueId
|
||||
&& run.contextSnapshot?.wakeReason === "issue_tree_resumed" && !isStoryWorkspaceDeferral(run));
|
||||
const resumeCounts = interruptedIssues.map(issueId => ({ issueId, count: resumed(issueId).length }));
|
||||
const stray = input.runs.filter(run => run.contextSnapshot?.wakeReason === "issue_tree_resumed"
|
||||
&& !interruptedIssues.includes(runIssueId(run) ?? ""));
|
||||
return [
|
||||
check("tree.hold-interrupted-work", input.interruptedRuns.length > 0,
|
||||
input.interruptedRuns.length ? `The hold interrupted ${input.interruptedRuns.length} run(s).` : "Boundary unexercised: no child work was active or queued when the hold was created."),
|
||||
check("tree.no-start-during-hold", startedDuringHold.length === 0,
|
||||
startedDuringHold.length ? `Child runs started during the hold: ${startedDuringHold.map(run => run.id).join(", ")}.` : "No child run started while the tree was paused."),
|
||||
check("tree.one-resume-per-interrupted", resumeCounts.every(entry => entry.count === 1) && stray.length === 0,
|
||||
`Each interrupted task gets exactly one issue_tree_resumed run (observed ${resumeCounts.map(entry => entry.count).join(", ") || "none"}; stray ${stray.length}).`),
|
||||
];
|
||||
}
|
||||
|
||||
export function gradeQueueSteer(input: {
|
||||
runs: readonly StoryRun[];
|
||||
activeRunId: string;
|
||||
edited: { commentId: string; originalMarker: string; editedMarker: string };
|
||||
discarded: { commentId: string; marker: string };
|
||||
steered: { commentId: string; marker: string };
|
||||
steerAttempts: ReadonlyArray<{ status: number; code?: string }>;
|
||||
files: { edited?: string; discarded?: string; steered?: string };
|
||||
}): StoryCheck[] {
|
||||
const executed = executedRuns(input.runs);
|
||||
const anyInput = (needle: string) => executed.some(run => runNativeInput(run).includes(needle));
|
||||
const active = input.runs.find(run => run.id === input.activeRunId);
|
||||
const acks = (active?.resultJson?.queuedSteeringAcknowledgements ?? {}) as Record<string, { status?: string }>;
|
||||
const editedCarriers = successfulDeliveries(input.runs, input.edited.commentId);
|
||||
const steerReplays = runsCarrying(input.runs, input.steered.commentId);
|
||||
const finalAttempt = input.steerAttempts.at(-1);
|
||||
const retriesValid = input.steerAttempts.slice(0, -1).every(attempt => attempt.code === "steering_temporarily_unavailable");
|
||||
return [
|
||||
check("queue.edit-used", anyInput(input.edited.editedMarker) && !anyInput(input.edited.originalMarker)
|
||||
&& (input.files.edited ?? "").includes(input.edited.editedMarker),
|
||||
"The edited text reached the agent; the original text never did."),
|
||||
check("queue.remaining-handled-once", editedCarriers.length === 1,
|
||||
`The remaining queued message is handled by exactly one successful run (observed ${editedCarriers.length}).`),
|
||||
// History may retain a deleted comment's ID, but never its text or a delivery wake.
|
||||
check("queue.discard-never-delivered", !anyInput(input.discarded.marker) && runsCarrying(input.runs, input.discarded.commentId).length === 0
|
||||
&& !acks[input.discarded.commentId] && input.files.discarded === undefined,
|
||||
"The discarded message never reached the agent."),
|
||||
check("queue.steer-consumed-in-active-run", acks[input.steered.commentId]?.status === "acknowledged" && steerReplays.length === 0
|
||||
&& (input.files.steered ?? "").includes(input.steered.marker),
|
||||
"The steered message is acknowledged by the active turn, acted on there, and never replayed as a new run."),
|
||||
check("queue.steer-retry-bounded", Boolean(finalAttempt && finalAttempt.status >= 200 && finalAttempt.status < 300) && retriesValid,
|
||||
`Steering succeeded after ${input.steerAttempts.length} attempt(s); only steering_temporarily_unavailable was retried.`),
|
||||
];
|
||||
}
|
||||
|
||||
export function gradeQueuedAcrossRestart(input: {
|
||||
runs: readonly StoryRun[];
|
||||
commentId: string;
|
||||
/** When the controller stop was requested; the old controller cannot dispatch after it. */
|
||||
restartStartedAt: number;
|
||||
/** Whether a delivery already existed when the restart was requested. */
|
||||
dispatchedBeforeRestart: boolean;
|
||||
waitObservations: ReadonlyArray<{ at: string; phase?: string; label?: string; cause?: string | null }>;
|
||||
}): StoryCheck[] {
|
||||
const carriers = successfulDeliveries(input.runs, input.commentId);
|
||||
const start = ms(earliest(runsCarrying(input.runs, input.commentId))?.startedAt);
|
||||
const bare = input.waitObservations.filter(observation => observation.phase === "queued"
|
||||
&& (!observation.cause || observation.label === "Queued"));
|
||||
return [
|
||||
check("restart.boundary-exercised", !input.dispatchedBeforeRestart && (!Number.isFinite(start) || start >= input.restartStartedAt),
|
||||
"The queued follow-up had not dispatched when the controller restart was requested."),
|
||||
check("restart.dispatched-bounded", Number.isFinite(start) && start - input.restartStartedAt <= RELIABILITY_BOUNDS.restartDispatchMs,
|
||||
`The follow-up dispatches automatically within ${RELIABILITY_BOUNDS.restartDispatchMs / 60_000} minutes of the restart (observed ${Number.isFinite(start) ? Math.round((start - input.restartStartedAt) / 1000) + "s" : "never"}).`),
|
||||
check("restart.exactly-once", carriers.length === 1,
|
||||
`The follow-up is consumed by exactly one successful run (observed ${carriers.length}).`),
|
||||
check("restart.named-wait", bare.length === 0,
|
||||
bare.length ? `Bare Queued without a cause at ${bare.map(observation => observation.at).join(", ")}.`
|
||||
: `Every observed wait names its cause (${input.waitObservations.length} observations).`),
|
||||
];
|
||||
}
|
||||
@@ -268,9 +268,10 @@ describe("manual everyday workflow catalog", () => {
|
||||
);
|
||||
});
|
||||
it("does not claim unsupported remote crash/hiring coverage", () => {
|
||||
// The explicit reliability group is covered by everyday-reliability.test.ts.
|
||||
const remote = runnerMatrix.filter(
|
||||
(e) =>
|
||||
e.suite.id === "everyday-workflows" && e.environment.id === "daytona",
|
||||
e.suite.id === "everyday-workflows" && e.environment.id === "daytona" && !e.task.explicitGroup,
|
||||
);
|
||||
expect(remote).toHaveLength(8);
|
||||
expect(new Set(remote.map((e) => e.task.id))).toEqual(
|
||||
|
||||
@@ -9,7 +9,9 @@ import { afterEach, beforeEach, expect, it, vi } from "vitest";
|
||||
import { runnerMatrix } from "./catalog.js";
|
||||
import { assertInstalledCliSelection, installedCliKeys, probeInstalledCodexStartup, verifyInstalledCli } from "./installed-cli.js";
|
||||
import { buildPaperclipServerEnvironment, buildRunnerE2EProcessEnvironment } from "./harness-env.js";
|
||||
const cells = runnerMatrix.filter(e => !isRollingWebDeployment(e) && !isLegacyWorkspaceMigration(e) && e.profile.id === "runner-acpx-pi");
|
||||
const cells = runnerMatrix.filter(e => !isRollingWebDeployment(e) && !isLegacyWorkspaceMigration(e) && e.profile.id === "runner-acpx-pi"
|
||||
// The explicit reliability facade cell is not an installed-CLI Pi qualification case.
|
||||
&& e.task.explicitGroup === undefined);
|
||||
const roots: string[] = [];
|
||||
// pnpm's source test wrapper sets NODE_PATH; the disposable public-install fixture must start closed.
|
||||
beforeEach(() => { vi.stubEnv("NODE_PATH", undefined); vi.stubEnv("NODE_OPTIONS", undefined); });
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { isEvalRemoteOrigin } from "./eval-remote-gateway.js";
|
||||
import { isEverydayReliabilityCase } from "./everyday-reliability-cases.js";
|
||||
import type { AiConnectionBinding } from "../../packages/shared/src/ai-connections.js";
|
||||
import { verifyInstalledDaytonaPlugin } from "./installed-daytona-plugin.js";
|
||||
import { NATIVE_COMPLETION_BUDGET_CENTS } from "./native-completion-defaults.js";
|
||||
@@ -221,7 +222,8 @@ export async function setupLiveFixtures(input: {
|
||||
budgetMonthlyCents: execution.suite.id === CHAT_FEATURE_SUITE ? CHAT_FEATURE_BUDGET_CENTS
|
||||
: receiptSmoke ? CASE_RECEIPT_BUDGET_CENTS
|
||||
: ["native-completion", "native-instruction-consolidation", "native-connection-guidance"].includes(execution.suite.id)
|
||||
|| (execution.suite.id === "everyday-workflows" && ["hire-reuse", "delegate-feedback"].includes(execution.task.id)) ? NATIVE_COMPLETION_BUDGET_CENTS
|
||||
|| (execution.suite.id === "everyday-workflows" && (["hire-reuse", "delegate-feedback"].includes(execution.task.id)
|
||||
|| isEverydayReliabilityCase(execution.task.id))) ? NATIVE_COMPLETION_BUDGET_CENTS
|
||||
: execution.suite.id === "task-titles" ? TASK_TITLE_BUDGET_CENTS
|
||||
: ["cursor-cloud", "stock-harness", "task-workspaces", "case-handoffs", "feedback-persistence", "daytona-restart-continuity", "remote-workspace-storage", "rolling-web-deployment"].includes(execution.suite.id) ? 1_000 : 0,
|
||||
});
|
||||
@@ -419,7 +421,8 @@ export async function setupLiveFixtures(input: {
|
||||
}
|
||||
}
|
||||
if (["stock-harness", "native-connection-guidance", "feedback-persistence", "daytona-restart-continuity", "remote-workspace-storage", "rolling-web-deployment"].includes(execution.suite.id)
|
||||
|| (execution.suite.id === "everyday-workflows" && ["hire-reuse", "delegate-feedback"].includes(execution.task.id))) {
|
||||
|| (execution.suite.id === "everyday-workflows" && (["hire-reuse", "delegate-feedback"].includes(execution.task.id)
|
||||
|| isEverydayReliabilityCase(execution.task.id)))) {
|
||||
agent.budgetMonthlyCents = 1_000;
|
||||
}
|
||||
if (receiptSmoke) {
|
||||
|
||||
@@ -195,6 +195,12 @@ export function selectRunnerExecutions(
|
||||
const selected = matrix.filter((execution) => {
|
||||
if (options.ids.length > 0) return options.ids.includes(execution.id);
|
||||
if (execution.suite.manualOnly && !options.suites.includes(execution.suite.id) && !options.list) return false;
|
||||
const unfilteredList = options.list && options.groups.length + options.suites.length + options.profiles.length
|
||||
+ options.environments.length + options.cases.length === 0;
|
||||
const explicitGroup = execution.task.explicitGroup;
|
||||
// Naming any of the task's own groups (for example `slow`) also opts in.
|
||||
if (explicitGroup && !unfilteredList && !options.groups.some(group => group === explicitGroup || execution.task.groups.includes(group))
|
||||
&& !options.cases.includes(execution.task.id)) return false;
|
||||
if (
|
||||
options.all ||
|
||||
(options.list &&
|
||||
|
||||
@@ -160,6 +160,12 @@ export interface RunnerTaskFixture {
|
||||
id: string;
|
||||
label: string;
|
||||
groups: readonly string[];
|
||||
/**
|
||||
* Opt-in group. When set, the task is selected only by naming this group
|
||||
* (`--group`), the case (`--case`), or an exact execution ID. Suite, profile
|
||||
* and environment selectors alone never select it.
|
||||
*/
|
||||
explicitGroup?: string;
|
||||
workMode: RunnerTaskWorkMode;
|
||||
flow: RunnerTaskFlow;
|
||||
expectedRunCount: number;
|
||||
|
||||
Reference in new issue
Block a user