mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
test(restart-chaos): measure TTFT in a real repository workspace, after an idle restart, and on Daytona (#20047)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos TTFT driver measures time to first token, but only with a tiny fixture workspace and only locally > - Dotta asked whether speed is good with a workspace like the paperclip repository itself, and that had never been measured > - This pull request lets the driver run in a real repository workspace, measure the first follow-up after an idle restart, run on Daytona (per-turn or warm reusable sandboxes), and report each run's workspace operations > - The benefit is a repeatable large-workspace speed measurement; its first run found a full workspace re-upload on every remote turn ## Linked Issues or Issue Description **What happened?** The TTFT driver seeded only a tiny fixture repository, created tasks outside any project, ran only locally, and could not show workspace operation durations. So large-workspace speed (worktree creation, provision, remote upload) was never measured. **Steps to reproduce** 1. Run `ttft-baseline.ts` with any options. 2. Every task runs without a project workspace and on the local host. 3. Nothing in the report shows worktree, provision or sync cost. **Expected behavior** The driver can measure a real repository workspace, locally and on Daytona, and names the workspace operations each run waited on. ## What Changed - `--workspace-repo <path>`: the seed uses that repository as the project's primary workspace (git_worktree isolation), and tasks are created in the project. - A `postfault` phase: K1 while idle, then a follow-up on each task; reported in its own table. - `--env daytona`, `--daytona-image`, `--daytona-cpu`, `--daytona-warm` (the reusable warm Daytona fixture). The Daytona controller preparation moved to `daytona-controller.ts`, and every sandbox is cleaned up at the end; incomplete cleanup fails the run. - Each run's workspace operations (`/api/heartbeat-runs/:id/workspace-operations`) are saved, and printed ranked by duration. ## Verification - `tsc -p tests/restart-chaos`: no errors in harness files. - Used for the large-workspace measurement, a fresh paperclip clone on local and Daytona (per-turn and warm). First turn / follow-up: local 31–34 s / 7 s; Daytona per-turn 58–63 s / 30 s; Daytona warm 45–51 s / 8–10 s. Every Daytona sandbox was cleaned up (environment deleted, none remaining). ## Risks - Test harness only; no product code. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, test execution, Daytona measurements). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
ce1ee9e89f
commit
10bc6431da
10 files changed
+368
-29
No files matched your search
@@ -9,6 +9,7 @@ import { parseTimeFaultOptions } from "./time-faults-config.js";
|
||||
import { parseStorageTrigger } from "./storage-trigger.js";
|
||||
import { parseNativeTrigger, parseNativeTriggerOutage } from "./native-trigger.js";
|
||||
import { parseRepeatedRestartOptions } from "./repeated-restarts.js";
|
||||
import { daytonaRetainsSandboxes } from "./ttft-baseline-stats.js";
|
||||
import { PROVIDER_FAULT_KINDS, PROVIDER_FAULT_WORKLOADS, isProviderFault, parseProviderDelays, parseProviderMessages, type ProviderFaultKind } from "./provider-faults.js";
|
||||
import { PROVIDER_FAULT_ADAPTERS, parseProviderFaultRestart } from "./acpx-provider-faults.js";
|
||||
import {
|
||||
@@ -99,6 +100,9 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
|
||||
"daytona-image": { type: "string" },
|
||||
"daytona-cpu": { type: "string" },
|
||||
"daytona-reuse-lease": { type: "boolean" },
|
||||
// The reusable warm Daytona fixture, or an environment that inherits the lifecycle (a user's default).
|
||||
"daytona-warm": { type: "boolean" },
|
||||
"daytona-inherit": { type: "boolean" },
|
||||
keep: { type: "boolean" },
|
||||
"rc06-masked": { type: "boolean" },
|
||||
"storm-minutes": { type: "string" },
|
||||
@@ -131,7 +135,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
|
||||
const pollMs = integer(values["poll-ms"], "poll-ms", 3_000, 2_000, 5_000);
|
||||
const budgetCents = integer(values["budget-cents"], "budget-cents", 3_000, 1, 100_000);
|
||||
if (values.replay !== undefined) {
|
||||
for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-cpu", "daytona-reuse-lease", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds", "storage-trigger", "native-trigger", "native-trigger-outage", "fault-after-seconds", "provider-fault-message", "provider-fault-delays", "db-pressure", "matrix", "server-from", "time-age-hours", "provider-fault-restart", "provider-fault-restart-after-seconds", "rr-gap-seconds", "rr-stop-on"] as const) {
|
||||
for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-cpu", "daytona-reuse-lease", "daytona-warm", "daytona-inherit", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds", "storage-trigger", "native-trigger", "native-trigger-outage", "fault-after-seconds", "provider-fault-message", "provider-fault-delays", "db-pressure", "matrix", "server-from", "time-age-hours", "provider-fault-restart", "provider-fault-restart-after-seconds", "rr-gap-seconds", "rr-stop-on"] as const) {
|
||||
if (values[key] !== undefined) throw new Error(`--replay reproduces the saved plan; do not combine it with --${key}`);
|
||||
}
|
||||
return { mode: "replay", ref: parseReplayRef(values.replay), keep,
|
||||
@@ -185,7 +189,8 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
|
||||
budgetCents,
|
||||
pollMs,
|
||||
...(env === "daytona" ? { daytonaImage, daytonaCpu: integer(values["daytona-cpu"], "daytona-cpu", DAYTONA_DEFAULT_CPU, 1, 4),
|
||||
...(values["daytona-reuse-lease"] ? { daytonaReuseLease: true } : {}) } : {}),
|
||||
...(values["daytona-reuse-lease"] ? { daytonaReuseLease: true } : {}),
|
||||
...(values["daytona-warm"] ? { daytonaWarm: true } : {}), ...(values["daytona-inherit"] ? { daytonaInherit: true } : {}) } : {}),
|
||||
...(values["rc06-masked"] ? { masks: ["rc06"] } : {}),
|
||||
...(lifecycle ? { lifecycle } : {}),
|
||||
// Seeded K4 hold upper end; the controller-outage cap is 600 s (#19797).
|
||||
@@ -272,9 +277,22 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
|
||||
}
|
||||
if (env !== "daytona" && values["daytona-cpu"] !== undefined) throw new Error("--daytona-cpu applies only to --env daytona");
|
||||
if (env !== "daytona" && values["daytona-reuse-lease"]) throw new Error("--daytona-reuse-lease applies only to --env daytona");
|
||||
for (const key of ["daytona-warm", "daytona-inherit"] as const) {
|
||||
if (env !== "daytona" && values[key]) throw new Error(`--${key} applies only to --env daytona`);
|
||||
}
|
||||
if (values["daytona-warm"] && values["daytona-inherit"]) throw new Error("--daytona-warm and --daytona-inherit cannot be combined: pick the warm fixture or an inheriting environment");
|
||||
if (options.daytonaCpu !== undefined) {
|
||||
const budget = integer(process.env.RESTART_CHAOS_DAYTONA_VCPU_BUDGET, "RESTART_CHAOS_DAYTONA_VCPU_BUDGET", DAYTONA_VCPU_BUDGET, 1, 1_000);
|
||||
if (options.concurrency * options.daytonaCpu > budget) {
|
||||
// A warm sandbox outlives its task for the idle window, and a new task cannot reuse
|
||||
// another task's sandbox, so with --daytona-warm (or --daytona-inherit with a warm
|
||||
// agent lifecycle) every task can hold one at once.
|
||||
const retains = daytonaRetainsSandboxes({ warmEnvironment: options.daytonaWarm === true,
|
||||
inheritEnvironment: options.daytonaInherit === true, lifecycle: options.lifecycle });
|
||||
const holders = retains ? Math.max(options.concurrency, options.tasks) : options.concurrency;
|
||||
if (holders * options.daytonaCpu > budget) {
|
||||
if (retains) throw new Error(`${options.daytonaWarm ? "--daytona-warm" : `--daytona-inherit with --lifecycle ${options.lifecycle}`} keeps each task's sandbox for the idle window: ${holders} tasks × `
|
||||
+ `--daytona-cpu ${options.daytonaCpu} vCPU exceeds the Daytona vCPU budget of ${budget}. Lower --tasks or --daytona-cpu `
|
||||
+ "(RESTART_CHAOS_DAYTONA_VCPU_BUDGET sets the budget).");
|
||||
throw new Error(`--concurrency ${options.concurrency} × --daytona-cpu ${options.daytonaCpu} vCPU exceeds the Daytona vCPU budget of ${budget} `
|
||||
+ "for chaos tasks. Lower one of them (RESTART_CHAOS_DAYTONA_VCPU_BUDGET sets the budget).");
|
||||
}
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { writeFileSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { assertControllerMatchesImage, defaultDaytonaImageCacheRoot, prepareDaytonaImageArtifacts } from "./daytona-image.js";
|
||||
import { PLUGIN_BUILD_STAMP, pluginBuildState } from "./plugin-build.js";
|
||||
import type { Timeline } from "./timeline.js";
|
||||
|
||||
function run(command: string, args: string[], cwd: string, label: string) {
|
||||
const result = spawnSync(command, args, { cwd, stdio: "inherit" });
|
||||
if (result.status !== 0) throw new Error(`${label} failed (${command} ${args.join(" ")})`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Mirrors restart-chaos main.ts's Daytona preparation for the TTFT driver.
|
||||
* The bundled Daytona plugin is outside the root workspace; prepare it as CI
|
||||
* does, and rebuild it whenever its sources (or the plugin SDK's) changed since
|
||||
* the last build, so a worktree that moved to a new master tests that master's
|
||||
* plugin rather than an older build.
|
||||
*/
|
||||
export function ensureDaytonaPlugin(repositoryRoot: string, timeline: Timeline) {
|
||||
const root = path.join(repositoryRoot, "packages/plugins/sandbox-providers/daytona");
|
||||
// Inputs outside the plugin directory: the linked plugin SDK and the tsconfig chain both packages extend.
|
||||
const extraInputs = [
|
||||
...["src", "package.json", "tsconfig.json"].map(file => path.join(repositoryRoot, "packages/plugins/sdk", file)),
|
||||
...["tsconfig.json", "tsconfig.base.json"].map(file => path.join(repositoryRoot, file)),
|
||||
];
|
||||
const state = pluginBuildState(root, extraInputs);
|
||||
timeline.add("daytona_plugin_build_state", { reason: state.reason, inputHash: state.inputHash });
|
||||
if (!state.needsBuild) return;
|
||||
run("pnpm", ["install", "--ignore-workspace", "--frozen-lockfile", "--ignore-scripts"], root, "Daytona plugin install");
|
||||
run(process.execPath, [path.join(repositoryRoot, "scripts/link-plugin-dev-sdk.mjs")], repositoryRoot, "Plugin SDK link");
|
||||
run("pnpm", ["--dir", root, "build"], repositoryRoot, "Daytona plugin build");
|
||||
writeFileSync(path.join(root, PLUGIN_BUILD_STAMP), `${state.inputHash}\n`);
|
||||
timeline.add("daytona_plugin_built", { inputHash: state.inputHash });
|
||||
}
|
||||
|
||||
/**
|
||||
* Controller environment for `--env daytona`: extract the image's Linux runnerd
|
||||
* and provider pack (cached per image digest, no Docker), refuse a controller
|
||||
* whose runner source differs from the image, and build the Daytona plugin.
|
||||
*/
|
||||
export async function prepareDaytonaController(input: { image: string; repositoryRoot: string; timeline: Timeline }):
|
||||
Promise<Record<string, string>> {
|
||||
const artifacts = await prepareDaytonaImageArtifacts({ image: input.image, cacheRoot: defaultDaytonaImageCacheRoot(),
|
||||
log: line => console.info(`[daytona-image] ${line}`) });
|
||||
const { localContentId } = await assertControllerMatchesImage(artifacts, input.repositoryRoot);
|
||||
input.timeline.add("daytona_image_matched", { image: input.image, contentId: artifacts.contentId, localContentId, sourceRevision: artifacts.sourceRevision,
|
||||
runnerdSha256: artifacts.runnerdSha256, providerPackDigest: artifacts.providerPackDigest, platformManifestDigest: artifacts.platformManifestDigest });
|
||||
ensureDaytonaPlugin(input.repositoryRoot, input.timeline);
|
||||
return { PAPERCLIP_RUNNER_REMOTE_BINARY_PATH: artifacts.runnerdPath, PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH: artifacts.providerPackPath };
|
||||
}
|
||||
@@ -330,6 +330,7 @@ async function main(): Promise<number> {
|
||||
const api: ChaosApi = new TimeShiftedApi(supervisor.baseUrl, supervisor.agingShifts);
|
||||
const lab = await seedChaosLab({ api, env: plan.options.env, waitForReadyAgents: true, cancelled: () => interrupted, daytonaImage: plan.options.daytonaImage, daytonaCpu: plan.options.daytonaCpu,
|
||||
daytonaReuseLease: plan.options.daytonaReuseLease,
|
||||
daytonaWarm: plan.options.daytonaWarm, daytonaInherit: plan.options.daytonaInherit,
|
||||
adapters: [...new Set([...plan.tasks.map(task => task.adapter),
|
||||
...(plan.tasks.some(task => task.workload === "M1") ? plan.options.adapters : [])])],
|
||||
concurrency: plan.options.concurrency, budgetCents: plan.options.budgetCents, masks: plan.options.masks,
|
||||
|
||||
@@ -261,6 +261,19 @@ describe("Daytona options", () => {
|
||||
expect(() => parseCommand(["--env", "local", "--daytona-cpu", "2"])).toThrow(/only to --env daytona/);
|
||||
expect(() => parseCommand(["--env", "local", "--daytona-reuse-lease"])).toThrow(/only to --env daytona/);
|
||||
});
|
||||
it("counts every task's sandbox against the budget when warm sandboxes outlive their task", () => {
|
||||
const wave = (...extra: string[]) => parseCommand(["--env", "daytona", "--daytona-image", image, "--tasks", "20", "--concurrency", "4", ...extra]);
|
||||
expect(() => wave("--daytona-warm")).toThrow(/--daytona-warm keeps each task's sandbox.*20 tasks/);
|
||||
expect(() => wave("--daytona-inherit", "--lifecycle", "warm")).toThrow(/--daytona-inherit with --lifecycle warm keeps each task's sandbox/);
|
||||
// --lifecycle default leaves the lifecycle unset everywhere, and a remote run then stays per-turn.
|
||||
expect(wave("--daytona-inherit", "--lifecycle", "default")).toMatchObject({ mode: "wave" });
|
||||
// The catalog agents pin per-turn, so an inheriting environment alone releases each sandbox after its turn.
|
||||
expect(wave("--daytona-inherit")).toMatchObject({ mode: "wave", options: { daytonaInherit: true, tasks: 20 } });
|
||||
expect(wave("--daytona-inherit", "--lifecycle", "per_turn")).toMatchObject({ mode: "wave" });
|
||||
expect(() => wave("--daytona-warm", "--daytona-inherit")).toThrow(/cannot be combined/);
|
||||
expect(() => parseCommand(["--env", "local", "--daytona-warm"])).toThrow(/only to --env daytona/);
|
||||
expect(() => parseCommand(["--env", "local", "--daytona-inherit"])).toThrow(/only to --env daytona/);
|
||||
});
|
||||
it("keeps reusable Daytona leases only when asked", () => {
|
||||
const reuse = parseCommand(["--env", "daytona", "--daytona-image", image, "--daytona-reuse-lease"]);
|
||||
expect(reuse).toMatchObject({ mode: "wave", options: { daytonaReuseLease: true } });
|
||||
|
||||
@@ -5,7 +5,7 @@ import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import type { ChaosApi } from "./api.js";
|
||||
import { cloneWorkspaceRepository, createFixtureRepository, seedChaosLab, waitForAgentsReady } from "./seed.js";
|
||||
import { cloneWorkspaceRepository, createFixtureRepository, inheritingEnvironmentConfig, seedChaosLab, waitForAgentsReady } from "./seed.js";
|
||||
import { Timeline } from "./timeline.js";
|
||||
|
||||
type Call = { method: string; route: string; body?: unknown };
|
||||
@@ -129,6 +129,14 @@ describe("seedChaosLab RC-06 mask", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("inheritingEnvironmentConfig", () => {
|
||||
it("drops the lifecycle and lease pins and keeps everything else", () => {
|
||||
const fixture = { provider: "daytona", image: "img", runnerLifecycleMode: "per_turn", reuseLease: false };
|
||||
expect(inheritingEnvironmentConfig(fixture)).toEqual({ provider: "daytona", image: "img" });
|
||||
expect(fixture).toHaveProperty("runnerLifecycleMode", "per_turn");
|
||||
});
|
||||
});
|
||||
|
||||
describe("cloneWorkspaceRepository", () => {
|
||||
it("clones a fresh, shallow working copy of the source repository", async () => {
|
||||
const root = await mkdtemp(path.join(os.tmpdir(), "restart-chaos-clone-"));
|
||||
|
||||
@@ -2,7 +2,7 @@ import { execFileSync } from "node:child_process";
|
||||
import { chmod, mkdir, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import type { RunnerApi } from "../runner-e2e/api.js";
|
||||
import { runnerEnvironments, runnerProfiles } from "../runner-e2e/catalog.js";
|
||||
import { daytonaWarmEnvironment, runnerEnvironments, runnerProfiles } from "../runner-e2e/catalog.js";
|
||||
import { installFixtureDaytonaProvider, waitForFixtureDaytonaProvider } from "../runner-e2e/live-fixtures.js";
|
||||
import type { MatrixExecution, SecretReference, SecretReferenceMap } from "../runner-e2e/types.js";
|
||||
import type { ChaosApi } from "./api.js";
|
||||
@@ -115,16 +115,25 @@ export function chaosDaytonaShape(cpu: number) {
|
||||
return { cpu, memory: 4, disk: 10 };
|
||||
}
|
||||
|
||||
/** A user's default environment config: the lifecycle and lease reuse are left to the agent. */
|
||||
export function inheritingEnvironmentConfig(config: Record<string, unknown>): Record<string, unknown> {
|
||||
const { runnerLifecycleMode: _lifecycle, reuseLease: _reuse, ...rest } = config;
|
||||
return rest;
|
||||
}
|
||||
|
||||
async function createDaytonaEnvironment(input: { api: ChaosApi; companyId: string; image: string; cpu: number; reuseLease: boolean; waveId: string;
|
||||
secretRefs: SecretReferenceMap; timeline: Timeline }): Promise<string> {
|
||||
const fixture = runnerEnvironments.find(candidate => candidate.id === "daytona");
|
||||
secretRefs: SecretReferenceMap; timeline: Timeline; warm?: boolean; inherit?: boolean }): Promise<string> {
|
||||
// The warm fixture keeps a reusable sandbox and a warm runner between turns.
|
||||
const fixture = input.warm ? daytonaWarmEnvironment : runnerEnvironments.find(candidate => candidate.id === "daytona");
|
||||
if (!fixture) throw new Error("runner-e2e catalog has no daytona environment fixture");
|
||||
const view = runnerApiView(input.api);
|
||||
await waitForFixtureDaytonaProvider(view, { companyId: input.companyId, deadlineAt: Date.now() + 120_000 });
|
||||
const payload = fixture.buildEnvironment({ secretRefs: input.secretRefs, daytonaImage: input.image, executionId: input.waveId });
|
||||
const shape = chaosDaytonaShape(input.cpu);
|
||||
// A user's default environment: the agent decides the lifecycle, and nothing pins lease reuse.
|
||||
const baseConfig = input.inherit ? inheritingEnvironmentConfig(payload.config as Record<string, unknown>) : payload.config as Record<string, unknown>;
|
||||
const environment = await input.api.post<{ id: string; driver: string }>(`/api/companies/${input.companyId}/environments`,
|
||||
{ ...payload, config: { ...(payload.config as Record<string, unknown>), ...shape, ...(input.reuseLease ? { reuseLease: true } : {}) } });
|
||||
{ ...payload, config: { ...baseConfig, ...shape, ...(input.reuseLease ? { reuseLease: true } : {}) } });
|
||||
input.timeline.add("seed_environment", { environmentId: environment.id, driver: environment.driver, image: input.image, ...shape,
|
||||
reuseLease: input.reuseLease });
|
||||
return environment.id;
|
||||
@@ -182,6 +191,12 @@ export async function seedChaosLab(input: {
|
||||
timeline: Timeline;
|
||||
/** Merged into each agent's adapterConfig (for example a runner lifecycle mode). */
|
||||
adapterConfigOverrides?: Record<string, unknown>;
|
||||
/** Use the reusable warm Daytona environment fixture (reuseLease, warm runner). */
|
||||
daytonaWarm?: boolean;
|
||||
/** Use a Daytona environment that inherits the lifecycle and leaves lease reuse unset, as a user's default does. */
|
||||
daytonaInherit?: boolean;
|
||||
/** A real repository to use as the project's primary workspace instead of the tiny fixture. */
|
||||
repositoryPath?: string;
|
||||
/** Called as soon as the wave owns external resources that need cleanup. */
|
||||
onEnvironment?: (lab: Pick<ChaosLab, "companyId" | "environmentId" | "environmentKind">) => void;
|
||||
}): Promise<ChaosLab> {
|
||||
@@ -225,7 +240,7 @@ export async function seedChaosLab(input: {
|
||||
let environmentId: string;
|
||||
if (input.env === "daytona") {
|
||||
environmentId = await createDaytonaEnvironment({ api, companyId: company.id, image: input.daytonaImage!, cpu: input.daytonaCpu ?? 4,
|
||||
reuseLease: input.daytonaReuseLease === true, waveId: input.waveId, secretRefs, timeline });
|
||||
reuseLease: input.daytonaReuseLease === true, waveId: input.waveId, secretRefs, timeline, warm: input.daytonaWarm, inherit: input.daytonaInherit });
|
||||
input.onEnvironment?.({ companyId: company.id, environmentId, environmentKind: "daytona" });
|
||||
if (input.adapters.includes("codex")) await qualifyNativeCodex({ api, companyId: company.id, environmentId, timeline });
|
||||
} else {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
baselineRows, distribution, openShellExecutions, percentile, rankPreTurnSteps, recoverySamples, runMetrics, runWarmth,
|
||||
baselineRows, daytonaBaselineHolders, daytonaRetainsSandboxes, distribution, parseBaselineEnvironment, parseBaselinePhases, openShellExecutions, percentile, rankPreTurnSteps, recoverySamples, runMetrics, runWarmth,
|
||||
type BaselineEvent, type BaselineRun,
|
||||
} from "./ttft-baseline-stats.js";
|
||||
|
||||
@@ -77,6 +77,52 @@ describe("ttft baseline statistics", () => {
|
||||
ttft: { n: 2, p50: 6_000, p95: 12_000 } });
|
||||
});
|
||||
|
||||
it("leaves post-restart and verification runs out of the baseline rows", () => {
|
||||
const rows = baselineRows([coldRun("a"), run("p", "postfault", [span("task.time_to_first_token", 0, 20_000)]),
|
||||
run("v", "verify", [span("task.time_to_first_token", 0, 30_000)])]);
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0]).toMatchObject({ warmth: "cold", ttft: { n: 1, p50: 6_000 } });
|
||||
});
|
||||
|
||||
it("counts held Daytona sandboxes per concurrent task, or per sampled task when they stay warm", () => {
|
||||
const phases = new Set(["coldwarm", "midturn", "restart"]);
|
||||
expect(daytonaBaselineHolders({ adapters: 3, samples: 5, phases, restarts: 2, retainsSandboxes: false })).toBe(3);
|
||||
// 5 samples x (coldwarm + midturn) + K1 and K2 for each of 2 restarts = 14 tasks per adapter.
|
||||
expect(daytonaBaselineHolders({ adapters: 3, samples: 5, phases, restarts: 2, retainsSandboxes: true })).toBe(42);
|
||||
// Restart tasks do not scale with --samples.
|
||||
expect(daytonaBaselineHolders({ adapters: 3, samples: 1, phases: new Set(["restart"]), restarts: 2, retainsSandboxes: true })).toBe(12);
|
||||
expect(daytonaBaselineHolders({ adapters: 3, samples: 5, phases: new Set(["restart"]), restarts: 2, retainsSandboxes: true })).toBe(12);
|
||||
expect(daytonaBaselineHolders({ adapters: 2, samples: 0, phases: new Set(["postfault"]), restarts: 0, retainsSandboxes: true })).toBe(2);
|
||||
});
|
||||
|
||||
it("treats an inheriting environment as warm only when the agent's lifecycle is not pinned per-turn", () => {
|
||||
expect(daytonaRetainsSandboxes({ warmEnvironment: true, inheritEnvironment: false, lifecycle: undefined })).toBe(true);
|
||||
expect(daytonaRetainsSandboxes({ warmEnvironment: false, inheritEnvironment: true, lifecycle: undefined })).toBe(false);
|
||||
expect(daytonaRetainsSandboxes({ warmEnvironment: false, inheritEnvironment: true, lifecycle: "per_turn" })).toBe(false);
|
||||
expect(daytonaRetainsSandboxes({ warmEnvironment: false, inheritEnvironment: true, lifecycle: "default" })).toBe(false);
|
||||
expect(daytonaRetainsSandboxes({ warmEnvironment: false, inheritEnvironment: true, lifecycle: "warm" })).toBe(true);
|
||||
expect(daytonaRetainsSandboxes({ warmEnvironment: false, inheritEnvironment: false, lifecycle: "warm" })).toBe(false);
|
||||
});
|
||||
|
||||
it("rejects unknown phases and a postfault phase without the coldwarm tasks it follows up on", () => {
|
||||
expect([...parseBaselinePhases("coldwarm, postfault")]).toEqual(["coldwarm", "postfault"]);
|
||||
expect([...parseBaselinePhases("verify")]).toEqual(["verify"]);
|
||||
expect(() => parseBaselinePhases("postfault")).toThrow(/add coldwarm/);
|
||||
expect(() => parseBaselinePhases("restart,postfualt")).toThrow(/got postfualt/);
|
||||
expect(() => parseBaselinePhases(" , ")).toThrow(/got nothing/);
|
||||
});
|
||||
|
||||
it("rejects contradictory or out-of-range environment options", () => {
|
||||
const base = { env: "daytona", image: "img", cpu: "4", warm: false, inherit: false };
|
||||
expect(parseBaselineEnvironment(base)).toEqual({ env: "daytona", cpu: 4 });
|
||||
expect(parseBaselineEnvironment({ ...base, env: "local", image: undefined })).toMatchObject({ env: "local" });
|
||||
expect(() => parseBaselineEnvironment({ ...base, env: "dayton" })).toThrow(/local or daytona/);
|
||||
expect(() => parseBaselineEnvironment({ ...base, image: undefined })).toThrow(/needs --daytona-image/);
|
||||
for (const cpu of ["0", "5", "abc", "2.5"]) expect(() => parseBaselineEnvironment({ ...base, cpu })).toThrow(/1 to 4/);
|
||||
expect(() => parseBaselineEnvironment({ ...base, warm: true, inherit: true })).toThrow(/cannot be combined/);
|
||||
expect(() => parseBaselineEnvironment({ ...base, env: "local", image: undefined, warm: true })).toThrow(/only to --env daytona/);
|
||||
});
|
||||
|
||||
it("reports first visible activity from run creation next to first text", () => {
|
||||
const toolFirst = run("t", "warm", [
|
||||
reuse("reused"),
|
||||
|
||||
@@ -16,7 +16,7 @@ export interface BaselineRun {
|
||||
adapter: string;
|
||||
environment: string;
|
||||
/** Driver intent: a new task, a follow-up comment on the same task, or a run crossed by a restart. */
|
||||
phase: "cold" | "warm" | "restart" | "midturn" | "verify";
|
||||
phase: "cold" | "warm" | "restart" | "midturn" | "verify" | "postfault";
|
||||
events: BaselineEvent[];
|
||||
workspaceCreated?: boolean;
|
||||
}
|
||||
@@ -222,8 +222,8 @@ export interface BaselineRow { adapter: string; environment: string; warmth: "co
|
||||
export function baselineRows(runs: readonly BaselineRun[]): BaselineRow[] {
|
||||
const groups = new Map<string, { adapter: string; environment: string; warmth: "cold" | "warm"; metrics: RunMetrics[] }>();
|
||||
for (const run of runs) {
|
||||
// Restart samples and new-agent verification samples are reported on their own.
|
||||
if (run.phase === "restart" || run.phase === "verify") continue;
|
||||
// Restart, post-restart and new-agent verification samples are reported on their own.
|
||||
if (run.phase === "restart" || run.phase === "verify" || run.phase === "postfault") continue;
|
||||
const warmth = runWarmth(run);
|
||||
const key = `${run.adapter}|${run.environment}|${warmth}`;
|
||||
const group = groups.get(key) ?? { adapter: run.adapter, environment: run.environment, warmth, metrics: [] };
|
||||
@@ -309,3 +309,57 @@ export function openShellExecutions(events: readonly BaselineEvent[], marker?: s
|
||||
}
|
||||
return [...open.entries()].map(([executionId, entry]) => ({ executionId, matched: entry.matched, unlabelled: !entry.labelled }));
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether finished tasks keep their Daytona sandbox for the idle window. The warm
|
||||
* fixture always does. An inheriting environment defers to the agent, whose catalog
|
||||
* profile pins per_turn unless --lifecycle warm replaces it. --lifecycle default
|
||||
* leaves the agent unset, and a remote run with no lifecycle anywhere is per-turn.
|
||||
*/
|
||||
export function daytonaRetainsSandboxes(input: { warmEnvironment: boolean; inheritEnvironment: boolean; lifecycle: string | undefined }): boolean {
|
||||
if (input.warmEnvironment) return true;
|
||||
return input.inheritEnvironment && input.lifecycle === "warm";
|
||||
}
|
||||
|
||||
/** Daytona sandboxes a TTFT baseline can hold at once, for the vCPU budget check. */
|
||||
export function daytonaBaselineHolders(input: { adapters: number; samples: number; phases: ReadonlySet<string>; restarts: number;
|
||||
retainsSandboxes: boolean }): number {
|
||||
// The default fixture releases each sandbox after its turn, so only the concurrent tasks hold one.
|
||||
if (!input.retainsSandboxes) return input.adapters;
|
||||
// A warm sandbox outlives its task for the idle window and no other task reuses it.
|
||||
// coldwarm, midturn and verify create one task per sample; restart creates one per
|
||||
// fault, K1 and K2 for each --restarts, independent of --samples.
|
||||
const sampledTasks = Math.max(1, input.samples) * ["coldwarm", "midturn", "verify"].filter(phase => input.phases.has(phase)).length;
|
||||
const restartTasks = input.phases.has("restart") ? 2 * input.restarts : 0;
|
||||
return input.adapters * Math.max(1, sampledTasks + restartTasks);
|
||||
}
|
||||
|
||||
/** Validates the TTFT driver's environment options; returns the environment. */
|
||||
const BASELINE_PHASES = ["coldwarm", "midturn", "restart", "verify", "postfault"] as const;
|
||||
|
||||
/** `--phases`: known phases only; postfault follows up on the coldwarm tasks, so it needs coldwarm. */
|
||||
export function parseBaselinePhases(value: string): Set<string> {
|
||||
const phases = new Set(value.split(",").map(phase => phase.trim()).filter(Boolean));
|
||||
const unknown = [...phases].filter(phase => !(BASELINE_PHASES as readonly string[]).includes(phase));
|
||||
if (unknown.length || !phases.size) throw new Error(`--phases takes ${BASELINE_PHASES.join(", ")}; got ${unknown.join(", ") || "nothing"}`);
|
||||
if (phases.has("postfault") && !phases.has("coldwarm")) {
|
||||
throw new Error("--phases postfault follows up on the coldwarm tasks after an idle restart; add coldwarm");
|
||||
}
|
||||
return phases;
|
||||
}
|
||||
|
||||
export function parseBaselineEnvironment(input: { env: string | undefined; image: string | undefined; cpu: string | undefined;
|
||||
warm: boolean; inherit: boolean }): { env: "local" | "daytona"; cpu: number } {
|
||||
const env = input.env ?? "local";
|
||||
if (env !== "local" && env !== "daytona") throw new Error(`--env must be local or daytona, got ${env}`);
|
||||
const cpu = Number(input.cpu ?? "4");
|
||||
if (env === "local") {
|
||||
if (input.warm || input.inherit || input.image) throw new Error("--daytona-* options apply only to --env daytona");
|
||||
return { env, cpu };
|
||||
}
|
||||
if (!input.image) throw new Error("--env daytona needs --daytona-image");
|
||||
if (!Number.isInteger(cpu) || cpu < 1 || cpu > 4) throw new Error(`--daytona-cpu must be an integer from 1 to 4, got ${input.cpu}`);
|
||||
// The warm fixture pins its lifecycle; inherit removes the pins. The two contradict.
|
||||
if (input.warm && input.inherit) throw new Error("--daytona-warm and --daytona-inherit cannot be combined");
|
||||
return { env, cpu };
|
||||
}
|
||||
@@ -21,15 +21,17 @@ import { parseArgs } from "node:util";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import { releaseInstance } from "../runner-e2e/instance-retention.js";
|
||||
import { ChaosApi } from "./api.js";
|
||||
import { CREDENTIAL_FOR_ADAPTER } from "./config.js";
|
||||
import { CREDENTIAL_FOR_ADAPTER, DAYTONA_VCPU_BUDGET } from "./config.js";
|
||||
import { ACTIVE_RUN_STATUSES } from "./grader.js";
|
||||
import { chaosLifecycleOverrides, seedChaosLab, type ChaosLab } from "./seed.js";
|
||||
import { ChaosSupervisor } from "./supervisor.js";
|
||||
import { createDb } from "../../packages/db/src/index.js";
|
||||
import { prepareDaytonaController } from "./daytona-controller.js";
|
||||
import { DaytonaWaveSweeper } from "./daytona-sweep.js";
|
||||
import { Timeline } from "./timeline.js";
|
||||
import type { AdapterId, FaultKind } from "./types.js";
|
||||
import {
|
||||
baselineRows, rankPreTurnSteps, recoverySamples, distribution, openShellExecutions,
|
||||
baselineRows, rankPreTurnSteps, recoverySamples, distribution, openShellExecutions, daytonaBaselineHolders, daytonaRetainsSandboxes, parseBaselineEnvironment, parseBaselinePhases,
|
||||
type BaselineEvent, type BaselineFault, type BaselineRun,
|
||||
} from "./ttft-baseline-stats.js";
|
||||
|
||||
@@ -89,9 +91,13 @@ async function steerQueuedComment(api: ChaosApi, issueId: string, commentId: str
|
||||
return { acknowledgedAt: null, disposition };
|
||||
}
|
||||
|
||||
/** Set by --workspace-repo: tasks then run in the project's isolated workspace of that repository. */
|
||||
let projectScopedTasks = false;
|
||||
|
||||
async function createTask(api: ChaosApi, lab: ChaosLab, adapter: AdapterId, title: string, description: string) {
|
||||
const issue = await api.post<Json>(`/api/companies/${lab.companyId}/issues`, {
|
||||
title, description, status: "todo", priority: "medium", assigneeAgentId: lab.agents[adapter].id,
|
||||
...(projectScopedTasks ? { projectId: lab.projectId } : {}),
|
||||
});
|
||||
return String(issue.id);
|
||||
}
|
||||
@@ -201,6 +207,11 @@ function report(runs: BaselineRun[], faults: BaselineFault[]): string {
|
||||
lines.push("", "New agent behind harness verification (TTFT counts from run creation; the verification wait precedes it):", "", lines[0]!, lines[1]!);
|
||||
for (const row of baselineRows(verifyRuns)) lines.push(rowLine(row));
|
||||
}
|
||||
const postfault = runs.filter(run => run.phase === "postfault").map(run => ({ ...run, phase: "cold" as const }));
|
||||
if (postfault.length) {
|
||||
lines.push("", "First follow-up after an idle control-plane restart (K1):", "", lines[0]!, lines[1]!);
|
||||
for (const row of baselineRows(postfault)) lines.push(rowLine(row));
|
||||
}
|
||||
|
||||
lines.push("", "| Pre-turn step | n | p50 | p95 | Adapters |", "|---|---|---|---|---|");
|
||||
for (const step of rankPreTurnSteps(runs).slice(0, 25)) lines.push(`| \`${step.step}\` | ${step.n} | ${ms(step.p50)} | ${ms(step.p95)} | ${step.adapters.join(", ")} |`);
|
||||
@@ -238,6 +249,17 @@ async function main() {
|
||||
"midturn-fault": { type: "string" },
|
||||
// verify phase: fault the control plane while a newly hired agent runs harness verification.
|
||||
"verify-fault": { type: "string" },
|
||||
// A real repository (for example a fresh clone of paperclip) as the project's
|
||||
// primary workspace; every task then runs in its isolated git worktree.
|
||||
"workspace-repo": { type: "string" },
|
||||
// local (default) or daytona; daytona needs --daytona-image (the signed Runner image).
|
||||
env: { type: "string", default: "local" },
|
||||
"daytona-image": { type: "string" },
|
||||
"daytona-cpu": { type: "string", default: "4" },
|
||||
// Use the reusable warm Daytona environment (reuseLease, warm runner) instead of the per-turn one.
|
||||
"daytona-warm": { type: "boolean", default: false },
|
||||
// A Daytona environment as a user creates it by default: lifecycle inherited, lease reuse unset.
|
||||
"daytona-inherit": { type: "boolean", default: false },
|
||||
"idle-ms": { type: "string", default: "600000" },
|
||||
} });
|
||||
if (values.report) {
|
||||
@@ -245,11 +267,28 @@ async function main() {
|
||||
process.stdout.write(report(data.runs, data.faults));
|
||||
return;
|
||||
}
|
||||
const phases = new Set(values.phases!.split(",").map(value => value.trim()));
|
||||
const phases = parseBaselinePhases(values.phases!);
|
||||
const requested = values.adapters!.split(",").map(value => value.trim()) as AdapterId[];
|
||||
const samples = Math.max(1, Number(values.samples));
|
||||
const restarts = Math.max(0, Number(values.restarts));
|
||||
const missing = requested.map(adapter => CREDENTIAL_FOR_ADAPTER[adapter]).filter(key => !process.env[key]);
|
||||
const lifecycle = values.lifecycle;
|
||||
if (lifecycle !== undefined && lifecycle !== "per_turn" && lifecycle !== "warm" && lifecycle !== "default") {
|
||||
throw new Error("--lifecycle must be per_turn, warm or default");
|
||||
}
|
||||
const { env, cpu } = parseBaselineEnvironment({ env: values.env, image: values["daytona-image"], cpu: values["daytona-cpu"],
|
||||
warm: values["daytona-warm"] === true, inherit: values["daytona-inherit"] === true });
|
||||
if (env === "daytona") {
|
||||
const budget = Number(process.env.RESTART_CHAOS_DAYTONA_VCPU_BUDGET ?? DAYTONA_VCPU_BUDGET);
|
||||
const retainsSandboxes = daytonaRetainsSandboxes({ warmEnvironment: values["daytona-warm"] === true,
|
||||
inheritEnvironment: values["daytona-inherit"] === true, lifecycle });
|
||||
const holders = daytonaBaselineHolders({ adapters: requested.length, samples, phases, restarts, retainsSandboxes });
|
||||
if (holders * cpu > budget) {
|
||||
throw new Error(`${holders} ${retainsSandboxes ? "sampled tasks (warm sandboxes outlive their task)" : "concurrent tasks"} × --daytona-cpu ${cpu} vCPU can be held at once, `
|
||||
+ `exceeding the Daytona vCPU budget of ${budget}. Lower --samples, --adapters or --daytona-cpu (RESTART_CHAOS_DAYTONA_VCPU_BUDGET sets the budget).`);
|
||||
}
|
||||
}
|
||||
const missing = [...requested.map(adapter => CREDENTIAL_FOR_ADAPTER[adapter]), ...(env === "daytona" ? ["DAYTONA_API_KEY"] : [])]
|
||||
.filter(key => !process.env[key]);
|
||||
if (missing.length) throw new Error(`Missing ${missing.join(", ")}; run: set -a; source ~/.secrets; set +a`);
|
||||
const midturnMode = values["midturn-mode"];
|
||||
if (midturnMode !== "queue" && midturnMode !== "steer") throw new Error("--midturn-mode must be queue or steer");
|
||||
@@ -258,10 +297,6 @@ async function main() {
|
||||
if (midturnFault && midturnMode !== "steer") throw new Error("--midturn-fault applies to --midturn-mode steer");
|
||||
const verifyFault = values["verify-fault"] as FaultKind | undefined;
|
||||
if (verifyFault !== undefined && !["K1", "K2", "K6"].includes(verifyFault)) throw new Error("--verify-fault must be K1, K2 or K6");
|
||||
const lifecycle = values.lifecycle;
|
||||
if (lifecycle !== undefined && lifecycle !== "per_turn" && lifecycle !== "warm" && lifecycle !== "default") {
|
||||
throw new Error("--lifecycle must be per_turn, warm or default");
|
||||
}
|
||||
// default removes the catalog's pinned per_turn so the server resolves the
|
||||
// product default; absent keeps the catalog setting.
|
||||
const adapterConfigOverrides = lifecycle === "warm"
|
||||
@@ -278,10 +313,35 @@ async function main() {
|
||||
const agentLifecycles: Record<string, string> = {};
|
||||
const midturn: Json[] = [];
|
||||
const timedOutIssues = new Set<string>();
|
||||
let sweeper: DaytonaWaveSweeper | null = null;
|
||||
const workspaceOperations: Array<{ runId: string; adapter: string; phase: string; operation: string; status: string;
|
||||
durationMs: number | null; command: string | null }> = [];
|
||||
// Every coldwarm sample that finished at rest gets its own post-restart follow-up.
|
||||
const followUpTasks: Array<{ adapter: AdapterId; issueId: string; nonce: string; runs: number }> = [];
|
||||
// One shutdown path for normal completion and for SIGINT/SIGTERM: the server
|
||||
// runs in a detached process group and the embedded Postgres has no exit hook.
|
||||
let stopping: Promise<void> | null = null;
|
||||
const shutdown = () => (stopping ??= supervisor.stop().then(() => false, error => {
|
||||
// Daytona resources are released while the server still runs, before it stops;
|
||||
// the same path serves normal completion and SIGINT/SIGTERM.
|
||||
const releaseDaytona = async () => {
|
||||
if (!sweeper) return;
|
||||
const owned = sweeper as DaytonaWaveSweeper;
|
||||
sweeper = null;
|
||||
try {
|
||||
const cleanup = await owned.cleanUp();
|
||||
timeline.add("daytona_cleanup", cleanup);
|
||||
await writeFile(path.join(outDir, "sandbox-cleanup.json"), `${JSON.stringify(cleanup, null, 2)}\n`, { mode: 0o600 });
|
||||
if (cleanup.remainingSandboxes.length || cleanup.errors.length || cleanup.environmentDeleteBlocked || !cleanup.environmentDeleted) {
|
||||
process.stderr.write(`Daytona cleanup incomplete: remaining ${cleanup.remainingSandboxes.join(", ") || "none"}; `
|
||||
+ `environment deleted ${cleanup.environmentDeleted}${cleanup.environmentDeleteBlocked ? ` (blocked: ${String(cleanup.environmentDeleteBlocked)})` : ""}; ${cleanup.errors.join("; ")}\n`);
|
||||
process.exitCode = 1;
|
||||
}
|
||||
} catch (error) {
|
||||
timeline.add("daytona_cleanup_failed", { error: String(error) });
|
||||
process.exitCode = 1;
|
||||
}
|
||||
};
|
||||
const shutdown = () => (stopping ??= releaseDaytona().then(() => supervisor.stop()).then(() => false, error => {
|
||||
process.stderr.write(`supervisor stop: ${String(error)}\n`);
|
||||
return true;
|
||||
}).then(stopFailed => releaseInstance(supervisor.paths.temporaryRoot, { keep: false, stopFailed, owner: "ttft-baseline" }))
|
||||
@@ -293,10 +353,18 @@ async function main() {
|
||||
process.once("SIGINT", onSignal);
|
||||
process.once("SIGTERM", onSignal);
|
||||
try {
|
||||
await supervisor.start();
|
||||
await supervisor.start(env === "daytona"
|
||||
? await prepareDaytonaController({ image: values["daytona-image"]!, repositoryRoot, timeline }) : {});
|
||||
const api = new ChaosApi(supervisor.baseUrl);
|
||||
const lab = await seedChaosLab({ api, env: "local", adapters: requested, concurrency: requested.length, budgetCents: 3000,
|
||||
masks: ["rc06"], fixtureRoot: supervisor.paths.temporaryRoot, waveId, timeline, adapterConfigOverrides });
|
||||
const workspaceRepo = values["workspace-repo"] ? path.resolve(values["workspace-repo"]) : undefined;
|
||||
projectScopedTasks = Boolean(workspaceRepo);
|
||||
const lab = await seedChaosLab({ api, env, daytonaImage: values["daytona-image"], daytonaCpu: cpu,
|
||||
daytonaWarm: values["daytona-warm"], daytonaInherit: values["daytona-inherit"],
|
||||
adapters: requested, concurrency: requested.length, budgetCents: 3000,
|
||||
masks: ["rc06"], fixtureRoot: supervisor.paths.temporaryRoot, waveId, timeline, adapterConfigOverrides, workspaceRepo,
|
||||
onEnvironment: owned => {
|
||||
sweeper = new DaytonaWaveSweeper({ apiKey: process.env.DAYTONA_API_KEY!, companyId: owned.companyId, environmentId: owned.environmentId, api });
|
||||
} });
|
||||
const adapters = await readyAdapters(api, lab, requested, timeline);
|
||||
if (!adapters.length) throw new Error("No agent passed lifecycle verification");
|
||||
// Record what each agent actually stores: "unset" means the server resolves
|
||||
@@ -332,6 +400,7 @@ async function main() {
|
||||
continue;
|
||||
}
|
||||
timeline.add("baseline_warm_done", { adapter, issueId, runs: warm.runs.length - cold.runs.length });
|
||||
followUpTasks.push({ adapter, issueId, nonce, runs: warm.runs.length });
|
||||
}
|
||||
}));
|
||||
|
||||
@@ -346,7 +415,8 @@ async function main() {
|
||||
const hired = await api.post<Json>(`/api/companies/${lab.companyId}/agents`, { ...source.body, name: `${source.name} verify ${nonce}` });
|
||||
const agentId = String(hired.id);
|
||||
const issue = await api.post<Json>(`/api/companies/${lab.companyId}/issues`, {
|
||||
title: `TTFT verify ${nonce}`, description: coldTask(nonce), status: "todo", priority: "medium", assigneeAgentId: agentId });
|
||||
title: `TTFT verify ${nonce}`, description: coldTask(nonce), status: "todo", priority: "medium", assigneeAgentId: agentId,
|
||||
...(projectScopedTasks ? { projectId: lab.projectId } : {}) });
|
||||
const issueId = String(issue.id);
|
||||
tasks.push({ issueId, adapter, phase: "verify" });
|
||||
let fault: Awaited<ReturnType<typeof supervisor.fault>> | null = null;
|
||||
@@ -387,6 +457,45 @@ async function main() {
|
||||
}
|
||||
}
|
||||
|
||||
// First turn after an idle control-plane restart: K1 while nothing runs,
|
||||
// then one more follow-up on each task the cold/warm phase left at rest.
|
||||
// The fault must be idle: no timed-out sample may still be running.
|
||||
// A failed runs read counts as not at rest: the fault must never land on a live turn.
|
||||
const allAtRest = timedOutIssues.size === 0 && (await Promise.all(tasks.map(async task =>
|
||||
(await api.get<Json[]>(`/api/issues/${task.issueId}/runs`, 10_000).catch(() => null))
|
||||
?.every(run => !ACTIVE_RUN_STATUSES.has(String(run.status))) ?? false))).every(Boolean);
|
||||
if (phases.has("postfault") && followUpTasks.length && !allAtRest) {
|
||||
timeline.add("baseline_postfault_skipped", { reason: "work_not_at_rest", timedOut: timedOutIssues.size });
|
||||
}
|
||||
if (phases.has("postfault") && followUpTasks.length && allAtRest) {
|
||||
// One idle K1 per round, then the first follow-up on one task per adapter, so each
|
||||
// sample is a genuine first turn after a restart. A timed-out follow-up may still be
|
||||
// running, so it ends the rounds: the next fault must stay idle.
|
||||
const rounds = Math.max(...adapters.map(adapter => followUpTasks.filter(entry => entry.adapter === adapter).length));
|
||||
for (let round = 0; round < rounds; round += 1) {
|
||||
const batch = adapters.flatMap(adapter => followUpTasks.filter(entry => entry.adapter === adapter).slice(round, round + 1));
|
||||
const receipt = await supervisor.fault("K1", 0, `ttft-baseline-postfault-${round}`);
|
||||
faults.push({ kind: "K1", readyAt: receipt.readyAt ?? Date.now(), runIds: [] });
|
||||
let timedOut = false;
|
||||
await Promise.all(batch.map(async entry => {
|
||||
const { adapter } = entry;
|
||||
await api.post(`/api/issues/${entry.issueId}/comments`, { body: warmFollowUp(`${entry.nonce}-postfault`), reopen: true });
|
||||
const after = await waitForRest(api, supervisor, entry.issueId, entry.runs + 1);
|
||||
for (const run of after.runs) if (!runPhase.has(runIdOf(run))) runPhase.set(runIdOf(run), "postfault");
|
||||
if (after.timedOut) {
|
||||
timedOut = true;
|
||||
timedOutIssues.add(entry.issueId);
|
||||
timeline.add("baseline_sample_timeout", { adapter, issueId: entry.issueId, phase: "postfault" });
|
||||
}
|
||||
else timeline.add("baseline_postfault_done", { adapter, issueId: entry.issueId, round });
|
||||
}));
|
||||
if (timedOut) {
|
||||
if (round + 1 < rounds) timeline.add("baseline_postfault_skipped", { reason: "work_not_at_rest", round: round + 1 });
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Mid-turn follow-up: a comment posted while the agent is inside a long
|
||||
// tool call. Records when the comment landed and every run of the task,
|
||||
// so the wait splits into "rest of the turn" and "hand-off".
|
||||
@@ -476,15 +585,34 @@ async function main() {
|
||||
if (timedOutIssues.has(task.issueId)) continue;
|
||||
for (const run of await runsFor(api, task.issueId)) {
|
||||
const runId = runIdOf(run);
|
||||
runs.push({ runId, adapter: task.adapter, environment: "local",
|
||||
runs.push({ runId, adapter: task.adapter, environment: env,
|
||||
phase: runPhase.get(runId) ?? task.phase, events: await allEvents(api, runId) });
|
||||
// Workspace commands (worktree, provision, sync) the run waited on, with their durations.
|
||||
const operations = await api.get<Json[]>(`/api/heartbeat-runs/${runId}/workspace-operations`, 10_000).catch((error: unknown) => {
|
||||
// A failed read is recorded as such, never as "no workspace operations".
|
||||
workspaceOperations.push({ runId, adapter: task.adapter, phase: runPhase.get(runId) ?? task.phase,
|
||||
operation: "read_failed", status: error instanceof Error ? error.message.slice(0, 160) : String(error), durationMs: null, command: null });
|
||||
return [] as Json[];
|
||||
});
|
||||
for (const operation of operations) workspaceOperations.push({ runId, adapter: task.adapter,
|
||||
phase: runPhase.get(runId) ?? task.phase, operation: String(operation.phase), status: String(operation.status),
|
||||
durationMs: operation.startedAt && operation.finishedAt
|
||||
? Date.parse(String(operation.finishedAt)) - Date.parse(String(operation.startedAt)) : null,
|
||||
command: typeof operation.command === "string" ? operation.command.slice(0, 120) : null });
|
||||
}
|
||||
}
|
||||
const data = { waveId, lifecycle: lifecycle ?? "catalog", agentLifecycles, adapters, notReady: requested.filter(adapter => !adapters.includes(adapter)),
|
||||
const data = { workspaceOperations, waveId, lifecycle: lifecycle ?? "catalog", agentLifecycles, adapters, notReady: requested.filter(adapter => !adapters.includes(adapter)),
|
||||
timedOutTasks: timedOutIssues.size, samples, restarts, midturn, gitSha: process.env.GIT_SHA ?? null, runs, faults };
|
||||
await writeFile(path.join(outDir, "baseline-data.json"), `${JSON.stringify(data)}\n`, { mode: 0o600 });
|
||||
await writeFile(path.join(outDir, "baseline-report.md"), report(runs, faults), { mode: 0o600 });
|
||||
process.stdout.write(report(runs, faults));
|
||||
if (workspaceOperations.length) {
|
||||
const table = ["", "| Run phase | Adapter | Workspace operation | Status | Duration | Command |", "|---|---|---|---|---|---|",
|
||||
...workspaceOperations.slice().sort((a, b) => (b.durationMs ?? 0) - (a.durationMs ?? 0))
|
||||
.map(op => `| ${op.phase} | ${op.adapter} | ${op.operation} | ${op.status} | ${op.durationMs === null ? "—" : `${(op.durationMs / 1000).toFixed(2)} s`} | ${op.command ? `\`${op.command}\`` : "—"} |`)].join("\n");
|
||||
process.stdout.write(`${table}\n`);
|
||||
await writeFile(path.join(outDir, "workspace-operations.md"), `${table}\n`, { mode: 0o600 });
|
||||
}
|
||||
if (timedOutIssues.size) process.stdout.write(`\n${timedOutIssues.size} sample(s) timed out and were left out.\n`);
|
||||
// Every answered mid-turn comment must be answered exactly once.
|
||||
const wrongReplies = midturn.filter(sample => sample.replyCount !== 1);
|
||||
@@ -495,7 +623,8 @@ async function main() {
|
||||
if (wrongReplies.length) process.exitCode = 1;
|
||||
process.stdout.write(`\nWrote ${outDir}\n`);
|
||||
} finally {
|
||||
// Always stop the wrapper-owned embedded Postgres (SysV shared memory is scarce).
|
||||
// Every Daytona sandbox this run created is archived or destroyed before the server
|
||||
// stops, then the wrapper-owned embedded Postgres stops (SysV shared memory is scarce).
|
||||
await shutdown();
|
||||
process.off("SIGINT", onSignal);
|
||||
process.off("SIGTERM", onSignal);
|
||||
|
||||
@@ -59,6 +59,10 @@ export interface WaveOptions {
|
||||
daytonaCpu?: number;
|
||||
/** `--daytona-reuse-lease`: the chaos environment keeps its sandboxes between turns (a per-turn run stops and keeps its sandbox). */
|
||||
daytonaReuseLease?: boolean;
|
||||
/** `--daytona-warm`: the reusable warm Daytona environment fixture. */
|
||||
daytonaWarm?: boolean;
|
||||
/** `--daytona-inherit`: a Daytona environment that inherits the lifecycle and leaves lease reuse unset. */
|
||||
daytonaInherit?: boolean;
|
||||
/** `--k4-hold-max-seconds`: upper end of the seeded K4 hold (default 300 s, cap 600 s). */
|
||||
k4HoldMaxMs?: number;
|
||||
faultAfterMs?: number;
|
||||
|
||||
Reference in new issue
Block a user