test(restart-chaos): measure TTFT in a real repository workspace, after an idle restart, and on Daytona (#20047)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos TTFT driver measures time to first token, but only
with a tiny fixture workspace and only locally
> - Dotta asked whether speed is good with a workspace like the
paperclip repository itself, and that had never been measured
> - This pull request lets the driver run in a real repository
workspace, measure the first follow-up after an idle restart, run on
Daytona (per-turn or warm reusable sandboxes), and report each run's
workspace operations
> - The benefit is a repeatable large-workspace speed measurement; its
first run found a full workspace re-upload on every remote turn

## Linked Issues or Issue Description

**What happened?**
The TTFT driver seeded only a tiny fixture repository, created tasks
outside any project, ran only locally, and could not show workspace
operation durations. So large-workspace speed (worktree creation,
provision, remote upload) was never measured.

**Steps to reproduce**
1. Run `ttft-baseline.ts` with any options.
2. Every task runs without a project workspace and on the local host.
3. Nothing in the report shows worktree, provision or sync cost.

**Expected behavior**
The driver can measure a real repository workspace, locally and on
Daytona, and names the workspace operations each run waited on.

## What Changed

- `--workspace-repo <path>`: the seed uses that repository as the
project's primary workspace (git_worktree isolation), and tasks are
created in the project.
- A `postfault` phase: K1 while idle, then a follow-up on each task;
reported in its own table.
- `--env daytona`, `--daytona-image`, `--daytona-cpu`, `--daytona-warm`
(the reusable warm Daytona fixture). The Daytona controller preparation
moved to `daytona-controller.ts`, and every sandbox is cleaned up at the
end; incomplete cleanup fails the run.
- Each run's workspace operations
(`/api/heartbeat-runs/:id/workspace-operations`) are saved, and printed
ranked by duration.

## Verification

- `tsc -p tests/restart-chaos`: no errors in harness files.
- Used for the large-workspace measurement, a fresh paperclip clone on
local and Daytona (per-turn and warm). First turn / follow-up: local
31–34 s / 7 s; Daytona per-turn 58–63 s / 30 s; Daytona warm 45–51 s /
8–10 s. Every Daytona sandbox was cleaned up (environment deleted, none
remaining).

## Risks

- Test harness only; no product code.
## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file editing, test execution, Daytona measurements).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-10 10:00:45 -06:00
1 parent ce1ee9e89f
commit 10bc6431da
10 files changed
+368 -29

No files matched your search

+21 -3
View File
@@ -9,6 +9,7 @@ import { parseTimeFaultOptions } from "./time-faults-config.js";
import { parseStorageTrigger } from "./storage-trigger.js";
import { parseNativeTrigger, parseNativeTriggerOutage } from "./native-trigger.js";
import { parseRepeatedRestartOptions } from "./repeated-restarts.js";
import { daytonaRetainsSandboxes } from "./ttft-baseline-stats.js";
import { PROVIDER_FAULT_KINDS, PROVIDER_FAULT_WORKLOADS, isProviderFault, parseProviderDelays, parseProviderMessages, type ProviderFaultKind } from "./provider-faults.js";
import { PROVIDER_FAULT_ADAPTERS, parseProviderFaultRestart } from "./acpx-provider-faults.js";
import {
@@ -99,6 +100,9 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
"daytona-image": { type: "string" },
"daytona-cpu": { type: "string" },
"daytona-reuse-lease": { type: "boolean" },
// The reusable warm Daytona fixture, or an environment that inherits the lifecycle (a user's default).
"daytona-warm": { type: "boolean" },
"daytona-inherit": { type: "boolean" },
keep: { type: "boolean" },
"rc06-masked": { type: "boolean" },
"storm-minutes": { type: "string" },
@@ -131,7 +135,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
const pollMs = integer(values["poll-ms"], "poll-ms", 3_000, 2_000, 5_000);
const budgetCents = integer(values["budget-cents"], "budget-cents", 3_000, 1, 100_000);
if (values.replay !== undefined) {
for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-cpu", "daytona-reuse-lease", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds", "storage-trigger", "native-trigger", "native-trigger-outage", "fault-after-seconds", "provider-fault-message", "provider-fault-delays", "db-pressure", "matrix", "server-from", "time-age-hours", "provider-fault-restart", "provider-fault-restart-after-seconds", "rr-gap-seconds", "rr-stop-on"] as const) {
for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-cpu", "daytona-reuse-lease", "daytona-warm", "daytona-inherit", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds", "storage-trigger", "native-trigger", "native-trigger-outage", "fault-after-seconds", "provider-fault-message", "provider-fault-delays", "db-pressure", "matrix", "server-from", "time-age-hours", "provider-fault-restart", "provider-fault-restart-after-seconds", "rr-gap-seconds", "rr-stop-on"] as const) {
if (values[key] !== undefined) throw new Error(`--replay reproduces the saved plan; do not combine it with --${key}`);
}
return { mode: "replay", ref: parseReplayRef(values.replay), keep,
@@ -185,7 +189,8 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
budgetCents,
pollMs,
...(env === "daytona" ? { daytonaImage, daytonaCpu: integer(values["daytona-cpu"], "daytona-cpu", DAYTONA_DEFAULT_CPU, 1, 4),
...(values["daytona-reuse-lease"] ? { daytonaReuseLease: true } : {}) } : {}),
...(values["daytona-reuse-lease"] ? { daytonaReuseLease: true } : {}),
...(values["daytona-warm"] ? { daytonaWarm: true } : {}), ...(values["daytona-inherit"] ? { daytonaInherit: true } : {}) } : {}),
...(values["rc06-masked"] ? { masks: ["rc06"] } : {}),
...(lifecycle ? { lifecycle } : {}),
// Seeded K4 hold upper end; the controller-outage cap is 600 s (#19797).
@@ -272,9 +277,22 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
}
if (env !== "daytona" && values["daytona-cpu"] !== undefined) throw new Error("--daytona-cpu applies only to --env daytona");
if (env !== "daytona" && values["daytona-reuse-lease"]) throw new Error("--daytona-reuse-lease applies only to --env daytona");
for (const key of ["daytona-warm", "daytona-inherit"] as const) {
if (env !== "daytona" && values[key]) throw new Error(`--${key} applies only to --env daytona`);
}
if (values["daytona-warm"] && values["daytona-inherit"]) throw new Error("--daytona-warm and --daytona-inherit cannot be combined: pick the warm fixture or an inheriting environment");
if (options.daytonaCpu !== undefined) {
const budget = integer(process.env.RESTART_CHAOS_DAYTONA_VCPU_BUDGET, "RESTART_CHAOS_DAYTONA_VCPU_BUDGET", DAYTONA_VCPU_BUDGET, 1, 1_000);
if (options.concurrency * options.daytonaCpu > budget) {
// A warm sandbox outlives its task for the idle window, and a new task cannot reuse
// another task's sandbox, so with --daytona-warm (or --daytona-inherit with a warm
// agent lifecycle) every task can hold one at once.
const retains = daytonaRetainsSandboxes({ warmEnvironment: options.daytonaWarm === true,
inheritEnvironment: options.daytonaInherit === true, lifecycle: options.lifecycle });
const holders = retains ? Math.max(options.concurrency, options.tasks) : options.concurrency;
if (holders * options.daytonaCpu > budget) {
if (retains) throw new Error(`${options.daytonaWarm ? "--daytona-warm" : `--daytona-inherit with --lifecycle ${options.lifecycle}`} keeps each task's sandbox for the idle window: ${holders} tasks × `
+ `--daytona-cpu ${options.daytonaCpu} vCPU exceeds the Daytona vCPU budget of ${budget}. Lower --tasks or --daytona-cpu `
+ "(RESTART_CHAOS_DAYTONA_VCPU_BUDGET sets the budget).");
throw new Error(`--concurrency ${options.concurrency} × --daytona-cpu ${options.daytonaCpu} vCPU exceeds the Daytona vCPU budget of ${budget} `
+ "for chaos tasks. Lower one of them (RESTART_CHAOS_DAYTONA_VCPU_BUDGET sets the budget).");
}
+51
View File
@@ -0,0 +1,51 @@
import { spawnSync } from "node:child_process";
import { writeFileSync } from "node:fs";
import path from "node:path";
import { assertControllerMatchesImage, defaultDaytonaImageCacheRoot, prepareDaytonaImageArtifacts } from "./daytona-image.js";
import { PLUGIN_BUILD_STAMP, pluginBuildState } from "./plugin-build.js";
import type { Timeline } from "./timeline.js";
function run(command: string, args: string[], cwd: string, label: string) {
const result = spawnSync(command, args, { cwd, stdio: "inherit" });
if (result.status !== 0) throw new Error(`${label} failed (${command} ${args.join(" ")})`);
}
/**
* Mirrors restart-chaos main.ts's Daytona preparation for the TTFT driver.
* The bundled Daytona plugin is outside the root workspace; prepare it as CI
* does, and rebuild it whenever its sources (or the plugin SDK's) changed since
* the last build, so a worktree that moved to a new master tests that master's
* plugin rather than an older build.
*/
export function ensureDaytonaPlugin(repositoryRoot: string, timeline: Timeline) {
const root = path.join(repositoryRoot, "packages/plugins/sandbox-providers/daytona");
// Inputs outside the plugin directory: the linked plugin SDK and the tsconfig chain both packages extend.
const extraInputs = [
...["src", "package.json", "tsconfig.json"].map(file => path.join(repositoryRoot, "packages/plugins/sdk", file)),
...["tsconfig.json", "tsconfig.base.json"].map(file => path.join(repositoryRoot, file)),
];
const state = pluginBuildState(root, extraInputs);
timeline.add("daytona_plugin_build_state", { reason: state.reason, inputHash: state.inputHash });
if (!state.needsBuild) return;
run("pnpm", ["install", "--ignore-workspace", "--frozen-lockfile", "--ignore-scripts"], root, "Daytona plugin install");
run(process.execPath, [path.join(repositoryRoot, "scripts/link-plugin-dev-sdk.mjs")], repositoryRoot, "Plugin SDK link");
run("pnpm", ["--dir", root, "build"], repositoryRoot, "Daytona plugin build");
writeFileSync(path.join(root, PLUGIN_BUILD_STAMP), `${state.inputHash}\n`);
timeline.add("daytona_plugin_built", { inputHash: state.inputHash });
}
/**
* Controller environment for `--env daytona`: extract the image's Linux runnerd
* and provider pack (cached per image digest, no Docker), refuse a controller
* whose runner source differs from the image, and build the Daytona plugin.
*/
export async function prepareDaytonaController(input: { image: string; repositoryRoot: string; timeline: Timeline }):
Promise<Record<string, string>> {
const artifacts = await prepareDaytonaImageArtifacts({ image: input.image, cacheRoot: defaultDaytonaImageCacheRoot(),
log: line => console.info(`[daytona-image] ${line}`) });
const { localContentId } = await assertControllerMatchesImage(artifacts, input.repositoryRoot);
input.timeline.add("daytona_image_matched", { image: input.image, contentId: artifacts.contentId, localContentId, sourceRevision: artifacts.sourceRevision,
runnerdSha256: artifacts.runnerdSha256, providerPackDigest: artifacts.providerPackDigest, platformManifestDigest: artifacts.platformManifestDigest });
ensureDaytonaPlugin(input.repositoryRoot, input.timeline);
return { PAPERCLIP_RUNNER_REMOTE_BINARY_PATH: artifacts.runnerdPath, PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH: artifacts.providerPackPath };
}
+1
View File
@@ -330,6 +330,7 @@ async function main(): Promise<number> {
const api: ChaosApi = new TimeShiftedApi(supervisor.baseUrl, supervisor.agingShifts);
const lab = await seedChaosLab({ api, env: plan.options.env, waitForReadyAgents: true, cancelled: () => interrupted, daytonaImage: plan.options.daytonaImage, daytonaCpu: plan.options.daytonaCpu,
daytonaReuseLease: plan.options.daytonaReuseLease,
daytonaWarm: plan.options.daytonaWarm, daytonaInherit: plan.options.daytonaInherit,
adapters: [...new Set([...plan.tasks.map(task => task.adapter),
...(plan.tasks.some(task => task.workload === "M1") ? plan.options.adapters : [])])],
concurrency: plan.options.concurrency, budgetCents: plan.options.budgetCents, masks: plan.options.masks,
+13
View File
@@ -261,6 +261,19 @@ describe("Daytona options", () => {
expect(() => parseCommand(["--env", "local", "--daytona-cpu", "2"])).toThrow(/only to --env daytona/);
expect(() => parseCommand(["--env", "local", "--daytona-reuse-lease"])).toThrow(/only to --env daytona/);
});
it("counts every task's sandbox against the budget when warm sandboxes outlive their task", () => {
const wave = (...extra: string[]) => parseCommand(["--env", "daytona", "--daytona-image", image, "--tasks", "20", "--concurrency", "4", ...extra]);
expect(() => wave("--daytona-warm")).toThrow(/--daytona-warm keeps each task's sandbox.*20 tasks/);
expect(() => wave("--daytona-inherit", "--lifecycle", "warm")).toThrow(/--daytona-inherit with --lifecycle warm keeps each task's sandbox/);
// --lifecycle default leaves the lifecycle unset everywhere, and a remote run then stays per-turn.
expect(wave("--daytona-inherit", "--lifecycle", "default")).toMatchObject({ mode: "wave" });
// The catalog agents pin per-turn, so an inheriting environment alone releases each sandbox after its turn.
expect(wave("--daytona-inherit")).toMatchObject({ mode: "wave", options: { daytonaInherit: true, tasks: 20 } });
expect(wave("--daytona-inherit", "--lifecycle", "per_turn")).toMatchObject({ mode: "wave" });
expect(() => wave("--daytona-warm", "--daytona-inherit")).toThrow(/cannot be combined/);
expect(() => parseCommand(["--env", "local", "--daytona-warm"])).toThrow(/only to --env daytona/);
expect(() => parseCommand(["--env", "local", "--daytona-inherit"])).toThrow(/only to --env daytona/);
});
it("keeps reusable Daytona leases only when asked", () => {
const reuse = parseCommand(["--env", "daytona", "--daytona-image", image, "--daytona-reuse-lease"]);
expect(reuse).toMatchObject({ mode: "wave", options: { daytonaReuseLease: true } });
+9 -1
View File
@@ -5,7 +5,7 @@ import os from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { ChaosApi } from "./api.js";
import { cloneWorkspaceRepository, createFixtureRepository, seedChaosLab, waitForAgentsReady } from "./seed.js";
import { cloneWorkspaceRepository, createFixtureRepository, inheritingEnvironmentConfig, seedChaosLab, waitForAgentsReady } from "./seed.js";
import { Timeline } from "./timeline.js";
type Call = { method: string; route: string; body?: unknown };
@@ -129,6 +129,14 @@ describe("seedChaosLab RC-06 mask", () => {
});
});
describe("inheritingEnvironmentConfig", () => {
it("drops the lifecycle and lease pins and keeps everything else", () => {
const fixture = { provider: "daytona", image: "img", runnerLifecycleMode: "per_turn", reuseLease: false };
expect(inheritingEnvironmentConfig(fixture)).toEqual({ provider: "daytona", image: "img" });
expect(fixture).toHaveProperty("runnerLifecycleMode", "per_turn");
});
});
describe("cloneWorkspaceRepository", () => {
it("clones a fresh, shallow working copy of the source repository", async () => {
const root = await mkdtemp(path.join(os.tmpdir(), "restart-chaos-clone-"));
+20 -5
View File
@@ -2,7 +2,7 @@ import { execFileSync } from "node:child_process";
import { chmod, mkdir, writeFile } from "node:fs/promises";
import path from "node:path";
import type { RunnerApi } from "../runner-e2e/api.js";
import { runnerEnvironments, runnerProfiles } from "../runner-e2e/catalog.js";
import { daytonaWarmEnvironment, runnerEnvironments, runnerProfiles } from "../runner-e2e/catalog.js";
import { installFixtureDaytonaProvider, waitForFixtureDaytonaProvider } from "../runner-e2e/live-fixtures.js";
import type { MatrixExecution, SecretReference, SecretReferenceMap } from "../runner-e2e/types.js";
import type { ChaosApi } from "./api.js";
@@ -115,16 +115,25 @@ export function chaosDaytonaShape(cpu: number) {
return { cpu, memory: 4, disk: 10 };
}
/** A user's default environment config: the lifecycle and lease reuse are left to the agent. */
export function inheritingEnvironmentConfig(config: Record<string, unknown>): Record<string, unknown> {
const { runnerLifecycleMode: _lifecycle, reuseLease: _reuse, ...rest } = config;
return rest;
}
async function createDaytonaEnvironment(input: { api: ChaosApi; companyId: string; image: string; cpu: number; reuseLease: boolean; waveId: string;
secretRefs: SecretReferenceMap; timeline: Timeline }): Promise<string> {
const fixture = runnerEnvironments.find(candidate => candidate.id === "daytona");
secretRefs: SecretReferenceMap; timeline: Timeline; warm?: boolean; inherit?: boolean }): Promise<string> {
// The warm fixture keeps a reusable sandbox and a warm runner between turns.
const fixture = input.warm ? daytonaWarmEnvironment : runnerEnvironments.find(candidate => candidate.id === "daytona");
if (!fixture) throw new Error("runner-e2e catalog has no daytona environment fixture");
const view = runnerApiView(input.api);
await waitForFixtureDaytonaProvider(view, { companyId: input.companyId, deadlineAt: Date.now() + 120_000 });
const payload = fixture.buildEnvironment({ secretRefs: input.secretRefs, daytonaImage: input.image, executionId: input.waveId });
const shape = chaosDaytonaShape(input.cpu);
// A user's default environment: the agent decides the lifecycle, and nothing pins lease reuse.
const baseConfig = input.inherit ? inheritingEnvironmentConfig(payload.config as Record<string, unknown>) : payload.config as Record<string, unknown>;
const environment = await input.api.post<{ id: string; driver: string }>(`/api/companies/${input.companyId}/environments`,
{ ...payload, config: { ...(payload.config as Record<string, unknown>), ...shape, ...(input.reuseLease ? { reuseLease: true } : {}) } });
{ ...payload, config: { ...baseConfig, ...shape, ...(input.reuseLease ? { reuseLease: true } : {}) } });
input.timeline.add("seed_environment", { environmentId: environment.id, driver: environment.driver, image: input.image, ...shape,
reuseLease: input.reuseLease });
return environment.id;
@@ -182,6 +191,12 @@ export async function seedChaosLab(input: {
timeline: Timeline;
/** Merged into each agent's adapterConfig (for example a runner lifecycle mode). */
adapterConfigOverrides?: Record<string, unknown>;
/** Use the reusable warm Daytona environment fixture (reuseLease, warm runner). */
daytonaWarm?: boolean;
/** Use a Daytona environment that inherits the lifecycle and leaves lease reuse unset, as a user's default does. */
daytonaInherit?: boolean;
/** A real repository to use as the project's primary workspace instead of the tiny fixture. */
repositoryPath?: string;
/** Called as soon as the wave owns external resources that need cleanup. */
onEnvironment?: (lab: Pick<ChaosLab, "companyId" | "environmentId" | "environmentKind">) => void;
}): Promise<ChaosLab> {
@@ -225,7 +240,7 @@ export async function seedChaosLab(input: {
let environmentId: string;
if (input.env === "daytona") {
environmentId = await createDaytonaEnvironment({ api, companyId: company.id, image: input.daytonaImage!, cpu: input.daytonaCpu ?? 4,
reuseLease: input.daytonaReuseLease === true, waveId: input.waveId, secretRefs, timeline });
reuseLease: input.daytonaReuseLease === true, waveId: input.waveId, secretRefs, timeline, warm: input.daytonaWarm, inherit: input.daytonaInherit });
input.onEnvironment?.({ companyId: company.id, environmentId, environmentKind: "daytona" });
if (input.adapters.includes("codex")) await qualifyNativeCodex({ api, companyId: company.id, environmentId, timeline });
} else {
@@ -1,6 +1,6 @@
import { describe, expect, it } from "vitest";
import {
baselineRows, distribution, openShellExecutions, percentile, rankPreTurnSteps, recoverySamples, runMetrics, runWarmth,
baselineRows, daytonaBaselineHolders, daytonaRetainsSandboxes, distribution, parseBaselineEnvironment, parseBaselinePhases, openShellExecutions, percentile, rankPreTurnSteps, recoverySamples, runMetrics, runWarmth,
type BaselineEvent, type BaselineRun,
} from "./ttft-baseline-stats.js";
@@ -77,6 +77,52 @@ describe("ttft baseline statistics", () => {
ttft: { n: 2, p50: 6_000, p95: 12_000 } });
});
it("leaves post-restart and verification runs out of the baseline rows", () => {
const rows = baselineRows([coldRun("a"), run("p", "postfault", [span("task.time_to_first_token", 0, 20_000)]),
run("v", "verify", [span("task.time_to_first_token", 0, 30_000)])]);
expect(rows).toHaveLength(1);
expect(rows[0]).toMatchObject({ warmth: "cold", ttft: { n: 1, p50: 6_000 } });
});
it("counts held Daytona sandboxes per concurrent task, or per sampled task when they stay warm", () => {
const phases = new Set(["coldwarm", "midturn", "restart"]);
expect(daytonaBaselineHolders({ adapters: 3, samples: 5, phases, restarts: 2, retainsSandboxes: false })).toBe(3);
// 5 samples x (coldwarm + midturn) + K1 and K2 for each of 2 restarts = 14 tasks per adapter.
expect(daytonaBaselineHolders({ adapters: 3, samples: 5, phases, restarts: 2, retainsSandboxes: true })).toBe(42);
// Restart tasks do not scale with --samples.
expect(daytonaBaselineHolders({ adapters: 3, samples: 1, phases: new Set(["restart"]), restarts: 2, retainsSandboxes: true })).toBe(12);
expect(daytonaBaselineHolders({ adapters: 3, samples: 5, phases: new Set(["restart"]), restarts: 2, retainsSandboxes: true })).toBe(12);
expect(daytonaBaselineHolders({ adapters: 2, samples: 0, phases: new Set(["postfault"]), restarts: 0, retainsSandboxes: true })).toBe(2);
});
it("treats an inheriting environment as warm only when the agent's lifecycle is not pinned per-turn", () => {
expect(daytonaRetainsSandboxes({ warmEnvironment: true, inheritEnvironment: false, lifecycle: undefined })).toBe(true);
expect(daytonaRetainsSandboxes({ warmEnvironment: false, inheritEnvironment: true, lifecycle: undefined })).toBe(false);
expect(daytonaRetainsSandboxes({ warmEnvironment: false, inheritEnvironment: true, lifecycle: "per_turn" })).toBe(false);
expect(daytonaRetainsSandboxes({ warmEnvironment: false, inheritEnvironment: true, lifecycle: "default" })).toBe(false);
expect(daytonaRetainsSandboxes({ warmEnvironment: false, inheritEnvironment: true, lifecycle: "warm" })).toBe(true);
expect(daytonaRetainsSandboxes({ warmEnvironment: false, inheritEnvironment: false, lifecycle: "warm" })).toBe(false);
});
it("rejects unknown phases and a postfault phase without the coldwarm tasks it follows up on", () => {
expect([...parseBaselinePhases("coldwarm, postfault")]).toEqual(["coldwarm", "postfault"]);
expect([...parseBaselinePhases("verify")]).toEqual(["verify"]);
expect(() => parseBaselinePhases("postfault")).toThrow(/add coldwarm/);
expect(() => parseBaselinePhases("restart,postfualt")).toThrow(/got postfualt/);
expect(() => parseBaselinePhases(" , ")).toThrow(/got nothing/);
});
it("rejects contradictory or out-of-range environment options", () => {
const base = { env: "daytona", image: "img", cpu: "4", warm: false, inherit: false };
expect(parseBaselineEnvironment(base)).toEqual({ env: "daytona", cpu: 4 });
expect(parseBaselineEnvironment({ ...base, env: "local", image: undefined })).toMatchObject({ env: "local" });
expect(() => parseBaselineEnvironment({ ...base, env: "dayton" })).toThrow(/local or daytona/);
expect(() => parseBaselineEnvironment({ ...base, image: undefined })).toThrow(/needs --daytona-image/);
for (const cpu of ["0", "5", "abc", "2.5"]) expect(() => parseBaselineEnvironment({ ...base, cpu })).toThrow(/1 to 4/);
expect(() => parseBaselineEnvironment({ ...base, warm: true, inherit: true })).toThrow(/cannot be combined/);
expect(() => parseBaselineEnvironment({ ...base, env: "local", image: undefined, warm: true })).toThrow(/only to --env daytona/);
});
it("reports first visible activity from run creation next to first text", () => {
const toolFirst = run("t", "warm", [
reuse("reused"),
+57 -3
View File
@@ -16,7 +16,7 @@ export interface BaselineRun {
adapter: string;
environment: string;
/** Driver intent: a new task, a follow-up comment on the same task, or a run crossed by a restart. */
phase: "cold" | "warm" | "restart" | "midturn" | "verify";
phase: "cold" | "warm" | "restart" | "midturn" | "verify" | "postfault";
events: BaselineEvent[];
workspaceCreated?: boolean;
}
@@ -222,8 +222,8 @@ export interface BaselineRow { adapter: string; environment: string; warmth: "co
export function baselineRows(runs: readonly BaselineRun[]): BaselineRow[] {
const groups = new Map<string, { adapter: string; environment: string; warmth: "cold" | "warm"; metrics: RunMetrics[] }>();
for (const run of runs) {
// Restart samples and new-agent verification samples are reported on their own.
if (run.phase === "restart" || run.phase === "verify") continue;
// Restart, post-restart and new-agent verification samples are reported on their own.
if (run.phase === "restart" || run.phase === "verify" || run.phase === "postfault") continue;
const warmth = runWarmth(run);
const key = `${run.adapter}|${run.environment}|${warmth}`;
const group = groups.get(key) ?? { adapter: run.adapter, environment: run.environment, warmth, metrics: [] };
@@ -309,3 +309,57 @@ export function openShellExecutions(events: readonly BaselineEvent[], marker?: s
}
return [...open.entries()].map(([executionId, entry]) => ({ executionId, matched: entry.matched, unlabelled: !entry.labelled }));
}
/**
* Whether finished tasks keep their Daytona sandbox for the idle window. The warm
* fixture always does. An inheriting environment defers to the agent, whose catalog
* profile pins per_turn unless --lifecycle warm replaces it. --lifecycle default
* leaves the agent unset, and a remote run with no lifecycle anywhere is per-turn.
*/
export function daytonaRetainsSandboxes(input: { warmEnvironment: boolean; inheritEnvironment: boolean; lifecycle: string | undefined }): boolean {
if (input.warmEnvironment) return true;
return input.inheritEnvironment && input.lifecycle === "warm";
}
/** Daytona sandboxes a TTFT baseline can hold at once, for the vCPU budget check. */
export function daytonaBaselineHolders(input: { adapters: number; samples: number; phases: ReadonlySet<string>; restarts: number;
retainsSandboxes: boolean }): number {
// The default fixture releases each sandbox after its turn, so only the concurrent tasks hold one.
if (!input.retainsSandboxes) return input.adapters;
// A warm sandbox outlives its task for the idle window and no other task reuses it.
// coldwarm, midturn and verify create one task per sample; restart creates one per
// fault, K1 and K2 for each --restarts, independent of --samples.
const sampledTasks = Math.max(1, input.samples) * ["coldwarm", "midturn", "verify"].filter(phase => input.phases.has(phase)).length;
const restartTasks = input.phases.has("restart") ? 2 * input.restarts : 0;
return input.adapters * Math.max(1, sampledTasks + restartTasks);
}
/** Validates the TTFT driver's environment options; returns the environment. */
const BASELINE_PHASES = ["coldwarm", "midturn", "restart", "verify", "postfault"] as const;
/** `--phases`: known phases only; postfault follows up on the coldwarm tasks, so it needs coldwarm. */
export function parseBaselinePhases(value: string): Set<string> {
const phases = new Set(value.split(",").map(phase => phase.trim()).filter(Boolean));
const unknown = [...phases].filter(phase => !(BASELINE_PHASES as readonly string[]).includes(phase));
if (unknown.length || !phases.size) throw new Error(`--phases takes ${BASELINE_PHASES.join(", ")}; got ${unknown.join(", ") || "nothing"}`);
if (phases.has("postfault") && !phases.has("coldwarm")) {
throw new Error("--phases postfault follows up on the coldwarm tasks after an idle restart; add coldwarm");
}
return phases;
}
export function parseBaselineEnvironment(input: { env: string | undefined; image: string | undefined; cpu: string | undefined;
warm: boolean; inherit: boolean }): { env: "local" | "daytona"; cpu: number } {
const env = input.env ?? "local";
if (env !== "local" && env !== "daytona") throw new Error(`--env must be local or daytona, got ${env}`);
const cpu = Number(input.cpu ?? "4");
if (env === "local") {
if (input.warm || input.inherit || input.image) throw new Error("--daytona-* options apply only to --env daytona");
return { env, cpu };
}
if (!input.image) throw new Error("--env daytona needs --daytona-image");
if (!Number.isInteger(cpu) || cpu < 1 || cpu > 4) throw new Error(`--daytona-cpu must be an integer from 1 to 4, got ${input.cpu}`);
// The warm fixture pins its lifecycle; inherit removes the pins. The two contradict.
if (input.warm && input.inherit) throw new Error("--daytona-warm and --daytona-inherit cannot be combined");
return { env, cpu };
}
+145 -16
View File
@@ -21,15 +21,17 @@ import { parseArgs } from "node:util";
import { setTimeout as delay } from "node:timers/promises";
import { releaseInstance } from "../runner-e2e/instance-retention.js";
import { ChaosApi } from "./api.js";
import { CREDENTIAL_FOR_ADAPTER } from "./config.js";
import { CREDENTIAL_FOR_ADAPTER, DAYTONA_VCPU_BUDGET } from "./config.js";
import { ACTIVE_RUN_STATUSES } from "./grader.js";
import { chaosLifecycleOverrides, seedChaosLab, type ChaosLab } from "./seed.js";
import { ChaosSupervisor } from "./supervisor.js";
import { createDb } from "../../packages/db/src/index.js";
import { prepareDaytonaController } from "./daytona-controller.js";
import { DaytonaWaveSweeper } from "./daytona-sweep.js";
import { Timeline } from "./timeline.js";
import type { AdapterId, FaultKind } from "./types.js";
import {
baselineRows, rankPreTurnSteps, recoverySamples, distribution, openShellExecutions,
baselineRows, rankPreTurnSteps, recoverySamples, distribution, openShellExecutions, daytonaBaselineHolders, daytonaRetainsSandboxes, parseBaselineEnvironment, parseBaselinePhases,
type BaselineEvent, type BaselineFault, type BaselineRun,
} from "./ttft-baseline-stats.js";
@@ -89,9 +91,13 @@ async function steerQueuedComment(api: ChaosApi, issueId: string, commentId: str
return { acknowledgedAt: null, disposition };
}
/** Set by --workspace-repo: tasks then run in the project's isolated workspace of that repository. */
let projectScopedTasks = false;
async function createTask(api: ChaosApi, lab: ChaosLab, adapter: AdapterId, title: string, description: string) {
const issue = await api.post<Json>(`/api/companies/${lab.companyId}/issues`, {
title, description, status: "todo", priority: "medium", assigneeAgentId: lab.agents[adapter].id,
...(projectScopedTasks ? { projectId: lab.projectId } : {}),
});
return String(issue.id);
}
@@ -201,6 +207,11 @@ function report(runs: BaselineRun[], faults: BaselineFault[]): string {
lines.push("", "New agent behind harness verification (TTFT counts from run creation; the verification wait precedes it):", "", lines[0]!, lines[1]!);
for (const row of baselineRows(verifyRuns)) lines.push(rowLine(row));
}
const postfault = runs.filter(run => run.phase === "postfault").map(run => ({ ...run, phase: "cold" as const }));
if (postfault.length) {
lines.push("", "First follow-up after an idle control-plane restart (K1):", "", lines[0]!, lines[1]!);
for (const row of baselineRows(postfault)) lines.push(rowLine(row));
}
lines.push("", "| Pre-turn step | n | p50 | p95 | Adapters |", "|---|---|---|---|---|");
for (const step of rankPreTurnSteps(runs).slice(0, 25)) lines.push(`| \`${step.step}\` | ${step.n} | ${ms(step.p50)} | ${ms(step.p95)} | ${step.adapters.join(", ")} |`);
@@ -238,6 +249,17 @@ async function main() {
"midturn-fault": { type: "string" },
// verify phase: fault the control plane while a newly hired agent runs harness verification.
"verify-fault": { type: "string" },
// A real repository (for example a fresh clone of paperclip) as the project's
// primary workspace; every task then runs in its isolated git worktree.
"workspace-repo": { type: "string" },
// local (default) or daytona; daytona needs --daytona-image (the signed Runner image).
env: { type: "string", default: "local" },
"daytona-image": { type: "string" },
"daytona-cpu": { type: "string", default: "4" },
// Use the reusable warm Daytona environment (reuseLease, warm runner) instead of the per-turn one.
"daytona-warm": { type: "boolean", default: false },
// A Daytona environment as a user creates it by default: lifecycle inherited, lease reuse unset.
"daytona-inherit": { type: "boolean", default: false },
"idle-ms": { type: "string", default: "600000" },
} });
if (values.report) {
@@ -245,11 +267,28 @@ async function main() {
process.stdout.write(report(data.runs, data.faults));
return;
}
const phases = new Set(values.phases!.split(",").map(value => value.trim()));
const phases = parseBaselinePhases(values.phases!);
const requested = values.adapters!.split(",").map(value => value.trim()) as AdapterId[];
const samples = Math.max(1, Number(values.samples));
const restarts = Math.max(0, Number(values.restarts));
const missing = requested.map(adapter => CREDENTIAL_FOR_ADAPTER[adapter]).filter(key => !process.env[key]);
const lifecycle = values.lifecycle;
if (lifecycle !== undefined && lifecycle !== "per_turn" && lifecycle !== "warm" && lifecycle !== "default") {
throw new Error("--lifecycle must be per_turn, warm or default");
}
const { env, cpu } = parseBaselineEnvironment({ env: values.env, image: values["daytona-image"], cpu: values["daytona-cpu"],
warm: values["daytona-warm"] === true, inherit: values["daytona-inherit"] === true });
if (env === "daytona") {
const budget = Number(process.env.RESTART_CHAOS_DAYTONA_VCPU_BUDGET ?? DAYTONA_VCPU_BUDGET);
const retainsSandboxes = daytonaRetainsSandboxes({ warmEnvironment: values["daytona-warm"] === true,
inheritEnvironment: values["daytona-inherit"] === true, lifecycle });
const holders = daytonaBaselineHolders({ adapters: requested.length, samples, phases, restarts, retainsSandboxes });
if (holders * cpu > budget) {
throw new Error(`${holders} ${retainsSandboxes ? "sampled tasks (warm sandboxes outlive their task)" : "concurrent tasks"} × --daytona-cpu ${cpu} vCPU can be held at once, `
+ `exceeding the Daytona vCPU budget of ${budget}. Lower --samples, --adapters or --daytona-cpu (RESTART_CHAOS_DAYTONA_VCPU_BUDGET sets the budget).`);
}
}
const missing = [...requested.map(adapter => CREDENTIAL_FOR_ADAPTER[adapter]), ...(env === "daytona" ? ["DAYTONA_API_KEY"] : [])]
.filter(key => !process.env[key]);
if (missing.length) throw new Error(`Missing ${missing.join(", ")}; run: set -a; source ~/.secrets; set +a`);
const midturnMode = values["midturn-mode"];
if (midturnMode !== "queue" && midturnMode !== "steer") throw new Error("--midturn-mode must be queue or steer");
@@ -258,10 +297,6 @@ async function main() {
if (midturnFault && midturnMode !== "steer") throw new Error("--midturn-fault applies to --midturn-mode steer");
const verifyFault = values["verify-fault"] as FaultKind | undefined;
if (verifyFault !== undefined && !["K1", "K2", "K6"].includes(verifyFault)) throw new Error("--verify-fault must be K1, K2 or K6");
const lifecycle = values.lifecycle;
if (lifecycle !== undefined && lifecycle !== "per_turn" && lifecycle !== "warm" && lifecycle !== "default") {
throw new Error("--lifecycle must be per_turn, warm or default");
}
// default removes the catalog's pinned per_turn so the server resolves the
// product default; absent keeps the catalog setting.
const adapterConfigOverrides = lifecycle === "warm"
@@ -278,10 +313,35 @@ async function main() {
const agentLifecycles: Record<string, string> = {};
const midturn: Json[] = [];
const timedOutIssues = new Set<string>();
let sweeper: DaytonaWaveSweeper | null = null;
const workspaceOperations: Array<{ runId: string; adapter: string; phase: string; operation: string; status: string;
durationMs: number | null; command: string | null }> = [];
// Every coldwarm sample that finished at rest gets its own post-restart follow-up.
const followUpTasks: Array<{ adapter: AdapterId; issueId: string; nonce: string; runs: number }> = [];
// One shutdown path for normal completion and for SIGINT/SIGTERM: the server
// runs in a detached process group and the embedded Postgres has no exit hook.
let stopping: Promise<void> | null = null;
const shutdown = () => (stopping ??= supervisor.stop().then(() => false, error => {
// Daytona resources are released while the server still runs, before it stops;
// the same path serves normal completion and SIGINT/SIGTERM.
const releaseDaytona = async () => {
if (!sweeper) return;
const owned = sweeper as DaytonaWaveSweeper;
sweeper = null;
try {
const cleanup = await owned.cleanUp();
timeline.add("daytona_cleanup", cleanup);
await writeFile(path.join(outDir, "sandbox-cleanup.json"), `${JSON.stringify(cleanup, null, 2)}\n`, { mode: 0o600 });
if (cleanup.remainingSandboxes.length || cleanup.errors.length || cleanup.environmentDeleteBlocked || !cleanup.environmentDeleted) {
process.stderr.write(`Daytona cleanup incomplete: remaining ${cleanup.remainingSandboxes.join(", ") || "none"}; `
+ `environment deleted ${cleanup.environmentDeleted}${cleanup.environmentDeleteBlocked ? ` (blocked: ${String(cleanup.environmentDeleteBlocked)})` : ""}; ${cleanup.errors.join("; ")}\n`);
process.exitCode = 1;
}
} catch (error) {
timeline.add("daytona_cleanup_failed", { error: String(error) });
process.exitCode = 1;
}
};
const shutdown = () => (stopping ??= releaseDaytona().then(() => supervisor.stop()).then(() => false, error => {
process.stderr.write(`supervisor stop: ${String(error)}\n`);
return true;
}).then(stopFailed => releaseInstance(supervisor.paths.temporaryRoot, { keep: false, stopFailed, owner: "ttft-baseline" }))
@@ -293,10 +353,18 @@ async function main() {
process.once("SIGINT", onSignal);
process.once("SIGTERM", onSignal);
try {
await supervisor.start();
await supervisor.start(env === "daytona"
? await prepareDaytonaController({ image: values["daytona-image"]!, repositoryRoot, timeline }) : {});
const api = new ChaosApi(supervisor.baseUrl);
const lab = await seedChaosLab({ api, env: "local", adapters: requested, concurrency: requested.length, budgetCents: 3000,
masks: ["rc06"], fixtureRoot: supervisor.paths.temporaryRoot, waveId, timeline, adapterConfigOverrides });
const workspaceRepo = values["workspace-repo"] ? path.resolve(values["workspace-repo"]) : undefined;
projectScopedTasks = Boolean(workspaceRepo);
const lab = await seedChaosLab({ api, env, daytonaImage: values["daytona-image"], daytonaCpu: cpu,
daytonaWarm: values["daytona-warm"], daytonaInherit: values["daytona-inherit"],
adapters: requested, concurrency: requested.length, budgetCents: 3000,
masks: ["rc06"], fixtureRoot: supervisor.paths.temporaryRoot, waveId, timeline, adapterConfigOverrides, workspaceRepo,
onEnvironment: owned => {
sweeper = new DaytonaWaveSweeper({ apiKey: process.env.DAYTONA_API_KEY!, companyId: owned.companyId, environmentId: owned.environmentId, api });
} });
const adapters = await readyAdapters(api, lab, requested, timeline);
if (!adapters.length) throw new Error("No agent passed lifecycle verification");
// Record what each agent actually stores: "unset" means the server resolves
@@ -332,6 +400,7 @@ async function main() {
continue;
}
timeline.add("baseline_warm_done", { adapter, issueId, runs: warm.runs.length - cold.runs.length });
followUpTasks.push({ adapter, issueId, nonce, runs: warm.runs.length });
}
}));
@@ -346,7 +415,8 @@ async function main() {
const hired = await api.post<Json>(`/api/companies/${lab.companyId}/agents`, { ...source.body, name: `${source.name} verify ${nonce}` });
const agentId = String(hired.id);
const issue = await api.post<Json>(`/api/companies/${lab.companyId}/issues`, {
title: `TTFT verify ${nonce}`, description: coldTask(nonce), status: "todo", priority: "medium", assigneeAgentId: agentId });
title: `TTFT verify ${nonce}`, description: coldTask(nonce), status: "todo", priority: "medium", assigneeAgentId: agentId,
...(projectScopedTasks ? { projectId: lab.projectId } : {}) });
const issueId = String(issue.id);
tasks.push({ issueId, adapter, phase: "verify" });
let fault: Awaited<ReturnType<typeof supervisor.fault>> | null = null;
@@ -387,6 +457,45 @@ async function main() {
}
}
// First turn after an idle control-plane restart: K1 while nothing runs,
// then one more follow-up on each task the cold/warm phase left at rest.
// The fault must be idle: no timed-out sample may still be running.
// A failed runs read counts as not at rest: the fault must never land on a live turn.
const allAtRest = timedOutIssues.size === 0 && (await Promise.all(tasks.map(async task =>
(await api.get<Json[]>(`/api/issues/${task.issueId}/runs`, 10_000).catch(() => null))
?.every(run => !ACTIVE_RUN_STATUSES.has(String(run.status))) ?? false))).every(Boolean);
if (phases.has("postfault") && followUpTasks.length && !allAtRest) {
timeline.add("baseline_postfault_skipped", { reason: "work_not_at_rest", timedOut: timedOutIssues.size });
}
if (phases.has("postfault") && followUpTasks.length && allAtRest) {
// One idle K1 per round, then the first follow-up on one task per adapter, so each
// sample is a genuine first turn after a restart. A timed-out follow-up may still be
// running, so it ends the rounds: the next fault must stay idle.
const rounds = Math.max(...adapters.map(adapter => followUpTasks.filter(entry => entry.adapter === adapter).length));
for (let round = 0; round < rounds; round += 1) {
const batch = adapters.flatMap(adapter => followUpTasks.filter(entry => entry.adapter === adapter).slice(round, round + 1));
const receipt = await supervisor.fault("K1", 0, `ttft-baseline-postfault-${round}`);
faults.push({ kind: "K1", readyAt: receipt.readyAt ?? Date.now(), runIds: [] });
let timedOut = false;
await Promise.all(batch.map(async entry => {
const { adapter } = entry;
await api.post(`/api/issues/${entry.issueId}/comments`, { body: warmFollowUp(`${entry.nonce}-postfault`), reopen: true });
const after = await waitForRest(api, supervisor, entry.issueId, entry.runs + 1);
for (const run of after.runs) if (!runPhase.has(runIdOf(run))) runPhase.set(runIdOf(run), "postfault");
if (after.timedOut) {
timedOut = true;
timedOutIssues.add(entry.issueId);
timeline.add("baseline_sample_timeout", { adapter, issueId: entry.issueId, phase: "postfault" });
}
else timeline.add("baseline_postfault_done", { adapter, issueId: entry.issueId, round });
}));
if (timedOut) {
if (round + 1 < rounds) timeline.add("baseline_postfault_skipped", { reason: "work_not_at_rest", round: round + 1 });
break;
}
}
}
// Mid-turn follow-up: a comment posted while the agent is inside a long
// tool call. Records when the comment landed and every run of the task,
// so the wait splits into "rest of the turn" and "hand-off".
@@ -476,15 +585,34 @@ async function main() {
if (timedOutIssues.has(task.issueId)) continue;
for (const run of await runsFor(api, task.issueId)) {
const runId = runIdOf(run);
runs.push({ runId, adapter: task.adapter, environment: "local",
runs.push({ runId, adapter: task.adapter, environment: env,
phase: runPhase.get(runId) ?? task.phase, events: await allEvents(api, runId) });
// Workspace commands (worktree, provision, sync) the run waited on, with their durations.
const operations = await api.get<Json[]>(`/api/heartbeat-runs/${runId}/workspace-operations`, 10_000).catch((error: unknown) => {
// A failed read is recorded as such, never as "no workspace operations".
workspaceOperations.push({ runId, adapter: task.adapter, phase: runPhase.get(runId) ?? task.phase,
operation: "read_failed", status: error instanceof Error ? error.message.slice(0, 160) : String(error), durationMs: null, command: null });
return [] as Json[];
});
for (const operation of operations) workspaceOperations.push({ runId, adapter: task.adapter,
phase: runPhase.get(runId) ?? task.phase, operation: String(operation.phase), status: String(operation.status),
durationMs: operation.startedAt && operation.finishedAt
? Date.parse(String(operation.finishedAt)) - Date.parse(String(operation.startedAt)) : null,
command: typeof operation.command === "string" ? operation.command.slice(0, 120) : null });
}
}
const data = { waveId, lifecycle: lifecycle ?? "catalog", agentLifecycles, adapters, notReady: requested.filter(adapter => !adapters.includes(adapter)),
const data = { workspaceOperations, waveId, lifecycle: lifecycle ?? "catalog", agentLifecycles, adapters, notReady: requested.filter(adapter => !adapters.includes(adapter)),
timedOutTasks: timedOutIssues.size, samples, restarts, midturn, gitSha: process.env.GIT_SHA ?? null, runs, faults };
await writeFile(path.join(outDir, "baseline-data.json"), `${JSON.stringify(data)}\n`, { mode: 0o600 });
await writeFile(path.join(outDir, "baseline-report.md"), report(runs, faults), { mode: 0o600 });
process.stdout.write(report(runs, faults));
if (workspaceOperations.length) {
const table = ["", "| Run phase | Adapter | Workspace operation | Status | Duration | Command |", "|---|---|---|---|---|---|",
...workspaceOperations.slice().sort((a, b) => (b.durationMs ?? 0) - (a.durationMs ?? 0))
.map(op => `| ${op.phase} | ${op.adapter} | ${op.operation} | ${op.status} | ${op.durationMs === null ? "—" : `${(op.durationMs / 1000).toFixed(2)} s`} | ${op.command ? `\`${op.command}\`` : "—"} |`)].join("\n");
process.stdout.write(`${table}\n`);
await writeFile(path.join(outDir, "workspace-operations.md"), `${table}\n`, { mode: 0o600 });
}
if (timedOutIssues.size) process.stdout.write(`\n${timedOutIssues.size} sample(s) timed out and were left out.\n`);
// Every answered mid-turn comment must be answered exactly once.
const wrongReplies = midturn.filter(sample => sample.replyCount !== 1);
@@ -495,7 +623,8 @@ async function main() {
if (wrongReplies.length) process.exitCode = 1;
process.stdout.write(`\nWrote ${outDir}\n`);
} finally {
// Always stop the wrapper-owned embedded Postgres (SysV shared memory is scarce).
// Every Daytona sandbox this run created is archived or destroyed before the server
// stops, then the wrapper-owned embedded Postgres stops (SysV shared memory is scarce).
await shutdown();
process.off("SIGINT", onSignal);
process.off("SIGTERM", onSignal);
+4
View File
@@ -59,6 +59,10 @@ export interface WaveOptions {
daytonaCpu?: number;
/** `--daytona-reuse-lease`: the chaos environment keeps its sandboxes between turns (a per-turn run stops and keeps its sandbox). */
daytonaReuseLease?: boolean;
/** `--daytona-warm`: the reusable warm Daytona environment fixture. */
daytonaWarm?: boolean;
/** `--daytona-inherit`: a Daytona environment that inherits the lifecycle and leaves lease reuse unset. */
daytonaInherit?: boolean;
/** `--k4-hold-max-seconds`: upper end of the seeded K4 hold (default 300 s, cap 600 s). */
k4HoldMaxMs?: number;
faultAfterMs?: number;