test(chaos): run waves in Cloud's volume-free server mode; make W2 phase 1 yield (#20930)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) runs real agent
tasks against an isolated control plane and injects faults
> - Cloud servers run in the volume-free mode
(`PAPERCLIP_EPHEMERAL_SERVER=true`, durable agent files, run logs in
object storage)
> - The harness never set that mode, so every wave ran a different
server mode from staging. Paths that exist only in that mode were never
tested. One of them is the per-agent directory operation lock, so
`--fail-recovery-step directory-lock` could never reach its step with
any workload
> - The re-wake workloads W2 and W2C also lost coverage: an agent that
reported phase 1 with disposition `done` closed the task, and the
harness never sent the re-wake (four of eight Claude cells on
2026-11-11)
> - This pull request adds `--ephemeral-server` and makes the W2/W2C
brief ask for a `yielded` phase-1 report
> - The benefit is that chaos waves can run in staging's server mode and
the re-wake cells test the re-wake on every provider

## Linked Issues or Issue Description

**What is the problem?**
The restart chaos harness cannot run the control plane in the
volume-free mode that Cloud and staging use. The `directory-lock`
recovery step (`agent_directory.operation_lock.before_acquire`) is
reached only for disposable remote copies, which need
`PAPERCLIP_EPHEMERAL_SERVER=true`. W2 and W2C cells also stop before the
re-wake when the agent reports phase 1 as `done`.

**What is the expected behavior?**
A wave can opt in to the volume-free mode on Daytona, and every restart
of the wave keeps that mode. The W2 and W2C cells reach the re-wake.

**Steps to reproduce**
Run `--fail-recovery-step
agent_directory.operation_lock.before_acquire:1` on Daytona with any
workload: the check `recovery_step_failed_once` reports "never reached".
Run W2 on Claude: the first run ends with the task `done` and
`rewake_sent` is "never sent".

## What Changed

- `assertIsolatedServerEnvironment` (`tests/runner-e2e/harness-env.ts`)
takes `expected.generatedSecretsMasterKey`. It admits
`PAPERCLIP_SECRETS_MASTER_KEY` only when that parameter is set and
equals the value exactly. `PAPERCLIP_SECRETS_MASTER_KEY_FILE` stays
refused. The check runs over the final server environment.
- `--k15-fresh-home` implies `--ephemeral-server` and defaults
`--storage` to `s3`. A fresh instance home without the volume-free mode
loses the agents' instruction bundles, which staging keeps in Postgres
and object storage.
- `--ephemeral-server` (`config.ts`, `types.ts`, `main.ts`,
`supervisor.ts`, new `ephemeral-server.ts` with its unit test
`ephemeral-server.test.ts`): it needs `--storage s3` and `--env daytona`
and refuses K14. Every server of the wave gets
`PAPERCLIP_EPHEMERAL_SERVER`, `PAPERCLIP_DURABLE_AGENT_FILES`,
`PAPERCLIP_SHARED_UI_ASSETS`, the wave's Postgres as
`PAPERCLIP_DATABASE_LISTEN_URL`, one stable
`PAPERCLIP_SECRETS_MASTER_KEY` (also written to the config key file) and
`RUN_LOG_S3_*` in the wave's object store.
- `rewake.ts`: the W2/W2C phase-1 brief asks for `paperclip_finish` with
disposition `yielded` and a `response_wake` continuation.
- W2/W2C tasks are created in `backlog` with their assignee, and a board
comment starts phase 1 (`startComment` in `workloads.ts`, used by
`observer.ts`). A passive board wait needs a comment-started run;
without it the yield queues an immediate continuation and phase 1 wakes
itself. The start comment is posted once (saved-comment check plus a
stable `clientRequestId`), and the brief keeps phase 2 out of blocking
`remainingWork`.
- Volume-free mode drops the repository workloads (B1, B5, F1, R1) from
the default list and refuses them when listed: their fixture is a
`local_path` project source, which a volume-free server refuses.
- `README.md`: a section for the new flag and a note on the brief.
- `shape-cells.test.ts`: option parsing and refusals for the new flag.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/shape-cells.test.ts` passes (16 tests).
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes.
- Live proof wave (Daytona, Codex, W2, K1 and K15 with a fresh home,
`--storage s3 --ephemeral-server`): wave `w-20261111T205259-s6301` (2
tasks). The first server and both replacements (K15 with a fresh home,
then K1) started and reached ready in the volume-free mode. The server
process had `PAPERCLIP_EPHEMERAL_SERVER=true`,
`PAPERCLIP_DURABLE_AGENT_FILES=true` and `RUN_LOG_S3_BUCKET` set. Both
W2 tasks left phase 1 `in_progress` (yielded), received the re-wake and
ended `done`. The wave also found product faults (reported separately,
not fixed here): a re-wake after K1 failed with
`runner_harness_state_mismatch: backup_without_reusable_lease` and
needed Retry.
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/support.test.ts` passes, including the new master-key
admission test (a matching wave key passes; an inherited key, a
mismatched key and `PAPERCLIP_SECRETS_MASTER_KEY_FILE` are refused).

## Risks

- Low risk. The flag is opt-in and the harness is test-only. The brief
change alters the W2/W2C prompt, so results from older W2/W2C seeds are
not directly comparable.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

tests/restart-chaos/README.md

@@ -26,7 +26,7 @@ Options: `--triggers time,comment,retry,queued,tool,result` (fault timing, below
 default 3000), `--poll-ms` (2000–5000), `--keep` (keep the temporary instance),
 `--k4-hold-max-seconds` (raise the seeded K4 hold upper end from 300 s, up to the 600 s outage cap).
 `--faults K0` plans cells with no fault, for workloads whose shape is the fault (W2, below). `--fail-recovery-step`,
-`--long-turn-minutes`, `--k15-overlap-seconds` and `--k15-fresh-home` are described in their own sections.
+`--long-turn-minutes`, `--k15-overlap-seconds`, `--k15-fresh-home` and `--ephemeral-server` are described in their own sections.
 `--fault-after-seconds <n>` fires every planned fault `n` seconds after its task launches (5–3600; with `--triggers time`), keeping
 the rest of the seeded plan, for example to restart a Daytona R1 task after its workspace preparation.
 `--matrix` crosses every selected workload with every fault kind (below), and `--server-from <dir>`
@@ -496,6 +496,28 @@ deletes a sandbox. An archived or stopped sandbox is recorded by its state alone
 | `w-20261111T204144-s9503` | CR1 360 s twice, OpenCode and Claude | No refusal. The Runner and the provider survived both outages. |
 | `w-20261111T210055-s9504` | CR3 45 s, OpenCode and Claude | Not the staging code, a separate failure. `runnerd process supervision failed: RPC call "environmentExecute" timed out after 40000ms` failed the run as `notification_transport_failed`. The close path suspended a healthy Runner (`lifecycle: suspended`, no stderr). Retry worked. |
 
+### Volume-free servers (`--ephemeral-server`)
+
+```sh
+node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env daytona --daytona-image $IMG --storage s3 --ephemeral-server \
+  --adapters codex --workloads W2 --faults K1,K2,K15 --k15-fresh-home --k15-overlap-seconds 30-90 --concurrency 3 --tasks 3 --seed 5501
+```
+
+Cloud servers run volume-free (`ephemeral-server-contract.ts`): agent files, run logs and UI assets live in object
+storage, secrets use one stable master key, and remote agent directories are disposable copies whose collection and
+cleanup take the per-agent Postgres operation lock. Every other wave runs the volume-backed mode, so those paths,
+including the `directory-lock` recovery step, are never reached there. `--ephemeral-server` gives every server of the
+wave `PAPERCLIP_EPHEMERAL_SERVER`, `PAPERCLIP_DURABLE_AGENT_FILES` and `PAPERCLIP_SHARED_UI_ASSETS`, the wave's
+Postgres as `PAPERCLIP_DATABASE_LISTEN_URL`, a stable `PAPERCLIP_SECRETS_MASTER_KEY` (also written to the config's
+key file inside the wave's temporary root; `assertIsolatedServerEnvironment` admits that one generated key and still
+refuses an inherited key or `PAPERCLIP_SECRETS_MASTER_KEY_FILE`) and `RUN_LOG_S3_*` in the wave's object store. It needs
+`--storage s3` and `--env daytona` (a local workspace blocks a volume-free server) and does not combine with K14. Workloads with the fixture repository (B1, B5,
+F1, R1) are a `local_path` project source, which a volume-free server refuses: they are dropped from the default
+workloads and refused when listed.
+`--k15-fresh-home` implies it (and defaults `--storage` to `s3`): a new instance home without the volume-free mode
+loses the agents' instruction bundles, which staging keeps in Postgres and object storage. Every restart replays the server's startup
+eligibility check, so a replacement that refuses to start fails the wave.
+
 ## Absolute symlink in the workspace (F1)
 
 ```sh
@@ -1084,7 +1106,12 @@ timed out`, `runner did not durably suspend before checkpoint` (class C+), or
 
 The workload is the fault, so these cells usually run with `--faults K0` (no fault). Every task of an
 adapter runs on that adapter's one agent: `--adapters codex --tasks 10 --concurrency 10` is ten tasks on one
-agent. Phase 1 ends with the task left `in_progress` (`in_review` would gate completion on a review). Then the harness, as the board, wakes the same task the
+agent. Phase 1 ends with the task left `in_progress` (`in_review` would gate completion on a review): the W2 and W2C
+brief asks the agent to report phase 1 with `paperclip_finish` disposition `yielded` and a `response_wake`
+continuation, because an agent that reports `done` closes the task and the wake is never sent (on 2026-11-11 four of
+eight Claude cells tested nothing). A passive board wait needs a run that a board comment started, so W2 and W2C
+create the task in `backlog` and start phase 1 with a board comment; without it the yield queues an immediate
+continuation and phase 1 wakes itself. Then the harness, as the board, wakes the same task the
 seeded 0–60 s after that run settled (`rewake.ts`):
 
 - **W2**: a board comment, or, by the task's nonce, a board-scheduled issue monitor due now, so the

tests/restart-chaos/config.ts

@@ -9,6 +9,7 @@ import { parseK15DeployShape, parseK15HoldSeconds } from "./redeploy.js";
 import { validateRecoveryStep } from "./recovery-step-faults.js";
 import { parseLongTurnMinutes } from "./long-turn.js";
 import { isRewakeWorkload } from "./rewake.js";
+import { workloadDefinitions } from "./workloads.js";
 import { parseClassROptions } from "./class-r.js";
 import { parseTimeFaultOptions } from "./time-faults-config.js";
 import { parseStorageTrigger } from "./storage-trigger.js";
@@ -143,6 +144,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
       "k15-hold-seconds": { type: "string" },
       "k15-overlap-seconds": { type: "string" },
       "k15-fresh-home": { type: "boolean" },
+      "ephemeral-server": { type: "boolean" },
       "tm1-hold-seconds": { type: "string" },
       "fail-recovery-step": { type: "string" },
       "long-turn-minutes": { type: "string" },
@@ -156,7 +158,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
   const pollMs = integer(values["poll-ms"], "poll-ms", 3_000, 2_000, 5_000);
   const budgetCents = integer(values["budget-cents"], "budget-cents", 3_000, 1, 100_000);
   if (values.replay !== undefined) {
-    for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-pack-mismatch", "daytona-cpu", "daytona-reuse-lease", "daytona-warm", "daytona-inherit", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds", "storage-trigger", "native-trigger", "native-trigger-outage", "fault-after-seconds", "provider-fault-message", "provider-fault-delays", "db-pressure", "matrix", "server-from", "time-age-hours", "provider-fault-restart", "provider-fault-restart-after-seconds", "rr-gap-seconds", "rr-stop-on", "k15-hold-seconds", "k15-overlap-seconds", "k15-fresh-home", "tm1-hold-seconds", "fail-recovery-step", "long-turn-minutes", "cr1-hold-seconds", "cr2-hold-seconds", "cr3-stall-seconds", "sandbox-watch-seconds"] as const) {
+    for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-pack-mismatch", "daytona-cpu", "daytona-reuse-lease", "daytona-warm", "daytona-inherit", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds", "storage-trigger", "native-trigger", "native-trigger-outage", "fault-after-seconds", "provider-fault-message", "provider-fault-delays", "db-pressure", "matrix", "server-from", "time-age-hours", "provider-fault-restart", "provider-fault-restart-after-seconds", "rr-gap-seconds", "rr-stop-on", "k15-hold-seconds", "k15-overlap-seconds", "k15-fresh-home", "ephemeral-server", "tm1-hold-seconds", "fail-recovery-step", "long-turn-minutes", "cr1-hold-seconds", "cr2-hold-seconds", "cr3-stall-seconds", "sandbox-watch-seconds"] as const) {
       if (values[key] !== undefined) throw new Error(`--replay reproduces the saved plan; do not combine it with --${key}`);
     }
     return { mode: "replay", ref: parseReplayRef(values.replay), keep,
@@ -205,11 +207,12 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
   const diskFullFaults = allFaults.filter(isDiskFullFault);
   const sessionFaults = allFaults.filter(isSessionFault);
   const baseline = allFaults.includes("K0");
+  const volumeFree = values["ephemeral-server"] === true || values["k15-fresh-home"] === true;
   const options: WaveOptions = {
     env,
     adapters,
     // The governance scenarios (G1, H1), the journeys (Q1–Q4) and the contention run (L1) are opt-in: H1 needs a wave of its own.
-    workloads: list<WorkloadId>(values.workloads, workloadChoices, "workloads", workloadChoices.filter(id => id !== "G1" && id !== "H1" && id !== "R1" && id !== "S1" && id !== "L1" && id !== "F1" && id !== "T1" && id !== "Q4" && id !== "H2"
+    workloads: list<WorkloadId>(values.workloads, workloadChoices, "workloads", workloadChoices.filter(id => !(volumeFree && workloadDefinitions[id].usesRepository) && id !== "G1" && id !== "H1" && id !== "R1" && id !== "S1" && id !== "L1" && id !== "F1" && id !== "T1" && id !== "Q4" && id !== "H2"
       && !takesJourneyTriggers(id) && !isRewakeWorkload(id) && (id !== "M1" || adapters.length >= 2))),
     // A crash-point wave schedules no fault; K1 is the restart its self-kill gets.
     faults: allFaults.filter((fault): fault is FaultKind => !isProviderFault(fault) && !isDiskFullFault(fault) && !isSessionFault(fault) && fault !== "K0"),
@@ -331,7 +334,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
   }
   applyJourneyOptions(options, { matrix: values.matrix === true, serverFrom: values["server-from"], tasksGiven: values.tasks !== undefined,
     triggersGiven: values.triggers !== undefined });
-  const storage = values.storage ?? "local";
+  const storage = values.storage ?? (values["k15-fresh-home"] ? "s3" : "local");
   const workspaceRepo = values["workspace-repo"];
   if (options.workloads.includes("R1") !== (workspaceRepo !== undefined)) {
     throw new Error("R1 (code change in a cloned repository) and --workspace-repo <git repository> go together: pass both or neither");
@@ -384,6 +387,15 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
   if (values["db-pressure"] !== undefined) options.dbPressure = integer(values["db-pressure"], "db-pressure", 0, 1, 8);
   if (storage !== "local" && storage !== "s3") throw new Error("--storage must be local or s3");
   if (storage === "s3") options.storage = "s3";
+  if (values["ephemeral-server"] || options.k15FreshHome) {
+    const flag = values["ephemeral-server"] ? "--ephemeral-server" : "--k15-fresh-home (a volume-free deploy)";
+    if (storage !== "s3") throw new Error(`${flag} runs volume-free and needs shared object storage: pass --storage s3`);
+    if (env !== "daytona") throw new Error(`${flag} runs only with --env daytona: a local workspace blocks a volume-free server`);
+    if (options.faults.includes("K14")) throw new Error(`${flag} does not combine with K14 (--upgrade-from)`);
+    const repository = options.workloads.filter(workload => workloadDefinitions[workload].usesRepository);
+    if (repository.length) throw new Error(`${flag} cannot run ${repository.join(", ")}: the fixture repository is a local_path project source, which a volume-free server refuses; pick other --workloads`);
+    options.ephemeralServer = true;
+  }
   if (values["storage-faults"] !== undefined) {
     if (storage !== "s3") throw new Error("--storage-faults needs --storage s3");
     options.storageFaults = list<StorageFaultKind>(values["storage-faults"], STORAGE_FAULT_KINDS, "storage-faults", []);

tests/restart-chaos/ephemeral-server.test.ts

@@ -0,0 +1,45 @@
+import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { afterEach, describe, expect, it } from "vitest";
+import { ephemeralServerEnvironment } from "./ephemeral-server.js";
+
+const roots: string[] = [];
+afterEach(async () => { await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); });
+
+async function fixture(keyFilePath?: (root: string) => string) {
+  const root = await mkdtemp(path.join(os.tmpdir(), "chaos-ephemeral-"));
+  roots.push(root);
+  const configPath = path.join(root, "home", "instances", "default", "config.json");
+  await mkdir(path.dirname(configPath), { recursive: true });
+  const keyFile = keyFilePath ? keyFilePath(root) : path.join(path.dirname(configPath), "secrets", "master.key");
+  await writeFile(configPath, JSON.stringify({ database: { connectionString: "postgres://u:p@127.0.0.1:5432/paperclip" },
+    secrets: { localEncrypted: { keyFilePath: keyFile } } }));
+  return { root, configPath, keyFile };
+}
+
+const storage = { PAPERCLIP_STORAGE_PROVIDER: "s3", PAPERCLIP_STORAGE_S3_BUCKET: "chaos-b", PAPERCLIP_STORAGE_S3_ENDPOINT: "http://127.0.0.1:9000",
+  PAPERCLIP_STORAGE_S3_REGION: "us-east-1", PAPERCLIP_STORAGE_S3_PREFIX: "chaos/w1/" };
+
+describe("ephemeralServerEnvironment", () => {
+  it("builds the volume-free environment and keeps one master key across restarts", async () => {
+    const { root, configPath, keyFile } = await fixture();
+    const first = await ephemeralServerEnvironment({ configPath, temporaryRoot: root, extraEnvironment: storage });
+    expect(first).toMatchObject({ PAPERCLIP_EPHEMERAL_SERVER: "true", PAPERCLIP_DURABLE_AGENT_FILES: "true", PAPERCLIP_SHARED_UI_ASSETS: "true",
+      PAPERCLIP_DATABASE_LISTEN_URL: "postgres://u:p@127.0.0.1:5432/paperclip", RUN_LOG_S3_BUCKET: "chaos-b",
+      RUN_LOG_S3_ENDPOINT: "http://127.0.0.1:9000", RUN_LOG_S3_REGION: "us-east-1", RUN_LOG_S3_PREFIX: "chaos/w1/run-logs" });
+    expect(Buffer.from(first.PAPERCLIP_SECRETS_MASTER_KEY!, "base64")).toHaveLength(32);
+    expect((await readFile(keyFile, "utf8")).trim()).toBe(first.PAPERCLIP_SECRETS_MASTER_KEY);
+    const second = await ephemeralServerEnvironment({ configPath, temporaryRoot: root, extraEnvironment: storage });
+    expect(second.PAPERCLIP_SECRETS_MASTER_KEY).toBe(first.PAPERCLIP_SECRETS_MASTER_KEY);
+  });
+
+  it("refuses a key file outside the wave's temporary root and a wave without object storage", async () => {
+    const outside = await fixture(() => path.join(os.tmpdir(), "elsewhere", "master.key"));
+    await expect(ephemeralServerEnvironment({ configPath: outside.configPath, temporaryRoot: outside.root, extraEnvironment: storage }))
+      .rejects.toThrow("escapes its temporary root");
+    const local = await fixture();
+    await expect(ephemeralServerEnvironment({ configPath: local.configPath, temporaryRoot: local.root, extraEnvironment: {} }))
+      .rejects.toThrow("--storage s3");
+  });
+});

tests/restart-chaos/ephemeral-server.ts

@@ -0,0 +1,25 @@
+import { randomBytes } from "node:crypto";
+import { mkdir, readFile, writeFile } from "node:fs/promises";
+import path from "node:path";
+import { databaseUrlFromConfig } from "./time-faults.js";
+
+export async function ephemeralServerEnvironment(input: { configPath: string; temporaryRoot: string; extraEnvironment: Record<string, string> }): Promise<Record<string, string>> {
+  const { configPath, temporaryRoot, extraEnvironment } = input;
+  if (extraEnvironment.PAPERCLIP_STORAGE_PROVIDER !== "s3") throw new Error("--ephemeral-server needs the wave's object store (--storage s3)");
+  const config = JSON.parse(await readFile(configPath, "utf8")) as { secrets?: { localEncrypted?: { keyFilePath?: string } } };
+  const keyFile = String(config.secrets?.localEncrypted?.keyFilePath ?? path.join(path.dirname(configPath), "secrets", "master.key"));
+  if (!path.resolve(keyFile).startsWith(`${path.resolve(temporaryRoot)}${path.sep}`)) throw new Error("--ephemeral-server: the wave's master key file escapes its temporary root");
+  let masterKey = await readFile(keyFile, "utf8").then(value => value.trim(), () => "");
+  if (!masterKey) {
+    masterKey = randomBytes(32).toString("base64");
+    await mkdir(path.dirname(keyFile), { recursive: true, mode: 0o700 });
+    await writeFile(keyFile, masterKey, { mode: 0o600, flag: "wx" });
+  }
+  return {
+    PAPERCLIP_EPHEMERAL_SERVER: "true", PAPERCLIP_DURABLE_AGENT_FILES: "true", PAPERCLIP_SHARED_UI_ASSETS: "true",
+    PAPERCLIP_DATABASE_LISTEN_URL: databaseUrlFromConfig(configPath), PAPERCLIP_SECRETS_MASTER_KEY: masterKey,
+    RUN_LOG_S3_BUCKET: extraEnvironment.PAPERCLIP_STORAGE_S3_BUCKET!, RUN_LOG_S3_ENDPOINT: extraEnvironment.PAPERCLIP_STORAGE_S3_ENDPOINT!,
+    RUN_LOG_S3_REGION: extraEnvironment.PAPERCLIP_STORAGE_S3_REGION!, RUN_LOG_S3_FORCE_PATH_STYLE: "true",
+    RUN_LOG_S3_PREFIX: `${extraEnvironment.PAPERCLIP_STORAGE_S3_PREFIX ?? ""}run-logs`,
+  };
+}

tests/restart-chaos/main.ts

@@ -248,6 +248,7 @@ async function main(): Promise<number> {
   const violationKey = (violation: { rule: string; kind: string; recordId: string }) => `${violation.rule}:${violation.kind}:${violation.recordId}`;
   // `--k15-overlap-seconds`, `--k15-fresh-home`: the staging deploy shape (redeploy.ts).
   supervisor.deployShape = { overlap: plan.options.k15Overlap === true, freshHome: plan.options.k15FreshHome === true };
+  supervisor.ephemeralServer = plan.options.ephemeralServer === true;
   // `--fail-recovery-step`: fail the nth hit of one recovery step once, after the wave's first restart fault.
   const recoveryStepReceipts = path.join(waveDir, "recovery-step-receipts.jsonl");
   if (plan.options.failRecoveryStep) {

tests/restart-chaos/observer.ts

@@ -1,5 +1,5 @@
 import { spawnSync } from "node:child_process";
-import { randomUUID } from "node:crypto";
+import { createHash, randomUUID } from "node:crypto";
 import { access, mkdir, mkdtemp, readFile, readdir, rm, writeFile } from "node:fs/promises";
 import os from "node:os";
 import path from "node:path";
@@ -30,6 +30,11 @@ const IGNORED_DIRECTORIES = new Set([".git", ".bench", "__pycache__", "node_modu
 const MAX_SNAPSHOT_FILES = 2_000;
 
 /** Relative paths of workspace files, excluding VCS and cache directories. */
+
+export function stableRequestId(key: string): string {
+  const hex = createHash("sha256").update(key).digest("hex");
+  return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-5${hex.slice(13, 16)}-${(8 + (Number.parseInt(hex[16]!, 16) & 3)).toString(16)}${hex.slice(17, 20)}-${hex.slice(20, 32)}`;
+}
 export async function listWorkspaceFiles(root: string): Promise<string[]> {
   const out: string[] = [];
   const walk = async (relative: string) => {
@@ -255,6 +260,7 @@ export class TaskObserver {
     for (;;) {
       try {
         await this.api.patch(`/api/issues/${this.record.issueId}`, { assigneeAgentId: this.journey?.assigneeAgentId ?? this.lab.agents[this.plan.adapter].id });
+        await this.postStartComment();
         break;
       } catch (error) {
         const transient = error instanceof ChaosApiError && (error.status === null || error.status >= 500);
@@ -272,6 +278,14 @@ export class TaskObserver {
     this.timeline.add("task_released", { taskId: this.plan.taskId, issueId: this.record.issueId });
   }
 
+  private async postStartComment(): Promise<void> {
+    if (!this.definition.startComment || !this.record.issueId) return;
+    const body = this.definition.startComment(this.plan.nonce);
+    const existing = await this.api.get<Json[]>(`/api/issues/${this.record.issueId}/comments`);
+    if (existing.some(comment => comment.body === body)) return;
+    await this.api.post<Json>(`/api/issues/${this.record.issueId}/comments`, { body, clientRequestId: stableRequestId(`start-${this.plan.taskId}-${this.plan.nonce}`) });
+  }
+
   async launch(options: { assign?: boolean } = {}): Promise<void> {
     const definition = this.definition;
     const assign = options.assign !== false;
@@ -282,13 +296,14 @@ export class TaskObserver {
       const issue = await this.api.post<Json>(`/api/companies/${this.lab.companyId}/issues`, {
         title: definition.title(this.plan.nonce),
         description: definition.description(this.plan.nonce, this.workloadAgents()),
-        status: "todo",
+        status: definition.startComment ? "backlog" : "todo",
         priority: "medium",
         ...(assign ? { assigneeAgentId: this.journey?.assigneeAgentId ?? this.lab.agents[this.plan.adapter].id } : {}),
         ...(definition.usesRepository ? { projectId: this.lab.projectId } : {}),
       });
       this.record.issueId = String(issue.id);
       this.record.identifier = issue.identifier ?? null;
+      if (assign) await this.postStartComment();
       // K14: which build launched the task, so a replay starts it on the same one.
       if (this.supervisor.serverBuild) this.record.serverBuildAtLaunch = this.supervisor.serverBuild;
       this.record.createdAt = Date.now();

tests/restart-chaos/rewake.ts

@@ -82,10 +82,13 @@ const phaseTwo = (nonce: string) => [
 
 const w2Description = (nonce: string) => [
   "This task has two phases.",
-  "Phase 1 (now): write a short brief on why idempotent retries matter for background job queues.",
+  "Phase 1 (when the board's comment asks you to start it): write a short brief on why idempotent retries matter for background job queues.",
   "Save it as this issue's document with key `brief` (title \"Brief\"), in markdown.",
   `It must contain the line \`${briefMarker(nonce)}\` exactly, on its own line, and at least three bullet points that start with "- ".`,
   "Then leave this task `in_progress` and end your turn. Do not mark it done or move it to review in phase 1. Do not create files or other tasks.",
+  "Phase 1 does not finish the task: report this turn with `paperclip_finish` and `reportedWorkDisposition` `yielded`, never `done`,",
+  `with a continuation of kind \`response_wake\` (summary "Waiting for the board's phase 2 wake", idempotencyKey \`w2-${nonce}\`).`,
+  "List phase 2 in remainingWork with blocksCompletion false: phase 1 is complete, and phase 2 waits for the board.",
   phaseTwo(nonce),
 ].join("\n");
 
@@ -271,6 +274,8 @@ export function gradeRewake(plan: Pick<TaskPlan, "workload" | "nonce">, artifact
   return { verdict: checks.every(check => check.ok) ? "pass" : "fail", checks };
 }
 
+export const rewakeStartComment = (nonce: string) => `Start phase 1 of this task now (${briefMarker(nonce)}).`;
+
 const definition = (id: RewakeWorkload, label: string, description: (nonce: string) => string): WorkloadDefinition => ({
   id,
   label,
@@ -282,6 +287,7 @@ const definition = (id: RewakeWorkload, label: string, description: (nonce: stri
   budgetMs: 20 * 60_000,
   title: nonce => `Chaos ${id} re-wake ${nonce}`,
   description,
+  ...(id === "W2Q" ? {} : { startComment: rewakeStartComment }),
   collect: { documents: ["brief"], files: () => [], commands: [] },
   grade: (nonce, artifacts) => gradeRewake({ workload: id, nonce }, artifacts),
 });

tests/restart-chaos/shape-cells.test.ts

@@ -8,6 +8,7 @@ import { parseRecoveryStep, parseRecoveryStepReceipts, recoveryStepControlSpec,
 import { generationHome, parseK15DeployShape, redeployEnvironment, redeployIdentity } from "./redeploy.js";
 import { awaitingRewake, gradeRewake, noteMarker, rewakeKind, rewakeMarker, rewakeStep, type RewakeArtifacts } from "./rewake.js";
 import { planTasks, planWave, taskFaultSchedule } from "./scheduler.js";
+import { stableRequestId } from "./observer.js";
 import type { OracleArtifacts, RunSample, TaskPlan, TaskRecord, TaskSample } from "./types.js";
 import { workloadDefinitions } from "./workloads.js";
 
@@ -214,6 +215,27 @@ describe("K15 staging deploy shape", () => {
     expect(() => parseK15DeployShape({ env: "daytona", faults: ["K15"] }, "2-3", false)).toThrow("between 5 and 600");
   });
 
+  it("runs volume-free servers only on Daytona with the wave's object store", () => {
+    const image = `ghcr.io/x/y@sha256:${"a".repeat(64)}`;
+    expect(wave(["--env", "daytona", "--daytona-image", image, "--storage", "s3", "--ephemeral-server", "--faults", "K15", "--k15-fresh-home"]))
+      .toMatchObject({ ephemeralServer: true, storage: "s3", k15FreshHome: true });
+    expect(wave(["--env", "daytona", "--daytona-image", image])).not.toHaveProperty("ephemeralServer");
+    expect(wave(["--env", "daytona", "--daytona-image", image, "--storage", "s3", "--ephemeral-server"]).workloads.every(workload => !workloadDefinitions[workload].usesRepository)).toBe(true);
+    expect(workloadDefinitions.W2.startComment?.("n1")).toContain("CHAOS-n1");
+    expect(workloadDefinitions.W2C.startComment).toBeDefined();
+    expect(workloadDefinitions.W2Q.startComment).toBeUndefined();
+    expect(stableRequestId("start-t01-n1")).toMatch(/^[0-9a-f]{8}-[0-9a-f]{4}-5[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/);
+    expect(stableRequestId("start-t01-n1")).toBe(stableRequestId("start-t01-n1"));
+    expect(stableRequestId("start-t02-n1")).not.toBe(stableRequestId("start-t01-n1"));
+    expect(wave(["--env", "daytona", "--daytona-image", image, "--faults", "K15", "--k15-fresh-home"]))
+      .toMatchObject({ ephemeralServer: true, storage: "s3", k15FreshHome: true });
+    expect(() => wave(["--env", "daytona", "--daytona-image", image, "--storage", "local", "--faults", "K15", "--k15-fresh-home"]))
+      .toThrow("--k15-fresh-home");
+    expect(() => wave(["--env", "daytona", "--daytona-image", image, "--ephemeral-server"])).toThrow("--storage s3");
+    expect(() => wave(["--env", "local", "--storage", "s3", "--ephemeral-server"])).toThrow("--env daytona");
+    expect(() => wave(["--env", "daytona", "--daytona-image", image, "--storage", "s3", "--ephemeral-server", "--workloads", "A1,B1"])).toThrow("B1");
+  });
+
   it("drops the volume variables for a fresh home, and gives each generation its own home", () => {
     const identity = redeployIdentity("w-20261111T000000-s1", 1);
     expect(redeployEnvironment(identity, "/tmp/home")).toMatchObject({ RAILWAY_VOLUME_MOUNT_PATH: "/tmp/home" });

tests/restart-chaos/supervisor.ts

@@ -41,6 +41,7 @@ import type { FaultKind, FaultReceipt, TriggerId } from "./types.js";
 import { isMultiServerFault, runMultiServerFault, type MultiServerKind, type MultiServerRecord, type ServerProcessWindow } from "./multi-server.js";
 import { generationHome, nextRedeployIdentity, redeployEnvironment, redeployIdentity, redeployNodeArgs, redeployReceipt, type RedeployIdentity } from "./redeploy.js";
 import { startEdgeProxy, type EdgeProxy } from "./edge-proxy.js";
+import { ephemeralServerEnvironment } from "./ephemeral-server.js";
 import { CLASS_R_PRE_FAULT_DEADLINE_MS, classRRedeploys, isClassRFault, runPluginStall, type ClassRHooks } from "./class-r.js";
 
 /** First boot applies every migration; a loaded host can need longer (RESTART_CHAOS_HEALTH_TIMEOUT_MS). */
@@ -173,6 +174,7 @@ export class ChaosSupervisor {
   redeployEvidence: ((faultId: string, phase: "stopped" | "before_start") => Promise<void>) | null = null;
   /** K15 deploy shape (redeploy.ts `parseK15DeployShape`); main.ts sets it before `start`. */
   deployShape: { overlap: boolean; freshHome: boolean } = { overlap: false, freshHome: false };
+  ephemeralServer = false;
   /** `--k15-fresh-home`: the instance home and config of the serving generation (generation 0 uses `paths`). */
   private serverHome: { home: string; configPath: string };
   /** `--k15-overlap-seconds`: the edge on the wave's port, and the serving generation's own port behind it. */
@@ -286,10 +288,12 @@ export class ChaosSupervisor {
       // K15: the generation's emulated host name and Railway deployment identity; the volume is the instance home.
       ...(this.redeploy ? redeployEnvironment(this.redeploy.identity, this.deployShape.freshHome ? null : home) : {}),
     };
-    const environment = buildPaperclipServerEnvironment(source, overrides);
+    const ephemeral = this.ephemeralServer ? await ephemeralServerEnvironment({ configPath: this.paths.configPath, temporaryRoot: this.paths.temporaryRoot, extraEnvironment }) : null;
+    const environment = buildPaperclipServerEnvironment(source, { ...overrides, ...ephemeral });
     const workspaceStorage = extraEnvironment.PAPERCLIP_STORAGE_PROVIDER === "s3"
       ? Object.fromEntries(Object.entries(extraEnvironment).filter(([key]) => key.startsWith("PAPERCLIP_STORAGE_"))) : undefined;
-    assertIsolatedServerEnvironment(environment, { temporaryRoot, paperclipHome: home, configPath, ...(workspaceStorage ? { workspaceStorage } : {}) });
+    assertIsolatedServerEnvironment(environment, { temporaryRoot, paperclipHome: home, configPath, ...(workspaceStorage ? { workspaceStorage } : {}),
+      ...(ephemeral ? { generatedSecretsMasterKey: ephemeral.PAPERCLIP_SECRETS_MASTER_KEY } : {}) });
     this.environment = Object.fromEntries(Object.entries(environment).filter((entry): entry is [string, string] => entry[1] !== undefined));
     await prepareSourceServerDependencies(root, this.environment, false);
   }

tests/restart-chaos/types.ts

@@ -86,6 +86,7 @@ export interface WaveOptions {
   k15Overlap?: boolean;
   /** `--k15-fresh-home`: each deploy generation gets a new, empty instance home and no volume (redeploy.ts). */
   k15FreshHome?: boolean;
+  ephemeralServer?: boolean;
   /** `--faults K0`: cells with no fault, for workloads whose shape is the fault (W2, W2Q, W2C). */
   baseline?: boolean;
   /**

tests/restart-chaos/workloads.ts

@@ -20,6 +20,7 @@ export interface WorkloadDefinition {
   title(nonce: string): string;
   description(nonce: string, agents?: WorkloadAgents): string;
   followUp?(nonce: string): string;
+  startComment?(nonce: string): string;
   stopBeforeFollowUp?: boolean;
   /** Files and commands the observer must collect for the T-oracle. */
   collect: { documents: string[]; files(nonce: string): string[]; commands: Array<{ name: string; argv: string[] | ((nonce: string) => string[]) }>;

tests/runner-e2e/harness-env.ts

@@ -257,6 +257,7 @@ export function assertIsolatedServerEnvironment(
     paperclipHome: string;
     configPath: string;
     workspaceStorage?: NodeJS.ProcessEnv;
+    generatedSecretsMasterKey?: string;
   },
 ) {
   const home = env.PAPERCLIP_HOME;
@@ -285,6 +286,7 @@ export function assertIsolatedServerEnvironment(
     ...AMBIENT_PAPERCLIP_CREDENTIAL_KEYS,
     ...AMBIENT_EXTERNAL_STATE_KEYS,
   ]) {
+    if (key === "PAPERCLIP_SECRETS_MASTER_KEY" && expected.generatedSecretsMasterKey && env[key] === expected.generatedSecretsMasterKey) continue;
     if (env[key] && !(AMBIENT_EXTERNAL_STATE_KEYS.includes(key as typeof AMBIENT_EXTERNAL_STATE_KEYS[number])
       && expected.workspaceStorage?.[key] === env[key] && expected.workspaceStorage?.PAPERCLIP_STORAGE_PROVIDER === "s3"))
       throw new Error(

tests/runner-e2e/support.test.ts

@@ -990,6 +990,32 @@ describe("runner E2E server isolation", () => {
     ).not.toThrow();
   });
 
+  it("admits only the wave's own generated secrets master key", () => {
+    const expected = {
+      temporaryRoot: "/tmp/cell",
+      paperclipHome: "/tmp/cell/paperclip-home",
+      configPath: "/tmp/cell/paperclip-home/instances/e2e/config.json",
+    };
+    const env = (extra: NodeJS.ProcessEnv) => ({
+      PAPERCLIP_HOME: expected.paperclipHome,
+      PAPERCLIP_CONFIG: expected.configPath,
+      XDG_CACHE_HOME: "/tmp/cell/xdg-cache",
+      PAPERCLIP_AGENT_JWT_SECRET: "generated-agent-jwt",
+      PAPERCLIP_DECISION_SIGNING_SECRET: "generated-decision-key",
+      PAPERCLIP_TOOL_ACTION_SIGNING_SECRET: "generated-tool-key",
+      BETTER_AUTH_SECRET: "generated-auth-key",
+      ...extra,
+    });
+    expect(() => assertIsolatedServerEnvironment(env({ PAPERCLIP_SECRETS_MASTER_KEY: "wave-key" }),
+      { ...expected, generatedSecretsMasterKey: "wave-key" })).not.toThrow();
+    expect(() => assertIsolatedServerEnvironment(env({ PAPERCLIP_SECRETS_MASTER_KEY: "ambient-key" }), expected))
+      .toThrow("PAPERCLIP_SECRETS_MASTER_KEY");
+    expect(() => assertIsolatedServerEnvironment(env({ PAPERCLIP_SECRETS_MASTER_KEY: "ambient-key" }),
+      { ...expected, generatedSecretsMasterKey: "wave-key" })).toThrow("PAPERCLIP_SECRETS_MASTER_KEY");
+    expect(() => assertIsolatedServerEnvironment(env({ PAPERCLIP_SECRETS_MASTER_KEY: "wave-key", PAPERCLIP_SECRETS_MASTER_KEY_FILE: "/tmp/cell/key" }),
+      { ...expected, generatedSecretsMasterKey: "wave-key" })).toThrow("PAPERCLIP_SECRETS_MASTER_KEY_FILE");
+  });
+
   it("uses absolute repository paths for the Playwright web server", () => {
     const command = runnerE2EWebServerCommand("/workspace/paperclip");
     expect(command).toContain(