From 068d8147357342c59f68b21519a1ed50cbcdf7fc Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 11 Nov 2026 17:12:14 -0600 Subject: [PATCH] test(chaos): run waves in Cloud's volume-free server mode; make W2 phase 1 yield (#20930) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) runs real agent tasks against an isolated control plane and injects faults > - Cloud servers run in the volume-free mode (`PAPERCLIP_EPHEMERAL_SERVER=true`, durable agent files, run logs in object storage) > - The harness never set that mode, so every wave ran a different server mode from staging. Paths that exist only in that mode were never tested. One of them is the per-agent directory operation lock, so `--fail-recovery-step directory-lock` could never reach its step with any workload > - The re-wake workloads W2 and W2C also lost coverage: an agent that reported phase 1 with disposition `done` closed the task, and the harness never sent the re-wake (four of eight Claude cells on 2026-11-11) > - This pull request adds `--ephemeral-server` and makes the W2/W2C brief ask for a `yielded` phase-1 report > - The benefit is that chaos waves can run in staging's server mode and the re-wake cells test the re-wake on every provider ## Linked Issues or Issue Description **What is the problem?** The restart chaos harness cannot run the control plane in the volume-free mode that Cloud and staging use. The `directory-lock` recovery step (`agent_directory.operation_lock.before_acquire`) is reached only for disposable remote copies, which need `PAPERCLIP_EPHEMERAL_SERVER=true`. W2 and W2C cells also stop before the re-wake when the agent reports phase 1 as `done`. **What is the expected behavior?** A wave can opt in to the volume-free mode on Daytona, and every restart of the wave keeps that mode. The W2 and W2C cells reach the re-wake. **Steps to reproduce** Run `--fail-recovery-step agent_directory.operation_lock.before_acquire:1` on Daytona with any workload: the check `recovery_step_failed_once` reports "never reached". Run W2 on Claude: the first run ends with the task `done` and `rewake_sent` is "never sent". ## What Changed - `assertIsolatedServerEnvironment` (`tests/runner-e2e/harness-env.ts`) takes `expected.generatedSecretsMasterKey`. It admits `PAPERCLIP_SECRETS_MASTER_KEY` only when that parameter is set and equals the value exactly. `PAPERCLIP_SECRETS_MASTER_KEY_FILE` stays refused. The check runs over the final server environment. - `--k15-fresh-home` implies `--ephemeral-server` and defaults `--storage` to `s3`. A fresh instance home without the volume-free mode loses the agents' instruction bundles, which staging keeps in Postgres and object storage. - `--ephemeral-server` (`config.ts`, `types.ts`, `main.ts`, `supervisor.ts`, new `ephemeral-server.ts` with its unit test `ephemeral-server.test.ts`): it needs `--storage s3` and `--env daytona` and refuses K14. Every server of the wave gets `PAPERCLIP_EPHEMERAL_SERVER`, `PAPERCLIP_DURABLE_AGENT_FILES`, `PAPERCLIP_SHARED_UI_ASSETS`, the wave's Postgres as `PAPERCLIP_DATABASE_LISTEN_URL`, one stable `PAPERCLIP_SECRETS_MASTER_KEY` (also written to the config key file) and `RUN_LOG_S3_*` in the wave's object store. - `rewake.ts`: the W2/W2C phase-1 brief asks for `paperclip_finish` with disposition `yielded` and a `response_wake` continuation. - W2/W2C tasks are created in `backlog` with their assignee, and a board comment starts phase 1 (`startComment` in `workloads.ts`, used by `observer.ts`). A passive board wait needs a comment-started run; without it the yield queues an immediate continuation and phase 1 wakes itself. The start comment is posted once (saved-comment check plus a stable `clientRequestId`), and the brief keeps phase 2 out of blocking `remainingWork`. - Volume-free mode drops the repository workloads (B1, B5, F1, R1) from the default list and refuses them when listed: their fixture is a `local_path` project source, which a volume-free server refuses. - `README.md`: a section for the new flag and a note on the brief. - `shape-cells.test.ts`: option parsing and refusals for the new flag. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/shape-cells.test.ts` passes (16 tests). - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live proof wave (Daytona, Codex, W2, K1 and K15 with a fresh home, `--storage s3 --ephemeral-server`): wave `w-20261111T205259-s6301` (2 tasks). The first server and both replacements (K15 with a fresh home, then K1) started and reached ready in the volume-free mode. The server process had `PAPERCLIP_EPHEMERAL_SERVER=true`, `PAPERCLIP_DURABLE_AGENT_FILES=true` and `RUN_LOG_S3_BUCKET` set. Both W2 tasks left phase 1 `in_progress` (yielded), received the re-wake and ended `done`. The wave also found product faults (reported separately, not fixed here): a re-wake after K1 failed with `runner_harness_state_mismatch: backup_without_reusable_lease` and needed Retry. - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/support.test.ts` passes, including the new master-key admission test (a matching wave key passes; an inherited key, a mismatched key and `PAPERCLIP_SECRETS_MASTER_KEY_FILE` are refused). ## Risks - Low risk. The flag is opt-in and the harness is test-only. The brief change alters the W2/W2C prompt, so results from older W2/W2C seeds are not directly comparable. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- tests/restart-chaos/README.md | 31 +++++++++++++- tests/restart-chaos/config.ts | 18 ++++++-- tests/restart-chaos/ephemeral-server.test.ts | 45 ++++++++++++++++++++ tests/restart-chaos/ephemeral-server.ts | 25 +++++++++++ tests/restart-chaos/main.ts | 1 + tests/restart-chaos/observer.ts | 19 ++++++++- tests/restart-chaos/rewake.ts | 8 +++- tests/restart-chaos/shape-cells.test.ts | 22 ++++++++++ tests/restart-chaos/supervisor.ts | 8 +++- tests/restart-chaos/types.ts | 1 + tests/restart-chaos/workloads.ts | 1 + tests/runner-e2e/harness-env.ts | 2 + tests/runner-e2e/support.test.ts | 26 +++++++++++ 13 files changed, 197 insertions(+), 10 deletions(-) create mode 100644 tests/restart-chaos/ephemeral-server.test.ts create mode 100644 tests/restart-chaos/ephemeral-server.ts diff --git a/tests/restart-chaos/README.md b/tests/restart-chaos/README.md index 2c4505b1687..a1684280485 100644 --- a/tests/restart-chaos/README.md +++ b/tests/restart-chaos/README.md @@ -26,7 +26,7 @@ Options: `--triggers time,comment,retry,queued,tool,result` (fault timing, below default 3000), `--poll-ms` (2000–5000), `--keep` (keep the temporary instance), `--k4-hold-max-seconds` (raise the seeded K4 hold upper end from 300 s, up to the 600 s outage cap). `--faults K0` plans cells with no fault, for workloads whose shape is the fault (W2, below). `--fail-recovery-step`, -`--long-turn-minutes`, `--k15-overlap-seconds` and `--k15-fresh-home` are described in their own sections. +`--long-turn-minutes`, `--k15-overlap-seconds`, `--k15-fresh-home` and `--ephemeral-server` are described in their own sections. `--fault-after-seconds ` fires every planned fault `n` seconds after its task launches (5–3600; with `--triggers time`), keeping the rest of the seeded plan, for example to restart a Daytona R1 task after its workspace preparation. `--matrix` crosses every selected workload with every fault kind (below), and `--server-from ` @@ -496,6 +496,28 @@ deletes a sandbox. An archived or stopped sandbox is recorded by its state alone | `w-20261111T204144-s9503` | CR1 360 s twice, OpenCode and Claude | No refusal. The Runner and the provider survived both outages. | | `w-20261111T210055-s9504` | CR3 45 s, OpenCode and Claude | Not the staging code, a separate failure. `runnerd process supervision failed: RPC call "environmentExecute" timed out after 40000ms` failed the run as `notification_transport_failed`. The close path suspended a healthy Runner (`lifecycle: suspended`, no stderr). Retry worked. | +### Volume-free servers (`--ephemeral-server`) + +```sh +node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env daytona --daytona-image $IMG --storage s3 --ephemeral-server \ + --adapters codex --workloads W2 --faults K1,K2,K15 --k15-fresh-home --k15-overlap-seconds 30-90 --concurrency 3 --tasks 3 --seed 5501 +``` + +Cloud servers run volume-free (`ephemeral-server-contract.ts`): agent files, run logs and UI assets live in object +storage, secrets use one stable master key, and remote agent directories are disposable copies whose collection and +cleanup take the per-agent Postgres operation lock. Every other wave runs the volume-backed mode, so those paths, +including the `directory-lock` recovery step, are never reached there. `--ephemeral-server` gives every server of the +wave `PAPERCLIP_EPHEMERAL_SERVER`, `PAPERCLIP_DURABLE_AGENT_FILES` and `PAPERCLIP_SHARED_UI_ASSETS`, the wave's +Postgres as `PAPERCLIP_DATABASE_LISTEN_URL`, a stable `PAPERCLIP_SECRETS_MASTER_KEY` (also written to the config's +key file inside the wave's temporary root; `assertIsolatedServerEnvironment` admits that one generated key and still +refuses an inherited key or `PAPERCLIP_SECRETS_MASTER_KEY_FILE`) and `RUN_LOG_S3_*` in the wave's object store. It needs +`--storage s3` and `--env daytona` (a local workspace blocks a volume-free server) and does not combine with K14. Workloads with the fixture repository (B1, B5, +F1, R1) are a `local_path` project source, which a volume-free server refuses: they are dropped from the default +workloads and refused when listed. +`--k15-fresh-home` implies it (and defaults `--storage` to `s3`): a new instance home without the volume-free mode +loses the agents' instruction bundles, which staging keeps in Postgres and object storage. Every restart replays the server's startup +eligibility check, so a replacement that refuses to start fails the wave. + ## Absolute symlink in the workspace (F1) ```sh @@ -1084,7 +1106,12 @@ timed out`, `runner did not durably suspend before checkpoint` (class C+), or The workload is the fault, so these cells usually run with `--faults K0` (no fault). Every task of an adapter runs on that adapter's one agent: `--adapters codex --tasks 10 --concurrency 10` is ten tasks on one -agent. Phase 1 ends with the task left `in_progress` (`in_review` would gate completion on a review). Then the harness, as the board, wakes the same task the +agent. Phase 1 ends with the task left `in_progress` (`in_review` would gate completion on a review): the W2 and W2C +brief asks the agent to report phase 1 with `paperclip_finish` disposition `yielded` and a `response_wake` +continuation, because an agent that reports `done` closes the task and the wake is never sent (on 2026-11-11 four of +eight Claude cells tested nothing). A passive board wait needs a run that a board comment started, so W2 and W2C +create the task in `backlog` and start phase 1 with a board comment; without it the yield queues an immediate +continuation and phase 1 wakes itself. Then the harness, as the board, wakes the same task the seeded 0–60 s after that run settled (`rewake.ts`): - **W2**: a board comment, or, by the task's nonce, a board-scheduled issue monitor due now, so the diff --git a/tests/restart-chaos/config.ts b/tests/restart-chaos/config.ts index 29921a57e1c..f52210945b2 100644 --- a/tests/restart-chaos/config.ts +++ b/tests/restart-chaos/config.ts @@ -9,6 +9,7 @@ import { parseK15DeployShape, parseK15HoldSeconds } from "./redeploy.js"; import { validateRecoveryStep } from "./recovery-step-faults.js"; import { parseLongTurnMinutes } from "./long-turn.js"; import { isRewakeWorkload } from "./rewake.js"; +import { workloadDefinitions } from "./workloads.js"; import { parseClassROptions } from "./class-r.js"; import { parseTimeFaultOptions } from "./time-faults-config.js"; import { parseStorageTrigger } from "./storage-trigger.js"; @@ -143,6 +144,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC "k15-hold-seconds": { type: "string" }, "k15-overlap-seconds": { type: "string" }, "k15-fresh-home": { type: "boolean" }, + "ephemeral-server": { type: "boolean" }, "tm1-hold-seconds": { type: "string" }, "fail-recovery-step": { type: "string" }, "long-turn-minutes": { type: "string" }, @@ -156,7 +158,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC const pollMs = integer(values["poll-ms"], "poll-ms", 3_000, 2_000, 5_000); const budgetCents = integer(values["budget-cents"], "budget-cents", 3_000, 1, 100_000); if (values.replay !== undefined) { - for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-pack-mismatch", "daytona-cpu", "daytona-reuse-lease", "daytona-warm", "daytona-inherit", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds", "storage-trigger", "native-trigger", "native-trigger-outage", "fault-after-seconds", "provider-fault-message", "provider-fault-delays", "db-pressure", "matrix", "server-from", "time-age-hours", "provider-fault-restart", "provider-fault-restart-after-seconds", "rr-gap-seconds", "rr-stop-on", "k15-hold-seconds", "k15-overlap-seconds", "k15-fresh-home", "tm1-hold-seconds", "fail-recovery-step", "long-turn-minutes", "cr1-hold-seconds", "cr2-hold-seconds", "cr3-stall-seconds", "sandbox-watch-seconds"] as const) { + for (const key of ["env", "adapters", "workloads", "faults", "triggers", "concurrency", "tasks", "seed", "daytona-image", "daytona-pack-mismatch", "daytona-cpu", "daytona-reuse-lease", "daytona-warm", "daytona-inherit", "storm-minutes", "lifecycle", "k4-hold-max-seconds", "crash-point", "crash-point-hold-ms", "upgrade-from", "workspace-repo", "storage", "storage-faults", "storage-fault-seconds", "storage-trigger", "native-trigger", "native-trigger-outage", "fault-after-seconds", "provider-fault-message", "provider-fault-delays", "db-pressure", "matrix", "server-from", "time-age-hours", "provider-fault-restart", "provider-fault-restart-after-seconds", "rr-gap-seconds", "rr-stop-on", "k15-hold-seconds", "k15-overlap-seconds", "k15-fresh-home", "ephemeral-server", "tm1-hold-seconds", "fail-recovery-step", "long-turn-minutes", "cr1-hold-seconds", "cr2-hold-seconds", "cr3-stall-seconds", "sandbox-watch-seconds"] as const) { if (values[key] !== undefined) throw new Error(`--replay reproduces the saved plan; do not combine it with --${key}`); } return { mode: "replay", ref: parseReplayRef(values.replay), keep, @@ -205,11 +207,12 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC const diskFullFaults = allFaults.filter(isDiskFullFault); const sessionFaults = allFaults.filter(isSessionFault); const baseline = allFaults.includes("K0"); + const volumeFree = values["ephemeral-server"] === true || values["k15-fresh-home"] === true; const options: WaveOptions = { env, adapters, // The governance scenarios (G1, H1), the journeys (Q1–Q4) and the contention run (L1) are opt-in: H1 needs a wave of its own. - workloads: list(values.workloads, workloadChoices, "workloads", workloadChoices.filter(id => id !== "G1" && id !== "H1" && id !== "R1" && id !== "S1" && id !== "L1" && id !== "F1" && id !== "T1" && id !== "Q4" && id !== "H2" + workloads: list(values.workloads, workloadChoices, "workloads", workloadChoices.filter(id => !(volumeFree && workloadDefinitions[id].usesRepository) && id !== "G1" && id !== "H1" && id !== "R1" && id !== "S1" && id !== "L1" && id !== "F1" && id !== "T1" && id !== "Q4" && id !== "H2" && !takesJourneyTriggers(id) && !isRewakeWorkload(id) && (id !== "M1" || adapters.length >= 2))), // A crash-point wave schedules no fault; K1 is the restart its self-kill gets. faults: allFaults.filter((fault): fault is FaultKind => !isProviderFault(fault) && !isDiskFullFault(fault) && !isSessionFault(fault) && fault !== "K0"), @@ -331,7 +334,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC } applyJourneyOptions(options, { matrix: values.matrix === true, serverFrom: values["server-from"], tasksGiven: values.tasks !== undefined, triggersGiven: values.triggers !== undefined }); - const storage = values.storage ?? "local"; + const storage = values.storage ?? (values["k15-fresh-home"] ? "s3" : "local"); const workspaceRepo = values["workspace-repo"]; if (options.workloads.includes("R1") !== (workspaceRepo !== undefined)) { throw new Error("R1 (code change in a cloned repository) and --workspace-repo go together: pass both or neither"); @@ -384,6 +387,15 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC if (values["db-pressure"] !== undefined) options.dbPressure = integer(values["db-pressure"], "db-pressure", 0, 1, 8); if (storage !== "local" && storage !== "s3") throw new Error("--storage must be local or s3"); if (storage === "s3") options.storage = "s3"; + if (values["ephemeral-server"] || options.k15FreshHome) { + const flag = values["ephemeral-server"] ? "--ephemeral-server" : "--k15-fresh-home (a volume-free deploy)"; + if (storage !== "s3") throw new Error(`${flag} runs volume-free and needs shared object storage: pass --storage s3`); + if (env !== "daytona") throw new Error(`${flag} runs only with --env daytona: a local workspace blocks a volume-free server`); + if (options.faults.includes("K14")) throw new Error(`${flag} does not combine with K14 (--upgrade-from)`); + const repository = options.workloads.filter(workload => workloadDefinitions[workload].usesRepository); + if (repository.length) throw new Error(`${flag} cannot run ${repository.join(", ")}: the fixture repository is a local_path project source, which a volume-free server refuses; pick other --workloads`); + options.ephemeralServer = true; + } if (values["storage-faults"] !== undefined) { if (storage !== "s3") throw new Error("--storage-faults needs --storage s3"); options.storageFaults = list(values["storage-faults"], STORAGE_FAULT_KINDS, "storage-faults", []); diff --git a/tests/restart-chaos/ephemeral-server.test.ts b/tests/restart-chaos/ephemeral-server.test.ts new file mode 100644 index 00000000000..fec2c15a377 --- /dev/null +++ b/tests/restart-chaos/ephemeral-server.test.ts @@ -0,0 +1,45 @@ +import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import { ephemeralServerEnvironment } from "./ephemeral-server.js"; + +const roots: string[] = []; +afterEach(async () => { await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); }); + +async function fixture(keyFilePath?: (root: string) => string) { + const root = await mkdtemp(path.join(os.tmpdir(), "chaos-ephemeral-")); + roots.push(root); + const configPath = path.join(root, "home", "instances", "default", "config.json"); + await mkdir(path.dirname(configPath), { recursive: true }); + const keyFile = keyFilePath ? keyFilePath(root) : path.join(path.dirname(configPath), "secrets", "master.key"); + await writeFile(configPath, JSON.stringify({ database: { connectionString: "postgres://u:p@127.0.0.1:5432/paperclip" }, + secrets: { localEncrypted: { keyFilePath: keyFile } } })); + return { root, configPath, keyFile }; +} + +const storage = { PAPERCLIP_STORAGE_PROVIDER: "s3", PAPERCLIP_STORAGE_S3_BUCKET: "chaos-b", PAPERCLIP_STORAGE_S3_ENDPOINT: "http://127.0.0.1:9000", + PAPERCLIP_STORAGE_S3_REGION: "us-east-1", PAPERCLIP_STORAGE_S3_PREFIX: "chaos/w1/" }; + +describe("ephemeralServerEnvironment", () => { + it("builds the volume-free environment and keeps one master key across restarts", async () => { + const { root, configPath, keyFile } = await fixture(); + const first = await ephemeralServerEnvironment({ configPath, temporaryRoot: root, extraEnvironment: storage }); + expect(first).toMatchObject({ PAPERCLIP_EPHEMERAL_SERVER: "true", PAPERCLIP_DURABLE_AGENT_FILES: "true", PAPERCLIP_SHARED_UI_ASSETS: "true", + PAPERCLIP_DATABASE_LISTEN_URL: "postgres://u:p@127.0.0.1:5432/paperclip", RUN_LOG_S3_BUCKET: "chaos-b", + RUN_LOG_S3_ENDPOINT: "http://127.0.0.1:9000", RUN_LOG_S3_REGION: "us-east-1", RUN_LOG_S3_PREFIX: "chaos/w1/run-logs" }); + expect(Buffer.from(first.PAPERCLIP_SECRETS_MASTER_KEY!, "base64")).toHaveLength(32); + expect((await readFile(keyFile, "utf8")).trim()).toBe(first.PAPERCLIP_SECRETS_MASTER_KEY); + const second = await ephemeralServerEnvironment({ configPath, temporaryRoot: root, extraEnvironment: storage }); + expect(second.PAPERCLIP_SECRETS_MASTER_KEY).toBe(first.PAPERCLIP_SECRETS_MASTER_KEY); + }); + + it("refuses a key file outside the wave's temporary root and a wave without object storage", async () => { + const outside = await fixture(() => path.join(os.tmpdir(), "elsewhere", "master.key")); + await expect(ephemeralServerEnvironment({ configPath: outside.configPath, temporaryRoot: outside.root, extraEnvironment: storage })) + .rejects.toThrow("escapes its temporary root"); + const local = await fixture(); + await expect(ephemeralServerEnvironment({ configPath: local.configPath, temporaryRoot: local.root, extraEnvironment: {} })) + .rejects.toThrow("--storage s3"); + }); +}); diff --git a/tests/restart-chaos/ephemeral-server.ts b/tests/restart-chaos/ephemeral-server.ts new file mode 100644 index 00000000000..0960edcec6f --- /dev/null +++ b/tests/restart-chaos/ephemeral-server.ts @@ -0,0 +1,25 @@ +import { randomBytes } from "node:crypto"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import path from "node:path"; +import { databaseUrlFromConfig } from "./time-faults.js"; + +export async function ephemeralServerEnvironment(input: { configPath: string; temporaryRoot: string; extraEnvironment: Record }): Promise> { + const { configPath, temporaryRoot, extraEnvironment } = input; + if (extraEnvironment.PAPERCLIP_STORAGE_PROVIDER !== "s3") throw new Error("--ephemeral-server needs the wave's object store (--storage s3)"); + const config = JSON.parse(await readFile(configPath, "utf8")) as { secrets?: { localEncrypted?: { keyFilePath?: string } } }; + const keyFile = String(config.secrets?.localEncrypted?.keyFilePath ?? path.join(path.dirname(configPath), "secrets", "master.key")); + if (!path.resolve(keyFile).startsWith(`${path.resolve(temporaryRoot)}${path.sep}`)) throw new Error("--ephemeral-server: the wave's master key file escapes its temporary root"); + let masterKey = await readFile(keyFile, "utf8").then(value => value.trim(), () => ""); + if (!masterKey) { + masterKey = randomBytes(32).toString("base64"); + await mkdir(path.dirname(keyFile), { recursive: true, mode: 0o700 }); + await writeFile(keyFile, masterKey, { mode: 0o600, flag: "wx" }); + } + return { + PAPERCLIP_EPHEMERAL_SERVER: "true", PAPERCLIP_DURABLE_AGENT_FILES: "true", PAPERCLIP_SHARED_UI_ASSETS: "true", + PAPERCLIP_DATABASE_LISTEN_URL: databaseUrlFromConfig(configPath), PAPERCLIP_SECRETS_MASTER_KEY: masterKey, + RUN_LOG_S3_BUCKET: extraEnvironment.PAPERCLIP_STORAGE_S3_BUCKET!, RUN_LOG_S3_ENDPOINT: extraEnvironment.PAPERCLIP_STORAGE_S3_ENDPOINT!, + RUN_LOG_S3_REGION: extraEnvironment.PAPERCLIP_STORAGE_S3_REGION!, RUN_LOG_S3_FORCE_PATH_STYLE: "true", + RUN_LOG_S3_PREFIX: `${extraEnvironment.PAPERCLIP_STORAGE_S3_PREFIX ?? ""}run-logs`, + }; +} diff --git a/tests/restart-chaos/main.ts b/tests/restart-chaos/main.ts index 3664d7f6b26..f0de3073dca 100644 --- a/tests/restart-chaos/main.ts +++ b/tests/restart-chaos/main.ts @@ -248,6 +248,7 @@ async function main(): Promise { const violationKey = (violation: { rule: string; kind: string; recordId: string }) => `${violation.rule}:${violation.kind}:${violation.recordId}`; // `--k15-overlap-seconds`, `--k15-fresh-home`: the staging deploy shape (redeploy.ts). supervisor.deployShape = { overlap: plan.options.k15Overlap === true, freshHome: plan.options.k15FreshHome === true }; + supervisor.ephemeralServer = plan.options.ephemeralServer === true; // `--fail-recovery-step`: fail the nth hit of one recovery step once, after the wave's first restart fault. const recoveryStepReceipts = path.join(waveDir, "recovery-step-receipts.jsonl"); if (plan.options.failRecoveryStep) { diff --git a/tests/restart-chaos/observer.ts b/tests/restart-chaos/observer.ts index 910f5e28c9a..fb3c0fd1509 100644 --- a/tests/restart-chaos/observer.ts +++ b/tests/restart-chaos/observer.ts @@ -1,5 +1,5 @@ import { spawnSync } from "node:child_process"; -import { randomUUID } from "node:crypto"; +import { createHash, randomUUID } from "node:crypto"; import { access, mkdir, mkdtemp, readFile, readdir, rm, writeFile } from "node:fs/promises"; import os from "node:os"; import path from "node:path"; @@ -30,6 +30,11 @@ const IGNORED_DIRECTORIES = new Set([".git", ".bench", "__pycache__", "node_modu const MAX_SNAPSHOT_FILES = 2_000; /** Relative paths of workspace files, excluding VCS and cache directories. */ + +export function stableRequestId(key: string): string { + const hex = createHash("sha256").update(key).digest("hex"); + return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-5${hex.slice(13, 16)}-${(8 + (Number.parseInt(hex[16]!, 16) & 3)).toString(16)}${hex.slice(17, 20)}-${hex.slice(20, 32)}`; +} export async function listWorkspaceFiles(root: string): Promise { const out: string[] = []; const walk = async (relative: string) => { @@ -255,6 +260,7 @@ export class TaskObserver { for (;;) { try { await this.api.patch(`/api/issues/${this.record.issueId}`, { assigneeAgentId: this.journey?.assigneeAgentId ?? this.lab.agents[this.plan.adapter].id }); + await this.postStartComment(); break; } catch (error) { const transient = error instanceof ChaosApiError && (error.status === null || error.status >= 500); @@ -272,6 +278,14 @@ export class TaskObserver { this.timeline.add("task_released", { taskId: this.plan.taskId, issueId: this.record.issueId }); } + private async postStartComment(): Promise { + if (!this.definition.startComment || !this.record.issueId) return; + const body = this.definition.startComment(this.plan.nonce); + const existing = await this.api.get(`/api/issues/${this.record.issueId}/comments`); + if (existing.some(comment => comment.body === body)) return; + await this.api.post(`/api/issues/${this.record.issueId}/comments`, { body, clientRequestId: stableRequestId(`start-${this.plan.taskId}-${this.plan.nonce}`) }); + } + async launch(options: { assign?: boolean } = {}): Promise { const definition = this.definition; const assign = options.assign !== false; @@ -282,13 +296,14 @@ export class TaskObserver { const issue = await this.api.post(`/api/companies/${this.lab.companyId}/issues`, { title: definition.title(this.plan.nonce), description: definition.description(this.plan.nonce, this.workloadAgents()), - status: "todo", + status: definition.startComment ? "backlog" : "todo", priority: "medium", ...(assign ? { assigneeAgentId: this.journey?.assigneeAgentId ?? this.lab.agents[this.plan.adapter].id } : {}), ...(definition.usesRepository ? { projectId: this.lab.projectId } : {}), }); this.record.issueId = String(issue.id); this.record.identifier = issue.identifier ?? null; + if (assign) await this.postStartComment(); // K14: which build launched the task, so a replay starts it on the same one. if (this.supervisor.serverBuild) this.record.serverBuildAtLaunch = this.supervisor.serverBuild; this.record.createdAt = Date.now(); diff --git a/tests/restart-chaos/rewake.ts b/tests/restart-chaos/rewake.ts index 54d87c5a49c..369d433ba56 100644 --- a/tests/restart-chaos/rewake.ts +++ b/tests/restart-chaos/rewake.ts @@ -82,10 +82,13 @@ const phaseTwo = (nonce: string) => [ const w2Description = (nonce: string) => [ "This task has two phases.", - "Phase 1 (now): write a short brief on why idempotent retries matter for background job queues.", + "Phase 1 (when the board's comment asks you to start it): write a short brief on why idempotent retries matter for background job queues.", "Save it as this issue's document with key `brief` (title \"Brief\"), in markdown.", `It must contain the line \`${briefMarker(nonce)}\` exactly, on its own line, and at least three bullet points that start with "- ".`, "Then leave this task `in_progress` and end your turn. Do not mark it done or move it to review in phase 1. Do not create files or other tasks.", + "Phase 1 does not finish the task: report this turn with `paperclip_finish` and `reportedWorkDisposition` `yielded`, never `done`,", + `with a continuation of kind \`response_wake\` (summary "Waiting for the board's phase 2 wake", idempotencyKey \`w2-${nonce}\`).`, + "List phase 2 in remainingWork with blocksCompletion false: phase 1 is complete, and phase 2 waits for the board.", phaseTwo(nonce), ].join("\n"); @@ -271,6 +274,8 @@ export function gradeRewake(plan: Pick, artifact return { verdict: checks.every(check => check.ok) ? "pass" : "fail", checks }; } +export const rewakeStartComment = (nonce: string) => `Start phase 1 of this task now (${briefMarker(nonce)}).`; + const definition = (id: RewakeWorkload, label: string, description: (nonce: string) => string): WorkloadDefinition => ({ id, label, @@ -282,6 +287,7 @@ const definition = (id: RewakeWorkload, label: string, description: (nonce: stri budgetMs: 20 * 60_000, title: nonce => `Chaos ${id} re-wake ${nonce}`, description, + ...(id === "W2Q" ? {} : { startComment: rewakeStartComment }), collect: { documents: ["brief"], files: () => [], commands: [] }, grade: (nonce, artifacts) => gradeRewake({ workload: id, nonce }, artifacts), }); diff --git a/tests/restart-chaos/shape-cells.test.ts b/tests/restart-chaos/shape-cells.test.ts index 6170f0e51af..9700241cc3f 100644 --- a/tests/restart-chaos/shape-cells.test.ts +++ b/tests/restart-chaos/shape-cells.test.ts @@ -8,6 +8,7 @@ import { parseRecoveryStep, parseRecoveryStepReceipts, recoveryStepControlSpec, import { generationHome, parseK15DeployShape, redeployEnvironment, redeployIdentity } from "./redeploy.js"; import { awaitingRewake, gradeRewake, noteMarker, rewakeKind, rewakeMarker, rewakeStep, type RewakeArtifacts } from "./rewake.js"; import { planTasks, planWave, taskFaultSchedule } from "./scheduler.js"; +import { stableRequestId } from "./observer.js"; import type { OracleArtifacts, RunSample, TaskPlan, TaskRecord, TaskSample } from "./types.js"; import { workloadDefinitions } from "./workloads.js"; @@ -214,6 +215,27 @@ describe("K15 staging deploy shape", () => { expect(() => parseK15DeployShape({ env: "daytona", faults: ["K15"] }, "2-3", false)).toThrow("between 5 and 600"); }); + it("runs volume-free servers only on Daytona with the wave's object store", () => { + const image = `ghcr.io/x/y@sha256:${"a".repeat(64)}`; + expect(wave(["--env", "daytona", "--daytona-image", image, "--storage", "s3", "--ephemeral-server", "--faults", "K15", "--k15-fresh-home"])) + .toMatchObject({ ephemeralServer: true, storage: "s3", k15FreshHome: true }); + expect(wave(["--env", "daytona", "--daytona-image", image])).not.toHaveProperty("ephemeralServer"); + expect(wave(["--env", "daytona", "--daytona-image", image, "--storage", "s3", "--ephemeral-server"]).workloads.every(workload => !workloadDefinitions[workload].usesRepository)).toBe(true); + expect(workloadDefinitions.W2.startComment?.("n1")).toContain("CHAOS-n1"); + expect(workloadDefinitions.W2C.startComment).toBeDefined(); + expect(workloadDefinitions.W2Q.startComment).toBeUndefined(); + expect(stableRequestId("start-t01-n1")).toMatch(/^[0-9a-f]{8}-[0-9a-f]{4}-5[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/); + expect(stableRequestId("start-t01-n1")).toBe(stableRequestId("start-t01-n1")); + expect(stableRequestId("start-t02-n1")).not.toBe(stableRequestId("start-t01-n1")); + expect(wave(["--env", "daytona", "--daytona-image", image, "--faults", "K15", "--k15-fresh-home"])) + .toMatchObject({ ephemeralServer: true, storage: "s3", k15FreshHome: true }); + expect(() => wave(["--env", "daytona", "--daytona-image", image, "--storage", "local", "--faults", "K15", "--k15-fresh-home"])) + .toThrow("--k15-fresh-home"); + expect(() => wave(["--env", "daytona", "--daytona-image", image, "--ephemeral-server"])).toThrow("--storage s3"); + expect(() => wave(["--env", "local", "--storage", "s3", "--ephemeral-server"])).toThrow("--env daytona"); + expect(() => wave(["--env", "daytona", "--daytona-image", image, "--storage", "s3", "--ephemeral-server", "--workloads", "A1,B1"])).toThrow("B1"); + }); + it("drops the volume variables for a fresh home, and gives each generation its own home", () => { const identity = redeployIdentity("w-20261111T000000-s1", 1); expect(redeployEnvironment(identity, "/tmp/home")).toMatchObject({ RAILWAY_VOLUME_MOUNT_PATH: "/tmp/home" }); diff --git a/tests/restart-chaos/supervisor.ts b/tests/restart-chaos/supervisor.ts index 09f536120a2..7bc0cacc885 100644 --- a/tests/restart-chaos/supervisor.ts +++ b/tests/restart-chaos/supervisor.ts @@ -41,6 +41,7 @@ import type { FaultKind, FaultReceipt, TriggerId } from "./types.js"; import { isMultiServerFault, runMultiServerFault, type MultiServerKind, type MultiServerRecord, type ServerProcessWindow } from "./multi-server.js"; import { generationHome, nextRedeployIdentity, redeployEnvironment, redeployIdentity, redeployNodeArgs, redeployReceipt, type RedeployIdentity } from "./redeploy.js"; import { startEdgeProxy, type EdgeProxy } from "./edge-proxy.js"; +import { ephemeralServerEnvironment } from "./ephemeral-server.js"; import { CLASS_R_PRE_FAULT_DEADLINE_MS, classRRedeploys, isClassRFault, runPluginStall, type ClassRHooks } from "./class-r.js"; /** First boot applies every migration; a loaded host can need longer (RESTART_CHAOS_HEALTH_TIMEOUT_MS). */ @@ -173,6 +174,7 @@ export class ChaosSupervisor { redeployEvidence: ((faultId: string, phase: "stopped" | "before_start") => Promise) | null = null; /** K15 deploy shape (redeploy.ts `parseK15DeployShape`); main.ts sets it before `start`. */ deployShape: { overlap: boolean; freshHome: boolean } = { overlap: false, freshHome: false }; + ephemeralServer = false; /** `--k15-fresh-home`: the instance home and config of the serving generation (generation 0 uses `paths`). */ private serverHome: { home: string; configPath: string }; /** `--k15-overlap-seconds`: the edge on the wave's port, and the serving generation's own port behind it. */ @@ -286,10 +288,12 @@ export class ChaosSupervisor { // K15: the generation's emulated host name and Railway deployment identity; the volume is the instance home. ...(this.redeploy ? redeployEnvironment(this.redeploy.identity, this.deployShape.freshHome ? null : home) : {}), }; - const environment = buildPaperclipServerEnvironment(source, overrides); + const ephemeral = this.ephemeralServer ? await ephemeralServerEnvironment({ configPath: this.paths.configPath, temporaryRoot: this.paths.temporaryRoot, extraEnvironment }) : null; + const environment = buildPaperclipServerEnvironment(source, { ...overrides, ...ephemeral }); const workspaceStorage = extraEnvironment.PAPERCLIP_STORAGE_PROVIDER === "s3" ? Object.fromEntries(Object.entries(extraEnvironment).filter(([key]) => key.startsWith("PAPERCLIP_STORAGE_"))) : undefined; - assertIsolatedServerEnvironment(environment, { temporaryRoot, paperclipHome: home, configPath, ...(workspaceStorage ? { workspaceStorage } : {}) }); + assertIsolatedServerEnvironment(environment, { temporaryRoot, paperclipHome: home, configPath, ...(workspaceStorage ? { workspaceStorage } : {}), + ...(ephemeral ? { generatedSecretsMasterKey: ephemeral.PAPERCLIP_SECRETS_MASTER_KEY } : {}) }); this.environment = Object.fromEntries(Object.entries(environment).filter((entry): entry is [string, string] => entry[1] !== undefined)); await prepareSourceServerDependencies(root, this.environment, false); } diff --git a/tests/restart-chaos/types.ts b/tests/restart-chaos/types.ts index f3468a8be8a..8ad98fd558b 100644 --- a/tests/restart-chaos/types.ts +++ b/tests/restart-chaos/types.ts @@ -86,6 +86,7 @@ export interface WaveOptions { k15Overlap?: boolean; /** `--k15-fresh-home`: each deploy generation gets a new, empty instance home and no volume (redeploy.ts). */ k15FreshHome?: boolean; + ephemeralServer?: boolean; /** `--faults K0`: cells with no fault, for workloads whose shape is the fault (W2, W2Q, W2C). */ baseline?: boolean; /** diff --git a/tests/restart-chaos/workloads.ts b/tests/restart-chaos/workloads.ts index 52762f9660c..6f2a0a4299c 100644 --- a/tests/restart-chaos/workloads.ts +++ b/tests/restart-chaos/workloads.ts @@ -20,6 +20,7 @@ export interface WorkloadDefinition { title(nonce: string): string; description(nonce: string, agents?: WorkloadAgents): string; followUp?(nonce: string): string; + startComment?(nonce: string): string; stopBeforeFollowUp?: boolean; /** Files and commands the observer must collect for the T-oracle. */ collect: { documents: string[]; files(nonce: string): string[]; commands: Array<{ name: string; argv: string[] | ((nonce: string) => string[]) }>; diff --git a/tests/runner-e2e/harness-env.ts b/tests/runner-e2e/harness-env.ts index 3c222b4995f..47759191876 100644 --- a/tests/runner-e2e/harness-env.ts +++ b/tests/runner-e2e/harness-env.ts @@ -257,6 +257,7 @@ export function assertIsolatedServerEnvironment( paperclipHome: string; configPath: string; workspaceStorage?: NodeJS.ProcessEnv; + generatedSecretsMasterKey?: string; }, ) { const home = env.PAPERCLIP_HOME; @@ -285,6 +286,7 @@ export function assertIsolatedServerEnvironment( ...AMBIENT_PAPERCLIP_CREDENTIAL_KEYS, ...AMBIENT_EXTERNAL_STATE_KEYS, ]) { + if (key === "PAPERCLIP_SECRETS_MASTER_KEY" && expected.generatedSecretsMasterKey && env[key] === expected.generatedSecretsMasterKey) continue; if (env[key] && !(AMBIENT_EXTERNAL_STATE_KEYS.includes(key as typeof AMBIENT_EXTERNAL_STATE_KEYS[number]) && expected.workspaceStorage?.[key] === env[key] && expected.workspaceStorage?.PAPERCLIP_STORAGE_PROVIDER === "s3")) throw new Error( diff --git a/tests/runner-e2e/support.test.ts b/tests/runner-e2e/support.test.ts index 76d8b478ce0..54ce763b2f0 100644 --- a/tests/runner-e2e/support.test.ts +++ b/tests/runner-e2e/support.test.ts @@ -990,6 +990,32 @@ describe("runner E2E server isolation", () => { ).not.toThrow(); }); + it("admits only the wave's own generated secrets master key", () => { + const expected = { + temporaryRoot: "/tmp/cell", + paperclipHome: "/tmp/cell/paperclip-home", + configPath: "/tmp/cell/paperclip-home/instances/e2e/config.json", + }; + const env = (extra: NodeJS.ProcessEnv) => ({ + PAPERCLIP_HOME: expected.paperclipHome, + PAPERCLIP_CONFIG: expected.configPath, + XDG_CACHE_HOME: "/tmp/cell/xdg-cache", + PAPERCLIP_AGENT_JWT_SECRET: "generated-agent-jwt", + PAPERCLIP_DECISION_SIGNING_SECRET: "generated-decision-key", + PAPERCLIP_TOOL_ACTION_SIGNING_SECRET: "generated-tool-key", + BETTER_AUTH_SECRET: "generated-auth-key", + ...extra, + }); + expect(() => assertIsolatedServerEnvironment(env({ PAPERCLIP_SECRETS_MASTER_KEY: "wave-key" }), + { ...expected, generatedSecretsMasterKey: "wave-key" })).not.toThrow(); + expect(() => assertIsolatedServerEnvironment(env({ PAPERCLIP_SECRETS_MASTER_KEY: "ambient-key" }), expected)) + .toThrow("PAPERCLIP_SECRETS_MASTER_KEY"); + expect(() => assertIsolatedServerEnvironment(env({ PAPERCLIP_SECRETS_MASTER_KEY: "ambient-key" }), + { ...expected, generatedSecretsMasterKey: "wave-key" })).toThrow("PAPERCLIP_SECRETS_MASTER_KEY"); + expect(() => assertIsolatedServerEnvironment(env({ PAPERCLIP_SECRETS_MASTER_KEY: "wave-key", PAPERCLIP_SECRETS_MASTER_KEY_FILE: "/tmp/cell/key" }), + { ...expected, generatedSecretsMasterKey: "wave-key" })).toThrow("PAPERCLIP_SECRETS_MASTER_KEY_FILE"); + }); + it("uses absolute repository paths for the Playwright web server", () => { const command = runnerE2EWebServerCommand("/workspace/paperclip"); expect(command).toContain(