test(restart-chaos): K15 redeploy, F1 workspace symlink, and the staging failure census (#20679)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The restart-chaos harness (`tests/restart-chaos/`) restarts the
control plane while real agent tasks run. It grades whether every task
finishes without a human Retry.
> - The hosted staging stack deploys about every 40 minutes. In 48
hours, 276 runs failed and 63 tasks stayed blocked. About a quarter of
the failures came within 10 minutes of a deploy. The harness passed the
same areas.
> - A hosted deploy replaces the server container. The new server runs
in another process space and has a new deployment id. The PID-based "the
earlier owner exited" proofs never apply across that boundary. A harness
restart on one machine keeps the process space, so every PID proof
succeeds and the hosted recovery path is never tested.
> - This pull request adds K15, a redeploy fault. It also adds the
staging census that shows the gap.
> - The benefit: chaos waves can now exercise the recovery path that
staging runs after every deploy, on local and Daytona.

## Linked Issues or Issue Description

**What happened?**
Many staging tasks that a deploy crossed ended blocked. Examples are
`native_remote_recovery_lease_mismatch`,
`runner_remote_process_identity_unavailable`,
`prior_owner_terminal_state_indeterminate` and `PRP provider resume
state is unavailable`. The restart-chaos Daytona waves passed for the
same adapters.

**What did you expect?**
The chaos harness should reproduce the deploy conditions that staging
has, so that these failure classes show up in a wave first.

**Steps to reproduce**
1. Compare the staging run records' `controllerProcess.hostHash` and
`deployment.deploymentId` across a deploy. Both change on every deploy.
2. Compare the same fields across a harness K2 restart. Both stay the
same.
3. Read `controllerProcessExited` and `sameProcessSpace` in
`server/src/services/controller-process-identity.ts`. A different
`hostHash` means that no PID proof applies.

**Additional context**
The full census, the failure classes and the per-class harness gaps are
in `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`. The
task identifiers in it are neutral labels.

## What Changed

- New fault `K15` (redeploy), in `tests/restart-chaos/redeploy.ts` and
`redeploy-preload.mjs`. A K15 wave gives every server an emulated host
name through a `--import` preload, which changes the process-space hash.
It also gives Railway-shaped deployment variables with the instance home
as the exclusive volume. A K15 stops the server with SIGTERM and starts
the next generation, with a new host name and deployment id.
- `supervisor.ts`, `main.ts`, `config.ts` and `types.ts`: select K15
with `--faults K15`. It works on `--env local` and `--env daytona`. The
receipt records `redeploy` and the timeline records `redeploy_switched`.
Config refuses K15 together with MS1–MS3.
- New opt-in workload `F1` (`workloads.ts`): the agent keeps an absolute
symlink in its repository workspace. A staging save-back refused such a
link and blocked the task. The oracle needs `done` and the saved notes
file, with no Retry. `f1-symlink.test.ts` covers the prompt, the grading
and the option planning.
- `redeploy.test.ts`: identity derivation, the environment, the argument
order, a real child process that proves the preload changes both CJS and
ESM `hostname()`, option parsing and plan detection.
- `tests/restart-chaos/README.md`: a K15 section and a row in the
"Pieces" table.
- `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`: the
48-hour staging census, 12 failure classes, the deploy timeline, the
reason the harness missed each class, and proposed scenarios.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/redeploy.test.ts
tests/restart-chaos/scheduler.test.ts
tests/restart-chaos/repeated-restarts.test.ts`: 68 tests pass. The full
restart-chaos unit suite passes: 43 files, 516 tests.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`:
passes.
- Live Daytona waves on 8f6ca25976, image
`paperclip-daytona-runner@sha256:bc7ca92d…`:

  | Wave | Faults and cells | Result |
  | --- | --- | --- |
| `w-20261111T002313-s9301` | K15, opencode and claude, A1 and C1, reuse
lease, 6 tasks | 5 of 6 pass. t05 (OpenCode, C1) failed R5
`silent_dead_run`: 183 s without evidence after the redeploy, then
succeeded. |
  | `w-20261111T004241-s9301` | K2 control, same seed | 6 of 6 pass |
| `w-20261111T005008-s9302` | K15, warm sandbox, comment trigger, 4
tasks | 4 of 4 pass |
| `w-20261111T010000-s9303` | K15, opencode only, 8 tasks | 8 of 8 pass
|
  | `w-20261111T011044-s9303` | K2 control, same seed | 8 of 8 pass |
| `w-20261111T012523-s9304` | F1 with K2, codex and opencode, `--storage
s3` | 2 of 2 pass |

- Every K15 fault switched the generation, with a new host name and
deployment id on the same volume. The server was ready again in 14–24 s.
- K15 does not yet reproduce the staging refusal codes on current
master. The census document explains the open conditions, such as a code
change at the deploy.

## Risks

- Low risk. The change is in the test harness and docs only. No product
code changes.
- K15 is an emulation. It changes the host name only in the server
process (and forked Node children). It does not emulate a new kernel
boot or PID namespace. The cloud stop-proof service is not emulated, so
stop proofs that need it stay unsupported, as on a self-hosted Railway
deploy. The README lists these limits.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edit, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
DottaandClaude authored and GitHub committed 2026-11-10 20:05:17 -06:00
1 parent 7435abcc63
commit 52160eba35
12 files changed
+723 -13

No files matched your search

+72 -1
View File
@@ -292,6 +292,76 @@ receipt names both commits (`upgrade` in `faults.json`, `upgrade_switched` in th
timeline). K14 is local-only, because a Daytona runner must match its image, and
it is never part of a storm's default faults.
## Redeploy (K15, `redeploy.ts`)
```sh
node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env daytona --daytona-image $IMG --daytona-cpu 3 \
--adapters opencode,claude --workloads A1,C1 --faults K15 --daytona-reuse-lease --concurrency 3 --tasks 6 --seed 9301
```
The hosted staging stack deploys about every 40 minutes. Each deploy replaces
the server's container. The replacement has the same database and volume, but it
runs in another process space (host name, kernel boot and PID namespace) and
has a new Railway deployment id. The server's PID-based proof that an earlier
owner exited (`controller-process-identity.ts`, `owner-proof.ts`) never applies
across a deploy. Recovery must use the deployment proof or the lease deadlines
instead. K1 and K2 restart on the same machine, so every PID proof succeeds and
that path stays untested. See `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`.
K15 emulates the container replacement without a container:
- Every server in a wave that has a K15 is launched with a `--import` preload
(`redeploy-preload.mjs`). The preload makes `os.hostname()` return a
per-generation name, which changes the process-space hash.
- Every server gets Railway-shaped variables: `RAILWAY_PROJECT_ID`,
`RAILWAY_ENVIRONMENT_ID`, `RAILWAY_SERVICE_ID` and `RAILWAY_DEPLOYMENT_ID`,
plus `RAILWAY_VOLUME_ID` and `RAILWAY_VOLUME_MOUNT_PATH`, which is the
instance home.
- A K15 stops the server with SIGTERM, like K2. The replacement starts as the
next generation, with a new host name and deployment id, on the same project,
service and volume.
- Other faults in the same wave restart within the current generation, like a
platform restart of the same deployment.
- Identities derive from the wave id; a replay uses the source wave id, so it sees the same ones.
- Each K15 receipt records `redeploy` (both generations) in `faults.json`, and
`redeploy_switched` in the timeline.
Limits:
- The Daytona sandboxes and remote runners survive, as they do on staging.
- Local runners are stopped with the server's process tree.
- The cloud stop-proof service (`PAPERCLIP_CLOUD_*`) is not emulated, so stop
proofs that need it stay unsupported, as on any self-hosted Railway deploy.
- K15 does not combine with MS1–MS3, because a second server would run in the
real host's process space.
- Combine K15 with `--adapters opencode` and `--daytona-reuse-lease` or
`--daytona-warm` to match staging's workload mix.
## Absolute symlink in the workspace (F1)
```sh
node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env daytona --daytona-image $IMG --daytona-cpu 3 \
--adapters codex,opencode --workloads F1 --faults K2 --storage s3 --concurrency 2 --tasks 2 --seed 9304
```
F1 is opt-in. The agent creates `hosts-link-<nonce>`, which is an absolute
symlink to `/etc/hosts`, in its repository workspace and keeps it. It also
writes a notes file, and then marks the task done.
On staging, a remote save-back refused such a link
(`remote_workspace_archive_unsafe_symlink`). The save-back then used up its
retries and left the task blocked. Agents create absolute links in ordinary
work, for example in tool caches or `node_modules/.bin`.
The oracle passes when:
- the task is `done`;
- the notes file is saved, without a Retry.
How the product saves the link itself (skip it, keep it, or record it) is not
graded. F1 is meaningful with a remote workspace (`--env daytona`, with
`--storage s3` as on staging). Locally it is a plain repository task.
## Config faults (CF1–CF3, `config-wave.ts`)
```sh
@@ -991,7 +1061,7 @@ re-arms the task's own triggered faults instead of using their offsets.
| `grader.ts` | Pure R1–R6 grading over the samples, plus the universal checks. |
| `universal.ts`, `universal-audit.ts` | U1–U4 on every task, and their audit over saved waves. |
| `contention.ts` | L1 workload, its accounting oracle and the `--db-pressure` writers. |
| `workloads.ts` | A1 brief document, B1 python CLI with tests, B5 long bench, C1 comment while running, M1 delegation, G1 approval across a restart, H1 budget hard stop across a restart, with their T-oracles. |
| `workloads.ts` | A1 brief document, B1 python CLI with tests, B5 long bench, C1 comment while running, M1 delegation, G1 approval across a restart, H1 budget hard stop across a restart, F1 absolute symlink kept in the workspace, with their T-oracles. |
| `journeys.ts`, `journey-driver.ts`, `journey-oauth-app.ts`, `journey-matrix.ts` | Q2 and Q3 definitions and oracles, the board's side of each, Q3's loopback OAuth/MCP app, and the `--matrix` planner. |
| `evidence.ts`, `ledger.ts` | Evidence bundle for each failure and the JSONL ledger plus `SUMMARY.md`. |
| `daytona-image.ts` | Docker-free extraction of the image's runnerd and provider pack, cached per digest, and the controller/image content-ID check. |
@@ -1001,6 +1071,7 @@ re-arms the task's own triggered faults instead of using their offsets.
| `time-faults.ts`, `time-faults-config.ts` | TM1–TM3: process freezing, database aging with recorded rows, and the time-corrected API the grader reads through. |
| `multi-server.ts` | MS1–MS3: the second server's steps, the seeded holds, and the cross-server invariant over the wave database. |
| `repeated-restarts.ts` | RR1–RR3: seeded restart steps, `--rr-gap-seconds`, and the stop-and-replace loop the supervisor runs. |
| `redeploy.ts`, `redeploy-preload.mjs` | K15: per-generation emulated host name and Railway deployment identity, and the preload that applies the host name. |
| `daytona-sweep.ts` | Daytona R6: read-only reconciliation of this wave's labelled sandboxes with leases; cleanup only through Paperclip. |
## Output (gitignored)
+10 -3
View File
@@ -4,7 +4,7 @@ import { parseArgs } from "node:util";
import { applyJourneyOptions } from "./journey-matrix.js";
import { takesJourneyTriggers } from "./journeys.js";
import { STORAGE_FAULT_KINDS, type StorageFaultKind } from "./storage-chaos.js";
import { validateMultiServerOptions } from "./multi-server.js";
import { isMultiServerFault, validateMultiServerOptions } from "./multi-server.js";
import { parseTimeFaultOptions } from "./time-faults-config.js";
import { parseStorageTrigger } from "./storage-trigger.js";
import { parseNativeTrigger, parseNativeTriggerOutage } from "./native-trigger.js";
@@ -17,7 +17,10 @@ import {
type AdapterId, type EnvironmentKind, type FaultKind, type TriggerId, type WaveOptions, type WorkloadId,
} from "./types.js";
/** A storm's default faults: every restart kind except K14, which needs an older build (`--upgrade-from`). */
/**
* A storm's default faults: every restart kind except K14, which needs an older build (`--upgrade-from`),
* and K15, which changes the deploy identity of every later server (pass `--faults` to include it).
*/
export const STORM_DEFAULT_FAULTS: FaultKind[] = ["K1", "K2", "K4", "K6"];
export const CREDENTIAL_FOR_ADAPTER: Record<AdapterId, "OPENAI_API_KEY" | "ANTHROPIC_API_KEY" | "OPENROUTER_API_KEY"> = {
@@ -176,7 +179,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
env,
adapters,
// The governance scenarios (G1, H1), the journeys (Q1–Q3) and the contention run (L1) are opt-in: H1 needs a wave of its own.
workloads: list<WorkloadId>(values.workloads, workloadChoices, "workloads", workloadChoices.filter(id => id !== "G1" && id !== "H1" && id !== "R1" && id !== "S1" && id !== "L1"
workloads: list<WorkloadId>(values.workloads, workloadChoices, "workloads", workloadChoices.filter(id => id !== "G1" && id !== "H1" && id !== "R1" && id !== "S1" && id !== "L1" && id !== "F1"
&& !takesJourneyTriggers(id) && (id !== "M1" || adapters.length >= 2))),
// A crash-point wave schedules no fault; K1 is the restart its self-kill gets.
faults: allFaults.filter((fault): fault is FaultKind => !isProviderFault(fault)),
@@ -233,6 +236,10 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
throw new Error("--provider-fault-message, --provider-fault-delays and --provider-fault-restart need a provider fault (PV1, PV2 or PV3) in --faults");
}
validateMultiServerOptions(options);
// K15 emulates one deploy identity per generation; a second server beside it would run in the real host's process space.
if (options.faults.includes("K15") && options.faults.some(isMultiServerFault)) {
throw new Error("K15 (redeploy) does not combine with MS1-MS3: the second server would not share the emulated deploy identity");
}
Object.assign(options, parseTimeFaultOptions(options, values["time-age-hours"]));
Object.assign(options, parseRepeatedRestartOptions(options, values["rr-gap-seconds"], values["rr-stop-on"]));
const upgradeFrom = values["upgrade-from"];
+53
View File
@@ -0,0 +1,53 @@
import { describe, expect, it } from "vitest";
import { parseCommand } from "./config.js";
import { WORKLOAD_TRIGGERS } from "./journeys.js";
import { planTasks } from "./scheduler.js";
import type { OracleArtifacts } from "./types.js";
import { f1LinkName, workloadDefinitions } from "./workloads.js";
const NONCE = "abc123de";
const LISTING = `lrwxrwxrwx 1 daytona daytona 10 Nov 11 01:30 hosts-link-${NONCE} -> /etc/hosts`;
const notes = (text: string) => ({ [`NOTES-${NONCE}.md`]: text });
const artifacts = (overrides: Partial<OracleArtifacts> = {}): OracleArtifacts => ({
issueStatus: "done", documents: {}, agentComments: [], files: notes(`F1-${NONCE}\n${LISTING}\n`), commands: {},
workspacePath: "/workspace/chaos", ...overrides,
}) as OracleArtifacts;
describe("F1: absolute symlink kept in the repository workspace", () => {
const f1 = workloadDefinitions.F1;
it("asks for the absolute link and the notes file in a repository workspace", () => {
expect(f1.usesRepository).toBe(true);
const text = f1.description(NONCE);
expect(text).toContain(`ln -s /etc/hosts ${f1LinkName(NONCE)}`);
expect(text).toContain(`NOTES-${NONCE}.md`);
expect(f1.collect.files(NONCE)).toEqual([`NOTES-${NONCE}.md`]);
});
it("passes only when the task is done and its saved notes file is in the workspace", () => {
expect(f1.grade(NONCE, artifacts()).verdict).toBe("pass");
expect(f1.grade(NONCE, artifacts({ issueStatus: "blocked" })).verdict).toBe("fail");
expect(f1.grade(NONCE, artifacts({ files: {} })).verdict).toBe("fail");
expect(f1.grade(NONCE, artifacts({ workspacePath: null })).verdict).toBe("fail");
});
it("needs the agent's ls -l line proving the absolute link existed before the save-back", () => {
expect(f1.description(NONCE)).toContain(`ls -l ${f1LinkName(NONCE)}`);
// Skipped link: notes only.
expect(f1.grade(NONCE, artifacts({ files: notes(`F1-${NONCE}\n`) })).verdict).toBe("fail");
// A regular file of the same name, or a link elsewhere, is not the link under test.
expect(f1.grade(NONCE, artifacts({ files: notes(`F1-${NONCE}\n-rw-r--r-- 1 d d 3 Nov 11 01:30 hosts-link-${NONCE}\n`) })).verdict).toBe("fail");
expect(f1.grade(NONCE, artifacts({ files: notes(`F1-${NONCE}\n${LISTING.replace("/etc/hosts", "etc/hosts")}\n`) })).verdict).toBe("fail");
expect(f1.grade(NONCE, artifacts({ files: notes(`F1-${NONCE}\n${LISTING.replace(NONCE, "ffffffff")}\n`) })).verdict).toBe("fail");
});
it("is opt-in, time-triggered, and plans with restart faults on Daytona", () => {
expect(WORKLOAD_TRIGGERS.F1).toEqual(["time"]);
const defaults = parseCommand(["--seed", "1"]);
expect(defaults.mode === "wave" && defaults.options.workloads).not.toContain("F1");
const parsed = parseCommand(["--env", "daytona", "--daytona-image", `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:${"b".repeat(64)}`,
"--workloads", "F1", "--faults", "K2,K15", "--adapters", "codex", "--tasks", "2", "--seed", "3"]);
if (parsed.mode !== "wave") throw new Error("expected a wave");
expect(planTasks(parsed.options).map(task => task.workload)).toEqual(["F1", "F1"]);
});
});
+1 -1
View File
@@ -35,7 +35,7 @@ export type JourneyTrigger = (typeof JOURNEY_TRIGGERS)[number];
/** Each workload's own fault boundaries, for the journey × fault matrix. */
export const WORKLOAD_TRIGGERS: Record<WorkloadId, readonly TriggerId[]> = {
A1: ["time"], B1: ["time"], B5: ["tool"], C1: ["comment", "result"], M1: ["time"], G1: ["approval"],
H1: ["paused"], R1: ["time"], S1: ["stop"], L1: ["time"], Q1: JOURNEY_TRIGGERS, Q2: JOURNEY_TRIGGERS, Q3: JOURNEY_TRIGGERS,
H1: ["paused"], R1: ["time"], S1: ["stop"], L1: ["time"], F1: ["time"], Q1: JOURNEY_TRIGGERS, Q2: JOURNEY_TRIGGERS, Q3: JOURNEY_TRIGGERS,
};
/** Q3 reviews must survive a delay longer than the implicit deadline #20340 removed. */
+4 -1
View File
@@ -38,6 +38,7 @@ import { collectWaveSpeed, speedMarkdown } from "./wave-speed.js";
import { TaskObserver, childSpeedTasksInTurn } from "./observer.js";
import { closeJourneyApps } from "./journey-driver.js";
import { matrixMayLaunch, planMatrixWave } from "./journey-matrix.js";
import { redeployWave } from "./redeploy.js";
import { StormRun, TASK_FAULT_LOG, observedFaultSchedule, planWave, replayOnCurrentBuild, replayRecordFor, replayWave, type TaskFaultLog } from "./scheduler.js";
import { chaosLifecycleOverrides, seedChaosLab } from "./seed.js";
import { ChaosSupervisor } from "./supervisor.js";
@@ -200,7 +201,9 @@ async function main(): Promise<number> {
const batchSize = crashPoints.length ? Math.max(1, Math.floor(plan.tasks.length / crashPoints.length)) : 0;
if (crashPoint && plan.tasks.length !== batchSize * crashPoints.length) throw new Error(`--tasks must be a multiple of the ${crashPoints.length} crash point(s)`);
const supervisor = await ChaosSupervisor.create({ repositoryRoot, waveDir, timeline,
...(plan.options.upgradeFrom ?? plan.options.serverFrom ? { upgradeFrom: plan.options.upgradeFrom ?? plan.options.serverFrom } : {}) });
...(plan.options.upgradeFrom ?? plan.options.serverFrom ? { upgradeFrom: plan.options.upgradeFrom ?? plan.options.serverFrom } : {}),
// K15: every server of the wave runs under an emulated deploy identity; a replay derives the source wave's ones.
...(redeployWave(plan) ? { redeployWaveId: plan.replayOf?.waveId ?? plan.waveId } : {}) });
// Crash-point suite: one booted server, one point at a time. Each point gets
// its own batch of tasks; the point is armed once the batch launched and the
// server kills itself at its nth hit (restarted like K1). Invariants are
+12
View File
@@ -0,0 +1,12 @@
// K15 (redeploy.ts): give the server process an emulated host name, so its process-space hash
// differs from the previous deploy's, as a replaced container's does. Loaded with `--import`
// before the server. Node children forked with the server's execArgv inherit it, as they share its container.
import os from "node:os";
import { syncBuiltinESMExports } from "node:module";
const hostname = process.env.RESTART_CHAOS_REDEPLOY_HOSTNAME?.trim();
if (hostname) {
os.hostname = () => hostname;
// `import { hostname } from "node:os"` reads the ESM export, which only follows the CJS object after a sync.
syncBuiltinESMExports();
}
+93
View File
@@ -0,0 +1,93 @@
import { spawnSync } from "node:child_process";
import { describe, expect, it } from "vitest";
import { parseCommand } from "./config.js";
import {
REDEPLOY_HOSTNAME_ENV, REDEPLOY_PRELOAD_PATH, nextRedeployIdentity, redeployEnvironment, redeployIdentity, redeployNodeArgs,
redeployReceipt, redeployWave,
} from "./redeploy.js";
import { planTasks } from "./scheduler.js";
import { FAULT_SIGNAL } from "./supervisor.js";
import type { WavePlan } from "./types.js";
const UUID = /^[a-f0-9]{8}-[a-f0-9]{4}-5[a-f0-9]{3}-[89ab][a-f0-9]{3}-[a-f0-9]{12}$/;
const IMAGE = `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:${"a".repeat(64)}`;
describe("K15 redeploy identity", () => {
it("keeps the service and volume and changes the host and deployment on every generation", () => {
const first = redeployIdentity("w-1", 0);
const second = nextRedeployIdentity("w-1", first);
expect(second.generation).toBe(1);
expect(second.hostname).not.toBe(first.hostname);
expect(second.deploymentId).not.toBe(first.deploymentId);
for (const key of ["projectId", "environmentId", "serviceId", "volumeId"] as const) expect(second[key]).toBe(first[key]);
for (const id of [first.projectId, first.environmentId, first.serviceId, first.deploymentId, first.volumeId]) expect(id).toMatch(UUID);
});
it("is deterministic per wave, so a replay derives the same identities", () => {
expect(redeployIdentity("w-1", 3)).toEqual(redeployIdentity("w-1", 3));
expect(redeployIdentity("w-2", 0).deploymentId).not.toBe(redeployIdentity("w-1", 0).deploymentId);
expect(() => redeployIdentity("w-1", -1)).toThrow(/non-negative/);
});
it("names the Railway variables the server reads as its deployment, with the instance home as the exclusive volume", () => {
const identity = redeployIdentity("w-1", 2);
expect(redeployEnvironment(identity, "/tmp/chaos/home")).toEqual({
[REDEPLOY_HOSTNAME_ENV]: identity.hostname,
RAILWAY_PROJECT_ID: identity.projectId, RAILWAY_ENVIRONMENT_ID: identity.environmentId, RAILWAY_SERVICE_ID: identity.serviceId,
RAILWAY_DEPLOYMENT_ID: identity.deploymentId, RAILWAY_VOLUME_ID: identity.volumeId, RAILWAY_VOLUME_MOUNT_PATH: "/tmp/chaos/home",
});
expect(redeployReceipt(redeployIdentity("w-1", 0), identity)).toMatchObject({ fromGeneration: 0, toGeneration: 2,
toDeploymentId: identity.deploymentId, volumeId: identity.volumeId });
});
it("loads the preload ahead of the server's own --import loader", () => {
expect(redeployNodeArgs(["--import", "/x/tsx/loader.mjs", "entry.ts"])).toEqual(["--import", REDEPLOY_PRELOAD_PATH, "--import", "/x/tsx/loader.mjs", "entry.ts"]);
});
it("changes os.hostname for both the CommonJS object and ESM named imports, as processSpaceHash reads it", () => {
const script = "import os from 'node:os'; import { hostname } from 'node:os'; console.log(JSON.stringify([os.hostname(), hostname()]));";
const run = (env: NodeJS.ProcessEnv) => spawnSync(process.execPath, ["--import", REDEPLOY_PRELOAD_PATH, "--input-type=module", "-e", script],
{ env: { PATH: process.env.PATH, ...env }, encoding: "utf8" });
const emulated = run({ [REDEPLOY_HOSTNAME_ENV]: "chaos-deploy-test" });
expect(emulated.status).toBe(0);
expect(JSON.parse(emulated.stdout)).toEqual(["chaos-deploy-test", "chaos-deploy-test"]);
// Without the variable the preload changes nothing.
const real = run({});
expect(real.status).toBe(0);
const [cjs, esm] = JSON.parse(real.stdout) as string[];
expect(cjs).toBe(esm);
expect(cjs).not.toBe("chaos-deploy-test");
});
});
describe("K15 wave options", () => {
it("is a graceful stop, selectable on local and Daytona like K2", () => {
expect(FAULT_SIGNAL.K15).toBe("SIGTERM");
const local = parseCommand(["--faults", "K15", "--adapters", "opencode", "--workloads", "A1,C1", "--seed", "1"]);
expect(local.mode === "wave" && local.options.faults).toEqual(["K15"]);
const daytona = parseCommand(["--env", "daytona", "--daytona-image", IMAGE, "--faults", "K2,K15", "--adapters", "opencode",
"--workloads", "C1", "--daytona-reuse-lease", "--seed", "1"]);
if (daytona.mode !== "wave") throw new Error("expected a wave");
expect(daytona.options.faults).toEqual(["K2", "K15"]);
expect(daytona.options.daytonaReuseLease).toBe(true);
});
it("refuses a second server beside an emulated deploy", () => {
expect(() => parseCommand(["--faults", "K15,MS1", "--workloads", "B5,Q1", "--concurrency", "1", "--seed", "1"])).toThrow(/K15 \(redeploy\) does not combine with MS1-MS3/);
});
it("plans K15 cells with no hold and marks the wave as a redeploy wave", () => {
const parsed = parseCommand(["--faults", "K15", "--adapters", "codex,opencode", "--workloads", "A1,C1", "--tasks", "4", "--seed", "7"]);
if (parsed.mode !== "wave") throw new Error("expected a wave");
const tasks = planTasks(parsed.options);
expect(tasks.every(task => task.fault === "K15" && task.holdMs === 0)).toBe(true);
const plan = { options: parsed.options, tasks } as Pick<WavePlan, "options" | "tasks" | "storm">;
expect(redeployWave(plan)).toBe(true);
const k2 = parseCommand(["--faults", "K2", "--workloads", "A1", "--tasks", "2", "--seed", "7"]);
if (k2.mode !== "wave") throw new Error("expected a wave");
expect(redeployWave({ options: k2.options, tasks: planTasks(k2.options) })).toBe(false);
// A replay of a K2 task that a K15 crossed still needs the emulated identity.
expect(redeployWave({ options: k2.options, tasks: [{ ...planTasks(k2.options)[0]!,
replayFaults: [{ kind: "K15", holdMs: 0, atMs: 1_000 }] }] })).toBe(true);
});
});
+96
View File
@@ -0,0 +1,96 @@
/**
* K15 (redeploy): what a hosted deploy does to the control plane, beyond K2's graceful stop.
*
* On the hosted staging stack every deploy replaces the server's container. The replacement runs
* in another process space (host name, kernel boot and PID namespace), so the server's PID-based
* "the previous owner has exited" proofs (`server/src/services/controller-process-identity.ts`,
* `owner-proof.ts`) never apply. It also carries a new Railway deployment id on the same exclusive
* volume, which is the proof hosted recovery uses instead. A K1/K2 restart on one machine keeps the
* process space, so every PID proof succeeds there and this path is never exercised.
*
* K15 emulates both changes without a container: each server of a K15 wave sees an emulated host
* name (a `--import` preload replaces `os.hostname`, which changes the process-space hash) and
* Railway-shaped deployment variables naming one stable project, environment, service and volume
* and a deployment id per generation. A K15 stops the server with SIGTERM like K2 and starts the
* replacement as the next generation. Other faults in the same wave restart within a generation,
* like a platform restart of the same deployment.
*/
import { createHash } from "node:crypto";
import { fileURLToPath } from "node:url";
import type { WavePlan } from "./types.js";
/** Whether any server of this wave can be redeployed: a planned, storm or replayed K15. */
export function redeployWave(plan: Pick<WavePlan, "options" | "tasks" | "storm">): boolean {
return plan.options.faults.includes("K15") || (plan.storm?.faults.some(fault => fault.kind === "K15") ?? false)
|| plan.tasks.some(task => task.fault === "K15" || (task.replayFaults?.some(fault => fault.kind === "K15") ?? false));
}
/** The preload that applies the emulated host name before the server loads. */
export const REDEPLOY_PRELOAD_PATH = fileURLToPath(new URL("./redeploy-preload.mjs", import.meta.url));
/** Read by the preload: the host name `os.hostname()` returns in the server process. */
export const REDEPLOY_HOSTNAME_ENV = "RESTART_CHAOS_REDEPLOY_HOSTNAME";
export interface RedeployIdentity {
/** 0 for the wave's first server; each K15 adds one. */
generation: number;
hostname: string;
projectId: string;
environmentId: string;
serviceId: string;
deploymentId: string;
volumeId: string;
}
/** A deterministic UUID-shaped id for one wave, role and generation (no randomness: replays match). */
function uuidFor(waveId: string, role: string, generation = 0): string {
const hex = createHash("sha256").update(`restart-chaos-redeploy:${waveId}:${role}:${generation}`).digest("hex");
// Version 5-style layout so validators that require a UUID shape accept it.
const variant = ((parseInt(hex[16]!, 16) & 0x3) | 0x8).toString(16);
return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-5${hex.slice(13, 16)}-${variant}${hex.slice(17, 20)}-${hex.slice(20, 32)}`;
}
/** The identity of the server that serves generation `generation` of this wave. */
export function redeployIdentity(waveId: string, generation: number): RedeployIdentity {
if (!Number.isInteger(generation) || generation < 0) throw new Error(`Redeploy generation must be a non-negative integer, got ${generation}`);
return {
generation,
hostname: `chaos-deploy-${createHash("sha256").update(`${waveId}:${generation}`).digest("hex").slice(0, 10)}`,
projectId: uuidFor(waveId, "project"),
environmentId: uuidFor(waveId, "environment"),
serviceId: uuidFor(waveId, "service"),
deploymentId: uuidFor(waveId, "deployment", generation),
volumeId: uuidFor(waveId, "volume"),
};
}
/** The next deploy of the same service: same project, environment, service and volume. */
export function nextRedeployIdentity(waveId: string, current: RedeployIdentity): RedeployIdentity {
return redeployIdentity(waveId, current.generation + 1);
}
/**
* Server environment for one generation. The volume mount is the instance home: on staging the
* instance directory lives on the exclusive volume that survives each deploy.
*/
export function redeployEnvironment(identity: RedeployIdentity, volumeMountPath: string): Record<string, string> {
return {
[REDEPLOY_HOSTNAME_ENV]: identity.hostname,
RAILWAY_PROJECT_ID: identity.projectId,
RAILWAY_ENVIRONMENT_ID: identity.environmentId,
RAILWAY_SERVICE_ID: identity.serviceId,
RAILWAY_DEPLOYMENT_ID: identity.deploymentId,
RAILWAY_VOLUME_ID: identity.volumeId,
RAILWAY_VOLUME_MOUNT_PATH: volumeMountPath,
};
}
/** Node arguments that load the preload ahead of the server entry's own `--import` loader. */
export function redeployNodeArgs(args: readonly string[]): string[] {
return ["--import", REDEPLOY_PRELOAD_PATH, ...args];
}
/** What a K15 receipt records: both generations, so a replay and the evidence name them. */
export function redeployReceipt(from: RedeployIdentity, to: RedeployIdentity) {
return { fromGeneration: from.generation, toGeneration: to.generation, fromHostname: from.hostname, toHostname: to.hostname,
fromDeploymentId: from.deploymentId, toDeploymentId: to.deploymentId, volumeId: to.volumeId };
}
+31 -5
View File
@@ -39,6 +39,7 @@ import { isRepeatedRestartFault, newRepeatedRestartRecord, recordFirstRestart, r
type RepeatedRestartRecord, type RestartStep } from "./repeated-restarts.js";
import type { FaultKind, FaultReceipt, TriggerId } from "./types.js";
import { isMultiServerFault, runMultiServerFault, type MultiServerKind, type MultiServerRecord, type ServerProcessWindow } from "./multi-server.js";
import { nextRedeployIdentity, redeployEnvironment, redeployIdentity, redeployNodeArgs, redeployReceipt, type RedeployIdentity } from "./redeploy.js";
/** First boot applies every migration; a loaded host can need longer (RESTART_CHAOS_HEALTH_TIMEOUT_MS). */
const HEALTH_TIMEOUT_MS = (() => {
@@ -48,7 +49,7 @@ const HEALTH_TIMEOUT_MS = (() => {
const READY_TIMEOUT_MS = 300_000;
const STOP_TIMEOUT_MS = 30_000;
export const FAULT_SIGNAL: Record<FaultKind, FaultReceipt["signal"]> = { K1: "SIGKILL", K2: "SIGTERM", K4: "SIGKILL", K6: "SIGKILL", K14: "SIGTERM",
export const FAULT_SIGNAL: Record<FaultKind, FaultReceipt["signal"]> = { K1: "SIGKILL", K2: "SIGTERM", K4: "SIGKILL", K6: "SIGKILL", K14: "SIGTERM", K15: "SIGTERM",
TM1: "SIGSTOP", TM2: "SIGSTOP", TM3: "SIGKILL", RR1: "SIGKILL", RR2: "SIGKILL", RR3: "SIGKILL",
// MS1 leaves the primary alone; MS2 freezes it; MS3 stops it gracefully (multi-server.ts).
MS1: "none", MS2: "SIGSTOP", MS3: "SIGTERM" };
@@ -154,13 +155,16 @@ export class ChaosSupervisor {
private serverRoot: string;
/** The environment overrides the wave passed to `start`, kept for every restart and for the K14 switch. */
private extraEnvironment: Record<string, string> = {};
/** K15 waves: the wave id the emulated deploy identities derive from, and the generation serving now. */
private redeploy: { waveId: string; identity: RedeployIdentity } | null = null;
/** Wave secrets, kept across a K14 switch as an in-place upgrade keeps its configuration. */
private readonly secrets = { PAPERCLIP_AGENT_JWT_SECRET: secret(), PAPERCLIP_DECISION_SIGNING_SECRET: secret(),
PAPERCLIP_TOOL_ACTION_SIGNING_SECRET: secret(), BETTER_AUTH_SECRET: secret() };
private constructor(readonly repositoryRoot: string, readonly port: number, readonly paths: SupervisorPaths,
private readonly timeline: Timeline, upgradeFrom: string | null) {
private readonly timeline: Timeline, upgradeFrom: string | null, redeployWaveId: string | null) {
this.serverRoot = upgradeFrom ?? repositoryRoot;
if (redeployWaveId) this.redeploy = { waveId: redeployWaveId, identity: redeployIdentity(redeployWaveId, 0) };
this.baseUrl = `http://127.0.0.1:${port}`;
this.log = createWriteStream(paths.serverLogPath, { flags: "a", mode: 0o600 });
this.stopper = createRunnerE2EServerStopper({
@@ -171,13 +175,16 @@ export class ChaosSupervisor {
});
}
static async create(input: { repositoryRoot: string; waveDir: string; timeline: Timeline; upgradeFrom?: string }): Promise<ChaosSupervisor> {
/** `redeployWaveId`: a K15 wave; every server gets the emulated deploy identity of its generation (redeploy.ts). */
static async create(input: { repositoryRoot: string; waveDir: string; timeline: Timeline; upgradeFrom?: string;
redeployWaveId?: string }): Promise<ChaosSupervisor> {
const temporaryRoot = await createRunnerE2ETemporaryRoot();
const home = path.join(temporaryRoot, "home");
const paths: SupervisorPaths = { temporaryRoot, home, configPath: path.join(home, "instances", "default", "config.json"),
serverLogPath: path.join(input.waveDir, "server.log") };
const port = await reserveRunnerE2EServerPort();
const supervisor = new ChaosSupervisor(input.repositoryRoot, port, paths, input.timeline, input.upgradeFrom ?? null);
const supervisor = new ChaosSupervisor(input.repositoryRoot, port, paths, input.timeline, input.upgradeFrom ?? null,
input.redeployWaveId ?? null);
input.timeline.add("supervisor_created", { temporaryRoot, port, ...(input.upgradeFrom
? { upgradeFrom: input.upgradeFrom, upgradeFromSha: checkoutSha(input.upgradeFrom) } : {}) });
return supervisor;
@@ -241,6 +248,8 @@ export class ChaosSupervisor {
PAPERCLIP_IN_WORKTREE: "false",
...waveServerUrlEnvironment(this.port),
...extraEnvironment,
// K15: the generation's emulated host name and Railway deployment identity; the volume is the instance home.
...(this.redeploy ? redeployEnvironment(this.redeploy.identity, home) : {}),
};
const environment = buildPaperclipServerEnvironment(source, overrides);
const workspaceStorage = extraEnvironment.PAPERCLIP_STORAGE_PROVIDER === "s3"
@@ -257,7 +266,9 @@ export class ChaosSupervisor {
const control = this.crashPointControl;
const env = control ? { ...this.environment, PAPERCLIP_CRASH_POINT_CONTROL: control.controlPath,
PAPERCLIP_CRASH_POINT_RECEIPT: control.receiptPath } : this.environment;
const child = spawn(process.execPath, runnerE2ETypeScriptProcessArgs(root, entry, ["onboard", "--yes", "--run"]),
const args = runnerE2ETypeScriptProcessArgs(root, entry, ["onboard", "--yes", "--run"]);
// K15 waves: the emulated host name of the serving deploy generation (redeploy.ts).
const child = spawn(process.execPath, this.redeploy ? redeployNodeArgs(args) : args,
{ cwd: root, env, stdio: ["ignore", "pipe", "pipe"], detached: runnerE2EServerDetached });
this.child = child;
const window = this.trackWindow(child, "primary");
@@ -484,6 +495,8 @@ export class ChaosSupervisor {
receipt.restartedAt = Date.now();
// K14: the replacement is the current build, on the same database and instance directory.
if (kind === "K14") receipt.upgrade = await this.switchToCurrentBuild(receipt.faultId);
// K15: the replacement is the next deploy: a new process space and deployment id on the same volume.
if (kind === "K15") receipt.redeploy = this.advanceDeploy(receipt.faultId);
this.timeline.add("fault_restarting", { faultId: receipt.faultId });
return this.spawnServer();
},
@@ -668,6 +681,19 @@ export class ChaosSupervisor {
throw new Error(`Second server did not report ready within ${READY_TIMEOUT_MS}ms`);
}
/** K15: move the next server to the next deploy generation (redeploy.ts). */
private advanceDeploy(faultId: string): NonNullable<FaultReceipt["redeploy"]> {
if (!this.redeploy || !this.environment) throw new Error("K15 needs a redeploy wave (ChaosSupervisor.create redeployWaveId)");
const from = this.redeploy.identity;
const to = nextRedeployIdentity(this.redeploy.waveId, from);
this.redeploy.identity = to;
Object.assign(this.environment, redeployEnvironment(to, this.paths.home));
const receipt = redeployReceipt(from, to);
this.timeline.add("redeploy_switched", { faultId, ...receipt });
this.log.write(`\n[restart-chaos] ${faultId} K15 redeploy ${from.hostname}/${from.deploymentId} -> ${to.hostname}/${to.deploymentId}\n`);
return receipt;
}
/**
* K14: move the server from the older build to this checkout. Only the first K14 switches; a later
* one restarts the current build (recorded with `switched: false`).
+7 -2
View File
@@ -6,11 +6,13 @@ import type { TimeFaultRecord } from "./time-faults.js";
/**
* K14 (upgrade mid-run): a graceful stop, then the replacement starts from the current build (`--upgrade-from`).
* K15 (redeploy, redeploy.ts): a graceful stop, then the replacement starts in a new emulated process space with a new
* Railway-shaped deployment id on the same volume, as a hosted deploy replaces the server's container.
* TM1–TM3 (time faults, time-faults.ts): freeze the server, sleep with aged rows, or an aged outage.
* RR1–RR3 (repeated restarts, repeated-restarts.ts): a K1, then more restarts while the replacement recovers.
* MS1–MS3 (two servers, one database, multi-server.ts): a second server beside the first.
*/
export const FAULT_KINDS = ["K1", "K2", "K4", "K6", "K14", "TM1", "TM2", "TM3", "RR1", "RR2", "RR3", "MS1", "MS2", "MS3"] as const;
export const FAULT_KINDS = ["K1", "K2", "K4", "K6", "K14", "K15", "TM1", "TM2", "TM3", "RR1", "RR2", "RR3", "MS1", "MS2", "MS3"] as const;
export type FaultKind = (typeof FAULT_KINDS)[number];
/** The shortest seeded K4 outage (scheduler.ts K4_HOLD_RANGE_MS), also the K4 hold after a provider fault. */
export const K4_MIN_HOLD_MS = 120_000;
@@ -25,8 +27,9 @@ export type TaskFaultKind = FaultKind | "K13" | "K0" | ProviderFaultKind;
* G1: an approval-gated action across a restart. H1: a budget hard stop crossed mid-run with a restart (its own wave).
* Q1: a saved question (question-handoff.ts). Q2, Q3: journeys.ts.
* L1: high-output, high-accounting run for database contention (contention.ts).
* F1: an absolute symlink pointing out of the repository workspace, kept at rest (staging save-back class F).
*/
export const WORKLOAD_IDS = ["A1", "B1", "B5", "C1", "M1", "G1", "H1", "R1", "S1", "Q1", "L1", "Q2", "Q3"] as const;
export const WORKLOAD_IDS = ["A1", "B1", "B5", "C1", "M1", "G1", "H1", "R1", "S1", "Q1", "L1", "Q2", "Q3", "F1"] as const;
export type WorkloadId = (typeof WORKLOAD_IDS)[number];
export const ADAPTER_IDS = ["codex", "claude", "opencode"] as const;
@@ -281,6 +284,8 @@ export interface FaultReceipt {
crashPoint?: { name: string; hit: number; pid: number; token?: string };
/** K14: the build the stopped server ran and the build its replacement runs. */
upgrade?: { fromRoot: string; fromSha: string | null; toRoot: string; toSha: string | null; switched: boolean };
/** K15: the deploy generation, emulated host name and deployment id before and after. */
redeploy?: ReturnType<typeof import("./redeploy.js").redeployReceipt>;
/** TM1–TM3: what was frozen and which rows were aged. */
timeFault?: TimeFaultRecord;
/** RR1–RR3: every stop and replacement of the sequence, the first kill included. */
+35
View File
@@ -41,6 +41,10 @@ const briefMarker = (nonce: string) => `CHAOS-${nonce}`;
const followUpMarker = (nonce: string) => `FOLLOWUP-${nonce}`;
const continueMarker = (nonce: string) => `CONTINUE-${nonce}`;
const notesFile = (nonce: string) => `NOTES-${nonce}.md`;
/** F1: the absolute symlink the agent creates and keeps in the repository root. */
export const f1LinkName = (nonce: string) => `hosts-link-${nonce}`;
/** F1: an `ls -l` line for that link: a symlink (`l` mode) naming the link and its absolute target. */
export const f1LinkListing = (nonce: string) => new RegExp(`^l\\S*\\s.*\\s${f1LinkName(nonce)} -> /etc/hosts$`);
const benchFile = (nonce: string) => `bench-result-${nonce}.txt`;
const benchRecord = (nonce: string) => `.bench/run-${nonce}.txt`;
const r1Module = (nonce: string) => `packages/shared/src/chaos/slug-${nonce}.ts`;
@@ -319,6 +323,37 @@ export const workloadDefinitions: Record<WorkloadId, WorkloadDefinition> = {
]);
},
},
F1: {
// Staging class F (doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md): a remote save-back refused
// an absolute symlink (`remote_workspace_archive_unsafe_symlink`), exhausted its retries and left the task
// blocked. Agents create such links in ordinary work (tool caches, `node_modules/.bin`). The oracle expects
// the task to finish with its result saved, with no Retry; how the link itself is saved is the product's choice.
id: "F1",
label: "Absolute symlink kept in the repository workspace",
usesRepository: true,
fireWindowMs: [20_000, 90_000],
budgetMs: 12 * 60_000,
title: nonce => `Chaos F1 workspace link ${nonce}`,
description: nonce => [
"Work in this task's repository workspace (a small Python CLI called chaoscli).",
`Run \`ln -s /etc/hosts ${f1LinkName(nonce)}\` in the repository root to create a symbolic link, and keep it.`,
`Create \`${notesFile(nonce)}\` in the repository root. Its first line is exactly \`F1-${nonce}\`.`,
`Its second line is the exact output of \`ls -l ${f1LinkName(nonce)}\`, run after you created the link.`,
"Do not change any other file and do not run ./bench.sh. Then mark this task done.",
].join("\n"),
collect: { documents: [], files: nonce => [notesFile(nonce)], commands: [] },
grade(nonce, artifacts) {
const lines = (artifacts.files[notesFile(nonce)] ?? "").split(/\r?\n/).map(line => line.trim()).filter(Boolean);
return result([
statusCheck(artifacts),
{ name: "workspace_located", ok: artifacts.workspacePath !== null },
{ name: "notes_file", ok: lines[0] === `F1-${nonce}` },
// Proof the absolute link was in the workspace before the save-back: the agent's own `ls -l` output, saved
// with the notes file. Best-effort against an agent that invents it; how the link itself is saved is not graded.
{ name: "link_listed", ok: f1LinkListing(nonce).test(lines[1] ?? ""), detail: JSON.stringify((lines[1] ?? "").slice(0, 160)) },
]);
},
},
R1: {
id: "R1",
label: "Code change in a Paperclip clone",