mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
test(restart-chaos): K15 redeploy, F1 workspace symlink, and the staging failure census (#20679)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The restart-chaos harness (`tests/restart-chaos/`) restarts the
control plane while real agent tasks run. It grades whether every task
finishes without a human Retry.
> - The hosted staging stack deploys about every 40 minutes. In 48
hours, 276 runs failed and 63 tasks stayed blocked. About a quarter of
the failures came within 10 minutes of a deploy. The harness passed the
same areas.
> - A hosted deploy replaces the server container. The new server runs
in another process space and has a new deployment id. The PID-based "the
earlier owner exited" proofs never apply across that boundary. A harness
restart on one machine keeps the process space, so every PID proof
succeeds and the hosted recovery path is never tested.
> - This pull request adds K15, a redeploy fault. It also adds the
staging census that shows the gap.
> - The benefit: chaos waves can now exercise the recovery path that
staging runs after every deploy, on local and Daytona.
## Linked Issues or Issue Description
**What happened?**
Many staging tasks that a deploy crossed ended blocked. Examples are
`native_remote_recovery_lease_mismatch`,
`runner_remote_process_identity_unavailable`,
`prior_owner_terminal_state_indeterminate` and `PRP provider resume
state is unavailable`. The restart-chaos Daytona waves passed for the
same adapters.
**What did you expect?**
The chaos harness should reproduce the deploy conditions that staging
has, so that these failure classes show up in a wave first.
**Steps to reproduce**
1. Compare the staging run records' `controllerProcess.hostHash` and
`deployment.deploymentId` across a deploy. Both change on every deploy.
2. Compare the same fields across a harness K2 restart. Both stay the
same.
3. Read `controllerProcessExited` and `sameProcessSpace` in
`server/src/services/controller-process-identity.ts`. A different
`hostHash` means that no PID proof applies.
**Additional context**
The full census, the failure classes and the per-class harness gaps are
in `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`. The
task identifiers in it are neutral labels.
## What Changed
- New fault `K15` (redeploy), in `tests/restart-chaos/redeploy.ts` and
`redeploy-preload.mjs`. A K15 wave gives every server an emulated host
name through a `--import` preload, which changes the process-space hash.
It also gives Railway-shaped deployment variables with the instance home
as the exclusive volume. A K15 stops the server with SIGTERM and starts
the next generation, with a new host name and deployment id.
- `supervisor.ts`, `main.ts`, `config.ts` and `types.ts`: select K15
with `--faults K15`. It works on `--env local` and `--env daytona`. The
receipt records `redeploy` and the timeline records `redeploy_switched`.
Config refuses K15 together with MS1–MS3.
- New opt-in workload `F1` (`workloads.ts`): the agent keeps an absolute
symlink in its repository workspace. A staging save-back refused such a
link and blocked the task. The oracle needs `done` and the saved notes
file, with no Retry. `f1-symlink.test.ts` covers the prompt, the grading
and the option planning.
- `redeploy.test.ts`: identity derivation, the environment, the argument
order, a real child process that proves the preload changes both CJS and
ESM `hostname()`, option parsing and plan detection.
- `tests/restart-chaos/README.md`: a K15 section and a row in the
"Pieces" table.
- `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`: the
48-hour staging census, 12 failure classes, the deploy timeline, the
reason the harness missed each class, and proposed scenarios.
## Verification
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/redeploy.test.ts
tests/restart-chaos/scheduler.test.ts
tests/restart-chaos/repeated-restarts.test.ts`: 68 tests pass. The full
restart-chaos unit suite passes: 43 files, 516 tests.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`:
passes.
- Live Daytona waves on 8f6ca25976, image
`paperclip-daytona-runner@sha256:bc7ca92d…`:
| Wave | Faults and cells | Result |
| --- | --- | --- |
| `w-20261111T002313-s9301` | K15, opencode and claude, A1 and C1, reuse
lease, 6 tasks | 5 of 6 pass. t05 (OpenCode, C1) failed R5
`silent_dead_run`: 183 s without evidence after the redeploy, then
succeeded. |
| `w-20261111T004241-s9301` | K2 control, same seed | 6 of 6 pass |
| `w-20261111T005008-s9302` | K15, warm sandbox, comment trigger, 4
tasks | 4 of 4 pass |
| `w-20261111T010000-s9303` | K15, opencode only, 8 tasks | 8 of 8 pass
|
| `w-20261111T011044-s9303` | K2 control, same seed | 8 of 8 pass |
| `w-20261111T012523-s9304` | F1 with K2, codex and opencode, `--storage
s3` | 2 of 2 pass |
- Every K15 fault switched the generation, with a new host name and
deployment id on the same volume. The server was ready again in 14–24 s.
- K15 does not yet reproduce the staging refusal codes on current
master. The census document explains the open conditions, such as a code
change at the deploy.
## Risks
- Low risk. The change is in the test harness and docs only. No product
code changes.
- K15 is an emulation. It changes the host name only in the server
process (and forked Node children). It does not emulate a new kernel
boot or PID namespace. The cloud stop-proof service is not emulated, so
stop proofs that need it stay unsupported, as on a self-hosted Railway
deploy. The README lists these limits.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edit, GitHub CLI).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
1 parent
7435abcc63
commit
52160eba35
12 files changed
+723
-13
No files matched your search
@@ -292,6 +292,76 @@ receipt names both commits (`upgrade` in `faults.json`, `upgrade_switched` in th
|
||||
timeline). K14 is local-only, because a Daytona runner must match its image, and
|
||||
it is never part of a storm's default faults.
|
||||
|
||||
## Redeploy (K15, `redeploy.ts`)
|
||||
|
||||
```sh
|
||||
node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env daytona --daytona-image $IMG --daytona-cpu 3 \
|
||||
--adapters opencode,claude --workloads A1,C1 --faults K15 --daytona-reuse-lease --concurrency 3 --tasks 6 --seed 9301
|
||||
```
|
||||
|
||||
The hosted staging stack deploys about every 40 minutes. Each deploy replaces
|
||||
the server's container. The replacement has the same database and volume, but it
|
||||
runs in another process space (host name, kernel boot and PID namespace) and
|
||||
has a new Railway deployment id. The server's PID-based proof that an earlier
|
||||
owner exited (`controller-process-identity.ts`, `owner-proof.ts`) never applies
|
||||
across a deploy. Recovery must use the deployment proof or the lease deadlines
|
||||
instead. K1 and K2 restart on the same machine, so every PID proof succeeds and
|
||||
that path stays untested. See `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`.
|
||||
|
||||
K15 emulates the container replacement without a container:
|
||||
|
||||
- Every server in a wave that has a K15 is launched with a `--import` preload
|
||||
(`redeploy-preload.mjs`). The preload makes `os.hostname()` return a
|
||||
per-generation name, which changes the process-space hash.
|
||||
- Every server gets Railway-shaped variables: `RAILWAY_PROJECT_ID`,
|
||||
`RAILWAY_ENVIRONMENT_ID`, `RAILWAY_SERVICE_ID` and `RAILWAY_DEPLOYMENT_ID`,
|
||||
plus `RAILWAY_VOLUME_ID` and `RAILWAY_VOLUME_MOUNT_PATH`, which is the
|
||||
instance home.
|
||||
- A K15 stops the server with SIGTERM, like K2. The replacement starts as the
|
||||
next generation, with a new host name and deployment id, on the same project,
|
||||
service and volume.
|
||||
- Other faults in the same wave restart within the current generation, like a
|
||||
platform restart of the same deployment.
|
||||
- Identities derive from the wave id; a replay uses the source wave id, so it sees the same ones.
|
||||
- Each K15 receipt records `redeploy` (both generations) in `faults.json`, and
|
||||
`redeploy_switched` in the timeline.
|
||||
|
||||
Limits:
|
||||
|
||||
- The Daytona sandboxes and remote runners survive, as they do on staging.
|
||||
- Local runners are stopped with the server's process tree.
|
||||
- The cloud stop-proof service (`PAPERCLIP_CLOUD_*`) is not emulated, so stop
|
||||
proofs that need it stay unsupported, as on any self-hosted Railway deploy.
|
||||
- K15 does not combine with MS1–MS3, because a second server would run in the
|
||||
real host's process space.
|
||||
- Combine K15 with `--adapters opencode` and `--daytona-reuse-lease` or
|
||||
`--daytona-warm` to match staging's workload mix.
|
||||
|
||||
## Absolute symlink in the workspace (F1)
|
||||
|
||||
```sh
|
||||
node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env daytona --daytona-image $IMG --daytona-cpu 3 \
|
||||
--adapters codex,opencode --workloads F1 --faults K2 --storage s3 --concurrency 2 --tasks 2 --seed 9304
|
||||
```
|
||||
|
||||
F1 is opt-in. The agent creates `hosts-link-<nonce>`, which is an absolute
|
||||
symlink to `/etc/hosts`, in its repository workspace and keeps it. It also
|
||||
writes a notes file, and then marks the task done.
|
||||
|
||||
On staging, a remote save-back refused such a link
|
||||
(`remote_workspace_archive_unsafe_symlink`). The save-back then used up its
|
||||
retries and left the task blocked. Agents create absolute links in ordinary
|
||||
work, for example in tool caches or `node_modules/.bin`.
|
||||
|
||||
The oracle passes when:
|
||||
|
||||
- the task is `done`;
|
||||
- the notes file is saved, without a Retry.
|
||||
|
||||
How the product saves the link itself (skip it, keep it, or record it) is not
|
||||
graded. F1 is meaningful with a remote workspace (`--env daytona`, with
|
||||
`--storage s3` as on staging). Locally it is a plain repository task.
|
||||
|
||||
## Config faults (CF1–CF3, `config-wave.ts`)
|
||||
|
||||
```sh
|
||||
@@ -991,7 +1061,7 @@ re-arms the task's own triggered faults instead of using their offsets.
|
||||
| `grader.ts` | Pure R1–R6 grading over the samples, plus the universal checks. |
|
||||
| `universal.ts`, `universal-audit.ts` | U1–U4 on every task, and their audit over saved waves. |
|
||||
| `contention.ts` | L1 workload, its accounting oracle and the `--db-pressure` writers. |
|
||||
| `workloads.ts` | A1 brief document, B1 python CLI with tests, B5 long bench, C1 comment while running, M1 delegation, G1 approval across a restart, H1 budget hard stop across a restart, with their T-oracles. |
|
||||
| `workloads.ts` | A1 brief document, B1 python CLI with tests, B5 long bench, C1 comment while running, M1 delegation, G1 approval across a restart, H1 budget hard stop across a restart, F1 absolute symlink kept in the workspace, with their T-oracles. |
|
||||
| `journeys.ts`, `journey-driver.ts`, `journey-oauth-app.ts`, `journey-matrix.ts` | Q2 and Q3 definitions and oracles, the board's side of each, Q3's loopback OAuth/MCP app, and the `--matrix` planner. |
|
||||
| `evidence.ts`, `ledger.ts` | Evidence bundle for each failure and the JSONL ledger plus `SUMMARY.md`. |
|
||||
| `daytona-image.ts` | Docker-free extraction of the image's runnerd and provider pack, cached per digest, and the controller/image content-ID check. |
|
||||
@@ -1001,6 +1071,7 @@ re-arms the task's own triggered faults instead of using their offsets.
|
||||
| `time-faults.ts`, `time-faults-config.ts` | TM1–TM3: process freezing, database aging with recorded rows, and the time-corrected API the grader reads through. |
|
||||
| `multi-server.ts` | MS1–MS3: the second server's steps, the seeded holds, and the cross-server invariant over the wave database. |
|
||||
| `repeated-restarts.ts` | RR1–RR3: seeded restart steps, `--rr-gap-seconds`, and the stop-and-replace loop the supervisor runs. |
|
||||
| `redeploy.ts`, `redeploy-preload.mjs` | K15: per-generation emulated host name and Railway deployment identity, and the preload that applies the host name. |
|
||||
| `daytona-sweep.ts` | Daytona R6: read-only reconciliation of this wave's labelled sandboxes with leases; cleanup only through Paperclip. |
|
||||
|
||||
## Output (gitignored)
|
||||
|
||||
@@ -4,7 +4,7 @@ import { parseArgs } from "node:util";
|
||||
import { applyJourneyOptions } from "./journey-matrix.js";
|
||||
import { takesJourneyTriggers } from "./journeys.js";
|
||||
import { STORAGE_FAULT_KINDS, type StorageFaultKind } from "./storage-chaos.js";
|
||||
import { validateMultiServerOptions } from "./multi-server.js";
|
||||
import { isMultiServerFault, validateMultiServerOptions } from "./multi-server.js";
|
||||
import { parseTimeFaultOptions } from "./time-faults-config.js";
|
||||
import { parseStorageTrigger } from "./storage-trigger.js";
|
||||
import { parseNativeTrigger, parseNativeTriggerOutage } from "./native-trigger.js";
|
||||
@@ -17,7 +17,10 @@ import {
|
||||
type AdapterId, type EnvironmentKind, type FaultKind, type TriggerId, type WaveOptions, type WorkloadId,
|
||||
} from "./types.js";
|
||||
|
||||
/** A storm's default faults: every restart kind except K14, which needs an older build (`--upgrade-from`). */
|
||||
/**
|
||||
* A storm's default faults: every restart kind except K14, which needs an older build (`--upgrade-from`),
|
||||
* and K15, which changes the deploy identity of every later server (pass `--faults` to include it).
|
||||
*/
|
||||
export const STORM_DEFAULT_FAULTS: FaultKind[] = ["K1", "K2", "K4", "K6"];
|
||||
|
||||
export const CREDENTIAL_FOR_ADAPTER: Record<AdapterId, "OPENAI_API_KEY" | "ANTHROPIC_API_KEY" | "OPENROUTER_API_KEY"> = {
|
||||
@@ -176,7 +179,7 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
|
||||
env,
|
||||
adapters,
|
||||
// The governance scenarios (G1, H1), the journeys (Q1–Q3) and the contention run (L1) are opt-in: H1 needs a wave of its own.
|
||||
workloads: list<WorkloadId>(values.workloads, workloadChoices, "workloads", workloadChoices.filter(id => id !== "G1" && id !== "H1" && id !== "R1" && id !== "S1" && id !== "L1"
|
||||
workloads: list<WorkloadId>(values.workloads, workloadChoices, "workloads", workloadChoices.filter(id => id !== "G1" && id !== "H1" && id !== "R1" && id !== "S1" && id !== "L1" && id !== "F1"
|
||||
&& !takesJourneyTriggers(id) && (id !== "M1" || adapters.length >= 2))),
|
||||
// A crash-point wave schedules no fault; K1 is the restart its self-kill gets.
|
||||
faults: allFaults.filter((fault): fault is FaultKind => !isProviderFault(fault)),
|
||||
@@ -233,6 +236,10 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
|
||||
throw new Error("--provider-fault-message, --provider-fault-delays and --provider-fault-restart need a provider fault (PV1, PV2 or PV3) in --faults");
|
||||
}
|
||||
validateMultiServerOptions(options);
|
||||
// K15 emulates one deploy identity per generation; a second server beside it would run in the real host's process space.
|
||||
if (options.faults.includes("K15") && options.faults.some(isMultiServerFault)) {
|
||||
throw new Error("K15 (redeploy) does not combine with MS1-MS3: the second server would not share the emulated deploy identity");
|
||||
}
|
||||
Object.assign(options, parseTimeFaultOptions(options, values["time-age-hours"]));
|
||||
Object.assign(options, parseRepeatedRestartOptions(options, values["rr-gap-seconds"], values["rr-stop-on"]));
|
||||
const upgradeFrom = values["upgrade-from"];
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { parseCommand } from "./config.js";
|
||||
import { WORKLOAD_TRIGGERS } from "./journeys.js";
|
||||
import { planTasks } from "./scheduler.js";
|
||||
import type { OracleArtifacts } from "./types.js";
|
||||
import { f1LinkName, workloadDefinitions } from "./workloads.js";
|
||||
|
||||
const NONCE = "abc123de";
|
||||
const LISTING = `lrwxrwxrwx 1 daytona daytona 10 Nov 11 01:30 hosts-link-${NONCE} -> /etc/hosts`;
|
||||
const notes = (text: string) => ({ [`NOTES-${NONCE}.md`]: text });
|
||||
const artifacts = (overrides: Partial<OracleArtifacts> = {}): OracleArtifacts => ({
|
||||
issueStatus: "done", documents: {}, agentComments: [], files: notes(`F1-${NONCE}\n${LISTING}\n`), commands: {},
|
||||
workspacePath: "/workspace/chaos", ...overrides,
|
||||
}) as OracleArtifacts;
|
||||
|
||||
describe("F1: absolute symlink kept in the repository workspace", () => {
|
||||
const f1 = workloadDefinitions.F1;
|
||||
|
||||
it("asks for the absolute link and the notes file in a repository workspace", () => {
|
||||
expect(f1.usesRepository).toBe(true);
|
||||
const text = f1.description(NONCE);
|
||||
expect(text).toContain(`ln -s /etc/hosts ${f1LinkName(NONCE)}`);
|
||||
expect(text).toContain(`NOTES-${NONCE}.md`);
|
||||
expect(f1.collect.files(NONCE)).toEqual([`NOTES-${NONCE}.md`]);
|
||||
});
|
||||
|
||||
it("passes only when the task is done and its saved notes file is in the workspace", () => {
|
||||
expect(f1.grade(NONCE, artifacts()).verdict).toBe("pass");
|
||||
expect(f1.grade(NONCE, artifacts({ issueStatus: "blocked" })).verdict).toBe("fail");
|
||||
expect(f1.grade(NONCE, artifacts({ files: {} })).verdict).toBe("fail");
|
||||
expect(f1.grade(NONCE, artifacts({ workspacePath: null })).verdict).toBe("fail");
|
||||
});
|
||||
|
||||
it("needs the agent's ls -l line proving the absolute link existed before the save-back", () => {
|
||||
expect(f1.description(NONCE)).toContain(`ls -l ${f1LinkName(NONCE)}`);
|
||||
// Skipped link: notes only.
|
||||
expect(f1.grade(NONCE, artifacts({ files: notes(`F1-${NONCE}\n`) })).verdict).toBe("fail");
|
||||
// A regular file of the same name, or a link elsewhere, is not the link under test.
|
||||
expect(f1.grade(NONCE, artifacts({ files: notes(`F1-${NONCE}\n-rw-r--r-- 1 d d 3 Nov 11 01:30 hosts-link-${NONCE}\n`) })).verdict).toBe("fail");
|
||||
expect(f1.grade(NONCE, artifacts({ files: notes(`F1-${NONCE}\n${LISTING.replace("/etc/hosts", "etc/hosts")}\n`) })).verdict).toBe("fail");
|
||||
expect(f1.grade(NONCE, artifacts({ files: notes(`F1-${NONCE}\n${LISTING.replace(NONCE, "ffffffff")}\n`) })).verdict).toBe("fail");
|
||||
});
|
||||
|
||||
it("is opt-in, time-triggered, and plans with restart faults on Daytona", () => {
|
||||
expect(WORKLOAD_TRIGGERS.F1).toEqual(["time"]);
|
||||
const defaults = parseCommand(["--seed", "1"]);
|
||||
expect(defaults.mode === "wave" && defaults.options.workloads).not.toContain("F1");
|
||||
const parsed = parseCommand(["--env", "daytona", "--daytona-image", `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:${"b".repeat(64)}`,
|
||||
"--workloads", "F1", "--faults", "K2,K15", "--adapters", "codex", "--tasks", "2", "--seed", "3"]);
|
||||
if (parsed.mode !== "wave") throw new Error("expected a wave");
|
||||
expect(planTasks(parsed.options).map(task => task.workload)).toEqual(["F1", "F1"]);
|
||||
});
|
||||
});
|
||||
@@ -35,7 +35,7 @@ export type JourneyTrigger = (typeof JOURNEY_TRIGGERS)[number];
|
||||
/** Each workload's own fault boundaries, for the journey × fault matrix. */
|
||||
export const WORKLOAD_TRIGGERS: Record<WorkloadId, readonly TriggerId[]> = {
|
||||
A1: ["time"], B1: ["time"], B5: ["tool"], C1: ["comment", "result"], M1: ["time"], G1: ["approval"],
|
||||
H1: ["paused"], R1: ["time"], S1: ["stop"], L1: ["time"], Q1: JOURNEY_TRIGGERS, Q2: JOURNEY_TRIGGERS, Q3: JOURNEY_TRIGGERS,
|
||||
H1: ["paused"], R1: ["time"], S1: ["stop"], L1: ["time"], F1: ["time"], Q1: JOURNEY_TRIGGERS, Q2: JOURNEY_TRIGGERS, Q3: JOURNEY_TRIGGERS,
|
||||
};
|
||||
|
||||
/** Q3 reviews must survive a delay longer than the implicit deadline #20340 removed. */
|
||||
|
||||
@@ -38,6 +38,7 @@ import { collectWaveSpeed, speedMarkdown } from "./wave-speed.js";
|
||||
import { TaskObserver, childSpeedTasksInTurn } from "./observer.js";
|
||||
import { closeJourneyApps } from "./journey-driver.js";
|
||||
import { matrixMayLaunch, planMatrixWave } from "./journey-matrix.js";
|
||||
import { redeployWave } from "./redeploy.js";
|
||||
import { StormRun, TASK_FAULT_LOG, observedFaultSchedule, planWave, replayOnCurrentBuild, replayRecordFor, replayWave, type TaskFaultLog } from "./scheduler.js";
|
||||
import { chaosLifecycleOverrides, seedChaosLab } from "./seed.js";
|
||||
import { ChaosSupervisor } from "./supervisor.js";
|
||||
@@ -200,7 +201,9 @@ async function main(): Promise<number> {
|
||||
const batchSize = crashPoints.length ? Math.max(1, Math.floor(plan.tasks.length / crashPoints.length)) : 0;
|
||||
if (crashPoint && plan.tasks.length !== batchSize * crashPoints.length) throw new Error(`--tasks must be a multiple of the ${crashPoints.length} crash point(s)`);
|
||||
const supervisor = await ChaosSupervisor.create({ repositoryRoot, waveDir, timeline,
|
||||
...(plan.options.upgradeFrom ?? plan.options.serverFrom ? { upgradeFrom: plan.options.upgradeFrom ?? plan.options.serverFrom } : {}) });
|
||||
...(plan.options.upgradeFrom ?? plan.options.serverFrom ? { upgradeFrom: plan.options.upgradeFrom ?? plan.options.serverFrom } : {}),
|
||||
// K15: every server of the wave runs under an emulated deploy identity; a replay derives the source wave's ones.
|
||||
...(redeployWave(plan) ? { redeployWaveId: plan.replayOf?.waveId ?? plan.waveId } : {}) });
|
||||
// Crash-point suite: one booted server, one point at a time. Each point gets
|
||||
// its own batch of tasks; the point is armed once the batch launched and the
|
||||
// server kills itself at its nth hit (restarted like K1). Invariants are
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
// K15 (redeploy.ts): give the server process an emulated host name, so its process-space hash
|
||||
// differs from the previous deploy's, as a replaced container's does. Loaded with `--import`
|
||||
// before the server. Node children forked with the server's execArgv inherit it, as they share its container.
|
||||
import os from "node:os";
|
||||
import { syncBuiltinESMExports } from "node:module";
|
||||
|
||||
const hostname = process.env.RESTART_CHAOS_REDEPLOY_HOSTNAME?.trim();
|
||||
if (hostname) {
|
||||
os.hostname = () => hostname;
|
||||
// `import { hostname } from "node:os"` reads the ESM export, which only follows the CJS object after a sync.
|
||||
syncBuiltinESMExports();
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { parseCommand } from "./config.js";
|
||||
import {
|
||||
REDEPLOY_HOSTNAME_ENV, REDEPLOY_PRELOAD_PATH, nextRedeployIdentity, redeployEnvironment, redeployIdentity, redeployNodeArgs,
|
||||
redeployReceipt, redeployWave,
|
||||
} from "./redeploy.js";
|
||||
import { planTasks } from "./scheduler.js";
|
||||
import { FAULT_SIGNAL } from "./supervisor.js";
|
||||
import type { WavePlan } from "./types.js";
|
||||
|
||||
const UUID = /^[a-f0-9]{8}-[a-f0-9]{4}-5[a-f0-9]{3}-[89ab][a-f0-9]{3}-[a-f0-9]{12}$/;
|
||||
const IMAGE = `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:${"a".repeat(64)}`;
|
||||
|
||||
describe("K15 redeploy identity", () => {
|
||||
it("keeps the service and volume and changes the host and deployment on every generation", () => {
|
||||
const first = redeployIdentity("w-1", 0);
|
||||
const second = nextRedeployIdentity("w-1", first);
|
||||
expect(second.generation).toBe(1);
|
||||
expect(second.hostname).not.toBe(first.hostname);
|
||||
expect(second.deploymentId).not.toBe(first.deploymentId);
|
||||
for (const key of ["projectId", "environmentId", "serviceId", "volumeId"] as const) expect(second[key]).toBe(first[key]);
|
||||
for (const id of [first.projectId, first.environmentId, first.serviceId, first.deploymentId, first.volumeId]) expect(id).toMatch(UUID);
|
||||
});
|
||||
|
||||
it("is deterministic per wave, so a replay derives the same identities", () => {
|
||||
expect(redeployIdentity("w-1", 3)).toEqual(redeployIdentity("w-1", 3));
|
||||
expect(redeployIdentity("w-2", 0).deploymentId).not.toBe(redeployIdentity("w-1", 0).deploymentId);
|
||||
expect(() => redeployIdentity("w-1", -1)).toThrow(/non-negative/);
|
||||
});
|
||||
|
||||
it("names the Railway variables the server reads as its deployment, with the instance home as the exclusive volume", () => {
|
||||
const identity = redeployIdentity("w-1", 2);
|
||||
expect(redeployEnvironment(identity, "/tmp/chaos/home")).toEqual({
|
||||
[REDEPLOY_HOSTNAME_ENV]: identity.hostname,
|
||||
RAILWAY_PROJECT_ID: identity.projectId, RAILWAY_ENVIRONMENT_ID: identity.environmentId, RAILWAY_SERVICE_ID: identity.serviceId,
|
||||
RAILWAY_DEPLOYMENT_ID: identity.deploymentId, RAILWAY_VOLUME_ID: identity.volumeId, RAILWAY_VOLUME_MOUNT_PATH: "/tmp/chaos/home",
|
||||
});
|
||||
expect(redeployReceipt(redeployIdentity("w-1", 0), identity)).toMatchObject({ fromGeneration: 0, toGeneration: 2,
|
||||
toDeploymentId: identity.deploymentId, volumeId: identity.volumeId });
|
||||
});
|
||||
|
||||
it("loads the preload ahead of the server's own --import loader", () => {
|
||||
expect(redeployNodeArgs(["--import", "/x/tsx/loader.mjs", "entry.ts"])).toEqual(["--import", REDEPLOY_PRELOAD_PATH, "--import", "/x/tsx/loader.mjs", "entry.ts"]);
|
||||
});
|
||||
|
||||
it("changes os.hostname for both the CommonJS object and ESM named imports, as processSpaceHash reads it", () => {
|
||||
const script = "import os from 'node:os'; import { hostname } from 'node:os'; console.log(JSON.stringify([os.hostname(), hostname()]));";
|
||||
const run = (env: NodeJS.ProcessEnv) => spawnSync(process.execPath, ["--import", REDEPLOY_PRELOAD_PATH, "--input-type=module", "-e", script],
|
||||
{ env: { PATH: process.env.PATH, ...env }, encoding: "utf8" });
|
||||
const emulated = run({ [REDEPLOY_HOSTNAME_ENV]: "chaos-deploy-test" });
|
||||
expect(emulated.status).toBe(0);
|
||||
expect(JSON.parse(emulated.stdout)).toEqual(["chaos-deploy-test", "chaos-deploy-test"]);
|
||||
// Without the variable the preload changes nothing.
|
||||
const real = run({});
|
||||
expect(real.status).toBe(0);
|
||||
const [cjs, esm] = JSON.parse(real.stdout) as string[];
|
||||
expect(cjs).toBe(esm);
|
||||
expect(cjs).not.toBe("chaos-deploy-test");
|
||||
});
|
||||
});
|
||||
|
||||
describe("K15 wave options", () => {
|
||||
it("is a graceful stop, selectable on local and Daytona like K2", () => {
|
||||
expect(FAULT_SIGNAL.K15).toBe("SIGTERM");
|
||||
const local = parseCommand(["--faults", "K15", "--adapters", "opencode", "--workloads", "A1,C1", "--seed", "1"]);
|
||||
expect(local.mode === "wave" && local.options.faults).toEqual(["K15"]);
|
||||
const daytona = parseCommand(["--env", "daytona", "--daytona-image", IMAGE, "--faults", "K2,K15", "--adapters", "opencode",
|
||||
"--workloads", "C1", "--daytona-reuse-lease", "--seed", "1"]);
|
||||
if (daytona.mode !== "wave") throw new Error("expected a wave");
|
||||
expect(daytona.options.faults).toEqual(["K2", "K15"]);
|
||||
expect(daytona.options.daytonaReuseLease).toBe(true);
|
||||
});
|
||||
|
||||
it("refuses a second server beside an emulated deploy", () => {
|
||||
expect(() => parseCommand(["--faults", "K15,MS1", "--workloads", "B5,Q1", "--concurrency", "1", "--seed", "1"])).toThrow(/K15 \(redeploy\) does not combine with MS1-MS3/);
|
||||
});
|
||||
|
||||
it("plans K15 cells with no hold and marks the wave as a redeploy wave", () => {
|
||||
const parsed = parseCommand(["--faults", "K15", "--adapters", "codex,opencode", "--workloads", "A1,C1", "--tasks", "4", "--seed", "7"]);
|
||||
if (parsed.mode !== "wave") throw new Error("expected a wave");
|
||||
const tasks = planTasks(parsed.options);
|
||||
expect(tasks.every(task => task.fault === "K15" && task.holdMs === 0)).toBe(true);
|
||||
const plan = { options: parsed.options, tasks } as Pick<WavePlan, "options" | "tasks" | "storm">;
|
||||
expect(redeployWave(plan)).toBe(true);
|
||||
const k2 = parseCommand(["--faults", "K2", "--workloads", "A1", "--tasks", "2", "--seed", "7"]);
|
||||
if (k2.mode !== "wave") throw new Error("expected a wave");
|
||||
expect(redeployWave({ options: k2.options, tasks: planTasks(k2.options) })).toBe(false);
|
||||
// A replay of a K2 task that a K15 crossed still needs the emulated identity.
|
||||
expect(redeployWave({ options: k2.options, tasks: [{ ...planTasks(k2.options)[0]!,
|
||||
replayFaults: [{ kind: "K15", holdMs: 0, atMs: 1_000 }] }] })).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,96 @@
|
||||
/**
|
||||
* K15 (redeploy): what a hosted deploy does to the control plane, beyond K2's graceful stop.
|
||||
*
|
||||
* On the hosted staging stack every deploy replaces the server's container. The replacement runs
|
||||
* in another process space (host name, kernel boot and PID namespace), so the server's PID-based
|
||||
* "the previous owner has exited" proofs (`server/src/services/controller-process-identity.ts`,
|
||||
* `owner-proof.ts`) never apply. It also carries a new Railway deployment id on the same exclusive
|
||||
* volume, which is the proof hosted recovery uses instead. A K1/K2 restart on one machine keeps the
|
||||
* process space, so every PID proof succeeds there and this path is never exercised.
|
||||
*
|
||||
* K15 emulates both changes without a container: each server of a K15 wave sees an emulated host
|
||||
* name (a `--import` preload replaces `os.hostname`, which changes the process-space hash) and
|
||||
* Railway-shaped deployment variables naming one stable project, environment, service and volume
|
||||
* and a deployment id per generation. A K15 stops the server with SIGTERM like K2 and starts the
|
||||
* replacement as the next generation. Other faults in the same wave restart within a generation,
|
||||
* like a platform restart of the same deployment.
|
||||
*/
|
||||
import { createHash } from "node:crypto";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import type { WavePlan } from "./types.js";
|
||||
|
||||
/** Whether any server of this wave can be redeployed: a planned, storm or replayed K15. */
|
||||
export function redeployWave(plan: Pick<WavePlan, "options" | "tasks" | "storm">): boolean {
|
||||
return plan.options.faults.includes("K15") || (plan.storm?.faults.some(fault => fault.kind === "K15") ?? false)
|
||||
|| plan.tasks.some(task => task.fault === "K15" || (task.replayFaults?.some(fault => fault.kind === "K15") ?? false));
|
||||
}
|
||||
|
||||
/** The preload that applies the emulated host name before the server loads. */
|
||||
export const REDEPLOY_PRELOAD_PATH = fileURLToPath(new URL("./redeploy-preload.mjs", import.meta.url));
|
||||
/** Read by the preload: the host name `os.hostname()` returns in the server process. */
|
||||
export const REDEPLOY_HOSTNAME_ENV = "RESTART_CHAOS_REDEPLOY_HOSTNAME";
|
||||
|
||||
export interface RedeployIdentity {
|
||||
/** 0 for the wave's first server; each K15 adds one. */
|
||||
generation: number;
|
||||
hostname: string;
|
||||
projectId: string;
|
||||
environmentId: string;
|
||||
serviceId: string;
|
||||
deploymentId: string;
|
||||
volumeId: string;
|
||||
}
|
||||
|
||||
/** A deterministic UUID-shaped id for one wave, role and generation (no randomness: replays match). */
|
||||
function uuidFor(waveId: string, role: string, generation = 0): string {
|
||||
const hex = createHash("sha256").update(`restart-chaos-redeploy:${waveId}:${role}:${generation}`).digest("hex");
|
||||
// Version 5-style layout so validators that require a UUID shape accept it.
|
||||
const variant = ((parseInt(hex[16]!, 16) & 0x3) | 0x8).toString(16);
|
||||
return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-5${hex.slice(13, 16)}-${variant}${hex.slice(17, 20)}-${hex.slice(20, 32)}`;
|
||||
}
|
||||
|
||||
/** The identity of the server that serves generation `generation` of this wave. */
|
||||
export function redeployIdentity(waveId: string, generation: number): RedeployIdentity {
|
||||
if (!Number.isInteger(generation) || generation < 0) throw new Error(`Redeploy generation must be a non-negative integer, got ${generation}`);
|
||||
return {
|
||||
generation,
|
||||
hostname: `chaos-deploy-${createHash("sha256").update(`${waveId}:${generation}`).digest("hex").slice(0, 10)}`,
|
||||
projectId: uuidFor(waveId, "project"),
|
||||
environmentId: uuidFor(waveId, "environment"),
|
||||
serviceId: uuidFor(waveId, "service"),
|
||||
deploymentId: uuidFor(waveId, "deployment", generation),
|
||||
volumeId: uuidFor(waveId, "volume"),
|
||||
};
|
||||
}
|
||||
|
||||
/** The next deploy of the same service: same project, environment, service and volume. */
|
||||
export function nextRedeployIdentity(waveId: string, current: RedeployIdentity): RedeployIdentity {
|
||||
return redeployIdentity(waveId, current.generation + 1);
|
||||
}
|
||||
|
||||
/**
|
||||
* Server environment for one generation. The volume mount is the instance home: on staging the
|
||||
* instance directory lives on the exclusive volume that survives each deploy.
|
||||
*/
|
||||
export function redeployEnvironment(identity: RedeployIdentity, volumeMountPath: string): Record<string, string> {
|
||||
return {
|
||||
[REDEPLOY_HOSTNAME_ENV]: identity.hostname,
|
||||
RAILWAY_PROJECT_ID: identity.projectId,
|
||||
RAILWAY_ENVIRONMENT_ID: identity.environmentId,
|
||||
RAILWAY_SERVICE_ID: identity.serviceId,
|
||||
RAILWAY_DEPLOYMENT_ID: identity.deploymentId,
|
||||
RAILWAY_VOLUME_ID: identity.volumeId,
|
||||
RAILWAY_VOLUME_MOUNT_PATH: volumeMountPath,
|
||||
};
|
||||
}
|
||||
|
||||
/** Node arguments that load the preload ahead of the server entry's own `--import` loader. */
|
||||
export function redeployNodeArgs(args: readonly string[]): string[] {
|
||||
return ["--import", REDEPLOY_PRELOAD_PATH, ...args];
|
||||
}
|
||||
|
||||
/** What a K15 receipt records: both generations, so a replay and the evidence name them. */
|
||||
export function redeployReceipt(from: RedeployIdentity, to: RedeployIdentity) {
|
||||
return { fromGeneration: from.generation, toGeneration: to.generation, fromHostname: from.hostname, toHostname: to.hostname,
|
||||
fromDeploymentId: from.deploymentId, toDeploymentId: to.deploymentId, volumeId: to.volumeId };
|
||||
}
|
||||
@@ -39,6 +39,7 @@ import { isRepeatedRestartFault, newRepeatedRestartRecord, recordFirstRestart, r
|
||||
type RepeatedRestartRecord, type RestartStep } from "./repeated-restarts.js";
|
||||
import type { FaultKind, FaultReceipt, TriggerId } from "./types.js";
|
||||
import { isMultiServerFault, runMultiServerFault, type MultiServerKind, type MultiServerRecord, type ServerProcessWindow } from "./multi-server.js";
|
||||
import { nextRedeployIdentity, redeployEnvironment, redeployIdentity, redeployNodeArgs, redeployReceipt, type RedeployIdentity } from "./redeploy.js";
|
||||
|
||||
/** First boot applies every migration; a loaded host can need longer (RESTART_CHAOS_HEALTH_TIMEOUT_MS). */
|
||||
const HEALTH_TIMEOUT_MS = (() => {
|
||||
@@ -48,7 +49,7 @@ const HEALTH_TIMEOUT_MS = (() => {
|
||||
const READY_TIMEOUT_MS = 300_000;
|
||||
const STOP_TIMEOUT_MS = 30_000;
|
||||
|
||||
export const FAULT_SIGNAL: Record<FaultKind, FaultReceipt["signal"]> = { K1: "SIGKILL", K2: "SIGTERM", K4: "SIGKILL", K6: "SIGKILL", K14: "SIGTERM",
|
||||
export const FAULT_SIGNAL: Record<FaultKind, FaultReceipt["signal"]> = { K1: "SIGKILL", K2: "SIGTERM", K4: "SIGKILL", K6: "SIGKILL", K14: "SIGTERM", K15: "SIGTERM",
|
||||
TM1: "SIGSTOP", TM2: "SIGSTOP", TM3: "SIGKILL", RR1: "SIGKILL", RR2: "SIGKILL", RR3: "SIGKILL",
|
||||
// MS1 leaves the primary alone; MS2 freezes it; MS3 stops it gracefully (multi-server.ts).
|
||||
MS1: "none", MS2: "SIGSTOP", MS3: "SIGTERM" };
|
||||
@@ -154,13 +155,16 @@ export class ChaosSupervisor {
|
||||
private serverRoot: string;
|
||||
/** The environment overrides the wave passed to `start`, kept for every restart and for the K14 switch. */
|
||||
private extraEnvironment: Record<string, string> = {};
|
||||
/** K15 waves: the wave id the emulated deploy identities derive from, and the generation serving now. */
|
||||
private redeploy: { waveId: string; identity: RedeployIdentity } | null = null;
|
||||
/** Wave secrets, kept across a K14 switch as an in-place upgrade keeps its configuration. */
|
||||
private readonly secrets = { PAPERCLIP_AGENT_JWT_SECRET: secret(), PAPERCLIP_DECISION_SIGNING_SECRET: secret(),
|
||||
PAPERCLIP_TOOL_ACTION_SIGNING_SECRET: secret(), BETTER_AUTH_SECRET: secret() };
|
||||
|
||||
private constructor(readonly repositoryRoot: string, readonly port: number, readonly paths: SupervisorPaths,
|
||||
private readonly timeline: Timeline, upgradeFrom: string | null) {
|
||||
private readonly timeline: Timeline, upgradeFrom: string | null, redeployWaveId: string | null) {
|
||||
this.serverRoot = upgradeFrom ?? repositoryRoot;
|
||||
if (redeployWaveId) this.redeploy = { waveId: redeployWaveId, identity: redeployIdentity(redeployWaveId, 0) };
|
||||
this.baseUrl = `http://127.0.0.1:${port}`;
|
||||
this.log = createWriteStream(paths.serverLogPath, { flags: "a", mode: 0o600 });
|
||||
this.stopper = createRunnerE2EServerStopper({
|
||||
@@ -171,13 +175,16 @@ export class ChaosSupervisor {
|
||||
});
|
||||
}
|
||||
|
||||
static async create(input: { repositoryRoot: string; waveDir: string; timeline: Timeline; upgradeFrom?: string }): Promise<ChaosSupervisor> {
|
||||
/** `redeployWaveId`: a K15 wave; every server gets the emulated deploy identity of its generation (redeploy.ts). */
|
||||
static async create(input: { repositoryRoot: string; waveDir: string; timeline: Timeline; upgradeFrom?: string;
|
||||
redeployWaveId?: string }): Promise<ChaosSupervisor> {
|
||||
const temporaryRoot = await createRunnerE2ETemporaryRoot();
|
||||
const home = path.join(temporaryRoot, "home");
|
||||
const paths: SupervisorPaths = { temporaryRoot, home, configPath: path.join(home, "instances", "default", "config.json"),
|
||||
serverLogPath: path.join(input.waveDir, "server.log") };
|
||||
const port = await reserveRunnerE2EServerPort();
|
||||
const supervisor = new ChaosSupervisor(input.repositoryRoot, port, paths, input.timeline, input.upgradeFrom ?? null);
|
||||
const supervisor = new ChaosSupervisor(input.repositoryRoot, port, paths, input.timeline, input.upgradeFrom ?? null,
|
||||
input.redeployWaveId ?? null);
|
||||
input.timeline.add("supervisor_created", { temporaryRoot, port, ...(input.upgradeFrom
|
||||
? { upgradeFrom: input.upgradeFrom, upgradeFromSha: checkoutSha(input.upgradeFrom) } : {}) });
|
||||
return supervisor;
|
||||
@@ -241,6 +248,8 @@ export class ChaosSupervisor {
|
||||
PAPERCLIP_IN_WORKTREE: "false",
|
||||
...waveServerUrlEnvironment(this.port),
|
||||
...extraEnvironment,
|
||||
// K15: the generation's emulated host name and Railway deployment identity; the volume is the instance home.
|
||||
...(this.redeploy ? redeployEnvironment(this.redeploy.identity, home) : {}),
|
||||
};
|
||||
const environment = buildPaperclipServerEnvironment(source, overrides);
|
||||
const workspaceStorage = extraEnvironment.PAPERCLIP_STORAGE_PROVIDER === "s3"
|
||||
@@ -257,7 +266,9 @@ export class ChaosSupervisor {
|
||||
const control = this.crashPointControl;
|
||||
const env = control ? { ...this.environment, PAPERCLIP_CRASH_POINT_CONTROL: control.controlPath,
|
||||
PAPERCLIP_CRASH_POINT_RECEIPT: control.receiptPath } : this.environment;
|
||||
const child = spawn(process.execPath, runnerE2ETypeScriptProcessArgs(root, entry, ["onboard", "--yes", "--run"]),
|
||||
const args = runnerE2ETypeScriptProcessArgs(root, entry, ["onboard", "--yes", "--run"]);
|
||||
// K15 waves: the emulated host name of the serving deploy generation (redeploy.ts).
|
||||
const child = spawn(process.execPath, this.redeploy ? redeployNodeArgs(args) : args,
|
||||
{ cwd: root, env, stdio: ["ignore", "pipe", "pipe"], detached: runnerE2EServerDetached });
|
||||
this.child = child;
|
||||
const window = this.trackWindow(child, "primary");
|
||||
@@ -484,6 +495,8 @@ export class ChaosSupervisor {
|
||||
receipt.restartedAt = Date.now();
|
||||
// K14: the replacement is the current build, on the same database and instance directory.
|
||||
if (kind === "K14") receipt.upgrade = await this.switchToCurrentBuild(receipt.faultId);
|
||||
// K15: the replacement is the next deploy: a new process space and deployment id on the same volume.
|
||||
if (kind === "K15") receipt.redeploy = this.advanceDeploy(receipt.faultId);
|
||||
this.timeline.add("fault_restarting", { faultId: receipt.faultId });
|
||||
return this.spawnServer();
|
||||
},
|
||||
@@ -668,6 +681,19 @@ export class ChaosSupervisor {
|
||||
throw new Error(`Second server did not report ready within ${READY_TIMEOUT_MS}ms`);
|
||||
}
|
||||
|
||||
/** K15: move the next server to the next deploy generation (redeploy.ts). */
|
||||
private advanceDeploy(faultId: string): NonNullable<FaultReceipt["redeploy"]> {
|
||||
if (!this.redeploy || !this.environment) throw new Error("K15 needs a redeploy wave (ChaosSupervisor.create redeployWaveId)");
|
||||
const from = this.redeploy.identity;
|
||||
const to = nextRedeployIdentity(this.redeploy.waveId, from);
|
||||
this.redeploy.identity = to;
|
||||
Object.assign(this.environment, redeployEnvironment(to, this.paths.home));
|
||||
const receipt = redeployReceipt(from, to);
|
||||
this.timeline.add("redeploy_switched", { faultId, ...receipt });
|
||||
this.log.write(`\n[restart-chaos] ${faultId} K15 redeploy ${from.hostname}/${from.deploymentId} -> ${to.hostname}/${to.deploymentId}\n`);
|
||||
return receipt;
|
||||
}
|
||||
|
||||
/**
|
||||
* K14: move the server from the older build to this checkout. Only the first K14 switches; a later
|
||||
* one restarts the current build (recorded with `switched: false`).
|
||||
|
||||
@@ -6,11 +6,13 @@ import type { TimeFaultRecord } from "./time-faults.js";
|
||||
|
||||
/**
|
||||
* K14 (upgrade mid-run): a graceful stop, then the replacement starts from the current build (`--upgrade-from`).
|
||||
* K15 (redeploy, redeploy.ts): a graceful stop, then the replacement starts in a new emulated process space with a new
|
||||
* Railway-shaped deployment id on the same volume, as a hosted deploy replaces the server's container.
|
||||
* TM1–TM3 (time faults, time-faults.ts): freeze the server, sleep with aged rows, or an aged outage.
|
||||
* RR1–RR3 (repeated restarts, repeated-restarts.ts): a K1, then more restarts while the replacement recovers.
|
||||
* MS1–MS3 (two servers, one database, multi-server.ts): a second server beside the first.
|
||||
*/
|
||||
export const FAULT_KINDS = ["K1", "K2", "K4", "K6", "K14", "TM1", "TM2", "TM3", "RR1", "RR2", "RR3", "MS1", "MS2", "MS3"] as const;
|
||||
export const FAULT_KINDS = ["K1", "K2", "K4", "K6", "K14", "K15", "TM1", "TM2", "TM3", "RR1", "RR2", "RR3", "MS1", "MS2", "MS3"] as const;
|
||||
export type FaultKind = (typeof FAULT_KINDS)[number];
|
||||
/** The shortest seeded K4 outage (scheduler.ts K4_HOLD_RANGE_MS), also the K4 hold after a provider fault. */
|
||||
export const K4_MIN_HOLD_MS = 120_000;
|
||||
@@ -25,8 +27,9 @@ export type TaskFaultKind = FaultKind | "K13" | "K0" | ProviderFaultKind;
|
||||
* G1: an approval-gated action across a restart. H1: a budget hard stop crossed mid-run with a restart (its own wave).
|
||||
* Q1: a saved question (question-handoff.ts). Q2, Q3: journeys.ts.
|
||||
* L1: high-output, high-accounting run for database contention (contention.ts).
|
||||
* F1: an absolute symlink pointing out of the repository workspace, kept at rest (staging save-back class F).
|
||||
*/
|
||||
export const WORKLOAD_IDS = ["A1", "B1", "B5", "C1", "M1", "G1", "H1", "R1", "S1", "Q1", "L1", "Q2", "Q3"] as const;
|
||||
export const WORKLOAD_IDS = ["A1", "B1", "B5", "C1", "M1", "G1", "H1", "R1", "S1", "Q1", "L1", "Q2", "Q3", "F1"] as const;
|
||||
export type WorkloadId = (typeof WORKLOAD_IDS)[number];
|
||||
|
||||
export const ADAPTER_IDS = ["codex", "claude", "opencode"] as const;
|
||||
@@ -281,6 +284,8 @@ export interface FaultReceipt {
|
||||
crashPoint?: { name: string; hit: number; pid: number; token?: string };
|
||||
/** K14: the build the stopped server ran and the build its replacement runs. */
|
||||
upgrade?: { fromRoot: string; fromSha: string | null; toRoot: string; toSha: string | null; switched: boolean };
|
||||
/** K15: the deploy generation, emulated host name and deployment id before and after. */
|
||||
redeploy?: ReturnType<typeof import("./redeploy.js").redeployReceipt>;
|
||||
/** TM1–TM3: what was frozen and which rows were aged. */
|
||||
timeFault?: TimeFaultRecord;
|
||||
/** RR1–RR3: every stop and replacement of the sequence, the first kill included. */
|
||||
|
||||
@@ -41,6 +41,10 @@ const briefMarker = (nonce: string) => `CHAOS-${nonce}`;
|
||||
const followUpMarker = (nonce: string) => `FOLLOWUP-${nonce}`;
|
||||
const continueMarker = (nonce: string) => `CONTINUE-${nonce}`;
|
||||
const notesFile = (nonce: string) => `NOTES-${nonce}.md`;
|
||||
/** F1: the absolute symlink the agent creates and keeps in the repository root. */
|
||||
export const f1LinkName = (nonce: string) => `hosts-link-${nonce}`;
|
||||
/** F1: an `ls -l` line for that link: a symlink (`l` mode) naming the link and its absolute target. */
|
||||
export const f1LinkListing = (nonce: string) => new RegExp(`^l\\S*\\s.*\\s${f1LinkName(nonce)} -> /etc/hosts$`);
|
||||
const benchFile = (nonce: string) => `bench-result-${nonce}.txt`;
|
||||
const benchRecord = (nonce: string) => `.bench/run-${nonce}.txt`;
|
||||
const r1Module = (nonce: string) => `packages/shared/src/chaos/slug-${nonce}.ts`;
|
||||
@@ -319,6 +323,37 @@ export const workloadDefinitions: Record<WorkloadId, WorkloadDefinition> = {
|
||||
]);
|
||||
},
|
||||
},
|
||||
F1: {
|
||||
// Staging class F (doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md): a remote save-back refused
|
||||
// an absolute symlink (`remote_workspace_archive_unsafe_symlink`), exhausted its retries and left the task
|
||||
// blocked. Agents create such links in ordinary work (tool caches, `node_modules/.bin`). The oracle expects
|
||||
// the task to finish with its result saved, with no Retry; how the link itself is saved is the product's choice.
|
||||
id: "F1",
|
||||
label: "Absolute symlink kept in the repository workspace",
|
||||
usesRepository: true,
|
||||
fireWindowMs: [20_000, 90_000],
|
||||
budgetMs: 12 * 60_000,
|
||||
title: nonce => `Chaos F1 workspace link ${nonce}`,
|
||||
description: nonce => [
|
||||
"Work in this task's repository workspace (a small Python CLI called chaoscli).",
|
||||
`Run \`ln -s /etc/hosts ${f1LinkName(nonce)}\` in the repository root to create a symbolic link, and keep it.`,
|
||||
`Create \`${notesFile(nonce)}\` in the repository root. Its first line is exactly \`F1-${nonce}\`.`,
|
||||
`Its second line is the exact output of \`ls -l ${f1LinkName(nonce)}\`, run after you created the link.`,
|
||||
"Do not change any other file and do not run ./bench.sh. Then mark this task done.",
|
||||
].join("\n"),
|
||||
collect: { documents: [], files: nonce => [notesFile(nonce)], commands: [] },
|
||||
grade(nonce, artifacts) {
|
||||
const lines = (artifacts.files[notesFile(nonce)] ?? "").split(/\r?\n/).map(line => line.trim()).filter(Boolean);
|
||||
return result([
|
||||
statusCheck(artifacts),
|
||||
{ name: "workspace_located", ok: artifacts.workspacePath !== null },
|
||||
{ name: "notes_file", ok: lines[0] === `F1-${nonce}` },
|
||||
// Proof the absolute link was in the workspace before the save-back: the agent's own `ls -l` output, saved
|
||||
// with the notes file. Best-effort against an agent that invents it; how the link itself is saved is not graded.
|
||||
{ name: "link_listed", ok: f1LinkListing(nonce).test(lines[1] ?? ""), detail: JSON.stringify((lines[1] ?? "").slice(0, 160)) },
|
||||
]);
|
||||
},
|
||||
},
|
||||
R1: {
|
||||
id: "R1",
|
||||
label: "Code change in a Paperclip clone",
|
||||
|
||||
Reference in new issue
Block a user