From 874fed09a479f748554b97e9d6146403828674c4 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 12:33:13 -0600 Subject: [PATCH] test(restart-chaos): enable isolated workspaces for repo tasks (#19536) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos`) restarts the control plane while agents work, then grades the result > - Rule R2 says files an agent wrote before a restart must still be present afterwards > - Repo workloads failed R2 with `pre_fault_files_lost`. The first run worked in the shared project checkout, not in its own worktree > - The seed sets a `git_worktree` project policy, but the server applies that policy only when the instance `enableIsolatedWorkspaces` setting is on. The seed did not turn it on > - This pull request makes the seed turn on the setting (as users do on the Experimental page, and as `tests/runner-e2e` does) and fail if the server does not confirm it > - The benefit is that repo tasks get their own worktree, so one task's files cannot appear in another task's R2 snapshot ## Linked Issues or Issue Description Refs #19477 **What happened?** In a restart chaos wave, two repo tasks (B1 and B5) ran at the same time. Both ran in the project primary checkout (`strategyType: project_primary`). Task B1's `NOTES-*.md` file and its `cli.py` edit appeared in the B5 task's workspace before any fault. After a Retry, the B5 task moved to a fresh attempt directory, and R2 reported the B1 file as lost from B5. **Expected behavior** Each repo task gets its own git worktree, as the harness project policy asks. The R2 snapshot of a task contains only that task's files. **Steps to reproduce** 1. Run `tests/restart-chaos/main.ts --env local --adapters codex --workloads B1,B5 --concurrency 2`. 2. Read `currentExecutionWorkspace` for each task. It shows `project_primary` and the fixture checkout as `cwd`. 3. Read `GET /api/instance/settings/experimental`. `enableIsolatedWorkspaces` is `false`. **Root cause** `server/src/services/heartbeat.ts` reads `enableIsolatedWorkspaces` and `gateProjectExecutionWorkspacePolicy` (`server/src/services/execution-workspace-policy.ts`) returns `null` when the setting is off. The project policy is then ignored, and the task resolves to the shared primary checkout. `tests/restart-chaos/seed.ts` created the project policy but did not turn on the setting. ## What Changed - `tests/restart-chaos/seed.ts`: before it creates the project, the seed sends `PATCH /api/instance/settings/experimental` with `enableIsolatedWorkspaces: true`. If the response does not confirm the setting, the seed stops with an error. It records a `seed_isolated_workspaces` timeline entry. - `tests/restart-chaos/seed.test.ts`: new unit tests. One test checks that the seed turns on the setting before it creates the `git_worktree` project. One test checks that the seed refuses to continue when the server does not confirm the setting. - `tests/restart-chaos/README.md`: the `seed.ts` row says that the seed turns on the setting, and why. ## Verification - Red before the fix: `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts seed.test.ts` failed 2/2 (`expected -1 to be greater than or equal to 0`). - Green after the fix: the full restart-chaos unit suite passes, 5 files and 42 tests. - Live wave on this branch (`16f5c8c5d7`, local runner, native Codex, macOS, load average about 40): `tests/restart-chaos/main.ts --env local --adapters codex --workloads B1 --tasks 3 --concurrency 1 --faults K1 --seed 21` (wave `w-20261106T181436-s21`). The timeline shows `seed_isolated_workspaces`. Result: 3/3 pass. Each task crossed a SIGKILL restart. R1, R2, R4, R5, R6 and the workload oracle passed for each task, with no lost files. - Each task's pre-fault file list now has a `.git` file (a git worktree, not the primary checkout) and only its own `NOTES-*.md` file. In the failing wave, the B5 task's list had the B1 task's `NOTES-*.md` file. - Not exercised: no task needed a Retry in this wave, so the Retry-to-fresh-attempt path did not run. - Ledger preflight: install and runner builds passed. The provider-free restart suite had 2 failures in `runnerd-codex-transport-warm-attachment.test.ts` at load average 44; a rerun of that file passed 18/18. ## Risks - Low risk. The change is in the test harness only. It changes no product code. - The harness instance is temporary, so the experimental setting does not reach a real instance. - Repo tasks now run in git worktrees. Earlier waves ran in the shared checkout, so R2 results for repo workloads from earlier waves are not comparable with new waves. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). Standard context window; reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- tests/restart-chaos/README.md | 2 +- tests/restart-chaos/seed.test.ts | 68 ++++++++++++++++++++++++++++++++ tests/restart-chaos/seed.ts | 10 +++++ 3 files changed, 79 insertions(+), 1 deletion(-) create mode 100644 tests/restart-chaos/seed.test.ts diff --git a/tests/restart-chaos/README.md b/tests/restart-chaos/README.md index 393d5fe9848..14207e23782 100644 --- a/tests/restart-chaos/README.md +++ b/tests/restart-chaos/README.md @@ -31,7 +31,7 @@ are wired (see the TODO in `seed.ts`). | File | Role | | --- | --- | | `supervisor.ts` | Owns the server. Reuses `tests/runner-e2e` process code (config, ports, env isolation, wrapper-owned Postgres, process-tree stopper, `replaceController`). Faults: K1 SIGKILL + immediate restart, K2 SIGTERM + restart, K4 SIGKILL + 2–5 min hold. The controller runtime cache is kept (discarding it is K11). | -| `seed.ts` | Chaos Lab company, encrypted provider secrets, native Codex / ACPX Claude / OpenCode agents from the runner-e2e catalog, and the `chaos-sandbox` project (git repo: python CLI, one failing test, `bench.sh` ≈ 4 min) with isolated git-worktree workspaces. | +| `seed.ts` | Chaos Lab company, encrypted provider secrets, native Codex / ACPX Claude / OpenCode agents from the runner-e2e catalog, and the `chaos-sandbox` project (git repo: python CLI, one failing test, `bench.sh` ≈ 4 min) with isolated git-worktree workspaces. It first turns on the instance `enableIsolatedWorkspaces` setting; without it the server ignores the project policy and every repo task shares the primary checkout. | | `scheduler.ts` | Seeded plan: workload × adapter × fault cells and a random fire time per task. Saved to `results//plan.json`. `--replay` reruns one task alone with the same choices and fires the recorded sequence of faults that crossed it (other tasks' faults included) at the same offsets from launch. | | `observer.ts` | Polls each issue and its runs. When a task rests needing Retry it calls `POST /api/issues/:id/retry` once, with the body `useTaskRetry.ts` sends. Collects T-oracle artifacts and the R6 sweep. | | `grader.ts` | Pure R1–R6 grading over the samples. | diff --git a/tests/restart-chaos/seed.test.ts b/tests/restart-chaos/seed.test.ts new file mode 100644 index 00000000000..25e4c348844 --- /dev/null +++ b/tests/restart-chaos/seed.test.ts @@ -0,0 +1,68 @@ +import { mkdtemp, rm } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import type { ChaosApi } from "./api.js"; +import { seedChaosLab } from "./seed.js"; +import { Timeline } from "./timeline.js"; + +type Call = { method: string; route: string; body?: unknown }; + +/** Records the seed's board API calls; answers like a fresh local_trusted instance. */ +function fakeApi(experimentalResponse: (body: Record) => Record) { + const calls: Call[] = []; + let ids = 0; + const api = { + get: async (route: string) => { + calls.push({ method: "GET", route }); + if (route.includes("/environments")) return [{ id: "environment-1", driver: "local" }]; + throw new Error(`unexpected GET ${route}`); + }, + post: async (route: string, body?: unknown) => { + calls.push({ method: "POST", route, body }); + return { id: `id-${++ids}`, name: "x" }; + }, + patch: async (route: string, body?: unknown) => { + calls.push({ method: "PATCH", route, body }); + if (route === "/api/instance/settings/experimental") return experimentalResponse(body as Record); + throw new Error(`unexpected PATCH ${route}`); + }, + }; + return { api: api as unknown as ChaosApi, calls }; +} + +describe("seedChaosLab workspace isolation", () => { + let fixtureRoot: string; + beforeEach(async () => { + fixtureRoot = await mkdtemp(path.join(os.tmpdir(), "rc21-seed-")); + vi.stubEnv("OPENAI_API_KEY", "test-only-not-a-key"); + }); + afterEach(async () => { + vi.unstubAllEnvs(); + await rm(fixtureRoot, { recursive: true, force: true }); + }); + + const seed = (api: ChaosApi) => seedChaosLab({ api, env: "local", adapters: ["codex"], concurrency: 2, + budgetCents: 3_000, fixtureRoot, waveId: "w-test", timeline: new Timeline(null) }); + + // RC-21: the project's git_worktree policy is gated behind the instance's + // isolated-workspaces setting. Without it every repo task ran in the shared + // primary checkout, so concurrent tasks wrote into one tree. + it("enables isolated workspaces before creating the git_worktree project", async () => { + const { api, calls } = fakeApi(body => ({ ...body })); + await seed(api); + const enable = calls.findIndex(call => call.method === "PATCH" && call.route === "/api/instance/settings/experimental" + && (call.body as Record).enableIsolatedWorkspaces === true); + const project = calls.findIndex(call => call.method === "POST" && call.route.endsWith("/projects")); + expect(enable).toBeGreaterThanOrEqual(0); + expect(project).toBeGreaterThan(enable); + expect((calls[project]!.body as { executionWorkspacePolicy: unknown }).executionWorkspacePolicy) + .toMatchObject({ enabled: true, defaultMode: "isolated_workspace", workspaceStrategy: { type: "git_worktree" } }); + }); + + it("refuses to seed when the instance does not confirm isolated workspaces", async () => { + const { api, calls } = fakeApi(() => ({ enableIsolatedWorkspaces: false })); + await expect(seed(api)).rejects.toThrow(/isolated workspaces/i); + expect(calls.some(call => call.method === "POST" && call.route.endsWith("/projects"))).toBe(false); + }); +}); diff --git a/tests/restart-chaos/seed.ts b/tests/restart-chaos/seed.ts index b9ee5f26c93..2799fb2384f 100644 --- a/tests/restart-chaos/seed.ts +++ b/tests/restart-chaos/seed.ts @@ -102,6 +102,16 @@ export async function seedChaosLab(input: { timeline.add("seed_agent", { adapter, agentId: created.id, profile: profile.id, model: profile.model }); } + // The project's git_worktree policy only takes effect when the instance has + // isolated workspaces enabled (the Experimental setting users turn on, as in + // runner-e2e). Without it every repo task shares the primary checkout (RC-21). + const experimental = await api.patch<{ enableIsolatedWorkspaces?: boolean }>( + "/api/instance/settings/experimental", { enableIsolatedWorkspaces: true }); + if (experimental?.enableIsolatedWorkspaces !== true) { + throw new Error("Isolated instance did not enable isolated workspaces; repo tasks would share one checkout"); + } + timeline.add("seed_isolated_workspaces", { enableIsolatedWorkspaces: true }); + const project = await api.post<{ id: string }>(`/api/companies/${company.id}/projects`, { name: "chaos-sandbox", description: "Fixture repository for restart chaos workloads",