test(restart-chaos): enable isolated workspaces for repo tasks (#19536)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos`) restarts the
control plane while agents work, then grades the result
> - Rule R2 says files an agent wrote before a restart must still be
present afterwards
> - Repo workloads failed R2 with `pre_fault_files_lost`. The first run
worked in the shared project checkout, not in its own worktree
> - The seed sets a `git_worktree` project policy, but the server
applies that policy only when the instance `enableIsolatedWorkspaces`
setting is on. The seed did not turn it on
> - This pull request makes the seed turn on the setting (as users do on
the Experimental page, and as `tests/runner-e2e` does) and fail if the
server does not confirm it
> - The benefit is that repo tasks get their own worktree, so one task's
files cannot appear in another task's R2 snapshot

## Linked Issues or Issue Description

Refs #19477

**What happened?**
In a restart chaos wave, two repo tasks (B1 and B5) ran at the same
time. Both ran in the project primary checkout (`strategyType:
project_primary`). Task B1's `NOTES-*.md` file and its `cli.py` edit
appeared in the B5 task's workspace before any fault. After a Retry, the
B5 task moved to a fresh attempt directory, and R2 reported the B1 file
as lost from B5.

**Expected behavior**
Each repo task gets its own git worktree, as the harness project policy
asks. The R2 snapshot of a task contains only that task's files.

**Steps to reproduce**
1. Run `tests/restart-chaos/main.ts --env local --adapters codex
--workloads B1,B5 --concurrency 2`.
2. Read `currentExecutionWorkspace` for each task. It shows
`project_primary` and the fixture checkout as `cwd`.
3. Read `GET /api/instance/settings/experimental`.
`enableIsolatedWorkspaces` is `false`.

**Root cause**
`server/src/services/heartbeat.ts` reads `enableIsolatedWorkspaces` and
`gateProjectExecutionWorkspacePolicy`
(`server/src/services/execution-workspace-policy.ts`) returns `null`
when the setting is off. The project policy is then ignored, and the
task resolves to the shared primary checkout.
`tests/restart-chaos/seed.ts` created the project policy but did not
turn on the setting.

## What Changed

- `tests/restart-chaos/seed.ts`: before it creates the project, the seed
sends `PATCH /api/instance/settings/experimental` with
`enableIsolatedWorkspaces: true`. If the response does not confirm the
setting, the seed stops with an error. It records a
`seed_isolated_workspaces` timeline entry.
- `tests/restart-chaos/seed.test.ts`: new unit tests. One test checks
that the seed turns on the setting before it creates the `git_worktree`
project. One test checks that the seed refuses to continue when the
server does not confirm the setting.
- `tests/restart-chaos/README.md`: the `seed.ts` row says that the seed
turns on the setting, and why.

## Verification

- Red before the fix: `pnpm exec vitest run --config
tests/restart-chaos/vitest.config.ts seed.test.ts` failed 2/2 (`expected
-1 to be greater than or equal to 0`).
- Green after the fix: the full restart-chaos unit suite passes, 5 files
and 42 tests.
- Live wave on this branch (`16f5c8c5d7`, local runner, native Codex,
macOS, load average about 40): `tests/restart-chaos/main.ts --env local
--adapters codex --workloads B1 --tasks 3 --concurrency 1 --faults K1
--seed 21` (wave `w-20261106T181436-s21`). The timeline shows
`seed_isolated_workspaces`. Result: 3/3 pass. Each task crossed a
SIGKILL restart. R1, R2, R4, R5, R6 and the workload oracle passed for
each task, with no lost files.
- Each task's pre-fault file list now has a `.git` file (a git worktree,
not the primary checkout) and only its own `NOTES-*.md` file. In the
failing wave, the B5 task's list had the B1 task's `NOTES-*.md` file.
- Not exercised: no task needed a Retry in this wave, so the
Retry-to-fresh-attempt path did not run.
- Ledger preflight: install and runner builds passed. The provider-free
restart suite had 2 failures in
`runnerd-codex-transport-warm-attachment.test.ts` at load average 44; a
rerun of that file passed 18/18.

## Risks

- Low risk. The change is in the test harness only. It changes no
product code.
- The harness instance is temporary, so the experimental setting does
not reach a real instance.
- Repo tasks now run in git worktrees. Earlier waves ran in the shared
checkout, so R2 results for repo workloads from earlier waves are not
comparable with new waves.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits). Standard context window; reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

tests/restart-chaos/README.md

@@ -31,7 +31,7 @@ are wired (see the TODO in `seed.ts`).
 | File | Role |
 | --- | --- |
 | `supervisor.ts` | Owns the server. Reuses `tests/runner-e2e` process code (config, ports, env isolation, wrapper-owned Postgres, process-tree stopper, `replaceController`). Faults: K1 SIGKILL + immediate restart, K2 SIGTERM + restart, K4 SIGKILL + 2–5 min hold. The controller runtime cache is kept (discarding it is K11). |
-| `seed.ts` | Chaos Lab company, encrypted provider secrets, native Codex / ACPX Claude / OpenCode agents from the runner-e2e catalog, and the `chaos-sandbox` project (git repo: python CLI, one failing test, `bench.sh` ≈ 4 min) with isolated git-worktree workspaces. |
+| `seed.ts` | Chaos Lab company, encrypted provider secrets, native Codex / ACPX Claude / OpenCode agents from the runner-e2e catalog, and the `chaos-sandbox` project (git repo: python CLI, one failing test, `bench.sh` ≈ 4 min) with isolated git-worktree workspaces. It first turns on the instance `enableIsolatedWorkspaces` setting; without it the server ignores the project policy and every repo task shares the primary checkout. |
 | `scheduler.ts` | Seeded plan: workload × adapter × fault cells and a random fire time per task. Saved to `results/<wave>/plan.json`. `--replay` reruns one task alone with the same choices and fires the recorded sequence of faults that crossed it (other tasks' faults included) at the same offsets from launch. |
 | `observer.ts` | Polls each issue and its runs. When a task rests needing Retry it calls `POST /api/issues/:id/retry` once, with the body `useTaskRetry.ts` sends. Collects T-oracle artifacts and the R6 sweep. |
 | `grader.ts` | Pure R1–R6 grading over the samples. |

tests/restart-chaos/seed.test.ts

@@ -0,0 +1,68 @@
+import { mkdtemp, rm } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+import type { ChaosApi } from "./api.js";
+import { seedChaosLab } from "./seed.js";
+import { Timeline } from "./timeline.js";
+
+type Call = { method: string; route: string; body?: unknown };
+
+/** Records the seed's board API calls; answers like a fresh local_trusted instance. */
+function fakeApi(experimentalResponse: (body: Record<string, unknown>) => Record<string, unknown>) {
+  const calls: Call[] = [];
+  let ids = 0;
+  const api = {
+    get: async (route: string) => {
+      calls.push({ method: "GET", route });
+      if (route.includes("/environments")) return [{ id: "environment-1", driver: "local" }];
+      throw new Error(`unexpected GET ${route}`);
+    },
+    post: async (route: string, body?: unknown) => {
+      calls.push({ method: "POST", route, body });
+      return { id: `id-${++ids}`, name: "x" };
+    },
+    patch: async (route: string, body?: unknown) => {
+      calls.push({ method: "PATCH", route, body });
+      if (route === "/api/instance/settings/experimental") return experimentalResponse(body as Record<string, unknown>);
+      throw new Error(`unexpected PATCH ${route}`);
+    },
+  };
+  return { api: api as unknown as ChaosApi, calls };
+}
+
+describe("seedChaosLab workspace isolation", () => {
+  let fixtureRoot: string;
+  beforeEach(async () => {
+    fixtureRoot = await mkdtemp(path.join(os.tmpdir(), "rc21-seed-"));
+    vi.stubEnv("OPENAI_API_KEY", "test-only-not-a-key");
+  });
+  afterEach(async () => {
+    vi.unstubAllEnvs();
+    await rm(fixtureRoot, { recursive: true, force: true });
+  });
+
+  const seed = (api: ChaosApi) => seedChaosLab({ api, env: "local", adapters: ["codex"], concurrency: 2,
+    budgetCents: 3_000, fixtureRoot, waveId: "w-test", timeline: new Timeline(null) });
+
+  // RC-21: the project's git_worktree policy is gated behind the instance's
+  // isolated-workspaces setting. Without it every repo task ran in the shared
+  // primary checkout, so concurrent tasks wrote into one tree.
+  it("enables isolated workspaces before creating the git_worktree project", async () => {
+    const { api, calls } = fakeApi(body => ({ ...body }));
+    await seed(api);
+    const enable = calls.findIndex(call => call.method === "PATCH" && call.route === "/api/instance/settings/experimental"
+      && (call.body as Record<string, unknown>).enableIsolatedWorkspaces === true);
+    const project = calls.findIndex(call => call.method === "POST" && call.route.endsWith("/projects"));
+    expect(enable).toBeGreaterThanOrEqual(0);
+    expect(project).toBeGreaterThan(enable);
+    expect((calls[project]!.body as { executionWorkspacePolicy: unknown }).executionWorkspacePolicy)
+      .toMatchObject({ enabled: true, defaultMode: "isolated_workspace", workspaceStrategy: { type: "git_worktree" } });
+  });
+
+  it("refuses to seed when the instance does not confirm isolated workspaces", async () => {
+    const { api, calls } = fakeApi(() => ({ enableIsolatedWorkspaces: false }));
+    await expect(seed(api)).rejects.toThrow(/isolated workspaces/i);
+    expect(calls.some(call => call.method === "POST" && call.route.endsWith("/projects"))).toBe(false);
+  });
+});

tests/restart-chaos/seed.ts

@@ -102,6 +102,16 @@ export async function seedChaosLab(input: {
     timeline.add("seed_agent", { adapter, agentId: created.id, profile: profile.id, model: profile.model });
   }
 
+  // The project's git_worktree policy only takes effect when the instance has
+  // isolated workspaces enabled (the Experimental setting users turn on, as in
+  // runner-e2e). Without it every repo task shares the primary checkout (RC-21).
+  const experimental = await api.patch<{ enableIsolatedWorkspaces?: boolean }>(
+    "/api/instance/settings/experimental", { enableIsolatedWorkspaces: true });
+  if (experimental?.enableIsolatedWorkspaces !== true) {
+    throw new Error("Isolated instance did not enable isolated workspaces; repo tasks would share one checkout");
+  }
+  timeline.add("seed_isolated_workspaces", { enableIsolatedWorkspaces: true });
+
   const project = await api.post<{ id: string }>(`/api/companies/${company.id}/projects`, {
     name: "chaos-sandbox",
     description: "Fixture repository for restart chaos workloads",