perf(workspaces): group same-root cohorts with bounded workers (#20437)

## Thinking Path > - Paperclip preserves agent work when its control-plane server is replaced. > - Online migration imports local workspaces into authenticated snapshots. > - Several workspace records can point to one physical directory. > - Shared child verification avoids repeated work inside one held cohort fence. > - The eight-member limit splits 30 identical-root aliases into four cohorts and repeats the child passes. > - This pull request admits up to 32 members only for one canonical physical root, while keeping each work stage bounded to eight peers. > - The benefit is fewer repeated child checks while capture and publication guards stay in place. ## Linked Issues or Issue Description Refs: #19873, #19882, #20418, #20424, #20429. Related PR search found shared verification and restart fixes, but no larger same-root cohort owner. The shared verifier and archived-source changes are merged into this branch from current master. **Current behavior** Thirty fresh aliases of one physical root require cohorts of 8, 8, 8 and 6. Each cohort performs an initial and final local child-validation pass. The same children are checked eight times in total. **Proposed behavior** Admit one cohort with nine to 32 members only when every canonical resolved physical root is exactly equal. Keep the existing complete source-scope equality and admission guards. Keep mixed-leaf cohorts limited to eight members. Bound source reads, reservations, recovery checks, final readers, replay and cleanup to eight joined peers. Capture concurrency stays at most eight. **Reason and benefit** For N distinct canonical covered children, the 30-alias case changes 8N local child checks to 2N, a 75% reduction in repeated child validation. If N is 68, that is 544 checks versus 136. Thirty parent captures, uploads and restores remain required. No total wall-clock improvement is claimed. **Breaking changes** Saved cohort fields and exact membership remain unchanged. Older tools that enforce the eight-member bound refuse larger cohorts. Use the newer tool to resume them; never split or regroup their saved membership. ## What Changed - Raise the canonical service, saved-cohort and maintenance-request member bounds to 32. Keep the capture-worker bound at eight and single-workspace modes at one. - Require one exact canonical physical root for more than eight members. Keep the existing full source-scope comparison, leaf rule and P/I/L/F guards. - Use one private stage mapper with batches of at most eight. Join every started peer before an error can leave the stage. Stop future batches after a failure or cancellation. - Preserve the existing cancellation and pending-snapshot ordering for stages of at most eight members. Check cancellation around extended batches. - Keep all saved cohort, member, operation and snapshot identities, proof refresh and atomic publication checks. - Bound cleanup without skipping later scratch cleanup after an earlier cleanup error. - Extend the existing PostgreSQL fixture with ten real aliases, partial-upload resume, cold restoration, replay and mixed-root refusal. Extend the request boundary test. - Document old-tool refusal, work counts and the retained-scratch capacity bound. ## Verification - Canonical Node24 worktree preflight and Plugin SDK prerequisites passed. After merging current master, preflight passed again with unchanged cached build inputs. - Genuine RED on the old bound: both real ten-alias import/resume cases and the 32-member request reject the eight-member limit (3 failed; 47 not selected; 18.23 seconds). - First focused GREEN: 4 passed in 24.20 seconds. The fresh ten-alias case performs exactly four local child captures for two children: one initial and one final pass. Actual reader, transaction and upload maxima are eight. All ten archives cold-restore after the original source is parked; replay retains the same cohort. - The partial-upload case keeps all ten saved operation IDs and the existing snapshot IDs. More than eight members across different canonical leaf roots refuse before any journal or storage work. The request parser accepts 32 members and eight workers, and refuses 33 members or nine workers. - The first expanded corpus had 55 passes and one real compatibility failure: a new eager cancellation check skipped the old pending snapshot boundary. The mapper now retains existing cancellation ordering for at most eight peers. That failure remains recorded as a nonpass. - Final sources include current master at `18697b8d21`, with the verifier, archived-source and UI repairs. The selected four-file corpus passes 47 cases in 78.88 seconds, with 191 intentionally not selected. The full maintenance suite separately passes 23 cases in 10.21 seconds. Nine cases overlap; the final union is 61 distinct passing checks. - Final coverage includes all 27 cohort cases, all 23 maintenance cases, all six actual old-format compatibility cases and five archived prepared-source controls. Existing source mutation, finalizer contention, actual fence loss, cancellation peer joins, journal compare-and-swap and outer-commit rollback controls pass. - Final server TypeScript, module boundaries and whitespace checks pass. Only the resource-risk documentation changed after that test run; runtime and fixture bytes stayed unchanged. - Commands: `pnpm exec vitest run server/src/services/remote-workspace-cohort-migration.test.ts server/src/maintenance/online-workspace-migration.test.ts server/src/__tests__/online-workspace-migration-compatibility.test.ts`; archived controls use the existing `remote-workspace-migration.test.ts` cases named `unselected archived recipe` and `archived prepared source`; `pnpm --filter @paperclipai/server exec tsc --noEmit`; `node scripts/check-module-boundaries.mjs`. - Initial source review was 4/5 and requested a control for the extended reader-batch stop boundary. The test-only follow-up `ca80ae9cc8791a72dab632090488093c5d0916f3` adds actual ten-member failure and cancellation cases. They hold a first-batch peer, verify that the source fence stays held, and prove members nine and ten never start preflight. Both pass in 10.87 seconds with no new journal, snapshot or upload effects. Runtime source hashes are unchanged; the local union is now 63 distinct checks. - The first stop-test attempt observed the two controlled readers before all SQL-backed peers had entered the inspection callback. Its assertion was too early. The corrected test checks the held first-batch readers at that point, then requires all eight to join before completion. Both cases pass; the earlier attempt remains a nonpass. - Final head `ca80ae9cc8791a72dab632090488093c5d0916f3`: [required CI](https://github.com/paperclipai/paperclip/actions/runs/42737676664) and [aggregate verify](https://github.com/paperclipai/paperclip/actions/runs/42737676664/job/129575813765) passed. Full rollup: 56 successes, four intentional skips and one advisory neutral result. Classic Snyk passed. [Greptile](https://github.com/paperclipai/paperclip/pull/20437#issuecomment-6756419093) and [Ponytail](https://github.com/paperclipai/paperclip/pull/20437#issuecomment-6756388693) both reviewed that exact head at 5/5. The one review finding is addressed and resolved; no open findings remain. Older partial CI and the initial 4/5 review remain nonfinal evidence. Hosted acceptance and measured wall-clock savings remain pending. ## Risks - Completed parent captures retain their scratch until the atomic publication completes. A 32-member cohort can retain up to four times the parent scratch of an eight-member cohort. Check actual parent capture, verification and restore sizes against current target filesystem capacity before selecting a larger cohort. Keep the existing resource guards. - Larger member arrays must not raise execution fanout. Every reader/reservation/final stage is bounded to eight and joins current peers before failure. Capture workers remain at most eight; child-verification workers remain two. - More than eight members require the exact same canonical physical root and complete source scope. A different root, new writer, changed source, recovery owner or journal still refuses through the existing owners. - Shared verification is limited to the held invocation fence. The separate fresh final pass remains required after all captures join. Different child sets and restart attempts do not share a local-byte proof. - The old saved membership and archive identities remain authoritative. Older tools fail closed on the larger bound. No schema, archive or provider format changes are included. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. The runtime does not expose a more specific deployment identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-11 05:11:49 +01:00 · 2026-11-09 20:03:25 -06:00
1 parent 1ca3e8ea2a
commit 07f6e8ff75
@@ -38,9 +38,9 @@ their existing reconstruction path.
 Preparation, lifecycle, runtime-service, expiration, and reopen owners retain
 the same physical keys before filesystem mutation or lifecycle exclusion. The
 importer holds all cohort task/project and lifecycle keys through publication.
-Each cohort reserves its at most eight members in independent transactions in
-parallel. It joins every reservation before starting capture or releasing the
-source fence, including on failure. Partial journals keep the full exact member
+Each cohort reserves its members in fully joined batches of at most eight
+independent transactions. It joins every reservation before starting capture or
+releasing the source fence, including on failure. Partial journals keep the full exact member
 and operation set for resume; publication stays serial in one transaction.
 It also holds the existing per-run physical finalizer keys for each exact native
 recovery proof. It re-inspects the complete source proof before capture and
@@ -128,3 +128,28 @@ reacquires the source locks and captures current local bytes again. Read-only
 verifier callers do not opt into checkpoint publication.
 A change that strengthens immutable archive verification must advance the proof
 version so earlier verification cannot bypass the new checks.
+
+
+A cohort with nine to 32 members is admitted only when every canonical resolved
+physical root is exactly the same. Cohorts of at most eight members retain the
+existing mixed-leaf rules. Source resolution, admission, reservation, recovery
+checks, final readers, replay and cleanup remain bounded to eight peers per
+stage. Capture concurrency remains at most eight, and all running peers join
+before an error or cancellation can release the source fence.
+
+With shared initial and final child verification, 30 fresh aliases need one
+initial and one final child pass instead of four pairs. For N distinct covered
+children this changes 8N child checks to 2N. Thirty parent captures, uploads and
+restores remain required. This is a work-count reduction; hosted wall-clock
+improvement is not yet measured. Saved six- and eight-member cohorts retain their
+exact membership. The canonical maintenance request accepts the same bounded
+32-member list; single-workspace and capture-worker limits stay unchanged.
+The journal shape is unchanged; older tools that enforce the
+eight-member bound refuse larger cohorts and must not regroup them.
+
+Completed parent captures retain scratch until atomic publication. A larger
+cohort can therefore retain up to four times the parent scratch of an
+eight-member cohort, even with the same worker limit. Select a larger cohort
+only after checking the actual parent capture footprint, its verification and
+restore scratch, and current free capacity on the target filesystem. The worker
+limit does not bound retained scratch. Keep the existing resource guards.
@@ -1,3 +1,4 @@
+import { randomUUID } from "node:crypto";
 import { describe, expect, it, vi } from "vitest";
 import { assertOnlineMigrationManifest, assertOnlineMigrationRuntime, assertOnlineMigrationSchema, onlineWorkspaceMigrationReceipt, parseOnlineWorkspaceMigrationRequest, runOnlineWorkspaceMigration } from "./online-workspace-migration.js";
 import type { Db } from "@paperclipai/db";
@@ -52,7 +53,10 @@ describe("online workspace migration request", () => {
  it("refuses duplicate membership and bounds the exact cohort", () => {
    const cohort = { ...request(), mode: "cohort", concurrency: 2 };
    expect(() => parseOnlineWorkspaceMigrationRequest({ ...cohort, workspaceIds: [cohort.workspaceIds[0], cohort.workspaceIds[0]] })).toThrow();
-    expect(() => parseOnlineWorkspaceMigrationRequest({ ...cohort, workspaceIds: Array(9).fill(cohort.workspaceIds[0]) })).toThrow();
+    const workspaceIds = Array.from({ length: 32 }, () => randomUUID());
+    expect(parseOnlineWorkspaceMigrationRequest({ ...cohort, workspaceIds, concurrency: 8 }).workspaceIds).toHaveLength(32);
+    expect(() => parseOnlineWorkspaceMigrationRequest({ ...cohort, workspaceIds: [...workspaceIds, randomUUID()] })).toThrow();
+    expect(() => parseOnlineWorkspaceMigrationRequest({ ...cohort, workspaceIds, concurrency: 9 })).toThrow();
  });
 });

@@ -24,7 +24,7 @@ const requestSchema = z.object({
  toolRoot: z.string().min(1).max(4096),
  source: z.object({ commit, deploymentId: z.string().uuid() }).strict(),
  companyId: z.string().uuid(),
-  workspaceIds: z.array(z.string().uuid()).min(1).max(8),
+  workspaceIds: z.array(z.string().uuid()).min(1).max(32),
  mode: z.enum(["inspect", "single", "cohort", "resume"]),
  operationId: z.string().uuid().optional(),
  snapshotId: z.string().uuid().optional(),
@@ -70,6 +70,15 @@ describe.skipIf(!support.supported)("bounded workspace cohort migration", () =>
    }
    return { companyId, projectId, root, parent: members[0], leaves: members.slice(1) };
  }
+  async function aliasMembers(f: Awaited<ReturnType<typeof fixture>>, base: typeof f.parent, count: number) {
+    const tasks = await db.insert(issues).values(Array.from({ length: count - 1 }, (_, index) => ({
+      companyId: f.companyId, projectId: f.projectId, title: `Alias ${index}` }))).returning();
+    const aliases = await db.insert(executionWorkspaces).values(tasks.map(task => ({ companyId: f.companyId,
+      projectId: f.projectId, sourceIssueId: task.id, name: "Alias", mode: base.mode, strategyType: base.strategyType,
+      providerType: base.providerType, cwd: base.cwd, providerRef: base.providerRef, repoUrl: base.repoUrl, branchName: base.branchName }))).returning();
+    for (const alias of aliases) await db.update(issues).set({ executionWorkspaceId: alias.id }).where(eq(issues.id, alias.sourceIssueId!));
+    return [base, ...aliases];
+  }
  const run = (f: Awaited<ReturnType<typeof fixture>>, extra: Partial<Parameters<typeof migrateRemoteWorkspaceCohort>[0]> = {}) =>
    migrateRemoteWorkspaceCohort({ db, storage, companyId: f.companyId, workspaceIds: f.leaves.map(row => row.id),
      apply: true, concurrency: 2, signal: AbortSignal.timeout(50_000), ...extra });
@@ -214,6 +223,123 @@ describe.skipIf(!support.supported)("bounded workspace cohort migration", () =>
    await assertLocal(f);
    expect(await db.select().from(workspaceOperations)).toHaveLength(mode === "foreign" ? 3 : 2);
  });
+  it.each(["fresh", "resume"] as const)("imports a larger same-root cohort with bounded work (%s)", async mode => {
+    const f = await fixture(), selected = await aliasMembers(f, f.parent, 10), workspaceIds = selected.map(row => row.id);
+    for (const child of f.leaves) await migrateRemoteWorkspace({ db, storage, companyId: f.companyId, workspaceId: child.id,
+      apply: true, signal: AbortSignal.timeout(50_000) });
+    await fs.writeFile(path.join(f.root, "root-untracked.txt"), "parent retained work\n");
+    const inspect = onlineWorkspaceSource.inspectOnlineWorkspaceSource, transaction = db.transaction.bind(db), put = storage.putFile.bind(storage);
+    let readers = 0, maximumReaders = 0, reservations = 0, maximumReservations = 0, uploads = 0, maximumUploads = 0;
+    vi.spyOn(onlineWorkspaceSource, "inspectOnlineWorkspaceSource").mockImplementation(async (...args) => {
+      readers++; maximumReaders = Math.max(maximumReaders, readers);
+      try { return await inspect(...args); } finally { readers--; }
+    });
+    vi.spyOn(db, "transaction").mockImplementation(async (...args) => {
+      reservations++; maximumReservations = Math.max(maximumReservations, reservations);
+      try { return await transaction(...args); } finally { reservations--; }
+    });
+    const capture = vi.spyOn(workspaceArchive, "captureRemoteWorkspaceDirectory");
+    let entered = 0, release!: () => void, interrupt = mode === "resume";
+    let timer: ReturnType<typeof setTimeout> | undefined;
+    const barrier = new Promise<void>(done => { release = done; });
+    vi.spyOn(storage, "putFile").mockImplementation(async input => {
+      uploads++; maximumUploads = Math.max(maximumUploads, uploads);
+      try {
+        if (interrupt && ++entered === 2) throw new Error("large_cohort_upload_interrupted");
+        if (!interrupt) {
+          if (++entered === 1) timer = setTimeout(() => release(), 5000);
+          if (entered === 8) release();
+          await barrier;
+        }
+        return await put(input);
+      } finally { uploads--; }
+    });
+    let savedOperations: Array<{ id: string; metadata: Record<string, unknown> | null }> = [], savedSnapshotIds: string[] = [];
+    try {
+      if (mode === "resume") {
+        await expect(run(f, { workspaceIds, concurrency: 1 })).rejects.toThrow("large_cohort_upload_interrupted");
+        savedOperations = (await db.select().from(workspaceOperations)).filter(row => workspaceIds.includes(row.executionWorkspaceId!));
+        savedSnapshotIds = (await db.select().from(executionWorkspaceSnapshots)).filter(row => workspaceIds.includes(row.executionWorkspaceId)).map(row => row.id);
+        expect(savedOperations).toHaveLength(10); expect(savedSnapshotIds).toHaveLength(2);
+        interrupt = false; entered = 0;
+      }
+      await expect(run(f, { workspaceIds, concurrency: 8 })).resolves.toMatchObject({ applied: true });
+    } finally { clearTimeout(timer); }
+    expect(maximumReaders).toBe(8); expect(maximumReservations).toBe(8); expect(maximumUploads).toBe(8);
+    expect(readers).toBe(0); expect(reservations).toBe(0); expect(uploads).toBe(0);
+    const operations = (await db.select().from(workspaceOperations)).filter(row => workspaceIds.includes(row.executionWorkspaceId!));
+    expect(operations).toHaveLength(10); expect(operations.every(row => row.status === "succeeded")).toBe(true);
+    if (mode === "fresh") expect(capture.mock.calls.filter(([input]) => f.leaves.some(child => input.cwd === child.cwd)
+      && path.basename(input.archivePath) === "current.pcws")).toHaveLength(4);
+    else {
+      expect(operations.map(row => row.id).sort()).toEqual(savedOperations.map(row => row.id).sort());
+      expect(operations.every(row => JSON.stringify(row.metadata!.cohortMigration) === JSON.stringify(savedOperations[0].metadata!.cohortMigration))).toBe(true);
+      expect((await db.select().from(executionWorkspaceSnapshots)).filter(row => savedSnapshotIds.includes(row.id))).toHaveLength(2);
+    }
+    await expect(run(f, { workspaceIds })).resolves.toMatchObject({ applied: true, replayed: true });
+    await fs.rename(f.root, f.root + "-original");
+    for (const workspace of selected) {
+      const [row] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace.id));
+      const ref = (await getWorkspaceSnapshot(db, { companyId: f.companyId, workspaceId: row.id,
+        snapshotId: readRemoteWorkspaceAuthority(row.metadata)!.snapshotId! }))!;
+      const archivePath = path.join(home, `${row.id}.pcws`), targetDir = path.join(home, `${row.id}-cold`);
+      await loadWorkspaceSnapshotArchive({ storage, reference: ref, targetPath: archivePath });
+      await restoreRemoteWorkspaceArchive({ archivePath, targetDir, expectedManifestDigest: ref.manifestDigest });
+      expect(await fs.readFile(path.join(targetDir, "saved.txt"), "utf8")).toBe("committed base\n");
+      expect(await fs.readFile(path.join(targetDir, "root-untracked.txt"), "utf8")).toBe("parent retained work\n");
+      expect(await git(targetDir, "log", "-1", "--format=%s")).toBe("Original history");
+    }
+  }, 90_000);
+  it.each(["failure", "cancel"] as const)("joins the first extended reader batch before stopping later members (%s)", async mode => {
+    const f = await fixture(), selected = await aliasMembers(f, f.parent, 10), ids = selected.map(row => row.id).sort();
+    for (const child of f.leaves) await migrateRemoteWorkspace({ db, storage, companyId: f.companyId, workspaceId: child.id,
+      apply: true, signal: AbortSignal.timeout(50_000) });
+    const operations = await db.select().from(workspaceOperations), snapshots = await db.select().from(executionWorkspaceSnapshots);
+    const inspect = onlineWorkspaceSource.inspectOnlineWorkspaceSource, calls = new Map<string, number>(), started: string[] = [];
+    let bothEntered!: () => void, release!: () => void, failed!: () => void, settled = false;
+    const both = new Promise<void>(done => { bothEntered = done; }), held = new Promise<void>(done => { release = done; });
+    const observed = new Promise<void>(done => { failed = done; }), controller = new AbortController();
+    vi.spyOn(onlineWorkspaceSource, "inspectOnlineWorkspaceSource").mockImplementation(async (...args) => {
+      const id = args[1].id, call = (calls.get(id) ?? 0) + 1; calls.set(id, call);
+      if (ids.includes(id) && call === 2) {
+        started.push(id);
+        if (id === ids[0]) {
+          await both; failed();
+          if (mode === "failure") throw new Error("extended_reader_failure");
+          controller.abort(new Error("extended_reader_cancelled"));
+        } else if (id === ids[1]) { bothEntered(); await held; }
+      }
+      return inspect(...args);
+    });
+    const put = vi.spyOn(storage, "putFile"), other = createDb(database.connectionString, { maxConnections: 1 });
+    const pending = run(f, { workspaceIds: ids, signal: controller.signal })
+      .then(value => ({ value }), error => ({ error })).finally(() => { settled = true; });
+    try {
+      await both; await observed;
+      expect(started).toContain(ids[0]); expect(started).toContain(ids[1]);
+      expect(started.every(id => ids.slice(0, 8).includes(id))).toBe(true);
+      expect(settled).toBe(false);
+      await expect(withWorkspaceSourceLocks(other, { companyId: f.companyId, projectIds: [f.projectId] }, async () => true, true))
+        .rejects.toBeInstanceOf(IsolatedRepositorySourceBusy);
+      expect(put).not.toHaveBeenCalled();
+    } finally { release(); await other.$client.end({ timeout: 1 }); }
+    expect(await pending).toMatchObject({ error: expect.objectContaining({ message: mode === "failure"
+      ? "extended_reader_failure" : "extended_reader_cancelled" }) });
+    expect(started.sort()).toEqual(ids.slice(0, 8));
+    expect(calls.get(ids[8])).toBe(1); expect(calls.get(ids[9])).toBe(1);
+    expect(await db.select().from(workspaceOperations)).toEqual(operations);
+    expect(await db.select().from(executionWorkspaceSnapshots)).toEqual(snapshots);
+    expect(put).not.toHaveBeenCalled();
+  }, 60_000);
+  it("refuses more than eight members across different canonical leaf roots", async () => {
+    const f = await fixture(), aliases = await aliasMembers(f, f.leaves[0], 8);
+    const workspaces = [f.leaves[1], ...aliases], noStorage = new Proxy(storage, { get() { throw new Error("storage_must_not_be_used"); } });
+    await expect(run(f, { storage: noStorage, workspaceIds: workspaces.map(row => row.id) }))
+      .rejects.toThrow("workspace_cohort_same_root_required");
+    expect(await db.select().from(workspaceOperations)).toHaveLength(0);
+    expect(await db.select().from(executionWorkspaceSnapshots)).toHaveLength(0);
+    await assertLocal(f);
+  });
  it("resumes the same uploaded cohort after an archived companion is independently migrated", async () => {
    const f = await fixture("cohort"), held = await recoverable(f), cwd = path.join(f.root, ".paperclip", "worktrees", "retired"), closedAt = new Date();
    await git(f.root, "worktree", "add", "-qb", "retired", cwd);
@@ -422,13 +422,12 @@ async function lockMigrationRecovery(lock: MigrationTransaction, plans: Migratio
    sources: online.flatMap(({ reserved, onlineSource }) => [...reserved.legacyRecoverySources, ...(onlineSource?.companions.flatMap(proof => proof.sources) ?? [])]),
    holds: online.flatMap(({ reserved, onlineSource }) => [...reserved.onlineRecoveryHolds, ...(onlineSource?.companions.flatMap(proof => proof.holds) ?? [])]) }))
    throw new Error("remote_workspace_migration_blocked:remote_copyback_owned");
-  const readers = await Promise.allSettled(online.map(async ({ input, onlineSource, reserved }) => {
+  await mapCohortMembers(online, async ({ input, onlineSource, reserved }) => {
    const current = await inspect(input.db, { ...input, onlineSource, operationId: reserved.operation!.id }, cohort);
    if (current.reasons.length || canonical(current.legacyRecoverySources) !== canonical(reserved.legacyRecoverySources)
      || canonical(current.onlineRecoveryHolds) !== canonical(reserved.onlineRecoveryHolds))
      throw new Error("remote_workspace_migration_recovery_source_changed");
-  }));
-  for (const reader of readers) if (reader.status === "rejected") throw reader.reason;
+  }, online[0].input.signal);
 }

 function createMigrationChildVerifier(input: MigrationInput, onlineSource: OnlineWorkspaceSource | undefined, sourceLock: MigrationTransaction, cohort = false) {
@@ -700,15 +699,29 @@ export async function migrateRemoteWorkspace(input: MigrationInput) {
 const cohortSchema = z.object({
  schema: z.literal("paperclip.remote-workspace-cohort.v1"), id: z.string().uuid(), companyId: z.string().uuid(),
  sourceProof: z.string().regex(/^[a-f0-9]{64}$/),
-  members: z.array(z.object({ workspaceId: z.string().uuid(), operationId: z.string().uuid() }).strict()).min(1).max(8),
+  members: z.array(z.object({ workspaceId: z.string().uuid(), operationId: z.string().uuid() }).strict()).min(1).max(32),
 }).strict();

+async function mapCohortMembers<T, R>(members: readonly T[], read: (member: T, index: number) => Promise<R>, signal?: AbortSignal): Promise<R[]> {
+  const results: R[] = [], cancellation = members.length > 8 ? signal : undefined;
+  for (let start = 0; start < members.length; start += 8) {
+    cancellation?.throwIfAborted();
+    const readers = await Promise.allSettled(members.slice(start, start + 8).map((member, index) => read(member, start + index)));
+    for (const reader of readers) {
+      if (reader.status === "rejected") throw reader.reason;
+      results.push(reader.value);
+    }
+    cancellation?.throwIfAborted();
+  }
+  return results;
+}
+
 export async function migrateRemoteWorkspaceCohort(input: {
  db: Db; storage: StorageService; companyId: string; workspaceIds: readonly string[];
  apply?: boolean; concurrency?: number; signal: AbortSignal; sourceLockWaitMs?: number;
 }) {
  const companyId = z.string().uuid().parse(input.companyId).toLowerCase();
-  const workspaceIds = z.array(z.string().uuid()).min(1).max(8).parse(input.workspaceIds).map(id => id.toLowerCase()).sort();
+  const workspaceIds = z.array(z.string().uuid()).min(1).max(32).parse(input.workspaceIds).map(id => id.toLowerCase()).sort();
  const concurrency = z.number().int().min(1).max(8).parse(input.concurrency ?? 2);
  if (new Set(workspaceIds).size !== workspaceIds.length) throw new Error("workspace_cohort_duplicate_member");
  input.signal.throwIfAborted();
@@ -737,20 +750,22 @@ export async function migrateRemoteWorkspaceCohort(input: {
        || cohort.members.find(member => member.workspaceId === snapshot.executionWorkspaceId)?.operationId !== operation.id
        || canonical(operation.metadata?.cohortMigration) !== canonical(cohort))) throw new Error("workspace_cohort_membership_changed");
    return { companyId, cohortId: cohort.id, applied: Boolean(input.apply), replayed: true,
-      members: await Promise.all(workspaceIds.map(workspaceId => inventoryRemoteWorkspaceMigration(input.db, { companyId, workspaceId }))) };
+      members: await mapCohortMembers(workspaceIds, workspaceId => inventoryRemoteWorkspaceMigration(input.db, { companyId, workspaceId }), input.signal) };
  }
  if (imported.some(Boolean)) throw new Error("workspace_cohort_membership_changed");
  const stop = new AbortController(), signal = AbortSignal.any([input.signal, stop.signal]);
-  const members = await Promise.all(workspaceIds.map(async workspaceId => {
+  const members = await mapCohortMembers(workspaceIds, async workspaceId => {
    const memberInput: MigrationInput = { db: input.db, storage: input.storage, companyId, workspaceId, apply: true, signal };
    const onlineSource = await resolveMigrationSource(memberInput);
    if (!onlineSource) throw new Error("workspace_cohort_online_source_required");
    return { input: memberInput, onlineSource };
-  }));
+  }, signal);
  const scopeIdentity = (scope: OnlineWorkspaceSource) => canonical({ kind: scope.kind,
    projectIds: scope.projectIds, issueIds: scope.issueIds, workspaceIds: scope.workspaceIds });
  if (members.some(member => scopeIdentity(member.onlineSource) !== scopeIdentity(members[0].onlineSource)))
    throw new Error("workspace_cohort_source_scope_mismatch");
+  if (members.length > 8 && members.some(member => member.onlineSource.root !== members[0].onlineSource.root))
+    throw new Error("workspace_cohort_same_root_required");
  for (const member of members) for (const other of members) {
    if (member !== other && member.onlineSource.root !== other.onlineSource.root
      && member.onlineSource.root.startsWith(other.onlineSource.root + path.sep)) throw new Error("workspace_cohort_leaf_members_required");
@@ -779,13 +794,12 @@ export async function migrateRemoteWorkspaceCohort(input: {
    || new Set(cohort.members.map(member => member.operationId)).size !== workspaceIds.length
    || pending.some(marker => canonical(marker) !== canonical(cohort))) throw new Error("workspace_cohort_membership_changed");
  const preflight = async (db: ReadDb) => {
-    const readers = await Promise.allSettled(members.map(async member => {
+    await mapCohortMembers(members, async member => {
      const current = await inspect(db, { ...member.input, onlineSource: member.onlineSource }, cohort);
      if (current.reasons.length) throw new Error(`remote_workspace_migration_blocked:${current.reasons.join(",")}`);
      if (fingerprint(current.workspace) !== fingerprint(initial.find(workspace => workspace.id === member.input.workspaceId)!))
        throw new Error("workspace_cohort_source_changed");
-    }));
-    for (const reader of readers) if (reader.status === "rejected") throw reader.reason;
+    }, signal);
  };
  if (!input.apply) { await preflight(input.db); return { companyId, cohortId: cohort.id, applied: false, eligible: true, workspaceIds }; }
  const publications: Parameters<typeof publishActivity>[0][] = [];
@@ -795,16 +809,12 @@ export async function migrateRemoteWorkspaceCohort(input: {
      if (!await tryAcquireExecutionWorkspaceLifecycleLock(lock, workspaceId)) throw new IsolatedRepositorySourceBusy("workspace_source_busy");
    }
    await preflight(input.db);
-    const reservations = await Promise.allSettled(members.map(async member => {
+    const plans: MigrationPlan[] = await mapCohortMembers(members, async member => {
      const reserved = await reserveMigration(member.input, member.onlineSource, cohort,
        fingerprint(initial.find(workspace => workspace.id === member.input.workspaceId)!));
      if (reserved.authority) throw new Error("workspace_cohort_membership_changed");
      return { ...member, reserved };
-    }));
-    const plans: MigrationPlan[] = reservations.map(reservation => {
-      if (reservation.status === "rejected") throw reservation.reason;
-      return reservation.value;
-    });
+    }, signal);
    await lockMigrationRecovery(lock, plans, cohort);
    if (refreshSourceProof) {
      const refreshedCohort = { ...cohort, sourceProof };
@@ -857,15 +867,14 @@ export async function migrateRemoteWorkspaceCohort(input: {
      for (const peer of peers) if (peer.status === "rejected") failure ??= peer.reason;
      if (failure) throw failure;
      signal.throwIfAborted();
-      const nestedReaders = await Promise.allSettled(plans.map(async (plan, index) => {
+      await mapCohortMembers(plans, async (plan, index) => {
        const captured = captures[index]!;
        if (!captured.nested.length) return;
        const scratch = path.join(captured.scratch, "verify-nested"); await fs.mkdir(scratch);
        const verified = await (childVerifier?.finalize ?? prepareNestedWorkspaceImports)({ ...plan.input, workspace: plan.reserved.workspace,
          scratch, recordedChildren: captured.nested, gitSourceRoots: plan.onlineSource?.gitSourceRoots });
        if (canonical(verified) !== canonical(captured.nested)) throw new Error("nested_workspace_source_changed");
-      }));
-      for (const reader of nestedReaders) if (reader.status === "rejected") throw reader.reason;
+      }, signal);
      await lock.execute(sql`select 1`);
      await lock.transaction(async tx => {
        await lockMigrationParents(tx, companyId, null, plans.flatMap(plan => [
@@ -891,9 +900,9 @@ export async function migrateRemoteWorkspaceCohort(input: {
    } catch (error) { failure = error; throw error; }
    finally {
      await childVerifier?.close();
-      const cleanup = await Promise.allSettled(captures.filter((capture): capture is CapturedMigration => Boolean(capture))
-        .map(capture => removeTaskDirectoryTree(capture.scratch)));
-      const errors = cleanup.flatMap(outcome => outcome.status === "rejected" ? [outcome.reason] : []);
+      const cleanup = await mapCohortMembers(captures.filter((capture): capture is CapturedMigration => Boolean(capture)),
+        capture => removeTaskDirectoryTree(capture.scratch).then(() => [], error => [error]));
+      const errors = cleanup.flat();
      if (errors.length) throw new AggregateError(failure ? [failure, ...errors] : errors, "workspace_cohort_cleanup_failed");
      if (!failure) signal.throwIfAborted();
    }