fix(restart-chaos): retry a failed image copy from an empty directory; share the image cache (#20021)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness tests recovery on Daytona sandboxes
> - The harness extracts the runner from the Daytona image with skopeo
before a wave starts
> - A skopeo blob read that timed out failed the whole run before any
test started
> - This pull request makes the image extraction retry cleanly and share
one cache
> - The benefit is that one network hiccup no longer costs a 20+ minute
run

## Linked Issues or Issue Description

Related: #19544 (Daytona chaos waves without Docker). No duplicate PR
was found.

**What happened?**

The Daytona wave `w-20261108T082021` and the crash-point run
`cp-20261108T101304` stopped before any test ran. skopeo reported `read:
operation timed out` while it copied a blob. The retry then reported
`not a containers image directory, don't want to overwrite`, because the
destination already had partial files. The crash-point suite also used
its own image cache, so it downloaded the image again on each run.

**Expected behavior**

A failed copy retries into an empty directory. Every wave and
crash-point run reuses an image that another run already extracted.

**Steps to reproduce**

1. Run `node scripts/crash-points.mjs --env daytona --daytona-image
<image>` on a slow or unstable network.
2. Let skopeo time out on one blob read.
3. See the harness stop with exit 2 before any point runs.

**Paperclip version or commit**

`master` at `8475826d2d`.

## What Changed

- With a shared cache, two runs can extract the same image at once. The
final cache directory is created only by an atomic rename, and no run
deletes a cache entry. A run that loses the rename adopts the valid
entry. An invalid leftover stays in place, and the run publishes its
extraction to a unique sibling directory.

- `copyImageToDir` runs up to 3 `skopeo copy` attempts. Each attempt
starts from an empty destination directory.
- `defaultDaytonaImageCacheRoot()` is now the one cache for both the
wave harness and the CLI: `tests/restart-chaos/results/daytona-images`
by default, or `RESTART_CHAOS_IMAGE_CACHE` if set. Crash-point runs
reuse an image that a wave already extracted.
- README updated.
- Three more tests: two runs against one cache through a barrier, a
corrupt leftover, and two runs that both meet a corrupt leftover.
- Unit tests use a fake skopeo that behaves like the real one: it
refuses a non-empty destination and leaves partial blobs behind when it
fails.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/daytona-image.test.ts`: 17/17 passed.
- `tsc -p tests/restart-chaos/tsconfig.json`: no errors in
`tests/restart-chaos`.
- Live run: the six Daytona crash points were rerun with this patch,
reusing the cached image.

## Risks

- Low. Test harness only; no product code changes.
- A shared cache means all runs depend on one receipt per image digest.
Receipts are already written atomically by renaming the staging dir, and
each one is keyed by the immutable digest.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

tests/restart-chaos/README.md

@@ -92,7 +92,9 @@ through Paperclip, by deleting the wave's own environment.
 top to bottom (honouring `.wh.` and `.wh..wh..opq` whiteouts), extracts only
 those two paths, checks the ELF header, the pack manifest and the image labels,
 records their digests in `receipt.json`, and deletes the layer blobs. The result
-is cached in `results/daytona-images/<digest>/`, so it runs once per image.
+is cached in `tests/restart-chaos/results/daytona-images/<digest>/` (or
+`$RESTART_CHAOS_IMAGE_CACHE`), shared by every wave and crash-point run, so it
+runs once per image. A copy that fails part-way retries from an empty directory.
 Run it alone with `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/daytona-image.ts <image>`.
 
 Before starting, the harness compares the image's `io.paperclip.runner.content-id`

tests/restart-chaos/daytona-image.test.ts

@@ -1,8 +1,9 @@
-import { mkdtemp, readFile, readlink, rm, stat } from "node:fs/promises";
+import { createHash } from "node:crypto";
+import { chmod, mkdir, mkdtemp, readFile, readlink, rm, stat, writeFile } from "node:fs/promises";
 import os from "node:os";
 import path from "node:path";
 import { afterEach, describe, expect, it } from "vitest";
-import { assertControllerMatchesImage, extractFromLayers, normalizeLayerPath, walkTar, type LayerSource } from "./daytona-image.js";
+import { assertControllerMatchesImage, copyImageToDir, extractFromLayers, prepareDaytonaImageArtifacts, normalizeLayerPath, walkTar, type LayerSource } from "./daytona-image.js";
 
 type Entry =
   | { path: string; type: "file"; content: string; mode?: number }
@@ -191,3 +192,117 @@ describe("assertControllerMatchesImage", () => {
     await expect(assertControllerMatchesImage(image, "/unused", "e".repeat(64))).rejects.toThrow(/does not match the Daytona image[\s\S]*d{40}[\s\S]*e{64}/);
   });
 });
+
+describe("copyImageToDir", () => {
+  async function fakeSkopeo(root: string, failures: number): Promise<string> {
+    const script = path.join(root, "skopeo");
+    await writeFile(script, `#!/bin/sh
+dest="\${9#dir:}"
+if [ -n "$(ls -A "$dest")" ]; then echo "not a containers image directory, don't want to overwrite" >&2; exit 1; fi
+n=$(cat "${root}/count" 2>/dev/null || echo 0); n=$((n+1)); echo $n > "${root}/count"
+if [ $n -le ${failures} ]; then echo partial > "$dest/blob"; echo "read: operation timed out" >&2; exit 1; fi
+echo '{}' > "$dest/manifest.json"
+`);
+    await chmod(script, 0o755);
+    return script;
+  }
+
+  it("retries a copy that failed part-way from an empty directory", async () => {
+    const root = await mkdtemp(path.join(os.tmpdir(), "daytona-skopeo-"));
+    try {
+      const blobs = path.join(root, "blobs");
+      await copyImageToDir({ skopeo: await fakeSkopeo(root, 2), image: "ghcr.io/x/y@sha256:" + "a".repeat(64), blobs });
+      expect(await readFile(path.join(root, "count"), "utf8")).toBe("3\n");
+      expect(await readFile(path.join(blobs, "manifest.json"), "utf8")).toBe("{}\n");
+      await expect(stat(path.join(blobs, "blob"))).rejects.toThrow();
+    } finally { await rm(root, { recursive: true, force: true }); }
+  });
+
+  it("gives up after the last attempt with skopeo's error", async () => {
+    const root = await mkdtemp(path.join(os.tmpdir(), "daytona-skopeo-"));
+    try {
+      await expect(copyImageToDir({ skopeo: await fakeSkopeo(root, 5), image: "i", blobs: path.join(root, "blobs"), attempts: 2 }))
+        .rejects.toThrow(/operation timed out/);
+      expect(await readFile(path.join(root, "count"), "utf8")).toBe("2\n");
+    } finally { await rm(root, { recursive: true, force: true }); }
+  });
+});
+
+describe("prepareDaytonaImageArtifacts with a shared cache", () => {
+  const sha = (bytes: Buffer | string) => createHash("sha256").update(bytes).digest("hex");
+  const revision = "a".repeat(40);
+
+  async function fakeImage(root: string): Promise<string> {
+    const source = path.join(root, "image");
+    await mkdir(source, { recursive: true });
+    const elf = Buffer.alloc(64); elf.writeUInt32BE(0x7f454c46, 0); elf[4] = 2; elf[5] = 1; elf.writeUInt16LE(62, 18);
+    const pack = JSON.stringify({ schema: "paperclip-runner/remote-provider-pack/v1", digest: `sha256:${"b".repeat(64)}`,
+      payload: { runnerSourceRevision: revision } });
+    const layerBytes = tar([
+      { path: RUNNERD, type: "file", content: elf.toString("latin1"), mode: 0o755 },
+      { path: `${PACK}/provider-pack.json`, type: "file", content: pack },
+    ]);
+    const config = Buffer.from(JSON.stringify({ os: "linux", architecture: "amd64",
+      config: { Labels: { "io.paperclip.runner.content-id": "c".repeat(64), "org.opencontainers.image.revision": revision } } }));
+    await writeFile(path.join(source, sha(layerBytes)), layerBytes);
+    await writeFile(path.join(source, sha(config)), config);
+    await writeFile(path.join(source, "manifest.json"), JSON.stringify({ config: { digest: `sha256:${sha(config)}` },
+      layers: [{ digest: `sha256:${sha(layerBytes)}`, mediaType: "application/vnd.oci.image.layer.v1.tar" }] }));
+    const barrier = path.join(root, "barrier");
+    await mkdir(barrier);
+    const script = path.join(root, "skopeo");
+    await writeFile(script, `#!/bin/sh
+dest="\${9#dir:}"
+touch "${barrier}/$$"
+while [ "$(ls "${barrier}" | wc -l)" -lt 2 ]; do sleep 0.05; done
+cp -R "${source}/." "$dest/"
+`);
+    await chmod(script, 0o755);
+    return script;
+  }
+
+  it("lets two runs extract the same image at once; both get a usable receipt", async () => {
+    const root = await mkdtemp(path.join(os.tmpdir(), "daytona-cache-"));
+    directories.push(root);
+    const skopeo = await fakeImage(root);
+    const image = `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:${"d".repeat(64)}`;
+    const cacheRoot = path.join(root, "cache");
+    const [first, second] = await Promise.all([
+      prepareDaytonaImageArtifacts({ image, cacheRoot, skopeo }),
+      prepareDaytonaImageArtifacts({ image, cacheRoot, skopeo }),
+    ]);
+    expect(second.runnerdPath).toBe(first.runnerdPath);
+    for (const receipt of [first, second]) {
+      expect((await stat(receipt.runnerdPath)).isFile()).toBe(true);
+      expect(await readFile(path.join(receipt.providerPackPath, "provider-pack.json"), "utf8")).toContain(revision);
+    }
+  });
+
+  it("leaves a corrupt leftover in place and publishes beside it", async () => {
+    const root = await mkdtemp(path.join(os.tmpdir(), "daytona-cache-"));
+    directories.push(root);
+    const skopeo = await fakeImage(root);
+    await writeFile(path.join(root, "barrier", "other"), "");
+    const image = `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:${"e".repeat(64)}`;
+    const cacheRoot = path.join(root, "cache");
+    await mkdir(path.join(cacheRoot, "e".repeat(64)), { recursive: true });
+    await writeFile(path.join(cacheRoot, "e".repeat(64), "junk"), "partial");
+    const receipt = await prepareDaytonaImageArtifacts({ image, cacheRoot, skopeo });
+    expect((await stat(receipt.runnerdPath)).isFile()).toBe(true);
+    expect(receipt.runnerdPath.startsWith(path.join(cacheRoot, `${"e".repeat(64)}.`))).toBe(true);
+    expect(await exists(cacheRoot, `${"e".repeat(64)}/junk`)).toBe(true);
+  });
+
+  it("two runs replacing a corrupt leftover at once both keep a usable cache", async () => {
+    const root = await mkdtemp(path.join(os.tmpdir(), "daytona-cache-"));
+    directories.push(root);
+    const skopeo = await fakeImage(root);
+    const image = `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:${"f".repeat(64)}`;
+    const cacheRoot = path.join(root, "cache");
+    await mkdir(path.join(cacheRoot, "f".repeat(64)), { recursive: true });
+    await writeFile(path.join(cacheRoot, "f".repeat(64), "junk"), "partial");
+    const receipts = await Promise.all([1, 2].map(() => prepareDaytonaImageArtifacts({ image, cacheRoot, skopeo })));
+    for (const receipt of receipts) expect((await stat(receipt.runnerdPath)).isFile()).toBe(true);
+    expect(await exists(cacheRoot, `${"f".repeat(64)}/junk`)).toBe(true);
+  });
+});

tests/restart-chaos/daytona-image.ts

@@ -327,6 +327,29 @@ async function readReceipt(directory: string): Promise<DaytonaImageArtifacts | n
   return receipt;
 }
 
+export function defaultDaytonaImageCacheRoot(): string {
+  return path.resolve(process.env.RESTART_CHAOS_IMAGE_CACHE ?? path.join(import.meta.dirname, "results", "daytona-images"));
+}
+
+export async function copyImageToDir(input: { skopeo: string; image: string; blobs: string; attempts?: number;
+  log?: (line: string) => void }): Promise<void> {
+  const attempts = input.attempts ?? 3;
+  const log = input.log ?? (() => undefined);
+  for (let attempt = 1; ; attempt += 1) {
+    await rm(input.blobs, { recursive: true, force: true });
+    await mkdir(input.blobs, { recursive: true, mode: 0o700 });
+    log(`skopeo copy ${input.image} (linux/amd64) → ${input.blobs} (attempt ${attempt}/${attempts})`);
+    try {
+      await execFileAsync(input.skopeo, ["copy", "--retry-times", "3", "--override-os", "linux", "--override-arch", "amd64",
+        `docker://${input.image}`, `dir:${input.blobs}`], { maxBuffer: 16 * 1024 * 1024 });
+      return;
+    } catch (error) {
+      if (attempt >= attempts) throw error;
+      log(`skopeo copy failed (${error instanceof Error ? error.message.split("\n")[0] : String(error)}); retrying into an empty directory`);
+    }
+  }
+}
+
 /**
  * Returns the verified runnerd and provider pack of `image`, extracting them
  * once per image digest into `cacheRoot/<digest>/`.
@@ -338,17 +361,12 @@ export async function prepareDaytonaImageArtifacts(input: { image: string; cache
   const finalDir = path.join(path.resolve(input.cacheRoot), match[1]!);
   const cached = await readReceipt(finalDir);
   if (cached) { log(`Daytona image artifacts cached at ${finalDir}`); return cached; }
-  await rm(finalDir, { recursive: true, force: true });
 
   const staging = path.join(path.resolve(input.cacheRoot), `.staging-${match[1]!.slice(0, 12)}-${randomBytes(4).toString("hex")}`);
   const blobs = path.join(staging, "blobs");
   const rootfs = path.join(staging, "rootfs");
   try {
-    await mkdir(blobs, { recursive: true, mode: 0o700 });
-    log(`skopeo copy ${input.image} (linux/amd64) → ${blobs}`);
-    await execFileAsync(input.skopeo ?? process.env.SKOPEO ?? "skopeo", ["copy", "--retry-times", "3",
-      "--override-os", "linux", "--override-arch", "amd64", `docker://${input.image}`, `dir:${blobs}`],
-    { maxBuffer: 16 * 1024 * 1024 });
+    await copyImageToDir({ skopeo: input.skopeo ?? process.env.SKOPEO ?? "skopeo", image: input.image, blobs, log });
     const manifestBytes = await readFile(path.join(blobs, "manifest.json"));
     const manifest = JSON.parse(manifestBytes.toString("utf8")) as { config: { digest: string }; layers: Array<{ digest: string; mediaType: string }> };
     if (!manifest.config?.digest || !Array.isArray(manifest.layers)) throw new Error("skopeo did not produce a single-platform image manifest");
@@ -390,11 +408,27 @@ export async function prepareDaytonaImageArtifacts(input: { image: string; cache
       providerPackPath: path.join(root, "rootfs", ...PROVIDER_PACK_PATH.split("/")), providerPackManifestSha256: sha256(packBytes),
       providerPackDigest: pack.digest!, providerPackRunnerSourceRevision: sourceRevision, extractedAt: new Date().toISOString(),
     });
-    const receipt = { ...receiptFor(finalDir), runnerdSha256: await fileSha256(runnerd) };
-    await writeFile(path.join(staging, "receipt.json"), `${JSON.stringify(receipt, null, 2)}\n`, { mode: 0o600 });
-    await rename(staging, finalDir);
-    log(`Daytona image artifacts ready at ${finalDir}`);
-    return receipt;
+    const runnerdSha256 = await fileSha256(runnerd);
+    const publishTo = async (root: string) => {
+      const receipt = { ...receiptFor(root), runnerdSha256 };
+      await writeFile(path.join(staging, "receipt.json"), `${JSON.stringify(receipt, null, 2)}\n`, { mode: 0o600 });
+      await rename(staging, root);
+      log(`Daytona image artifacts ready at ${root}`);
+      return receipt;
+    };
+    try {
+      return await publishTo(finalDir);
+    } catch (error) {
+      const concurrent = await readReceipt(finalDir);
+      if (concurrent) {
+        await rm(staging, { recursive: true, force: true });
+        log(`Daytona image artifacts extracted concurrently at ${finalDir}`);
+        return concurrent;
+      }
+      if (!(await stat(finalDir).then(() => true, () => false))) throw error;
+      log(`Invalid cache entry at ${finalDir} left in place; publishing this extraction beside it`);
+      return await publishTo(`${finalDir}.${randomBytes(4).toString("hex")}`);
+    }
   } catch (error) {
     await rm(staging, { recursive: true, force: true }).catch(() => undefined);
     throw error;
@@ -428,7 +462,7 @@ if (invoked && import.meta.url === pathToFileURL(invoked).href) {
   const image = process.argv[2];
   if (!image) { console.error("Usage: daytona-image.ts <registry/repository@sha256:digest>"); process.exit(2); }
   const repositoryRoot = path.resolve(import.meta.dirname, "../..");
-  const cacheRoot = process.env.RESTART_CHAOS_IMAGE_CACHE ?? path.join(import.meta.dirname, "results", "daytona-images");
+  const cacheRoot = defaultDaytonaImageCacheRoot();
   try {
     const artifacts = await prepareDaytonaImageArtifacts({ image, cacheRoot, log: line => console.error(line) });
     const check = await assertControllerMatchesImage(artifacts, repositoryRoot).then(() => "match", (error: Error) => error.message);

tests/restart-chaos/main.ts

@@ -14,7 +14,7 @@ import path from "node:path";
 import { setTimeout as delay } from "node:timers/promises";
 import { ChaosApi, isSocketTypeOfServiceRace } from "./api.js";
 import { CREDENTIAL_FOR_ADAPTER, parseCommand, replayCommand, waveIdFor } from "./config.js";
-import { assertControllerMatchesImage, prepareDaytonaImageArtifacts } from "./daytona-image.js";
+import { assertControllerMatchesImage, defaultDaytonaImageCacheRoot, prepareDaytonaImageArtifacts } from "./daytona-image.js";
 import { PLUGIN_BUILD_STAMP, pluginBuildState } from "./plugin-build.js";
 import { DaytonaWaveSweeper, leaksForTask, unattributedLeaks, type DaytonaSweepResult } from "./daytona-sweep.js";
 import { knownSecretValues, redactKnownSecrets, writeEvidenceBundle } from "./evidence.js";
@@ -91,7 +91,7 @@ function ensureDaytonaPlugin(timeline: Timeline) {
  */
 async function prepareDaytona(plan: WavePlan, timeline: Timeline): Promise<Record<string, string>> {
   const image = plan.options.daytonaImage!;
-  const artifacts = await prepareDaytonaImageArtifacts({ image, cacheRoot: path.join(resultsRoot, "daytona-images"),
+  const artifacts = await prepareDaytonaImageArtifacts({ image, cacheRoot: defaultDaytonaImageCacheRoot(),
     log: line => console.info(`[daytona-image] ${line}`) });
   const { localContentId } = await assertControllerMatchesImage(artifacts, repositoryRoot);
   timeline.add("daytona_image_matched", { image, contentId: artifacts.contentId, localContentId, sourceRevision: artifacts.sourceRevision,