Files
PaperClipAI/tests/restart-chaos/api.ts
T
DottaandClaude Opus 5.5 1634f8b66b test(restart-chaos): run chaos waves against Daytona sandboxes without Docker (#19544)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) restarts a local
control plane on purpose while agent tasks run, and grades every task on
rules R1 to R6
> - The plan targets agent runs in Daytona sandboxes, but the harness
only supported `--env local`
> - A local controller needs two inputs that match the sandbox image
exactly: the image's Linux `paperclip-runnerd` and its provider pack.
The documented way to get them uses Docker, and Docker is not available
on the operator machine
> - The Daytona org that chaos uses also runs the live Cloud fleet, so
the harness must never change a sandbox it did not create through its
own Paperclip environment
> - This pull request adds `--env daytona`. It extracts both inputs from
the image without Docker, and refuses a controller whose runner source
differs from the image. It seeds a Daytona environment from the
runner-e2e fixtures and adds a read-only Daytona R6 sweep
> - The benefit is that chaos waves can now cross real Daytona sandboxes
with a local, killable control plane

## Linked Issues or Issue Description

**Feature description**
Run restart chaos waves with a local control plane and agent runs in
Daytona sandboxes (plan
`doc/plans/2026-11-06-control-plane-restart-chaos.md`, Phase 1b).

**Problem it solves**
`--env daytona` threw "not wired yet". The controller-side runnerd and
provider pack had to come from `docker cp`, which needs a Docker daemon.

**Proposed solution**
Use `skopeo copy` (no daemon) to fetch the linux/amd64 image into a
directory, walk the OCI layers from top to bottom, and extract only the
two paths. Before the wave starts, compare the image's content-ID label
with this checkout.

Refs #19477

## What Changed

- `tests/restart-chaos/daytona-image.ts`: a streaming tar reader and an
OCI layer walk (top to bottom, `.wh.<name>` and `.wh..wh..opq`
whiteouts, hard links, symlinks written last, no `..` paths). It
extracts `/usr/local/bin/paperclip-runnerd` and
`/opt/paperclip-runner/provider-pack/`. It checks the layer blob
digests, the ELF header (x86-64), the provider-pack manifest schema, and
its `runnerSourceRevision` against the image revision label. It records
digests in `receipt.json`, caches the result by image digest under
`results/daytona-images/`, and deletes the layer blobs. It also runs
standalone.
- Controller/image check: `assertControllerMatchesImage` compares the
image's `io.paperclip.runner.content-id` label with
`computeDaytonaImageContentId` for this checkout. On a mismatch it fails
and prints both IDs and the image's source revision.
- `seed.ts`: for `--env daytona`, it installs the bundled Daytona plugin
(`installFixtureDaytonaProvider`) and waits for provider readiness. It
creates the `daytona` environment fixture from `catalog.ts` with a
`DAYTONA_API_KEY` company secret and runs the native Codex
platform-qualification probe. Then it assigns the agents and the project
to that environment. `--daytona-cpu` sets the sandbox vCPU (default 4, 4
GiB, 10 GiB).
- `daytona-sweep.ts`: a read-only R6 for Daytona. The Daytona client is
a frozen wrapper that exposes only a label-filtered `list`, and no stop,
archive or delete exists. The sweep keeps only rows whose
`paperclip-company-id` and `paperclip-environment-id` both equal the
wave's own UUIDs, and it re-checks every row. It reconciles sandboxes
with the environment's leases:
  - A sandbox archived after its lease was released is expected.
- An orphaned sandbox or an `active`/`pending_cleanup` lease fails R6
for the owning task, or the wave.
- `assertHarnessMayMutate` refuses `paperclip-create-*` names and
`fleet-sbx-*` snapshots.
- Cleanup only deletes the wave's environment through Paperclip and
reports what is still running.
- `config.ts`: `--daytona-image` (or `PAPERCLIP_E2E_DAYTONA_IMAGE`)
needs an immutable digest. `--concurrency × --daytona-cpu` must stay
within `RESTART_CHAOS_DAYTONA_VCPU_BUDGET` (default 40). B5 is
local-only, because its oracle reads the live workspace.
- `main.ts`: Daytona preflight (credentials, image extraction,
content-ID check, Daytona plugin build as CI does). It passes
`PAPERCLIP_RUNNER_REMOTE_BINARY_PATH` and
`PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH` to every server start and
writes `sandbox-sweep.json`. Cleanup errors never skip stopping the
controller.
- README: a Daytona section (image matching, image-only CI dispatch,
vCPU budget, fleet safety, R6).
- Unit tests: synthetic tar layers for the tar reader, whiteouts, opaque
directories, overrides, hard links, symlinks and parent replacement.
Further tests cover the content-ID check, sweep reconciliation, exact
wave selection (unlabelled rows and other labels are never selected),
the read-only client, the fleet guard, and the CLI options.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
62/62 pass, credential-free.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean.
- Real extraction without Docker, image `…@sha256:b0c71f4d…`: 42 layers,
about 8 minutes, a 3 GB pack.
- The content-ID check refused the current checkout. The image is
`b8b16e15…` from `7e6d882558`; the checkout is `aec45589…`.
- A checkout of `7e6d882558` computes `b8b16e15…`, so the check compares
like with like.
- Current image:
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:a84579522192de0393ae24d09e67c4e752df4c701bd315ff54ad9bbca42e2dc0`
(source `5d25117ffa`, content `4e9a221f…`). It matches this branch, and
the receipt shows the content ID match.
- Paid waves, `--env daytona --adapters codex,claude --tasks 3
--concurrency 2 --faults K1 --seed 11`:
- Wave 1 (10 vCPU org, 4 vCPU sandboxes): 0/3. Every create returned
HTTP 400 "Total CPU limit exceeded. Maximum allowed: 10", reproduced
directly with the SDK. This led to the vCPU budget, and the
archived-sandbox fix in R6.
- Wave 2 (same org, 2 vCPU sandboxes): **3/3 PASS**, all
restart-crossing. Codex completed through one working Retry; both Claude
tasks continued automatically. R6: 4 sandboxes archived after release, 0
orphans, 0 held leases.
- Wave 3 (company org, 4 vCPU sandboxes): 0/3. Two product failures were
found and reported to the campaign ledger: an ACPX Claude recovery
blocked on `ambiguous_remote_recovery_evidence` with no Retry, and a
false budget hard stop at 10.75¢ of 3000¢ that paused the company. The
harness changed no sandbox. R6 reported 2 held leases, and the
environment delete returned 409 `pending_sandbox_cleanup` because of
them.
- Fleet-org key check: one sandbox with a unique non-`paperclip-create`
name and label was created, then deleted by the exact ID that create
returned.

## Risks

- This change touches the test harness only, with no product code.
- The extracted provider pack is about 3 GB per image in the gitignored
results directory. Delete old `results/daytona-images/<digest>`
directories by hand.
- The harness never changes Daytona sandboxes. If Paperclip's
environment cleanup is blocked, sandboxes stay until their 60-minute ttl
or a person acts. The harness reports them in `sandbox-sweep.json` and
fails the wave.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, GitHub CLI). Default context window,
extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 14:26:45 -06:00

41 lines
2.1 KiB
TypeScript

/** Minimal board API client for an isolated local_trusted instance. */
export function isSocketTypeOfServiceRace(error: unknown): boolean {
const candidate = error as { code?: unknown; syscall?: unknown } | null;
return candidate?.code === "EINVAL" && candidate.syscall === "setTypeOfService";
}
export class ChaosApiError extends Error {
constructor(readonly status: number | null, message: string) { super(message); this.name = "ChaosApiError"; }
}
export class ChaosApi {
constructor(readonly baseUrl: string, private readonly timeoutMs = 10_000) {}
private async request<T>(method: string, route: string, body?: unknown, timeoutMs = this.timeoutMs): Promise<T> {
let response: Response;
try {
response = await fetch(`${this.baseUrl}${route}`, {
method,
headers: { origin: this.baseUrl, ...(body === undefined ? {} : { "content-type": "application/json" }) },
body: body === undefined ? undefined : JSON.stringify(body),
signal: AbortSignal.timeout(timeoutMs),
cache: "no-store",
});
} catch (error) {
throw new ChaosApiError(null, `${method} ${route} unreachable: ${error instanceof Error ? error.message : String(error)}`);
}
const text = await response.text();
if (!response.ok) throw new ChaosApiError(response.status, `${method} ${route} returned ${response.status}: ${text.slice(0, 500)}`);
return (text ? JSON.parse(text) : null) as T;
}
get<T>(route: string, timeoutMs?: number) { return this.request<T>("GET", route, undefined, timeoutMs); }
post<T>(route: string, body?: unknown, timeoutMs?: number) { return this.request<T>("POST", route, body ?? {}, timeoutMs); }
put<T>(route: string, body?: unknown) { return this.request<T>("PUT", route, body ?? {}); }
patch<T>(route: string, body?: unknown) { return this.request<T>("PATCH", route, body ?? {}); }
async delete(route: string, options?: { allowNotFound?: boolean; timeout?: number }): Promise<void> {
try { await this.request<unknown>("DELETE", route, undefined, options?.timeout); }
catch (error) { if (!(options?.allowNotFound && error instanceof ChaosApiError && error.status === 404)) throw error; }
}
}