mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) restarts a local control plane on purpose while agent tasks run, and grades every task on rules R1 to R6 > - The plan targets agent runs in Daytona sandboxes, but the harness only supported `--env local` > - A local controller needs two inputs that match the sandbox image exactly: the image's Linux `paperclip-runnerd` and its provider pack. The documented way to get them uses Docker, and Docker is not available on the operator machine > - The Daytona org that chaos uses also runs the live Cloud fleet, so the harness must never change a sandbox it did not create through its own Paperclip environment > - This pull request adds `--env daytona`. It extracts both inputs from the image without Docker, and refuses a controller whose runner source differs from the image. It seeds a Daytona environment from the runner-e2e fixtures and adds a read-only Daytona R6 sweep > - The benefit is that chaos waves can now cross real Daytona sandboxes with a local, killable control plane ## Linked Issues or Issue Description **Feature description** Run restart chaos waves with a local control plane and agent runs in Daytona sandboxes (plan `doc/plans/2026-11-06-control-plane-restart-chaos.md`, Phase 1b). **Problem it solves** `--env daytona` threw "not wired yet". The controller-side runnerd and provider pack had to come from `docker cp`, which needs a Docker daemon. **Proposed solution** Use `skopeo copy` (no daemon) to fetch the linux/amd64 image into a directory, walk the OCI layers from top to bottom, and extract only the two paths. Before the wave starts, compare the image's content-ID label with this checkout. Refs #19477 ## What Changed - `tests/restart-chaos/daytona-image.ts`: a streaming tar reader and an OCI layer walk (top to bottom, `.wh.<name>` and `.wh..wh..opq` whiteouts, hard links, symlinks written last, no `..` paths). It extracts `/usr/local/bin/paperclip-runnerd` and `/opt/paperclip-runner/provider-pack/`. It checks the layer blob digests, the ELF header (x86-64), the provider-pack manifest schema, and its `runnerSourceRevision` against the image revision label. It records digests in `receipt.json`, caches the result by image digest under `results/daytona-images/`, and deletes the layer blobs. It also runs standalone. - Controller/image check: `assertControllerMatchesImage` compares the image's `io.paperclip.runner.content-id` label with `computeDaytonaImageContentId` for this checkout. On a mismatch it fails and prints both IDs and the image's source revision. - `seed.ts`: for `--env daytona`, it installs the bundled Daytona plugin (`installFixtureDaytonaProvider`) and waits for provider readiness. It creates the `daytona` environment fixture from `catalog.ts` with a `DAYTONA_API_KEY` company secret and runs the native Codex platform-qualification probe. Then it assigns the agents and the project to that environment. `--daytona-cpu` sets the sandbox vCPU (default 4, 4 GiB, 10 GiB). - `daytona-sweep.ts`: a read-only R6 for Daytona. The Daytona client is a frozen wrapper that exposes only a label-filtered `list`, and no stop, archive or delete exists. The sweep keeps only rows whose `paperclip-company-id` and `paperclip-environment-id` both equal the wave's own UUIDs, and it re-checks every row. It reconciles sandboxes with the environment's leases: - A sandbox archived after its lease was released is expected. - An orphaned sandbox or an `active`/`pending_cleanup` lease fails R6 for the owning task, or the wave. - `assertHarnessMayMutate` refuses `paperclip-create-*` names and `fleet-sbx-*` snapshots. - Cleanup only deletes the wave's environment through Paperclip and reports what is still running. - `config.ts`: `--daytona-image` (or `PAPERCLIP_E2E_DAYTONA_IMAGE`) needs an immutable digest. `--concurrency × --daytona-cpu` must stay within `RESTART_CHAOS_DAYTONA_VCPU_BUDGET` (default 40). B5 is local-only, because its oracle reads the live workspace. - `main.ts`: Daytona preflight (credentials, image extraction, content-ID check, Daytona plugin build as CI does). It passes `PAPERCLIP_RUNNER_REMOTE_BINARY_PATH` and `PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH` to every server start and writes `sandbox-sweep.json`. Cleanup errors never skip stopping the controller. - README: a Daytona section (image matching, image-only CI dispatch, vCPU budget, fleet safety, R6). - Unit tests: synthetic tar layers for the tar reader, whiteouts, opaque directories, overrides, hard links, symlinks and parent replacement. Further tests cover the content-ID check, sweep reconciliation, exact wave selection (unlabelled rows and other labels are never selected), the read-only client, the fleet guard, and the CLI options. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 62/62 pass, credential-free. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. - Real extraction without Docker, image `…@sha256:b0c71f4d…`: 42 layers, about 8 minutes, a 3 GB pack. - The content-ID check refused the current checkout. The image is `b8b16e15…` from `7e6d882558`; the checkout is `aec45589…`. - A checkout of `7e6d882558` computes `b8b16e15…`, so the check compares like with like. - Current image: `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:a84579522192de0393ae24d09e67c4e752df4c701bd315ff54ad9bbca42e2dc0` (source `5d25117ffa`, content `4e9a221f…`). It matches this branch, and the receipt shows the content ID match. - Paid waves, `--env daytona --adapters codex,claude --tasks 3 --concurrency 2 --faults K1 --seed 11`: - Wave 1 (10 vCPU org, 4 vCPU sandboxes): 0/3. Every create returned HTTP 400 "Total CPU limit exceeded. Maximum allowed: 10", reproduced directly with the SDK. This led to the vCPU budget, and the archived-sandbox fix in R6. - Wave 2 (same org, 2 vCPU sandboxes): **3/3 PASS**, all restart-crossing. Codex completed through one working Retry; both Claude tasks continued automatically. R6: 4 sandboxes archived after release, 0 orphans, 0 held leases. - Wave 3 (company org, 4 vCPU sandboxes): 0/3. Two product failures were found and reported to the campaign ledger: an ACPX Claude recovery blocked on `ambiguous_remote_recovery_evidence` with no Retry, and a false budget hard stop at 10.75¢ of 3000¢ that paused the company. The harness changed no sandbox. R6 reported 2 held leases, and the environment delete returned 409 `pending_sandbox_cleanup` because of them. - Fleet-org key check: one sandbox with a unique non-`paperclip-create` name and label was created, then deleted by the exact ID that create returned. ## Risks - This change touches the test harness only, with no product code. - The extracted provider pack is about 3 GB per image in the gitignored results directory. Delete old `results/daytona-images/<digest>` directories by hand. - The harness never changes Daytona sandboxes. If Paperclip's environment cleanup is blocked, sandboxes stay until their 60-minute ttl or a person acts. The harness reports them in `sandbox-sweep.json` and fails the wave. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, GitHub CLI). Default context window, extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
41 lines
2.1 KiB
TypeScript
41 lines
2.1 KiB
TypeScript
/** Minimal board API client for an isolated local_trusted instance. */
|
|
export function isSocketTypeOfServiceRace(error: unknown): boolean {
|
|
const candidate = error as { code?: unknown; syscall?: unknown } | null;
|
|
return candidate?.code === "EINVAL" && candidate.syscall === "setTypeOfService";
|
|
}
|
|
|
|
export class ChaosApiError extends Error {
|
|
constructor(readonly status: number | null, message: string) { super(message); this.name = "ChaosApiError"; }
|
|
}
|
|
|
|
export class ChaosApi {
|
|
constructor(readonly baseUrl: string, private readonly timeoutMs = 10_000) {}
|
|
|
|
private async request<T>(method: string, route: string, body?: unknown, timeoutMs = this.timeoutMs): Promise<T> {
|
|
let response: Response;
|
|
try {
|
|
response = await fetch(`${this.baseUrl}${route}`, {
|
|
method,
|
|
headers: { origin: this.baseUrl, ...(body === undefined ? {} : { "content-type": "application/json" }) },
|
|
body: body === undefined ? undefined : JSON.stringify(body),
|
|
signal: AbortSignal.timeout(timeoutMs),
|
|
cache: "no-store",
|
|
});
|
|
} catch (error) {
|
|
throw new ChaosApiError(null, `${method} ${route} unreachable: ${error instanceof Error ? error.message : String(error)}`);
|
|
}
|
|
const text = await response.text();
|
|
if (!response.ok) throw new ChaosApiError(response.status, `${method} ${route} returned ${response.status}: ${text.slice(0, 500)}`);
|
|
return (text ? JSON.parse(text) : null) as T;
|
|
}
|
|
|
|
get<T>(route: string, timeoutMs?: number) { return this.request<T>("GET", route, undefined, timeoutMs); }
|
|
post<T>(route: string, body?: unknown, timeoutMs?: number) { return this.request<T>("POST", route, body ?? {}, timeoutMs); }
|
|
put<T>(route: string, body?: unknown) { return this.request<T>("PUT", route, body ?? {}); }
|
|
patch<T>(route: string, body?: unknown) { return this.request<T>("PATCH", route, body ?? {}); }
|
|
async delete(route: string, options?: { allowNotFound?: boolean; timeout?: number }): Promise<void> {
|
|
try { await this.request<unknown>("DELETE", route, undefined, options?.timeout); }
|
|
catch (error) { if (!(options?.allowNotFound && error instanceof ChaosApiError && error.status === 404)) throw error; }
|
|
}
|
|
}
|