mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart-chaos harness (`tests/restart-chaos/`) restarts or stalls the control plane while real agent tasks run. > - The staging census (#20679, `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`) found two open classes that the harness could not create. In class R, a deploy lands mid-turn, the server stays down for 4–7 minutes, and recovery then finds the remote Runner gone (`runner_remote_process_identity_unavailable`). In class L, the server stalls for about 100 s without a restart, and legacy runs that are still `preparing` end up waiting for a manual Retry. > - K15 restarted at once. No Daytona workload kept a tool running across the fault. TM1 was local-only with a fixed range. Nothing recorded what the remote Runner did while the controller was away. > - This pull request adds a K15 outage hold, a mid-tool workload, sandbox-side Runner evidence, and TM1 stalls on Daytona with a configurable hold. > - The benefit: these staging failure shapes can now be reproduced, and each wave records the Runner's own state. ## Linked Issues or Issue Description Refs #20679 **What happened?** On staging, two failure shapes had no matching chaos scenario: - Class R: a deploy landed mid-turn and the server stayed down for 4–7 minutes. Recovery then found the remote Runner gone. Nothing captured why the Runner exited. - Class L: a 100 s server stall without a restart lapsed every controller lease. Legacy `preparing` runs then waited for a manual Retry. **What did you expect?** The harness should create a long mid-turn deploy outage and a stall without a restart on Daytona. It should also save the sandbox-side Runner evidence. **Steps to reproduce** 1. Use the census report and the staging run records listed in it. 2. Compare them with the harness options before this change: K15 has no hold, TM1 is local-only, and there is no Daytona tool workload. ## What Changed - `--k15-hold-seconds <n|min-max>` (0–600). A K15 stops the server with SIGTERM, keeps it down for the seeded hold, then starts the next deploy generation. The range is drawn only for K15 cells, so existing seeds keep their plans. Storms use it too. - New opt-in workload `T1`. The agent runs one shell loop in the foreground for about 100 s, then saves `TICKS-<nonce>=50`. Its fault window, 35–80 s after launch, lands mid-tool. - New `daytona-runner-logs.ts`, a read-only probe of each wave sandbox through the Daytona toolbox. It runs only on sandboxes with the wave's labels, and only on `started` ones. It records every process, memory, disk, `dmesg` OOM lines, and the tails of `runnerd.stderr.log`, `runner-process.identity` and `runner-state.json`. It redacts known secrets. It runs right after a K15 stop, just before the next deploy starts, and at wave end. Output goes to `results/<wave>/daytona-runner/` and to `runner_logs_captured` in the timeline. The probe never stops, starts or deletes a sandbox, and a failed capture is recorded instead of thrown. - TM1 now runs on `--env daytona`, and `--tm1-hold-seconds <n|min-max>` (10–600) replaces its default 95–150 s range with the same single draw. TM2 and TM3 stay local. - Tests: `k15-long-hold.test.ts` and `tm1-stall.test.ts`. The README is updated. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 45 files and 531 tests pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Daytona waves, all with T1 and K15: | Wave | Revision and image | Hold | Result | | --- | --- | --- | --- | | `w-20261111T015441-s9310` | master8f6ca25976, image `bc7ca92d…` | 267–408 s, OpenCode | 4 of 4 pass | | `w-20261111T022637-s9311` | master8f6ca25976, image `bc7ca92d…` | 300 s, OpenCode and Claude | 4 of 4 pass. Every stop capture shows the T1 loop running. | | `w-20261111T025307-s9320` | mastera1ec05a1a4, image `09321542…` | 460–482 s, past the Runner's 7-minute reconnect grace | 4 of 4 pass | - In every pre-restart capture, the same `runnerd` PID was alive for the whole outage (etimes 491–549 s in the last wave). Every sandbox was `started`. There was no `transport_reconnect_grace_exceeded` and no OOM. - So the class R Runner loss does not reproduce on current master with the e2e image. The census document lists the remaining differences from staging. - TM1 Daytona waves on mastera1ec05a1a4found a **new product gap** (RED; the harness only reports it): - `w-20261111T034105-s9330` (TM1 at `--fault-after-seconds 10`): 2 of 3 pass. The failing OpenCode run was stalled at `native.process_start_requested`. Afterwards it waited forever with `controller_still_alive` / `restartKind: hard`, where the controller it waited for was its own live server. - `w-20261111T040439-s9331` (TM1 mid-tool, T1): 0 of 3 pass. After the stall the run had no further events and stayed `running` for more than 20 minutes, with no Retry. - Review fixes in3540bfe700: matrix cells take both holds, T1's `fault_crossed_tool` check, and one 45 s deadline per Runner capture. The suite is now 45 files and 531 tests. ## Risks - Low risk. This change is test harness and docs only, with no product code. - The Runner probe executes one fixed read-only command in this wave's own sandboxes through the Daytona toolbox. It never targets fleet sandboxes, because it filters on the wave's company and environment labels. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edit, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
123 lines
6.5 KiB
TypeScript
123 lines
6.5 KiB
TypeScript
/**
|
||
* K15 (redeploy): what a hosted deploy does to the control plane, beyond K2's graceful stop.
|
||
*
|
||
* On the hosted staging stack every deploy replaces the server's container. The replacement runs
|
||
* in another process space (host name, kernel boot and PID namespace), so the server's PID-based
|
||
* "the previous owner has exited" proofs (`server/src/services/controller-process-identity.ts`,
|
||
* `owner-proof.ts`) never apply. It also carries a new Railway deployment id on the same exclusive
|
||
* volume, which is the proof hosted recovery uses instead. A K1/K2 restart on one machine keeps the
|
||
* process space, so every PID proof succeeds there and this path is never exercised.
|
||
*
|
||
* K15 emulates both changes without a container: each server of a K15 wave sees an emulated host
|
||
* name (a `--import` preload replaces `os.hostname`, which changes the process-space hash) and
|
||
* Railway-shaped deployment variables naming one stable project, environment, service and volume
|
||
* and a deployment id per generation. A K15 stops the server with SIGTERM like K2 and starts the
|
||
* replacement as the next generation. Other faults in the same wave restart within a generation,
|
||
* like a platform restart of the same deployment.
|
||
*/
|
||
import { createHash } from "node:crypto";
|
||
import { fileURLToPath } from "node:url";
|
||
import type { WavePlan } from "./types.js";
|
||
|
||
/** Whether any server of this wave can be redeployed: a planned, storm or replayed K15. */
|
||
export function redeployWave(plan: Pick<WavePlan, "options" | "tasks" | "storm">): boolean {
|
||
return plan.options.faults.includes("K15") || (plan.storm?.faults.some(fault => fault.kind === "K15") ?? false)
|
||
|| plan.tasks.some(task => task.fault === "K15" || (task.replayFaults?.some(fault => fault.kind === "K15") ?? false));
|
||
}
|
||
|
||
/** `--k15-hold-seconds` upper bound: the controller-outage cap (tests/runner-e2e/controller-outage.ts). */
|
||
export const K15_MAX_HOLD_SECONDS = 600;
|
||
|
||
/**
|
||
* `--k15-hold-seconds <n>` or `<min>-<max>` (0–600 s): how long a K15 keeps the server down between its SIGTERM
|
||
* and the next deploy's start. Staging deploys were down 4–7 minutes, so the remote Runner has to survive the
|
||
* gap mid-turn. Without it a K15 restarts at once.
|
||
*/
|
||
export function parseK15HoldSeconds(options: { faults: readonly string[] }, raw: string | undefined): { k15HoldMs?: [number, number] } {
|
||
if (raw === undefined) return {};
|
||
if (!options.faults.includes("K15")) throw new Error("--k15-hold-seconds sets the K15 outage; add K15 to --faults");
|
||
const match = /^([0-9]+)(?:-([0-9]+))?$/.exec(raw.trim());
|
||
if (!match) throw new Error("--k15-hold-seconds takes <seconds> or <min>-<max> seconds");
|
||
const min = Number(match[1]);
|
||
const max = match[2] === undefined ? min : Number(match[2]);
|
||
if (max > K15_MAX_HOLD_SECONDS || max < min) {
|
||
throw new Error(`--k15-hold-seconds must be between 0 and ${K15_MAX_HOLD_SECONDS} seconds, min not above max`);
|
||
}
|
||
return { k15HoldMs: [min * 1000, max * 1000] };
|
||
}
|
||
|
||
/** The seeded hold of one K15. Draws only when a range is set, so existing seeds keep their plans. */
|
||
export function k15HoldMs(range: readonly [number, number] | undefined, int: (min: number, max: number) => number): number {
|
||
return range ? int(range[0], range[1]) : 0;
|
||
}
|
||
|
||
/** The preload that applies the emulated host name before the server loads. */
|
||
export const REDEPLOY_PRELOAD_PATH = fileURLToPath(new URL("./redeploy-preload.mjs", import.meta.url));
|
||
/** Read by the preload: the host name `os.hostname()` returns in the server process. */
|
||
export const REDEPLOY_HOSTNAME_ENV = "RESTART_CHAOS_REDEPLOY_HOSTNAME";
|
||
|
||
export interface RedeployIdentity {
|
||
/** 0 for the wave's first server; each K15 adds one. */
|
||
generation: number;
|
||
hostname: string;
|
||
projectId: string;
|
||
environmentId: string;
|
||
serviceId: string;
|
||
deploymentId: string;
|
||
volumeId: string;
|
||
}
|
||
|
||
/** A deterministic UUID-shaped id for one wave, role and generation (no randomness: replays match). */
|
||
function uuidFor(waveId: string, role: string, generation = 0): string {
|
||
const hex = createHash("sha256").update(`restart-chaos-redeploy:${waveId}:${role}:${generation}`).digest("hex");
|
||
// Version 5-style layout so validators that require a UUID shape accept it.
|
||
const variant = ((parseInt(hex[16]!, 16) & 0x3) | 0x8).toString(16);
|
||
return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-5${hex.slice(13, 16)}-${variant}${hex.slice(17, 20)}-${hex.slice(20, 32)}`;
|
||
}
|
||
|
||
/** The identity of the server that serves generation `generation` of this wave. */
|
||
export function redeployIdentity(waveId: string, generation: number): RedeployIdentity {
|
||
if (!Number.isInteger(generation) || generation < 0) throw new Error(`Redeploy generation must be a non-negative integer, got ${generation}`);
|
||
return {
|
||
generation,
|
||
hostname: `chaos-deploy-${createHash("sha256").update(`${waveId}:${generation}`).digest("hex").slice(0, 10)}`,
|
||
projectId: uuidFor(waveId, "project"),
|
||
environmentId: uuidFor(waveId, "environment"),
|
||
serviceId: uuidFor(waveId, "service"),
|
||
deploymentId: uuidFor(waveId, "deployment", generation),
|
||
volumeId: uuidFor(waveId, "volume"),
|
||
};
|
||
}
|
||
|
||
/** The next deploy of the same service: same project, environment, service and volume. */
|
||
export function nextRedeployIdentity(waveId: string, current: RedeployIdentity): RedeployIdentity {
|
||
return redeployIdentity(waveId, current.generation + 1);
|
||
}
|
||
|
||
/**
|
||
* Server environment for one generation. The volume mount is the instance home: on staging the
|
||
* instance directory lives on the exclusive volume that survives each deploy.
|
||
*/
|
||
export function redeployEnvironment(identity: RedeployIdentity, volumeMountPath: string): Record<string, string> {
|
||
return {
|
||
[REDEPLOY_HOSTNAME_ENV]: identity.hostname,
|
||
RAILWAY_PROJECT_ID: identity.projectId,
|
||
RAILWAY_ENVIRONMENT_ID: identity.environmentId,
|
||
RAILWAY_SERVICE_ID: identity.serviceId,
|
||
RAILWAY_DEPLOYMENT_ID: identity.deploymentId,
|
||
RAILWAY_VOLUME_ID: identity.volumeId,
|
||
RAILWAY_VOLUME_MOUNT_PATH: volumeMountPath,
|
||
};
|
||
}
|
||
|
||
/** Node arguments that load the preload ahead of the server entry's own `--import` loader. */
|
||
export function redeployNodeArgs(args: readonly string[]): string[] {
|
||
return ["--import", REDEPLOY_PRELOAD_PATH, ...args];
|
||
}
|
||
|
||
/** What a K15 receipt records: both generations, so a replay and the evidence name them. */
|
||
export function redeployReceipt(from: RedeployIdentity, to: RedeployIdentity) {
|
||
return { fromGeneration: from.generation, toGeneration: to.generation, fromHostname: from.hostname, toHostname: to.hostname,
|
||
fromDeploymentId: from.deploymentId, toDeploymentId: to.deploymentId, volumeId: to.volumeId };
|
||
}
|