mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-14 17:22:00 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs execute through a Runner (`paperclip-runnerd`) inside a remote sandbox. A warm Daytona sandbox is reused by later runs. > - When a Runner exits before `session.open`, the run does a bounded same-run bootstrap retry. Durable evidence proves that no provider session and no provider event exist. > - The retry fails when the Runner wrote `runner-state.json` but exited before it wrote its provider state. Host preparation then calls the sandbox state "unavailable" and refuses with `runner_harness_state_mismatch: resumed_unavailable`. The task needs Retry. > - This pull request reads the Runner state on its own when the full check fails, but only for a proven bootstrap restart of the same run. A non-suspended Runner state of this run is then the run's own interrupted bootstrap, and the existing restart path moves it aside and claims a new session root. > - The benefit is that a Runner that exits before `session.open` on a reused sandbox recovers without a person. Saved state, a verified backup and every other mismatch keep their refusal. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T230537-s6402` (Claude through ACPX, Daytona warm sandboxes, `--ephemeral-server`, K1/K2/K15 with `--k15-fresh-home`, seed 6402) needed Retry on 4 of 6 tasks, with the same signature after each restart kind. The first attempt of each run reused the warm sandbox. Its Runner failed with `runner_ingress_bind_conflict: failed to bind listener: Address already in use` and exited with `lifecycle=connecting`. Recovery logged "Restarting an incomplete native runner bootstrap on the same heartbeat run". The second attempt reused the environment and failed with `runner_harness_state_mismatch: resumed_unavailable`. The earlier wave `w-20261111T212429-s6402` had one run with the same error code. **Root cause** `inspectRemoteHarnessState` in `server/src/services/native-runtime/native-session-executor.ts` checks the persistence directories, `runner-state.json` and the provider state file in one shell test. A Runner that exits at bind time wrote `runner-state.json` but no provider state, so the one test fails and the result is `incompleteReason: "unavailable"` before the Runner state is read. The restart path (`restartableBootstrapState`) accepts only `not_suspended` and `provider_unopened`, so it is skipped. `claimUntouchedRemoteSession` then cannot create the session root that the first attempt created, and the resumed branch throws `resumed_unavailable`. On a created lease the same state fell through without a restart. **Expected behavior** A proven, interrupted bootstrap of the same run restarts on the same sandbox. The run finishes without Retry. **Steps to reproduce** Run the s6402 cell above, or add `--runner-bind-fault` (this pull request) to force the Runner exit on every run's first launch. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: when the full harness check fails during host preparation, `inspectOwnUnpreparedBootstrap` reads `runner-state.json` alone. It returns `not_suspended` for this run only when all of these are true: this run has a proven bootstrap restart (`bootstrap_incomplete` recovery plus the stop proof), no failover backup exists, no archived rotation snapshot is in use, and the Runner state names this run, this Runner instance and this session with a lifecycle other than `suspended`. Otherwise the result stays `unavailable`, as before. The existing restart path then moves the session root aside (it is kept, not deleted) and claims a new one. - Chaos coverage: `--runner-bind-fault` in `tests/restart-chaos` gives every server of the wave `PAPERCLIP_TEST_RUNNER_BIND_FAULT=first_launch`. With `NODE_ENV=test` only, the remote Runner launcher starts a short-lived listener on the Runner's listen port before a Runner's first launch on that server, so the Runner exits with a real `runner_ingress_bind_conflict`. Documented in `tests/restart-chaos/README.md`. ## Verification - Red on master: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-session-executor.test.ts -t "bootstraps only an untouched provider session"` failed 2 of the new cells. Resumed lease: `Error: runner_harness_state_mismatch: resumed_unavailable` (the staging error). Created lease: no restart happened (`expected [] to have a length of 1 but got +0`). - Green: the same cells pass. The negative cells keep `resumed_unavailable`: no restart proof, a Runner state of another run, a `suspended` Runner state with missing provider state, and a failover backup present. - `native-session-executor.test.ts`: 1406 of 1406 pass. A new test proves the bind fault holds the listen port (a second bind gets `EADDRINUSE`) and fires once per Runner instance. - `tests/restart-chaos`: `runner-bind-fault.test.ts`, `question-handoff.test.ts`, `shape-cells.test.ts` pass. Server and harness `tsc --noEmit` pass. - Chaos wave: see the comment below for the before/after counts. ## Risks - The new read runs only for a proven bootstrap restart of the same run with a failed full check, so the normal path has no extra command and no wait before `turn.start`. - A partly saved session (`suspended` Runner state without provider state), a Runner state of another run and any failover backup keep today's refusal. The moved-aside root is kept for inspection. - A Runner that started under an older server and is restarted by this version takes the same path, because it reads only files that older Runners already write. - The bind fault is test instrumentation. It is inert unless `NODE_ENV=test` and the variable are both set. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edit, test runs). Default context window, extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #20928 (restart a Runner that exited before session.open), #20874, #20863, #20953. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
499 lines
24 KiB
TypeScript
499 lines
24 KiB
TypeScript
/** Shared types for the control-plane restart chaos harness (plan §5). */
|
||
import type { DiskFullFaultKind, DiskFullRecord } from "./disk-full-faults.js";
|
||
import type { ProviderFaultEvent, ProviderFaultKind, ProviderFaultRule, ProviderMessage, ProviderRestartKind } from "./provider-faults.js";
|
||
import type { QuestionArtifacts, QuestionRecord } from "./question-handoff.js";
|
||
import type { RepeatedRestartRecord, RestartStep } from "./repeated-restarts.js";
|
||
import type { TimeFaultRecord } from "./time-faults.js";
|
||
|
||
/**
|
||
* K14 (upgrade mid-run): a graceful stop, then the replacement starts from the current build (`--upgrade-from`).
|
||
* K15 (redeploy, redeploy.ts): a graceful stop, then the replacement starts in a new emulated process space with a new
|
||
* Railway-shaped deployment id on the same volume, as a hosted deploy replaces the server's container.
|
||
* TM1–TM3 (time faults, time-faults.ts): freeze the server, sleep with aged rows, or an aged outage.
|
||
* RR1–RR3 (repeated restarts, repeated-restarts.ts): a K1, then more restarts while the replacement recovers.
|
||
* MS1–MS3 (two servers, one database, multi-server.ts): a second server beside the first.
|
||
* CR1–CR3 (staging class R, class-r.ts): a long-held redeploy, a provider-only kill in the sandbox, a plugin RPC stall.
|
||
*/
|
||
export const FAULT_KINDS = ["K1", "K2", "K4", "K6", "K14", "K15", "TM1", "TM2", "TM3", "RR1", "RR2", "RR3", "MS1", "MS2", "MS3", "CR1", "CR2", "CR3"] as const;
|
||
export type FaultKind = (typeof FAULT_KINDS)[number];
|
||
/** The shortest seeded K4 outage (scheduler.ts K4_HOLD_RANGE_MS), also the K4 hold after a provider fault. */
|
||
export const K4_MIN_HOLD_MS = 120_000;
|
||
/**
|
||
* A task's planned fault. "K13" (restart storm) tasks have none of their own; the wave's storm crosses them.
|
||
* PV1–PV3 (provider message faults, provider-faults.ts) restart nothing: the task's rule is armed in the Codex shim.
|
||
* "K0" is a `--matrix` baseline cell: the same journey with no fault.
|
||
* SR1 (session faults, session-faults.ts) restarts nothing: the board replaces the saved session before answering.
|
||
* DF1 (disk-full-faults.ts) restarts nothing either: the task's final workspace saves find the sandbox disk full.
|
||
*/
|
||
export type TaskFaultKind = FaultKind | "K13" | "K0" | ProviderFaultKind | DiskFullFaultKind | import("./session-faults.js").SessionFaultKind;
|
||
|
||
/**
|
||
* G1: an approval-gated action across a restart. H1: a budget hard stop crossed mid-run with a restart (its own wave).
|
||
* Q1: a saved question (question-handoff.ts). Q2, Q3: journeys.ts.
|
||
* L1: high-output, high-accounting run for database contention (contention.ts).
|
||
* F1: an absolute symlink pointing out of the repository workspace, kept at rest (staging save-back class F).
|
||
* T1: a 100-second shell tool in the agent's workspace, so a fault lands while the remote Runner runs a tool (staging class R).
|
||
* H2: the agent's own last step hands the task to the person who created it, in review (staging class H).
|
||
* W2, W2Q, W2C: the same task woken again 0–60 s after its run succeeded (rewake.ts), many tasks on one agent.
|
||
*/
|
||
export const WORKLOAD_IDS = ["A1", "B1", "B5", "C1", "M1", "G1", "H1", "R1", "S1", "Q1", "L1", "Q2", "Q3", "Q4", "F1", "T1", "W2", "W2Q", "W2C", "H2"] as const;
|
||
export type WorkloadId = (typeof WORKLOAD_IDS)[number];
|
||
|
||
export const ADAPTER_IDS = ["codex", "claude", "opencode"] as const;
|
||
export type AdapterId = (typeof ADAPTER_IDS)[number];
|
||
|
||
/** `asked` and `answered`: the journey boundaries of Q1–Q3 (journeys.ts, question-handoff.ts). */
|
||
export const TRIGGER_IDS = ["time", "comment", "retry", "queued", "tool", "result", "approval", "paused", "stop", "asked", "answered"] as const;
|
||
/** `provider` is not a `--triggers` choice: `--provider-fault-restart` schedules it (K1 after a provider fault applied). */
|
||
export type TriggerId = (typeof TRIGGER_IDS)[number] | "provider";
|
||
|
||
export const ENVIRONMENT_KINDS = ["local", "daytona"] as const;
|
||
export type EnvironmentKind = (typeof ENVIRONMENT_KINDS)[number];
|
||
|
||
export interface WaveOptions {
|
||
env: EnvironmentKind;
|
||
adapters: AdapterId[];
|
||
workloads: WorkloadId[];
|
||
faults: FaultKind[];
|
||
triggers?: TriggerId[];
|
||
concurrency: number;
|
||
tasks: number;
|
||
seed: number;
|
||
/** Company budget hard cap in cents; the product hard-stops at this value. */
|
||
budgetCents: number;
|
||
/** Poll interval for the observer, 2–5 s per plan §5. */
|
||
pollMs: number;
|
||
/** `--env daytona`: the immutable runner image (registry/repository@sha256:…). */
|
||
daytonaImage?: string;
|
||
/** `--daytona-pack-mismatch <image@sha256:…>`: sandboxes start from this
|
||
* image while the controller keeps `daytonaImage`'s runnerd and provider pack,
|
||
* so the sandbox's preinstalled pack does not match and every fresh sandbox
|
||
* takes the provider-pack upload and admission path (staging run 3f317172). */
|
||
daytonaSandboxImage?: string;
|
||
/** `--env daytona`: vCPU per sandbox (see seed.ts chaosDaytonaShape). */
|
||
daytonaCpu?: number;
|
||
/** `--daytona-reuse-lease`: the chaos environment keeps its sandboxes between turns (a per-turn run stops and keeps its sandbox). */
|
||
daytonaReuseLease?: boolean;
|
||
/** `--daytona-warm`: the reusable warm Daytona environment fixture. */
|
||
daytonaWarm?: boolean;
|
||
/** `--daytona-inherit`: a Daytona environment that inherits the lifecycle and leaves lease reuse unset. */
|
||
daytonaInherit?: boolean;
|
||
/** `--k4-hold-max-seconds`: upper end of the seeded K4 hold (default 300 s, cap 600 s). */
|
||
k4HoldMaxMs?: number;
|
||
/** `--k15-hold-seconds`: the seeded K15 outage range (redeploy.ts); absent restarts at once. With `--k15-overlap-seconds`, the overlap range. */
|
||
k15HoldMs?: [number, number];
|
||
/** `--k15-overlap-seconds`: a K15 boots the next deploy beside the old server and SIGTERMs the old one after the seeded overlap (redeploy.ts). */
|
||
k15Overlap?: boolean;
|
||
/** `--k15-fresh-home`: each deploy generation gets a new, empty instance home and no volume (redeploy.ts). */
|
||
k15FreshHome?: boolean;
|
||
ephemeralServer?: boolean;
|
||
runnerBindFault?: boolean;
|
||
/** `--faults K0`: cells with no fault, for workloads whose shape is the fault (W2, W2Q, W2C). */
|
||
baseline?: boolean;
|
||
/**
|
||
* `--fail-recovery-step <point>[:<nth>]` (recovery-step-faults.ts): once the wave's first restart fault fired, the
|
||
* nth hit of that crash point in the replacement throws its step's own error, once.
|
||
*/
|
||
failRecoveryStep?: string;
|
||
/** `--long-turn-minutes`: every task's first turn does this much real tool work before its workload (long-turn.ts). */
|
||
longTurnMinutes?: number;
|
||
/** `--sandbox-watch-seconds`: on Daytona, capture every wave sandbox on this clock (daytona-runner-logs.ts class R probe). */
|
||
sandboxWatchMs?: number;
|
||
/** `--cr1-hold-seconds`: the CR1 outage range (class-r.ts); absent on a CR1 wave means 360 s. */
|
||
cr1HoldMs?: [number, number];
|
||
/** `--cr2-hold-seconds`: a K15 outage around CR2's provider kill; absent or 0 keeps the controller up. */
|
||
cr2HoldMs?: [number, number];
|
||
/** `--cr3-stall-seconds`: how long CR3 stops the Daytona plugin worker; absent means 45 s. */
|
||
cr3StallMs?: [number, number];
|
||
/** `--tm1-hold-seconds`: the seeded TM1 stall range (time-faults-config.ts); absent uses 95–150 s. */
|
||
tm1HoldMs?: [number, number];
|
||
faultAfterMs?: number;
|
||
/**
|
||
* Known product signatures this wave deliberately masks. "rc06" keeps the
|
||
* company budget amount but disables its hard stop, so a stuck
|
||
* accounting row (RC-06) cannot wedge every later task. Spend is still
|
||
* recorded per wave. Absent on plans written before masks existed.
|
||
*/
|
||
masks?: string[];
|
||
/**
|
||
* K13 restart storm: fire faults drawn from `faults` every 2–4 minutes for
|
||
* this many minutes, keeping `concurrency` tasks in flight, then drain.
|
||
*/
|
||
stormMinutes?: number;
|
||
/**
|
||
* Runner lifecycle forced onto every seeded agent (`--lifecycle`). Absent
|
||
* keeps the catalog profile's own setting.
|
||
*/
|
||
lifecycle?: "per_turn" | "warm" | "default";
|
||
/**
|
||
* Crash-point suite (`--crash-point <name>[:<nth>][,...]`): no external
|
||
* fault is scheduled. One server serves every listed point in turn: each
|
||
* point's batch of tasks launches, then the point is armed and the server
|
||
* SIGKILLs itself at its nth hit and is restarted like K1.
|
||
*/
|
||
crashPoint?: string;
|
||
/** Crash-point suite: hold the control plane down this long after each self-kill (K4-style). */
|
||
crashPointHoldMs?: number;
|
||
/**
|
||
* K14 upgrade mid-run (`--upgrade-from <dir>`): an older, prepared checkout. The wave's server starts
|
||
* from it; the first K14 stops it gracefully and starts the current checkout on the same database
|
||
* and instance directory. Later K14 faults restart the current build, like K2.
|
||
*/
|
||
upgradeFrom?: string;
|
||
workspaceRepo?: string;
|
||
storage?: "s3";
|
||
storageFaults?: Array<import("./storage-chaos.js").StorageFaultKind>;
|
||
storageFaultMs?: [number, number];
|
||
storageWindows?: import("./storage-chaos.js").StorageFaultWindow[];
|
||
storageTrigger?: import("./storage-trigger.js").StorageTriggerTarget;
|
||
nativeTrigger?: import("./native-trigger.js").NativeTriggerSpec;
|
||
workspaceCommit?: string;
|
||
workspaceGitUrl?: string;
|
||
/** Provider message faults from `--faults` (PV1 delay, PV2 drop, PV3 duplicate); see provider-faults.ts. */
|
||
providerFaults?: ProviderFaultKind[];
|
||
/** Sandbox disk-full faults from `--faults` (DF1); see disk-full-faults.ts. */
|
||
diskFullFaults?: DiskFullFaultKind[];
|
||
/** Session faults from `--faults` (SR1); see session-faults.ts. */
|
||
sessionFaults?: Array<import("./session-faults.js").SessionFaultKind>;
|
||
providerFaultMessage?: ProviderMessage;
|
||
/** `--provider-fault-message S1=interrupt_ack,...`: a workload's own message, over `providerFaultMessage`. */
|
||
providerFaultMessageByWorkload?: Partial<Record<WorkloadId, ProviderMessage>>;
|
||
/** PV1 delays, cycled across tasks (`--provider-fault-delays`). */
|
||
providerFaultDelaysMs?: number[];
|
||
/** `--db-pressure <writers>`: harness-owned writers touching the wave's running run rows back to back (contention.ts). */
|
||
dbPressure?: number;
|
||
/** `--matrix`: one task per workload × adapter × (K0, or each fault at each of the workload's triggers). */
|
||
matrix?: boolean;
|
||
/** `--server-from <dir>`: every server of the wave runs from this prepared checkout (for example a pre-fix commit). */
|
||
serverFrom?: string;
|
||
/** TM2/TM3: how far every database timestamp is shifted back (`--time-age-hours`, default 26 h). */
|
||
timeAgeMs?: number;
|
||
/** `--provider-fault-restart K1`: each PV1/PV2 task also restarts the control plane after its fault applied. */
|
||
providerFaultRestart?: ProviderRestartKind;
|
||
/** `--provider-fault-restart K2,K4,K1`: restart kinds cycled across PV1/PV2 tasks (the first is `providerFaultRestart`). */
|
||
providerFaultRestartKinds?: ProviderRestartKind[];
|
||
providerFaultRestartAfterMs?: number;
|
||
/** RR1–RR3: the gap from a replacement's health to the next stop (`--rr-gap-seconds`); absent keeps each kind's default. */
|
||
rrGapMs?: [number, number];
|
||
/** RR1–RR3 (`--rr-stop-on recovering`): stop each replacement on its first `recovering` status instead of after a gap. */
|
||
rrStopOn?: "recovering";
|
||
}
|
||
|
||
export interface StormFault {
|
||
kind: FaultKind;
|
||
holdMs: number;
|
||
/** Wait this long after the previous storm fault settled (or the storm started). */
|
||
gapMs: number;
|
||
}
|
||
|
||
export interface StormPlan {
|
||
durationMs: number;
|
||
faults: StormFault[];
|
||
}
|
||
|
||
export interface TaskPlan {
|
||
taskId: string;
|
||
index: number;
|
||
workload: WorkloadId;
|
||
adapter: AdapterId;
|
||
fault: TaskFaultKind;
|
||
/** Fire the fault this long after the task's issue is created. */
|
||
fireAfterMs: number;
|
||
/** Outage hold between stop and restart. 0 for K1/K2. */
|
||
holdMs: number;
|
||
trigger?: TriggerId;
|
||
triggerDelayMs?: number;
|
||
/** C1 only: post the follow-up comment this long after creation. */
|
||
followUpAfterMs: number | null;
|
||
/** Unique marker used by the deterministic T-oracle. */
|
||
nonce: string;
|
||
/**
|
||
* Replay only: every fault that crossed this task in the source wave, with
|
||
* its offset from the task's launch. Replaces the single planned fault.
|
||
*/
|
||
replayFaults?: ScheduledFault[];
|
||
/** PV1–PV3 tasks: the provider message rule armed for this task's nonce. */
|
||
providerFault?: ProviderFaultRule;
|
||
/** RR1–RR3: the seeded restarts after the first kill. */
|
||
restartSteps?: RestartStep[];
|
||
/** `--long-turn-minutes`: the first turn's tool work before the workload (long-turn.ts). */
|
||
longTurnMinutes?: number;
|
||
}
|
||
|
||
export interface ScheduledFault {
|
||
kind: FaultKind; holdMs: number;
|
||
atMs: number;
|
||
trigger?: TriggerId;
|
||
delayMs?: number;
|
||
/** RR1–RR3: the restarts after the first kill. */
|
||
restartSteps?: RestartStep[];
|
||
}
|
||
|
||
export interface WavePlan {
|
||
schema: "paperclip.restart-chaos.wave-plan/v1";
|
||
waveId: string;
|
||
seed: number;
|
||
createdAt: string;
|
||
options: WaveOptions;
|
||
tasks: TaskPlan[];
|
||
/** K13 only: the seeded storm fault sequence. */
|
||
storm?: StormPlan;
|
||
/** Present when this wave replays one task of an earlier wave. */
|
||
replayOf?: { waveId: string; taskId: string };
|
||
}
|
||
|
||
export type Verdict = "pass" | "fail" | "n/a";
|
||
export interface RuleResult { verdict: Verdict; detail: string }
|
||
export type RuleId = "R1" | "R2" | "R3" | "R4" | "R5" | "R6";
|
||
|
||
export interface RunSample {
|
||
id: string;
|
||
status: string;
|
||
createdAt: string | null;
|
||
startedAt: string | null;
|
||
finishedAt: string | null;
|
||
errorCode: string | null;
|
||
/** Bounded code naming why a generic `errorCode` (such as `adapter_failed`) failed, e.g. `40001`. */
|
||
failureCause?: string | null;
|
||
retryOfRunId: string | null;
|
||
wakeCommentIds: string[];
|
||
livenessState: string | null;
|
||
lastEvidenceAt?: string | null;
|
||
/** A tool started with no matching end yet (a long `bench.sh` is quiet but live). */
|
||
toolInFlight?: boolean;
|
||
/** The run's recorded local runner process is still running with the same start time. */
|
||
processAlive?: boolean;
|
||
/** A provider turn started and has no turn terminal yet. */
|
||
providerTurnActive?: boolean;
|
||
}
|
||
|
||
export interface InterruptionSample {
|
||
state: string;
|
||
canRetry: boolean;
|
||
retryWait: boolean;
|
||
workspaceFailure: string | null;
|
||
observedRunId: string | null;
|
||
userAction: string | null;
|
||
}
|
||
|
||
/** One observer poll. Unreachable polls keep only the timestamp. */
|
||
export interface TaskSample {
|
||
at: number;
|
||
reachable: boolean;
|
||
serverReady: boolean;
|
||
issueStatus?: string;
|
||
interruption?: InterruptionSample | null;
|
||
activeRecoveryAction?: boolean;
|
||
/** Owner of the active recovery action; "board" means it waits on a person. */
|
||
recoveryOwner?: string | null;
|
||
executionBlocker?: boolean;
|
||
scheduledRetry?: boolean;
|
||
runs?: RunSample[];
|
||
}
|
||
|
||
export interface RetryAttempt {
|
||
at: number;
|
||
acceptedAt?: number;
|
||
observedRunId: string | null;
|
||
httpStatus: number | null;
|
||
accepted: boolean | null;
|
||
error: string | null;
|
||
via?: "ui" | "api";
|
||
bannerText?: string | null;
|
||
diagnostics?: { screenshot: string | null; dom: string | null } | null;
|
||
dismissedDialogs?: string[];
|
||
}
|
||
|
||
export const UI_RETRY_BUTTON_MISSING = "ui_retry_button_missing";
|
||
export const UI_RETRY_NO_REQUEST = "ui_retry_no_request";
|
||
export const UI_RETRY_BLOCKED_BY_MODAL = "ui_retry_blocked_by_modal";
|
||
|
||
export interface FaultReceipt {
|
||
faultId: string;
|
||
kind: FaultKind;
|
||
/** Task that scheduled the fault; every other running task is also exposed. */
|
||
requestedBy: string;
|
||
/** "none": MS1 sends no signal to the primary server. */
|
||
signal: "SIGKILL" | "SIGTERM" | "SIGSTOP" | "none";
|
||
holdMs: number;
|
||
requestedAt: number;
|
||
startedAt: number;
|
||
stoppedAt: number | null;
|
||
restartedAt: number | null;
|
||
healthyAt: number | null;
|
||
readyAt: number | null;
|
||
phases: Array<{ phase: string; at: number }>;
|
||
trigger?: TriggerId;
|
||
triggerDelayMs?: number;
|
||
loadAvg1m?: number;
|
||
crashLoop?: Array<{ pid: number | null; target: string; phase: string; at: number }>;
|
||
/** Crash-point suite: the server killed itself at this named point (no external signal). */
|
||
crashPoint?: { name: string; hit: number; pid: number; token?: string };
|
||
/** K14: the build the stopped server ran and the build its replacement runs. */
|
||
upgrade?: { fromRoot: string; fromSha: string | null; toRoot: string; toSha: string | null; switched: boolean };
|
||
/** K15: the deploy generation, emulated host name and deployment id before and after. */
|
||
redeploy?: ReturnType<typeof import("./redeploy.js").redeployReceipt>;
|
||
/** `--fail-recovery-step`: the step armed (fail mode) when this fault's replacement started. */
|
||
recoveryStep?: { spec: string; token: string };
|
||
/** TM1–TM3: what was frozen and which rows were aged. */
|
||
timeFault?: TimeFaultRecord;
|
||
/** RR1–RR3: every stop and replacement of the sequence, the first kill included. */
|
||
repeatedRestart?: RepeatedRestartRecord;
|
||
/** MS1–MS3: the second server and what happened to the first. */
|
||
multiServer?: import("./multi-server.js").MultiServerRecord;
|
||
/** CR1–CR3: the provider kill or plugin stall and the sandbox captures taken around the fault (class-r.ts). */
|
||
classR?: import("./class-r.js").ClassRRecord;
|
||
serverLogOffsetAtStart: number;
|
||
serverLogOffsetAtReady: number | null;
|
||
error: string | null;
|
||
}
|
||
|
||
export interface OracleArtifacts {
|
||
issueStatus: string | null;
|
||
documents: Record<string, string | null>;
|
||
agentComments: string[];
|
||
files: Record<string, string | null>;
|
||
commands: Record<string, { exitCode: number | null; stdout: string; stderr: string }>;
|
||
workspacePath: string | null;
|
||
/** B5: distinct `.bench/progress` values the harness itself saw while the task was live. */
|
||
benchProgressObserved?: number;
|
||
stop?: { runId: string | null; runStatus: string | null; runsStartedBeforeContinue: string[]; continuePosted: boolean; missed?: boolean };
|
||
children?: Array<{ id: string; title?: string | null; status: string | null; assigneeAgentId: string | null; documents: Record<string, string | null> }>;
|
||
childStalls?: Array<{ childIssueId: string; recoveryActionId: string | null }>;
|
||
stallWakes?: Array<{ stalledIssueId: string | null; recoveryActionId: string | null }>;
|
||
/** G1: approvals linked to the task. */
|
||
approvals?: Array<{ id: string; status: string; title: string | null }>;
|
||
/** G1 and H1: the board's actions and what the observer saw around them. */
|
||
governance?: GovernanceRecord;
|
||
/** Q1, Q4: the saved questions and the task's runs. */
|
||
question?: QuestionArtifacts;
|
||
/** PV1–PV3: the shim's log lines for this task. */
|
||
providerFaultEvents?: ProviderFaultEvent[];
|
||
/** L1: accounting and run-log evidence read from the wave database at rest. */
|
||
contention?: import("./contention.js").ContentionEvidence;
|
||
/** Q2, Q3: what the board did and what the task, its runs and the harness's app saw. */
|
||
journey?: import("./journeys.js").JourneyArtifacts;
|
||
/** W2, W2Q, W2C: the re-wake and the task's runs (rewake.ts). */
|
||
rewake?: import("./rewake.js").RewakeArtifacts;
|
||
/** `--long-turn-minutes`: the task's runs, for the harness's own long-turn evidence (long-turn.ts). */
|
||
runs?: import("./rewake.js").RewakeArtifacts["runs"];
|
||
/** H2: who owns the task at rest, and how its runs ended. */
|
||
handoff?: HandoffArtifacts;
|
||
}
|
||
|
||
/** H2: the task's owner and execution state at rest, and each run's terminal state. */
|
||
export interface HandoffArtifacts {
|
||
assigneeAgentId: string | null;
|
||
assigneeUserId: string | null;
|
||
createdByUserId: string | null;
|
||
executionBlocker: boolean;
|
||
activeRecoveryAction: boolean;
|
||
/** Null when the run history could not be read: the run checks then fail. */
|
||
runs: Array<{ id: string; status: string; errorCode: string | null }> | null;
|
||
}
|
||
|
||
/** What the board did for a governance task (G1, H1), and what the observer saw. */
|
||
export interface GovernanceRecord {
|
||
/** G1: first time the observer saw the task's approval pending. */
|
||
approvalSeenAt?: number | null;
|
||
approvalId?: string | null;
|
||
/** G1: the board's decision, made once the task's faults fired and the server was ready. */
|
||
decision?: { kind: "approve" | "reject"; at: number; httpStatus: number | null; error: string | null;
|
||
/** The approval was still pending when the board decided (it survived the restart). */
|
||
pendingWhenDecided: boolean; afterFault: boolean };
|
||
/** H1: the company budget the harness set before launch, in cents (observed spend + 1). */
|
||
budgetSetCents?: number | null;
|
||
budgetSetError?: string | null;
|
||
/** H1: first time the observer saw the company paused for its budget. */
|
||
budgetPausedAt?: number | null;
|
||
/** H1: runs of the task that started while the company was paused (must be none). */
|
||
runsStartedWhilePaused?: string[];
|
||
/** H1: the board's raise-and-resume, made once the task's faults fired and the pause held. */
|
||
resume?: { at: number; amount: number | null; incidentId: string | null; httpStatus: number | null; error: string | null; afterFault: boolean;
|
||
/** When the resume request returned: the pause window ends here. */
|
||
doneAt?: number };
|
||
/** H1: the company was active again at rest. */
|
||
companyActiveAtRest?: boolean;
|
||
/** H1: the pause cleared on its own (before the board resumed it). */
|
||
pauseLostAt?: number | null;
|
||
/** G1: subtasks named after the nonce that existed before the board decided (must be 0). */
|
||
childrenBeforeDecision?: number | null;
|
||
}
|
||
|
||
export interface OracleResult { verdict: Exclude<Verdict, "n/a">; checks: Array<{ name: string; ok: boolean; detail?: string }> }
|
||
|
||
export interface RestSweep {
|
||
activeRuns: string[];
|
||
runningWorkspaceOperations: string[];
|
||
activeRecoveryAction: boolean;
|
||
/** `--env daytona`: orphaned sandboxes and held leases belonging to this task's runs. */
|
||
daytonaLeaks?: string[];
|
||
error: string | null;
|
||
}
|
||
|
||
export interface TaskRecord {
|
||
plan: TaskPlan;
|
||
issueId: string | null;
|
||
identifier: string | null;
|
||
createdAt: number | null;
|
||
launchError: string | null;
|
||
samples: TaskSample[];
|
||
retry: RetryAttempt | null;
|
||
restAt: number | null;
|
||
finalStatus: string | null;
|
||
deadlineAt: number | null;
|
||
timedOut: boolean;
|
||
followUp: { at: number; sentAt?: number; postedAt?: number; commentId: string | null; error: string | null; duringRun?: boolean } | null;
|
||
stop?: { at: number; runId: string | null; httpStatus: number | null; error: string | null; missed?: boolean };
|
||
faultIds: string[];
|
||
sweep: RestSweep | null;
|
||
/** Workspace files present just before the first fault that hit this task (R2). */
|
||
preFaultFiles: string[] | null;
|
||
preFaultWorkspacePath?: string;
|
||
lastFaultAt?: number;
|
||
restSnapshotError?: string;
|
||
priorRetries?: RetryAttempt[];
|
||
childStalls?: Array<{ childIssueId: string; recoveryActionId: string | null }>;
|
||
childRetries?: Array<RetryAttempt & { childIssueId: string }>;
|
||
workspaceMovedTo?: string;
|
||
/** Set when the pre-fault snapshot could not read the workspace; R2 then fails visibly. */
|
||
preFaultSnapshotError?: string | null;
|
||
remoteWorkspace?: boolean;
|
||
remoteFiles?: Record<string, string | null>;
|
||
remoteFileErrors?: Record<string, string>;
|
||
/** Pre-fault files missing at rest; null when not checked. */
|
||
lostFiles: string[] | null;
|
||
/** B5: distinct bench progress values the observer read during polling. */
|
||
benchProgressSeen?: string[];
|
||
benchFirstSeenAt?: number | null;
|
||
/** C1: first time the observer saw the run that received the follow-up propose or accept its result. */
|
||
resultAcceptedAt?: number | null;
|
||
/** The run whose result armed the `result` trigger; the fault fires only while it is still running. */
|
||
resultRunId?: string | null;
|
||
triggerMissed?: TriggerId[];
|
||
governance?: GovernanceRecord;
|
||
/** K14 waves: the build serving when the task launched. */
|
||
serverBuildAtLaunch?: "upgrade_from" | "current";
|
||
/** Q1, Q4: the saved questions and the board's answers. */
|
||
question?: QuestionRecord;
|
||
/** Q2, Q3: the board's journey actions (journey-driver.ts). */
|
||
journey?: import("./journeys.js").JourneyRecord;
|
||
/** W2, W2Q, W2C: the first run, the board's wake and the run that served it (rewake.ts). */
|
||
rewake?: import("./rewake.js").RewakeRecord;
|
||
/** PV with `--provider-fault-restart`: when the shim logged the task's fault (arms the `provider` trigger). */
|
||
providerFaultAppliedAt?: number;
|
||
/** DF1: when the fault was armed and cleared, and what the board showed at the disk-full rest. */
|
||
diskFull?: DiskFullRecord;
|
||
}
|
||
|
||
export interface TaskGrade {
|
||
crossed: boolean;
|
||
crossingFaults: string[];
|
||
rules: Record<RuleId, RuleResult>;
|
||
T: OracleResult;
|
||
/** Universal checks U1–U5 (universal.ts); absent on grades written before them. */
|
||
universal?: import("./universal.js").UniversalGrade;
|
||
pass: boolean;
|
||
signature: string | null;
|
||
}
|