Files
PaperClipAI/tests/restart-chaos/types.ts
T
DottaandClaude Opus 5.5 66344b76ae fix(native): restart a bootstrap whose Runner exited before writing provider state on a resumed sandbox (#20972)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs execute through a Runner (`paperclip-runnerd`) inside a
remote sandbox. A warm Daytona sandbox is reused by later runs.
> - When a Runner exits before `session.open`, the run does a bounded
same-run bootstrap retry. Durable evidence proves that no provider
session and no provider event exist.
> - The retry fails when the Runner wrote `runner-state.json` but exited
before it wrote its provider state. Host preparation then calls the
sandbox state "unavailable" and refuses with
`runner_harness_state_mismatch: resumed_unavailable`. The task needs
Retry.
> - This pull request reads the Runner state on its own when the full
check fails, but only for a proven bootstrap restart of the same run. A
non-suspended Runner state of this run is then the run's own interrupted
bootstrap, and the existing restart path moves it aside and claims a new
session root.
> - The benefit is that a Runner that exits before `session.open` on a
reused sandbox recovers without a person. Saved state, a verified backup
and every other mismatch keep their refusal.

## Linked Issues or Issue Description

**What happened?**
Restart-chaos wave `w-20261111T230537-s6402` (Claude through ACPX,
Daytona warm sandboxes, `--ephemeral-server`, K1/K2/K15 with
`--k15-fresh-home`, seed 6402) needed Retry on 4 of 6 tasks, with the
same signature after each restart kind. The first attempt of each run
reused the warm sandbox. Its Runner failed with
`runner_ingress_bind_conflict: failed to bind listener: Address already
in use` and exited with `lifecycle=connecting`. Recovery logged
"Restarting an incomplete native runner bootstrap on the same heartbeat
run". The second attempt reused the environment and failed with
`runner_harness_state_mismatch: resumed_unavailable`. The earlier wave
`w-20261111T212429-s6402` had one run with the same error code.

**Root cause**
`inspectRemoteHarnessState` in
`server/src/services/native-runtime/native-session-executor.ts` checks
the persistence directories, `runner-state.json` and the provider state
file in one shell test. A Runner that exits at bind time wrote
`runner-state.json` but no provider state, so the one test fails and the
result is `incompleteReason: "unavailable"` before the Runner state is
read. The restart path (`restartableBootstrapState`) accepts only
`not_suspended` and `provider_unopened`, so it is skipped.
`claimUntouchedRemoteSession` then cannot create the session root that
the first attempt created, and the resumed branch throws
`resumed_unavailable`. On a created lease the same state fell through
without a restart.

**Expected behavior**
A proven, interrupted bootstrap of the same run restarts on the same
sandbox. The run finishes without Retry.

**Steps to reproduce**
Run the s6402 cell above, or add `--runner-bind-fault` (this pull
request) to force the Runner exit on every run's first launch.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`: when
the full harness check fails during host preparation,
`inspectOwnUnpreparedBootstrap` reads `runner-state.json` alone. It
returns `not_suspended` for this run only when all of these are true:
this run has a proven bootstrap restart (`bootstrap_incomplete` recovery
plus the stop proof), no failover backup exists, no archived rotation
snapshot is in use, and the Runner state names this run, this Runner
instance and this session with a lifecycle other than `suspended`.
Otherwise the result stays `unavailable`, as before. The existing
restart path then moves the session root aside (it is kept, not deleted)
and claims a new one.
- Chaos coverage: `--runner-bind-fault` in `tests/restart-chaos` gives
every server of the wave
`PAPERCLIP_TEST_RUNNER_BIND_FAULT=first_launch`. With `NODE_ENV=test`
only, the remote Runner launcher starts a short-lived listener on the
Runner's listen port before a Runner's first launch on that server, so
the Runner exits with a real `runner_ingress_bind_conflict`. Documented
in `tests/restart-chaos/README.md`.

## Verification

- Red on master: `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"bootstraps only an untouched provider session"` failed 2 of the new
cells. Resumed lease: `Error: runner_harness_state_mismatch:
resumed_unavailable` (the staging error). Created lease: no restart
happened (`expected [] to have a length of 1 but got +0`).
- Green: the same cells pass. The negative cells keep
`resumed_unavailable`: no restart proof, a Runner state of another run,
a `suspended` Runner state with missing provider state, and a failover
backup present.
- `native-session-executor.test.ts`: 1406 of 1406 pass. A new test
proves the bind fault holds the listen port (a second bind gets
`EADDRINUSE`) and fires once per Runner instance.
- `tests/restart-chaos`: `runner-bind-fault.test.ts`,
`question-handoff.test.ts`, `shape-cells.test.ts` pass. Server and
harness `tsc --noEmit` pass.
- Chaos wave: see the comment below for the before/after counts.

## Risks

- The new read runs only for a proven bootstrap restart of the same run
with a failed full check, so the normal path has no extra command and no
wait before `turn.start`.
- A partly saved session (`suspended` Runner state without provider
state), a Runner state of another run and any failover backup keep
today's refusal. The moved-aside root is kept for inspection.
- A Runner that started under an older server and is restarted by this
version takes the same path, because it reads only files that older
Runners already write.
- The bind fault is test instrumentation. It is inert unless
`NODE_ENV=test` and the variable are both set.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, test runs). Default context window,
extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #20928 (restart a Runner that exited before session.open),
#20874, #20863, #20953.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 18:36:33 -06:00

499 lines
24 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
/** Shared types for the control-plane restart chaos harness (plan §5). */
import type { DiskFullFaultKind, DiskFullRecord } from "./disk-full-faults.js";
import type { ProviderFaultEvent, ProviderFaultKind, ProviderFaultRule, ProviderMessage, ProviderRestartKind } from "./provider-faults.js";
import type { QuestionArtifacts, QuestionRecord } from "./question-handoff.js";
import type { RepeatedRestartRecord, RestartStep } from "./repeated-restarts.js";
import type { TimeFaultRecord } from "./time-faults.js";
/**
* K14 (upgrade mid-run): a graceful stop, then the replacement starts from the current build (`--upgrade-from`).
* K15 (redeploy, redeploy.ts): a graceful stop, then the replacement starts in a new emulated process space with a new
* Railway-shaped deployment id on the same volume, as a hosted deploy replaces the server's container.
* TM1–TM3 (time faults, time-faults.ts): freeze the server, sleep with aged rows, or an aged outage.
* RR1–RR3 (repeated restarts, repeated-restarts.ts): a K1, then more restarts while the replacement recovers.
* MS1–MS3 (two servers, one database, multi-server.ts): a second server beside the first.
* CR1–CR3 (staging class R, class-r.ts): a long-held redeploy, a provider-only kill in the sandbox, a plugin RPC stall.
*/
export const FAULT_KINDS = ["K1", "K2", "K4", "K6", "K14", "K15", "TM1", "TM2", "TM3", "RR1", "RR2", "RR3", "MS1", "MS2", "MS3", "CR1", "CR2", "CR3"] as const;
export type FaultKind = (typeof FAULT_KINDS)[number];
/** The shortest seeded K4 outage (scheduler.ts K4_HOLD_RANGE_MS), also the K4 hold after a provider fault. */
export const K4_MIN_HOLD_MS = 120_000;
/**
* A task's planned fault. "K13" (restart storm) tasks have none of their own; the wave's storm crosses them.
* PV1–PV3 (provider message faults, provider-faults.ts) restart nothing: the task's rule is armed in the Codex shim.
* "K0" is a `--matrix` baseline cell: the same journey with no fault.
* SR1 (session faults, session-faults.ts) restarts nothing: the board replaces the saved session before answering.
* DF1 (disk-full-faults.ts) restarts nothing either: the task's final workspace saves find the sandbox disk full.
*/
export type TaskFaultKind = FaultKind | "K13" | "K0" | ProviderFaultKind | DiskFullFaultKind | import("./session-faults.js").SessionFaultKind;
/**
* G1: an approval-gated action across a restart. H1: a budget hard stop crossed mid-run with a restart (its own wave).
* Q1: a saved question (question-handoff.ts). Q2, Q3: journeys.ts.
* L1: high-output, high-accounting run for database contention (contention.ts).
* F1: an absolute symlink pointing out of the repository workspace, kept at rest (staging save-back class F).
* T1: a 100-second shell tool in the agent's workspace, so a fault lands while the remote Runner runs a tool (staging class R).
* H2: the agent's own last step hands the task to the person who created it, in review (staging class H).
* W2, W2Q, W2C: the same task woken again 0–60 s after its run succeeded (rewake.ts), many tasks on one agent.
*/
export const WORKLOAD_IDS = ["A1", "B1", "B5", "C1", "M1", "G1", "H1", "R1", "S1", "Q1", "L1", "Q2", "Q3", "Q4", "F1", "T1", "W2", "W2Q", "W2C", "H2"] as const;
export type WorkloadId = (typeof WORKLOAD_IDS)[number];
export const ADAPTER_IDS = ["codex", "claude", "opencode"] as const;
export type AdapterId = (typeof ADAPTER_IDS)[number];
/** `asked` and `answered`: the journey boundaries of Q1–Q3 (journeys.ts, question-handoff.ts). */
export const TRIGGER_IDS = ["time", "comment", "retry", "queued", "tool", "result", "approval", "paused", "stop", "asked", "answered"] as const;
/** `provider` is not a `--triggers` choice: `--provider-fault-restart` schedules it (K1 after a provider fault applied). */
export type TriggerId = (typeof TRIGGER_IDS)[number] | "provider";
export const ENVIRONMENT_KINDS = ["local", "daytona"] as const;
export type EnvironmentKind = (typeof ENVIRONMENT_KINDS)[number];
export interface WaveOptions {
env: EnvironmentKind;
adapters: AdapterId[];
workloads: WorkloadId[];
faults: FaultKind[];
triggers?: TriggerId[];
concurrency: number;
tasks: number;
seed: number;
/** Company budget hard cap in cents; the product hard-stops at this value. */
budgetCents: number;
/** Poll interval for the observer, 2–5 s per plan §5. */
pollMs: number;
/** `--env daytona`: the immutable runner image (registry/repository@sha256:…). */
daytonaImage?: string;
/** `--daytona-pack-mismatch <image@sha256:…>`: sandboxes start from this
* image while the controller keeps `daytonaImage`'s runnerd and provider pack,
* so the sandbox's preinstalled pack does not match and every fresh sandbox
* takes the provider-pack upload and admission path (staging run 3f317172). */
daytonaSandboxImage?: string;
/** `--env daytona`: vCPU per sandbox (see seed.ts chaosDaytonaShape). */
daytonaCpu?: number;
/** `--daytona-reuse-lease`: the chaos environment keeps its sandboxes between turns (a per-turn run stops and keeps its sandbox). */
daytonaReuseLease?: boolean;
/** `--daytona-warm`: the reusable warm Daytona environment fixture. */
daytonaWarm?: boolean;
/** `--daytona-inherit`: a Daytona environment that inherits the lifecycle and leaves lease reuse unset. */
daytonaInherit?: boolean;
/** `--k4-hold-max-seconds`: upper end of the seeded K4 hold (default 300 s, cap 600 s). */
k4HoldMaxMs?: number;
/** `--k15-hold-seconds`: the seeded K15 outage range (redeploy.ts); absent restarts at once. With `--k15-overlap-seconds`, the overlap range. */
k15HoldMs?: [number, number];
/** `--k15-overlap-seconds`: a K15 boots the next deploy beside the old server and SIGTERMs the old one after the seeded overlap (redeploy.ts). */
k15Overlap?: boolean;
/** `--k15-fresh-home`: each deploy generation gets a new, empty instance home and no volume (redeploy.ts). */
k15FreshHome?: boolean;
ephemeralServer?: boolean;
runnerBindFault?: boolean;
/** `--faults K0`: cells with no fault, for workloads whose shape is the fault (W2, W2Q, W2C). */
baseline?: boolean;
/**
* `--fail-recovery-step <point>[:<nth>]` (recovery-step-faults.ts): once the wave's first restart fault fired, the
* nth hit of that crash point in the replacement throws its step's own error, once.
*/
failRecoveryStep?: string;
/** `--long-turn-minutes`: every task's first turn does this much real tool work before its workload (long-turn.ts). */
longTurnMinutes?: number;
/** `--sandbox-watch-seconds`: on Daytona, capture every wave sandbox on this clock (daytona-runner-logs.ts class R probe). */
sandboxWatchMs?: number;
/** `--cr1-hold-seconds`: the CR1 outage range (class-r.ts); absent on a CR1 wave means 360 s. */
cr1HoldMs?: [number, number];
/** `--cr2-hold-seconds`: a K15 outage around CR2's provider kill; absent or 0 keeps the controller up. */
cr2HoldMs?: [number, number];
/** `--cr3-stall-seconds`: how long CR3 stops the Daytona plugin worker; absent means 45 s. */
cr3StallMs?: [number, number];
/** `--tm1-hold-seconds`: the seeded TM1 stall range (time-faults-config.ts); absent uses 95–150 s. */
tm1HoldMs?: [number, number];
faultAfterMs?: number;
/**
* Known product signatures this wave deliberately masks. "rc06" keeps the
* company budget amount but disables its hard stop, so a stuck
* accounting row (RC-06) cannot wedge every later task. Spend is still
* recorded per wave. Absent on plans written before masks existed.
*/
masks?: string[];
/**
* K13 restart storm: fire faults drawn from `faults` every 2–4 minutes for
* this many minutes, keeping `concurrency` tasks in flight, then drain.
*/
stormMinutes?: number;
/**
* Runner lifecycle forced onto every seeded agent (`--lifecycle`). Absent
* keeps the catalog profile's own setting.
*/
lifecycle?: "per_turn" | "warm" | "default";
/**
* Crash-point suite (`--crash-point <name>[:<nth>][,...]`): no external
* fault is scheduled. One server serves every listed point in turn: each
* point's batch of tasks launches, then the point is armed and the server
* SIGKILLs itself at its nth hit and is restarted like K1.
*/
crashPoint?: string;
/** Crash-point suite: hold the control plane down this long after each self-kill (K4-style). */
crashPointHoldMs?: number;
/**
* K14 upgrade mid-run (`--upgrade-from <dir>`): an older, prepared checkout. The wave's server starts
* from it; the first K14 stops it gracefully and starts the current checkout on the same database
* and instance directory. Later K14 faults restart the current build, like K2.
*/
upgradeFrom?: string;
workspaceRepo?: string;
storage?: "s3";
storageFaults?: Array<import("./storage-chaos.js").StorageFaultKind>;
storageFaultMs?: [number, number];
storageWindows?: import("./storage-chaos.js").StorageFaultWindow[];
storageTrigger?: import("./storage-trigger.js").StorageTriggerTarget;
nativeTrigger?: import("./native-trigger.js").NativeTriggerSpec;
workspaceCommit?: string;
workspaceGitUrl?: string;
/** Provider message faults from `--faults` (PV1 delay, PV2 drop, PV3 duplicate); see provider-faults.ts. */
providerFaults?: ProviderFaultKind[];
/** Sandbox disk-full faults from `--faults` (DF1); see disk-full-faults.ts. */
diskFullFaults?: DiskFullFaultKind[];
/** Session faults from `--faults` (SR1); see session-faults.ts. */
sessionFaults?: Array<import("./session-faults.js").SessionFaultKind>;
providerFaultMessage?: ProviderMessage;
/** `--provider-fault-message S1=interrupt_ack,...`: a workload's own message, over `providerFaultMessage`. */
providerFaultMessageByWorkload?: Partial<Record<WorkloadId, ProviderMessage>>;
/** PV1 delays, cycled across tasks (`--provider-fault-delays`). */
providerFaultDelaysMs?: number[];
/** `--db-pressure <writers>`: harness-owned writers touching the wave's running run rows back to back (contention.ts). */
dbPressure?: number;
/** `--matrix`: one task per workload × adapter × (K0, or each fault at each of the workload's triggers). */
matrix?: boolean;
/** `--server-from <dir>`: every server of the wave runs from this prepared checkout (for example a pre-fix commit). */
serverFrom?: string;
/** TM2/TM3: how far every database timestamp is shifted back (`--time-age-hours`, default 26 h). */
timeAgeMs?: number;
/** `--provider-fault-restart K1`: each PV1/PV2 task also restarts the control plane after its fault applied. */
providerFaultRestart?: ProviderRestartKind;
/** `--provider-fault-restart K2,K4,K1`: restart kinds cycled across PV1/PV2 tasks (the first is `providerFaultRestart`). */
providerFaultRestartKinds?: ProviderRestartKind[];
providerFaultRestartAfterMs?: number;
/** RR1–RR3: the gap from a replacement's health to the next stop (`--rr-gap-seconds`); absent keeps each kind's default. */
rrGapMs?: [number, number];
/** RR1–RR3 (`--rr-stop-on recovering`): stop each replacement on its first `recovering` status instead of after a gap. */
rrStopOn?: "recovering";
}
export interface StormFault {
kind: FaultKind;
holdMs: number;
/** Wait this long after the previous storm fault settled (or the storm started). */
gapMs: number;
}
export interface StormPlan {
durationMs: number;
faults: StormFault[];
}
export interface TaskPlan {
taskId: string;
index: number;
workload: WorkloadId;
adapter: AdapterId;
fault: TaskFaultKind;
/** Fire the fault this long after the task's issue is created. */
fireAfterMs: number;
/** Outage hold between stop and restart. 0 for K1/K2. */
holdMs: number;
trigger?: TriggerId;
triggerDelayMs?: number;
/** C1 only: post the follow-up comment this long after creation. */
followUpAfterMs: number | null;
/** Unique marker used by the deterministic T-oracle. */
nonce: string;
/**
* Replay only: every fault that crossed this task in the source wave, with
* its offset from the task's launch. Replaces the single planned fault.
*/
replayFaults?: ScheduledFault[];
/** PV1–PV3 tasks: the provider message rule armed for this task's nonce. */
providerFault?: ProviderFaultRule;
/** RR1–RR3: the seeded restarts after the first kill. */
restartSteps?: RestartStep[];
/** `--long-turn-minutes`: the first turn's tool work before the workload (long-turn.ts). */
longTurnMinutes?: number;
}
export interface ScheduledFault {
kind: FaultKind; holdMs: number;
atMs: number;
trigger?: TriggerId;
delayMs?: number;
/** RR1–RR3: the restarts after the first kill. */
restartSteps?: RestartStep[];
}
export interface WavePlan {
schema: "paperclip.restart-chaos.wave-plan/v1";
waveId: string;
seed: number;
createdAt: string;
options: WaveOptions;
tasks: TaskPlan[];
/** K13 only: the seeded storm fault sequence. */
storm?: StormPlan;
/** Present when this wave replays one task of an earlier wave. */
replayOf?: { waveId: string; taskId: string };
}
export type Verdict = "pass" | "fail" | "n/a";
export interface RuleResult { verdict: Verdict; detail: string }
export type RuleId = "R1" | "R2" | "R3" | "R4" | "R5" | "R6";
export interface RunSample {
id: string;
status: string;
createdAt: string | null;
startedAt: string | null;
finishedAt: string | null;
errorCode: string | null;
/** Bounded code naming why a generic `errorCode` (such as `adapter_failed`) failed, e.g. `40001`. */
failureCause?: string | null;
retryOfRunId: string | null;
wakeCommentIds: string[];
livenessState: string | null;
lastEvidenceAt?: string | null;
/** A tool started with no matching end yet (a long `bench.sh` is quiet but live). */
toolInFlight?: boolean;
/** The run's recorded local runner process is still running with the same start time. */
processAlive?: boolean;
/** A provider turn started and has no turn terminal yet. */
providerTurnActive?: boolean;
}
export interface InterruptionSample {
state: string;
canRetry: boolean;
retryWait: boolean;
workspaceFailure: string | null;
observedRunId: string | null;
userAction: string | null;
}
/** One observer poll. Unreachable polls keep only the timestamp. */
export interface TaskSample {
at: number;
reachable: boolean;
serverReady: boolean;
issueStatus?: string;
interruption?: InterruptionSample | null;
activeRecoveryAction?: boolean;
/** Owner of the active recovery action; "board" means it waits on a person. */
recoveryOwner?: string | null;
executionBlocker?: boolean;
scheduledRetry?: boolean;
runs?: RunSample[];
}
export interface RetryAttempt {
at: number;
acceptedAt?: number;
observedRunId: string | null;
httpStatus: number | null;
accepted: boolean | null;
error: string | null;
via?: "ui" | "api";
bannerText?: string | null;
diagnostics?: { screenshot: string | null; dom: string | null } | null;
dismissedDialogs?: string[];
}
export const UI_RETRY_BUTTON_MISSING = "ui_retry_button_missing";
export const UI_RETRY_NO_REQUEST = "ui_retry_no_request";
export const UI_RETRY_BLOCKED_BY_MODAL = "ui_retry_blocked_by_modal";
export interface FaultReceipt {
faultId: string;
kind: FaultKind;
/** Task that scheduled the fault; every other running task is also exposed. */
requestedBy: string;
/** "none": MS1 sends no signal to the primary server. */
signal: "SIGKILL" | "SIGTERM" | "SIGSTOP" | "none";
holdMs: number;
requestedAt: number;
startedAt: number;
stoppedAt: number | null;
restartedAt: number | null;
healthyAt: number | null;
readyAt: number | null;
phases: Array<{ phase: string; at: number }>;
trigger?: TriggerId;
triggerDelayMs?: number;
loadAvg1m?: number;
crashLoop?: Array<{ pid: number | null; target: string; phase: string; at: number }>;
/** Crash-point suite: the server killed itself at this named point (no external signal). */
crashPoint?: { name: string; hit: number; pid: number; token?: string };
/** K14: the build the stopped server ran and the build its replacement runs. */
upgrade?: { fromRoot: string; fromSha: string | null; toRoot: string; toSha: string | null; switched: boolean };
/** K15: the deploy generation, emulated host name and deployment id before and after. */
redeploy?: ReturnType<typeof import("./redeploy.js").redeployReceipt>;
/** `--fail-recovery-step`: the step armed (fail mode) when this fault's replacement started. */
recoveryStep?: { spec: string; token: string };
/** TM1–TM3: what was frozen and which rows were aged. */
timeFault?: TimeFaultRecord;
/** RR1–RR3: every stop and replacement of the sequence, the first kill included. */
repeatedRestart?: RepeatedRestartRecord;
/** MS1–MS3: the second server and what happened to the first. */
multiServer?: import("./multi-server.js").MultiServerRecord;
/** CR1–CR3: the provider kill or plugin stall and the sandbox captures taken around the fault (class-r.ts). */
classR?: import("./class-r.js").ClassRRecord;
serverLogOffsetAtStart: number;
serverLogOffsetAtReady: number | null;
error: string | null;
}
export interface OracleArtifacts {
issueStatus: string | null;
documents: Record<string, string | null>;
agentComments: string[];
files: Record<string, string | null>;
commands: Record<string, { exitCode: number | null; stdout: string; stderr: string }>;
workspacePath: string | null;
/** B5: distinct `.bench/progress` values the harness itself saw while the task was live. */
benchProgressObserved?: number;
stop?: { runId: string | null; runStatus: string | null; runsStartedBeforeContinue: string[]; continuePosted: boolean; missed?: boolean };
children?: Array<{ id: string; title?: string | null; status: string | null; assigneeAgentId: string | null; documents: Record<string, string | null> }>;
childStalls?: Array<{ childIssueId: string; recoveryActionId: string | null }>;
stallWakes?: Array<{ stalledIssueId: string | null; recoveryActionId: string | null }>;
/** G1: approvals linked to the task. */
approvals?: Array<{ id: string; status: string; title: string | null }>;
/** G1 and H1: the board's actions and what the observer saw around them. */
governance?: GovernanceRecord;
/** Q1, Q4: the saved questions and the task's runs. */
question?: QuestionArtifacts;
/** PV1–PV3: the shim's log lines for this task. */
providerFaultEvents?: ProviderFaultEvent[];
/** L1: accounting and run-log evidence read from the wave database at rest. */
contention?: import("./contention.js").ContentionEvidence;
/** Q2, Q3: what the board did and what the task, its runs and the harness's app saw. */
journey?: import("./journeys.js").JourneyArtifacts;
/** W2, W2Q, W2C: the re-wake and the task's runs (rewake.ts). */
rewake?: import("./rewake.js").RewakeArtifacts;
/** `--long-turn-minutes`: the task's runs, for the harness's own long-turn evidence (long-turn.ts). */
runs?: import("./rewake.js").RewakeArtifacts["runs"];
/** H2: who owns the task at rest, and how its runs ended. */
handoff?: HandoffArtifacts;
}
/** H2: the task's owner and execution state at rest, and each run's terminal state. */
export interface HandoffArtifacts {
assigneeAgentId: string | null;
assigneeUserId: string | null;
createdByUserId: string | null;
executionBlocker: boolean;
activeRecoveryAction: boolean;
/** Null when the run history could not be read: the run checks then fail. */
runs: Array<{ id: string; status: string; errorCode: string | null }> | null;
}
/** What the board did for a governance task (G1, H1), and what the observer saw. */
export interface GovernanceRecord {
/** G1: first time the observer saw the task's approval pending. */
approvalSeenAt?: number | null;
approvalId?: string | null;
/** G1: the board's decision, made once the task's faults fired and the server was ready. */
decision?: { kind: "approve" | "reject"; at: number; httpStatus: number | null; error: string | null;
/** The approval was still pending when the board decided (it survived the restart). */
pendingWhenDecided: boolean; afterFault: boolean };
/** H1: the company budget the harness set before launch, in cents (observed spend + 1). */
budgetSetCents?: number | null;
budgetSetError?: string | null;
/** H1: first time the observer saw the company paused for its budget. */
budgetPausedAt?: number | null;
/** H1: runs of the task that started while the company was paused (must be none). */
runsStartedWhilePaused?: string[];
/** H1: the board's raise-and-resume, made once the task's faults fired and the pause held. */
resume?: { at: number; amount: number | null; incidentId: string | null; httpStatus: number | null; error: string | null; afterFault: boolean;
/** When the resume request returned: the pause window ends here. */
doneAt?: number };
/** H1: the company was active again at rest. */
companyActiveAtRest?: boolean;
/** H1: the pause cleared on its own (before the board resumed it). */
pauseLostAt?: number | null;
/** G1: subtasks named after the nonce that existed before the board decided (must be 0). */
childrenBeforeDecision?: number | null;
}
export interface OracleResult { verdict: Exclude<Verdict, "n/a">; checks: Array<{ name: string; ok: boolean; detail?: string }> }
export interface RestSweep {
activeRuns: string[];
runningWorkspaceOperations: string[];
activeRecoveryAction: boolean;
/** `--env daytona`: orphaned sandboxes and held leases belonging to this task's runs. */
daytonaLeaks?: string[];
error: string | null;
}
export interface TaskRecord {
plan: TaskPlan;
issueId: string | null;
identifier: string | null;
createdAt: number | null;
launchError: string | null;
samples: TaskSample[];
retry: RetryAttempt | null;
restAt: number | null;
finalStatus: string | null;
deadlineAt: number | null;
timedOut: boolean;
followUp: { at: number; sentAt?: number; postedAt?: number; commentId: string | null; error: string | null; duringRun?: boolean } | null;
stop?: { at: number; runId: string | null; httpStatus: number | null; error: string | null; missed?: boolean };
faultIds: string[];
sweep: RestSweep | null;
/** Workspace files present just before the first fault that hit this task (R2). */
preFaultFiles: string[] | null;
preFaultWorkspacePath?: string;
lastFaultAt?: number;
restSnapshotError?: string;
priorRetries?: RetryAttempt[];
childStalls?: Array<{ childIssueId: string; recoveryActionId: string | null }>;
childRetries?: Array<RetryAttempt & { childIssueId: string }>;
workspaceMovedTo?: string;
/** Set when the pre-fault snapshot could not read the workspace; R2 then fails visibly. */
preFaultSnapshotError?: string | null;
remoteWorkspace?: boolean;
remoteFiles?: Record<string, string | null>;
remoteFileErrors?: Record<string, string>;
/** Pre-fault files missing at rest; null when not checked. */
lostFiles: string[] | null;
/** B5: distinct bench progress values the observer read during polling. */
benchProgressSeen?: string[];
benchFirstSeenAt?: number | null;
/** C1: first time the observer saw the run that received the follow-up propose or accept its result. */
resultAcceptedAt?: number | null;
/** The run whose result armed the `result` trigger; the fault fires only while it is still running. */
resultRunId?: string | null;
triggerMissed?: TriggerId[];
governance?: GovernanceRecord;
/** K14 waves: the build serving when the task launched. */
serverBuildAtLaunch?: "upgrade_from" | "current";
/** Q1, Q4: the saved questions and the board's answers. */
question?: QuestionRecord;
/** Q2, Q3: the board's journey actions (journey-driver.ts). */
journey?: import("./journeys.js").JourneyRecord;
/** W2, W2Q, W2C: the first run, the board's wake and the run that served it (rewake.ts). */
rewake?: import("./rewake.js").RewakeRecord;
/** PV with `--provider-fault-restart`: when the shim logged the task's fault (arms the `provider` trigger). */
providerFaultAppliedAt?: number;
/** DF1: when the fault was armed and cleared, and what the board showed at the disk-full rest. */
diskFull?: DiskFullRecord;
}
export interface TaskGrade {
crossed: boolean;
crossingFaults: string[];
rules: Record<RuleId, RuleResult>;
T: OracleResult;
/** Universal checks U1–U5 (universal.ts); absent on grades written before them. */
universal?: import("./universal.js").UniversalGrade;
pass: boolean;
signature: string | null;
}