Files
PaperClipAI/tests/restart-chaos/universal.ts
T
DottaandClaude Opus 5.5 edd00526e7 fix(native): park a response_wake with no admitted response instead of self-waking (#20935)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native agent run ends with a structured result. The status arbiter
turns that result into an issue status and next actions.
> - A result can report `yielded` with `continuation.kind:
"response_wake"`. This means "wake me when somebody responds".
> - Only a direct Board comment gave this kind a passive wait. Every
other `response_wake` became an immediate `issue_status_changed` wake to
the same agent.
> - The agent woke, saw nothing new, reported `response_wake` again, and
woke again. Nothing bounded this loop. It spent provider budget and its
back-to-back runs caused other failures on warm sandboxes.
> - This pull request parks such a result as a visible wait, and puts
declared continuation wakes under the rewake throttle as a backstop.
> - The benefit is that an agent that waits for input stops and costs
nothing until a real response arrives.

## Linked Issues or Issue Description

**What happened?**

Restart-chaos waves on master `683fd1e9a5` (no faults injected) showed
one task run 30 to 66 times, about 12 s apart. Example: wave
`w-20261111T190352-s5101`, task t02 (W2C), 65 runs in 13 minutes. Each
run reported `yielded` with `continuation.kind: "response_wake"` and the
summary "No changes made. This status-change wake does not trigger phase
2". Each run was woken by `issue_status_changed`, and each finalization
committed `live_continuation_registered` with an `enqueue_continuation`
effect of kind `response_wake`.

Root cause: `arbitrateNativeStatus()`
(`server/src/services/native-runtime/status-arbiter.ts`) only treats
`response_wake` as passive when `boardResponseWaitAuthorized` (from
`readNativeBoardResponseWaitSource`), external chat, or plan wait is
true. Otherwise it falls to the generic yielded branch, and
`status-decision-committer.ts` (`enqueue_continuation`) enqueues an
`issue_status_changed` wake to the same agent. `issue_status_changed` is
not in `THROTTLED_ISSUE_REWAKE_REASONS`, and the native committer writes
an `issue.updated` activity row with the run's id for every decision, so
the rewake throttle saw every native run as progress.

**Expected behavior**

A `response_wake` with no admitted response source parks the task. A
comment, an interaction answer or a status change by somebody else wakes
the agent. No path can loop without a bound.

**Steps to reproduce**

Run `tests/restart-chaos/main.ts --env local --adapters codex
--workloads W2,W2Q,W2C --concurrency 10 --tasks 10 --seed 5101
--lifecycle warm --triggers time`. W2/W2C tasks loop after phase 1.

## What Changed

- `status-arbiter.ts`: a `yielded` / `response_wake` result with no
admitted response source and no blocking remaining work now commits
`response_wake_waiting_for_input`, keeps the task `in_progress`, and has
no effects. Policy version is `phase6-v14`. Board, external chat, plan,
governance, dependency, pause and unfinished-work branches are unchanged
and still come first.
- `native-runtime/native-response-wake-wait.ts` (new) and
`recovery/service.ts`: recovery treats that applied decision as a
durable wait while it is the issue's latest decision, the issue is
`in_progress` with the same agent assignee, and no user comment arrived
after the run started. This applies to the stranded-work sweep and to
the paused-agent passive-wait check.
- `execution-projection.ts`: a committed run with this reason shows the
phase `waiting_for_answer`, label "Waiting for a reply", and a next
action that tells the user to comment or change the status. This shows
only while that decision is still the issue's latest decision and the
issue is `in_progress`; an older run shows "Completed".
- Backstop in `issue-rewake-throttle.ts` and `heartbeat/queue.ts`: an
`issue_status_changed` wake that carries a declared `continuationKind`
(the `enqueue_continuation` effect) is now a throttle candidate. The
native status committer's own `issue.updated` / status-decision activity
no longer counts as run progress or as new input. After two consecutive
successful runs with no issue-visible progress, such a wake waits for
the existing escalating cooldown (2 min, doubling, max 30 min). The
native status dispatcher (`dispatchNativeStatusWakeBatch`) does not drop
a throttled intent: it keeps the intent `queued`, reports the cooldown
deadline to the deferred-receipt worker, and dispatches it again when
the cooldown ends. Status wakes without a declared continuation (blocker
owner, notify owner, retry) are unchanged.
- Restart chaos: new enforced universal check U5 (`universal.ts`). A
task with more than 10 runs fails as `U5:run_budget_exceeded`
(`RESTART_CHAOS_U5_MAX_RUNS`). The README records the calibration.
- `doc/architecture/native-status-arbitration.md`: the decision table
and the wait rules.

## Verification

- RED on master (same reason as the waves):
`heartbeat-process-recovery.test.ts` "parks a native response_wake that
no response admitted…" fails with decision `reasonCode:
"live_continuation_registered"` and effect `{ kind:
"enqueue_continuation", continuationKind: "response_wake" }`.
`heartbeat-issue-rewake-throttle.test.ts` "throttles a
same_agent/response_wake/retry continuation…" fails 3/3 because the wake
is admitted.
- GREEN: the integration test finalizes the run, dispatches native
status wakes and runs the stranded-work sweep three times. It asserts
one run total, one wake request, no recovery action, the task
`in_progress`, and the projection "Waiting for a reply".
- `heartbeat-process-recovery.test.ts` "delays a no-progress native
continuation until the rewake cooldown…" finalizes a real `same_agent`
continuation after a no-progress run, dispatches twice (one throttle
receipt, intent stays `queued`, deadline equals `nextAllowedAt`), then
dispatches after the cooldown and gets the run. It fails when the
dispatcher drops the throttled intent.
- Focused suites pass: `heartbeat-process-recovery`,
`heartbeat-issue-rewake-throttle`, `status-arbiter`,
`native-status-arbiter-corpus`, `runtime-mode`, `execution-projection`,
`chat-run-milestone-work`, `heartbeat-dependency-scheduling`,
`heartbeat-run-summary` (904 tests), and all restart-chaos unit tests
(584 tests). `pnpm --filter @paperclipai/server typecheck` passes.
- Universal audit over the 17 saved waves of 2026-11-11: U5 flags 12
tasks, all looping tasks (12 to 66 runs), and 0 tasks on all-pass waves.
- Chaos rerun of local warm seed 5101 (W2, W2Q, W2C): result is added in
a comment on this PR.

## Risks

- Behavioral shift: a model that reports `response_wake` to mean
"continue now" no longer gets an immediate run. The task waits visibly
for a reply. The runner prompt already describes `response_wake` as a
real response wait, and `same_agent` stays available for server-owned
continuations.
- The throttle now applies to native runs. A chain of declared
continuations with no issue-visible output (no comment, document, work
product or other activity) waits for the cooldown after the second such
run. `same_agent` continuations are server-owned (repository
preparation), so this affects only repeated no-output chains.
- Old saved state: decisions committed under `phase6-v13` keep their
recorded effects. A run that finalizes after deploy uses the new policy.
No migration.
- Budget hard-stop behavior is unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 15:40:44 -06:00

326 lines
20 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
/**
* Universal checks U1–U5: scenario-independent rules graded on every task of
* every wave, whatever its workload. Pure functions over observer samples (no
* I/O), so they also run over saved evidence (`universal-audit.ts`).
*
* U1 liveness every non-terminal task has a live owner or a visible, actionable item
* U2 retry loop the same failure does not repeat without the task changing state
* U3 diagnosable every failed run names a specific cause (its code, or a recorded failureCause)
* U4 stale notices no failure or Retry notice outlives later successful work
*
* Each rule is either `enforced` (a violation fails the task) or a warning
* (reported in the grade and ledger, never fails the task). The split comes
* from the false-positive audit over saved waves; see README "Universal checks".
*/
import type { RunSample, TaskRecord, TaskSample } from "./types.js";
export type UniversalId = "U1" | "U2" | "U3" | "U4" | "U5";
export interface UniversalResult { verdict: "pass" | "fail" | "n/a"; detail: string; enforced: boolean;
/** Measurements behind the verdict (milliseconds or counts), for bound tuning. */
metrics?: Record<string, number> }
export type UniversalGrade = Record<UniversalId, UniversalResult>;
export interface UniversalBounds {
/** U1: longest ready-server time a non-terminal task may spend with no live owner and no actionable item. */
orphanMs: number;
/** U1: longest ready-server time a run may stay queued without starting. */
queuedMs: number;
/** U1: a `running` run with no event, activity or start evidence for this long is not a live owner. */
silentRunMs: number;
/** U2: most consecutive failed runs with one failure code while the issue status stays the same. */
sameFailureRuns: number;
/** U4: longest a failure or Retry notice may remain after a later run succeeded. */
staleNoticeMs: number;
maxRuns: number;
}
/**
* Defaults from the 2026-11-09 audit of 79 saved waves (1,620 tasks; README
* "Universal checks"). Override with RESTART_CHAOS_U1_ORPHAN_SECONDS,
* RESTART_CHAOS_U1_QUEUED_SECONDS, RESTART_CHAOS_U2_SAME_FAILURE_RUNS and
* RESTART_CHAOS_U4_STALE_SECONDS.
*/
export const DEFAULT_UNIVERSAL_BOUNDS: UniversalBounds = {
orphanMs: 3 * 60_000,
queuedMs: 5 * 60_000,
silentRunMs: 3 * 60_000,
sameFailureRuns: 3,
staleNoticeMs: 60_000,
maxRuns: 10,
};
/** Rules that fail a task. The rest are warnings until the audit shows no false positive on known-good waves. */
export const ENFORCED_UNIVERSAL: ReadonlySet<UniversalId> = new Set<UniversalId>(["U1", "U2", "U4", "U5"]);
export function universalBounds(env: NodeJS.ProcessEnv = process.env): UniversalBounds {
const seconds = (name: string, fallback: number) => {
const raw = env[name];
if (raw === undefined || raw === "") return fallback;
const value = Number(raw);
if (!Number.isFinite(value) || value <= 0) throw new Error(`${name} must be a positive number`);
return value;
};
return {
orphanMs: seconds("RESTART_CHAOS_U1_ORPHAN_SECONDS", DEFAULT_UNIVERSAL_BOUNDS.orphanMs / 1000) * 1000,
queuedMs: seconds("RESTART_CHAOS_U1_QUEUED_SECONDS", DEFAULT_UNIVERSAL_BOUNDS.queuedMs / 1000) * 1000,
silentRunMs: DEFAULT_UNIVERSAL_BOUNDS.silentRunMs,
sameFailureRuns: Math.round(seconds("RESTART_CHAOS_U2_SAME_FAILURE_RUNS", DEFAULT_UNIVERSAL_BOUNDS.sameFailureRuns)),
staleNoticeMs: seconds("RESTART_CHAOS_U4_STALE_SECONDS", DEFAULT_UNIVERSAL_BOUNDS.staleNoticeMs / 1000) * 1000,
maxRuns: Math.round(seconds("RESTART_CHAOS_U5_MAX_RUNS", DEFAULT_UNIVERSAL_BOUNDS.maxRuns)),
};
}
const TERMINAL_ISSUE = new Set(["done", "cancelled"]);
const ms = (iso: string | null | undefined) => (iso ? Date.parse(iso) : NaN);
const seconds = (value: number) => `${Math.round(value / 1000)}s`;
const ready = (sample: TaskSample) => sample.reachable && sample.serverReady;
/**
* Codes that name no cause a person can act on. `adapter_failed` is the
* catch-all an adapter returns when it did not classify its failure; the board
* shows it as "adapter failed".
*/
export const GENERIC_FAILURE_CODES: ReadonlySet<string> = new Set(["", "error", "unknown", "unknown_error", "failed", "adapter_failed", "internal_error"]);
/** Server fallbacks that record only that the adapter gave no code: they name nothing specific. */
const VAGUE_FAILURE_CAUSES: ReadonlySet<string> = new Set(["adapter_reported_error", "object", "string", "undefined"]);
/** A recorded failureCause that names something: not empty, a bare class, `unknown_error` or a no-code fallback. */
export function diagnosableFailureCause(cause: string | null | undefined): boolean {
const text = (cause ?? "").trim();
return text !== "" && !GENERIC_FAILURE_CODES.has(text.toLowerCase()) && !VAGUE_FAILURE_CAUSES.has(text)
&& !bareCauseMessage(text);
}
const FAILED_RUN = new Set(["failed", "timed_out"]);
/** A message whose only cause is a bare exception class or "unknown" (the #20385 "last cause: Error" class). */
export function bareCauseMessage(message: string | null | undefined): boolean {
const text = (message ?? "").trim();
if (!text) return true;
if (/^(error|unknown|unknown_error|internal error)\.?$/i.test(text)) return true;
return /\b(?:last cause|cause|reason)\s*[:=]\s*\(?\s*(?:Error|unknown_error|unknown|undefined|null)\s*\)?(?:\s|[;,.)]|$)/.test(text)
|| /^Error at source seq \d+$/.test(text) || /:\s*Error at source seq \d+/.test(text);
}
/** Latest view of each run across the samples. */
function finalRuns(record: TaskRecord): Map<string, RunSample & { firstSeenAt: number; finishedSeenAt: number | null; statusWhenFinished: string | null }> {
const runs = new Map<string, RunSample & { firstSeenAt: number; finishedSeenAt: number | null; statusWhenFinished: string | null }>();
for (const sample of record.samples) {
if (!sample.reachable) continue;
for (const run of sample.runs ?? []) {
const prior = runs.get(run.id);
const finished = !["queued", "running", "scheduled_retry"].includes(run.status);
runs.set(run.id, { ...run, firstSeenAt: prior?.firstSeenAt ?? sample.at,
finishedSeenAt: prior?.finishedSeenAt ?? (finished ? sample.at : null),
statusWhenFinished: prior?.statusWhenFinished ?? (finished ? sample.issueStatus ?? null : null) });
}
}
return runs;
}
/** U1: what keeps this task moving at this sample, or null when nothing does. */
/** A retry timer or an accepted Retry that waits on the system, with no run dispatched yet. */
export function retryTimer(sample: TaskSample): boolean {
const dispatched = (sample.runs ?? []).some(run => ["queued", "running", "scheduled_retry"].includes(run.status));
return !dispatched && Boolean(sample.scheduledRetry || sample.interruption?.retryWait || sample.interruption?.state === "retry_scheduled");
}
/** `timerFor`: ready-server time an accepted Retry has waited on the system with no run dispatched. */
export function liveOwner(sample: TaskSample, queuedFor: ReadonlyMap<string, number>, bounds: UniversalBounds, timerFor = 0): string | null {
for (const run of sample.runs ?? []) {
if (run.status === "running") {
if (run.toolInFlight || run.processAlive) return `run ${run.id} running (tool or process alive)`;
const evidence = Math.max(ms(run.lastEvidenceAt) || 0, ms(run.startedAt) || 0);
if (!evidence || sample.at - evidence <= bounds.silentRunMs) return `run ${run.id} running`;
}
if (run.status === "queued" || run.status === "scheduled_retry") {
if ((queuedFor.get(run.id) ?? 0) <= bounds.queuedMs) return `run ${run.id} ${run.status}`;
}
}
// A retry row stays on the issue after its run is dispatched; once a run exists the run's own bounds above decide.
// A scheduled retry is a timer (samples carry no due time, so it is not bounded here). An accepted Retry
// that waits on the system (`retryWait`) is an owner only within the queue bound.
if (retryTimer(sample) && (!sample.interruption?.retryWait || timerFor <= bounds.queuedMs)) return "retry timer";
if (sample.activeRecoveryAction && sample.recoveryOwner !== "board") return "system recovery action";
return null;
}
/** U1: an item a person can see and act on (Retry, a question, an approval, a board-owned recovery, a named blocker). */
export function actionableItem(sample: TaskSample): string | null {
const interruption = sample.interruption;
// A named user action (for example `confirm_run_stopped`) is shown even while an accepted Retry waits.
if (interruption?.userAction) return `user action ${interruption.userAction}`;
// Otherwise an accepted Retry that waits on the system ("Retry is waiting for Paperclip to ...") leaves nothing
// for the person to do: the Retry button, the failed notice and the blocker describe work they already asked for.
if (interruption?.retryWait) return null;
if (interruption?.canRetry) return "Retry";
if (interruption && (interruption.state === "waiting_for_user" || interruption.state === "failed")) return `interruption ${interruption.state}`;
if (sample.activeRecoveryAction && sample.recoveryOwner === "board") return "board recovery action";
if (sample.executionBlocker) return "execution blocker";
// Blocked and in-review issues wait on another issue, an approval or a person by definition.
if (sample.issueStatus === "blocked" || sample.issueStatus === "in_review") return `status ${sample.issueStatus}`;
return null;
}
export function gradeU1(record: TaskRecord, bounds: UniversalBounds): Omit<UniversalResult, "enforced"> {
if (!record.samples.some(ready)) return { verdict: "n/a", detail: "no ready sample" };
// Ready-server time each run has spent queued (outages do not count).
const queuedFor = new Map<string, number>();
let orphanMs = 0, worstOrphan = 0, orphanFrom: number | null = null, lastAt: number | null = null;
let worstQueued = 0, worstQueuedRun = "";
let timerFor = 0, worstTimer = 0;
for (const sample of record.samples) {
// Outages pause the clock: a down server is the fault's time, not the task's.
if (!ready(sample)) { lastAt = null; continue; }
for (const run of sample.runs ?? []) {
if (run.status === "queued" || run.status === "scheduled_retry") {
const waited = (queuedFor.get(run.id) ?? 0) + (lastAt !== null && queuedFor.has(run.id) ? sample.at - lastAt : 0);
queuedFor.set(run.id, waited);
if (waited > worstQueued) { worstQueued = waited; worstQueuedRun = run.id; }
}
}
timerFor = retryTimer(sample) && sample.interruption?.retryWait ? timerFor + (lastAt !== null ? sample.at - lastAt : 0) : 0;
worstTimer = Math.max(worstTimer, timerFor);
if (TERMINAL_ISSUE.has(sample.issueStatus ?? "")) { orphanMs = 0; orphanFrom = null; lastAt = sample.at; continue; }
const owned = liveOwner(sample, queuedFor, bounds, timerFor) ?? actionableItem(sample);
if (owned) { orphanMs = 0; orphanFrom = null; }
else {
if (orphanFrom === null) orphanFrom = sample.at;
else if (lastAt !== null) orphanMs += sample.at - lastAt;
worstOrphan = Math.max(worstOrphan, orphanMs);
if (orphanMs > bounds.orphanMs) {
const queued = (sample.runs ?? []).filter(run => run.status === "queued" || run.status === "scheduled_retry");
return { verdict: "fail", metrics: { ownerlessMs: orphanMs, queuedMs: worstQueued, retryTimerMs: worstTimer }, detail: `no_owner_no_action ${seconds(orphanMs)} in ${sample.issueStatus ?? "unknown"}`
+ (queued.length ? ` with ${queued.map(run => `${run.id} queued ${seconds(queuedFor.get(run.id) ?? 0)}`).join(",")}` : "")
+ (timerFor ? ` after a retry wait of ${seconds(timerFor)}${sample.interruption?.retryWait ? " (accepted Retry waiting on the system)" : ""}` : "")
+ ` (interruption ${sample.interruption?.state ?? "none"})` };
}
}
lastAt = sample.at;
}
return { verdict: "pass", metrics: { ownerlessMs: worstOrphan, queuedMs: worstQueued, retryTimerMs: worstTimer }, detail: `longest ownerless ${seconds(worstOrphan)}; longest queued ${seconds(worstQueued)}${worstQueuedRun ? ` (${worstQueuedRun})` : ""}` };
}
/** The issue status seen between two failures (both finish polls included) was not one value throughout. */
function issueStatusChanged(record: TaskRecord, before: { finishedSeenAt: number | null; statusWhenFinished: string | null },
after: { finishedSeenAt: number | null; statusWhenFinished: string | null }): boolean {
if (before.statusWhenFinished !== after.statusWhenFinished) return true;
const [from, to] = [before.finishedSeenAt ?? 0, after.finishedSeenAt ?? Number.POSITIVE_INFINITY];
return record.samples.some(sample => sample.reachable && sample.at >= from && sample.at <= to && sample.issueStatus !== before.statusWhenFinished);
}
export function gradeU2(record: TaskRecord, bounds: UniversalBounds): Omit<UniversalResult, "enforced"> {
// Every finished run in creation order: a succeeded run between two failures is progress and ends the streak.
const finished = [...finalRuns(record).values()].filter(run => run.finishedSeenAt !== null)
.sort((a, b) => (ms(a.createdAt) || a.firstSeenAt) - (ms(b.createdAt) || b.firstSeenAt));
const isFailure = (run: RunSample) => FAILED_RUN.has(run.status) || run.status === "interrupted";
if (!finished.some(isFailure)) return { verdict: "n/a", detail: "no failed run" };
let streak = 0, worst = 0, worstCode = "";
let previous: (typeof finished)[number] | null = null;
for (const current of finished) {
if (!isFailure(current)) {
// A cancellation (for example a superseded continuation) is neither progress nor a repeat.
if (current.status !== "cancelled") { streak = 0; previous = null; }
continue;
}
const same = previous !== null && `${previous.status}:${previous.errorCode}` === `${current.status}:${current.errorCode}`
&& !issueStatusChanged(record, previous, current);
streak = same ? streak + 1 : 1;
previous = current;
if (streak > worst) { worst = streak; worstCode = `${current.status}:${current.errorCode ?? "none"}`; }
}
return worst > bounds.sameFailureRuns
? { verdict: "fail", metrics: { sameFailureRuns: worst }, detail: `same_failure_repeated ${worst}x ${worstCode} without a task state change` }
: { verdict: "pass", metrics: { sameFailureRuns: worst }, detail: `most repeats of one failure ${worst} (${worstCode})` };
}
/** What the wave database adds for U3: each run's error text and the task's recovery actions. */
export interface UniversalDbEvidence {
runMessages: ReadonlyMap<string, string | null>;
recoveryActions: ReadonlyArray<{ id: string; kind: string; cause: string | null; nextAction: string | null }>;
}
/** U3: failure codes from the samples; with database evidence, also run error text and recovery-action causes. */
export function gradeU3(record: TaskRecord, db?: UniversalDbEvidence): Omit<UniversalResult, "enforced"> {
const failed = [...finalRuns(record).values()].filter(run => FAILED_RUN.has(run.status));
const actions = db?.recoveryActions ?? [];
if (!failed.length && !actions.length) return { verdict: "n/a", detail: "no failed run or recovery action" };
const messages = db?.runMessages;
const vague: string[] = [];
for (const action of actions) {
if (bareCauseMessage(action.cause) || bareCauseMessage(action.nextAction)) {
vague.push(`recovery ${action.id}(${action.kind}) cause=${JSON.stringify((action.cause ?? "").slice(0, 80))} next=${JSON.stringify((action.nextAction ?? "").slice(0, 80))}`);
}
}
for (const run of failed) {
const code = run.errorCode ?? "";
// A generic code with a recorded cause (for example `adapter_failed` + `40001`) names what failed.
if (diagnosableFailureCause(run.failureCause)) continue;
if (GENERIC_FAILURE_CODES.has(code.trim().toLowerCase())) {
vague.push(`${run.id}:code=${code || "none"}${run.failureCause ? ` cause=${run.failureCause.slice(0, 80)}` : ""}`);
} else if (messages?.has(run.id) && bareCauseMessage(messages.get(run.id))) {
vague.push(`${run.id}:${code} message=${JSON.stringify((messages.get(run.id) ?? "").slice(0, 120))}`);
}
}
return vague.length
? { verdict: "fail", detail: `undiagnosable_failure ${vague.slice(0, 4).join("; ")}${vague.length > 4 ? ` (+${vague.length - 4})` : ""}` }
: { verdict: "pass", detail: `${failed.length} failed run(s) name a specific cause${messages ? " and message" : ""}; ${actions.length} recovery action(s)` };
}
/** A failure or Retry notice the board shows for this sample. */
function failureNotice(sample: TaskSample): string | null {
const interruption = sample.interruption;
if (!interruption) return null;
if (interruption.canRetry) return "Retry offered";
if (interruption.state === "failed") return "failed notice";
return null;
}
export function gradeU4(record: TaskRecord, bounds: UniversalBounds): Omit<UniversalResult, "enforced"> {
let staleFrom: number | null = null, worst = 0, what = "";
const latestFinish = new Map<string, RunSample>();
for (const sample of record.samples) {
if (!ready(sample)) continue;
for (const run of sample.runs ?? []) latestFinish.set(run.id, run);
const runs = [...latestFinish.values()].sort((a, b) => (ms(a.createdAt) || 0) - (ms(b.createdAt) || 0));
const latest = runs[runs.length - 1];
const active = runs.some(run => ["queued", "running", "scheduled_retry"].includes(run.status));
const succeededLast = latest?.status === "succeeded" && !active;
const notice = succeededLast ? failureNotice(sample) : null;
if (!notice) { staleFrom = null; continue; }
staleFrom ??= sample.at;
if (sample.at - staleFrom > worst) { worst = sample.at - staleFrom; what = `${notice} after ${latest!.id} succeeded (status ${sample.issueStatus ?? "unknown"})`; }
}
// A notice still shown at rest is stale however briefly it was observed.
const last = [...record.samples].reverse().find(ready);
const atRest = record.restAt !== null && staleFrom !== null && last !== undefined && failureNotice(last) !== null;
if (worst > bounds.staleNoticeMs || atRest) return { verdict: "fail", metrics: { staleNoticeMs: worst }, detail: `stale_notice ${what} for ${seconds(worst)}${atRest ? ", still shown at rest" : ""}` };
return { verdict: "pass", metrics: { staleNoticeMs: worst }, detail: worst ? `notice cleared ${seconds(worst)} after success` : "no notice after success" };
}
export function gradeU5(record: TaskRecord, bounds: UniversalBounds): Omit<UniversalResult, "enforced"> {
const runs = finalRuns(record).size;
return runs > bounds.maxRuns
? { verdict: "fail", metrics: { runs }, detail: `run_budget_exceeded ${runs} runs > ${bounds.maxRuns}` }
: { verdict: "pass", metrics: { runs }, detail: `${runs} runs` };
}
export function gradeUniversal(record: TaskRecord, options: { bounds?: UniversalBounds; db?: UniversalDbEvidence;
enforced?: ReadonlySet<UniversalId> } = {}): UniversalGrade {
const bounds = options.bounds ?? universalBounds();
const enforced = options.enforced ?? ENFORCED_UNIVERSAL;
const results = {
U1: gradeU1(record, bounds),
U2: gradeU2(record, bounds),
U3: gradeU3(record, options.db),
U4: gradeU4(record, bounds),
U5: gradeU5(record, bounds),
};
return Object.fromEntries(Object.entries(results).map(([id, result]) => [id, { ...result, enforced: enforced.has(id as UniversalId) }])) as UniversalGrade;
}
/** Violations that fail the task (enforced rules only). */
export const enforcedUniversalFailures = (grade: UniversalGrade) =>
(Object.entries(grade) as Array<[UniversalId, UniversalResult]>).filter(([, result]) => result.enforced && result.verdict === "fail");
/** Violations reported as warnings. */
export const universalWarnings = (grade: UniversalGrade) =>
(Object.entries(grade) as Array<[UniversalId, UniversalResult]>).filter(([, result]) => !result.enforced && result.verdict === "fail");