mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-15 18:56:05 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native agent run ends with a structured result. The status arbiter
turns that result into an issue status and next actions.
> - A result can report `yielded` with `continuation.kind:
"response_wake"`. This means "wake me when somebody responds".
> - Only a direct Board comment gave this kind a passive wait. Every
other `response_wake` became an immediate `issue_status_changed` wake to
the same agent.
> - The agent woke, saw nothing new, reported `response_wake` again, and
woke again. Nothing bounded this loop. It spent provider budget and its
back-to-back runs caused other failures on warm sandboxes.
> - This pull request parks such a result as a visible wait, and puts
declared continuation wakes under the rewake throttle as a backstop.
> - The benefit is that an agent that waits for input stops and costs
nothing until a real response arrives.
## Linked Issues or Issue Description
**What happened?**
Restart-chaos waves on master `683fd1e9a5` (no faults injected) showed
one task run 30 to 66 times, about 12 s apart. Example: wave
`w-20261111T190352-s5101`, task t02 (W2C), 65 runs in 13 minutes. Each
run reported `yielded` with `continuation.kind: "response_wake"` and the
summary "No changes made. This status-change wake does not trigger phase
2". Each run was woken by `issue_status_changed`, and each finalization
committed `live_continuation_registered` with an `enqueue_continuation`
effect of kind `response_wake`.
Root cause: `arbitrateNativeStatus()`
(`server/src/services/native-runtime/status-arbiter.ts`) only treats
`response_wake` as passive when `boardResponseWaitAuthorized` (from
`readNativeBoardResponseWaitSource`), external chat, or plan wait is
true. Otherwise it falls to the generic yielded branch, and
`status-decision-committer.ts` (`enqueue_continuation`) enqueues an
`issue_status_changed` wake to the same agent. `issue_status_changed` is
not in `THROTTLED_ISSUE_REWAKE_REASONS`, and the native committer writes
an `issue.updated` activity row with the run's id for every decision, so
the rewake throttle saw every native run as progress.
**Expected behavior**
A `response_wake` with no admitted response source parks the task. A
comment, an interaction answer or a status change by somebody else wakes
the agent. No path can loop without a bound.
**Steps to reproduce**
Run `tests/restart-chaos/main.ts --env local --adapters codex
--workloads W2,W2Q,W2C --concurrency 10 --tasks 10 --seed 5101
--lifecycle warm --triggers time`. W2/W2C tasks loop after phase 1.
## What Changed
- `status-arbiter.ts`: a `yielded` / `response_wake` result with no
admitted response source and no blocking remaining work now commits
`response_wake_waiting_for_input`, keeps the task `in_progress`, and has
no effects. Policy version is `phase6-v14`. Board, external chat, plan,
governance, dependency, pause and unfinished-work branches are unchanged
and still come first.
- `native-runtime/native-response-wake-wait.ts` (new) and
`recovery/service.ts`: recovery treats that applied decision as a
durable wait while it is the issue's latest decision, the issue is
`in_progress` with the same agent assignee, and no user comment arrived
after the run started. This applies to the stranded-work sweep and to
the paused-agent passive-wait check.
- `execution-projection.ts`: a committed run with this reason shows the
phase `waiting_for_answer`, label "Waiting for a reply", and a next
action that tells the user to comment or change the status. This shows
only while that decision is still the issue's latest decision and the
issue is `in_progress`; an older run shows "Completed".
- Backstop in `issue-rewake-throttle.ts` and `heartbeat/queue.ts`: an
`issue_status_changed` wake that carries a declared `continuationKind`
(the `enqueue_continuation` effect) is now a throttle candidate. The
native status committer's own `issue.updated` / status-decision activity
no longer counts as run progress or as new input. After two consecutive
successful runs with no issue-visible progress, such a wake waits for
the existing escalating cooldown (2 min, doubling, max 30 min). The
native status dispatcher (`dispatchNativeStatusWakeBatch`) does not drop
a throttled intent: it keeps the intent `queued`, reports the cooldown
deadline to the deferred-receipt worker, and dispatches it again when
the cooldown ends. Status wakes without a declared continuation (blocker
owner, notify owner, retry) are unchanged.
- Restart chaos: new enforced universal check U5 (`universal.ts`). A
task with more than 10 runs fails as `U5:run_budget_exceeded`
(`RESTART_CHAOS_U5_MAX_RUNS`). The README records the calibration.
- `doc/architecture/native-status-arbitration.md`: the decision table
and the wait rules.
## Verification
- RED on master (same reason as the waves):
`heartbeat-process-recovery.test.ts` "parks a native response_wake that
no response admitted…" fails with decision `reasonCode:
"live_continuation_registered"` and effect `{ kind:
"enqueue_continuation", continuationKind: "response_wake" }`.
`heartbeat-issue-rewake-throttle.test.ts` "throttles a
same_agent/response_wake/retry continuation…" fails 3/3 because the wake
is admitted.
- GREEN: the integration test finalizes the run, dispatches native
status wakes and runs the stranded-work sweep three times. It asserts
one run total, one wake request, no recovery action, the task
`in_progress`, and the projection "Waiting for a reply".
- `heartbeat-process-recovery.test.ts` "delays a no-progress native
continuation until the rewake cooldown…" finalizes a real `same_agent`
continuation after a no-progress run, dispatches twice (one throttle
receipt, intent stays `queued`, deadline equals `nextAllowedAt`), then
dispatches after the cooldown and gets the run. It fails when the
dispatcher drops the throttled intent.
- Focused suites pass: `heartbeat-process-recovery`,
`heartbeat-issue-rewake-throttle`, `status-arbiter`,
`native-status-arbiter-corpus`, `runtime-mode`, `execution-projection`,
`chat-run-milestone-work`, `heartbeat-dependency-scheduling`,
`heartbeat-run-summary` (904 tests), and all restart-chaos unit tests
(584 tests). `pnpm --filter @paperclipai/server typecheck` passes.
- Universal audit over the 17 saved waves of 2026-11-11: U5 flags 12
tasks, all looping tasks (12 to 66 runs), and 0 tasks on all-pass waves.
- Chaos rerun of local warm seed 5101 (W2, W2Q, W2C): result is added in
a comment on this PR.
## Risks
- Behavioral shift: a model that reports `response_wake` to mean
"continue now" no longer gets an immediate run. The task waits visibly
for a reply. The runner prompt already describes `response_wake` as a
real response wait, and `same_agent` stays available for server-owned
continuations.
- The throttle now applies to native runs. A chain of declared
continuations with no issue-visible output (no comment, document, work
product or other activity) waits for the cooldown after the second such
run. `same_agent` continuations are server-owned (repository
preparation), so this affects only repeated no-output chains.
- Old saved state: decisions committed under `phase6-v13` keep their
recorded effects. A run that finalizes after deploy uses the new policy.
No migration.
- Budget hard-stop behavior is unchanged.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
218 lines
15 KiB
TypeScript
218 lines
15 KiB
TypeScript
import { describe, expect, it } from "vitest";
|
|
import { gradeTask } from "./grader.js";
|
|
import type { OracleArtifacts, RunSample, TaskRecord, TaskSample } from "./types.js";
|
|
import { bareCauseMessage, DEFAULT_UNIVERSAL_BOUNDS as BOUNDS, diagnosableFailureCause, gradeU1, gradeU2, gradeU3, gradeU4, gradeU5, gradeUniversal, universalBounds } from "./universal.js";
|
|
import { workloadDefinitions } from "./workloads.js";
|
|
|
|
const T0 = Date.parse("2026-11-09T12:00:00Z");
|
|
const iso = (offsetMs: number) => new Date(T0 + offsetMs).toISOString();
|
|
const SEC = 1_000;
|
|
|
|
function run(id: string, status: string, created: number, finished: number | null = null, extra: Partial<RunSample> = {}): RunSample {
|
|
return { id, status, createdAt: iso(created), startedAt: status === "queued" ? null : iso(created + 500),
|
|
finishedAt: finished === null ? null : iso(finished), errorCode: null, retryOfRunId: null, wakeCommentIds: [], livenessState: null,
|
|
lastEvidenceAt: status === "running" ? iso(created + 500) : null, ...extra };
|
|
}
|
|
function sample(at: number, issueStatus: string, runs: RunSample[], extra: Partial<TaskSample> = {}): TaskSample {
|
|
return { at: T0 + at, reachable: true, serverReady: true, issueStatus, interruption: null, activeRecoveryAction: false,
|
|
executionBlocker: false, scheduledRetry: false, runs, ...extra };
|
|
}
|
|
/** A poll every 3 s from `from` to `to` (ms), built by `make`. */
|
|
function polls(from: number, to: number, make: (at: number) => TaskSample): TaskSample[] {
|
|
const out: TaskSample[] = [];
|
|
for (let at = from; at <= to; at += 3 * SEC) out.push(make(at));
|
|
return out;
|
|
}
|
|
function record(samples: TaskSample[], extra: Partial<TaskRecord> = {}): TaskRecord {
|
|
return {
|
|
plan: { taskId: "t01", index: 0, workload: "A1", adapter: "codex", fault: "K1", fireAfterMs: 20_000, holdMs: 0, followUpAfterMs: null, nonce: "abcd1234" },
|
|
issueId: "issue-1", identifier: "CHA-1", createdAt: T0, launchError: null, samples, retry: null, restAt: samples.at(-1)?.at ?? null,
|
|
finalStatus: "done", deadlineAt: T0 + 3_600_000, timedOut: false, followUp: null, faultIds: [],
|
|
sweep: { activeRuns: [], runningWorkspaceOperations: [], activeRecoveryAction: false, error: null },
|
|
preFaultFiles: null, lostFiles: null, ...extra,
|
|
};
|
|
}
|
|
const retryable = { state: "failed", canRetry: true, retryWait: false, workspaceFailure: null, observedRunId: "r1", userAction: null };
|
|
|
|
describe("U1 liveness", () => {
|
|
it("passes a task whose run keeps working, then finishes", () => {
|
|
const samples = [...polls(0, 120 * SEC, at => sample(at, "in_progress", [run("r1", "running", 0, null, { lastEvidenceAt: iso(at) })])),
|
|
sample(123 * SEC, "done", [run("r1", "succeeded", 0, 121 * SEC)])];
|
|
expect(gradeU1(record(samples), BOUNDS).verdict).toBe("pass");
|
|
});
|
|
|
|
it("fails a run that stays queued past the queue bound with nothing else for the user (the #20361 shape)", () => {
|
|
const samples = polls(0, 9 * 60 * SEC, at => sample(at, "todo", [run("r1", "queued", 0)]));
|
|
const result = gradeU1(record(samples, { restAt: null, finalStatus: null, timedOut: true }), BOUNDS);
|
|
expect(result.verdict).toBe("fail");
|
|
expect(result.detail).toMatch(/^no_owner_no_action .* in todo with r1 queued/);
|
|
});
|
|
|
|
it("treats a visible Retry or a waiting question as actionable", () => {
|
|
const failedRun = run("r1", "failed", 0, 5 * SEC, { errorCode: "process_lost" });
|
|
const withRetry = polls(0, 10 * 60 * SEC, at => sample(at, "in_progress", [failedRun], { interruption: retryable }));
|
|
expect(gradeU1(record(withRetry), BOUNDS).verdict).toBe("pass");
|
|
const question = { ...retryable, state: "waiting_for_user", canRetry: false };
|
|
expect(gradeU1(record(polls(0, 10 * 60 * SEC, at => sample(at, "in_progress", [failedRun], { interruption: question }))), BOUNDS).verdict).toBe("pass");
|
|
});
|
|
|
|
it("does not count outage time toward the bound", () => {
|
|
const quiet = (at: number) => sample(at, "in_progress", [run("r1", "failed", 0, 5 * SEC, { errorCode: "process_lost" })]);
|
|
const samples = [...polls(10 * SEC, 100 * SEC, quiet), { at: T0 + 101 * SEC, reachable: false, serverReady: false },
|
|
{ at: T0 + 400 * SEC, reachable: false, serverReady: false }, ...polls(401 * SEC, 480 * SEC, quiet)];
|
|
expect(gradeU1(record(samples, { restAt: null }), BOUNDS).verdict).toBe("pass");
|
|
});
|
|
|
|
it("does not let a retry row keep a stuck dispatched retry run alive", () => {
|
|
const samples = polls(0, 9 * 60 * SEC, at => sample(at, "todo", [run("r1", "queued", 0)], { scheduledRetry: true }));
|
|
expect(gradeU1(record(samples, { restAt: null }), BOUNDS).verdict).toBe("fail");
|
|
const pending = polls(0, 9 * 60 * SEC, at => sample(at, "todo", [run("r0", "failed", 0, SEC, { errorCode: "process_lost" })], { scheduledRetry: true }));
|
|
expect(gradeU1(record(pending, { restAt: null }), BOUNDS).verdict).toBe("pass");
|
|
});
|
|
|
|
it("fails an accepted Retry that waits on the system past the bound with no run (w-20261110T050818-s20386)", () => {
|
|
const waiting = { ...retryable, retryWait: true };
|
|
const failedRun = run("r1", "failed", 0, SEC, { errorCode: "adapter_failed" });
|
|
const samples = polls(0, 9 * 60 * SEC, at => sample(at, "blocked", [failedRun], { interruption: waiting, executionBlocker: true }));
|
|
expect(gradeU1(record(samples, { restAt: null }), BOUNDS)).toMatchObject({ verdict: "fail",
|
|
detail: expect.stringMatching(/accepted Retry waiting on the system/) });
|
|
// A wait that names a person's action (confirm the previous run stopped) is actionable.
|
|
const confirm = polls(0, 9 * 60 * SEC, at => sample(at, "blocked", [failedRun],
|
|
{ interruption: { ...waiting, userAction: "confirm_run_stopped" } }));
|
|
expect(gradeU1(record(confirm, { restAt: null }), BOUNDS).verdict).toBe("pass");
|
|
// A short wait that ends in a dispatched run is fine.
|
|
const short = [...polls(0, 60 * SEC, at => sample(at, "blocked", [failedRun], { interruption: waiting })),
|
|
sample(63 * SEC, "in_progress", [failedRun, run("r2", "running", 62 * SEC, null, { lastEvidenceAt: iso(63 * SEC) })])];
|
|
expect(gradeU1(record(short, { restAt: null }), BOUNDS).verdict).toBe("pass");
|
|
});
|
|
|
|
it("fails a silent running run once it stops counting as an owner", () => {
|
|
const samples = polls(0, 7 * 60 * SEC, at => sample(at, "in_progress", [run("r1", "running", 0)]));
|
|
expect(gradeU1(record(samples, { restAt: null }), BOUNDS).verdict).toBe("fail");
|
|
});
|
|
});
|
|
|
|
describe("U2 retry loops", () => {
|
|
const failed = (id: string, at: number, code = "adapter_failed") => run(id, "failed", at, at + 5 * SEC, { errorCode: code });
|
|
it("fails the same failure repeating without the task changing state", () => {
|
|
const runs: RunSample[] = [];
|
|
const samples = [0, 1, 2, 3].map(index => { runs.push(failed(`r${index}`, index * 20 * SEC)); return sample(index * 20 * SEC + 6 * SEC, "in_progress", [...runs]); });
|
|
expect(gradeU2(record(samples), BOUNDS)).toMatchObject({ verdict: "fail", detail: expect.stringMatching(/same_failure_repeated 4x failed:adapter_failed/) });
|
|
});
|
|
it("resets the streak when a run succeeds between failures", () => {
|
|
const succeeded = run("ok", "succeeded", 7 * SEC, 9 * SEC);
|
|
const samples = [sample(60 * SEC, "in_progress", [failed("r0", 0), succeeded, failed("r1", 10 * SEC), failed("r2", 20 * SEC), failed("r3", 30 * SEC)])];
|
|
expect(gradeU2(record(samples), BOUNDS)).toMatchObject({ verdict: "pass", metrics: { sameFailureRuns: 3 } });
|
|
});
|
|
it("resets the streak when the issue changed state between failures", () => {
|
|
const all = [failed("r0", 0), failed("r1", 20 * SEC), failed("r2", 40 * SEC), failed("r3", 60 * SEC)];
|
|
const samples = [sample(6 * SEC, "in_progress", all.slice(0, 1)), sample(26 * SEC, "in_progress", all.slice(0, 2)),
|
|
sample(30 * SEC, "blocked", all.slice(0, 2)), sample(46 * SEC, "in_progress", all.slice(0, 3)), sample(66 * SEC, "in_progress", all)];
|
|
expect(gradeU2(record(samples), BOUNDS)).toMatchObject({ verdict: "pass", metrics: { sameFailureRuns: 2 } });
|
|
});
|
|
it("passes when the failures differ or stay within the bound", () => {
|
|
const samples = [sample(10 * SEC, "in_progress", [failed("r1", 0), failed("r2", 6 * SEC, "process_lost"), failed("r3", 12 * SEC)])];
|
|
expect(gradeU2(record(samples), BOUNDS).verdict).toBe("pass");
|
|
});
|
|
});
|
|
|
|
describe("U3 diagnosable failures", () => {
|
|
it("flags a generic code, and a bare cause in the stored message", () => {
|
|
const samples = [sample(10 * SEC, "done", [run("r1", "failed", 0, 5 * SEC, { errorCode: "adapter_failed" })])];
|
|
expect(gradeU3(record(samples)).verdict).toBe("fail");
|
|
const replay = [sample(10 * SEC, "done", [run("r2", "failed", 0, 5 * SEC, { errorCode: "native_event_replay_conflict" })])];
|
|
expect(gradeU3(record(replay)).verdict).toBe("pass");
|
|
const stalled = "native_event_replay_conflict: runner_event_commit_stalled: runner events kept failing to commit without progress "
|
|
+ "(last cause: Error at source seq 12); automatic recovery is stopped.";
|
|
expect(gradeU3(record(replay), { runMessages: new Map([["r2", stalled]]), recoveryActions: [] }).verdict).toBe("fail");
|
|
const coded = stalled.replace("last cause: Error", "last cause: 40001");
|
|
expect(gradeU3(record(replay), { runMessages: new Map([["r2", coded]]), recoveryActions: [] }).verdict).toBe("pass");
|
|
});
|
|
it("treats a generic code with a recorded failure cause as diagnosable, and stays a warning", () => {
|
|
const failed = (failureCause?: string) => [sample(10 * SEC, "done", [run("r1", "failed", 0, 5 * SEC,
|
|
{ errorCode: "adapter_failed", ...(failureCause === undefined ? {} : { failureCause }) })])];
|
|
for (const cause of ["40001", "ECONNRESET", "provider_http_429", "exit_code_137", "TypeError"]) {
|
|
expect(gradeU3(record(failed(cause))).verdict).toBe("pass");
|
|
}
|
|
// An old row without the field, or a cause that is itself bare, still names nothing.
|
|
for (const cause of [undefined, "", "Error", "unknown_error", "adapter_reported_error"]) {
|
|
const result = gradeU3(record(failed(cause)));
|
|
expect(result.verdict).toBe("fail");
|
|
expect(result.detail).toContain("r1:code=adapter_failed");
|
|
}
|
|
expect(gradeUniversal(record(failed("Error"))).U3).toMatchObject({ verdict: "fail", enforced: false });
|
|
// The cause also excuses a bare stored message: "Adapter failed (40001)" names what went wrong.
|
|
expect(gradeU3(record(failed("40001")), { runMessages: new Map([["r1", "Error"]]), recoveryActions: [] }).verdict).toBe("pass");
|
|
});
|
|
it("recognizes diagnosable failure causes", () => {
|
|
expect(diagnosableFailureCause("40001")).toBe(true);
|
|
expect(diagnosableFailureCause("signal_SIGKILL")).toBe(true);
|
|
for (const cause of [null, undefined, "", " ", "Error", "unknown", "adapter_failed", "adapter_reported_error", "object"]) {
|
|
expect(diagnosableFailureCause(cause)).toBe(false);
|
|
}
|
|
});
|
|
it("flags a recovery action with an empty or bare cause", () => {
|
|
const db = { runMessages: new Map(), recoveryActions: [{ id: "a1", kind: "stalled_issue", cause: "Error", nextAction: "Retry the task" }] };
|
|
expect(gradeU3(record([sample(0, "done", [])]), db).verdict).toBe("fail");
|
|
});
|
|
it("recognizes bare causes", () => {
|
|
expect(bareCauseMessage("Error")).toBe(true);
|
|
expect(bareCauseMessage("")).toBe(true);
|
|
expect(bareCauseMessage("failed (cause: unknown_error)")).toBe(true);
|
|
expect(bareCauseMessage("Error at source seq 7")).toBe(true);
|
|
expect(bareCauseMessage("Process exited with code 1: ENOENT spawn codex")).toBe(false);
|
|
});
|
|
});
|
|
|
|
describe("U4 stale notices", () => {
|
|
it("fails a Retry notice still shown after a later run succeeded", () => {
|
|
const runs = [run("r1", "failed", 0, 5 * SEC, { errorCode: "process_lost" }), run("r2", "succeeded", 10 * SEC, 30 * SEC)];
|
|
const samples = polls(31 * SEC, 31 * SEC + 2 * 60 * SEC, at => sample(at, "done", runs, { interruption: retryable }));
|
|
expect(gradeU4(record(samples), BOUNDS)).toMatchObject({ verdict: "fail", detail: expect.stringMatching(/^stale_notice Retry offered after r2 succeeded/) });
|
|
});
|
|
it("passes when the notice clears", () => {
|
|
const runs = [run("r1", "failed", 0, 5 * SEC, { errorCode: "process_lost" }), run("r2", "succeeded", 10 * SEC, 30 * SEC)];
|
|
const samples = [sample(31 * SEC, "done", runs, { interruption: retryable }), sample(34 * SEC, "done", runs)];
|
|
expect(gradeU4(record(samples), BOUNDS).verdict).toBe("pass");
|
|
});
|
|
});
|
|
|
|
describe("U5 run budget", () => {
|
|
it("fails a task that keeps starting runs (the response_wake loop shape)", () => {
|
|
const runs = Array.from({ length: 65 }, (_, index) => run(`r${index}`, "succeeded", index * 12 * SEC, index * 12 * SEC + 10 * SEC));
|
|
const samples = [sample(800 * SEC, "in_progress", runs)];
|
|
expect(gradeU5(record(samples, { restAt: null, finalStatus: null, timedOut: true }), BOUNDS))
|
|
.toMatchObject({ verdict: "fail", detail: "run_budget_exceeded 65 runs > 10", metrics: { runs: 65 } });
|
|
});
|
|
it("passes a task within the budget and reads the bound from the environment", () => {
|
|
const runs = Array.from({ length: 5 }, (_, index) => run(`r${index}`, "succeeded", index * 20 * SEC, index * 20 * SEC + 10 * SEC));
|
|
expect(gradeU5(record([sample(120 * SEC, "done", runs)]), BOUNDS).verdict).toBe("pass");
|
|
expect(universalBounds({ RESTART_CHAOS_U5_MAX_RUNS: "4" }).maxRuns).toBe(4);
|
|
expect(gradeU5(record([sample(120 * SEC, "done", runs)]), { ...BOUNDS, maxRuns: 4 }).verdict).toBe("fail");
|
|
});
|
|
});
|
|
|
|
describe("universal checks in the task grade", () => {
|
|
const artifacts: OracleArtifacts = { issueStatus: "done", documents: { brief: "CHAOS-abcd1234\n- one\n- two\n- three\n" },
|
|
agentComments: [], files: {}, commands: {}, workspacePath: null };
|
|
it("fails a task on an enforced rule and keeps a warning rule from failing it", () => {
|
|
const runs = [run("r1", "failed", 0, 5 * SEC, { errorCode: "adapter_failed" }), run("r2", "succeeded", 10 * SEC, 30 * SEC)];
|
|
const stale = polls(31 * SEC, 31 * SEC + 2 * 60 * SEC, at => sample(at, "done", runs, { interruption: retryable }));
|
|
const grade = gradeTask(record(stale), [], 600_000, workloadDefinitions.A1.grade("abcd1234", artifacts));
|
|
expect(grade.universal?.U4).toMatchObject({ verdict: "fail", enforced: true });
|
|
expect(grade.universal?.U3).toMatchObject({ verdict: "fail", enforced: false });
|
|
expect(grade.pass).toBe(false);
|
|
expect(grade.signature).toMatch(/^U4:stale_notice:A1:K1$/);
|
|
|
|
const clean = [sample(31 * SEC, "done", runs)];
|
|
const warned = gradeTask(record(clean), [], 600_000, workloadDefinitions.A1.grade("abcd1234", artifacts));
|
|
expect(warned.universal?.U3.verdict).toBe("fail");
|
|
expect(warned.pass).toBe(true);
|
|
});
|
|
it("reads bounds from the environment", () => {
|
|
expect(universalBounds({ RESTART_CHAOS_U1_ORPHAN_SECONDS: "60", RESTART_CHAOS_U2_SAME_FAILURE_RUNS: "5" })).toMatchObject({ orphanMs: 60_000, sameFailureRuns: 5 });
|
|
expect(() => universalBounds({ RESTART_CHAOS_U1_ORPHAN_SECONDS: "-1" })).toThrow();
|
|
expect(gradeUniversal(record([sample(0, "done", [])])).U1.verdict).toBe("pass");
|
|
});
|
|
});
|