mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-31 22:24:16 +01:00
Operationalize Product E2E accounting, tracing, and scheduling (#17009)
## Thinking Path > - Paperclip manages agents and the work they perform. > - Product E2E tests exercise real agents, tools, setup and cleanup. > - Repeated runs need retained costs and traces, including failed attempts. > - A passing retry must not hide earlier spending or feedback. > - Recurring work also needs admission limits, private reports and failure detection. > - This change supplies those controls while keeping paid activation behind existing gates. ## Linked Issues or Issue Description Refs #16996 and #17158. This is the consolidated landing of the reviewed eval operations stack: #17019, #17024, #17025, #17027, #17030, #17033, #17035, #17038, #17043, #17044, #17048, #17050, #17051, #17092, #17096, #17100, #17107, #17109, #17112, #17115, #17117, #17121, #17123, #17127, #17130, #17133, #17134, #17137, #17140, #17143, #17154. The repository permits squash merges only. A local rehearsal showed that squashing each slice into master causes repeated conflicts in dependent slices. This PR preserves their commits and lands their combined implementation in one reviewed integration. Superseded slices will close after this merge. ## What Changed - Retain every attempt's cost receipt, including setup, failures, interruptions and retries. Separate reported amounts, estimates, missing costs and reserved capacity in reports and dashboards. - Add bounded request admission, exact model pricecards, gateway error accounting, cache policy and reconciliation evidence. Refuse unsupported routes and requests that cannot fit the allowance. - Integrate Braintrust gateway traces and OTLP correlation, with secret filtering and experiment imports. Record phase timing, including setup, feedback capture and cleanup. - Collect complaint and suggestion evidence after each attempt, including recovery after failures. Retain explicit review dispositions independently from the product grade. - Add gated daily and weekly selections, measured forecasts, setup/readiness checks, wall-time limits and retained monitor notifications. Include declared non-model overhead in admission while reporting the existing subscription separately. - Keep recurring history private, bound uploads and retention, and use isolated archive environments. Keep public publication an explicit operation. Report and publication jobs install only the committed lockfile, with scripts disabled and isolated stores; stale pins fail closed. - Integrate current master, preserve its Product E2E journey billing and privacy checks, and test that shared setup receipts do not double-count estimated agent usage. Preserve upstream fixture repairs. ## Verification - Complete integration baseline `0649a5ce7544ab20825c2183d2f421157db49f24` includes master `a3e1d915a7110a4e115b46396a3f79e6dfb4d05e` and every original stack head as ancestors. - On that integration baseline, `pnpm test:e2e:runner:unit`: 4,012 Vitest tests passed, two skipped, across 193 files; all 137 Node source-contract tests passed. - Focused billing and Product E2E journey contract checks: 24 tests passed. They cover estimates, unknown receipts, single-owner setup costs, browser-only execution and private publication guards. - Monitor and workflow wiring: all 96 Node tests passed. `node scripts/check-token-gates.mjs`: all gates clean. - `pnpm test:e2e:runner:typecheck` passes after the standard Runner TypeScript build refreshed local declarations. The first attempt rejected the new master `runner_prepared_launch` union member because this checkout retained an older generated declaration; the source union already contained that member. - Current head `492b0708a1c531a023d9ac10f0f3dc6383cc1754` removes dependency re-resolution from archive-authorized reporting jobs after the security scan flagged it. All 15 workflow-security tests pass after updating the old two-install assertion. - Hosted [CI run 39965460180](https://github.com/paperclipai/paperclip/actions/runs/39965460180) passes on the current repaired head, including typecheck, build, general tests, Runner checks, browser shards and canary dry run. Greptile is 5/5 on this exact head with no unresolved threads. The security scan passes after the fixed lockfile-resolution thread was resolved. - Paperclip Review returned a terminal neutral check but an incomplete 0-file assessment on the repaired head: its platform blocked both diff responses as prompt-injection content. This is not a successful code review. No bypass or fabricated pass was attempted; current-head Greptile, security and hosted CI are independent gates. - No paid test, reservation release or recurring-run activation accompanies this landing. Live route qualification and invoice reconciliation remain separate evidence. <details> <summary>Retained earlier validation failures</summary> Earlier broad local runs had two 15-second test timeouts (CLI poll cadence and Retry validation), one embedded-database setup timeout, and later PostgreSQL semaphore exhaustion. The exact timed-out tests passed separately without larger timeouts; the broad-run causes were not established. Those local failures remain distinct from subsequent hosted CI results. The independent #17158 fix records its Muse fixture collision, its UUID correction and hosted verification. Local Muse execution was also blocked by macOS embedded-PostgreSQL startup. No historical failure is treated as a successful paid qualification. </details> ## Risks - Admission is a local control, not a provider invoice cap. Missing provider receipts or resource charges retain unknown amounts and holds; estimates do not become billed dollars. - The bounded path admits only documented local suites and provider routes. Ordinary legacy or separate Runner protocol campaigns have different controls. Merging this code does not qualify additional routes. - Recurrence still needs the existing enable gates and current qualification/allowance evidence. Archive permissions and resources must match the documented private-environment contract. - Legacy reports may lack complete attempt or phase evidence. Reports preserve that limitation and do not infer zero usage from missing data. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The exact deployment variant and context-window size are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted checks; full-suite host limitation disclosed above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
4173059081
commit
3bd709df8b
141 files changed
+12644
-299
No files matched your search
@@ -0,0 +1,196 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import test from "node:test";
|
||||
import { acknowledgeEvalNotifications, expectedCampaign, inspectEvalSchedule, monitorTransition, runEvalMonitor, MONITORED_SCHEDULES } from "../eval-schedule-monitor.mjs";
|
||||
|
||||
const now = Date.parse("2026-10-25T13:00:00Z");
|
||||
const enabledAt = "2026-10-24T00:00:00Z";
|
||||
const variables = { total_count: 3, variables: [
|
||||
{ name: "RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED", value: "true" },
|
||||
{ name: "RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED", value: "true" },
|
||||
{ name: "RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED_AT", value: enabledAt },
|
||||
] };
|
||||
const workflow = { id: 17, path: ".github/workflows/runner-full-stack-e2e.yml", state: "active" };
|
||||
const run = (cadence, id, extra = {}) => ({ id, workflow_id: 17, run_attempt: 1, event: "schedule", status: "completed", conclusion: "success",
|
||||
display_title: `Runner Full-Stack E2E / ${MONITORED_SCHEDULES[cadence]}`,
|
||||
created_at: cadence === "daily" ? "2026-10-25T08:50:00Z" : "2026-10-25T09:50:00Z", ...extra });
|
||||
function fixture(overrides = {}) {
|
||||
const calls = [];
|
||||
const get = async endpoint => {
|
||||
calls.push(endpoint);
|
||||
if (endpoint.includes("/variables?")) return overrides.variables ?? structuredClone(variables);
|
||||
if (endpoint.endsWith("runner-full-stack-e2e.yml")) return overrides.workflow ?? workflow;
|
||||
if (endpoint.includes("/runs?")) {
|
||||
const workflow_runs = overrides.runs ?? [run("daily", 101), run("weekly", 102)];
|
||||
return { total_count: overrides.total_count ?? workflow_runs.length, workflow_runs };
|
||||
}
|
||||
if (endpoint.includes("/artifacts?")) {
|
||||
const id = Number(endpoint.match(/\/runs\/(\d+)\//)[1]);
|
||||
const artifacts = overrides.artifacts ?? [{ name: `runner-e2e-report-${id}-1`, expired: false, size_in_bytes: 100 }];
|
||||
return { artifacts, total_count: artifacts.length };
|
||||
}
|
||||
throw new Error("Unexpected read");
|
||||
};
|
||||
return { get, calls };
|
||||
}
|
||||
|
||||
test("UTC due windows honor grace, activation and the weekly boundary", () => {
|
||||
assert.equal(expectedCampaign("daily", now, Date.parse(enabledAt)), "2026-10-25T08:47:00.000Z");
|
||||
assert.equal(expectedCampaign("weekly", now, Date.parse(enabledAt)), "2026-10-25T09:47:00.000Z");
|
||||
assert.equal(expectedCampaign("weekly", Date.parse("2026-10-25T10:00:00Z"), Date.parse(enabledAt)), undefined);
|
||||
assert.equal(expectedCampaign("daily", now, now), undefined);
|
||||
assert.equal(expectedCampaign("daily", Date.parse("2026-11-01T10:17:00Z"), 0), "2026-11-01T08:47:00.000Z");
|
||||
});
|
||||
|
||||
test("disabled gates do not query or diagnose old paid campaigns", async () => {
|
||||
const f = fixture({ variables: { total_count: 0, variables: [] } });
|
||||
assert.equal((await inspectEvalSchedule({ ...f, now })).status, "disabled");
|
||||
assert.equal(f.calls.length, 1);
|
||||
});
|
||||
|
||||
for (const budgetGate of [undefined, "false"]) test(`an enabled legacy schedule with ${budgetGate ?? "missing"} budget gate alerts once`, async () => {
|
||||
const rows = [{ name: "RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED", value: "true" }];
|
||||
if (budgetGate !== undefined) rows.push({ name: "RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED", value: budgetGate });
|
||||
const f = fixture({ variables: { total_count: rows.length, variables: rows } });
|
||||
const inspection = await inspectEvalSchedule({ ...f, now });
|
||||
assert.equal(inspection.status, "configuration_incomplete");
|
||||
assert.equal(inspection.reason, "legacy_schedule_enabled_without_budget_gate");
|
||||
assert.equal(f.calls.length, 1);
|
||||
const first = monitorTransition(undefined, inspection);
|
||||
assert.equal(first.notifications.length, 1);
|
||||
assert.equal(first.notifications[0].kind, "problem");
|
||||
assert.deepEqual(monitorTransition(first, inspection).notifications, first.notifications);
|
||||
});
|
||||
|
||||
test("an explicitly disabled legacy gate stays quiet even if the new budget gate is enabled", async () => {
|
||||
const changed = structuredClone(variables); changed.variables[1].value = "false";
|
||||
const f = fixture({ variables: changed });
|
||||
const inspection = await inspectEvalSchedule({ ...f, now });
|
||||
assert.equal(inspection.status, "disabled"); assert.equal(f.calls.length, 1);
|
||||
assert.deepEqual(monitorTransition(undefined, inspection).notifications, []);
|
||||
});
|
||||
|
||||
test("missing activation time and a disabled workflow stay distinct from missing runs", async () => {
|
||||
const invalid = structuredClone(variables); invalid.variables[2].value = "not a date";
|
||||
assert.equal((await inspectEvalSchedule({ ...fixture({ variables: invalid }), now })).status, "configuration_incomplete");
|
||||
assert.equal((await inspectEvalSchedule({ ...fixture({ workflow: { ...workflow, state: "disabled_inactivity" } }), now })).status, "workflow_disabled");
|
||||
});
|
||||
|
||||
test("requires one completed scheduled campaign and its unexpired report for each cadence", async () => {
|
||||
const result = await inspectEvalSchedule({ ...fixture(), now });
|
||||
assert.deepEqual(result.observations.map(row => row.status), ["healthy", "healthy"]);
|
||||
assert.deepEqual(result.observations.map(row => row.runId), [101, 102]);
|
||||
assert.deepEqual(monitorTransition(undefined, result).notifications, []);
|
||||
});
|
||||
|
||||
for (const [name, overrides, status] of [
|
||||
["no campaign", { runs: [] }, "missing"],
|
||||
["manual run cannot substitute for schedule", { runs: [run("daily", 101, { event: "workflow_dispatch" })] }, "missing"],
|
||||
["failed campaign", { runs: [run("daily", 101, { conclusion: "failure" })] }, "failed"],
|
||||
["canceled campaign", { runs: [run("daily", 101, { conclusion: "cancelled" })] }, "failed"],
|
||||
["overdue queued campaign", { runs: [run("daily", 101, { status: "queued" })] }, "overdue"],
|
||||
["unexpected paid rerun", { runs: [run("daily", 101, { run_attempt: 2 })] }, "unexpected_retry"],
|
||||
["multiple matching runs", { runs: [run("daily", 101), run("daily", 103)] }, "ambiguous_runs"],
|
||||
["missing report", { artifacts: [] }, "missing_report"],
|
||||
["expired report", { artifacts: [{ name: "runner-e2e-report-101-1", expired: true, size_in_bytes: 100 }] }, "missing_report"],
|
||||
["empty report", { artifacts: [{ name: "runner-e2e-report-101-1", expired: false, size_in_bytes: 0 }] }, "missing_report"],
|
||||
]) test(name, async () => {
|
||||
const result = await inspectEvalSchedule({ ...fixture(overrides), now });
|
||||
assert.equal(result.observations[0].status, status);
|
||||
});
|
||||
|
||||
test("new-day early run does not collide with the prior day during grace", async () => {
|
||||
const runs = [run("daily", 100, { created_at: "2026-10-24T08:50:00Z" }), run("daily", 101)];
|
||||
const result = await inspectEvalSchedule({ ...fixture({ runs }), now: Date.parse("2026-10-25T09:00:00Z") });
|
||||
assert.equal(result.observations[0].runId, 100);
|
||||
assert.equal(result.observations[0].status, "healthy");
|
||||
});
|
||||
|
||||
test("running is quiet inside the runtime limit", async () => {
|
||||
const result = await inspectEvalSchedule({ ...fixture({ runs: [run("daily", 101, { status: "in_progress", created_at: "2026-10-25T10:00:00Z" })] }), now });
|
||||
assert.equal(result.observations[0].status, "running");
|
||||
});
|
||||
|
||||
test("API errors and incomplete lists cannot imply a missing or successful run", async () => {
|
||||
assert.equal((await inspectEvalSchedule({ ...fixture({ total_count: 101 }), now })).status, "api_unavailable");
|
||||
assert.equal((await inspectEvalSchedule({ get: async () => { throw new Error("private upstream body"); }, now })).status, "api_unavailable");
|
||||
});
|
||||
|
||||
test("alerts once per incident and recovery while keeping last observed run IDs through API failure", async () => {
|
||||
const failed = await inspectEvalSchedule({ ...fixture({ runs: [run("daily", 101, { conclusion: "failure" }), run("weekly", 102)] }), now });
|
||||
const first = monitorTransition(undefined, failed);
|
||||
assert.equal(first.notifications.length, 1);
|
||||
assert.deepEqual(monitorTransition(first, failed).notifications, first.notifications);
|
||||
const outage = { observedAt: new Date(now + 1000).toISOString(), status: "api_unavailable", observations: [] };
|
||||
const unavailable = monitorTransition({ ...first, notifications: [] }, outage);
|
||||
assert.equal(unavailable.lastSuccessfulInspectionAt, first.lastSuccessfulInspectionAt);
|
||||
assert.deepEqual(unavailable.lastInspectedRunIds, { daily: 101, weekly: 102 });
|
||||
assert.equal(unavailable.notifications[0].status, "api_unavailable");
|
||||
assert.deepEqual(monitorTransition(unavailable, outage).notifications, unavailable.notifications);
|
||||
const resumed = monitorTransition({ ...unavailable, notifications: [] }, failed);
|
||||
assert.deepEqual(resumed.notifications.map(row => [row.kind, row.cadence]), [["recovery", "monitor"]]);
|
||||
const healthy = monitorTransition({ ...resumed, notifications: [] }, await inspectEvalSchedule({ ...fixture(), now }));
|
||||
assert.deepEqual(healthy.notifications.map(row => [row.kind, row.cadence]), [["recovery", "daily"]]);
|
||||
});
|
||||
|
||||
test("persists private state across invocations and refuses corrupt history", async () => {
|
||||
const directory = await mkdtemp(path.join(os.tmpdir(), "eval-monitor-"));
|
||||
try {
|
||||
const file = path.join(directory, "state.json");
|
||||
const first = await runEvalMonitor(file, fixture({ runs: [] }).get, now);
|
||||
assert.equal(first.notifications.length, 2);
|
||||
const repeated = await runEvalMonitor(file, fixture({ runs: [] }).get, now);
|
||||
assert.deepEqual(repeated.notifications, first.notifications);
|
||||
assert.deepEqual(await acknowledgeEvalNotifications(file, [first.notifications[0].id]), { acknowledged: 1, pending: 1 });
|
||||
const afterPartialDelivery = await runEvalMonitor(file, fixture({ runs: [] }).get, now);
|
||||
assert.deepEqual(afterPartialDelivery.notifications, first.notifications.slice(1));
|
||||
assert.deepEqual(await acknowledgeEvalNotifications(file, first.notifications.map(row => row.id)), { acknowledged: 1, pending: 0 });
|
||||
assert.deepEqual((await runEvalMonitor(file, fixture({ runs: [] }).get, now)).notifications, []);
|
||||
assert.equal(JSON.parse(await readFile(file, "utf8")).repository, "paperclipai/paperclip");
|
||||
await writeFile(file, "bad history");
|
||||
await assert.rejects(runEvalMonitor(file, fixture().get, now));
|
||||
assert.equal(await readFile(file, "utf8"), "bad history");
|
||||
} finally { await rm(directory, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test("pending alerts survive later recovery and are acknowledged only after delivery", async () => {
|
||||
const directory = await mkdtemp(path.join(os.tmpdir(), "eval-monitor-outbox-"));
|
||||
try {
|
||||
const file = path.join(directory, "state.json");
|
||||
const lostOutput = await runEvalMonitor(file, fixture({ runs: [] }).get, now);
|
||||
const recovered = await runEvalMonitor(file, fixture().get, now + 1000);
|
||||
assert.deepEqual(recovered.notifications.slice(0, 2), lostOutput.notifications);
|
||||
assert.deepEqual(recovered.notifications.map(row => row.kind), ["problem", "problem", "recovery", "recovery"]);
|
||||
const unchanged = await runEvalMonitor(file, fixture().get, now + 2000);
|
||||
assert.deepEqual(unchanged.notifications, recovered.notifications);
|
||||
await assert.rejects(acknowledgeEvalNotifications(file, ["invalid"]));
|
||||
assert.deepEqual(await acknowledgeEvalNotifications(file, recovered.notifications.map(row => row.id)), { acknowledged: 4, pending: 0 });
|
||||
assert.deepEqual((await runEvalMonitor(file, fixture().get, now + 3000)).notifications, []);
|
||||
} finally { await rm(directory, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test("rejects valid JSON with malformed history without querying or overwriting it", async () => {
|
||||
const directory = await mkdtemp(path.join(os.tmpdir(), "eval-monitor-corrupt-"));
|
||||
try {
|
||||
const file = path.join(directory, "state.json");
|
||||
await runEvalMonitor(file, fixture().get, now);
|
||||
const valid = JSON.parse(await readFile(file, "utf8"));
|
||||
for (const malformed of [null, false, 0, [], {}, { ...valid, signals: [] }, { ...valid, signals: { daily: null } },
|
||||
{ ...valid, notifications: null }, { ...valid, notifications: [{ id: "invalid" }] }]) {
|
||||
const original = JSON.stringify(malformed);
|
||||
await writeFile(file, original);
|
||||
const f = fixture();
|
||||
await assert.rejects(runEvalMonitor(file, f.get, now), /Invalid retained monitor state/);
|
||||
assert.equal(f.calls.length, 0);
|
||||
assert.equal(await readFile(file, "utf8"), original);
|
||||
}
|
||||
} finally { await rm(directory, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test("workflow titles carry the literal schedule used to distinguish daily and weekly runs", async () => {
|
||||
const text = await readFile(new URL("../../.github/workflows/runner-full-stack-e2e.yml", import.meta.url), "utf8");
|
||||
assert.ok(text.includes("run-name: Runner Full-Stack E2E / ${{ github.event.schedule || inputs.cadence || 'manual' }}"));
|
||||
for (const cron of Object.values(MONITORED_SCHEDULES)) assert.ok(text.includes(`cron: "${cron}"`));
|
||||
});
|
||||
@@ -698,8 +698,22 @@ test("Runner eval workflows pin actions and gate paid live execution", () => {
|
||||
const crons = [...workflow.matchAll(/cron:\s*"([^"]+)"/g)].map(
|
||||
(match) => match[1],
|
||||
);
|
||||
assert.equal(crons.length, 1, `${name} must have one schedule`);
|
||||
assert.match(crons[0], /^\d{1,2} \d{1,2} \* \* 0$/);
|
||||
if (name === "runner-full-stack-e2e.yml") {
|
||||
assert.deepEqual(crons, ["47 8 * * *", "47 9 * * 0"]);
|
||||
assert.ok(workflow.includes("vars.RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED == 'true' && vars.RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED == 'true'"));
|
||||
assert.ok(workflow.includes("Prepare bounded campaign before credentials"));
|
||||
assert.ok(workflow.includes("--max-parallel 1 --max-automatic-retries 0"));
|
||||
const catalog = workflow.slice(workflow.indexOf(" catalog:"), workflow.indexOf(" daytona_image:"));
|
||||
assert.ok(catalog.includes("EVAL_OPERATING_OVERHEAD_JSON: ${{ vars.RUNNER_E2E_OPERATING_OVERHEAD_JSON }}"));
|
||||
assert.ok(catalog.includes("operating_overhead: ${{ steps.catalog.outputs.operating_overhead }}"));
|
||||
const prepare = workflow.slice(workflow.indexOf(" - name: Prepare bounded campaign before credentials"), workflow.indexOf(" - name: Run bounded campaign and retain feedback review"));
|
||||
assert.ok(prepare.includes("EVAL_OPERATING_OVERHEAD_JSON: ${{ needs.catalog.outputs.operating_overhead }}"));
|
||||
assert.ok(prepare.includes("EVAL_TEST_RUNNER: ${{ needs.authorize.outputs.test_runner }}"));
|
||||
assert.ok(!prepare.includes("vars.RUNNER_E2E_OPERATING_OVERHEAD_JSON"));
|
||||
} else {
|
||||
assert.equal(crons.length, 1, `${name} must have one schedule`);
|
||||
assert.match(crons[0], /^\d{1,2} \d{1,2} \* \* 0$/);
|
||||
}
|
||||
}
|
||||
|
||||
const chaosWorkflow = actionPinWorkflows[2];
|
||||
|
||||
@@ -0,0 +1,176 @@
|
||||
import { execFile } from "node:child_process";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { mkdir, readFile, rename, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { promisify } from "node:util";
|
||||
import { pathToFileURL } from "node:url";
|
||||
|
||||
export const MONITORED_REPOSITORY = "paperclipai/paperclip";
|
||||
export const MONITORED_WORKFLOW = "runner-full-stack-e2e.yml";
|
||||
export const MONITORED_SCHEDULES = { daily: "47 8 * * *", weekly: "47 9 * * 0" };
|
||||
const hour = 3_600_000;
|
||||
const prefix = `repos/${MONITORED_REPOSITORY}/actions`;
|
||||
const numericId = value => Number.isSafeInteger(value) && value > 0;
|
||||
|
||||
export function expectedCampaign(cadence, now, enabledAt) {
|
||||
if (!Object.hasOwn(MONITORED_SCHEDULES, cadence) || !Number.isFinite(now) || !Number.isFinite(enabledAt)) {
|
||||
throw new Error("Invalid monitor schedule input");
|
||||
}
|
||||
const due = new Date(now - 90 * 60_000);
|
||||
due.setUTCHours(cadence === "daily" ? 8 : 9, 47, 0, 0);
|
||||
if (due.getTime() > now - 90 * 60_000) due.setUTCDate(due.getUTCDate() - 1);
|
||||
if (cadence === "weekly") due.setUTCDate(due.getUTCDate() - due.getUTCDay());
|
||||
return due.getTime() >= enabledAt ? due.toISOString() : undefined;
|
||||
}
|
||||
|
||||
function signal(cadence, status, dueAt, runId) {
|
||||
return { cadence, status, ...(dueAt ? { dueAt } : {}), ...(runId ? {
|
||||
runId, url: `https://github.com/${MONITORED_REPOSITORY}/actions/runs/${runId}`,
|
||||
} : {}) };
|
||||
}
|
||||
|
||||
export async function inspectEvalSchedule({ get, now = Date.now() }) {
|
||||
const observations = [];
|
||||
try {
|
||||
const variables = await get(`${prefix}/variables?per_page=100`);
|
||||
if (!Array.isArray(variables.variables) || !Number.isSafeInteger(variables.total_count)
|
||||
|| variables.total_count !== variables.variables.length) throw new Error("Incomplete variables");
|
||||
const values = Object.fromEntries(variables.variables.map(row => [row.name, row.value]));
|
||||
if (values.RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED !== "true") {
|
||||
return { observedAt: new Date(now).toISOString(), status: "disabled", observations: [] };
|
||||
}
|
||||
if (values.RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED !== "true") {
|
||||
return { observedAt: new Date(now).toISOString(), status: "configuration_incomplete",
|
||||
reason: "legacy_schedule_enabled_without_budget_gate", observations: [] };
|
||||
}
|
||||
const enabledAt = Date.parse(values.RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED_AT ?? "");
|
||||
if (!Number.isFinite(enabledAt) || enabledAt > now) {
|
||||
return { observedAt: new Date(now).toISOString(), status: "configuration_incomplete", observations: [] };
|
||||
}
|
||||
const workflow = await get(`${prefix}/workflows/${MONITORED_WORKFLOW}`);
|
||||
if (workflow.path !== `.github/workflows/${MONITORED_WORKFLOW}` || !numericId(workflow.id)) throw new Error("Wrong workflow");
|
||||
if (workflow.state !== "active") return { observedAt: new Date(now).toISOString(), status: "workflow_disabled", observations: [] };
|
||||
const expected = Object.keys(MONITORED_SCHEDULES).map(cadence => ({ cadence, dueAt: expectedCampaign(cadence, now, enabledAt) }));
|
||||
const due = expected.filter(row => row.dueAt);
|
||||
if (!due.length) return { observedAt: new Date(now).toISOString(), status: "waiting", observations: [] };
|
||||
const since = due.map(row => row.dueAt).sort()[0];
|
||||
const runs = await get(`${prefix}/workflows/${MONITORED_WORKFLOW}/runs?event=schedule&per_page=100&created=${encodeURIComponent(`>=${since}`)}`);
|
||||
if (!Array.isArray(runs.workflow_runs) || !Number.isSafeInteger(runs.total_count)
|
||||
|| runs.total_count !== runs.workflow_runs.length) throw new Error("Incomplete runs");
|
||||
for (const { cadence, dueAt } of expected) {
|
||||
if (!dueAt) { observations.push(signal(cadence, "waiting")); continue; }
|
||||
const matches = runs.workflow_runs.filter(run => run.event === "schedule" && run.workflow_id === workflow.id
|
||||
&& run.display_title === `Runner Full-Stack E2E / ${MONITORED_SCHEDULES[cadence]}`
|
||||
&& Date.parse(run.created_at) >= Date.parse(dueAt) && Date.parse(run.created_at) <= now
|
||||
&& Date.parse(run.created_at) < Date.parse(dueAt) + (cadence === "daily" ? 24 : 168) * hour);
|
||||
if (!matches.length) { observations.push(signal(cadence, "missing", dueAt)); continue; }
|
||||
if (matches.length !== 1) { observations.push(signal(cadence, "ambiguous_runs", dueAt)); continue; }
|
||||
const run = matches[0];
|
||||
if (!numericId(run.id) || !numericId(run.run_attempt) || typeof run.status !== "string") throw new Error("Invalid run identity");
|
||||
if (run.run_attempt !== 1) { observations.push(signal(cadence, "unexpected_retry", dueAt, run.id)); continue; }
|
||||
if (run.status !== "completed") {
|
||||
observations.push(signal(cadence, now - Date.parse(run.created_at) > 4 * hour ? "overdue" : "running", dueAt, run.id));
|
||||
continue;
|
||||
}
|
||||
if (run.conclusion !== "success") { observations.push(signal(cadence, "failed", dueAt, run.id)); continue; }
|
||||
const artifacts = await get(`${prefix}/runs/${run.id}/artifacts?per_page=100`);
|
||||
if (!Array.isArray(artifacts.artifacts) || !Number.isSafeInteger(artifacts.total_count)
|
||||
|| artifacts.total_count !== artifacts.artifacts.length) throw new Error("Incomplete artifacts");
|
||||
const reports = artifacts.artifacts.filter(row => row.name === `runner-e2e-report-${run.id}-1`
|
||||
&& row.expired === false && Number.isSafeInteger(row.size_in_bytes) && row.size_in_bytes > 0);
|
||||
observations.push(signal(cadence, reports.length === 1 ? "healthy" : "missing_report", dueAt, run.id));
|
||||
}
|
||||
return { observedAt: new Date(now).toISOString(), status: "observed", observations };
|
||||
} catch {
|
||||
return { observedAt: new Date(now).toISOString(), status: "api_unavailable", observations };
|
||||
}
|
||||
}
|
||||
|
||||
const quiet = new Set(["disabled", "waiting", "running", "healthy"]);
|
||||
const statuses = new Set([...quiet, "observed", "configuration_incomplete", "workflow_disabled", "api_unavailable",
|
||||
"missing", "ambiguous_runs", "unexpected_retry", "overdue", "failed", "missing_report"]);
|
||||
const object = value => value !== null && typeof value === "object" && !Array.isArray(value);
|
||||
const timestamp = value => typeof value === "string" && Number.isFinite(Date.parse(value));
|
||||
const notificationId = value => typeof value === "string" && /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/.test(value);
|
||||
const signalKey = row => !quiet.has(row.status) ? JSON.stringify([row.status, row.dueAt ?? null, row.runId ?? null]) : row.status;
|
||||
const validSignal = row => object(row) && ["monitor", "daily", "weekly"].includes(row.cadence) && statuses.has(row.status)
|
||||
&& (row.dueAt === undefined || timestamp(row.dueAt)) && (row.runId === undefined || numericId(row.runId))
|
||||
&& (row.runId === undefined ? row.url === undefined : row.url === `https://github.com/${MONITORED_REPOSITORY}/actions/runs/${row.runId}`);
|
||||
|
||||
function validateState(previous) {
|
||||
if (!object(previous) || previous.version !== 1 || previous.repository !== MONITORED_REPOSITORY
|
||||
|| previous.workflow !== MONITORED_WORKFLOW || !object(previous.signals)
|
||||
|| !timestamp(previous.checkedAt) || !(previous.lastSuccessfulInspectionAt === null || timestamp(previous.lastSuccessfulInspectionAt))
|
||||
|| !object(previous.lastInspectedRunIds) || Object.entries(previous.lastInspectedRunIds).some(([cadence, id]) => !["daily", "weekly"].includes(cadence) || !numericId(id))
|
||||
|| Object.entries(previous.signals).some(([cadence, row]) => !validSignal(row) || row.cadence !== cadence
|
||||
|| row.issue !== !quiet.has(row.status) || row.key !== signalKey(row))
|
||||
|| !Array.isArray(previous.notifications) || previous.notifications.some(row => !validSignal(row)
|
||||
|| !notificationId(row.id) || !["problem", "recovery"].includes(row.kind))
|
||||
|| new Set(previous.notifications.map(row => row.id)).size !== previous.notifications.length) {
|
||||
throw new Error("Invalid retained monitor state; refusing to reset notification history");
|
||||
}
|
||||
}
|
||||
|
||||
export function monitorTransition(previous, inspection) {
|
||||
if (previous !== undefined) validateState(previous);
|
||||
const current = inspection.status === "observed" ? inspection.observations : [signal("monitor", inspection.status)];
|
||||
const signals = {}, notifications = [];
|
||||
const prior = previous?.signals ?? {};
|
||||
for (const row of current) {
|
||||
const issue = !quiet.has(row.status);
|
||||
const key = signalKey(row);
|
||||
signals[row.cadence] = { ...row, issue, key };
|
||||
if (issue && prior[row.cadence]?.key !== key) notifications.push({ kind: "problem", ...row });
|
||||
else if (!issue && prior[row.cadence]?.issue && row.status !== "disabled") notifications.push({ kind: "recovery", ...row });
|
||||
}
|
||||
if (inspection.status === "observed" && prior.monitor?.issue) notifications.push({ kind: "recovery", cadence: "monitor", status: "observed" });
|
||||
const readable = !["api_unavailable", "configuration_incomplete"].includes(inspection.status);
|
||||
return { version: 1, repository: MONITORED_REPOSITORY, workflow: MONITORED_WORKFLOW,
|
||||
checkedAt: inspection.observedAt,
|
||||
lastSuccessfulInspectionAt: readable ? inspection.observedAt : previous?.lastSuccessfulInspectionAt ?? null,
|
||||
lastInspectedRunIds: { ...previous?.lastInspectedRunIds,
|
||||
...Object.fromEntries(inspection.observations.filter(row => row.runId).map(row => [row.cadence, row.runId])) },
|
||||
signals: inspection.status === "observed" || quiet.has(inspection.status) ? signals : { ...prior, ...signals },
|
||||
notifications: [...previous?.notifications ?? [], ...notifications.map(row => ({ id: randomUUID(), ...row }))] };
|
||||
}
|
||||
|
||||
async function saveState(stateFile, next) {
|
||||
await mkdir(path.dirname(stateFile), { recursive: true, mode: 0o700 });
|
||||
await writeFile(`${stateFile}.pending`, JSON.stringify(next, null, 2) + "\n", { mode: 0o600 });
|
||||
await rename(`${stateFile}.pending`, stateFile);
|
||||
}
|
||||
|
||||
export async function acknowledgeEvalNotifications(stateFile, ids) {
|
||||
if (!Array.isArray(ids) || !ids.length || ids.some(id => !notificationId(id))) throw new Error("Invalid notification acknowledgments");
|
||||
const previous = JSON.parse(await readFile(stateFile, "utf8"));
|
||||
validateState(previous);
|
||||
const next = { ...previous, notifications: previous.notifications.filter(row => !ids.includes(row.id)) };
|
||||
await saveState(stateFile, next);
|
||||
return { acknowledged: previous.notifications.length - next.notifications.length, pending: next.notifications.length };
|
||||
}
|
||||
|
||||
export async function runEvalMonitor(stateFile, get, now = Date.now()) {
|
||||
let previous;
|
||||
try { previous = JSON.parse(await readFile(stateFile, "utf8")); validateState(previous); }
|
||||
catch (error) { if (error.code !== "ENOENT") throw error; }
|
||||
const inspection = await inspectEvalSchedule({ get, now });
|
||||
const next = monitorTransition(previous, inspection);
|
||||
await saveState(stateFile, next);
|
||||
return { ...inspection, lastSuccessfulInspectionAt: next.lastSuccessfulInspectionAt,
|
||||
lastInspectedRunIds: next.lastInspectedRunIds, notifications: next.notifications };
|
||||
}
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
if (process.argv.length !== 3 && !(process.argv[3] === "--ack" && process.argv.length >= 5)) {
|
||||
throw new Error("Usage: node scripts/eval-schedule-monitor.mjs <private-state-file> [--ack <notification-id> ...]");
|
||||
}
|
||||
const execute = promisify(execFile);
|
||||
const get = async endpoint => {
|
||||
if (!endpoint.startsWith(`${prefix}/`)) throw new Error("Unapproved monitor endpoint");
|
||||
const { stdout } = await execute("gh", ["api", "--method", "GET", endpoint], { timeout: 15_000, maxBuffer: 4 * 1024 * 1024 });
|
||||
return JSON.parse(stdout);
|
||||
};
|
||||
console.log(JSON.stringify(process.argv[3] === "--ack"
|
||||
? await acknowledgeEvalNotifications(path.resolve(process.argv[2]), process.argv.slice(4))
|
||||
: await runEvalMonitor(path.resolve(process.argv[2]), get), null, 2));
|
||||
}
|
||||
Reference in new issue
Block a user