test(restart-chaos): keep wake evidence and early run events, and word staging-mix rounds unambiguously (#21173)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness runs agent tasks across server swaps and
keeps evidence for each failure
> - Wave `w-20261112T181948-s7101` had two "run started after the task
was done" cases that could not be traced
> - The wave deleted its database before anyone saved which wake started
those runs, which server created them, or how the task status changed
> - The evidence bundle also kept only the last 200 events of a run, so
it dropped the first 291 events of a Retry run, including its wake
payload
> - The staging-mix wording made one Codex agent refuse to redo an
interrupted round, and the task blocked on a harness artefact
> - This pull request saves that evidence for every task, keeps a run's
first events, and rewords the round rules without changing the grade

## Linked Issues or Issue Description

Refs #21125, #21168.

**What existing behavior does this improve?**
Restart chaos evidence capture and the staging-mix task wording.

**Subsystem affected**
Test harness only: `tests/restart-chaos/`. No product code changes.

**Current behavior**
- The harness does not save a task's wake rows, run context snapshots or
status history before the wave database is deleted.
- `evidence.ts` keeps only the last 200 events per run.
- The task tells the agent "do not skip, repeat or shorten any" round,
and the follow-up says "do not restart any". An agent read that as
"never re-run an interrupted round".

**Proposed behavior**
- Save, for every task, the wake rows, run snapshots, status changes and
serving server generations.
- Keep a run's first 400 and last 200 events, with a marker between
them.
- Say plainly that every round must appear in the ledger exactly once,
that an interrupted round must be run again, and that a round already in
the ledger must never be run again.

**Reason and benefit**
Wake and run origins can be traced after the wave, and the wording no
longer creates failures that come only from the harness.

## What Changed

- New `wake-evidence.ts`, called once at the end of each wave while the
database still exists. It writes `wake-evidence/<task>.json` for every
task. Each file holds:
- for each run: its `context_snapshot`, invocation source, trigger and
wakeup request id;
- the `agent_wakeup_requests` rows for the issue and for each run:
reason, source, trigger detail, requesting actor, idempotency key,
payload and times;
  - the issue's status changes from `activity_log`.

Each wake and run names the server generations alive when it was created
(`servingAtCreate`): pid, boot id, epoch and process space from
`controller_instances`, plus the harness's primary or secondary role
during an overlap. Each run also names its owning controller (`owner`).
`wake-evidence/servers.json` lists every controller and harness server
window. Known secret values are redacted.
- `evidence.ts` keeps all of a run's events, or the first 400 and the
last 200 with an `omittedEvents` marker between them.
- `staging-mix.ts` only changes the wording; the grading is unchanged.
The new text:
- every round must appear in the ledger file exactly once, with the hash
the command printed;
- before a round, read the ledger; a round whose line is there is
finished and must never run again;
  - a round interrupted before its line was written must be run again.

  The follow-up comment says the same.
- `main.ts` gains one call; the file stays at 1,000 lines. README
updated.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
59 files, 711 tests pass. That includes a real embedded PostgreSQL test
that writes a wake, a run, a status change and a controller row, then
checks the evidence file.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`:
passes.
- Unit tests cover the serving-generation join during an overlap, the
per-issue join, the first 400 and last 200 events of a 1,291-event run,
and the new wording.
- Not run: a live wave with this change.

## Risks

- Low risk. Test harness only.
- Evidence capture is one set of reads per wave. A failure is recorded
as `wake_evidence_failed` and does not fail the wave.
- Known master CI failures, not touched here: the storage S3 multipart
throttle test and the Runner native-distribution-integrity test.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits). Extended reasoning was available.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-12 13:41:14 -06:00
1 parent 4363a3115a
commit edf7edac60
6 files changed
+354 -19

No files matched your search

+22 -4
View File
@@ -1906,10 +1906,28 @@ re-arms the task's own triggered faults instead of using their offsets.
- `ledger.jsonl`, `SUMMARY.md` — every graded task, per-cell pass rate, open signatures.
- `<wave>/plan.json`, `timeline.jsonl`, `server.log`, `faults.json`, `<task>.grade.json`.
- `<wave>/evidence/<task>/` on failure — run ids and statuses, last 200 events per
run, workspace operations, server log window around each fault, startup phase
timings, fault timeline, seed, git SHA and the replay command. Known provider
key values are redacted.
- `<wave>/evidence/<task>/` on failure — run ids and statuses, each run's events (all of them, or the first 400
and the last 200 with an `omittedEvents` marker between, so the wake payload and `turn.submitted` are always
kept), workspace operations, server log window around each fault, startup phase timings, fault timeline, seed,
git SHA and the replay command. Known provider key values are redacted.
- `<wave>/wake-evidence/<task>.json` for every task, read from the wave database before the instance is deleted.
It holds:
- each run's `context_snapshot`, invocation source and trigger;
- the `agent_wakeup_requests` row behind each run and every wake for the issue: reason, source, trigger detail,
requesting actor, idempotency key, payload and times;
- the issue's status changes from the activity log.
Each wake and run also names the server generations alive when it was created (`servingAtCreate`): the pid, boot
id, epoch and process space from `controller_instances`, and the harness's own primary or secondary role during an
overlap. The run's owning controller is listed as `owner`. `wake-evidence/servers.json` lists every controller and
harness server window.
- **Process exit:** a controller's lifetime also ends at the process exit the harness recorded, so a hard-killed
server is not listed as serving later work.
- **Aged waves:** in TM2/TM3 waves, the database times are put back at their real times before they are compared
with the harness clock.
- **Failed waves:** the capture also runs, once, on the failure path before PostgreSQL stops. If the task list does
not exist yet, it writes one `issue-<id>.json` per issue that has runs.
## Grading notes
+16 -8
View File
@@ -5,9 +5,12 @@ import { replayCommand } from "./config.js";
import type { TimelineEntry } from "./timeline.js";
import type { FaultReceipt, OracleArtifacts, TaskGrade, TaskRecord, WavePlan } from "./types.js";
export { captureWakeEvidence, wakeEvidenceOnce } from "./wake-evidence.js";
const LOG_BEFORE_BYTES = 128 * 1024;
const LOG_AFTER_BYTES = 512 * 1024;
const EVENTS_PER_RUN = 200;
export const EVENTS_HEAD = 400;
export const EVENTS_TAIL = 200;
const OBSERVER_POLL_LINE = /INFO\S*: \S*(?:GET|HEAD) \/(?:issues\/[0-9a-f-]+(?:\/runs)?|ready|liveness|health) 200/;
type Json = Record<string, any>;
@@ -24,19 +27,24 @@ export function knownSecretValues(env: NodeJS.ProcessEnv = process.env): string[
return Object.entries(env).filter(([key, value]) => value && /(?:KEY|TOKEN|SECRET|PASSWORD)$/.test(key)).map(([, value]) => value!);
}
async function lastEvents(api: ChaosApi, runId: string): Promise<Json[]> {
const kept: Json[] = [];
let afterSeq = 0;
export async function runEvents(api: ChaosApi, runId: string): Promise<Json[]> {
const head: Json[] = [], tail: Json[] = [];
let total = 0, afterSeq = 0, omittedFrom: unknown = null, omittedTo: unknown = null;
for (let page = 0; page < 200; page += 1) {
const rows = await api.get<Json[]>(`/api/heartbeat-runs/${runId}/events?afterSeq=${afterSeq}&limit=500`);
if (!rows?.length) break;
kept.push(...rows);
if (kept.length > EVENTS_PER_RUN) kept.splice(0, kept.length - EVENTS_PER_RUN);
for (const row of rows) {
total += 1;
if (head.length < EVENTS_HEAD) { head.push(row); continue; }
tail.push(row);
if (tail.length > EVENTS_TAIL) { const dropped = tail.shift()!; omittedFrom ??= dropped.seq ?? null; omittedTo = dropped.seq ?? null; }
}
const next = Number(rows[rows.length - 1]?.seq);
if (!Number.isFinite(next) || next <= afterSeq || rows.length < 500) break;
afterSeq = next;
}
return kept;
const omitted = total - head.length - tail.length;
return omitted > 0 ? [...head, { omittedEvents: omitted, fromSeq: omittedFrom, toSeq: omittedTo }, ...tail] : [...head, ...tail];
}
async function logWindow(serverLogPath: string, faults: readonly FaultReceipt[]): Promise<string> {
@@ -94,7 +102,7 @@ export async function writeEvidenceBundle(input: {
await attempt("runs", async () => { runs = await api.get<Json[]>(`/api/issues/${record.issueId}/runs`); await write("runs.json", runs); });
for (const run of runs) {
const runId = String(run.runId ?? run.id);
await attempt(`events ${runId}`, async () => write(`run-${runId}-events.json`, await lastEvents(api, runId)));
await attempt(`events ${runId}`, async () => write(`run-${runId}-events.json`, await runEvents(api, runId)));
await attempt(`workspace-operations ${runId}`, async () => write(`run-${runId}-workspace-operations.json`,
await api.get(`/api/heartbeat-runs/${runId}/workspace-operations`)));
}
+5 -5
View File
@@ -28,7 +28,7 @@ import { CREDENTIAL_FOR_ADAPTER, parseCommand, replayCommand, waveIdFor } from "
import { assertControllerMatchesImage, assertProviderPackMismatch, defaultDaytonaImageCacheRoot, prepareDaytonaImageArtifacts } from "./daytona-image.js";
import { PLUGIN_BUILD_STAMP, pluginBuildState } from "./plugin-build.js";
import { DaytonaWaveSweeper, leaksForTask, unattributedLeaks, type DaytonaSweepResult } from "./daytona-sweep.js";
import { knownSecretValues, redactKnownSecrets, writeEvidenceBundle } from "./evidence.js";
import { knownSecretValues, redactKnownSecrets, wakeEvidenceOnce, writeEvidenceBundle } from "./evidence.js";
import { gradeTask } from "./grader.js";
import { releaseInstance } from "../runner-e2e/instance-retention.js";
import { checkPlanInvariants, readUniversalDbEvidence, summarizeViolations, type InvariantCheck } from "./invariants.js";
@@ -320,8 +320,8 @@ async function main(): Promise<number> {
harnessError ??= error;
interrupted = true;
});
// Spend is read once per wave: after the tasks, or during cleanup when the wave stopped early.
let spendTarget: { api: ChaosApi; companyId: string } | null = null;
let captureEvidence: (() => Promise<void>) | null = null;
let spendRecorded = false;
const recordWaveSpend = async () => {
if (!spendTarget || spendRecorded) return;
@@ -463,6 +463,7 @@ async function main(): Promise<number> {
};
} });
spendTarget = { api, companyId: lab.companyId };
captureEvidence = wakeEvidenceOnce({ databaseUrl: waveDatabaseUrl, companyId: lab.companyId, waveDir, observers: () => launched, windows: supervisor.serverWindows, shifts: supervisor.agingShifts, redact: text => redactKnownSecrets(text, knownSecretValues()), note: (kind, data) => timeline.add(kind, data) });
const releaseSwap = plan.options.releaseSwap ? await ReleaseSwapWave.start({ plan, lab, api, timeline, supervisor, databaseUrl: waveDatabaseUrl(), cancelled: () => interrupted }) : null;
// --db-pressure: contend with the wave's running run rows from outside the server (contention.ts).
if (plan.options.dbPressure) {
@@ -782,8 +783,8 @@ async function main(): Promise<number> {
}
const attributed = new Set<string>();
const secrets = knownSecretValues();
// U3 and L1 read the wave database once, before the instance is removed.
const evidenceDb = createDb(waveDatabaseUrl());
await captureEvidence?.();
const universalByIssue = await readUniversalDbEvidence(evidenceDb, lab.companyId).catch(error => {
timeline.add("universal_db_evidence_failed", { error: String(error) }); return null; });
const allRunMessages = new Map([...(universalByIssue?.values() ?? [])].flatMap(entry => [...entry.runMessages]));
@@ -907,11 +908,10 @@ async function main(): Promise<number> {
sandboxWatchStopped = true;
await mix?.stop().catch(() => undefined);
await stopPressure().catch(error => timeline.add("db_pressure_stop_failed", { error: String(error) }));
// Best effort for an interrupted or failed wave; the server may already be down.
await captureEvidence?.();
if (supervisor.serverReady) await recordWaveSpend().catch(() => undefined);
await board?.close().catch(() => undefined);
await closeJourneyApps();
// A failure here (for example writing the report) must not skip stopping the controller and Postgres.
try { if (sweeper) try {
const owned = sweeper as DaytonaWaveSweeper;
// An interrupted wave still reports what it leaked before cleanup.
+5 -2
View File
@@ -313,8 +313,11 @@ export function roundCommand(nonce: string): string {
function roundsSteps(nonce: string, rounds: number): string[] {
return [
`Run this shell command ${rounds} times, as separate tool calls in the foreground, replacing every <N> with the round number 1 to ${rounds}`,
`(each takes about ${ROUND_SECONDS} seconds; do not run rounds in the background or in parallel, and do not skip, repeat or shorten any):`,
`(each takes about ${ROUND_SECONDS} seconds; run them one at a time in the foreground, never in the background or in parallel):`,
`\`${roundCommand(nonce)}\``,
`Every round from 1 to ${rounds} must end up in \`${ledgerFile(nonce)}\` exactly once, with the hash the command printed.`,
"Before each round, read the ledger file: if that round's ROUND line is already there, the round is finished, so never run it again.",
"If a round was interrupted (for example the tool call failed or your session restarted) before its ROUND line was written, you must run that round again from the start.",
];
}
@@ -355,7 +358,7 @@ export function stagingMixDescription(nonce: string, task: StagingMixTask, agent
export const followUpRequestId = (issueId: string, nonce: string) => stableRequestId(`followup-${issueId}-${nonce}`);
export function followUpComment(nonce: string): string {
return `Additional requirement: include the line \`${followUpMarker(nonce)}\` on its own line in this task's \`ledger\` document. Keep going with your rounds; do not restart any.`;
return `Additional requirement: include the line \`${followUpMarker(nonce)}\` on its own line in this task's \`ledger\` document. Keep going with your rounds: a round whose ROUND line is already in the ledger file is finished, and a round without one must still be run.`;
}
export interface LedgerRound { round: number; start: number; end: number; hash: string }
+150
View File
@@ -0,0 +1,150 @@
import { randomUUID } from "node:crypto";
import { mkdtemp, readFile, rm } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { activityLog, agentWakeupRequests, agents, companies, controllerInstances, createDb, heartbeatRuns, issues, type Db } from "../../packages/db/src/index.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../server/src/__tests__/helpers/embedded-postgres.js";
import { boundByProcessExit, captureWakeEvidence, servingAt, unshiftRows, wakeEvidenceForIssue, wakeEvidenceOnce, type ControllerRow } from "./wake-evidence.js";
import { runEvents } from "./evidence.js";
import { stagingMixDescription, followUpComment } from "./staging-mix.js";
import type { ChaosApi } from "./api.js";
const support = await getEmbeddedPostgresTestSupport();
const controllers: ControllerRow[] = [
{ bootId: "boot-a", epoch: 1, processSpace: "space-a", pid: 100, startedAt: 0, endedAt: 600_000 },
{ bootId: "boot-b", epoch: 2, processSpace: "space-b", pid: 200, startedAt: 500_000, endedAt: null },
];
describe("wake evidence", () => {
it("names every server generation alive when a wake or run was created, overlap included", () => {
const windows = [{ pid: 100, role: "primary" as const, spawnedAt: 0, exitedAt: 600_000, frozen: [] },
{ pid: 200, role: "secondary" as const, spawnedAt: 500_000, exitedAt: null, frozen: [] },
{ pid: 300, role: "primary" as const, spawnedAt: 700_000, exitedAt: null, frozen: [] }];
expect(servingAt(100_000, controllers, windows).map(server => server.pid)).toEqual([100]);
expect(servingAt(550_000, controllers, windows).map(server => [server.pid, server.processSpace, server.harnessRole])).toEqual([[100, "space-a", "primary"], [200, "space-b", "secondary"]]);
expect(servingAt(800_000, controllers, windows).map(server => [server.pid, server.bootId])).toEqual([[200, "boot-b"], [300, null]]);
expect(servingAt(null, controllers, windows)).toEqual([]);
});
it("ends a hard-killed controller at its recorded process exit, so it is not listed as serving a later run", () => {
const killed: ControllerRow[] = [{ bootId: "boot-k", epoch: 1, processSpace: "space-k", pid: 100, startedAt: 1_000, endedAt: null },
{ bootId: "boot-n", epoch: 2, processSpace: "space-n", pid: 200, startedAt: 61_000, endedAt: null }];
const windows = [{ pid: 100, role: "primary" as const, spawnedAt: 0, exitedAt: 60_000, frozen: [] },
{ pid: 200, role: "primary" as const, spawnedAt: 60_500, exitedAt: null, frozen: [] }];
expect(boundByProcessExit(killed, windows).map(controller => controller.endedAt)).toEqual([60_000, null]);
expect(servingAt(90_000, boundByProcessExit(killed, windows), windows).map(server => server.pid)).toEqual([200]);
const evidence = wakeEvidenceForIssue("i1", { runs: [{ id: "r1", issueId: "i1", createdAt: 90_000, controllerBootId: "boot-n" }], wakes: [], statusChanges: [],
controllers: killed, windows });
expect(evidence.runs[0]!.servingAtCreate.map(server => server.pid)).toEqual([200]);
expect(boundByProcessExit(killed, [{ pid: 100, role: "primary", spawnedAt: 500_000, exitedAt: 600_000, frozen: [] }])[0]!.endedAt).toBeNull();
});
it("puts aged database times back at their real times before comparing them with the harness clock", () => {
const shifts = [{ faultId: "f1", ageMs: 26 * 3_600_000, appliedAt: 2_000_000_000_000, waveStartedAt: 1_999_999_000_000 }];
const aged = 1_999_999_500_000 - 26 * 3_600_000;
const [row] = unshiftRows([{ createdAt: aged, finishedAt: null, contextSnapshot: { at: new Date(aged).toISOString() } }], ["createdAt", "finishedAt"], ["contextSnapshot"], shifts);
expect(row).toEqual({ createdAt: 1_999_999_500_000, finishedAt: null, contextSnapshot: { at: new Date(1_999_999_500_000).toISOString() } });
expect(unshiftRows([{ createdAt: 1_999_999_600_000 }], ["createdAt"], [], shifts)[0]!.createdAt).toBe(1_999_999_600_000);
expect(unshiftRows([{ createdAt: aged }], ["createdAt"], [], [])[0]!.createdAt).toBe(aged);
});
it("captures once, falls back to every issue when the task list is not available yet, and records a failure", async () => {
const notes: Array<[string, Record<string, unknown>]> = [];
let opened = 0;
const capture = wakeEvidenceOnce({ databaseUrl: () => "postgres://unused", companyId: "c1", waveDir: "/nonexistent/wave", windows: [], shifts: [],
observers: () => { throw new ReferenceError("launched is not initialized"); }, redact: text => text, note: (kind, data) => notes.push([kind, data]),
open: () => { opened += 1; return { $client: { unsafe: async () => { throw new Error("database is gone"); }, end: async () => undefined } } as never; } });
await Promise.all([capture(), capture()]);
await capture();
expect(opened).toBe(1);
expect(notes).toEqual([["wake_evidence_failed", { error: "Error: database is gone" }]]);
});
it("joins each run to its wake, owner and status changes for one issue", () => {
const evidence = wakeEvidenceForIssue("i1", {
runs: [{ id: "r1", issueId: "i1", status: "succeeded", wakeupRequestId: "w2", controllerBootId: "boot-b", createdAt: 550_000, contextSnapshot: { wakeReason: "retry" } },
{ id: "r9", issueId: "i9", status: "succeeded", wakeupRequestId: "w9", createdAt: 1 }],
wakes: [{ id: "w1", issueId: "i1", reason: "issue_assigned", source: "assignment", createdAt: 10 },
{ id: "w2", issueId: null, reason: "retry", source: "on_demand", requestedByActorType: "user", idempotencyKey: "k", createdAt: 540_000 },
{ id: "w9", issueId: "i9", createdAt: 1 }],
statusChanges: [{ issueId: "i1", action: "issue.updated", fromStatus: "done", toStatus: "in_progress", createdAt: 560_000 }, { issueId: "i9", toStatus: "done" }],
controllers, windows: [],
});
expect(evidence.runs).toHaveLength(1);
expect(evidence.runs[0]).toMatchObject({ runId: "r1", wakeupRequestId: "w2", contextSnapshot: { wakeReason: "retry" }, owner: { pid: 200, epoch: 2 } });
expect(evidence.runs[0]!.servingAtCreate.map(server => server.pid)).toEqual([100, 200]);
expect(evidence.wakes.map(wake => [wake.id, wake.reason, wake.requestedByActorType, wake.idempotencyKey])).toEqual([["w1", "issue_assigned", null, null], ["w2", "retry", "user", "k"]]);
expect(evidence.statusChanges).toEqual([{ at: 560_000, action: "issue.updated", actorType: null, actorId: null, runId: null, from: "done", to: "in_progress" }]);
});
it("keeps the first 400 events and the last 200 of a long run, and says what it left out", async () => {
const all = Array.from({ length: 1_291 }, (_, index) => ({ seq: index + 1, eventType: index === 0 ? "turn.submitted" : "log" }));
const api = { get: async <T>(url: string) => {
const afterSeq = Number(/afterSeq=(\d+)/.exec(url)![1]);
return all.filter(event => event.seq > afterSeq).slice(0, 500) as T;
} } as unknown as ChaosApi;
const kept = await runEvents(api, "run-1");
expect(kept).toHaveLength(601);
expect(kept[0]).toMatchObject({ seq: 1, eventType: "turn.submitted" });
expect(kept[400]).toEqual({ omittedEvents: 691, fromSeq: 401, toSeq: 1_091 });
expect(kept.at(-1)).toMatchObject({ seq: 1_291 });
const short = await runEvents({ get: async <T>(url: string) => (/afterSeq=0/.test(url) ? all.slice(0, 50) : []) as T } as unknown as ChaosApi, "run-2");
expect(short).toHaveLength(50);
});
});
describe("staging-mix round wording", () => {
it("tells the agent to rerun an interrupted round and never one already in the ledger", () => {
const text = stagingMixDescription("abcd1234", { turnMinutes: 5, rounds: 4, subtasks: 2, followUpAfterMs: null, arriveAtMs: 0, holdMs: 0 });
expect(text).not.toMatch(/do not skip, repeat or shorten/);
expect(text).toContain("must end up in `ledger-abcd1234.log` exactly once");
expect(text).toContain("if that round's ROUND line is already there, the round is finished, so never run it again");
expect(text).toContain("before its ROUND line was written, you must run that round again from the start");
expect(followUpComment("abcd1234")).not.toMatch(/do not restart/);
expect(followUpComment("abcd1234")).toContain("a round without one must still be run");
});
});
describe.skipIf(!support.supported)("wake evidence against real PostgreSQL", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: Db;
beforeAll(async () => {
database = await startEmbeddedPostgresTestDatabase("restart-chaos-wake-evidence-");
db = createDb(database.connectionString);
}, 120_000);
afterAll(async () => { await db?.$client.end(); await database?.cleanup(); });
it("writes one file per task with the wake row, context snapshot, status changes and serving controller", async () => {
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), wakeId = randomUUID(), runId = randomUUID(), bootId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Wake evidence", issuePrefix: companyId.slice(0, 8) });
await db.insert(agents).values({ id: agentId, companyId, name: "Agent", adapterType: "process", status: "active" });
await db.insert(issues).values({ id: issueId, companyId, title: "Task" });
await db.insert(controllerInstances).values({ bootId, processSpace: "space-wake", pid: 4242, leaseExpiresAt: new Date(Date.now() + 60_000) });
await db.insert(agentWakeupRequests).values({ id: wakeId, companyId, agentId, source: "on_demand", reason: "retry", triggerDetail: "manual",
requestedByActorType: "user", requestedByActorId: "u1", idempotencyKey: "idem-1", payload: { issueId }, runId });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId, status: "succeeded", invocationSource: "on_demand", wakeupRequestId: wakeId,
contextSnapshot: { wakeReason: "retry", taskId: issueId }, controllerBootId: bootId });
await db.insert(activityLog).values({ companyId, actorType: "agent", actorId: agentId, action: "issue.updated", entityType: "issue", entityId: issueId,
details: { status: "done", _previous: { status: "in_progress" } } });
const waveDir = await mkdtemp(path.join(os.tmpdir(), "wake-evidence-"));
try {
const written = await captureWakeEvidence({ db, companyId, waveDir, observers: [{ plan: { taskId: "t01" }, record: { issueId } }, { plan: { taskId: "t02" }, record: { issueId: null } }],
windows: [], redact: text => text.replaceAll("u1", "[REDACTED]") });
expect(written).toBe(1);
const evidence = JSON.parse(await readFile(path.join(waveDir, "wake-evidence", "t01.json"), "utf8"));
expect(evidence.runs[0]).toMatchObject({ runId, invocationSource: "on_demand", wakeupRequestId: wakeId, contextSnapshot: { wakeReason: "retry" },
owner: { pid: 4242, processSpace: "space-wake" } });
expect(evidence.wakes[0]).toMatchObject({ id: wakeId, reason: "retry", source: "on_demand", triggerDetail: "manual", requestedByActorType: "user",
requestedByActorId: "[REDACTED]", idempotencyKey: "idem-1", runId });
expect(evidence.wakes[0].servingAtCreate.map((server: { pid: number }) => server.pid)).toEqual([4242]);
expect(evidence.statusChanges).toMatchObject([{ action: "issue.updated", from: "in_progress", to: "done" }]);
expect(JSON.parse(await readFile(path.join(waveDir, "wake-evidence", "servers.json"), "utf8")).controllers[0]).toMatchObject({ pid: 4242 });
expect(await captureWakeEvidence({ db, companyId, waveDir, observers: null, windows: [], redact: text => text })).toBe(1);
expect(JSON.parse(await readFile(path.join(waveDir, "wake-evidence", `issue-${issueId}.json`), "utf8")).runs[0]).toMatchObject({ runId });
} finally {
await rm(waveDir, { recursive: true, force: true });
}
});
});
+156
View File
@@ -0,0 +1,156 @@
import { mkdir, writeFile } from "node:fs/promises";
import path from "node:path";
import type { ServerProcessWindow } from "./multi-server.js";
import { unshiftJson, unshiftMs, type AgingShift } from "./time-faults.js";
import { createDb, type Db } from "../../packages/db/src/index.js";
type Row = Record<string, unknown>;
export interface ControllerRow { bootId: string; epoch: number; processSpace: string; pid: number; startedAt: number; endedAt: number | null }
export interface ServingServer { pid: number; bootId: string | null; epoch: number | null; processSpace: string | null; harnessRole: string | null }
const ms = (value: unknown) => value === null || value === undefined ? null : Number(value);
const text = (value: unknown) => value === null || value === undefined ? null : String(value);
export const PROCESS_MATCH_SLACK_MS = 60_000;
export function boundByProcessExit(controllers: readonly ControllerRow[], windows: readonly ServerProcessWindow[]): ControllerRow[] {
return controllers.map(controller => {
const window = windows.filter(candidate => candidate.pid === controller.pid && candidate.spawnedAt <= controller.startedAt + PROCESS_MATCH_SLACK_MS
&& (candidate.exitedAt === null || candidate.exitedAt >= controller.startedAt - PROCESS_MATCH_SLACK_MS))
.sort((a, b) => b.spawnedAt - a.spawnedAt)[0];
if (!window || window.exitedAt === null) return controller;
return { ...controller, endedAt: controller.endedAt === null ? window.exitedAt : Math.min(controller.endedAt, window.exitedAt) };
});
}
export function unshiftRows(rows: readonly Row[], timeKeys: readonly string[], jsonKeys: readonly string[], shifts: readonly AgingShift[]): Row[] {
if (!shifts.length) return [...rows];
return rows.map(row => {
const copy: Row = { ...row };
for (const key of timeKeys) if (copy[key] !== null && copy[key] !== undefined) copy[key] = unshiftMs(Number(copy[key]), shifts);
for (const key of jsonKeys) if (copy[key] !== null && copy[key] !== undefined) copy[key] = unshiftJson(copy[key], shifts);
return copy;
});
}
export function servingAt(at: number | null, controllers: readonly ControllerRow[], windows: readonly ServerProcessWindow[]): ServingServer[] {
if (at === null) return [];
const alive = controllers.filter(controller => controller.startedAt <= at && (controller.endedAt === null || controller.endedAt >= at));
const fromDb = alive.map(controller => ({ pid: controller.pid, bootId: controller.bootId, epoch: controller.epoch, processSpace: controller.processSpace,
harnessRole: windows.find(window => window.pid === controller.pid && window.spawnedAt <= at && (window.exitedAt === null || window.exitedAt >= at))?.role ?? null }));
const known = new Set(fromDb.map(server => server.pid));
const fromHarness = windows.filter(window => !known.has(window.pid) && window.spawnedAt <= at && (window.exitedAt === null || window.exitedAt >= at))
.map(window => ({ pid: window.pid, bootId: null, epoch: null, processSpace: null, harnessRole: window.role }));
return [...fromDb, ...fromHarness];
}
export interface WakeEvidenceInput {
runs: readonly Row[];
wakes: readonly Row[];
statusChanges: readonly Row[];
controllers: readonly ControllerRow[];
windows: readonly ServerProcessWindow[];
}
export function wakeEvidenceForIssue(issueId: string, input: WakeEvidenceInput) {
const runs = input.runs.filter(run => run.issueId === issueId);
const wakeIds = new Set(runs.map(run => text(run.wakeupRequestId)).filter(Boolean));
const wakes = input.wakes.filter(wake => wake.issueId === issueId || wakeIds.has(text(wake.id)));
const controllers = boundByProcessExit(input.controllers, input.windows);
const controllerByBoot = new Map(controllers.map(controller => [controller.bootId, controller]));
return {
issueId,
runs: runs.map(run => {
const owner = controllerByBoot.get(String(run.controllerBootId ?? ""));
return { runId: text(run.id), status: text(run.status), invocationSource: text(run.invocationSource), triggerDetail: text(run.triggerDetail),
wakeupRequestId: text(run.wakeupRequestId), createdAt: ms(run.createdAt), startedAt: ms(run.startedAt), finishedAt: ms(run.finishedAt),
contextSnapshot: run.contextSnapshot ?? null,
owner: owner ? { pid: owner.pid, bootId: owner.bootId, epoch: owner.epoch, processSpace: owner.processSpace } : null,
servingAtCreate: servingAt(ms(run.createdAt), controllers, input.windows) };
}),
wakes: wakes.map(wake => ({ id: text(wake.id), reason: text(wake.reason), source: text(wake.source), triggerDetail: text(wake.triggerDetail),
requestedByActorType: text(wake.requestedByActorType), requestedByActorId: text(wake.requestedByActorId), idempotencyKey: text(wake.idempotencyKey),
status: text(wake.status), runId: text(wake.runId), payload: wake.payload ?? null, error: text(wake.error),
createdAt: ms(wake.createdAt), requestedAt: ms(wake.requestedAt), claimedAt: ms(wake.claimedAt), finishedAt: ms(wake.finishedAt),
servingAtCreate: servingAt(ms(wake.createdAt), controllers, input.windows) })),
statusChanges: input.statusChanges.filter(change => change.issueId === issueId).map(change => ({ at: ms(change.createdAt), action: text(change.action),
actorType: text(change.actorType), actorId: text(change.actorId), runId: text(change.runId), from: text(change.fromStatus), to: text(change.toStatus) })),
};
}
async function readRows(db: Pick<Db, "$client">, companyId: string, shifts: readonly AgingShift[]): Promise<Omit<WakeEvidenceInput, "windows">> {
const client = db.$client;
const runs = await client.unsafe(
`select id::text as id, issue_id::text as "issueId", status, invocation_source as "invocationSource", trigger_detail as "triggerDetail",
wakeup_request_id::text as "wakeupRequestId", context_snapshot as "contextSnapshot", controller_boot_id::text as "controllerBootId",
floor(extract(epoch from created_at) * 1000) as "createdAt", floor(extract(epoch from started_at) * 1000) as "startedAt",
floor(extract(epoch from finished_at) * 1000) as "finishedAt"
from heartbeat_runs where company_id::text = $1 order by created_at`, [companyId]) as Row[];
const wakes = await client.unsafe(
`select id::text as id, payload->>'issueId' as "issueId", source, trigger_detail as "triggerDetail", reason, payload, status,
requested_by_actor_type as "requestedByActorType", requested_by_actor_id as "requestedByActorId", idempotency_key as "idempotencyKey",
run_id::text as "runId", error, floor(extract(epoch from created_at) * 1000) as "createdAt",
floor(extract(epoch from requested_at) * 1000) as "requestedAt", floor(extract(epoch from claimed_at) * 1000) as "claimedAt",
floor(extract(epoch from finished_at) * 1000) as "finishedAt"
from agent_wakeup_requests where company_id::text = $1 order by created_at`, [companyId]) as Row[];
const statusChanges = await client.unsafe(
`select entity_id as "issueId", action, actor_type as "actorType", actor_id as "actorId", run_id::text as "runId",
coalesce(details->'_previous'->>'status', details->>'previousStatus') as "fromStatus", details->>'status' as "toStatus",
floor(extract(epoch from created_at) * 1000) as "createdAt"
from activity_log where company_id::text = $1 and entity_type = 'issue'
and (details ? 'status' or details->'_previous' ? 'status' or details ? 'previousStatus') order by created_at`, [companyId]) as Row[];
const controllers = (await client.unsafe(
`select boot_id as "bootId", epoch, process_space as "processSpace", pid, floor(extract(epoch from started_at) * 1000) as "startedAt",
floor(extract(epoch from coalesce(stopped_at, fenced_at)) * 1000) as "endedAt" from controller_instances order by epoch`) as Row[])
.map(row => ({ bootId: String(row.bootId), epoch: Number(row.epoch), processSpace: String(row.processSpace), pid: Number(row.pid),
startedAt: Number(row.startedAt), endedAt: ms(row.endedAt) }));
return {
runs: unshiftRows(runs, ["createdAt", "startedAt", "finishedAt"], ["contextSnapshot"], shifts),
wakes: unshiftRows(wakes, ["createdAt", "requestedAt", "claimedAt", "finishedAt"], ["payload"], shifts),
statusChanges: unshiftRows(statusChanges, ["createdAt"], [], shifts),
controllers: controllers.map(controller => ({ ...controller, startedAt: unshiftMs(controller.startedAt, shifts),
endedAt: controller.endedAt === null ? null : unshiftMs(controller.endedAt, shifts) })),
};
}
export async function captureWakeEvidence(input: { db: Pick<Db, "$client">; companyId: string; waveDir: string;
observers: ReadonlyArray<{ plan: { taskId: string }; record: { issueId: string | null } }> | null; windows: readonly ServerProcessWindow[];
shifts?: readonly AgingShift[]; redact: (text: string) => string }): Promise<number> {
const rows = await readRows(input.db, input.companyId, input.shifts ?? []);
const observers = input.observers ?? [...new Set(rows.runs.map(run => text(run.issueId)).filter((id): id is string => id !== null))]
.map(issueId => ({ plan: { taskId: `issue-${issueId}` }, record: { issueId } }));
const directory = path.join(input.waveDir, "wake-evidence");
await mkdir(directory, { recursive: true, mode: 0o700 });
let written = 0;
for (const observer of observers) {
if (!observer.record.issueId) continue;
const evidence = { schema: "paperclip.restart-chaos.wake-evidence/v1", taskId: observer.plan.taskId,
...wakeEvidenceForIssue(observer.record.issueId, { ...rows, windows: input.windows }) };
await writeFile(path.join(directory, `${observer.plan.taskId}.json`), input.redact(`${JSON.stringify(evidence, null, 2)}\n`), { mode: 0o600 });
written += 1;
}
await writeFile(path.join(directory, "servers.json"), `${JSON.stringify({ controllers: rows.controllers, harnessWindows: input.windows }, null, 2)}\n`, { mode: 0o600 });
return written;
}
export function wakeEvidenceOnce(input: { databaseUrl: () => string; companyId: string; waveDir: string;
observers: () => ReadonlyArray<{ plan: { taskId: string }; record: { issueId: string | null } }>; windows: readonly ServerProcessWindow[];
shifts: readonly AgingShift[]; redact: (text: string) => string; note: (kind: string, data: Record<string, unknown>) => void;
open?: (url: string) => Pick<Db, "$client"> }): () => Promise<void> {
let done: Promise<void> | null = null;
return () => done ??= (async () => {
let observers: ReturnType<typeof input.observers> | null = null;
try { observers = input.observers(); } catch { observers = null; }
const db = (input.open ?? createDb)(input.databaseUrl());
try {
const written = await captureWakeEvidence({ db, companyId: input.companyId, waveDir: input.waveDir, observers, windows: input.windows,
shifts: input.shifts, redact: input.redact });
input.note("wake_evidence_captured", { written, byTask: observers !== null });
} catch (error) {
input.note("wake_evidence_failed", { error: String(error) });
} finally {
await db.$client.end({ timeout: 5 }).catch(() => undefined);
}
})();
}