Files
PaperClipAI/tests/restart-chaos/timeline.ts
T
DottaandClaude Opus 5.5 aca908a165 fix(restart-chaos): stop the timeline echo loop when stdout breaks (#19695)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) runs waves of
real agent tasks while it restarts the control plane on purpose
> - A wave writes a timeline file and also echoes every timeline entry
to stdout
> - When the process that reads that stdout exits, every echo fails with
EPIPE. The uncaught-exception handler then records the error with
another echo, so the loop never ends
> - In one Daytona wave this wrote 3.2 million entries (2.5 GB) in 30
minutes. It also stopped the wave from grading its tasks
> - This pull request stops the echo after the first stdout error, and
records the stdout loss once
> - The benefit is that a wave keeps running and grading when its log
reader goes away

## Linked Issues or Issue Description

**Bug description**
`Timeline.add` echoes with `console.info`. After stdout breaks, each
echo raises EPIPE as an uncaught exception. The handler in `main.ts`
calls `timeline.add("harness_uncaught_exception")`, and that call echoes
again.

**Steps to reproduce**
Run a wave as a child process that reads its stdout, for example from a
wave-loop script. Kill the parent while the wave runs. `timeline.jsonl`
then grows without bound with `harness_uncaught_exception` / `write
EPIPE` entries.

**Expected behavior**
The wave goes on writing its files (timeline, grades, ledger) and stops
writing to the broken stdout.

Refs #19544

## What Changed

- `tests/restart-chaos/timeline.ts`: a `process.stdout` error listener
and a guard around `console.info` turn off the echo for good after a
broken-pipe error. The timeline file is still written. On the first
entry after the echo stops, the timeline writes one
`harness_stdout_lost` entry (file only) with the error. The file adds
`isBrokenStdoutError` and `markTimelineEchoBroken`.
- `tests/restart-chaos/main.ts`: the uncaught-exception handler passes
broken-stdout errors to `markTimelineEchoBroken`. It does not mark the
wave as interrupted and does not set a harness error.
- `tests/restart-chaos/timeline.test.ts` (new): the error
classification, and the file-but-no-echo behaviour with exactly one
`harness_stdout_lost` entry.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
102/102 pass.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean.
- Incident: wave `w-20261107T001552-s13`. Its timeline had 76 real
entries, then 3,242,762 `harness_uncaught_exception` entries. Those 3.2
million entries came from the loop this pull request removes.

## Risks

- This change touches the test harness only.
- After stdout breaks, the console shows no wave progress, but every
result is still in the wave's files. A different uncaught exception
still stops the wave as before.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, GitHub CLI). Default context window,
extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 20:06:02 -06:00

59 lines
2.5 KiB
TypeScript

import { appendFileSync } from "node:fs";
export interface TimelineEntry { at: string; t: number; kind: string; [key: string]: unknown }
/**
* Console echo stops for good once stdout fails (for example EPIPE after the
* parent that read our output exited). Otherwise every echo raises another
* error, the uncaught-exception handler records it with another echo, and the
* timeline grows without bound.
*/
let echoBroken = false;
let echoBrokenBy: string | null = null;
function markEchoBroken(error: unknown) {
if (echoBroken) return;
echoBroken = true;
echoBrokenBy = error instanceof Error ? `${(error as NodeJS.ErrnoException).code ?? ""} ${error.message}`.trim() : String(error);
}
process.stdout.on("error", markEchoBroken);
export function isBrokenStdoutError(error: unknown): boolean {
const code = (error as NodeJS.ErrnoException | null)?.code;
return code === "EPIPE" || code === "ERR_STREAM_DESTROYED" || code === "ERR_STREAM_WRITE_AFTER_END";
}
/** Append-only action timeline. Every supervisor and launcher action lands here. */
export class Timeline {
readonly entries: TimelineEntry[] = [];
private stdoutLossRecorded = false;
constructor(private readonly path: string | null, readonly origin = Date.now()) {}
private record(kind: string, data: Record<string, unknown>): TimelineEntry {
const now = Date.now();
const entry: TimelineEntry = { ...data, at: new Date(now).toISOString(), t: now - this.origin, kind };
this.entries.push(entry);
if (this.path) appendFileSync(this.path, `${JSON.stringify(entry)}\n`, { mode: 0o600 });
return entry;
}
add(kind: string, data: Record<string, unknown> = {}): TimelineEntry {
const entry = this.record(kind, data);
if (process.env.RESTART_CHAOS_QUIET !== "1" && !echoBroken && process.stdout.writable) {
try { console.info(`[chaos +${Math.round(entry.t / 1000)}s] ${kind} ${JSON.stringify(data)}`); }
catch (error) { if (isBrokenStdoutError(error)) markEchoBroken(error); else throw error; }
}
// Say once, in the file only, why the console went quiet.
if (echoBroken && !this.stdoutLossRecorded) {
this.stdoutLossRecorded = true;
this.record("harness_stdout_lost", { error: echoBrokenBy });
}
return entry;
}
}
/** For a broken-stdout error that surfaced elsewhere (for example as an uncaught exception). */
export function markTimelineEchoBroken(error: unknown): void { markEchoBroken(error); }
/** Test hook: whether console echo has been disabled by a broken stdout. */
export function timelineEchoBroken(): boolean { return echoBroken; }