Files
PaperClipAI/tests/restart-chaos/observer.test.ts
T
DottaandClaude Opus 5.5 0c04ff76eb fix(native-runtime): a follow-up that arrives as a warm session closes continues the task (#21279)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native agent session on a remote sandbox stays warm after a run,
and an idle timeout closes it after about 5 minutes
> - When a person's follow-up comment arrived just as that idle close
began, the new run failed with `native_session_supervisor_busy`
> - The idle close of the finished owner could also fail its final
checkpoint with `native_runner_checkpoint_owner_lost`, because the task
already pointed at the queued follow-up run
> - Then the retry was held for workspace repair and the task stayed
blocked; the environment could not be deleted (409) because its warm
runner could not be stopped
> - This pull request makes the follow-up wait for the idle close that
is in flight, and lets the finished owner save its final checkpoint
while its successor still waits in the queue
> - The benefit is that a follow-up that lands at the warm session's
idle close continues the task with no person involved

## Linked Issues or Issue Description

Refs #18011 (the checkpoint ownership fence this change extends).

**What happened?**

A follow-up comment arrived 295–340 s after the previous run ended,
while the warm session's idle close was in progress. The new run failed
with `native_session_supervisor_busy`. In another case, the idle close
failed its final checkpoint with `native_runner_checkpoint_owner_lost`,
the next run failed
`native_instruction_materialization_handoff_unavailable`, and the retry
was cancelled with `execution_reconciliation_required`. The task stayed
blocked. A hosted instance showed the same class as "previous run's
session on this sandbox could not be retired".

**Expected behavior**

The follow-up run waits for the idle close (inside the existing bounded
release wait) and then starts. The finished owner saves its final
checkpoint and closes. The task completes hands-off.

**Steps to reproduce**

1. Run an opencode agent on a Daytona sandbox with a warm session (idle
timeout 300 s).
2. Let a run finish.
3. Post a comment on the task 298–306 s after the run ended.

**Paperclip version or commit**

master `444dc18d98`.

**Deployment mode**

Local server with Daytona remote sandboxes (restart-chaos harness, light
lane). The same class was seen on a hosted staging instance.

## What Changed

- `claimWarmNativeInstructionCopy` now also waits when an idle close is
in flight for the session scope (`closingWarmNativeSessions`). The wait
uses the same deadline as the existing release wait
(`warmSessionReleaseWait`), so it adds no new wait and nothing before
`turn.start` on the normal path.
- `validateNativeRunnerCheckpointOwner` accepts the finished owner's
final checkpoint (accepted result, no new work) when
`issues.executionRunId` points at a successor run that is `queued` or
`scheduled_retry` for the same company and task and has no
`native_run_finalizations` row.
- Tests: a server test for a follow-up that arrives while the idle
timeout closes the warm session; four new scenarios (`queued_successor`,
`scheduled_retry_successor` × `runner.suspend`/`turn.stop`) and four new
negatives (`queued_wrong_issue`, `queued_new_work` × both commands) in
the early-retirement checkpoint matrix.
- Restart-chaos harness:
`RESTART_CHAOS_MIX_FOLLOW_UP_AFTER_END_MS=<min>-<max>` and
`RESTART_CHAOS_MIX_FOLLOW_UP_COUNT` post the staging-mix follow-up a set
delay after the task's last run ended, so a cell can aim the follow-up
at the warm idle close. The restart-chaos README describes both
variables. A failed post is retried with the same `clientRequestId`, the
grade check `after_end_follow_ups_served` needs an agent reply after
each post and the ledger marker, and the task budget includes the
planned waits.
- Harness: the wave database cleanup
(`tests/runner-e2e/controller-database.ts`) no longer waits forever when
a group signal (for example `timeout` or Ctrl-C) already stopped the
embedded Postgres. Before, `embedded-postgres` `stop()` waited for an
exit event that had already fired, and an interrupted wave hung after
the Daytona R6 sweep.

## Verification

- Fix 1, red without the product hunk: `native-session-executor.test.ts
-t "idle timeout is closing"` fails with `promise rejected "Error:
native_session_supervisor_busy"`. Green with the fix: 1 passed.
- Fix 2, red without the product hunk:
`native-runner-restart-recovery.integration.test.ts` fails 4 of 208
(`queued_successor` and `scheduled_retry_successor` ×
`runner.suspend`/`turn.stop`) with
`native_runner_checkpoint_owner_lost`. Green with the fix: 208 passed,
14 skipped. The two new negative cases pass in both states.
- Harness hang fix, red without the fix: the new
`controller-database.test.ts` case ("finishes cleanup when the database
already stopped because a group signal reached it") times out on cleanup
(`expected 'hung' to be 'cleaned'`). Green with the fix: 5 passed, 1
skipped.
- After the rebase onto master `7154030ef6`, both server test files
together: 1639 passed, 21 skipped.
- Restart-chaos harness tests (`staging-mix`, `staging-mix-retry`,
`observer`): 75 passed, including a test for the after-end retry, the
`after_end_follow_ups_served` grade check and the added budget. `tsc
--noEmit` passes for `server`, `tests/runner-e2e` and
`tests/restart-chaos`.
- `pnpm verify` exits 2 because `tests/restart-chaos/observer.ts` is a
global CI input. Required CI supplies the broad coverage.
- Daytona cell on this branch (4 opencode tasks, warm sandboxes, 1 vCPU,
follow-up comment posted 300.8–308.2 s after each task's last run ended,
image built for master `444dc18d98`): 4/4 tasks done hands-off, 8/8 runs
succeeded, 0 Retry banners, 0 `owner_lost` / `supervisor_busy` / "could
not be retired" / `execution_reconciliation_required` lines in the
server log, environment delete clean, warm-shutdown grade pass, 0
sandboxes left. The harness exited by itself after 1076 s.
- Same cell before the fix (master, 8 tasks): 1/8 tasks blocked
(`native_session_supervisor_busy` →
`native_runner_checkpoint_owner_lost` →
`execution_reconciliation_required`). With fix 1 only: 1/8 still blocked
(`owner_lost` on the idle close). With both fixes: 8/8 done and 0 failed
runs (cut by the 20-minute cap before cleanup), then 4/4 above.
- The cell grade `T:completion_once` fails on all 4 tasks by design: the
timed follow-up comment arrives after the task is done, so the agent
posts a second completion comment.

## Risks

- Fix 1 can delay a follow-up by up to the existing warm release wait
(30 s) while an idle close finishes. Before this change, that run
failed. If the close fails or the wait expires, the claim returns null
and the run uses a fresh instruction copy, as before.
- Fix 2 widens the checkpoint ownership fence from #18011. It does not
reopen what #18011 closed:
- The exemption applies only to the finished owner's final save: the
checkpoint must hold an accepted result and no new work, so no provider
turn can be granted through it.
- The successor must still be unclaimed (`queued` or `scheduled_retry`)
and must have no native coordinator (`native_run_finalizations`) row. No
controller has started it, so nothing can have consumed or depended on
the predecessor's checkpoint. Once the successor is claimed, the
existing preparing/claimed rules from #18011 apply unchanged.
  - The successor must belong to the same company and the same task.
- Cases that must still reject, with tests in the same matrix:
- `queued_wrong_issue`: a queued successor for a different task is
rejected with `native_runner_checkpoint_owner_lost` and the stored
checkpoint does not change.
- `queued_new_work`: a `turn.start` (new work) while the successor is
queued is rejected with `native_runner_checkpoint_owner_lost` and the
stored checkpoint does not change.
- All earlier rejections stay: unclaimed native coordinator,
dispatching, expired or foreign preparation, foreign native preparation,
wrong task, new work, unaccepted result, replaced controller.
- Not changed: an owner whose close fails still blocks environment
delete (409). That is a separate product decision.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edit, test runs). Earlier commits on
this branch were written by the same model in an earlier session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:42:50 -06:00

539 lines
33 KiB
TypeScript

import { execFileSync } from "node:child_process";
import { mkdtemp, readFile, rm } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it } from "vitest";
import type { ChaosApi } from "./api.js";
import { CHILD_WAKE_GRACE_MS, REST_CONFIRM_MS, STALL_GIVE_UP_MS, STOP_CONTINUE_AFTER_MS, TaskObserver, childSpeedTasksInTurn, childrenPending, listWorkspaceFiles, noteLivenessEvent, stopArmed, toRunSample } from "./observer.js";
import { gradeTask } from "./grader.js";
import { createFixtureRepository, type ChaosLab } from "./seed.js";
import { Timeline } from "./timeline.js";
import type { TaskPlan } from "./types.js";
import { BENCH_EXPECTED, parseBenchRecord } from "./workloads.js";
process.env.RESTART_CHAOS_QUIET = "1";
const lab: ChaosLab = { companyId: "c1", environmentId: "e1", environmentKind: "local", projectId: "p1", fixtureRepositoryPath: "/tmp/none",
agents: { codex: { id: "a1", name: "Chaos codex" }, claude: { id: "a2", name: "Chaos claude" }, opencode: { id: "a3", name: "Chaos opencode" } } };
const plan: TaskPlan = { taskId: "t01", index: 0, workload: "C1", adapter: "codex", fault: "K1", fireAfterMs: 999_999,
holdMs: 0, followUpAfterMs: 1_000, nonce: "abcd1234" };
const ready = { serverReady: true, faultInProgress: false };
const directories: string[] = [];
afterEach(async () => { await Promise.all(directories.splice(0).map(directory => rm(directory, { recursive: true, force: true }))); });
/** Fake board API: a C1 issue whose first run already finished. */
function fakeApi(overrides: Partial<Record<string, (route: string) => unknown>> = {}, status = "done") {
const posted: string[] = [];
const api = {
async get(route: string) {
for (const [prefix, handler] of Object.entries(overrides)) if (route.includes(prefix)) return handler!(route);
if (route.endsWith("/runs")) return [{ runId: "r1", status: "succeeded", createdAt: new Date(0).toISOString(), finishedAt: new Date(1).toISOString() }];
return { id: "i1", status, interruption: null };
},
async post(route: string) { posted.push(route); return route.endsWith("/issues") ? { id: "i1", identifier: "CHA-1" } : { id: "comment-1" }; },
};
return { api: api as unknown as ChaosApi, posted };
}
describe("TaskObserver", () => {
it("does not finish a C1 task on the poll that posts the follow-up", async () => {
const { api, posted } = fakeApi();
const observer = new TaskObserver(plan, api, lab, new Timeline(null), ready);
await observer.launch();
const start = observer.record.createdAt!;
// The task rests before the follow-up is due, for longer than the rest window.
await observer.poll(start + 100);
await observer.poll(start + 100 + REST_CONFIRM_MS + 1);
expect(observer.done).toBe(false);
expect(observer.record.followUp).toMatchObject({ commentId: "comment-1" });
expect(posted.some(route => route.endsWith("/comments"))).toBe(true);
// A fresh rest window must elapse after the comment before the task counts as resting.
await observer.poll(start + 200 + REST_CONFIRM_MS + 2);
expect(observer.done).toBe(false);
await observer.poll(start + 300 + 2 * REST_CONFIRM_MS + 3);
expect(observer.done).toBe(true);
});
it("does not finish a staging-mix task on the poll that posts its last after-end follow-up", async () => {
const previous = { ...process.env };
const delay = REST_CONFIRM_MS + 50;
Object.assign(process.env, { RESTART_CHAOS_MIX_FOLLOW_UP_AFTER_END_MS: `${delay}-${delay}`, RESTART_CHAOS_MIX_FOLLOW_UP_COUNT: "1" });
try {
let endedAt = 0;
const { api, posted } = fakeApi({ "/runs": () => [{ runId: "r1", status: "succeeded", createdAt: new Date(endedAt - 1_000).toISOString(),
finishedAt: new Date(endedAt).toISOString() }] });
const mixPlan: TaskPlan = { ...plan, workload: "SM", fault: "K0", followUpAfterMs: null,
stagingMix: { turnMinutes: 2, rounds: 2, subtasks: 0, followUpAfterMs: 1_000, arriveAtMs: 0, holdMs: 0 } };
const observer = new TaskObserver(mixPlan, api, lab, new Timeline(null), ready);
await observer.launch();
const start = observer.record.createdAt!;
endedAt = start;
const comments = () => posted.filter(route => route.endsWith("/comments")).length;
await observer.poll(start + 100);
expect(comments()).toBe(0);
await observer.poll(start + 100 + REST_CONFIRM_MS + 1);
expect(comments()).toBe(1);
expect(observer.done).toBe(false);
await observer.poll(start + 200 + REST_CONFIRM_MS + 2);
expect(observer.done).toBe(false);
await observer.poll(start + 300 + 2 * REST_CONFIRM_MS + 3);
expect(observer.done).toBe(true);
} finally {
for (const key of ["RESTART_CHAOS_MIX_FOLLOW_UP_AFTER_END_MS", "RESTART_CHAOS_MIX_FOLLOW_UP_COUNT"]) {
if (previous[key] === undefined) delete process.env[key]; else process.env[key] = previous[key];
}
}
});
it("keeps watching unexplained idle work until the stall limit, then records the stall", async () => {
const { api } = fakeApi({}, "in_progress");
const observer = new TaskObserver({ ...plan, workload: "A1", followUpAfterMs: null }, api, lab, new Timeline(null), ready);
await observer.launch();
const start = observer.record.createdAt!;
await observer.poll(start + 1_000);
await observer.poll(start + 1_000 + REST_CONFIRM_MS + 1);
expect(observer.done).toBe(false);
await observer.poll(start + 1_000 + STALL_GIVE_UP_MS + 1);
expect(observer.done).toBe(true);
expect(observer.record.finalStatus).toBe("in_progress");
expect(observer.record.restAt).toBe(start + 1_000);
});
it("reports a failed workspace-operation read as a sweep error, not a clean rest", async () => {
const { api } = fakeApi({ "/workspace-operations": () => { throw new Error("timeout"); } });
const observer = new TaskObserver(plan, api, lab, new Timeline(null), ready);
await observer.launch();
const sweep = await observer.sweep();
expect(sweep.error).toMatch(/timeout/);
});
it("leaves R2 unchecked instead of failing the wave when the pre-fault snapshot cannot read the workspace", async () => {
const { api } = fakeApi({ "/api/issues/i1": route => route.endsWith("/runs") ? []
: { id: "i1", status: "in_progress", currentExecutionWorkspace: { cwd: "/nonexistent/restart-chaos-workspace" } } });
const observer = new TaskObserver({ ...plan, workload: "B1", followUpAfterMs: null }, api, lab, new Timeline(null), ready);
await observer.launch();
await observer.poll(observer.record.createdAt! + 1_000);
await expect(observer.snapshotBeforeFault()).resolves.toBeUndefined();
expect(observer.record.preFaultFiles).toBeNull();
expect(observer.record.preFaultSnapshotError).toMatch(/ENOENT/);
});
it("keeps an M1 parent watched while a subtask runs, then rests it once a subtask is blocked", async () => {
let childStatus = "in_progress";
const { api } = fakeApi({ "parentIds=": () => [{ id: "c1", parentId: "i1", status: "done" }, { id: "c2", parentId: "i1", status: childStatus }],
"/documents/": () => null }, "blocked");
const timeline = new Timeline(null);
const observer = new TaskObserver({ ...plan, workload: "M1", followUpAfterMs: null }, api, lab, timeline, ready);
await observer.launch();
let now = observer.record.createdAt! + 1_000;
const restWindow = async () => { await observer.poll(now); now += REST_CONFIRM_MS + 1; await observer.poll(now); now += 1_000; };
await restWindow();
await restWindow();
expect(observer.done).toBe(false);
childStatus = "blocked";
await restWindow();
expect(observer.done).toBe(false);
expect(timeline.entries.filter(entry => entry.kind === "task_child_blocked")).toHaveLength(1);
now += CHILD_WAKE_GRACE_MS;
await restWindow();
expect(observer.done).toBe(true);
expect(observer.record.finalStatus).toBe("blocked");
});
it("clicks the board Retry on a blocked subtask, as a user would, and keeps watching the parent", async () => {
let childStatus = "blocked";
const base = fakeApi({ "parentIds=": () => [{ id: "c1", parentId: "i1", status: "done" }, { id: "c2", parentId: "i1", status: childStatus }],
"/documents/": () => null, "/recovery-actions": () => ({ active: { id: "a1", status: "active", ownerType: "board" }, actions: [] }), "/api/issues/c2": () => ({ id: "c2", status: childStatus,
interruption: { canRetry: childStatus === "blocked", observedRunId: "child-run" } }) }, "blocked");
const childRetries: string[] = [];
const api = { ...base.api, async get(route: string) {
if (route === "/api/issues/i1") return { id: "i1", status: "blocked", stalledDependencies: childStatus === "blocked"
? [{ stalledIssueId: "c2", relation: "child", recoveryAction: { id: "a1", status: "active", ownerType: "board" } }] : [] };
return base.api.get(route);
},
async post(route: string, body: unknown) {
if (route === "/api/issues/c2/retry") { childRetries.push(route); childStatus = "in_progress"; return { accepted: true }; }
return (base.api as unknown as { post(route: string, body: unknown): Promise<unknown> }).post(route, body);
} } as unknown as ChaosApi;
const timeline = new Timeline(null);
const observer = new TaskObserver({ ...plan, workload: "M1", followUpAfterMs: null }, api, lab, timeline, ready);
await observer.launch();
let now = observer.record.createdAt! + 1_000;
const restWindow = async () => { await observer.poll(now); now += REST_CONFIRM_MS + 1; await observer.poll(now); now += 1_000; };
await restWindow();
expect(childRetries).toEqual(["/api/issues/c2/retry"]);
expect(observer.record.childRetries).toMatchObject([{ childIssueId: "c2", accepted: true, observedRunId: "child-run" }]);
expect(observer.record.childStalls).toEqual([{ childIssueId: "c2", recoveryActionId: "a1" }]);
now += CHILD_WAKE_GRACE_MS;
await restWindow();
expect(observer.done).toBe(false);
childStatus = "done";
await restWindow();
now += CHILD_WAKE_GRACE_MS;
await restWindow();
expect(childRetries).toHaveLength(1);
});
it("M1: rereads a parent whose read failed on the next poll, and reads each stall wake from the parent's activity", async () => {
const base = fakeApi({ "parentIds=": () => [{ id: "c2", parentId: "i1", status: "blocked" }], "/documents/": () => null, "/comments": () => [],
"/workspace-operations": () => [], "/api/issues/c2": () => ({ id: "c2", status: "blocked", interruption: { canRetry: false } }),
"/activity": () => [{ action: "issue.blocker_stalled_wake", details: { stalledIssueId: "c2", recoveryActionId: "a1" } },
{ action: "issue.updated", details: {} }, { action: "issue.blocker_stalled_wake", details: {} }] }, "blocked");
let failNextStallRead = false;
const api = { ...base.api, async get(route: string, timeoutMs?: number) {
if (route === "/api/issues/i1") {
if (timeoutMs === undefined && failNextStallRead) { failNextStallRead = false; throw new Error("503"); }
return { id: "i1", status: "blocked", stalledDependencies: [{ stalledIssueId: "c2", relation: "child", recoveryAction: { id: "a1", ownerType: "board" } }] };
}
return base.api.get(route);
}, post: base.api.post.bind(base.api) } as unknown as ChaosApi;
const observer = new TaskObserver({ ...plan, workload: "M1", followUpAfterMs: null }, api, lab, new Timeline(null), ready);
await observer.launch();
const start = observer.record.createdAt!;
failNextStallRead = true;
await observer.poll(start + 1_000);
expect(observer.record.childStalls ?? []).toEqual([]);
await observer.poll(start + 2_000);
expect(observer.record.childStalls).toEqual([{ childIssueId: "c2", recoveryActionId: "a1" }]);
const artifacts = await observer.collectArtifacts();
expect(artifacts.stallWakes).toEqual([{ stalledIssueId: "c2", recoveryActionId: "a1" }, { stalledIssueId: null, recoveryActionId: null }]);
});
it("S1: stops the running run as the board would, then posts the continue comment only after it settles", async () => {
let runStatus = "running";
const base = fakeApi({ "/runs": () => [{ id: "r1", status: runStatus, createdAt: new Date(0).toISOString(), finishedAt: null }] }, "in_progress");
const posted: string[] = [];
const api = { ...base.api, get: base.api.get.bind(base.api), async post(route: string) {
posted.push(route);
if (route === "/api/heartbeat-runs/r1/cancel") { runStatus = "cancelled"; return { id: "r1", status: "cancelled" }; }
return route.endsWith("/issues") ? { id: "i1", identifier: "CHA-1" } : { id: "comment-1" };
} } as unknown as ChaosApi;
const observer = new TaskObserver({ ...plan, workload: "S1", followUpAfterMs: 1_000 }, api, lab, new Timeline(null), ready);
await observer.launch();
const start = observer.record.createdAt!;
await observer.poll(start + 2_000);
expect(posted).toContain("/api/heartbeat-runs/r1/cancel");
expect(observer.record.stop).toMatchObject({ runId: "r1", httpStatus: 200, error: null });
await observer.poll(start + 2_000 + STOP_CONTINUE_AFTER_MS - 1);
expect(observer.record.followUp).toBeNull();
await observer.poll(start + 2_000 + STOP_CONTINUE_AFTER_MS + 1);
expect(observer.record.followUp).toMatchObject({ commentId: "comment-1", duringRun: false });
expect(posted.filter(route => route.endsWith("/cancel"))).toHaveLength(1);
});
it("S1 with a provider fault: stops on the first live provider turn, not the wall-clock window", async () => {
// Waves w-20261110T082702-s212 and w-20261110T091057-s213: the agent finished
// before the seeded window, so the PV2 interrupt_ack fault was never injected.
let events: unknown[] = [];
let runStatus = "running";
const base = fakeApi({
"/events": () => events,
"/runs": () => [{ id: "r1", status: runStatus, createdAt: new Date(0).toISOString(), finishedAt: null }],
}, "in_progress");
const posted: string[] = [];
const api = { ...base.api, get: base.api.get.bind(base.api), async post(route: string) {
posted.push(route);
if (route === "/api/heartbeat-runs/r1/cancel") { runStatus = "cancelled"; return { id: "r1" }; }
return route.endsWith("/issues") ? { id: "i1", identifier: "CHA-1" } : { id: "comment-1" };
} } as unknown as ChaosApi;
const pvPlan: TaskPlan = { ...plan, workload: "S1", adapter: "claude", followUpAfterMs: 60_000,
providerFault: { kind: "PV2", message: "interrupt_ack" } };
const observer = new TaskObserver(pvPlan, api, lab, new Timeline(null), ready);
await observer.launch();
const start = observer.record.createdAt!;
// Running, but no provider turn yet: the prompt has not reached the provider.
await observer.poll(start + 500);
expect(posted.filter(route => route.endsWith("/cancel"))).toEqual([]);
events = [{ seq: 1, eventType: "turn.started", createdAt: new Date(start + 600).toISOString(), payload: {} }];
await observer.poll(start + 1_000);
expect(posted.filter(route => route.endsWith("/cancel"))).toEqual(["/api/heartbeat-runs/r1/cancel"]);
expect(observer.record.stop).toMatchObject({ runId: "r1", httpStatus: 200 });
// The continue comment still waits for the seeded window.
events = [];
await observer.poll(start + 1_000 + STOP_CONTINUE_AFTER_MS + 1);
expect(observer.record.followUp).toBeNull();
await observer.poll(start + 60_000 + 1);
expect(observer.record.followUp).toMatchObject({ commentId: "comment-1" });
});
it("S1 with a provider fault: does not stop after a turn terminal, and stops once when the next turn starts", async () => {
const events = [
{ seq: 1, eventType: "turn.started", createdAt: new Date(1).toISOString(), payload: {} },
{ seq: 2, eventType: "turn.completed", createdAt: new Date(2).toISOString(), payload: {} },
];
let nextTurn: unknown[] = [];
const base = fakeApi({
"/events": route => route.includes("afterSeq=0") ? events
: route.includes("afterSeq=2") ? nextTurn : [],
"/runs": () => [{ id: "r1", status: "running", createdAt: new Date(0).toISOString(), finishedAt: null }],
}, "in_progress");
const pvPlan: TaskPlan = { ...plan, workload: "S1", adapter: "claude", followUpAfterMs: 1_000,
providerFault: { kind: "PV2", message: "interrupt_ack" } };
const observer = new TaskObserver(pvPlan, base.api, lab, new Timeline(null), ready);
await observer.launch();
await observer.poll(observer.record.createdAt! + 5_000);
expect(base.posted.filter(route => route.endsWith("/cancel"))).toEqual([]);
expect(observer.record.stop ?? null).toBeNull();
// The next turn re-arms the Stop, and it fires once.
nextTurn = [{ seq: 3, eventType: "turn.started", createdAt: new Date(3).toISOString(), payload: {} }];
await observer.poll(observer.record.createdAt! + 6_000);
nextTurn = [];
await observer.poll(observer.record.createdAt! + 7_000);
expect(base.posted.filter(route => route.endsWith("/cancel"))).toEqual(["/api/heartbeat-runs/r1/cancel"]);
expect(observer.record.stop).toMatchObject({ runId: "r1" });
});
it("S1 with a provider fault: records a missed Stop when the run ends before any live turn and the task rests after the window", async () => {
const base = fakeApi({ "/events": () => [],
"/runs": () => [{ id: "r1", status: "succeeded", createdAt: new Date(0).toISOString(), finishedAt: new Date(1).toISOString() }],
"/comments": () => [], "/documents/": () => null, "/workspace-operations": () => [] }, "done");
const timeline = new Timeline(null);
const pvPlan: TaskPlan = { ...plan, workload: "S1", adapter: "claude", followUpAfterMs: 1_000,
providerFault: { kind: "PV2", message: "interrupt_ack" } };
const observer = new TaskObserver(pvPlan, base.api, lab, timeline, ready);
await observer.launch();
const start = observer.record.createdAt!;
await observer.poll(start + 500);
expect(observer.record.stop ?? null).toBeNull();
await observer.poll(start + 2_000);
expect(observer.record.stop).toMatchObject({ runId: null, missed: true });
expect(timeline.entries.some(entry => entry.kind === "task_stop_missed")).toBe(true);
expect(base.posted.filter(route => route.endsWith("/cancel"))).toEqual([]);
});
it("noteLivenessEvent: a runner replacement or stop clears the live provider turn", () => {
const replaced = { seq: 0, lastAt: null, openTools: 0, process: null } as Parameters<typeof noteLivenessEvent>[0];
noteLivenessEvent(replaced, { eventType: "native.process_identity_recorded", payload: { targetKind: "local", processPid: 1, processStartedAt: "a" } });
noteLivenessEvent(replaced, { eventType: "turn.started", payload: {} });
expect(replaced.turnActive).toBe(true);
noteLivenessEvent(replaced, { eventType: "native.process_identity_recorded", payload: { targetKind: "local", processPid: 2, processStartedAt: "b" } });
expect(replaced.turnActive).toBe(false);
noteLivenessEvent(replaced, { eventType: "turn.started", payload: {} });
noteLivenessEvent(replaced, { eventType: "native.local_process_stopped", payload: { processPid: 2 } });
expect(replaced.turnActive).toBe(false);
});
it("stopArmed: a provider-fault Stop needs a live provider turn; others keep the window", () => {
const live = { id: "r1", status: "running", providerTurnActive: true } as never;
const idle = { id: "r1", status: "running" } as never;
expect(stopArmed({ now: 0, windowAt: 1_000, providerFault: true, running: live })).toBe(true);
expect(stopArmed({ now: 5_000, windowAt: 1_000, providerFault: true, running: idle })).toBe(false);
expect(stopArmed({ now: 5_000, windowAt: 1_000, providerFault: true, running: undefined })).toBe(false);
expect(stopArmed({ now: 0, windowAt: 1_000, providerFault: false, running: live })).toBe(false);
expect(stopArmed({ now: 1_000, windowAt: 1_000, providerFault: false, running: undefined })).toBe(true);
});
it("S1: holds the Stop while another task's fault has the server down or recovering", async () => {
const base = fakeApi({ "/runs": () => [{ id: "r1", status: "running", createdAt: new Date(0).toISOString(), finishedAt: null }] }, "in_progress");
const posted: string[] = [];
const api = { ...base.api, get: base.api.get.bind(base.api), async post(route: string) {
posted.push(route);
return route.endsWith("/issues") ? { id: "i1", identifier: "CHA-1" } : { id: "r1" };
} } as unknown as ChaosApi;
const supervisor = { serverReady: true, faultInProgress: false };
const observer = new TaskObserver({ ...plan, workload: "S1", followUpAfterMs: 1_000 }, api, lab, new Timeline(null), supervisor);
await observer.launch();
supervisor.faultInProgress = true;
await observer.poll(observer.record.createdAt! + 2_000);
expect(posted.filter(route => route.endsWith("/cancel"))).toEqual([]);
supervisor.faultInProgress = false;
await observer.poll(observer.record.createdAt! + 3_000);
expect(posted.filter(route => route.endsWith("/cancel"))).toEqual(["/api/heartbeat-runs/r1/cancel"]);
});
it("S1: records a missed Stop and still posts the continue comment when the run finished before the stop window", async () => {
const base = fakeApi({ "/runs": () => [{ id: "r1", status: "succeeded", createdAt: new Date(0).toISOString(), finishedAt: new Date(1).toISOString() }],
"/comments": () => [], "/documents/": () => null, "/workspace-operations": () => [] }, "done");
const posted: string[] = [];
const api = { ...base.api, get: base.api.get.bind(base.api), async post(route: string) {
posted.push(route);
return route.endsWith("/issues") ? { id: "i1", identifier: "CHA-1" } : { id: "comment-1" };
} } as unknown as ChaosApi;
const timeline = new Timeline(null);
const observer = new TaskObserver({ ...plan, workload: "S1", followUpAfterMs: 1_000 }, api, lab, timeline, ready);
await observer.launch();
const start = observer.record.createdAt!;
await observer.poll(start + 2_000);
expect(observer.record.stop).toMatchObject({ runId: null, missed: true });
expect(timeline.entries.some(entry => entry.kind === "task_stop_missed")).toBe(true);
await observer.poll(start + 2_000 + STOP_CONTINUE_AFTER_MS + 1);
expect(observer.record.followUp).toMatchObject({ commentId: "comment-1" });
expect(posted.filter(route => route.endsWith("/cancel"))).toEqual([]);
expect((await observer.collectArtifacts()).stop).toMatchObject({ missed: true, continuePosted: true, runsStartedBeforeContinue: [] });
});
it("S1: arms a stop-triggered fault at the Stop click plus its delay, and misses it when the Stop was missed", async () => {
const running = fakeApi({ "/runs": () => [{ id: "r1", status: "running", createdAt: new Date(0).toISOString(), finishedAt: null }] }, "in_progress");
const stopPlan: TaskPlan = { ...plan, workload: "S1", followUpAfterMs: 1_000, trigger: "stop", triggerDelayMs: 500 };
const observer = new TaskObserver(stopPlan, running.api, lab, new Timeline(null), ready);
await observer.launch();
const start = observer.record.createdAt!;
expect(observer.faultDue(start + 5_000)).toBeNull();
await observer.poll(start + 2_000);
const clicked = observer.record.stop!.at;
expect(observer.faultDue(clicked + 499)).toBeNull();
expect(observer.faultDue(clicked + 500)).toMatchObject({ trigger: "stop", kind: "K1" });
const finished = fakeApi({ "/runs": () => [{ id: "r1", status: "succeeded", createdAt: new Date(0).toISOString(), finishedAt: new Date(1).toISOString() }] }, "done");
const timeline = new Timeline(null);
const missed = new TaskObserver(stopPlan, finished.api, lab, timeline, ready);
await missed.launch();
await missed.poll(missed.record.createdAt! + 2_000);
expect(missed.record.stop).toMatchObject({ missed: true });
expect(missed.faultDue(missed.record.createdAt! + 2_600)).toBeNull();
expect(missed.record.triggerMissed).toEqual(["stop"]);
});
it("S1: does not count the continue comment's own run as a run after the stop, even when it starts before the comment returns", async () => {
let runs: Array<Record<string, unknown>> = [{ id: "r1", status: "running", createdAt: new Date(0).toISOString(), finishedAt: null }];
const base = fakeApi({ "/runs": () => runs, "/comments": () => [], "/documents/": () => null, "/workspace-operations": () => [] }, "in_progress");
let clock = 0;
const api = { ...base.api, get: base.api.get.bind(base.api), async post(route: string) {
if (route === "/api/heartbeat-runs/r1/cancel") { runs = [{ ...runs[0]!, status: "cancelled" }]; return { id: "r1" }; }
if (route.endsWith("/comments")) {
runs = [...runs, { id: "r2", status: "running", createdAt: new Date(clock - 10).toISOString(), finishedAt: null, wakeCommentIds: ["continue-1"] }];
return { id: "continue-1" };
}
return { id: "i1", identifier: "CHA-1" };
} } as unknown as ChaosApi;
const observer = new TaskObserver({ ...plan, workload: "S1", followUpAfterMs: 1_000 }, api, lab, new Timeline(null), ready);
await observer.launch();
const start = observer.record.createdAt!;
await observer.poll(start + 2_000);
clock = start + 2_000 + STOP_CONTINUE_AFTER_MS + 1;
await observer.poll(clock);
await observer.poll(clock + 3_000);
const artifacts = await observer.collectArtifacts();
expect(artifacts.stop).toMatchObject({ runId: "r1", runStatus: "cancelled", runsStartedBeforeContinue: [], continuePosted: true });
});
it("S1: counts a run recovery started between the poll's reads and the continue request", async () => {
let runs: Array<Record<string, unknown>> = [{ id: "r1", status: "running", createdAt: new Date(0).toISOString(), finishedAt: null }];
let stopped = false;
const base = fakeApi({ "/runs": () => runs, "/comments": () => [], "/documents/": () => null, "/workspace-operations": () => [] }, "in_progress");
let sentAt = 0;
const api = { ...base.api, async get(route: string) {
const value = await base.api.get(route);
if (stopped && route.endsWith("/runs") && runs.length === 1) {
runs = [...runs, { id: "r2", status: "queued", createdAt: new Date(sentAt - 5).toISOString(), finishedAt: null }];
}
return value;
}, async post(route: string) {
if (route === "/api/heartbeat-runs/r1/cancel") { stopped = true; runs = [{ ...runs[0]!, status: "cancelled" }]; return { id: "r1" }; }
if (route.endsWith("/comments")) return { id: "continue-1" };
return { id: "i1", identifier: "CHA-1" };
} } as unknown as ChaosApi;
const observer = new TaskObserver({ ...plan, workload: "S1", followUpAfterMs: 1_000 }, api, lab, new Timeline(null), ready);
await observer.launch();
const start = observer.record.createdAt!;
await observer.poll(start + 2_000);
const at = start + 2_000 + STOP_CONTINUE_AFTER_MS + 1;
sentAt = at + 100;
runs = [runs[0]!];
observer.record.followUp = { at, sentAt, postedAt: sentAt + 50, commentId: "continue-1", error: null, duringRun: false };
observer.record.samples.push({ at: sentAt + 200, reachable: true, serverReady: true, issueStatus: "in_progress", interruption: null,
activeRecoveryAction: false, recoveryOwner: null, executionBlocker: false, scheduledRetry: false,
runs: [runs[0], { id: "r2", status: "queued", createdAt: new Date(sentAt - 5).toISOString(), finishedAt: null }] } as never);
const artifacts = await observer.collectArtifacts();
expect(artifacts.stop?.runsStartedBeforeContinue).toEqual(["r2"]);
});
it.each([
["agent-owned interrupted session", { id: "a-agent", status: "cancelled", ownerType: "agent" }],
["system reconciliation resolved as cancelled", { id: "a-system", status: "resolved", ownerType: "system", outcome: "cancelled" }],
])("records no stall for a blocked subtask the product does not list as stalled (%s), but still clicks its Retry", async (_name, action) => {
let childStatus = "blocked";
const base = fakeApi({ "parentIds=": () => [{ id: "c1", parentId: "i1", status: "done" }, { id: "c2", parentId: "i1", status: childStatus }],
"/documents/": () => null, "/recovery-actions": () => ({ active: null, actions: [action] }),
"/api/issues/i1": route => route === "/api/issues/i1" ? { id: "i1", status: "blocked", stalledDependencies: [] } : null,
"/api/issues/c2": () => ({ id: "c2", status: childStatus, interruption: { canRetry: childStatus === "blocked", observedRunId: "child-run" } }) }, "blocked");
const childRetries: string[] = [];
const api = { ...base.api, get: base.api.get.bind(base.api), async post(route: string, body: unknown) {
if (route === "/api/issues/c2/retry") { childRetries.push(route); childStatus = "in_progress"; return { accepted: true }; }
return (base.api as unknown as { post(route: string, body: unknown): Promise<unknown> }).post(route, body);
} } as unknown as ChaosApi;
const observer = new TaskObserver({ ...plan, workload: "M1", followUpAfterMs: null }, api, lab, new Timeline(null), ready);
await observer.launch();
await observer.poll(observer.record.createdAt! + 1_000);
expect(observer.record.childStalls ?? []).toEqual([]);
expect(childRetries).toEqual(["/api/issues/c2/retry"]);
});
it("closes an open task as unfinished at interrupt, so the wave grades it as a failure", async () => {
const { api } = fakeApi({}, "in_progress");
const timeline = new Timeline(null);
const observer = new TaskObserver({ ...plan, workload: "A1", followUpAfterMs: null }, api, lab, timeline, ready);
await observer.launch();
await observer.poll(observer.record.createdAt! + 1_000);
observer.markUnfinished();
expect(observer.done).toBe(true);
expect(observer.record).toMatchObject({ timedOut: true, finalStatus: "in_progress" });
expect(timeline.entries.some(entry => entry.kind === "task_unfinished_at_interrupt")).toBe(true);
const grade = gradeTask(observer.record, [], observer.definition.budgetMs, { verdict: "pass", checks: [] });
expect(grade.pass).toBe(false);
expect(grade.rules.R1.detail).toMatch(/no_rest_within_budget/);
});
it("fires the planned fault once, after its offset", async () => {
const { api } = fakeApi();
const observer = new TaskObserver({ ...plan, fireAfterMs: 5_000 }, api, lab, new Timeline(null), ready);
await observer.launch();
const start = observer.record.createdAt!;
expect(observer.faultDue(start + 4_999)).toBeNull();
expect(observer.faultDue(start + 5_000)).toEqual({ kind: "K1", holdMs: 0, atMs: 5_000 });
expect(observer.faultDue(start + 6_000)).toBeNull();
});
});
describe("fixture repository", () => {
it("lists workspace files without VCS or cache directories", async () => {
const root = await mkdtemp(path.join(os.tmpdir(), "restart-chaos-fixture-"));
directories.push(root);
await createFixtureRepository(root);
const files = await listWorkspaceFiles(root);
expect(files).toContain("chaoscli/cli.py");
expect(files.some(file => file.startsWith(".git/"))).toBe(false);
});
it("bench.sh writes a tagged completion record with the deterministic result", async () => {
const root = await mkdtemp(path.join(os.tmpdir(), "restart-chaos-bench-"));
directories.push(root);
await createFixtureRepository(root, 1);
const output = execFileSync("./bench.sh", ["abcd1234"], { cwd: root, encoding: "utf8", timeout: 60_000 });
expect(output.trim().split("\n").at(-1)).toBe(BENCH_EXPECTED);
const record = parseBenchRecord(await readFile(path.join(root, ".bench", "run-abcd1234.txt"), "utf8"));
expect(record?.result).toBe(BENCH_EXPECTED);
}, 60_000);
});
describe("childrenPending", () => {
it("waits on running subtasks and stops waiting once each is done, cancelled or blocked", () => {
expect(childrenPending(null)).toBe(true);
expect(childrenPending([{ status: "done" }, { status: "in_progress" }])).toBe(true);
expect(childrenPending([{ status: "done" }, { status: "cancelled" }])).toBe(false);
expect(childrenPending([{ status: "done" }, { status: "blocked" }])).toBe(false);
});
});
describe("childSpeedTasksInTurn", () => {
it("reads each observer's subtasks one at a time so a large wave does not flood the API", async () => {
let inFlight = 0, peak = 0;
const observers = [1, 2, 3].map(id => ({ childSpeedTasks: async () => {
inFlight += 1; peak = Math.max(peak, inFlight);
await new Promise(resolve => setTimeout(resolve, 5));
inFlight -= 1;
return [{ issueId: `c${id}`, adapter: "codex" as const, samples: [] }];
} }));
expect((await childSpeedTasksInTurn(observers)).map(task => task.issueId)).toEqual(["c1", "c2", "c3"]);
expect(peak).toBe(1);
});
});
describe("run samples", () => {
it("keeps the board's failure cause for U3 and omits it for older servers", () => {
const base = { runId: "r1", status: "failed", errorCode: "adapter_failed" };
expect(toRunSample({ ...base, failureCause: "40001" }).failureCause).toBe("40001");
expect(toRunSample(base)).not.toHaveProperty("failureCause");
expect(toRunSample({ ...base, failureCause: null })).not.toHaveProperty("failureCause");
});
});