test(restart-chaos): count distinct warm close failures and let R5 wait out environment acquire (#21852)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness (`tests/restart-chaos`) runs zero-downtime
gate waves and grades each wave and each task
> - Gate wave A1 of rerun 6 (w-20261116T001158-s7106) failed. Its "warm
sessions that did not close" count was 28, but only 2 sessions failed to
close. Task t010 also failed R5, but its run was healthy.
> - The harness's environment-delete retry loop re-ran one failing close
every 3 s, so one failed session was logged 27 times. The harness
counted log lines, not failures. Also, no sandbox capture existed from
the time of the failed idle close, so the cause was not recorded.
> - R5 saw 253 s of silence on t010's run aedc1f6e. The run spent 251 s
in `environment.acquire` (a Daytona restore of an archived sandbox, with
one 90 s plugin call timeout). The "run started" event is written only
after acquire, so the run had no event during that time. The run then
succeeded.
> - This pull request makes the warm-shutdown grade count distinct (run,
reason) failures, captures the sandbox at the first idle-close failure,
and lets R5 wait up to 10 minutes while a running run has no run event
yet.
> - The benefit is a wave grade that shows the real number of failed
sessions, records why a close failed, and does not report a slow
environment acquire as a dead run.

## Linked Issues or Issue Description

**Bug description**
Three findings from gate wave A1 of rerun 6 (w-20261116T001158-s7106):
1. `warm-shutdown.ts` reports one failure per log line. The wave showed
29 shutdown close failures (28 "did not close" lines and 1 "cleanup
failed before controller shutdown" line) for 2 sessions: run 300f0d61 (2
lines) and run 70d909cc (27 lines from the environment-delete retries).
No sandbox capture was taken when the idle close of 70d909cc failed, so
why the Runner did not suspend is not known.
2. R5 reported `silent_dead_run aedc1f6e… 181s` for task t010. The run
was in environment acquire for 251 s. It then succeeded.
3. Task t024 failed its content check `subtask_1_done`. The child saved
its ledger with Read-tool line-number prefixes (`1\tROUND 1 ...`). The
grader requires lines that start with `ROUND`. This is model behaviour:
a ledger that the agent corrupts is a real content failure. This pull
request does NOT change that check.

**Steps to reproduce**
1. Grade the A1 server log lines (fixture
`fixtures/a1-warm-close-failures.log`) with `gradeWarmShutdown`. The
result has 29 `shutdownCloseFailures` entries.
2. Grade the t010 samples (fixture
`fixtures/t010-slow-environment-acquire.json`) with `gradeR5`. The
result is `fail: silent_dead_run aedc1f6e…`.

**Expected behavior**
1. The grade shows 2 distinct shutdown close failures, each with its
line count, and the verdict stays "fail".
2. R5 passes a run that has no run event yet for up to 10 minutes. R5
still fails a run with no event past 10 minutes, and a started run that
is silent past 3 minutes.

## What Changed

- `warm-shutdown.ts`: `shutdownCloseFailures` and `idleCloseFailures`
are now lists of distinct `{ runId, reason, lines, firstLine }` entries,
keyed by run and failure reason. The reason drops the JSON detail after
the error text (for example timings), so retries of the same failure
merge. A line with no parsable run keeps its own message as the reason.
Each distinct shutdown failure still adds a problem, so the verdict is
unchanged in meaning: any distinct failure fails the wave.
- `warm-shutdown.ts`: new `watchFirstIdleCloseFailure` splits server
output into lines per stream and reports the first idle-close failure
once. New optional `sandboxCaptures` input and output, listed in the
markdown section.
- `supervisor.ts`: new `onServerOutput(chunk, source)` hook, called for
the primary and the second server's stdout and stderr. A hook error is
recorded in the timeline and never breaks logging.
- `main.ts`: on a Daytona warm wave, the first idle-close failure
triggers one `captureRunnerLogs("idle-close-failure-<run>")` (the class
R probe: processes, sockets, Runner runtime files including
`runner-state.json`, and logs). The harness waits for that capture
before the environment cleanup and before the warm-shutdown grade.
- `observer.ts` / `types.ts`: a running run whose events read back empty
(the read succeeded and there is no event yet) gets `preparing: true` in
its sample. A failed events read does not set it.
- `grader.ts`: new `PREPARATION_LIMIT_MS` (10 minutes). `deadRunningRun`
uses it when the run sample has `preparing`.
- Why 10 minutes: the observed acquire took 251 s, including one 90 s
plugin call timeout. The Daytona provider bounds each start or restore
at 5 minutes by default (`DEFAULT_DAYTONA_OPERATION_TIMEOUT_MS`), so 10
minutes covers one timed-out restore and its retry. This is the same
bound #21730 gave an active provider turn.
- New fixtures: `fixtures/a1-warm-close-failures.log` (the 34 real
close-failure lines of the A1 server log) and
`fixtures/t010-slow-environment-acquire.json` (the 87 real t010 samples
of run aedc1f6e, with the time of its first run event).
- `README.md`: document the distinct-failure count, the idle-close
capture and the preparation bound for R5.
- t024: no change. Decision: the ledger check stays strict, because a
ledger that the agent saved with line-number prefixes is a real content
failure.

## Verification

- Red on `origin/master` sources, green with this change (9 tests):
- `warm-shutdown.test.ts` › "reports each distinct (run, reason) once,
not each log line (w-20261116T001158-s7106 A1: 2 sessions, 29 lines)": 2
failures (300f0d61 with 2 lines, 70d909cc with 27 lines), 2 problems,
verdict fail.
- `warm-shutdown.test.ts` › "still fails on one distinct failure, and
keeps different reasons for the same run apart" (guards: the 27 lines of
one session still fail; two reasons for one run give two problems; an
unparsable line still fails).
- `warm-shutdown.test.ts` › "names the first idle-close failure once,
across chunk boundaries and interleaved streams" (a forced-stop line
does not trigger it).
- `warm-shutdown.test.ts` › "lists the sandbox captures taken at the
first idle-close failure", plus the two updated shape tests.
- `triggers.test.ts` › R5 live evidence › "does not flag a run that is
still acquiring its environment before its first run event
(w-20261116T001158-s7106/t010)": the real samples still fail without
`preparing`, and pass with `preparing` on the samples before the first
run event.
- `triggers.test.ts` › R5 live evidence › "still fails a run silent past
the preparation bound, and a started run silent past the stall bound"
(guards: 9.5 min preparing passes; 10.5 min preparing fails; 4 min quiet
without `preparing` fails).
- `triggers.test.ts` › R5 live evidence › "marks a running run as
preparing only while its events read back empty" (the flag clears after
the "run started" event; a failed events read never sets it).
- `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 71
files, 833 tests pass.
- `npx tsc -p tests/restart-chaos/tsconfig.json --noEmit`: clean.
- No wave was run. The live idle-close capture path in `main.ts` is not
exercised by a unit test.

## Risks

- Low risk. The change touches only the test harness.
- `warm-shutdown.json` changes shape: `shutdownCloseFailures` and
`idleCloseFailures` are now objects, not strings. Scripts that read
these fields as strings must read `firstLine`. Summary counts now show
sessions, not lines, so they are lower than before for the same log.
- A run that dies before its first run event now fails R5 after 10
minutes instead of 3. The fault is still caught, but later.
- The idle-close capture can add up to 45 s (the capture deadline)
before the environment cleanup on a wave that had an idle-close failure.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code
with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-15 20:55:32 -06:00
1 parent d149e30323
commit c5218f5ec4
15 files changed
+335 -28

No files matched your search

+9 -3
View File
@@ -199,8 +199,14 @@ also grades "no leaked sandboxes / clean warm shutdown" (`warm-shutdown.ts`). It
the wave when the environment delete stayed blocked, when a wave sandbox is still
running after cleanup, or when the server log shows a warm session that did not close
on environment delete or controller shutdown. Idle closes that needed a forced Runner
stop, or whose sandbox was already destroyed, are counted but do not fail. The result
is in `<wave>/warm-shutdown.json` and in the wave's `SUMMARY.md`.
stop, or whose sandbox was already destroyed, are counted but do not fail. Both counts
are distinct (run, reason) failures, not log lines: the harness's environment-delete
retries re-log the same failed close every few seconds (one session logged 27 times in
w-20261116T001158-s7106). Each failure keeps its line count. On a Daytona warm wave, the
first idle-close failure in the server output triggers one sandbox capture
(`daytona-runner/idle-close-failure-<run>/`: processes, Runner state and logs), listed
under `sandboxCaptures`. The result is in `<wave>/warm-shutdown.json` and in the wave's
`SUMMARY.md`.
## Hosted mode (`--hosted-base-url`, `--hosted-allow`)
@@ -411,7 +417,7 @@ part of U3 is checked only in waves run with this harness.
temporary instance is deleted.
- Faults are serialized; each one hits every task running at that moment.
- A `todo` or `in_progress` task with no live run and no named wait is not a rest. The observer keeps watching it until it resumes or passes the R5 stall limit.
- R5 also fails a run that stays `running` with no live evidence (no new run event, `lastUsefulActionAt` or `execution.lastConfirmedActivityAt`, and no start) for more than 3 minutes after the server is ready. A `running` status alone is not proof of life. A quiet run is still live while a tool it started has not ended (for example a 4-minute `bench.sh`) or while its recorded local runner process still runs with the same PID and start time. While a provider turn is active and no tool runs, one model request is outstanding: R5 allows up to 10 minutes of quiet before it counts a dead run. A slow OpenRouter request took 303.6 s in w-20261115T044649-s7106/t028, and common provider clients time out a request at 10 minutes.
- R5 also fails a run that stays `running` with no live evidence (no new run event, `lastUsefulActionAt` or `execution.lastConfirmedActivityAt`, and no start) for more than 3 minutes after the server is ready. A `running` status alone is not proof of life. A quiet run is still live while a tool it started has not ended (for example a 4-minute `bench.sh`) or while its recorded local runner process still runs with the same PID and start time. While a provider turn is active and no tool runs, one model request is outstanding: R5 allows up to 10 minutes of quiet before it counts a dead run. A slow OpenRouter request took 303.6 s in w-20261115T044649-s7106/t028, and common provider clients time out a request at 10 minutes. A run with no run event yet (`preparing`: its events read back empty) is still acquiring or preparing its environment, because "run started" is written only after that. R5 allows it 10 minutes too: in w-20261116T001158-s7106/t010 a Daytona restore of an archived sandbox took 251 s, including one 90 s plugin call timeout, and the Daytona provider bounds each start or restore at 5 minutes by default, so 10 minutes covers one timed-out restore and its retry. U1 uses the same bound (`PREPARATION_LIMIT_MS`) before it stops counting a preparing run as the task's owner.
- The wave fingerprints the checkout and `packages/paperclip-runner/dist` at start and after every fault. A build, install or branch switch during a wave would make later restarts run different code, so it fails the wave as a harness error (`worktree_changed_mid_wave`). The baseline is taken after the harness's own setup (preflight rebuilds provider entrypoints) and covers the tracked diff, untracked file contents, the runner `dist` and the runnerd binary. Build before the wave, never during it.
- `faults.json` and `task-faults.json` (each task's launch time and the faults that crossed it) are written after every fault, so `--replay` of a task in an interrupted or wedged wave, or a storm task, re-fires the faults that crossed it at the same offsets.
- A fault that fails to complete fails the wave as a harness error (exit 2), even when recovery brings the server back.
+2 -2
View File
@@ -21,9 +21,9 @@ export function daytonaCleanupLine(sweep: DaytonaSweepResult, cleanup: WaveClean
}
export async function finishDaytonaCleanup(input: { sweeper: Pick<DaytonaWaveSweeper, "sweep" | "cleanUp">; sweep: DaytonaSweepResult | null; waveDir: string;
timeline: { add(kind: string, data: Record<string, unknown>): void }; log?: (line: string) => void }): Promise<DaytonaCleanupOutcome> {
timeline: { add(kind: string, data: Record<string, unknown>): void }; log?: (line: string) => void; beforeSandboxDelete?: () => Promise<void> }): Promise<DaytonaCleanupOutcome> {
const sweep = input.sweep ?? await input.sweeper.sweep(0);
const cleanup = await input.sweeper.cleanUp();
const cleanup = await input.sweeper.cleanUp(undefined, undefined, input.beforeSandboxDelete);
input.timeline.add("daytona_cleanup", cleanup);
(input.log ?? console.info)(daytonaCleanupLine(sweep, cleanup));
if (cleanup.environmentDeleteBlocked) input.timeline.add("daytona_environment_delete_blocked", { detail: cleanup.environmentDeleteBlocked });
+16
View File
@@ -61,6 +61,22 @@ describe("cleanup through Paperclip", () => {
expect(rows.find(row => row.id === "other-env")!.state).toBe("archived");
});
it("waits for a capture the environment delete triggered before it deletes any wave sandbox", async () => {
const rows = waveRows(["started"]);
const order: string[] = [];
const { client, removeSandbox } = deletingDaytona(rows);
const sweeper = new DaytonaWaveSweeper({ apiKey: "unused", companyId: COMPANY, environmentId: ENVIRONMENT,
api: { delete: async () => { order.push("environment delete"); } } as never,
removeSandbox: async id => { order.push(`delete ${id}`); await removeSandbox(id); } });
(sweeper as unknown as { client: unknown }).client = client;
const result = await sweeper.cleanUp(0, 1_000, async () => { await new Promise(resolve => setTimeout(resolve, 20)); order.push("capture"); });
expect(order).toEqual(["environment delete", "capture", "delete started"]);
expect(result.errors).toEqual([]);
const failing = new DaytonaWaveSweeper({ apiKey: "unused", companyId: COMPANY, environmentId: ENVIRONMENT, api: { delete: async () => undefined } as never });
(failing as unknown as { client: unknown }).client = readOnlyDaytona(listing);
expect((await failing.cleanUp(0, 1_000, async () => { throw new Error("probe failed"); })).errors).toEqual(["before sandbox delete: probe failed"]);
});
it("fails cleanup when a wave sandbox survives deletion, and never deletes a fleet snapshot even with the wave's labels", async () => {
const rows = [...waveRows(["archived"]), ...waveRows(["fleet"], { snapshot: "fleet-sbx-9", state: "archived" })];
const { client, removeSandbox, removed } = deletingDaytona(rows);
+2 -1
View File
@@ -215,7 +215,7 @@ export class DaytonaWaveSweeper {
}
}
async cleanUp(settleMs = 90_000, deleteWindowMs = R6_SETTLE_MS): Promise<WaveCleanup> {
async cleanUp(settleMs = 90_000, deleteWindowMs = R6_SETTLE_MS, beforeSandboxDelete?: () => Promise<void>): Promise<WaveCleanup> {
const errors: string[] = [];
let environmentDeleted = false;
let environmentDeleteBlocked: string | null = null;
@@ -229,6 +229,7 @@ export class DaytonaWaveSweeper {
if (/returned 409\b/.test(message)) environmentDeleteBlocked = message;
else errors.push(`environment delete: ${message}`);
}
await beforeSandboxDelete?.().catch(error => { errors.push(`before sandbox delete: ${error instanceof Error ? error.message : String(error)}`); });
let remainingSandboxes: string[] = [];
let archivedSandboxes: string[] = [];
// Provider teardown is asynchronous; give it a bounded settle before reporting.
@@ -0,0 +1,34 @@
[18:30:55] WARN: idle warm native session close failed {"runId":"300f0d61-548b-4630-8f6a-f79292ed9d24","issueId":"6577ff15-9c90-4b0c-812c-ee1f6e2226b0","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"287cbe35-0b93-46f4-89f4-5a0430592870","closeFailure":"runner_remote_process_identity_unavailable: runner_remote_process_identity_unavailable [process_verification/process_identity_changed]","closeAttempt":1}
[18:31:00] WARN: idle warm native session close failed {"runId":"300f0d61-548b-4630-8f6a-f79292ed9d24","issueId":"6577ff15-9c90-4b0c-812c-ee1f6e2226b0","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"287cbe35-0b93-46f4-89f4-5a0430592870","closeFailure":"runner_remote_process_identity_unavailable: runner_remote_process_identity_unavailable [process_verification/process_identity_changed]","closeAttempt":2}
[18:31:05] WARN: idle warm native session close failed {"runId":"300f0d61-548b-4630-8f6a-f79292ed9d24","issueId":"6577ff15-9c90-4b0c-812c-ee1f6e2226b0","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"287cbe35-0b93-46f4-89f4-5a0430592870","closeFailure":"runner_remote_process_identity_unavailable: runner_remote_process_identity_unavailable [process_verification/process_identity_changed]","closeAttempt":3}
[18:31:10] WARN: idle warm native session close failed {"runId":"300f0d61-548b-4630-8f6a-f79292ed9d24","issueId":"6577ff15-9c90-4b0c-812c-ee1f6e2226b0","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"287cbe35-0b93-46f4-89f4-5a0430592870","closeFailure":"runner_remote_process_identity_unavailable: runner_remote_process_identity_unavailable [process_verification/process_identity_changed]","closeAttempt":4}
[18:32:50] WARN: idle warm native sessions did not close {"reason":"controller restart","failures":[{"runId":"300f0d61-548b-4630-8f6a-f79292ed9d24","issueId":"6577ff15-9c90-4b0c-812c-ee1f6e2226b0","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"287cbe35-0b93-46f4-89f4-5a0430592870","reason":"close failed: runner_remote_process_identity_unavailable: runner_remote_process_identity_unavailable [process_verification/process_identity_changed]"}]}
[18:32:50] WARN: idle native session cleanup failed before controller shutdown {"idleSessions":{"closed":0,"busy":0,"failed":1,"failures":[{"runId":"300f0d61-548b-4630-8f6a-f79292ed9d24","issueId":"6577ff15-9c90-4b0c-812c-ee1f6e2226b0","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"287cbe35-0b93-46f4-89f4-5a0430592870","reason":"close failed: runner_remote_process_identity_unavailable: runner_remote_process_identity_unavailable [process_verification/process_identity_changed]"}]}}
[19:11:36] WARN: idle warm native session close failed {"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","closeFailure":"native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}","closeAttempt":1}
[19:12:13] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:12:14] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:12:16] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:12:19] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:12:22] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:12:25] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:12:28] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:12:31] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:12:34] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:12:40] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:12:43] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:12:46] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:12:49] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:12:52] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:12:55] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:12:58] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:13:01] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:13:04] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:13:12] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:13:15] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:13:18] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:13:21] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:13:24] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:13:27] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:13:30] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:13:33] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
[19:13:36] WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[{"runId":"70d909cc-8b04-40ae-a0bf-0846df140cf0","issueId":"0605dbde-cd34-4728-bc68-3c1fe9ca4c06","environmentId":"d729f12b-4d52-4e10-8cae-e03bdf73b2a3","providerLeaseId":"2f139bcc-8503-4722-8943-0defefeecbab","reason":"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint {\"runnerSuspended\":false,\"runnerRetained\":true,\"providerDrained\":true,\"suspensionState\":{\"commandStatus\":\"failed\",\"runnerLifecycle\":\"ready\",\"runnerIdentityMatches\":true},\"semanticTools\":{\"persistenceFailed\":false,\"failures\":[],\"pending\":[],\"deliveries\":[]},\"finalProviderState\":null,\"timings\":{\"exitProbeMs\":0,\"stopMs\":0,\"drainMs\":225,\"suspendQueuedAfterMs\":225,\"proofReadMs\":4290,\"barrierExitProbeMs\":0,\"suspendMs\":5239,\"retainProbeMs\":0,\"releaseMs\":0,\"retireMs\":0,\"checkpointMs\":591,\"totalMs\":6055},\"proofWindowEnd\":\"suspend_not_completed\"}"}]}
@@ -0,0 +1 @@
{"source":"w-20261116T001158-s7106/t010 (CHA-14): follow-up run aedc1f6e spent 251 s in environment.acquire (Daytona restore of an archived sandbox, one 90 s plugin RPC timeout); its first run event (\"run started\") was written after acquire; the run then succeeded","runId":"aedc1f6e-3a5e-42d2-97ab-295f19e70bb8","firstRunEventAt":"2026-11-16T01:08:19.648Z","samples":[{"at":1794791045930,"reachable":true,"serverReady":true,"issueStatus":"in_progress","interruption":{"state":"working","canRetry":false,"retryWait":false,"workspaceFailure":null,"observedRunId":"aedc1f6e-3a5e-42d2-97ab-295f19e70bb8","userAction":null},"activeRecoveryAction":false,"recoveryOwner":null,"executionBlocker":false,"scheduledRetry":false,"runs":[{"id":"aedc1f6e-3a5e-42d2-97ab-295f19e70bb8","status":"running","createdAt":"2026-11-16T00:59:12.296Z","startedAt":"2026-11-16T01:04:05.882Z","finishedAt":null,"errorCode":null,"retryOfRunId":null,"wakeCommentIds":[],"livenessState":null,"lastEvidenceAt":"2026-11-16T01:04:05.882Z"},{"id":"91d61efb-4a8b-4ed2-87ba-f640d6065cbf","status":"succeeded","createdAt":"2026-11-16T00:26:58.075Z","startedAt":"2026-11-16T00:26:58.346Z","finishedAt":"2026-11-16T00:38:17.936Z","errorCode":null,"retryOfRunId":null,"wakeCommentIds":[],"livenessState":"blocked","lastEvidenceAt":"2026-11-16T00:38:17.875Z"}]},{"at":1794791049786,"reachable":true,"serverReady":true,"issueStatus":"in_progress","interruption":{"state":"working","canRetry":false,"retryWait":false,"workspaceFailure":null,"observedRunId":"aedc1f6e-3a5e-42d2-97ab-295f19e70bb8","userAction":null},"activeRecoveryAction":false,"recoveryOwner":null,"executionBlocker":false,"scheduledRetry":false,"runs":[{"id":"aedc1f6e-3a5e-42d2-97ab-295f19e70bb8","status":"running","createdAt":"2026-11-16T00:59:12.296Z","startedAt":"2026-11-16T01:04:05.882Z","finishedAt":null,"errorCode":null,"retryOfRunId":null,"wakeCommentIds":[],"livenessState":null,"lastEvidenceAt":"2026-11-16T01:04:05.882Z"},{"id":"91d61efb-4a8b-4ed2-87ba-f640d6065cbf","status":"succeeded","createdAt":"2026-11-16T00:26:58.075Z","startedAt":"2026-11-16T00:26:58.346Z","finishedAt":"2026-11-16T00:38:17.936Z","errorCode":null,"retryOfRunId":null,"wakeCommentIds":[],"livenessState":"blocked","lastEvidenceAt":"2026-11-16T00:38:17.875Z"}]},{"at":1794791053308,"reachable":true,"serverReady":true,"issueStatus":"in_progress","interruption":{"state":"working","canRetry":false,"retryWait":false,"workspaceFailure":null,"observedRunId":"aedc1f6e-3a5e-42d2-97ab-295f19e70bb8","userAction":null},"activeRecoveryAction":false,"recoveryOwner":null,"executionBlocker":false,"scheduledRetry":false,"runs":[{"id":"aedc1f6e-3a5e-42d2-97ab-295f19e70bb8","status":"running","createdAt":"2026-11-16T00:59:12.296Z","startedAt":"2026-11-16T01:04:05.882Z","finishedAt":null,"errorCode":null,"retryOfRunId":null,"wakeCommentIds":[],"livenessState":null,"lastEvidenceAt":"2026-11-16T01:04:05.882Z"},{"id":"91d61efb-4a8b-4ed2-87ba-f640d6065cbf","status":"succeeded","createdAt":"2026-11-16T00:26:58.075Z","startedAt":"2026-11-16T00:26:58.346Z","finishedAt":"2026-11-16T00:38:17.936Z","errorCode":null,"retryOfRunId":null,"wakeCommentIds":[],"livenessState":"blocked","lastEvidenceAt":"2026-11-16T00:38:17.875Z"}]},{"at":1794791056820,"reachable":true,"serverReady":true,"issueStatus":"in_progress","interruption":{"state":"working","canRetry":false,"retryWait":false,"workspaceFailure":null,"observedRunId":"aedc1f6e-3a5e-42d2-97ab-295f19e70bb8","userAction":null},"activeRecoveryAction":false,"recoveryOwner":null,"executionBlocker":false,"scheduledRetry":false,"runs":[{"id":"aedc1f6e-3a5e-42d2-97ab-295f19e70bb8","status":"running","createdAt":"2026-11-16T00:59:12.296Z","startedAt":"2026-11-16T01:04:05.882Z","finishedAt":null,"errorCode":null,"retryOfRunId":null,"wakeCommentIds":[],"livenessState":null,"lastEvidenceAt":"2026-11-16T01:04:05.882Z"},{"id":"91d61efb-4a8b-4ed2-87ba-f640d6065cbf","status":"succeeded","createdAt":"2026-11-16T00:26:58.075Z","startedAt":"2026-11-16T00:26:58.346Z","finishedAt":"2026-11-16T00:38:17.936Z","errorCode":null,"retryOfRunId":null,"wakeCommentIds":[],"livenessState":"blocked","lastEvidenceAt":"2026-11-16T00:38:17.875Z"}]},{"at":1794791060250,"reachable":true,"serverReady":true,"issueStatus":"in_progress","interruption":{"state":"working","canRetry":false,"retryWait":false,"workspaceFailure":null,"observedRunId":"aedc1f6e-3a5e-42d2-97ab-295f19e70bb8","userAction":null},"activeRecoveryAction":false,"recoveryOwner":null,"executionBlocker":false,"scheduledRetry":false,"runs":[{"id":"aedc1f6e-3a5e-42d2-97ab-295f19e70bb8","status":"running","createdAt":"2026-11-16T00:59:12.296Z","startedAt":"2026-11-16T01:04:05.882Z","finishedAt":null,"errorCode":null,"retryOfRunId":null,"wakeCommentIds":[],"livenessState":null,"lastEvidenceAt":"2026-11-16T01:04:05.882Z"},{"id":"91d61efb-4a8b-4ed2-87ba-f640d6065cbf","status":"succeeded","createdAt":"2026-11-16T00:26:58.075Z","startedAt":"2026-11-16T00:26:58.346Z","finishedAt":"2026-11-16T00:Line truncated
+2 -1
View File
@@ -8,6 +8,7 @@ import { enforcedUniversalFailures, gradeUniversal, type SlotFree, type Universa
export const ACTIVE_RUN_STATUSES = new Set(["queued", "running", "scheduled_retry"]);
export const STALL_LIMIT_MS = 3 * 60_000;
export const PROVIDER_TURN_LIMIT_MS = 10 * 60_000;
export const PREPARATION_LIMIT_MS = 10 * 60_000;
const REST_STATUSES_WITH_WORK = new Set(["todo", "in_progress"]);
const ms = (iso: string | null | undefined) => (iso ? Date.parse(iso) : NaN);
@@ -211,7 +212,7 @@ export function deadRunningRun(sample: TaskSample, readySince: number): { id: st
if (run.status !== "running") continue;
// Quiet is not dead while a tool is still in flight or the run's process is verifiably alive.
if (run.toolInFlight || run.processAlive) continue;
const limit = run.providerTurnActive ? PROVIDER_TURN_LIMIT_MS : STALL_LIMIT_MS;
const limit = run.preparing ? PREPARATION_LIMIT_MS : run.providerTurnActive ? PROVIDER_TURN_LIMIT_MS : STALL_LIMIT_MS;
const evidence = Math.max(ms(run.lastEvidenceAt) || 0, ms(run.startedAt) || 0, readySince);
if (sample.at - evidence > limit) return { id: run.id, idleMs: sample.at - Math.max(ms(run.lastEvidenceAt) || 0, ms(run.startedAt) || 0) };
}
+19 -4
View File
@@ -44,7 +44,7 @@ import { matrixMayLaunch, planMatrixWave } from "./journey-matrix.js";
import { redeployWave } from "./redeploy.js";
import { withT1CrossingCheck } from "./workloads.js";
import { parseRecoveryStepReceipts, withRecoveryStepCheck } from "./recovery-step-faults.js";
import { captureWaveRunnerLogs, executeInWaveSandboxes } from "./daytona-runner-logs.js";
import { captureWaveRunnerLogs, executeInWaveSandboxes, RUNNER_LOG_CAPTURE_DEADLINE_MS } from "./daytona-runner-logs.js";
import { isClassRFault, killedPidsFromOutput, providerKillCommand, watchLabels, withClassRCheck, type ProviderKillRecord } from "./class-r.js";
import { TASK_FAULT_LOG, observedFaultSchedule, planWave, replayOnCurrentBuild, replayRecordFor, replayWave, type TaskFaultLog } from "./scheduler.js";
import { chaosLifecycleOverrides, seedChaosLab } from "./seed.js";
@@ -59,7 +59,7 @@ import { guardedRunnerOutputs, worktreeFingerprint } from "./wave-guard.js";
import { checkCrossServer, readCrossServerEvidence } from "./multi-server.js";
import { ReleaseSwapWave, stormRunFor } from "./release-swap-setup.js";
import { runnerBindFaultServerEnvironment, sessionFaultServerEnvironment } from "./session-faults.js";
import { controllerLogNames, gradeWarmShutdown, isWarmWave, warmShutdownMarkdown, type WarmShutdownCleanup } from "./warm-shutdown.js";
import { controllerLogNames, gradeWarmShutdown, isWarmWave, waitBounded, warmShutdownMarkdown, watchFirstIdleCloseFailure, type WarmShutdownCleanup } from "./warm-shutdown.js";
import type { FaultKind, FaultReceipt, OracleArtifacts, TriggerId, WavePlan } from "./types.js";
const repositoryRoot = path.resolve(import.meta.dirname, "../..");
@@ -318,6 +318,12 @@ async function main(): Promise<number> {
let storageSection: string | null = null;
let warmShutdownSection: string | null = null;
let daytonaCleanup: WarmShutdownCleanup | null = null;
let idleCloseCapture: Promise<void> | null = null;
const idleCloseCaptures: string[] = [];
const waitForIdleCloseCapture = async (phase: string) => {
const outcome = await waitBounded(() => idleCloseCapture, RUNNER_LOG_CAPTURE_DEADLINE_MS + 5_000);
if (outcome === "timed_out") timeline.add("idle_close_failure_capture_wait_timed_out", { phase });
};
let harnessError: unknown;
process.on("uncaughtException", error => {
if (isSocketTypeOfServiceRace(error)) { timeline.add("harness_socket_race_ignored", { error: String(error) }); return; }
@@ -461,6 +467,12 @@ async function main(): Promise<number> {
if (sandboxEvidence) {
runnerLogTarget = { companyId: owned.companyId, environmentId: owned.environmentId };
if (redeployWave(plan)) supervisor.redeployEvidence = (faultId, phase) => captureRunnerLogs(`${faultId}-${phase}`);
if (isWarmWave(plan.options)) supervisor.onServerOutput = watchFirstIdleCloseFailure(failure => {
const label = `idle-close-failure-${(failure.runId ?? "unknown").slice(0, 8)}`;
idleCloseCaptures.push(label);
timeline.add("idle_close_failure_seen", { runId: failure.runId, label, line: failure.line.slice(0, 2_000) });
idleCloseCapture = captureRunnerLogs(label).catch(error => { timeline.add("idle_close_failure_capture_failed", { label, error: String(error) }); });
});
// `--sandbox-watch-seconds`: capture on a clock too, for Runner exits that no fault announces (W2 re-wake).
if (plan.options.sandboxWatchMs) faultWork.push(watchSandboxes(plan.options.sandboxWatchMs));
}
@@ -919,8 +931,10 @@ async function main(): Promise<number> {
if (supervisor.serverReady) await recordWaveSpend().catch(() => undefined);
await board?.close().catch(() => undefined);
await closeJourneyApps();
await waitForIdleCloseCapture("before_cleanup");
try { if (sweeper) try {
const outcome = await finishDaytonaCleanup({ sweeper: sweeper as DaytonaWaveSweeper, sweep: sandboxSweep, waveDir, timeline });
const outcome = await finishDaytonaCleanup({ sweeper: sweeper as DaytonaWaveSweeper, sweep: sandboxSweep, waveDir, timeline,
beforeSandboxDelete: () => waitForIdleCloseCapture("before_sandbox_delete") });
sandboxSweep = outcome.sweep;
daytonaCleanup = outcome.cleanup;
if (outcome.failed) anyFailure = true;
@@ -942,7 +956,8 @@ async function main(): Promise<number> {
const logDir = path.dirname(supervisor.paths.serverLogPath);
const controllerLogs = controllerLogNames(await readdir(logDir).catch(() => [] as string[]));
const serverLog = (await Promise.all(controllerLogs.map(name => readFile(path.join(logDir, name), "utf8").catch(() => "")))).join("\n");
const warmShutdown = gradeWarmShutdown({ cleanup: daytonaCleanup, serverLog });
await waitForIdleCloseCapture("before_grade");
const warmShutdown = gradeWarmShutdown({ cleanup: daytonaCleanup, serverLog, sandboxCaptures: idleCloseCaptures });
timeline.add("warm_shutdown", { ...warmShutdown });
if (warmShutdown.verdict === "fail") anyFailure = true;
warmShutdownSection = warmShutdownMarkdown(warmShutdown);
+3
View File
@@ -555,8 +555,10 @@ export class TaskObserver {
private async runEvidence(run: RunSample): Promise<RunSample> {
if (run.status !== "running") return run;
const cursor = this.runEventCursors.get(run.id) ?? { seq: 0, lastAt: null, openTools: 0, process: null };
let eventsRead = false;
for (let page = 0; page < 3; page += 1) {
const events = await this.api.get<Json[]>(`/api/heartbeat-runs/${run.id}/events?afterSeq=${cursor.seq}&limit=${RUN_EVENT_PAGE}`, 5_000).catch(() => null);
if (Array.isArray(events)) eventsRead = true;
if (!events?.length) break;
for (const event of events) {
if (typeof event.seq === "number" && event.seq > cursor.seq) cursor.seq = event.seq;
@@ -571,6 +573,7 @@ export class TaskObserver {
}
this.runEventCursors.set(run.id, cursor);
return { ...run, lastEvidenceAt: latestIso([run.lastEvidenceAt, cursor.lastAt]),
...(eventsRead && cursor.seq === 0 ? { preparing: true } : {}),
...(cursor.openTools > 0 ? { toolInFlight: true } : {}),
...(cursor.turnActive ? { providerTurnActive: true } : {}),
...(cursor.process && this.processAlive(cursor.process.pid, cursor.process.startedAt) ? { processAlive: true } : {}) };
+9 -4
View File
@@ -183,6 +183,7 @@ export class ChaosSupervisor {
deployShape: { overlap: boolean; freshHome: boolean } = { overlap: false, freshHome: false };
ephemeralServer = false;
swapHook: ((info: SwapPhaseInfo) => Promise<string | void>) | null = null;
onServerOutput: ((chunk: Buffer | string, source: string) => void) | null = null;
private rollingRedeploys = 0;
/** `--k15-fresh-home`: the instance home and config of the serving generation (generation 0 uses `paths`). */
private serverHome: { home: string; configPath: string };
@@ -232,6 +233,10 @@ export class ChaosSupervisor {
/** K14 waves: whether the older build or this checkout is serving. */
get serverBuild(): "upgrade_from" | "current" { return this.serverRoot === this.repositoryRoot ? "current" : "upgrade_from"; }
private noteServerOutput(chunk: Buffer | string, source: string): void {
try { this.onServerOutput?.(chunk, source); } catch (error) { this.timeline.add("server_output_hook_failed", { error: String(error) }); }
}
serverLogOffset(): number {
try { return statSync(this.paths.serverLogPath).size + this.log.writableLength; } catch { return 0; }
}
@@ -321,8 +326,8 @@ export class ChaosSupervisor {
this.child = child;
const window = this.trackWindow(child, "primary");
child.stdout?.on("data", chunk => this.log.write(chunk));
child.stderr?.on("data", chunk => this.log.write(chunk));
child.stdout?.on("data", chunk => { this.log.write(chunk); this.noteServerOutput(chunk, `${child.pid}:stdout`); });
child.stderr?.on("data", chunk => { this.log.write(chunk); this.noteServerOutput(chunk, `${child.pid}:stderr`); });
child.once("error", error => { this.spawnErrors.set(child, error); });
child.once("exit", (code, signal) => {
if (window) window.exitedAt = Date.now();
@@ -741,8 +746,8 @@ export class ChaosSupervisor {
record.secondary = { pid: child.pid ?? null, port, spawnedAt: Date.now(), readyAt: null, stoppedAt: null };
const secondary = { child, port, log, exit: null as string | null };
this.secondary = secondary;
child.stdout?.on("data", chunk => log.write(chunk));
child.stderr?.on("data", chunk => log.write(chunk));
child.stdout?.on("data", chunk => { log.write(chunk); this.noteServerOutput(chunk, `${child.pid}:stdout`); });
child.stderr?.on("data", chunk => { log.write(chunk); this.noteServerOutput(chunk, `${child.pid}:stderr`); });
child.once("error", error => { this.spawnErrors.set(child, error); });
child.once("exit", (code, signal) => {
if (window) window.exitedAt = Date.now();
+59 -2
View File
@@ -6,7 +6,8 @@ import path from "node:path";
import { describe, expect, it } from "vitest";
import { isSocketTypeOfServiceRace, type ChaosApi } from "./api.js";
import { parseCommand } from "./config.js";
import { gradeR5 } from "./grader.js";
import { gradeR5, PREPARATION_LIMIT_MS } from "./grader.js";
import { DEFAULT_UNIVERSAL_BOUNDS, gradeU1 } from "./universal.js";
import { latestIso, localProcessAlive, noteLivenessEvent, QUEUED_TRIGGER_MAX_EMPTY_SAMPLES, TaskObserver, TRIGGER_LATE_MS } from "./observer.js";
import { observedFaultSchedule, planTasks, taskFaultSchedule, TRIGGER_DELAY_RANGE_MS } from "./scheduler.js";
import type { ChaosLab } from "./seed.js";
@@ -424,7 +425,7 @@ describe("R5 live evidence", () => {
expect(gradeR5(record(slow.samples)).verdict).toBe("pass");
});
const quietTurn = (live: { providerTurnActive?: boolean }, quietMs: number) => {
const quietTurn = (live: { providerTurnActive?: boolean; preparing?: boolean }, quietMs: number) => {
const start = Date.parse("2026-11-15T05:00:00Z");
const run = { id: "r1", status: "running", createdAt: null, startedAt: new Date(start - 600_000).toISOString(), finishedAt: null,
errorCode: null, retryOfRunId: null, wakeCommentIds: [], livenessState: null, lastEvidenceAt: new Date(start).toISOString(), ...live };
@@ -440,6 +441,62 @@ describe("R5 live evidence", () => {
detail: expect.stringMatching(/^silent_dead_run r1 .* for 630s/) });
});
it("does not flag a run that is still acquiring its environment before its first run event (w-20261116T001158-s7106/t010)", () => {
const slow = JSON.parse(readFileSync(path.join(import.meta.dirname, "fixtures/t010-slow-environment-acquire.json"), "utf8")) as
{ runId: string; firstRunEventAt: string; samples: TaskSample[] };
const firstEventAt = Date.parse(slow.firstRunEventAt);
const quiet = slow.samples.filter(sample => sample.at < firstEventAt);
expect(quiet.at(-1)!.at - quiet[0]!.at).toBeGreaterThan(240_000);
expect(quiet.every(sample => sample.runs!.find(run => run.id === slow.runId)!.lastEvidenceAt
=== sample.runs!.find(run => run.id === slow.runId)!.startedAt)).toBe(true);
expect(gradeR5(record(slow.samples))).toMatchObject({ verdict: "fail", detail: expect.stringMatching(/^silent_dead_run aedc1f6e/) });
const observed = slow.samples.map(sample => ({ ...sample, runs: sample.runs?.map(run =>
run.id === slow.runId && run.status === "running" && sample.at < firstEventAt ? { ...run, preparing: true } : run) }));
expect(gradeR5(record(observed)).verdict).toBe("pass");
expect(gradeU1(record(observed), DEFAULT_UNIVERSAL_BOUNDS).verdict).toBe("pass");
});
it("R5 and U1 agree on a preparing run: a 6 and 7 minute restore passes both, silence past the preparation bound fails both", () => {
for (const quietMs of [360_000, 420_000]) {
expect(gradeR5(quietTurn({ preparing: true }, quietMs)).verdict).toBe("pass");
expect(gradeU1(quietTurn({ preparing: true }, quietMs), DEFAULT_UNIVERSAL_BOUNDS).verdict).toBe("pass");
}
const pastBound = PREPARATION_LIMIT_MS + DEFAULT_UNIVERSAL_BOUNDS.orphanMs + 60_000;
expect(gradeR5(quietTurn({ preparing: true }, pastBound))).toMatchObject({ verdict: "fail", detail: expect.stringMatching(/^silent_dead_run r1/) });
expect(gradeU1(quietTurn({ preparing: true }, pastBound), DEFAULT_UNIVERSAL_BOUNDS)).toMatchObject({ verdict: "fail",
detail: expect.stringMatching(/^no_owner_no_action/) });
expect(gradeU1(quietTurn({}, 420_000), DEFAULT_UNIVERSAL_BOUNDS).verdict).toBe("fail");
});
it("still fails a run silent past the preparation bound, and a started run silent past the stall bound", () => {
expect(gradeR5(quietTurn({ preparing: true }, 570_000)).verdict).toBe("pass");
expect(gradeR5(quietTurn({ preparing: true }, 630_000))).toMatchObject({ verdict: "fail",
detail: expect.stringMatching(/^silent_dead_run r1 .* for 630s/) });
expect(gradeR5(quietTurn({ preparing: false }, 240_000))).toMatchObject({ verdict: "fail", detail: expect.stringMatching(/^silent_dead_run r1/) });
});
it("marks a running run as preparing only while its events read back empty", async () => {
const startedAt = new Date().toISOString();
const state = { runs: [{ runId: "r1", status: "running", startedAt }] as Json[], issue: { id: "i1", status: "in_progress" }, events: [] as Json[] };
const observer = new TaskObserver({ ...base, workload: "B5" }, scriptedApi(state), lab, new Timeline(null), ready);
await observer.launch();
await observer.poll(Date.now());
expect(observer.record.samples.at(-1)!.runs![0]).toMatchObject({ preparing: true });
state.events.push({ seq: 1, eventType: "lifecycle", message: "run started", payload: {}, createdAt: startedAt });
await observer.poll(Date.now() + 3_000);
expect(observer.record.samples.at(-1)!.runs![0]!.preparing).toBeUndefined();
const unread = scriptedApi({ ...state, events: [] });
const failing = { ...unread, get: async (route: string) => {
if (route.includes("/events")) throw new Error("events unavailable");
return (unread as unknown as { get(route: string): Promise<unknown> }).get(route);
} } as unknown as ChaosApi;
const blind = new TaskObserver({ ...base, workload: "B5" }, failing, lab, new Timeline(null), ready);
await blind.launch();
await blind.poll(Date.now());
expect(blind.record.samples.at(-1)!.runs![0]!.preparing).toBeUndefined();
});
it("derives tool-in-flight and process liveness from run events", async () => {
const startedAt = new Date().toISOString();
const identity = { processPid: 4242, processGroupId: 4242, processStartedAt: "2026-11-06T10:00:00.000Z", targetKind: "local" };
+1
View File
@@ -271,6 +271,7 @@ export interface RunSample {
processAlive?: boolean;
/** A provider turn started and has no turn terminal yet. */
providerTurnActive?: boolean;
preparing?: boolean;
}
export interface InterruptionSample {
+3 -1
View File
@@ -12,6 +12,7 @@
* (reported in the grade and ledger, never fails the task). The split comes
* from the false-positive audit over saved waves; see README "Universal checks".
*/
import { PREPARATION_LIMIT_MS } from "./grader.js";
import type { RunSample, TaskRecord, TaskSample } from "./types.js";
export type UniversalId = "U1" | "U2" | "U3" | "U4" | "U5";
@@ -137,7 +138,8 @@ export function liveOwner(sample: TaskSample, queuedFor: ReadonlyMap<string, num
if (run.status === "running") {
if (run.toolInFlight || run.processAlive) return `run ${run.id} running (tool or process alive)`;
const evidence = Math.max(ms(run.lastEvidenceAt) || 0, ms(run.startedAt) || 0);
if (!evidence || sample.at - evidence <= bounds.silentRunMs) return `run ${run.id} running`;
const silentLimit = run.preparing ? Math.max(bounds.silentRunMs, PREPARATION_LIMIT_MS) : bounds.silentRunMs;
if (!evidence || sample.at - evidence <= silentLimit) return `run ${run.id} ${run.preparing ? "preparing" : "running"}`;
}
if (run.status === "queued" || run.status === "scheduled_retry") {
if ((queuedFor.get(run.id) ?? 0) <= bounds.queuedMs) return `run ${run.id} ${run.status}`;
+69 -3
View File
@@ -1,5 +1,7 @@
import { readFileSync } from "node:fs";
import path from "node:path";
import { describe, expect, it } from "vitest";
import { controllerLogNames, gradeWarmShutdown, isWarmWave, warmShutdownMarkdown } from "./warm-shutdown.js";
import { controllerLogNames, gradeWarmShutdown, isWarmWave, waitBounded, warmShutdownMarkdown, watchFirstIdleCloseFailure } from "./warm-shutdown.js";
const esc = String.fromCharCode(27);
const clean = { environmentDeleted: true, environmentDeleteBlocked: null, remainingSandboxes: [] };
@@ -31,7 +33,9 @@ describe("clean warm shutdown check", () => {
expect(check.verdict).toBe("fail");
expect(check.environmentDeleteBlocked).toBe(blocked);
expect(check.leakedSandboxes).toEqual(["sbx-1:started", "sbx-2:started"]);
expect(check.shutdownCloseFailures).toEqual(['[14:20:17] WARN: idle native session cleanup failed before controller shutdown {"idleSessions":{"failed":2}}']);
expect(check.shutdownCloseFailures).toEqual([{ runId: null, lines: 1,
reason: 'WARN: idle native session cleanup failed before controller shutdown {"idleSessions":{"failed":2}}',
firstLine: '[14:20:17] WARN: idle native session cleanup failed before controller shutdown {"idleSessions":{"failed":2}}' }]);
expect(check.problems).toHaveLength(3);
expect(warmShutdownMarkdown(check)).toContain("Verdict: FAIL");
});
@@ -39,7 +43,7 @@ describe("clean warm shutdown check", () => {
it("reports a forced stop without failing when cleanup still finished", () => {
const check = gradeWarmShutdown({ cleanup: clean,
serverLog: "WARN: warm native session close did not settle; its Runner was stopped and the session discarded {}\n" });
expect(check).toMatchObject({ verdict: "pass", idleCloseFailures: [expect.stringContaining("did not settle")] });
expect(check).toMatchObject({ verdict: "pass", idleCloseFailures: [{ firstLine: expect.stringContaining("did not settle") }] });
expect(gradeWarmShutdown({ cleanup: clean,
serverLog: "WARN: warm native session's sandbox was already destroyed; released its owner {}\n" }).idleCloseFailures).toHaveLength(1);
});
@@ -51,4 +55,66 @@ describe("clean warm shutdown check", () => {
expect(gradeWarmShutdown({ cleanup: null,
serverLog: 'WARN: idle warm native sessions did not close {"reason":"terminal heartbeat run destroyed its environment lease"}' }).verdict).toBe("pass");
});
const a1Log = readFileSync(path.join(import.meta.dirname, "fixtures/a1-warm-close-failures.log"), "utf8");
const run300 = "300f0d61-548b-4630-8f6a-f79292ed9d24";
const run70d = "70d909cc-8b04-40ae-a0bf-0846df140cf0";
it("reports each distinct (run, reason) once, not each log line (w-20261116T001158-s7106 A1: 2 sessions, 29 lines)", () => {
expect(a1Log.split("\n").filter(line => /did not close|cleanup failed before controller shutdown/.test(line))).toHaveLength(29);
const check = gradeWarmShutdown({ cleanup: clean, serverLog: a1Log });
expect(check.verdict).toBe("fail");
expect(check.shutdownCloseFailures.map(({ runId, lines }) => ({ runId, lines }))).toEqual([
{ runId: run300, lines: 2 }, { runId: run70d, lines: 27 }]);
expect(check.shutdownCloseFailures[0]!.reason).toMatch(/process_identity_changed\]$/);
expect(check.shutdownCloseFailures[1]!.reason).toBe(
"close failed: native_session_close_unrecoverable: provider_transport_failed: runner did not durably suspend before checkpoint");
expect(check.idleCloseFailures.map(({ runId, lines }) => ({ runId, lines }))).toEqual([
{ runId: run300, lines: 4 }, { runId: run70d, lines: 1 }]);
expect(check.problems).toHaveLength(2);
expect(warmShutdownMarkdown(check)).toContain("warm sessions that did not close: 2; idle closes that needed a forced stop or failed: 2.");
});
it("still fails on one distinct failure, and keeps different reasons for the same run apart", () => {
const one = a1Log.split("\n").filter(line => line.includes(run70d) && line.includes("did not close"));
expect(gradeWarmShutdown({ cleanup: clean, serverLog: one.join("\n") })).toMatchObject({ verdict: "fail",
shutdownCloseFailures: [{ runId: run70d, lines: 27 }] });
const twoReasons = [
'WARN: idle warm native sessions did not close {"reason":"controller restart","failures":[{"runId":"r1","reason":"close failed: a"}]}',
'WARN: idle warm native sessions did not close {"reason":"controller restart","failures":[{"runId":"r1","reason":"close failed: b"}]}',
].join("\n");
expect(gradeWarmShutdown({ cleanup: clean, serverLog: twoReasons }).problems).toHaveLength(2);
const unparsed = gradeWarmShutdown({ cleanup: clean, serverLog: 'WARN: idle warm native sessions did not close {"reason":"environment deleted","failures":[' });
expect(unparsed).toMatchObject({ verdict: "fail", shutdownCloseFailures: [{ runId: null, lines: 1 }] });
});
it("names the first idle-close failure once, across chunk boundaries and interleaved streams", () => {
const seen: Array<{ runId: string | null }> = [];
const watch = watchFirstIdleCloseFailure(failure => { seen.push(failure); });
const failed = a1Log.split("\n").find(line => line.includes("idle warm native session close failed") && line.includes(run70d))!;
watch("INFO: server ready\n", "1:stdout");
watch(failed.slice(0, 40), "1:stdout");
watch("WARN: unrelated stderr line\n", "1:stderr");
expect(seen).toEqual([]);
watch(`${failed.slice(40)}\n`, "1:stdout");
watch(`${failed}\n`, "1:stdout");
expect(seen).toEqual([expect.objectContaining({ runId: run70d })]);
const quiet: unknown[] = [];
const none = watchFirstIdleCloseFailure(failure => { quiet.push(failure); });
none("WARN: warm native session close did not settle; its Runner was stopped and the session discarded {}\n");
expect(quiet).toEqual([]);
});
it("lists the sandbox captures taken at the first idle-close failure", () => {
const check = gradeWarmShutdown({ cleanup: clean, serverLog: a1Log, sandboxCaptures: ["idle-close-failure-300f0d61"] });
expect(check.sandboxCaptures).toEqual(["idle-close-failure-300f0d61"]);
expect(warmShutdownMarkdown(check)).toContain("idle-close-failure-300f0d61");
});
it("waits for a pending capture only up to its bound", async () => {
expect(await waitBounded(() => null, 10)).toBe("none");
expect(await waitBounded(() => Promise.resolve(), 1_000)).toBe("settled");
expect(await waitBounded(() => Promise.reject(new Error("probe failed")), 1_000)).toBe("settled");
expect(await waitBounded(() => new Promise(() => undefined), 20)).toBe("timed_out");
});
});
+106 -7
View File
@@ -5,19 +5,29 @@ export interface WarmShutdownCleanup {
undeletedSandboxes?: string[];
}
export interface WarmCloseFailure {
runId: string | null;
reason: string;
lines: number;
firstLine: string;
}
export interface WarmShutdownCheck {
verdict: "pass" | "fail";
environmentDeleted: boolean | null;
environmentDeleteBlocked: string | null;
leakedSandboxes: string[];
shutdownCloseFailures: string[];
idleCloseFailures: string[];
shutdownCloseFailures: WarmCloseFailure[];
idleCloseFailures: WarmCloseFailure[];
sandboxCaptures: string[];
problems: string[];
}
const SHUTDOWN_CLOSE_FAILURE = /idle native session cleanup failed before controller shutdown|idle warm native sessions did not close.*"reason":"(environment deleted|controller restart)"/;
const IDLE_CLOSE_FAILURE = /idle warm native session (close|detach) failed|warm native session close did not settle|warm native session's sandbox was already destroyed/;
const FIRST_IDLE_CLOSE_FAILURE = /idle warm native session (close|detach) failed|idle warm native sessions did not close|idle native session cleanup failed before controller shutdown/;
const ANSI = new RegExp(`${String.fromCharCode(27)}\\[[0-9;]*m`, "g");
const MAX_PARTIAL_LINE = 256 * 1024;
export function controllerLogNames(names: readonly string[]): string[] {
return names.filter(name => /^server(-[a-z0-9]+)?\.log$/.test(name)).sort();
@@ -27,16 +37,75 @@ export function isWarmWave(options: { lifecycle?: string; daytonaWarm?: boolean
return options.lifecycle === "warm" || options.daytonaWarm === true;
}
export function gradeWarmShutdown(input: { cleanup: WarmShutdownCleanup | null; serverLog: string }): WarmShutdownCheck {
const lines = input.serverLog.split("\n").map(line => line.replace(ANSI, "").trim()).filter(Boolean);
const shutdownCloseFailures = lines.filter(line => SHUTDOWN_CLOSE_FAILURE.test(line));
const idleCloseFailures = lines.filter(line => IDLE_CLOSE_FAILURE.test(line));
function cleanLine(line: string): string {
return line.replace(ANSI, "").trim();
}
function lineDetail(line: string): Record<string, unknown> | null {
const start = line.indexOf("{");
if (start < 0) return null;
try {
const parsed = JSON.parse(line.slice(start)) as unknown;
return parsed && typeof parsed === "object" && !Array.isArray(parsed) ? parsed as Record<string, unknown> : null;
} catch {
return null;
}
}
function lineMessage(line: string): string {
const brace = line.indexOf("{");
return (brace < 0 ? line : line.slice(0, brace)).replace(/^\[[^\]]*\]\s*/, "").trim();
}
function failureReason(value: unknown): string | null {
if (typeof value !== "string" || !value.trim()) return null;
return value.split(" {")[0]!.trim();
}
function lineFailures(line: string): Array<{ runId: string | null; reason: string }> {
const detail = lineDetail(line);
const listed = Array.isArray(detail?.failures) ? detail.failures
: Array.isArray((detail?.idleSessions as Record<string, unknown> | undefined)?.failures)
? (detail!.idleSessions as Record<string, unknown>).failures as unknown[] : null;
const fromList = (listed ?? []).flatMap(entry => {
if (!entry || typeof entry !== "object") return [];
const item = entry as Record<string, unknown>;
const reason = failureReason(item.reason) ?? failureReason(item.closeFailure);
return [{ runId: typeof item.runId === "string" ? item.runId : null, reason: reason ?? lineMessage(line) }];
});
if (fromList.length) return fromList;
if (detail && typeof detail.runId === "string") {
return [{ runId: detail.runId, reason: failureReason(detail.closeFailure) ?? lineMessage(line) }];
}
const body = line.replace(/^\[[^\]]*\]\s*/, "");
return [{ runId: null, reason: body }];
}
export function distinctCloseFailures(lines: readonly string[]): WarmCloseFailure[] {
const byKey = new Map<string, WarmCloseFailure>();
for (const line of lines) {
for (const failure of lineFailures(line)) {
const key = `${failure.runId ?? ""}\u0000${failure.reason}`;
const existing = byKey.get(key);
if (existing) existing.lines += 1;
else byKey.set(key, { runId: failure.runId, reason: failure.reason, lines: 1, firstLine: line });
}
}
return [...byKey.values()];
}
export function gradeWarmShutdown(input: { cleanup: WarmShutdownCleanup | null; serverLog: string; sandboxCaptures?: readonly string[] }): WarmShutdownCheck {
const lines = input.serverLog.split("\n").map(cleanLine).filter(Boolean);
const shutdownCloseFailures = distinctCloseFailures(lines.filter(line => SHUTDOWN_CLOSE_FAILURE.test(line)));
const idleCloseFailures = distinctCloseFailures(lines.filter(line => IDLE_CLOSE_FAILURE.test(line)));
const problems: string[] = [];
if (input.cleanup?.environmentDeleteBlocked) problems.push(`environment delete blocked: ${input.cleanup.environmentDeleteBlocked}`);
else if (input.cleanup && !input.cleanup.environmentDeleted) problems.push("environment was not deleted");
if (input.cleanup?.remainingSandboxes.length) problems.push(`sandboxes still running after cleanup: ${input.cleanup.remainingSandboxes.join(", ")}`);
if (input.cleanup?.undeletedSandboxes?.length) problems.push(`wave sandboxes left after the harness deleted them: ${input.cleanup.undeletedSandboxes.join(", ")}`);
for (const line of shutdownCloseFailures) problems.push(`warm session did not close: ${line}`);
for (const failure of shutdownCloseFailures) {
problems.push(`warm session did not close: run ${failure.runId ?? "unknown"} (${failure.lines} log line${failure.lines === 1 ? "" : "s"}): ${failure.reason}`);
}
return {
verdict: problems.length ? "fail" : "pass",
environmentDeleted: input.cleanup?.environmentDeleted ?? null,
@@ -44,15 +113,45 @@ export function gradeWarmShutdown(input: { cleanup: WarmShutdownCleanup | null;
leakedSandboxes: [...new Set([...input.cleanup?.remainingSandboxes ?? [], ...input.cleanup?.undeletedSandboxes ?? []])],
shutdownCloseFailures,
idleCloseFailures,
sandboxCaptures: [...input.sandboxCaptures ?? []],
problems,
};
}
export function watchFirstIdleCloseFailure(onFailure: (failure: { runId: string | null; line: string }) => void): (chunk: Buffer | string, source?: string) => void {
const partials = new Map<string, string>();
let fired = false;
return (chunk, source = "") => {
if (fired) return;
const lines = `${partials.get(source) ?? ""}${typeof chunk === "string" ? chunk : chunk.toString("utf8")}`.split("\n");
const rest = lines.pop() ?? "";
partials.set(source, rest.length > MAX_PARTIAL_LINE ? "" : rest);
for (const raw of lines) {
const line = cleanLine(raw);
if (!FIRST_IDLE_CLOSE_FAILURE.test(line)) continue;
fired = true;
onFailure({ runId: lineFailures(line)[0]?.runId ?? null, line });
return;
}
};
}
export async function waitBounded(pending: () => Promise<unknown> | null, ms: number): Promise<"settled" | "timed_out" | "none"> {
const work = pending();
if (!work) return "none";
let timer: NodeJS.Timeout | undefined;
const outcome = await Promise.race([work.then(() => "settled" as const, () => "settled" as const),
new Promise<"timed_out">(resolve => { timer = setTimeout(() => resolve("timed_out"), ms); })]);
clearTimeout(timer);
return outcome;
}
export function warmShutdownMarkdown(check: WarmShutdownCheck): string {
const out = ["## Clean warm shutdown (no leaked sandboxes)", "",
`Verdict: ${check.verdict.toUpperCase()}. Environment deleted: ${check.environmentDeleted ?? "n/a"}; sandboxes left running: ${check.leakedSandboxes.length}; `
+ `warm sessions that did not close: ${check.shutdownCloseFailures.length}; idle closes that needed a forced stop or failed: ${check.idleCloseFailures.length}.`, ""];
for (const problem of check.problems) out.push(`- ${problem.slice(0, 2_000)}`);
if (check.problems.length) out.push("");
if (check.sandboxCaptures.length) out.push(`Sandbox captures at the first idle-close failure: ${check.sandboxCaptures.join(", ")} (daytona-runner/).`, "");
return out.join("\n");
}