mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 02:43:25 +01:00
fix(runner): wait for a slow Codex turn start while it reports progress (#21329)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Codex agents run through the Paperclip runner, and the server-side transport starts each provider turn > - After Codex accepts a turn, the transport waits a fixed 30 s for that turn's `turn/started` before it reports the turn identity > - Codex starts its MCP servers before it starts the turn, so a healthy cold start in a loaded sandbox often takes longer than 30 s > - The transport then failed a live turn with `runnerd did not report the provider turn identity`, and the abandoned turn's late notifications broke the reattachment (`did not name its execution owner`, `runner_harness_state_mismatch: resumed_not_suspended`) > - This pull request sizes the start window to the measured cold start and extends it while the provider keeps reporting startup progress > - The benefit is that slow but healthy Codex starts succeed on the first attempt, and a dead start still fails at a bound ## Linked Issues or Issue Description **What happened?** On staging, Codex runs on Daytona failed at turn start with `native_session_interrupted: runnerd did not report the provider turn identity`. The run log shows the same sequence each time. The runner accepts the turn. Codex then reports startup progress (a bubblewrap config warning, `account/updated`, and `mcpServer/startupStatus/updated` failures for an MCP server that is not logged in) for about 30 s. The transport gives up 30 s after acceptance. A few seconds later, Codex sends the turn's `turn/started`, which now arrives at a closed turn (`protocol_failure:turn_binding_mismatch`). The same-run reattachment then hits that orphaned turn. It fails with `Provider authoritative notification item/started did not name its execution owner.` or `runner_harness_state_mismatch: resumed_not_suspended`. Of 198 recent runs in the staging company, 52 contain this turn identity failure. In 68 successful starts, submit to `turn/started` took p50 25 s, p75 37 s, p90 50 s, p95 54 s and max 76 s. The fixed 30 s wait sits near the median. **Expected behavior** A slow but healthy Codex start succeeds. A start that makes no progress, whose Runner exits, or that the runner fails still fails at a bound and recovers through the normal same-run resume. **Steps to reproduce** 1. Start a Codex turn whose provider accepts the turn but sends `turn/started` after the 30 s default window (on staging: several MCP servers and a loaded sandbox). 2. Let the provider report startup notices during the wait, and let the runner not prove the accepted turn live in time (on staging no turn was adopted). 3. See `runnerd did not report the provider turn identity`, then the reattachment conflict. The new tests reproduce this with the fake Codex app server (`--turn-started-delay-ms` with `--turn-started-progress-ms`) and a runner snapshot that cannot prove liveness. **Paperclip version or commit** Staging Core `c730872974` (`2026.1106.0+2253`). Base of this PR: `origin/master`. **Deployment mode** Authenticated, public, managed cloud (staging). Remote runner on Daytona. **Agent adapter(s) involved** `paperclip_runner` with the Codex provider (`gpt-6.1-sol`). ## What Changed - `runnerd-codex-transport.ts`: the default wait for an accepted turn's `turn/started` is now 90 s (measured max 76 s), not 30 s. An explicit `turnStartTimeoutMs` keeps its old bound. - The wait restarts its window each time the runner commits a provider notice, or a runner diagnostic that names the accepted turn. The total wait stops at 180 s (option `turnStartProgressCeilingMs`, so tests can lower it). A provider relaunch after the accepted `turn.start` does not extend it. - The adoption check (runner snapshot proves the accepted turn live) now runs before each window end, not only before the first one. A Runner exit and a runner failure of the pending turn still fail at once. - `fake-codex-app-server`: new `--turn-started-progress-ms` flag sends `mcpServer/startupStatus/updated` failures while it delays `turn/started`. - `doc/execution-semantics.md`: documents the start window, the progress rule and the staging measurements. ## Verification - Red on `origin/master` transport (with the new fake flag): `keeps waiting for an accepted Codex turn while its startup keeps reporting progress` failed after 4.6 s, and `gives a silent Codex cold start the measured default start window` failed after 30.9 s. Both failed with `runnerd did not report the provider turn identity`. - Green: `npx vitest run src/live/runnerd-codex-transport-deadlines.test.ts` passes 9/9. This includes a ceiling test: provider notices every 1 s and no `turn/started` fail at an 8 s test ceiling, and the test goes red with the ceiling cap removed. It also includes the existing adoption, no-adoption-within-bound and runner-failure tests, and a new test that a silent start still fails within its bound. - `npx vitest run` on the adoption, cut-turn-start, stall, turn-start-timeout-retention, remote-retention, restart-continuation and controller-outage transport suites passes 48/48. - `npx vitest run src/live/runnerd-codex-transport.test.ts src/live/live-session.test.ts src/live/turn-stream.test.ts`: 391/392 pass. The one failure (`stops a lost runnerd's provider and escaped tool job from its lineage journal (journal=true)`, 1 s, process-lineage timing under full-suite load) passes when run alone, both on this branch and on `origin/master`. - `tsc --noEmit` for `packages/paperclip-runner` and `cargo fmt --check` pass. ## Risks - A silent dead start (no Runner exit, no provider output, no runner failure) now fails after 90 s, not 30 s. The Runner exit and runner-failure paths still fail at once, so this applies only to a provider that stops without a trace. - A provider that keeps sending notices but never starts the turn waits at most 180 s. - No schema, API or UI change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, browser reads of staging run data). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Refs #21272, #21120 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
314b893678
commit
d9a4bb23fe
4 files changed
+196
-62
No files matched your search
@@ -1775,6 +1775,18 @@ starts by itself once the gate clears. A Retry accepted during the pre-deploy
|
||||
drain stays saved and is dispatched by the deferred-receipt worker after the
|
||||
restart.
|
||||
|
||||
After the provider accepts a turn, the transport waits for that turn's
|
||||
`turn/started` before it reports the turn identity. Codex starts its MCP
|
||||
servers before it starts the turn, so a healthy cold start in a loaded sandbox
|
||||
can take most of a minute: on staging, submit to `turn/started` measured p50
|
||||
25 s, p90 50 s and max 76 s across 68 successful starts, while a fixed 30 s wait
|
||||
hit 52 of 198 recent runs with `runnerd did not report the provider turn identity`.
|
||||
The default wait is 90 s. Each provider notice, or runner diagnostic naming the
|
||||
accepted turn, restarts that window, up to 180 s in all. A provider relaunch
|
||||
does not count as progress. A silent start with no proof that the accepted turn
|
||||
is still live fails when its window ends, and a Runner exit or a runner
|
||||
failure of the pending turn still fails at once.
|
||||
|
||||
When a remote native run fails and its close cannot prove that the Runner
|
||||
suspended (for example, the Runner is unreachable after `provider_transport_failed`
|
||||
or a `turn.start` timeout), the controller no longer force-stops that Runner. It
|
||||
|
||||
+18
-1
@@ -1989,7 +1989,24 @@ fn run() -> Result<(), Box<dyn std::error::Error>> {
|
||||
}
|
||||
// A loaded host can accept a turn long before it starts it.
|
||||
if let Some(delay) = argument(&args, "--turn-started-delay-ms") {
|
||||
thread::sleep(Duration::from_millis(delay.parse()?));
|
||||
let delay = Duration::from_millis(delay.parse()?);
|
||||
let started = Instant::now();
|
||||
if let Some(interval) = argument(&args, "--turn-started-progress-ms") {
|
||||
let interval = Duration::from_millis(interval.parse()?);
|
||||
let mut server = 0;
|
||||
while started.elapsed() + interval < delay {
|
||||
thread::sleep(interval);
|
||||
server += 1;
|
||||
send(
|
||||
json!({"method": "mcpServer/startupStatus/updated", "params": {
|
||||
"name": format!("startup-{server}"),
|
||||
"status": "failed",
|
||||
"error": "fixture server unavailable"
|
||||
}}),
|
||||
)?;
|
||||
}
|
||||
}
|
||||
thread::sleep(delay.saturating_sub(started.elapsed()));
|
||||
}
|
||||
// A test can hold the start until it has detached the controller.
|
||||
if let Some(gate) = argument(&args, "--turn-started-gate").map(PathBuf::from) {
|
||||
|
||||
@@ -106,6 +106,96 @@ it("fails a pending turn start at once with the runner's own terminal failure",
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
async function withUnprovableLiveness<T>(run: (unprove: () => void) => Promise<T>): Promise<T> {
|
||||
let unproven = false;
|
||||
const OriginalCore = durableControlPlane.DurablePrpControlPlane;
|
||||
const coreSpy = vi.spyOn(durableControlPlane, "DurablePrpControlPlane")
|
||||
.mockImplementation(function(options: ConstructorParameters<typeof OriginalCore>[0]) {
|
||||
const core = new OriginalCore(options);
|
||||
const queueCommand = core.queueCommand.bind(core);
|
||||
core.queueCommand = ((type: string, ...rest: unknown[]) => {
|
||||
if (unproven && type === "session.snapshot") throw new Error("runner snapshot unavailable");
|
||||
return (queueCommand as (...args: unknown[]) => unknown)(type, ...rest);
|
||||
}) as typeof core.queueCommand;
|
||||
return core;
|
||||
} as unknown as typeof OriginalCore);
|
||||
try {
|
||||
return await run(() => { unproven = true; });
|
||||
} finally {
|
||||
coreSpy.mockRestore();
|
||||
}
|
||||
}
|
||||
|
||||
async function startSlowCodexTurn(input: {
|
||||
label: string;
|
||||
delayMs: number;
|
||||
progressMs?: number;
|
||||
turnStartTimeoutMs?: number;
|
||||
turnStartProgressCeilingMs?: number;
|
||||
}) {
|
||||
const stateDirectory = await mkdtemp(join(tmpdir(), `runnerd-${input.label}-`));
|
||||
const callsPath = join(stateDirectory, "calls.log");
|
||||
return withUnprovableLiveness(async (unprove) => {
|
||||
const bundle = createCapabilityRunnerdCodexTransport({
|
||||
runnerBinary: defaultCapabilityRunnerdBinary(), codexCommand: fakeCodex,
|
||||
codexArgs: fakeCodexArgs(stateDirectory, "--turn-started-delay-ms", String(input.delayMs),
|
||||
...(input.progressMs === undefined ? [] : ["--turn-started-progress-ms", String(input.progressMs)]),
|
||||
"--call-log", callsPath),
|
||||
stateDirectory,
|
||||
...(input.turnStartTimeoutMs === undefined ? {} : { turnStartTimeoutMs: input.turnStartTimeoutMs }),
|
||||
...(input.turnStartProgressCeilingMs === undefined ? {} : { turnStartProgressCeilingMs: input.turnStartProgressCeilingMs }),
|
||||
});
|
||||
try {
|
||||
await bundle.transport.request("thread/start", { cwd: tmpdir(), dynamicTools: [] });
|
||||
unprove();
|
||||
const started = Date.now();
|
||||
const outcome = await bundle.transport.request("turn/start", { input: [{ type: "text", text: "Start slowly." }] })
|
||||
.then(result => ({ result, error: null }), (error: unknown) => ({ result: null, error }));
|
||||
const elapsedMs = Date.now() - started;
|
||||
const calls = (await readFile(callsPath, "utf8")).trim().split(/\r?\n/);
|
||||
return { ...outcome, elapsedMs, turnStarts: calls.filter(call => call === "turn/start").length, diagnostics: bundle.evidence().diagnostics };
|
||||
} finally {
|
||||
await bundle.transport.close().catch(() => undefined);
|
||||
await rm(stateDirectory, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
it("keeps waiting for an accepted Codex turn while its startup keeps reporting progress", async () => {
|
||||
const outcome = await startSlowCodexTurn({ label: "progressing-turn-start", delayMs: 12_000, progressMs: 1_500, turnStartTimeoutMs: 4_000 });
|
||||
expect(outcome.error).toBeNull();
|
||||
expect(outcome.result).toMatchObject({ turn: { id: "provider-turn-1" } });
|
||||
expect(outcome.elapsedMs).toBeGreaterThan(10_000);
|
||||
expect(outcome.turnStarts).toBe(1);
|
||||
expect(outcome.diagnostics.join("\n")).not.toContain("adopting the durably accepted");
|
||||
}, 60_000);
|
||||
|
||||
it("fails an accepted Codex start at the total ceiling while progress never reaches turn/started", async () => {
|
||||
const outcome = await startSlowCodexTurn({
|
||||
label: "endless-progress-turn-start", delayMs: 30_000, progressMs: 1_000,
|
||||
turnStartTimeoutMs: 3_000, turnStartProgressCeilingMs: 8_000,
|
||||
});
|
||||
expect(String(outcome.error)).toContain("runnerd did not report the provider turn identity");
|
||||
expect(outcome.elapsedMs).toBeGreaterThan(7_000);
|
||||
expect(outcome.elapsedMs).toBeLessThan(8_000 + 2_000);
|
||||
expect(outcome.turnStarts).toBe(1);
|
||||
}, 60_000);
|
||||
|
||||
it("still fails a silent accepted Codex start at its bound when liveness cannot be proven", async () => {
|
||||
const outcome = await startSlowCodexTurn({ label: "silent-turn-start", delayMs: 20_000, turnStartTimeoutMs: 3_000 });
|
||||
expect(String(outcome.error)).toContain("runnerd did not report the provider turn identity");
|
||||
expect(outcome.elapsedMs).toBeLessThan(3_000 + 1_500);
|
||||
expect(outcome.turnStarts).toBe(1);
|
||||
}, 60_000);
|
||||
|
||||
it("gives a silent Codex cold start the measured default start window", async () => {
|
||||
const outcome = await startSlowCodexTurn({ label: "cold-turn-start", delayMs: 40_000 });
|
||||
expect(outcome.error).toBeNull();
|
||||
expect(outcome.result).toMatchObject({ turn: { id: "provider-turn-1" } });
|
||||
expect(outcome.elapsedMs).toBeGreaterThan(38_000);
|
||||
expect(outcome.turnStarts).toBe(1);
|
||||
}, 90_000);
|
||||
|
||||
it("keeps a quiet active Codex turn in the same process across connection lease expiry", async () => {
|
||||
const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-renew-active-"));
|
||||
const callsPath = join(stateDirectory, "calls.log");
|
||||
|
||||
@@ -962,6 +962,8 @@ function terminalMayBelongToPendingTurnStart(input: {
|
||||
|
||||
/** Part of the turn-start bound kept for the accepted-turn adoption check. */
|
||||
const TURN_START_ADOPTION_CHECK_RESERVE_MS = 5_000;
|
||||
const TURN_START_EVENT_DEFAULT_TIMEOUT_MS = 90_000;
|
||||
const TURN_START_PROGRESS_CEILING_MS = 180_000;
|
||||
|
||||
/**
|
||||
* The runner's own final failure of a turn whose start is still pending: a
|
||||
@@ -1669,6 +1671,7 @@ export interface CapabilityRunnerdCodexTransportOptions extends RunnerExecutionD
|
||||
closeGraceMs?: number;
|
||||
/** Bounded provider turn-admission wait; defaults to 30 seconds. */
|
||||
turnStartTimeoutMs?: number;
|
||||
turnStartProgressCeilingMs?: number;
|
||||
/** Controller-owned opt-in operation timing. No exporter lives in Runner. */
|
||||
measureOperation?<T>(phase: string, work: () => Promise<T>, attributes?: Record<string, number | boolean>): Promise<T>;
|
||||
recordTimingReceipt?(receipt: unknown): void;
|
||||
@@ -7618,13 +7621,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
}): boolean {
|
||||
if (input.snapshot?.activeProviderTurnId !== input.expectedProviderTurnId) return false;
|
||||
if (!Number.isSafeInteger(input.acceptingCommandSeq)) return false;
|
||||
const relaunched = (this.#core?.store.state.committedEvents ?? []).some((event) => {
|
||||
if (event.eventType !== "harness.diagnostic") return false;
|
||||
const payload = record(record(event.envelope.payload).payload);
|
||||
return payload.code === "provider_startup_ownership" &&
|
||||
Number(record(record(payload.startup).command).controllerSeq) > input.acceptingCommandSeq;
|
||||
});
|
||||
if (relaunched) return false;
|
||||
if (this.#providerRelaunchedAfter(input.acceptingCommandSeq)) return false;
|
||||
if (input.acceptingProviderPid === null || this.#evidence.providerPid !== input.acceptingProviderPid) return false;
|
||||
// A remote provider's pid names no local process; the runner's own
|
||||
// launch reports above are its replacement evidence.
|
||||
@@ -7632,6 +7629,37 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
return readLocalProcessStartedAt(input.acceptingProviderPid) === input.acceptingProviderStartedAt;
|
||||
}
|
||||
|
||||
#providerRelaunchedAfter(acceptingCommandSeq: number): boolean {
|
||||
return (this.#core?.store.state.committedEvents ?? []).some((event) => {
|
||||
if (event.eventType !== "harness.diagnostic") return false;
|
||||
const payload = record(record(event.envelope.payload).payload);
|
||||
return payload.code === "provider_startup_ownership" &&
|
||||
Number(record(record(payload.startup).command).controllerSeq) > acceptingCommandSeq;
|
||||
});
|
||||
}
|
||||
|
||||
#latestCommittedSourceSeq(): number {
|
||||
return this.#core?.store.state.committedEvents.at(-1)?.sourceSeq ?? 0;
|
||||
}
|
||||
|
||||
#pendingTurnStartProgressSince(sourceSeq: number, expectedProviderTurnId: string): { progressed: boolean; latestSeq: number } {
|
||||
const events = this.#core?.store.state.committedEvents ?? [];
|
||||
let latestSeq = sourceSeq;
|
||||
let progressed = false;
|
||||
for (let index = events.length - 1; index >= 0; index -= 1) {
|
||||
const event = events[index]!;
|
||||
if (event.sourceSeq <= sourceSeq) break;
|
||||
if (event.sourceSeq > latestSeq) latestSeq = event.sourceSeq;
|
||||
if (progressed) continue;
|
||||
if (event.eventType === "provider.notice.recorded") progressed = true;
|
||||
else if (event.eventType === "harness.diagnostic") {
|
||||
const payload = record(record(event.envelope.payload).payload);
|
||||
progressed = payload.expectedTurnId === expectedProviderTurnId;
|
||||
}
|
||||
}
|
||||
return { progressed, latestSeq };
|
||||
}
|
||||
|
||||
async #startTurnWork(
|
||||
params: Record<string, unknown>,
|
||||
): Promise<Record<string, unknown>> {
|
||||
@@ -7757,10 +7785,19 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
// command's durable result so a delayed prior-turn event cannot satisfy
|
||||
// the new response fence. ACPX echoes the requested identity, while
|
||||
// Codex and OpenCode return their provider-assigned identity.
|
||||
const deadline =
|
||||
const quietWindowMs =
|
||||
this.options.turnStartTimeoutMs === undefined
|
||||
? this.#waitNow() + 30_000
|
||||
: commandDeadline;
|
||||
? TURN_START_EVENT_DEFAULT_TIMEOUT_MS
|
||||
: Math.max(0, commandDeadline - this.#waitNow());
|
||||
let deadline = this.#waitNow() + quietWindowMs;
|
||||
const progressCeiling = this.#waitNow() + Math.max(quietWindowMs, this.options.turnStartProgressCeilingMs ?? TURN_START_PROGRESS_CEILING_MS);
|
||||
let progressSeq = this.#latestCommittedSourceSeq();
|
||||
const extendOnProgress = () => {
|
||||
const progress = this.#pendingTurnStartProgressSince(progressSeq, expectedProviderTurnId);
|
||||
progressSeq = progress.latestSeq;
|
||||
if (progress.progressed && !this.#providerRelaunchedAfter(acceptingCommandSeq))
|
||||
deadline = Math.min(progressCeiling, Math.max(deadline, this.#waitNow() + quietWindowMs));
|
||||
};
|
||||
const providerTurnStarted = () =>
|
||||
turnStartResponseReady({
|
||||
responseEpoch,
|
||||
@@ -7768,14 +7805,10 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
expectedProviderTurnId,
|
||||
boundTurnId: this.#turnId,
|
||||
});
|
||||
// Keep the adoption check inside the configured bound: stop waiting for
|
||||
// the start event early enough to read the runner's snapshot before it.
|
||||
const adoptionCheckAt = deadline - Math.min(
|
||||
const adoptionReserveMs = Math.min(
|
||||
TURN_START_ADOPTION_CHECK_RESERVE_MS,
|
||||
Math.floor(Math.max(0, deadline - this.#waitNow()) / 4),
|
||||
Math.floor(quietWindowMs / 4),
|
||||
);
|
||||
// The runner can end the accepted turn before it starts. Its terminal is
|
||||
// held for the pending start, so report that real cause at once.
|
||||
const throwIfRunnerFailedStart = () => {
|
||||
const failure = runnerdPendingTurnStartFailure([
|
||||
...this.#deferredTurnStartEvents,
|
||||
@@ -7787,59 +7820,41 @@ class DurablePrpCodexTransport implements CodexAppServerTransport {
|
||||
throw new Error(`runnerd failed the provider turn before it started: ${failure}`);
|
||||
};
|
||||
const waitForStarted = async () => {
|
||||
while (!providerTurnStarted() && this.#waitNow() < adoptionCheckAt) {
|
||||
this.#throwIfFailed();
|
||||
this.#pumpEvents();
|
||||
if (providerTurnStarted()) break;
|
||||
throwIfRunnerFailedStart();
|
||||
if (await this.#runnerHasExited())
|
||||
throw new Error("runnerd exited before provider turn startup");
|
||||
await new Promise((resolveWait) => setTimeout(resolveWait, 10));
|
||||
}
|
||||
if (!providerTurnStarted()) {
|
||||
// The durable command result already carries the provider's own
|
||||
// acceptance and turn id. A loaded provider can start that turn after
|
||||
// this bound, and it still runs it. Failing would abandon a live turn
|
||||
// that no controller owns, so its tools and close could never settle.
|
||||
// Bind the exact accepted identity when the runner proves it is live;
|
||||
// its turn/started is then accepted in order, and the turn's own
|
||||
// terminal, timeout and Stop paths own it from here.
|
||||
this.#throwIfFailed();
|
||||
if (await this.#runnerHasExited())
|
||||
throw new Error("runnerd exited before provider turn startup");
|
||||
const snapshot = await this.#commandResult(
|
||||
"session.snapshot", {}, deadline,
|
||||
).catch(() => null);
|
||||
// Ingest any runner-reported provider launch committed before it. A
|
||||
// pump that rejected a stale start has failed the transport; never
|
||||
// report a successful start on it.
|
||||
this.#pumpEventsSafely();
|
||||
this.#throwIfFailed();
|
||||
if (!providerTurnStarted()) throwIfRunnerFailedStart();
|
||||
if (!providerTurnStarted() && this.#acceptedTurnProvablyLive({
|
||||
snapshot, expectedProviderTurnId, acceptingProviderPid,
|
||||
acceptingProviderStartedAt, acceptingCommandSeq,
|
||||
})) {
|
||||
this.#turnId = expectedProviderTurnId;
|
||||
this.#observedTurnStartEpoch = responseEpoch;
|
||||
this.#diagnostic(
|
||||
"provider turn start outlived its bound; adopting the durably accepted active provider turn",
|
||||
);
|
||||
}
|
||||
// Not provably live: keep the original contract and wait for the
|
||||
// start event until the configured bound, then fail as before.
|
||||
while (!providerTurnStarted() && this.#waitNow() < deadline) {
|
||||
let adoptionCheckedFor: number | null = null;
|
||||
while (!providerTurnStarted()) {
|
||||
this.#throwIfFailed();
|
||||
this.#pumpEvents();
|
||||
if (providerTurnStarted()) break;
|
||||
throwIfRunnerFailedStart();
|
||||
extendOnProgress();
|
||||
if (await this.#runnerHasExited())
|
||||
throw new Error("runnerd exited before provider turn startup");
|
||||
if (this.#waitNow() >= deadline - adoptionReserveMs && adoptionCheckedFor !== deadline) {
|
||||
adoptionCheckedFor = deadline;
|
||||
const snapshot = await this.#commandResult(
|
||||
"session.snapshot", {}, deadline,
|
||||
).catch(() => null);
|
||||
this.#pumpEventsSafely();
|
||||
this.#throwIfFailed();
|
||||
if (providerTurnStarted()) break;
|
||||
throwIfRunnerFailedStart();
|
||||
if (this.#acceptedTurnProvablyLive({
|
||||
snapshot, expectedProviderTurnId, acceptingProviderPid,
|
||||
acceptingProviderStartedAt, acceptingCommandSeq,
|
||||
})) {
|
||||
this.#turnId = expectedProviderTurnId;
|
||||
this.#observedTurnStartEpoch = responseEpoch;
|
||||
this.#diagnostic(
|
||||
"provider turn start outlived its bound; adopting the durably accepted active provider turn",
|
||||
);
|
||||
break;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (this.#waitNow() >= deadline)
|
||||
throw new Error("runnerd did not report the provider turn identity");
|
||||
await new Promise((resolveWait) => setTimeout(resolveWait, 10));
|
||||
}
|
||||
if (!providerTurnStarted())
|
||||
throw new Error("runnerd did not report the provider turn identity");
|
||||
}
|
||||
};
|
||||
if (this.options.measureOperation) await this.options.measureOperation("runner.transport.started_event", waitForStarted);
|
||||
else await waitForStarted();
|
||||
|
||||
Reference in new issue
Block a user