From 82ce8e64bae1e4f299102ed4b08101028ca2cd65 Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Fri, 13 Nov 2026 20:45:49 -0800 Subject: [PATCH] feat(runner): let the session driver admit a turn that the provider started (#21453) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core runs Claude Code through Paperclip Runner. Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. > - Today the run ends when the turn ends. The command stops, and the run fails with `native_finalization_missing`. > - The Runner part of this work is complete in #21316 and the pull requests below it: when a controller sends `providerStartedTurns` on `turn.start`, the Runner keeps Claude while a background shell is running, and gives the cycle of Claude to the controller as a turn that the provider started. > - Core reads Runner events through the Codex app-server driver, also for ACPX agents. The driver refuses a turn that the controller did not start, except a goal turn of Codex. > - This pull request lets the driver accept a turn that the provider started, only directly after a turn that the Runner marked as waiting. > - The benefit is that the next parts can use that turn. Nothing reaches this code until the last part sends the opt-in. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one of four parts on the Core side. They are stacked on #21316: 1. The driver admits a turn that the provider started. 2. The consumer waits for that turn, for a limited time. 3. The server ends the wait when a person comments, and keeps the clock across a restart. 4. The transport sends the opt-in. Only this part changes what a production run does. This is part 1. Its base branch is `feat/adopt-claude-provider-started-turn`. Refs #19912 (design). **What existing behavior does this improve?** Nothing in production yet. What the driver does with a turn that the provider started. **Subsystem affected** Paperclip Runner package, the controller side: `src/drivers/codex` (the driver that Core uses to read a native session). **Current behavior** - A `turn/started` that the controller did not ask for fails the session with `turn_binding_mismatch`, except for an active Codex goal. - An interrupt with no active turn fails with "already terminal". **Proposed behavior** - A turn terminal that the Runner marked with `awaitingProviderTurn` and that has no result puts the driver in a waiting state. The driver event `turn.completed` carries the marker. - In that state, a `turn/started` with `origin: "provider"` is admitted the same way as a goal turn, and the driver event carries `origin: "provider"`. In each other state it still fails closed. - In that state with no active turn, an interrupt gives an acknowledgement item with the status `provider_turn_wait_ended`. It sends nothing to the Runner. The driver also gives that acknowledgement when it gets the internal `paperclip/runTerminal` notification with a private marker; a notification without the marker does nothing. - An interrupt for a provider-started turn that is running is sent to the Runner only when it names that turn. - The driver saves the waiting state in the persisted session (`awaitingProviderTurn`, written only during a wait). After a controller restart, the restored session still admits the provider-started turn and still acknowledges an interrupt during the wait. The driver restores the state only when the session has no active turn, no stored result and at least one finished turn, so a new run cannot take the wait of the run before it. - When a restart finds that the saved active turn is finished, the turn read of the transport and `reconcile()` of the driver carry the same marker as the live `turn/completed` notification. `runnerd-codex-transport.ts` copies `awaitingProviderTurn: true` from the saved `turn.completed` event to the turn object of a read. - The driver ends the waiting state when the Runner reports the run terminal. **Reason and benefit** The consumer in the next part needs these three signals: a marked terminal, an admitted turn, and a clear end of the wait. ## What Changed - `codex-session-terminal.ts`, `codex-session-notifications.ts`, `codex-session-state.ts`, `codex-harness-session.ts`, `codex-driver-values.ts`: 37 lines added, 4 removed. - `codex-app-server-driver.provider-turn.test.ts` (new, 232 lines). - `contracts/harness-driver.ts`, `contracts/native-session-backend.ts`, `backends/harness-driver-backend.ts`, `codex-app-server-driver-impl.ts`: the optional field in the two persisted shapes and its copy in each direction. - `live/runnerd-codex-transport.ts`: the marker on a recovered turn terminal. The loop that builds the earlier terminals of a `thread/read` is now the function `runnerdPriorTurnTerminals`, with the same logic, so that a test can call it. ## State saved by the previous release - One new optional field in the persisted driver session: `awaitingProviderTurn`. A snapshot outside a wait has the shape of the previous release. - **State from the previous release, read by this release:** the field is absent, so the driver does not wait. This is the behavior of today. - **State from this release, read by the previous release:** the previous reader copies the fields that it knows and ignores this one. No reader rejects it, and the integrity digest covers the stored object as written. ## Verification At the head of this branch: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json` and `pnpm run build:typescript` in `packages/paperclip-runner`; `node scripts/verify-runner-vendor-dependencies.mjs` and `pnpm typecheck` in `server`; `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`: all passed. - 6 driver tests: a provider-started turn after a marked terminal is admitted and can report a result; a provider-started turn with no marked terminal fails closed; an interrupt during the wait is acknowledged; a running provider-started turn is not interrupted by the end of the wait; the marked notification ends the wait and a notification with no marker does not; a normal terminal is not changed. - Restart tests: a session restored from a snapshot with the waiting state admits the provider-started turn and settles its result; a snapshot outside a wait has no new field; a restored session with no waiting state refuses a provider-started turn as today; an interrupt during a restored wait is acknowledged. A saved active turn that the read reports finished with the marker gives a `turn.completed` with `awaitingProviderTurn: true`; with no marker it is an ordinary terminal. - `harness-driver-backend.test.ts`: the field goes through a serialized checkpoint only during a wait. `runnerd-codex-transport.test.ts` (324 passed): the marker is on a recovered terminal only for a `turn.completed` with a boolean `true`. - `src/native-session-runtime.test.ts`: 297 passed, with the real `paperclip-runnerd`. - Mutation checks for the restart work, each made a named test fail: no restore of the state; no copy in the backend; no copy in the transport; no marker argument in `reconcile()`; no end of the wait at the run terminal. - Mutation checks: the acknowledgement branch removed, the admission removed, and the marker check removed each made a test fail. Not run: the full suite. CI supplies that coverage. ## Risks - **No production effect.** The Runner sends the marker only after the opt-in of part 4. - **The admission rule is shared with Codex goal turns.** The existing goal tests pass with no change. - **Known limit after a controller restart.** If the Runner already runs a provider-started turn and the saved active turn is the turn before it, recovery fails closed, as today for each unexpected active turn. - **A controller turn that starts during a wait** clears the waiting state when its `turn/started` arrives, not when it is sent. A snapshot in that short time can hold an active turn and the field together; the restore rule drops the field. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 --- .../backends/harness-driver-backend.test.ts | 24 ++ .../src/backends/harness-driver-backend.ts | 2 + .../src/contracts/harness-driver.ts | 1 + .../src/contracts/native-session-backend.ts | 1 + .../codex/codex-app-server-driver-impl.ts | 2 + ...ex-app-server-driver.provider-turn.test.ts | 232 ++++++++++++++++++ .../src/drivers/codex/codex-driver-values.ts | 1 + .../drivers/codex/codex-harness-session.ts | 16 +- .../codex/codex-session-notifications.ts | 22 +- .../src/drivers/codex/codex-session-state.ts | 4 + .../drivers/codex/codex-session-terminal.ts | 5 +- .../src/live/runnerd-codex-transport.test.ts | 21 ++ .../src/live/runnerd-codex-transport.ts | 46 ++-- 13 files changed, 358 insertions(+), 19 deletions(-) create mode 100644 packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.provider-turn.test.ts diff --git a/packages/paperclip-runner/src/backends/harness-driver-backend.test.ts b/packages/paperclip-runner/src/backends/harness-driver-backend.test.ts index 33f017f8e0f..ac6562d55d6 100644 --- a/packages/paperclip-runner/src/backends/harness-driver-backend.test.ts +++ b/packages/paperclip-runner/src/backends/harness-driver-backend.test.ts @@ -219,6 +219,30 @@ describe("HarnessDriverBackend", () => { expect(recover.mock.calls[0]![0].codexUsageBaseline!.latest.inputTokens).toBe(150); }); + it.each([true, false])("carries the provider-turn wait marker through a serialized native checkpoint only while waiting (%s)", async (awaiting) => { + class WaitSession extends FakeHarnessSession { + override async snapshot(): Promise { + return { ...(await super.snapshot()), semanticResult: undefined, activeTurnId: null, + ...(awaiting ? { awaitingProviderTurn: true } : {}) }; + } + } + const original = new HarnessDriverBackend({ ...driver, openSession: async () => new WaitSession() }); + const session = await original.openSession({ identity: { + runId: "run-1", sessionId: "session-1", companyId: "company-1", issueId: "issue-1", agentId: "agent-1", + } }); + const checkpoint = JSON.parse(JSON.stringify(await session.snapshot())); + const recover = vi.fn(async (_snapshot: PersistedHarnessSession) => ({ recovered: true, session: new WaitSession() })); + await new HarnessDriverBackend({ ...driver, recoverSession: recover }) + .recoverSession(checkpoint, { signal: new AbortController().signal }); + if (awaiting) { + expect(checkpoint.awaitingProviderTurn).toBe(true); + expect(recover.mock.calls[0]![0].awaitingProviderTurn).toBe(true); + } else { + expect(checkpoint).not.toHaveProperty("awaitingProviderTurn"); + expect(recover.mock.calls[0]![0]).not.toHaveProperty("awaitingProviderTurn"); + } + }); + it("rejects and closes a provider session without a durable provider identity", async () => { let closed = false; class MissingProviderIdentitySession extends FakeHarnessSession { diff --git a/packages/paperclip-runner/src/backends/harness-driver-backend.ts b/packages/paperclip-runner/src/backends/harness-driver-backend.ts index e3922fb36b9..b45e0b9bf50 100644 --- a/packages/paperclip-runner/src/backends/harness-driver-backend.ts +++ b/packages/paperclip-runner/src/backends/harness-driver-backend.ts @@ -153,6 +153,7 @@ export class HarnessDriverBackend implements NativeSessionBackend { pendingRuntimeRequests: snapshot.pendingRuntimeRequests ?? [], goal: snapshot.goal ?? null, lineage: snapshot.lineage ?? [], + ...(snapshot.awaitingProviderTurn === true ? { awaitingProviderTurn: true } : {}), }; const recoveryOptions: HarnessSessionRecoveryOptions = { signal: options.signal, @@ -918,6 +919,7 @@ class HarnessNativeSession implements NativeSession { pendingRuntimeRequests: snapshot.pendingRuntimeRequests ?? [], goal: snapshot.goal ?? null, lineage: snapshot.lineage ?? [], + ...(snapshot.awaitingProviderTurn === true ? { awaitingProviderTurn: true } : {}), }; } diff --git a/packages/paperclip-runner/src/contracts/harness-driver.ts b/packages/paperclip-runner/src/contracts/harness-driver.ts index 14b3aae458f..37091f4042d 100644 --- a/packages/paperclip-runner/src/contracts/harness-driver.ts +++ b/packages/paperclip-runner/src/contracts/harness-driver.ts @@ -505,6 +505,7 @@ export interface PersistedHarnessSession { | "same_session_only" | "allow_replacement_after_governed_wait" | "allow_replacement_after_resume_failure"; + awaitingProviderTurn?: boolean; } /** Runner proof that saved provider state cannot be restored by this runner diff --git a/packages/paperclip-runner/src/contracts/native-session-backend.ts b/packages/paperclip-runner/src/contracts/native-session-backend.ts index fcd43490e2f..e8ce03c19dc 100644 --- a/packages/paperclip-runner/src/contracts/native-session-backend.ts +++ b/packages/paperclip-runner/src/contracts/native-session-backend.ts @@ -80,6 +80,7 @@ export interface PersistedNativeSession { pendingRuntimeRequests?: HarnessRuntimeRequest[]; goal?: HarnessThreadGoal | null; lineage?: HarnessThreadLineageEntry[]; + awaitingProviderTurn?: boolean; } export interface NativeSessionRecoveryResult { diff --git a/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts b/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts index f5fa26a6a28..83feb7983a0 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts @@ -771,6 +771,7 @@ export class CodexAppServerDriver implements HarnessDriver { codexUsageBaseline: snapshot.codexUsageBaseline, dispositionOnlyRecoveryConsumed, dispositionOnlyRecoveryTurnId, + awaitingProviderTurn: snapshot.awaitingProviderTurn, adoptedAcceptedTurnId, lostAcceptedTurnId, stalePendingRuntimeRequests: snapshot.pendingRuntimeRequests ?? [], @@ -1108,6 +1109,7 @@ export class CodexAppServerDriver implements HarnessDriver { codexUsageBaseline?: PersistedHarnessSession["codexUsageBaseline"]; dispositionOnlyRecoveryConsumed?: boolean; dispositionOnlyRecoveryTurnId?: string | null; + awaitingProviderTurn?: boolean; adoptedAcceptedTurnId?: string | null; lostAcceptedTurnId?: string | null; stalePendingRuntimeRequests?: HarnessRuntimeRequest[]; diff --git a/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.provider-turn.test.ts b/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.provider-turn.test.ts new file mode 100644 index 00000000000..ad30df81663 --- /dev/null +++ b/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.provider-turn.test.ts @@ -0,0 +1,232 @@ +import { + FakeCodexTransport, + HarnessOperationAlreadyTerminalError, + HarnessStaleTurnError, + WORKSPACE, + collectUntilTerminal, + describe, + expect, + it, + makeDriver, + result, +} from "./codex-app-server-driver.test-support.js"; +import { RUNNERD_RUN_TERMINAL } from "./codex-driver-values.js"; + +async function completedFirstTurn(awaiting: boolean) { + const transport = new FakeCodexTransport(); + const session = await makeDriver([transport]).openSession({ + runId: "provider-turn", normalizedSessionId: "provider-turn-session", workingDirectory: WORKSPACE, + }); + await session.startTurn({ message: { role: "user", text: "Start a background install." } }); + transport.push("turn/started", { threadId: "thread-1", turn: { id: "turn-1", status: "inProgress" } }); + transport.push("turn/completed", { threadId: "thread-1", turn: { id: "turn-1", status: "completed", items: [] }, + ...(awaiting ? { awaitingProviderTurn: true } : {}) }); + const events = session.events()[Symbol.asyncIterator](); + const first = await collectUntilTerminal({ [Symbol.asyncIterator]: () => events }); + return { transport, session, events, first }; +} + +async function restoredAfterFirstTurn(awaiting: boolean) { + const first = new FakeCodexTransport(); + const second = new FakeCodexTransport(); + const driver = makeDriver([first, second]); + const original = await driver.openSession({ + runId: "provider-turn", normalizedSessionId: "provider-turn-session", workingDirectory: WORKSPACE, + }); + await original.startTurn({ message: { role: "user", text: "Start a background install." } }); + first.push("turn/started", { threadId: "thread-1", turn: { id: "turn-1", status: "inProgress" } }); + first.push("turn/completed", { threadId: "thread-1", turn: { id: "turn-1", status: "completed", items: [] }, + ...(awaiting ? { awaitingProviderTurn: true } : {}) }); + await collectUntilTerminal(original.events()); + const checkpoint = await original.snapshot(); + await original.close({ reason: "controller restarted" }); + second.readResponse = { thread: { id: "thread-1", sessionId: "provider-session-1", cwd: WORKSPACE, + turns: [{ id: "turn-1", status: "completed", items: [] }] } }; + const recovery = await driver.recoverSession!(checkpoint); + expect(recovery).toMatchObject({ recovered: true }); + const session = recovery.session!; + return { transport: second, session, checkpoint, events: session.events()[Symbol.asyncIterator]() }; +} + +async function reconciledActiveTurn(awaiting: boolean) { + const first = new FakeCodexTransport(); + const second = new FakeCodexTransport(); + const driver = makeDriver([first, second]); + const original = await driver.openSession({ + runId: "provider-turn", normalizedSessionId: "provider-turn-session", workingDirectory: WORKSPACE, + }); + await original.startTurn({ message: { role: "user", text: "Start a background install." } }); + const checkpoint = await original.snapshot(); + expect(checkpoint.activeTurnId).toBe("turn-1"); + await original.close({ reason: "controller restarted" }); + second.readResponse = { thread: { id: "thread-1", sessionId: "provider-session-1", cwd: WORKSPACE, + turns: [{ id: "turn-1", status: "completed", items: [], ...(awaiting ? { awaitingProviderTurn: true } : {}) }] } }; + const recovery = await driver.recoverSession!(checkpoint); + expect(recovery).toMatchObject({ recovered: true }); + const session = recovery.session!; + return { transport: second, session, events: session.events()[Symbol.asyncIterator]() }; +} + +async function nextUntil(events: AsyncIterator<{ eventType: string; payload: Record }>, done: (eventType: string) => boolean) { + const collected: Array<{ eventType: string; payload: Record; turnId?: string }> = []; + for (;;) { + const next = await events.next(); + if (next.done) return collected; + collected.push(next.value); + if (done(next.value.eventType)) return collected; + } +} + +describe("Codex app-server driver with provider-started turns", () => { + it("marks an awaited terminal and adopts the provider's own next turn with its result", async () => { + const { transport, session, events, first } = await completedFirstTurn(true); + expect(first.at(-1)).toMatchObject({ eventType: "turn.completed", payload: { status: "completed", awaitingProviderTurn: true } }); + transport.push("turn/started", { threadId: "thread-1", turn: { id: "provider-1", status: "inProgress" }, origin: "provider" }); + transport.push("paperclip/runResult", { threadId: "thread-1", turnId: "provider-1", result }); + transport.push("turn/completed", { threadId: "thread-1", turn: { id: "provider-1", status: "completed", items: [] } }); + const second = await nextUntil(events, eventType => eventType === "turn.completed"); + expect(second.some(event => event.eventType === "session.failed")).toBe(false); + expect(second.find(event => event.eventType === "turn.started")).toMatchObject({ turnId: "provider-1", payload: { origin: "provider" } }); + expect(second.at(-1)).toMatchObject({ eventType: "turn.completed", turnId: "provider-1" }); + expect(second.at(-1)!.payload).not.toHaveProperty("awaitingProviderTurn"); + expect((await session.snapshot()).semanticResult).toMatchObject({ result, turnId: "provider-1" }); + await session.close({ reason: "fixture complete" }); + }); + + it("still fails closed on a provider-started turn the Runner did not await", async () => { + const { transport, session, events, first } = await completedFirstTurn(false); + expect(first.at(-1)!.payload).not.toHaveProperty("awaitingProviderTurn"); + transport.push("turn/started", { threadId: "thread-1", turn: { id: "provider-1", status: "inProgress" }, origin: "provider" }); + const failed = await nextUntil(events, eventType => eventType === "session.failed"); + expect(failed.at(-1)).toMatchObject({ eventType: "session.failed", payload: { code: "turn_binding_mismatch" } }); + await session.close({ reason: "fixture complete" }).catch(() => undefined); + }); + + it("acknowledges an interrupt during the wait without sending one to the provider", async () => { + const { transport, session, events } = await completedFirstTurn(true); + await session.interrupt!({ reason: "Stop" }); + const acknowledged = await nextUntil(events, eventType => eventType === "item.completed"); + expect(acknowledged.at(-1)).toMatchObject({ payload: { kind: "interrupt_acknowledgement", status: "provider_turn_wait_ended" } }); + expect(transport.calls.some(request => request.method === "turn/interrupt")).toBe(false); + await session.close({ reason: "fixture complete" }); + }); + + it("never interrupts a running provider-started turn for the wait's own end", async () => { + const { transport, session, events } = await completedFirstTurn(true); + transport.push("turn/started", { threadId: "thread-1", turn: { id: "provider-1", status: "inProgress" }, origin: "provider" }); + await nextUntil(events, eventType => eventType === "turn.started"); + await expect(session.interrupt!({ turnId: "turn-1", reason: "cap" })).rejects.toBeInstanceOf(HarnessStaleTurnError); + expect(transport.calls.some(request => request.method === "turn/interrupt")).toBe(false); + await session.close({ reason: "fixture complete" }); + }); + + it("ends the wait on the Runner's run.terminal only while awaiting with no turn, and ignores a forged one", async () => { + const forged = await completedFirstTurn(true); + forged.transport.push("paperclip/runTerminal", { threadId: "thread-1" }); + forged.transport.push("turn/started", { threadId: "thread-1", turn: { id: "provider-1", status: "inProgress" }, origin: "provider" }); + const untouched = await nextUntil(forged.events, eventType => eventType === "turn.started"); + expect(untouched.some(event => event.payload.kind === "interrupt_acknowledgement")).toBe(false); + await forged.session.close({ reason: "fixture complete" }); + + const awaiting = await completedFirstTurn(true); + awaiting.transport.push("paperclip/runTerminal", { [RUNNERD_RUN_TERMINAL]: true, threadId: "thread-1" }); + const ended = await nextUntil(awaiting.events, eventType => eventType === "item.completed"); + expect(ended.filter(event => event.payload.kind === "interrupt_acknowledgement")).toHaveLength(1); + expect(ended.at(-1)).toMatchObject({ payload: { kind: "interrupt_acknowledgement", status: "provider_turn_wait_ended", cause: "run_terminal" } }); + expect(await awaiting.session.snapshot()).not.toHaveProperty("awaitingProviderTurn"); + awaiting.transport.push("paperclip/runTerminal", { [RUNNERD_RUN_TERMINAL]: true, threadId: "thread-1" }); + awaiting.transport.push("turn/started", { threadId: "thread-1", turn: { id: "provider-1", status: "inProgress" }, origin: "provider" }); + const afterSecond = await nextUntil(awaiting.events, eventType => eventType === "session.failed"); + expect(afterSecond.some(event => event.payload.kind === "interrupt_acknowledgement")).toBe(false); + await awaiting.session.close({ reason: "fixture complete" }).catch(() => undefined); + + const ordinary = await completedFirstTurn(false); + ordinary.transport.push("paperclip/runTerminal", { [RUNNERD_RUN_TERMINAL]: true, threadId: "thread-1" }); + ordinary.transport.push("turn/started", { threadId: "thread-1", turn: { id: "provider-1", status: "inProgress" }, origin: "provider" }); + const after = await nextUntil(ordinary.events, eventType => eventType === "session.failed"); + expect(after.some(event => event.payload.kind === "interrupt_acknowledgement")).toBe(false); + await ordinary.session.close({ reason: "fixture complete" }).catch(() => undefined); + }); + + it("keeps the wait across a controller restart and adopts the provider's own next turn", async () => { + const { transport, session, checkpoint, events } = await restoredAfterFirstTurn(true); + expect(checkpoint).toMatchObject({ activeTurnId: null, semanticResult: null, awaitingProviderTurn: true }); + expect(await session.snapshot()).toMatchObject({ awaitingProviderTurn: true }); + transport.push("turn/started", { threadId: "thread-1", turn: { id: "provider-1", status: "inProgress" }, origin: "provider" }); + transport.push("paperclip/runResult", { threadId: "thread-1", turnId: "provider-1", result }); + transport.push("turn/completed", { threadId: "thread-1", turn: { id: "provider-1", status: "completed", items: [] } }); + const adopted = await nextUntil(events, eventType => eventType === "turn.completed" || eventType === "session.failed"); + expect(adopted.some(event => event.eventType === "session.failed")).toBe(false); + expect(adopted.find(event => event.eventType === "turn.started")).toMatchObject({ turnId: "provider-1", payload: { origin: "provider" } }); + expect(adopted.at(-1)).toMatchObject({ eventType: "turn.completed", turnId: "provider-1" }); + const settled = await session.snapshot(); + expect(settled.semanticResult).toMatchObject({ result, turnId: "provider-1" }); + expect(settled).not.toHaveProperty("awaitingProviderTurn"); + await session.close({ reason: "fixture complete" }); + }); + + it("marks a persisted active turn that a restart reconciles as awaiting, then adopts the provider's own next turn", async () => { + const { transport, session, events } = await reconciledActiveTurn(true); + const reconciled = await nextUntil(events, eventType => eventType === "turn.completed"); + expect(reconciled.at(-1)).toMatchObject({ eventType: "turn.completed", turnId: "turn-1", + payload: { status: "completed", awaitingProviderTurn: true } }); + expect(await session.snapshot()).toMatchObject({ activeTurnId: null, semanticResult: null, awaitingProviderTurn: true }); + transport.push("turn/started", { threadId: "thread-1", turn: { id: "provider-1", status: "inProgress" }, origin: "provider" }); + transport.push("paperclip/runResult", { threadId: "thread-1", turnId: "provider-1", result }); + transport.push("turn/completed", { threadId: "thread-1", turn: { id: "provider-1", status: "completed", items: [] } }); + const adopted = await nextUntil(events, eventType => eventType === "turn.completed" || eventType === "session.failed"); + expect(adopted.some(event => event.eventType === "session.failed")).toBe(false); + expect(adopted.find(event => event.eventType === "turn.started")).toMatchObject({ turnId: "provider-1", payload: { origin: "provider" } }); + expect(adopted.at(-1)).toMatchObject({ eventType: "turn.completed", turnId: "provider-1" }); + expect((await session.snapshot()).semanticResult).toMatchObject({ result, turnId: "provider-1" }); + await session.close({ reason: "fixture complete" }); + }); + + it("reconciles an unmarked persisted active turn as an ordinary terminal", async () => { + const { transport, session, events } = await reconciledActiveTurn(false); + const reconciled = await nextUntil(events, eventType => eventType === "turn.completed"); + expect(reconciled.at(-1)).toMatchObject({ eventType: "turn.completed", turnId: "turn-1" }); + expect(reconciled.at(-1)!.payload).not.toHaveProperty("awaitingProviderTurn"); + expect(await session.snapshot()).not.toHaveProperty("awaitingProviderTurn"); + transport.push("turn/started", { threadId: "thread-1", turn: { id: "provider-1", status: "inProgress" }, origin: "provider" }); + const failed = await nextUntil(events, eventType => eventType === "session.failed"); + expect(failed.at(-1)).toMatchObject({ eventType: "session.failed", payload: { code: "turn_binding_mismatch" } }); + await session.close({ reason: "fixture complete" }).catch(() => undefined); + }); + + it("writes no wait marker in a snapshot taken outside a wait", async () => { + const ordinary = await completedFirstTurn(false); + expect(await ordinary.session.snapshot()).not.toHaveProperty("awaitingProviderTurn"); + await ordinary.session.close({ reason: "fixture complete" }); + const opened = await makeDriver([new FakeCodexTransport()]).openSession({ + runId: "provider-turn", normalizedSessionId: "provider-turn-session", workingDirectory: WORKSPACE, + }); + expect(await opened.snapshot()).not.toHaveProperty("awaitingProviderTurn"); + await opened.close({ reason: "fixture complete" }); + }); + + it("still fails closed on a provider-started turn after a restore with no wait", async () => { + const { transport, session, checkpoint, events } = await restoredAfterFirstTurn(false); + expect(checkpoint).not.toHaveProperty("awaitingProviderTurn"); + transport.push("turn/started", { threadId: "thread-1", turn: { id: "provider-1", status: "inProgress" }, origin: "provider" }); + const failed = await nextUntil(events, eventType => eventType === "session.failed"); + expect(failed.at(-1)).toMatchObject({ eventType: "session.failed", payload: { code: "turn_binding_mismatch" } }); + await session.close({ reason: "fixture complete" }).catch(() => undefined); + }); + + it("acknowledges an interrupt during a restored wait without sending one to the provider", async () => { + const { transport, session, events } = await restoredAfterFirstTurn(true); + await session.interrupt!({ reason: "Stop" }); + let acknowledged = await events.next(); + while (!acknowledged.done && acknowledged.value.payload.kind !== "interrupt_acknowledgement") acknowledged = await events.next(); + expect(acknowledged.value).toMatchObject({ payload: { kind: "interrupt_acknowledgement", status: "provider_turn_wait_ended" } }); + expect(transport.calls.some(request => request.method === "turn/interrupt")).toBe(false); + await session.close({ reason: "fixture complete" }); + }); + + it("keeps rejecting an interrupt after a terminal that does not await", async () => { + const { session } = await completedFirstTurn(false); + await expect(session.interrupt!({ reason: "Stop" })).rejects.toBeInstanceOf(HarnessOperationAlreadyTerminalError); + await session.close({ reason: "fixture complete" }); + }); +}); diff --git a/packages/paperclip-runner/src/drivers/codex/codex-driver-values.ts b/packages/paperclip-runner/src/drivers/codex/codex-driver-values.ts index 241d91e5663..02283ba02ce 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-driver-values.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-driver-values.ts @@ -43,6 +43,7 @@ export const RUNNERD_CANONICAL_ITEM = Symbol.for( ); export const RUNNERD_RUNTIME_ACTIVITY = Symbol("paperclip.runnerd.runtime-activity"); +export const RUNNERD_RUN_TERMINAL = Symbol("paperclip.runnerd.run-terminal"); export function text(value: unknown, fallback = ""): string { return typeof value === "string" ? value : fallback; diff --git a/packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts b/packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts index ca2de721d59..ac4cd9d581e 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts @@ -476,6 +476,19 @@ export class CodexHarnessSession ); return; } + if (this.awaitingProviderTurn && this.activeTurnId === null && !this.turnStartPending) { + this.emit( + "item.completed", + { + kind: "interrupt_acknowledgement", + text: "Paperclip stopped waiting for the provider's own next turn.", + status: "provider_turn_wait_ended", + reason: boundedText(input.reason), + }, + { itemId: `interrupt:provider-turn-wait:${++this.interruptSequence}` }, + ); + return; + } if (this.terminal || this.activeTurnId === null) { throw new HarnessOperationAlreadyTerminalError("interruption"); } @@ -836,7 +849,7 @@ export class CodexHarnessSession ["completed", "failed", "interrupted", "cancelled"].includes(status) ) { reconciledTerminalTurnId = expectedTurnId; - mapTerminalTurn(this, expectedTurn, expectedTurnId, true); + mapTerminalTurn(this, expectedTurn, expectedTurnId, true, expectedTurn.awaitingProviderTurn === true); } else { throw new HarnessReconciliationError( `persisted active turn ${expectedTurnId} has unreconcilable status ${status || "missing"}`, @@ -920,6 +933,7 @@ export class CodexHarnessSession this.currentGoal === null ? null : structuredClone(this.currentGoal), lineage: this.lineage(), lastSourceSequence: this.sourceSequence, + ...(this.awaitingProviderTurn ? { awaitingProviderTurn: true } : {}), }; } diff --git a/packages/paperclip-runner/src/drivers/codex/codex-session-notifications.ts b/packages/paperclip-runner/src/drivers/codex/codex-session-notifications.ts index 5a2ba36ad28..e33481395e9 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-session-notifications.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-session-notifications.ts @@ -30,6 +30,7 @@ import { recordWorkspaceChanges, } from "./codex-session-workspace.js"; import { + RUNNERD_RUN_TERMINAL, RUNNERD_RUNTIME_ACTIVITY, boundedText, differingJsonPaths, @@ -152,6 +153,19 @@ async function mapNotification(state: CodexSessionState, notification: CodexRpcN } async function mapNotificationBody(state: CodexSessionState, notification: CodexRpcNotification): Promise { + if (notification.method === "paperclip/runTerminal") { + if ((notification.params as Record)[RUNNERD_RUN_TERMINAL] !== true) return; + if (state.awaitingProviderTurn && state.activeTurnId === null && !state.turnStartPending) { + state.awaitingProviderTurn = false; + state.emit("item.completed", { + kind: "interrupt_acknowledgement", + text: "The Runner ended the wait for the provider's own next turn.", + status: "provider_turn_wait_ended", + cause: "run_terminal", + }, { itemId: `interrupt:provider-turn-wait:run-terminal:${state.sourceSequence + 1}` }); + } + return; + } if (notification.method === "paperclip/runtimeActivityRecorded") { const params = notification.params; if ((params as Record)[RUNNERD_RUNTIME_ACTIVITY] !== true) return; @@ -538,7 +552,8 @@ async function mapNotificationBody(state: CodexSessionState, notification: Codex state.recordHistoryEvent(notification.method, turnId); return; } - const autonomousGoalTurn = state.currentGoal?.status === "active" + const providerStartedTurn = params.origin === "provider" && state.awaitingProviderTurn; + const autonomousGoalTurn = (state.currentGoal?.status === "active" || providerStartedTurn) && state.activeTurnId === null && !state.turnStartPending && !state.protocolFailed @@ -569,9 +584,10 @@ async function mapNotificationBody(state: CodexSessionState, notification: Codex state.activeTurnId = turnId; state.turnStartPending = false; state.turnStarted = true; + state.awaitingProviderTurn = false; state.emit( "turn.started", - { status: text(turn.status, "inProgress") }, + { status: text(turn.status, "inProgress"), ...(providerStartedTurn ? { origin: "provider" } : {}) }, { turnId }, ); return; @@ -608,7 +624,7 @@ async function mapNotificationBody(state: CodexSessionState, notification: Codex } if (ignoreUnboundRestoredTurnLoss(state, turn, turnId)) return; if (!state.notificationNamesActiveTurn(turnId, "turn terminal")) return; - mapTerminalTurn(state, turn, turnId); + mapTerminalTurn(state, turn, turnId, false, params.awaitingProviderTurn === true); return; } if (notification.method === "item/started") { diff --git a/packages/paperclip-runner/src/drivers/codex/codex-session-state.ts b/packages/paperclip-runner/src/drivers/codex/codex-session-state.ts index 5314dd5ae66..3a7ca9270db 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-session-state.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-session-state.ts @@ -161,6 +161,7 @@ export class CodexSessionState { * stricter consumed marker. */ restartContinuationTurnId: string | null = null; + awaitingProviderTurn = false; turnStarted = false; readonly terminalTurns = new Map(); /** @@ -217,6 +218,7 @@ export class CodexSessionState { codexUsageBaseline?: CodexUsageBaseline; dispositionOnlyRecoveryConsumed?: boolean; dispositionOnlyRecoveryTurnId?: string | null; + awaitingProviderTurn?: boolean; /** A recovered turn whose turn.accepted the previous controller never emitted. */ adoptedAcceptedTurnId?: string | null; /** A turn the runner accepted for an earlier generation that the restore lost. */ @@ -309,6 +311,8 @@ export class CodexSessionState { if (this.activeTurnId && this.terminalTurns.has(this.activeTurnId)) { this.activeTurnId = null; } + this.awaitingProviderTurn = input.awaitingProviderTurn === true && this.activeTurnId === null + && this.result === null && this.terminalTurns.size > 0; const dispositionOnlyRecoveryPreviouslyConsumed = input.dispositionOnlyRecoveryConsumed ?? false; const dispositionOnlyRecoveryTurnId = diff --git a/packages/paperclip-runner/src/drivers/codex/codex-session-terminal.ts b/packages/paperclip-runner/src/drivers/codex/codex-session-terminal.ts index f344b5af03b..736c0166349 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-session-terminal.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-session-terminal.ts @@ -153,6 +153,7 @@ export function mapTerminalTurn( turn: Record, fallbackTurnId: string, reconciling = false, + awaitingProviderTurn = false, ): void { const turnId = text(turn.id, fallbackTurnId || state.activeTurnId || ""); const reported = turn; @@ -214,9 +215,11 @@ export function mapTerminalTurn( state.cancelPendingRequests("turn_terminal"); state.activeTurnId = null; state.turnStarted = false; + state.awaitingProviderTurn = awaitingProviderTurn && eventType === "turn.completed" && state.result === null; const providerFailure = status === "failed" ? readProviderFailure(record(turn.error).providerFailure) ?? (state.driverKind === "codex_app_server" ? codexProviderFailure(turn.error, "turn", state.diagnosticEnvironment) : null) : null; - const payload = boundedPayload({ status, error: turn.error ?? null }); + const payload = boundedPayload({ status, error: turn.error ?? null, + ...(state.awaitingProviderTurn ? { awaitingProviderTurn: true } : {}) }); if (providerFailure) { payload.status = status; payload.error = { ...record(payload.error), providerFailure }; diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts index b983446b565..5169606853d 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts @@ -1962,6 +1962,27 @@ it("settles a restored turn from the runner's reconciled result terminal", () => expect(recovered([runTerminal], "turn-1")).toBeNull(); }); +it("carries the provider-turn wait marker on a recovered turn terminal only when the PRP turn.completed has it", () => { + const { runnerdRecoveredTurnTerminal: recovered, runnerdPriorTurnTerminals: prior } = runnerdRecoveryInternals; + const event = (eventType: string, payload: Record) => ({ eventType, envelope: { runId: "run-1", payload: { payload } } }); + const awaited = event("turn.completed", { providerTurnId: "turn-1", awaitingProviderTurn: true }); + const plain = event("turn.completed", { providerTurnId: "turn-1" }); + const failed = event("turn.failed", { providerTurnId: "turn-1", awaitingProviderTurn: true }); + const forged = event("turn.completed", { providerTurnId: "turn-1", awaitingProviderTurn: "true" }); + expect(recovered([awaited], "turn-1")).toEqual({ status: "completed", error: null, awaitingProviderTurn: true }); + expect(recovered([plain], "turn-1")).toStrictEqual({ status: "completed", error: null }); + expect(recovered([failed], "turn-1")).toStrictEqual({ status: "failed", error: null }); + expect(recovered([forged], "turn-1")).toStrictEqual({ status: "completed", error: null }); + expect([...prior([awaited], "run-1", "turn-2").values()]) + .toEqual([{ id: "turn-1", status: "completed", error: null, awaitingProviderTurn: true }]); + for (const unmarked of [plain, failed, forged]) { + expect([...prior([unmarked], "run-1", "turn-2").values()]) + .toStrictEqual([{ id: "turn-1", status: unmarked.eventType.slice("turn.".length), error: null }]); + } + expect(prior([awaited], "run-1", "turn-1").size).toBe(0); + expect(prior([awaited], "run-other", "turn-2").size).toBe(0); +}); + it("requires ACPX command results to preserve the requested turn identity", () => { expect( runnerdRecoveryInternals.turnStartCommandResultValid({ diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts index 0fbecce161f..6fc75db84f2 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts @@ -886,8 +886,35 @@ function turnStartResponseReady(input: { type RunnerdRecoveredTurnTerminal = { status: "completed" | "failed" | "interrupted" | "cancelled"; error: unknown; + awaitingProviderTurn?: true; }; +function awaitingProviderTurnMarker(eventType: string, payload: Record): { awaitingProviderTurn?: true } { + return eventType === "turn.completed" && payload.awaitingProviderTurn === true ? { awaitingProviderTurn: true } : {}; +} + +function runnerdPriorTurnTerminals( + events: readonly Pick[], + runId: string | undefined, + boundTurnId: string, +): Map> { + const priorTerminals = new Map>(); + for (const event of events) { + if (event.envelope.runId !== runId || + !["turn.completed", "turn.failed", "turn.interrupted", "turn.cancelled"].includes(event.eventType)) continue; + const payload = record(record(event.envelope.payload).payload); + const turnId = payload.providerTurnId ?? payload.turnId ?? record(payload.turn).id; + if (typeof turnId !== "string" || !turnId || turnId === boundTurnId) continue; + priorTerminals.set(turnId, { + id: turnId, + status: event.eventType.slice("turn.".length), + error: payload.error ?? record(payload.turn).error ?? null, + ...awaitingProviderTurnMarker(event.eventType, payload), + }); + } + return priorTerminals; +} + const RUNNERD_TURN_TERMINAL_STATUSES = new Set(["completed", "failed", "interrupted", "cancelled"]); /** @@ -916,7 +943,8 @@ function runnerdRecoveredTurnTerminal( // Reconciliation must retain the cause, including the runner's explicit // process-loss marker, rather than inventing error:null. terminal = { status: event.eventType.slice("turn.".length) as RunnerdRecoveredTurnTerminal["status"], - error: payload.error ?? record(payload.turn).error ?? null }; + error: payload.error ?? record(payload.turn).error ?? null, + ...awaitingProviderTurnMarker(event.eventType, payload) }; reconciledEndedTurn = false; } else if (event.eventType === "run.terminal" && reconciledEndedTurn) { const status = String(payload.turnTerminalState); @@ -4652,19 +4680,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { // Keep its prior terminal as the history anchor, so driver recovery can // adopt the later accepted turn instead of submitting it again. Items // remain lazy and fail closed if their start left the retained window. - const priorTerminals = new Map>(); - for (const event of this.#core?.store.state.committedEvents ?? []) { - if (event.envelope.runId !== this.#core?.store.state.identity.runId || - !["turn.completed", "turn.failed", "turn.interrupted", "turn.cancelled"].includes(event.eventType)) continue; - const payload = record(record(event.envelope.payload).payload); - const turnId = payload.providerTurnId ?? payload.turnId ?? record(payload.turn).id; - if (typeof turnId !== "string" || !turnId || turnId === this.#turnId) continue; - priorTerminals.set(turnId, { - id: turnId, - status: event.eventType.slice("turn.".length), - error: payload.error ?? record(payload.turn).error ?? null, - }); - } + const priorTerminals = runnerdPriorTurnTerminals( + this.#core?.store.state.committedEvents ?? [], this.#core?.store.state.identity.runId, this.#turnId); recoveredTurns.unshift(...priorTerminals.values()); this.#recoveryTurnBindingPending = false; this.#pumpEvents(); @@ -9786,6 +9803,7 @@ export const runnerdRecoveryInternals = Object.freeze({ rotatedRunAttachPayload, localRunnerStateArchive, rotateAuthorityEpoch, + runnerdPriorTurnTerminals, runnerdRecoveredTurnTerminal, runnerdTerminalProviderTurnId, terminalMayBelongToPendingTurnStart,