mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core runs Claude Code through Paperclip Runner. Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. > - The Runner can keep Claude while a background shell is running and give the cycle of Claude to the controller as a turn that the provider started. Core can wait for that turn. #21607 lets a caller ask the transport for it. > - A controller can restart at any time, for example at a deploy. A wait can last 10 minutes and a provider-started turn can last longer, so a restart during one of them is a normal event. > - Tests with the real `paperclip-runnerd` showed that such a restart failed the session. The checkpoint that Core had was the checkpoint of the turn before, and the recovery of the driver refused or mislabelled the turns that it did not know. > - This pull request makes that restart safe. It changes no production run, because no caller asks for provider-started turns before part 3. > - The benefit is that part 3 can turn the behavior on without a failed run at each deploy. ## Linked Issues or Issue Description No public issue exists. The issue is described here. #21456 turned the behavior on and had grown too large to review. It is now three stacked pull requests: 1. #21607 (merged): the transport can ask the Runner for provider-started turns. No caller asks. 2. **This pull request:** the recovery after a controller restart during a wait or during a provider-started turn. 3. #21456: Core sets the option for native ACPX Claude runs. Only that part changes what a production run does. This is part 2. Its base branch is `master`. Refs #19912 (design), #21456. **What existing behavior does this improve?** None in production. It makes a controller restart safe for a run that asked for provider-started turns. **Subsystem affected** Paperclip Runner package: the consumer (`src/native-session-runtime.ts`), the session backend (`src/backends/harness-driver-backend.ts`), the driver (`src/drivers/codex/`) and the transport (`src/live/`). Only for a run that asked for provider-started turns. **Current behavior** - The consumer saves no checkpoint when a wait starts or when a provider-started turn starts. During a wait and during a provider-started turn, the checkpoint that Core has is the checkpoint from the start of the turn before. It names that turn as active. - A restart from that checkpoint fails: the driver has the old turn as active, gets the events of a provider-started turn, and stops the session with `turn_binding_mismatch`. - The recovery of the driver adopts one turn that it does not know as a disposition recovery turn, and refuses two or more. With two or more, it closes the transport, and that stops the Runner and Claude. - The Runner does not send again an event that the controller acknowledged. If the start of a provider-started turn was acknowledged before the restart, the new driver gets the other events of that turn with no start. - A turn that ends waiting for a provider-started turn sets the terminal of the session, as a turn that ends the run does. After a restart, if the provider-started turn already has its result when recovery completes, the consumer finds a result and a terminal, settles the run, and does not read the events of that turn. The start, the items, the usage event and the terminal of that turn do not reach Core. - The recovered driver reads notifications from its construction, and `reconcile()` maps the terminal of the saved turn only after its reads. If the saved checkpoint is still the checkpoint of the controller turn and the start of the provider-started turn is already queued, the driver judges that start too early and stops the session with `turn_binding_mismatch`. - A checkpoint that waits can also have a consumed disposition marker (a recovery turn after an earlier restart ended waiting). The driver restores that session as closed and reads no notifications, and the consumer settles the run on the earlier terminal with no result. **Proposed behavior** - **Checkpoints.** The consumer saves a checkpoint when a wait starts and when a provider-started turn starts, with the function that it already uses for a governed wait. A checkpoint that fails stops the turn, as for a governed wait. - **A checkpoint as of the event that Core stored.** The driver gives each event its sequence when it reads it from the provider, before the consumer stores it. A snapshot that is taken later can be ahead of Core. For these two checkpoints, the driver captures its snapshot when it has handled the notification that made the event (the `turn.completed` that waits, or the `turn.started` of the provider), with the sequence of that event. The consumer asks for the snapshot as of the event that it has just appended. The saved state and cursor are then those of an event that Core has. - **Session terminal.** A turn that ends waiting for a provider-started turn does not set the terminal of the session (`HarnessNativeSession`). The terminal of the session is the end of the run, and that turn does not end the run. A checkpoint that is saved during a wait or during a provider-started turn has no terminal, so a recovered session settles only from the events that it reads, in order, to the terminal of the turn that has the result. - **Driver, a checkpoint that waits.** Recovery keeps the wait. It does not adopt and does not refuse the turns after the last terminal that it knows. The Runner sends again, in order, each event that was not acknowledged, and the driver admits each provider-started turn from that stream, as it does with no restart. - **Driver, a checkpoint with an active turn.** `reconcile()` accepts one turn in progress that comes after the saved active turn, when the saved turn ended with the wait marker. It refuses each other unexpected active turn as before, and it refuses this one too when the reconciled turn settled a result. - **Driver, a start that is queued before recovery reconciled the saved turn.** A `turn/started` of the provider that arrives while the session does not wait for one first waits until `reconcile()` has settled, and is then judged as before. `reconcile()` judges the live state after its reads; if the stream already settled the saved turn into a wait, it accepts exactly one turn in progress after the last terminal turn and does not adopt it. The driver admits that turn at its start. - **A wait with a consumed disposition marker.** The driver keeps reading notifications for a restored session that waits, as it does for a wait with no restart. It still refuses a second controller-started turn. The consumer does not take the consumed-marker shortcut for a recovered snapshot that waits, so the wait is resumed. - **Transport, the start comes first.** For each driver, the transport gives the `turn/started` of a provider-started turn before each other notification and before a tool call of that turn. If it did not give that start to this driver, it builds it from the `turn.started` event in its stored events. It does not do this for the turn that the recovered driver already has as active (`resumeActiveTurnId`). It binds the events and the tool call of a provider-started turn to that turn, not to the turn that the Runner reports as active at the time of the recovery read. - **Transport, which start owns an event.** The stored events can have gaps: the window removes old events but keeps the latest of each anchor type, and the controller checkpoint drops streamed deltas. The owner of an event is the last stored `turn.started` before it, if it is a provider start, and if the first turn terminal that is stored after that start is not before the event and is the terminal of the same provider turn (`providerTurnId`). If the first stored terminal after the start is the terminal of a different turn, the end of the provider turn is not known, and the transport builds no start. **Reason and benefit** - The checkpoint is the cause. With a true checkpoint, the two states of a restart are states that recovery can handle: a wait, or an active turn. - The Runner already sends again each event that was not acknowledged. Recovery uses that stream and does not build the turns again from a read. - The window removes a terminal only while a later terminal of the same type is stored. Thus the first stored terminal after a start shows if a terminal of that turn was removed. A rule that only ignores the gaps is not safe: the Runner can make a `turn.failed` for a turn that did not emit its start (`provider_turn_lost_on_restore`). ## What Changed Production code: 113 lines added, 19 removed. - `src/native-session-runtime.ts` (15 added, 4 removed): the two checkpoints in `consumeTurn`, with the sequence of the appended event, and the resumed wait with a consumed disposition marker. - `src/drivers/codex/codex-session-state.ts` (24 added, 1 removed), `codex-session-terminal.ts` (2 added), `codex-session-notifications.ts` (2 added), `codex-harness-session.ts` (24 added, 7 removed), `codex-app-server-driver-impl.ts` (4 added, 1 removed): the snapshot at the event, and the recovery of the driver. - `src/contracts/harness-driver.ts` and `src/contracts/native-session-backend.ts` (1 added each): an optional `boundarySnapshot` on a driver session, and an optional sequence in the snapshot options. Other drivers do not implement it and are not changed. - `src/backends/harness-driver-backend.ts` (7 added, 5 removed): the session terminal, and the snapshot as of a sequence. - `src/live/runnerd-codex-transport.ts` (15 added, 1 removed) and `src/live/provider-turn-tool-order.ts` (18 added): the start-first rule and the owner of an event. - Tests: `src/live/runnerd-acpx-provider-turn.test.ts` (572 added, 5 removed), `src/drivers/codex/codex-app-server-driver.provider-turn.test.ts` (348 added), `src/live/runnerd-codex-transport.test.ts` (137 added), `src/control-plane/durable-prp-control-plane.test.ts` (84 added), `src/live/provider-turn-tool-order.test.ts` (new, 61), `src/native-session-runtime.provider-turn.test.ts` (92 added, 6 removed), `src/backends/harness-driver-backend.test.ts` (56 added). - `runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs` (test binary only, 123 added, 14 removed): a delay before a provider-started turn, a goal state at its terminal, a list of provider-started turns for one session, a gap between them, a turn that stays open until the new controller reads the snapshot, and a turn that starts after that read. The modes that existed are not changed, and the fake reads its input on one thread as before. ## State saved by the previous release - No new persisted field. The driver session already has `awaitingProviderTurn` (#21453). - **More checkpoints:** one when a wait starts and one when a provider-started turn starts. An ordinary turn saves the same number of checkpoints as before (4 in the unit test, measured before and after). - **No terminal in a checkpoint of a wait.** A checkpoint that is saved during a wait or during a provider-started turn has `terminal: null`. When a wait ends with no provider-started turn (the limit, a comment of a person, or the Runner ends the run), the last checkpoint of the run also has `terminal: null`; before, it had the terminal of the turn that waited. One reader uses that field with no result: the text of a cleanup failure (`nativeCheckpointRecordsCompletedTurn`), which then does not say that the turn completed. In the tests with the real daemon (the limit, the provider process lost, the sidecar exits), the run still ends on the path of today (`native_finalization_missing`). - **A checkpoint of the previous release** has no waiting state. Recovery takes the path of today. - **Rollback of the Runner package during a wait:** the previous code reads the checkpoint, ignores `awaitingProviderTurn`, and takes its recovery path of today. ## Verification With nothing from part 3. The Rust checks and the build checks in the first item were run before the last seven commits; these commits change no Rust. The other items were run at the head of this branch (`36d5e5e587`), or at an earlier commit of this branch for a suite whose files the later commits do not change. - Before the last seven commits: `pnpm run typecheck:typescript`, `pnpm run build:typescript` and `node --test test/acpx-sidecar-contract.test.mjs` (18 passed) in `packages/paperclip-runner`; `node scripts/verify-runner-vendor-dependencies.mjs` in `server`; `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`; `cargo fmt --check`, `cargo build`, the 17 `acpx_*` suites (241 passed) and the `acpx` unit tests (102 passed): all passed. - `npx tsc --noEmit -p tsconfig.json` and `npx tsc --noEmit -p tsconfig.test.json` in `packages/paperclip-runner`; `pnpm typecheck` in `server`: all passed. - `src/live/runnerd-acpx-provider-turn.test.ts`, 4 runs with the production code of this head, 26 of 26 passed in each. The Core of these tests follows the production append (`appendNativeSourceEvent`): it stores an event that is out of order, keeps its contiguous cursor below a gap, and reports a duplicate or a conflict for an event that it has. It does not reject a gap in the sequence; production does not. The restart cases, all with the real `paperclip-runnerd`: - **From the checkpoint that the real consumer saved.** The first `executeNativeSession` is fenced and detached as the server does for a restart, and a second one recovers from the last checkpoint that the first one saved. (a) Restart during the wait; the provider-started turn starts after the new controller is attached. (b) Restart while a provider-started turn is in progress. In each: the test asserts the shape of that checkpoint, the run settles with the result of the provider-started turn, each event is appended one time and in order, the turn has one `turn.started` with `origin: "provider"`, and the session closes with no error. - **Without the two checkpoints, each of these cases fails:** the saved checkpoint is the checkpoint of the turn before, and the second controller fails with `turn_binding_mismatch` or with a replay conflict. - **A slow `appendEvent`, with events that arrive behind the stored event** (3 cases, each run 3 times with the fix and 3 times without it, with the same result in each run). R7: the append of the `turn.completed` that waits is slow, and the provider-started turn starts behind it. With the snapshot at the event, the checkpoint is the waiting state at the sequence that Core has, and Core gets the `turn.started` of the turn one time. Without it, the checkpoint names the provider-started turn as active, the transport builds no start, and Core never gets that `turn.started`. R8: the first controller answers the `paperclip_finish` call while the append of the provider `turn.started` is slow. With the snapshot at the event, the checkpoint has no result and the run ends with `native_finalization_missing` (see Risks). Without it, the checkpoint has a result that Core does not have, and recovery stops with "session cannot start another turn". R6: the append of the provider `turn.started` is slow and a delta is queued behind it. With the snapshot at the event, the new controller continues the sequence that Core has. Without it, the only difference is one sequence number that Core never gets. - **The start was acknowledged before the restart** (3 cases: one turn open across the restart; two turns; a turn that waits for the answer to its `paperclip_finish` call). The new driver gets the start of each turn one time before its other events, the finish call is answered exactly one time, and the session closes with no error. - **The turn ran with no controller attached** (2 cases, with and with no result) and **two turns in one outage** (2 cases): the turns arrive in order after the recovery. - **A provider-started turn that is active in the checkpoint** gets no second start, and the next provider-started turn arrives after it. - **The turn already has its result when recovery completes** (3 cases: the turn made its `paperclip_finish` call with no controller attached; the turn waited for the answer to that call at the restart; two turns across the restart). The test asserts the exact list of events that Core gets: each provider-started turn has one start, its usage item, its message item and its terminal, and then `run.result.accepted` and `run.terminal`. `completeRun` is called one time with the turn that has the result. Before the change to the session terminal, the second controller appended only `run.result.accepted` and `run.terminal` in each of these cases. - **The wait checkpoint is not saved, then a restart** (R10). The last checkpoint is the checkpoint of the controller turn, Core has the terminal of that turn, and the start of the provider-started turn is queued. The run settles with the result of the provider-started turn, Core has one `turn.started` for it, and the finish call is answered one time. Without the wait in the driver, recovery stops with `turn_binding_mismatch`. - **An ordinary turn**, with no option. One that waits for the answer to its finish call at a restart: Core gets its result, its terminal and the run terminal one time. One whose checkpoint after `startTurn` is ahead of Core at a restart (R9): the run settles, and Core does not have only the one event that the first controller did not store. These cases pass with and without this pull request. - **No checkpoint with a terminal and no result** is saved by the two controllers in the cases above. - `src/live/provider-turn-tool-order.test.ts` and `src/live/runnerd-codex-transport.test.ts`: 345 passed. The helper tests cover a stored start with removed events after it, a dropped delta, the terminal of the turn before and after the event, a terminal of a different turn after a removed terminal, and a controller start. With no option, with a different agent or provider, and for a turn that the controller started, the notifications of the transport are the same as before this pull request. - `src/control-plane/durable-prp-control-plane.test.ts`: 230 passed. Three new tests use the real commit path: a full window of 4,096 events removes the events after a provider start and the start is found; the window removes the `turn.failed` of a provider turn after a later `turn.failed` with no start, and no start is found; a restored checkpoint that dropped the deltas of the turn, and the start is found. - `src/drivers/codex` and `src/backends/harness-driver-backend.test.ts`: 491 passed. The driver tests cover a provider start that is queued before `recoverSession()` is called, the wait state with one turn in progress at `reconcile()` (and the two cases that it must still refuse), a restored wait with a consumed disposition marker (the provider turn is admitted and `startTurn` is refused), the snapshot at the two events (also after the driver moved on, and for a restart continuation that ends waiting), one later turn in progress, one completed and a second in progress, two completed with a result, and the cases that `reconcile()` must still refuse. The backend tests cover the session terminal and the snapshot as of a sequence. - `src/native-session-runtime.provider-turn.test.ts` and `src/native-session-runtime.test.ts`: 333 passed. Two new cases resume a restored wait with a consumed disposition marker: the provider-started turn arrives and settles the run, or the wait ends at its limit and the run takes the path of today with no new turn. - `server`: `native-session-executor.test.ts`, `native-provider-turn-wait.test.ts` and `native-runner-restart-recovery.integration.test.ts`: 1,743 passed, 21 skipped. - Mutation checks, each made named tests fail: no checkpoint at the start of a wait; no checkpoint at the start of a provider-started turn; the consumer does not pass the sequence (R6, R7 and R8 fail: a hole in the sequence, no `turn.started` in Core, a different failure); the backend ignores the sequence; the driver has no snapshot at the event; no start-first rule; no exclusion of the active turn of the recovered driver; the turn binding only at the first delivery; the contiguous sequence required again; no check of the first terminal after the start; the adoption of later turns put back; no exception in `reconcile()`; no check after the settlement in `reconcile()`; the terminal of the session set again by a turn that ends waiting; no wait of the provider start for `reconcile()`; no exception in `reconcile()` for a session that waits; the pump deferred for a restored wait; the consumed-marker shortcut taken for a snapshot that waits. Not run: the full suite. CI supplies that coverage. ## Risks - **No production effect.** Each change is behind the waiting state, a provider `turn.started`, or the option of the transport. No caller sets the option before part 3. - **An event that the first controller received but Core did not store is not sent again after a restart.** This is the same on `master` and is not changed here. Only the terminal of a turn is built again. If the lost event is the result of a provider-started turn (the first controller answered the `paperclip_finish` call and stopped before Core stored the proposal), the run ends on the path of today for a turn with no result. R8 shows this. R9 shows the same loss for one event of an ordinary turn. - **A second `turn.started` for one provider-started turn can reach Core.** This occurs if the restart comes after Core stored the start and before the checkpoint of that start is saved (the time of one checkpoint write). The last checkpoint then waits, and the transport builds the start again for the new driver. Seen with the real daemon when the checkpoint write is held; not in a committed test. The usage accounting of the server uses the turn id as its key, so there is no second count, but the second start resets the "restored" mark of that turn. The transcript does not read `turn.started`. - **The older checkpoints can also save a cursor that is ahead of Core** (measured for the checkpoint after `startTurn` of an ordinary turn). Production accepts the gap in the sequence that follows, so this is not a failure there. It is on `master` and is not changed here. - **When the terminal of a provider-started turn was removed from the stored events,** the transport builds no start for an event of that turn, and the session stops on that event, as before. This needs a later terminal of the same type with no start of its own (a turn that was lost while it started) and a full window. - **The last checkpoint of a wait that ends with no provider-started turn has no terminal,** as the section above says. The change is only in a run that waits, and no run waits before part 3. - **A wait with a consumed disposition marker has no test with the real daemon.** The state needs two restarts in one run, and the test helper does one. Unit tests of the driver and of the consumer cover it. - **A provider start that waits for `reconcile()`** waits as long as the reads of `reconcile()` take. Each read ends at the command deadline of the Runner (30 seconds by default) or when the transport closes. - **`item.delta` events across a restart are not asserted.** A streamed delta that the first controller received is not sent again. The completed items are asserted. - **The driver keeps a maximum of 8 snapshots at events** and drops each one when the consumer asks for it or for a later one. If the consumer is more than 8 such events behind, the checkpoint uses the current snapshot, as before this change. - **A second provider-started turn that also ends during the same outage** is not tested with the real daemon: the fake gives each terminal the same goal revision, and the Runner then holds the second terminal until a controller is attached. The end of the reconnect time of the Runner is not tested here. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, decided the designs, reviewed the diffs, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as Claude Code subagents with tool use (shell, file edits, local test runs), wrote the change and the tests, reproduced the review findings with the real daemon, and verified this slice alone. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
696 lines
40 KiB
TypeScript
696 lines
40 KiB
TypeScript
import { describe, expect, it, vi } from "vitest";
|
|
|
|
import type { NativeExecutionInputV1 } from "./contracts/native-execution.js";
|
|
import type { NativeSession, PersistedNativeSession } from "./contracts/native-session-backend.js";
|
|
import { executeNativeSession } from "./native-session-runtime.js";
|
|
import { providerTurnWaitDeadline, PROVIDER_TURN_WAIT_TIMEOUT_MARGIN_MS, type ProviderTurnWaitOptions } from "./native-provider-turn-wait.js";
|
|
import type { PrpEvent, PrpStructuredRunResult } from "./protocol/replay-contract.js";
|
|
|
|
const identity = {
|
|
runId: "run-provider-turn",
|
|
sessionId: "session-provider-turn",
|
|
companyId: "company-provider-turn",
|
|
issueId: "issue-provider-turn",
|
|
agentId: "agent-provider-turn",
|
|
};
|
|
|
|
const input: NativeExecutionInputV1 = {
|
|
schema: "paperclip.native-execution-input.v1",
|
|
binding: { companyId: identity.companyId, runId: identity.runId, issueId: identity.issueId, agentId: identity.agentId,
|
|
executionWorkspaceId: "workspace-provider-turn" },
|
|
task: { identifier: "PAP-PROVIDER", title: "Wait for a background install", description: null,
|
|
prompt: "# PAP-PROVIDER: Wait for a background install", workMode: "standard" },
|
|
workspace: { cwd: "/workspace", repoUrl: null, repoRef: null, branchName: null },
|
|
session: { normalizedSessionId: identity.sessionId, driverKind: "codex_app_server", protocolVersion: 1,
|
|
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
|
|
provider: { kind: "codex", model: null },
|
|
completionContract: { id: "contract-provider-turn", sha256: "contract-provider-turn-sha", schemaVersion: "paperclip.completion-contract.v1",
|
|
contract: { revision: "1", objective: "Install and test", criteria: [{ id: "objective", requirement: "Install and test" }] } },
|
|
interactionResponses: [],
|
|
credentialBindings: [],
|
|
};
|
|
|
|
const result: PrpStructuredRunResult = {
|
|
schema: "paperclip.run_result.v1",
|
|
reportedWorkDisposition: "done",
|
|
summary: "Installed and tested after the background install finished.",
|
|
completionClaim: { contractRevision: "1", objectiveSatisfied: true,
|
|
criteria: [{ criterionId: "objective", status: "satisfied", evidenceRefs: [] }], remainingWork: [] },
|
|
evidence: [],
|
|
verification: [{ commandOrCheck: "pnpm test", status: "passed" }],
|
|
attentionRequests: [],
|
|
artifacts: [],
|
|
};
|
|
|
|
const planReviewWait: PrpStructuredRunResult = {
|
|
schema: "paperclip.run_result.v1",
|
|
reportedWorkDisposition: "yielded",
|
|
summary: "Waiting for the plan review.",
|
|
completionClaim: { contractRevision: "1", objectiveSatisfied: false,
|
|
criteria: [{ criterionId: "objective", status: "unknown", evidenceRefs: ["interaction:plan"] }],
|
|
remainingWork: [{ description: "Resume after the plan review.", blocksCompletion: true }] },
|
|
evidence: [{ ref: "interaction:plan" }],
|
|
verification: [],
|
|
attentionRequests: [],
|
|
artifacts: [{ kind: "issue_thread_interaction", ref: "interaction:plan" }],
|
|
continuation: { kind: "response_wake", summary: "Resume from the plan review.", idempotencyKey: "interaction-response:plan" },
|
|
};
|
|
|
|
let sequence = 0;
|
|
const event = (turnId: string, eventType: PrpEvent["eventType"], payload: Record<string, unknown> = {}): PrpEvent => {
|
|
sequence += 1;
|
|
return {
|
|
schema: "paperclip.prp.event.v1",
|
|
sourceEventId: `runner-provider-turn:${identity.runId}:${sequence}`,
|
|
sourceSeq: sequence,
|
|
sourceInstanceId: "runner-provider-turn",
|
|
sourceKind: "runner",
|
|
runId: identity.runId,
|
|
normalizedSessionId: identity.sessionId,
|
|
turnId,
|
|
eventType,
|
|
schemaVersion: 1,
|
|
priority: 0,
|
|
emittedAt: "2026-11-12T00:00:00.000Z",
|
|
payload,
|
|
};
|
|
};
|
|
|
|
const usage = { kind: "usage", usage: { runDeltaComplete: true, runDelta: { inputTokens: 6, outputTokens: 4 } } };
|
|
const turnOne = (awaiting: boolean) => [
|
|
event("turn-1", "turn.started", { status: "inProgress" }),
|
|
event("turn-1", "item.completed", usage),
|
|
event("turn-1", "turn.completed", { status: "completed", error: null, ...(awaiting ? { awaitingProviderTurn: true } : {}) }),
|
|
];
|
|
const providerTurn = (turnId: string, options: { result?: boolean; awaiting?: boolean } = {}) => [
|
|
event(turnId, "turn.started", { status: "inProgress", origin: "provider" }),
|
|
event(turnId, "item.completed", usage),
|
|
...(options.result ? [event(turnId, "run.result.proposed", result as unknown as Record<string, unknown>)] : []),
|
|
event(turnId, "turn.completed", { status: "completed", error: null, ...(options.awaiting ? { awaitingProviderTurn: true } : {}) }),
|
|
];
|
|
|
|
const restartFailure = { code: "provider_turn_lost_on_restore", recoverable: true };
|
|
const lostTerminal = { turnId: "turn-recovery", fingerprint: JSON.stringify({ terminalState: "failed", error: restartFailure, result: null }) };
|
|
const completedTerminal = (turnId: string) => ({ turnId, fingerprint: JSON.stringify({ terminalState: "completed", error: null, result: null }) });
|
|
const restartCheckpoint: PersistedNativeSession = {
|
|
backendKind: "mock", driverKind: "codex_app_server", sessionId: identity.sessionId, identity,
|
|
providerSessionId: "provider-session", cursor: "0", activeTurnId: null,
|
|
terminalTurns: [lostTerminal], semanticResult: null, pendingRuntimeRequests: [], lineage: [],
|
|
};
|
|
|
|
interface RestartContinuation {
|
|
onDisposition?: (push: (...values: Array<PrpEvent | "end">) => void) => void;
|
|
recovery?: { checkpoint: PersistedNativeSession; snapshot: Partial<PersistedNativeSession>; replay?: PrpEvent[] };
|
|
}
|
|
|
|
function harness(options: { withoutInterrupt?: boolean; restart?: RestartContinuation } = {}) {
|
|
const queue: Array<PrpEvent | "end"> = [];
|
|
let wake: (() => void) | null = null;
|
|
let activeProviderTurn: string | null = null;
|
|
const observed = { activeTurnId: null as string | null, awaiting: false };
|
|
const snapshotSequences: Array<number | undefined> = [];
|
|
const push = (...values: Array<PrpEvent | "end">) => {
|
|
for (const value of values) {
|
|
if (value !== "end" && value.eventType === "turn.started") activeProviderTurn = value.turnId ?? null;
|
|
if (value !== "end" && value.eventType.startsWith("turn.") && value.eventType !== "turn.started") activeProviderTurn = null;
|
|
queue.push(value);
|
|
}
|
|
wake?.();
|
|
};
|
|
const interrupt = vi.fn(async (_input: { turnId?: string; reason?: string }) => {
|
|
if (activeProviderTurn) push(event(activeProviderTurn, "turn.cancelled", { status: "cancelled", error: null }));
|
|
else push(event("turn-1", "item.completed", { kind: "interrupt_acknowledgement", status: "provider_turn_wait_ended" }));
|
|
});
|
|
const capabilities = { resume: options.restart !== undefined, typedEvents: true, steering: false, interruption: true, structuredResult: true };
|
|
const startTurn = vi.fn(async (_input: Parameters<NativeSession["startTurn"]>[0]) => {
|
|
if (!options.restart) return { turnId: "turn-1" };
|
|
if (options.restart.recovery) return { turnId: "turn-fresh" };
|
|
if (startTurn.mock.calls.length === 1) {
|
|
push(event("turn-continuation", "turn.started", { status: "inProgress" }), event("turn-continuation", "item.completed", usage),
|
|
event("turn-continuation", "turn.completed", { status: "completed", error: null, awaitingProviderTurn: true }));
|
|
return { turnId: "turn-continuation", borrowedDispositionMarker: true as const };
|
|
}
|
|
options.restart.onDisposition?.(push);
|
|
return { turnId: "turn-disposition" };
|
|
});
|
|
const session: NativeSession = {
|
|
identity: () => identity,
|
|
async capabilities() { return capabilities; },
|
|
async *events() {
|
|
for (;;) {
|
|
const next = queue.shift();
|
|
if (next === "end") return;
|
|
if (next) {
|
|
if (next.eventType === "turn.started") Object.assign(observed, { activeTurnId: next.turnId ?? null, awaiting: false });
|
|
else if (next.eventType.startsWith("turn.")) Object.assign(observed, { activeTurnId: null, awaiting: next.payload.awaitingProviderTurn === true });
|
|
yield structuredClone(next);
|
|
continue;
|
|
}
|
|
await new Promise<void>(resolve => { wake = resolve; });
|
|
wake = null;
|
|
}
|
|
},
|
|
startTurn,
|
|
...(options.withoutInterrupt ? {} : { interrupt }),
|
|
async result() { return null; },
|
|
async snapshot(snapshotOptions?: { sourceSeq?: number }) {
|
|
snapshotSequences.push(snapshotOptions?.sourceSeq);
|
|
const base: PersistedNativeSession = { backendKind: "mock", sessionId: identity.sessionId, identity,
|
|
providerSessionId: "provider-session", cursor: String(sequence), activeTurnId: null, pendingRuntimeRequests: [], lineage: [] };
|
|
if (!options.restart) return { ...base, activeTurnId: observed.activeTurnId, ...(observed.awaiting ? { awaitingProviderTurn: true } : {}) };
|
|
if (options.restart.recovery) return { ...base, ...options.restart.recovery.snapshot };
|
|
const starts = startTurn.mock.calls.length;
|
|
return { ...base, semanticResult: null,
|
|
terminalTurns: [lostTerminal, ...(starts >= 1 ? [completedTerminal("turn-continuation")] : []),
|
|
...(starts >= 2 ? [completedTerminal("turn-disposition")] : [])],
|
|
...(starts >= 1 ? { dispositionOnlyRecoveryConsumed: starts >= 2, dispositionOnlyRecoveryTurnId: starts >= 2 ? "turn-disposition" : null } : {}) };
|
|
},
|
|
async close() { push("end"); },
|
|
};
|
|
return { session, push, interrupt, startTurn, snapshotSequences };
|
|
}
|
|
|
|
async function run(options: {
|
|
stream: PrpEvent[];
|
|
after?: (control: ReturnType<typeof harness>, appended: PrpEvent[]) => Promise<void> | void;
|
|
providerTurnWait?: ProviderTurnWaitOptions;
|
|
turnTimeoutMs?: number;
|
|
missingResult?: (input: { turnId: string | null; terminalEvent: PrpEvent }) => PrpStructuredRunResult | null;
|
|
withoutInterrupt?: boolean;
|
|
pipelinesDeltas?: boolean;
|
|
restart?: RestartContinuation;
|
|
}) {
|
|
const control = harness({ withoutInterrupt: options.withoutInterrupt, restart: options.restart });
|
|
const appended: PrpEvent[] = [];
|
|
const checkpoints: PersistedNativeSession[] = [];
|
|
const checkpointedAfter: number[] = [];
|
|
let queued: Array<() => void> = [];
|
|
const flushStreamingEvents = vi.fn(() => {
|
|
const flushed = queued;
|
|
queued = [];
|
|
for (const commit of flushed) commit();
|
|
});
|
|
let returned: Awaited<ReturnType<typeof executeNativeSession>> | null = null;
|
|
const completeRun = vi.fn(async () => undefined);
|
|
const resolveMissingResult = vi.fn(async (value: { turnId: string | null; terminalEvent: PrpEvent }) => options.missingResult?.(value) ?? null);
|
|
control.push(...options.stream);
|
|
const capabilities = { resume: options.restart !== undefined, typedEvents: true, steering: false, interruption: true, structuredResult: true };
|
|
const started = Date.now();
|
|
const execution = executeNativeSession({
|
|
input,
|
|
backend: { async descriptor() { return { kind: "mock", name: "provider-turn", version: "1", capabilities }; },
|
|
async openSession() { return control.session; },
|
|
...(options.restart ? { async recoverSession() { return { recovered: true, session: control.session }; } } : {}) },
|
|
controlPlane: {
|
|
...(options.pipelinesDeltas ? { pipelinesStreamingEvents: true, flushStreamingEvents } : {}),
|
|
async openRun() {}, async checkpointSession(value: PersistedNativeSession) { checkpoints.push(structuredClone(value)); checkpointedAfter.push(appended.length); },
|
|
async appendEvent(value: PrpEvent) {
|
|
appended.push(structuredClone(value));
|
|
if (options.pipelinesDeltas && value.eventType === "item.delta") await new Promise<void>(resolve => queued.push(resolve));
|
|
return { cursor: value.sourceSeq, highestContiguousSourceSeq: value.sourceSeq, disposition: "committed" };
|
|
},
|
|
async replayEvents(request: { sourceInstanceId: string; afterSourceSeq: number }) {
|
|
const events = (options.restart?.recovery?.replay ?? [])
|
|
.filter(value => value.sourceInstanceId === request.sourceInstanceId && value.sourceSeq > request.afterSourceSeq);
|
|
return { events, highestContiguousSourceSeq: Math.max(0, ...events.map(value => value.sourceSeq)) };
|
|
},
|
|
completeRun,
|
|
},
|
|
runnerInstanceId: "runner-provider-turn", controlPlaneInstanceId: "control-provider-turn", timeoutMs: 30_000,
|
|
...(options.turnTimeoutMs === undefined ? {} : { turnTimeoutMs: options.turnTimeoutMs }),
|
|
resolveMissingResult,
|
|
providerTurnWait: options.providerTurnWait,
|
|
...(options.restart ? { persistedSession: options.restart.recovery?.checkpoint ?? restartCheckpoint, resumeInterruptedTurn: true } : {}),
|
|
}).then((value) => { returned = value; return "finalized"; }, (error: unknown) => String(error instanceof Error ? error.message : error));
|
|
await options.after?.(control, appended);
|
|
const outcome = await execution;
|
|
return { outcome, returned, appended, completeRun, resolveMissingResult, interrupt: control.interrupt, flushStreamingEvents,
|
|
startTurn: control.startTurn, checkpoints, checkpointedAfter, snapshotSequences: control.snapshotSequences, elapsedMs: Date.now() - started };
|
|
}
|
|
|
|
const types = (events: PrpEvent[]) => events.map(value => value.eventType);
|
|
const waitFor = async (condition: () => boolean) => vi.waitFor(() => expect(condition()).toBe(true), { timeout: 5_000, interval: 5 });
|
|
|
|
describe("the controller consumer with provider-started turn sequences", () => {
|
|
it("finalizes a no-result turn exactly as before when the Runner did not await", async () => {
|
|
const outcome = await run({ stream: [...turnOne(false), ...providerTurn("provider-1", { result: true })] });
|
|
expect(outcome.outcome).toContain("native_finalization_missing");
|
|
expect(outcome.completeRun).not.toHaveBeenCalled();
|
|
expect(types(outcome.appended)).toEqual(["turn.started", "item.completed", "turn.completed"]);
|
|
expect(outcome.interrupt).not.toHaveBeenCalled();
|
|
});
|
|
|
|
const ordinaryTurn = () => [
|
|
event("turn-1", "turn.started", { status: "inProgress" }),
|
|
event("turn-1", "item.completed", usage),
|
|
event("turn-1", "run.result.proposed", result as unknown as Record<string, unknown>),
|
|
event("turn-1", "turn.completed", { status: "completed", error: null }),
|
|
];
|
|
const checkpointsAfter = (outcome: Awaited<ReturnType<typeof run>>, matches: (value: PrpEvent) => boolean) => {
|
|
const index = outcome.appended.findIndex(matches);
|
|
return outcome.checkpoints.filter((_value, at) => outcome.checkpointedAfter[at] === index + 1);
|
|
};
|
|
|
|
it("persists a checkpoint with the waiting shape right after the awaiting terminal", async () => {
|
|
const outcome = await run({ stream: turnOne(true), providerTurnWait: { capMs: 100 } });
|
|
expect(checkpointsAfter(outcome, value => value.payload.awaitingProviderTurn === true))
|
|
.toEqual([expect.objectContaining({ activeTurnId: null, awaitingProviderTurn: true })]);
|
|
});
|
|
|
|
it("persists a checkpoint naming a provider-started turn active right after its start", async () => {
|
|
const outcome = await run({ stream: [...turnOne(true), ...providerTurn("provider-1", { awaiting: true }), ...providerTurn("provider-2", { result: true })] });
|
|
expect(outcome.outcome).toBe("finalized");
|
|
for (const turnId of ["provider-1", "provider-2"]) {
|
|
expect(checkpointsAfter(outcome, value => value.turnId === turnId && value.eventType === "turn.started"))
|
|
.toEqual([expect.objectContaining({ activeTurnId: turnId })]);
|
|
}
|
|
});
|
|
|
|
it("asks for the snapshot as of the awaiting terminal and the provider start it has just appended", async () => {
|
|
const outcome = await run({ stream: [...turnOne(true), ...providerTurn("provider-1", { result: true })] });
|
|
expect(outcome.outcome).toBe("finalized");
|
|
const awaited = outcome.appended.find(value => value.payload.awaitingProviderTurn === true)!;
|
|
const started = outcome.appended.find(value => value.payload.origin === "provider")!;
|
|
expect(outcome.snapshotSequences.filter(value => value !== undefined)).toEqual([awaited.sourceSeq, started.sourceSeq]);
|
|
});
|
|
|
|
it("persists no checkpoint of either kind for a turn that does not await", async () => {
|
|
const outcome = await run({ stream: ordinaryTurn() });
|
|
expect(outcome.outcome).toBe("finalized");
|
|
const checkpointsWithoutProviderTurns = 4;
|
|
expect(outcome.checkpoints).toHaveLength(checkpointsWithoutProviderTurns);
|
|
expect(outcome.checkpoints.filter(value => value.awaitingProviderTurn || value.activeTurnId?.startsWith("provider-"))).toEqual([]);
|
|
});
|
|
|
|
it("consumes the provider-started turn as the next turn of the same run and settles its result normally", async () => {
|
|
const outcome = await run({ stream: [...turnOne(true), ...providerTurn("provider-1", { result: true })] });
|
|
expect(outcome.outcome).toBe("finalized");
|
|
expect(outcome.completeRun).toHaveBeenCalledOnce();
|
|
expect(outcome.resolveMissingResult).not.toHaveBeenCalled();
|
|
expect(outcome.appended.filter(value => value.payload.origin === "provider").map(value => value.turnId)).toEqual(["provider-1"]);
|
|
expect(outcome.appended.filter(value => value.payload.kind === "usage").map(value => value.turnId)).toEqual(["turn-1", "provider-1"]);
|
|
expect(outcome.interrupt).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("keeps waiting when a provider-started turn also ends awaiting, then settles the later turn's result", async () => {
|
|
const outcome = await run({ stream: [...turnOne(true), ...providerTurn("provider-1", { awaiting: true }),
|
|
...providerTurn("provider-2", { result: true })] });
|
|
expect(outcome.outcome).toBe("finalized");
|
|
expect(outcome.appended.filter(value => value.payload.origin === "provider").map(value => value.turnId)).toEqual(["provider-1", "provider-2"]);
|
|
});
|
|
|
|
it("ends an idle wait at the cap by stopping the wait, then takes today's missing-result path with the awaited terminal", async () => {
|
|
const outcome = await run({ stream: turnOne(true), providerTurnWait: { capMs: 100 } });
|
|
expect(outcome.outcome).toContain("native_finalization_missing");
|
|
expect(outcome.interrupt).toHaveBeenCalledOnce();
|
|
expect(outcome.interrupt.mock.calls[0]![0]).toMatchObject({ turnId: "turn-1" });
|
|
expect(outcome.resolveMissingResult).toHaveBeenCalledWith(expect.objectContaining({
|
|
turnId: "turn-1", terminalEvent: expect.objectContaining({ eventType: "turn.completed", payload: expect.objectContaining({ awaitingProviderTurn: true }) }) }));
|
|
expect(outcome.elapsedMs).toBeGreaterThanOrEqual(90);
|
|
});
|
|
|
|
it("does not restart the cap when a later provider-started turn also ends without a result", async () => {
|
|
const outcome = await run({
|
|
stream: turnOne(true), providerTurnWait: { capMs: 300 },
|
|
after: async (control) => {
|
|
await new Promise(resolve => setTimeout(resolve, 200));
|
|
control.push(...providerTurn("provider-1", { awaiting: true }));
|
|
},
|
|
});
|
|
expect(outcome.outcome).toContain("native_finalization_missing");
|
|
expect(outcome.elapsedMs).toBeLessThan(450);
|
|
expect(outcome.resolveMissingResult).toHaveBeenCalledWith(expect.objectContaining({ turnId: "provider-1" }));
|
|
});
|
|
|
|
it.each(["cap", "comment"] as const)("lets a running provider-started turn finish after the %s and settles its result normally", async (trigger) => {
|
|
let release!: () => void;
|
|
const comment = new Promise<void>(resolve => { release = resolve; });
|
|
const onWait = vi.fn(() => comment);
|
|
const outcome = await run({
|
|
stream: turnOne(true), providerTurnWait: trigger === "cap" ? { capMs: 150 } : { capMs: 60_000, onWait },
|
|
after: async (control) => {
|
|
control.push(event("provider-1", "turn.started", { status: "inProgress", origin: "provider" }));
|
|
if (trigger === "comment") { await waitFor(() => onWait.mock.calls.length === 1); release(); }
|
|
await new Promise(resolve => setTimeout(resolve, 300));
|
|
control.push(event("provider-1", "item.completed", usage), event("provider-1", "run.result.proposed", result as unknown as Record<string, unknown>),
|
|
event("provider-1", "turn.completed", { status: "completed", error: null }));
|
|
},
|
|
});
|
|
expect(outcome.outcome).toBe("finalized");
|
|
expect(outcome.interrupt).not.toHaveBeenCalled();
|
|
expect(outcome.completeRun).toHaveBeenCalledOnce();
|
|
expect(outcome.appended.some(value => value.eventType === "turn.cancelled")).toBe(false);
|
|
});
|
|
|
|
it.each(["cap", "comment"] as const)("takes today's path at once when a provider-started turn running past the %s ends awaiting again", async (trigger) => {
|
|
let release!: () => void;
|
|
const comment = new Promise<void>(resolve => { release = resolve; });
|
|
const onWait = vi.fn(() => comment);
|
|
let endedAt = 0;
|
|
const outcome = await run({
|
|
stream: turnOne(true), providerTurnWait: trigger === "cap" ? { capMs: 150 } : { capMs: 60_000, onWait },
|
|
after: async (control) => {
|
|
control.push(event("provider-1", "turn.started", { status: "inProgress", origin: "provider" }));
|
|
if (trigger === "comment") { await waitFor(() => onWait.mock.calls.length === 1); release(); }
|
|
await new Promise(resolve => setTimeout(resolve, 300));
|
|
endedAt = Date.now();
|
|
control.push(event("provider-1", "item.completed", usage),
|
|
event("provider-1", "turn.completed", { status: "completed", error: null, awaitingProviderTurn: true }));
|
|
},
|
|
});
|
|
expect(outcome.outcome).toContain("native_finalization_missing");
|
|
expect(outcome.interrupt).not.toHaveBeenCalled();
|
|
expect(outcome.resolveMissingResult).toHaveBeenCalledWith(expect.objectContaining({ turnId: "provider-1" }));
|
|
expect(Date.now() - endedAt).toBeLessThan(1_000);
|
|
});
|
|
|
|
it("settles a batched delta that arrived during the idle wait before the limit ends the turn", async () => {
|
|
let delta!: PrpEvent;
|
|
const outcome = await run({
|
|
stream: turnOne(true), providerTurnWait: { capMs: 150 }, withoutInterrupt: true, pipelinesDeltas: true,
|
|
missingResult: () => result,
|
|
after: async (control, appended) => {
|
|
await waitFor(() => appended.some(value => value.payload.awaitingProviderTurn === true));
|
|
delta = { ...event("turn-1", "item.delta", { kind: "agentMessage", text: "late chunk" }), itemId: "message-1", priority: 1 };
|
|
control.push(delta);
|
|
},
|
|
});
|
|
expect(outcome.outcome).toBe("finalized");
|
|
expect(outcome.interrupt).not.toHaveBeenCalled();
|
|
expect(outcome.flushStreamingEvents).toHaveBeenCalled();
|
|
expect(outcome.appended.filter(value => value.eventType === "item.delta")).toHaveLength(1);
|
|
expect(outcome.returned).toMatchObject({ nativeEventCount: outcome.appended.length, highestContiguousSourceSeq: delta.sourceSeq });
|
|
});
|
|
|
|
it("never waits longer than the run's remaining timeout", async () => {
|
|
const turnTimeoutMs = PROVIDER_TURN_WAIT_TIMEOUT_MARGIN_MS + 200;
|
|
const outcome = await run({ stream: turnOne(true), turnTimeoutMs });
|
|
expect(outcome.outcome).toContain("native_finalization_missing");
|
|
expect(outcome.outcome).not.toContain("timed out");
|
|
expect(outcome.elapsedMs).toBeLessThan(2_000);
|
|
expect(providerTurnWaitDeadline({ startedAtMs: 1_000, capMs: 600_000, overallDeadlineMs: null })).toBe(601_000);
|
|
expect(providerTurnWaitDeadline({ startedAtMs: 1_000, capMs: 600_000, overallDeadlineMs: 60_000 }))
|
|
.toBe(60_000 - PROVIDER_TURN_WAIT_TIMEOUT_MARGIN_MS);
|
|
});
|
|
|
|
it("resumes the cap from the wait start a previous controller recorded", async () => {
|
|
const outcome = await run({ stream: turnOne(true), providerTurnWait: { startedAtMs: Date.now() - 599_900 } });
|
|
expect(outcome.outcome).toContain("native_finalization_missing");
|
|
expect(outcome.elapsedMs).toBeLessThan(2_000);
|
|
});
|
|
|
|
it("ends the wait early when the control plane reports a new comment", async () => {
|
|
let release!: () => void;
|
|
const comment = new Promise<void>(resolve => { release = resolve; });
|
|
const onWait = vi.fn(() => comment);
|
|
const outcome = await run({
|
|
stream: turnOne(true), providerTurnWait: { capMs: 60_000, onWait },
|
|
after: async () => { await waitFor(() => onWait.mock.calls.length === 1); release(); },
|
|
});
|
|
expect(outcome.outcome).toContain("native_finalization_missing");
|
|
expect(outcome.interrupt).toHaveBeenCalledOnce();
|
|
expect(outcome.elapsedMs).toBeLessThan(5_000);
|
|
});
|
|
|
|
it("ends the wait at once when Stop interrupts the session", async () => {
|
|
const outcome = await run({
|
|
stream: turnOne(true), providerTurnWait: { capMs: 60_000 },
|
|
after: async (control, appended) => {
|
|
await waitFor(() => appended.some(value => value.payload.awaitingProviderTurn === true));
|
|
await control.session.interrupt!({ reason: "Stop" });
|
|
},
|
|
});
|
|
expect(outcome.outcome).toContain("native_finalization_missing");
|
|
expect(outcome.interrupt).toHaveBeenCalledOnce();
|
|
expect(outcome.elapsedMs).toBeLessThan(5_000);
|
|
});
|
|
|
|
it("tolerates the Runner's held run.terminal during the wait and after Core ends it", async () => {
|
|
const heldTerminal = () => event("turn-1", "run.terminal", { schema: "paperclip.prp.terminal.v1", status: "succeeded",
|
|
turnTerminalState: "completed", runTerminalState: "succeeded", reportedWorkDisposition: "done", provider: "acpx" });
|
|
const outcome = await run({
|
|
stream: [...turnOne(true), heldTerminal()], providerTurnWait: { capMs: 150 },
|
|
after: async (control) => {
|
|
await waitFor(() => control.interrupt.mock.calls.length === 1);
|
|
control.push(heldTerminal());
|
|
},
|
|
});
|
|
expect(outcome.outcome).toContain("native_finalization_missing");
|
|
expect(outcome.appended.filter(value => value.eventType === "run.terminal")).toHaveLength(1);
|
|
expect(outcome.resolveMissingResult).toHaveBeenCalledOnce();
|
|
});
|
|
|
|
it.each([false, true])("settles a plan proposed in the provider-started turn as the plan review wait (cap passed during the turn: %s)", async (capPassed) => {
|
|
let seen: PrpEvent[] = [];
|
|
const outcome = await run({
|
|
stream: turnOne(true), providerTurnWait: { capMs: capPassed ? 100 : 60_000 },
|
|
missingResult: ({ terminalEvent }) => seen.some(value => value.eventType === "plan.proposed")
|
|
&& terminalEvent.eventType === "turn.completed" ? planReviewWait : null,
|
|
after: async (control, appended) => {
|
|
seen = appended;
|
|
control.push(event("provider-1", "turn.started", { status: "inProgress", origin: "provider" }));
|
|
await waitFor(() => appended.some(value => value.payload.origin === "provider"));
|
|
if (capPassed) await new Promise(resolve => setTimeout(resolve, 250));
|
|
control.push(
|
|
event("provider-1", "plan.proposed", { schema: "paperclip.plan.proposed.v1", proposalId: "toolu_plan", status: "completed",
|
|
markdown: "# Plan\n\n1. Install.\n2. Test.", source: "claude_exit_plan_mode" }),
|
|
event("provider-1", "item.completed", usage),
|
|
event("provider-1", "turn.completed", { status: "completed", error: null }),
|
|
event("provider-1", "run.terminal", { schema: "paperclip.prp.terminal.v1", status: "succeeded", turnTerminalState: "completed",
|
|
runTerminalState: "succeeded", reportedWorkDisposition: "done", provider: "acpx" }),
|
|
);
|
|
},
|
|
});
|
|
expect(outcome.outcome).toBe("finalized");
|
|
expect(outcome.interrupt).not.toHaveBeenCalled();
|
|
expect(outcome.appended.filter(value => value.eventType === "plan.proposed")).toEqual([expect.objectContaining({ turnId: "provider-1", payload: expect.objectContaining({ status: "completed" }) })]);
|
|
expect(outcome.resolveMissingResult).toHaveBeenCalledOnce();
|
|
expect(outcome.resolveMissingResult).toHaveBeenCalledWith(expect.objectContaining({
|
|
turnId: "provider-1", terminalEvent: expect.objectContaining({ eventType: "turn.completed", turnId: "provider-1" }) }));
|
|
expect(outcome.resolveMissingResult.mock.calls[0]![0].terminalEvent.payload).not.toHaveProperty("awaitingProviderTurn");
|
|
expect((outcome.completeRun.mock.calls[0] as unknown[])[0]).toMatchObject({ result: planReviewWait, turnId: "provider-1",
|
|
terminal: { runTerminalState: "succeeded", reportedWorkDisposition: "yielded" } });
|
|
});
|
|
|
|
const dispositionRequests = (startTurn: ReturnType<typeof harness>["startTurn"]) =>
|
|
startTurn.mock.calls.filter(([value]) => value.message.text.includes("Paperclip semantic-result recovery")).length;
|
|
const dispositionTurn = () => [
|
|
event("turn-disposition", "turn.started", { status: "inProgress" }),
|
|
event("turn-disposition", "run.result.proposed", result as unknown as Record<string, unknown>),
|
|
event("turn-disposition", "turn.completed", { status: "completed", error: null }),
|
|
];
|
|
|
|
const ids = (events: PrpEvent[]) => events.map(value => value.sourceEventId);
|
|
const appendedOf = (appended: PrpEvent[], events: PrpEvent[]) =>
|
|
ids(appended).filter(id => ids(events).includes(id));
|
|
|
|
it("keeps reading the follow-up turn that a forced end of an idle wait starts and settles its result", async () => {
|
|
const turn = dispositionTurn();
|
|
const outcome = await run({
|
|
stream: [], providerTurnWait: { capMs: 100 }, withoutInterrupt: true,
|
|
restart: { onDisposition: push => push(...turn) },
|
|
});
|
|
expect(outcome.outcome).toBe("finalized");
|
|
expect(outcome.returned).toMatchObject({ result, turnId: "turn-disposition" });
|
|
expect(outcome.resolveMissingResult).not.toHaveBeenCalled();
|
|
expect(outcome.interrupt).not.toHaveBeenCalled();
|
|
expect(outcome.startTurn).toHaveBeenCalledTimes(2);
|
|
expect(dispositionRequests(outcome.startTurn)).toBe(1);
|
|
expect(appendedOf(outcome.appended, turn)).toEqual(ids(turn));
|
|
});
|
|
|
|
it("appends exactly once an event that reaches the outstanding read after the forced end was taken", async () => {
|
|
const turn = dispositionTurn();
|
|
const outcome = await run({
|
|
stream: [], providerTurnWait: { capMs: 100 }, withoutInterrupt: true, restart: {},
|
|
after: async (control) => {
|
|
await waitFor(() => control.startTurn.mock.calls.length === 2);
|
|
const [first, ...rest] = turn;
|
|
await new Promise(resolve => setTimeout(resolve, 50));
|
|
control.push(first!);
|
|
await new Promise(resolve => setTimeout(resolve, 50));
|
|
control.push(...rest);
|
|
},
|
|
});
|
|
expect(outcome.outcome).toBe("finalized");
|
|
expect(outcome.appended.filter(value => value.sourceEventId === turn[0]!.sourceEventId)).toHaveLength(1);
|
|
expect(appendedOf(outcome.appended, turn)).toEqual(ids(turn));
|
|
expect(dispositionRequests(outcome.startTurn)).toBe(1);
|
|
});
|
|
|
|
it.each([true, false])("keeps waiting after a controller restart while the checkpoint still names the awaited turn as active (provider turn arrives: %s)", async (arrives) => {
|
|
const recovery = {
|
|
checkpoint: { ...restartCheckpoint, activeTurnId: "turn-1", terminalTurns: [] },
|
|
snapshot: { terminalTurns: [completedTerminal("turn-1")], semanticResult: null, awaitingProviderTurn: true },
|
|
};
|
|
const stream = [event("turn-1", "turn.completed", { status: "completed", error: null, awaitingProviderTurn: true })];
|
|
const turn = providerTurn("provider-1", { result: true });
|
|
const outcome = await run({
|
|
stream, restart: { recovery }, turnTimeoutMs: 10_000,
|
|
providerTurnWait: { startedAtMs: Date.now() - (arrives ? 599_000 : 599_700) },
|
|
after: async (control) => {
|
|
if (!arrives) return;
|
|
await new Promise(resolve => setTimeout(resolve, 200));
|
|
control.push(...turn);
|
|
},
|
|
});
|
|
expect(outcome.startTurn).not.toHaveBeenCalled();
|
|
if (arrives) {
|
|
expect(outcome.outcome).toBe("finalized");
|
|
expect(outcome.returned).toMatchObject({ result, turnId: "provider-1" });
|
|
expect(outcome.interrupt).not.toHaveBeenCalled();
|
|
expect(appendedOf(outcome.appended, turn)).toEqual(ids(turn));
|
|
} else {
|
|
expect(outcome.outcome).toContain("native_finalization_missing");
|
|
expect(outcome.interrupt).toHaveBeenCalledOnce();
|
|
expect(outcome.resolveMissingResult).toHaveBeenCalledWith(expect.objectContaining({ turnId: "turn-1",
|
|
terminalEvent: expect.objectContaining({ eventType: "turn.completed", payload: expect.objectContaining({ awaitingProviderTurn: true }) }) }));
|
|
expect(outcome.elapsedMs).toBeLessThan(5_000);
|
|
}
|
|
});
|
|
|
|
const awaitingSnapshot = { terminalTurns: [completedTerminal("turn-1")], semanticResult: null, awaitingProviderTurn: true };
|
|
const afterTerminalRecovery = (options: { flag?: boolean; replay?: PrpEvent[] } = {}) => ({
|
|
checkpoint: { ...restartCheckpoint, terminalTurns: [completedTerminal("turn-1")], ...(options.flag === false ? {} : { awaitingProviderTurn: true }) },
|
|
snapshot: options.flag === false ? { terminalTurns: [completedTerminal("turn-1")], semanticResult: null } : awaitingSnapshot,
|
|
replay: options.replay ?? turnOne(true),
|
|
});
|
|
|
|
it.each([true, false])("keeps waiting after a controller restart saved after the awaiting terminal (provider turn arrives: %s)", async (arrives) => {
|
|
const turn = providerTurn("provider-1", { result: true });
|
|
const recovery = afterTerminalRecovery();
|
|
const outcome = await run({
|
|
stream: [], restart: { recovery }, turnTimeoutMs: 10_000,
|
|
providerTurnWait: { startedAtMs: Date.now() - (arrives ? 599_000 : 599_700) },
|
|
after: async (control) => {
|
|
if (!arrives) return;
|
|
await new Promise(resolve => setTimeout(resolve, 200));
|
|
control.push(...turn);
|
|
},
|
|
});
|
|
expect(outcome.startTurn).not.toHaveBeenCalled();
|
|
expect(appendedOf(outcome.appended, recovery.replay)).toEqual([]);
|
|
if (arrives) {
|
|
expect(outcome.outcome).toBe("finalized");
|
|
expect(outcome.returned).toMatchObject({ result, turnId: "provider-1" });
|
|
expect(outcome.interrupt).not.toHaveBeenCalled();
|
|
expect(appendedOf(outcome.appended, turn)).toEqual(ids(turn));
|
|
} else {
|
|
expect(outcome.outcome).toContain("native_finalization_missing");
|
|
expect(outcome.interrupt).toHaveBeenCalledOnce();
|
|
expect(outcome.resolveMissingResult).toHaveBeenCalledWith(expect.objectContaining({ turnId: "turn-1",
|
|
terminalEvent: expect.objectContaining({ eventType: "turn.completed", payload: expect.objectContaining({ awaitingProviderTurn: true }) }) }));
|
|
expect(outcome.elapsedMs).toBeLessThan(5_000);
|
|
}
|
|
});
|
|
|
|
it.each([false, true])("ends a resumed wait at once through the no-result path when its limit passed before the restart (without interrupt: %s)", async (withoutInterrupt) => {
|
|
const outcome = await run({
|
|
stream: [], restart: { recovery: afterTerminalRecovery() }, turnTimeoutMs: 10_000, withoutInterrupt,
|
|
providerTurnWait: { startedAtMs: Date.now() - 700_000 },
|
|
});
|
|
expect(outcome.startTurn).not.toHaveBeenCalled();
|
|
expect(outcome.outcome).toContain("native_finalization_missing");
|
|
expect(outcome.interrupt).toHaveBeenCalledTimes(withoutInterrupt ? 0 : 1);
|
|
expect(outcome.resolveMissingResult).toHaveBeenCalledOnce();
|
|
expect(outcome.resolveMissingResult).toHaveBeenCalledWith(expect.objectContaining({ turnId: "turn-1",
|
|
terminalEvent: expect.objectContaining({ payload: expect.objectContaining({ awaitingProviderTurn: true }) }) }));
|
|
expect(outcome.elapsedMs).toBeLessThan(1_000);
|
|
});
|
|
|
|
it.each([
|
|
{ label: "the replayed terminal has no marker", recovery: afterTerminalRecovery({ replay: turnOne(false) }) },
|
|
{ label: "replay has no terminal", recovery: afterTerminalRecovery({ replay: [] }) },
|
|
{ label: "the recovered session has no awaiting flag", recovery: afterTerminalRecovery({ flag: false }) },
|
|
])("sends the disposition-only recovery prompt after a restart as before when $label", async ({ recovery }) => {
|
|
const fresh = [
|
|
event("turn-fresh", "turn.started", { status: "inProgress" }),
|
|
event("turn-fresh", "run.result.proposed", result as unknown as Record<string, unknown>),
|
|
event("turn-fresh", "turn.completed", { status: "completed", error: null }),
|
|
];
|
|
const outcome = await run({
|
|
stream: [], restart: { recovery }, turnTimeoutMs: 10_000, providerTurnWait: { startedAtMs: Date.now() - 599_000 },
|
|
after: async (control) => {
|
|
await waitFor(() => control.startTurn.mock.calls.length === 1);
|
|
control.push(...fresh);
|
|
},
|
|
});
|
|
expect(outcome.startTurn).toHaveBeenCalledOnce();
|
|
expect(dispositionRequests(outcome.startTurn)).toBe(1);
|
|
expect(outcome.outcome).toBe("finalized");
|
|
expect(outcome.returned).toMatchObject({ result, turnId: "turn-fresh" });
|
|
expect(outcome.interrupt).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("resumes the wait again after a second restart during a resumed wait", async () => {
|
|
const startedAtMs = Date.now() - 599_000;
|
|
const first = await run({
|
|
stream: [], restart: { recovery: afterTerminalRecovery() }, turnTimeoutMs: 10_000, providerTurnWait: { startedAtMs },
|
|
});
|
|
expect(first.startTurn).not.toHaveBeenCalled();
|
|
expect(first.checkpoints.length).toBeGreaterThan(0);
|
|
const persisted = first.checkpoints.at(-1)!;
|
|
expect(persisted).toMatchObject({ activeTurnId: null, awaitingProviderTurn: true });
|
|
expect(persisted.dispositionOnlyRecoveryConsumed).toBeFalsy();
|
|
const turn = providerTurn("provider-1", { result: true });
|
|
const second = await run({
|
|
stream: [], restart: { recovery: { ...afterTerminalRecovery(), checkpoint: persisted } }, turnTimeoutMs: 10_000,
|
|
providerTurnWait: { startedAtMs: Date.now() - 599_000 },
|
|
after: async (control) => {
|
|
await new Promise(resolve => setTimeout(resolve, 200));
|
|
control.push(...turn);
|
|
},
|
|
});
|
|
expect(second.startTurn).not.toHaveBeenCalled();
|
|
expect(second.outcome).toBe("finalized");
|
|
expect(second.returned).toMatchObject({ result, turnId: "provider-1" });
|
|
});
|
|
|
|
it.each([true, false])("resumes a restored wait whose disposition marker is consumed instead of settling on that turn's earlier terminal (provider turn arrives: %s)", async (arrives) => {
|
|
const awaitedTurn = [
|
|
event("turn-disposition", "turn.started", { status: "inProgress" }),
|
|
event("turn-disposition", "item.completed", usage),
|
|
event("turn-disposition", "turn.completed", { status: "completed", error: null, awaitingProviderTurn: true }),
|
|
];
|
|
const terminalTurns = [completedTerminal("turn-1"), completedTerminal("turn-disposition")];
|
|
const waiting = { terminalTurns, semanticResult: null, awaitingProviderTurn: true, dispositionOnlyRecoveryConsumed: true, dispositionOnlyRecoveryTurnId: "turn-disposition" };
|
|
const recovery = { checkpoint: { ...restartCheckpoint, ...waiting }, snapshot: waiting, replay: [...turnOne(false), ...awaitedTurn] };
|
|
const turn = providerTurn("provider-1", { result: true });
|
|
const outcome = await run({
|
|
stream: [], restart: { recovery }, turnTimeoutMs: 10_000,
|
|
providerTurnWait: { startedAtMs: Date.now() - (arrives ? 599_000 : 599_700) },
|
|
after: async (control) => {
|
|
if (!arrives) return;
|
|
await new Promise(resolve => setTimeout(resolve, 200));
|
|
control.push(...turn);
|
|
},
|
|
});
|
|
expect(outcome.startTurn).not.toHaveBeenCalled();
|
|
if (arrives) {
|
|
expect(outcome.outcome).toBe("finalized");
|
|
expect(outcome.returned).toMatchObject({ result, turnId: "provider-1" });
|
|
expect(outcome.resolveMissingResult).not.toHaveBeenCalled();
|
|
expect(outcome.interrupt).not.toHaveBeenCalled();
|
|
expect(appendedOf(outcome.appended, turn)).toEqual(ids(turn));
|
|
} else {
|
|
expect(outcome.outcome).toContain("native_finalization_missing");
|
|
expect(outcome.interrupt).toHaveBeenCalledOnce();
|
|
expect(outcome.resolveMissingResult).toHaveBeenCalledOnce();
|
|
expect(outcome.resolveMissingResult).toHaveBeenCalledWith(expect.objectContaining({ turnId: "turn-disposition",
|
|
terminalEvent: expect.objectContaining({ eventType: "turn.completed", payload: expect.objectContaining({ awaitingProviderTurn: true }) }) }));
|
|
}
|
|
});
|
|
|
|
it("fails the run as today when the Runner's stream ends during the wait", async () => {
|
|
const outcome = await run({
|
|
stream: turnOne(true), providerTurnWait: { capMs: 60_000 },
|
|
after: async (control, appended) => {
|
|
await waitFor(() => appended.some(value => value.payload.awaitingProviderTurn === true));
|
|
control.push("end");
|
|
},
|
|
});
|
|
expect(outcome.outcome).toContain("native event stream closed before a turn terminal fact");
|
|
expect(outcome.elapsedMs).toBeLessThan(5_000);
|
|
});
|
|
});
|