diff --git a/doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md b/doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md new file mode 100644 index 00000000000..a7828ba4e0f --- /dev/null +++ b/doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md @@ -0,0 +1,56 @@ +# Durable records for cut-off turn starts and orphaned tool calls + +Status: approved by Dotta (2026-11-10), being implemented in two PRs +Goal: after a control-plane restart or freeze, 0% of tasks need Retry, and Retry still works whenever it is shown. +Parent plan: `2026-11-07-durable-ownership-and-reconciliation.md`, section B (write-ahead intents) and section F (charge only on recorded outcomes). + +## Rule + +Recovery may resend, replay or continue only when a durable record proves that no provider work and no tool effect can be duplicated or lost. A duplicated prompt or a second write is worse than a Retry click. When the record cannot prove it, the existing behaviour stays: a working Retry. + +## Change 1: a turn start cut off before the prompt reached the provider + +**What happens today.** Restart-chaos wave `w-20261110T185315-s9107`, task t05 (claude, workload C1, fault TM2). A follow-up run's `turn.start` reaches the Runner, which saves the ACPX session as `turn_starting` and asks the ACPX sidecar to start the turn. For a resumed Claude session the sidecar first starts Claude and loads the conversation; the prompt is sent only after that. Under load this took more than 24 s. Then the harness froze the server, the Runner, the sidecar and Claude together (SIGSTOP) for 33 s. On thaw the Runner's 30 s sidecar request deadline had passed, so it reaped the sidecar and saved the session as `closed`. Every same-run resume then failed with `ACPX session is not open`, recovery stopped with `native_session_retry_exhausted`, and a person had to click Retry. + +The Runner cannot tell from its own state whether Claude received the prompt. Resending could duplicate it; a continuation could lose it. + +**The record: a sidecar prompt-dispatch journal.** The ACPX sidecar writes one small file in the session's private runtime directory (`0700`) for each turn: + +1. When the runtime port starts the turn, before any provider work: one line `{"schema":…,"turnId":X,"phase":"received"}`, truncating the previous turn's file. No fsync. +2. When ACPX is about to write that turn's first `session/prompt` request to the provider's stdin: one appended line `{"phase":"dispatched"}`, then `fsync`, and only then the write. ACPX calls the host's `onAcpMessage("outbound", …)` hook synchronously before it hands the bytes to the provider stream, so a failed journal write aborts the prompt write. + +The file name and record shape are declared in `protocol/provider-schemas/acpx-sidecar.schema.json` and generated into TypeScript and Rust, so the pack contract digest covers them. + +**What it proves.** The Runner reads the journal only after it has proved that the sidecar and the provider are gone (it terminated the sidecar and acquired the provider lifetime fence, the same proof the existing lost-turn restore uses). A dead sidecar cannot write or send anything more. + +| Journal for the active turn | Meaning | Recovery | +|---|---|---| +| exactly one complete `received` line | the prompt was never written to the provider | restore the session with no active turn; emit `harness.diagnostic` `acpx_turn_start_not_dispatched`; the same-run resume sends the run's `turn.start` once | +| a `dispatched` line | the provider may have the prompt | unchanged: the existing lost-turn path (restore and continue in the same conversation, or close) | +| missing, another turn, malformed, torn | unknown | unchanged | + +Both cut-off shapes use it: a sidecar request that failed in flight (timeout or sidecar exit), and a Runner killed while the session was `turn_starting`. + +**Resend exactly once.** The cut-off `turn.start` command stays failed; it is never executed again. The restored session has no active turn, so the next same-run resume starts the turn with the run's own input. If that start is cut off too, the journal decides again. + +## Change 2: a tool call orphaned by a restart + +**What happens today.** Daytona wave `w-20261110T174930-s9205`, task t01 (codex, C1, RR2): a restart cut a `write_document` callback after the Runner journaled its input and before a result was saved. Close could not prove the provider drained, so the session quarantined (`native_session_cleanup_quarantined`). Since #20628 a callback admitted by an earlier controller settles as `tool_call_interrupted_by_restart` (unknown outcome, never executed again); a callback the current controller admitted still stays pending. + +**The record already exists.** The intent is the Runner's durable semantic input (committed before dispatch, keyed by call ID). The outcome is the server's idempotency receipt in `heartbeat_runs.result_json.semanticToolReceipts`. For `write_document` the revision, the activity row and the receipt commit in one transaction under the run and issue row locks, and the revision is compare-and-swap on `baseRevisionId`. + +**What it proves.** Receipt present: applied. Receipt absent: not applied, because the effect cannot commit without it. So recovery runs the same receipt-guarded mutation. It replays the saved result or applies the write for the first time, never twice. A late original callback then replays the receipt. A stale base, a reused key or invalid input is the call's definite outcome and settles as a tool error. Any other failure leaves the call unsettled, as before. The set of operations with this proof is explicit (`TRANSACTIONAL_RECEIPT_OPERATIONS`, `write_document` only); other tools keep their current rules. + +This makes the gateway call idempotent on its durable input, which is the cleaner proof the brief allowed. It needs no Runner change and no new image. + +**Not done.** Exposing the cleanup-gate owner's run id so the server can drop the 50-run scan bound from #20639 (optional in the brief) is left out to keep scope. + +## Upgrade safety + +- Old saved state, new code. A `turn_starting` session saved by an older Runner has no journal (or one for another turn): unknown, so the old lost-turn path runs unchanged. Old receipts already have the shape the transactional recovery reads. +- New state, old code mid-deploy. Change 1 adds no field to `acpx-provider-state.json` (its decoder denies unknown fields). It only saves lifecycles an old Runner already handles (`prepared`, then `suspended`/`session_open`). An old Runner ignores the journal file. Change 2 changes no saved shape. +- New Runner, old sidecar. Possible only with a stale pack; the contract digest prevents reuse. If it happens anyway, an old sidecar writes no `received` line, so the proof never forms. + +## Time to first token + +Change 2 touches only recovery. Change 1 adds to every turn, before the prompt reaches the provider: one small unsynced file write when the turn starts, and one append plus `fsync` immediately before the prompt write. It adds no wait, no file transfer and no round trip. The measured cost is in the change 1 PR (target: well under 5 ms p95 on local disks, against a 9.5–16.5 s TTFT p50 budget). Recovery-time work (fence proof, journal read) runs only after a failure. diff --git a/packages/paperclip-runner/docs/durable-recovery.md b/packages/paperclip-runner/docs/durable-recovery.md index 84c8e0101bc..d62874891e0 100644 --- a/packages/paperclip-runner/docs/durable-recovery.md +++ b/packages/paperclip-runner/docs/durable-recovery.md @@ -603,6 +603,18 @@ callback that this process is not running (`dispatchToolCallback`) as earlier controller can never record a result, so this is the callback's only settlement, and it never executes the operation again. Every other unresolved input stays unsettled. + +`write_document` needs neither fallback. Its revision and its idempotency +receipt commit in one transaction under the run and issue row locks, and the +revision is compare-and-swap on `baseRevisionId`. The receipt is therefore the +durable outcome record: present means the write applied, absent means it did +not. Recovery runs the same receipt-guarded mutation, whichever controller +admitted the callback. It returns the saved result when the receipt exists and +applies the write once when it does not. A stale base, a reused key or invalid +input is the call's real outcome and settles as a tool error. Any other failure +leaves the call unsettled. A late original callback then replays the receipt, +so the document is never written twice. + A different case is a runner crash after the controller saved the result but before its delivery command completed. Codex/OpenCode explicitly opt in to reconciling that exact `semantic_tool.result` command through their durable tool diff --git a/packages/paperclip-runner/src/control-plane/orphaned-tool-callback.ts b/packages/paperclip-runner/src/control-plane/orphaned-tool-callback.ts index ef0ce5bf10a..b187c0bb13f 100644 --- a/packages/paperclip-runner/src/control-plane/orphaned-tool-callback.ts +++ b/packages/paperclip-runner/src/control-plane/orphaned-tool-callback.ts @@ -15,6 +15,13 @@ * callback settles as interrupted with an unknown outcome. It is never * executed again. A callback that this process is still running, or that this * process dispatched and saw fail, stays pending. + * + * Exception: an authority may settle a transactional write itself, for any + * admission, when its effect and receipt commit in one locked transaction + * (server `TRANSACTIONAL_RECEIPT_OPERATIONS`, today only `write_document`). + * Its `recover` then runs the receipt-guarded write, which replays the saved + * result or applies it once; the owner-gone guard here is not what makes that + * safe. */ import type { SemanticInputAdmission } from "./durable-prp-control-plane.js"; diff --git a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts index 7dcb86f8345..0e401700d39 100644 --- a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts +++ b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts @@ -929,8 +929,11 @@ process.on("SIGTERM", () => child.kill("SIGTERM")); } } }).where(eq(heartbeatRuns.id, fixture.runId)); await expect(recover(call)) .resolves.toEqual({ result: { revision: "revision-1" } }); + // write_document's receipt commits with its revision, so a key whose + // receipt holds other input proves this call can never apply under it. await expect(recover({ ...call, input: { ...call.input, body: "changed" } })) - .resolves.toBeNull(); + .resolves.toMatchObject({ isError: true, result: { success: false, contentItems: [ + { type: "inputText", text: expect.stringContaining("paperclip_runner_tool_idempotency_conflict") }] } }); await expect(recover({ ...call, operationId: "unknown_external_write" })) .resolves.toBeNull(); expect(executeRead).not.toHaveBeenCalled(); diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts index a7efadbf80a..262c166c0b2 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts @@ -1314,6 +1314,72 @@ describe("PaperclipRunnerToolAuthority", () => { ).rejects.toThrow("paperclip_runner_tool_idempotency_conflict"); }); + it("settles a write_document a restart orphaned from its transactional receipt, applying it at most once", async () => { + // A restart can cut a write_document callback after the runner journaled + // its input but before any result was saved. The revision and its receipt + // commit in one transaction, so recovery can prove which side it is on. + const authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId }); + const notApplied = { + tool: "write_document", + callId: "orphaned-write-not-applied", + arguments: { idempotencyKey: "orphaned-write-not-applied", key: "orphan-brief", title: "Brief", + body: "Recovered once.", baseRevisionId: null, changeSummary: "Initial brief" }, + }; + // No receipt: the write provably never applied. Recovery applies it once. + const recovered = await authority.recover(notApplied); + expect(recovered).toMatchObject({ result: { disposition: "applied", created: true, document: { key: "orphan-brief" } } }); + expect(recovered?.isError).not.toBe(true); + const applied = await documentService(db).getIssueDocumentByKey(issueId, "orphan-brief"); + expect(applied).toMatchObject({ body: "Recovered once.", latestRevisionNumber: 1 }); + // A second recovery, and the original callback arriving late, both replay + // the receipt instead of writing again. + expect(await authority.recover(notApplied)).toEqual(recovered); + expect(await authority.execute(notApplied)).toEqual(recovered!.result); + expect(await documentService(db).getIssueDocumentByKey(issueId, "orphan-brief")) + .toMatchObject({ latestRevisionNumber: 1 }); + + // Receipt present: the write applied before the restart. Recovery returns + // the saved result and writes nothing. + const update = { + tool: "write_document", + callId: "orphaned-write-applied", + arguments: { idempotencyKey: "orphaned-write-applied", key: "orphan-brief", title: "Brief", + body: "Second revision.", baseRevisionId: applied!.latestRevisionId, changeSummary: "Update" }, + }; + const original = await authority.execute(update); + expect(await authority.recover(update)).toEqual({ result: original }); + expect(await documentService(db).getIssueDocumentByKey(issueId, "orphan-brief")) + .toMatchObject({ body: "Second revision.", latestRevisionNumber: 2 }); + + // A write whose base moved on cannot apply. Recovery reports the definite + // conflict as a tool error and changes nothing. + const stale = { + tool: "write_document", + callId: "orphaned-write-stale", + arguments: { idempotencyKey: "orphaned-write-stale", key: "orphan-brief", title: "Brief", + body: "Stale.", baseRevisionId: applied!.latestRevisionId, changeSummary: "Stale" }, + }; + // A task switched to ask mode refuses a write that never applied, while + // a write that did apply still replays its receipt. + await db.update(issues).set({ workMode: "ask" }).where(eq(issues.id, issueId)); + try { + const askMode = await authority.recover({ ...notApplied, callId: "orphaned-write-ask", + arguments: { ...notApplied.arguments, idempotencyKey: "orphaned-write-ask", key: "orphan-ask" } }); + expect(JSON.stringify(askMode)).toContain("paperclip_runner_tool_mode_denied"); + expect(askMode).toMatchObject({ isError: true }); + expect(await documentService(db).getIssueDocumentByKey(issueId, "orphan-ask")).toBeNull(); + expect(await authority.recover(update)).toEqual({ result: original }); + } finally { + await db.update(issues).set({ workMode: "standard" }).where(eq(issues.id, issueId)); + } + + const conflict = await authority.recover(stale); + expect(conflict).toMatchObject({ isError: true, result: { success: false } }); + expect(JSON.stringify(conflict)).toContain("Document was updated by someone else"); + expect(await documentService(db).getIssueDocumentByKey(issueId, "orphan-brief")) + .toMatchObject({ body: "Second revision.", latestRevisionNumber: 2 }); + }); + it("returns the exact accepted plan revision in task context", async () => { const plan = await documentService(db).getIssueDocumentByKey( issueId, diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts index c3ed5300321..7a718df59be 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts @@ -174,6 +174,19 @@ const RECOVERABLE_READ_OPERATIONS = new Set([ READ_CURRENT_WAKE_COMMENTS_TOOL_NAME, LIST_CHAT_ATTACHMENTS_TOOL_NAME, READ_CHAT_ATTACHMENT_TOOL_NAME, READ_TOOL_RESULT, "connections_search", ]); +// Writes whose effect and idempotency receipt commit in one transaction, with +// no outside effect before commit, and whose effect is compare-and-swap on the +// state it read. Recovery may run them again: the receipt proves whether the +// first attempt applied (see #recoverTransactionalWrite). +const TRANSACTIONAL_RECEIPT_OPERATIONS = new Set(["write_document"]); +const TRANSACTIONAL_RECOVERY_REFUSALS = new Set([ + "paperclip_runner_tool_idempotency_conflict", "paperclip_runner_tool_input_invalid", + "paperclip_runner_tool_mode_denied", +]); +function nativeToolAllowedInMode(operationId: string, workMode: string): boolean { + return CAPABILITY_SEMANTIC_TOOL_CATALOG.some(candidate => candidate.operationId === operationId + && candidate.allowedModes.includes(workMode as "standard" | "planning" | "ask")); +} const RECOVERABLE_READ_ERRORS = new Set([ "paperclip_runner_document_not_found", "paperclip_runner_agent_not_found", "paperclip_runner_approval_not_found", "paperclip_runner_tool_input_invalid", @@ -477,6 +490,7 @@ export class PaperclipRunnerToolAuthority { if (!Object.hasOwn(receipt, "result")) return null; return { result: await this.#withInstructionMutationReceipt(call.tool, call.callId, input, true) }; } + if (TRANSACTIONAL_RECEIPT_OPERATIONS.has(call.tool)) return this.#recoverTransactionalWrite(call.tool, input); // Recovery goes through the receipt owner: it must use the same parsed // input, current authority and safe post-commit dispatch as an ordinary // retry, without ever falling through to an uncertain mutation. @@ -541,6 +555,32 @@ export class PaperclipRunnerToolAuthority { + " Call list_agents and copy a valid id exactly; nothing was created or changed."); } + /** + * Settle a transactional write whose callback a restart cut off. The effect + * and its idempotency receipt commit in one transaction under the run and + * issue row locks, so the receipt is the durable outcome record: present + * means applied, absent means not applied. Running the same receipt-guarded + * mutation therefore either replays the saved result or applies the write + * for the first time, never twice. A definite pre-commit refusal (a stale + * base, a reused key, invalid input) is the call's real outcome and returns + * as a tool error; any other failure stays unsettled. + */ + async #recoverTransactionalWrite(tool: string, input: Record) { + try { + if (tool === "write_document") return { result: await this.#writeDocument(input, true) }; + throw new Error(`paperclip_runner_transactional_recovery_unsupported: ${tool}`); + } catch (error) { + // Each of these is raised before the transaction commits, so it rolls the + // write back: the call's outcome is this refusal, not an unknown one. + const definite = error instanceof HttpError ? error.status >= 400 && error.status < 500 + : error instanceof Error && TRANSACTIONAL_RECOVERY_REFUSALS.has(error.message); + if (!definite) throw error; + return { isError: true, result: { + success: false, contentItems: [{ type: "inputText", text: (error as Error).message }], + } }; + } + } + async #recoverRead(call: Parameters[0]) { try { return { result: await this.execute(call) }; @@ -1850,10 +1890,16 @@ export class PaperclipRunnerToolAuthority { return result; } - async #writeDocument(input: Record): Promise { + async #writeDocument(input: Record, recovering = false): Promise { const idempotencyKey = requiredString(input.idempotencyKey); let publication: Awaited>["publication"] | null = null; const result = await this.#withMutationReceipt("write_document", idempotencyKey, input, async (tx, context) => { + // execute() checks the work mode before dispatch. A recovered write + // applies later, so it checks the mode it locked; a saved receipt + // still replays without this check because its effect already applied. + if (recovering && !nativeToolAllowedInMode("write_document", context.issue.workMode)) { + throw new Error("paperclip_runner_tool_mode_denied"); + } const sourceTrust = this.binding.museRuntime ? await resolveActorSourceTrustForIssue({ db: tx, issue: context.issue, actor: { actorType: "agent", actorId: this.binding.agentId, agentId: this.binding.agentId, runId: this.binding.runId } }) : undefined; const write = await documentService(tx).upsertIssueDocument({