From e33ede9a8add1fd83b4ea16c8a803d943c74204a Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 10 Nov 2026 19:05:15 -0600 Subject: [PATCH] fix(native): settle a restart-orphaned write_document from its transactional receipt (#20674) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run calls Paperclip tools such as `write_document`. The runner saves each tool input durably before the server runs the tool, and the server saves a result receipt > - A control-plane restart can cut a `write_document` call after the runner saved its input and before any result was saved > - Recovery then has no proof of the outcome. A call from an earlier server process settles as "unknown outcome" (#20628). A call the current server process admitted stays pending, close cannot prove that the provider drained, and the session quarantines (`native_session_cleanup_quarantined`). A person must click Retry > - `write_document` saves its revision and its idempotency receipt in one database transaction, under the run and issue row locks. The revision is also compare-and-swap on `baseRevisionId`. So the receipt is a durable outcome record: present means applied, absent means not applied > - This pull request makes recovery use that proof. Recovery runs the same receipt-guarded write. It replays the saved result, or it applies the write exactly once > - The benefit is that an orphaned `write_document` settles with its real result. The document is never written twice, and the run does not need Retry for this call ## Linked Issues or Issue Description **What happened?** Daytona restart-chaos wave `w-20261110T174930-s9205`, task t01 (codex, workload C1, fault RR2). A restart cut a `write_document` call. The close settlement showed `semanticTools.pending: [{ operationId: "write_document", callId: "exec-8b79ab78-…" }]` and `providerDrained: false`. The session quarantined with `native_session_cleanup_quarantined`, and the task needed Retry. Since #20628, a call that an earlier server process admitted settles as `tool_call_interrupted_by_restart` with an unknown outcome. The agent then has to guess whether the write applied. **Expected behavior** When a durable record proves the outcome, recovery uses it. A write that applied returns its saved result. A write that did not apply runs once. When the record cannot prove it, the call stays as it is today. **Steps to reproduce** 1. On master, run `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/paperclip-runner-tool-authority.test.ts -t "orphaned"`. 2. `authority.recover()` returns `null` for a `write_document` that has no receipt, so the call stays unsettled. Refs #20628, #20639, #20507 ## What Changed - `PaperclipRunnerToolAuthority.recover()` now settles `write_document` through its transactional receipt (`#recoverTransactionalWrite`). It runs the same receipt-guarded mutation as an ordinary call. If the receipt exists, it returns the saved result and writes nothing. If no receipt exists, it applies the write once. - A definite refusal that rolls the transaction back (a 4xx such as a stale `baseRevisionId`, a reused idempotency key, invalid input) settles as a tool error. Any other failure leaves the call unsettled, as before. - The set of operations with this proof is explicit: `TRANSACTIONAL_RECEIPT_OPERATIONS` contains only `write_document`. Other tools keep their current recovery rules. - New test: an orphaned `write_document` with no receipt applies once; a second recovery and a late original call replay the receipt; an applied write recovers its saved result; a stale base settles as a tool error with no change. - Updated test: a reused idempotency key with other input now settles as an `paperclip_runner_tool_idempotency_conflict` tool error instead of staying pending. The receipt proves that this call can never apply under that key. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` (semantic operations that outlive a turn) and a new design note, `doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md`. The note also covers the second change (an ACPX turn start cut off before the prompt reached the provider), which comes in a separate pull request. ## Verification - Red on master, green here: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/paperclip-runner-tool-authority.test.ts` (43 passed). - `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`: 178 passed. One unrelated ACPX timing test failed once under load and passed when run alone. - Server typecheck: `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify` exits 2 (a runner doc changed), so required CI supplies the broad coverage. - Live restart-chaos wave: see the comment on this pull request. ## Risks - Recovery can now apply a write that never applied before the restart. This is the call the agent made, applied once. The write is compare-and-swap on `baseRevisionId`, so it cannot overwrite a revision that someone wrote after the agent read the document. That case settles as a conflict error. - Upgrade safety: no saved shape changes. Receipts saved by the previous release have the same shape. A server on the previous release keeps its current recovery for these calls. - Time to first token: no change. This code runs only during recovery. - No runner source changes, so no new runner image is needed. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file editing) in Claude Code. Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude --- ...urable-turn-start-and-tool-call-records.md | 56 ++++++++++++++++ .../paperclip-runner/docs/durable-recovery.md | 12 ++++ .../control-plane/orphaned-tool-callback.ts | 7 ++ ...unner-restart-recovery.integration.test.ts | 5 +- .../paperclip-runner-tool-authority.test.ts | 66 +++++++++++++++++++ .../paperclip-runner-tool-authority.ts | 48 +++++++++++++- 6 files changed, 192 insertions(+), 2 deletions(-) create mode 100644 doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md diff --git a/doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md b/doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md new file mode 100644 index 00000000000..a7828ba4e0f --- /dev/null +++ b/doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md @@ -0,0 +1,56 @@ +# Durable records for cut-off turn starts and orphaned tool calls + +Status: approved by Dotta (2026-11-10), being implemented in two PRs +Goal: after a control-plane restart or freeze, 0% of tasks need Retry, and Retry still works whenever it is shown. +Parent plan: `2026-11-07-durable-ownership-and-reconciliation.md`, section B (write-ahead intents) and section F (charge only on recorded outcomes). + +## Rule + +Recovery may resend, replay or continue only when a durable record proves that no provider work and no tool effect can be duplicated or lost. A duplicated prompt or a second write is worse than a Retry click. When the record cannot prove it, the existing behaviour stays: a working Retry. + +## Change 1: a turn start cut off before the prompt reached the provider + +**What happens today.** Restart-chaos wave `w-20261110T185315-s9107`, task t05 (claude, workload C1, fault TM2). A follow-up run's `turn.start` reaches the Runner, which saves the ACPX session as `turn_starting` and asks the ACPX sidecar to start the turn. For a resumed Claude session the sidecar first starts Claude and loads the conversation; the prompt is sent only after that. Under load this took more than 24 s. Then the harness froze the server, the Runner, the sidecar and Claude together (SIGSTOP) for 33 s. On thaw the Runner's 30 s sidecar request deadline had passed, so it reaped the sidecar and saved the session as `closed`. Every same-run resume then failed with `ACPX session is not open`, recovery stopped with `native_session_retry_exhausted`, and a person had to click Retry. + +The Runner cannot tell from its own state whether Claude received the prompt. Resending could duplicate it; a continuation could lose it. + +**The record: a sidecar prompt-dispatch journal.** The ACPX sidecar writes one small file in the session's private runtime directory (`0700`) for each turn: + +1. When the runtime port starts the turn, before any provider work: one line `{"schema":…,"turnId":X,"phase":"received"}`, truncating the previous turn's file. No fsync. +2. When ACPX is about to write that turn's first `session/prompt` request to the provider's stdin: one appended line `{"phase":"dispatched"}`, then `fsync`, and only then the write. ACPX calls the host's `onAcpMessage("outbound", …)` hook synchronously before it hands the bytes to the provider stream, so a failed journal write aborts the prompt write. + +The file name and record shape are declared in `protocol/provider-schemas/acpx-sidecar.schema.json` and generated into TypeScript and Rust, so the pack contract digest covers them. + +**What it proves.** The Runner reads the journal only after it has proved that the sidecar and the provider are gone (it terminated the sidecar and acquired the provider lifetime fence, the same proof the existing lost-turn restore uses). A dead sidecar cannot write or send anything more. + +| Journal for the active turn | Meaning | Recovery | +|---|---|---| +| exactly one complete `received` line | the prompt was never written to the provider | restore the session with no active turn; emit `harness.diagnostic` `acpx_turn_start_not_dispatched`; the same-run resume sends the run's `turn.start` once | +| a `dispatched` line | the provider may have the prompt | unchanged: the existing lost-turn path (restore and continue in the same conversation, or close) | +| missing, another turn, malformed, torn | unknown | unchanged | + +Both cut-off shapes use it: a sidecar request that failed in flight (timeout or sidecar exit), and a Runner killed while the session was `turn_starting`. + +**Resend exactly once.** The cut-off `turn.start` command stays failed; it is never executed again. The restored session has no active turn, so the next same-run resume starts the turn with the run's own input. If that start is cut off too, the journal decides again. + +## Change 2: a tool call orphaned by a restart + +**What happens today.** Daytona wave `w-20261110T174930-s9205`, task t01 (codex, C1, RR2): a restart cut a `write_document` callback after the Runner journaled its input and before a result was saved. Close could not prove the provider drained, so the session quarantined (`native_session_cleanup_quarantined`). Since #20628 a callback admitted by an earlier controller settles as `tool_call_interrupted_by_restart` (unknown outcome, never executed again); a callback the current controller admitted still stays pending. + +**The record already exists.** The intent is the Runner's durable semantic input (committed before dispatch, keyed by call ID). The outcome is the server's idempotency receipt in `heartbeat_runs.result_json.semanticToolReceipts`. For `write_document` the revision, the activity row and the receipt commit in one transaction under the run and issue row locks, and the revision is compare-and-swap on `baseRevisionId`. + +**What it proves.** Receipt present: applied. Receipt absent: not applied, because the effect cannot commit without it. So recovery runs the same receipt-guarded mutation. It replays the saved result or applies the write for the first time, never twice. A late original callback then replays the receipt. A stale base, a reused key or invalid input is the call's definite outcome and settles as a tool error. Any other failure leaves the call unsettled, as before. The set of operations with this proof is explicit (`TRANSACTIONAL_RECEIPT_OPERATIONS`, `write_document` only); other tools keep their current rules. + +This makes the gateway call idempotent on its durable input, which is the cleaner proof the brief allowed. It needs no Runner change and no new image. + +**Not done.** Exposing the cleanup-gate owner's run id so the server can drop the 50-run scan bound from #20639 (optional in the brief) is left out to keep scope. + +## Upgrade safety + +- Old saved state, new code. A `turn_starting` session saved by an older Runner has no journal (or one for another turn): unknown, so the old lost-turn path runs unchanged. Old receipts already have the shape the transactional recovery reads. +- New state, old code mid-deploy. Change 1 adds no field to `acpx-provider-state.json` (its decoder denies unknown fields). It only saves lifecycles an old Runner already handles (`prepared`, then `suspended`/`session_open`). An old Runner ignores the journal file. Change 2 changes no saved shape. +- New Runner, old sidecar. Possible only with a stale pack; the contract digest prevents reuse. If it happens anyway, an old sidecar writes no `received` line, so the proof never forms. + +## Time to first token + +Change 2 touches only recovery. Change 1 adds to every turn, before the prompt reaches the provider: one small unsynced file write when the turn starts, and one append plus `fsync` immediately before the prompt write. It adds no wait, no file transfer and no round trip. The measured cost is in the change 1 PR (target: well under 5 ms p95 on local disks, against a 9.5–16.5 s TTFT p50 budget). Recovery-time work (fence proof, journal read) runs only after a failure. diff --git a/packages/paperclip-runner/docs/durable-recovery.md b/packages/paperclip-runner/docs/durable-recovery.md index 84c8e0101bc..d62874891e0 100644 --- a/packages/paperclip-runner/docs/durable-recovery.md +++ b/packages/paperclip-runner/docs/durable-recovery.md @@ -603,6 +603,18 @@ callback that this process is not running (`dispatchToolCallback`) as earlier controller can never record a result, so this is the callback's only settlement, and it never executes the operation again. Every other unresolved input stays unsettled. + +`write_document` needs neither fallback. Its revision and its idempotency +receipt commit in one transaction under the run and issue row locks, and the +revision is compare-and-swap on `baseRevisionId`. The receipt is therefore the +durable outcome record: present means the write applied, absent means it did +not. Recovery runs the same receipt-guarded mutation, whichever controller +admitted the callback. It returns the saved result when the receipt exists and +applies the write once when it does not. A stale base, a reused key or invalid +input is the call's real outcome and settles as a tool error. Any other failure +leaves the call unsettled. A late original callback then replays the receipt, +so the document is never written twice. + A different case is a runner crash after the controller saved the result but before its delivery command completed. Codex/OpenCode explicitly opt in to reconciling that exact `semantic_tool.result` command through their durable tool diff --git a/packages/paperclip-runner/src/control-plane/orphaned-tool-callback.ts b/packages/paperclip-runner/src/control-plane/orphaned-tool-callback.ts index ef0ce5bf10a..b187c0bb13f 100644 --- a/packages/paperclip-runner/src/control-plane/orphaned-tool-callback.ts +++ b/packages/paperclip-runner/src/control-plane/orphaned-tool-callback.ts @@ -15,6 +15,13 @@ * callback settles as interrupted with an unknown outcome. It is never * executed again. A callback that this process is still running, or that this * process dispatched and saw fail, stays pending. + * + * Exception: an authority may settle a transactional write itself, for any + * admission, when its effect and receipt commit in one locked transaction + * (server `TRANSACTIONAL_RECEIPT_OPERATIONS`, today only `write_document`). + * Its `recover` then runs the receipt-guarded write, which replays the saved + * result or applies it once; the owner-gone guard here is not what makes that + * safe. */ import type { SemanticInputAdmission } from "./durable-prp-control-plane.js"; diff --git a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts index 7dcb86f8345..0e401700d39 100644 --- a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts +++ b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts @@ -929,8 +929,11 @@ process.on("SIGTERM", () => child.kill("SIGTERM")); } } }).where(eq(heartbeatRuns.id, fixture.runId)); await expect(recover(call)) .resolves.toEqual({ result: { revision: "revision-1" } }); + // write_document's receipt commits with its revision, so a key whose + // receipt holds other input proves this call can never apply under it. await expect(recover({ ...call, input: { ...call.input, body: "changed" } })) - .resolves.toBeNull(); + .resolves.toMatchObject({ isError: true, result: { success: false, contentItems: [ + { type: "inputText", text: expect.stringContaining("paperclip_runner_tool_idempotency_conflict") }] } }); await expect(recover({ ...call, operationId: "unknown_external_write" })) .resolves.toBeNull(); expect(executeRead).not.toHaveBeenCalled(); diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts index a7efadbf80a..262c166c0b2 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts @@ -1314,6 +1314,72 @@ describe("PaperclipRunnerToolAuthority", () => { ).rejects.toThrow("paperclip_runner_tool_idempotency_conflict"); }); + it("settles a write_document a restart orphaned from its transactional receipt, applying it at most once", async () => { + // A restart can cut a write_document callback after the runner journaled + // its input but before any result was saved. The revision and its receipt + // commit in one transaction, so recovery can prove which side it is on. + const authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId }); + const notApplied = { + tool: "write_document", + callId: "orphaned-write-not-applied", + arguments: { idempotencyKey: "orphaned-write-not-applied", key: "orphan-brief", title: "Brief", + body: "Recovered once.", baseRevisionId: null, changeSummary: "Initial brief" }, + }; + // No receipt: the write provably never applied. Recovery applies it once. + const recovered = await authority.recover(notApplied); + expect(recovered).toMatchObject({ result: { disposition: "applied", created: true, document: { key: "orphan-brief" } } }); + expect(recovered?.isError).not.toBe(true); + const applied = await documentService(db).getIssueDocumentByKey(issueId, "orphan-brief"); + expect(applied).toMatchObject({ body: "Recovered once.", latestRevisionNumber: 1 }); + // A second recovery, and the original callback arriving late, both replay + // the receipt instead of writing again. + expect(await authority.recover(notApplied)).toEqual(recovered); + expect(await authority.execute(notApplied)).toEqual(recovered!.result); + expect(await documentService(db).getIssueDocumentByKey(issueId, "orphan-brief")) + .toMatchObject({ latestRevisionNumber: 1 }); + + // Receipt present: the write applied before the restart. Recovery returns + // the saved result and writes nothing. + const update = { + tool: "write_document", + callId: "orphaned-write-applied", + arguments: { idempotencyKey: "orphaned-write-applied", key: "orphan-brief", title: "Brief", + body: "Second revision.", baseRevisionId: applied!.latestRevisionId, changeSummary: "Update" }, + }; + const original = await authority.execute(update); + expect(await authority.recover(update)).toEqual({ result: original }); + expect(await documentService(db).getIssueDocumentByKey(issueId, "orphan-brief")) + .toMatchObject({ body: "Second revision.", latestRevisionNumber: 2 }); + + // A write whose base moved on cannot apply. Recovery reports the definite + // conflict as a tool error and changes nothing. + const stale = { + tool: "write_document", + callId: "orphaned-write-stale", + arguments: { idempotencyKey: "orphaned-write-stale", key: "orphan-brief", title: "Brief", + body: "Stale.", baseRevisionId: applied!.latestRevisionId, changeSummary: "Stale" }, + }; + // A task switched to ask mode refuses a write that never applied, while + // a write that did apply still replays its receipt. + await db.update(issues).set({ workMode: "ask" }).where(eq(issues.id, issueId)); + try { + const askMode = await authority.recover({ ...notApplied, callId: "orphaned-write-ask", + arguments: { ...notApplied.arguments, idempotencyKey: "orphaned-write-ask", key: "orphan-ask" } }); + expect(JSON.stringify(askMode)).toContain("paperclip_runner_tool_mode_denied"); + expect(askMode).toMatchObject({ isError: true }); + expect(await documentService(db).getIssueDocumentByKey(issueId, "orphan-ask")).toBeNull(); + expect(await authority.recover(update)).toEqual({ result: original }); + } finally { + await db.update(issues).set({ workMode: "standard" }).where(eq(issues.id, issueId)); + } + + const conflict = await authority.recover(stale); + expect(conflict).toMatchObject({ isError: true, result: { success: false } }); + expect(JSON.stringify(conflict)).toContain("Document was updated by someone else"); + expect(await documentService(db).getIssueDocumentByKey(issueId, "orphan-brief")) + .toMatchObject({ body: "Second revision.", latestRevisionNumber: 2 }); + }); + it("returns the exact accepted plan revision in task context", async () => { const plan = await documentService(db).getIssueDocumentByKey( issueId, diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts index c3ed5300321..7a718df59be 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts @@ -174,6 +174,19 @@ const RECOVERABLE_READ_OPERATIONS = new Set([ READ_CURRENT_WAKE_COMMENTS_TOOL_NAME, LIST_CHAT_ATTACHMENTS_TOOL_NAME, READ_CHAT_ATTACHMENT_TOOL_NAME, READ_TOOL_RESULT, "connections_search", ]); +// Writes whose effect and idempotency receipt commit in one transaction, with +// no outside effect before commit, and whose effect is compare-and-swap on the +// state it read. Recovery may run them again: the receipt proves whether the +// first attempt applied (see #recoverTransactionalWrite). +const TRANSACTIONAL_RECEIPT_OPERATIONS = new Set(["write_document"]); +const TRANSACTIONAL_RECOVERY_REFUSALS = new Set([ + "paperclip_runner_tool_idempotency_conflict", "paperclip_runner_tool_input_invalid", + "paperclip_runner_tool_mode_denied", +]); +function nativeToolAllowedInMode(operationId: string, workMode: string): boolean { + return CAPABILITY_SEMANTIC_TOOL_CATALOG.some(candidate => candidate.operationId === operationId + && candidate.allowedModes.includes(workMode as "standard" | "planning" | "ask")); +} const RECOVERABLE_READ_ERRORS = new Set([ "paperclip_runner_document_not_found", "paperclip_runner_agent_not_found", "paperclip_runner_approval_not_found", "paperclip_runner_tool_input_invalid", @@ -477,6 +490,7 @@ export class PaperclipRunnerToolAuthority { if (!Object.hasOwn(receipt, "result")) return null; return { result: await this.#withInstructionMutationReceipt(call.tool, call.callId, input, true) }; } + if (TRANSACTIONAL_RECEIPT_OPERATIONS.has(call.tool)) return this.#recoverTransactionalWrite(call.tool, input); // Recovery goes through the receipt owner: it must use the same parsed // input, current authority and safe post-commit dispatch as an ordinary // retry, without ever falling through to an uncertain mutation. @@ -541,6 +555,32 @@ export class PaperclipRunnerToolAuthority { + " Call list_agents and copy a valid id exactly; nothing was created or changed."); } + /** + * Settle a transactional write whose callback a restart cut off. The effect + * and its idempotency receipt commit in one transaction under the run and + * issue row locks, so the receipt is the durable outcome record: present + * means applied, absent means not applied. Running the same receipt-guarded + * mutation therefore either replays the saved result or applies the write + * for the first time, never twice. A definite pre-commit refusal (a stale + * base, a reused key, invalid input) is the call's real outcome and returns + * as a tool error; any other failure stays unsettled. + */ + async #recoverTransactionalWrite(tool: string, input: Record) { + try { + if (tool === "write_document") return { result: await this.#writeDocument(input, true) }; + throw new Error(`paperclip_runner_transactional_recovery_unsupported: ${tool}`); + } catch (error) { + // Each of these is raised before the transaction commits, so it rolls the + // write back: the call's outcome is this refusal, not an unknown one. + const definite = error instanceof HttpError ? error.status >= 400 && error.status < 500 + : error instanceof Error && TRANSACTIONAL_RECOVERY_REFUSALS.has(error.message); + if (!definite) throw error; + return { isError: true, result: { + success: false, contentItems: [{ type: "inputText", text: (error as Error).message }], + } }; + } + } + async #recoverRead(call: Parameters[0]) { try { return { result: await this.execute(call) }; @@ -1850,10 +1890,16 @@ export class PaperclipRunnerToolAuthority { return result; } - async #writeDocument(input: Record): Promise { + async #writeDocument(input: Record, recovering = false): Promise { const idempotencyKey = requiredString(input.idempotencyKey); let publication: Awaited>["publication"] | null = null; const result = await this.#withMutationReceipt("write_document", idempotencyKey, input, async (tx, context) => { + // execute() checks the work mode before dispatch. A recovered write + // applies later, so it checks the mode it locked; a saved receipt + // still replays without this check because its effect already applied. + if (recovering && !nativeToolAllowedInMode("write_document", context.issue.workMode)) { + throw new Error("paperclip_runner_tool_mode_denied"); + } const sourceTrust = this.binding.museRuntime ? await resolveActorSourceTrustForIssue({ db: tx, issue: context.issue, actor: { actorType: "agent", actorId: this.binding.agentId, agentId: this.binding.agentId, runId: this.binding.runId } }) : undefined; const write = await documentService(tx).upsertIssueDocument({