mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:38:22 +01:00
DottaandClaude
e33ede9a8a
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run calls Paperclip tools such as `write_document`. The runner saves each tool input durably before the server runs the tool, and the server saves a result receipt > - A control-plane restart can cut a `write_document` call after the runner saved its input and before any result was saved > - Recovery then has no proof of the outcome. A call from an earlier server process settles as "unknown outcome" (#20628). A call the current server process admitted stays pending, close cannot prove that the provider drained, and the session quarantines (`native_session_cleanup_quarantined`). A person must click Retry > - `write_document` saves its revision and its idempotency receipt in one database transaction, under the run and issue row locks. The revision is also compare-and-swap on `baseRevisionId`. So the receipt is a durable outcome record: present means applied, absent means not applied > - This pull request makes recovery use that proof. Recovery runs the same receipt-guarded write. It replays the saved result, or it applies the write exactly once > - The benefit is that an orphaned `write_document` settles with its real result. The document is never written twice, and the run does not need Retry for this call ## Linked Issues or Issue Description **What happened?** Daytona restart-chaos wave `w-20261110T174930-s9205`, task t01 (codex, workload C1, fault RR2). A restart cut a `write_document` call. The close settlement showed `semanticTools.pending: [{ operationId: "write_document", callId: "exec-8b79ab78-…" }]` and `providerDrained: false`. The session quarantined with `native_session_cleanup_quarantined`, and the task needed Retry. Since #20628, a call that an earlier server process admitted settles as `tool_call_interrupted_by_restart` with an unknown outcome. The agent then has to guess whether the write applied. **Expected behavior** When a durable record proves the outcome, recovery uses it. A write that applied returns its saved result. A write that did not apply runs once. When the record cannot prove it, the call stays as it is today. **Steps to reproduce** 1. On master, run `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/paperclip-runner-tool-authority.test.ts -t "orphaned"`. 2. `authority.recover()` returns `null` for a `write_document` that has no receipt, so the call stays unsettled. Refs #20628, #20639, #20507 ## What Changed - `PaperclipRunnerToolAuthority.recover()` now settles `write_document` through its transactional receipt (`#recoverTransactionalWrite`). It runs the same receipt-guarded mutation as an ordinary call. If the receipt exists, it returns the saved result and writes nothing. If no receipt exists, it applies the write once. - A definite refusal that rolls the transaction back (a 4xx such as a stale `baseRevisionId`, a reused idempotency key, invalid input) settles as a tool error. Any other failure leaves the call unsettled, as before. - The set of operations with this proof is explicit: `TRANSACTIONAL_RECEIPT_OPERATIONS` contains only `write_document`. Other tools keep their current recovery rules. - New test: an orphaned `write_document` with no receipt applies once; a second recovery and a late original call replay the receipt; an applied write recovers its saved result; a stale base settles as a tool error with no change. - Updated test: a reused idempotency key with other input now settles as an `paperclip_runner_tool_idempotency_conflict` tool error instead of staying pending. The receipt proves that this call can never apply under that key. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` (semantic operations that outlive a turn) and a new design note, `doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md`. The note also covers the second change (an ACPX turn start cut off before the prompt reached the provider), which comes in a separate pull request. ## Verification - Red on master, green here: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/paperclip-runner-tool-authority.test.ts` (43 passed). - `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`: 178 passed. One unrelated ACPX timing test failed once under load and passed when run alone. - Server typecheck: `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify` exits 2 (a runner doc changed), so required CI supplies the broad coverage. - Live restart-chaos wave: see the comment on this pull request. ## Risks - Recovery can now apply a write that never applied before the restart. This is the call the agent made, applied once. The write is compare-and-swap on `baseRevisionId`, so it cannot overwrite a revision that someone wrote after the agent read the document. That case settles as a conflict error. - Upgrade safety: no saved shape changes. Receipts saved by the previous release have the same shape. A server on the previous release keeps its current recovery for these calls. - Time to first token: no change. This code runs only during recovery. - No runner source changes, so no new runner image is needed. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file editing) in Claude Code. Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>