fix(native): settle a restart-orphaned write_document from its transactional receipt (#20674)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run calls Paperclip tools such as `write_document`. The
runner saves each tool input durably before the server runs the tool,
and the server saves a result receipt
> - A control-plane restart can cut a `write_document` call after the
runner saved its input and before any result was saved
> - Recovery then has no proof of the outcome. A call from an earlier
server process settles as "unknown outcome" (#20628). A call the current
server process admitted stays pending, close cannot prove that the
provider drained, and the session quarantines
(`native_session_cleanup_quarantined`). A person must click Retry
> - `write_document` saves its revision and its idempotency receipt in
one database transaction, under the run and issue row locks. The
revision is also compare-and-swap on `baseRevisionId`. So the receipt is
a durable outcome record: present means applied, absent means not
applied
> - This pull request makes recovery use that proof. Recovery runs the
same receipt-guarded write. It replays the saved result, or it applies
the write exactly once
> - The benefit is that an orphaned `write_document` settles with its
real result. The document is never written twice, and the run does not
need Retry for this call

## Linked Issues or Issue Description

**What happened?**
Daytona restart-chaos wave `w-20261110T174930-s9205`, task t01 (codex,
workload C1, fault RR2). A restart cut a `write_document` call. The
close settlement showed `semanticTools.pending: [{ operationId:
"write_document", callId: "exec-8b79ab78-…" }]` and `providerDrained:
false`. The session quarantined with
`native_session_cleanup_quarantined`, and the task needed Retry. Since
#20628, a call that an earlier server process admitted settles as
`tool_call_interrupted_by_restart` with an unknown outcome. The agent
then has to guess whether the write applied.

**Expected behavior**
When a durable record proves the outcome, recovery uses it. A write that
applied returns its saved result. A write that did not apply runs once.
When the record cannot prove it, the call stays as it is today.

**Steps to reproduce**
1. On master, run `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/paperclip-runner-tool-authority.test.ts -t
"orphaned"`.
2. `authority.recover()` returns `null` for a `write_document` that has
no receipt, so the call stays unsettled.

Refs #20628, #20639, #20507

## What Changed

- `PaperclipRunnerToolAuthority.recover()` now settles `write_document`
through its transactional receipt (`#recoverTransactionalWrite`). It
runs the same receipt-guarded mutation as an ordinary call. If the
receipt exists, it returns the saved result and writes nothing. If no
receipt exists, it applies the write once.
- A definite refusal that rolls the transaction back (a 4xx such as a
stale `baseRevisionId`, a reused idempotency key, invalid input) settles
as a tool error. Any other failure leaves the call unsettled, as before.
- The set of operations with this proof is explicit:
`TRANSACTIONAL_RECEIPT_OPERATIONS` contains only `write_document`. Other
tools keep their current recovery rules.
- New test: an orphaned `write_document` with no receipt applies once; a
second recovery and a late original call replay the receipt; an applied
write recovers its saved result; a stale base settles as a tool error
with no change.
- Updated test: a reused idempotency key with other input now settles as
an `paperclip_runner_tool_idempotency_conflict` tool error instead of
staying pending. The receipt proves that this call can never apply under
that key.
- Docs: `packages/paperclip-runner/docs/durable-recovery.md` (semantic
operations that outlive a turn) and a new design note,
`doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md`. The
note also covers the second change (an ACPX turn start cut off before
the prompt reached the provider), which comes in a separate pull
request.

## Verification

- Red on master, green here: `pnpm --filter @paperclipai/server exec
vitest run
src/services/native-runtime/paperclip-runner-tool-authority.test.ts` (43
passed).
- `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
178 passed. One unrelated ACPX timing test failed once under load and
passed when run alone.
- Server typecheck: `npx tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify` exits 2 (a runner doc changed), so required CI supplies
the broad coverage.
- Live restart-chaos wave: see the comment on this pull request.

## Risks

- Recovery can now apply a write that never applied before the restart.
This is the call the agent made, applied once. The write is
compare-and-swap on `baseRevisionId`, so it cannot overwrite a revision
that someone wrote after the agent read the document. That case settles
as a conflict error.
- Upgrade safety: no saved shape changes. Receipts saved by the previous
release have the same shape. A server on the previous release keeps its
current recovery for these calls.
- Time to first token: no change. This code runs only during recovery.
- No runner source changes, so no new runner image is needed.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file editing) in Claude Code. Extended reasoning was used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
DottaandClaude authored and GitHub committed 2026-11-10 19:05:15 -06:00
1 parent cec903c6f0
commit e33ede9a8a
6 files changed
+192 -2

No files matched your search

@@ -929,8 +929,11 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
} } }).where(eq(heartbeatRuns.id, fixture.runId));
await expect(recover(call))
.resolves.toEqual({ result: { revision: "revision-1" } });
// write_document's receipt commits with its revision, so a key whose
// receipt holds other input proves this call can never apply under it.
await expect(recover({ ...call, input: { ...call.input, body: "changed" } }))
.resolves.toBeNull();
.resolves.toMatchObject({ isError: true, result: { success: false, contentItems: [
{ type: "inputText", text: expect.stringContaining("paperclip_runner_tool_idempotency_conflict") }] } });
await expect(recover({ ...call, operationId: "unknown_external_write" }))
.resolves.toBeNull();
expect(executeRead).not.toHaveBeenCalled();
@@ -1314,6 +1314,72 @@ describe("PaperclipRunnerToolAuthority", () => {
).rejects.toThrow("paperclip_runner_tool_idempotency_conflict");
});
it("settles a write_document a restart orphaned from its transactional receipt, applying it at most once", async () => {
// A restart can cut a write_document callback after the runner journaled
// its input but before any result was saved. The revision and its receipt
// commit in one transaction, so recovery can prove which side it is on.
const authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId });
const notApplied = {
tool: "write_document",
callId: "orphaned-write-not-applied",
arguments: { idempotencyKey: "orphaned-write-not-applied", key: "orphan-brief", title: "Brief",
body: "Recovered once.", baseRevisionId: null, changeSummary: "Initial brief" },
};
// No receipt: the write provably never applied. Recovery applies it once.
const recovered = await authority.recover(notApplied);
expect(recovered).toMatchObject({ result: { disposition: "applied", created: true, document: { key: "orphan-brief" } } });
expect(recovered?.isError).not.toBe(true);
const applied = await documentService(db).getIssueDocumentByKey(issueId, "orphan-brief");
expect(applied).toMatchObject({ body: "Recovered once.", latestRevisionNumber: 1 });
// A second recovery, and the original callback arriving late, both replay
// the receipt instead of writing again.
expect(await authority.recover(notApplied)).toEqual(recovered);
expect(await authority.execute(notApplied)).toEqual(recovered!.result);
expect(await documentService(db).getIssueDocumentByKey(issueId, "orphan-brief"))
.toMatchObject({ latestRevisionNumber: 1 });
// Receipt present: the write applied before the restart. Recovery returns
// the saved result and writes nothing.
const update = {
tool: "write_document",
callId: "orphaned-write-applied",
arguments: { idempotencyKey: "orphaned-write-applied", key: "orphan-brief", title: "Brief",
body: "Second revision.", baseRevisionId: applied!.latestRevisionId, changeSummary: "Update" },
};
const original = await authority.execute(update);
expect(await authority.recover(update)).toEqual({ result: original });
expect(await documentService(db).getIssueDocumentByKey(issueId, "orphan-brief"))
.toMatchObject({ body: "Second revision.", latestRevisionNumber: 2 });
// A write whose base moved on cannot apply. Recovery reports the definite
// conflict as a tool error and changes nothing.
const stale = {
tool: "write_document",
callId: "orphaned-write-stale",
arguments: { idempotencyKey: "orphaned-write-stale", key: "orphan-brief", title: "Brief",
body: "Stale.", baseRevisionId: applied!.latestRevisionId, changeSummary: "Stale" },
};
// A task switched to ask mode refuses a write that never applied, while
// a write that did apply still replays its receipt.
await db.update(issues).set({ workMode: "ask" }).where(eq(issues.id, issueId));
try {
const askMode = await authority.recover({ ...notApplied, callId: "orphaned-write-ask",
arguments: { ...notApplied.arguments, idempotencyKey: "orphaned-write-ask", key: "orphan-ask" } });
expect(JSON.stringify(askMode)).toContain("paperclip_runner_tool_mode_denied");
expect(askMode).toMatchObject({ isError: true });
expect(await documentService(db).getIssueDocumentByKey(issueId, "orphan-ask")).toBeNull();
expect(await authority.recover(update)).toEqual({ result: original });
} finally {
await db.update(issues).set({ workMode: "standard" }).where(eq(issues.id, issueId));
}
const conflict = await authority.recover(stale);
expect(conflict).toMatchObject({ isError: true, result: { success: false } });
expect(JSON.stringify(conflict)).toContain("Document was updated by someone else");
expect(await documentService(db).getIssueDocumentByKey(issueId, "orphan-brief"))
.toMatchObject({ body: "Second revision.", latestRevisionNumber: 2 });
});
it("returns the exact accepted plan revision in task context", async () => {
const plan = await documentService(db).getIssueDocumentByKey(
issueId,
@@ -174,6 +174,19 @@ const RECOVERABLE_READ_OPERATIONS = new Set([
READ_CURRENT_WAKE_COMMENTS_TOOL_NAME, LIST_CHAT_ATTACHMENTS_TOOL_NAME, READ_CHAT_ATTACHMENT_TOOL_NAME,
READ_TOOL_RESULT, "connections_search",
]);
// Writes whose effect and idempotency receipt commit in one transaction, with
// no outside effect before commit, and whose effect is compare-and-swap on the
// state it read. Recovery may run them again: the receipt proves whether the
// first attempt applied (see #recoverTransactionalWrite).
const TRANSACTIONAL_RECEIPT_OPERATIONS = new Set(["write_document"]);
const TRANSACTIONAL_RECOVERY_REFUSALS = new Set([
"paperclip_runner_tool_idempotency_conflict", "paperclip_runner_tool_input_invalid",
"paperclip_runner_tool_mode_denied",
]);
function nativeToolAllowedInMode(operationId: string, workMode: string): boolean {
return CAPABILITY_SEMANTIC_TOOL_CATALOG.some(candidate => candidate.operationId === operationId
&& candidate.allowedModes.includes(workMode as "standard" | "planning" | "ask"));
}
const RECOVERABLE_READ_ERRORS = new Set([
"paperclip_runner_document_not_found", "paperclip_runner_agent_not_found",
"paperclip_runner_approval_not_found", "paperclip_runner_tool_input_invalid",
@@ -477,6 +490,7 @@ export class PaperclipRunnerToolAuthority {
if (!Object.hasOwn(receipt, "result")) return null;
return { result: await this.#withInstructionMutationReceipt(call.tool, call.callId, input, true) };
}
if (TRANSACTIONAL_RECEIPT_OPERATIONS.has(call.tool)) return this.#recoverTransactionalWrite(call.tool, input);
// Recovery goes through the receipt owner: it must use the same parsed
// input, current authority and safe post-commit dispatch as an ordinary
// retry, without ever falling through to an uncertain mutation.
@@ -541,6 +555,32 @@ export class PaperclipRunnerToolAuthority {
+ " Call list_agents and copy a valid id exactly; nothing was created or changed.");
}
/**
* Settle a transactional write whose callback a restart cut off. The effect
* and its idempotency receipt commit in one transaction under the run and
* issue row locks, so the receipt is the durable outcome record: present
* means applied, absent means not applied. Running the same receipt-guarded
* mutation therefore either replays the saved result or applies the write
* for the first time, never twice. A definite pre-commit refusal (a stale
* base, a reused key, invalid input) is the call's real outcome and returns
* as a tool error; any other failure stays unsettled.
*/
async #recoverTransactionalWrite(tool: string, input: Record<string, unknown>) {
try {
if (tool === "write_document") return { result: await this.#writeDocument(input, true) };
throw new Error(`paperclip_runner_transactional_recovery_unsupported: ${tool}`);
} catch (error) {
// Each of these is raised before the transaction commits, so it rolls the
// write back: the call's outcome is this refusal, not an unknown one.
const definite = error instanceof HttpError ? error.status >= 400 && error.status < 500
: error instanceof Error && TRANSACTIONAL_RECOVERY_REFUSALS.has(error.message);
if (!definite) throw error;
return { isError: true, result: {
success: false, contentItems: [{ type: "inputText", text: (error as Error).message }],
} };
}
}
async #recoverRead(call: Parameters<PaperclipRunnerToolAuthority["execute"]>[0]) {
try {
return { result: await this.execute(call) };
@@ -1850,10 +1890,16 @@ export class PaperclipRunnerToolAuthority {
return result;
}
async #writeDocument(input: Record<string, unknown>): Promise<unknown> {
async #writeDocument(input: Record<string, unknown>, recovering = false): Promise<unknown> {
const idempotencyKey = requiredString(input.idempotencyKey);
let publication: Awaited<ReturnType<typeof persistActivity>>["publication"] | null = null;
const result = await this.#withMutationReceipt("write_document", idempotencyKey, input, async (tx, context) => {
// execute() checks the work mode before dispatch. A recovered write
// applies later, so it checks the mode it locked; a saved receipt
// still replays without this check because its effect already applied.
if (recovering && !nativeToolAllowedInMode("write_document", context.issue.workMode)) {
throw new Error("paperclip_runner_tool_mode_denied");
}
const sourceTrust = this.binding.museRuntime ? await resolveActorSourceTrustForIssue({ db: tx, issue: context.issue,
actor: { actorType: "agent", actorId: this.binding.agentId, agentId: this.binding.agentId, runId: this.binding.runId } }) : undefined;
const write = await documentService(tx).upsertIssueDocument({