mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-15 18:56:05 +01:00
fix(native): settle a restart-orphaned write_document from its transactional receipt (#20674)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run calls Paperclip tools such as `write_document`. The runner saves each tool input durably before the server runs the tool, and the server saves a result receipt > - A control-plane restart can cut a `write_document` call after the runner saved its input and before any result was saved > - Recovery then has no proof of the outcome. A call from an earlier server process settles as "unknown outcome" (#20628). A call the current server process admitted stays pending, close cannot prove that the provider drained, and the session quarantines (`native_session_cleanup_quarantined`). A person must click Retry > - `write_document` saves its revision and its idempotency receipt in one database transaction, under the run and issue row locks. The revision is also compare-and-swap on `baseRevisionId`. So the receipt is a durable outcome record: present means applied, absent means not applied > - This pull request makes recovery use that proof. Recovery runs the same receipt-guarded write. It replays the saved result, or it applies the write exactly once > - The benefit is that an orphaned `write_document` settles with its real result. The document is never written twice, and the run does not need Retry for this call ## Linked Issues or Issue Description **What happened?** Daytona restart-chaos wave `w-20261110T174930-s9205`, task t01 (codex, workload C1, fault RR2). A restart cut a `write_document` call. The close settlement showed `semanticTools.pending: [{ operationId: "write_document", callId: "exec-8b79ab78-…" }]` and `providerDrained: false`. The session quarantined with `native_session_cleanup_quarantined`, and the task needed Retry. Since #20628, a call that an earlier server process admitted settles as `tool_call_interrupted_by_restart` with an unknown outcome. The agent then has to guess whether the write applied. **Expected behavior** When a durable record proves the outcome, recovery uses it. A write that applied returns its saved result. A write that did not apply runs once. When the record cannot prove it, the call stays as it is today. **Steps to reproduce** 1. On master, run `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/paperclip-runner-tool-authority.test.ts -t "orphaned"`. 2. `authority.recover()` returns `null` for a `write_document` that has no receipt, so the call stays unsettled. Refs #20628, #20639, #20507 ## What Changed - `PaperclipRunnerToolAuthority.recover()` now settles `write_document` through its transactional receipt (`#recoverTransactionalWrite`). It runs the same receipt-guarded mutation as an ordinary call. If the receipt exists, it returns the saved result and writes nothing. If no receipt exists, it applies the write once. - A definite refusal that rolls the transaction back (a 4xx such as a stale `baseRevisionId`, a reused idempotency key, invalid input) settles as a tool error. Any other failure leaves the call unsettled, as before. - The set of operations with this proof is explicit: `TRANSACTIONAL_RECEIPT_OPERATIONS` contains only `write_document`. Other tools keep their current recovery rules. - New test: an orphaned `write_document` with no receipt applies once; a second recovery and a late original call replay the receipt; an applied write recovers its saved result; a stale base settles as a tool error with no change. - Updated test: a reused idempotency key with other input now settles as an `paperclip_runner_tool_idempotency_conflict` tool error instead of staying pending. The receipt proves that this call can never apply under that key. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` (semantic operations that outlive a turn) and a new design note, `doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md`. The note also covers the second change (an ACPX turn start cut off before the prompt reached the provider), which comes in a separate pull request. ## Verification - Red on master, green here: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/paperclip-runner-tool-authority.test.ts` (43 passed). - `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`: 178 passed. One unrelated ACPX timing test failed once under load and passed when run alone. - Server typecheck: `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify` exits 2 (a runner doc changed), so required CI supplies the broad coverage. - Live restart-chaos wave: see the comment on this pull request. ## Risks - Recovery can now apply a write that never applied before the restart. This is the call the agent made, applied once. The write is compare-and-swap on `baseRevisionId`, so it cannot overwrite a revision that someone wrote after the agent read the document. That case settles as a conflict error. - Upgrade safety: no saved shape changes. Receipts saved by the previous release have the same shape. A server on the previous release keeps its current recovery for these calls. - Time to first token: no change. This code runs only during recovery. - No runner source changes, so no new runner image is needed. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file editing) in Claude Code. Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
1 parent
cec903c6f0
commit
e33ede9a8a
6 files changed
+192
-2
No files matched your search
+4
-1
@@ -929,8 +929,11 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
|
||||
} } }).where(eq(heartbeatRuns.id, fixture.runId));
|
||||
await expect(recover(call))
|
||||
.resolves.toEqual({ result: { revision: "revision-1" } });
|
||||
// write_document's receipt commits with its revision, so a key whose
|
||||
// receipt holds other input proves this call can never apply under it.
|
||||
await expect(recover({ ...call, input: { ...call.input, body: "changed" } }))
|
||||
.resolves.toBeNull();
|
||||
.resolves.toMatchObject({ isError: true, result: { success: false, contentItems: [
|
||||
{ type: "inputText", text: expect.stringContaining("paperclip_runner_tool_idempotency_conflict") }] } });
|
||||
await expect(recover({ ...call, operationId: "unknown_external_write" }))
|
||||
.resolves.toBeNull();
|
||||
expect(executeRead).not.toHaveBeenCalled();
|
||||
|
||||
@@ -1314,6 +1314,72 @@ describe("PaperclipRunnerToolAuthority", () => {
|
||||
).rejects.toThrow("paperclip_runner_tool_idempotency_conflict");
|
||||
});
|
||||
|
||||
it("settles a write_document a restart orphaned from its transactional receipt, applying it at most once", async () => {
|
||||
// A restart can cut a write_document callback after the runner journaled
|
||||
// its input but before any result was saved. The revision and its receipt
|
||||
// commit in one transaction, so recovery can prove which side it is on.
|
||||
const authority = new PaperclipRunnerToolAuthority(db, { companyId, agentId, issueId, runId });
|
||||
const notApplied = {
|
||||
tool: "write_document",
|
||||
callId: "orphaned-write-not-applied",
|
||||
arguments: { idempotencyKey: "orphaned-write-not-applied", key: "orphan-brief", title: "Brief",
|
||||
body: "Recovered once.", baseRevisionId: null, changeSummary: "Initial brief" },
|
||||
};
|
||||
// No receipt: the write provably never applied. Recovery applies it once.
|
||||
const recovered = await authority.recover(notApplied);
|
||||
expect(recovered).toMatchObject({ result: { disposition: "applied", created: true, document: { key: "orphan-brief" } } });
|
||||
expect(recovered?.isError).not.toBe(true);
|
||||
const applied = await documentService(db).getIssueDocumentByKey(issueId, "orphan-brief");
|
||||
expect(applied).toMatchObject({ body: "Recovered once.", latestRevisionNumber: 1 });
|
||||
// A second recovery, and the original callback arriving late, both replay
|
||||
// the receipt instead of writing again.
|
||||
expect(await authority.recover(notApplied)).toEqual(recovered);
|
||||
expect(await authority.execute(notApplied)).toEqual(recovered!.result);
|
||||
expect(await documentService(db).getIssueDocumentByKey(issueId, "orphan-brief"))
|
||||
.toMatchObject({ latestRevisionNumber: 1 });
|
||||
|
||||
// Receipt present: the write applied before the restart. Recovery returns
|
||||
// the saved result and writes nothing.
|
||||
const update = {
|
||||
tool: "write_document",
|
||||
callId: "orphaned-write-applied",
|
||||
arguments: { idempotencyKey: "orphaned-write-applied", key: "orphan-brief", title: "Brief",
|
||||
body: "Second revision.", baseRevisionId: applied!.latestRevisionId, changeSummary: "Update" },
|
||||
};
|
||||
const original = await authority.execute(update);
|
||||
expect(await authority.recover(update)).toEqual({ result: original });
|
||||
expect(await documentService(db).getIssueDocumentByKey(issueId, "orphan-brief"))
|
||||
.toMatchObject({ body: "Second revision.", latestRevisionNumber: 2 });
|
||||
|
||||
// A write whose base moved on cannot apply. Recovery reports the definite
|
||||
// conflict as a tool error and changes nothing.
|
||||
const stale = {
|
||||
tool: "write_document",
|
||||
callId: "orphaned-write-stale",
|
||||
arguments: { idempotencyKey: "orphaned-write-stale", key: "orphan-brief", title: "Brief",
|
||||
body: "Stale.", baseRevisionId: applied!.latestRevisionId, changeSummary: "Stale" },
|
||||
};
|
||||
// A task switched to ask mode refuses a write that never applied, while
|
||||
// a write that did apply still replays its receipt.
|
||||
await db.update(issues).set({ workMode: "ask" }).where(eq(issues.id, issueId));
|
||||
try {
|
||||
const askMode = await authority.recover({ ...notApplied, callId: "orphaned-write-ask",
|
||||
arguments: { ...notApplied.arguments, idempotencyKey: "orphaned-write-ask", key: "orphan-ask" } });
|
||||
expect(JSON.stringify(askMode)).toContain("paperclip_runner_tool_mode_denied");
|
||||
expect(askMode).toMatchObject({ isError: true });
|
||||
expect(await documentService(db).getIssueDocumentByKey(issueId, "orphan-ask")).toBeNull();
|
||||
expect(await authority.recover(update)).toEqual({ result: original });
|
||||
} finally {
|
||||
await db.update(issues).set({ workMode: "standard" }).where(eq(issues.id, issueId));
|
||||
}
|
||||
|
||||
const conflict = await authority.recover(stale);
|
||||
expect(conflict).toMatchObject({ isError: true, result: { success: false } });
|
||||
expect(JSON.stringify(conflict)).toContain("Document was updated by someone else");
|
||||
expect(await documentService(db).getIssueDocumentByKey(issueId, "orphan-brief"))
|
||||
.toMatchObject({ body: "Second revision.", latestRevisionNumber: 2 });
|
||||
});
|
||||
|
||||
it("returns the exact accepted plan revision in task context", async () => {
|
||||
const plan = await documentService(db).getIssueDocumentByKey(
|
||||
issueId,
|
||||
|
||||
@@ -174,6 +174,19 @@ const RECOVERABLE_READ_OPERATIONS = new Set([
|
||||
READ_CURRENT_WAKE_COMMENTS_TOOL_NAME, LIST_CHAT_ATTACHMENTS_TOOL_NAME, READ_CHAT_ATTACHMENT_TOOL_NAME,
|
||||
READ_TOOL_RESULT, "connections_search",
|
||||
]);
|
||||
// Writes whose effect and idempotency receipt commit in one transaction, with
|
||||
// no outside effect before commit, and whose effect is compare-and-swap on the
|
||||
// state it read. Recovery may run them again: the receipt proves whether the
|
||||
// first attempt applied (see #recoverTransactionalWrite).
|
||||
const TRANSACTIONAL_RECEIPT_OPERATIONS = new Set(["write_document"]);
|
||||
const TRANSACTIONAL_RECOVERY_REFUSALS = new Set([
|
||||
"paperclip_runner_tool_idempotency_conflict", "paperclip_runner_tool_input_invalid",
|
||||
"paperclip_runner_tool_mode_denied",
|
||||
]);
|
||||
function nativeToolAllowedInMode(operationId: string, workMode: string): boolean {
|
||||
return CAPABILITY_SEMANTIC_TOOL_CATALOG.some(candidate => candidate.operationId === operationId
|
||||
&& candidate.allowedModes.includes(workMode as "standard" | "planning" | "ask"));
|
||||
}
|
||||
const RECOVERABLE_READ_ERRORS = new Set([
|
||||
"paperclip_runner_document_not_found", "paperclip_runner_agent_not_found",
|
||||
"paperclip_runner_approval_not_found", "paperclip_runner_tool_input_invalid",
|
||||
@@ -477,6 +490,7 @@ export class PaperclipRunnerToolAuthority {
|
||||
if (!Object.hasOwn(receipt, "result")) return null;
|
||||
return { result: await this.#withInstructionMutationReceipt(call.tool, call.callId, input, true) };
|
||||
}
|
||||
if (TRANSACTIONAL_RECEIPT_OPERATIONS.has(call.tool)) return this.#recoverTransactionalWrite(call.tool, input);
|
||||
// Recovery goes through the receipt owner: it must use the same parsed
|
||||
// input, current authority and safe post-commit dispatch as an ordinary
|
||||
// retry, without ever falling through to an uncertain mutation.
|
||||
@@ -541,6 +555,32 @@ export class PaperclipRunnerToolAuthority {
|
||||
+ " Call list_agents and copy a valid id exactly; nothing was created or changed.");
|
||||
}
|
||||
|
||||
/**
|
||||
* Settle a transactional write whose callback a restart cut off. The effect
|
||||
* and its idempotency receipt commit in one transaction under the run and
|
||||
* issue row locks, so the receipt is the durable outcome record: present
|
||||
* means applied, absent means not applied. Running the same receipt-guarded
|
||||
* mutation therefore either replays the saved result or applies the write
|
||||
* for the first time, never twice. A definite pre-commit refusal (a stale
|
||||
* base, a reused key, invalid input) is the call's real outcome and returns
|
||||
* as a tool error; any other failure stays unsettled.
|
||||
*/
|
||||
async #recoverTransactionalWrite(tool: string, input: Record<string, unknown>) {
|
||||
try {
|
||||
if (tool === "write_document") return { result: await this.#writeDocument(input, true) };
|
||||
throw new Error(`paperclip_runner_transactional_recovery_unsupported: ${tool}`);
|
||||
} catch (error) {
|
||||
// Each of these is raised before the transaction commits, so it rolls the
|
||||
// write back: the call's outcome is this refusal, not an unknown one.
|
||||
const definite = error instanceof HttpError ? error.status >= 400 && error.status < 500
|
||||
: error instanceof Error && TRANSACTIONAL_RECOVERY_REFUSALS.has(error.message);
|
||||
if (!definite) throw error;
|
||||
return { isError: true, result: {
|
||||
success: false, contentItems: [{ type: "inputText", text: (error as Error).message }],
|
||||
} };
|
||||
}
|
||||
}
|
||||
|
||||
async #recoverRead(call: Parameters<PaperclipRunnerToolAuthority["execute"]>[0]) {
|
||||
try {
|
||||
return { result: await this.execute(call) };
|
||||
@@ -1850,10 +1890,16 @@ export class PaperclipRunnerToolAuthority {
|
||||
return result;
|
||||
}
|
||||
|
||||
async #writeDocument(input: Record<string, unknown>): Promise<unknown> {
|
||||
async #writeDocument(input: Record<string, unknown>, recovering = false): Promise<unknown> {
|
||||
const idempotencyKey = requiredString(input.idempotencyKey);
|
||||
let publication: Awaited<ReturnType<typeof persistActivity>>["publication"] | null = null;
|
||||
const result = await this.#withMutationReceipt("write_document", idempotencyKey, input, async (tx, context) => {
|
||||
// execute() checks the work mode before dispatch. A recovered write
|
||||
// applies later, so it checks the mode it locked; a saved receipt
|
||||
// still replays without this check because its effect already applied.
|
||||
if (recovering && !nativeToolAllowedInMode("write_document", context.issue.workMode)) {
|
||||
throw new Error("paperclip_runner_tool_mode_denied");
|
||||
}
|
||||
const sourceTrust = this.binding.museRuntime ? await resolveActorSourceTrustForIssue({ db: tx, issue: context.issue,
|
||||
actor: { actorType: "agent", actorId: this.binding.agentId, agentId: this.binding.agentId, runId: this.binding.runId } }) : undefined;
|
||||
const write = await documentService(tx).upsertIssueDocument({
|
||||
|
||||
Reference in new issue
Block a user