mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 11:54:35 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #20645 lets a local same-run scheduled resume send the one same-conversation continuation for a turn lost on restore. It applies only when durable records prove three facts: the previous runner stopped, the attempt was admitted within the budget, and it is the same checkpointed conversation. > - It read the third proof from the coordinator's saved `failure_detail` while the phase was `retryable_failure`. > - In production, the resume reconciler (`claimNativeSessionResumptions`) claims the run first. It moves the coordinator to `observed` and clears `failure_detail`. So the executor never saw that evidence, and #20645 refused every real scheduled resume. > - The restart-chaos L7 wave showed this live. Codex B5 TM2 hit the exact path (`runner.drain` timeout, scheduled resume, `provider_turn_lost_on_restore`), sent no continuation, and needed Retry. > - This pull request makes the reconciler record that evidence durably when it claims the run, and makes the executor read it from there. > - The benefit is that #20645 works on the real path. The proof rule is unchanged. ## Linked Issues or Issue Description Refs #20645 **What happened?** Wave `w-20261110T212307-s9107` (L7, seed 9107, local, master plus #20645), task codex B5 TM2: - `native_session_recovery_failed: runner_local_connect_failed: PRP command runner.drain timed out`; - `retryable_failure` with `recoveryMode: "exact_checkpoint_resume"`, `recoverable`, `providerSessionEstablished` and `checkpointExists` all set; - `native.local_process_stopped`; - attempt 2 started; - `turn.failed` with `provider_turn_lost_on_restore` for the same turn ID that the checkpoint was running. No continuation was sent. The run failed with `native_provider_terminal_failed` and was preserved with `uncertain_provider_action`. **Expected behavior** The proven scheduled resume sends the single continuation. **Steps to reproduce** The new integration test below. The live path is timing-dependent (the L7 command in #20645). ## What Changed - `native-finalization-reconciler.ts`: when it claims a same-run resume, the reconciler appends one `same_run_resume` entry to `recovery_history`. The append is bounded like other history writes, and it runs in the same update that clears the failure. The entry copies `recoveryMode`, `recoverable`, `providerSessionEstablished`, `checkpointExists`, `successorRunId` and `replacementDenied` from the saved failure, with the controller boot and provider attempt. - `scheduled-resume-continuation.ts`: the evidence for the third proof now comes from either of two durable records: - the saved failure, when the executor claims a `retryable_failure` coordinator directly (unchanged); or - a reconciler claim: an `observed` coordinator whose last history entry is a `same_run_resume` entry naming the executor's own controller boot. The two other proofs are unchanged: the stop receipt, and an admitted attempt from 1 to 3. - `native-session-executor.ts`: passes the coordinator's last history entry and the controller boot to the decision. - `doc/execution-semantics.md`: describes the recorded evidence. Readers of `recovery_history` that look at the last entry do so only for restart claims, Stop recovery and shutdown suspension. Restart recovery's previous-claim lookup skips non-claim dispositions, so a `same_run_resume` entry does not change those paths. **Still refused** (tested): - every #20645 case; - the last history entry is not the resume claim, or there is no history; - the resume claim was recorded by another boot; - the claimed evidence is a bootstrap retry, has no checkpoint, or records a replacement; - a reconciler-claimed resume without a stop receipt. ## Verification - `native-scheduled-resume-continuation.integration.test.ts` now also runs through the real reconciler claim (`claimNativeSessionResumptions`) before `executePaperclipNativeSession`. On master the reconciler-claimed allowed case fails (`resumeInterruptedTurn` is false). With this change it is true, and the claim admits attempt 2. A reconciler-claimed resume without a stop receipt still passes false. - `scheduled-resume-continuation.test.ts`: decision cases for the reconciler-claim entry. - `native-session-resumption.test.ts` passes. 54 tests pass across the three files. - `cd server && npx tsc --noEmit` passes. - Live: not yet. The failing wave above is the reproduction. A rerun is timing-dependent. ## Risks - One more `recovery_history` entry per scheduled resume, inside the existing 20-entry bound. - The proof rule itself is unchanged. This only reads the same evidence from the record where it now survives. - No schema change. A run claimed by an earlier build has no `same_run_resume` entry, so it keeps the existing failure: refused, with Retry offered as today. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). Extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
147 lines
8.3 KiB
TypeScript
147 lines
8.3 KiB
TypeScript
import { describe, expect, it } from "vitest";
|
|
import { boundOriginalMessage, describeRefusedExactResume, currentRunFailureText, interruptedExactResumeCause } from "./native-exact-resume-failure.js";
|
|
|
|
const interruption = {
|
|
at: "2099-01-01T00:00:00.000Z",
|
|
disposition: "retryable_failure",
|
|
reason: "native_session_interrupted",
|
|
providerAttempt: 1,
|
|
stderrTail: "PRP command turn.start failed: classification=session_resume_required",
|
|
providerSessionEstablished: true,
|
|
checkpointExists: true,
|
|
};
|
|
const claimed = { phase: "observed", attempt: 1, resultId: null, recoveryHistory: [interruption] };
|
|
|
|
describe("interruptedExactResumeCause", () => {
|
|
it("returns the interrupted attempt that a pending exact-checkpoint resume claim was resuming", () => {
|
|
expect(interruptedExactResumeCause(claimed)).toEqual({
|
|
originalFailureCode: "native_session_interrupted",
|
|
originalMessage: "PRP command turn.start failed: classification=session_resume_required",
|
|
interruptedAttempt: 1,
|
|
});
|
|
});
|
|
|
|
it("uses only the latest recovery history entry", () => {
|
|
const earlier = { ...interruption, reason: "provider_transport_failed", providerAttempt: 1, stderrTail: "older" };
|
|
expect(interruptedExactResumeCause({ ...claimed, attempt: 2,
|
|
recoveryHistory: [earlier, { ...interruption, providerAttempt: 2 }] })?.originalMessage).toBe(interruption.stderrTail);
|
|
});
|
|
|
|
it.each([
|
|
["no coordinator", null],
|
|
["a first attempt without recovery history", { ...claimed, attempt: 0, recoveryHistory: [] }],
|
|
["an executor-claimed later attempt", { ...claimed, attempt: 2 }],
|
|
["a coordinator that is not a pending claim", { ...claimed, phase: "retryable_failure" }],
|
|
["an accepted result", { ...claimed, resultId: "result" }],
|
|
["a bootstrap retry without a checkpoint", { ...claimed, recoveryHistory: [{ ...interruption, checkpointExists: false, providerSessionEstablished: false }] }],
|
|
["an exhausted terminal entry", { ...claimed, recoveryHistory: [{ ...interruption, disposition: "terminal_failure" }] }],
|
|
["malformed history", { ...claimed, recoveryHistory: { reason: "native_session_interrupted" } }],
|
|
])("returns null for %s", (_name, coordinator) => {
|
|
expect(interruptedExactResumeCause(coordinator)).toBeNull();
|
|
});
|
|
|
|
it("finds the interruption behind restart-recovery waits for the same attempt", () => {
|
|
const blocked = { at: "2099-01-01T00:01:00.000Z", disposition: "blocked", reason: "native_retry_admission_ineligible", providerAttempt: 1 };
|
|
const awaiting = { at: "2099-01-01T00:02:00.000Z", disposition: "awaiting_evidence", reason: "current_controller_preparing", providerAttempt: 1 };
|
|
expect(interruptedExactResumeCause({ ...claimed, recoveryHistory: [interruption, blocked, awaiting] })).toEqual({
|
|
originalFailureCode: "native_session_interrupted",
|
|
originalMessage: "PRP command turn.start failed: classification=session_resume_required",
|
|
interruptedAttempt: 1,
|
|
});
|
|
});
|
|
|
|
it("finds the interruption behind the resume reconciler's claim entry for the same attempt", () => {
|
|
const resumeClaim = { at: "2099-01-01T00:01:00.000Z", disposition: "same_run_resume", controllerBootId: "boot-1",
|
|
providerAttempt: 1, evidence: { recoveryMode: "exact_checkpoint_resume" } };
|
|
expect(interruptedExactResumeCause({ ...claimed, recoveryHistory: [interruption, resumeClaim] })).toEqual({
|
|
originalFailureCode: "native_session_interrupted",
|
|
originalMessage: "PRP command turn.start failed: classification=session_resume_required",
|
|
interruptedAttempt: 1,
|
|
});
|
|
});
|
|
|
|
it.each([
|
|
["a resume claim entry for a later provider attempt", { disposition: "same_run_resume", providerAttempt: 2 }],
|
|
["a wait for a later provider attempt", { disposition: "blocked", reason: "held", providerAttempt: 2 }],
|
|
["a later recovery entry that is not a wait", { disposition: "resume_dead_runner", reason: "restart", providerAttempt: 1 }],
|
|
])("does not reach past %s", (_name, later) => {
|
|
expect(interruptedExactResumeCause({ ...claimed, recoveryHistory: [interruption, later] })).toBeNull();
|
|
});
|
|
|
|
it("returns the whole original message so it can be masked before any cut", () => {
|
|
const cause = interruptedExactResumeCause({ ...claimed, recoveryHistory: [{ ...interruption, stderrTail: "x".repeat(4_096) }] });
|
|
expect(cause?.originalMessage).toHaveLength(4_096);
|
|
expect(boundOriginalMessage(cause!.originalMessage)).toHaveLength(1_500);
|
|
const message = describeRefusedExactResume(cause!, "Plugin Runner retained session is not authorized");
|
|
expect(message.length).toBeLessThanOrEqual(2_000);
|
|
expect(message).toContain(`native_session_interrupted: ${"x".repeat(1_500)}; exact checkpoint`);
|
|
});
|
|
});
|
|
|
|
describe("describeRefusedExactResume", () => {
|
|
it("names the original failure before the refusal", () => {
|
|
expect(describeRefusedExactResume(interruptedExactResumeCause(claimed)!, "Plugin Runner retained session is not authorized"))
|
|
.toBe("native_session_interrupted: PRP command turn.start failed: classification=session_resume_required; "
|
|
+ "exact checkpoint resume refused: Plugin Runner retained session is not authorized");
|
|
});
|
|
|
|
it("keeps the refusal readable when both messages are long", () => {
|
|
const cause = interruptedExactResumeCause({ ...claimed, recoveryHistory: [{ ...interruption, stderrTail: "o".repeat(4_096) }] })!;
|
|
const refusal = `Plugin Runner retained session is not authorized: ${"r".repeat(600)}`;
|
|
const message = describeRefusedExactResume(cause, refusal);
|
|
expect(message).toHaveLength(2_000);
|
|
expect(message.startsWith("native_session_interrupted: ooo")).toBe(true);
|
|
expect(message.endsWith(`; exact checkpoint resume refused: ${refusal.slice(0, 480)}`)).toBe(true);
|
|
});
|
|
|
|
it("keeps a short refusal whole after a maximal original message", () => {
|
|
const cause = interruptedExactResumeCause({ ...claimed, recoveryHistory: [{ ...interruption, stderrTail: "o".repeat(4_096) }] })!;
|
|
const message = describeRefusedExactResume(cause, "Plugin Runner retained session is not authorized");
|
|
expect(message.length).toBeLessThanOrEqual(2_000);
|
|
expect(message.endsWith("; exact checkpoint resume refused: Plugin Runner retained session is not authorized")).toBe(true);
|
|
});
|
|
|
|
it("keeps the original failure code when no message was persisted", () => {
|
|
expect(describeRefusedExactResume({ originalFailureCode: "native_session_interrupted", originalMessage: "", interruptedAttempt: 1 }, "refused"))
|
|
.toBe("native_session_interrupted; exact checkpoint resume refused: refused");
|
|
});
|
|
});
|
|
|
|
describe("currentRunFailureText", () => {
|
|
it("classifies a refused exact resume from its refusal, not the chained error", () => {
|
|
expect(currentRunFailureText({
|
|
error: "native_session_interrupted: spawn agent ENOENT; exact checkpoint resume refused: retained session is not authorized",
|
|
resultJson: { nativeExactResumeRefusal: { refusal: "retained session is not authorized" } },
|
|
})).toBe("retained session is not authorized");
|
|
});
|
|
|
|
it.each([
|
|
["no result", { error: "spawn agent ENOENT", resultJson: null }],
|
|
["a result without a refusal", { error: "spawn agent ENOENT", resultJson: { errorMessage: "other" } }],
|
|
["a malformed refusal", { error: "spawn agent ENOENT", resultJson: { nativeExactResumeRefusal: { refusal: 7 } } }],
|
|
])("uses the run error for %s", (_name, run) => {
|
|
expect(currentRunFailureText(run)).toBe("spawn agent ENOENT");
|
|
});
|
|
});
|
|
|
|
describe("cuts at whole Unicode characters", () => {
|
|
const emoji = "\u{1F642}";
|
|
const wellFormed = (text: string) => !/[\ud800-\udbff](?![\udc00-\udfff])|(?<![\ud800-\udbff])[\udc00-\udfff]/.test(text);
|
|
|
|
it("does not split an emoji at the original-message bound", () => {
|
|
const bounded = boundOriginalMessage(`${"x".repeat(1_499)}${emoji}tail`);
|
|
expect(bounded).toBe("x".repeat(1_499));
|
|
expect(wellFormed(bounded)).toBe(true);
|
|
});
|
|
|
|
it("does not split an emoji at the refusal or dynamic original-message cuts", () => {
|
|
const message = describeRefusedExactResume(
|
|
{ originalFailureCode: "native_session_interrupted", originalMessage: `${"o".repeat(1_449)}${emoji}${"o".repeat(200)}`, interruptedAttempt: 1 },
|
|
`${"r".repeat(479)}${emoji}${"r".repeat(100)}`,
|
|
);
|
|
expect(wellFormed(message)).toBe(true);
|
|
expect(message.endsWith(`; exact checkpoint resume refused: ${"r".repeat(479)}`)).toBe(true);
|
|
expect(message.length).toBeLessThanOrEqual(2_000);
|
|
});
|
|
});
|