Files
PaperClipAI/server/src/services/native-runtime/native-exact-resume-failure.test.ts
T
DottaandClaude c14f2fe45d fix(native): read the scheduled-resume evidence the resume reconciler claimed (#20656)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #20645 lets a local same-run scheduled resume send the one
same-conversation continuation for a turn lost on restore. It applies
only when durable records prove three facts: the previous runner
stopped, the attempt was admitted within the budget, and it is the same
checkpointed conversation.
> - It read the third proof from the coordinator's saved
`failure_detail` while the phase was `retryable_failure`.
> - In production, the resume reconciler
(`claimNativeSessionResumptions`) claims the run first. It moves the
coordinator to `observed` and clears `failure_detail`. So the executor
never saw that evidence, and #20645 refused every real scheduled resume.
> - The restart-chaos L7 wave showed this live. Codex B5 TM2 hit the
exact path (`runner.drain` timeout, scheduled resume,
`provider_turn_lost_on_restore`), sent no continuation, and needed
Retry.
> - This pull request makes the reconciler record that evidence durably
when it claims the run, and makes the executor read it from there.
> - The benefit is that #20645 works on the real path. The proof rule is
unchanged.

## Linked Issues or Issue Description

Refs #20645

**What happened?**
Wave `w-20261110T212307-s9107` (L7, seed 9107, local, master plus
#20645), task codex B5 TM2:
- `native_session_recovery_failed: runner_local_connect_failed: PRP
command runner.drain timed out`;
- `retryable_failure` with `recoveryMode: "exact_checkpoint_resume"`,
`recoverable`, `providerSessionEstablished` and `checkpointExists` all
set;
- `native.local_process_stopped`;
- attempt 2 started;
- `turn.failed` with `provider_turn_lost_on_restore` for the same turn
ID that the checkpoint was running.

No continuation was sent. The run failed with
`native_provider_terminal_failed` and was preserved with
`uncertain_provider_action`.

**Expected behavior**
The proven scheduled resume sends the single continuation.

**Steps to reproduce**
The new integration test below. The live path is timing-dependent (the
L7 command in #20645).

## What Changed

- `native-finalization-reconciler.ts`: when it claims a same-run resume,
the reconciler appends one `same_run_resume` entry to
`recovery_history`. The append is bounded like other history writes, and
it runs in the same update that clears the failure. The entry copies
`recoveryMode`, `recoverable`, `providerSessionEstablished`,
`checkpointExists`, `successorRunId` and `replacementDenied` from the
saved failure, with the controller boot and provider attempt.
- `scheduled-resume-continuation.ts`: the evidence for the third proof
now comes from either of two durable records:
- the saved failure, when the executor claims a `retryable_failure`
coordinator directly (unchanged); or
- a reconciler claim: an `observed` coordinator whose last history entry
is a `same_run_resume` entry naming the executor's own controller boot.
The two other proofs are unchanged: the stop receipt, and an admitted
attempt from 1 to 3.
- `native-session-executor.ts`: passes the coordinator's last history
entry and the controller boot to the decision.
- `doc/execution-semantics.md`: describes the recorded evidence.

Readers of `recovery_history` that look at the last entry do so only for
restart claims, Stop recovery and shutdown suspension. Restart
recovery's previous-claim lookup skips non-claim dispositions, so a
`same_run_resume` entry does not change those paths.

**Still refused** (tested):
- every #20645 case;
- the last history entry is not the resume claim, or there is no
history;
- the resume claim was recorded by another boot;
- the claimed evidence is a bootstrap retry, has no checkpoint, or
records a replacement;
- a reconciler-claimed resume without a stop receipt.

## Verification

- `native-scheduled-resume-continuation.integration.test.ts` now also
runs through the real reconciler claim (`claimNativeSessionResumptions`)
before `executePaperclipNativeSession`. On master the reconciler-claimed
allowed case fails (`resumeInterruptedTurn` is false). With this change
it is true, and the claim admits attempt 2. A reconciler-claimed resume
without a stop receipt still passes false.
- `scheduled-resume-continuation.test.ts`: decision cases for the
reconciler-claim entry.
- `native-session-resumption.test.ts` passes. 54 tests pass across the
three files.
- `cd server && npx tsc --noEmit` passes.
- Live: not yet. The failing wave above is the reproduction. A rerun is
timing-dependent.

## Risks

- One more `recovery_history` entry per scheduled resume, inside the
existing 20-entry bound.
- The proof rule itself is unchanged. This only reads the same evidence
from the record where it now survives.
- No schema change. A run claimed by an earlier build has no
`same_run_resume` entry, so it keeps the existing failure: refused, with
Retry offered as today.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs). Extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 18:21:25 -06:00

147 lines
8.3 KiB
TypeScript

import { describe, expect, it } from "vitest";
import { boundOriginalMessage, describeRefusedExactResume, currentRunFailureText, interruptedExactResumeCause } from "./native-exact-resume-failure.js";
const interruption = {
at: "2099-01-01T00:00:00.000Z",
disposition: "retryable_failure",
reason: "native_session_interrupted",
providerAttempt: 1,
stderrTail: "PRP command turn.start failed: classification=session_resume_required",
providerSessionEstablished: true,
checkpointExists: true,
};
const claimed = { phase: "observed", attempt: 1, resultId: null, recoveryHistory: [interruption] };
describe("interruptedExactResumeCause", () => {
it("returns the interrupted attempt that a pending exact-checkpoint resume claim was resuming", () => {
expect(interruptedExactResumeCause(claimed)).toEqual({
originalFailureCode: "native_session_interrupted",
originalMessage: "PRP command turn.start failed: classification=session_resume_required",
interruptedAttempt: 1,
});
});
it("uses only the latest recovery history entry", () => {
const earlier = { ...interruption, reason: "provider_transport_failed", providerAttempt: 1, stderrTail: "older" };
expect(interruptedExactResumeCause({ ...claimed, attempt: 2,
recoveryHistory: [earlier, { ...interruption, providerAttempt: 2 }] })?.originalMessage).toBe(interruption.stderrTail);
});
it.each([
["no coordinator", null],
["a first attempt without recovery history", { ...claimed, attempt: 0, recoveryHistory: [] }],
["an executor-claimed later attempt", { ...claimed, attempt: 2 }],
["a coordinator that is not a pending claim", { ...claimed, phase: "retryable_failure" }],
["an accepted result", { ...claimed, resultId: "result" }],
["a bootstrap retry without a checkpoint", { ...claimed, recoveryHistory: [{ ...interruption, checkpointExists: false, providerSessionEstablished: false }] }],
["an exhausted terminal entry", { ...claimed, recoveryHistory: [{ ...interruption, disposition: "terminal_failure" }] }],
["malformed history", { ...claimed, recoveryHistory: { reason: "native_session_interrupted" } }],
])("returns null for %s", (_name, coordinator) => {
expect(interruptedExactResumeCause(coordinator)).toBeNull();
});
it("finds the interruption behind restart-recovery waits for the same attempt", () => {
const blocked = { at: "2099-01-01T00:01:00.000Z", disposition: "blocked", reason: "native_retry_admission_ineligible", providerAttempt: 1 };
const awaiting = { at: "2099-01-01T00:02:00.000Z", disposition: "awaiting_evidence", reason: "current_controller_preparing", providerAttempt: 1 };
expect(interruptedExactResumeCause({ ...claimed, recoveryHistory: [interruption, blocked, awaiting] })).toEqual({
originalFailureCode: "native_session_interrupted",
originalMessage: "PRP command turn.start failed: classification=session_resume_required",
interruptedAttempt: 1,
});
});
it("finds the interruption behind the resume reconciler's claim entry for the same attempt", () => {
const resumeClaim = { at: "2099-01-01T00:01:00.000Z", disposition: "same_run_resume", controllerBootId: "boot-1",
providerAttempt: 1, evidence: { recoveryMode: "exact_checkpoint_resume" } };
expect(interruptedExactResumeCause({ ...claimed, recoveryHistory: [interruption, resumeClaim] })).toEqual({
originalFailureCode: "native_session_interrupted",
originalMessage: "PRP command turn.start failed: classification=session_resume_required",
interruptedAttempt: 1,
});
});
it.each([
["a resume claim entry for a later provider attempt", { disposition: "same_run_resume", providerAttempt: 2 }],
["a wait for a later provider attempt", { disposition: "blocked", reason: "held", providerAttempt: 2 }],
["a later recovery entry that is not a wait", { disposition: "resume_dead_runner", reason: "restart", providerAttempt: 1 }],
])("does not reach past %s", (_name, later) => {
expect(interruptedExactResumeCause({ ...claimed, recoveryHistory: [interruption, later] })).toBeNull();
});
it("returns the whole original message so it can be masked before any cut", () => {
const cause = interruptedExactResumeCause({ ...claimed, recoveryHistory: [{ ...interruption, stderrTail: "x".repeat(4_096) }] });
expect(cause?.originalMessage).toHaveLength(4_096);
expect(boundOriginalMessage(cause!.originalMessage)).toHaveLength(1_500);
const message = describeRefusedExactResume(cause!, "Plugin Runner retained session is not authorized");
expect(message.length).toBeLessThanOrEqual(2_000);
expect(message).toContain(`native_session_interrupted: ${"x".repeat(1_500)}; exact checkpoint`);
});
});
describe("describeRefusedExactResume", () => {
it("names the original failure before the refusal", () => {
expect(describeRefusedExactResume(interruptedExactResumeCause(claimed)!, "Plugin Runner retained session is not authorized"))
.toBe("native_session_interrupted: PRP command turn.start failed: classification=session_resume_required; "
+ "exact checkpoint resume refused: Plugin Runner retained session is not authorized");
});
it("keeps the refusal readable when both messages are long", () => {
const cause = interruptedExactResumeCause({ ...claimed, recoveryHistory: [{ ...interruption, stderrTail: "o".repeat(4_096) }] })!;
const refusal = `Plugin Runner retained session is not authorized: ${"r".repeat(600)}`;
const message = describeRefusedExactResume(cause, refusal);
expect(message).toHaveLength(2_000);
expect(message.startsWith("native_session_interrupted: ooo")).toBe(true);
expect(message.endsWith(`; exact checkpoint resume refused: ${refusal.slice(0, 480)}`)).toBe(true);
});
it("keeps a short refusal whole after a maximal original message", () => {
const cause = interruptedExactResumeCause({ ...claimed, recoveryHistory: [{ ...interruption, stderrTail: "o".repeat(4_096) }] })!;
const message = describeRefusedExactResume(cause, "Plugin Runner retained session is not authorized");
expect(message.length).toBeLessThanOrEqual(2_000);
expect(message.endsWith("; exact checkpoint resume refused: Plugin Runner retained session is not authorized")).toBe(true);
});
it("keeps the original failure code when no message was persisted", () => {
expect(describeRefusedExactResume({ originalFailureCode: "native_session_interrupted", originalMessage: "", interruptedAttempt: 1 }, "refused"))
.toBe("native_session_interrupted; exact checkpoint resume refused: refused");
});
});
describe("currentRunFailureText", () => {
it("classifies a refused exact resume from its refusal, not the chained error", () => {
expect(currentRunFailureText({
error: "native_session_interrupted: spawn agent ENOENT; exact checkpoint resume refused: retained session is not authorized",
resultJson: { nativeExactResumeRefusal: { refusal: "retained session is not authorized" } },
})).toBe("retained session is not authorized");
});
it.each([
["no result", { error: "spawn agent ENOENT", resultJson: null }],
["a result without a refusal", { error: "spawn agent ENOENT", resultJson: { errorMessage: "other" } }],
["a malformed refusal", { error: "spawn agent ENOENT", resultJson: { nativeExactResumeRefusal: { refusal: 7 } } }],
])("uses the run error for %s", (_name, run) => {
expect(currentRunFailureText(run)).toBe("spawn agent ENOENT");
});
});
describe("cuts at whole Unicode characters", () => {
const emoji = "\u{1F642}";
const wellFormed = (text: string) => !/[\ud800-\udbff](?![\udc00-\udfff])|(?<![\ud800-\udbff])[\udc00-\udfff]/.test(text);
it("does not split an emoji at the original-message bound", () => {
const bounded = boundOriginalMessage(`${"x".repeat(1_499)}${emoji}tail`);
expect(bounded).toBe("x".repeat(1_499));
expect(wellFormed(bounded)).toBe(true);
});
it("does not split an emoji at the refusal or dynamic original-message cuts", () => {
const message = describeRefusedExactResume(
{ originalFailureCode: "native_session_interrupted", originalMessage: `${"o".repeat(1_449)}${emoji}${"o".repeat(200)}`, interruptedAttempt: 1 },
`${"r".repeat(479)}${emoji}${"r".repeat(100)}`,
);
expect(wellFormed(message)).toBe(true);
expect(message.endsWith(`; exact checkpoint resume refused: ${"r".repeat(479)}`)).toBe(true);
expect(message.length).toBeLessThanOrEqual(2_000);
});
});