mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-15 18:56:05 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #20645 lets a local same-run scheduled resume send the one same-conversation continuation for a turn lost on restore. It applies only when durable records prove three facts: the previous runner stopped, the attempt was admitted within the budget, and it is the same checkpointed conversation. > - It read the third proof from the coordinator's saved `failure_detail` while the phase was `retryable_failure`. > - In production, the resume reconciler (`claimNativeSessionResumptions`) claims the run first. It moves the coordinator to `observed` and clears `failure_detail`. So the executor never saw that evidence, and #20645 refused every real scheduled resume. > - The restart-chaos L7 wave showed this live. Codex B5 TM2 hit the exact path (`runner.drain` timeout, scheduled resume, `provider_turn_lost_on_restore`), sent no continuation, and needed Retry. > - This pull request makes the reconciler record that evidence durably when it claims the run, and makes the executor read it from there. > - The benefit is that #20645 works on the real path. The proof rule is unchanged. ## Linked Issues or Issue Description Refs #20645 **What happened?** Wave `w-20261110T212307-s9107` (L7, seed 9107, local, master plus #20645), task codex B5 TM2: - `native_session_recovery_failed: runner_local_connect_failed: PRP command runner.drain timed out`; - `retryable_failure` with `recoveryMode: "exact_checkpoint_resume"`, `recoverable`, `providerSessionEstablished` and `checkpointExists` all set; - `native.local_process_stopped`; - attempt 2 started; - `turn.failed` with `provider_turn_lost_on_restore` for the same turn ID that the checkpoint was running. No continuation was sent. The run failed with `native_provider_terminal_failed` and was preserved with `uncertain_provider_action`. **Expected behavior** The proven scheduled resume sends the single continuation. **Steps to reproduce** The new integration test below. The live path is timing-dependent (the L7 command in #20645). ## What Changed - `native-finalization-reconciler.ts`: when it claims a same-run resume, the reconciler appends one `same_run_resume` entry to `recovery_history`. The append is bounded like other history writes, and it runs in the same update that clears the failure. The entry copies `recoveryMode`, `recoverable`, `providerSessionEstablished`, `checkpointExists`, `successorRunId` and `replacementDenied` from the saved failure, with the controller boot and provider attempt. - `scheduled-resume-continuation.ts`: the evidence for the third proof now comes from either of two durable records: - the saved failure, when the executor claims a `retryable_failure` coordinator directly (unchanged); or - a reconciler claim: an `observed` coordinator whose last history entry is a `same_run_resume` entry naming the executor's own controller boot. The two other proofs are unchanged: the stop receipt, and an admitted attempt from 1 to 3. - `native-session-executor.ts`: passes the coordinator's last history entry and the controller boot to the decision. - `doc/execution-semantics.md`: describes the recorded evidence. Readers of `recovery_history` that look at the last entry do so only for restart claims, Stop recovery and shutdown suspension. Restart recovery's previous-claim lookup skips non-claim dispositions, so a `same_run_resume` entry does not change those paths. **Still refused** (tested): - every #20645 case; - the last history entry is not the resume claim, or there is no history; - the resume claim was recorded by another boot; - the claimed evidence is a bootstrap retry, has no checkpoint, or records a replacement; - a reconciler-claimed resume without a stop receipt. ## Verification - `native-scheduled-resume-continuation.integration.test.ts` now also runs through the real reconciler claim (`claimNativeSessionResumptions`) before `executePaperclipNativeSession`. On master the reconciler-claimed allowed case fails (`resumeInterruptedTurn` is false). With this change it is true, and the claim admits attempt 2. A reconciler-claimed resume without a stop receipt still passes false. - `scheduled-resume-continuation.test.ts`: decision cases for the reconciler-claim entry. - `native-session-resumption.test.ts` passes. 54 tests pass across the three files. - `cd server && npx tsc --noEmit` passes. - Live: not yet. The failing wave above is the reproduction. A rerun is timing-dependent. ## Risks - One more `recovery_history` entry per scheduled resume, inside the existing 20-entry bound. - The proof rule itself is unchanged. This only reads the same evidence from the record where it now survives. - No schema change. A run claimed by an earlier build has no `same_run_resume` entry, so it keeps the existing failure: refused, with Retry offered as today. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). Extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
114 lines
5.4 KiB
TypeScript
114 lines
5.4 KiB
TypeScript
/**
|
|
* The interrupted provider attempt that a same-run exact-checkpoint retry was
|
|
* resuming when the retry failed before it took its own provider claim.
|
|
*/
|
|
export type InterruptedExactResume = {
|
|
originalFailureCode: string;
|
|
originalMessage: string;
|
|
interruptedAttempt: number;
|
|
};
|
|
|
|
/** Bounds the original message in the chained message and in the run result. */
|
|
const ORIGINAL_MESSAGE_MAX_CHARS = 1_500;
|
|
/** The coordinator failure detail keeps at most this many characters. */
|
|
const FAILURE_DETAIL_MAX_CHARS = 2_000;
|
|
/** Leaves the refusal readable however long the original message is. */
|
|
const REFUSAL_MAX_CHARS = 480;
|
|
const REFUSAL_SEPARATOR = "; exact checkpoint resume refused: ";
|
|
/** Restart-recovery waits that keep the interrupted provider attempt. */
|
|
const RECOVERY_WAIT_DISPOSITIONS: ReadonlySet<unknown> = new Set(["blocked", "awaiting_evidence", "same_run_resume"]);
|
|
|
|
/**
|
|
* Cuts to at most `max` UTF-16 code units without splitting a surrogate pair.
|
|
* A lone surrogate is invalid Unicode, and PostgreSQL `jsonb` rejects it, so
|
|
* a split emoji would make the failure write itself fail.
|
|
*/
|
|
function cutWholeCharacters(text: string, max: number): string {
|
|
if (text.length <= max) return text;
|
|
const last = max > 0 ? text.charCodeAt(max - 1) : 0;
|
|
return text.slice(0, last >= 0xd800 && last <= 0xdbff ? max - 1 : Math.max(0, max));
|
|
}
|
|
|
|
function record(value: unknown): Record<string, unknown> {
|
|
return value && typeof value === "object" && !Array.isArray(value)
|
|
? (value as Record<string, unknown>)
|
|
: {};
|
|
}
|
|
|
|
/**
|
|
* A same-run resume claim clears the coordinator failure fields, so the
|
|
* interrupted attempt survives only in its recovery history entry. That entry
|
|
* explains the retry only while the claim is still `observed` and no later
|
|
* provider attempt was claimed: the executor increments `attempt` when it
|
|
* claims, and it records its own failures for that attempt. A retryable entry
|
|
* with a checkpoint was an exact-checkpoint resume, because a bootstrap retry
|
|
* has no checkpoint and an ambiguous state is never retryable. Restart
|
|
* recovery can record waits for the same attempt after that entry (for
|
|
* example a held remote retry); those are skipped. Any other entry, or any
|
|
* entry for a later provider attempt, ends the search without a cause.
|
|
*/
|
|
export function interruptedExactResumeCause(coordinator: {
|
|
phase: string;
|
|
attempt: number;
|
|
resultId: string | null;
|
|
recoveryHistory: unknown;
|
|
} | null | undefined): InterruptedExactResume | null {
|
|
if (!coordinator || coordinator.phase !== "observed" || coordinator.resultId !== null || coordinator.attempt < 1) return null;
|
|
const history = Array.isArray(coordinator.recoveryHistory) ? coordinator.recoveryHistory : [];
|
|
let entry: Record<string, unknown> = {};
|
|
for (let index = history.length - 1; index >= 0; index -= 1) {
|
|
entry = record(history[index]);
|
|
if (!RECOVERY_WAIT_DISPOSITIONS.has(entry.disposition) || entry.providerAttempt !== coordinator.attempt) break;
|
|
}
|
|
if (
|
|
entry.disposition !== "retryable_failure" ||
|
|
entry.providerAttempt !== coordinator.attempt ||
|
|
entry.checkpointExists !== true ||
|
|
entry.providerSessionEstablished !== true ||
|
|
typeof entry.reason !== "string" || !entry.reason
|
|
) return null;
|
|
// The executor removed credentials and bounded this tail before it
|
|
// persisted the entry. It is returned whole so the caller can mask
|
|
// usernames before any length budget cuts it.
|
|
const message = typeof entry.stderrTail === "string" ? entry.stderrTail.trim() : "";
|
|
return {
|
|
originalFailureCode: entry.reason,
|
|
originalMessage: message,
|
|
interruptedAttempt: coordinator.attempt,
|
|
};
|
|
}
|
|
|
|
/** The original message as the run result keeps it. Mask it before calling this. */
|
|
export function boundOriginalMessage(message: string): string {
|
|
return cutWholeCharacters(message, ORIGINAL_MESSAGE_MAX_CHARS);
|
|
}
|
|
|
|
/**
|
|
* The run's current failure text. A refused exact resume reports the
|
|
* interrupted attempt in `error` for operators in company-scoped run records,
|
|
* but retry and recovery decisions and Sentry failure reports must use only
|
|
* the current refusal, which the run result keeps separately. The earlier
|
|
* attempt's provider text stays out of Sentry.
|
|
*/
|
|
export function currentRunFailureText(run: { error?: string | null; resultJson?: unknown } | null | undefined): string | null {
|
|
const refusal = record(record(run?.resultJson).nativeExactResumeRefusal).refusal;
|
|
return typeof refusal === "string" ? refusal : run?.error ?? null;
|
|
}
|
|
|
|
/**
|
|
* Names the interrupted attempt first, then why its exact checkpoint could not
|
|
* resume. The refusal keeps its own budget, and the original message is cut to
|
|
* the room left, so the whole message fits the failure-detail bound. Both
|
|
* texts must already be masked, because a cut can split a username.
|
|
*/
|
|
export function describeRefusedExactResume(cause: InterruptedExactResume, refusal: string): string {
|
|
const refused = `${REFUSAL_SEPARATOR}${cutWholeCharacters(refusal, REFUSAL_MAX_CHARS)}`;
|
|
const originalBudget = Math.max(0, FAILURE_DETAIL_MAX_CHARS - refused.length);
|
|
const code = cutWholeCharacters(cause.originalFailureCode, originalBudget);
|
|
const messageBudget = originalBudget - code.length - 2;
|
|
const original = cause.originalMessage && messageBudget > 0
|
|
? `${code}: ${cutWholeCharacters(cause.originalMessage, Math.min(ORIGINAL_MESSAGE_MAX_CHARS, messageBudget))}`
|
|
: code;
|
|
return `${original}${refused}`;
|
|
}
|