mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 21:59:38 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs retain their provider conversation across server restarts. > - A dead runner can restore a Codex conversation after its active turn is lost. > - The old recovery path synthesized a failed task result from that interruption. > - The task then required operator action even though its conversation and workspace were available. > - This pull request preserves the interruption cause and uses the admitted restart attempt for one continuation in the same conversation. > - The agent can reconcile unfinished actions and complete the current request without resending the original task. ## Linked Issues or Issue Description Related: #12845 added native restart recovery. #15042 covers admission during shutdown. #14796 covers legacy shutdown recovery. This change covers a lost native Codex turn after successful conversation restoration. **What happened?** After a server restart killed the local runner, Paperclip restored the saved Codex thread. Runnerd found that the old turn was no longer active. It synthesized a failed terminal and a needs-review result from the last progress message. The transport also discarded the terminal error when it reconstructed thread history. The task failed instead of continuing. **Expected behavior** After proving that the old process stopped and admitting a bounded recovery attempt, resume the current request in the same conversation. Preserve the workspace. Inspect unfinished actions before proceeding. Keep real provider failures, accepted results, intentional stops, unknown unreconciled effects, and exhausted attempts subject to their existing rules. **Steps to reproduce** 1. Start a local native Codex run and leave its turn active. 2. Kill the isolated runner and provider processes, as can happen during a server restart. 3. Restore the same provider thread with no active turn. 4. Observe the synthetic task failure. The new real-process regression reproduces this boundary with a scripted provider. ## What Changed - Record an explicit recoverable process-loss cause without inventing a task result. - Preserve terminal errors and prior turns in reconstructed provider history. Recover the authoritative saved result when adopting an accepted continuation. - Send one continuation in the same conversation for an admitted dead-runner recovery. Require reconciliation of unfinished commands and external actions. - Persist the interrupted terminal before submission and retain the existing recovery marker across another controller loss. - Keep provider attempt limits, terminal failures, and intentional cancellation behavior. - Add red/green regressions, real process-kill coverage, restart checkpoint coverage, and retry-budget coverage. Document the behavior and run-log evidence. ## Verification - Red: the new native runtime regression rejected with `NativeProviderTerminalFailure` on the original code; the Rust restore regression found a missing recovery cause. - Red/green: if restoring the conversation fails and replacement is allowed, the replacement receives the full task and fresh-session handoff. Both prepared and legacy execution inputs are covered. - Red: a second controller crash after the provider accepted the continuation caused an extra `turn/start`. The regression now proves there are exactly two submissions total: the original and its continuation. - Green: focused runtime, backend, driver recovery, and real-process restart suites (207 tests). After the final history/result changes, driver recovery and real-process restart suites passed again (36 tests). - Green: complete Codex transport suite (186 tests), server restart classification/database integration suites (34 tests), and Rust Codex provider suite (92 passed, 2 ignored). - Full `pnpm -r typecheck` and `pnpm build` passed on `60141e649`. The subsequent replacement-prompt guard passed the Runner TypeScript check and the complete runtime plus process-restart suites (148 tests). - Local full-suite attempt: `pnpm test:run` reported two failures in the untouched chat integration suite. Both passed individually, and the complete chat suite passed on rerun (1,063 tests). After all remote test shards passed, the duplicate serial local run was stopped with SIGINT; it is not claimed as a full local-suite pass. - Latest-head CI (`b1297dcd4`): 55 successful checks and 4 intentionally skipped checks, including all test shards, typecheck, build, native Runner verification, end-to-end tests, and canary dry run. [CI run](https://github.com/paperclipai/paperclip/actions/runs/37405346303). - Greptile: 5/5 on the latest head, with no unresolved review threads. The PR is mergeable. - The process tests use the real runner binary and a scripted Codex provider. They do not call a live model service. ## Risks - This changes local Codex recovery after process loss. A continuation can execute more work in the retained conversation. Its prompt requires state inspection before repeating an uncertain action; the system does not replay tool calls. - Recovery shares the existing three-attempt budget and one-shot continuation marker. Real failures and older unmarked failed checkpoints are not reopened. - No database migration or API change is required. ## Model Used - OpenAI Codex, based on GPT-6. The exact model ID and context-window size are not exposed in this session. - Capabilities: reasoning, source inspection, tool use, code editing, code execution, and test analysis. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
271 lines
10 KiB
TypeScript
271 lines
10 KiB
TypeScript
import type {
|
|
NativeRunIdentity,
|
|
NativeSessionCapabilities,
|
|
NativeUserMessage,
|
|
} from "./types.js";
|
|
import type {
|
|
PrpEvent,
|
|
PrpStructuredRunResult,
|
|
PrpTerminalState,
|
|
} from "../protocol/replay-contract.js";
|
|
import type {
|
|
HarnessRuntimeRequest,
|
|
HarnessRuntimeRequestHandoff,
|
|
HarnessRuntimeRequestResolution,
|
|
HarnessGoalOperation,
|
|
HarnessThreadGoal,
|
|
HarnessThreadLineageEntry,
|
|
NativeRuntimeContextCapabilities,
|
|
PersistedHarnessProviderIdentity,
|
|
PersistedHarnessSession,
|
|
PersistedHarnessTurnTerminal,
|
|
} from "./harness-driver.js";
|
|
|
|
export interface NativeSessionBackendDescriptor {
|
|
kind: "runner" | "remote" | "mock";
|
|
name: string;
|
|
version: string;
|
|
capabilities: NativeSessionCapabilities;
|
|
runtimeContextCapabilities?: NativeRuntimeContextCapabilities;
|
|
}
|
|
|
|
export interface OpenNativeSessionInput {
|
|
identity: NativeRunIdentity;
|
|
workingDirectory?: string;
|
|
/**
|
|
* When present, aborts provider bootstrap if the caller's recovery deadline
|
|
* expires. Backends own cleanup for work that has not returned a session yet.
|
|
*/
|
|
signal?: AbortSignal;
|
|
}
|
|
|
|
export interface NativeSessionRecoveryOptions {
|
|
/** Abort provider recovery and release any not-yet-returned provider state. */
|
|
signal: AbortSignal;
|
|
}
|
|
|
|
export interface PersistedNativeSession {
|
|
backendKind: NativeSessionBackendDescriptor["kind"];
|
|
driverKind?: string | null;
|
|
sessionId: string;
|
|
identity: NativeRunIdentity;
|
|
providerSessionId?: string | null;
|
|
/** Tagged provider-owned identity required for safe driver recovery. */
|
|
providerIdentity?: PersistedHarnessProviderIdentity;
|
|
workingDirectory?: string;
|
|
codexUsageBaseline?: PersistedHarnessSession["codexUsageBaseline"];
|
|
providerRecoveryPolicy?:
|
|
| "same_session_only"
|
|
| "allow_replacement_after_governed_wait"
|
|
| "allow_replacement_after_resume_failure";
|
|
cursor?: string | null;
|
|
semanticResult?: PrpStructuredRunResult | null;
|
|
terminal?: PrpTerminalState | null;
|
|
activeTurnId?: string | null;
|
|
terminalTurns?: PersistedHarnessTurnTerminal[];
|
|
/** At-most-once marker for resultless disposition repair or restart continuation. */
|
|
dispositionOnlyRecoveryConsumed?: boolean;
|
|
dispositionOnlyRecoveryTurnId?: string | null;
|
|
pendingRuntimeRequests?: HarnessRuntimeRequest[];
|
|
goal?: HarnessThreadGoal | null;
|
|
lineage?: HarnessThreadLineageEntry[];
|
|
}
|
|
|
|
export interface NativeSessionRecoveryResult {
|
|
recovered: boolean;
|
|
session?: NativeSession;
|
|
reason?: string;
|
|
}
|
|
|
|
/** Runner-owned cause: the process was restored, but its active turn was lost. */
|
|
export const NATIVE_RESTART_INTERRUPTION_CODE = "provider_turn_lost_on_restore";
|
|
|
|
export function isNativeRestartInterruption(error: unknown): boolean {
|
|
if (!error || typeof error !== "object" || Array.isArray(error)) return false;
|
|
const failure = error as Record<string, unknown>;
|
|
return failure.code === NATIVE_RESTART_INTERRUPTION_CODE && failure.recoverable === true;
|
|
}
|
|
|
|
/** The terminal fingerprint survives a crash between interruption and continuation. */
|
|
export function nativeRestartInterruptedTurnId(snapshot: Pick<PersistedNativeSession,
|
|
"driverKind" | "activeTurnId" | "semanticResult" | "terminalTurns" | "dispositionOnlyRecoveryConsumed" | "pendingRuntimeRequests" | "goal"
|
|
>): string | null {
|
|
if (snapshot.driverKind !== "codex_app_server" || snapshot.activeTurnId != null ||
|
|
snapshot.semanticResult != null || snapshot.dispositionOnlyRecoveryConsumed ||
|
|
snapshot.goal != null || snapshot.pendingRuntimeRequests?.length) return null;
|
|
const terminal = Array.isArray(snapshot.terminalTurns) ? snapshot.terminalTurns.at(-1) : null;
|
|
if (!terminal || typeof terminal.fingerprint !== "string" || !terminal.turnId) return null;
|
|
try {
|
|
const fingerprint = JSON.parse(terminal.fingerprint);
|
|
return fingerprint?.terminalState === "failed" && fingerprint.result === null &&
|
|
isNativeRestartInterruption(fingerprint.error) ? terminal.turnId : null;
|
|
} catch { return null; }
|
|
}
|
|
|
|
/**
|
|
* Cancellation is a synchronous authority transition followed by passive
|
|
* provider cleanup. Once `cancel` returns, the provider session must no longer
|
|
* be able to publish accepted output or acquire new mutation authority for the
|
|
* cancelled turn. Cleanup may stop processes or transports, but it must not
|
|
* perform durable control-plane mutations.
|
|
*/
|
|
export interface NativeSessionCancellation {
|
|
cleanup: Promise<void>;
|
|
}
|
|
|
|
export interface NativeSessionSnapshotOptions {
|
|
/** Stop provider snapshot work that outlives the execution deadline. */
|
|
signal: AbortSignal;
|
|
}
|
|
|
|
/** A dispatched operation may have taken effect, but has no proven result. */
|
|
export class SemanticToolOutcomeUnknownError extends Error {
|
|
readonly code = "semantic_tool_outcome_unknown";
|
|
|
|
constructor(message: string, options?: ErrorOptions) {
|
|
super(message, options);
|
|
this.name = "SemanticToolOutcomeUnknownError";
|
|
}
|
|
}
|
|
|
|
export function isSemanticToolOutcomeUnknownError(error: unknown): error is SemanticToolOutcomeUnknownError {
|
|
// Match across the server's vendored and source-mode runtime boundaries.
|
|
return error instanceof Error && "code" in error && error.code === "semantic_tool_outcome_unknown";
|
|
}
|
|
|
|
/** The exact close owner has torn down its controller without proving suspension. */
|
|
export class NativeSessionCloseUnrecoverableError extends Error {
|
|
readonly code = "native_session_close_unrecoverable";
|
|
|
|
constructor(readonly settlement?: Record<string, unknown>) {
|
|
super(
|
|
"provider_transport_failed: runner did not durably suspend before checkpoint",
|
|
);
|
|
this.name = "NativeSessionCloseUnrecoverableError";
|
|
}
|
|
}
|
|
|
|
/** An authenticated, exactly bound runner event failed permanent integrity checks. */
|
|
export class NativeSessionProtocolIntegrityError extends Error {
|
|
readonly code = "native_event_replay_conflict";
|
|
readonly recovery = "operator_required";
|
|
|
|
constructor(
|
|
readonly reason:
|
|
| "semantic_input_digest_mismatch"
|
|
| "source_event_replay_conflict",
|
|
) {
|
|
super(
|
|
reason === "semantic_input_digest_mismatch"
|
|
? "native_event_replay_conflict: authenticated runner semantic input failed integrity validation; automatic recovery is stopped."
|
|
: "native_event_replay_conflict: authenticated runner event conflicts with committed history; automatic recovery is stopped.",
|
|
);
|
|
this.name = "NativeSessionProtocolIntegrityError";
|
|
}
|
|
}
|
|
|
|
/** Admission is blocked by a retained owner that has no safe automatic close retry. */
|
|
export class NativeSessionCleanupQuarantinedError extends Error {
|
|
readonly code = "native_session_cleanup_quarantined";
|
|
readonly recovery = "operator_required";
|
|
|
|
constructor() {
|
|
super(
|
|
"native_session_cleanup_quarantined: prior session cleanup requires operator recovery; verify its retained process ownership and checkpoint before a controlled restart. Clearing a task session does not resolve this quarantine.",
|
|
);
|
|
this.name = "NativeSessionCleanupQuarantinedError";
|
|
}
|
|
}
|
|
|
|
export interface NativeSession {
|
|
identity(): NativeRunIdentity;
|
|
capabilities(): Promise<NativeSessionCapabilities>;
|
|
attachRun?(input: { identity: NativeRunIdentity }): Promise<void>;
|
|
/** Relinquish controller authority without suspending provider execution. */
|
|
detachControllerForRestart?(): Promise<void>;
|
|
events(input?: { afterCursor?: string | null }): AsyncIterable<PrpEvent>;
|
|
startTurn(input: {
|
|
message: NativeUserMessage;
|
|
/** Set by orchestration only after successful provider-session recovery. */
|
|
continuation?: true;
|
|
requestedCollaborationMode?: "default" | "plan";
|
|
}): Promise<{
|
|
turnId: string;
|
|
effectiveCollaborationMode?: "default" | "plan";
|
|
}>;
|
|
steer?(input: {
|
|
mode?: "steer" | "follow_up";
|
|
turnId: string;
|
|
message: NativeUserMessage;
|
|
correlationId?: string;
|
|
}): Promise<void>;
|
|
interrupt?(input: { turnId?: string; reason?: string }): Promise<void>;
|
|
/** Commit cancellation synchronously; the returned promise owns cleanup only. */
|
|
cancel?(input: {
|
|
reason: string;
|
|
signal: AbortSignal;
|
|
}): NativeSessionCancellation;
|
|
resolveRuntimeRequest?(input: {
|
|
requestId: string;
|
|
turnId: string;
|
|
resolution: HarnessRuntimeRequestResolution;
|
|
}): Promise<void>;
|
|
handoffRuntimeRequest?(input: {
|
|
requestId: string;
|
|
turnId: string;
|
|
reason: "durable_handoff";
|
|
/**
|
|
* Revokes durable mutation authority when event consumption fails. The
|
|
* method must commit synchronously before returning; its returned promise
|
|
* owns provider cleanup only and must not mutate durable request state.
|
|
*/
|
|
signal: AbortSignal;
|
|
}): HarnessRuntimeRequestHandoff;
|
|
goal?(input: HarnessGoalOperation): Promise<HarnessThreadGoal | null>;
|
|
result(): Promise<{
|
|
result: PrpStructuredRunResult;
|
|
terminal: PrpTerminalState;
|
|
turnId: string | null;
|
|
} | null>;
|
|
usage?(): Promise<Record<string, unknown> | null>;
|
|
snapshot(
|
|
options?: NativeSessionSnapshotOptions,
|
|
): Promise<PersistedNativeSession>;
|
|
/**
|
|
* Idempotently stop provider work and release every pending `events().next()`
|
|
* before this promise resolves. Implementations must settle every promise
|
|
* previously returned by the session (including interrupt, cancel, handoff,
|
|
* and iterator teardown). The runtime bounds its wait for a broken provider,
|
|
* revokes that session's mutation authority, removes it from reuse, and keeps
|
|
* observing late cleanup so a contract violation cannot defeat a run timeout
|
|
* or become an unhandled rejection.
|
|
*/
|
|
close(input: { reason: string }): Promise<void>;
|
|
}
|
|
|
|
/** Normalized control-plane boundary shared by runner and hosted backends. */
|
|
export interface NativeSessionBackend {
|
|
/** Existing task rules at user-message priority, for prepared native envelopes. */
|
|
readonly preparedTaskConstraints?: readonly string[];
|
|
descriptor(): Promise<NativeSessionBackendDescriptor>;
|
|
openSession(input: OpenNativeSessionInput): Promise<NativeSession>;
|
|
/** Open a fresh provider session after an explicitly governed continuity break. */
|
|
openReplacementSession?(
|
|
input: OpenNativeSessionInput,
|
|
previous: PersistedNativeSession,
|
|
): Promise<NativeSession>;
|
|
recoverSession?(
|
|
snapshot: PersistedNativeSession,
|
|
options: NativeSessionRecoveryOptions,
|
|
): Promise<NativeSessionRecoveryResult>;
|
|
}
|
|
|
|
/** A provider failed terminal is not a missing completion proposal. */
|
|
export class NativeProviderTerminalFailure extends Error {
|
|
readonly code = "native_provider_terminal_failed";
|
|
constructor(readonly providerCode: string, readonly recoverable: boolean, message = "Provider session ended with a failed terminal") {
|
|
super(message);
|
|
this.name = "NativeProviderTerminalFailure";
|
|
}
|
|
}
|