mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-28 12:04:41 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Paperclip Runner keeps durable run and provider state outside one server process. > - A server restart can leave that runner alive or can interrupt it after a provider checkpoint. > - The old startup path used handoff intent and PID evidence, but it did not reconstruct native ownership. > - That gap could block the issue, create a replacement run, or start duplicate provider work. > - This pull request adds durable same-run recovery for coordinated and uncoordinated restarts. > - The benefit is exact recovery of the run, runner, session, provider, steering, and finalization state. ## Linked Issues or Issue Description Refs #9628. That pull request added earlier local-adapter hot-restart work. This change adds native PRP authority reconstruction and same-run provider resume. Refs #10935. That pull request handles missing hot-restart snapshots. This change also supports hard restarts with no snapshot. Refs #11624. That pull request prevents unsafe retry after an adopted legacy process exits. This change reconciles native terminal evidence before provider recovery. Refs #12070. That pull request improves process liveness checks. This change also binds recovery to a process-start fingerprint and fails closed on ambiguity. **What happened?** The server could record hot-restart intent, but startup did not rebuild native runner ownership. A live runner could not re-register its PRP authority. A dead runner could not resume the exact native and provider session on the same heartbeat run. Generic recovery could then block the issue or create replacement work. **Expected behavior** A live native runner must reconnect with the same PID and logical identities. A dead runner must resume the same durable session and heartbeat run with only a new operating-system PID. A proposed or terminal result must finalize once before any provider turn starts. Ambiguous process or session evidence must stay blocked without a signal or duplicate spawn. **Steps to reproduce** 1. Start a Paperclip Runner heartbeat and wait for an active provider turn. 2. Restart only the Paperclip server, with or without a hot-restart marker. 3. Observe that the old startup path does not reconstruct the native control-plane authority. 4. Kill both the server and runner after a provider checkpoint. 5. Observe that the old path cannot resume the exact native session on the original heartbeat run. **Paperclip version or commit** The defect was reproduced from commit `1991f31fd53e7f7794d5c2e4b93be384ade2b41d`. This branch is rebased onto the current `master`. **Deployment mode** Local development and self-hosted server deployments that use the local Paperclip Runner. ## What Changed - Added correlated hot-restart requests and version-compatible native handoff fields. - Added controller boot identity, process-start identity, controller generation, recovery state, request id, and bounded history to the native finalization ledger. - Added transactional recovery claims for live-runner reattach, dead-runner resume, and incomplete bootstrap. - Added fail-closed ownership takeover rules and process identity validation. - Added live runner adoption to the local runner transport without a duplicate spawn. - Added same-run provider checkpoint resume and legacy retry-row compatibility. - Reconciled proposed and terminal results before runner or provider recovery. - Bound the HTTP and PRP listener before startup recovery and delayed scheduling and generic reapers until classification completes. - Added restart-aware health diagnostics, run-log recovery transitions, durable runner diagnostics, and bounded shutdown finalizer draining. - Moved restart-survivable diagnostics into runner-owned, pre-redacted bounded writes; raw stdout and stderr are never persisted. - Added process-start fencing for controller, runner, and provider PIDs; startup classifies every candidate without an implicit cap. - Added crash-recoverable, contention-safe development restart-request coordination and failed-startup listener cleanup. - Added a credential-free real-process restart suite for eight restart, scale, and identity scenarios. - Documented native restart operation, persistence, diagnostics, and verification. ## Verification - The documented native restart commands passed. They ran eight real-process/database recovery scenarios and the live runner adoption transport test. - Native executor tests passed: 111 tests. - Heartbeat recovery tests passed: 124 tests. - Hot restart, health, and shutdown tests passed: 52 tests. - The broader affected server suite passed: 350 tests. - Focused native recovery and startup tests passed: 49 tests. - Runner transport and control-plane tests passed: 63 tests. - Runner-owned diagnostic tests passed for write-time bounding, credential redaction, private file modes, and raw stream non-persistence. - Development restart coordination tests passed: 11 tests. - Database migration checks and the partial-application/replay regression test passed. - Server, database, and Paperclip Runner typechecks passed. - `git diff --check` passed. - Full Paperclip PR CI passed, including build, canary, all five general server shards, all five serialized server shards, all three browser E2E shards, workspace suites, and release-registry verification. - Greptile completed at 5/5 with no outstanding findings, recommendations, follow-ups, or open review threads. ## Risks - Moderate risk. This changes startup ordering and ownership transfer for active native runs. - The migration adds nullable columns and does not rewrite existing rows. - Recovery fails closed when process or durable session identity is incomplete or contradictory. - The first implementation supports the local Paperclip Runner. Remote targets keep their existing behavior. - The real-process suite covers cleanup and asserts that no runner or provider process survives each test. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex with GPT-5. The runtime did not expose a more specific model revision or context-window size. Repository editing, shell execution, database tests, and real-process test execution were enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
148 lines
5.5 KiB
TypeScript
148 lines
5.5 KiB
TypeScript
import type {
|
|
NativeExecutionInput,
|
|
PersistedNativeSession,
|
|
} from "../../vendor/paperclip-runner/index.js";
|
|
import { parseNativeExecutionInput } from "../../vendor/paperclip-runner/index.js";
|
|
|
|
function record(value: unknown): Record<string, unknown> {
|
|
return typeof value === "object" && value !== null && !Array.isArray(value)
|
|
? value as Record<string, unknown>
|
|
: {};
|
|
}
|
|
|
|
export function isNativeSessionId(value: unknown): value is string {
|
|
return typeof value === "string"
|
|
&& /^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i.test(value);
|
|
}
|
|
|
|
const LEGACY_RETRY_SOURCE_TERMINAL_STATUSES = new Set([
|
|
"succeeded",
|
|
"interrupted",
|
|
"failed",
|
|
"cancelled",
|
|
"timed_out",
|
|
]);
|
|
|
|
/**
|
|
* Legacy compatibility is deliberately narrower than ordinary task-session
|
|
* continuation: only a replacement row that never acquired any native process
|
|
* or provider authority may be rebound to a terminal native source. The exact
|
|
* checkpoint/session/workspace/provider binding is validated separately by
|
|
* rebindNativeSessionCheckpoint.
|
|
*/
|
|
export function isUnusedLegacyNativeRetryReplacement(input: {
|
|
replacement: {
|
|
processPid: number | null;
|
|
processGroupId: number | null;
|
|
processStartedAt: Date | null;
|
|
runnerProfileJson: unknown;
|
|
};
|
|
source: {
|
|
runtimeMode: string | null;
|
|
status: string;
|
|
nativeSessionId: string | null;
|
|
} | null;
|
|
hasProviderEvents: boolean;
|
|
}): boolean {
|
|
const replacementProfile = record(input.replacement.runnerProfileJson);
|
|
return Boolean(
|
|
input.source?.runtimeMode === "native" &&
|
|
LEGACY_RETRY_SOURCE_TERMINAL_STATUSES.has(input.source.status) &&
|
|
isNativeSessionId(input.source.nativeSessionId) &&
|
|
input.replacement.processPid === null &&
|
|
input.replacement.processGroupId === null &&
|
|
input.replacement.processStartedAt === null &&
|
|
replacementProfile.sessionCheckpoint == null &&
|
|
!input.hasProviderEvents,
|
|
);
|
|
}
|
|
|
|
function sameProvider(
|
|
previous: NativeExecutionInput["provider"],
|
|
current: NativeExecutionInput["provider"],
|
|
): boolean {
|
|
return JSON.stringify(previous) === JSON.stringify(current);
|
|
}
|
|
|
|
/**
|
|
* Rebind a completed prior run's provider checkpoint to a new heartbeat run.
|
|
* The provider/driver session identity is retained, while every per-turn and
|
|
* per-event field is reset so the new run starts one clean turn via resume.
|
|
*/
|
|
export function rebindNativeSessionCheckpoint(input: {
|
|
previousRun: {
|
|
id: string;
|
|
companyId: string;
|
|
agentId: string;
|
|
nativeSessionId: string | null;
|
|
runnerProfileJson: unknown;
|
|
};
|
|
currentExecution: NativeExecutionInput;
|
|
}): PersistedNativeSession | null {
|
|
const previousProfile = record(input.previousRun.runnerProfileJson);
|
|
const rawCheckpoint = record(previousProfile.sessionCheckpoint);
|
|
const checkpointIdentity = record(rawCheckpoint.identity);
|
|
const current = input.currentExecution;
|
|
const normalizedSessionId = current.session.normalizedSessionId;
|
|
if (
|
|
!isNativeSessionId(normalizedSessionId)
|
|
|| input.previousRun.companyId !== current.binding.companyId
|
|
|| input.previousRun.agentId !== current.binding.agentId
|
|
|| input.previousRun.nativeSessionId !== normalizedSessionId
|
|
|| typeof rawCheckpoint.sessionId !== "string"
|
|
|| checkpointIdentity.runId !== input.previousRun.id
|
|
|| checkpointIdentity.companyId !== current.binding.companyId
|
|
|| checkpointIdentity.issueId !== current.binding.issueId
|
|
|| checkpointIdentity.agentId !== current.binding.agentId
|
|
|| checkpointIdentity.sessionId !== normalizedSessionId
|
|
) return null;
|
|
|
|
let previousExecution: NativeExecutionInput;
|
|
try {
|
|
previousExecution = parseNativeExecutionInput(previousProfile.nativeExecutionInput);
|
|
} catch {
|
|
return null;
|
|
}
|
|
if (
|
|
previousExecution.binding.runId !== input.previousRun.id
|
|
|| previousExecution.binding.companyId !== current.binding.companyId
|
|
|| previousExecution.binding.issueId !== current.binding.issueId
|
|
|| previousExecution.binding.agentId !== current.binding.agentId
|
|
|| previousExecution.session.normalizedSessionId !== normalizedSessionId
|
|
|| previousExecution.session.driverKind !== current.session.driverKind
|
|
|| previousExecution.workspace.cwd !== current.workspace.cwd
|
|
|| ("executionMode" in previousExecution ? previousExecution.executionMode : "default")
|
|
!== ("executionMode" in current ? current.executionMode : "default")
|
|
|| !sameProvider(previousExecution.provider, current.provider)
|
|
|| previousExecution.schema !== current.schema
|
|
|| ("runtimeContext" in previousExecution && "runtimeContext" in current
|
|
&& previousExecution.runtimeContext.aggregateDigest !== current.runtimeContext.aggregateDigest)
|
|
) return null;
|
|
|
|
const priorSemanticResult = record(rawCheckpoint.semanticResult);
|
|
const priorContinuation = record(priorSemanticResult.continuation);
|
|
const providerRecoveryPolicy =
|
|
priorSemanticResult.reportedWorkDisposition === "yielded"
|
|
&& priorContinuation.kind === "response_wake"
|
|
? "allow_replacement_after_governed_wait" as const
|
|
: "allow_replacement_after_resume_failure" as const;
|
|
|
|
return {
|
|
...(structuredClone(rawCheckpoint) as unknown as PersistedNativeSession),
|
|
identity: {
|
|
runId: current.binding.runId,
|
|
sessionId: normalizedSessionId,
|
|
companyId: current.binding.companyId,
|
|
issueId: current.binding.issueId,
|
|
agentId: current.binding.agentId,
|
|
},
|
|
cursor: null,
|
|
semanticResult: null,
|
|
terminal: null,
|
|
activeTurnId: null,
|
|
terminalTurns: [],
|
|
pendingRuntimeRequests: [],
|
|
providerRecoveryPolicy,
|
|
};
|
|
}
|