Files
PaperClipAI/server/src/services/native-runtime/native-session-resume.ts
T
Dotta 7b094724e6 fix(runner): recover native sessions across restarts (#12845)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Paperclip Runner keeps durable run and provider state outside
one server process.
> - A server restart can leave that runner alive or can interrupt it
after a provider checkpoint.
> - The old startup path used handoff intent and PID evidence, but it
did not reconstruct native ownership.
> - That gap could block the issue, create a replacement run, or start
duplicate provider work.
> - This pull request adds durable same-run recovery for coordinated and
uncoordinated restarts.
> - The benefit is exact recovery of the run, runner, session, provider,
steering, and finalization state.

## Linked Issues or Issue Description

Refs #9628. That pull request added earlier local-adapter hot-restart
work. This change adds native PRP authority reconstruction and same-run
provider resume.

Refs #10935. That pull request handles missing hot-restart snapshots.
This change also supports hard restarts with no snapshot.

Refs #11624. That pull request prevents unsafe retry after an adopted
legacy process exits. This change reconciles native terminal evidence
before provider recovery.

Refs #12070. That pull request improves process liveness checks. This
change also binds recovery to a process-start fingerprint and fails
closed on ambiguity.

**What happened?**

The server could record hot-restart intent, but startup did not rebuild
native runner ownership. A live runner could not re-register its PRP
authority. A dead runner could not resume the exact native and provider
session on the same heartbeat run. Generic recovery could then block the
issue or create replacement work.

**Expected behavior**

A live native runner must reconnect with the same PID and logical
identities. A dead runner must resume the same durable session and
heartbeat run with only a new operating-system PID. A proposed or
terminal result must finalize once before any provider turn starts.
Ambiguous process or session evidence must stay blocked without a signal
or duplicate spawn.

**Steps to reproduce**

1. Start a Paperclip Runner heartbeat and wait for an active provider
turn.
2. Restart only the Paperclip server, with or without a hot-restart
marker.
3. Observe that the old startup path does not reconstruct the native
control-plane authority.
4. Kill both the server and runner after a provider checkpoint.
5. Observe that the old path cannot resume the exact native session on
the original heartbeat run.

**Paperclip version or commit**

The defect was reproduced from commit
`1991f31fd53e7f7794d5c2e4b93be384ade2b41d`. This branch is rebased onto
the current `master`.

**Deployment mode**

Local development and self-hosted server deployments that use the local
Paperclip Runner.

## What Changed

- Added correlated hot-restart requests and version-compatible native
handoff fields.
- Added controller boot identity, process-start identity, controller
generation, recovery state, request id, and bounded history to the
native finalization ledger.
- Added transactional recovery claims for live-runner reattach,
dead-runner resume, and incomplete bootstrap.
- Added fail-closed ownership takeover rules and process identity
validation.
- Added live runner adoption to the local runner transport without a
duplicate spawn.
- Added same-run provider checkpoint resume and legacy retry-row
compatibility.
- Reconciled proposed and terminal results before runner or provider
recovery.
- Bound the HTTP and PRP listener before startup recovery and delayed
scheduling and generic reapers until classification completes.
- Added restart-aware health diagnostics, run-log recovery transitions,
durable runner diagnostics, and bounded shutdown finalizer draining.
- Moved restart-survivable diagnostics into runner-owned, pre-redacted
bounded writes; raw stdout and stderr are never persisted.
- Added process-start fencing for controller, runner, and provider PIDs;
startup classifies every candidate without an implicit cap.
- Added crash-recoverable, contention-safe development restart-request
coordination and failed-startup listener cleanup.
- Added a credential-free real-process restart suite for eight restart,
scale, and identity scenarios.
- Documented native restart operation, persistence, diagnostics, and
verification.

## Verification

- The documented native restart commands passed. They ran eight
real-process/database recovery scenarios and the live runner adoption
transport test.
- Native executor tests passed: 111 tests.
- Heartbeat recovery tests passed: 124 tests.
- Hot restart, health, and shutdown tests passed: 52 tests.
- The broader affected server suite passed: 350 tests.
- Focused native recovery and startup tests passed: 49 tests.
- Runner transport and control-plane tests passed: 63 tests.
- Runner-owned diagnostic tests passed for write-time bounding,
credential redaction, private file modes, and raw stream
non-persistence.
- Development restart coordination tests passed: 11 tests.
- Database migration checks and the partial-application/replay
regression test passed.
- Server, database, and Paperclip Runner typechecks passed.
- `git diff --check` passed.
- Full Paperclip PR CI passed, including build, canary, all five general
server shards, all five serialized server shards, all three browser E2E
shards, workspace suites, and release-registry verification.
- Greptile completed at 5/5 with no outstanding findings,
recommendations, follow-ups, or open review threads.

## Risks

- Moderate risk. This changes startup ordering and ownership transfer
for active native runs.
- The migration adds nullable columns and does not rewrite existing
rows.
- Recovery fails closed when process or durable session identity is
incomplete or contradictory.
- The first implementation supports the local Paperclip Runner. Remote
targets keep their existing behavior.
- The real-process suite covers cleanup and asserts that no runner or
provider process survives each test.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex with GPT-5. The runtime did not expose a more specific
model revision or context-window size. Repository editing, shell
execution, database tests, and real-process test execution were enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-09-04 15:03:53 -05:00

148 lines
5.5 KiB
TypeScript

import type {
NativeExecutionInput,
PersistedNativeSession,
} from "../../vendor/paperclip-runner/index.js";
import { parseNativeExecutionInput } from "../../vendor/paperclip-runner/index.js";
function record(value: unknown): Record<string, unknown> {
return typeof value === "object" && value !== null && !Array.isArray(value)
? value as Record<string, unknown>
: {};
}
export function isNativeSessionId(value: unknown): value is string {
return typeof value === "string"
&& /^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i.test(value);
}
const LEGACY_RETRY_SOURCE_TERMINAL_STATUSES = new Set([
"succeeded",
"interrupted",
"failed",
"cancelled",
"timed_out",
]);
/**
* Legacy compatibility is deliberately narrower than ordinary task-session
* continuation: only a replacement row that never acquired any native process
* or provider authority may be rebound to a terminal native source. The exact
* checkpoint/session/workspace/provider binding is validated separately by
* rebindNativeSessionCheckpoint.
*/
export function isUnusedLegacyNativeRetryReplacement(input: {
replacement: {
processPid: number | null;
processGroupId: number | null;
processStartedAt: Date | null;
runnerProfileJson: unknown;
};
source: {
runtimeMode: string | null;
status: string;
nativeSessionId: string | null;
} | null;
hasProviderEvents: boolean;
}): boolean {
const replacementProfile = record(input.replacement.runnerProfileJson);
return Boolean(
input.source?.runtimeMode === "native" &&
LEGACY_RETRY_SOURCE_TERMINAL_STATUSES.has(input.source.status) &&
isNativeSessionId(input.source.nativeSessionId) &&
input.replacement.processPid === null &&
input.replacement.processGroupId === null &&
input.replacement.processStartedAt === null &&
replacementProfile.sessionCheckpoint == null &&
!input.hasProviderEvents,
);
}
function sameProvider(
previous: NativeExecutionInput["provider"],
current: NativeExecutionInput["provider"],
): boolean {
return JSON.stringify(previous) === JSON.stringify(current);
}
/**
* Rebind a completed prior run's provider checkpoint to a new heartbeat run.
* The provider/driver session identity is retained, while every per-turn and
* per-event field is reset so the new run starts one clean turn via resume.
*/
export function rebindNativeSessionCheckpoint(input: {
previousRun: {
id: string;
companyId: string;
agentId: string;
nativeSessionId: string | null;
runnerProfileJson: unknown;
};
currentExecution: NativeExecutionInput;
}): PersistedNativeSession | null {
const previousProfile = record(input.previousRun.runnerProfileJson);
const rawCheckpoint = record(previousProfile.sessionCheckpoint);
const checkpointIdentity = record(rawCheckpoint.identity);
const current = input.currentExecution;
const normalizedSessionId = current.session.normalizedSessionId;
if (
!isNativeSessionId(normalizedSessionId)
|| input.previousRun.companyId !== current.binding.companyId
|| input.previousRun.agentId !== current.binding.agentId
|| input.previousRun.nativeSessionId !== normalizedSessionId
|| typeof rawCheckpoint.sessionId !== "string"
|| checkpointIdentity.runId !== input.previousRun.id
|| checkpointIdentity.companyId !== current.binding.companyId
|| checkpointIdentity.issueId !== current.binding.issueId
|| checkpointIdentity.agentId !== current.binding.agentId
|| checkpointIdentity.sessionId !== normalizedSessionId
) return null;
let previousExecution: NativeExecutionInput;
try {
previousExecution = parseNativeExecutionInput(previousProfile.nativeExecutionInput);
} catch {
return null;
}
if (
previousExecution.binding.runId !== input.previousRun.id
|| previousExecution.binding.companyId !== current.binding.companyId
|| previousExecution.binding.issueId !== current.binding.issueId
|| previousExecution.binding.agentId !== current.binding.agentId
|| previousExecution.session.normalizedSessionId !== normalizedSessionId
|| previousExecution.session.driverKind !== current.session.driverKind
|| previousExecution.workspace.cwd !== current.workspace.cwd
|| ("executionMode" in previousExecution ? previousExecution.executionMode : "default")
!== ("executionMode" in current ? current.executionMode : "default")
|| !sameProvider(previousExecution.provider, current.provider)
|| previousExecution.schema !== current.schema
|| ("runtimeContext" in previousExecution && "runtimeContext" in current
&& previousExecution.runtimeContext.aggregateDigest !== current.runtimeContext.aggregateDigest)
) return null;
const priorSemanticResult = record(rawCheckpoint.semanticResult);
const priorContinuation = record(priorSemanticResult.continuation);
const providerRecoveryPolicy =
priorSemanticResult.reportedWorkDisposition === "yielded"
&& priorContinuation.kind === "response_wake"
? "allow_replacement_after_governed_wait" as const
: "allow_replacement_after_resume_failure" as const;
return {
...(structuredClone(rawCheckpoint) as unknown as PersistedNativeSession),
identity: {
runId: current.binding.runId,
sessionId: normalizedSessionId,
companyId: current.binding.companyId,
issueId: current.binding.issueId,
agentId: current.binding.agentId,
},
cursor: null,
semanticResult: null,
terminal: null,
activeTurnId: null,
terminalTurns: [],
pendingRuntimeRequests: [],
providerRecoveryPolicy,
};
}