Files
PaperClipAI/server/src/services/native-runtime/native-controller-recovery.ts
T
DottaandClaude Opus 5.5 77ddf260bf fix(runner-e2e): keep the supervisor off the heartbeat graph and ship the plugin SDK build (#21860)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The paid Runner Full-Stack E2E workflow proves chat features against
a real source server
> - Since master f7a3497e9d, every paid cell fails before any test runs:
the source server cannot find `@paperclipai/plugin-sdk/dist/index.js`
> - The supervisor builds the plugin SDK before it starts Paperclip, but
its own static imports now load the SDK first
> - This pull request removes the heavy import edge, adds a regression
test for that class, and builds and checks the SDK once per campaign
> - The benefit is that paid campaigns run again, and a missing
workspace build fails once in the build job instead of in 199 paid cells

## Linked Issues or Issue Description

Refs #21791

**What happened?**
Chat-eval cohort 17 (master f7a3497e9d) failed in all 199 cells. Each
cell exited in the "Run paid cell" step. The private Playwright log
shows `ERR_MODULE_NOT_FOUND` for
`server/node_modules/@paperclipai/plugin-sdk/dist/index.js`, imported
from `server/src/services/reusable-sandbox-resume-error.ts`, and then
`Process from config.webServer was not able to start`. Cohort 16 (master
819f61f1fd) passed 169/199 with the same workflow steps.

**Root cause**
#21791 (df9914dbd9) added `import { NATIVE_RETAINED_OWNER_WAIT } from
"./native-controller-recovery.js"` to `native-runner-checkpoint.ts`.
`tests/runner-e2e/server.ts` statically imports
`controller-recovery-observer.ts`, which imports the checkpoint reader.
The new edge connects it to the full heartbeat graph:
`native-runner-checkpoint -> native-controller-recovery ->
native-finalization-notifications -> external-agents -> muse ->
heartbeat -> heartbeat/run-completion -> reusable-sandbox-resume-error
-> @paperclipai/plugin-sdk`. The plugin SDK exports only its generated
`dist`. The supervisor builds that `dist` in
`prepareSourceServerDependencies`, but ES static imports load before
that call. The workflow installs with `--ignore-scripts` and never built
the SDK, so the supervisor crashes at load. #21554 is not involved: it
changes only release workflows.

**Expected behavior**
The supervisor loads without generated SDK output. The campaign build
job builds and checks the SDK once, and each cell restores it.

**Steps to reproduce**
1. Remove `packages/plugins/sdk/dist`.
2. Run `node cli/node_modules/tsx/dist/cli.mjs
tests/runner-e2e/server.ts`.
3. Before this change: `ERR_MODULE_NOT_FOUND` for the SDK. After this
change: the normal `PAPERCLIP_RUNNER_E2E_SERVER_LOG is required` error,
which shows that all static imports loaded.

## What Changed

- Moved `NATIVE_RETAINED_OWNER_WAIT` and
`NATIVE_AGENT_DIRECTORY_BUSY_WAIT` to a new leaf module
`native-recovery-wait-codes.ts`. `native-controller-recovery.ts`
re-exports them, so other importers do not change.
`native-runner-checkpoint.ts` imports the leaf. This also removes a
checkpoint/controller-recovery import cycle.
- Added `tests/runner-e2e/server-startup-import-graph.test.ts`. It uses
esbuild to walk the static import graph of `tests/runner-e2e/server.ts`
and fails with the full chain if it reaches `@paperclipai/plugin-sdk`.
- `runner-full-stack-e2e.yml`: the "Build reusable runner campaign
artifacts" job runs `scripts/ensure-plugin-build-deps.mjs` (the same
certified helper the cell uses), checks that `shared` and `plugin-sdk`
dist and their completion markers exist with a clear `::error::`, and
ships both dist trees in the campaign bundle. The cell restore step
checks both entry points with a clear `::error::`. The helper's
fingerprint receipt lets the cell reuse the shipped output instead of a
rebuild.
- `workflow-security.test.ts` pins the build, packaging and guard.

## Verification

- Red/green: `npx vitest run --config tests/runner-e2e/vitest.config.ts
server-startup-import-graph` fails with the old import and passes with
the fix.
- Local reproduction above: fails before, loads after.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
workflow-security server-startup-import-graph native-completion`: 137
passed.
- `node --test tests/runner-e2e/native-completion-git-source.test.mjs`:
36 passed.
- Server `tsc --noEmit`: clean.
- Paid proof of the source fix: one cell
(`chat-features.runner-opencode.local.access.private-resource`),
dispatched with `--ref master -f
target_branch=fix/runner-e2e-plugin-sdk-dist`:
https://github.com/paperclipai/paperclip/actions/runs/43804619166 . That
run uses the master workflow and this branch's source, so it proves the
import-graph fix alone restores the cells.
- The workflow changes (SDK build, bundle and guards) cannot run in a
paid campaign before merge, because paid runs use the default-branch
workflow. `workflow-security.test.ts` pins them. A 1-case paid check
from master follows the merge.
- The failed "Authorize paid campaign" check on this commit is from a
dispatch with a branch `--ref`. The default-branch gate rejected it as
designed. It is not a PR-event check.

## Risks

- Low risk. The constants keep their values and are re-exported from the
old module.
- The campaign bundle grows by the `shared` and `plugin-sdk` dist trees
(small). If the fingerprints do not match in a cell, the helper
rebuilds, as before.

## Model Used

- Claude, `claude-opus-5-5`, via Claude Code with tool use (shell, file
edit, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 10:08:11 -06:00

411 lines
28 KiB
TypeScript

import { posix } from "node:path";
import { and, eq, inArray, isNull, ne, or, sql } from "drizzle-orm";
import { environmentLeases, heartbeatRunEvents, heartbeatRuns, issues, nativeRunFinalizations, nativeRunnerCheckpoints, type Db } from "@paperclipai/db";
import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
import { captureLegacyDurablePrpCheckpoint, parseNativeExecutionInput, type DurablePrpCheckpoint, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js";
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
import { readRunCancellation } from "../run-cancellation.js";
import { OWN_PROCESS_START_EVENT_TYPES } from "../native-process-start-events.js";
import { nativeSha256 } from "./canonical.js";
import { findNativeSessionResumeRun, isNativeSessionId, isUnusedNativeSessionBootstrap, nativeSessionProviderEvidence } from "./native-session-resume.js";
import { insertMigratedNativeRunnerCheckpoint, loadNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
import { offlineMigrationExecutionWhere } from "../offline-migration-execution.js";
import { redactRunnerApiPayloads } from "./runner-api-log-redaction.js";
import { NATIVE_AGENT_DIRECTORY_BUSY_WAIT, NATIVE_RETAINED_OWNER_WAIT } from "./native-recovery-wait-codes.js";
export { NATIVE_AGENT_DIRECTORY_BUSY_WAIT, NATIVE_RETAINED_OWNER_WAIT };
export const NATIVE_CONTROLLER_RECOVERY_MAX_BYTES = 4 * 1024;
/** Private run-profile metadata. No credentials or controller-local paths. */
export interface NativeControllerRecovery {
schema: "paperclip.native-controller-recovery.v1";
snapshotSha256: string;
configDigest: string;
processConfigDigest: string | null;
checkpoint: {
runId: string;
runnerInstanceId: string;
normalizedSessionId: string;
} | null;
}
export type NativeControllerCompatibility = Pick<NativeControllerRecovery, "configDigest" | "processConfigDigest">;
export function createNativeControllerRecovery(
snapshot: PersistedNativeSession,
compatibility: NativeControllerCompatibility,
checkpoint: NativeControllerRecovery["checkpoint"],
): NativeControllerRecovery {
const value: NativeControllerRecovery = {
schema: "paperclip.native-controller-recovery.v1",
snapshotSha256: nativeSha256(snapshot),
configDigest: compatibility.configDigest,
processConfigDigest: compatibility.processConfigDigest,
checkpoint,
};
if (Buffer.byteLength(JSON.stringify(value)) > NATIVE_CONTROLLER_RECOVERY_MAX_BYTES) {
throw new Error("native_controller_recovery_metadata_too_large");
}
return value;
}
function record(value: unknown): Record<string, unknown> {
return value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
}
export function readNativeControllerRecovery(value: unknown, snapshot: unknown): NativeControllerRecovery | null {
const envelope = record(value), checkpoint = record(envelope.checkpoint);
if (Buffer.byteLength(JSON.stringify(value) ?? "null") > NATIVE_CONTROLLER_RECOVERY_MAX_BYTES
|| envelope.schema !== "paperclip.native-controller-recovery.v1"
|| typeof envelope.snapshotSha256 !== "string" || envelope.snapshotSha256 !== nativeSha256(snapshot)
|| typeof envelope.configDigest !== "string" || !/^sha256:[a-f0-9]{64}$/.test(envelope.configDigest)
|| !(envelope.processConfigDigest === null || (typeof envelope.processConfigDigest === "string" && /^[a-f0-9]{64}$/.test(envelope.processConfigDigest)))
|| ![checkpoint.runId, checkpoint.runnerInstanceId, checkpoint.normalizedSessionId].every(v => typeof v === "string" && v.length > 0)) return null;
return envelope as unknown as NativeControllerRecovery;
}
export type NativeControllerRecoveryOutcome =
| { kind: "recoverable"; snapshot: PersistedNativeSession; metadata: NativeControllerRecovery; checkpoint: DurablePrpCheckpoint }
| { kind: "waiting_for_owner"; deadline: Date }
| { kind: "missing_evidence" | "unsupported" | "dependency_unavailable"; reason: string; cause?: unknown }
| { kind: "no_prior_execution" };
export type NativeRetainedOwnerWait = { runId: string; controllerBootId: string };
export class NativeControllerRecoveryWaitingError extends Error {
constructor(readonly retryAt: Date, readonly owner?: NativeRetainedOwnerWait) {
super(NATIVE_RETAINED_OWNER_WAIT);
this.name = "NativeControllerRecoveryWaitingError";
}
}
/** Admission has not crossed the provider boundary. Retain this same business
* run on the existing retry owner; never turn an ownership wait into a failure
* or invent a new provider attempt. The usual resumption gates recheck Stop,
* permissions and task ownership when this deadline arrives. */
export async function deferNativeControllerRecovery(db: Db, input: {
companyId: string; agentId: string; issueId: string; runId: string;
leaseOwner?: string; retryAt: Date; waitingForOwner?: NativeRetainedOwnerWait;
}): Promise<void> {
await db.transaction(async tx => {
const [task] = await tx.select({ executionRunId: issues.executionRunId }).from(issues)
.where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("update");
const [owner] = await tx.select().from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId),
eq(nativeRunFinalizations.runId, input.runId))).for("update");
const [run] = await tx.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId),
eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.nativeIssueId, input.issueId))).for("update");
const [authority] = await tx.select({ runId: nativeRunnerCheckpoints.runId }).from(nativeRunnerCheckpoints)
.where(and(eq(nativeRunnerCheckpoints.companyId, input.companyId), eq(nativeRunnerCheckpoints.runId, input.runId))).limit(1);
const progress = await nativeSessionProviderEvidence(tx as unknown as Db, [input.runId]);
const cancellation = record(run?.resultJson?.nativeCancellation);
const stop = readRunCancellation(run?.resultJson);
const [clock] = await tx.execute<{ now: Date }>(sql`select clock_timestamp() as now`);
const hasProcessIdentity = Boolean(run?.processPid || run?.processGroupId || run?.processStartedAt);
const inheritedProcessIdentity = hasProcessIdentity
&& typeof record(run?.runnerProfileJson).inheritedProcessIdentityRunId === "string"
&& !(await tx.select({ id: heartbeatRunEvents.id }).from(heartbeatRunEvents).where(and(
eq(heartbeatRunEvents.companyId, input.companyId), eq(heartbeatRunEvents.runId, input.runId),
isNull(heartbeatRunEvents.sourceEventId),
inArray(heartbeatRunEvents.eventType, [...OWN_PROCESS_START_EVENT_TYPES]))).limit(1)).length;
if (!run || run.runtimeMode !== "native" || run.status !== "running" || task?.executionRunId !== input.runId
|| !owner || owner.phase !== "observed" || owner.resultId || owner.attempt !== 0
|| owner.controllerGeneration !== 0 || owner.leaseOwner !== (input.leaseOwner ?? null)
|| (input.leaseOwner && (!owner.leaseExpiresAt || owner.leaseExpiresAt <= new Date(clock.now)))
|| (hasProcessIdentity && !inheritedProcessIdentity) || authority || progress.size
|| !record(run.runnerProfileJson).nativeExecutionInput || run.resultJson?.startupCancellation
|| (stop?.source === "operator" && stop.expected)
|| (["run", "issue"].includes(String(cancellation.scope)) && ["pending", "acknowledged"].includes(String(cancellation.dispatchState)))) {
throw new Error("native_controller_recovery_wait_authority_changed");
}
if (inheritedProcessIdentity) {
await tx.update(heartbeatRuns).set({ processPid: null, processGroupId: null, processStartedAt: null, updatedAt: new Date() })
.where(eq(heartbeatRuns.id, input.runId));
}
await notifyNativeFinalizationWork(tx, input);
await tx.update(nativeRunFinalizations).set({
phase: "retryable_failure", leaseOwner: null, leaseExpiresAt: null, controlDeadlineAt: null,
nextAttemptAt: input.retryAt, recoveryState: "resuming_session", failureCode: NATIVE_RETAINED_OWNER_WAIT,
failureDetail: { originalFailureCode: NATIVE_RETAINED_OWNER_WAIT,
message: "Waiting for the retained native controller owner to release its lease.",
recoveryMode: record(run.runnerProfileJson).sessionCheckpoint ? "same_session_resume" : "bootstrap_retry",
providerSessionEstablished: false, providerEventsExist: false,
...(input.waitingForOwner ? { waitingForOwner: { runId: input.waitingForOwner.runId,
controllerBootId: input.waitingForOwner.controllerBootId } } : {}) },
updatedAt: new Date(),
}).where(eq(nativeRunFinalizations.runId, input.runId));
});
}
export async function deferNativeRunForBusyAgentDirectory(db: Db, input: {
companyId: string; runId: string; leaseOwner?: string; retryAt: Date; maxDeferrals: number;
}): Promise<"deferred" | "pre_admission" | "refused"> {
return db.transaction(async tx => {
const [current] = await tx.select({ issueId: heartbeatRuns.nativeIssueId }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId)));
const [existing] = await tx.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations)
.where(and(eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, input.runId)));
if (!current?.issueId || !existing) return "pre_admission";
const [task] = await tx.select({ executionRunId: issues.executionRunId }).from(issues)
.where(and(eq(issues.companyId, input.companyId), eq(issues.id, current.issueId))).for("update");
const [owner] = await tx.select().from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, current.issueId),
eq(nativeRunFinalizations.runId, input.runId))).for("update");
const [run] = await tx.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId),
eq(heartbeatRuns.nativeIssueId, current.issueId))).for("update");
const cancellation = record(run?.resultJson?.nativeCancellation);
const stop = readRunCancellation(run?.resultJson);
const context = record(run?.contextSnapshot);
const deferrals = typeof context.agentDirectoryBusyDeferrals === "number" ? context.agentDirectoryBusyDeferrals : 0;
const [clock] = await tx.execute<{ now: Date }>(sql`select clock_timestamp() as now`);
if (!run || run.runtimeMode !== "native" || run.status !== "running" || task?.executionRunId !== input.runId
|| !owner || owner.resultId || owner.phase !== "observed"
|| owner.leaseOwner !== (input.leaseOwner ?? null)
|| (input.leaseOwner && (!owner.leaseExpiresAt || owner.leaseExpiresAt <= new Date(clock.now)))
|| run.resultJson?.startupCancellation || stop
|| (["run", "issue"].includes(String(cancellation.scope)) && ["pending", "acknowledged"].includes(String(cancellation.dispatchState)))
|| deferrals >= input.maxDeferrals) return "refused";
await notifyNativeFinalizationWork(tx, { companyId: input.companyId, issueId: current.issueId, runId: input.runId });
await tx.update(heartbeatRuns).set({ contextSnapshot: { ...context, agentDirectoryBusyDeferrals: deferrals + 1 }, updatedAt: new Date() })
.where(eq(heartbeatRuns.id, input.runId));
const detail = record(owner.failureDetail);
await tx.update(nativeRunFinalizations).set({
phase: "retryable_failure", leaseOwner: null, leaseExpiresAt: null, controlDeadlineAt: null,
nextAttemptAt: input.retryAt, recoveryState: "resuming_session", failureCode: NATIVE_AGENT_DIRECTORY_BUSY_WAIT,
failureDetail: {
recoveryMode: record(run.runnerProfileJson).sessionCheckpoint ? "same_session_resume" : "bootstrap_retry",
providerSessionEstablished: false, providerEventsExist: false,
...detail,
originalFailureCode: detail.originalFailureCode ?? NATIVE_AGENT_DIRECTORY_BUSY_WAIT,
message: "Waiting for another run of this agent to finish with its files.",
},
updatedAt: new Date(),
}).where(eq(nativeRunFinalizations.runId, input.runId));
return "deferred";
});
}
/** The retained Runner keeps its state under the directory its own lease
* recorded. A workspace can move to a new allocation of the SAME sandbox: its
* prior epoch was abandoned, so the last save was restored into a fresh root
* named for the new lease. Both roots are canonical siblings under the
* sandbox's `.paperclip-workspaces` and carry the same execution-workspace ID.
* Anything else (another workspace, another parent, a non-canonical path) is
* not the retained allocation and stays refused. */
function sameSandboxWorkspaceAllocation(recorded: string, current: string): boolean {
if (recorded === current) return true;
const canonical = (path: string) => posix.isAbsolute(path) && posix.normalize(path) === path && !path.endsWith("/");
if (!canonical(recorded) || !canonical(current)) return false;
const parent = posix.dirname(recorded);
if (posix.basename(parent) !== ".paperclip-workspaces" || posix.dirname(current) !== parent) return false;
const workspace = WORKSPACE_ALLOCATION.exec(posix.basename(recorded))?.[1];
return Boolean(workspace && WORKSPACE_ALLOCATION.exec(posix.basename(current))?.[1] === workspace);
}
/** The published allocation names (remote-workspaces publishedWorkspaceDirectory):
* exactly `<workspaceId>`, or `<workspaceId>-<epoch>-<leaseId>`. */
const UUID = "[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}";
const WORKSPACE_ALLOCATION = new RegExp(`^(${UUID})(?:-[1-9][0-9]*-${UUID})?$`);
/** Bind a read of retained Runner lifecycle to the surviving sandbox and
* return the directory that Runner wrote under. The ordinary admission owner
* still checks current permissions before execution. */
export async function assertNativeRetainedSandboxBinding(db: Db, input: {
companyId: string; issueId: string; runId: string;
target: Extract<AdapterExecutionTarget, { kind: "remote" }>;
}): Promise<{ remoteCwd: string }> {
const { target } = input;
if (target.transport !== "sandbox" || target.sandboxLeaseAcquisition?.outcome !== "resumed"
|| !target.providerKey || !target.environmentId || !target.leaseId) {
throw new Error("native_controller_recovery_dependency_unavailable:retained_sandbox");
}
const [owner] = await db.select().from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId),
eq(nativeRunFinalizations.runId, input.runId)));
const leases = await db.select({ metadata: environmentLeases.metadata, status: environmentLeases.status }).from(environmentLeases).where(and(
eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.issueId, input.issueId),
eq(environmentLeases.heartbeatRunId, input.runId), eq(environmentLeases.environmentId, target.environmentId),
eq(environmentLeases.provider, target.providerKey), eq(environmentLeases.providerLeaseId, target.sandboxLeaseAcquisition.providerLeaseId)));
const recordedCwd = leases.length === 1 ? leases[0].metadata?.remoteCwd : undefined;
if (!owner || !owner.resultId || owner.recoveryState || typeof recordedCwd !== "string"
|| !sameSandboxWorkspaceAllocation(recordedCwd, target.remoteCwd)
|| !["active", "released", "expired"].includes(leases[0].status)) {
throw new Error("native_controller_recovery_missing_evidence:retained_sandbox_binding");
}
if (owner.leaseOwner && owner.leaseExpiresAt && owner.leaseExpiresAt > new Date()) {
throw new NativeControllerRecoveryWaitingError(owner.leaseExpiresAt);
}
if (owner.leaseOwner || owner.phase !== "committed") {
throw new Error("native_controller_recovery_missing_evidence:retained_sandbox_binding");
}
return { remoteCwd: recordedCwd };
}
/** Only the existing bounded selector chooses history. Never search backwards
* again when its selected record is incomplete or incompatible. This operation
* reads evidence; it grants neither command nor acknowledgement authority. */
export async function resolveNativeControllerRecovery(db: Db, input: {
companyId: string; agentId: string; issueId: string; normalizedSessionId: string; currentRunId: string;
}): Promise<NativeControllerRecoveryOutcome> {
if (!isNativeSessionId(input.normalizedSessionId)) return { kind: "missing_evidence", reason: "session_binding" };
const [current] = await db.select({ createdAt: heartbeatRuns.createdAt, processPid: heartbeatRuns.processPid,
processGroupId: heartbeatRuns.processGroupId, processStartedAt: heartbeatRuns.processStartedAt,
runnerProfileJson: heartbeatRuns.runnerProfileJson }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, input.currentRunId), eq(heartbeatRuns.companyId, input.companyId),
eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.nativeIssueId, input.issueId),
eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId)));
if (!current) return { kind: "missing_evidence", reason: "current_binding" };
const source = await findNativeSessionResumeRun(db, { ...input, beforeCreatedAt: current.createdAt, includeProgressBarrier: true });
if (!source) {
// An exhausted candidate window is unknown history, not fresh-bootstrap
// proof. Read only IDs, and retain the selector's existing 32-row boundary.
const prior = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
ne(heartbeatRuns.id, input.currentRunId), eq(heartbeatRuns.companyId, input.companyId),
eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.nativeIssueId, input.issueId),
eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId))).limit(33);
if (prior.length > 32) return { kind: "missing_evidence", reason: "candidate_window_exhausted" };
const progress = await nativeSessionProviderEvidence(db, [input.currentRunId]);
if (!isUnusedNativeSessionBootstrap(current, progress.has(input.currentRunId))) {
return { kind: "missing_evidence", reason: "current_execution_progress" };
}
// The normal admission and remote untouched-session claim still execute.
return { kind: "no_prior_execution" };
}
const [owner] = await db.select({ leaseOwner: nativeRunFinalizations.leaseOwner, leaseExpiresAt: nativeRunFinalizations.leaseExpiresAt }).from(nativeRunFinalizations)
.where(and(eq(nativeRunFinalizations.runId, source.id), eq(nativeRunFinalizations.companyId, input.companyId),
eq(nativeRunFinalizations.issueId, input.issueId)));
if (owner?.leaseOwner && owner.leaseExpiresAt && owner.leaseExpiresAt.getTime() > Date.now()) {
return { kind: "waiting_for_owner", deadline: owner.leaseExpiresAt };
}
if (source.runtimeMode !== "native" || !["succeeded", "failed", "cancelled", "interrupted", "timed_out"].includes(source.status)
|| source.createdAt > current.createdAt) return { kind: "missing_evidence", reason: "newer_progress_barrier" };
const profile = record(source.runnerProfileJson), snapshot = record(profile.sessionCheckpoint);
const identity = record(snapshot.identity);
if (identity.companyId !== input.companyId || identity.agentId !== input.agentId
|| identity.issueId !== input.issueId || identity.runId !== source.id || identity.sessionId !== input.normalizedSessionId) {
return { kind: "missing_evidence", reason: "snapshot_binding" };
}
if (record(profile.nativeControllerRecovery).schema !== "paperclip.native-controller-recovery.v1") {
return { kind: "unsupported", reason: "metadata_format" };
}
const metadata = readNativeControllerRecovery(profile.nativeControllerRecovery, snapshot);
if (!metadata?.checkpoint || metadata.checkpoint.runId !== source.id
|| metadata.checkpoint.normalizedSessionId !== input.normalizedSessionId
|| metadata.checkpoint.runnerInstanceId !== source.runnerInstanceId) {
return { kind: "missing_evidence", reason: "snapshot_digest_or_checkpoint_binding" };
}
let checkpoint: DurablePrpCheckpoint | null;
try {
checkpoint = await loadNativeRunnerCheckpoint(db, { companyId: input.companyId, issueId: input.issueId, runId: source.id });
} catch (cause) {
return { kind: "dependency_unavailable", reason: "checkpoint_authority_unreadable", cause };
}
if (!checkpoint || checkpoint.recoveryEvidence.runnerInstanceId !== metadata.checkpoint.runnerInstanceId
|| checkpoint.recoveryEvidence.normalizedSessionId !== input.normalizedSessionId) {
return { kind: "missing_evidence", reason: "checkpoint_binding" };
}
return { kind: "recoverable", snapshot: snapshot as unknown as PersistedNativeSession, metadata, checkpoint };
}
/** Bounded internal migration of ONE retained source while its disk is available.
* The migration owner supplies compatibility computed from the admitted input.
* This never repairs/replaces the saved snapshot or claims a live execution. */
export async function importNativeControllerRecovery(db: Db, input: {
companyId: string; agentId: string; issueId: string; runId: string; normalizedSessionId: string;
expectedConfigDigest: string;
supervisor?: unknown;
offline?: { writersStopped: true; legacyControllerState?: string; legacyRunnerArtifact?: { version: string; digest: string } };
legacyCheckpoint?: { legacyControllerState?: string; legacyRunnerArtifact?: { version: string; digest: string } };
}): Promise<{ kind: "imported" | "unchanged" } | Exclude<NativeControllerRecoveryOutcome, { kind: "recoverable" } | { kind: "no_prior_execution" }>> {
if (input.offline && input.legacyCheckpoint) throw new Error("Conflicting native controller migration authority");
const legacy = input.offline ?? input.legacyCheckpoint;
const supervisor = record(input.supervisor);
if (input.supervisor !== undefined && (supervisor.schema !== "paperclip.native-session-supervisor.v1" || supervisor.configDigest !== input.expectedConfigDigest)) {
return { kind: "unsupported", reason: "supervisor_compatibility" };
}
return db.transaction(async tx => {
const [task] = await tx.select({ executionRunId: issues.executionRunId, checkoutRunId: issues.checkoutRunId }).from(issues)
.where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("share");
const [owner] = await tx.select().from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId),
eq(nativeRunFinalizations.runId, input.runId))).for("update");
if (!task || !owner) return { kind: "missing_evidence" as const, reason: "owner_binding" };
if (input.offline) {
const [active] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(offlineMigrationExecutionWhere()).limit(1);
if (!input.offline.writersStopped || active) return { kind: "missing_evidence" as const, reason: "offline_writers_required" };
}
const [clock] = await tx.execute<{ now: Date }>(sql`select clock_timestamp() as now`);
const liveOwner = owner.leaseOwner && (!owner.leaseExpiresAt || owner.leaseExpiresAt > new Date(clock.now));
if (input.offline ? liveOwner : owner.leaseOwner || task.executionRunId) {
return owner.leaseExpiresAt && owner.leaseExpiresAt > new Date(clock.now)
? { kind: "waiting_for_owner" as const, deadline: owner.leaseExpiresAt }
: { kind: "missing_evidence" as const, reason: "normal_ownership_recovery_required" };
}
if (!input.offline && (!(owner.phase === "committed" && owner.resultId) || owner.recoveryState)) {
return { kind: "missing_evidence" as const, reason: "pending_finalization" };
}
if (input.legacyCheckpoint) {
const [active] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, input.companyId), inArray(heartbeatRuns.status, ["queued", "running"]),
or(eq(heartbeatRuns.nativeIssueId, input.issueId), eq(heartbeatRuns.issueId, input.issueId),
eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId)))).limit(1);
if (task.checkoutRunId || active) return { kind: "missing_evidence" as const, reason: "legacy_source_active" };
}
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.runId),
eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.agentId, input.agentId),
eq(heartbeatRuns.nativeIssueId, input.issueId), eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId))).for("update");
const profile = record(run?.runnerProfileJson), snapshot = record(profile.sessionCheckpoint), identity = record(snapshot.identity);
if (run?.runtimeMode !== "native" || !(run.status === "succeeded" || input.legacyCheckpoint && run.status === "failed"
|| input.offline && ["failed", "cancelled", "interrupted", "timed_out"].includes(run.status)) || !run.finishedAt
|| identity.runId !== input.runId || identity.companyId !== input.companyId || identity.issueId !== input.issueId
|| identity.agentId !== input.agentId || identity.sessionId !== input.normalizedSessionId
|| input.supervisor !== undefined && nativeSha256(snapshot) !== nativeSha256(redactRunnerApiPayloads(supervisor.snapshot))) {
return { kind: "missing_evidence" as const, reason: "stale_supervisor_snapshot" };
}
let checkpoint = await loadNativeRunnerCheckpoint(tx, input);
if (input.supervisor === undefined) {
try {
const admitted = parseNativeExecutionInput(profile.nativeExecutionInput);
if (!legacy || (!checkpoint && admitted.session.lifecyclePolicy.mode !== "per_turn")
|| admitted.session.normalizedSessionId !== input.normalizedSessionId
|| admitted.binding.runId !== input.runId || admitted.binding.companyId !== input.companyId
|| admitted.binding.agentId !== input.agentId || admitted.binding.issueId !== input.issueId) {
return { kind: "missing_evidence" as const, reason: "admitted_per_turn_binding" };
}
} catch { return { kind: "unsupported" as const, reason: "admitted_input_format" }; }
}
const importing = !checkpoint && legacy?.legacyControllerState !== undefined;
if (importing) {
if (!run.runnerInstanceId) return { kind: "missing_evidence" as const, reason: "checkpoint_binding" };
checkpoint = captureLegacyDurablePrpCheckpoint(legacy!.legacyControllerState!, {
runId: input.runId, runnerInstanceId: run.runnerInstanceId, normalizedSessionId: input.normalizedSessionId,
}, legacy!.legacyRunnerArtifact);
}
if (!checkpoint || checkpoint.recoveryEvidence.normalizedSessionId !== input.normalizedSessionId
|| checkpoint.recoveryEvidence.runnerInstanceId !== run.runnerInstanceId) {
return { kind: "missing_evidence" as const, reason: "checkpoint_binding" };
}
const metadata = createNativeControllerRecovery(snapshot as unknown as PersistedNativeSession, {
configDigest: input.expectedConfigDigest,
processConfigDigest: typeof supervisor.processConfigDigest === "string" ? supervisor.processConfigDigest : null,
}, { runId: input.runId, runnerInstanceId: checkpoint.recoveryEvidence.runnerInstanceId, normalizedSessionId: input.normalizedSessionId });
if (!readNativeControllerRecovery(metadata, snapshot)) return { kind: "unsupported" as const, reason: "metadata_format" };
if (profile.nativeControllerRecovery) {
if (nativeSha256(profile.nativeControllerRecovery) !== nativeSha256(metadata)) {
return { kind: "missing_evidence" as const, reason: "existing_metadata_conflict" };
}
if (importing) await insertMigratedNativeRunnerCheckpoint(tx, input, checkpoint);
return { kind: importing ? "imported" as const : "unchanged" as const };
}
if (importing) await insertMigratedNativeRunnerCheckpoint(tx, input, checkpoint);
await tx.update(heartbeatRuns).set({ runnerProfileJson: { ...profile, nativeControllerRecovery: metadata }, updatedAt: new Date() })
.where(eq(heartbeatRuns.id, input.runId));
return { kind: "imported" as const };
});
}