mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 18:26:40 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The paid Runner Full-Stack E2E workflow proves chat features against a real source server > - Since masterf7a3497e9d, every paid cell fails before any test runs: the source server cannot find `@paperclipai/plugin-sdk/dist/index.js` > - The supervisor builds the plugin SDK before it starts Paperclip, but its own static imports now load the SDK first > - This pull request removes the heavy import edge, adds a regression test for that class, and builds and checks the SDK once per campaign > - The benefit is that paid campaigns run again, and a missing workspace build fails once in the build job instead of in 199 paid cells ## Linked Issues or Issue Description Refs #21791 **What happened?** Chat-eval cohort 17 (masterf7a3497e9d) failed in all 199 cells. Each cell exited in the "Run paid cell" step. The private Playwright log shows `ERR_MODULE_NOT_FOUND` for `server/node_modules/@paperclipai/plugin-sdk/dist/index.js`, imported from `server/src/services/reusable-sandbox-resume-error.ts`, and then `Process from config.webServer was not able to start`. Cohort 16 (master819f61f1fd) passed 169/199 with the same workflow steps. **Root cause** #21791 (df9914dbd9) added `import { NATIVE_RETAINED_OWNER_WAIT } from "./native-controller-recovery.js"` to `native-runner-checkpoint.ts`. `tests/runner-e2e/server.ts` statically imports `controller-recovery-observer.ts`, which imports the checkpoint reader. The new edge connects it to the full heartbeat graph: `native-runner-checkpoint -> native-controller-recovery -> native-finalization-notifications -> external-agents -> muse -> heartbeat -> heartbeat/run-completion -> reusable-sandbox-resume-error -> @paperclipai/plugin-sdk`. The plugin SDK exports only its generated `dist`. The supervisor builds that `dist` in `prepareSourceServerDependencies`, but ES static imports load before that call. The workflow installs with `--ignore-scripts` and never built the SDK, so the supervisor crashes at load. #21554 is not involved: it changes only release workflows. **Expected behavior** The supervisor loads without generated SDK output. The campaign build job builds and checks the SDK once, and each cell restores it. **Steps to reproduce** 1. Remove `packages/plugins/sdk/dist`. 2. Run `node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/server.ts`. 3. Before this change: `ERR_MODULE_NOT_FOUND` for the SDK. After this change: the normal `PAPERCLIP_RUNNER_E2E_SERVER_LOG is required` error, which shows that all static imports loaded. ## What Changed - Moved `NATIVE_RETAINED_OWNER_WAIT` and `NATIVE_AGENT_DIRECTORY_BUSY_WAIT` to a new leaf module `native-recovery-wait-codes.ts`. `native-controller-recovery.ts` re-exports them, so other importers do not change. `native-runner-checkpoint.ts` imports the leaf. This also removes a checkpoint/controller-recovery import cycle. - Added `tests/runner-e2e/server-startup-import-graph.test.ts`. It uses esbuild to walk the static import graph of `tests/runner-e2e/server.ts` and fails with the full chain if it reaches `@paperclipai/plugin-sdk`. - `runner-full-stack-e2e.yml`: the "Build reusable runner campaign artifacts" job runs `scripts/ensure-plugin-build-deps.mjs` (the same certified helper the cell uses), checks that `shared` and `plugin-sdk` dist and their completion markers exist with a clear `::error::`, and ships both dist trees in the campaign bundle. The cell restore step checks both entry points with a clear `::error::`. The helper's fingerprint receipt lets the cell reuse the shipped output instead of a rebuild. - `workflow-security.test.ts` pins the build, packaging and guard. ## Verification - Red/green: `npx vitest run --config tests/runner-e2e/vitest.config.ts server-startup-import-graph` fails with the old import and passes with the fix. - Local reproduction above: fails before, loads after. - `npx vitest run --config tests/runner-e2e/vitest.config.ts workflow-security server-startup-import-graph native-completion`: 137 passed. - `node --test tests/runner-e2e/native-completion-git-source.test.mjs`: 36 passed. - Server `tsc --noEmit`: clean. - Paid proof of the source fix: one cell (`chat-features.runner-opencode.local.access.private-resource`), dispatched with `--ref master -f target_branch=fix/runner-e2e-plugin-sdk-dist`: https://github.com/paperclipai/paperclip/actions/runs/43804619166 . That run uses the master workflow and this branch's source, so it proves the import-graph fix alone restores the cells. - The workflow changes (SDK build, bundle and guards) cannot run in a paid campaign before merge, because paid runs use the default-branch workflow. `workflow-security.test.ts` pins them. A 1-case paid check from master follows the merge. - The failed "Authorize paid campaign" check on this commit is from a dispatch with a branch `--ref`. The default-branch gate rejected it as designed. It is not a PR-event check. ## Risks - Low risk. The constants keep their values and are re-exported from the old module. - The campaign bundle grows by the `shared` and `plugin-sdk` dist trees (small). If the fingerprints do not match in a cell, the helper rebuilds, as before. ## Model Used - Claude, `claude-opus-5-5`, via Claude Code with tool use (shell, file edit, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
411 lines
28 KiB
TypeScript
411 lines
28 KiB
TypeScript
import { posix } from "node:path";
|
|
import { and, eq, inArray, isNull, ne, or, sql } from "drizzle-orm";
|
|
import { environmentLeases, heartbeatRunEvents, heartbeatRuns, issues, nativeRunFinalizations, nativeRunnerCheckpoints, type Db } from "@paperclipai/db";
|
|
import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
|
|
import { captureLegacyDurablePrpCheckpoint, parseNativeExecutionInput, type DurablePrpCheckpoint, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js";
|
|
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
|
|
import { readRunCancellation } from "../run-cancellation.js";
|
|
import { OWN_PROCESS_START_EVENT_TYPES } from "../native-process-start-events.js";
|
|
import { nativeSha256 } from "./canonical.js";
|
|
import { findNativeSessionResumeRun, isNativeSessionId, isUnusedNativeSessionBootstrap, nativeSessionProviderEvidence } from "./native-session-resume.js";
|
|
import { insertMigratedNativeRunnerCheckpoint, loadNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
|
|
import { offlineMigrationExecutionWhere } from "../offline-migration-execution.js";
|
|
import { redactRunnerApiPayloads } from "./runner-api-log-redaction.js";
|
|
import { NATIVE_AGENT_DIRECTORY_BUSY_WAIT, NATIVE_RETAINED_OWNER_WAIT } from "./native-recovery-wait-codes.js";
|
|
|
|
export { NATIVE_AGENT_DIRECTORY_BUSY_WAIT, NATIVE_RETAINED_OWNER_WAIT };
|
|
|
|
export const NATIVE_CONTROLLER_RECOVERY_MAX_BYTES = 4 * 1024;
|
|
|
|
/** Private run-profile metadata. No credentials or controller-local paths. */
|
|
export interface NativeControllerRecovery {
|
|
schema: "paperclip.native-controller-recovery.v1";
|
|
snapshotSha256: string;
|
|
configDigest: string;
|
|
processConfigDigest: string | null;
|
|
checkpoint: {
|
|
runId: string;
|
|
runnerInstanceId: string;
|
|
normalizedSessionId: string;
|
|
} | null;
|
|
}
|
|
|
|
export type NativeControllerCompatibility = Pick<NativeControllerRecovery, "configDigest" | "processConfigDigest">;
|
|
|
|
export function createNativeControllerRecovery(
|
|
snapshot: PersistedNativeSession,
|
|
compatibility: NativeControllerCompatibility,
|
|
checkpoint: NativeControllerRecovery["checkpoint"],
|
|
): NativeControllerRecovery {
|
|
const value: NativeControllerRecovery = {
|
|
schema: "paperclip.native-controller-recovery.v1",
|
|
snapshotSha256: nativeSha256(snapshot),
|
|
configDigest: compatibility.configDigest,
|
|
processConfigDigest: compatibility.processConfigDigest,
|
|
checkpoint,
|
|
};
|
|
if (Buffer.byteLength(JSON.stringify(value)) > NATIVE_CONTROLLER_RECOVERY_MAX_BYTES) {
|
|
throw new Error("native_controller_recovery_metadata_too_large");
|
|
}
|
|
return value;
|
|
}
|
|
|
|
function record(value: unknown): Record<string, unknown> {
|
|
return value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
|
|
}
|
|
|
|
export function readNativeControllerRecovery(value: unknown, snapshot: unknown): NativeControllerRecovery | null {
|
|
const envelope = record(value), checkpoint = record(envelope.checkpoint);
|
|
if (Buffer.byteLength(JSON.stringify(value) ?? "null") > NATIVE_CONTROLLER_RECOVERY_MAX_BYTES
|
|
|| envelope.schema !== "paperclip.native-controller-recovery.v1"
|
|
|| typeof envelope.snapshotSha256 !== "string" || envelope.snapshotSha256 !== nativeSha256(snapshot)
|
|
|| typeof envelope.configDigest !== "string" || !/^sha256:[a-f0-9]{64}$/.test(envelope.configDigest)
|
|
|| !(envelope.processConfigDigest === null || (typeof envelope.processConfigDigest === "string" && /^[a-f0-9]{64}$/.test(envelope.processConfigDigest)))
|
|
|| ![checkpoint.runId, checkpoint.runnerInstanceId, checkpoint.normalizedSessionId].every(v => typeof v === "string" && v.length > 0)) return null;
|
|
return envelope as unknown as NativeControllerRecovery;
|
|
}
|
|
|
|
export type NativeControllerRecoveryOutcome =
|
|
| { kind: "recoverable"; snapshot: PersistedNativeSession; metadata: NativeControllerRecovery; checkpoint: DurablePrpCheckpoint }
|
|
| { kind: "waiting_for_owner"; deadline: Date }
|
|
| { kind: "missing_evidence" | "unsupported" | "dependency_unavailable"; reason: string; cause?: unknown }
|
|
| { kind: "no_prior_execution" };
|
|
|
|
|
|
export type NativeRetainedOwnerWait = { runId: string; controllerBootId: string };
|
|
|
|
export class NativeControllerRecoveryWaitingError extends Error {
|
|
constructor(readonly retryAt: Date, readonly owner?: NativeRetainedOwnerWait) {
|
|
super(NATIVE_RETAINED_OWNER_WAIT);
|
|
this.name = "NativeControllerRecoveryWaitingError";
|
|
}
|
|
}
|
|
|
|
/** Admission has not crossed the provider boundary. Retain this same business
|
|
* run on the existing retry owner; never turn an ownership wait into a failure
|
|
* or invent a new provider attempt. The usual resumption gates recheck Stop,
|
|
* permissions and task ownership when this deadline arrives. */
|
|
export async function deferNativeControllerRecovery(db: Db, input: {
|
|
companyId: string; agentId: string; issueId: string; runId: string;
|
|
leaseOwner?: string; retryAt: Date; waitingForOwner?: NativeRetainedOwnerWait;
|
|
}): Promise<void> {
|
|
await db.transaction(async tx => {
|
|
const [task] = await tx.select({ executionRunId: issues.executionRunId }).from(issues)
|
|
.where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("update");
|
|
const [owner] = await tx.select().from(nativeRunFinalizations).where(and(
|
|
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId),
|
|
eq(nativeRunFinalizations.runId, input.runId))).for("update");
|
|
const [run] = await tx.select().from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId),
|
|
eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.nativeIssueId, input.issueId))).for("update");
|
|
const [authority] = await tx.select({ runId: nativeRunnerCheckpoints.runId }).from(nativeRunnerCheckpoints)
|
|
.where(and(eq(nativeRunnerCheckpoints.companyId, input.companyId), eq(nativeRunnerCheckpoints.runId, input.runId))).limit(1);
|
|
const progress = await nativeSessionProviderEvidence(tx as unknown as Db, [input.runId]);
|
|
const cancellation = record(run?.resultJson?.nativeCancellation);
|
|
const stop = readRunCancellation(run?.resultJson);
|
|
const [clock] = await tx.execute<{ now: Date }>(sql`select clock_timestamp() as now`);
|
|
const hasProcessIdentity = Boolean(run?.processPid || run?.processGroupId || run?.processStartedAt);
|
|
const inheritedProcessIdentity = hasProcessIdentity
|
|
&& typeof record(run?.runnerProfileJson).inheritedProcessIdentityRunId === "string"
|
|
&& !(await tx.select({ id: heartbeatRunEvents.id }).from(heartbeatRunEvents).where(and(
|
|
eq(heartbeatRunEvents.companyId, input.companyId), eq(heartbeatRunEvents.runId, input.runId),
|
|
isNull(heartbeatRunEvents.sourceEventId),
|
|
inArray(heartbeatRunEvents.eventType, [...OWN_PROCESS_START_EVENT_TYPES]))).limit(1)).length;
|
|
if (!run || run.runtimeMode !== "native" || run.status !== "running" || task?.executionRunId !== input.runId
|
|
|| !owner || owner.phase !== "observed" || owner.resultId || owner.attempt !== 0
|
|
|| owner.controllerGeneration !== 0 || owner.leaseOwner !== (input.leaseOwner ?? null)
|
|
|| (input.leaseOwner && (!owner.leaseExpiresAt || owner.leaseExpiresAt <= new Date(clock.now)))
|
|
|| (hasProcessIdentity && !inheritedProcessIdentity) || authority || progress.size
|
|
|| !record(run.runnerProfileJson).nativeExecutionInput || run.resultJson?.startupCancellation
|
|
|| (stop?.source === "operator" && stop.expected)
|
|
|| (["run", "issue"].includes(String(cancellation.scope)) && ["pending", "acknowledged"].includes(String(cancellation.dispatchState)))) {
|
|
throw new Error("native_controller_recovery_wait_authority_changed");
|
|
}
|
|
if (inheritedProcessIdentity) {
|
|
await tx.update(heartbeatRuns).set({ processPid: null, processGroupId: null, processStartedAt: null, updatedAt: new Date() })
|
|
.where(eq(heartbeatRuns.id, input.runId));
|
|
}
|
|
await notifyNativeFinalizationWork(tx, input);
|
|
await tx.update(nativeRunFinalizations).set({
|
|
phase: "retryable_failure", leaseOwner: null, leaseExpiresAt: null, controlDeadlineAt: null,
|
|
nextAttemptAt: input.retryAt, recoveryState: "resuming_session", failureCode: NATIVE_RETAINED_OWNER_WAIT,
|
|
failureDetail: { originalFailureCode: NATIVE_RETAINED_OWNER_WAIT,
|
|
message: "Waiting for the retained native controller owner to release its lease.",
|
|
recoveryMode: record(run.runnerProfileJson).sessionCheckpoint ? "same_session_resume" : "bootstrap_retry",
|
|
providerSessionEstablished: false, providerEventsExist: false,
|
|
...(input.waitingForOwner ? { waitingForOwner: { runId: input.waitingForOwner.runId,
|
|
controllerBootId: input.waitingForOwner.controllerBootId } } : {}) },
|
|
updatedAt: new Date(),
|
|
}).where(eq(nativeRunFinalizations.runId, input.runId));
|
|
});
|
|
}
|
|
|
|
|
|
export async function deferNativeRunForBusyAgentDirectory(db: Db, input: {
|
|
companyId: string; runId: string; leaseOwner?: string; retryAt: Date; maxDeferrals: number;
|
|
}): Promise<"deferred" | "pre_admission" | "refused"> {
|
|
return db.transaction(async tx => {
|
|
const [current] = await tx.select({ issueId: heartbeatRuns.nativeIssueId }).from(heartbeatRuns)
|
|
.where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId)));
|
|
const [existing] = await tx.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations)
|
|
.where(and(eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, input.runId)));
|
|
if (!current?.issueId || !existing) return "pre_admission";
|
|
const [task] = await tx.select({ executionRunId: issues.executionRunId }).from(issues)
|
|
.where(and(eq(issues.companyId, input.companyId), eq(issues.id, current.issueId))).for("update");
|
|
const [owner] = await tx.select().from(nativeRunFinalizations).where(and(
|
|
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, current.issueId),
|
|
eq(nativeRunFinalizations.runId, input.runId))).for("update");
|
|
const [run] = await tx.select().from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId),
|
|
eq(heartbeatRuns.nativeIssueId, current.issueId))).for("update");
|
|
const cancellation = record(run?.resultJson?.nativeCancellation);
|
|
const stop = readRunCancellation(run?.resultJson);
|
|
const context = record(run?.contextSnapshot);
|
|
const deferrals = typeof context.agentDirectoryBusyDeferrals === "number" ? context.agentDirectoryBusyDeferrals : 0;
|
|
const [clock] = await tx.execute<{ now: Date }>(sql`select clock_timestamp() as now`);
|
|
if (!run || run.runtimeMode !== "native" || run.status !== "running" || task?.executionRunId !== input.runId
|
|
|| !owner || owner.resultId || owner.phase !== "observed"
|
|
|| owner.leaseOwner !== (input.leaseOwner ?? null)
|
|
|| (input.leaseOwner && (!owner.leaseExpiresAt || owner.leaseExpiresAt <= new Date(clock.now)))
|
|
|| run.resultJson?.startupCancellation || stop
|
|
|| (["run", "issue"].includes(String(cancellation.scope)) && ["pending", "acknowledged"].includes(String(cancellation.dispatchState)))
|
|
|| deferrals >= input.maxDeferrals) return "refused";
|
|
await notifyNativeFinalizationWork(tx, { companyId: input.companyId, issueId: current.issueId, runId: input.runId });
|
|
await tx.update(heartbeatRuns).set({ contextSnapshot: { ...context, agentDirectoryBusyDeferrals: deferrals + 1 }, updatedAt: new Date() })
|
|
.where(eq(heartbeatRuns.id, input.runId));
|
|
const detail = record(owner.failureDetail);
|
|
await tx.update(nativeRunFinalizations).set({
|
|
phase: "retryable_failure", leaseOwner: null, leaseExpiresAt: null, controlDeadlineAt: null,
|
|
nextAttemptAt: input.retryAt, recoveryState: "resuming_session", failureCode: NATIVE_AGENT_DIRECTORY_BUSY_WAIT,
|
|
failureDetail: {
|
|
recoveryMode: record(run.runnerProfileJson).sessionCheckpoint ? "same_session_resume" : "bootstrap_retry",
|
|
providerSessionEstablished: false, providerEventsExist: false,
|
|
...detail,
|
|
originalFailureCode: detail.originalFailureCode ?? NATIVE_AGENT_DIRECTORY_BUSY_WAIT,
|
|
message: "Waiting for another run of this agent to finish with its files.",
|
|
},
|
|
updatedAt: new Date(),
|
|
}).where(eq(nativeRunFinalizations.runId, input.runId));
|
|
return "deferred";
|
|
});
|
|
}
|
|
|
|
/** The retained Runner keeps its state under the directory its own lease
|
|
* recorded. A workspace can move to a new allocation of the SAME sandbox: its
|
|
* prior epoch was abandoned, so the last save was restored into a fresh root
|
|
* named for the new lease. Both roots are canonical siblings under the
|
|
* sandbox's `.paperclip-workspaces` and carry the same execution-workspace ID.
|
|
* Anything else (another workspace, another parent, a non-canonical path) is
|
|
* not the retained allocation and stays refused. */
|
|
function sameSandboxWorkspaceAllocation(recorded: string, current: string): boolean {
|
|
if (recorded === current) return true;
|
|
const canonical = (path: string) => posix.isAbsolute(path) && posix.normalize(path) === path && !path.endsWith("/");
|
|
if (!canonical(recorded) || !canonical(current)) return false;
|
|
const parent = posix.dirname(recorded);
|
|
if (posix.basename(parent) !== ".paperclip-workspaces" || posix.dirname(current) !== parent) return false;
|
|
const workspace = WORKSPACE_ALLOCATION.exec(posix.basename(recorded))?.[1];
|
|
return Boolean(workspace && WORKSPACE_ALLOCATION.exec(posix.basename(current))?.[1] === workspace);
|
|
}
|
|
|
|
/** The published allocation names (remote-workspaces publishedWorkspaceDirectory):
|
|
* exactly `<workspaceId>`, or `<workspaceId>-<epoch>-<leaseId>`. */
|
|
const UUID = "[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}";
|
|
const WORKSPACE_ALLOCATION = new RegExp(`^(${UUID})(?:-[1-9][0-9]*-${UUID})?$`);
|
|
|
|
/** Bind a read of retained Runner lifecycle to the surviving sandbox and
|
|
* return the directory that Runner wrote under. The ordinary admission owner
|
|
* still checks current permissions before execution. */
|
|
export async function assertNativeRetainedSandboxBinding(db: Db, input: {
|
|
companyId: string; issueId: string; runId: string;
|
|
target: Extract<AdapterExecutionTarget, { kind: "remote" }>;
|
|
}): Promise<{ remoteCwd: string }> {
|
|
const { target } = input;
|
|
if (target.transport !== "sandbox" || target.sandboxLeaseAcquisition?.outcome !== "resumed"
|
|
|| !target.providerKey || !target.environmentId || !target.leaseId) {
|
|
throw new Error("native_controller_recovery_dependency_unavailable:retained_sandbox");
|
|
}
|
|
const [owner] = await db.select().from(nativeRunFinalizations).where(and(
|
|
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId),
|
|
eq(nativeRunFinalizations.runId, input.runId)));
|
|
const leases = await db.select({ metadata: environmentLeases.metadata, status: environmentLeases.status }).from(environmentLeases).where(and(
|
|
eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.issueId, input.issueId),
|
|
eq(environmentLeases.heartbeatRunId, input.runId), eq(environmentLeases.environmentId, target.environmentId),
|
|
eq(environmentLeases.provider, target.providerKey), eq(environmentLeases.providerLeaseId, target.sandboxLeaseAcquisition.providerLeaseId)));
|
|
const recordedCwd = leases.length === 1 ? leases[0].metadata?.remoteCwd : undefined;
|
|
if (!owner || !owner.resultId || owner.recoveryState || typeof recordedCwd !== "string"
|
|
|| !sameSandboxWorkspaceAllocation(recordedCwd, target.remoteCwd)
|
|
|| !["active", "released", "expired"].includes(leases[0].status)) {
|
|
throw new Error("native_controller_recovery_missing_evidence:retained_sandbox_binding");
|
|
}
|
|
if (owner.leaseOwner && owner.leaseExpiresAt && owner.leaseExpiresAt > new Date()) {
|
|
throw new NativeControllerRecoveryWaitingError(owner.leaseExpiresAt);
|
|
}
|
|
if (owner.leaseOwner || owner.phase !== "committed") {
|
|
throw new Error("native_controller_recovery_missing_evidence:retained_sandbox_binding");
|
|
}
|
|
return { remoteCwd: recordedCwd };
|
|
}
|
|
|
|
/** Only the existing bounded selector chooses history. Never search backwards
|
|
* again when its selected record is incomplete or incompatible. This operation
|
|
* reads evidence; it grants neither command nor acknowledgement authority. */
|
|
export async function resolveNativeControllerRecovery(db: Db, input: {
|
|
companyId: string; agentId: string; issueId: string; normalizedSessionId: string; currentRunId: string;
|
|
}): Promise<NativeControllerRecoveryOutcome> {
|
|
if (!isNativeSessionId(input.normalizedSessionId)) return { kind: "missing_evidence", reason: "session_binding" };
|
|
const [current] = await db.select({ createdAt: heartbeatRuns.createdAt, processPid: heartbeatRuns.processPid,
|
|
processGroupId: heartbeatRuns.processGroupId, processStartedAt: heartbeatRuns.processStartedAt,
|
|
runnerProfileJson: heartbeatRuns.runnerProfileJson }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, input.currentRunId), eq(heartbeatRuns.companyId, input.companyId),
|
|
eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.nativeIssueId, input.issueId),
|
|
eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId)));
|
|
if (!current) return { kind: "missing_evidence", reason: "current_binding" };
|
|
const source = await findNativeSessionResumeRun(db, { ...input, beforeCreatedAt: current.createdAt, includeProgressBarrier: true });
|
|
if (!source) {
|
|
// An exhausted candidate window is unknown history, not fresh-bootstrap
|
|
// proof. Read only IDs, and retain the selector's existing 32-row boundary.
|
|
const prior = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
|
ne(heartbeatRuns.id, input.currentRunId), eq(heartbeatRuns.companyId, input.companyId),
|
|
eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.nativeIssueId, input.issueId),
|
|
eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId))).limit(33);
|
|
if (prior.length > 32) return { kind: "missing_evidence", reason: "candidate_window_exhausted" };
|
|
const progress = await nativeSessionProviderEvidence(db, [input.currentRunId]);
|
|
if (!isUnusedNativeSessionBootstrap(current, progress.has(input.currentRunId))) {
|
|
return { kind: "missing_evidence", reason: "current_execution_progress" };
|
|
}
|
|
// The normal admission and remote untouched-session claim still execute.
|
|
return { kind: "no_prior_execution" };
|
|
}
|
|
const [owner] = await db.select({ leaseOwner: nativeRunFinalizations.leaseOwner, leaseExpiresAt: nativeRunFinalizations.leaseExpiresAt }).from(nativeRunFinalizations)
|
|
.where(and(eq(nativeRunFinalizations.runId, source.id), eq(nativeRunFinalizations.companyId, input.companyId),
|
|
eq(nativeRunFinalizations.issueId, input.issueId)));
|
|
if (owner?.leaseOwner && owner.leaseExpiresAt && owner.leaseExpiresAt.getTime() > Date.now()) {
|
|
return { kind: "waiting_for_owner", deadline: owner.leaseExpiresAt };
|
|
}
|
|
if (source.runtimeMode !== "native" || !["succeeded", "failed", "cancelled", "interrupted", "timed_out"].includes(source.status)
|
|
|| source.createdAt > current.createdAt) return { kind: "missing_evidence", reason: "newer_progress_barrier" };
|
|
const profile = record(source.runnerProfileJson), snapshot = record(profile.sessionCheckpoint);
|
|
const identity = record(snapshot.identity);
|
|
if (identity.companyId !== input.companyId || identity.agentId !== input.agentId
|
|
|| identity.issueId !== input.issueId || identity.runId !== source.id || identity.sessionId !== input.normalizedSessionId) {
|
|
return { kind: "missing_evidence", reason: "snapshot_binding" };
|
|
}
|
|
if (record(profile.nativeControllerRecovery).schema !== "paperclip.native-controller-recovery.v1") {
|
|
return { kind: "unsupported", reason: "metadata_format" };
|
|
}
|
|
const metadata = readNativeControllerRecovery(profile.nativeControllerRecovery, snapshot);
|
|
if (!metadata?.checkpoint || metadata.checkpoint.runId !== source.id
|
|
|| metadata.checkpoint.normalizedSessionId !== input.normalizedSessionId
|
|
|| metadata.checkpoint.runnerInstanceId !== source.runnerInstanceId) {
|
|
return { kind: "missing_evidence", reason: "snapshot_digest_or_checkpoint_binding" };
|
|
}
|
|
let checkpoint: DurablePrpCheckpoint | null;
|
|
try {
|
|
checkpoint = await loadNativeRunnerCheckpoint(db, { companyId: input.companyId, issueId: input.issueId, runId: source.id });
|
|
} catch (cause) {
|
|
return { kind: "dependency_unavailable", reason: "checkpoint_authority_unreadable", cause };
|
|
}
|
|
if (!checkpoint || checkpoint.recoveryEvidence.runnerInstanceId !== metadata.checkpoint.runnerInstanceId
|
|
|| checkpoint.recoveryEvidence.normalizedSessionId !== input.normalizedSessionId) {
|
|
return { kind: "missing_evidence", reason: "checkpoint_binding" };
|
|
}
|
|
return { kind: "recoverable", snapshot: snapshot as unknown as PersistedNativeSession, metadata, checkpoint };
|
|
}
|
|
|
|
/** Bounded internal migration of ONE retained source while its disk is available.
|
|
* The migration owner supplies compatibility computed from the admitted input.
|
|
* This never repairs/replaces the saved snapshot or claims a live execution. */
|
|
export async function importNativeControllerRecovery(db: Db, input: {
|
|
companyId: string; agentId: string; issueId: string; runId: string; normalizedSessionId: string;
|
|
expectedConfigDigest: string;
|
|
supervisor?: unknown;
|
|
offline?: { writersStopped: true; legacyControllerState?: string; legacyRunnerArtifact?: { version: string; digest: string } };
|
|
legacyCheckpoint?: { legacyControllerState?: string; legacyRunnerArtifact?: { version: string; digest: string } };
|
|
}): Promise<{ kind: "imported" | "unchanged" } | Exclude<NativeControllerRecoveryOutcome, { kind: "recoverable" } | { kind: "no_prior_execution" }>> {
|
|
if (input.offline && input.legacyCheckpoint) throw new Error("Conflicting native controller migration authority");
|
|
const legacy = input.offline ?? input.legacyCheckpoint;
|
|
const supervisor = record(input.supervisor);
|
|
if (input.supervisor !== undefined && (supervisor.schema !== "paperclip.native-session-supervisor.v1" || supervisor.configDigest !== input.expectedConfigDigest)) {
|
|
return { kind: "unsupported", reason: "supervisor_compatibility" };
|
|
}
|
|
return db.transaction(async tx => {
|
|
const [task] = await tx.select({ executionRunId: issues.executionRunId, checkoutRunId: issues.checkoutRunId }).from(issues)
|
|
.where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("share");
|
|
const [owner] = await tx.select().from(nativeRunFinalizations).where(and(
|
|
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId),
|
|
eq(nativeRunFinalizations.runId, input.runId))).for("update");
|
|
if (!task || !owner) return { kind: "missing_evidence" as const, reason: "owner_binding" };
|
|
if (input.offline) {
|
|
const [active] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(offlineMigrationExecutionWhere()).limit(1);
|
|
if (!input.offline.writersStopped || active) return { kind: "missing_evidence" as const, reason: "offline_writers_required" };
|
|
}
|
|
const [clock] = await tx.execute<{ now: Date }>(sql`select clock_timestamp() as now`);
|
|
const liveOwner = owner.leaseOwner && (!owner.leaseExpiresAt || owner.leaseExpiresAt > new Date(clock.now));
|
|
if (input.offline ? liveOwner : owner.leaseOwner || task.executionRunId) {
|
|
return owner.leaseExpiresAt && owner.leaseExpiresAt > new Date(clock.now)
|
|
? { kind: "waiting_for_owner" as const, deadline: owner.leaseExpiresAt }
|
|
: { kind: "missing_evidence" as const, reason: "normal_ownership_recovery_required" };
|
|
}
|
|
if (!input.offline && (!(owner.phase === "committed" && owner.resultId) || owner.recoveryState)) {
|
|
return { kind: "missing_evidence" as const, reason: "pending_finalization" };
|
|
}
|
|
if (input.legacyCheckpoint) {
|
|
const [active] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, input.companyId), inArray(heartbeatRuns.status, ["queued", "running"]),
|
|
or(eq(heartbeatRuns.nativeIssueId, input.issueId), eq(heartbeatRuns.issueId, input.issueId),
|
|
eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId)))).limit(1);
|
|
if (task.checkoutRunId || active) return { kind: "missing_evidence" as const, reason: "legacy_source_active" };
|
|
}
|
|
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.runId),
|
|
eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.agentId, input.agentId),
|
|
eq(heartbeatRuns.nativeIssueId, input.issueId), eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId))).for("update");
|
|
const profile = record(run?.runnerProfileJson), snapshot = record(profile.sessionCheckpoint), identity = record(snapshot.identity);
|
|
if (run?.runtimeMode !== "native" || !(run.status === "succeeded" || input.legacyCheckpoint && run.status === "failed"
|
|
|| input.offline && ["failed", "cancelled", "interrupted", "timed_out"].includes(run.status)) || !run.finishedAt
|
|
|| identity.runId !== input.runId || identity.companyId !== input.companyId || identity.issueId !== input.issueId
|
|
|| identity.agentId !== input.agentId || identity.sessionId !== input.normalizedSessionId
|
|
|| input.supervisor !== undefined && nativeSha256(snapshot) !== nativeSha256(redactRunnerApiPayloads(supervisor.snapshot))) {
|
|
return { kind: "missing_evidence" as const, reason: "stale_supervisor_snapshot" };
|
|
}
|
|
let checkpoint = await loadNativeRunnerCheckpoint(tx, input);
|
|
if (input.supervisor === undefined) {
|
|
try {
|
|
const admitted = parseNativeExecutionInput(profile.nativeExecutionInput);
|
|
if (!legacy || (!checkpoint && admitted.session.lifecyclePolicy.mode !== "per_turn")
|
|
|| admitted.session.normalizedSessionId !== input.normalizedSessionId
|
|
|| admitted.binding.runId !== input.runId || admitted.binding.companyId !== input.companyId
|
|
|| admitted.binding.agentId !== input.agentId || admitted.binding.issueId !== input.issueId) {
|
|
return { kind: "missing_evidence" as const, reason: "admitted_per_turn_binding" };
|
|
}
|
|
} catch { return { kind: "unsupported" as const, reason: "admitted_input_format" }; }
|
|
}
|
|
const importing = !checkpoint && legacy?.legacyControllerState !== undefined;
|
|
if (importing) {
|
|
if (!run.runnerInstanceId) return { kind: "missing_evidence" as const, reason: "checkpoint_binding" };
|
|
checkpoint = captureLegacyDurablePrpCheckpoint(legacy!.legacyControllerState!, {
|
|
runId: input.runId, runnerInstanceId: run.runnerInstanceId, normalizedSessionId: input.normalizedSessionId,
|
|
}, legacy!.legacyRunnerArtifact);
|
|
}
|
|
if (!checkpoint || checkpoint.recoveryEvidence.normalizedSessionId !== input.normalizedSessionId
|
|
|| checkpoint.recoveryEvidence.runnerInstanceId !== run.runnerInstanceId) {
|
|
return { kind: "missing_evidence" as const, reason: "checkpoint_binding" };
|
|
}
|
|
const metadata = createNativeControllerRecovery(snapshot as unknown as PersistedNativeSession, {
|
|
configDigest: input.expectedConfigDigest,
|
|
processConfigDigest: typeof supervisor.processConfigDigest === "string" ? supervisor.processConfigDigest : null,
|
|
}, { runId: input.runId, runnerInstanceId: checkpoint.recoveryEvidence.runnerInstanceId, normalizedSessionId: input.normalizedSessionId });
|
|
if (!readNativeControllerRecovery(metadata, snapshot)) return { kind: "unsupported" as const, reason: "metadata_format" };
|
|
if (profile.nativeControllerRecovery) {
|
|
if (nativeSha256(profile.nativeControllerRecovery) !== nativeSha256(metadata)) {
|
|
return { kind: "missing_evidence" as const, reason: "existing_metadata_conflict" };
|
|
}
|
|
if (importing) await insertMigratedNativeRunnerCheckpoint(tx, input, checkpoint);
|
|
return { kind: importing ? "imported" as const : "unchanged" as const };
|
|
}
|
|
if (importing) await insertMigratedNativeRunnerCheckpoint(tx, input, checkpoint);
|
|
await tx.update(heartbeatRuns).set({ runnerProfileJson: { ...profile, nativeControllerRecovery: metadata }, updatedAt: new Date() })
|
|
.where(eq(heartbeatRuns.id, input.runId));
|
|
return { kind: "imported" as const };
|
|
});
|
|
}
|