mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Cloud servers run volume-free (`PAPERCLIP_EPHEMERAL_SERVER=true`). Postgres and object storage hold the durable state. A startup gate refuses legacy volume data that was not migrated. > - The gate reads one family of data from disk: Runner controller files under `runtime/paperclip-runner/durable-sessions/`. It treats a controller file that the database does not prove as legacy data. > - A volume-free server also writes these files during normal work. A warm follow-up restores the previous run's checkpoint as a full `durable.control-plane-state.v2` file. The file stays full until the new controller registers its database checkpoint writer. > - When the server crashes in that window and restarts on the same filesystem, the gate finds its own file and refuses to start. Every later start refuses the same way. This is a full outage that no Retry can fix. > - This pull request gives the gate a durable marker. A volume-free server claims an empty state directory before any controller can write. Files in a claimed directory belong to a volume-free server and do not block startup. > - The benefit is that a crash restart of a volume-free server starts again. Genuine legacy data keeps its protection, and the refusal now names the run, task and file. ## Linked Issues or Issue Description **What happened?** The first restart-chaos waves in volume-free server mode (harness flag `--ephemeral-server`, #20930) found this outage. Wave `w-20261111T212429-s6402` (Claude through ACPX, Daytona warm sandboxes) did a fresh-home deploy and then a `kill -9` of that server. The replacement server exited with `Volume-free server data is not eligible: native_controllers_require_migration`. A second start at +608 s exited the same way. The server stayed down. **Root cause** `hasLocalNativeControllerObligations` in `server/src/services/native-runtime/native-controller-migration.ts` scans every Runner session in the state directory. The kept instance shows the cause. The session's newest run `4a4d7ca1` was `running` (admitted 3 s before the kill). Its controller root held the full `paperclip.runner.durable.control-plane-state.v2` file of its warm predecessor `9c13b7f8`. A controller writes that full file when it opens and replaces it with the `external-control-plane-state.v1` marker only in `setCheckpointPersistence`. `preservedActiveCheckpoint` accepts only the marker, so the gate returned `true`. A Codex wave with the same fault sequence passed because no warm follow-up was in this window at the kill. Claude W2 cells start a new warm follow-up every 15 to 35 s, so the window is open much more often. **Can staging hit it?** Yes, with a restart that keeps the container. Railway documents that a manual Restart "restarts the process within the deployment's container", so the filesystem stays. Railway does not document whether an automatic crash restart reuses the container. A redeploy makes a new container, so it does not hit this. **Expected behavior** A volume-free server starts again after any restart. State that it wrote itself is disposable, because the database checkpoint is the durable record. Data that a volume-backed server wrote keeps its current protection. ## What Changed - Add `server/src/services/native-runtime/volume-free-controller-state.ts`. `claimVolumeFreeControllerState(true)` writes `.volume-free-owner.json` atomically (temporary file, fsync, rename, directory fsync) when the state directory is absent or empty. It returns `owned` for a directory that has the marker and `unclaimed` for a directory with other files. `claimVolumeFreeControllerState(false)` removes the marker. - `server/src/index.ts` calls the claim right after the volume-free configuration check. This is before any controller or recovery can write. A server started without volume-free mode releases the claim, because it may write files that exist only on its disk. An `unclaimed` result logs a warning. - `hasLocalNativeControllerObligations` ignores the marker entries. It returns no obligation for a claimed directory. It now delegates to `findLocalNativeControllerObligation`, which returns the run, task and file (or `database_checkpoint`). - `assertEphemeralServerData` adds that detail to the `native_controllers_require_migration` refusal. The error says which run and file the database does not prove and why startup refuses. - Document the marker in `doc/DEVELOPING.md` ("Runner controller state on a restarted volume-free server"). ## Verification - Red, before the gate reads the marker: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-controller-migration.integration.test.ts -t volume_free_owner` failed with `AssertionError: expected true to be false` at the first gate check. The fixture is a running warm run whose root holds a full `durable.control-plane-state.v2` file, the same shape as the incident. - Green: the same test passes. It checks: claimed directory gives no obligation; release gives the obligation back; a non-empty unclaimed directory stays blocked; the refusal names the run; the controller file is never changed or removed. - `ephemeral-server-data.test.ts`, `ephemeral-server-contract.test.ts` and all 57 cells of `native-controller-migration.integration.test.ts` pass. Server `tsc --noEmit` passes. - Probe on a copy of the kept instance database and the killed server's state directory: the unmodified gate returned an obligation for run `4a4d7ca1` (source file `.../7dbf2d5c.../control-plane/control-plane-state.json`). - Chaos: the s6402 shape (`--ephemeral-server --adapters claude --workloads W2 --faults K1,K2,K15 --k15-fresh-home --seed 6402`) runs on this branch merged with #20930. The result is added in a comment. ## Risks - A filesystem that a volume-free server claimed and that a pre-marker, volume-backed server then wrote is not detected. This needs a downgrade and a mode change on one disk. A current server in volume mode removes the marker, which closes the normal path. - A volume-free server on a directory written before this change (no marker) keeps today's refusal. A new deploy makes a new container, so staging claims the empty directory on first start after this deploy. - Other scanners of the state directory filter on the 64-hex session pattern, so the marker file does not change them. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edit, test runs). Default context window, extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
488 lines
35 KiB
TypeScript
488 lines
35 KiB
TypeScript
import { constants } from "node:fs";
|
|
import { lstat, open, readdir } from "node:fs/promises";
|
|
import { join } from "node:path";
|
|
import { and, eq, inArray, or, sql } from "drizzle-orm";
|
|
import { heartbeatRuns, issues, nativeRunFinalizations, type Db } from "@paperclipai/db";
|
|
import { remoteWorkspaceRunReferenceSchema } from "@paperclipai/shared";
|
|
import { captureLegacyDurablePrpCheckpoint, inspectDurablePrpCheckpointHeader, parseNativeExecutionInput } from "../../vendor/paperclip-runner/index.js";
|
|
import { offlineMigrationExecutionWhere } from "../offline-migration-execution.js";
|
|
import { findNativeSessionResumeRun, isNativeSessionId } from "./native-session-resume.js";
|
|
import { nativeControllerMigrationDescriptor } from "./native-session-executor.js";
|
|
import { hasNativeRunnerCheckpoint, loadNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
|
|
import { importNativeControllerRecovery, readNativeControllerRecovery } from "./native-controller-recovery.js";
|
|
import { nativeHarnessStateBase } from "./native-harness-backup-stamp.js";
|
|
import { providerPackWorkspaceId, resolveRemoteProviderPackAuthority } from "./remote-provider-pack-authority.js";
|
|
import { nativeSha256 } from "./canonical.js";
|
|
import { readNativeWorkspaceSyncReference } from "./native-workspace-sync.js";
|
|
import type { StorageService } from "../../storage/types.js";
|
|
import { durableNativeHarnessStorage } from "./native-harness-backup-storage.js";
|
|
import { archiveFailedLegacyNativeController, hasArchivedLegacyNativeController } from "./native-controller-archive.js";
|
|
import { redactRunnerApiPayloads } from "./runner-api-log-redaction.js";
|
|
import { archiveSettledFailedNativeController } from "./native-controller-archive-source.js";
|
|
import { nativeControllerSourceDigest } from "./native-controller-source-digest.js";
|
|
import { isVolumeFreeControllerStateOwnerEntry, volumeFreeControllerStateOwned } from "./volume-free-controller-state.js";
|
|
|
|
type Selected = { id: string; companyId: string; agentId: string; issueId: string; sessionId: string; createdAt: Date; lastRunId: string };
|
|
|
|
function record(value: unknown): Record<string, unknown> {
|
|
return value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
|
|
}
|
|
|
|
async function confirmedOlderSuccessfulSnapshot(db: Db, selected: Selected,
|
|
source: NonNullable<Awaited<ReturnType<typeof findNativeSessionResumeRun>>>,
|
|
paths: Awaited<ReturnType<typeof localPaths>>, olderRunId: unknown) {
|
|
const snapshot = record(source.runnerProfileJson.sessionCheckpoint), identity = record(snapshot.identity);
|
|
const scope = { companyId: selected.companyId, agentId: selected.agentId, issueId: selected.issueId, sessionId: selected.sessionId };
|
|
if (source.status !== "failed" || identity.runId !== source.id
|
|
|| Object.entries(scope).some(([key, value]) => identity[key] !== value)
|
|
|| paths.execution.binding.runId !== source.id || paths.execution.binding.companyId !== selected.companyId
|
|
|| paths.execution.binding.agentId !== selected.agentId || paths.execution.binding.issueId !== selected.issueId
|
|
|| paths.execution.session.normalizedSessionId !== selected.sessionId
|
|
|| typeof snapshot.sessionId !== "string" || !snapshot.sessionId.trim()
|
|
|| typeof olderRunId !== "string" || olderRunId === source.id
|
|
|| !/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/i.test(olderRunId)) return null;
|
|
const [prior] = await db.select({ createdAt: heartbeatRuns.createdAt, finishedAt: heartbeatRuns.finishedAt,
|
|
runnerInstanceId: heartbeatRuns.runnerInstanceId, snapshot: sql<unknown>`${heartbeatRuns.runnerProfileJson}->'sessionCheckpoint'` })
|
|
.from(heartbeatRuns).where(and(eq(heartbeatRuns.id, olderRunId), eq(heartbeatRuns.companyId, selected.companyId),
|
|
eq(heartbeatRuns.agentId, selected.agentId), eq(heartbeatRuns.nativeIssueId, selected.issueId),
|
|
eq(heartbeatRuns.nativeSessionId, selected.sessionId), eq(heartbeatRuns.runtimeMode, "native"), eq(heartbeatRuns.status, "succeeded")));
|
|
const priorSnapshot = record(prior?.snapshot), priorIdentity = record(priorSnapshot.identity);
|
|
if (!prior?.finishedAt || prior.createdAt >= source.createdAt
|
|
|| !source.runnerInstanceId || prior.runnerInstanceId !== source.runnerInstanceId
|
|
|| priorIdentity.runId !== olderRunId || Object.entries(scope).some(([key, value]) => priorIdentity[key] !== value)
|
|
|| priorSnapshot.sessionId !== snapshot.sessionId || priorSnapshot.backendKind !== snapshot.backendKind) return null;
|
|
const [failed] = await db.select({ finishedAt: heartbeatRuns.finishedAt, startedAt: heartbeatRuns.startedAt }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, source.id), eq(heartbeatRuns.companyId, selected.companyId), eq(heartbeatRuns.agentId, selected.agentId),
|
|
eq(heartbeatRuns.nativeIssueId, selected.issueId), eq(heartbeatRuns.nativeSessionId, selected.sessionId),
|
|
eq(heartbeatRuns.runtimeMode, "native"), eq(heartbeatRuns.status, "failed")));
|
|
return failed?.finishedAt && prior.finishedAt <= (failed.startedAt ?? source.createdAt) ? priorSnapshot : null;
|
|
}
|
|
|
|
async function hasConfirmedOlderSupervisor(db: Db, selected: Selected,
|
|
source: NonNullable<Awaited<ReturnType<typeof findNativeSessionResumeRun>>>,
|
|
paths: Awaited<ReturnType<typeof localPaths>>, supervisor: unknown) {
|
|
const olderSnapshot = record(record(supervisor).snapshot);
|
|
const prior = await confirmedOlderSuccessfulSnapshot(db, selected, source, paths, record(olderSnapshot.identity).runId);
|
|
return prior !== null && nativeSha256(prior) === nativeSha256(redactRunnerApiPayloads(olderSnapshot));
|
|
}
|
|
|
|
async function hasConfirmedOlderController(db: Db, selected: Selected,
|
|
source: NonNullable<Awaited<ReturnType<typeof findNativeSessionResumeRun>>>,
|
|
paths: Awaited<ReturnType<typeof localPaths>>, controller: string, artifact?: { version: string; digest: string }) {
|
|
const identity = record(record(JSON.parse(controller)).identity);
|
|
if (!await confirmedOlderSuccessfulSnapshot(db, selected, source, paths, identity.runId)) return false;
|
|
try {
|
|
captureLegacyDurablePrpCheckpoint(controller, { runId: String(identity.runId),
|
|
runnerInstanceId: source.runnerInstanceId!, normalizedSessionId: selected.sessionId }, artifact);
|
|
return true;
|
|
} catch { return false; }
|
|
}
|
|
|
|
async function* retainedSources(db: Db, signal?: AbortSignal, concurrency = 1, inspectDatabaseRecovery = false) {
|
|
let after: string | undefined;
|
|
for (;;) {
|
|
signal?.throwIfAborted();
|
|
const rows = await db.execute<Selected>(sql`select s.id, s.company_id as "companyId", s.agent_id as "agentId",
|
|
r.native_issue_id as "issueId", r.native_session_id as "sessionId", r.created_at as "createdAt", r.id as "lastRunId"
|
|
from agent_task_sessions s join heartbeat_runs r on r.id = s.last_run_id
|
|
and r.company_id = s.company_id and r.agent_id = s.agent_id
|
|
where s.adapter_type = 'paperclip_runner' and r.native_issue_id is not null and r.native_session_id is not null
|
|
${after ? sql`and s.id > ${after}::uuid` : sql``} order by s.id limit 100`);
|
|
for (let offset = 0; offset < rows.length; offset += concurrency) {
|
|
signal?.throwIfAborted();
|
|
const batch = await Promise.allSettled(rows.slice(offset, offset + concurrency).map(async selected => {
|
|
const source = await findNativeSessionResumeRun(db, { companyId: selected.companyId, agentId: selected.agentId,
|
|
issueId: selected.issueId, normalizedSessionId: selected.sessionId,
|
|
beforeCreatedAt: new Date(selected.createdAt), includeProgressBarrier: true });
|
|
const databaseRecoveryPreserved = Boolean(inspectDatabaseRecovery && source && source.status !== "running"
|
|
&& readNativeControllerRecovery(source.runnerProfileJson.nativeControllerRecovery, source.runnerProfileJson.sessionCheckpoint)
|
|
&& await hasNativeRunnerCheckpoint(db, { companyId: selected.companyId, issueId: selected.issueId, runId: source.id }));
|
|
return { selected, source, databaseRecoveryPreserved };
|
|
}));
|
|
for (const result of batch) {
|
|
signal?.throwIfAborted();
|
|
if (result.status === "rejected") throw result.reason;
|
|
if (result.value.source) yield { selected: result.value.selected, source: result.value.source,
|
|
databaseRecoveryPreserved: result.value.databaseRecoveryPreserved };
|
|
}
|
|
}
|
|
if (rows.length < 100) return;
|
|
after = rows.at(-1)!.id;
|
|
}
|
|
}
|
|
|
|
async function readPrivateSource(file: string, maxBytes: number): Promise<string | undefined> {
|
|
const handle = await open(file, constants.O_RDONLY | constants.O_NOFOLLOW).catch((error: NodeJS.ErrnoException) => {
|
|
if (error.code === "ENOENT") return undefined; throw error;
|
|
});
|
|
if (!handle) return undefined;
|
|
try {
|
|
const stat = await handle.stat();
|
|
if (!stat.isFile() || stat.nlink !== 1 || stat.size > maxBytes) throw new Error("native_controller_migration_unsafe_source");
|
|
return await handle.readFile("utf8");
|
|
} finally { await handle.close(); }
|
|
}
|
|
|
|
async function localPaths(source: NonNullable<Awaited<ReturnType<typeof findNativeSessionResumeRun>>>) {
|
|
const execution = parseNativeExecutionInput(source.runnerProfileJson.nativeExecutionInput);
|
|
if (execution.binding.companyId !== source.companyId || execution.binding.agentId !== source.agentId
|
|
|| execution.binding.issueId !== source.nativeIssueId || execution.binding.runId !== source.id
|
|
|| execution.session.normalizedSessionId !== source.nativeSessionId) throw new Error("native_controller_migration_artifact_binding");
|
|
const descriptor = nativeControllerMigrationDescriptor(execution, source.runnerProfileJson.nativeToolContractFingerprint);
|
|
const root = await lstat(descriptor.root).catch((error: NodeJS.ErrnoException) => {
|
|
if (error.code === "ENOENT") return undefined; throw error;
|
|
});
|
|
if (root && (!root.isDirectory() || root.isSymbolicLink())) throw new Error("native_controller_migration_unsafe_source");
|
|
const control = join(descriptor.root, "control-plane");
|
|
const directory = await lstat(control).catch((error: NodeJS.ErrnoException) => {
|
|
if (error.code === "ENOENT") return undefined; throw error;
|
|
});
|
|
if (directory && (!directory.isDirectory() || directory.isSymbolicLink())) throw new Error("native_controller_migration_unsafe_source");
|
|
return { ...descriptor, execution, controlFile: join(control, "control-plane-state.json") };
|
|
}
|
|
|
|
export async function readLegacyNativeControllerArchiveSource(source: NonNullable<Awaited<ReturnType<typeof findNativeSessionResumeRun>>>) {
|
|
const paths = await localPaths(source);
|
|
const controller = await readPrivateSource(paths.controlFile, 256 * 1024 * 1024);
|
|
const supervisor = paths.execution.session.lifecyclePolicy.mode === "warm"
|
|
? await readPrivateSource(paths.supervisorPath, 4 * 1024 * 1024) : undefined;
|
|
return { paths, controller, supervisor };
|
|
}
|
|
|
|
export async function readLegacyNativeControllerArchiveDigests(source: NonNullable<Awaited<ReturnType<typeof findNativeSessionResumeRun>>>) {
|
|
const paths = await localPaths(source);
|
|
return { controllerDigest: await nativeControllerSourceDigest(paths.controlFile, 256 * 1024 * 1024),
|
|
supervisorDigest: paths.execution.session.lifecyclePolicy.mode === "warm"
|
|
? await nativeControllerSourceDigest(paths.supervisorPath, 4 * 1024 * 1024) : undefined };
|
|
}
|
|
|
|
function preservedFailedCheckpoint(source: NonNullable<Awaited<ReturnType<typeof findNativeSessionResumeRun>>>,
|
|
sessionId: string, checkpoint: Awaited<ReturnType<typeof loadNativeRunnerCheckpoint>>) {
|
|
return source.status === "failed" && source.runnerProfileJson.sessionCheckpoint == null
|
|
&& checkpoint?.runId === source.id && checkpoint.recoveryEvidence.runnerInstanceId === source.runnerInstanceId
|
|
&& checkpoint.recoveryEvidence.normalizedSessionId === sessionId;
|
|
}
|
|
|
|
async function preservedDatabaseOnlyFailedCheckpoint(selected: Selected,
|
|
source: NonNullable<Awaited<ReturnType<typeof findNativeSessionResumeRun>>>,
|
|
checkpoint: Awaited<ReturnType<typeof loadNativeRunnerCheckpoint>>) {
|
|
const snapshot = record(source.runnerProfileJson.sessionCheckpoint), identity = record(snapshot.identity);
|
|
if (source.status !== "failed" || source.runtimeMode !== "native" || !checkpoint
|
|
|| source.companyId !== selected.companyId || source.agentId !== selected.agentId
|
|
|| source.nativeIssueId !== selected.issueId || source.nativeSessionId !== selected.sessionId
|
|
|| identity.companyId !== selected.companyId || identity.agentId !== selected.agentId
|
|
|| identity.issueId !== selected.issueId || identity.runId !== source.id || identity.sessionId !== selected.sessionId
|
|
|| typeof snapshot.sessionId !== "string" || !snapshot.sessionId.trim()
|
|
|| checkpoint.runId !== source.id || checkpoint.recoveryEvidence.runnerInstanceId !== source.runnerInstanceId
|
|
|| checkpoint.recoveryEvidence.normalizedSessionId !== selected.sessionId) return false;
|
|
const header = inspectDurablePrpCheckpointHeader(checkpoint.records.get("header") ?? "");
|
|
if (header?.runId !== source.id || header.runnerInstanceId !== source.runnerInstanceId
|
|
|| header.normalizedSessionId !== selected.sessionId) return false;
|
|
const paths = await localPaths(source);
|
|
for (const file of [paths.controlFile, paths.supervisorPath]) {
|
|
const present = await lstat(file).catch((error: NodeJS.ErrnoException) => {
|
|
if (error.code === "ENOENT") return undefined; throw error;
|
|
});
|
|
if (present) return false;
|
|
}
|
|
return true;
|
|
}
|
|
|
|
async function preservedWarmMarker(db: Db, selected: Selected,
|
|
source: NonNullable<Awaited<ReturnType<typeof findNativeSessionResumeRun>>>, markerIdentity: Record<string, unknown>) {
|
|
if (!isNativeSessionId(markerIdentity.runId) || markerIdentity.runId === source.id) return false;
|
|
const [prior] = await db.execute<{ id: string }>(sql`
|
|
select r.id from heartbeat_runs r join heartbeat_runs current on current.id=${source.id}::uuid and current.company_id=r.company_id
|
|
join native_run_finalizations f on f.run_id=r.id and f.company_id=r.company_id and f.issue_id=r.native_issue_id
|
|
join native_run_results result on result.id=f.result_id and result.run_id=r.id and result.company_id=r.company_id
|
|
and result.issue_id=r.native_issue_id and result.completion_contract_id=r.completion_contract_id and result.schema_status='accepted'
|
|
where r.id=${markerIdentity.runId}::uuid and r.company_id=${selected.companyId}::uuid and r.agent_id=${selected.agentId}::uuid
|
|
and r.native_issue_id=${selected.issueId}::uuid and r.native_session_id=${selected.sessionId}::uuid
|
|
and r.runner_instance_id=${source.runnerInstanceId} and r.runtime_mode='native' and r.status='succeeded'
|
|
and r.created_at<current.created_at and r.finished_at is not null and r.finished_at<=coalesce(current.started_at,current.created_at)
|
|
and f.phase='committed' and f.lease_owner is null and f.recovery_state is null and f.failure_code is null`);
|
|
if (!prior) return false;
|
|
const checkpoint = await loadNativeRunnerCheckpoint(db, { companyId: selected.companyId, issueId: selected.issueId, runId: prior.id }).catch(() => null);
|
|
const header = checkpoint && inspectDurablePrpCheckpointHeader(checkpoint.records.get("header") ?? "");
|
|
return Boolean(header && nativeSha256(header) === nativeSha256(markerIdentity));
|
|
}
|
|
|
|
async function preservedActiveCheckpoint(db: Db, selected: Selected,
|
|
source: NonNullable<Awaited<ReturnType<typeof findNativeSessionResumeRun>>>, paths: Awaited<ReturnType<typeof localPaths>>) {
|
|
if (source.status !== "running" || source.runtimeMode !== "native") return false;
|
|
let marker: Record<string, unknown> | null, bytes: string | undefined;
|
|
try {
|
|
bytes = await readPrivateSource(paths.controlFile, 4096);
|
|
marker = bytes === undefined ? null : record(JSON.parse(bytes));
|
|
} catch { return false; }
|
|
if (marker !== null && (marker.schema !== "paperclip.runner.external-control-plane-state.v1"
|
|
|| Object.keys(marker).sort().join(",") !== "identity,schema")) return false;
|
|
const binding = { companyId: selected.companyId, issueId: selected.issueId, runId: source.id };
|
|
const checkpoint = await loadNativeRunnerCheckpoint(db, binding).catch(() => null);
|
|
const header = checkpoint && inspectDurablePrpCheckpointHeader(checkpoint.records.get("header") ?? "");
|
|
if (!header || header.runId !== source.id || header.runnerInstanceId !== source.runnerInstanceId
|
|
|| header.normalizedSessionId !== selected.sessionId) return false;
|
|
const exact = marker === null || nativeSha256(header) === nativeSha256(marker.identity), markerIdentity = record(marker?.identity);
|
|
if (!exact && (paths.execution.session.lifecyclePolicy.mode !== "warm"
|
|
|| ["runnerInstanceId", "normalizedSessionId", "environmentLeaseId"].some(field => markerIdentity[field] !== header[field as keyof typeof header])
|
|
|| !await preservedWarmMarker(db, selected, source, markerIdentity))) return false;
|
|
const [current] = await db.execute<{ workspace: unknown; remoteWorkspace: unknown; hasRemoteWorkspace: boolean; pluginLeaseId: string | null; liveOwner: boolean }>(sql`
|
|
select r.runner_profile_json->'nativeWorkspaceSync' as workspace,
|
|
r.runner_profile_json->'remoteWorkspace' as "remoteWorkspace",
|
|
r.runner_profile_json ? 'remoteWorkspace' as "hasRemoteWorkspace", p.id as "pluginLeaseId",
|
|
(r.native_phase='observed' and r.process_pid is not null and r.process_started_at is not null
|
|
and i.execution_run_id=r.id and i.assignee_agent_id=r.agent_id
|
|
and f.controller_boot_id is not null and f.controller_boot_id<>'' and f.controller_generation>0
|
|
and f.lease_owner is not null and f.lease_owner<>'' and f.lease_expires_at>clock_timestamp()
|
|
and f.phase='observed' and f.result_id is null and f.failure_code is null) is true as "liveOwner"
|
|
from heartbeat_runs r left join native_run_finalizations f on f.run_id=r.id and f.company_id=r.company_id and f.issue_id=r.native_issue_id
|
|
left join issues i on i.id=r.native_issue_id and i.company_id=r.company_id
|
|
left join lateral (select l.id from environment_leases l where l.company_id=r.company_id and l.heartbeat_run_id=r.id
|
|
and l.issue_id=r.native_issue_id and l.status='active' and l.released_at is null and l.cleanup_status is null
|
|
and (l.expires_at is null or l.expires_at>clock_timestamp()) and l.metadata->>'agentId'=r.agent_id::text
|
|
and l.metadata->>'runnerTaskLease'='true' and l.metadata->>'runnerTaskDispatched'='true'
|
|
and coalesce(l.metadata->>'runnerTaskExecutionStopped','false')!='true'
|
|
and coalesce(l.metadata->>'runnerTaskRetiring','false')!='true' order by l.created_at desc limit 1) p on true
|
|
where r.id=${source.id}::uuid and r.company_id=${selected.companyId}::uuid and r.agent_id=${selected.agentId}::uuid
|
|
and r.native_issue_id=${selected.issueId}::uuid and r.native_session_id=${selected.sessionId}::uuid
|
|
and r.runner_instance_id=${source.runnerInstanceId} and r.runtime_mode='native' and r.status='running'`);
|
|
if (!current) return false;
|
|
const remote = remoteWorkspaceRunReferenceSchema.safeParse(current.remoteWorkspace);
|
|
const workspaceId = current.hasRemoteWorkspace
|
|
? remote.success && !current.pluginLeaseId ? remote.data.workspaceId : null
|
|
: current.pluginLeaseId ?? readNativeWorkspaceSyncReference(current.workspace)?.workspaceId;
|
|
return workspaceId === header.environmentLeaseId
|
|
&& (current.pluginLeaseId !== null || paths.execution.binding.executionWorkspaceId === workspaceId)
|
|
&& (current.liveOwner || exact && hasMatchingControllerRecovery(selected, source, checkpoint))
|
|
&& bytes === await readPrivateSource(paths.controlFile, 4096);
|
|
}
|
|
|
|
export async function hasLocalNativeControllerObligations(db: Db, storage?: StorageService | null): Promise<boolean> {
|
|
return await findLocalNativeControllerObligation(db, storage) !== null;
|
|
}
|
|
|
|
export async function findLocalNativeControllerObligation(db: Db, storage?: StorageService | null) {
|
|
const base = nativeHarnessStateBase();
|
|
const roots = await readdir(base).catch((error: NodeJS.ErrnoException) => {
|
|
if (error.code === "ENOENT") return [] as string[]; throw error;
|
|
});
|
|
if (!roots.some(name => !isVolumeFreeControllerStateOwnerEntry(name)) || await volumeFreeControllerStateOwned(base)) return null;
|
|
for await (const { selected, source, databaseRecoveryPreserved } of retainedSources(db, undefined, 8, true)) {
|
|
const binding = { companyId: selected.companyId, issueId: selected.issueId, runId: source.id };
|
|
if (databaseRecoveryPreserved) continue;
|
|
if (source.status === "failed" && source.runnerProfileJson.sessionCheckpoint == null
|
|
&& preservedFailedCheckpoint(source, selected.sessionId, await loadNativeRunnerCheckpoint(db, binding))) continue;
|
|
const paths = await localPaths(source);
|
|
if (await preservedActiveCheckpoint(db, selected, source, paths)) continue;
|
|
const present = await lstat(paths.controlFile).catch((error: NodeJS.ErrnoException) => {
|
|
if (error.code === "ENOENT") return undefined; throw error;
|
|
});
|
|
if (!present && source.status === "failed" && source.runnerProfileJson.sessionCheckpoint != null
|
|
&& await preservedDatabaseOnlyFailedCheckpoint(selected, source, await loadNativeRunnerCheckpoint(db, binding))) continue;
|
|
if (present && source.status === "failed") {
|
|
const archiveStorage = storage === undefined ? durableNativeHarnessStorage() : storage;
|
|
const controller = await readPrivateSource(paths.controlFile, 256 * 1024 * 1024);
|
|
const supervisor = paths.execution.session.lifecyclePolicy.mode === "warm"
|
|
? await readPrivateSource(paths.supervisorPath, 4 * 1024 * 1024) : undefined;
|
|
if (controller !== undefined && archiveStorage && await hasArchivedLegacyNativeController(db, { ...binding, storage: archiveStorage, controller, supervisor })) continue;
|
|
}
|
|
if (present) return { runId: source.id, issueId: selected.issueId, source: paths.controlFile };
|
|
if (await hasNativeRunnerCheckpoint(db, binding)) return { runId: source.id, issueId: selected.issueId, source: "database_checkpoint" };
|
|
}
|
|
return null;
|
|
}
|
|
|
|
function hasMatchingControllerRecovery(selected: Selected,
|
|
source: NonNullable<Awaited<ReturnType<typeof findNativeSessionResumeRun>>>,
|
|
checkpoint: Awaited<ReturnType<typeof loadNativeRunnerCheckpoint>>) {
|
|
const metadata = readNativeControllerRecovery(source.runnerProfileJson.nativeControllerRecovery, source.runnerProfileJson.sessionCheckpoint);
|
|
const snapshot = source.runnerProfileJson.sessionCheckpoint as { identity?: Record<string, unknown> } | null;
|
|
return Boolean(checkpoint && metadata?.checkpoint?.runId === source.id
|
|
&& metadata.checkpoint.runnerInstanceId === source.runnerInstanceId
|
|
&& metadata.checkpoint.normalizedSessionId === selected.sessionId
|
|
&& snapshot?.identity?.runId === source.id && snapshot.identity.companyId === selected.companyId
|
|
&& snapshot.identity.agentId === selected.agentId && snapshot.identity.issueId === selected.issueId
|
|
&& snapshot.identity.sessionId === selected.sessionId
|
|
&& checkpoint.recoveryEvidence.runnerInstanceId === source.runnerInstanceId
|
|
&& checkpoint.recoveryEvidence.normalizedSessionId === selected.sessionId);
|
|
}
|
|
|
|
async function readMigrationMaterial(db: Db, selected: Selected,
|
|
source: NonNullable<Awaited<ReturnType<typeof findNativeSessionResumeRun>>>,
|
|
checkpoint: Awaited<ReturnType<typeof loadNativeRunnerCheckpoint>>) {
|
|
const paths = await localPaths(source);
|
|
const legacyControllerState = checkpoint ? undefined : await readPrivateSource(paths.controlFile, 256 * 1024 * 1024);
|
|
let legacyRunnerArtifact: { version: string; digest: string } | undefined;
|
|
if (!checkpoint && legacyControllerState === undefined) return { paths, legacyControllerState, legacyRunnerArtifact, supervisor: undefined };
|
|
if (legacyControllerState !== undefined && await readPrivateSource(join(paths.root, "provider-pack-authority.json"), 512 * 1024)) {
|
|
const identity = JSON.parse(legacyControllerState).identity;
|
|
const execution = paths.execution;
|
|
const [admission] = await db.select({ workspace: sql<unknown>`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'` })
|
|
.from(heartbeatRuns).where(and(eq(heartbeatRuns.id, source.id), eq(heartbeatRuns.companyId, selected.companyId),
|
|
eq(heartbeatRuns.agentId, selected.agentId)));
|
|
const workspace = readNativeWorkspaceSyncReference(admission?.workspace);
|
|
const remoteCwd = workspace?.remoteCwd ?? execution.workspace.cwd;
|
|
if (typeof identity?.environmentLeaseId !== "string" || !remoteCwd
|
|
|| (workspace && workspace.workspaceId !== execution.binding.executionWorkspaceId)) {
|
|
throw new Error("native_controller_migration_artifact_binding");
|
|
}
|
|
const unavailable = (): never => { throw new Error("native_controller_migration_artifact_unavailable"); };
|
|
legacyRunnerArtifact = resolveRemoteProviderPackAuthority({ stateDirectory: paths.root, reattach: true,
|
|
binding: { companyId: selected.companyId, agentId: selected.agentId,
|
|
workspaceId: providerPackWorkspaceId(execution.binding.executionWorkspaceId, source.id, identity.environmentLeaseId),
|
|
sessionId: selected.sessionId, runnerInstanceId: source.runnerInstanceId!,
|
|
environmentLeaseId: identity.environmentLeaseId, remoteCwd,
|
|
providerDigest: nativeSha256(execution.provider) },
|
|
current: unavailable, runnerArtifact: unavailable }).runnerArtifact;
|
|
}
|
|
const supervisor = !checkpoint && paths.execution.session.lifecyclePolicy.mode === "warm"
|
|
? await readPrivateSource(paths.supervisorPath, 4 * 1024 * 1024) : undefined;
|
|
return { paths, legacyControllerState, legacyRunnerArtifact, supervisor };
|
|
}
|
|
|
|
export async function migrateNativeControllers(db: Db, input: { writersStopped: true; signal: AbortSignal; storage?: StorageService }) {
|
|
const [active] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(offlineMigrationExecutionWhere()).limit(1);
|
|
if (!input.writersStopped || active) throw new Error("native_controller_migration_requires_offline_writers");
|
|
const counts = { inspected: 0, imported: 0, unchanged: 0, sourceAlreadyAbsent: 0, archivedFailed: 0, preservedFailed: 0 };
|
|
for await (const { selected, source } of retainedSources(db, input.signal)) {
|
|
counts.inspected++;
|
|
const binding = { companyId: selected.companyId, issueId: selected.issueId, runId: source.id };
|
|
const checkpoint = await loadNativeRunnerCheckpoint(db, binding);
|
|
if (preservedFailedCheckpoint(source, selected.sessionId, checkpoint)) { counts.preservedFailed++; continue; }
|
|
if (await preservedDatabaseOnlyFailedCheckpoint(selected, source, checkpoint)) { counts.preservedFailed++; continue; }
|
|
if (hasMatchingControllerRecovery(selected, source, checkpoint)) { counts.unchanged++; continue; }
|
|
const { paths, legacyControllerState, legacyRunnerArtifact, supervisor } = await readMigrationMaterial(db, selected, source, checkpoint);
|
|
if (!checkpoint && legacyControllerState === undefined) { counts.sourceAlreadyAbsent++; continue; }
|
|
input.signal.throwIfAborted();
|
|
let result: Awaited<ReturnType<typeof importNativeControllerRecovery>>;
|
|
try {
|
|
result = await importNativeControllerRecovery(db, { ...binding, agentId: selected.agentId,
|
|
normalizedSessionId: selected.sessionId, expectedConfigDigest: paths.configDigest,
|
|
supervisor: supervisor === undefined ? undefined : JSON.parse(supervisor),
|
|
offline: { writersStopped: true, legacyControllerState, legacyRunnerArtifact } });
|
|
} catch (error) {
|
|
const storage = input.storage ?? durableNativeHarnessStorage();
|
|
if (!(error instanceof Error) || source.status !== "failed" || legacyControllerState === undefined || !storage) throw error;
|
|
const archivable = error.message === "native_runner_checkpoint_artifact_unproven"
|
|
|| error.message === "native_runner_checkpoint_binding_mismatch"
|
|
&& await hasConfirmedOlderController(db, selected, source, paths, legacyControllerState, legacyRunnerArtifact);
|
|
if (!archivable) throw error;
|
|
await archiveFailedLegacyNativeController(db, { ...binding, storage, controller: legacyControllerState, supervisor,
|
|
writersStopped: true, signal: input.signal });
|
|
counts.archivedFailed++;
|
|
continue;
|
|
}
|
|
if (result.kind !== "imported" && result.kind !== "unchanged") {
|
|
if (result.kind === "missing_evidence" && result.reason === "stale_supervisor_snapshot"
|
|
&& legacyControllerState !== undefined && supervisor !== undefined
|
|
&& await hasConfirmedOlderSupervisor(db, selected, source, paths, JSON.parse(supervisor))) {
|
|
const storage = input.storage ?? durableNativeHarnessStorage();
|
|
if (storage) {
|
|
await archiveFailedLegacyNativeController(db, { ...binding, storage, controller: legacyControllerState, supervisor,
|
|
writersStopped: true, signal: input.signal });
|
|
counts.archivedFailed++;
|
|
continue;
|
|
}
|
|
}
|
|
throw new Error(`native_controller_migration_blocked:${source.id}:${"reason" in result ? result.reason : result.kind}`);
|
|
}
|
|
counts[result.kind]++;
|
|
}
|
|
return counts;
|
|
}
|
|
|
|
function onlineControllerSourceDeferral(error: unknown): string | null {
|
|
if (!(error instanceof Error)) return null;
|
|
const reasons: Record<string, string> = {
|
|
native_runner_checkpoint_artifact_unproven: "artifact_unproven",
|
|
native_runner_checkpoint_binding_mismatch: "checkpoint_binding",
|
|
native_controller_migration_source_changed: "source_changed",
|
|
native_controller_migration_unsafe_source: "unsafe_source",
|
|
native_controller_migration_artifact_binding: "artifact_binding",
|
|
native_controller_migration_artifact_unavailable: "artifact_unavailable",
|
|
native_controller_archive_source_unsettled: "archive_source_unsettled",
|
|
};
|
|
if (reasons[error.message]) return reasons[error.message];
|
|
if (error.message.startsWith("runner_remote_provider_artifact_incompatible:")) return "artifact_incompatible";
|
|
if (error instanceof SyntaxError) return "invalid_source_json";
|
|
if (["ENOENT", "ENOTDIR", "EISDIR", "ELOOP", "EACCES", "EPERM"].includes((error as NodeJS.ErrnoException).code ?? "")) return "source_unavailable";
|
|
return null;
|
|
}
|
|
|
|
export async function prestageNativeControllers(db: Db, input: { signal: AbortSignal; storage?: StorageService }) {
|
|
const counts = { inspected: 0, imported: 0, unchanged: 0, archivedFailed: 0, sourceAlreadyAbsent: 0, deferred: {} as Record<string, number> };
|
|
const defer = (reason: string) => { counts.deferred[reason] = (counts.deferred[reason] ?? 0) + 1; };
|
|
for await (const { selected, source } of retainedSources(db, input.signal)) {
|
|
counts.inspected++;
|
|
input.signal.throwIfAborted();
|
|
let outcome = await db.transaction(async tx => {
|
|
await tx.execute(sql`set local lock_timeout = '1s'`);
|
|
const [task] = await tx.select().from(issues).where(and(eq(issues.companyId, selected.companyId),
|
|
eq(issues.id, selected.issueId))).for("update");
|
|
if (!task || task.executionRunId || task.checkoutRunId) return "source_active";
|
|
const [owner] = await tx.select().from(nativeRunFinalizations).where(and(
|
|
eq(nativeRunFinalizations.companyId, selected.companyId), eq(nativeRunFinalizations.issueId, selected.issueId),
|
|
eq(nativeRunFinalizations.runId, source.id))).for("update");
|
|
if (!owner || owner.leaseOwner || owner.phase !== "committed" || !owner.resultId || owner.recoveryState) return "owner_not_released";
|
|
const [current] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, source.id),
|
|
eq(heartbeatRuns.companyId, selected.companyId), eq(heartbeatRuns.agentId, selected.agentId),
|
|
eq(heartbeatRuns.nativeIssueId, selected.issueId), eq(heartbeatRuns.nativeSessionId, selected.sessionId))).for("update");
|
|
if (!current || !["succeeded", "failed"].includes(current.status) || !current.finishedAt) return "source_not_successful";
|
|
const [active] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, selected.companyId), inArray(heartbeatRuns.status, ["queued", "running"]),
|
|
or(eq(heartbeatRuns.nativeIssueId, selected.issueId), eq(heartbeatRuns.issueId, selected.issueId),
|
|
eq(heartbeatRuns.nativeSessionId, selected.sessionId)))).limit(1);
|
|
if (active) return "source_active";
|
|
const scoped = tx as unknown as Db;
|
|
const binding = { companyId: selected.companyId, issueId: selected.issueId, runId: source.id };
|
|
const checkpoint = await loadNativeRunnerCheckpoint(scoped, binding);
|
|
const profile = record(current.runnerProfileJson);
|
|
const freshSource = { ...source, status: current.status, runnerInstanceId: current.runnerInstanceId,
|
|
runnerProfileJson: { nativeToolContractFingerprint: profile.nativeToolContractFingerprint,
|
|
nativeExecutionInput: profile.nativeExecutionInput, sessionCheckpoint: profile.sessionCheckpoint,
|
|
nativeControllerRecovery: profile.nativeControllerRecovery } };
|
|
if (hasMatchingControllerRecovery(selected, freshSource, checkpoint)) return "unchanged";
|
|
const material = await readMigrationMaterial(scoped, selected, freshSource, checkpoint);
|
|
if (!checkpoint && material.legacyControllerState === undefined) return "source_already_absent";
|
|
input.signal.throwIfAborted();
|
|
const result = await importNativeControllerRecovery(scoped, { ...binding, agentId: selected.agentId,
|
|
normalizedSessionId: selected.sessionId, expectedConfigDigest: material.paths.configDigest,
|
|
supervisor: material.supervisor === undefined ? undefined : JSON.parse(material.supervisor),
|
|
legacyCheckpoint: { legacyControllerState: material.legacyControllerState,
|
|
legacyRunnerArtifact: material.legacyRunnerArtifact } });
|
|
if (result.kind !== "imported" && result.kind !== "unchanged") return "source_unproven";
|
|
if (material.legacyControllerState !== undefined
|
|
&& await readPrivateSource(material.paths.controlFile, 256 * 1024 * 1024) !== material.legacyControllerState) {
|
|
throw new Error("native_controller_migration_source_changed");
|
|
}
|
|
if (material.supervisor !== undefined
|
|
&& await readPrivateSource(material.paths.supervisorPath, 4 * 1024 * 1024) !== material.supervisor) {
|
|
throw new Error("native_controller_migration_source_changed");
|
|
}
|
|
input.signal.throwIfAborted();
|
|
return result.kind;
|
|
}).catch(error => {
|
|
input.signal.throwIfAborted();
|
|
const reason = onlineControllerSourceDeferral(error);
|
|
if (reason) return reason;
|
|
if (error?.code === "55P03" || error?.cause?.code === "55P03") return "source_busy";
|
|
throw error;
|
|
});
|
|
if (source.status === "failed" && ["owner_not_released", "artifact_unproven", "checkpoint_binding", "source_unproven", "artifact_binding"].includes(outcome)) {
|
|
const storage = input.storage ?? durableNativeHarnessStorage();
|
|
if (storage) outcome = await archiveSettledFailedNativeController(db, { selected, source, storage, signal: input.signal }).catch(error => {
|
|
input.signal.throwIfAborted();
|
|
const reason = onlineControllerSourceDeferral(error);
|
|
if (reason) return reason;
|
|
if (error?.code === "55P03" || error?.cause?.code === "55P03") return "source_busy";
|
|
throw error;
|
|
});
|
|
}
|
|
if (outcome === "archived_failed") counts.archivedFailed++;
|
|
else if (outcome === "imported" || outcome === "unchanged") counts[outcome]++;
|
|
else if (outcome === "source_already_absent") counts.sourceAlreadyAbsent++;
|
|
else defer(outcome);
|
|
}
|
|
return counts;
|
|
}
|