mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 14:19:00 +01:00
fix: recover interrupted native preparations and unblock upgrades (#20177)
## Thinking Path > - Paperclip manages AI-agent companies and their work. > - A native run prepares its workspace before a runner PID or checkpoint exists. > - A server crash can interrupt that stage and leave its controller on another host. > - Recovery needs a durable owner proof, a delivered recovery wake, and the saved execution contract. > - The old paths missed controller lease registration and boot proof, skipped checkpoint-free preparations, and counted retained work as live upgrade blockers. > - This PR connects preparation to the existing controller lease and recovery paths, and separates retained work from the upgrade census. > - The result is bounded recovery without a second executor or loss of saved work. ## Linked Issues or Issue Description **What happened?** A native preparation interrupted by a server crash could remain running indefinitely with no runner PID or checkpoint. Native controller stamps did not wake the shared instance lease. That lease was disabled on volume-backed Railway services, and native takeover omitted the boot binding needed for its cross-host proof. Scoped recovery selected checkpoint-backed runs but skipped these preparations. The upgrade probe then counted the retained row as current execution and could block every later upgrade. **Expected behavior** Track the preparation's controller before a runner starts. Recover it through the normal classifier after proving its owner stopped or fencing that boot. Keep live or unprovable owners held. Deliver recovery wakes to preparations without checkpoints. Preserve unresolved historical work and report it separately from current execution during upgrade observations. **Steps to reproduce** 1. Save an observed native preparation with no runner process, result, provider-start evidence or checkpoint. 2. Replace its controller with a process in another PID space. Let its owner lease expire and send a scoped recovery wake. 3. Before this fix, recovery skips the checkpoint-free preparation or cannot prove its foreign controller stopped. Upgrade readiness continues to count that old row as a live blocker. Related work: #18037, #19651, #19868, and #20168. The foreign-PID guard from #19868 remains. The queued first-turn input fix from #20168 is included through the current base. ## What Changed - Register the existing instance lease when native execution and restart claims are written. Keep passive identity reads free of ownership effects. Keep that lease enabled on volume-backed Railway services unless explicitly disabled. - Preserve the existing PID/start-time proof within the same readable PID space. Bind foreign or unknown-space takeover to its saved boot and last write. A live lease or grace period keeps the run held with a scheduled check. Repeated evidence waits preserve their timestamp. - Include explicitly fenced, checkpoint-free preparations in the existing finite recovery candidate inventory. Normal launch, checkpoint, provider evidence and admission checks still decide whether work may resume. - Expose the existing board stop-confirmation operation for historical foreign-host owners. Bind confirmation to the exact run, boot, PID, process start and host. Reject changed ownership. - Report narrowly defined old, unowned preparations as `retainedBlockedRuns` outside the live upgrade census. Local executions, pending admission, drain state, truncation and positive checkpoint proof retain their barriers. - Update the execution contract and add PostgreSQL, real-route, race and preservation regressions. ## Verification - Genuine red/green regressions cover the old upgrade census, missing native lease registration, the volume-backed default, foreign-host confirmation, and scoped recovery skipping checkpoint-free preparation. Test bodies ran against real PostgreSQL before their corresponding production changes. - Combined verification before the final idle correction (head `055be6c84d9`): **459 passed, 22 skipped, 11 files passed**. Covers controller lease/identity, native restart classification and integration, host confirmation, heartbeat recovery, instance route authorization, upgrade readiness, and queued first-turn input. The focused command used `pnpm exec vitest run` with those 11 suites and `--maxWorkers=1 --no-file-parallelism`. The 22 skipped cases require the Rust debug fake-Codex fixture, which was not built locally; the real-process CI coverage must pass. - `pnpm --filter @paperclipai/server typecheck`: passed on head `055be6c84d9b058064daad0af71bb5d893ab9529`. - Actual route proof uses the real heartbeat service and PostgreSQL. Unauthorized access returns 403. The retained run and owner records remain unchanged. Recovery integration admits one bootstrap claim across two replacement controllers, keeps a live owner held, and preserves local execution. - `pnpm verify`: exit 2, requests broad CI because the execution contract changed. Full local suite not run. Current-head CI must supply broad coverage. - Local source checkout on macOS ARM64, Node v25.6.1, pnpm 9.15.4. Tests use isolated PostgreSQL. No live provider restart is claimed by those fixtures. - CI found an idle-query regression in the first version of the lease wiring. Two focused PostgreSQL regressions reproduced the misplaced identity-read notification and the missing claim notification. The fix moves registration to actual ownership writes. `env -u npm_execpath pnpm exec vitest run server/src/__tests__/maintenance-server-routes.test.ts --maxWorkers=1 --no-file-parallelism`: **17 passed**, including 90 real seconds and two qualified hourly windows with no extra idle queries. An initial local setup attempt inherited npm metadata and did not run tests; clearing that metadata selected the correct pnpm helper. Three focused controller/native suites also passed **168 tests, 22 fixture-gated skips**. Server typecheck passed after the correction on head `4a527326dcd5b3a9e286f0e4c3e761d3056af4d9`. - The older CI server shard also exposed a same-space graceful-restart regression. The new boot fallback had overridden local process-exit proof. Seven suspension cases and a focused unit regression reproduced it; the fix limits boot fallback to foreign or unknown PID spaces. Fresh verification on `acf1065d98150210c1e227c7f3b05a1eb489730b`: **251 passed, 22 fixture-gated skips, 4 files passed**, including all **40** graceful-restart suspension tests. Server typecheck passed. The local regression failed before this correction, and the same-space and foreign-space paths pass together after it. - Current-head CI completed successfully on `acf1065d98150210c1e227c7f3b05a1eb489730b` (run 42568109784). Greptile reviewed that exact head at 5/5 with no unresolved review threads. - Secret and private-data scans passed. No workflow, lockfile or migration changes. ## Risks - The instance lease now applies to volume-backed Railway services. Its existing 30-second renewal, quiescence, grace and self-fence behavior applies while work is held. Idle servers retain the existing no-query behavior. An explicit `PAPERCLIP_CONTROLLER_INSTANCE_LEASE=off` remains available. - A controller lease proves only the server owner's state. It does not prove an agent process stopped or authorize replay, resource release or volume removal. The existing classifier and provider/runner checks remain responsible for those decisions. - Previous-release records keep their shape and saved state. A missing boot lease stays unprovable and requires the scoped stop-confirmation path. A changed confirmation target is rejected. - The upgrade observation does not recover a held task by itself. Source tests do not qualify a hosted replacement. Deployment and the affected task's useful result must be verified separately. - This repairs recovery after a crash; it does not claim to identify or fix every source of heap exhaustion. ## Model Used OpenAI `gpt-6.1-sol`, xhigh reasoning effort, through Codex in T3 Code, with repository, shell, and GitHub tools. Context window size is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin FoleyandPaperclip authored and GitHub committed
1 parent
2d93413085
commit
9b71d03215
14 files changed
+301
-42
No files matched your search
@@ -103,7 +103,7 @@ export async function requestNativeAcceptedResultLeaseOperatorAction(db: Db, rec
|
||||
if (!held) return;
|
||||
await requestNativeControllerStopConfirmation(db, {
|
||||
runId: record.recordId, companyId: record.companyId, issueId: record.issueId,
|
||||
controllerBootId: stamp.bootId, controllerPid: stamp.pid,
|
||||
controllerBootId: stamp.bootId, controllerPid: stamp.pid, controllerHostHash: null,
|
||||
controllerProcessStartedAt: stamp.processStartedAt ? new Date(stamp.processStartedAt) : null,
|
||||
});
|
||||
}
|
||||
@@ -77,4 +77,28 @@ describe("native controller rows saved before their host was recorded", () => {
|
||||
await expect(confirmNativeControllerStopped({ db, companyId, issueId, actionId: actions[0]!.id, runId,
|
||||
stopEvidence, actorId: "board-user" })).rejects.toMatchObject({ status: 409 });
|
||||
});
|
||||
|
||||
it("binds stop confirmation to an exact foreign-host controller and rejects changed ownership", async () => {
|
||||
const { companyId, issueId, runId, coordinator } = await seed();
|
||||
const owner = { ...coordinator, controllerHostHash: "previous-pid-space" };
|
||||
await db.update(nativeRunFinalizations).set({ controllerHostHash: owner.controllerHostHash })
|
||||
.where(eq(nativeRunFinalizations.runId, runId));
|
||||
await requestNativeControllerStopConfirmation(db, owner);
|
||||
const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId));
|
||||
const input = { db, companyId, issueId, runId, actionId: action!.id, actorId: "board-user",
|
||||
stopEvidence: "The hosting provider confirms the exact previous container stopped." };
|
||||
for (const patch of [{ controllerHostHash: "changed-space" }, { controllerBootId: randomUUID() },
|
||||
{ controllerPid: 1_999_999_999 }, { controllerProcessStartedAt: new Date(owner.controllerProcessStartedAt!.getTime() + 60_000) }]) {
|
||||
await db.update(nativeRunFinalizations).set(patch).where(eq(nativeRunFinalizations.runId, runId));
|
||||
await expect(confirmNativeControllerStopped(input)).rejects.toMatchObject({ status: 409 });
|
||||
await db.update(nativeRunFinalizations).set({ controllerHostHash: owner.controllerHostHash,
|
||||
controllerBootId: owner.controllerBootId, controllerPid: owner.controllerPid,
|
||||
controllerProcessStartedAt: owner.controllerProcessStartedAt }).where(eq(nativeRunFinalizations.runId, runId));
|
||||
}
|
||||
await expect(confirmNativeControllerStopped(input)).resolves.toEqual({ runId, status: "queued" });
|
||||
const [attested] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
|
||||
expect(await evaluateNativeControllerTakeover({ owner: attested!, now: new Date() }))
|
||||
.toEqual({ allowed: true, reason: "operator_attested_controller_stopped" });
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]?.status).toBe("running");
|
||||
});
|
||||
});
|
||||
@@ -5,12 +5,6 @@ import { conflict } from "../../errors.js";
|
||||
import { persistActivity, publishActivity } from "../activity-log.js";
|
||||
import { notifyLiveRunRecovery } from "../live-run-recovery-notifications.js";
|
||||
|
||||
/**
|
||||
* An ephemeral (multi-host) server cannot judge a native controller row saved
|
||||
* before controller_host_hash existed: its PID may name a process on another
|
||||
* host. Restart recovery leaves such a row waiting and asks the board, through
|
||||
* this recovery action, to confirm that the previous server stopped.
|
||||
*/
|
||||
export const NATIVE_CONTROLLER_HOST_UNVERIFIED = "native_controller_host_unverified";
|
||||
/** controller_host_hash value written once the board confirms the stop. */
|
||||
export const OPERATOR_ATTESTED_CONTROLLER_HOST = "operator_attested";
|
||||
@@ -20,7 +14,7 @@ const changed = () => conflict("The recorded controller is no longer current. Re
|
||||
{ code: "native_controller_host_recovery_stale" });
|
||||
|
||||
type Coordinator = Pick<typeof nativeRunFinalizations.$inferSelect,
|
||||
"runId" | "companyId" | "issueId" | "controllerBootId" | "controllerPid" | "controllerProcessStartedAt">;
|
||||
"runId" | "companyId" | "issueId" | "controllerBootId" | "controllerPid" | "controllerProcessStartedAt" | "controllerHostHash">;
|
||||
|
||||
/** Idempotent. Another active recovery action on the task keeps precedence. */
|
||||
export async function requestNativeControllerStopConfirmation(tx: Db, coordinator: Coordinator) {
|
||||
@@ -29,9 +23,10 @@ export async function requestNativeControllerStopConfirmation(tx: Db, coordinato
|
||||
status: "active", ownerType: "board", cause: NATIVE_CONTROLLER_HOST_UNVERIFIED,
|
||||
fingerprint: fingerprint(coordinator.runId), attemptCount: 1,
|
||||
evidence: { runId: coordinator.runId, controller: { bootId: coordinator.controllerBootId, pid: coordinator.controllerPid,
|
||||
hostHash: coordinator.controllerHostHash,
|
||||
processStartedAt: coordinator.controllerProcessStartedAt?.toISOString() ?? null } },
|
||||
nextAction: "This run's previous server was recorded before Paperclip stored which host it ran on, so Paperclip "
|
||||
+ "cannot verify that it stopped. Confirm that the previous server process has stopped (for example, its "
|
||||
nextAction: "Paperclip cannot verify that this run's previous server stopped from the current host. "
|
||||
+ "Confirm that the previous server process has stopped (for example, its "
|
||||
+ "container was replaced), and the run resumes from its saved state.",
|
||||
}).onConflictDoNothing();
|
||||
}
|
||||
@@ -57,13 +52,15 @@ export async function confirmNativeControllerStopped(input: {
|
||||
const recorded = (action?.evidence?.controller ?? {}) as Record<string, unknown>;
|
||||
if (!coordinator || !action || !run || action.cause !== NATIVE_CONTROLLER_HOST_UNVERIFIED || action.ownerType !== "board"
|
||||
|| !["active", "escalated"].includes(action.status) || action.evidence?.runId !== input.runId
|
||||
|| coordinator.controllerHostHash !== null || coordinator.controllerPid !== recorded.pid
|
||||
|| coordinator.controllerBootId !== recorded.bootId) throw changed();
|
||||
|| coordinator.controllerHostHash !== (recorded.hostHash ?? null) || coordinator.controllerPid !== recorded.pid
|
||||
|| coordinator.controllerBootId !== recorded.bootId
|
||||
|| (coordinator.controllerProcessStartedAt?.toISOString() ?? null) !== recorded.processStartedAt) throw changed();
|
||||
// Work intent before the write it announces (sleep-aware background work).
|
||||
await notifyLiveRunRecovery(tx, { companyId: input.companyId, runId: input.runId, agentId: run.agentId });
|
||||
const now = new Date();
|
||||
const updated = await tx.update(nativeRunFinalizations).set({ controllerHostHash: OPERATOR_ATTESTED_CONTROLLER_HOST, updatedAt: now })
|
||||
.where(and(eq(nativeRunFinalizations.runId, input.runId), isNull(nativeRunFinalizations.controllerHostHash),
|
||||
.where(and(eq(nativeRunFinalizations.runId, input.runId), coordinator.controllerHostHash === null
|
||||
? isNull(nativeRunFinalizations.controllerHostHash) : eq(nativeRunFinalizations.controllerHostHash, coordinator.controllerHostHash),
|
||||
eq(nativeRunFinalizations.controllerPid, coordinator.controllerPid!),
|
||||
eq(nativeRunFinalizations.controllerBootId, coordinator.controllerBootId!)))
|
||||
.returning({ runId: nativeRunFinalizations.runId });
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { currentProcessSpaceHash } from "../controller-process-identity.js";
|
||||
import { currentControllerBootId, currentProcessSpaceHash } from "../controller-process-identity.js";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
import {
|
||||
@@ -297,6 +297,14 @@ describe("native controller takeover fencing", () => {
|
||||
now, isProcessAlive: () => true })).resolves.toEqual({ allowed: true, reason: "operator_attested_controller_stopped" });
|
||||
});
|
||||
|
||||
it("preserves same-space process proof when boot metadata is present", async () => {
|
||||
const local = { ...owner(), controllerBootId: currentControllerBootId(), controllerHostHash: currentProcessSpaceHash() };
|
||||
expect(await evaluateNativeControllerTakeover({ owner: local, now, isProcessAlive: () => false }))
|
||||
.toEqual({ allowed: true, reason: "controller_process_dead" });
|
||||
expect(await evaluateNativeControllerTakeover({ owner: local, now, isProcessAlive: () => true,
|
||||
readProcessStartedAt: async () => recordedStart })).toEqual({ allowed: false, reason: "controller_still_alive" });
|
||||
});
|
||||
|
||||
it.each([
|
||||
["another host", "another-host-pid-space"],
|
||||
["an unreadable PID space", "unknown"],
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
import { readPendingNativeRunStop } from "./native-cancellation-request.js";
|
||||
import { CONTROLLER_LEASE_RENEW_MS } from "../controller-instance-lease.js";
|
||||
import { currentControllerProcessStamp, UNKNOWN_PROCESS_SPACE } from "../controller-process-identity.js";
|
||||
import { readNativeSessionResumptionAgent, restoreNativeSessionExecution } from "./native-session-resumption-admission.js";
|
||||
import { notifyLiveRunRecovery } from "../live-run-recovery-notifications.js";
|
||||
import { NATIVE_RETAINED_OWNER_WAIT } from "./native-controller-recovery.js";
|
||||
@@ -401,6 +403,8 @@ export async function evaluateNativeControllerTakeover(input: {
|
||||
* existed: those keep the previous local proof for one release, outside
|
||||
* ephemeral mode only (see LEGACY_HOST_FALLBACK_EVENT). */
|
||||
controllerHostHash?: string | null;
|
||||
controllerBootId?: string | null;
|
||||
updatedAt?: Date;
|
||||
};
|
||||
now: Date;
|
||||
coordinatedPreviousController?: {
|
||||
@@ -454,15 +458,19 @@ export async function evaluateNativeControllerTakeover(input: {
|
||||
pidExists: input.isProcessAlive ?? base.pidExists,
|
||||
readStartedAt: input.readProcessStartedAt ?? base.readStartedAt,
|
||||
};
|
||||
const needsBootProof = owner.controllerHostHash != null
|
||||
&& (owner.controllerHostHash !== probe.processSpace || owner.controllerHostHash === UNKNOWN_PROCESS_SPACE);
|
||||
const proof = await proveOwnerGone({
|
||||
kind: "local", scope: "native_controller", generation: null, host: null,
|
||||
processSpace: recordsHost ? owner.controllerHostHash ?? null : probe.processSpace,
|
||||
processSpaceUnrecorded: recordsHost && owner.controllerHostHash == null,
|
||||
bootId: null, pid: priorPid, processStartedAt: owner.controllerProcessStartedAt?.toISOString() ?? null,
|
||||
bootId: needsBootProof ? owner.controllerBootId ?? null : null, touchedAt: owner.updatedAt?.toISOString() ?? null,
|
||||
pid: priorPid, processStartedAt: owner.controllerProcessStartedAt?.toISOString() ?? null,
|
||||
processGroupId: null,
|
||||
}, probe);
|
||||
const fallback = proof.legacyHostFallback ? { legacyHostFallback: true as const } : {};
|
||||
if (proof.verdict === "gone") {
|
||||
if (proof.reason === "controller_fenced" || proof.reason === "controller_stopped") return { allowed: true, reason: proof.reason };
|
||||
if (proof.reason === "pid_reused") return { allowed: true, reason: "controller_pid_recycled", ...fallback };
|
||||
return {
|
||||
allowed: true,
|
||||
@@ -473,6 +481,7 @@ export async function evaluateNativeControllerTakeover(input: {
|
||||
...fallback,
|
||||
};
|
||||
}
|
||||
if (proof.reason === "controller_lease_held" || proof.reason === "controller_lease_grace") return { allowed: false, reason: proof.reason };
|
||||
if (proof.reason === "process_space_unrecorded") return { allowed: false, reason: "controller_host_unrecorded" };
|
||||
if (proof.reason === "different_process_space" || proof.reason === "unknown_process_space"
|
||||
|| proof.reason === "different_host") {
|
||||
@@ -1187,7 +1196,7 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
input.coordinatedPreviousController ?? null,
|
||||
});
|
||||
await recordLegacyHostFallbackProof(tx as unknown as Db, row.run, takeover, "native_restart_recovery");
|
||||
if (!takeover.allowed && takeover.reason === "controller_host_unrecorded") {
|
||||
if (!takeover.allowed && ["controller_host_unrecorded", "controller_host_unprovable"].includes(takeover.reason)) {
|
||||
await requestNativeControllerStopConfirmation(tx as unknown as Db, row.coordinator);
|
||||
}
|
||||
if (!takeover.allowed) {
|
||||
@@ -1217,7 +1226,8 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
// Repeated scans of one ownership wait record it once, so the
|
||||
// bounded history keeps the drain record the next claim reads.
|
||||
...(waitAlreadyRecorded ? {} : { recoveryHistory: appendBoundedRecoveryHistory(event) }),
|
||||
updatedAt: now,
|
||||
updatedAt: waitAlreadyRecorded && row.coordinator.recoveryState === "awaiting_evidence"
|
||||
&& row.coordinator.recoveryRequestId === (input.recoveryRequestId ?? null) ? row.coordinator.updatedAt : now,
|
||||
})
|
||||
.where(
|
||||
and(
|
||||
@@ -1230,7 +1240,9 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
kind: "awaiting_evidence",
|
||||
runId: row.run.id,
|
||||
reason: takeover.reason,
|
||||
...(durableAuthority && row.coordinator.leaseExpiresAt ? { retryAt: row.coordinator.leaseExpiresAt } : {}),
|
||||
...(durableAuthority && row.coordinator.leaseExpiresAt ? { retryAt: row.coordinator.leaseExpiresAt }
|
||||
: ["controller_lease_held", "controller_lease_grace"].includes(takeover.reason)
|
||||
? { retryAt: new Date(now.getTime() + CONTROLLER_LEASE_RENEW_MS) } : {}),
|
||||
} as const;
|
||||
}
|
||||
|
||||
@@ -1648,6 +1660,7 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
// The attempt the executor runs this resume as (see its execution claim).
|
||||
const progressAttempt = nextNativeProviderAttempt(providerAttempt, claimKind, !resumeKeepsAttempt);
|
||||
await notifyLiveRunRecovery(tx, { companyId: row.run.companyId, runId: row.run.id, agentId: row.run.agentId });
|
||||
currentControllerProcessStamp();
|
||||
const claimed = await tx
|
||||
.update(nativeRunFinalizations)
|
||||
.set({
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { readNativeCheckpointRetirement } from "./native-runner-checkpoint-retirement.js";
|
||||
import { NATIVE_PREPARATION_FENCING_VERSION } from "./native-active-runs.js";
|
||||
import { notifyLiveRunRecovery } from "../live-run-recovery-notifications.js";
|
||||
import { and, eq, exists, gt, isNull, lte, inArray, notExists, or, sql } from "drizzle-orm";
|
||||
import { checkpointDb, heartbeatRuns, issues, nativeRunFinalizations, nativeRunnerCheckpoints, type Db } from "@paperclipai/db";
|
||||
@@ -349,10 +350,12 @@ export async function listExpiredNativeRunnerAuthorities(db: Db, now: Date, runI
|
||||
eq(heartbeatRuns.status, "running"), eq(nativeRunFinalizations.phase, "observed"),
|
||||
isNull(nativeRunFinalizations.resultId),
|
||||
or(isNull(nativeRunFinalizations.leaseExpiresAt), lte(nativeRunFinalizations.leaseExpiresAt, now)),
|
||||
exists(db.select({ runId: nativeRunnerCheckpoints.runId }).from(nativeRunnerCheckpoints).where(and(
|
||||
or(exists(db.select({ runId: nativeRunnerCheckpoints.runId }).from(nativeRunnerCheckpoints).where(and(
|
||||
eq(nativeRunnerCheckpoints.runId, nativeRunFinalizations.runId),
|
||||
eq(nativeRunnerCheckpoints.issueId, nativeRunFinalizations.issueId),
|
||||
eq(nativeRunnerCheckpoints.companyId, nativeRunFinalizations.companyId))))));
|
||||
eq(nativeRunnerCheckpoints.companyId, nativeRunFinalizations.companyId)))),
|
||||
and(sql`${heartbeatRuns.runnerProfileJson}->'nativePreparationFencingVersion' = ${JSON.stringify(NATIVE_PREPARATION_FENCING_VERSION)}::jsonb`,
|
||||
isNull(heartbeatRuns.processPid), isNull(heartbeatRuns.processGroupId), isNull(heartbeatRuns.processStartedAt)))));
|
||||
return rows.map(row => row.runId);
|
||||
}
|
||||
|
||||
|
||||
@@ -18,6 +18,7 @@ import {
|
||||
agents,
|
||||
authUsers,
|
||||
companies,
|
||||
controllerInstances,
|
||||
companyMemberships,
|
||||
completionContracts,
|
||||
environmentLeases,
|
||||
@@ -80,6 +81,8 @@ import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.
|
||||
import { RUNNER_BRIDGE_READ_OPERATIONS, RUNNER_BRIDGE_SCHEMAS } from "./runner-bridge-tools.js";
|
||||
import { HttpError } from "../../errors.js";
|
||||
import { legacyControllerBootId } from "../legacy-controller-lease.js";
|
||||
import { onControllerStamp } from "../controller-process-identity.js";
|
||||
import { registerControllerInstanceDb, resetControllerInstanceLeaseForTests } from "../controller-instance-lease.js";
|
||||
|
||||
const mockCaptureRunFailure = vi.hoisted(() => vi.fn());
|
||||
vi.mock("../../sentry.js", async () => {
|
||||
@@ -2277,6 +2280,42 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
|
||||
]);
|
||||
});
|
||||
|
||||
it("recovers a preparation with no runner PID once its foreign controller stops", async () => {
|
||||
const fixture = await seedRun("REMOTE-PREPARATION-OWNER");
|
||||
const oldBoot = randomUUID();
|
||||
await fixture.db.update(heartbeatRuns).set({
|
||||
runnerProfileJson: { ...fixture.preparedProfile, nativeWorkspaceSync: remotePreparedWorkspace() },
|
||||
}).where(eq(heartbeatRuns.id, fixture.runId));
|
||||
await fixture.db.update(nativeRunFinalizations).set({ controllerBootId: oldBoot, controllerPid: 7,
|
||||
controllerProcessStartedAt: new Date(Date.now() - 600_000), controllerHostHash: "previous-pid-space",
|
||||
leaseOwner: "previous-preparation", leaseExpiresAt: new Date(0), attempt: 1, controllerGeneration: 1,
|
||||
}).where(eq(nativeRunFinalizations.runId, fixture.runId));
|
||||
await fixture.db.insert(controllerInstances).values({ bootId: oldBoot, processSpace: "previous-pid-space",
|
||||
pid: 7, leaseExpiresAt: new Date(Date.now() + 60_000) });
|
||||
registerControllerInstanceDb(fixture.db);
|
||||
const stamped = vi.fn(), unsubscribe = onControllerStamp(stamped);
|
||||
try {
|
||||
const [waiting] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
|
||||
restartKind: "hard", runIds: [fixture.runId] });
|
||||
expect(waiting).toMatchObject({ kind: "awaiting_evidence", reason: "controller_lease_held", retryAt: expect.any(Date) });
|
||||
expect(stamped).not.toHaveBeenCalled();
|
||||
const [before] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId));
|
||||
await claimNativeRestartRecoveries({ db: fixture.db, controller: successor, restartKind: "hard", runIds: [fixture.runId] });
|
||||
const [after] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId));
|
||||
expect(after!.updatedAt).toEqual(before!.updatedAt);
|
||||
await fixture.db.update(controllerInstances).set({ stoppedAt: new Date() }).where(eq(controllerInstances.bootId, oldBoot));
|
||||
const dispositions = (await Promise.all([successor, { ...successor, bootId: randomUUID() }].map(controller =>
|
||||
claimNativeRestartRecoveries({ db: fixture.db, controller, restartKind: "hard", runIds: [fixture.runId] })))).flat();
|
||||
expect(dispositions.filter(row => row.kind === "bootstrap_incomplete")).toHaveLength(1);
|
||||
expect(dispositions.filter(row => row.kind === "awaiting_evidence")).toHaveLength(1);
|
||||
expect(stamped).toHaveBeenCalledOnce();
|
||||
const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
|
||||
expect(run).toMatchObject({ status: "running", processPid: null, nativeSessionId: fixture.native.normalizedSessionId });
|
||||
const [owner] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId));
|
||||
expect(owner).toMatchObject({ attempt: 1, controllerGeneration: 2 });
|
||||
} finally { unsubscribe(); resetControllerInstanceLeaseForTests(); }
|
||||
});
|
||||
|
||||
it("recovers an unclaimed remote startup once without requiring a provider checkpoint", async () => {
|
||||
const fixture = await seedRun("REMOTE-UNCLAIMED");
|
||||
await fixture.db.update(heartbeatRuns).set({
|
||||
|
||||
@@ -303,7 +303,7 @@ import { noteNativeRunnerEvidence } from "./native-runner-evidence-signal.js";
|
||||
import { removeNativeHarnessBackup } from "./native-harness-backup-cleanup.js";
|
||||
import { registerLiveRunnerGoalController } from "../runner-goal-control-broker.js";
|
||||
import { applyRunnerGoalPrpEvent } from "../runner-goals.js";
|
||||
import { currentProcessSpaceHash } from "../controller-process-identity.js";
|
||||
import { currentControllerProcessStamp, currentProcessSpaceHash } from "../controller-process-identity.js";
|
||||
import { proveOwnerGone } from "../owner-proof.js";
|
||||
import {
|
||||
NativeRunnerOwnershipUnverifiedError,
|
||||
@@ -9996,6 +9996,7 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
}
|
||||
const leaseNow = new Date();
|
||||
confirmedLeaseDeadline = performance.now() + executionLeaseTtlMs;
|
||||
currentControllerProcessStamp();
|
||||
const claimed = await tx
|
||||
.update(nativeRunFinalizations)
|
||||
.set({
|
||||
|
||||
Reference in new issue
Block a user