From bc453342ef4a93e3ddc7fd4bc8cf177311f5c5c6 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 11 Nov 2026 04:42:05 -0600 Subject: [PATCH] fix(native): start a cold server from active database checkpoints (#20792) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs save their controller state in complete encrypted database checkpoints. > - A rolling update starts a replacement with an empty local home while the old server still owns active remote work. > - Startup required a local controller marker before it checked the active database authority. > - The replacement could report `native_controllers_require_migration` even when the checkpoint, workspace and live owner were valid. > - This change uses the existing complete checkpoint proof when the local marker is absent. A marker that exists keeps its strict validation. > - The benefit is that a cold replacement can start while supported remote work continues. ## Linked Issues or Issue Description **What happened?** A volume-free replacement failed startup with `native_controllers_require_migration`. The old server had a local external-control marker and a valid active database checkpoint. The replacement had the checkpoint but no local marker. **Expected behavior** An absent local marker must not make a complete, valid database checkpoint into a local migration obligation. Missing, corrupt or mismatched durable evidence must still refuse startup. **Steps to reproduce** 1. Save a full native checkpoint with its real checkpoint writer. Keep its source run active with a live generation-one owner and its bound remote workspace. 2. Remove only the local controller marker to represent a cold replacement home. 3. Inspect native controller obligations. Before this change, the inspection returns true. **Paperclip version or commit** Reproduced on `eb75c9bb855ce83915256efc9d6b308a272cdeff`. The fix is based on `c44a422607f9f08c39ad98f9c830fe6e3dba8dc2`. **Deployment mode** Volume-free server with native remote execution and database checkpoint persistence. ## What Changed - Let an absent local marker use the existing full checkpoint, run, session, workspace and owner checks. - Keep validation of every present marker and the final file reread that detects a change during inspection. - Add a cold-home variant of the existing real PostgreSQL active-checkpoint regression. Its live source owner has a different boot ID from the inspecting process. - Run the same refusal controls for missing checkpoint parts, invalid identity, stale leases, changed task/workspace bindings and invalid local markers. ## Verification - Genuine RED: the cold-home case failed on unchanged production code with `expected true to be false` after the real checkpoint was saved and only its local marker was removed. - Rebased GREEN, Node 24: `node node_modules/vitest/vitest.mjs run --project @paperclipai/server server/src/services/native-runtime/native-controller-migration.integration.test.ts -t 'offline native controller migration \(active_(durable|durable_cold|warm)\)'` — 3 passed, 53 unrelated cases filtered out. These controls use disposable real PostgreSQL and the canonical encrypted checkpoint writer/reader. - Server layered TypeScript check: `cd server && node scripts/typecheck-layers.mjs` — PASS for all source and test layers. - `node scripts/check-module-boundaries.mjs` and `git diff --check` — PASS. - [Required CI](https://github.com/paperclipai/paperclip/actions/runs/43003635878) passed on `7c5cb8fc16f3f7e3bcfec0ce10bf0b43d638d09b`. All 60 check runs are terminal: 55 success, 4 skipped and one neutral 5/5 review. Greptile is 5/5 with no review threads. Live rolling-update acceptance remains a separate delivery check. ## Risks - No schema or saved-state format changes. State saved by the previous release uses the same checkpoint reader. - The absent-marker path still needs a complete valid checkpoint and the existing identity, workspace and owner proof. Present legacy or malformed markers still refuse. - This changes eligibility only. It does not take ownership, launch work, copy state, release a lease or alter admission. - No command or configuration changes need operator documentation. ## Model Used OpenAI Codex, GPT-6, with reasoning, tool use and code execution. The host does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal issue id - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- ...tive-controller-migration.integration.test.ts | 16 ++++++++++------ .../native-controller-migration.ts | 10 +++++----- 2 files changed, 15 insertions(+), 11 deletions(-) diff --git a/server/src/services/native-runtime/native-controller-migration.integration.test.ts b/server/src/services/native-runtime/native-controller-migration.integration.test.ts index 693022dd224..62d1ff52e78 100644 --- a/server/src/services/native-runtime/native-controller-migration.integration.test.ts +++ b/server/src/services/native-runtime/native-controller-migration.integration.test.ts @@ -26,7 +26,7 @@ import { legacyNativeControllerArchive } from "./native-controller-archive.js"; import { restoreDurableNativeHarnessBackup } from "./native-harness-backup-storage.js"; describeEmbeddedPostgres.each(["archive_online", "archive_online_owned", "archive_online_changed", "archive_online_commit", "archive_online_active", "archive_online_foreign", - "active_warm", "source_scan", "active_durable", "online_success", "online_owned", "online_same_source", "online_busy", "online_changed", "online_evidence", "import", "archive", "success", "batch", "durable_failed", "durable_inline_failed", + "active_warm", "source_scan", "active_durable", "active_durable_cold", "online_success", "online_owned", "online_same_source", "online_busy", "online_changed", "online_evidence", "import", "archive", "success", "batch", "durable_failed", "durable_inline_failed", "stale_confirmed", "stale_foreign", "stale_changed", "stale_queued", "stale_overlap", "stale_success", "stale_busy", "stale_durable", "stale_durable_absent", "stale_durable_runner", "stale_durable_session", "stale_durable_snapshot", "controller_confirmed", "controller_foreign", "controller_thread", "controller_runner", "controller_success", "controller_busy", "controller_invalid", "controller_durable", @@ -231,7 +231,7 @@ describeEmbeddedPostgres.each(["archive_online", "archive_online_owned", "archiv expect(await readFile(f.file, "utf8")).toBe(f.source); } finally { checkpointSpy.mockRestore(); spy.mockRestore(); } }); - if (scenario === "active_durable") it("accepts a complete active database authority without terminal metadata and refuses stale or incomplete handoff proof", async () => { + if (scenario === "active_durable" || scenario === "active_durable_cold") it("accepts a complete active database authority without terminal metadata and refuses stale or incomplete handoff proof", async () => { const f = await fixture(1, false, true), controller = await currentNativeControllerIdentity(); const binding = { companyId: f.companyId, issueId: f.issueId, runId: f.runId }; const { nativeWorkspaceSync, ...baseProfile } = f.profile; @@ -241,25 +241,28 @@ describeEmbeddedPostgres.each(["archive_online", "archive_online_owned", "archiv const profile = { ...baseProfile, sessionCheckpoint: null, remoteWorkspace }; await f.db.update(heartbeatRuns).set({ status: "running", finishedAt: null, nativePhase: "observed", processPid: process.pid, processStartedAt: new Date(), runnerProfileJson: profile }).where(eq(heartbeatRuns.id, f.runId)); - const owner = { controllerBootId: controller.bootId, controllerGeneration: 1, leaseOwner: "live-checkpoint-writer", + const owner = { controllerBootId: scenario === "active_durable_cold" ? randomUUID() : controller.bootId, + controllerGeneration: 1, leaseOwner: "live-checkpoint-writer", leaseExpiresAt: new Date(Date.now() + 60_000), phase: "observed", resultId: null, failureCode: null, recoveryState: null } as const; await f.db.update(nativeRunFinalizations).set(owner).where(eq(nativeRunFinalizations.runId, f.runId)); await f.db.update(issues).set({ executionRunId: f.runId, checkoutRunId: null }).where(eq(issues.id, f.issueId)); const core = new DurablePrpControlPlane({ stateDirectory: join(f.paths.root, "control-plane"), identity: f.identity, expectedRunnerVersion: "migration-fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` }); - core.setCheckpointPersistence(nativeRunnerCheckpointWriter(f.db, { ...binding, bootId: controller.bootId, + core.setCheckpointPersistence(nativeRunnerCheckpointWriter(f.db, { ...binding, bootId: owner.controllerBootId, leaseOwner: owner.leaseOwner, generation: 1, authorityRunId: f.runId })); try { await core.flushCheckpoint(); const marker = await readFile(f.file, "utf8"); const rows = await f.db.select().from(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId)); expect(rows.length).toBeGreaterThan(2); + if (scenario === "active_durable_cold") await rm(f.file); expect(await hasLocalNativeControllerObligations(f.db)).toBe(false); for (const value of [f.source, JSON.stringify({ schema: "unknown", identity: f.identity }), JSON.stringify({ schema: "paperclip.runner.external-control-plane-state.v1", identity: { ...f.identity, runId: randomUUID() } })]) { await writeFile(f.file, value); expect(await hasLocalNativeControllerObligations(f.db)).toBe(true); } - await writeFile(f.file, marker); + if (scenario === "active_durable_cold") await rm(f.file); + else await writeFile(f.file, marker); for (const part of ["cursor", "command:1"]) { await f.db.delete(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.part, part)); expect(await hasLocalNativeControllerObligations(f.db)).toBe(true); @@ -304,7 +307,8 @@ describeEmbeddedPostgres.each(["archive_online", "archive_online_owned", "archiv .where(eq(heartbeatRuns.id, f.runId)); expect(await hasLocalNativeControllerObligations(f.db)).toBe(false); await f.db.update(heartbeatRuns).set({ runnerProfileJson: profile }).where(eq(heartbeatRuns.id, f.runId)); - expect(await readFile(f.file, "utf8")).toBe(marker); + if (scenario === "active_durable_cold") await expect(readFile(f.file, "utf8")).rejects.toMatchObject({ code: "ENOENT" }); + else expect(await readFile(f.file, "utf8")).toBe(marker); expect(await f.db.select().from(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId))).toEqual(rows); } finally { await core.stop(); } }); diff --git a/server/src/services/native-runtime/native-controller-migration.ts b/server/src/services/native-runtime/native-controller-migration.ts index 2c127058e67..aeb22c5e517 100644 --- a/server/src/services/native-runtime/native-controller-migration.ts +++ b/server/src/services/native-runtime/native-controller-migration.ts @@ -209,19 +209,19 @@ async function preservedWarmMarker(db: Db, selected: Selected, async function preservedActiveCheckpoint(db: Db, selected: Selected, source: NonNullable>>, paths: Awaited>) { if (source.status !== "running" || source.runtimeMode !== "native") return false; - let marker: Record, bytes: string | undefined; + let marker: Record | null, bytes: string | undefined; try { bytes = await readPrivateSource(paths.controlFile, 4096); - marker = record(JSON.parse(bytes ?? "null")); + marker = bytes === undefined ? null : record(JSON.parse(bytes)); } catch { return false; } - if (marker.schema !== "paperclip.runner.external-control-plane-state.v1" - || Object.keys(marker).sort().join(",") !== "identity,schema") return false; + if (marker !== null && (marker.schema !== "paperclip.runner.external-control-plane-state.v1" + || Object.keys(marker).sort().join(",") !== "identity,schema")) return false; const binding = { companyId: selected.companyId, issueId: selected.issueId, runId: source.id }; const checkpoint = await loadNativeRunnerCheckpoint(db, binding).catch(() => null); const header = checkpoint && inspectDurablePrpCheckpointHeader(checkpoint.records.get("header") ?? ""); if (!header || header.runId !== source.id || header.runnerInstanceId !== source.runnerInstanceId || header.normalizedSessionId !== selected.sessionId) return false; - const exact = nativeSha256(header) === nativeSha256(marker.identity), markerIdentity = record(marker.identity); + const exact = marker === null || nativeSha256(header) === nativeSha256(marker.identity), markerIdentity = record(marker?.identity); if (!exact && (paths.execution.session.lifecyclePolicy.mode !== "warm" || ["runnerInstanceId", "normalizedSessionId", "environmentLeaseId"].some(field => markerIdentity[field] !== header[field as keyof typeof header]) || !await preservedWarmMarker(db, selected, source, markerIdentity))) return false;