mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(native): start a cold server from active database checkpoints (#20792)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs save their controller state in complete encrypted database checkpoints. > - A rolling update starts a replacement with an empty local home while the old server still owns active remote work. > - Startup required a local controller marker before it checked the active database authority. > - The replacement could report `native_controllers_require_migration` even when the checkpoint, workspace and live owner were valid. > - This change uses the existing complete checkpoint proof when the local marker is absent. A marker that exists keeps its strict validation. > - The benefit is that a cold replacement can start while supported remote work continues. ## Linked Issues or Issue Description **What happened?** A volume-free replacement failed startup with `native_controllers_require_migration`. The old server had a local external-control marker and a valid active database checkpoint. The replacement had the checkpoint but no local marker. **Expected behavior** An absent local marker must not make a complete, valid database checkpoint into a local migration obligation. Missing, corrupt or mismatched durable evidence must still refuse startup. **Steps to reproduce** 1. Save a full native checkpoint with its real checkpoint writer. Keep its source run active with a live generation-one owner and its bound remote workspace. 2. Remove only the local controller marker to represent a cold replacement home. 3. Inspect native controller obligations. Before this change, the inspection returns true. **Paperclip version or commit** Reproduced on `eb75c9bb855ce83915256efc9d6b308a272cdeff`. The fix is based on `c44a422607f9f08c39ad98f9c830fe6e3dba8dc2`. **Deployment mode** Volume-free server with native remote execution and database checkpoint persistence. ## What Changed - Let an absent local marker use the existing full checkpoint, run, session, workspace and owner checks. - Keep validation of every present marker and the final file reread that detects a change during inspection. - Add a cold-home variant of the existing real PostgreSQL active-checkpoint regression. Its live source owner has a different boot ID from the inspecting process. - Run the same refusal controls for missing checkpoint parts, invalid identity, stale leases, changed task/workspace bindings and invalid local markers. ## Verification - Genuine RED: the cold-home case failed on unchanged production code with `expected true to be false` after the real checkpoint was saved and only its local marker was removed. - Rebased GREEN, Node 24: `node node_modules/vitest/vitest.mjs run --project @paperclipai/server server/src/services/native-runtime/native-controller-migration.integration.test.ts -t 'offline native controller migration \(active_(durable|durable_cold|warm)\)'` — 3 passed, 53 unrelated cases filtered out. These controls use disposable real PostgreSQL and the canonical encrypted checkpoint writer/reader. - Server layered TypeScript check: `cd server && node scripts/typecheck-layers.mjs` — PASS for all source and test layers. - `node scripts/check-module-boundaries.mjs` and `git diff --check` — PASS. - [Required CI](https://github.com/paperclipai/paperclip/actions/runs/43003635878) passed on `7c5cb8fc16f3f7e3bcfec0ce10bf0b43d638d09b`. All 60 check runs are terminal: 55 success, 4 skipped and one neutral 5/5 review. Greptile is 5/5 with no review threads. Live rolling-update acceptance remains a separate delivery check. ## Risks - No schema or saved-state format changes. State saved by the previous release uses the same checkpoint reader. - The absent-marker path still needs a complete valid checkpoint and the existing identity, workspace and owner proof. Present legacy or malformed markers still refuse. - This changes eligibility only. It does not take ownership, launch work, copy state, release a lease or alter admission. - No command or configuration changes need operator documentation. ## Model Used OpenAI Codex, GPT-6, with reasoning, tool use and code execution. The host does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal issue id - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
c44a422607
commit
bc453342ef
2 files changed
+15
-11
No files matched your search
@@ -209,19 +209,19 @@ async function preservedWarmMarker(db: Db, selected: Selected,
|
||||
async function preservedActiveCheckpoint(db: Db, selected: Selected,
|
||||
source: NonNullable<Awaited<ReturnType<typeof findNativeSessionResumeRun>>>, paths: Awaited<ReturnType<typeof localPaths>>) {
|
||||
if (source.status !== "running" || source.runtimeMode !== "native") return false;
|
||||
let marker: Record<string, unknown>, bytes: string | undefined;
|
||||
let marker: Record<string, unknown> | null, bytes: string | undefined;
|
||||
try {
|
||||
bytes = await readPrivateSource(paths.controlFile, 4096);
|
||||
marker = record(JSON.parse(bytes ?? "null"));
|
||||
marker = bytes === undefined ? null : record(JSON.parse(bytes));
|
||||
} catch { return false; }
|
||||
if (marker.schema !== "paperclip.runner.external-control-plane-state.v1"
|
||||
|| Object.keys(marker).sort().join(",") !== "identity,schema") return false;
|
||||
if (marker !== null && (marker.schema !== "paperclip.runner.external-control-plane-state.v1"
|
||||
|| Object.keys(marker).sort().join(",") !== "identity,schema")) return false;
|
||||
const binding = { companyId: selected.companyId, issueId: selected.issueId, runId: source.id };
|
||||
const checkpoint = await loadNativeRunnerCheckpoint(db, binding).catch(() => null);
|
||||
const header = checkpoint && inspectDurablePrpCheckpointHeader(checkpoint.records.get("header") ?? "");
|
||||
if (!header || header.runId !== source.id || header.runnerInstanceId !== source.runnerInstanceId
|
||||
|| header.normalizedSessionId !== selected.sessionId) return false;
|
||||
const exact = nativeSha256(header) === nativeSha256(marker.identity), markerIdentity = record(marker.identity);
|
||||
const exact = marker === null || nativeSha256(header) === nativeSha256(marker.identity), markerIdentity = record(marker?.identity);
|
||||
if (!exact && (paths.execution.session.lifecyclePolicy.mode !== "warm"
|
||||
|| ["runnerInstanceId", "normalizedSessionId", "environmentLeaseId"].some(field => markerIdentity[field] !== header[field as keyof typeof header])
|
||||
|| !await preservedWarmMarker(db, selected, source, markerIdentity))) return false;
|
||||
|
||||
Reference in new issue
Block a user