From 55d7df2bad95418de71fdb3c28fef7b36c9f118d Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 4 Nov 2026 07:14:58 -0600 Subject: [PATCH] fix(runtime): bound checkpoint data and preserve durable failed sessions (#18959) ## Thinking Path > - Paperclip manages AI agents and their work. > - Native Runner checkpoints preserve controller state across server replacements. > - Enrollment imports legacy controller state into those durable checkpoints. > - The importer passed extra source fields into each encrypted record, duplicating the whole source and exhausting memory. > - Some failed runs already had durable checkpoints but no session snapshot. Re-importing them incorrectly required resumable state. > - This PR stores only each record's binding and contents, and preserves matching failed checkpoints without changing their recovery status. ## Linked Issues or Issue Description Refs #18957 and #18943. **What happened?** A 538-record migrated checkpoint stored 1,108,851,510 bytes of encrypted material. Every record included the full 1.4 MB legacy source. Large writes disconnected and later reads exhausted memory. **Expected behavior** Each encrypted record contains only its company, issue, run, part and contents. **Steps to reproduce** Run offline controller migration with a retained controller that has many commands. Decrypt a saved envelope and inspect its fields. The old writer also includes the entire offline migration input. **Paperclip version or commit** Reproduced on `a1f3637c96dfadaf3a15ae9c83aa8227f4fed20e`. ## What Changed - Serialize only company, issue, run, part and contents in encrypted checkpoint envelopes. - Preserve an already-durable failed checkpoint when its run, runner and session match and the legacy session snapshot is absent. Keep its failure, recovery blockers, owner and pointers unchanged. - Report the number of failed checkpoints preserved during maintenance. - Test exact encrypted contents, idempotent preservation after removing the old home, and rejection of successful or foreign-session sources. - Document the migration findings and verification boundaries. ## Verification - The oversized-envelope regression failed before the serializer fix. - The already-durable failed-source regression failed before the preservation fix. - Four focused test files: 44 tests passed. The migration suite passed all 5 tests again after adding the foreign-session control. - Server typecheck passed. Greptile reviewed `c4f02a064c` at 5/5 with no findings. The exact-head PR workflow passed, including typecheck, build and all required tests. - Live staging repair preserved all 16 affected checkpoint sets: 2,982 records, 6,400,144,676 bytes reduced to 15,419,528 bytes. Each record matched its legacy source and passed re-encryption verification. Original encrypted rows and source files remain on the retained volume. - Full hosted enrollment and active-agent deployment proof remain pending. Deployment will follow the normal staging release pipeline. ## Risks The serializer affects normal and migrated checkpoint writes. The decoder already accepts the same five fields and continues to accept older envelopes. Preservation applies only to a failed run with an exact, validated durable checkpoint and no session snapshot; it grants no new recovery authority. This change does not rewrite existing database records automatically. Existing failures and missing provider files remain failures until their owning recovery path resolves them. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use and code execution. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --- ...-04-preserve-native-workspace-migration.md | 26 ++++++++++ .../maintenance/ephemeral-migration-data.ts | 1 + ...e-controller-migration.integration.test.ts | 49 ++++++++++++++++++- .../native-controller-migration.ts | 12 ++++- .../native-runner-checkpoint.ts | 2 +- 5 files changed, 87 insertions(+), 3 deletions(-) diff --git a/doc/plans/2026-11-04-preserve-native-workspace-migration.md b/doc/plans/2026-11-04-preserve-native-workspace-migration.md index 92cb05cd216..1a0e88fce1f 100644 --- a/doc/plans/2026-11-04-preserve-native-workspace-migration.md +++ b/doc/plans/2026-11-04-preserve-native-workspace-migration.md @@ -142,3 +142,29 @@ forces a conflict in a later batch, verifies that earlier batches roll back, then imports and decrypts the complete original checkpoint. The precise server-side reason for the connection closure is not yet established; the unbatched failures and batched comparison remain in the verification record. + +## Checkpoint envelope expansion found during the live import + +The encryption boundary spread the caller's entire input into every record. +For offline imports that input includes the complete legacy controller source. +One 538-record checkpoint stored 1,108,851,510 bytes of encrypted material, +although the intended records total about 2.2 MB after encryption. This explains +the oversized writes and subsequent memory exhaustion; the earlier SQL-shape +probe passed a narrow binding and therefore did not reproduce the actual owner +payload. The previous write failures and killed attempts remain in the evidence. + +The checkpoint writer now selects company, issue, run, part and contents explicitly. +The real database migration test decrypts each imported envelope and requires +exactly those fields, with the original record content. It failed before the fix. +Existing envelopes remain readable. For the staging imports from this attempt, +a guarded offline repair backs up the encrypted rows on the retained volume, +checks every record against its original legacy source, and rewrites only the +envelope. Run, session, owner and task metadata remain unchanged. + +The complete 807-source audit also found five failed runs with existing durable +checkpoints and no provider-session snapshot. Their original controller state +is already preserved. Enrollment now verifies the durable run, Runner and +session binding and counts these failed states separately. It does not create +a session snapshot or recovery authority, clear a blocker, or change their +checkpoint rows, task pointers or failed status. A successful run with a missing +snapshot still blocks enrollment. diff --git a/server/src/maintenance/ephemeral-migration-data.ts b/server/src/maintenance/ephemeral-migration-data.ts index c45a5731fe3..cf1fc2ede75 100644 --- a/server/src/maintenance/ephemeral-migration-data.ts +++ b/server/src/maintenance/ephemeral-migration-data.ts @@ -44,6 +44,7 @@ export async function migrateEphemeralServerData(db: Db, input: { counts.nativeControllersUnchanged = controllers.unchanged; counts.nativeControllerSourcesAlreadyAbsent = controllers.sourceAlreadyAbsent; counts.nativeControllersArchivedFailed = controllers.archivedFailed; + counts.nativeControllersPreservedFailed = controllers.preservedFailed; phase("native_harness_backups"); counts.nativeHarnessBackups = await migrateNativeHarnessBackups(db, { storage, writersStopped: true, signal }); phase("workspaces"); diff --git a/server/src/services/native-runtime/native-controller-migration.integration.test.ts b/server/src/services/native-runtime/native-controller-migration.integration.test.ts index 1c01589d8df..480f475c183 100644 --- a/server/src/services/native-runtime/native-controller-migration.integration.test.ts +++ b/server/src/services/native-runtime/native-controller-migration.integration.test.ts @@ -14,12 +14,13 @@ import { resolveNativeControllerRecovery } from "./native-controller-recovery.js import { resolveRemoteProviderPackAuthority } from "./remote-provider-pack-authority.js"; import { canonicalProviderPackJson } from "./remote-provider-pack-verification.js"; import { nativeSha256 } from "./canonical.js"; +import { createLocalEncryptedCipher } from "../../secrets/local-encrypted-provider.js"; import { createLocalDiskStorageProvider } from "../../storage/local-disk-provider.js"; import { createStorageService } from "../../storage/service.js"; import { legacyNativeControllerArchive } from "./native-controller-archive.js"; import { restoreDurableNativeHarnessBackup } from "./native-harness-backup-storage.js"; -describeEmbeddedPostgres.each(["import", "archive", "success", "batch"])("offline native controller migration (%s)", scenario => { +describeEmbeddedPostgres.each(["import", "archive", "success", "batch", "durable_failed"])("offline native controller migration (%s)", scenario => { const environment = nativeRestartFixture(); afterEach(() => vi.unstubAllEnvs()); async function fixture(commandCount = 1) { @@ -155,11 +156,57 @@ describeEmbeddedPostgres.each(["import", "archive", "success", "batch"])("offlin await f.db.delete(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId)); expect(await migrateNativeControllers(f.db, { writersStopped: true, signal: AbortSignal.timeout(30_000) })) .toMatchObject({ imported: 1 }); + const rows = await f.db.select().from(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId)); + const cipher = createLocalEncryptedCipher("read"); + for (const row of rows) expect(JSON.parse(cipher.decrypt(row.material))).toEqual({ + ...binding, part: row.part, contents: checkpoint.records.get(row.part), + }); const restored = await loadNativeRunnerCheckpoint(f.db, binding); expect(restored?.records).toEqual(checkpoint.records); expect(await readFile(f.file, "utf8")).toBe(f.source); } finally { f.db.$client.options.debug = previousDebug; } }, 60_000); + if (scenario === "durable_failed") it("preserves a failed durable checkpoint without inventing a missing session snapshot", async () => { + const f = await fixture(); + const binding = { companyId: f.companyId, issueId: f.issueId, runId: f.runId }; + const checkpoint = captureLegacyDurablePrpCheckpoint(f.source, { runId: f.runId, + runnerInstanceId: f.native.runnerInstanceId, normalizedSessionId: f.native.normalizedSessionId }, + { version: "migration-fixture", digest: `sha256:${"a".repeat(64)}` }); + await f.db.transaction(tx => insertMigratedNativeRunnerCheckpoint(tx, binding, checkpoint)); + await f.db.update(heartbeatRuns).set({ processPid: 12345, runnerProfileJson: { ...f.profile, sessionCheckpoint: null } }) + .where(eq(heartbeatRuns.id, f.runId)); + const options = { writersStopped: true as const, signal: AbortSignal.timeout(30_000) }; + await f.db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, f.runId)); + await expect(migrateNativeControllers(f.db, options)).rejects.toThrow("stale_supervisor_snapshot"); + await f.db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, f.runId)); + const [run] = await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + const [owner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + const rows = await f.db.select().from(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId)); + const foreign = JSON.parse(f.source); + foreign.identity.normalizedSessionId = randomUUID(); + const foreignCheckpoint = captureLegacyDurablePrpCheckpoint(JSON.stringify(foreign), { runId: f.runId, + runnerInstanceId: f.native.runnerInstanceId, normalizedSessionId: foreign.identity.normalizedSessionId }, + { version: "migration-fixture", digest: `sha256:${"a".repeat(64)}` }); + await f.db.delete(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId)); + await f.db.transaction(tx => insertMigratedNativeRunnerCheckpoint(tx, binding, foreignCheckpoint)); + expect(await hasLocalNativeControllerObligations(f.db)).toBe(true); + await expect(migrateNativeControllers(f.db, options)).rejects.toThrow("stale_supervisor_snapshot"); + await f.db.delete(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId)); + await f.db.insert(nativeRunnerCheckpoints).values(rows); + expect(await migrateNativeControllers(f.db, options)).toMatchObject({ preservedFailed: 1, imported: 0 }); + expect(await hasLocalNativeControllerObligations(f.db)).toBe(false); + expect(await migrateNativeControllers(f.db, options)).toMatchObject({ preservedFailed: 1, imported: 0 }); + expect(await f.db.select().from(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId))).toEqual(rows); + expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(run); + expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toEqual(owner); + expect((await f.db.select().from(agentTaskSessions).where(eq(agentTaskSessions.id, f.session.id)))[0].lastRunId).toBe(f.runId); + expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.queuedId)))[0].status).toBe("queued"); + await rm(f.paperclipHome, { recursive: true, force: true }); + expect((await loadNativeRunnerCheckpoint(f.db, binding))?.records).toEqual(checkpoint.records); + const recovery = await resolveNativeControllerRecovery(f.db, { companyId: f.companyId, agentId: f.agentId, + issueId: f.issueId, normalizedSessionId: f.native.normalizedSessionId, currentRunId: f.queuedId }); + expect(recovery.kind).not.toBe("recoverable"); + }); if (scenario === "archive") it("preserves an already-failed legacy controller without inventing executable recovery authority", async () => { const f = await fixture(); await rm(join(f.paths.root, "provider-pack-authority.json")); diff --git a/server/src/services/native-runtime/native-controller-migration.ts b/server/src/services/native-runtime/native-controller-migration.ts index 6665845fb4c..3a250116e3a 100644 --- a/server/src/services/native-runtime/native-controller-migration.ts +++ b/server/src/services/native-runtime/native-controller-migration.ts @@ -68,6 +68,13 @@ async function localPaths(source: NonNullable>>, + sessionId: string, checkpoint: Awaited>) { + return source.status === "failed" && source.runnerProfileJson.sessionCheckpoint == null + && checkpoint?.runId === source.id && checkpoint.recoveryEvidence.runnerInstanceId === source.runnerInstanceId + && checkpoint.recoveryEvidence.normalizedSessionId === sessionId; +} + export async function hasLocalNativeControllerObligations(db: Db, storage?: StorageService | null): Promise { const roots = await readdir(nativeHarnessStateBase()).catch((error: NodeJS.ErrnoException) => { if (error.code === "ENOENT") return [] as string[]; throw error; @@ -77,6 +84,8 @@ export async function hasLocalNativeControllerObligations(db: Db, storage?: Stor const binding = { companyId: selected.companyId, issueId: selected.issueId, runId: source.id }; if (readNativeControllerRecovery(source.runnerProfileJson.nativeControllerRecovery, source.runnerProfileJson.sessionCheckpoint) && await hasNativeRunnerCheckpoint(db, binding)) continue; + if (source.status === "failed" && source.runnerProfileJson.sessionCheckpoint == null + && preservedFailedCheckpoint(source, selected.sessionId, await loadNativeRunnerCheckpoint(db, binding))) continue; const paths = await localPaths(source); const present = await lstat(paths.controlFile).catch((error: NodeJS.ErrnoException) => { if (error.code === "ENOENT") return undefined; throw error; @@ -96,11 +105,12 @@ export async function hasLocalNativeControllerObligations(db: Db, storage?: Stor export async function migrateNativeControllers(db: Db, input: { writersStopped: true; signal: AbortSignal; storage?: StorageService }) { const [active] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(offlineMigrationExecutionWhere()).limit(1); if (!input.writersStopped || active) throw new Error("native_controller_migration_requires_offline_writers"); - const counts = { inspected: 0, imported: 0, unchanged: 0, sourceAlreadyAbsent: 0, archivedFailed: 0 }; + const counts = { inspected: 0, imported: 0, unchanged: 0, sourceAlreadyAbsent: 0, archivedFailed: 0, preservedFailed: 0 }; for await (const { selected, source } of retainedSources(db, input.signal)) { counts.inspected++; const binding = { companyId: selected.companyId, issueId: selected.issueId, runId: source.id }; const checkpoint = await loadNativeRunnerCheckpoint(db, binding); + if (preservedFailedCheckpoint(source, selected.sessionId, checkpoint)) { counts.preservedFailed++; continue; } const metadata = readNativeControllerRecovery(source.runnerProfileJson.nativeControllerRecovery, source.runnerProfileJson.sessionCheckpoint); const snapshot = source.runnerProfileJson.sessionCheckpoint as { identity?: Record } | null; if (checkpoint && metadata?.checkpoint?.runId === source.id diff --git a/server/src/services/native-runtime/native-runner-checkpoint.ts b/server/src/services/native-runtime/native-runner-checkpoint.ts index 5d38781748f..4920ede5437 100644 --- a/server/src/services/native-runtime/native-runner-checkpoint.ts +++ b/server/src/services/native-runtime/native-runner-checkpoint.ts @@ -14,7 +14,7 @@ function encryptParts(binding: Binding, records: ReadonlyMap) { const cipher = createLocalEncryptedCipher("write"); return [...records].map(([part, contents]) => ({ companyId: binding.companyId, issueId: binding.issueId, runId: binding.runId, part, - material: cipher.encrypt(JSON.stringify({ ...binding, part, contents })), + material: cipher.encrypt(JSON.stringify({ companyId: binding.companyId, issueId: binding.issueId, runId: binding.runId, part, contents })), })); }