fix(runtime): bound checkpoint data and preserve durable failed sessions (#18959)

## Thinking Path

> - Paperclip manages AI agents and their work.
> - Native Runner checkpoints preserve controller state across server
replacements.
> - Enrollment imports legacy controller state into those durable
checkpoints.
> - The importer passed extra source fields into each encrypted record,
duplicating the whole source and exhausting memory.
> - Some failed runs already had durable checkpoints but no session
snapshot. Re-importing them incorrectly required resumable state.
> - This PR stores only each record's binding and contents, and
preserves matching failed checkpoints without changing their recovery
status.

## Linked Issues or Issue Description

Refs #18957 and #18943.

**What happened?**

A 538-record migrated checkpoint stored 1,108,851,510 bytes of encrypted
material. Every record included the full 1.4 MB legacy source. Large
writes disconnected and later reads exhausted memory.

**Expected behavior**

Each encrypted record contains only its company, issue, run, part and
contents.

**Steps to reproduce**

Run offline controller migration with a retained controller that has
many commands. Decrypt a saved envelope and inspect its fields. The old
writer also includes the entire offline migration input.

**Paperclip version or commit**

Reproduced on `a1f3637c96dfadaf3a15ae9c83aa8227f4fed20e`.

## What Changed

- Serialize only company, issue, run, part and contents in encrypted
checkpoint envelopes.
- Preserve an already-durable failed checkpoint when its run, runner and
session match and the legacy session snapshot is absent. Keep its
failure, recovery blockers, owner and pointers unchanged.
- Report the number of failed checkpoints preserved during maintenance.
- Test exact encrypted contents, idempotent preservation after removing
the old home, and rejection of successful or foreign-session sources.
- Document the migration findings and verification boundaries.

## Verification

- The oversized-envelope regression failed before the serializer fix.
- The already-durable failed-source regression failed before the
preservation fix.
- Four focused test files: 44 tests passed. The migration suite passed
all 5 tests again after adding the foreign-session control.
- Server typecheck passed. Greptile reviewed `c4f02a064c` at 5/5 with no
findings. The exact-head PR workflow passed, including typecheck, build
and all required tests.
- Live staging repair preserved all 16 affected checkpoint sets: 2,982
records, 6,400,144,676 bytes reduced to 15,419,528 bytes. Each record
matched its legacy source and passed re-encryption verification.
Original encrypted rows and source files remain on the retained volume.
- Full hosted enrollment and active-agent deployment proof remain
pending. Deployment will follow the normal staging release pipeline.

## Risks

The serializer affects normal and migrated checkpoint writes. The
decoder already accepts the same five fields and continues to accept
older envelopes. Preservation applies only to a failed run with an
exact, validated durable checkpoint and no session snapshot; it grants
no new recovery authority. This change does not rewrite existing
database records automatically. Existing failures and missing provider
files remain failures until their owning recovery path resolves them.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use and code execution.
The exact deployment model ID and context-window size are not exposed in
this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
This commit is contained in:
Dotta authored and GitHub committed 2026-11-04 07:14:58 -06:00
1 parent 81b01bcc16
commit 55d7df2bad
5 files changed
+87 -3

No files matched your search

@@ -68,6 +68,13 @@ async function localPaths(source: NonNullable<Awaited<ReturnType<typeof findNati
return { ...descriptor, execution, controlFile: join(control, "control-plane-state.json") };
}
function preservedFailedCheckpoint(source: NonNullable<Awaited<ReturnType<typeof findNativeSessionResumeRun>>>,
sessionId: string, checkpoint: Awaited<ReturnType<typeof loadNativeRunnerCheckpoint>>) {
return source.status === "failed" && source.runnerProfileJson.sessionCheckpoint == null
&& checkpoint?.runId === source.id && checkpoint.recoveryEvidence.runnerInstanceId === source.runnerInstanceId
&& checkpoint.recoveryEvidence.normalizedSessionId === sessionId;
}
export async function hasLocalNativeControllerObligations(db: Db, storage?: StorageService | null): Promise<boolean> {
const roots = await readdir(nativeHarnessStateBase()).catch((error: NodeJS.ErrnoException) => {
if (error.code === "ENOENT") return [] as string[]; throw error;
@@ -77,6 +84,8 @@ export async function hasLocalNativeControllerObligations(db: Db, storage?: Stor
const binding = { companyId: selected.companyId, issueId: selected.issueId, runId: source.id };
if (readNativeControllerRecovery(source.runnerProfileJson.nativeControllerRecovery, source.runnerProfileJson.sessionCheckpoint)
&& await hasNativeRunnerCheckpoint(db, binding)) continue;
if (source.status === "failed" && source.runnerProfileJson.sessionCheckpoint == null
&& preservedFailedCheckpoint(source, selected.sessionId, await loadNativeRunnerCheckpoint(db, binding))) continue;
const paths = await localPaths(source);
const present = await lstat(paths.controlFile).catch((error: NodeJS.ErrnoException) => {
if (error.code === "ENOENT") return undefined; throw error;
@@ -96,11 +105,12 @@ export async function hasLocalNativeControllerObligations(db: Db, storage?: Stor
export async function migrateNativeControllers(db: Db, input: { writersStopped: true; signal: AbortSignal; storage?: StorageService }) {
const [active] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(offlineMigrationExecutionWhere()).limit(1);
if (!input.writersStopped || active) throw new Error("native_controller_migration_requires_offline_writers");
const counts = { inspected: 0, imported: 0, unchanged: 0, sourceAlreadyAbsent: 0, archivedFailed: 0 };
const counts = { inspected: 0, imported: 0, unchanged: 0, sourceAlreadyAbsent: 0, archivedFailed: 0, preservedFailed: 0 };
for await (const { selected, source } of retainedSources(db, input.signal)) {
counts.inspected++;
const binding = { companyId: selected.companyId, issueId: selected.issueId, runId: source.id };
const checkpoint = await loadNativeRunnerCheckpoint(db, binding);
if (preservedFailedCheckpoint(source, selected.sessionId, checkpoint)) { counts.preservedFailed++; continue; }
const metadata = readNativeControllerRecovery(source.runnerProfileJson.nativeControllerRecovery, source.runnerProfileJson.sessionCheckpoint);
const snapshot = source.runnerProfileJson.sessionCheckpoint as { identity?: Record<string, unknown> } | null;
if (checkpoint && metadata?.checkpoint?.runId === source.id