From 3a283fbc9b8df4d80dd4194ddc7efc2fa2042817 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 4 Nov 2026 13:40:53 -0600 Subject: [PATCH] fix(runtime): preserve failed sources with older warm supervisors (#19024) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A volume-free web server needs durable recovery data for retained sessions. > - An old failed follow-up can retain its snapshot while the warm supervisor still holds the preceding successful turn. > - The migration refuses to treat those different snapshots as the same execution authority. > - That refusal also blocks the tenant even when the failed source can be preserved as an archive. > - This pull request archives that narrow case after it verifies the older successful snapshot and all scope bindings. > - The failed run keeps its history and files while valid controller recovery keeps its existing proof requirements. ## Linked Issues or Issue Description Refs #18959, which added the existing failed-controller archive path. No duplicate PR for this case was found. **What happened?** Offline controller migration stops with `stale_supervisor_snapshot` for an old failed warm follow-up. Its local supervisor still contains the prior successful turn. Read-only checks found the same database snapshot and supervisor bytes in the live source and the isolated copy. The mismatch predates the rehearsal. **Expected behavior** Preserve the failed run's database state and available local files in the existing durable archive. Only archive when the older supervisor matches a completed successful run in the same company, agent, task, native session and Runner instance. Keep current execution and controller authority checks intact. **Steps to reproduce** Create a successful native turn and a later failed warm follow-up. Keep the successful snapshot in the supervisor file and the failed snapshot in the database. Give the failed source local controller bytes but no durable controller checkpoint. Run the offline migration. The new regression fails with `stale_supervisor_snapshot` before this change. ## What Changed - Verify the older supervisor against scoped successful history and the same provider thread. - Use the existing encrypted archive owner for that failed legacy source. - Preserve snapshots, run status, ownership, queued work and the session's last-run pointer. - Keep foreign, changed, active, successful and already-durable sources outside this path. - Document the preservation and recovery boundary. ## Verification - New positive regression: red on the original owner, green after the change. - Native migration and controller recovery: 27 checks passed. Final migration checks also cover refusal to replace a durable checkpoint. - Storage failure blocks publication. Repeated migration is safe. The archive restores the exact controller and supervisor bytes after the old home is deleted. - The archived failed source does not gain executable recovery authority. - Local server typecheck is blocked by reused workspace declarations for accounting, bootstrap and ingress contracts. No diagnostic names either modified owner or test file. Full local typecheck, repo-wide tests and build are not claimed; clean CI is required before merge. The exact-head CI has now passed typecheck, build, policy, Runner checks, browser E2E, and all server test shards. - Rehearsal verification against the copied historical source remains required. Full tenant migration and hosted deployment qualification remain separate gates. ## Risks The archive preserves evidence for an already failed source. It does not certify that source as resumable. A prior snapshot must match exactly, and the normal archive owner checks the terminal failed binding before it publishes. Existing durable checkpoints remain authoritative. A missing or changed proof still blocks migration. ## Model Used OpenAI Codex, exact model ID `gpt-6.1-sol`, with reasoning, tool use and code execution. The host does not expose context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id or instance-derived details - [x] I have run the relevant tests locally; retained failures and limits are listed above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- ...-existing-workspace-ephemeral-migration.md | 12 ++++ ...e-controller-migration.integration.test.ts | 69 +++++++++++++++++-- .../native-controller-migration.ts | 46 +++++++++++++ 3 files changed, 123 insertions(+), 4 deletions(-) diff --git a/doc/plans/2026-11-03-existing-workspace-ephemeral-migration.md b/doc/plans/2026-11-03-existing-workspace-ephemeral-migration.md index f4150f62fc9..2b1f6ddcbe7 100644 --- a/doc/plans/2026-11-03-existing-workspace-ephemeral-migration.md +++ b/doc/plans/2026-11-03-existing-workspace-ephemeral-migration.md @@ -48,6 +48,18 @@ yet make the whole tenant eligible for enrollment. ## Legacy recovery storage transition +### Failed controllers with an older warm supervisor + +An old failed follow-up can retain a database snapshot while its warm supervisor +still contains the preceding successful turn. Migration preserves the failed +run's database state and archives its available controller and supervisor files. +The archive requires the exact older snapshot to match a completed successful +run in the same company, agent, task, native session, and Runner instance. Its +provider thread must also match the failed source. Existing durable controller +checkpoints, active owners, changed snapshots, and foreign bindings still block +this archive path. The archive remains readable after removing the old home. +Execution recovery continues to require independently valid authority. + The migration owner must distinguish file location from recovery completion. An offline import can preserve a terminal failed run's local workspace without claiming that its remote files were recovered. The snapshot and migration diff --git a/server/src/services/native-runtime/native-controller-migration.integration.test.ts b/server/src/services/native-runtime/native-controller-migration.integration.test.ts index 480f475c183..4e489090f6b 100644 --- a/server/src/services/native-runtime/native-controller-migration.integration.test.ts +++ b/server/src/services/native-runtime/native-controller-migration.integration.test.ts @@ -1,6 +1,6 @@ import { createHash, randomUUID } from "node:crypto"; import { mkdir, readFile, rm, writeFile } from "node:fs/promises"; -import { join } from "node:path"; +import { dirname, join } from "node:path"; import { eq } from "drizzle-orm"; import { afterEach, expect, it, vi } from "vitest"; import { agentTaskSessions, heartbeatRuns, issues, nativeHarnessBackups, nativeRunFinalizations, nativeRunnerCheckpoints } from "@paperclipai/db"; @@ -20,10 +20,11 @@ import { createStorageService } from "../../storage/service.js"; import { legacyNativeControllerArchive } from "./native-controller-archive.js"; import { restoreDurableNativeHarnessBackup } from "./native-harness-backup-storage.js"; -describeEmbeddedPostgres.each(["import", "archive", "success", "batch", "durable_failed"])("offline native controller migration (%s)", scenario => { +describeEmbeddedPostgres.each(["import", "archive", "success", "batch", "durable_failed", + "stale_confirmed", "stale_foreign", "stale_changed", "stale_success", "stale_busy", "stale_durable"])("offline native controller migration (%s)", scenario => { const environment = nativeRestartFixture(); afterEach(() => vi.unstubAllEnvs()); - async function fixture(commandCount = 1) { + async function fixture(commandCount = 1, warm = false) { vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "73".repeat(32)); const f = await environment.seedRun(randomUUID()); const { companyId, agentId, paperclipHome, runtimeRoot } = environment.current; @@ -33,7 +34,7 @@ describeEmbeddedPostgres.each(["import", "archive", "success", "batch", "durable task: { identifier: "MIG-1", title: "Retained work", description: null, prompt: "Continue the work.", workMode: "standard" }, workspace: { cwd: "/workspace", repoUrl: null, repoRef: null, branchName: null }, session: { normalizedSessionId: f.native.normalizedSessionId, driverKind: "codex_app_server", protocolVersion: 1, - lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } }, + lifecyclePolicy: warm ? { mode: "warm", idleTimeoutMs: 60_000 } : { mode: "per_turn", idleTimeoutMs: null } }, completionContract: { id: randomUUID(), sha256: "fixture", schemaVersion: "paperclip.completion-contract.v1", contract: { revision: "1", objective: "Continue", criteria: [{ id: "objective", requirement: "Save work." }] } }, credentialBindings: [], interactionResponses: [], @@ -238,6 +239,66 @@ describeEmbeddedPostgres.each(["import", "archive", "success", "batch", "durable issueId: f.issueId, normalizedSessionId: f.native.normalizedSessionId, currentRunId: f.queuedId }); expect(recovery.kind).not.toBe("recoverable"); }); + if (scenario.startsWith("stale_")) it("archives a failed source only when its older supervisor matches scoped history", async () => { + const f = await fixture(1, true), priorId = randomUUID(); + const priorSnapshot = { ...f.snapshot, identity: { ...f.snapshot.identity, runId: priorId } }; + if (scenario === "stale_foreign") priorSnapshot.identity.agentId = randomUUID(); + const priorProfile = { ...f.profile, nativeExecutionInput: { ...f.profile.nativeExecutionInput, + binding: { ...f.profile.nativeExecutionInput.binding, runId: priorId } }, sessionCheckpoint: priorSnapshot }; + await f.db.insert(heartbeatRuns).values({ id: priorId, companyId: f.companyId, agentId: f.agentId, + nativeIssueId: f.issueId, nativeSessionId: f.native.normalizedSessionId, runnerInstanceId: f.native.runnerInstanceId, + runtimeMode: "native", status: "succeeded", createdAt: new Date(Date.now() - 60_000), + finishedAt: new Date(Date.now() - 50_000), runnerProfileJson: priorProfile }); + const supervisor = JSON.stringify({ schema: "paperclip.native-session-supervisor.v1", + configDigest: f.paths.configDigest, snapshot: priorSnapshot }); + await mkdir(dirname(f.paths.supervisorPath), { recursive: true }); + await writeFile(f.paths.supervisorPath, supervisor, { mode: 0o600 }); + if (scenario === "stale_changed") await f.db.update(heartbeatRuns).set({ runnerProfileJson: { + ...priorProfile, sessionCheckpoint: { ...priorSnapshot, sessionId: "different-provider-thread" }, + } }).where(eq(heartbeatRuns.id, priorId)); + if (scenario === "stale_success") await f.db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, f.runId)); + if (scenario === "stale_busy") await f.db.update(nativeRunFinalizations).set({ leaseOwner: "live-owner", + leaseExpiresAt: new Date(Date.now() + 60_000) }).where(eq(nativeRunFinalizations.runId, f.runId)); + if (scenario === "stale_durable") { + const checkpoint = captureLegacyDurablePrpCheckpoint(f.source, { runId: f.runId, + runnerInstanceId: f.native.runnerInstanceId, normalizedSessionId: f.native.normalizedSessionId }, + { version: "migration-fixture", digest: `sha256:${"a".repeat(64)}` }); + await f.db.transaction(tx => insertMigratedNativeRunnerCheckpoint(tx, + { companyId: f.companyId, issueId: f.issueId, runId: f.runId }, checkpoint)); + } + const [originalRun] = await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + const [originalOwner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + const storage = createStorageService(createLocalDiskStorageProvider(join(f.runtimeRoot, "durable-objects"))); + const options = { storage, writersStopped: true as const, signal: AbortSignal.timeout(30_000) }; + if (scenario !== "stale_confirmed") { + await expect(migrateNativeControllers(f.db, options)).rejects.toThrow(); + expect(await f.db.select().from(nativeHarnessBackups)).toHaveLength(0); + expect(await hasLocalNativeControllerObligations(f.db, storage)).toBe(true); + expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(originalRun); + expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toEqual(originalOwner); + return; + } + const put = vi.spyOn(storage, "putFile").mockRejectedValue(new Error("storage unavailable")); + await expect(migrateNativeControllers(f.db, options)).rejects.toThrow("storage unavailable"); + put.mockRestore(); + expect(await migrateNativeControllers(f.db, options)).toMatchObject({ archivedFailed: 1, imported: 0 }); + expect(await migrateNativeControllers(f.db, options)).toMatchObject({ archivedFailed: 1, imported: 0 }); + expect(await hasLocalNativeControllerObligations(f.db, storage)).toBe(false); + expect(await f.db.select().from(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId))).toHaveLength(0); + expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(originalRun); + expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toEqual(originalOwner); + expect((await f.db.select().from(agentTaskSessions).where(eq(agentTaskSessions.id, f.session.id)))[0].lastRunId).toBe(f.runId); + await rm(f.paperclipHome, { recursive: true, force: true }); + const descriptor = legacyNativeControllerArchive({ companyId: f.companyId, issueId: f.issueId, + runId: f.runId, controller: f.source, supervisor }); + const replacement = join(f.runtimeRoot, "archived-stale-supervisor"); + expect(await restoreDurableNativeHarnessBackup({ db: f.db, storage, companyId: f.companyId, + scopeDigest: descriptor.scopeDigest, backupRoot: replacement })).toBe(true); + expect(await readFile(join(replacement, "current", "control-plane-state.json"), "utf8")).toBe(f.source); + expect(await readFile(join(replacement, "current", "supervisor.json"), "utf8")).toBe(supervisor); + expect((await resolveNativeControllerRecovery(f.db, { companyId: f.companyId, agentId: f.agentId, + issueId: f.issueId, normalizedSessionId: f.native.normalizedSessionId, currentRunId: f.queuedId })).kind).not.toBe("recoverable"); + }); if (scenario === "success") it("does not substitute a failure archive for a successful retained session", async () => { const f = await fixture(); await rm(join(f.paths.root, "provider-pack-authority.json")); diff --git a/server/src/services/native-runtime/native-controller-migration.ts b/server/src/services/native-runtime/native-controller-migration.ts index 3a250116e3a..93626245b40 100644 --- a/server/src/services/native-runtime/native-controller-migration.ts +++ b/server/src/services/native-runtime/native-controller-migration.ts @@ -16,9 +16,44 @@ import { readNativeWorkspaceSyncReference } from "./native-workspace-sync.js"; import type { StorageService } from "../../storage/types.js"; import { durableNativeHarnessStorage } from "./native-harness-backup-storage.js"; import { archiveFailedLegacyNativeController, hasArchivedLegacyNativeController } from "./native-controller-archive.js"; +import { redactRunnerApiPayloads } from "./runner-api-log-redaction.js"; type Selected = { id: string; companyId: string; agentId: string; issueId: string; sessionId: string; createdAt: Date }; +function record(value: unknown): Record { + return value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; +} + +async function hasConfirmedOlderSupervisor(db: Db, selected: Selected, + source: NonNullable>>, + paths: Awaited>, supervisor: unknown) { + const snapshot = record(source.runnerProfileJson.sessionCheckpoint), identity = record(snapshot.identity); + const olderSnapshot = record(record(supervisor).snapshot), olderIdentity = record(olderSnapshot.identity); + const scope = { companyId: selected.companyId, agentId: selected.agentId, issueId: selected.issueId, sessionId: selected.sessionId }; + if (source.status !== "failed" || identity.runId !== source.id + || Object.entries(scope).some(([key, value]) => identity[key] !== value || olderIdentity[key] !== value) + || paths.execution.binding.runId !== source.id || paths.execution.binding.companyId !== selected.companyId + || paths.execution.binding.agentId !== selected.agentId || paths.execution.binding.issueId !== selected.issueId + || paths.execution.session.normalizedSessionId !== selected.sessionId + || typeof snapshot.sessionId !== "string" || !snapshot.sessionId.trim() + || snapshot.sessionId !== olderSnapshot.sessionId || snapshot.backendKind !== olderSnapshot.backendKind + || typeof olderIdentity.runId !== "string" || olderIdentity.runId === source.id + || !/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/i.test(olderIdentity.runId)) return false; + const [prior] = await db.select({ createdAt: heartbeatRuns.createdAt, finishedAt: heartbeatRuns.finishedAt, + runnerInstanceId: heartbeatRuns.runnerInstanceId, snapshot: sql`${heartbeatRuns.runnerProfileJson}->'sessionCheckpoint'` }) + .from(heartbeatRuns).where(and(eq(heartbeatRuns.id, olderIdentity.runId), eq(heartbeatRuns.companyId, selected.companyId), + eq(heartbeatRuns.agentId, selected.agentId), eq(heartbeatRuns.nativeIssueId, selected.issueId), + eq(heartbeatRuns.nativeSessionId, selected.sessionId), eq(heartbeatRuns.runtimeMode, "native"), eq(heartbeatRuns.status, "succeeded"))); + if (!prior?.finishedAt || prior.createdAt >= source.createdAt || prior.finishedAt > source.createdAt + || !source.runnerInstanceId || prior.runnerInstanceId !== source.runnerInstanceId + || nativeSha256(prior.snapshot) !== nativeSha256(redactRunnerApiPayloads(olderSnapshot))) return false; + const [failed] = await db.select({ finishedAt: heartbeatRuns.finishedAt }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, source.id), eq(heartbeatRuns.companyId, selected.companyId), eq(heartbeatRuns.agentId, selected.agentId), + eq(heartbeatRuns.nativeIssueId, selected.issueId), eq(heartbeatRuns.nativeSessionId, selected.sessionId), + eq(heartbeatRuns.runtimeMode, "native"), eq(heartbeatRuns.status, "failed"))); + return Boolean(failed?.finishedAt); +} + async function* retainedSources(db: Db, signal?: AbortSignal) { let after: string | undefined; for (;;) { @@ -165,6 +200,17 @@ export async function migrateNativeControllers(db: Db, input: { writersStopped: continue; } if (result.kind !== "imported" && result.kind !== "unchanged") { + if (result.kind === "missing_evidence" && result.reason === "stale_supervisor_snapshot" + && legacyControllerState !== undefined && supervisor !== undefined + && await hasConfirmedOlderSupervisor(db, selected, source, paths, JSON.parse(supervisor))) { + const storage = input.storage ?? durableNativeHarnessStorage(); + if (storage) { + await archiveFailedLegacyNativeController(db, { ...binding, storage, controller: legacyControllerState, supervisor, + writersStopped: true, signal: input.signal }); + counts.archivedFailed++; + continue; + } + } throw new Error(`native_controller_migration_blocked:${source.id}:${"reason" in result ? result.reason : result.kind}`); } counts[result.kind]++;