From 4608744977cc64da1bd630c10557b96d6925fe4a Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 9 Nov 2026 19:49:00 -0600 Subject: [PATCH] fix(workspaces): preserve recovery for verified archived Git sources (#20436) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Durable workspace migration must retain saved Git files and pending native recovery. > - The archived-Git owner validates saved recipes, exact cleanup receipts, and surviving owned branches. > - Online recovery preservation currently accepts only the recipe variant. > - A valid cleanup receipt or owned branch therefore cannot migrate while an old native reference is still prepared. > - This pull request admits every verified archived-Git variant through the same recovery and finalization locks. > - Tasks stay done and keep their selected workspace, history, and pending provider obligations. ## Linked Issues or Issue Description Refs #20424, #20298, and #20418. The related cleanup work in #19812 does not cover this admission gap. **What happened?** Canonical inspection accepts the exact old commit of an archived Git workspace through a valid branch-deletion receipt or an owned surviving branch. Migration still reports `remote_copyback_unresolved` when a terminal native run retains a prepared reference. A read-only staging inventory reproduced this sole blocker for four such workspaces. Their original Git objects and canonical source proofs already exist. **Expected behavior** Preserve the verified Git baseline and every typed native/provider recovery obligation through the existing migration owner. Keep the task status, selection, history, archive state, and recovery records unchanged. Do not reopen the task or substitute the current default branch. **Steps to reproduce** 1. Save a clean archived Git workspace with an authentic cleanup receipt or a surviving owned branch, without a restore recipe. 2. Retain a terminal prepared native reference and its exact provider allocation and recovery action. 3. Remove the old task worktree. 4. Call `migrateRemoteWorkspace` online. **Paperclip version or commit** Reproduced on merged #20424, `915edcff49bb2cbacfbaee604b41e082e0397109`. **Deployment mode** Self-hosted server with PostgreSQL and configured object storage. ## What Changed - Change three conditions to accept a non-null canonical archived-Git source instead of requiring its recipe field. - Use the same typed recovery inspector, sorted native finalization locks, fresh reinspection, and source-task publication locks for all accepted variants. - Reuse the existing Git source validator, private scratch materializer, encrypted snapshot restore, journal, and recovery markers. No source-proof schema or provider protocol changes. - Extend the existing real fixture for cleanup receipts and retained branches. Verify exact old Git commits and unchanged task/recovery state. Refuse receipt revocation and branch movement during upload. ## Verification - Genuine RED on merged #20424: both new source-kind cases fail with `remote_workspace_migration_blocked:remote_copyback_unresolved,workspace_recovery_unresolved`. - GREEN: nine focused PostgreSQL/Git/encrypted-storage cases pass. They include both new variants, two actual upload-time source mutations, the existing recipe interrupted-journal resume, reference/provider mutations, and a competing native finalization owner. - Command: `pnpm exec vitest run server/src/services/remote-workspace-migration.test.ts -t 'no-recipe archived|unselected archived recipe|archived prepared source|competing native F owner'`. - All nine focused cases, server typecheck layers, module boundaries, and whitespace checks pass on exact published head `03065f97b31846a405a8911c30640ef62374e4a4`, after the clean rebase onto merged #20418. Required CI is now clean on this exact head:56 SUCCESS and4 expected SKIPPED checks, with no pending or failed checks. Fresh Greptile is5/5 with no actionable findings or unresolved threads. - Previous-release state: existing cleanup receipts, owned branches, prepared v3 references, and journal shapes carry forward unchanged. Saved snapshots and source identities retain their existing owners. Unrecognized or changed sources still refuse by the existing named errors. Migration does not declare remote files recovered or provider allocations stopped. - Hosted work in this PR was read-only source inspection and inventory. No staging import, source mutation, lifecycle reopen, or provider call was performed by this PR author. ## Risks The existing archived-Git validator now admits its other verified variants to recovery preservation. Incorrect source proof could publish incomplete authority. This PR keeps its exact commit/receipt/branch checks and full authenticated capture/readback. It also keeps all native ownership, reference, allocation, and publication checks. Missing or changed sources remain blocked, and pending remote data retains its original recovery obligation. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository tools, and code execution. The runtime does not expose a deployment suffix or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- .../remote-workspace-migration.test.ts | 103 +++++++++++++----- .../services/remote-workspace-migration.ts | 6 +- 2 files changed, 81 insertions(+), 28 deletions(-) diff --git a/server/src/services/remote-workspace-migration.test.ts b/server/src/services/remote-workspace-migration.test.ts index c31fad595eb..6ea398b49ae 100644 --- a/server/src/services/remote-workspace-migration.test.ts +++ b/server/src/services/remote-workspace-migration.test.ts @@ -637,28 +637,42 @@ describe.skipIf(!support.supported)("remote workspace migration", () => { branchName, strategyType: "git_worktree", providerType: "git_worktree", metadata: { createdByRuntime: true, gitBranchOwnershipVersion: 1 } }).where(eq(executionWorkspaces.id, f.workspaceId)); return { ...f, cwd, base, upstream, commit, projectId, projectWorkspaceId, repoUrl, branchName }; } - async function archivedPreparedFixture() { + async function archivedPreparedFixture(sourceKind: "recipe" | "cleanup" | "branch" = "recipe") { const base = await gitArchiveFixture(), f = { ...base, ...await failedCopybackFixture(base) }; - await fs.writeFile(path.join(f.cwd, "saved.txt"), "archived staged edit\n"); await git(f.cwd, "add", "saved.txt"); - await fs.writeFile(path.join(f.cwd, "saved.txt"), "archived unstaged edit\n"); - await fs.writeFile(path.join(f.cwd, "untracked.txt"), "archived private file\n"); - const captured = (await captureRestorePoint({ cwd: f.cwd, upstreamCommits: [f.upstream] }))!; - const id = randomUUID(), generation = 1, sealed = await sealRestorePoint(captured.payload, - { companyId: f.companyId, workspaceId: f.workspaceId, restorePointId: id, generation }); - const objectKey = `${f.companyId}/git-restore-points/${f.workspaceId}/${generation}/${id}.pcrp`; - await storage.putFile({ companyId: f.companyId, namespace: "git-restore-points", objectKey, - originalFilename: "saved.pcrp", contentType: "application/octet-stream", body: sealed.ciphertext }); - const restorePoint = { schema: "paperclip.git-restore-point.v1", id, generation, objectKey, byteSize: sealed.ciphertext.length, - sha256: sealed.sha256, key: sealed.key, head: captured.manifest.head, index: captured.manifest.index, - worktree: captured.manifest.worktree, ignored: captured.manifest.ignored, modes: captured.manifest.modes, headOnRemote: false }; - const contents = await captureGitWorkspaceContents(f.cwd, { companyId: f.companyId, workspaceId: f.workspaceId }, []); - await db.update(executionWorkspaces).set({ status: "archived", closedAt: new Date(), metadata: { - createdByRuntime: true, gitBranchOwnershipVersion: 1, gitCacheRestoration: { schema: "paperclip.git-cache-restoration.v2", - companyId: f.companyId, workspaceId: f.workspaceId, sourceIssueId: f.issueId, repoUrl: f.repoUrl, - commit: f.commit, branchName: f.branchName, sourceKind: "managed_project", projectId: f.projectId, - sourcePath: f.base, worktreePath: f.cwd, gitBranchOwnershipVersion: 1, verifiedAt: new Date().toISOString(), restorePoint, contents }, - } }).where(eq(executionWorkspaces.id, f.workspaceId)); - await git(f.base, "worktree", "remove", "--force", f.cwd); await git(f.base, "branch", "-D", f.branchName); + let cleanup: typeof workspaceOperations.$inferSelect | null = null; + if (sourceKind === "recipe") { + await fs.writeFile(path.join(f.cwd, "saved.txt"), "archived staged edit\n"); await git(f.cwd, "add", "saved.txt"); + await fs.writeFile(path.join(f.cwd, "saved.txt"), "archived unstaged edit\n"); + await fs.writeFile(path.join(f.cwd, "untracked.txt"), "archived private file\n"); + const captured = (await captureRestorePoint({ cwd: f.cwd, upstreamCommits: [f.upstream] }))!; + const id = randomUUID(), generation = 1, sealed = await sealRestorePoint(captured.payload, + { companyId: f.companyId, workspaceId: f.workspaceId, restorePointId: id, generation }); + const objectKey = `${f.companyId}/git-restore-points/${f.workspaceId}/${generation}/${id}.pcrp`; + await storage.putFile({ companyId: f.companyId, namespace: "git-restore-points", objectKey, + originalFilename: "saved.pcrp", contentType: "application/octet-stream", body: sealed.ciphertext }); + const restorePoint = { schema: "paperclip.git-restore-point.v1", id, generation, objectKey, byteSize: sealed.ciphertext.length, + sha256: sealed.sha256, key: sealed.key, head: captured.manifest.head, index: captured.manifest.index, + worktree: captured.manifest.worktree, ignored: captured.manifest.ignored, modes: captured.manifest.modes, headOnRemote: false }; + const contents = await captureGitWorkspaceContents(f.cwd, { companyId: f.companyId, workspaceId: f.workspaceId }, []); + await db.update(executionWorkspaces).set({ status: "archived", closedAt: new Date(), metadata: { + createdByRuntime: true, gitBranchOwnershipVersion: 1, gitCacheRestoration: { schema: "paperclip.git-cache-restoration.v2", + companyId: f.companyId, workspaceId: f.workspaceId, sourceIssueId: f.issueId, repoUrl: f.repoUrl, + commit: f.commit, branchName: f.branchName, sourceKind: "managed_project", projectId: f.projectId, + sourcePath: f.base, worktreePath: f.cwd, gitBranchOwnershipVersion: 1, verifiedAt: new Date().toISOString(), restorePoint, contents }, + } }).where(eq(executionWorkspaces.id, f.workspaceId)); + await git(f.base, "worktree", "remove", "--force", f.cwd); await git(f.base, "branch", "-D", f.branchName); + } else { + await db.update(executionWorkspaces).set({ status: "archived", closedAt: new Date() }).where(eq(executionWorkspaces.id, f.workspaceId)); + if (sourceKind === "cleanup") await git(f.base, "merge", "--ff-only", f.branchName); + await git(f.base, "worktree", "remove", f.cwd); + if (sourceKind === "cleanup") { + const receipt = await git(f.base, "branch", "-d", f.branchName); + [cleanup] = await db.insert(workspaceOperations).values({ companyId: f.companyId, executionWorkspaceId: f.workspaceId, + phase: "worktree_cleanup", status: "succeeded", exitCode: 0, command: `git branch -d ${f.branchName}`, + cwd: f.base, stdoutExcerpt: receipt, startedAt: new Date(), finishedAt: new Date(), + metadata: { cleanupAction: "branch_delete", workspaceId: f.workspaceId, workspacePath: f.cwd, branchName: f.branchName } }).returning(); + } + } const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); const [environment] = await db.insert(environments).values({ name: f.runId, driver: "daytona" }).returning(); const [lease] = await db.insert(environmentLeases).values({ id: f.reference.leaseId, companyId: f.companyId, @@ -670,16 +684,55 @@ describe.skipIf(!support.supported)("remote workspace migration", () => { const [hold] = await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: f.issueId, kind: "active_run_watchdog", cause, ownerType: "board", ownerAgentId: null, returnOwnerAgentId: run.agentId, fingerprint: nativeSha256({ runId: f.runId, failureCode: cause }), nextAction: "Preserve retained remote files", evidence: { runId: f.runId } }).returning(); - const [selected] = await db.insert(executionWorkspaces).values({ companyId: f.companyId, sourceIssueId: f.issueId, - name: "Current task workspace", providerType: "local_fs", mode: "isolated_workspace", strategyType: "project_primary", cwd: path.join(root, "current-task") }).returning(); - await db.update(issues).set({ status: "cancelled", executionWorkspaceId: selected.id }).where(eq(issues.id, f.issueId)); + if (sourceKind === "recipe") { + const [selected] = await db.insert(executionWorkspaces).values({ companyId: f.companyId, sourceIssueId: f.issueId, + name: "Current task workspace", providerType: "local_fs", mode: "isolated_workspace", strategyType: "project_primary", cwd: path.join(root, "current-task") }).returning(); + await db.update(issues).set({ status: "cancelled", executionWorkspaceId: selected.id }).where(eq(issues.id, f.issueId)); + } else await db.update(issues).set({ status: "done" }).where(eq(issues.id, f.issueId)); await db.insert(activityLog).values({ companyId: f.companyId, actorType: "user", actorId: "original owner", action: "issue.updated", entityType: "issue", entityId: f.issueId, details: { retained: "original task history" } }); const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId)); const [owner] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); const history = await db.select().from(activityLog).where(eq(activityLog.entityId, f.issueId)); - return { ...f, run, lease, hold, task, owner, history, before: await workspace(f.workspaceId) }; + return { ...f, run, lease, hold, task, owner, history, cleanup, before: await workspace(f.workspaceId) }; } + it.each(["cleanup", "branch"] as const)("preserves a no-recipe archived %s source with prepared recovery", async kind => { + const f = await archivedPreparedFixture(kind); + expect(await migrateRemoteWorkspace({ db, storage, ...f, apply: true })).toMatchObject({ applied: true }); + const row = await workspace(f.workspaceId), authority = readRemoteWorkspaceAuthority(row.metadata)!; + expect(row.status).toBe("archived"); expect(row.closedAt).toEqual(f.before.closedAt); + expect(row.metadata).not.toHaveProperty("gitCacheRestoration"); + expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toEqual(f.task); + expect(await db.select().from(activityLog).where(eq(activityLog.entityId, f.issueId))).toEqual(f.history); + expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0]).toEqual(f.lease); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toEqual(f.owner); + expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.hold.id)))[0]).toEqual(f.hold); + await expect(readMigratedLegacyWorkspaceSource(db, { companyId: f.companyId, runId: f.runId, reference: f.reference })) + .resolves.toMatchObject({ runId: f.runId, reference: f.reference }); + if (f.cleanup) expect((await db.select().from(workspaceOperations).where(eq(workspaceOperations.id, f.cleanup.id)))[0]).toEqual(f.cleanup); + else expect(await git(f.base, "rev-parse", `refs/heads/${f.branchName}`)).toBe(f.commit); + await expect(fs.access(f.cwd)).rejects.toMatchObject({ code: "ENOENT" }); + const ref = (await getWorkspaceSnapshot(db, { ...f, snapshotId: authority.snapshotId! }))!; + const archivePath = path.join(root, "no-recipe.pcws"), targetDir = path.join(root, "restored-no-recipe"); + await loadWorkspaceSnapshotArchive({ storage, reference: ref, targetPath: archivePath }); + await restoreRemoteWorkspaceArchive({ archivePath, targetDir, expectedManifestDigest: ref.manifestDigest }); + expect(await git(targetDir, "rev-parse", "HEAD")).toBe(f.commit); + expect(await fs.readFile(path.join(targetDir, "saved.txt"), "utf8")).toBe("original saved work\n"); + }); + it.each(["cleanup", "branch"] as const)("refuses a no-recipe archived %s source changed during upload", async kind => { + const f = await archivedPreparedFixture(kind), put = storage.putFile.bind(storage); let uploaded = false; + storage.putFile = async input => { const saved = await put(input); uploaded = true; + if (f.cleanup) await db.update(workspaceOperations).set({ status: "failed" }).where(eq(workspaceOperations.id, f.cleanup.id)); + else await git(f.base, "update-ref", `refs/heads/${f.branchName}`, f.upstream); + return saved; }; + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow(); + expect(uploaded).toBe(true); + expect(readRemoteWorkspaceAuthority((await workspace(f.workspaceId)).metadata)).toBeNull(); + expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toEqual(f.task); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]).toEqual(f.run); + expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0]).toEqual(f.lease); + await expect(fs.access(f.cwd)).rejects.toMatchObject({ code: "ENOENT" }); + }); it("preserves an unselected archived recipe and its pending native files without reopening the task", async () => { const f = await archivedPreparedFixture(), put = storage.putFile.bind(storage); let interrupt = true; diff --git a/server/src/services/remote-workspace-migration.ts b/server/src/services/remote-workspace-migration.ts index cf068eebde6..f512dc5552d 100644 --- a/server/src/services/remote-workspace-migration.ts +++ b/server/src/services/remote-workspace-migration.ts @@ -161,7 +161,7 @@ async function inspect(db: ReadDb, input: { companyId: string; workspaceId: stri )); const legacyRecoverySources: LegacyWorkspaceRecoverySource[] = []; const onlineRecoveryAllowed = !input.writersStopped - && Boolean(input.onlineSource || workspace.status === "archived" && archivedGitSource?.recipe); + && Boolean(input.onlineSource || workspace.status === "archived" && archivedGitSource); for (const run of referencedRuns.filter(run => object(run.reference).state === "prepared").sort((a, b) => a.id.localeCompare(b.id))) { if (onlineRecoveryAllowed) { const source = await inspectOnlineLegacyWorkspaceRecoverySource(db, { workspace, runId: run.id }); @@ -416,7 +416,7 @@ type MigrationPlan = { input: MigrationInput; onlineSource?: OnlineWorkspaceSour async function lockMigrationRecovery(lock: MigrationTransaction, plans: MigrationPlan[], cohort?: WorkspaceMigrationCohort) { const online = plans.filter(plan => !plan.input.writersStopped - && (plan.onlineSource || plan.reserved.workspace.status === "archived" && plan.reserved.archivedGitSource?.recipe)); + && (plan.onlineSource || plan.reserved.workspace.status === "archived" && plan.reserved.archivedGitSource)); if (!online.length) return; if (!await lockOnlineLegacyWorkspaceRecoverySources(lock as unknown as Db, { companyId: online[0].input.companyId, sources: online.flatMap(({ reserved, onlineSource }) => [...reserved.legacyRecoverySources, ...(onlineSource?.companions.flatMap(proof => proof.sources) ?? [])]), @@ -548,7 +548,7 @@ async function validateMigrationPublication(tx: MigrationTransaction, input: Mig || journal.issueId !== operation.issueId || journal.cwd !== operation.cwd || canonical(journal.metadata) !== canonical(operation.metadata)) throw new Error("workspace_cohort_journal_changed"); } - await lockMigrationParents(tx, input.companyId, reserved.workspace.sourceIssueId, onlineSource || reserved.archivedGitSource?.recipe ? [ + await lockMigrationParents(tx, input.companyId, reserved.workspace.sourceIssueId, onlineSource || reserved.archivedGitSource ? [ ...reserved.legacyRecoverySources.map(source => source.issueId), ...(onlineSource?.companions.flatMap(proof => proof.sources.map(source => source.issueId)) ?? []), ] : []);