From 4185e308659abf0887930e20049ca6e7cbfc0e81 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 7 Nov 2026 11:05:25 -0600 Subject: [PATCH] Recover archived commits from original bare sources (#19855) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Workspace migration must preserve saved files before the web server releases its local volume. > - An archived Git checkout can retain a trusted branch-deletion receipt after its private clone and generated project cache are removed. > - The original repository can still exist at its recorded file URL, but the archive owner did not check that source. > - This pull request permits a narrow, verified original bare-repository source for the exact recorded commit. > - The existing encrypted snapshot and publication owners then preserve the workspace through their normal migration path. ## Linked Issues or Issue Description No public issue exists. Searches for the archived-source error and original bare Git source found no duplicate change. **What happened?** Migration reported `archived_git_source_unavailable` for an archived checkout even though its original bare repository still contained the exact saved commit. The private clone and generated cache were absent. **Expected behavior** The canonical owner can use the recorded original local repository when its full commit identity and source ownership are proven. Uncertain or changed sources remain blocked. **Steps to reproduce** 1. Create an isolated checkout from a local bare repository. 2. Close it through the normal cleanup owner and retain the successful branch-deletion receipt. 3. Remove the private clone and generated cache, while retaining the recorded original repository and full base revision. 4. Inspect the archived workspace for migration. **Paperclip version or commit** Reproduced on `84908d176b2800f4c792c4249c610ec0baebdd28`. **Deployment mode** A source installation with existing local Git workspaces and shared object storage. ## What Changed - Consider the exact configured file-URL repository only after the canonical primary source is definitively absent. - Require a matching source record and successful cleanup receipt, the exact full base/deleted commit, a contained bare SHA-1 repository, bounded standard configuration and alias-free metadata. - Restrict this source path to local Git transport and a minimal environment. Reconstruct the tree only in private migration scratch. - Retain existing source fingerprints, journal identity, encrypted upload/readback and final publication checks. Lock and recheck the configured source and cleanup record before publication. - Add real PostgreSQL/Git coverage for encrypted cold restore, lost-reply retry and refused source or publication changes. ## Verification - The unchanged owner failed the real four-object original-source regression with an ineligible inventory. - The review regressions failed before each repair: four returned-primary retry/publication cases and two actual production metadata/reaper cases. - Final head `8925519310` passed all 40 focused PostgreSQL/Git tests in 56.38 seconds. Four positive formats use canonical/legacy metadata and ordinary/verified-tip cleanup. Each imports an encrypted snapshot, removes the original file URI from reach, then authenticates and cold-restores exact files, executable mode and Git history. - The same file covers source/receipt/writer guards, changed sources before publication, authenticated lost-reply reuse and 18 returned-primary directory/alias/unreadable cases. Refused publication retains returned edits and uploaded ciphertext. - Server TypeScript, module-boundary checks and `git diff --check` passed on this revision. Canonical test wrappers exited and cleaned up. - An earlier focused run also passed 22 existing archived-Git/restore-point controls. `pnpm verify --explain` completed without executing extra suites. Required CI supplies final broad coverage. - Final head passed all 60 CI/review contexts: 56 successful checks and four expected skips. Greptile reviewed this exact head at 5/5, with no open review threads. - One unchanged archive segment-boundary test timed out on both framework attempts in the initial server shard. The one job-only retry passed with unchanged code and timeout. The test and its resolved local import graph match a green control revision. The original failures remain retained; the cause is unresolved. ## Risks - The fallback is intentionally narrow. Original repositories with a different full tip, unsupported configuration, aliases (including an aliased instance root), external object storage or more than 10,000 metadata entries remain blocked. - The first source check and final publication check add bounded local filesystem and Git work to this maintenance path. They add no recurring worker or network Git access. - Original repository files remain intact. Importing a retained workspace does not make its configured local file URL portable for future fresh project work. That separate configuration dependency still requires preservation and qualification before volume removal. - The local regression uses the real encryption and restore owners with a local object-storage backend. It does not claim a hosted backfill or a deployed migration. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository tools and code execution. The exact serving snapshot and context-window limit are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/DATABASE.md | 24 +++ .../remote-workspace-migration-git.ts | 145 +++++++++++-- ...e-original-bare-source.integration.test.ts | 199 ++++++++++++++++++ 3 files changed, 347 insertions(+), 21 deletions(-) create mode 100644 server/src/services/remote-workspace-original-bare-source.integration.test.ts diff --git a/doc/DATABASE.md b/doc/DATABASE.md index fe8df430701..0bc895b9f84 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -126,6 +126,30 @@ Receipt validation adds bounded per-scope database reads to startup eligibility. It does not add a recurring scan or network work. Qualification must measure the selected cohort's startup latency on the serving revision before enrollment. +### Original local Git sources for archived workspaces + +When an archived runtime branch has an exact successful deletion receipt, its +saved commit can be imported from the recorded original file-URL repository if +the canonical private/project cache is definitively absent. A present, corrupt, +foreign or unreadable primary source never falls through to this path. + +This fallback requires the matching company/project source record, a real bare +repository under the same instance root, and the exact full saved base SHA matching +the deleted tip (`baseRefSnapshot.resolvedSha`, with the legacy `baseRefSha` form +accepted when no canonical snapshot exists). Conflicting records fail closed. It accepts self-contained SHA-1 storage with a bounded standard +configuration and at most 10,000 metadata entries. Aliases, special files, +alternates, replacement/graft/shallow state and execution hooks fail closed. +Every retained branch or lightweight tag must still name that full base commit. +Saved retries and final publication recheck that all recorded private and generated +primary sources remain absent. Returned or unreadable primary files block the +original-source path. Git reads use a restricted environment and local-file transport only. + +Materialization uses private migration scratch. Existing encrypted snapshot +readback, journal identity, source reinspection and publication fences still +apply. Original repository bytes and cleanup history remain unchanged. This +imports the retained workspace tree; it does not turn a configured local file +URL into a portable source for future fresh project work. + ### Proactive agent chat messages Migration `0419_slippery_impossible_man.sql` adds an agent-scoped unique constraint diff --git a/server/src/services/remote-workspace-migration-git.ts b/server/src/services/remote-workspace-migration-git.ts index a80623b7ec3..93ca8117cfd 100644 --- a/server/src/services/remote-workspace-migration-git.ts +++ b/server/src/services/remote-workspace-migration-git.ts @@ -18,6 +18,7 @@ import { materializeManagedProjectWorkspace } from "./managed-repository-checkou import { isRuntimeOwnedGitBranch } from "./execution-workspace-branch-ownership.js"; import { physicalPath } from "./remote-workspace-migration-tree.js"; import { formatCommandForDisplay } from "./workspace-command-display.js"; +import { resolvePaperclipInstanceRoot } from "../home-paths.js"; import { recordedIsolatedRemoteSource } from "./isolated-remote-workspace-source.js"; type Workspace = typeof executionWorkspaces.$inferSelect; @@ -30,7 +31,84 @@ export type ArchivedGitMigrationSource = { }; const execute = promisify(execFile); const unavailable = () => new Error("remote_workspace_migration_archived_git_source_unavailable"); -function verifiedReceiptObjectSource(workspace: Workspace, sourcePath: string) { +function originalLocalSource(workspace: Workspace, sourcePath: string) { + return Boolean(workspace.repoUrl && /^file:/i.test(workspace.repoUrl) && localRepositoryPath(workspace.repoUrl) === sourcePath + && !managedSourcePaths(workspace).includes(sourcePath) && recordedIsolatedRemoteSource(workspace) !== sourcePath); +} +function originalRecordedCommit(workspace: Workspace) { + const snapshot = workspace.metadata?.baseRefSnapshot, legacy = workspace.metadata?.baseRefSha; + if (snapshot != null) { + if (typeof snapshot !== "object" || Array.isArray(snapshot)) return null; + const canonical = (snapshot as Record).resolvedSha; + return typeof canonical === "string" && /^[a-f0-9]{40}$/.test(canonical) && (legacy == null || legacy === canonical) ? canonical : null; + } + return typeof legacy === "string" && /^[a-f0-9]{40}$/.test(legacy) ? legacy : null; +} +function originalDeletionReceipt(workspace: Workspace, operation: typeof workspaceOperations.$inferSelect | undefined) { + return Boolean(operation && workspace.closedAt && operation.phase === "worktree_cleanup" && operation.status === "succeeded" + && operation.exitCode === 0 && operation.finishedAt && operation.finishedAt >= operation.startedAt && operation.startedAt >= workspace.closedAt + && operation.companyId === workspace.companyId && operation.executionWorkspaceId === workspace.id + && (!operation.issueId || operation.issueId === workspace.sourceIssueId) + && operation.metadata?.cleanupAction === "branch_delete" + && operation.metadata.workspaceId === workspace.id && operation.metadata.workspacePath === (workspace.providerRef ?? workspace.cwd) + && operation.metadata.branchName === workspace.branchName); +} +async function localGit(cwd: string, args: string[]) { + return (await execute("git", ["--no-optional-locks", "-c", "core.fsmonitor=false", "-c", `core.hooksPath=${os.devNull}`, + "-c", "gc.auto=0", "-c", "maintenance.auto=false", "-c", "protocol.allow=never", "-c", "protocol.file.allow=always", "-C", cwd, ...args], { + timeout: 30_000, maxBuffer: 1024 * 1024, env: { PATH: process.env.PATH, LANG: "C", LC_ALL: "C", GIT_CONFIG_NOSYSTEM: "1", + GIT_CONFIG_GLOBAL: os.devNull, GIT_TERMINAL_PROMPT: "0", GIT_OPTIONAL_LOCKS: "0", GIT_NO_REPLACE_OBJECTS: "1", GIT_NO_LAZY_FETCH: "1" }, + })).stdout.trim(); +} +async function verifiedOriginalLocalSource(workspace: Workspace, sourcePath: string, configuredCwd?: string | null) { + for (const primary of new Set([recordedIsolatedRemoteSource(workspace), ...managedSourcePaths(workspace), configuredCwd].filter((value): value is string => typeof value === "string"))) { + const present = await fs.lstat(primary).catch((error: NodeJS.ErrnoException) => { if (error.code === "ENOENT") return null; throw error; }); + if (present || await physicalPath(primary) !== primary) throw unavailable(); + } + const root = resolvePaperclipInstanceRoot(), relative = path.relative(root, sourcePath), full = originalRecordedCommit(workspace); + if (!originalLocalSource(workspace, sourcePath) || !relative || relative === ".." || relative.startsWith(`..${path.sep}`) + || path.isAbsolute(relative) || path.resolve(sourcePath) !== sourcePath || await fs.realpath(root) !== root + || typeof full !== "string" || !/^[a-f0-9]{40}$/.test(full)) throw unavailable(); + let current = root; + for (const part of relative.split(path.sep)) { + current = path.join(current, part); + const entry = await fs.lstat(current); + if (!entry.isDirectory() || entry.isSymbolicLink()) throw unavailable(); + } + const pending = [sourcePath]; let entries = 0; + while (pending.length) { + const directory = await fs.opendir(pending.pop()!); + for await (const entry of directory) { + if (++entries > 10_000) throw unavailable(); + const target = path.join(directory.path, entry.name), stat = await fs.lstat(target); + if (stat.isSymbolicLink() || !stat.isDirectory() && !stat.isFile()) throw unavailable(); + if (stat.isDirectory()) pending.push(target); + } + } + for (const name of ["objects", "refs"]) { + const entry = await fs.lstat(path.join(sourcePath, name)); + if (!entry.isDirectory() || entry.isSymbolicLink()) throw unavailable(); + } + const config = await fs.lstat(path.join(sourcePath, "config")); + if (!config.isFile() || config.isSymbolicLink() || config.nlink !== 1 || config.size > 64 * 1024) throw unavailable(); + for (const name of ["objects/info/alternates", "objects/info/http-alternates", "refs/replace", "info/grafts", "shallow", "commondir"]) { + const entry = await fs.lstat(path.join(sourcePath, name)).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return null; throw error; + }); + if (entry) throw unavailable(); + } + const keys = (await localGit(sourcePath, ["config", "--local", "--no-includes", "--name-only", "--null", "--list"])).split("\0").filter(Boolean); + if (keys.some(key => !/^(core\.(repositoryformatversion|filemode|bare|logallrefupdates|ignorecase|precomposeunicode)|remote\.origin\.(url|fetch|mirror))$/.test(key)) + || await localGit(sourcePath, ["rev-parse", "--is-bare-repository"]) !== "true" + || await localGit(sourcePath, ["rev-parse", "--path-format=absolute", "--git-common-dir"]) !== sourcePath + || await localGit(sourcePath, ["rev-parse", "--show-object-format"]) !== "sha1" + || await localGit(sourcePath, ["rev-parse", "--verify", `${full}^{commit}`]) !== full) throw unavailable(); + const refs = (await localGit(sourcePath, ["for-each-ref", "--format=%(objectname) %(refname)"])).split("\n").filter(Boolean); + if (!refs.length || refs.length > 64 || refs.some(ref => !ref.startsWith(`${full} refs/heads/`) && !ref.startsWith(`${full} refs/tags/`))) throw unavailable(); + return sourcePath; +} +function verifiedReceiptObjectSource(workspace: Workspace, sourcePath: string, configuredCwd?: string | null) { + if (originalLocalSource(workspace, sourcePath)) return verifiedOriginalLocalSource(workspace, sourcePath, configuredCwd); return managedSourcePaths(workspace).includes(sourcePath) ? verifiedManagedGitObjectSource(workspace, sourcePath) : verifiedGitCacheSource(workspace, sourcePath); } @@ -102,7 +180,7 @@ export async function inspectArchivedGitMigrationSource(db: Pick, const saved = journal?.metadata?.archivedGitSource; const savedPath = saved && typeof saved === "object" && !Array.isArray(saved) && typeof (saved as { sourcePath?: unknown }).sourcePath === "string" ? (saved as { sourcePath: string }).sourcePath : null; - const sourcePath = savedPath ?? (privatePresent && privateSource ? privateSource : projectWorkspace?.cwd ?? managedSourcePaths(workspace)[0]); + let sourcePath = savedPath ?? (privatePresent && privateSource ? privateSource : projectWorkspace?.cwd ?? managedSourcePaths(workspace)[0]); if (!sourcePath || !privatePresent && projectWorkspace?.repoUrl && projectWorkspace.repoUrl !== workspace.repoUrl) throw unavailable(); const [operation] = archived && workspace.closedAt ? await db.select().from(workspaceOperations).where(and( eq(workspaceOperations.companyId, workspace.companyId), eq(workspaceOperations.executionWorkspaceId, workspace.id), @@ -157,9 +235,19 @@ export async function inspectArchivedGitMigrationSource(db: Pick, if (!operation || !receipt?.startsWith(prefix) || !receipt.endsWith(").")) return null; const abbreviation = receipt.slice(prefix.length, -2); if (!/^[a-f0-9]{7,40}$/.test(abbreviation)) throw unavailable(); - await verifiedReceiptObjectSource(workspace, sourcePath); - const commit = await git(sourcePath, ["rev-parse", "--verify", `${abbreviation}^{commit}`]); - if (!/^[a-f0-9]{40}$/.test(commit)) throw unavailable(); + if (!savedPath && managedSourcePaths(workspace).includes(sourcePath)) { + const present = await fs.lstat(sourcePath).catch((error: NodeJS.ErrnoException) => { if (error.code === "ENOENT") return null; throw error; }); + if (!present) { + const original = workspace.repoUrl && /^file:/i.test(workspace.repoUrl) ? localRepositoryPath(workspace.repoUrl) : null; + if (original && projectWorkspace?.repoUrl === workspace.repoUrl && originalDeletionReceipt(workspace, operation) + && await physicalPath(sourcePath) === sourcePath) sourcePath = original; + } + } + if (originalLocalSource(workspace, sourcePath) && (!projectWorkspace || projectWorkspace.repoUrl !== workspace.repoUrl + || !originalDeletionReceipt(workspace, operation))) throw unavailable(); + await verifiedReceiptObjectSource(workspace, sourcePath, projectWorkspace?.cwd); + const commit = await (originalLocalSource(workspace, sourcePath) ? localGit : git)(sourcePath, ["rev-parse", "--verify", `${abbreviation}^{commit}`]); + if (!/^[a-f0-9]{40}$/.test(commit) || originalLocalSource(workspace, sourcePath) && commit !== originalRecordedCommit(workspace)) throw unavailable(); return { commit, sourcePath, cleanupOperationId: operation.id, recipe: null }; } @@ -176,6 +264,18 @@ async function authorizeNestedSource(db: Pick, workspace: Workspac } export async function lockArchivedGitMigrationSources(db: Pick, workspace: Workspace, source: ArchivedGitMigrationSource) { + if (originalLocalSource(workspace, source.sourcePath)) { + const [configured] = await db.select({ repoUrl: projectWorkspaces.repoUrl }).from(projectWorkspaces).where(and( + eq(projectWorkspaces.companyId, workspace.companyId), eq(projectWorkspaces.id, workspace.projectWorkspaceId!), + eq(projectWorkspaces.projectId, workspace.projectId!))).for("share", { noWait: true }); + const [receipt] = await db.select().from(workspaceOperations).where(and(eq(workspaceOperations.companyId, workspace.companyId), + eq(workspaceOperations.executionWorkspaceId, workspace.id), eq(workspaceOperations.id, source.cleanupOperationId!))).for("share", { noWait: true }); + if (!configured || configured.repoUrl !== workspace.repoUrl || !receipt || !originalDeletionReceipt(workspace, receipt) + || source.commit !== originalRecordedCommit(workspace)) throw unavailable(); + const prefix = `Deleted branch ${workspace.branchName} (was `, output = receipt.stdoutExcerpt?.trim(); + if (!output?.startsWith(prefix) || !output.endsWith(").") || !/^[a-f0-9]{7,40}$/.test(output.slice(prefix.length, -2)) + || !source.commit.startsWith(output.slice(prefix.length, -2))) throw unavailable(); + } for (const checkout of [...(source.recipe?.contents?.checkouts ?? [])].sort((a, b) => a.source.id.localeCompare(b.source.id))) { await authorizeNestedSource(db, workspace, checkout.source, true); } @@ -187,7 +287,9 @@ export async function materializeArchivedGitMigrationSource(input: { }) { const { workspace, source } = input, cwd = path.join(input.scratch, "retained-git"); if (!workspace.sourceIssueId || !workspace.repoUrl || !workspace.branchName) throw unavailable(); - const resolveGitAuth = createIssueGitRemoteAuthProvider(input.db, workspace.companyId, workspace.sourceIssueId); + const original = originalLocalSource(workspace, source.sourcePath), runGit = original ? localGit : git; + if (original && (source.recipe || !source.cleanupOperationId || source.retainedBranch || source.commit !== originalRecordedCommit(workspace))) throw unavailable(); + const resolveGitAuth = original ? async () => { throw unavailable(); } : createIssueGitRemoteAuthProvider(input.db, workspace.companyId, workspace.sourceIssueId); const present = await fs.lstat(source.sourcePath).catch((error: NodeJS.ErrnoException) => { if (error.code === "ENOENT") return null; throw error; @@ -199,13 +301,13 @@ export async function materializeArchivedGitMigrationSource(input: { if (!source.recipe && !source.cleanupOperationId && source.retainedBranch !== workspace.branchName) throw unavailable(); const verified = source.recipe ? await verifiedGitCacheSource(workspace, source.sourcePath) : await verifiedReceiptObjectSource(workspace, source.sourcePath); - if (source.retainedBranch && (await git(verified, ["rev-parse", "--verify", `refs/heads/${source.retainedBranch}^{commit}`])) !== source.commit) throw unavailable(); - await fs.mkdir(cwd, { mode: 0o700 }); await git(cwd, ["init", "--quiet"]); + if (source.retainedBranch && (await runGit(verified, ["rev-parse", "--verify", `refs/heads/${source.retainedBranch}^{commit}`])) !== source.commit) throw unavailable(); + await fs.mkdir(cwd, { mode: 0o700 }); await runGit(cwd, ["init", "--quiet"]); if (source.recipe?.restorePoint) await copyVerifiedGitExcludes(verified, cwd).catch(() => { throw unavailable(); }); // Fetch objects, never the current working tree of a shared project root. - const presentCommit = await git(verified, ["cat-file", "-e", `${source.commit}^{commit}`]).then(() => true, () => false); + const presentCommit = await runGit(verified, ["cat-file", "-e", `${source.commit}^{commit}`]).then(() => true, () => false); if (presentCommit) { - await git(cwd, ["fetch", "--no-tags", "--no-recurse-submodules", "--", verified, source.commit]); + await runGit(cwd, ["fetch", "--no-tags", "--no-recurse-submodules", "--", verified, source.commit]); } else if (source.recipe?.restorePoint && !source.recipe.restorePoint.headOnRemote) { // The archived branch can have been pruned from the base while its // encrypted bundle still owns the unpushed commits. Import only that @@ -213,8 +315,8 @@ export async function materializeArchivedGitMigrationSource(input: { // verify and import the saved history below. const { manifest } = await loadGitRestorePoint({ storage: input.storage, workspace, ref: source.recipe.restorePoint }); for (const commit of manifest.prerequisites) { - const local = await git(verified, ["cat-file", "-e", `${commit}^{commit}`]).then(() => true, () => false); - if (local) await git(cwd, ["fetch", "--no-tags", "--no-recurse-submodules", "--", verified, commit]); + const local = await runGit(verified, ["cat-file", "-e", `${commit}^{commit}`]).then(() => true, () => false); + if (local) await runGit(cwd, ["fetch", "--no-tags", "--no-recurse-submodules", "--", verified, commit]); else await ensureSavedGitCommit(cwd, workspace.repoUrl, commit, resolveGitAuth); } } @@ -224,12 +326,13 @@ export async function materializeArchivedGitMigrationSource(input: { } if (source.recipe?.restorePoint) await ensureGitRestorePointCommit({ storage: input.storage, workspace, ref: source.recipe.restorePoint, repositoryCwd: cwd }); - await ensureSavedGitCommit(cwd, workspace.repoUrl, source.commit, resolveGitAuth); - await git(cwd, ["check-ref-format", `refs/heads/${workspace.branchName}`]); - await git(cwd, ["checkout", "--detach", source.commit]); - await git(cwd, ["update-ref", `refs/heads/${workspace.branchName}`, source.commit]); - await git(cwd, ["symbolic-ref", "HEAD", `refs/heads/${workspace.branchName}`]); - await git(cwd, ["config", "remote.origin.url", workspace.repoUrl]); + if (original) await runGit(cwd, ["cat-file", "-e", `${source.commit}^{commit}`]); + else await ensureSavedGitCommit(cwd, workspace.repoUrl, source.commit, resolveGitAuth); + await runGit(cwd, ["check-ref-format", `refs/heads/${workspace.branchName}`]); + await runGit(cwd, ["checkout", "--detach", source.commit]); + await runGit(cwd, ["update-ref", `refs/heads/${workspace.branchName}`, source.commit]); + await runGit(cwd, ["symbolic-ref", "HEAD", `refs/heads/${workspace.branchName}`]); + await runGit(cwd, ["config", "remote.origin.url", workspace.repoUrl]); if (source.recipe?.restorePoint) await applyGitRestorePointToWorktree({ storage: input.storage, workspace, ref: source.recipe.restorePoint, repositoryCwd: cwd, worktreePath: cwd, preserveImportedHistory: true }); const approvedLocalSources = new Set(); @@ -251,9 +354,9 @@ export async function materializeArchivedGitMigrationSource(input: { // the original repository and unstaged files remain untouched. for (const repository of [cwd, ...(source.recipe?.contents?.checkouts ?? []) .map(checkout => path.join(cwd, PROJECT_REPOSITORIES_DIR, checkout.name))]) { - const tree = await git(repository, ["write-tree"]); - await git(repository, ["read-tree", "--empty"]); - await git(repository, ["read-tree", tree]); + const tree = await runGit(repository, ["write-tree"]); + await runGit(repository, ["read-tree", "--empty"]); + await runGit(repository, ["read-tree", tree]); } return cwd; } diff --git a/server/src/services/remote-workspace-original-bare-source.integration.test.ts b/server/src/services/remote-workspace-original-bare-source.integration.test.ts new file mode 100644 index 00000000000..1c816f554d7 --- /dev/null +++ b/server/src/services/remote-workspace-original-bare-source.integration.test.ts @@ -0,0 +1,199 @@ +import { randomUUID, createHash } from "node:crypto"; +import { execFile } from "node:child_process"; +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { pathToFileURL } from "node:url"; +import { promisify } from "node:util"; +import { afterAll, afterEach, beforeAll, beforeEach, expect, it, vi } from "vitest"; +import { eq } from "drizzle-orm"; +import { activityLog, agents, companies, createDb, heartbeatRuns, executionWorkspaceSnapshots, executionWorkspaces, issues, projects, projectWorkspaces, workspaceOperations } from "@paperclipai/db"; +import { readRemoteWorkspaceAuthority } from "@paperclipai/shared"; +import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; +import { createLocalDiskStorageProvider } from "../storage/local-disk-provider.js"; +import { createStorageService } from "../storage/service.js"; +import type { StorageService } from "../storage/types.js"; +import { inspectArchivedGitMigrationSource } from "./remote-workspace-migration-git.js"; +import { inventoryRemoteWorkspaceMigration, migrateRemoteWorkspace } from "./remote-workspace-migration.js"; +import { getWorkspaceSnapshot, loadWorkspaceSnapshotArchive } from "./remote-workspace-snapshot-storage.js"; +import { restoreRemoteWorkspaceArchive } from "./remote-workspace-snapshot-archive.js"; +import { cleanupExecutionWorkspaceArtifacts } from "./workspace-runtime.js"; +import { workspaceOperationService } from "./workspace-operations.js"; +import { isolatedRemoteSourcePath } from "./isolated-remote-workspace-source.js"; +import { managedSourcePaths } from "./git-workspace-restoration.js"; +import { resolvePaperclipInstanceRoot } from "../home-paths.js"; +import { mergeExecutionWorkspaceMetadataForPersistence } from "./heartbeat/workspaces.js"; + +const execute = promisify(execFile); +const git = async (cwd: string, ...args: string[]) => (await execute("git", ["-C", cwd, ...args], { env: { ...process.env, + GIT_CONFIG_NOSYSTEM: "1", GIT_CONFIG_GLOBAL: os.devNull, GIT_AUTHOR_NAME: "Fixture", GIT_COMMITTER_NAME: "Fixture", + GIT_AUTHOR_EMAIL: "fixture@example.test", GIT_COMMITTER_EMAIL: "fixture@example.test" } })).stdout.trim(); +let database: Awaited>, db: ReturnType, root: string, storage: StorageService; +beforeAll(async () => { database = await startEmbeddedPostgresTestDatabase("original-bare-source-"); db = createDb(database.connectionString); }, 90_000); +beforeEach(async () => { root = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "original-bare-source-"))); + vi.stubEnv("PAPERCLIP_HOME", root); vi.stubEnv("PAPERCLIP_INSTANCE_ID", "default"); vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "61".repeat(32)); + storage = createStorageService(createLocalDiskStorageProvider(path.join(root, "objects"))); }); +afterEach(async () => { vi.restoreAllMocks(); vi.unstubAllEnvs(); await db.delete(activityLog); await db.delete(workspaceOperations); + await db.delete(executionWorkspaceSnapshots); await db.delete(heartbeatRuns); await db.delete(issues); await db.delete(agents); await db.delete(executionWorkspaces); await db.delete(projectWorkspaces); + await db.delete(projects); await db.delete(companies); await fs.rm(root, { recursive: true, force: true }); }); +afterAll(async () => { await db?.$client.end({ timeout: 1 }); await database?.cleanup(); }); +async function manifest(base: string, prefix = ""): Promise> { + const entries = []; + for (const name of (await fs.readdir(path.join(base, prefix))).sort()) { + const relative = path.join(prefix, name), file = path.join(base, relative), stat = await fs.lstat(file); + if (stat.isDirectory()) entries.push(...await manifest(base, relative)); + else entries.push({ path: relative, mode: stat.mode & 0o777, sha256: createHash("sha256").update(await fs.readFile(file)).digest("hex") }); + } + return entries; +} +async function fixture(options: { legacy?: boolean; verifiedCleanup?: boolean } = {}) { + const companyId = randomUUID(), projectId = randomUUID(), projectWorkspaceId = randomUUID(), issueId = randomUUID(), workspaceId = randomUUID(); + const seed = path.join(root, "seed"), original = path.join(resolvePaperclipInstanceRoot(), "acceptance", "original", "main.git"); + await fs.mkdir(seed, { recursive: true }); await fs.mkdir(path.dirname(original), { recursive: true }); + await git(seed, "init", "-q", "--initial-branch=main"); + await fs.writeFile(path.join(seed, "README.md"), "preserve exact original bytes\n"); + await fs.writeFile(path.join(seed, "run.sh"), "#!/bin/sh\necho retained\n", { mode: 0o755 }); + await git(seed, "add", "."); await git(seed, "commit", "-qm", "retained original"); + const commit = await git(seed, "rev-parse", "HEAD"); await git(root, "clone", "--bare", "--no-hardlinks", seed, original); + const repoUrl = pathToFileURL(original).href, base = isolatedRemoteSourcePath(companyId, issueId), branchName = "closed-acceptance"; + await fs.mkdir(path.dirname(base), { recursive: true }); await git(root, "clone", "--no-hardlinks", original, base); + await git(base, "remote", "set-url", "origin", repoUrl); + const cwd = path.join(base, ".paperclip", "worktrees", branchName); await git(base, "worktree", "add", "-qb", branchName, cwd); + await db.insert(companies).values({ id: companyId, name: "Original repository", issuePrefix: "BARE" }); + await db.insert(projects).values({ id: projectId, companyId, name: "Original source" }); + await db.insert(projectWorkspaces).values({ id: projectWorkspaceId, companyId, projectId, name: "Recorded source", cwd: null, repoUrl }); + await db.insert(issues).values({ id: issueId, companyId, title: "Closed task", status: "done" }); + const [workspace] = await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, projectId, projectWorkspaceId, sourceIssueId: issueId, + mode: "isolated_workspace", strategyType: "git_worktree", providerType: "git_worktree", name: "Closed checkout", status: "archived", closedAt: new Date(), + cwd, providerRef: cwd, branchName, repoUrl, baseRef: "main", metadata: options.legacy + ? { createdByRuntime: true, gitBranchOwnershipVersion: 1, baseRefSha: commit, lifecycleGeneration: 1, + isolatedRemoteSource: { schema: "paperclip.isolated-remote-source.v1", companyId, issueId, repoUrl } } + : mergeExecutionWorkspaceMetadataForPersistence({ existingMetadata: { lifecycleGeneration: 1, + isolatedRemoteSource: { schema: "paperclip.isolated-remote-source.v1", companyId, issueId, repoUrl } }, + source: "heartbeat", createdByRuntime: true, strategyType: "git_worktree", configSnapshot: null, shouldReuseExisting: false, + baseRef: "main", baseRefSha: commit }) }).returning(); + const cleaned = await cleanupExecutionWorkspaceArtifacts({ workspace: workspace!, projectWorkspace: { cwd: base, cleanupCommand: null }, runCleanupCommands: false, + recorder: workspaceOperationService(db).createRecorder({ companyId, executionWorkspaceId: workspaceId, issueId }), + expectedBranchHeadSha: options.verifiedCleanup ? commit : undefined }); + expect(cleaned.warnings).toEqual([]); await fs.rm(base, { recursive: true }); await fs.rm(seed, { recursive: true }); + expect((await git(original, "rev-list", "--objects", "--no-object-names", "--all")).split("\n")).toHaveLength(4); + return { companyId, projectId, projectWorkspaceId, issueId, workspaceId, original, base, cwd, branchName, repoUrl, commit, workspace: workspace! }; +} +it.each([{ label: "canonical metadata ordinary cleanup" }, { label: "legacy metadata ordinary cleanup", legacy: true }, + { label: "legacy metadata verified cleanup", legacy: true, verifiedCleanup: true }, { label: "canonical metadata verified cleanup", verifiedCleanup: true }]) +("imports the exact deleted commit with $label and cold-restores its encrypted files", async options => { + const f = await fixture(options), before = await manifest(f.original); + expect((await inspectArchivedGitMigrationSource(db, f.workspace))?.commit).toBe(f.commit); + expect((await inventoryRemoteWorkspaceMigration(db, f)).eligible).toBe(true); + await migrateRemoteWorkspace({ db, storage, ...f, apply: true }); + const row = (await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, f.workspaceId)))[0]!; + const authority = readRemoteWorkspaceAuthority(row.metadata)!; + expect(authority.source).toMatchObject({ kind: "git", ref: f.commit }); expect(row.status).toBe("archived"); expect(row.cwd).toBe(`/workspaces/${f.workspaceId}`); expect(row.sourceIssueId).toBe(f.issueId); + const reference = (await getWorkspaceSnapshot(db, { ...f, snapshotId: authority.snapshotId! }))!; + const cipher = await storage.getObject(f.companyId, reference.objectKey), parts: Buffer[] = []; + for await (const part of cipher.stream) parts.push(Buffer.from(part)); + const encrypted = Buffer.concat(parts); + expect(encrypted.subarray(0, 7).toString()).toBe("PCWSE1\n"); expect(encrypted.includes(Buffer.from("preserve exact original bytes"))).toBe(false); + const archivePath = path.join(root, "downloaded.pcws"), targetDir = path.join(root, "cold-restored"); + const parked = path.join(root, "parked-original.git"); await fs.rename(f.original, parked); + try { + expect(await fs.lstat(f.original).catch(() => null)).toBeNull(); + await loadWorkspaceSnapshotArchive({ storage, reference, targetPath: archivePath }); + await restoreRemoteWorkspaceArchive({ archivePath, targetDir, expectedManifestDigest: reference.manifestDigest }); + } finally { await fs.rename(parked, f.original); } + expect(await fs.readFile(path.join(targetDir, "README.md"), "utf8")).toBe("preserve exact original bytes\n"); + expect(await fs.readFile(path.join(targetDir, "run.sh"), "utf8")).toBe("#!/bin/sh\necho retained\n"); + expect((await fs.lstat(path.join(targetDir, "run.sh"))).mode & 0o777).toBe(0o755); + expect(await git(targetDir, "rev-parse", "HEAD")).toBe(f.commit); expect(await git(targetDir, "log", "--format=%H")).toBe(f.commit); + expect(await manifest(f.original)).toEqual(before); expect(await fs.lstat(f.cwd).catch(() => null)).toBeNull(); +}); + +it.each(["wrong-full-tip", "malformed-base-snapshot", "wrong-project-source", "missing-receipt", "wrong-receipt", "wrong-receipt-path", "reopened", "active-writer", "source-alias", "object-alias", "unsafe-config", "outside-instance", "present-primary", "unknown-primary"])("refuses %s without publishing", async kind => { + const f = await fixture(); + if (kind === "wrong-full-tip") await db.update(executionWorkspaces).set({ metadata: { ...f.workspace.metadata, baseRefSha: "f".repeat(40) } }).where(eq(executionWorkspaces.id, f.workspaceId)); + if (kind === "malformed-base-snapshot") await db.update(executionWorkspaces).set({ metadata: { ...f.workspace.metadata, baseRefSnapshot: { resolvedSha: "invalid" }, baseRefSha: f.commit } }).where(eq(executionWorkspaces.id, f.workspaceId)); + if (kind === "wrong-project-source") await db.update(projectWorkspaces).set({ repoUrl: "https://example.invalid/other.git" }).where(eq(projectWorkspaces.id, f.projectWorkspaceId)); + const operations = await db.select().from(workspaceOperations).where(eq(workspaceOperations.executionWorkspaceId, f.workspaceId)); + const deletion = operations.find(row => row.metadata?.cleanupAction === "branch_delete")!; + if (kind === "missing-receipt") await db.delete(workspaceOperations).where(eq(workspaceOperations.id, deletion.id)); + if (kind === "wrong-receipt") await db.update(workspaceOperations).set({ stdoutExcerpt: "Deleted branch other (was 1234567)." }).where(eq(workspaceOperations.id, deletion.id)); + if (kind === "wrong-receipt-path") await db.update(workspaceOperations).set({ metadata: { ...deletion.metadata, workspacePath: f.original } }).where(eq(workspaceOperations.id, deletion.id)); + if (kind === "reopened") await db.update(executionWorkspaces).set({ status: "active", closedAt: null }).where(eq(executionWorkspaces.id, f.workspaceId)); + if (kind === "active-writer") { const [agent] = await db.insert(agents).values({ companyId: f.companyId, name: "Existing writer" }).returning(); + const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: agent!.id, issueId: f.issueId, status: "running" }).returning(); + await db.update(issues).set({ executionRunId: run!.id }).where(eq(issues.id, f.issueId)); } + if (kind === "source-alias") { const original = f.original + ".retained"; await fs.rename(f.original, original); await fs.symlink(original, f.original); } + if (kind === "object-alias") { const objects = path.join(f.original, "objects"), retained = path.join(root, "retained-objects"); await fs.rename(objects, retained); await fs.symlink(retained, objects); } + if (kind === "unsafe-config") await git(f.original, "config", "uploadpack.packObjectsHook", "exit 97"); + if (kind === "outside-instance") { const original = path.join(root, "outside.git"); await fs.rename(f.original, original); const repoUrl = pathToFileURL(original).href; + await db.update(projectWorkspaces).set({ repoUrl }).where(eq(projectWorkspaces.id, f.projectWorkspaceId)); + await db.update(executionWorkspaces).set({ repoUrl, metadata: { ...f.workspace.metadata, isolatedRemoteSource: { schema: "paperclip.isolated-remote-source.v1", companyId: f.companyId, issueId: f.issueId, repoUrl } } }).where(eq(executionWorkspaces.id, f.workspaceId)); } + const primary = managedSourcePaths(f.workspace)[0]!; + if (kind === "present-primary") { await fs.mkdir(path.join(primary, ".git"), { recursive: true }); await fs.writeFile(path.join(primary, "keep.txt"), "unknown primary files"); } + if (kind === "unknown-primary") { const stat = fs.lstat.bind(fs); vi.spyOn(fs, "lstat").mockImplementation(async (...args) => { if (String(args[0]) === primary) throw Object.assign(new Error("not readable"), { code: "EACCES" }); return stat(...args); }); } + const put = vi.spyOn(storage, "putFile"); + expect((await inventoryRemoteWorkspaceMigration(db, f)).eligible).toBe(false); + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow(); + expect(put).not.toHaveBeenCalled(); expect(readRemoteWorkspaceAuthority((await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, f.workspaceId)))[0]!.metadata)).toBeNull(); +}); +it.each(["changed-ref", "changed-source-row", "changed-cleanup"])("preserves the uploaded object without authority when %s changes before publication", async kind => { + const f = await fixture(), put = storage.putFile.bind(storage); let changed = false; + storage.putFile = async input => { const result = await put(input); if (!changed) { changed = true; + if (kind === "changed-ref") { const tree = await git(f.original, "rev-parse", f.commit + "^{tree}"); + const newer = await git(f.original, "commit-tree", tree, "-p", f.commit, "-m", "newer ref"); await git(f.original, "update-ref", "refs/heads/main", newer); } + if (kind === "changed-source-row") await db.update(projectWorkspaces).set({ repoUrl: "https://example.invalid/changed.git" }).where(eq(projectWorkspaces.id, f.projectWorkspaceId)); + if (kind === "changed-cleanup") { const rows = await db.select().from(workspaceOperations).where(eq(workspaceOperations.executionWorkspaceId, f.workspaceId)); + const deletion = rows.find(row => row.metadata?.cleanupAction === "branch_delete")!; await db.update(workspaceOperations).set({ stdoutExcerpt: "Changed receipt" }).where(eq(workspaceOperations.id, deletion.id)); } + } return result; }; + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow(); expect(changed).toBe(true); + const row = (await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, f.workspaceId)))[0]!; expect(readRemoteWorkspaceAuthority(row.metadata)).toBeNull(); + const snapshots = await db.select().from(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.executionWorkspaceId, f.workspaceId)); + expect(snapshots).toHaveLength(1); expect(snapshots[0]!.status).toBe("uploaded"); expect(snapshots[0]!.objectKey).toBeTruthy(); +}); +it("reuses the same authenticated uploaded snapshot after losing the original local-source upload reply", async () => { + const f = await fixture(), put = storage.putFile.bind(storage); let lost = false; + storage.putFile = async input => { const result = await put(input); if (!lost) { lost = true; throw new Error("upload_ack_lost"); } return result; }; + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow("upload_ack_lost"); + const pending = (await db.select().from(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.executionWorkspaceId, f.workspaceId)))[0]!; + expect(pending.status).toBe("pending"); + const result = await migrateRemoteWorkspace({ db, storage, ...f, apply: true }); expect(result.applied).toBe(true); + const snapshots = await db.select().from(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.executionWorkspaceId, f.workspaceId)); + expect(snapshots).toHaveLength(1); expect(snapshots[0]!.id).toBe(pending.id); expect(snapshots[0]!.sourceOperationId).toBe(pending.sourceOperationId); + const reference = (await getWorkspaceSnapshot(db, { ...f, snapshotId: pending.id }))!, archivePath = path.join(root, "retry.pcws"), targetDir = path.join(root, "retry-restored"); + await loadWorkspaceSnapshotArchive({ storage, reference, targetPath: archivePath }); + await restoreRemoteWorkspaceArchive({ archivePath, targetDir, expectedManifestDigest: reference.manifestDigest }); + expect(await git(targetDir, "rev-parse", "HEAD")).toBe(f.commit); + expect(await fs.readFile(path.join(targetDir, "README.md"), "utf8")).toBe("preserve exact original bytes\n"); +}); + +it.each(["private", "generated", "hashed"].flatMap(primary => ["directory", "alias", "unreadable"].flatMap(state => ["retry", "publication"].map(boundary => ({ primary, state, boundary }))))) +("refuses returned primary $primary $state at $boundary without losing either copy", async ({ primary, state, boundary }) => { + const f = await fixture(), put = storage.putFile.bind(storage); + const target = primary === "private" ? f.base : managedSourcePaths(f.workspace)[primary === "generated" ? 0 : 1]!; + const retained = state === "alias" ? path.join(root, "returned-primary") : target; + const restorePrimary = async () => { + await fs.mkdir(path.dirname(retained), { recursive: true }); await git(root, "clone", "--no-hardlinks", f.original, retained); + await fs.writeFile(path.join(retained, "retained-edit.txt"), "returned primary work must survive\n"); + if (state === "alias") { await fs.mkdir(path.dirname(target), { recursive: true }); await fs.symlink(retained, target); } + if (state === "unreadable") { const stat = fs.lstat.bind(fs); vi.spyOn(fs, "lstat").mockImplementation(async (...args) => { + if (String(args[0]) === target) throw Object.assign(new Error("primary unreadable"), { code: "EACCES" }); return stat(...args); + }); } + }; + let uploaded = false, uploadedKey: string | null = null, uploadedSha256: string | null = null; + storage.putFile = async input => { const result = await put(input); uploadedKey = result.objectKey; uploadedSha256 = result.sha256; if (!uploaded) { uploaded = true; + if (boundary === "retry") throw new Error("upload_ack_lost"); await restorePrimary(); + } return result; }; + if (boundary === "retry") { await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow("upload_ack_lost"); await restorePrimary(); } + const snapshotsBefore = await db.select().from(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.executionWorkspaceId, f.workspaceId)); + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow(); + const row = (await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, f.workspaceId)))[0]!; + expect(readRemoteWorkspaceAuthority(row.metadata)).toBeNull(); + expect(await fs.readFile(path.join(retained, "retained-edit.txt"), "utf8")).toBe("returned primary work must survive\n"); + const snapshots = await db.select().from(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.executionWorkspaceId, f.workspaceId)); + expect(snapshots).toHaveLength(1); if (boundary === "retry") expect(snapshots[0]!.id).toBe(snapshotsBefore[0]!.id); + expect(uploadedKey).toBeTruthy(); if (boundary === "retry") expect(snapshots[0]!.objectKey).toBeNull(); else expect(snapshots[0]!.objectKey).toBe(uploadedKey); + const cipher = await storage.getObject(f.companyId, uploadedKey!), hash = createHash("sha256"); let bytes = 0; + for await (const chunk of cipher.stream) { bytes += Buffer.byteLength(chunk); hash.update(chunk); } + expect(bytes).toBeGreaterThan(0); expect(hash.digest("hex")).toBe(uploadedSha256); + expect(await git(f.original, "rev-parse", "HEAD")).toBe(f.commit); +});