From d5c4f7dd02d408a261cedd9a1926b9900a4aedc4 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 20:08:19 -0600 Subject: [PATCH] fix: preserve shared Git sources during online workspace migration (#19688) ## Thinking Path > - Paperclip preserves the work produced by its agents. > - Remote workspace migration must preserve legacy Git history and local edits. > - Linked checkouts can share administration and objects with other tasks. > - A task lock alone cannot exclude writers of a shared or borrowed store. > - The online source owner must derive and retain the full physical ownership closure. > - This change reuses task, project, lifecycle and finalizer locks for that closure. > - Idle legacy Git work can then migrate while unrelated remote turns continue. ## Linked Issues or Issue Description Refs: #19642. **What happened?** Online migration keeps legacy Git sources blocked because several task checkouts can share a project base or a private Git cache. A read-only source observation found 160 pending Git records, 120 distinct paths and nine common stores. Its largest project cohort contained 126 records and 110 task owners. An ordinary-directory fence cannot preserve these sources by itself. **Expected behavior** Derive the physical source owners from stable Git metadata. Hold their existing source and lifecycle keys through publication. Retain all history and recovery obligations. Reject foreign aliases, unknown stores and changed sources. Let unrelated object-backed work continue. **Steps to reproduce** 1. Create a managed project checkout and linked task worktrees. 2. Keep a settled prepared recovery source on a sibling task. 3. Keep unrelated remote work running. 4. Request an online import of an idle leaf checkout. 5. The existing owner keeps the shared source blocked. **Paperclip version or commit** This branch builds on merged #19642 (`7ac99718a4`) and fresh master `fb3cf1a072`. It adds the Git closure. It does not add a writers-stopped shortcut. **Deployment mode** Hosted native Runner with remote workspace storage. Local qualification uses owned synthetic Git repositories, disposable PostgreSQL databases and the actual archive and storage owners. ## What Changed - Read bounded stable Git pointers, common-directory metadata and local alternates. Derive the complete physical source ownership cohort. Preserve all object stores. - Retain common and borrowed store owners in preparation, lifecycle, runtime, expiration and actual reopen paths. Keep company boundaries and the existing 128-key bound. - Hold exact sibling native recovery proofs under their existing physical finalizer keys. Recheck descriptors, allocations, recovery actions and source identities before capture and publication. Preserve original states and warnings. - Require leaf-first nested imports. Reuse the existing immutable child download, restore and source comparison before parent exclusion. - Bind the self-contained archive owner to the held Git namespaces. Keep its three walks, history normalization, fsck, bytes and digest format. Reject bare, detached, network, unknown or changed Git storage. - Check both foreign path fields. Keep foreign metadata private. Document the proof, bounds, source observation and remaining eligibility limits. ## Verification - Final revision `2934e41c923b93e16508b978c849b747c86be645`: all checks are terminal, with 56 successful and four skipped, zero failures and zero pending. The [required CI aggregate](https://github.com/paperclipai/paperclip/actions/runs/42326464408/job/128266279210) passed. Build, types, all server/Runner/serialized suites, end-to-end aggregate and Snyk passed. [Fresh exact-head Greptile review](https://github.com/paperclipai/paperclip/pull/19688#issuecomment-6699760104) is 5/5 with no open threads. GitHub reports MERGEABLE and CLEAN. - Review then found two correctness gaps: an ordinary managed directory inside an unrelated host repository was blocked, and a Git parent could miss a completed nested operation after its child snapshot became missing or deleted. Three actual-owner regressions reproduced both gaps in 9.70 seconds. The corrected owner passes 12 focused cases in 17.08 seconds. It imports and restores only the ordinary root bytes, preserves the outer repository, rejects both missing/deleted child receipts before a parent journal, and retains valid/changed child coverage and private-cache contention. Its wrapper exited 0 and cleanup completed. The first bootstrap attempt failed before tests ran because the host exhausted its 32 shared-memory segment limit; no shared services were stopped and no pass is claimed for that attempt. The final qualified source/test type graph, module boundaries and diff checks pass. - Receipt scans keep canonical path resolution. A lexical SQL prefix can omit children reached through symlinked ancestors. The observed full metadata closure completed in 1.17 seconds; no measured bottleneck justifies weakening that alias proof. Writer admission still rejects uncertain Git ownership before mutation. Expiration returns its typed skip, and reopen returns its structured failure. Runtime and lifecycle do not bypass an unknown source. No broad error-contract change was added. - Full CI at `720104ea975e` exposed five existing boundary cases across three server shards. Four exact cases reproduced locally as failures in 25.98 seconds. Reopen now preserves its structured failure when source inspection rejects the path, without reconstructing it. Runtime recognizes only the exact regular ignore-only `.git/info/exclude` fragment; migration still rejects that fragment. The original arbitrary-path and file-URL import/delete-host/resume fixtures keep both supported input forms and use the genuine offline setup path after their sole writer finishes. The production mode gate is unchanged. A new real-owner control proves the offline flag still rejects a running legacy database writer. - The final focused run passed all ten cases across four suites in 22.75 seconds, including all five CI failures, invalid-pointer admission, fragment rejection, offline writer exclusion and actual nested reopen lock ordering. Its wrapper exited 0 and cleanup completed. The known invalid-pointer case remains fail-closed; the separate unexpected-I/O diagnostic retains safe error metadata without command text or credentials. Existing Git stderr redaction coverage remains intact. The final metadata reader suite passed 20 cases in 2.68 seconds. The qualified server/test source type graph, module boundaries and diff checks pass. No production assertions or timeout defaults were weakened. Fresh CI and review are required for the repair revision. - A later review found that the ordinary early return still used only one foreign path field. The actual ordinary providerRef-only regression failed as incorrectly eligible. One source-path owner now enumerates both local locators for company/foreign ordinary and Git scans, layout checks, early admission, lifecycle/reopen/runtime and expiration/move/preparation retention. Capture and display keep their original preferred path. The actual owner canary passed 49 cases across two suites in 109.15 seconds. It covers cwd-only/ref-only/different/symlink locators, same-company mismatches, foreign owners, lifecycle ordering, Git/recovery imports and expiration/reopen. Three selected-locator fallback cases also failed before their guard; the final eight-case control passed in 31.05 seconds, including the configured alias and valid import/private-child controls. All wrappers exited 0 and completed cleanup. The metadata/locator suite passed 19 cases. The final source/test type graph, module boundaries and diff checks pass. - Review found two P1 gaps. Both configured-directory symlink cases failed before canonicalization. The actual parent import lost an already-imported child's unique private cache and failed its archive fence instead of retaining that owner. The corrected real-owner canary passed four cases in 20.78 seconds: configured symlink runtime provisioning, private-cache contention before the parent journal, successful parent publication after release, and unchanged/changed child coverage. The metadata suite passed 15 cases in 4.14 seconds. Both wrappers exited 0 and cleanup completed. - A fresh source read-only observation found 160 pending Git records and nine common stores. An additional exact receipt query found 1,243 completed migration receipts and 557 nested source paths under pending Git roots. All 126 records in the largest project were present in the captured Git metadata set. For all 110 present candidate workspace records, the derived closure fits the unchanged 128-key cap. The worst case needs 110 issue keys and one project key, with 17 keys of headroom. Its 173 imported children introduce no physical issue owner outside those 110 logical task owners. This checks the observed key bound only. It does not claim migration eligibility or a coherent snapshot. - The initial stacked CI did not run four Runner test lanes: trusted master passed new --package-lane/--native-server options to the old base lane script. The branch is now rebased onto actual merged master and retargeted before synchronization. It includes the canonical current lane parser without an unrelated Runner patch. Fresh new-head CI is required. - The actual importer/source owner run passed 36 cases across three suites in 187.75 seconds. It covers sibling prepared recovery, unrelated running remote work, common-cache contention, changed coordinator/allocation/path state during upload, exact nested coverage, dirty unpushed work, and existing expiration/reopen owners. Wrapper exit 0 and cleanup completed. - A separate previously hanging lifecycle-worker restore passed in 17.33 seconds with the unchanged 15-second test-body limit. An earlier run timed out in three expiration/reopen cases and blocked cleanup; its wrapper later exited 1. These failures remain recorded. They are not counted as passes. - Two boundary controls first failed: a foreign providerRef-only alias was incorrectly eligible, and actual reopen did not admit its distinct cache owner at the initial source boundary. The narrow guards now inspect both foreign locators and derive physical Git keys before the raw lifecycle transaction. The foreign alias and existing real nested reopen contention passed. The private-cache control first retained contradictory recorded source provenance and stayed fail-closed after release. The corrected present legacy-worktree fixture passed in 12.61 seconds. These failed attempts remain recorded; they are not passes. - The metadata reader and serialized/local archive controls use actual Git repositories. They compare archive bytes and digests, remove every original source, restore staged and unstaged work, and run fsck. The metadata reader passed 13 cases in 2.85 seconds. The local/serialized archive controls passed nine cases in 9.27 seconds. All processes exited 0. An additional actual importer control changed a relative commondir pointer to the absolute name of the same store during upload. Publication was rejected, the uploaded journal remained available, and original files stayed intact (one case, 5.86 seconds; wrapper exit 0 and cleanup complete). - The final rebased full local server/test type graph passes with db/shared/Runner source aliases because reused dependency declarations predate new master exports. The first rebased attempt reported those stale declaration fields/tables, and is not counted as a pass. Module boundaries and diff checks pass. Standard package builds, types and broad regression checks remain required in CI. - No hosted import, admission hold, cancellation, provider mutation, source filesystem write or shared-cache Git command was performed for this change. The source observation is read-only and is not a locked eligibility proof. ## Risks - A missing writer owner could make physical source exclusion incomplete. Existing mutation owners retain the same company/task/project keys, and uncertain metadata fails closed. - A project can contain a large cohort. Limits remain finite. A bound failure keeps the source blocked. - Bare, detached, unowned or network Git stores remain blocked online. Missing archived sources retain their existing authenticated reconstruction path. - Exact recovery proofs preserve authority and obligations. They do not infer provider termination or mark a prepared descriptor finalized. - This change does not claim all observed Git records are immediately eligible. Existing preparation, provider, recovery, runtime and source-change blockers remain. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use and code execution. The host does not expose a more specific backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../2026-11-07-online-git-source-fence.md | 90 ++++++ .../git-workspace-expiration.test.ts | 42 ++- .../execution-workspace-lifecycle-lock.ts | 34 +- .../execution-workspace-source-roots.ts | 4 + server/src/services/execution-workspaces.ts | 20 +- .../src/services/git-workspace-expiration.ts | 12 +- .../heartbeat/workspace-preparation.test.ts | 11 +- server/src/services/heartbeat/workspaces.ts | 3 +- .../services/isolated-repository-rehome.ts | 5 +- .../src/services/online-workspace-source.ts | 181 +++++++++++ .../remote-workspace-migration-tree.ts | 156 ++++++---- .../remote-workspace-migration.test.ts | 11 +- .../services/remote-workspace-migration.ts | 28 +- .../remote-workspace-online-source.test.ts | 292 +++++++++++++++++- .../remote-workspace-snapshot-archive.test.ts | 33 +- .../remote-workspace-snapshot-archive.ts | 48 ++- .../workspace-git-source-roots.test.ts | 143 +++++++++ .../services/workspace-git-source-roots.ts | 115 +++++++ server/src/services/workspace-runtime.ts | 13 +- server/src/services/workspace-source-lock.ts | 44 +-- 20 files changed, 1151 insertions(+), 134 deletions(-) create mode 100644 doc/plans/2026-11-07-online-git-source-fence.md create mode 100644 server/src/services/online-workspace-source.ts create mode 100644 server/src/services/workspace-git-source-roots.test.ts create mode 100644 server/src/services/workspace-git-source-roots.ts diff --git a/doc/plans/2026-11-07-online-git-source-fence.md b/doc/plans/2026-11-07-online-git-source-fence.md new file mode 100644 index 00000000000..c0e7ec93699 --- /dev/null +++ b/doc/plans/2026-11-07-online-git-source-fence.md @@ -0,0 +1,90 @@ +# Online Git source preservation + +The online importer must exclude every writer of a selected checkout, its Git +administration, and all common or borrowed object stores. A checkout's task key +alone does not cover a shared project base or a private cache owned by another +task. Unrelated remote work must continue. + +The source owner reads bounded, stable Git pointer, `commondir`, and alternates +metadata. It maps each physical directory to the existing company/task source +key or managed-project source key. It derives a transitive company cohort from +those owners and declared project sources. It rejects foreign physical aliases, +unknown stores, bare repositories, symlink metadata, network alternates, and +uncertain or excessive metadata. It preserves every object store; HEAD alone is +not a source closure. The bounds are 1,000 company records, 256 cohort records, +128 source keys, 32 borrowed stores, and four alternate levels. Foreign minimal +paths use the existing bounded scan of 256 records per page, at most 40 pages. + +Configured directory aliases resolve through the existing canonical path owner +before Git inspection. Git pointer files and administration/object-store links +retain their strict checks. A configured symlink does not make a direct symlink +migration source admissible. + +The runtime retains the existing plain-task-directory treatment of an exact +regular `.git/info/exclude` fragment. It is not an operational Git repository. +Git pointers and object stores retain their topology checks, and migration +rejects the fragment instead of treating it as an ordinary snapshot. A blocked +reopen returns its existing structured failure without attempting reconstruction. +Arbitrary self-hosted Git paths and file URLs retain the genuine offline +maintenance path; online migration requires a proven physical source owner. + +Physical exclusion enumerates both `cwd` and `providerRef` through one source +path owner. Company/foreign alias checks, early admission, layout checks and +filesystem mutation owners use that set. A managed or Git source in either +locator requires a proven scope before the legacy fallback. Capture and display +keep their existing preferred-path selection. Missing archived sources retain +their existing reconstruction path. + +Preparation, lifecycle, runtime-service, expiration, and reopen owners retain +the same physical keys before filesystem mutation or lifecycle exclusion. The +importer holds all cohort task/project and lifecycle keys through publication. +It also holds the existing per-run physical finalizer keys for each exact native +recovery proof. It re-inspects the complete source proof before capture and +publication. Changed descriptors, allocations, owners, aliases, or Git metadata +reject publication. Prepared sources and unresolved recovery warnings retain +their original state and evidence. They are not marked finalized or cleared. + +Present nested workspaces must import before their parent. The existing owning +nested-import path downloads and restores each immutable child snapshot and +compares it with a fresh source capture before excluding that child. Missing or +changed coverage remains blocked. The archive retains its three source walks, +history normalization, Git/fsck checks, manifest ordering, and byte format. Its +self-contained remote function checks that every Git store belongs to the held +physical source roots. It preserves the original source files. + +Ordinary capture does not read an enclosing host repository outside its root. +Its canonical ordinary proof remains valid there. Any `.git` marker at the +selected root still requires Git admission. A completed nested operation stays +in the parent layout guard unless the derived child proof covers that exact +operation and physical path. Missing or revoked snapshot coverage cannot turn +the leftover child into an ordinary parent directory. + +Receipt path scans retain canonical resolution for aliases. A lexical SQL prefix +could hide a child reached through a symlinked ancestor. The finite receipt bound +remains in force; query optimization must preserve that alias coverage. + +Imported child paths come from the same exact completed migration/snapshot +selector used by nested source verification. Their physical Git owners remain +in the parent fence after their database rows become remote. The selector bounds +company migration receipts to 10,240 and binds each child path, operation, +snapshot, digest and Git metadata proof. Current remote child execution does not +reacquire authority over those historical host files. + +A source metadata observation at `2026-11-06T23:15:08.979Z` examined 160 pending +Git records and 120 unique paths. It found 92 Git pointers, 49 Git directories, +19 absent archived roots, and nine common/object stores. No alternate or uncertain +metadata was observed. Project cohort sizes were 126, 27, 4, 1, 1, and 1; the +largest contained 110 distinct task owners. These observations bound useful +cohorts. They are not a coherent snapshot, acquired lock proof, or claim that all +160 are eligible. Missing archived sources keep their existing authenticated +reconstruction path. Remaining provider, recovery, preparation, runtime, and +source-change blockers remain authoritative. + +Focused verification uses the real importer and local storage owner. It covers +prepared sibling recovery, unrelated running remote work, private cache ownership, +source changes during upload, leaf-first exact child coverage, and existing Git +expiration/reopen consumers. Archive tests exercise both local calls and the +serialized remote function, compare archive bytes and digests, remove every +original source, restore uncommitted/staged work, and run fsck. No hosted import, +hold, cancellation, provider call, or shared-cache Git command is part of these +local tests. diff --git a/server/src/__tests__/git-workspace-expiration.test.ts b/server/src/__tests__/git-workspace-expiration.test.ts index 9ab0acd7429..ee42fda684d 100644 --- a/server/src/__tests__/git-workspace-expiration.test.ts +++ b/server/src/__tests__/git-workspace-expiration.test.ts @@ -20,7 +20,7 @@ import { materializeIsolatedRemoteWorkspaceSource } from "../services/isolated-r import { realizeExecutionWorkspace, ensurePersistedExecutionWorkspaceAvailable } from "../services/workspace-runtime.js"; import { executionWorkspaceService } from "../services/execution-workspaces.js"; import { withExecutionWorkspaceLifecycleLock } from "../services/execution-workspace-lifecycle-lock.js"; -import { withWorkspaceSourceLocks } from "../services/workspace-source-lock.js"; +import { IsolatedRepositorySourceBusy, withWorkspaceSourceLocks } from "../services/workspace-source-lock.js"; import { prepareProjectRepositoryWorkspaces, mergeExecutionWorkspaceMetadataForPersistence } from "../services/heartbeat/workspaces.js"; import { describeGitExpirationFailure, expireReconstructableGitWorkspace } from "../services/git-workspace-expiration.js"; import { nestedWorktreeBranchPrefix, restoreNestedRepositoryWorktree, withNestedRepositoryBaseLock } from "../services/nested-repository-worktrees.js"; @@ -220,6 +220,30 @@ describeEmbeddedPostgres("reconstructable Git expiration", () => { expect((await fs.stat(f.nestedSettingsPath)).mode & 0o777).toBe(0o640); }); + it("retains a distinct private Git cache before the actual reopen lifecycle", async () => { + const f = await fixture(), sourceIssueId = randomUUID(); + await db.insert(issues).values({ id: sourceIssueId, companyId: f.companyId, title: "Reopen consumer" }); + const metadata = { ...(await f.row()).metadata }; + delete metadata.isolatedRemoteSource; + await db.update(executionWorkspaces).set({ sourceIssueId, metadata }).where(eq(executionWorkspaces.id, f.workspaceId)); + let entered!: () => void, release!: () => void; + const started = new Promise(resolve => { entered = resolve; }); + const pending = new Promise(resolve => { release = resolve; }); + const owner = withWorkspaceSourceLocks(db, { companyId: f.companyId, issueIds: [f.issueId] }, async () => { + entered(); await pending; + }); + await started; + try { + await expect(f.reopen()).rejects.toBeInstanceOf(IsolatedRepositorySourceBusy); + const key = `execution_workspace_lifecycle:${f.workspaceId}`; + const locks = await db.execute(sql`select count(*)::int n from pg_locks where locktype='advisory' and granted + and classid=((hashtextextended(${key},0)>>32)&4294967295)::oid and objid=(hashtextextended(${key},0)&4294967295)::oid`); + expect(locks[0].n).toBe(0); + } finally { release(); await owner; } + expect(await f.reopen()).toMatchObject({ ok: true, reopened: false }); + expect(git(f.cwd, "rev-parse", "HEAD")).toBe(f.commitA); + }); + it("takes the project fence before the lifecycle lock during an actual nested reopen", async () => { const f = await nestedFixture(true, true); expect(await f.expire({ expireBase: false })).toMatchObject({ expired: true, baseExpired: false }); @@ -897,10 +921,22 @@ describeEmbeddedPostgres("reconstructable Git expiration", () => { it("describes an unexpected inspection failure without the command or credentials", async () => { const f = await fixture(); - await fs.writeFile(path.join(f.cwd, ".git"), `gitdir: ${path.join(f.cwd, "missing-admin")}\n`); + const failure = Object.assign(new Error("inspection command token=sk-secret-fixture"), { code: "EACCES" }); + vi.spyOn(fs, "open").mockRejectedValueOnce(failure); const result = await f.expire(); expect(result).toMatchObject({ expired: false, skipReason: "inspection_failed", - failure: { name: "Error", code: expect.any(String), at: expect.stringMatching(/^at /), stderr: expect.stringMatching(/not a git repository|gitfile does not point to a valid repository/) } }); + failure: { name: "Error", code: "EACCES", at: expect.stringMatching(/^at /), stderr: null } }); + expect(JSON.stringify(result)).not.toContain("sk-secret-fixture"); + expect(JSON.stringify(result)).not.toContain("inspection command"); + expect(await exists(f.cwd)).toBe(true); + }); + + it("retains an invalid Git pointer rejected by source admission without exposing its command", async () => { + const f = await fixture(); + await fs.writeFile(path.join(f.cwd, ".git"), `gitdir: ${path.join(f.cwd, "missing-admin")}\n`); + const result = await f.expire(); + expect(result).toMatchObject({ expired: false, skipReason: "workspace_git_source_unproven" }); + expect(result.failure).toBeUndefined(); expect(await exists(f.cwd)).toBe(true); }); diff --git a/server/src/services/execution-workspace-lifecycle-lock.ts b/server/src/services/execution-workspace-lifecycle-lock.ts index 2f324d257ff..8063a9d5bcf 100644 --- a/server/src/services/execution-workspace-lifecycle-lock.ts +++ b/server/src/services/execution-workspace-lifecycle-lock.ts @@ -1,6 +1,8 @@ import { executionWorkspaces, withDedicatedDbConnection, type Db } from "@paperclipai/db"; import { eq, sql } from "drizzle-orm"; -import { isolatedTaskSourceIdentity, managedProjectSourceIdentity, withWorkspaceSourceLocks } from "./workspace-source-lock.js"; +import { isDeepStrictEqual } from "node:util"; +import { workspaceSourceIdentities, withWorkspaceSourceLocks } from "./workspace-source-lock.js"; +import { localWorkspaceSourcePaths } from "./execution-workspace-source-roots.js"; import { withWorkspaceLockKeepalive } from "./native-runtime/native-workspace-lock-keepalive.js"; type Transaction = Parameters[0]>[0]; @@ -20,23 +22,25 @@ export async function withExecutionWorkspaceLifecycleLock( work: (tx: Transaction) => Promise, ): Promise { const [source] = await db.select({ companyId: executionWorkspaces.companyId, projectId: executionWorkspaces.projectId, - providerType: executionWorkspaces.providerType, cwd: executionWorkspaces.cwd, providerRef: executionWorkspaces.providerRef }).from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId)); - const projectId = source && (source.cwd ?? source.providerRef) - ? await managedProjectSourceIdentity(source.companyId, (source.cwd ?? source.providerRef)!) : null; - const issueId = source && (source.cwd ?? source.providerRef) - ? await isolatedTaskSourceIdentity(source.companyId, (source.cwd ?? source.providerRef)!) : null; - const projectIds = [...new Set([projectId, source && ["local_fs", "git_worktree"].includes(source.providerType) - ? source.projectId : null].filter((id): id is string => Boolean(id)))]; + providerType: executionWorkspaces.providerType, sourceIssueId: executionWorkspaces.sourceIssueId, cwd: executionWorkspaces.cwd, providerRef: executionWorkspaces.providerRef }).from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId)); + const sourceRoots = source ? await Promise.all(localWorkspaceSourcePaths(source) + .map(cwd => workspaceSourceIdentities(source.companyId, cwd))) : []; + const projectIds = [...new Set([...sourceRoots.flatMap(root => root.projectIds), + source && ["local_fs", "git_worktree"].includes(source.providerType) ? source.projectId : null, + ].filter((id): id is string => Boolean(id)))].sort(); + const issueIds = [...new Set([...sourceRoots.flatMap(root => root.issueIds), + source && ["local_fs", "git_worktree"].includes(source.providerType) ? source.sourceIssueId : null, + ].filter((id): id is string => Boolean(id)))].sort(); const workWithLifecycle = () => withDedicatedDbConnection(db, (connection) => connection.transaction(async (tx) => { await acquireExecutionWorkspaceLifecycleLock(tx, workspaceId); - if ((projectIds.length || issueId) && source) { + if ((projectIds.length || issueIds.length) && source) { const [fresh] = await tx.select({ companyId: executionWorkspaces.companyId, projectId: executionWorkspaces.projectId, - providerType: executionWorkspaces.providerType, cwd: executionWorkspaces.cwd, providerRef: executionWorkspaces.providerRef }).from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId)); + providerType: executionWorkspaces.providerType, sourceIssueId: executionWorkspaces.sourceIssueId, cwd: executionWorkspaces.cwd, providerRef: executionWorkspaces.providerRef }).from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId)); + const freshRoots = fresh ? await Promise.all(localWorkspaceSourcePaths(fresh) + .map(cwd => workspaceSourceIdentities(fresh.companyId, cwd))) : []; if (!fresh || fresh.companyId !== source.companyId || fresh.projectId !== source.projectId - || fresh.providerType !== source.providerType || !(fresh.cwd ?? fresh.providerRef) - || await managedProjectSourceIdentity(fresh.companyId, (fresh.cwd ?? fresh.providerRef)!) !== projectId - || await isolatedTaskSourceIdentity(fresh.companyId, (fresh.cwd ?? fresh.providerRef)!) !== issueId) { + || fresh.providerType !== source.providerType || fresh.sourceIssueId !== source.sourceIssueId || !isDeepStrictEqual(freshRoots, sourceRoots)) { throw new Error("workspace_source_scope_changed"); } } @@ -50,8 +54,8 @@ export async function withExecutionWorkspaceLifecycleLock( ); }), ); - return (projectIds.length || issueId) && source - ? withWorkspaceSourceLocks(db, { companyId: source.companyId, projectIds, issueIds: issueId ? [issueId] : [] }, workWithLifecycle) + return (projectIds.length || issueIds.length) && source + ? withWorkspaceSourceLocks(db, { companyId: source.companyId, projectIds, issueIds }, workWithLifecycle) : workWithLifecycle(); } diff --git a/server/src/services/execution-workspace-source-roots.ts b/server/src/services/execution-workspace-source-roots.ts index f038e5f66ac..2ee6737cb9e 100644 --- a/server/src/services/execution-workspace-source-roots.ts +++ b/server/src/services/execution-workspace-source-roots.ts @@ -1,6 +1,10 @@ import fs from "node:fs/promises"; import path from "node:path"; +export function localWorkspaceSourcePaths(workspace: { cwd?: string | null; providerRef?: string | null }): string[] { + return [...new Set([workspace.cwd, workspace.providerRef].filter((value): value is string => typeof value === "string" && value.length > 0))]; +} + export async function canonicalRepositoryRoot(cwd: string): Promise { const absolute = path.resolve(cwd); try { return await fs.realpath(absolute); } diff --git a/server/src/services/execution-workspaces.ts b/server/src/services/execution-workspaces.ts index 69f9c6dd163..ab9462e811e 100644 --- a/server/src/services/execution-workspaces.ts +++ b/server/src/services/execution-workspaces.ts @@ -8,7 +8,8 @@ import { measureRunTransaction } from "./execution-transaction-timing.js"; import { measureRunPhase } from "./execution-preparation-timing.js"; import { assertExecutionWorkspaceRecordAccess, assertTaskWorkspaceExecutionAccess, assertTaskWorkspaceSourceProjectAccess } from "./task-workspace-access.js"; import { verifiedRecordedIsolatedRemoteSource, withIsolatedRepositorySourceLock } from "./isolated-remote-workspace-source.js"; -import { managedProjectSourceIdentity, retainCurrentWorkspaceSource, withWorkspaceSourceLocks } from "./workspace-source-lock.js"; +import { workspaceSourceIdentities, retainCurrentWorkspaceSource, withWorkspaceSourceLocks } from "./workspace-source-lock.js"; +import { localWorkspaceSourcePaths } from "./execution-workspace-source-roots.js"; import { expireReconstructableGitWorkspace, gitWorkspaceLastActivityAt, hasPendingGitExpiration, listIssuesUsingGitWorkspace } from "./git-workspace-expiration.js"; import { expireIsolatedTaskDirectory, hasPendingTaskDirectoryRetention, isIsolatedTaskDirectoryWorkspace, isRetryableTaskDirectorySkip, mayBeIsolatedTaskDirectoryWorkspace, readTaskDirectoryRetention, restoreIsolatedTaskDirectory, supersedeTaskDirectoryRestorePoint, @@ -4774,14 +4775,19 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic return result; }; if (freshSnapshot.providerType === "plugin") return reopenSource(); - const projectIds = new Set(); + const projectIds = new Set(), issueIds = new Set(); if (freshSnapshot.projectId) projectIds.add(freshSnapshot.projectId); - for (const cwd of new Set([freshSnapshot.cwd, freshSnapshot.providerRef])) { - if (!cwd) continue; - const projectId = await managedProjectSourceIdentity(freshSnapshot.companyId, cwd); - if (projectId) projectIds.add(projectId); + try { + for (const cwd of localWorkspaceSourcePaths(freshSnapshot)) { + const identities = await workspaceSourceIdentities(freshSnapshot.companyId, cwd); + for (const id of identities.projectIds) projectIds.add(id); + for (const id of identities.issueIds) issueIds.add(id); + } + } catch { + return { ok: false, code: "rebuild_failed", message: freshSnapshot.metadata?.gitCacheRestoration + ? new GitCacheRestorationFailure().message : "Failed to rebuild the execution workspace" }; } - return withWorkspaceSourceLocks(db, { companyId: freshSnapshot.companyId, projectIds: [...projectIds] }, reopenSource); + return withWorkspaceSourceLocks(db, { companyId: freshSnapshot.companyId, projectIds: [...projectIds], issueIds: [...issueIds] }, reopenSource); }; return authorizedSnapshot.sourceIssueId ? withIsolatedRepositorySourceLock(db, authorizedSnapshot.companyId, authorizedSnapshot.sourceIssueId, reopen) diff --git a/server/src/services/git-workspace-expiration.ts b/server/src/services/git-workspace-expiration.ts index 8dcf1ac7d6a..0b9157fa4d6 100644 --- a/server/src/services/git-workspace-expiration.ts +++ b/server/src/services/git-workspace-expiration.ts @@ -12,6 +12,8 @@ import { TASK_DIRECTORY_RETENTION_KEY } from "./isolated-task-directory-snapshot import { currentRestorePointSnapshot, RestorePointSkip, type RestorePointLimits } from "./git-restore-point.js"; import { abandonGitRestorePoint, abandonGitRestorePointsByKey, activateGitRestorePoints, restorePointHistoryUnchanged, restorePointSnapshotMatches, storeGitRestorePoint } from "./git-restore-point-storage.js"; import { withIsolatedRepositorySourceLock, verifiedRecordedIsolatedRemoteSource } from "./isolated-remote-workspace-source.js"; +import { retainCurrentWorkspaceSource } from "./workspace-source-lock.js"; +import { localWorkspaceSourcePaths } from "./execution-workspace-source-roots.js"; import { managedProjectCheckoutName } from "./managed-project-checkout-name.js"; import { nestedRepositoryBaseCandidates } from "./nested-repository-worktrees.js"; import type { GitRemoteAuthProvider } from "./git-credentials.js"; @@ -292,8 +294,11 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Worksp let pendingRestorePoints: string[] = []; if (!snapshot.sourceIssueId) return { ...result, skipReason: "missing_source_issue" }; try { - return await withIsolatedRepositorySourceLock(db, snapshot.companyId, snapshot.sourceIssueId, () => - withDedicatedDbConnection(db, connection => connection.transaction(async tx => { + return await withIsolatedRepositorySourceLock(db, snapshot.companyId, snapshot.sourceIssueId, async () => { + for (const cwd of localWorkspaceSourcePaths(snapshot)) await retainCurrentWorkspaceSource(cwd); + const retainedSource = readGitCacheRestoration(snapshot)?.sourcePath; + if (retainedSource) await retainCurrentWorkspaceSource(retainedSource); + return withDedicatedDbConnection(db, connection => connection.transaction(async tx => { const acquired = await tx.execute<{ locked: boolean }>(sql`select pg_try_advisory_xact_lock(hashtextextended(${`execution_workspace_lifecycle:${snapshot.id}`}, 0)) as locked`); if (!acquired[0]?.locked) throw new Error("lifecycle_busy"); let workspace = (await db.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, snapshot.id), eq(executionWorkspaces.companyId, snapshot.companyId))))[0]; @@ -487,7 +492,8 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Worksp state.completed = recipe.sourceKind !== "isolated_remote" || !options.expireBase || (result.baseExpired || !await exists(recipe.sourcePath)) && (!state.baseTrash || !await exists(state.baseTrash)); await persist(); return result; - })), true); + })); + }, true); } catch (error) { for (const ledgerId of pendingRestorePoints) await abandonGitRestorePoint(db, ledgerId).catch(() => undefined); options.signal?.throwIfAborted(); diff --git a/server/src/services/heartbeat/workspace-preparation.test.ts b/server/src/services/heartbeat/workspace-preparation.test.ts index cc2eac60aab..e68eead6fc6 100644 --- a/server/src/services/heartbeat/workspace-preparation.test.ts +++ b/server/src/services/heartbeat/workspace-preparation.test.ts @@ -662,8 +662,15 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace providerType: "local_fs", strategyType: "project_primary", mode: "shared_workspace", status: "active" }).returning(); await db.update(issues).set({ executionWorkspaceId: original.id }).where(eq(issues.id, issue.id)); const storage = createStorageService(createLocalDiskStorageProvider(join(home, `import-objects-${randomUUID()}`))); - await expect(migrateRemoteWorkspace({ db, storage, companyId: issue.companyId, workspaceId: original.id, apply: true })) - .resolves.toMatchObject({ applied: true }); + const migrationBefore = process.env.PAPERCLIP_EPHEMERAL_MIGRATION; + process.env.PAPERCLIP_EPHEMERAL_MIGRATION = "true"; + try { + await expect(migrateRemoteWorkspace({ db, storage, companyId: issue.companyId, workspaceId: original.id, + apply: true, writersStopped: true })).resolves.toMatchObject({ applied: true }); + } finally { + if (migrationBefore === undefined) delete process.env.PAPERCLIP_EPHEMERAL_MIGRATION; + else process.env.PAPERCLIP_EPHEMERAL_MIGRATION = migrationBefore; + } const [imported] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, original.id)); const saved = readRemoteWorkspaceAuthority(imported.metadata)!; expect(saved).toMatchObject({ source: { kind: "git", repoUrl, ref: "main", branchName: "main" }, diff --git a/server/src/services/heartbeat/workspaces.ts b/server/src/services/heartbeat/workspaces.ts index 49a1e3e1f3f..efccc4f145c 100644 --- a/server/src/services/heartbeat/workspaces.ts +++ b/server/src/services/heartbeat/workspaces.ts @@ -1,4 +1,5 @@ import { retainCurrentWorkspaceSource } from "../workspace-source-lock.js"; +import { localWorkspaceSourcePaths } from "../execution-workspace-source-roots.js"; import { effectiveProjectWorkspacePolicy } from "../project-workspace-provider.js"; import { canonicalRepositoryRoot } from "../execution-workspace-source-roots.js"; import { assertTaskWorkspaceExecutionAccess } from "../task-workspace-access.js"; @@ -3401,7 +3402,7 @@ export function createHeartbeatWorkspaceResolver(db: Db) { managedBase && workspace.repoUrl ? `${managedBase}-${createHash("sha256").update(workspace.repoUrl).digest("hex").slice(0, 12)}` : null, ].filter((value): value is string => Boolean(value)); - await retainCurrentWorkspaceSource(workspace.cwd ?? workspace.providerRef ?? "./"); + for (const sourceCwd of localWorkspaceSourcePaths(workspace)) await retainCurrentWorkspaceSource(sourceCwd); const cwd = await resolvePersistedGitWorkspaceSource({ companyId: agent.companyId, projectId: sourceProjectId, diff --git a/server/src/services/isolated-repository-rehome.ts b/server/src/services/isolated-repository-rehome.ts index c11ba8e0b60..8ab31be0da0 100644 --- a/server/src/services/isolated-repository-rehome.ts +++ b/server/src/services/isolated-repository-rehome.ts @@ -19,7 +19,7 @@ import { applyRestorePoint, captureRestorePoint, currentRestorePointSnapshot, im import { findRegenerableWorkspaceDirs, REGENERABLE_WORKSPACE_DIR_NAMES, WORKSPACE_COMPACTION_TRASH_DIR } from "./workspace-compaction.js"; import { ensureManagedRepositoriesIgnored } from "./managed-repository-checkout.js"; import { withWorkspaceLockKeepalive } from "./native-runtime/native-workspace-lock-keepalive.js"; -import { canonicalRepositoryRoot } from "./execution-workspace-source-roots.js"; +import { canonicalRepositoryRoot, localWorkspaceSourcePaths } from "./execution-workspace-source-roots.js"; import { withExecutionWorkspaceLifecycleLock } from "./execution-workspace-lifecycle-lock.js"; import { loadGitRestorePoint } from "./git-restore-point-storage.js"; import { nestedWorktreeBranchPrefix, readLinkedNestedCheckout, repairMovedNestedRepositoryWorktree, withNestedRepositoryBaseLock } from "./nested-repository-worktrees.js"; @@ -355,8 +355,7 @@ async function undoMove(db: Db | Tx, workspaceId: string, pending: IsolatedRepos export function withWorkspaceMoveLocks(db: Db, workspace: Workspace, work: (tx: Tx) => Promise) { return withIsolatedRepositorySourceLock(db, workspace.companyId, workspace.sourceIssueId!, () => withWorkspaceSourceLocks(db, { companyId: workspace.companyId, projectIds: workspace.projectId ? [workspace.projectId] : [] }, async () => { - const cwd = workspace.cwd ?? workspace.providerRef; - if (cwd) await retainCurrentWorkspaceSource(cwd); + for (const cwd of localWorkspaceSourcePaths(workspace)) await retainCurrentWorkspaceSource(cwd); return withDedicatedDbConnection(db, connection => connection.transaction(async tx => { const rows = await tx.execute<{ locked: boolean; idle_timeout_ms: string | number | null }>(sql`select pg_try_advisory_xact_lock(hashtextextended(${`execution_workspace_lifecycle:${workspace.id}`}, 0)) as locked, diff --git a/server/src/services/online-workspace-source.ts b/server/src/services/online-workspace-source.ts new file mode 100644 index 00000000000..d7fbb38bbf5 --- /dev/null +++ b/server/src/services/online-workspace-source.ts @@ -0,0 +1,181 @@ +import fs from "node:fs/promises"; +import path from "node:path"; +import { and, eq, gt, inArray, ne, sql } from "drizzle-orm"; +import { executionWorkspaces, heartbeatRuns, issueRecoveryActions, type Db } from "@paperclipai/db"; +import { resolvePaperclipInstanceRoot } from "../home-paths.js"; +import { inspectImportedWorkspaceSourcePaths, inspectOnlineOrdinarySource, physicalPath, requiresOnlineOrdinarySource, type WorkspaceSourceScope } from "./remote-workspace-migration-tree.js"; +import { workspaceSourceIdentities } from "./workspace-source-lock.js"; +import { readWorkspaceGitSourceRoots } from "./workspace-git-source-roots.js"; +import { inspectOnlineLegacyWorkspaceRecoveryHold, inspectOnlineLegacyWorkspaceRecoverySource, type OnlineLegacyWorkspaceRecoveryHold } from "./online-legacy-workspace-recovery.js"; +import type { LegacyWorkspaceRecoverySource } from "./legacy-workspace-recovery-migration.js"; +import { hasPendingGitExpiration } from "./git-workspace-expiration.js"; +import { readGitCacheRestoration } from "./git-workspace-restoration.js"; +import { localWorkspaceSourcePaths } from "./execution-workspace-source-roots.js"; + +type Workspace = typeof executionWorkspaces.$inferSelect; +type ReadDb = Pick; +type Identities = Awaited>; +export type OnlineWorkspaceSource = WorkspaceSourceScope & { + kind: "ordinary" | "git"; projectIds: string[]; + gitSourceFingerprints?: Array<{ workspaceId: string; fingerprint: string | null }>; + importedChildren?: Array>[number] & { gitFingerprint: string | null }>; + companions: Array<{ workspaceId: string; sources: LegacyWorkspaceRecoverySource[]; + holds: Array }>; +}; +const intersects = (values: string[], keys: Set) => values.some(value => keys.has(value)); + +export async function requiresOnlineWorkspaceSource(workspace: Workspace): Promise { + if (await requiresOnlineOrdinarySource(workspace)) return true; + if (!["local_fs", "git_worktree"].includes(workspace.providerType)) return false; + for (const cwd of localWorkspaceSourcePaths(workspace)) { + if (!path.isAbsolute(cwd)) continue; + const root = await physicalPath(cwd); + const entry = await fs.lstat(root).catch((error: NodeJS.ErrnoException) => { if (error.code !== "ENOENT") throw error; return null; }); + if (!entry?.isDirectory() || entry.isSymbolicLink()) continue; + if (workspace.repoUrl || workspace.branchName || workspace.strategyType === "git_worktree" + || await fs.lstat(path.join(root, ".git")).then(() => true, + (error: NodeJS.ErrnoException) => { if (error.code !== "ENOENT") throw error; return false; })) return true; + } + return false; +} + +async function companion(db: ReadDb, workspace: Workspace) { + const runs = await db.select({ id: heartbeatRuns.id, issueId: heartbeatRuns.nativeIssueId, + state: sql`${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,state}'` }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, workspace.companyId), + sql`${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,workspaceId}'=${workspace.id}`, + sql`${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,state}' in ('prepared','finalized')`, + )).orderBy(heartbeatRuns.id); + const sources: LegacyWorkspaceRecoverySource[] = []; + for (const run of runs.filter(run => run.state === "prepared")) { + const source = await inspectOnlineLegacyWorkspaceRecoverySource(db, { workspace, runId: run.id }); + if (!source) return null; + sources.push(source); + } + const issueIds = [...new Set([workspace.sourceIssueId, ...runs.map(run => run.issueId)].filter((id): id is string => Boolean(id)))]; + const actions = issueIds.length ? await db.select().from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, workspace.companyId), inArray(issueRecoveryActions.sourceIssueId, issueIds), + inArray(issueRecoveryActions.status, ["active", "escalated"]), + )).orderBy(issueRecoveryActions.id) : []; + const holds: Array = []; + for (const recovery of actions) { + const hold = await inspectOnlineLegacyWorkspaceRecoveryHold(db, { workspace, recovery }); + if (!hold) return null; + holds.push({ actionId: recovery.id, ...hold }); + } + return { workspaceId: workspace.id, sources, holds, recoveryIssueIds: issueIds }; +} + +export async function inspectOnlineWorkspaceSource(db: ReadDb, workspace: Workspace): Promise { + const cwd = workspace.cwd ?? workspace.providerRef; + if (!cwd || !path.isAbsolute(cwd) || !["local_fs", "git_worktree"].includes(workspace.providerType)) return null; + const ordinary = await inspectOnlineOrdinarySource(db, workspace); + if (ordinary) { + const marker = await fs.lstat(path.join(ordinary.root, ".git")).catch((error: NodeJS.ErrnoException) => { + if (error.code !== "ENOENT") throw error; return null; + }); + if (!marker) return { ...ordinary, kind: "ordinary", projectIds: ordinary.projectId ? [ordinary.projectId] : [], companions: [] }; + } + let selected: Identities; + try { selected = await workspaceSourceIdentities(workspace.companyId, cwd); } catch { return null; } + if (!selected.git || selected.git.root !== await physicalPath(cwd) || selected.unownedPaths.length || !workspace.sourceIssueId + || workspace.cwd && workspace.providerRef && workspace.cwd !== workspace.providerRef) return null; + const head = await fs.lstat(path.join(selected.git.gitDirectory.path, "HEAD")).catch((error: NodeJS.ErrnoException) => { + if (error.code !== "ENOENT") throw error; return null; + }); + if (!head || !head.isFile() && !head.isSymbolicLink()) return null; + const entry = await fs.lstat(cwd); + const rows = await db.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.companyId, workspace.companyId), + inArray(executionWorkspaces.providerType, ["local_fs", "git_worktree"]))).orderBy(executionWorkspaces.id).limit(1001); + if (rows.length > 1000) return null; + const nodes: Array<{ row: Workspace; identities: Identities }> = []; + for (const row of rows) { + const paths = localWorkspaceSourcePaths(row); + if (!paths.length || paths.some(cwd => !path.isAbsolute(cwd))) return null; + try { + const identities = await workspaceSourceIdentities(workspace.companyId, paths[0]); + for (const cwd of paths.slice(1)) { + const alternate = await workspaceSourceIdentities(workspace.companyId, cwd); + identities.projectIds = [...new Set([...identities.projectIds, ...alternate.projectIds])].sort(); + identities.issueIds = [...new Set([...identities.issueIds, ...alternate.issueIds])].sort(); + } + nodes.push({ row, identities }); + } catch { return null; } + } + const projectIds = new Set(selected.projectIds), physicalIssueIds = new Set(selected.issueIds), aliases = new Set(); + const importedChildren: NonNullable = []; + try { + for (const child of await inspectImportedWorkspaceSourcePaths(db, workspace)) { + const identities = await workspaceSourceIdentities(workspace.companyId, child.sourcePath); + if (identities.unownedPaths.length) return null; + for (const id of identities.projectIds) projectIds.add(id); + for (const id of identities.issueIds) physicalIssueIds.add(id); + importedChildren.push({ ...child, gitFingerprint: identities.git?.fingerprint ?? null }); + } + } catch { return null; } + for (;;) { + let changed = false; + for (const node of nodes) { + if (aliases.has(node.row.id)) continue; + if (node.row.id !== workspace.id && !intersects(node.identities.projectIds, projectIds) + && !intersects(node.identities.issueIds, physicalIssueIds) + && !(node.row.projectId && projectIds.has(node.row.projectId)) + && !(node.row.sourceIssueId && physicalIssueIds.has(node.row.sourceIssueId))) continue; + if (!node.row.sourceIssueId || node.row.cwd && node.row.providerRef && node.row.cwd !== node.row.providerRef + || !["active", "idle", "in_review", "archived"].includes(node.row.status) + || node.row.metadata?.remoteWorkspace || hasPendingGitExpiration(node.row)) return null; + aliases.add(node.row.id); changed = true; + for (const id of node.identities.projectIds) projectIds.add(id); + for (const id of node.identities.issueIds) physicalIssueIds.add(id); + if (node.row.projectId) projectIds.add(node.row.projectId); + } + if (!changed) break; + } + if (!aliases.has(workspace.id) || aliases.size > 256) return null; + const included = nodes.filter(node => aliases.has(node.row.id)); + const issueIds = [...new Set([...physicalIssueIds, ...included.map(node => node.row.sourceIssueId!)])].sort(); + if (issueIds.length + projectIds.size > 128) return null; + const instance = await physicalPath(resolvePaperclipInstanceRoot()); + const gitSourceRoots = [...[...projectIds].map(id => path.join(instance, "projects", workspace.companyId, id)), + ...issueIds.flatMap(id => ["isolated-workspaces", "isolated-workspace-attempts", "isolated-repository-sources"].map(kind => + path.join(instance, kind, workspace.companyId, id, ...(kind === "isolated-repository-sources" ? ["checkout"] : []))))].sort(); + const owned = (target: string) => gitSourceRoots.some(root => target === root || target.startsWith(root + path.sep)); + let after: string | undefined; + for (let page = 0; ; page++) { + const foreign = await db.select({ id: executionWorkspaces.id, cwd: executionWorkspaces.cwd, providerRef: executionWorkspaces.providerRef }) + .from(executionWorkspaces).where(and(ne(executionWorkspaces.companyId, workspace.companyId), + inArray(executionWorkspaces.providerType, ["local_fs", "git_worktree"]), ...(after ? [gt(executionWorkspaces.id, after)] : []))) + .orderBy(executionWorkspaces.id).limit(256); + for (const row of foreign) { + for (const cwd of localWorkspaceSourcePaths(row)) { + if (!path.isAbsolute(cwd)) return null; + try { + if (owned(await physicalPath(cwd))) return null; + const git = await readWorkspaceGitSourceRoots(cwd, { ancestors: true }); + if (git && [git.gitDirectory.path, git.commonDirectory.path, ...git.objectDirectories.map(root => root.path)].some(owned)) return null; + } catch { return null; } + } + } + if (foreign.length < 256) break; + if (page === 39) return null; + after = foreign.at(-1)!.id; + } + const recipe = readGitCacheRestoration(workspace); + if (recipe && (!recipe.restoredAt || recipe.contents && !recipe.contentsRestoredAt + || recipe.restorePoint && !recipe.restorePoint.expiredAt && !recipe.restorePoint.appliedAt)) return null; + const companions: OnlineWorkspaceSource["companions"] = [], settledPreparedWorkspaceIds: string[] = [], settledRecoveryIssueIds = new Set(); + for (const node of included) { + const proof = await companion(db, node.row); + if (!proof) return null; + companions.push({ workspaceId: proof.workspaceId, sources: proof.sources, holds: proof.holds }); + if (proof.sources.length) settledPreparedWorkspaceIds.push(proof.workspaceId); + for (const id of proof.recoveryIssueIds) settledRecoveryIssueIds.add(id); + } + return { kind: "git", root: selected.git.root, projectId: null, projectIds: [...projectIds].sort(), issueIds, + workspaceIds: [...aliases].sort(), device: entry.dev, inode: entry.ino, gitSourceRoots, companions, + physicalRoots: included.flatMap(node => localWorkspaceSourcePaths(node.row)).sort(), + settledPreparedWorkspaceIds: settledPreparedWorkspaceIds.sort(), settledRecoveryIssueIds: [...settledRecoveryIssueIds].sort(), + settledFinalizationRunIds: [...new Set(companions.flatMap(proof => proof.sources.map(source => source.runId)))].sort(), + gitSourceFingerprints: included.map(node => ({ workspaceId: node.row.id, fingerprint: node.identities.git?.fingerprint ?? null })), importedChildren, + }; +} diff --git a/server/src/services/remote-workspace-migration-tree.ts b/server/src/services/remote-workspace-migration-tree.ts index 48c771763f3..30145c2ad6f 100644 --- a/server/src/services/remote-workspace-migration-tree.ts +++ b/server/src/services/remote-workspace-migration-tree.ts @@ -1,11 +1,13 @@ import fs from "node:fs/promises"; import path from "node:path"; import { isDeepStrictEqual } from "node:util"; -import { and, eq, gt, inArray, ne, sql } from "drizzle-orm"; +import { and, asc, eq, gt, inArray, ne, sql } from "drizzle-orm"; import { executionWorkspaces, executionWorkspaceSnapshots, heartbeatRuns, workspaceOperations, type Db } from "@paperclipai/db"; import { resolvePaperclipInstanceRoot } from "../home-paths.js"; import { adapterExecutionControls } from "./adapter-execution-control.js"; import { managedProjectSourceIdentity } from "./workspace-source-lock.js"; +import { readWorkspaceGitSourceRoots } from "./workspace-git-source-roots.js"; +import { localWorkspaceSourcePaths } from "./execution-workspace-source-roots.js"; import { offlineMigrationExecutionWhere } from "./offline-migration-execution.js"; import type { StorageService } from "../storage/types.js"; import { captureRemoteWorkspaceDirectory, restoreRemoteWorkspaceArchive } from "./remote-workspace-snapshot-archive.js"; @@ -49,25 +51,32 @@ export async function offlineWorkspaceMigrationBlockers(db: ReadDb) { return active ? ["active_execution_requires_drain"] : []; } -export type OnlineOrdinarySource = { +export type WorkspaceSourceScope = { root: string; projectId: string | null; issueIds: string[]; workspaceIds: string[]; device: number; inode: number; + physicalRoots?: string[]; gitSourceRoots?: string[]; settledPreparedWorkspaceIds?: string[]; settledRecoveryIssueIds?: string[]; settledFinalizationRunIds?: string[]; + importedChildren?: Array<{ operationId: string; sourcePath: string }>; }; export async function requiresOnlineOrdinarySource(workspace: Workspace): Promise { - if (!workspace.cwd || workspace.providerType !== "local_fs" || workspace.repoUrl || workspace.branchName) return false; - const parts = path.relative(resolvePaperclipInstanceRoot(), workspace.cwd).split(path.sep); - const managed = parts[1] === workspace.companyId && Boolean(workspace.sourceIssueId) && ( - parts[0] === "projects" && parts.length === 4 && /^[a-f0-9-]{36}$/.test(parts[2]) - || parts[2] === workspace.sourceIssueId && (parts[0] === "isolated-workspaces" && parts.length === 3 - || parts[0] === "isolated-workspace-attempts" && parts.length === 4)); - if (!managed) return false; - const entry = await fs.lstat(workspace.cwd).catch((error: NodeJS.ErrnoException) => { - if (error.code === "ENOENT") return null; throw error; - }); - return Boolean(entry?.isDirectory() && !entry.isSymbolicLink()); + if (workspace.providerType !== "local_fs" || workspace.repoUrl || workspace.branchName) return false; + const instance = await physicalPath(resolvePaperclipInstanceRoot()); + for (const cwd of localWorkspaceSourcePaths(workspace)) { + const root = await physicalPath(cwd); + const parts = path.relative(instance, root).split(path.sep); + const managed = parts[1] === workspace.companyId && Boolean(workspace.sourceIssueId) && ( + parts[0] === "projects" && parts.length === 4 && /^[a-f0-9-]{36}$/.test(parts[2]) + || parts[2] === workspace.sourceIssueId && (parts[0] === "isolated-workspaces" && parts.length === 3 + || parts[0] === "isolated-workspace-attempts" && parts.length === 4)); + if (!managed) continue; + const entry = await fs.lstat(root).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return null; throw error; + }); + if (entry?.isDirectory() && !entry.isSymbolicLink()) return true; + } + return false; } -export async function inspectOnlineOrdinarySource(db: ReadDb, workspace: Workspace): Promise { +export async function inspectOnlineOrdinarySource(db: ReadDb, workspace: Workspace): Promise | null> { if (!workspace.cwd || workspace.providerType !== "local_fs" || workspace.repoUrl || workspace.branchName) return null; const root = await physicalPath(workspace.cwd); const instance = await physicalPath(resolvePaperclipInstanceRoot()); @@ -98,10 +107,10 @@ export async function inspectOnlineOrdinarySource(db: ReadDb, workspace: Workspa if (rows.length > 1000) return null; const aliases: typeof rows = []; for (const row of rows) { - const cwd = rootOf(row); - let physical: string; - try { physical = cwd ? await physicalPath(cwd) : ""; } catch { return null; } - if (!physical || !overlaps(root, physical)) continue; + const paths = localWorkspaceSourcePaths(row); + let physical: string[]; + try { physical = await Promise.all(paths.map(physicalPath)); } catch { return null; } + if (!physical.some(cwd => overlaps(root, cwd))) continue; if (row.cwd !== root || row.providerRef && row.providerRef !== root || row.providerType !== "local_fs" || row.repoUrl || row.branchName || !row.sourceIssueId || isolated && row.sourceIssueId !== workspace.sourceIssueId @@ -119,8 +128,9 @@ export async function inspectOnlineOrdinarySource(db: ReadDb, workspace: Workspa ...(after ? [gt(executionWorkspaces.id, after)] : []), )).orderBy(executionWorkspaces.id).limit(256); for (const row of foreign) { - const cwd = rootOf(row); - try { if (cwd && overlaps(root, await physicalPath(cwd))) return null; } catch { return null; } + for (const cwd of localWorkspaceSourcePaths(row)) { + try { if (overlaps(root, await physicalPath(cwd))) return null; } catch { return null; } + } } if (foreign.length < 256) break; if (page === 39) return null; @@ -132,9 +142,12 @@ export async function inspectOnlineOrdinarySource(db: ReadDb, workspace: Workspa } export async function onlineOrdinarySourceBlockers(db: Pick, companyId: string, - source: OnlineOrdinarySource, workspace: Pick, operationId?: string): Promise { + source: WorkspaceSourceScope, workspace: Pick, operationId?: string): Promise { const workspaceIds = sql.join(source.workspaceIds.map(id => sql`${id}::uuid`), sql`, `); const issueIds = sql.join(source.issueIds.map(id => sql`${id}::uuid`), sql`, `); + const settledWorkspaces = sql.join((source.settledPreparedWorkspaceIds?.length ? source.settledPreparedWorkspaceIds : ["00000000-0000-0000-0000-000000000000"]).map(id => sql`${id}`), sql`, `); + const settledFinalizers = sql.join((source.settledFinalizationRunIds?.length ? source.settledFinalizationRunIds : ["00000000-0000-0000-0000-000000000000"]).map(id => sql`${id}::uuid`), sql`, `); + const settledRecoveries = sql.join((source.settledRecoveryIssueIds?.length ? source.settledRecoveryIssueIds : ["00000000-0000-0000-0000-000000000000"]).map(id => sql`${id}::uuid`), sql`, `); const rows = await db.execute<{ reason: string }>(sql` select 'active_execution' reason where exists (select 1 from heartbeat_runs r where r.company_id=${companyId}::uuid and r.status in ('queued','running') and (r.issue_id in (${issueIds}) or r.native_issue_id in (${issueIds}) @@ -154,18 +167,19 @@ export async function onlineOrdinarySourceBlockers(db: Pick, comp and r.runner_profile_json#>>'{nativeWorkspaceSync,workspaceId}' in (${sql.join(source.workspaceIds.map(id => sql`${id}`), sql`, `)}) and (r.runner_profile_json#>>'{nativeWorkspaceSync,workspaceId}'<>${workspace.id} and r.runner_profile_json#>>'{nativeWorkspaceSync,state}'='prepared' + and r.runner_profile_json#>>'{nativeWorkspaceSync,workspaceId}' not in (${settledWorkspaces}) or r.runner_profile_json->'nativeWorkspaceFinalizationOwner' is not null or r.controller_lease_expires_at>clock_timestamp())) union all select 'execution_finalization_unresolved' where exists (select 1 from native_run_finalizations f where f.company_id=${companyId}::uuid and (f.issue_id in (${issueIds}) and f.issue_id<>${workspace.sourceIssueId}::uuid or f.run_id in (select r.id from heartbeat_runs r where r.company_id=${companyId}::uuid and r.runner_profile_json#>>'{nativeWorkspaceSync,workspaceId}' in (${sql.join(source.workspaceIds.map(id => sql`${id}`), sql`, `)}) - and r.runner_profile_json#>>'{nativeWorkspaceSync,workspaceId}'<>${workspace.id})) and (f.phase not in ('committed','terminal_failure') or f.lease_owner is not null)) + and r.runner_profile_json#>>'{nativeWorkspaceSync,workspaceId}'<>${workspace.id})) and f.run_id not in (${settledFinalizers}) and (f.phase not in ('committed','terminal_failure') or f.lease_owner is not null)) union all select 'workspace_recovery_unresolved' where exists (select 1 from issue_recovery_actions a where a.company_id=${companyId}::uuid and (a.source_issue_id in (${issueIds}) and a.source_issue_id<>${workspace.sourceIssueId}::uuid or a.source_issue_id in (select r.native_issue_id from heartbeat_runs r where r.company_id=${companyId}::uuid and r.runner_profile_json#>>'{nativeWorkspaceSync,workspaceId}' in (${sql.join(source.workspaceIds.map(id => sql`${id}`), sql`, `)}) - and r.runner_profile_json#>>'{nativeWorkspaceSync,workspaceId}'<>${workspace.id})) and a.status in ('active','escalated')) + and r.runner_profile_json#>>'{nativeWorkspaceSync,workspaceId}'<>${workspace.id})) and a.source_issue_id not in (${settledRecoveries}) and a.status in ('active','escalated')) union all select 'repository_preparation_unresolved' where exists (select 1 from execution_workspace_repositories r where r.company_id=${companyId}::uuid and r.execution_workspace_id in (${workspaceIds}) and r.state<>'ready')`); const services = await db.execute<{ cwd: string | null; provider: string }>(sql` @@ -174,9 +188,17 @@ export async function onlineOrdinarySourceBlockers(db: Pick, comp if (services.length > 1000) reasons.push("workspace_runtime_service_unsettled"); else for (const service of services) { try { - if (service.cwd ? overlaps(source.root, await physicalPath(service.cwd)) : service.provider !== "adapter_managed") { - reasons.push("workspace_runtime_service_unsettled"); break; + const cwd = service.cwd ? await physicalPath(service.cwd) : null; + let overlap = cwd ? (source.physicalRoots ?? [source.root]).some(root => overlaps(root, cwd)) : service.provider !== "adapter_managed"; + if (!overlap && cwd && source.gitSourceRoots) { + const owned = (target: string) => source.gitSourceRoots!.some(root => target === root || target.startsWith(root + path.sep)); + overlap = owned(cwd); + if (!overlap) { + const git = await readWorkspaceGitSourceRoots(cwd, { ancestors: true }); + overlap = Boolean(git && [git.gitDirectory.path, git.commonDirectory.path, ...git.objectDirectories.map(root => root.path)].some(owned)); + } } + if (overlap) { reasons.push("workspace_runtime_service_unsettled"); break; } } catch { reasons.push("workspace_runtime_service_unsettled"); break; } } const controlled = [...adapterExecutionControls.keys()]; @@ -191,7 +213,7 @@ export async function onlineOrdinarySourceBlockers(db: Pick, comp /** Only the maintenance entry point can acknowledge that every old deployment * has stopped. A task-level migration hold cannot fence other path aliases. */ -export async function workspaceMigrationLayoutBlockers(db: ReadDb, workspace: Workspace, writersStopped = false, onlineSource?: OnlineOrdinarySource) { +export async function workspaceMigrationLayoutBlockers(db: ReadDb, workspace: Workspace, writersStopped = false, onlineSource?: WorkspaceSourceScope) { if (writersStopped) return offlineWorkspaceMigrationBlockers(db); const cwd = rootOf(workspace); if (!cwd) return []; @@ -201,45 +223,73 @@ export async function workspaceMigrationLayoutBlockers(db: ReadDb, workspace: Wo inArray(executionWorkspaces.providerType, ["local_fs", "git_worktree"]), )); const root = await physicalPath(cwd); - for (const row of local) if (!onlineSource?.workspaceIds.includes(row.id) && rootOf(row) && overlaps(root, await physicalPath(rootOf(row)!))) - return ["overlapping_workspace_requires_maintenance"]; - const migrated = await db.select({ cwd: workspaceOperations.cwd }).from(workspaceOperations).where(and( + for (const row of local) { + for (const cwd of localWorkspaceSourcePaths(row)) { + const physical = await physicalPath(cwd); + if (onlineSource?.gitSourceRoots && physical.startsWith(`${root}/.paperclip/worktrees/`)) { + const entry = await fs.lstat(physical).catch((error: NodeJS.ErrnoException) => { if (error.code !== "ENOENT") throw error; return null; }); + if (entry) return ["nested_workspace_import_required"]; + } + if (!onlineSource?.workspaceIds.includes(row.id) && overlaps(root, physical)) return ["overlapping_workspace_requires_maintenance"]; + } + } + const migrated = await db.select({ id: workspaceOperations.id, cwd: workspaceOperations.cwd }).from(workspaceOperations).where(and( eq(workspaceOperations.companyId, workspace.companyId), eq(workspaceOperations.phase, "remote_workspace_migration"), eq(workspaceOperations.status, "succeeded"), ne(workspaceOperations.executionWorkspaceId, workspace.id), )); - for (const row of migrated) if (row.cwd && (await physicalPath(row.cwd)).startsWith(`${root}/.paperclip/worktrees/`)) - return ["overlapping_workspace_requires_maintenance"]; + for (const row of migrated) { + if (!row.cwd) continue; + const sourcePath = await physicalPath(row.cwd); + if (sourcePath.startsWith(`${root}/.paperclip/worktrees/`) + && !(onlineSource?.gitSourceRoots && onlineSource.importedChildren?.some(child => child.operationId === row.id && child.sourcePath === sourcePath))) + return ["overlapping_workspace_requires_maintenance"]; + } return []; } +async function importedWorkspaceCandidates(db: ReadDb, workspace: Workspace) { + const sourceRoot = rootOf(workspace); + if (!sourceRoot) return []; + const root = await physicalPath(sourceRoot); + const rows = await db.select({ operation: workspaceOperations, snapshot: executionWorkspaceSnapshots }) + .from(workspaceOperations).innerJoin(executionWorkspaceSnapshots, and( + eq(executionWorkspaceSnapshots.companyId, workspaceOperations.companyId), + eq(executionWorkspaceSnapshots.executionWorkspaceId, workspaceOperations.executionWorkspaceId), + eq(executionWorkspaceSnapshots.sourceOperationId, workspaceOperations.id), + )).where(and(eq(workspaceOperations.companyId, workspace.companyId), + ne(workspaceOperations.executionWorkspaceId, workspace.id), eq(workspaceOperations.phase, "remote_workspace_migration"), + eq(workspaceOperations.status, "succeeded"), inArray(executionWorkspaceSnapshots.status, ["committed", "superseded"]), + sql`${workspaceOperations.metadata}->>'snapshotId' = ${executionWorkspaceSnapshots.id}::text`, + )).orderBy(asc(workspaceOperations.id)).limit(10241); + if (rows.length > 10240) throw new Error("nested_workspace_scope_limit"); + const located = []; + for (const row of rows) { + if (!row.operation.cwd) continue; + located.push({ ...row, sourcePath: await physicalPath(row.operation.cwd) }); + } + return located.filter(row => row.sourcePath.startsWith(`${root}/.paperclip/worktrees/`)) + .sort((a, b) => a.sourcePath.length - b.sourcePath.length || a.operation.id.localeCompare(b.operation.id)); +} + +export async function inspectImportedWorkspaceSourcePaths(db: ReadDb, workspace: Workspace) { + return (await importedWorkspaceCandidates(db, workspace)).map(({ operation, snapshot, sourcePath }) => ({ + workspaceId: snapshot.executionWorkspaceId, operationId: operation.id, snapshotId: snapshot.id, + sourcePath, manifestDigest: snapshot.manifestDigest, operationManifestDigest: operation.metadata?.manifestDigest, + })); +} + /** Runtime-created nested task folders can be omitted only after an exact * committed import covers their current bytes. Ordinary user folders stay in * the parent archive. The snapshot owner retains the resulting dependencies. */ export async function prepareNestedWorkspaceImports(input: { db: Db; storage: StorageService; workspace: Workspace; scratch: string; signal?: AbortSignal; - recordedChildren?: ImportedWorkspaceChild[]; + recordedChildren?: ImportedWorkspaceChild[]; gitSourceRoots?: string[]; }): Promise { const sourceRoot = rootOf(input.workspace); - if (!sourceRoot) return []; - if (input.recordedChildren && !input.recordedChildren.length) return []; + if (!sourceRoot || input.recordedChildren && !input.recordedChildren.length) return []; const root = await physicalPath(sourceRoot); - const rows = await input.db.select({ operation: workspaceOperations, snapshot: executionWorkspaceSnapshots }) - .from(workspaceOperations).innerJoin(executionWorkspaceSnapshots, and( - eq(executionWorkspaceSnapshots.companyId, workspaceOperations.companyId), - eq(executionWorkspaceSnapshots.executionWorkspaceId, workspaceOperations.executionWorkspaceId), - eq(executionWorkspaceSnapshots.sourceOperationId, workspaceOperations.id), - )).where(and(eq(workspaceOperations.companyId, input.workspace.companyId), - ne(workspaceOperations.executionWorkspaceId, input.workspace.id), eq(workspaceOperations.phase, "remote_workspace_migration"), - eq(workspaceOperations.status, "succeeded"), inArray(executionWorkspaceSnapshots.status, ["committed", "superseded"]), - sql`${workspaceOperations.metadata}->>'snapshotId' = ${executionWorkspaceSnapshots.id}::text`, - )); - const located = []; - for (const row of rows) { - if (!row.operation.cwd || input.recordedChildren && !input.recordedChildren.some(child => child.operationId === row.operation.id)) continue; - located.push({ ...row, sourcePath: await physicalPath(row.operation.cwd) }); - } - const candidates = located.filter(row => row.sourcePath.startsWith(`${root}/.paperclip/worktrees/`)) - .sort((a, b) => a.sourcePath.length - b.sourcePath.length || a.operation.id.localeCompare(b.operation.id)); + const candidates = (await importedWorkspaceCandidates(input.db, input.workspace)).filter(row => + !input.recordedChildren || input.recordedChildren.some(child => child.operationId === row.operation.id)); const covered: ImportedWorkspaceChild[] = []; for (const { operation, snapshot, sourcePath } of candidates) { input.signal?.throwIfAborted(); @@ -259,7 +309,7 @@ export async function prepareNestedWorkspaceImports(input: { async function verifyCoveredWorkspace(input: { db: Db; storage: StorageService; sourcePath: string; snapshot: typeof executionWorkspaceSnapshots.$inferSelect; - operationId: string; expectedDigest: unknown; scratch: string; signal?: AbortSignal; depth: number; + operationId: string; expectedDigest: unknown; scratch: string; signal?: AbortSignal; depth: number; gitSourceRoots?: string[]; }) { input.signal?.throwIfAborted(); if (input.depth > 32) throw new Error("nested_workspace_coverage_invalid"); @@ -287,6 +337,6 @@ async function verifyCoveredWorkspace(input: { await loadWorkspaceSnapshotArchive({ storage: input.storage, reference, targetPath: archivePath, signal: input.signal }); await restoreRemoteWorkspaceArchive({ archivePath, targetDir: restored, expectedManifestDigest: reference.manifestDigest, signal: input.signal }); const current = await captureRemoteWorkspaceDirectory({ cwd: input.sourcePath, archivePath: path.join(area, "current.pcws"), - exclude: children.map(child => child.relativePath), signal: input.signal }); + exclude: children.map(child => child.relativePath), gitSourceRoots: input.gitSourceRoots, signal: input.signal }); if (current.manifestDigest !== reference.manifestDigest) throw new Error("nested_workspace_source_changed"); } diff --git a/server/src/services/remote-workspace-migration.test.ts b/server/src/services/remote-workspace-migration.test.ts index 24b511bdbfd..3df31c2d939 100644 --- a/server/src/services/remote-workspace-migration.test.ts +++ b/server/src/services/remote-workspace-migration.test.ts @@ -521,7 +521,8 @@ describe.skipIf(!support.supported)("remote workspace migration", () => { }); async function gitArchiveFixture(repoUrl = "https://example.invalid/repository.git", home = root) { const f = await seed(), projectId = randomUUID(), projectWorkspaceId = randomUUID(), branchName = "saved-task"; - vi.stubEnv("PAPERCLIP_HOME", home); vi.stubEnv("PAPERCLIP_INSTANCE_ID", "default"); + await fs.mkdir(home, { recursive: true }); + vi.stubEnv("PAPERCLIP_HOME", await fs.realpath(home)); vi.stubEnv("PAPERCLIP_INSTANCE_ID", "default"); const base = resolveManagedProjectWorkspaceDir({ companyId: f.companyId, projectId, repoName: "repository" }); await fs.mkdir(base, { recursive: true }); await git(base, "init", "-q", "-b", "main"); await fs.writeFile(path.join(base, "base.txt"), "base\n"); await git(base, "add", "."); await git(base, "commit", "-qm", "base"); @@ -1143,7 +1144,13 @@ describe.skipIf(!support.supported)("remote workspace migration", () => { expect(await fs.readFile(path.join(f.cwd, "saved.txt"), "utf8")).toBe("new local edits\n"); }); it("imports dirty unpushed Git work and restores it after the original folder disappears", async () => { - const f = await seed(); await git(f.cwd, "init", "-q", "-b", "feature"); + let f = await seed(); + vi.stubEnv("PAPERCLIP_HOME", await fs.realpath(root)); + const cwd = path.join(resolvePaperclipInstanceRoot(), "isolated-workspaces", f.companyId, f.issueId); + await fs.mkdir(path.dirname(cwd), { recursive: true }); await fs.rename(f.cwd, cwd); + await db.update(executionWorkspaces).set({ cwd, providerRef: cwd }).where(eq(executionWorkspaces.id, f.workspaceId)); + f = { ...f, cwd }; + await git(f.cwd, "init", "-q", "-b", "feature"); await git(f.cwd, "add", "."); await git(f.cwd, "commit", "-qm", "unpushed work"); await git(f.cwd, "remote", "add", "origin", "https://example.invalid/repository.git"); const head = await git(f.cwd, "rev-parse", "HEAD"); diff --git a/server/src/services/remote-workspace-migration.ts b/server/src/services/remote-workspace-migration.ts index 7cd94e97e21..07ea7d30095 100644 --- a/server/src/services/remote-workspace-migration.ts +++ b/server/src/services/remote-workspace-migration.ts @@ -22,8 +22,9 @@ import { loadTaskDirectoryRestorePoint, mayBeIsolatedTaskDirectoryWorkspace, rea import { inspectArchivedGitMigrationSource, lockArchivedGitMigrationSources, materializeArchivedGitMigrationSource, type ArchivedGitMigrationSource } from "./remote-workspace-migration-git.js"; import { readNativeWorkspaceSyncReference } from "./native-runtime/native-workspace-sync.js"; import { isLegacyWorkspaceTaskRelated, readLegacyWorkspaceRecoveryAllocations, LEGACY_WORKSPACE_RECOVERY_MIGRATION, type LegacyWorkspaceRecoverySource } from "./legacy-workspace-recovery-migration.js"; -import { inspectOnlineOrdinarySource, requiresOnlineOrdinarySource, onlineOrdinarySourceBlockers, type OnlineOrdinarySource, offlineWorkspaceMigrationBlockers, prepareNestedWorkspaceImports, validateImportedWorkspaceChildren, workspaceMigrationLayoutBlockers } from "./remote-workspace-migration-tree.js"; +import { onlineOrdinarySourceBlockers, offlineWorkspaceMigrationBlockers, prepareNestedWorkspaceImports, validateImportedWorkspaceChildren, workspaceMigrationLayoutBlockers } from "./remote-workspace-migration-tree.js"; +import { inspectOnlineWorkspaceSource, requiresOnlineWorkspaceSource, type OnlineWorkspaceSource } from "./online-workspace-source.js"; import { withWorkspaceSourceLocks, IsolatedRepositorySourceBusy } from "./workspace-source-lock.js"; import { inspectOnlineLegacyWorkspaceRecoverySource, inspectOnlineLegacyWorkspaceRecoveryHold, lockOnlineLegacyWorkspaceRecoverySources, type OnlineLegacyWorkspaceRecoveryHold } from "./online-legacy-workspace-recovery.js"; import { inspectNativeWorkspaceMigration, nativeWorkspaceMigrationMarker } from "./native-runtime/native-workspace-migration.js"; @@ -84,7 +85,7 @@ async function isUnmaterializedRepositoryRequest(workspace: Workspace, }); } -async function inspect(db: ReadDb, input: { companyId: string; workspaceId: string; operationId?: string; writersStopped?: boolean; onlineSource?: OnlineOrdinarySource; onlineSourceUnproven?: boolean }) { +async function inspect(db: ReadDb, input: { companyId: string; workspaceId: string; operationId?: string; writersStopped?: boolean; onlineSource?: OnlineWorkspaceSource; onlineSourceUnproven?: boolean }) { const [workspace] = await db.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, input.workspaceId))); if (!workspace) throw new Error("remote_workspace_migration_workspace_missing"); const authority = readRemoteWorkspaceAuthority(workspace.metadata); @@ -95,7 +96,7 @@ async function inspect(db: ReadDb, input: { companyId: string; workspaceId: stri if (input.writersStopped) reasons.push(...await offlineWorkspaceMigrationBlockers(db)); if (input.onlineSourceUnproven) reasons.push("online_workspace_source_scope_unproven"); if (input.onlineSource) { - const currentScope = await inspectOnlineOrdinarySource(db, workspace); + const currentScope = await inspectOnlineWorkspaceSource(db, workspace); if (canonical(currentScope) !== canonical(input.onlineSource)) reasons.push("online_workspace_source_scope_changed"); else reasons.push(...await onlineOrdinarySourceBlockers(db as Db, workspace.companyId, currentScope!, workspace, allowedOperation)); } @@ -311,9 +312,9 @@ export async function inventoryRemoteWorkspaceMigration(db: Db, input: { company const [workspace] = !input.writersStopped ? await db.select().from(executionWorkspaces).where(and( eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, input.workspaceId))) : []; const onlineSource = workspace && !readRemoteWorkspaceAuthority(workspace.metadata) - ? await inspectOnlineOrdinarySource(db, workspace) ?? undefined : undefined; + ? await inspectOnlineWorkspaceSource(db, workspace) ?? undefined : undefined; const result = await inspect(db, { ...input, onlineSource, onlineSourceUnproven: Boolean(workspace && !onlineSource - && !readRemoteWorkspaceAuthority(workspace.metadata) && await requiresOnlineOrdinarySource(workspace)) }); + && !readRemoteWorkspaceAuthority(workspace.metadata) && await requiresOnlineWorkspaceSource(workspace)) }); return { companyId: input.companyId, workspaceId: input.workspaceId, eligible: result.reasons.length === 0, alreadyMigrated: Boolean(result.authority), reasons: result.reasons, localPath: result.workspace.cwd ?? result.workspace.providerRef, source: result.source, operationId: typeof result.marker.operationId === "string" ? result.marker.operationId : null, @@ -329,9 +330,9 @@ export async function migrateRemoteWorkspace(input: { const [sourceWorkspace] = !input.writersStopped ? await input.db.select().from(executionWorkspaces).where(and( eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, input.workspaceId))) : []; const onlineSource = sourceWorkspace && !readRemoteWorkspaceAuthority(sourceWorkspace.metadata) - ? await inspectOnlineOrdinarySource(input.db, sourceWorkspace) ?? undefined : undefined; + ? await inspectOnlineWorkspaceSource(input.db, sourceWorkspace) ?? undefined : undefined; if (sourceWorkspace && !onlineSource && !readRemoteWorkspaceAuthority(sourceWorkspace.metadata) - && await requiresOnlineOrdinarySource(sourceWorkspace)) { + && await requiresOnlineWorkspaceSource(sourceWorkspace)) { throw new Error("remote_workspace_migration_blocked:online_workspace_source_scope_unproven"); } let committedPublication: Parameters[0] | undefined; @@ -390,7 +391,8 @@ export async function migrateRemoteWorkspace(input: { let operation = reserved.operation!; if (onlineSource) { if (!await lockOnlineLegacyWorkspaceRecoverySources(lock as unknown as Db, { companyId: input.companyId, - sources: reserved.legacyRecoverySources, holds: reserved.onlineRecoveryHolds })) + sources: [...reserved.legacyRecoverySources, ...onlineSource.companions.flatMap(proof => proof.sources)], + holds: [...reserved.onlineRecoveryHolds, ...onlineSource.companions.flatMap(proof => proof.holds)] })) throw new Error("remote_workspace_migration_blocked:remote_copyback_owned"); const current = await inspect(input.db, { ...input, onlineSource, operationId: operation.id }); if (current.reasons.length || canonical(current.legacyRecoverySources) !== canonical(reserved.legacyRecoverySources) @@ -420,8 +422,8 @@ export async function migrateRemoteWorkspace(input: { // archive identity even if a later batch has already imported the leaves. const recordedChildren = scope.state === "fixed" ? validateImportedWorkspaceChildren(scope.children) : scope.state === "pending" ? undefined : []; - const nested = input.writersStopped && !retained && !reserved.archivedGitSource - ? await prepareNestedWorkspaceImports({ ...input, workspace: reserved.workspace, scratch, recordedChildren }) : []; + const nested = (input.writersStopped || onlineSource?.kind === "git") && !retained && !reserved.archivedGitSource + ? await prepareNestedWorkspaceImports({ ...input, workspace: reserved.workspace, scratch, recordedChildren, gitSourceRoots: onlineSource?.gitSourceRoots }) : []; if (scope.state === "pending") { const [saved] = await input.db.update(workspaceOperations).set({ metadata: { ...operation.metadata, nestedWorkspaceScope: { version: 1, state: "fixed", children: nested } }, updatedAt: new Date() }) @@ -431,7 +433,7 @@ export async function migrateRemoteWorkspace(input: { operation = saved; } const exclude = nested.map(child => child.relativePath); - const captured = recovered ?? await captureRemoteWorkspaceDirectory({ cwd: sourceRoot!, archivePath, exclude, requireNoGit: Boolean(onlineSource), signal: input.signal }); + const captured = recovered ?? await captureRemoteWorkspaceDirectory({ cwd: sourceRoot!, archivePath, exclude, requireNoGit: onlineSource?.kind === "ordinary", gitSourceRoots: onlineSource?.gitSourceRoots, signal: input.signal }); const ref = await uploadWorkspaceSnapshot({ db: input.db, storage: input.storage, snapshotId: snapshot.id, companyId: input.companyId, workspaceId: input.workspaceId, archivePath, manifestDigest: captured.manifestDigest, signal: input.signal, // Private Git reconstruction can change administrative bytes between @@ -444,7 +446,7 @@ export async function migrateRemoteWorkspace(input: { await restoreRemoteWorkspaceArchive({ archivePath: restoredArchive, targetDir: restored, expectedManifestDigest: ref.manifestDigest, signal: input.signal }); if (!recovered) { const currentArchive = path.join(scratch, "current.pcws"); - const current = await captureRemoteWorkspaceDirectory({ cwd: sourceRoot!, archivePath: currentArchive, exclude, requireNoGit: Boolean(onlineSource), signal: input.signal }); + const current = await captureRemoteWorkspaceDirectory({ cwd: sourceRoot!, archivePath: currentArchive, exclude, requireNoGit: onlineSource?.kind === "ordinary", gitSourceRoots: onlineSource?.gitSourceRoots, signal: input.signal }); if (current.manifestDigest !== captured.manifestDigest) throw new Error("remote_workspace_migration_source_changed"); } if (nested.length) { @@ -561,7 +563,7 @@ export async function migrateRemoteWorkspace(input: { if (onlineSource) { const result = await withWorkspaceSourceLocks(input.db, { companyId: input.companyId, issueIds: onlineSource.issueIds, - projectIds: onlineSource.projectId ? [onlineSource.projectId] : [], + projectIds: onlineSource.projectIds, }, async lock => { for (const workspaceId of onlineSource.workspaceIds) { if (!await tryAcquireExecutionWorkspaceLifecycleLock(lock, workspaceId)) diff --git a/server/src/services/remote-workspace-online-source.test.ts b/server/src/services/remote-workspace-online-source.test.ts index 9f9e0c5cc2f..d32d690d06c 100644 --- a/server/src/services/remote-workspace-online-source.test.ts +++ b/server/src/services/remote-workspace-online-source.test.ts @@ -1,3 +1,5 @@ +import { execFile } from "node:child_process"; +import { promisify } from "node:util"; import { randomUUID } from "node:crypto"; import fs from "node:fs/promises"; import os from "node:os"; @@ -74,9 +76,153 @@ describe.skipIf(!support.supported)("online ordinary physical source fence", () } return { companyId, projectId, cwd, aliases, workspaceId: aliases[0].workspace.id }; } + const execute = promisify(execFile); + const git = async (cwd: string, ...args: string[]) => (await execute("git", ["-c", `core.hooksPath=${os.devNull}`, "-C", cwd, ...args], { + env: { ...process.env, GIT_CONFIG_NOSYSTEM: "1", GIT_CONFIG_GLOBAL: os.devNull, GIT_TERMINAL_PROMPT: "0", + GIT_AUTHOR_NAME: "Source fixture", GIT_AUTHOR_EMAIL: "source@example.test", GIT_COMMITTER_NAME: "Source fixture", GIT_COMMITTER_EMAIL: "source@example.test" }, + })).stdout.trim(); + async function gitFixture(f = undefined as Awaited> | undefined) { + const base = f ?? await fixture(); + await git(base.cwd, "init", "-q", "-b", "main"); + await git(base.cwd, "add", "."); await git(base.cwd, "commit", "-qm", "Saved legacy work"); + const repoUrl = "https://example.invalid/saved-work.git"; + await git(base.cwd, "remote", "add", "origin", repoUrl); + for (const alias of base.aliases) await db.update(executionWorkspaces).set({ repoUrl, branchName: "main" }) + .where(eq(executionWorkspaces.id, alias.workspace.id)); + const [issue] = await db.insert(issues).values({ companyId: base.companyId, projectId: base.projectId, title: "Shared Git child" }).returning(); + const cwd = path.join(base.cwd, ".paperclip", "worktrees", "child"); + await git(base.cwd, "worktree", "add", "-qb", "child", cwd); + const [workspace] = await db.insert(executionWorkspaces).values({ companyId: base.companyId, projectId: base.projectId, + sourceIssueId: issue.id, name: "Shared Git child", mode: "isolated_workspace", strategyType: "git_worktree", + providerType: "git_worktree", cwd, providerRef: cwd, repoUrl, branchName: "child", baseRef: "main" }).returning(); + await db.update(issues).set({ executionWorkspaceId: workspace.id }).where(eq(issues.id, issue.id)); + return { ...base, child: workspace, repoUrl }; + } async function stored(workspaceId: string) { return (await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId)))[0]; } + it("imports an ordinary managed directory without capturing its enclosing host repository", async () => { + await git(home, "init", "-q", "-b", "main"); + await fs.writeFile(path.join(home, "host-only.txt"), "outside the workspace\n"); + await git(home, "add", "host-only.txt"); await git(home, "commit", "-qm", "Enclosing host repository"); + const head = await git(home, "rev-parse", "HEAD"), f = await fixture(); + expect(await inventoryRemoteWorkspaceMigration(db, f)).toMatchObject({ eligible: true }); + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).resolves.toMatchObject({ applied: true }); + const authority = readRemoteWorkspaceAuthority((await stored(f.workspaceId)).metadata)!; + const reference = await getWorkspaceSnapshot(db, { ...f, snapshotId: authority.snapshotId! }); + const archivePath = path.join(home, "ordinary.pcws"), targetDir = path.join(home, "ordinary-restored"); + await loadWorkspaceSnapshotArchive({ storage, reference: reference!, targetPath: archivePath }); + await restoreRemoteWorkspaceArchive({ archivePath, targetDir, expectedManifestDigest: reference!.manifestDigest }); + expect(await fs.readFile(path.join(targetDir, "saved.txt"), "utf8")).toBe("saved legacy edits\n"); + expect(await fs.lstat(path.join(targetDir, ".git")).catch(() => null)).toBeNull(); + expect(await fs.lstat(path.join(targetDir, "host-only.txt")).catch(() => null)).toBeNull(); + expect(await git(home, "rev-parse", "HEAD")).toBe(head); + }); + it.each([false, true])("does not admit a Git fragment as an ordinary migration source (objects: %s)", async objects => { + const f = await fixture(), info = path.join(f.cwd, ".git", "info"); + await fs.mkdir(info, { recursive: true }); + await fs.writeFile(path.join(info, "exclude"), "/.paperclip-repositories/\n"); + if (objects) await fs.mkdir(path.join(f.cwd, ".git", "objects")); + expect(await inventoryRemoteWorkspaceMigration(db, f)).toMatchObject({ eligible: false, + reasons: expect.arrayContaining(["online_workspace_source_scope_unproven"]) }); + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow("online_workspace_source_scope_unproven"); + expect(await db.select().from(workspaceOperations)).toEqual([]); + expect(await fs.readFile(path.join(info, "exclude"), "utf8")).toBe("/.paperclip-repositories/\n"); + }); + it("preserves unmanaged Git import in genuine offline mode while retaining online and drain guards", async () => { + const f = await fixture(), cwd = path.join(home, "self-hosted-repository"), repoUrl = "https://example.invalid/self-hosted.git"; + await fs.mkdir(cwd); await git(cwd, "init", "-q", "-b", "main"); + await fs.writeFile(path.join(cwd, "saved.txt"), "self-hosted original work\n"); + await git(cwd, "add", "."); await git(cwd, "commit", "-qm", "Self-hosted source"); + await git(cwd, "remote", "add", "origin", repoUrl); + for (const alias of f.aliases.slice(1)) await db.update(executionWorkspaces).set({ providerType: "plugin" }) + .where(eq(executionWorkspaces.id, alias.workspace.id)); + await db.update(executionWorkspaces).set({ cwd, providerRef: cwd, repoUrl, branchName: "main" }) + .where(eq(executionWorkspaces.id, f.workspaceId)); + expect(await inventoryRemoteWorkspaceMigration(db, f)).toMatchObject({ eligible: false, + reasons: expect.arrayContaining(["online_workspace_source_scope_unproven"]) }); + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow("online_workspace_source_scope_unproven"); + vi.stubEnv("PAPERCLIP_EPHEMERAL_MIGRATION", "true"); + const [agent] = await db.insert(agents).values({ companyId: f.companyId, name: "Offline writer guard" }).returning(); + const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: agent.id, + issueId: f.aliases[0].issue.id, status: "running", runtimeMode: "legacy" }).returning(); + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true, writersStopped: true })) + .rejects.toThrow("active_execution_requires_drain"); + expect(await db.select().from(workspaceOperations)).toEqual([]); + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, run.id)); + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true, writersStopped: true })).resolves.toMatchObject({ applied: true }); + expect(await fs.readFile(path.join(cwd, "saved.txt"), "utf8")).toBe("self-hosted original work\n"); + }); + it.each(["ordinary", "Git"].flatMap(kind => ["cwd", "providerRef", "different", "alias"].map(locator => ({ kind, locator }))))( + "rejects foreign $kind ownership through $locator source locators", async ({ kind, locator }) => { + const base = await fixture(), f = kind === "Git" ? await gitFixture(base) : null; + const workspaceId = f?.child.id ?? base.workspaceId, sourcePath = f?.child.cwd ?? base.cwd; + const companyId = randomUUID(), unrelated = path.join(home, "foreign-display-root"), alias = path.join(home, "foreign-path-alias"); + await fs.mkdir(unrelated); await fs.symlink(sourcePath, alias); + await db.insert(companies).values({ id: companyId, name: "Foreign source owner", issuePrefix: "FOREIGN" }); + await db.insert(executionWorkspaces).values({ companyId, name: "Foreign source alias", mode: "shared_workspace", + strategyType: "project_primary", providerType: "local_fs", + cwd: locator === "cwd" ? sourcePath : locator === "providerRef" ? null : unrelated, + providerRef: locator === "cwd" ? null : locator === "alias" ? alias : sourcePath }); + const input = { companyId: base.companyId, workspaceId }; + expect(await inventoryRemoteWorkspaceMigration(db, input)).toMatchObject({ eligible: false }); + await expect(migrateRemoteWorkspace({ db, storage, ...input, apply: true })).rejects.toThrow("remote_workspace_migration_blocked"); + expect(readRemoteWorkspaceAuthority((await stored(workspaceId)).metadata)).toBeNull(); + expect(await db.select().from(executionWorkspaceSnapshots)).toEqual([]); + expect(await fs.readFile(path.join(sourcePath, "saved.txt"), "utf8")).toBe("saved legacy edits\n"); + }); + it.each(["cwd", "providerRef"] as const)("rejects a mismatched same-company ordinary %s owner", async locator => { + const f = await fixture(), unrelated = path.join(home, "another-source"); + await fs.mkdir(unrelated); + await db.update(executionWorkspaces).set({ cwd: locator === "cwd" ? f.cwd : unrelated, + providerRef: locator === "cwd" ? unrelated : f.cwd }).where(eq(executionWorkspaces.id, f.aliases[1].workspace.id)); + expect(await inventoryRemoteWorkspaceMigration(db, f)).toMatchObject({ eligible: false }); + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow("online_workspace_source_scope_unproven"); + expect(await db.select().from(workspaceOperations)).toEqual([]); + }); + it.each(["providerRef", "different", "alias", "secondary-Git"] as const)("requires a proven source for the selected %s locator before fallback", async locator => { + const base = await fixture(), f = locator === "secondary-Git" ? await gitFixture(base) : null; + for (const alias of base.aliases.slice(f ? 0 : 1)) await db.update(executionWorkspaces).set({ providerType: "plugin" }) + .where(eq(executionWorkspaces.id, alias.workspace.id)); + const workspaceId = f?.child.id ?? base.workspaceId, sourcePath = f?.child.cwd ?? base.cwd; + const unrelated = path.join(home, "selected-display-root"); await fs.mkdir(unrelated); + const alias = path.join(home, "selected-path-alias"); await fs.symlink(sourcePath, alias); + await db.update(executionWorkspaces).set({ cwd: locator === "providerRef" ? null : unrelated, + providerRef: locator === "alias" ? alias : sourcePath, repoUrl: null, branchName: null, + strategyType: "project_primary" }).where(eq(executionWorkspaces.id, workspaceId)); + const input = { companyId: base.companyId, workspaceId }; + expect(await inventoryRemoteWorkspaceMigration(db, input)).toMatchObject({ eligible: false, + reasons: expect.arrayContaining(["online_workspace_source_scope_unproven"]) }); + await expect(migrateRemoteWorkspace({ db, storage, ...input, apply: true })).rejects.toThrow("online_workspace_source_scope_unproven"); + expect(await db.select().from(workspaceOperations)).toEqual([]); + expect(await fs.readFile(path.join(sourcePath, "saved.txt"), "utf8")).toBe("saved legacy edits\n"); + }); + it.each(["cwd", "providerRef", "different", "alias"] as const)("reserves every %s source locator before the actual lifecycle owner", async locator => { + const f = await fixture(), unrelated = path.join(home, "lifecycle-display-root"), alias = path.join(home, "lifecycle-path-alias"); + await fs.mkdir(unrelated); await fs.symlink(f.cwd, alias); + await db.update(executionWorkspaces).set({ projectId: null, + cwd: locator === "cwd" ? f.cwd : locator === "providerRef" ? null : unrelated, + providerRef: locator === "cwd" ? null : locator === "alias" ? alias : f.cwd }).where(eq(executionWorkspaces.id, f.workspaceId)); + let entered!: () => void, release!: () => void, mutated = false; + const started = new Promise(resolve => { entered = resolve; }); + const pending = new Promise(resolve => { release = resolve; }); + const owner = withWorkspaceSourceLocks(db, { companyId: f.companyId, projectIds: [f.projectId] }, async () => { entered(); await pending; }); + await started; + const lifecycle = withExecutionWorkspaceLifecycleLock(db, f.workspaceId, async () => { mutated = true; }); + try { + await vi.waitFor(async () => { + const waiting = await db.execute(sql`select count(*)::int n from pg_stat_activity where datname=current_database() and wait_event='advisory'`); + expect(waiting[0].n).toBeGreaterThan(0); + }); + expect(mutated).toBe(false); + const key = `execution_workspace_lifecycle:${f.workspaceId}`; + const locks = await db.execute(sql`select count(*)::int n from pg_locks where locktype='advisory' and granted + and classid=((hashtextextended(${key},0)>>32)&4294967295)::oid and objid=(hashtextextended(${key},0)&4294967295)::oid`); + expect(locks[0].n).toBe(0); + } finally { release(); await owner; await lifecycle; } + expect(mutated).toBe(true); + expect(await fs.readFile(path.join(f.cwd, "saved.txt"), "utf8")).toBe("saved legacy edits\n"); + }); function uploadBarrier() { let entered!: () => void, release!: () => void; const started = new Promise(resolve => { entered = resolve; }); @@ -192,11 +338,13 @@ describe.skipIf(!support.supported)("online ordinary physical source fence", () repoUrl: null, repoRef: null, strategy: "project_primary", branchName: null, branchCreatedByRuntime: false, worktreePath: null, warnings: [], created: false } })).toEqual([]); }); - it.each(["workspace", "configured"] as const)("fences project command provisioning and its running %s CWD without an execution workspace ID", async location => { + it.each(["workspace", "configured", "symlink"] as const)("fences project command provisioning and its running %s CWD without an execution workspace ID", async location => { const f = await fixture(); const startProjectId = location === "configured" ? randomUUID() : f.projectId; if (startProjectId !== f.projectId) await db.insert(projects).values({ id: startProjectId, companyId: f.companyId, name: "Service owner" }); - const startCwd = resolveManagedProjectWorkspaceDir({ companyId: f.companyId, projectId: startProjectId }); + const startCwd = location === "symlink" ? path.join(home, "configured-service-source") + : resolveManagedProjectWorkspaceDir({ companyId: f.companyId, projectId: startProjectId }); + if (location === "symlink") await fs.symlink(f.cwd, startCwd); await fs.mkdir(startCwd, { recursive: true }); const [agent] = await db.insert(agents).values({ companyId: f.companyId, name: "Project command owner" }).returning(); const [projectWorkspace] = await db.insert(projectWorkspaces).values({ companyId: f.companyId, projectId: startProjectId, @@ -360,6 +508,146 @@ describe.skipIf(!support.supported)("online ordinary physical source fence", () status: "active", evidence: { runId } }); return { ...f, run, owner, lease, warning }; } + it("imports a shared Git child while a terminal sibling recovery and unrelated remote execution remain intact", async () => { + const recovery = await recoveryFixture(), f = await gitFixture(recovery); + const unrelated = await remoteWriter(f.companyId, recovery.run.agentId); + await fs.writeFile(path.join(f.child.cwd!, "saved.txt"), "staged child work\n"); + await git(f.child.cwd!, "add", "."); + await fs.writeFile(path.join(f.child.cwd!, "saved.txt"), "unstaged child work\n"); + const input = { companyId: f.companyId, workspaceId: f.child.id }; + expect(await inventoryRemoteWorkspaceMigration(db, input)).toMatchObject({ eligible: true }); + await expect(migrateRemoteWorkspace({ db, storage, ...input, apply: true })).resolves.toMatchObject({ applied: true }); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, unrelated.id)))[0]).toEqual(unrelated); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, recovery.run.id)))[0]).toEqual(recovery.run); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, recovery.run.id)))[0]).toEqual(recovery.owner); + expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, recovery.warning.id)))[0]).toEqual(recovery.warning); + expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, recovery.lease.id)))[0]).toEqual(recovery.lease); + const authority = readRemoteWorkspaceAuthority((await stored(f.child.id)).metadata)!; + const reference = await getWorkspaceSnapshot(db, { companyId: f.companyId, workspaceId: f.child.id, snapshotId: authority.snapshotId! }); + const archivePath = path.join(home, "git-child.pcws"), targetDir = path.join(home, "git-child-restored"); + await loadWorkspaceSnapshotArchive({ storage, reference: reference!, targetPath: archivePath }); + expect(await fs.readFile(path.join(f.cwd, "saved.txt"), "utf8")).toBe("saved legacy edits\n"); + await fs.rm(f.cwd, { recursive: true }); + await restoreRemoteWorkspaceArchive({ archivePath, targetDir, expectedManifestDigest: reference!.manifestDigest }); + expect(await git(targetDir, "show", ":saved.txt")).toBe("staged child work"); + expect(await fs.readFile(path.join(targetDir, "saved.txt"), "utf8")).toBe("unstaged child work\n"); + await git(targetDir, "fsck", "--full", "--no-dangling"); + }); + it("takes the original private Git store owner before reserving an import journal", async () => { + const f = await fixture(); + const [seed] = await db.insert(issues).values({ companyId: f.companyId, title: "Private seed owner" }).returning(); + const source = path.join(resolvePaperclipInstanceRoot(), "isolated-repository-sources", f.companyId, seed.id, "checkout"); + await fs.mkdir(source, { recursive: true }); await git(source, "init", "-q", "-b", "main"); + await fs.writeFile(path.join(source, "saved.txt"), "private seed work\n"); + await git(source, "add", "."); await git(source, "commit", "-qm", "Private seed"); + const repoUrl = "https://example.invalid/private-seed.git"; + await git(source, "remote", "add", "origin", repoUrl); + const [issue] = await db.insert(issues).values({ companyId: f.companyId, title: "Linked task" }).returning(); + const cwd = path.join(f.cwd, ".paperclip", "worktrees", "private-child"); + await git(source, "worktree", "add", "-qb", "private-child", cwd); + const [workspace] = await db.insert(executionWorkspaces).values({ companyId: f.companyId, projectId: f.projectId, + sourceIssueId: issue.id, name: "Private linked source", mode: "isolated_workspace", strategyType: "git_worktree", + providerType: "git_worktree", cwd, providerRef: cwd, repoUrl, branchName: "private-child" }).returning(); + let entered!: () => void, release!: () => void; + const started = new Promise(resolve => { entered = resolve; }); + const pending = new Promise(resolve => { release = resolve; }); + const owner = withIsolatedRepositorySourceLock(db, f.companyId, seed.id, async () => { entered(); await pending; }); + await started; + const input = { companyId: f.companyId, workspaceId: workspace.id }; + try { + expect(await inventoryRemoteWorkspaceMigration(db, input)).toMatchObject({ eligible: true }); + await expect(migrateRemoteWorkspace({ db, storage, ...input, apply: true })).rejects.toBeInstanceOf(IsolatedRepositorySourceBusy); + expect(await db.select().from(workspaceOperations)).toEqual([]); + } finally { release(); await owner; } + await expect(migrateRemoteWorkspace({ db, storage, ...input, apply: true })).resolves.toMatchObject({ applied: true }); + expect(await fs.readFile(path.join(source, "saved.txt"), "utf8")).toBe("private seed work\n"); + }); + it.each(["coordinator", "allocation", "locator", "git-metadata"] as const)("rechecks the shared Git companion %s before publication", async change => { + const recovery = await recoveryFixture(), f = await gitFixture(recovery), barrier = uploadBarrier(); + const migration = migrateRemoteWorkspace({ db, storage, companyId: f.companyId, workspaceId: f.child.id, apply: true }) + .then(value => ({ value }), error => ({ error })); + await Promise.race([barrier.started, migration.then(result => { if ("error" in result) throw result.error; throw new Error("capture did not upload"); })]); + try { + if (change === "coordinator") await db.update(nativeRunFinalizations).set({ leaseOwner: randomUUID(), leaseExpiresAt: new Date(Date.now() - 1000) }) + .where(eq(nativeRunFinalizations.runId, recovery.run.id)); + if (change === "allocation") await db.update(environmentLeases).set({ metadata: { remoteCwd: "/changed-companion-source" } }) + .where(eq(environmentLeases.id, recovery.lease.id)); + if (change === "locator") await db.update(executionWorkspaces).set({ providerRef: path.join(home, "unproven-companion") }) + .where(eq(executionWorkspaces.id, f.aliases[1].workspace.id)); + if (change === "git-metadata") { + const administration = await git(f.child.cwd!, "rev-parse", "--absolute-git-dir"); + await fs.writeFile(path.join(administration, "commondir"), path.join(f.cwd, ".git") + "\n"); + } + } finally { barrier.release(); } + const result = await migration; + expect(result).toHaveProperty("error"); + expect("error" in result ? String(result.error) : "").toContain("online_workspace_source_scope_changed"); + expect(readRemoteWorkspaceAuthority((await stored(f.child.id)).metadata)).toBeNull(); + expect((await db.select().from(executionWorkspaceSnapshots))[0].status).toBe("uploaded"); + expect(await fs.readFile(path.join(f.child.cwd!, "saved.txt"), "utf8")).toBe("saved legacy edits\n"); + }); + it("retains an imported nested child's private cache owner through parent source verification", async () => { + const f = await gitFixture(), sourceIssueId = randomUUID(); + const source = path.join(resolvePaperclipInstanceRoot(), "isolated-repository-sources", f.companyId, sourceIssueId, "checkout"); + await git(f.cwd, "worktree", "remove", f.child.cwd!); + await fs.mkdir(path.dirname(source), { recursive: true }); + await git(home, "clone", "--no-hardlinks", f.cwd, source); + await git(source, "remote", "set-url", "origin", f.repoUrl); + await git(source, "worktree", "add", "-qb", "private-child", f.child.cwd!); + await db.update(executionWorkspaces).set({ branchName: "private-child" }).where(eq(executionWorkspaces.id, f.child.id)); + await migrateRemoteWorkspace({ db, storage, companyId: f.companyId, workspaceId: f.child.id, apply: true }); + expect((await stored(f.child.id)).providerType).toBe("cloud_sandbox"); + let entered!: () => void, release!: () => void; + const started = new Promise(resolve => { entered = resolve; }); + const pending = new Promise(resolve => { release = resolve; }); + const owner = withWorkspaceSourceLocks(db, { companyId: f.companyId, issueIds: [sourceIssueId] }, async () => { + entered(); await pending; + }); + await started; + try { + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toBeInstanceOf(IsolatedRepositorySourceBusy); + expect(await db.select().from(workspaceOperations).where(eq(workspaceOperations.executionWorkspaceId, f.workspaceId))).toEqual([]); + } finally { release(); await owner; } + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).resolves.toMatchObject({ applied: true }); + const authority = readRemoteWorkspaceAuthority((await stored(f.workspaceId)).metadata)!; + const [snapshot] = await db.select().from(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.id, authority.snapshotId!)); + expect(snapshot.metadata?.importedWorkspaceChildren).toEqual([expect.objectContaining({ workspaceId: f.child.id })]); + expect(await fs.readFile(path.join(source, "saved.txt"), "utf8")).toBe("saved legacy edits\n"); + expect(await fs.readFile(path.join(f.child.cwd!, "saved.txt"), "utf8")).toBe("saved legacy edits\n"); + }); + + it.each(["missing", "deleted"] as const)("rejects a Git parent with %s imported-child snapshot coverage", async state => { + const f = await gitFixture(); + await migrateRemoteWorkspace({ db, storage, companyId: f.companyId, workspaceId: f.child.id, apply: true }); + const authority = readRemoteWorkspaceAuthority((await stored(f.child.id)).metadata)!; + if (state === "missing") await db.delete(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.id, authority.snapshotId!)); + else await db.update(executionWorkspaceSnapshots).set({ status: "deleted" }).where(eq(executionWorkspaceSnapshots.id, authority.snapshotId!)); + expect(await inventoryRemoteWorkspaceMigration(db, f)).toMatchObject({ eligible: false, + reasons: expect.arrayContaining(["overlapping_workspace_requires_maintenance"]) }); + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow("overlapping_workspace_requires_maintenance"); + expect(readRemoteWorkspaceAuthority((await stored(f.workspaceId)).metadata)).toBeNull(); + expect(await db.select().from(workspaceOperations).where(eq(workspaceOperations.executionWorkspaceId, f.workspaceId))).toEqual([]); + expect(await fs.readFile(path.join(f.child.cwd!, "saved.txt"), "utf8")).toBe("saved legacy edits\n"); + }); + it.each([false, true])("requires exact imported-child coverage for an online shared Git parent (%s)", async changed => { + const f = await gitFixture(); + expect(await inventoryRemoteWorkspaceMigration(db, f)).toMatchObject({ eligible: false, + reasons: expect.arrayContaining(["nested_workspace_import_required"]) }); + await migrateRemoteWorkspace({ db, storage, companyId: f.companyId, workspaceId: f.child.id, apply: true }); + if (changed) await fs.writeFile(path.join(f.child.cwd!, "new-work.txt"), "changed after child import\n"); + expect(await inventoryRemoteWorkspaceMigration(db, f)).toMatchObject({ eligible: true }); + const migration = migrateRemoteWorkspace({ db, storage, ...f, apply: true }); + if (changed) { + await expect(migration).rejects.toThrow("nested_workspace_source_changed"); + expect(readRemoteWorkspaceAuthority((await stored(f.workspaceId)).metadata)).toBeNull(); + expect(await fs.readFile(path.join(f.child.cwd!, "new-work.txt"), "utf8")).toBe("changed after child import\n"); + } else { + await expect(migration).resolves.toMatchObject({ applied: true }); + const authority = readRemoteWorkspaceAuthority((await stored(f.workspaceId)).metadata)!; + const [snapshot] = await db.select().from(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.id, authority.snapshotId!)); + expect(snapshot.metadata?.importedWorkspaceChildren).toEqual([expect.objectContaining({ workspaceId: f.child.id })]); + } + }); it("imports a terminal prepared source while retaining its original provider and finalization obligations", async () => { const f = await recoveryFixture(), put = storage.putFile.bind(storage); const unrelated = await remoteWriter(f.companyId, f.run.agentId); diff --git a/server/src/services/remote-workspace-snapshot-archive.test.ts b/server/src/services/remote-workspace-snapshot-archive.test.ts index b7f166ffec5..92326d5ae48 100644 --- a/server/src/services/remote-workspace-snapshot-archive.test.ts +++ b/server/src/services/remote-workspace-snapshot-archive.test.ts @@ -81,6 +81,33 @@ describe("portable remote workspace archives", () => { expect(await sourceBytes(cwd)).toBe(before); await expect(capture(cwd)).resolves.toMatchObject({ manifestDigest: expect.any(String) }); }); + it.each(["linked", "borrowed"].flatMap(kind => ["local", "serialized"].map(mode => ({ kind, mode }))))( + "enforces owned Git source roots for $kind $mode capture without changing archive bytes", async ({ kind, mode }) => { + const base = await source(); await git(base, "init", "-q", "-b", "main"); + await fs.writeFile(path.join(base, "work.txt"), "base work\n"); await git(base, "add", "."); await git(base, "commit", "-qm", "base"); + const cwd = path.join(root, "selected"); + if (kind === "linked") await git(base, "worktree", "add", "-qb", "selected", cwd); + else await git(root, "clone", "-q", "--shared", base, cwd); + await fs.writeFile(path.join(cwd, "work.txt"), "staged work\n"); await git(cwd, "add", "."); + await fs.writeFile(path.join(cwd, "work.txt"), "unstaged work\n"); + const baseline = await capture(cwd), archivePath = path.join(root, "owned.pcws"); + const prefix = remoteWorkspaceArchiveScript().split("const [kind, first, second, digest]")[0]; + const fenced = async (roots: string[]) => { + const input = { kind: "capture", cwd, archivePath, gitSourceRoots: roots }; + return mode === "local" ? captureRemoteWorkspaceDirectory(input) + : JSON.parse((await execute(process.execPath, ["--eval", `${prefix}archive(${JSON.stringify(input)}).then(value=>process.stdout.write(JSON.stringify(value))).catch(error=>{process.stderr.write(error.message);process.exitCode=1;});`], { env })).stdout); + }; + await expect(fenced([await fs.realpath(cwd)])).rejects.toThrow("online_source_git_requires_shared_fence"); + await expect(fs.access(archivePath)).rejects.toMatchObject({ code: "ENOENT" }); + const snapshot = { archivePath, ...await fenced([await fs.realpath(root)]) }; + expect(snapshot).toMatchObject({ manifestDigest: baseline.manifestDigest, sha256: baseline.sha256, byteSize: baseline.byteSize }); + expect(await fs.readFile(archivePath)).toEqual(await fs.readFile(baseline.archivePath)); + await fs.rm(base, { recursive: true }); await fs.rm(cwd, { recursive: true }); + const target = await restore(snapshot); + expect(await git(target, "show", ":work.txt")).toBe("staged work"); + expect(await fs.readFile(path.join(target, "work.txt"), "utf8")).toBe("unstaged work\n"); + await git(target, "fsck", "--full", "--no-dangling"); + }); it("requires a shared Git fence for a nested bare repository borrowing external objects", async () => { const cwd = await source(), upstream = path.join(root, "upstream"), repository = path.join(cwd, "bare.git"); await fs.mkdir(upstream); @@ -92,8 +119,10 @@ describe("portable remote workspace archives", () => { expect(await fs.readFile(path.join(repository, "objects", "info", "alternates"), "utf8")) .toContain(path.join(upstream, ".git", "objects")); const before = await sourceBytes(cwd), external = await sourceBytes(upstream); - await expect(captureRemoteWorkspaceDirectory({ cwd, archivePath: path.join(root, "bare.pcws"), requireNoGit: true })) - .rejects.toThrow("online_source_git_requires_shared_fence"); + for (const options of [{ requireNoGit: true }, { gitSourceRoots: [await fs.realpath(root)] }]) { + await expect(captureRemoteWorkspaceDirectory({ cwd, archivePath: path.join(root, "bare.pcws"), ...options })) + .rejects.toThrow("online_source_git_requires_shared_fence"); + } expect(await sourceBytes(cwd)).toBe(before); expect(await sourceBytes(upstream)).toBe(external); }); diff --git a/server/src/services/remote-workspace-snapshot-archive.ts b/server/src/services/remote-workspace-snapshot-archive.ts index f554e793720..4afe876aa32 100644 --- a/server/src/services/remote-workspace-snapshot-archive.ts +++ b/server/src/services/remote-workspace-snapshot-archive.ts @@ -11,6 +11,7 @@ async function workspaceArchive(input: { expectedManifestDigest?: string; exclude?: string[]; requireNoGit?: boolean; + gitSourceRoots?: string[]; signal?: AbortSignal; maxEntries?: number; maxManifestBytes?: number; @@ -273,6 +274,46 @@ async function workspaceArchive(input: { const output = await fs.open(destination, "w", 0o600); try { await readStableFile(source, bytes => output.writeFile(bytes)); } finally { await output.close(); } }; + const assertGitSourceFence = async (repo: string) => { + if (!input.gitSourceRoots) return null; + if (!input.gitSourceRoots.length || input.gitSourceRoots.length > 384 + || input.gitSourceRoots.some(root => !path.isAbsolute(root) || path.resolve(root) !== root)) fail("online_source_git_requires_shared_fence"); + const directory = async (target: string) => { + const actual = await fs.realpath(target), entry = await fs.lstat(target, { bigint: true }); + if (!entry.isDirectory() || entry.isSymbolicLink() + || !input.gitSourceRoots!.some(root => actual === root || actual.startsWith(`${root}${path.sep}`))) { + fail("online_source_git_requires_shared_fence"); + } + return { path: actual, device: String(entry.dev), inode: String(entry.ino) }; + }; + const gitDirectory = await directory((await git(repo, ["rev-parse", "--absolute-git-dir"])).trim()); + const commonDirectory = await directory((await git(repo, ["rev-parse", "--path-format=absolute", "--git-common-dir"])).trim()); + const stores: unknown[] = [], seen = new Set(); + const visit = async (target: string, depth: number): Promise => { + if (depth > 4 || seen.size >= 32) fail("online_source_git_requires_shared_fence"); + const current = await directory(target); + if (seen.has(current.path)) return; + seen.add(current.path); + if (await fs.lstat(path.join(current.path, "info", "http-alternates")).then(() => true, + (error: NodeJS.ErrnoException) => { if (error.code !== "ENOENT") throw error; return false; })) fail("online_source_git_requires_shared_fence"); + const alternatePath = path.join(current.path, "info", "alternates"); + const entry = await fs.lstat(alternatePath).catch((error: NodeJS.ErrnoException) => { if (error.code !== "ENOENT") throw error; return null; }); + if (!entry) { stores.push(current); return; } + if (!entry.isFile() || entry.isSymbolicLink() || entry.size > 64 * 1024) fail("online_source_git_requires_shared_fence"); + const chunks: Buffer[] = []; let size = 0; + const saved = await readStableFile(alternatePath, async bytes => { + size += bytes.length; + if (size > 64 * 1024) fail("online_source_git_requires_shared_fence"); + chunks.push(bytes); + }); + const values = Buffer.concat(chunks).toString("utf8").split("\n").filter(Boolean); + if (values.length > 32 || values.some(value => /[\0\r]/.test(value) || value.startsWith('"'))) fail("online_source_git_requires_shared_fence"); + stores.push({ ...current, alternatesSha256: saved.sha256 }); + for (const value of values) await visit(path.resolve(current.path, value), depth + 1); + }; + await visit(path.join(commonDirectory.path, "objects"), 0); + return { gitDirectory, commonDirectory, stores }; + }; const gitStatePath = async (repo: string, name: string) => (await git(repo, ["rev-parse", "--path-format=absolute", "--git-path", name])).trim(); const gitStateHash = async (repo: string, name: string) => { const file = await gitStatePath(repo, name); @@ -325,7 +366,7 @@ async function workspaceArchive(input: { const directoryStat = await fs.lstat(directory); if (!directoryStat.isDirectory() || directoryStat.isSymbolicLink() || await fs.realpath(directory) !== directory) fail("changed"); const names = (await fs.readdir(directory)).sort(); - if (input.requireNoGit && (names.includes("commondir") || names.includes("gitdir") + if (skipGit && (input.requireNoGit || input.gitSourceRoots) && (names.includes("commondir") || names.includes("gitdir") || names.includes("objects") && names.some(name => ["HEAD", "refs", "packed-refs"].includes(name)) || names.includes("HEAD") && names.includes("config") || path.basename(directory) === "info" && names.some(name => ["alternates", "http-alternates"].includes(name)))) { @@ -411,13 +452,14 @@ async function workspaceArchive(input: { const repositories = [...new Set(directories)]; const gitSignatures = new Map(); const signature = async (repo: string, settings?: Awaited>) => { + const fence = await assertGitSourceFence(repo); const operation = await assertGitOperationComplete(repo); const index = (await git(repo, ["rev-parse", "--path-format=absolute", "--git-path", "index"])).trim(); const indexHash = await hashFile(index).catch((error: NodeJS.ErrnoException) => error.code === "ENOENT" ? null : Promise.reject(error)); const config = settings ?? await portableGitSettings(repo), shallow = await gitStateHash(repo, "shallow"); const partial = config.some(setting => /^remote\..*\.promisor$/u.test(setting.key) && setting.value === "true"); return JSON.stringify([await git(repo, ["status", "--porcelain=v2", "-z", "--untracked-files=no"]), await git(repo, ["for-each-ref", "--format=%(objectname) %(refname)"]), await git(repo, ["symbolic-ref", "-q", "HEAD"]).catch(() => git(repo, ["rev-parse", "HEAD"])), indexHash, config, - shallow, await gitStateHash(repo, "info/sparse-checkout"), shallow || partial ? (await gitObjectFiles(repo)).files : null, operation]); + shallow, await gitStateHash(repo, "info/sparse-checkout"), shallow || partial ? (await gitObjectFiles(repo)).files : null, operation, fence]); }; for (let index = 0; index < repositories.length; index++) { const relative = repositories[index]!, repo = path.join(root, relative), normalized = path.join(scratch, `repo-${index}.git`); @@ -587,7 +629,7 @@ async function workspaceArchive(input: { } } -export function captureRemoteWorkspaceDirectory(input: { cwd: string; archivePath: string; manifestPath?: string; exclude?: string[]; requireNoGit?: boolean; signal?: AbortSignal }) { +export function captureRemoteWorkspaceDirectory(input: { cwd: string; archivePath: string; manifestPath?: string; exclude?: string[]; requireNoGit?: boolean; gitSourceRoots?: string[]; signal?: AbortSignal }) { return workspaceArchive({ ...input, kind: "capture" }); } diff --git a/server/src/services/workspace-git-source-roots.test.ts b/server/src/services/workspace-git-source-roots.test.ts new file mode 100644 index 00000000000..6dc649d4868 --- /dev/null +++ b/server/src/services/workspace-git-source-roots.test.ts @@ -0,0 +1,143 @@ +import { randomUUID } from "node:crypto"; +import { execFile } from "node:child_process"; +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { promisify } from "node:util"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { readWorkspaceGitSourceRoots } from "./workspace-git-source-roots.js"; +import { workspaceSourceIdentities } from "./workspace-source-lock.js"; +import { localWorkspaceSourcePaths } from "./execution-workspace-source-roots.js"; +import { resolveManagedProjectWorkspaceDir, resolvePaperclipInstanceRoot } from "../home-paths.js"; +const execute = promisify(execFile); +let root: string; +const git = async (cwd: string, ...args: string[]) => execute("git", ["-c", `core.hooksPath=${os.devNull}`, "-C", cwd, ...args], { + env: { ...process.env, GIT_CONFIG_GLOBAL: os.devNull, GIT_CONFIG_NOSYSTEM: "1", GIT_TERMINAL_PROMPT: "0", + GIT_AUTHOR_NAME: "Source fixture", GIT_AUTHOR_EMAIL: "source@example.test", GIT_COMMITTER_NAME: "Source fixture", GIT_COMMITTER_EMAIL: "source@example.test" }, +}); +beforeEach(async () => { root = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "pc-git-source-"))); }); +afterEach(async () => { vi.unstubAllEnvs(); await fs.rm(root, { recursive: true, force: true }); }); +async function repository(name = "repository") { + const cwd = path.join(root, name); + await fs.mkdir(cwd, { recursive: true }); await git(cwd, "init", "--initial-branch=main"); + await fs.writeFile(path.join(cwd, "saved.txt"), "original work\n"); + await git(cwd, "add", "."); await git(cwd, "commit", "-m", "Original"); + return cwd; +} +describe("Git physical source roots", () => { + it("distinguishes an ignore-only fragment from Git source metadata", async () => { + const cwd = path.join(root, "plain-task"), marker = path.join(cwd, ".git"); + await fs.mkdir(path.join(marker, "info"), { recursive: true }); + await fs.writeFile(path.join(marker, "info", "exclude"), "/.paperclip-repositories/\n"); + expect(await readWorkspaceGitSourceRoots(cwd)).toBeNull(); + for (const name of ["HEAD", "config", "commondir"]) { + await fs.writeFile(path.join(marker, name), name === "commondir" ? ".\n" : "partial Git state\n"); + await expect(readWorkspaceGitSourceRoots(cwd)).rejects.toThrow("workspace_git_source_unproven"); + await fs.rm(path.join(marker, name)); + } + await fs.mkdir(path.join(marker, "objects")); + expect(await readWorkspaceGitSourceRoots(cwd)).toMatchObject({ objectDirectories: [{ path: path.join(marker, "objects") }] }); + }); + it.each([ + { cwd: "/source/a", providerRef: null, paths: ["/source/a"] }, + { cwd: null, providerRef: "/source/a", paths: ["/source/a"] }, + { cwd: "/source/a", providerRef: "/source/b", paths: ["/source/a", "/source/b"] }, + { cwd: "/source/a", providerRef: "/source/a", paths: ["/source/a"] }, + ])("retains every distinct local locator $cwd $providerRef", ({ cwd, providerRef, paths }) => { + expect(localWorkspaceSourcePaths({ cwd, providerRef })).toEqual(paths); + }); + it.each([false, true])("maps a configured directory symlink before strict Git inspection (%s)", async isGit => { + vi.stubEnv("PAPERCLIP_HOME", root); + const companyId = randomUUID(), projectId = randomUUID(); + const cwd = resolveManagedProjectWorkspaceDir({ companyId, projectId }); + await fs.mkdir(cwd, { recursive: true }); + if (isGit) await git(cwd, "init", "-q"); + const alias = path.join(root, "configured-source"); + await fs.symlink(cwd, alias); + expect(await workspaceSourceIdentities(companyId, alias)).toMatchObject({ projectIds: [projectId], unownedPaths: [] }); + expect(await workspaceSourceIdentities(companyId, alias)).toEqual(await workspaceSourceIdentities(companyId, cwd)); + await expect(readWorkspaceGitSourceRoots(alias)).rejects.toThrow("workspace_git_source_unproven"); + }); + it("binds actual linked worktree administration and the full shared object-store root", async () => { + const source = await repository(), cwd = path.join(root, "task worktree"); + await git(source, "worktree", "add", "-b", "task", cwd); + const actual = await readWorkspaceGitSourceRoots(cwd); + expect(actual).toMatchObject({ root: cwd, commonDirectory: { path: path.join(source, ".git") }, + objectDirectories: [{ path: path.join(source, ".git", "objects") }] }); + expect(actual!.gitDirectory.path.startsWith(path.join(source, ".git", "worktrees") + path.sep)).toBe(true); + expect(actual!.metadata.map(file => path.basename(file.path)).sort()).toEqual([".git", "commondir"]); + expect(await readWorkspaceGitSourceRoots(cwd)).toEqual(actual); + expect(await fs.readFile(path.join(cwd, "saved.txt"), "utf8")).toBe("original work\n"); + }); + it("retains borrowed object stores instead of treating HEAD as the complete source", async () => { + const source = await repository(), cwd = path.join(root, "borrower"); + await git(root, "clone", "--shared", source, cwd); + const actual = await readWorkspaceGitSourceRoots(cwd); + expect(actual!.objectDirectories.map(directory => directory.path).sort()).toEqual([ + path.join(cwd, ".git", "objects"), path.join(source, ".git", "objects"), + ].sort()); + expect(actual!.metadata.some(file => file.path.endsWith("info/alternates"))).toBe(true); + }); + it.each(["symlink", "fifo", "oversized", "http-alternate", "bare"] as const)("refuses unproven %s metadata", async kind => { + const cwd = await repository(); + if (kind === "symlink") { + await fs.rename(path.join(cwd, ".git"), path.join(cwd, "administration")); + await fs.symlink("administration", path.join(cwd, ".git")); + } + if (kind === "fifo" || kind === "oversized") { + await fs.rm(path.join(cwd, ".git"), { recursive: true }); + if (kind === "fifo") await execute("mkfifo", [path.join(cwd, ".git")]); + else await fs.writeFile(path.join(cwd, ".git"), "x".repeat(4097)); + } + if (kind === "http-alternate") await fs.writeFile(path.join(cwd, ".git", "objects", "info", "http-alternates"), "https://example.invalid/objects\n"); + if (kind === "bare") { await fs.rm(cwd, { recursive: true }); await git(root, "init", "--bare", cwd); } + await expect(readWorkspaceGitSourceRoots(cwd)).rejects.toThrow("workspace_git_source_unproven"); + }); + it("changes the proof when a valid pointer selects another administration directory", async () => { + const source = await repository(), first = path.join(root, "first"), second = path.join(root, "second"); + await git(source, "worktree", "add", "-b", "first", first); + await git(source, "worktree", "add", "-b", "second", second); + const before = await readWorkspaceGitSourceRoots(first); + await fs.writeFile(path.join(first, ".git"), await fs.readFile(path.join(second, ".git"))); + const after = await readWorkspaceGitSourceRoots(first); + expect(after!.gitDirectory).not.toEqual(before!.gitDirectory); + expect(after!.fingerprint).not.toBe(before!.fingerprint); + }); + it.each([false, true])("maps a managed linked source and its service subdirectory to both physical owners (%s)", async nested => { + vi.stubEnv("PAPERCLIP_HOME", root); + const companyId = randomUUID(), projectId = randomUUID(), sourceIssueId = randomUUID(); + const source = await repository(path.relative(root, path.join(resolvePaperclipInstanceRoot(), + "isolated-repository-sources", companyId, sourceIssueId, "checkout"))); + const cwd = path.join(resolveManagedProjectWorkspaceDir({ companyId, projectId }), ".paperclip", "worktrees", "task"); + await fs.mkdir(path.dirname(cwd), { recursive: true }); + await git(source, "worktree", "add", "-b", "task", cwd); + const target = nested ? path.join(cwd, "service") : cwd; + if (nested) await fs.mkdir(target); + const identities = await workspaceSourceIdentities(companyId, target); + expect(identities).toMatchObject({ projectIds: [projectId], issueIds: [sourceIssueId], unownedPaths: [] }); + expect(identities.git!.commonDirectory.path).toBe(path.join(source, ".git")); + }); + it("rejects a physical Git source owned by another company", async () => { + vi.stubEnv("PAPERCLIP_HOME", root); + const companyId = randomUUID(), foreignCompanyId = randomUUID(); + const source = await repository(path.relative(root, path.join(resolvePaperclipInstanceRoot(), + "isolated-repository-sources", foreignCompanyId, randomUUID(), "checkout"))); + const cwd = resolveManagedProjectWorkspaceDir({ companyId, projectId: randomUUID() }); + await fs.mkdir(path.dirname(cwd), { recursive: true }); + await git(source, "worktree", "add", "-b", "task", cwd); + await expect(workspaceSourceIdentities(companyId, cwd)).rejects.toThrow("workspace_source_company_mismatch"); + }); + it("keeps an external borrowed store explicitly unowned", async () => { + vi.stubEnv("PAPERCLIP_HOME", root); + const companyId = randomUUID(), projectId = randomUUID(), source = await repository("external"); + const cwd = resolveManagedProjectWorkspaceDir({ companyId, projectId }); + await fs.mkdir(path.dirname(cwd), { recursive: true }); + await git(root, "clone", "--shared", source, cwd); + expect(await workspaceSourceIdentities(companyId, cwd)).toMatchObject({ projectIds: [projectId], + unownedPaths: [path.join(source, ".git", "objects")] }); + }); + it("leaves absent and ordinary directories outside the Git proof", async () => { + expect(await readWorkspaceGitSourceRoots(path.join(root, "absent"))).toBeNull(); + expect(await readWorkspaceGitSourceRoots(root)).toBeNull(); + }); +}); diff --git a/server/src/services/workspace-git-source-roots.ts b/server/src/services/workspace-git-source-roots.ts new file mode 100644 index 00000000000..2e0c87b02c7 --- /dev/null +++ b/server/src/services/workspace-git-source-roots.ts @@ -0,0 +1,115 @@ +import fs from "node:fs/promises"; +import { constants } from "node:fs"; +import path from "node:path"; +import { createHash } from "node:crypto"; + +type Identity = { path: string; device: string; inode: string }; +type Metadata = Identity & { sha256: string; bytes: number }; +export type WorkspaceGitSourceRoots = { + root: string; gitDirectory: Identity; commonDirectory: Identity; + objectDirectories: Identity[]; metadata: Metadata[]; fingerprint: string; +}; +const invalid = () => new Error("workspace_git_source_unproven"); +const sha256 = (value: string | Buffer) => createHash("sha256").update(value).digest("hex"); +const entry = (target: string) => fs.lstat(target, { bigint: true }).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return null; + throw error; +}); + +async function directoryIdentity(target: string): Promise { + const stat = await entry(target); + if (!stat?.isDirectory() || stat.isSymbolicLink()) throw invalid(); + const actual = await fs.realpath(target); + return { path: actual, device: String(stat.dev), inode: String(stat.ino) }; +} + +async function readMetadata(target: string, limit: number): Promise<{ metadata: Metadata; text: string } | null> { + const before = await entry(target); + if (!before) return null; + if (!before.isFile() || before.isSymbolicLink() || before.nlink !== 1n || before.size > BigInt(limit)) throw invalid(); + const handle = await fs.open(target, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); + try { + const opened = await handle.stat({ bigint: true }); + if (opened.dev !== before.dev || opened.ino !== before.ino || !opened.isFile()) throw invalid(); + const buffer = Buffer.alloc(limit + 1), { bytesRead } = await handle.read(buffer, 0, buffer.length, 0); + const after = await handle.stat({ bigint: true }), current = await entry(target); + if (!current || bytesRead !== Number(before.size) || bytesRead > limit + || [opened, after, current].some(stat => stat.dev !== before.dev || stat.ino !== before.ino || stat.size !== before.size + || stat.mtimeNs !== before.mtimeNs || stat.ctimeNs !== before.ctimeNs)) throw invalid(); + const bytes = buffer.subarray(0, bytesRead); + return { metadata: { path: target, device: String(before.dev), inode: String(before.ino), bytes: bytesRead, sha256: sha256(bytes) }, + text: bytes.toString("utf8") }; + } finally { await handle.close(); } +} + +function metadataPath(text: string): string { + const value = text.replace(/\r?\n$/, ""); + if (!value || /[\0\r\n]/.test(value)) throw invalid(); + return value; +} + +async function onlyEntry(directory: string, name: string): Promise { + let found = false; + for await (const child of await fs.opendir(directory)) { + if (found || child.name !== name) return false; + found = true; + } + return found; +} + +export async function readWorkspaceGitSourceRoots(cwd: string, options: { ancestors?: boolean } = {}): Promise { + let root = path.resolve(cwd), marker = "", markerEntry: Awaited>; + for (let depth = 0; ; depth++) { + if (depth > 32) throw invalid(); + const source = await entry(root); + if (!source) return null; + if (!source.isDirectory() || source.isSymbolicLink()) throw invalid(); + root = await fs.realpath(root); + marker = path.join(root, ".git"); markerEntry = await entry(marker); + if (markerEntry) break; + if (await entry(path.join(root, "objects")) && await entry(path.join(root, "HEAD"))) throw invalid(); + if (!options.ancestors || path.dirname(root) === root) return null; + root = path.dirname(root); + } + const metadata: Metadata[] = []; + let gitDirectory: Identity; + if (markerEntry.isDirectory() && !markerEntry.isSymbolicLink()) { + if (!await entry(path.join(marker, "objects")) && await onlyEntry(marker, "info")) { + const info = path.join(marker, "info"), infoEntry = await entry(info); + if (infoEntry?.isDirectory() && !infoEntry.isSymbolicLink() && await onlyEntry(info, "exclude")) { + const exclude = await entry(path.join(info, "exclude")); + if (exclude?.isFile() && !exclude.isSymbolicLink() && exclude.nlink === 1n) return null; + } + } + gitDirectory = await directoryIdentity(marker); + } + else { + const pointer = await readMetadata(marker, 4096); + if (!pointer || !pointer.text.startsWith("gitdir: ")) throw invalid(); + metadata.push(pointer.metadata); + gitDirectory = await directoryIdentity(path.resolve(root, metadataPath(pointer.text.slice(8)))); + } + const commonPointer = await readMetadata(path.join(gitDirectory.path, "commondir"), 4096); + if (commonPointer) metadata.push(commonPointer.metadata); + const commonDirectory = commonPointer + ? await directoryIdentity(path.resolve(gitDirectory.path, metadataPath(commonPointer.text))) : gitDirectory; + const objectDirectories: Identity[] = [], seen = new Set(); + const visit = async (target: string, depth: number): Promise => { + if (depth > 4 || seen.size >= 32) throw invalid(); + const directory = await directoryIdentity(target); + if (seen.has(directory.path)) return; + seen.add(directory.path); objectDirectories.push(directory); + if (await entry(path.join(directory.path, "info", "http-alternates"))) throw invalid(); + const alternates = await readMetadata(path.join(directory.path, "info", "alternates"), 64 * 1024); + if (!alternates) return; + metadata.push(alternates.metadata); + const values = alternates.text.split("\n").filter(Boolean); + if (values.length > 32 || values.some(value => /[\0\r]/.test(value) || value.startsWith('"'))) throw invalid(); + for (const value of values) await visit(path.resolve(directory.path, value), depth + 1); + }; + await visit(path.join(commonDirectory.path, "objects"), 0); + objectDirectories.sort((a, b) => a.path.localeCompare(b.path)); + metadata.sort((a, b) => a.path.localeCompare(b.path)); + const result = { root, gitDirectory, commonDirectory, objectDirectories, metadata }; + return { ...result, fingerprint: sha256(JSON.stringify(result)) }; +} diff --git a/server/src/services/workspace-runtime.ts b/server/src/services/workspace-runtime.ts index ef07a03aa08..9f6d949d54e 100644 --- a/server/src/services/workspace-runtime.ts +++ b/server/src/services/workspace-runtime.ts @@ -8,7 +8,8 @@ import { getStorageService } from "../storage/index.js"; import type { StorageService } from "../storage/types.js"; import { sanitizeRuntimeServiceBaseEnv } from "./runtime-service-env.js"; import { verifiedRecordedIsolatedRemoteSource } from "./isolated-remote-workspace-source.js"; -import { assertCurrentWorkspaceSourceHeld, isolatedTaskSourceIdentity, managedProjectSourceIdentity, withWorkspaceSourceLocks } from "./workspace-source-lock.js"; +import { assertCurrentWorkspaceSourceHeld, workspaceSourceIdentities, withWorkspaceSourceLocks } from "./workspace-source-lock.js"; +import { localWorkspaceSourcePaths } from "./execution-workspace-source-roots.js"; export { sanitizeRuntimeServiceBaseEnv } from "./runtime-service-env.js"; import { execFile, spawn, type ChildProcess } from "node:child_process"; import { existsSync, lstatSync, readdirSync, readFileSync, realpathSync } from "node:fs"; @@ -7803,14 +7804,14 @@ async function withRuntimeServiceSourceLocks(input: { issue: ExecutionWorkspaceIssueRef | null; adapterEnv: Record; services: Record[]; }, action: () => Promise): Promise { if (!input.db || input.services.length === 0) return action(); - const paths = [input.workspace.cwd, input.workspace.baseCwd, ...input.services.map(service => + const paths = [...localWorkspaceSourcePaths({ cwd: input.workspace.cwd, providerRef: input.workspace.worktreePath }), + input.workspace.baseCwd, ...input.services.map(service => resolveRuntimeServiceReuseIdentity({ ...input, service, scopeType: "project_workspace", scopeId: input.workspace.workspaceId }).serviceCwd)]; const projectIds: string[] = [], issueIds: string[] = []; for (const cwd of new Set(paths)) { - const projectId = await managedProjectSourceIdentity(input.agent.companyId, cwd); - const issueId = await isolatedTaskSourceIdentity(input.agent.companyId, cwd); - if (projectId) projectIds.push(projectId); - if (issueId) issueIds.push(issueId); + const identities = await workspaceSourceIdentities(input.agent.companyId, cwd); + projectIds.push(...identities.projectIds); + issueIds.push(...identities.issueIds); } return withWorkspaceSourceLocks(input.db, { companyId: input.agent.companyId, projectIds, issueIds }, action); } diff --git a/server/src/services/workspace-source-lock.ts b/server/src/services/workspace-source-lock.ts index b9f0ac2b7f3..6dceccd90d8 100644 --- a/server/src/services/workspace-source-lock.ts +++ b/server/src/services/workspace-source-lock.ts @@ -4,6 +4,7 @@ import { sql } from "drizzle-orm"; import path from "node:path"; import { resolvePaperclipInstanceRoot } from "../home-paths.js"; import { canonicalRepositoryRoot } from "./execution-workspace-source-roots.js"; +import { readWorkspaceGitSourceRoots } from "./workspace-git-source-roots.js"; import { withWorkspaceLockKeepalive } from "./native-runtime/native-workspace-lock-keepalive.js"; type SourceTransaction = Parameters[0]>[0]; @@ -38,7 +39,7 @@ export async function withWorkspaceSourceLocks(db: Db, if (held?.db === db) { if (new Set([...held.keys, ...keys]).size > 128) throw new Error("workspace_source_scope_limit"); if (held.companyId !== input.companyId) throw new Error("workspace_source_company_mismatch"); - const onlyProjectKeys = keys.every(key => key.startsWith("managed_project_source:")); + const onlyProjectKeys = keys.filter(key => !held.keys.has(key)).every(key => key.startsWith("managed_project_source:")); const alreadyHasProject = [...held.keys].some(key => key.startsWith("managed_project_source:")); await acquire(held, !onlyProjectKeys || alreadyHasProject); return action(held.transaction); @@ -65,7 +66,8 @@ export async function managedProjectSourceIdentity(companyId: string, cwd: strin export async function isolatedTaskSourceIdentity(companyId: string, cwd: string): Promise { const root = await canonicalRepositoryRoot(resolvePaperclipInstanceRoot()); const relative = path.relative(root, await canonicalRepositoryRoot(cwd)).split(path.sep); - if (!["isolated-workspaces", "isolated-workspace-attempts"].includes(relative[0])) return null; + if (!["isolated-workspaces", "isolated-workspace-attempts", "isolated-repository-sources"].includes(relative[0])) return null; + if (relative[0] === "isolated-repository-sources" && relative[3] !== "checkout") return null; if (relative[1] !== companyId) throw new Error("workspace_source_company_mismatch"); return relative[2] && /^[a-f0-9-]{36}$/.test(relative[2]) ? relative[2] : null; } @@ -77,31 +79,35 @@ export async function retainManagedProjectSource(db: Db, companyId: string, cwd: await withWorkspaceSourceLocks(db, { companyId, projectIds: [projectId] }, async () => undefined, true); } +export async function workspaceSourceIdentities(companyId: string, cwd: string) { + cwd = await canonicalRepositoryRoot(cwd); + const git = await readWorkspaceGitSourceRoots(cwd, { ancestors: true }); + const roots = [cwd, ...(git ? [git.root, git.gitDirectory.path, git.commonDirectory.path, + ...git.objectDirectories.map(directory => directory.path)] : [])]; + const projectIds = new Set(), issueIds = new Set(), unownedPaths: string[] = []; + for (const target of new Set(roots)) { + const projectId = await managedProjectSourceIdentity(companyId, target); + const issueId = await isolatedTaskSourceIdentity(companyId, target); + if (projectId) projectIds.add(projectId); + if (issueId) issueIds.add(issueId); + if (!projectId && !issueId) unownedPaths.push(target); + } + return { projectIds: [...projectIds].sort(), issueIds: [...issueIds].sort(), unownedPaths, git }; +} + export async function retainCurrentWorkspaceSource(cwd: string): Promise { const scope = sourceLocks.getStore(); if (!scope) return; - const instance = await canonicalRepositoryRoot(resolvePaperclipInstanceRoot()); - const task = path.relative(instance, await canonicalRepositoryRoot(cwd)).split(path.sep); - if (["isolated-workspaces", "isolated-workspace-attempts"].includes(task[0]) && task.length >= 3) { - const issueId = await isolatedTaskSourceIdentity(scope.companyId, cwd); - if (issueId) await withWorkspaceSourceLocks(scope.db, { companyId: scope.companyId, issueIds: [issueId] }, async () => undefined, true); - return; - } - const root = await canonicalRepositoryRoot(path.join(resolvePaperclipInstanceRoot(), "projects")); - const target = await canonicalRepositoryRoot(cwd), relative = path.relative(root, target); - if (relative === ".." || relative.startsWith(`..${path.sep}`) || path.isAbsolute(relative)) return; - const [companyId, projectId] = relative.split(path.sep); - if (!companyId || !projectId || !/^[a-f0-9-]{36}$/.test(companyId) || !/^[a-f0-9-]{36}$/.test(projectId)) return; - await retainManagedProjectSource(scope.db, companyId, cwd); + const identities = await workspaceSourceIdentities(scope.companyId, cwd); + await withWorkspaceSourceLocks(scope.db, { companyId: scope.companyId, ...identities }, async () => undefined, true); } export async function assertCurrentWorkspaceSourceHeld(cwd: string): Promise { const scope = sourceLocks.getStore(); if (!scope) throw new Error("workspace_source_admission_scope_required"); - const projectId = await managedProjectSourceIdentity(scope.companyId, cwd); - const issueId = await isolatedTaskSourceIdentity(scope.companyId, cwd); - if (projectId && !scope.keys.has(`managed_project_source:${scope.companyId}:${projectId}`) - || issueId && !scope.keys.has(`isolated_repository_source:${scope.companyId}:${issueId}`)) { + const identities = await workspaceSourceIdentities(scope.companyId, cwd); + if (identities.projectIds.some(id => !scope.keys.has(`managed_project_source:${scope.companyId}:${id}`)) + || identities.issueIds.some(id => !scope.keys.has(`isolated_repository_source:${scope.companyId}:${id}`))) { throw new IsolatedRepositorySourceBusy("workspace_source_busy"); } }