Files
PaperClipAI/server/src/services/isolated-remote-workspace-source.ts
T
DottaandPaperclip 7ac99718a4 fix(workspaces): fence legacy sources for online migration (#19642)
## Thinking Path

> - Paperclip coordinates AI agents and preserves their saved work.
> - Remote workspace storage replaces legacy host directories with
durable snapshots.
> - Several legacy workspace records can name one physical project
directory.
> - A task lock alone cannot exclude a new task that selects that
directory.
> - Terminal recovery records can retain valid sources after physical
writers settle.
> - This change fences the source through its real preparation and
lifecycle owners.
> - Idle ordinary sources can then migrate while unrelated remote work
continues.

## Linked Issues or Issue Description

**What happened?**

Online migration rejects ordinary legacy directories when several
workspace records name the same path. A source inspection found 45
aliases across three project roots with no root Git marker or previously
imported nested child. The instance also had unrelated running remote
work. Requiring the whole instance to drain prevented useful migration
of these idle sources.

Terminal native sources and superseded finalization attempts also remain
blocked after their exact physical and durable owners settle. Their
files, provider allocations and recovery records must remain available
through the change of file authority.

**Expected behavior**

Migrate an idle ordinary source under a derived physical source fence.
Let unrelated object-backed turns continue. Preserve provider and
recovery obligations. Keep unknown writers, shared Git stores and
uncertain source state blocked.

**Steps to reproduce**

1. Create several local workspace records for one managed project
directory.
2. Give the aliases different source tasks. Leave their local writers
idle.
3. Keep a valid remote workspace turn running on another source.
4. Inspect or apply migration to an alias. The old owner returns
`overlapping_workspace_requires_maintenance`.
5. Retain a terminal native prepared source with a null coordinator
owner, or an old finalize attempt with an exact later committed receipt.
The old online path keeps these sources blocked.

**Paperclip version or commit**

Current tested head: `7478c817f09f7eed396922c0f7831af07ae6ff14`, based
on master `8621bcf53ee2e4bbefb7d193add42dd544ca06e8`. Related recovery
work: #19131.

**Deployment mode**

Hosted native Runner with remote workspace storage. Qualification uses
disposable local PostgreSQL databases, actual archive/storage owners and
synthetic source trees. The source observations were read-only.

## What Changed

- Retain the existing company/task source keys and add one physical
managed-project key. Heartbeat source selection, materialization, reopen
and lifecycle owners share it. Configured aliases of canonical task
directories retain their original task key.
- Fence runtime provision and each configured service CWD through both
startup owners. Check all unsettled service CWDs for physical source
overlap, even when workspace IDs differ. Runs with no services keep
their filesystem-free path.
- Derive a bounded ordinary alias closure. Lock every alias task and
lifecycle key. Reject foreign companies, changed membership, unproven
descendants, included Git metadata and other unresolved writer
obligations.
- Commit remote authority on the connection that holds source exclusion.
Refresh a stale selected local workspace after admission if migration
published remote authority. Publish activity after commit.
- Admit exact terminal native recovery sources under the existing
physical finalizer keys. Bind descriptors, allocations and owner
checkpoints to journaled fingerprints. Recheck them before capture and
publication.
- Add a separate company-scoped online entry for exact superseded
finalization receipts. Keep the offline entry's deployment prerequisites
and keep inventory read-only.
- Document the proof and the remaining Git/common-store extension. The
original files stay in place. No new scheduler, deployment action or
database migration is added.

## Verification

- Review found that the real reopen path could hold lifecycle exclusion
while waiting for a project source. The new actual nested-reopen
regression failed before the repair: lifecycle lock count was 1 while
another connection held the project fence. The repair acquires declared
and physical project sources before reconstruction and lifecycle. The
contention case, ordinary-fetch restore and runtime nested restore
passed. The lifecycle-worker restore first completed in 16.64 seconds
but exceeded its unchanged 15-second body limit; a bounded warm rerun
passed at that same limit. Both wrappers exited and completed cleanup.
The source type graph, module-boundary check and diff check pass. Final
7478 CI passed: 58 successful jobs, three skipped, one neutral and a
successful Snyk context. Fresh Greptile is 5/5 for exact7478, with no
unresolved threads. [Final
CI](https://github.com/paperclipai/paperclip/actions/runs/42318150314/job/128240631573).

- Full CI on `e4283b3452` passed the other server, build and type
checks. One exposure owner case failed: two execution workspaces
received one preview pair after source admission serialized their
starts. The first local seven-case run reproduced that failure; adding
the owner argument alone did not change the hash payload. The completed
repair binds run-scoped reuse to its execution-workspace owner. All
seven existing real-process/fake-broker cases then passed in 13.81
seconds, with wrapper exit 0 and cleanup complete. Public scope IDs and
explicit project/agent sharing are unchanged. The final7478 head passed
all required CI and fresh review gates.

- The service/source/restore owner canary passed 25 cases across four
suites before the final no-service fast-path guard (41.33 seconds). This
includes actual project service processes with a different project ID
and no execution-workspace ID. Their startup waits for source ownership,
migration stays blocked while they run, and the source becomes eligible
after their verified stop. The final three-case runtime canary passed at
the final source inputs (18.70 seconds). It covers that fast path and
both service-CWD variants. Both wrappers exited 0 and completed cleanup.
- The boundary-repair canary passed 9 cases across four suites at
`7c18f1c170`. It covers the real rehome/conversion gateway, project
command provisioning, future task selection, stale imported authority,
foreign-company binding, and existing standalone filesystem callers. The
shared PostgreSQL wrapper exited 0 and completed cleanup. Duration:
94.15 seconds.
- The P1 diagnostic passed 46 cases across five suites at the pre-rebase
P1 revision. It covers the actual importer with an export warning
created by `recordNativeFinalizationFailure`, strict
prepared-source/action bindings, finalizer locks, source/lifecycle
races, unrelated remote execution, and bare Git with external
alternates. It uses the repository shared PostgreSQL wrapper,
`--fsModuleCache`, and a local-only 90-second hook allowance. Duration:
245.44 seconds. No product timeout changed.
- The canonical retained-directory compatibility check passed 22 cases
across two suites. Missing archived task directories still import from
authenticated restore points; they do not need a present-directory
source proof.
- Both initial P1 regressions were observed before the fix. The real
finalizer warning made the importer ineligible. The old archive owner
accepted a nested bare repository borrowing external objects. The new
cases pass and preserve the original warning, prepared descriptor,
allocations and files.
- Prior attempts remain non-passes: the first 23-case canary passed on
an earlier head; its exact-head rerun hit 30-second database
setup/cleanup limits. A read of the owned PostgreSQL server found a
template clone waiting on `DataFileExtend` for 58 seconds. A broader
owner run was intentionally interrupted during slow existing heartbeat
setup. One diagnostic invocation used an invalid CLI option and executed
no tests. A later review canary passed 21 cases and failed the new real
restore contention case: cwd-only admission missed the declared project
of an external worktree. Its fixture also omitted the required
file-repository authorization callback and produced an unhandled
rejection after the failed assertion. The repaired run passed 22 cases,
exited 0 and completed cleanup. These earlier failures are not reported
as passes.
- The recovery owner passed 37 PostgreSQL cases, including 14 new and 23
existing finalized-recovery cases. The operation reconciliation owner
passed 40 cases. The initial archive run passed 103 cases and found a
missing closed diagnostic entry; the corrected diagnostic and expanded
Git rejection cases pass in the P1 run.
- The full server type graph passes with the Runner source exports
mapped locally because this checkout has no built Runner declarations.
The latest module-boundary check and `git diff --check` pass. Standard
package build/typecheck and broad final-head regression coverage remain
required in CI.
- The initial CI exposed missing admission scopes for standalone
filesystem APIs and a changed foreign-company error. The latest repair
keeps those APIs independent of a database and holds physical fences at
their actual database-backed mutation owners. The branch also includes
the existing master fix for unchanged Runner default-model assertions.
New review findings and new-head CI are being checked.
- The local canary command and bounded source selection are documented
in `doc/plans/2026-11-06-online-ordinary-source-fence.md`. Source
observations only read data. A fresh bounded metadata probe found 254
pending legacy records and zero unsettled local-process services. No
live import, hold, cancellation, provider action or source filesystem
write was performed for this change.

## Risks

- Run-scoped reuse now includes execution-workspace ownership in its
private key. Explicit project/agent sharing keeps its previous identity.
- A missing writer owner could make a source proof unsafe. Admission
derives physical aliases, includes future project selection and retains
all existing source/lifecycle/finalizer locks. Unknown and changed state
fail closed.
- Cross-company physical alias checks use bounded minimal-path pages.
Lexical-prefix filtering cannot safely replace them because a symlink
can hide a foreign alias. Unknown paths or more than 10,240 foreign
records leave the source explicitly unproven. More than 1,000
same-company local records also fail closed.
- The project key can serialize local preparation within one project. A
first project waits; additional project acquisition fails on contention
to avoid a lock cycle. Remote authority keeps its remote preparation
path.
- Recovery preservation does not prove provider termination or
successful file recovery. Existing provider allocations, checkpoints,
task intent and recovery actions remain authoritative.
- This slice does not finish shared Git migration. Git
administrative/common-object/alternates closure and exact nested
imported-child coverage remain the next owned extension. No global
writers-stopped shortcut is used online.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, tool use and code execution.
The host does not expose a more specific backend model ID or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 18:42:45 -06:00

129 lines
6.5 KiB
TypeScript

import type { Db } from "@paperclipai/db";
import path from "node:path";
import fs from "node:fs/promises";
import { execFile as execFileCallback } from "node:child_process";
import { promisify } from "node:util";
import type { TaskWorkspaceIntent } from "@paperclipai/shared";
import { isGitHubHttpsRemoteUrl, type GitRemoteAuthProvider } from "./git-credentials.js";
import { resolvePaperclipInstanceRoot } from "../home-paths.js";
import { materializeManagedProjectWorkspace } from "./managed-repository-checkout.js";
import { measureRunPhase } from "./execution-preparation-timing.js";
import { isRestrictedWorkspaceSourceDenial } from "./workspace-source-restriction.js";
const execFile = promisify(execFileCallback);
export { IsolatedRepositorySourceBusy, withIsolatedRepositorySourceLock } from "./workspace-source-lock.js";
export function isolatedRemoteSourcePath(companyId: string, issueId: string): string {
if ([companyId, issueId].some(id => !/^[a-zA-Z0-9_-]+$/.test(id))) {
throw new Error("Invalid isolated repository source identity");
}
return path.join(resolvePaperclipInstanceRoot(), "isolated-repository-sources", companyId, issueId, "checkout");
}
export function recordedIsolatedRemoteSource(input: {
companyId?: string;
sourceIssueId: string | null;
repoUrl: string | null;
metadata?: Record<string, unknown> | null;
}): string | null {
const value = input.metadata?.isolatedRemoteSource;
if (value === undefined) return null;
const source = value as Record<string, unknown> | null;
if (!source || source.schema !== "paperclip.isolated-remote-source.v1"
|| !input.companyId || !input.sourceIssueId || source.companyId !== input.companyId
|| source.issueId !== input.sourceIssueId || source.repoUrl !== input.repoUrl) {
throw new Error("Recorded isolated repository source identity changed");
}
return isolatedRemoteSourcePath(input.companyId, input.sourceIssueId);
}
async function assertRemoteSourceIdentity(cwd: string, repoUrl: string) {
const root = resolvePaperclipInstanceRoot();
const canonicalCwd = await fs.realpath(cwd);
if (canonicalCwd !== path.join(await fs.realpath(root), path.relative(root, cwd))) {
throw new Error("Isolated repository source path changed");
}
const gitDir = path.join(canonicalCwd, ".git");
const entry = await fs.lstat(gitDir);
if (!entry.isDirectory() || entry.isSymbolicLink()) throw new Error("Isolated repository source Git storage changed");
const options = { timeout: 10_000, maxBuffer: 16 * 1024 };
const common = await execFile("git", ["-C", canonicalCwd, "rev-parse", "--path-format=absolute", "--git-common-dir"], options);
if (common.stdout.trim() !== gitDir) throw new Error("Isolated repository source Git storage changed");
const origin = await execFile("git", ["-C", canonicalCwd, "config", "--local", "--null", "--get-all", "remote.origin.url"], options);
if (origin.stdout !== `${repoUrl}\0`) throw new Error("Isolated repository source identity changed");
}
export async function verifiedRecordedIsolatedRemoteSource(input: Parameters<typeof recordedIsolatedRemoteSource>[0]): Promise<string | null> {
const cwd = recordedIsolatedRemoteSource(input);
if (cwd) await assertRemoteSourceIdentity(cwd, input.repoUrl!);
return cwd;
}
export function canIsolateRemoteWorkspaceSource(input: {
error: unknown;
repoUrl: string | null;
reuseExisting: boolean;
hasExistingWorkspace: boolean;
selectedProjectWorkspaceId: string | null;
selection: TaskWorkspaceIntent | null | undefined;
strategy: Record<string, unknown>;
trustPresetKind: string;
}): boolean {
const branchTemplate = input.strategy.branchTemplate;
const selectsFixedBranch = typeof branchTemplate === "string" && branchTemplate.length > 0
&& !/\{\{\s*(?:issue\.(?:id|identifier|title)|slug)\s*\}\}/.test(branchTemplate);
if (input.trustPresetKind !== "standard" || !isRestrictedWorkspaceSourceDenial(input.error)
|| input.reuseExisting || input.hasExistingWorkspace || input.strategy.existingBranch || selectsFixedBranch
|| (input.selection && !["project", "task_default", "recovery"].includes(input.selection.source))
|| (input.selectedProjectWorkspaceId && (input.selection?.selection.kind !== "configured_source"
|| input.selection.selection.projectWorkspaceId !== input.selectedProjectWorkspaceId))) return false;
if (!input.repoUrl) return false;
try {
const url = new URL(input.repoUrl);
return ["https:", "ssh:"].includes(url.protocol) && !!url.hostname && !url.password
&& (url.protocol === "ssh:" || !url.username);
} catch { return false; }
}
export async function materializeIsolatedRemoteWorkspaceSource(input: {
companyId: string;
issueId: string;
repoUrl: string;
resolveGitAuth: GitRemoteAuthProvider;
}): Promise<string> {
const cwd = isolatedRemoteSourcePath(input.companyId, input.issueId);
const canonicalCwd = await measureRunPhase("workspace.materialize.directories", async () => {
const root = resolvePaperclipInstanceRoot();
await fs.mkdir(root, { recursive: true });
const canonicalRoot = await fs.realpath(root);
let parent = canonicalRoot;
for (const segment of ["isolated-repository-sources", input.companyId, input.issueId]) {
parent = path.join(parent, segment);
await fs.mkdir(parent, { mode: 0o700 }).catch((error: NodeJS.ErrnoException) => {
if (error.code !== "EEXIST") throw error;
});
const entry = await fs.lstat(parent);
if (!entry.isDirectory() || entry.isSymbolicLink() || await fs.realpath(parent) !== parent) {
throw new Error("Isolated repository source path is not a private directory");
}
}
const checkout = path.join(parent, "checkout");
const entry = await fs.lstat(checkout).catch((error: NodeJS.ErrnoException) => {
if (error.code === "ENOENT") return null;
throw error;
});
if (entry && (!entry.isDirectory() || entry.isSymbolicLink())) {
throw new Error("Isolated repository source cannot be a symbolic link");
}
return checkout;
});
const result = await materializeManagedProjectWorkspace(canonicalCwd, {
repoUrl: input.repoUrl, resolveGitAuth: input.resolveGitAuth,
objectCacheCompanyDirectory: isGitHubHttpsRemoteUrl(input.repoUrl)
? path.join(await fs.realpath(resolvePaperclipInstanceRoot()), "repository-object-cache", input.companyId) : undefined,
});
if (result.warning) throw new Error("Isolated repository source is not a Git checkout");
await measureRunPhase("workspace.materialize.source_identity", () => assertRemoteSourceIdentity(cwd, input.repoUrl));
return cwd;
}