mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
## Thinking Path > - Paperclip coordinates AI agents and preserves their saved work. > - Remote workspace storage replaces legacy host directories with durable snapshots. > - Several legacy workspace records can name one physical project directory. > - A task lock alone cannot exclude a new task that selects that directory. > - Terminal recovery records can retain valid sources after physical writers settle. > - This change fences the source through its real preparation and lifecycle owners. > - Idle ordinary sources can then migrate while unrelated remote work continues. ## Linked Issues or Issue Description **What happened?** Online migration rejects ordinary legacy directories when several workspace records name the same path. A source inspection found 45 aliases across three project roots with no root Git marker or previously imported nested child. The instance also had unrelated running remote work. Requiring the whole instance to drain prevented useful migration of these idle sources. Terminal native sources and superseded finalization attempts also remain blocked after their exact physical and durable owners settle. Their files, provider allocations and recovery records must remain available through the change of file authority. **Expected behavior** Migrate an idle ordinary source under a derived physical source fence. Let unrelated object-backed turns continue. Preserve provider and recovery obligations. Keep unknown writers, shared Git stores and uncertain source state blocked. **Steps to reproduce** 1. Create several local workspace records for one managed project directory. 2. Give the aliases different source tasks. Leave their local writers idle. 3. Keep a valid remote workspace turn running on another source. 4. Inspect or apply migration to an alias. The old owner returns `overlapping_workspace_requires_maintenance`. 5. Retain a terminal native prepared source with a null coordinator owner, or an old finalize attempt with an exact later committed receipt. The old online path keeps these sources blocked. **Paperclip version or commit** Current tested head: `7478c817f09f7eed396922c0f7831af07ae6ff14`, based on master `8621bcf53ee2e4bbefb7d193add42dd544ca06e8`. Related recovery work: #19131. **Deployment mode** Hosted native Runner with remote workspace storage. Qualification uses disposable local PostgreSQL databases, actual archive/storage owners and synthetic source trees. The source observations were read-only. ## What Changed - Retain the existing company/task source keys and add one physical managed-project key. Heartbeat source selection, materialization, reopen and lifecycle owners share it. Configured aliases of canonical task directories retain their original task key. - Fence runtime provision and each configured service CWD through both startup owners. Check all unsettled service CWDs for physical source overlap, even when workspace IDs differ. Runs with no services keep their filesystem-free path. - Derive a bounded ordinary alias closure. Lock every alias task and lifecycle key. Reject foreign companies, changed membership, unproven descendants, included Git metadata and other unresolved writer obligations. - Commit remote authority on the connection that holds source exclusion. Refresh a stale selected local workspace after admission if migration published remote authority. Publish activity after commit. - Admit exact terminal native recovery sources under the existing physical finalizer keys. Bind descriptors, allocations and owner checkpoints to journaled fingerprints. Recheck them before capture and publication. - Add a separate company-scoped online entry for exact superseded finalization receipts. Keep the offline entry's deployment prerequisites and keep inventory read-only. - Document the proof and the remaining Git/common-store extension. The original files stay in place. No new scheduler, deployment action or database migration is added. ## Verification - Review found that the real reopen path could hold lifecycle exclusion while waiting for a project source. The new actual nested-reopen regression failed before the repair: lifecycle lock count was 1 while another connection held the project fence. The repair acquires declared and physical project sources before reconstruction and lifecycle. The contention case, ordinary-fetch restore and runtime nested restore passed. The lifecycle-worker restore first completed in 16.64 seconds but exceeded its unchanged 15-second body limit; a bounded warm rerun passed at that same limit. Both wrappers exited and completed cleanup. The source type graph, module-boundary check and diff check pass. Final 7478 CI passed: 58 successful jobs, three skipped, one neutral and a successful Snyk context. Fresh Greptile is 5/5 for exact7478, with no unresolved threads. [Final CI](https://github.com/paperclipai/paperclip/actions/runs/42318150314/job/128240631573). - Full CI on `e4283b3452` passed the other server, build and type checks. One exposure owner case failed: two execution workspaces received one preview pair after source admission serialized their starts. The first local seven-case run reproduced that failure; adding the owner argument alone did not change the hash payload. The completed repair binds run-scoped reuse to its execution-workspace owner. All seven existing real-process/fake-broker cases then passed in 13.81 seconds, with wrapper exit 0 and cleanup complete. Public scope IDs and explicit project/agent sharing are unchanged. The final7478 head passed all required CI and fresh review gates. - The service/source/restore owner canary passed 25 cases across four suites before the final no-service fast-path guard (41.33 seconds). This includes actual project service processes with a different project ID and no execution-workspace ID. Their startup waits for source ownership, migration stays blocked while they run, and the source becomes eligible after their verified stop. The final three-case runtime canary passed at the final source inputs (18.70 seconds). It covers that fast path and both service-CWD variants. Both wrappers exited 0 and completed cleanup. - The boundary-repair canary passed 9 cases across four suites at `7c18f1c170`. It covers the real rehome/conversion gateway, project command provisioning, future task selection, stale imported authority, foreign-company binding, and existing standalone filesystem callers. The shared PostgreSQL wrapper exited 0 and completed cleanup. Duration: 94.15 seconds. - The P1 diagnostic passed 46 cases across five suites at the pre-rebase P1 revision. It covers the actual importer with an export warning created by `recordNativeFinalizationFailure`, strict prepared-source/action bindings, finalizer locks, source/lifecycle races, unrelated remote execution, and bare Git with external alternates. It uses the repository shared PostgreSQL wrapper, `--fsModuleCache`, and a local-only 90-second hook allowance. Duration: 245.44 seconds. No product timeout changed. - The canonical retained-directory compatibility check passed 22 cases across two suites. Missing archived task directories still import from authenticated restore points; they do not need a present-directory source proof. - Both initial P1 regressions were observed before the fix. The real finalizer warning made the importer ineligible. The old archive owner accepted a nested bare repository borrowing external objects. The new cases pass and preserve the original warning, prepared descriptor, allocations and files. - Prior attempts remain non-passes: the first 23-case canary passed on an earlier head; its exact-head rerun hit 30-second database setup/cleanup limits. A read of the owned PostgreSQL server found a template clone waiting on `DataFileExtend` for 58 seconds. A broader owner run was intentionally interrupted during slow existing heartbeat setup. One diagnostic invocation used an invalid CLI option and executed no tests. A later review canary passed 21 cases and failed the new real restore contention case: cwd-only admission missed the declared project of an external worktree. Its fixture also omitted the required file-repository authorization callback and produced an unhandled rejection after the failed assertion. The repaired run passed 22 cases, exited 0 and completed cleanup. These earlier failures are not reported as passes. - The recovery owner passed 37 PostgreSQL cases, including 14 new and 23 existing finalized-recovery cases. The operation reconciliation owner passed 40 cases. The initial archive run passed 103 cases and found a missing closed diagnostic entry; the corrected diagnostic and expanded Git rejection cases pass in the P1 run. - The full server type graph passes with the Runner source exports mapped locally because this checkout has no built Runner declarations. The latest module-boundary check and `git diff --check` pass. Standard package build/typecheck and broad final-head regression coverage remain required in CI. - The initial CI exposed missing admission scopes for standalone filesystem APIs and a changed foreign-company error. The latest repair keeps those APIs independent of a database and holds physical fences at their actual database-backed mutation owners. The branch also includes the existing master fix for unchanged Runner default-model assertions. New review findings and new-head CI are being checked. - The local canary command and bounded source selection are documented in `doc/plans/2026-11-06-online-ordinary-source-fence.md`. Source observations only read data. A fresh bounded metadata probe found 254 pending legacy records and zero unsettled local-process services. No live import, hold, cancellation, provider action or source filesystem write was performed for this change. ## Risks - Run-scoped reuse now includes execution-workspace ownership in its private key. Explicit project/agent sharing keeps its previous identity. - A missing writer owner could make a source proof unsafe. Admission derives physical aliases, includes future project selection and retains all existing source/lifecycle/finalizer locks. Unknown and changed state fail closed. - Cross-company physical alias checks use bounded minimal-path pages. Lexical-prefix filtering cannot safely replace them because a symlink can hide a foreign alias. Unknown paths or more than 10,240 foreign records leave the source explicitly unproven. More than 1,000 same-company local records also fail closed. - The project key can serialize local preparation within one project. A first project waits; additional project acquisition fails on contention to avoid a lock cycle. Remote authority keeps its remote preparation path. - Recovery preservation does not prove provider termination or successful file recovery. Existing provider allocations, checkpoints, task intent and recovery actions remain authoritative. - This slice does not finish shared Git migration. Git administrative/common-object/alternates closure and exact nested imported-child coverage remain the next owned extension. No global writers-stopped shortcut is used online. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use and code execution. The host does not expose a more specific backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
108 lines
6.4 KiB
TypeScript
108 lines
6.4 KiB
TypeScript
import { AsyncLocalStorage } from "node:async_hooks";
|
|
import { withDedicatedDbConnection, type Db } from "@paperclipai/db";
|
|
import { sql } from "drizzle-orm";
|
|
import path from "node:path";
|
|
import { resolvePaperclipInstanceRoot } from "../home-paths.js";
|
|
import { canonicalRepositoryRoot } from "./execution-workspace-source-roots.js";
|
|
import { withWorkspaceLockKeepalive } from "./native-runtime/native-workspace-lock-keepalive.js";
|
|
|
|
type SourceTransaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
|
|
type SourceLockScope = { db: Db; companyId: string; keys: Set<string>; transaction: SourceTransaction };
|
|
const sourceLocks = new AsyncLocalStorage<SourceLockScope>();
|
|
export class IsolatedRepositorySourceBusy extends Error {}
|
|
|
|
export function withIsolatedRepositorySourceLock<T>(db: Db, companyId: string, issueId: string,
|
|
action: () => Promise<T>, tryOnly = false): Promise<T> {
|
|
return withWorkspaceSourceLocks(db, { companyId, issueIds: [issueId] }, action, tryOnly);
|
|
}
|
|
|
|
export async function withWorkspaceSourceLocks<T>(db: Db,
|
|
input: { companyId: string; issueIds?: readonly string[]; projectIds?: readonly string[] },
|
|
action: (transaction: SourceTransaction) => Promise<T>, tryOnly = false): Promise<T> {
|
|
const keys = [...new Set([
|
|
...(input.issueIds ?? []).map(id => `isolated_repository_source:${input.companyId}:${id}`),
|
|
...(input.projectIds ?? []).map(id => `managed_project_source:${input.companyId}:${id}`),
|
|
])].sort();
|
|
if (keys.length > 128) throw new Error("workspace_source_scope_limit");
|
|
const held = sourceLocks.getStore();
|
|
const acquire = async (scope: SourceLockScope, nonblocking: boolean) => {
|
|
for (const key of keys) {
|
|
if (scope.keys.has(key)) continue;
|
|
const lock = nonblocking ? sql`pg_try_advisory_xact_lock(hashtextextended(${key}, 0))`
|
|
: sql`pg_advisory_xact_lock(hashtextextended(${key}, 0))`;
|
|
const rows = await scope.transaction.execute(sql`select ${lock} as acquired`);
|
|
if (nonblocking && !rows[0]?.acquired) throw new IsolatedRepositorySourceBusy("workspace_source_busy");
|
|
scope.keys.add(key);
|
|
}
|
|
};
|
|
if (held?.db === db) {
|
|
if (new Set([...held.keys, ...keys]).size > 128) throw new Error("workspace_source_scope_limit");
|
|
if (held.companyId !== input.companyId) throw new Error("workspace_source_company_mismatch");
|
|
const onlyProjectKeys = keys.every(key => key.startsWith("managed_project_source:"));
|
|
const alreadyHasProject = [...held.keys].some(key => key.startsWith("managed_project_source:"));
|
|
await acquire(held, !onlyProjectKeys || alreadyHasProject);
|
|
return action(held.transaction);
|
|
}
|
|
return withDedicatedDbConnection(db, connection => connection.transaction(async transaction => {
|
|
const scope = { db, companyId: input.companyId, transaction, keys: new Set<string>() };
|
|
await acquire(scope, tryOnly);
|
|
const rows = await transaction.execute(sql`select setting::bigint as idle_timeout_ms from pg_settings where name = 'idle_in_transaction_session_timeout'`);
|
|
return withWorkspaceLockKeepalive(transaction, Number(rows[0]?.idle_timeout_ms),
|
|
() => sourceLocks.run(scope, () => action(transaction)));
|
|
}));
|
|
}
|
|
|
|
export async function managedProjectSourceIdentity(companyId: string, cwd: string): Promise<string | null> {
|
|
const root = await canonicalRepositoryRoot(path.join(resolvePaperclipInstanceRoot(), "projects"));
|
|
const target = await canonicalRepositoryRoot(cwd);
|
|
const relative = path.relative(root, target);
|
|
if (relative === ".." || relative.startsWith(`..${path.sep}`) || path.isAbsolute(relative)) return null;
|
|
const [owner, projectId] = relative.split(path.sep);
|
|
if (owner !== companyId) throw new Error("workspace_source_company_mismatch");
|
|
return projectId && /^[a-f0-9-]{36}$/.test(projectId) ? projectId : null;
|
|
}
|
|
|
|
export async function isolatedTaskSourceIdentity(companyId: string, cwd: string): Promise<string | null> {
|
|
const root = await canonicalRepositoryRoot(resolvePaperclipInstanceRoot());
|
|
const relative = path.relative(root, await canonicalRepositoryRoot(cwd)).split(path.sep);
|
|
if (!["isolated-workspaces", "isolated-workspace-attempts"].includes(relative[0])) return null;
|
|
if (relative[1] !== companyId) throw new Error("workspace_source_company_mismatch");
|
|
return relative[2] && /^[a-f0-9-]{36}$/.test(relative[2]) ? relative[2] : null;
|
|
}
|
|
|
|
export async function retainManagedProjectSource(db: Db, companyId: string, cwd: string): Promise<void> {
|
|
const projectId = await managedProjectSourceIdentity(companyId, cwd);
|
|
if (!projectId) return;
|
|
if (sourceLocks.getStore()?.db !== db) throw new Error("workspace_source_admission_scope_required");
|
|
await withWorkspaceSourceLocks(db, { companyId, projectIds: [projectId] }, async () => undefined, true);
|
|
}
|
|
|
|
export async function retainCurrentWorkspaceSource(cwd: string): Promise<void> {
|
|
const scope = sourceLocks.getStore();
|
|
if (!scope) return;
|
|
const instance = await canonicalRepositoryRoot(resolvePaperclipInstanceRoot());
|
|
const task = path.relative(instance, await canonicalRepositoryRoot(cwd)).split(path.sep);
|
|
if (["isolated-workspaces", "isolated-workspace-attempts"].includes(task[0]) && task.length >= 3) {
|
|
const issueId = await isolatedTaskSourceIdentity(scope.companyId, cwd);
|
|
if (issueId) await withWorkspaceSourceLocks(scope.db, { companyId: scope.companyId, issueIds: [issueId] }, async () => undefined, true);
|
|
return;
|
|
}
|
|
const root = await canonicalRepositoryRoot(path.join(resolvePaperclipInstanceRoot(), "projects"));
|
|
const target = await canonicalRepositoryRoot(cwd), relative = path.relative(root, target);
|
|
if (relative === ".." || relative.startsWith(`..${path.sep}`) || path.isAbsolute(relative)) return;
|
|
const [companyId, projectId] = relative.split(path.sep);
|
|
if (!companyId || !projectId || !/^[a-f0-9-]{36}$/.test(companyId) || !/^[a-f0-9-]{36}$/.test(projectId)) return;
|
|
await retainManagedProjectSource(scope.db, companyId, cwd);
|
|
}
|
|
|
|
export async function assertCurrentWorkspaceSourceHeld(cwd: string): Promise<void> {
|
|
const scope = sourceLocks.getStore();
|
|
if (!scope) throw new Error("workspace_source_admission_scope_required");
|
|
const projectId = await managedProjectSourceIdentity(scope.companyId, cwd);
|
|
const issueId = await isolatedTaskSourceIdentity(scope.companyId, cwd);
|
|
if (projectId && !scope.keys.has(`managed_project_source:${scope.companyId}:${projectId}`)
|
|
|| issueId && !scope.keys.has(`isolated_repository_source:${scope.companyId}:${issueId}`)) {
|
|
throw new IsolatedRepositorySourceBusy("workspace_source_busy");
|
|
}
|
|
}
|