mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-30 13:20:54 +01:00
## Thinking Path > - Paperclip manages agent work and its execution workspaces. > - Workspace recovery scans candidates and retains deadlines for unfinished work. > - Rounded database timestamps can prevent a scan from advancing. > - Separate clock reads can give one retry obligation different deadlines. > - Repeated inspection must preserve both scan progress and the saved retry time. > - This PR keeps full cursor precision and computes each retry deadline once. > - Recovery can finish its scan and wait until work is actually due. ## Linked Issues or Issue Description **What happened?** Terminal workspace recovery can repeatedly inspect the oldest skipped candidate when its timestamp contains microseconds. Eligible workspaces behind it never reach cleanup. Separately, nested-clone migration and isolated-repository rehome recovery use separate clock reads for their retry gate and advertised deadline. A later inspection can move the advertised deadline backward. **Expected behavior** Each bounded scan advances using the complete timestamp and ID. The retry gate and advertised deadline represent the same saved time, including when the clock advances between calls. **Steps to reproduce** For cursor starvation, place two skipped workspaces with microsecond timestamps before an eligible terminal workspace and use a one-row page. For deadline drift, inject a clock that advances one millisecond per read, run disabled migration or rehome recovery, then inspect it again before the retry time. Stacked on #17497. Related cursor fixes #17479 and #17482 have separate owners. This PR owns workspace recovery cursor and retry precision. ## What Changed - Preserve the database timestamp as text in the private timestamp/ID cursor. - Compute one retry time per migration or rehome outcome and reuse it in the existing maps. - Test the cursor with real PostgreSQL microsecond timestamps and the retry boundary with a deterministic advancing clock. - Support validated disposable external test databases in the affected fixtures, retaining embedded fallback. - Document the evidence in `doc/idle-terminal-workspace-pagination.md`. ## Verification Current test revision: `9d846c84e76c90893ab62a09548c11bbdb627771`, followed by docs-only head `a34dc50de3`, which received both 5/5 reviews and passing CI. This candidate normally integrates landed approval retention `553a004926`. Every runtime and acceptance-test source matches that tested/reviewed tree; only four independent Runner E2E accounting files and the ledger differ. Migration metadata is unchanged. Existing local checks are reused for their unchanged source; fresh final-head CI and reviews are required against master. This remains the terminal-precision duty. - **136 cases pass** in four complete suites: execution workspace service, terminal worker, isolated repository rehome and nested-clone migration. Server typecheck and production build pass with the current Runner vendor. - The preceding `f0f0de1f6f` run also passed all 136 on three newly created, unmigrated PostgreSQL databases. The suites themselves applied migrations. The current run reuses those clean fixtures after integrating the parent's approval wake repair; it is not presented as a fresh-database run. - The regressions retain real PostgreSQL microsecond timestamps and an advancing clock. Before repair, skipped rows starved eligible cleanup and early inspection moved a saved retry by one millisecond. Current assertions preserve exact progress/deadline behavior; no timeout or business policy is weakened. - Earlier external-fixture setup tests failed with missing company/restore-point schema. Their setup repair and red/green record remain in `doc/idle-terminal-workspace-pagination.md`. This PR adds no migration or new queue. - Earlier stack integration receipts and failures remain in the stack ledger and `doc/idle-approved-tool-recovery.md`; they are historical parent evidence, not this head's acceptance. No full local repository suite, deployed suspension, fleet cost or live provider qualification is claimed. Commands: server Vitest for `execution-workspaces-service`, `terminal-workspace-work`, `isolated-repository-rehome` and `nested-clone-migration`; server typecheck/build. Fresh exact-head CI and Greptile/Ponytail 5/5 remain required before merge in dependency order. ## Risks The cursor remains private query state. Retry intervals, eligibility, cleanup authority, and shutdown behavior are unchanged. No schema, queue, external alarm, or additional polling loop is introduced. The tests establish workspace recovery behavior, not host suspension or cost savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
531 lines
36 KiB
TypeScript
531 lines
36 KiB
TypeScript
import { promises as fs } from "node:fs";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import { execFileSync } from "node:child_process";
|
|
import { randomUUID } from "node:crypto";
|
|
import { pathToFileURL } from "node:url";
|
|
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import { eq, sql } from "drizzle-orm";
|
|
import { agents, applyPendingMigrations, companies, createDb, executionWorkspaceRestorePoints, executionWorkspaces, heartbeatRuns, issues, projects, projectWorkspaces } from "@paperclipai/db";
|
|
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
|
import { materializeManagedProjectWorkspace } from "../services/managed-repository-checkout.js";
|
|
import { materializeIsolatedRemoteWorkspaceSource } from "../services/isolated-remote-workspace-source.js";
|
|
import { realizeExecutionWorkspace } from "../services/workspace-runtime.js";
|
|
import { executionWorkspaceService } from "../services/execution-workspaces.js";
|
|
import { prepareProjectRepositoryWorkspaces } from "../services/heartbeat/workspaces.js";
|
|
import { expireReconstructableGitWorkspace } from "../services/git-workspace-expiration.js";
|
|
import { currentRestorePointSnapshot, restorePointHistoryDigest } from "../services/git-restore-point.js";
|
|
import { nestedRepositoryBaseCandidates, nestedWorktreeBranchPrefix, readLinkedNestedCheckout, withNestedRepositoryBaseLock } from "../services/nested-repository-worktrees.js";
|
|
import { listNestedCloneMigrationNames, migrateNestedRepositoryClone, nextNestedCloneMigrationBatch, type NestedCloneMigrationOptions } from "../services/nested-clone-migration.js";
|
|
import { getStorageService } from "../storage/index.js";
|
|
|
|
const gitEnv = { ...process.env, GIT_AUTHOR_NAME: "Fixture", GIT_AUTHOR_EMAIL: "fixture@example.test",
|
|
GIT_COMMITTER_NAME: "Fixture", GIT_COMMITTER_EMAIL: "fixture@example.test", GIT_TERMINAL_PROMPT: "0" };
|
|
const git = (cwd: string, ...args: string[]) => execFileSync("git", args, { cwd, env: gitEnv, encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }).trim();
|
|
const exists = (target: string) => fs.lstat(target).then(() => true, () => false);
|
|
const DAY_MS = 24 * 60 * 60 * 1000;
|
|
const noAuth = async () => null;
|
|
|
|
describe("nested clone migration batches", () => {
|
|
it("puts pending recoveries first, rotates through the rest and stops retrying an exhausted rotation until the list changes", () => {
|
|
const listed = Array.from({ length: 10 }, (_, index) => `repo-${index.toString(16).padStart(12, "0")}`);
|
|
const first = nextNestedCloneMigrationBatch(listed, [listed[9]!], undefined, 4);
|
|
expect(first).toMatchObject({ names: [listed[9], listed[0], listed[1], listed[2]], overflow: true, retryOverflow: true });
|
|
const second = nextNestedCloneMigrationBatch(listed, [], first.rotation, 4);
|
|
expect(second).toMatchObject({ names: [listed[3], listed[4], listed[5], listed[6]], retryOverflow: true });
|
|
const third = nextNestedCloneMigrationBatch(listed, [], second.rotation, 4);
|
|
expect(third).toMatchObject({ names: [listed[7], listed[8], listed[0], listed[1]], retryOverflow: false });
|
|
expect([...third.rotation.visited]).toEqual([]);
|
|
const fourth = nextNestedCloneMigrationBatch(listed, [], third.rotation, 4);
|
|
expect(fourth).toMatchObject({ names: listed.slice(0, 4), retryOverflow: false });
|
|
const grown = [...listed, "repo-aaaaaaaaaaaa"];
|
|
expect(nextNestedCloneMigrationBatch(grown, [], fourth.rotation, 4)).toMatchObject({ retryOverflow: true });
|
|
expect(nextNestedCloneMigrationBatch(listed.slice(0, 3), [], undefined, 4)).toMatchObject({ names: listed.slice(0, 3), overflow: false, retryOverflow: false });
|
|
});
|
|
});
|
|
|
|
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
|
|
|
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_NESTED_CLONE_TEST_DATABASE_URL");
|
|
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
|
const describeEmbeddedPostgres = support.supported ? describe : describe.skip;
|
|
describeEmbeddedPostgres("nested repository clone migration", () => {
|
|
let db: ReturnType<typeof createDb>;
|
|
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
|
let root: string;
|
|
const previousHome = process.env.PAPERCLIP_HOME;
|
|
const previousInstance = process.env.PAPERCLIP_INSTANCE_ID;
|
|
beforeAll(async () => {
|
|
if (external) await applyPendingMigrations(external);
|
|
else database = await startEmbeddedPostgresTestDatabase("nested-clone-migration-");
|
|
db = createDb(external ?? database!.connectionString);
|
|
root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-nested-migration-"));
|
|
process.env.PAPERCLIP_HOME = path.join(root, "home");
|
|
process.env.PAPERCLIP_INSTANCE_ID = "default";
|
|
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "51".repeat(32));
|
|
});
|
|
afterEach(async () => {
|
|
await db.delete(executionWorkspaceRestorePoints);
|
|
await db.execute(sql`TRUNCATE TABLE companies CASCADE`);
|
|
});
|
|
afterAll(async () => {
|
|
vi.unstubAllEnvs();
|
|
if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome;
|
|
if (previousInstance === undefined) delete process.env.PAPERCLIP_INSTANCE_ID; else process.env.PAPERCLIP_INSTANCE_ID = previousInstance;
|
|
await db.$client.end({ timeout: 1 }); await database?.cleanup(); await fs.rm(root, { recursive: true, force: true });
|
|
});
|
|
|
|
async function repository(location: string, name: string, files: Record<string, string>) {
|
|
const author = path.join(location, `${name}-author`), remote = path.join(location, `${name}.git`);
|
|
await fs.mkdir(author, { recursive: true });
|
|
git(author, "init", "--initial-branch=main");
|
|
for (const [file, contents] of Object.entries(files)) await fs.writeFile(path.join(author, file), contents);
|
|
git(author, "add", "."); git(author, "commit", "-m", `${name} A`);
|
|
git(location, "clone", "--bare", author, remote);
|
|
return { author, remote, url: pathToFileURL(remote).href, commit: git(author, "rev-parse", "HEAD") };
|
|
}
|
|
|
|
async function fixture(options: { issueStatus?: "in_progress" | "done"; sharedBase?: boolean } = {}) {
|
|
const companyId = randomUUID(), projectId = randomUUID();
|
|
const location = path.join(root, companyId);
|
|
const main = await repository(location, "remote", { "marker.txt": "commit A\n", ".gitignore": "node_modules/\n.claude/settings.local.json\n" });
|
|
const nested = await repository(location, "nested", { "nested.txt": "nested A\n", "remove-me.txt": "tracked\n",
|
|
".gitignore": "node_modules/\n.env\n.claude/settings.local.json\n*.log\n" });
|
|
await db.insert(companies).values({ id: companyId, name: "Nested migration fixture", issuePrefix: `NM${companyId.slice(0, 6)}` });
|
|
await db.insert(projects).values({ id: projectId, companyId, name: "Fixture project" });
|
|
const [source] = await db.insert(projectWorkspaces).values({ companyId, projectId, name: "Nested repository", repoUrl: nested.url, repoRef: "main" }).returning();
|
|
const candidates = nestedRepositoryBaseCandidates({ companyId, projectId, repoUrl: nested.url });
|
|
if (options.sharedBase !== false) await materializeManagedProjectWorkspace(candidates[0]!, { repoUrl: nested.url });
|
|
const done = options.issueStatus === "done";
|
|
const task = async (label: string) => {
|
|
const issueId = randomUUID(), workspaceId = randomUUID();
|
|
await db.insert(issues).values({ id: issueId, companyId, projectId, title: `Task ${label}`, status: done ? "done" : "in_progress",
|
|
...(done ? { completedAt: new Date(Date.now() - 2 * DAY_MS) } : {}) });
|
|
const baseCwd = await materializeIsolatedRemoteWorkspaceSource({ companyId, issueId, repoUrl: main.url, resolveGitAuth: noAuth });
|
|
const base = { baseCwd, source: "task_session" as const, projectId: null, workspaceId: null, repoUrl: main.url, repoRef: "origin/main" };
|
|
const realized = await realizeExecutionWorkspace({ base, issue: { id: issueId, identifier: `NM-${label}`, title: "Fixture" },
|
|
agent: { id: null, name: "Fixture", companyId }, skipHostProvision: true,
|
|
config: { workspaceStrategy: { type: "git_worktree", branchTemplate: `task-${label}`, worktreeParentDir: path.join(baseCwd, ".paperclip", "worktrees"), baseRef: "origin/main" } } });
|
|
await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, projectId, sourceIssueId: issueId, mode: "isolated_workspace",
|
|
strategyType: "git_worktree", name: `Task ${label}`, status: "active", providerType: "local_fs", cwd: realized.cwd, providerRef: realized.cwd,
|
|
repoUrl: main.url, baseRef: "origin/main", branchName: realized.branchName,
|
|
metadata: { createdByRuntime: realized.branchCreatedByRuntime, gitBranchOwnershipVersion: 1, baseRefSha: realized.baseRefSha,
|
|
isolatedRemoteSource: { schema: "paperclip.isolated-remote-source.v1", companyId, issueId, repoUrl: main.url } } });
|
|
await db.update(issues).set({ executionWorkspaceId: workspaceId }).where(eq(issues.id, issueId));
|
|
const [prepared] = await prepareProjectRepositoryWorkspaces({ cwd: realized.cwd, anchorRepoUrl: main.url, workspaces: [source!], resolveGitAuth: noAuth });
|
|
const checkout = await fs.realpath(prepared!.cwd);
|
|
const row = () => db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId)).then(rows => rows[0]!);
|
|
const migrate = async (migrationOptions: NestedCloneMigrationOptions = {}) =>
|
|
migrateNestedRepositoryClone(db, await row(), path.basename(checkout), { resolveGitAuth: noAuth, ...migrationOptions });
|
|
const expire = async () => expireReconstructableGitWorkspace(db, await row(), { dryRun: false, expireBase: false, now: new Date(),
|
|
restorePoint: { storage: getStorageService(), notBefore: new Date(Date.now() - DAY_MS), keyDurable: true } });
|
|
const reopen = () => executionWorkspaceService(db).reopenClosedIsolatedExecutionWorkspaceForIssue({ workspaceId,
|
|
issue: { id: issueId, companyId, projectId: null }, actor: { actorType: "user", agentId: null },
|
|
authorizationActor: { type: "board", source: "local_implicit", isInstanceAdmin: true } });
|
|
return { issueId, workspaceId, cwd: realized.cwd, checkout, name: path.basename(checkout), prefix: await nestedWorktreeBranchPrefix(checkout),
|
|
row, migrate, expire, reopen };
|
|
};
|
|
const first = await task("one");
|
|
return { companyId, projectId, main, nested, candidates, base: candidates[0]!, task, ...first };
|
|
}
|
|
|
|
async function nestedWork(checkout: string) {
|
|
await fs.writeFile(path.join(checkout, "nested.txt"), "nested B (unpushed)\n");
|
|
git(checkout, "commit", "-qam", "Nested B unpushed");
|
|
const unpushed = git(checkout, "rev-parse", "HEAD");
|
|
git(checkout, "checkout", "-q", "-b", "side");
|
|
await fs.writeFile(path.join(checkout, "side.txt"), "side branch work\n");
|
|
git(checkout, "add", "side.txt"); git(checkout, "commit", "-qm", "Side work");
|
|
const side = git(checkout, "rev-parse", "HEAD");
|
|
git(checkout, "tag", "local-tag");
|
|
git(checkout, "checkout", "-q", "main");
|
|
git(checkout, "tag", "-a", "-m", "annotated local release", "local-release");
|
|
await fs.writeFile(path.join(checkout, "nested.txt"), "staged edit\n");
|
|
git(checkout, "add", "nested.txt");
|
|
await fs.writeFile(path.join(checkout, "nested.txt"), "unstaged edit\n");
|
|
await fs.rm(path.join(checkout, "remove-me.txt"));
|
|
await fs.writeFile(path.join(checkout, "draft.md"), "untracked work\n");
|
|
await fs.chmod(path.join(checkout, "draft.md"), 0o600);
|
|
await fs.writeFile(path.join(checkout, "build.log"), "ignored output\n");
|
|
await fs.mkdir(path.join(checkout, "node_modules", "pkg"), { recursive: true });
|
|
await fs.writeFile(path.join(checkout, "node_modules", "pkg", "index.js"), "module.exports = 1;\n");
|
|
await fs.writeFile(path.join(checkout, ".env"), "LOCAL_SETTING=1\n");
|
|
await fs.mkdir(path.join(checkout, ".claude"));
|
|
await fs.writeFile(path.join(checkout, ".claude", "settings.local.json"), "{\"local\":true}\n");
|
|
return { unpushed, side };
|
|
}
|
|
|
|
const gitView = (cwd: string) => ({ head: git(cwd, "rev-parse", "HEAD"), status: git(cwd, "status", "--porcelain=v1", "--untracked-files=all"),
|
|
reflog: git(cwd, "reflog", "show", "--format=%H", "HEAD"), staged: git(cwd, "show", ":nested.txt") });
|
|
const ownedRefs = (base: string, prefix: string) => git(base, "for-each-ref", "--format=%(refname) %(objectname)", `refs/heads/${prefix}`);
|
|
const worktreePaths = (base: string) => git(base, "worktree", "list", "--porcelain").split("\n").filter(line => line.startsWith("worktree ")).map(line => line.slice(9));
|
|
const staging = (checkout: string) => [".clone-migrating", ".clone-standalone", ".clone-trash"].map(suffix => `${checkout}${suffix}`);
|
|
|
|
async function expectUntouched(f: Awaited<ReturnType<typeof fixture>>, before: Awaited<ReturnType<typeof currentRestorePointSnapshot>>, checkout = f.checkout, prefix = f.prefix) {
|
|
expect((await fs.lstat(path.join(checkout, ".git"))).isDirectory()).toBe(true);
|
|
expect(await currentRestorePointSnapshot(checkout)).toEqual(before);
|
|
for (const target of staging(checkout)) expect(await exists(target)).toBe(false);
|
|
expect(ownedRefs(f.base, prefix)).toBe("");
|
|
expect(worktreePaths(f.base)).toHaveLength(1);
|
|
expect((await f.row()).metadata?.nestedCloneMigrationPending).toBeUndefined();
|
|
}
|
|
|
|
it("converts a standalone nested clone into a worktree of the shared clone with its exact state and history", async () => {
|
|
const f = await fixture();
|
|
const { unpushed, side } = await nestedWork(f.checkout);
|
|
const before = await currentRestorePointSnapshot(f.checkout);
|
|
const view = gitView(f.checkout);
|
|
const realBase = await fs.realpath(f.base);
|
|
expect(await listNestedCloneMigrationNames(db, await f.row())).toEqual([f.name]);
|
|
|
|
const result = await f.migrate();
|
|
expect(result).toMatchObject({ outcome: "migrated", reason: null, branchName: `${f.prefix}main` });
|
|
expect(result.reclaimedBytes).toBeGreaterThan(0);
|
|
expect(result.candidateBytes).toBeGreaterThan(0);
|
|
|
|
expect((await fs.lstat(path.join(f.checkout, ".git"))).isFile()).toBe(true);
|
|
expect(git(f.checkout, "rev-parse", "--path-format=absolute", "--git-common-dir")).toBe(path.join(realBase, ".git"));
|
|
expect(await readLinkedNestedCheckout(f.checkout)).toMatchObject({ base: realBase, checkout: f.checkout });
|
|
expect(worktreePaths(f.base)).toContain(f.checkout);
|
|
expect(await currentRestorePointSnapshot(f.checkout)).toEqual(before);
|
|
expect(gitView(f.checkout)).toEqual(view);
|
|
expect(view.head).toBe(unpushed);
|
|
expect(git(f.checkout, "symbolic-ref", "--short", "HEAD")).toBe(`${f.prefix}main`);
|
|
expect(git(f.checkout, "rev-parse", "--abbrev-ref", "@{upstream}")).toBe("origin/main");
|
|
expect(await fs.readFile(path.join(f.checkout, "nested.txt"), "utf8")).toBe("unstaged edit\n");
|
|
expect(await exists(path.join(f.checkout, "remove-me.txt"))).toBe(false);
|
|
expect((await fs.stat(path.join(f.checkout, "draft.md"))).mode & 0o777).toBe(0o600);
|
|
expect(await fs.readFile(path.join(f.checkout, "build.log"), "utf8")).toBe("ignored output\n");
|
|
expect(await fs.readFile(path.join(f.checkout, "node_modules", "pkg", "index.js"), "utf8")).toBe("module.exports = 1;\n");
|
|
expect(await fs.readFile(path.join(f.checkout, ".env"), "utf8")).toBe("LOCAL_SETTING=1\n");
|
|
expect(await fs.readFile(path.join(f.checkout, ".claude", "settings.local.json"), "utf8")).toBe("{\"local\":true}\n");
|
|
|
|
expect(git(f.base, "rev-parse", `refs/heads/${f.prefix}main`)).toBe(unpushed);
|
|
expect(git(f.base, "rev-parse", `refs/heads/${f.prefix}side`)).toBe(side);
|
|
expect(git(f.base, "rev-parse", `refs/heads/${f.prefix}tags/local-tag`)).toBe(side);
|
|
expect(git(f.base, "rev-parse", `refs/heads/${f.prefix}tags/local-release`)).toBe(unpushed);
|
|
expect(git(f.base, "for-each-ref", "--format=%(refname)", "refs/heads").split("\n").filter(ref => !ref.startsWith(`refs/heads/${f.prefix}`))).toEqual(["refs/heads/main"]);
|
|
expect(git(f.base, "status", "--porcelain")).toBe("");
|
|
for (const target of staging(f.checkout)) expect(await exists(target)).toBe(false);
|
|
expect((await f.row()).metadata?.nestedCloneMigrationPending).toBeUndefined();
|
|
expect(await f.migrate()).toMatchObject({ outcome: "skipped", reason: "already_worktree_backed" });
|
|
expect(await listNestedCloneMigrationNames(db, await f.row())).toEqual([]);
|
|
});
|
|
|
|
it("keeps commits that only the clone's remote-tracking refs still hold", async () => {
|
|
const f = await fixture();
|
|
git(f.nested.author, "checkout", "-q", "-b", "squashed-away");
|
|
await fs.writeFile(path.join(f.nested.author, "feature.txt"), "feature\n");
|
|
git(f.nested.author, "add", "."); git(f.nested.author, "commit", "-qm", "Feature");
|
|
const feature = git(f.nested.author, "rev-parse", "HEAD");
|
|
git(f.nested.author, "push", "-q", f.nested.remote, "squashed-away");
|
|
git(f.checkout, "fetch", "-q", "origin");
|
|
git(f.nested.remote, "branch", "-D", "squashed-away");
|
|
git(f.nested.remote, "reflog", "expire", "--expire=now", "--all");
|
|
git(f.nested.remote, "gc", "-q", "--prune=now");
|
|
expect(() => git(f.base, "cat-file", "-e", `${feature}^{commit}`)).toThrow();
|
|
expect(await f.migrate()).toMatchObject({ outcome: "migrated" });
|
|
expect(git(f.base, "rev-parse", `refs/heads/${f.prefix}remotes/origin/squashed-away`)).toBe(feature);
|
|
expect(git(f.checkout, "cat-file", "-t", feature)).toBe("commit");
|
|
});
|
|
|
|
it("expires a converted checkout whose only extra history is remote-tracking and restores that history", async () => {
|
|
const f = await fixture({ issueStatus: "done" });
|
|
git(f.nested.author, "checkout", "-q", "-b", "squashed-away");
|
|
await fs.writeFile(path.join(f.nested.author, "feature.txt"), "feature\n");
|
|
git(f.nested.author, "add", "."); git(f.nested.author, "commit", "-qm", "Feature");
|
|
const feature = git(f.nested.author, "rev-parse", "HEAD");
|
|
git(f.nested.author, "push", "-q", f.nested.remote, "squashed-away");
|
|
git(f.checkout, "fetch", "-q", "origin");
|
|
git(f.nested.remote, "branch", "-D", "squashed-away");
|
|
git(f.nested.remote, "reflog", "expire", "--expire=now", "--all");
|
|
git(f.nested.remote, "gc", "-q", "--prune=now");
|
|
expect(await f.migrate()).toMatchObject({ outcome: "migrated" });
|
|
|
|
expect(await f.expire()).toMatchObject({ expired: true, restorePoint: true, skipReason: null });
|
|
for (const line of ownedRefs(f.base, f.prefix).split("\n").filter(Boolean)) git(f.base, "update-ref", "-d", line.split(" ")[0]!);
|
|
git(f.base, "reflog", "expire", "--expire=now", "--all");
|
|
git(f.base, "gc", "-q", "--prune=now");
|
|
expect(() => git(f.base, "cat-file", "-e", `${feature}^{commit}`)).toThrow();
|
|
expect(await f.reopen()).toMatchObject({ ok: true, reopened: true });
|
|
expect(git(f.checkout, "cat-file", "-t", feature)).toBe("commit");
|
|
});
|
|
|
|
it("fingerprints a converted checkout's own history rather than the shared clone's remote refs", async () => {
|
|
const f = await fixture();
|
|
expect(await f.migrate()).toMatchObject({ outcome: "migrated" });
|
|
const head = git(f.checkout, "rev-parse", "HEAD");
|
|
const before = await restorePointHistoryDigest(f.checkout);
|
|
git(f.base, "update-ref", "refs/remotes/origin/another-task", head);
|
|
expect(await restorePointHistoryDigest(f.checkout)).toBe(before);
|
|
git(f.base, "update-ref", `refs/heads/${f.prefix}remotes/origin/extra`, head);
|
|
expect(await restorePointHistoryDigest(f.checkout)).not.toBe(before);
|
|
for (let index = 0; index < 64; index++) git(f.base, "update-ref", `refs/heads/${f.prefix}remotes/origin/bulk-${index}`, head);
|
|
await expect(restorePointHistoryDigest(f.checkout)).rejects.toThrow("managed_checkout_local_tip_budget");
|
|
});
|
|
|
|
it("finishes an interrupted conversion after the workspace is archived and forgets one whose workspace is gone", async () => {
|
|
const f = await fixture();
|
|
const { unpushed } = await nestedWork(f.checkout);
|
|
const before = await currentRestorePointSnapshot(f.checkout);
|
|
expect(await f.migrate({ hooks: { afterMove: async () => { throw new Error("simulated crash"); } } })).toMatchObject({ outcome: "skipped" });
|
|
await db.update(executionWorkspaces).set({ status: "archived", closedAt: new Date() }).where(eq(executionWorkspaces.id, f.workspaceId));
|
|
const service = executionWorkspaceService(db, { nestedCloneMigrationEnabled: true });
|
|
await service.sweepTerminalWorkspaces();
|
|
expect((await f.row()).metadata?.nestedCloneMigrationPending).toBeUndefined();
|
|
expect(await exists(`${f.checkout}.clone-standalone`)).toBe(false);
|
|
expect(git(f.checkout, "rev-parse", "HEAD")).toBe(unpushed);
|
|
expect(await currentRestorePointSnapshot(f.checkout)).toEqual(before);
|
|
expect(await service.nextTerminalWorkspaceSweepAt()).toBeNull();
|
|
|
|
const second = await f.task("two");
|
|
await nestedWork(second.checkout);
|
|
expect(await second.migrate({ hooks: { afterAside: async () => { throw new Error("simulated crash"); } } })).toMatchObject({ outcome: "skipped" });
|
|
await db.update(executionWorkspaces).set({ status: "archived", closedAt: new Date() }).where(eq(executionWorkspaces.id, second.workspaceId));
|
|
await fs.rm(second.cwd, { recursive: true, force: true });
|
|
expect(await second.migrate()).toMatchObject({ outcome: "rolled_back", reason: "workspace_removed" });
|
|
expect((await second.row()).metadata?.nestedCloneMigrationPending).toBeUndefined();
|
|
expect(git(f.base, "rev-parse", `refs/heads/${second.prefix}main`)).toBeTruthy();
|
|
});
|
|
|
|
it("skips a nested clone with a stash without changing anything", async () => {
|
|
const f = await fixture();
|
|
await fs.writeFile(path.join(f.checkout, "nested.txt"), "stashed edit\n");
|
|
git(f.checkout, "stash", "-q");
|
|
const before = await currentRestorePointSnapshot(f.checkout);
|
|
expect(await f.migrate()).toMatchObject({ outcome: "skipped", reason: "nested_clone_has_stash" });
|
|
await expectUntouched(f, before);
|
|
expect(git(f.checkout, "stash", "list")).not.toBe("");
|
|
});
|
|
|
|
it("skips a nested clone whose local excludes the shared clone would not apply", async () => {
|
|
const f = await fixture();
|
|
await fs.appendFile(path.join(f.checkout, ".git", "info", "exclude"), "\nscratch/\n");
|
|
await fs.mkdir(path.join(f.checkout, "scratch"));
|
|
await fs.writeFile(path.join(f.checkout, "scratch", "notes.txt"), "locally excluded\n");
|
|
const before = await currentRestorePointSnapshot(f.checkout);
|
|
expect(await f.migrate()).toMatchObject({ outcome: "skipped", reason: "unsupported_info_exclude" });
|
|
await expectUntouched(f, before);
|
|
});
|
|
|
|
it("never checks out through filters planted in the shared clone during the conversion", async () => {
|
|
const f = await fixture();
|
|
await fs.writeFile(path.join(f.checkout, ".gitattributes"), "*.dat filter=evil\n");
|
|
await fs.writeFile(path.join(f.checkout, "blob.dat"), "payload\n");
|
|
git(f.checkout, "add", ".gitattributes", "blob.dat"); git(f.checkout, "commit", "-qm", "Attributes");
|
|
await fs.writeFile(path.join(f.checkout, "draft.md"), "untracked work\n");
|
|
const marker = path.join(path.dirname(f.nested.remote), "smudge-ran");
|
|
const before = await currentRestorePointSnapshot(f.checkout);
|
|
const result = await f.migrate({ hooks: { beforeWorktreeAdd: async () => {
|
|
git(f.base, "config", "filter.evil.smudge", `sh -c 'touch ${marker}; cat'`);
|
|
git(f.base, "config", "filter.evil.required", "true");
|
|
} } });
|
|
expect(result).toMatchObject({ outcome: "skipped", reason: "unsupported_base_git_config" });
|
|
expect(await exists(marker)).toBe(false);
|
|
git(f.base, "config", "--remove-section", "filter.evil");
|
|
await expectUntouched(f, before);
|
|
});
|
|
|
|
it("skips while a run is live", async () => {
|
|
const f = await fixture();
|
|
await nestedWork(f.checkout);
|
|
const agentId = randomUUID();
|
|
await db.insert(agents).values({ id: agentId, companyId: f.companyId, name: "Fixture", role: "engineer", adapterType: "process" });
|
|
await db.insert(heartbeatRuns).values({ id: randomUUID(), companyId: f.companyId, agentId, issueId: f.issueId, status: "running" });
|
|
const before = await currentRestorePointSnapshot(f.checkout);
|
|
expect(await f.migrate()).toMatchObject({ outcome: "skipped", reason: "live_or_unresolved_consumer" });
|
|
await expectUntouched(f, before);
|
|
});
|
|
|
|
it("skips a nested clone whose origin differs from the project repository", async () => {
|
|
const f = await fixture();
|
|
await nestedWork(f.checkout);
|
|
git(f.checkout, "remote", "set-url", "origin", f.main.url);
|
|
const before = await currentRestorePointSnapshot(f.checkout);
|
|
expect(await f.migrate()).toMatchObject({ outcome: "skipped", reason: "foreign_origin" });
|
|
await expectUntouched(f, before);
|
|
});
|
|
|
|
it("skips a checkout that is not a configured project repository", async () => {
|
|
const f = await fixture();
|
|
const stray = path.join(path.dirname(f.checkout), "stray-0123456789ab");
|
|
git(path.dirname(stray), "clone", "-q", f.nested.url, stray);
|
|
const before = await currentRestorePointSnapshot(stray);
|
|
for (let index = 0; index < 20; index += 1) await fs.mkdir(path.join(path.dirname(f.checkout), `agent-${index.toString(16).padStart(12, "0")}`, ".git"), { recursive: true });
|
|
expect(await listNestedCloneMigrationNames(db, await f.row())).toEqual([f.name]);
|
|
expect(await migrateNestedRepositoryClone(db, await f.row(), "stray-0123456789ab", { resolveGitAuth: noAuth }))
|
|
.toMatchObject({ outcome: "skipped", reason: "unknown_managed_checkout" });
|
|
await expectUntouched(f, before, stray, await nestedWorktreeBranchPrefix(stray));
|
|
});
|
|
|
|
it("skips when the project has no shared clone", async () => {
|
|
const f = await fixture({ sharedBase: false });
|
|
expect(await f.migrate()).toMatchObject({ outcome: "skipped", reason: "no_shared_base" });
|
|
expect((await fs.lstat(path.join(f.checkout, ".git"))).isDirectory()).toBe(true);
|
|
});
|
|
|
|
it("rolls back and keeps the clone when it changes between capture and verification", async () => {
|
|
const f = await fixture();
|
|
await nestedWork(f.checkout);
|
|
const before = await currentRestorePointSnapshot(f.checkout);
|
|
const result = await f.migrate({ hooks: { beforeVerify: async ({ from, to }) => {
|
|
expect(await exists(path.join(to, "node_modules", "pkg", "index.js"))).toBe(true);
|
|
await fs.writeFile(path.join(from, "late.txt"), "written by a concurrent run\n");
|
|
} } });
|
|
expect(result).toMatchObject({ outcome: "skipped", reason: "snapshot_mismatch" });
|
|
await fs.rm(path.join(f.checkout, "late.txt"));
|
|
await expectUntouched(f, before);
|
|
expect(await fs.readFile(path.join(f.checkout, "node_modules", "pkg", "index.js"), "utf8")).toBe("module.exports = 1;\n");
|
|
expect(await fs.readFile(path.join(f.checkout, ".env"), "utf8")).toBe("LOCAL_SETTING=1\n");
|
|
});
|
|
|
|
it("serializes two tasks converting against the same shared clone and keeps their branches apart", async () => {
|
|
const f = await fixture();
|
|
const second = await f.task("two");
|
|
const one = await nestedWork(f.checkout);
|
|
await fs.writeFile(path.join(second.checkout, "nested.txt"), "second task work\n");
|
|
git(second.checkout, "commit", "-qam", "Second task unpushed");
|
|
const twoHead = git(second.checkout, "rev-parse", "HEAD");
|
|
let active = 0, peak = 0, reached = 0;
|
|
const hooks = { beforeVerify: async () => {
|
|
reached += 1; active += 1; peak = Math.max(peak, active);
|
|
await new Promise(resolve => setTimeout(resolve, 200));
|
|
active -= 1;
|
|
} };
|
|
let release!: () => void;
|
|
const held = withNestedRepositoryBaseLock(f.base, () => new Promise<void>(resolve => { release = resolve; }));
|
|
const running = Promise.all([f.migrate({ hooks }), second.migrate({ hooks })]);
|
|
await new Promise(resolve => setTimeout(resolve, 3_000));
|
|
expect(reached).toBe(0);
|
|
expect((await fs.lstat(path.join(f.checkout, ".git"))).isDirectory()).toBe(true);
|
|
expect((await fs.lstat(path.join(second.checkout, ".git"))).isDirectory()).toBe(true);
|
|
expect(ownedRefs(f.base, f.prefix)).toBe("");
|
|
release();
|
|
await held;
|
|
const [a, b] = await running;
|
|
expect(a).toMatchObject({ outcome: "migrated" });
|
|
expect(b).toMatchObject({ outcome: "migrated" });
|
|
expect(peak).toBe(1);
|
|
expect(f.prefix).not.toBe(second.prefix);
|
|
expect(git(f.checkout, "rev-parse", "HEAD")).toBe(one.unpushed);
|
|
expect(git(second.checkout, "rev-parse", "HEAD")).toBe(twoHead);
|
|
expect(ownedRefs(f.base, second.prefix)).toBe(`refs/heads/${second.prefix}main ${twoHead}`);
|
|
expect(ownedRefs(f.base, f.prefix).split("\n")).toHaveLength(4);
|
|
expect(worktreePaths(f.base)).toEqual(expect.arrayContaining([f.checkout, second.checkout]));
|
|
});
|
|
|
|
it.each(["afterAside", "afterMove", "afterFinalize"] as const)("recovers a conversion interrupted at %s on the next pass", async (step) => {
|
|
const f = await fixture();
|
|
const { unpushed } = await nestedWork(f.checkout);
|
|
const before = await currentRestorePointSnapshot(f.checkout);
|
|
const crashed = await f.migrate({ hooks: { [step]: async () => { throw new Error("simulated crash"); } } });
|
|
expect(crashed).toMatchObject({ outcome: "skipped" });
|
|
expect(await exists(`${f.checkout}.clone-standalone`)).toBe(true);
|
|
expect((await f.row()).metadata?.nestedCloneMigrationPending).toMatchObject({ [f.name]: { phase: step === "afterFinalize" ? "cleanup" : "swap" } });
|
|
expect(await listNestedCloneMigrationNames(db, await f.row())).toEqual([f.name]);
|
|
expect(await f.migrate({ dryRun: true })).toMatchObject({ outcome: "dry_run", reason: "interrupted_migration_pending" });
|
|
expect(await exists(`${f.checkout}.clone-standalone`)).toBe(true);
|
|
|
|
const recovered = await f.migrate();
|
|
expect(recovered).toMatchObject({ outcome: "recovered", reason: null, branchName: `${f.prefix}main` });
|
|
expect(recovered.reclaimedBytes).toBeGreaterThan(0);
|
|
expect((await fs.lstat(path.join(f.checkout, ".git"))).isFile()).toBe(true);
|
|
expect(await currentRestorePointSnapshot(f.checkout)).toEqual(before);
|
|
expect(git(f.checkout, "rev-parse", "HEAD")).toBe(unpushed);
|
|
for (const target of staging(f.checkout)) expect(await exists(target)).toBe(false);
|
|
expect((await f.row()).metadata?.nestedCloneMigrationPending).toBeUndefined();
|
|
expect(worktreePaths(f.base)).toEqual([await fs.realpath(f.base), f.checkout]);
|
|
});
|
|
|
|
it("undoes a conversion interrupted before the swap and then completes it", async () => {
|
|
const f = await fixture();
|
|
const { unpushed } = await nestedWork(f.checkout);
|
|
const before = await currentRestorePointSnapshot(f.checkout);
|
|
const realBase = await fs.realpath(f.base);
|
|
const staged = `${f.checkout}.clone-migrating`;
|
|
const branch = `${f.prefix}main`;
|
|
git(realBase, "fetch", "-q", f.checkout, `refs/heads/main:refs/heads/${branch}`);
|
|
git(realBase, "worktree", "add", "-q", staged, branch);
|
|
await fs.rename(path.join(f.checkout, "node_modules"), path.join(staged, "node_modules"));
|
|
const pending = { base: realBase, head: unpushed, branch, refs: [{ ref: `refs/heads/${branch}`, commit: unpushed, upstream: "main" }],
|
|
moves: ["node_modules"], phase: "transfer" };
|
|
const current = await f.row();
|
|
await db.update(executionWorkspaces).set({ metadata: { ...current.metadata, nestedCloneMigrationPending: { [f.name]: pending } } })
|
|
.where(eq(executionWorkspaces.id, f.workspaceId));
|
|
expect(await f.migrate({ dryRun: true })).toMatchObject({ outcome: "dry_run", reason: "interrupted_migration_pending" });
|
|
expect(await exists(path.join(staged, "node_modules"))).toBe(true);
|
|
|
|
const result = await f.migrate();
|
|
expect(result).toMatchObject({ outcome: "migrated", reason: null });
|
|
expect(await currentRestorePointSnapshot(f.checkout)).toEqual(before);
|
|
expect(await fs.readFile(path.join(f.checkout, "node_modules", "pkg", "index.js"), "utf8")).toBe("module.exports = 1;\n");
|
|
for (const target of staging(f.checkout)) expect(await exists(target)).toBe(false);
|
|
expect(worktreePaths(f.base)).toEqual([realBase, f.checkout]);
|
|
expect((await f.row()).metadata?.nestedCloneMigrationPending).toBeUndefined();
|
|
});
|
|
|
|
it("expires and restores a converted checkout and protects its unpushed work", async () => {
|
|
const f = await fixture({ issueStatus: "done" });
|
|
await fs.writeFile(path.join(f.checkout, "nested.txt"), "nested B (unpushed)\n");
|
|
git(f.checkout, "commit", "-qam", "Nested B unpushed");
|
|
const unpushed = git(f.checkout, "rev-parse", "HEAD");
|
|
await fs.writeFile(path.join(f.checkout, "draft.md"), "nested untracked work\n");
|
|
expect(await f.migrate()).toMatchObject({ outcome: "migrated" });
|
|
const realBase = await fs.realpath(f.base);
|
|
|
|
expect(await f.expire()).toMatchObject({ expired: true, restorePoint: true, skipReason: null });
|
|
expect(await exists(f.cwd)).toBe(false);
|
|
const recipe = (await f.row()).metadata?.gitCacheRestoration as Record<string, any>;
|
|
expect(recipe.contents.checkouts[0]).toMatchObject({ name: f.name, commit: unpushed, branchName: `${f.prefix}main`, worktree: { base: realBase } });
|
|
expect(await f.reopen()).toMatchObject({ ok: true, reopened: true });
|
|
expect(git(f.checkout, "rev-parse", "--path-format=absolute", "--git-common-dir")).toBe(path.join(realBase, ".git"));
|
|
expect(git(f.checkout, "symbolic-ref", "--short", "HEAD")).toBe(`${f.prefix}main`);
|
|
expect(git(f.checkout, "rev-parse", "HEAD")).toBe(unpushed);
|
|
expect(await fs.readFile(path.join(f.checkout, "draft.md"), "utf8")).toBe("nested untracked work\n");
|
|
|
|
const second = await f.task("two");
|
|
await nestedWork(second.checkout);
|
|
await fs.rm(path.join(second.checkout, "node_modules"), { recursive: true });
|
|
expect(await second.migrate()).toMatchObject({ outcome: "migrated" });
|
|
expect(await second.expire()).toMatchObject({ expired: false, skipReason: "unpushed_managed_checkout_branches" });
|
|
expect(await exists(second.checkout)).toBe(true);
|
|
});
|
|
|
|
it("is disabled by default, changes nothing in dry run, and runs from the reaper when enabled", async () => {
|
|
const f = await fixture();
|
|
const { unpushed } = await nestedWork(f.checkout);
|
|
const before = await currentRestorePointSnapshot(f.checkout);
|
|
const disabled = executionWorkspaceService(db);
|
|
await disabled.sweepTerminalWorkspaces();
|
|
await expectUntouched(f, before);
|
|
expect(await disabled.nextTerminalWorkspaceSweepAt()).toBeNull();
|
|
|
|
const dryRun = executionWorkspaceService(db, { nestedCloneMigrationEnabled: true, nestedCloneMigrationDryRun: true });
|
|
await dryRun.sweepTerminalWorkspaces();
|
|
await expectUntouched(f, before);
|
|
expect(await dryRun.nextTerminalWorkspaceSweepAt()).toBeNull();
|
|
|
|
const agentId = randomUUID();
|
|
await db.insert(agents).values({ id: agentId, companyId: f.companyId, name: "Fixture", role: "engineer", adapterType: "process" });
|
|
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId, issueId: f.issueId, status: "running" }).returning();
|
|
// Make a second clock read observably different without changing the
|
|
// scheduler's saved business deadline during an early inspection.
|
|
let tick = Date.now();
|
|
const busy = executionWorkspaceService(db, { nestedCloneMigrationEnabled: true, now: () => new Date(tick++) });
|
|
await busy.sweepTerminalWorkspaces();
|
|
await expectUntouched(f, before);
|
|
const retryAt = await busy.nextTerminalWorkspaceSweepAt();
|
|
expect(retryAt).toBeGreaterThan(Date.now());
|
|
await busy.sweepTerminalWorkspaces();
|
|
expect(await busy.nextTerminalWorkspaceSweepAt()).toBe(retryAt);
|
|
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, run!.id));
|
|
|
|
const enabled = executionWorkspaceService(db, { nestedCloneMigrationEnabled: true });
|
|
await enabled.sweepTerminalWorkspaces();
|
|
expect((await fs.lstat(path.join(f.checkout, ".git"))).isFile()).toBe(true);
|
|
expect(git(f.checkout, "rev-parse", "HEAD")).toBe(unpushed);
|
|
expect(await currentRestorePointSnapshot(f.checkout)).toEqual(before);
|
|
expect(await enabled.nextTerminalWorkspaceSweepAt()).toBeNull();
|
|
});
|
|
});
|