Files
PaperClipAI/server/src/__tests__/git-reproducible-managed-checkouts.test.ts
T
DottaandPaperclip 790b729086 fix: restore nested project repositories and settings after expiration (#16691)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use isolated Git workspaces with managed project repository
copies.
> - Finished workspaces can expire when Git can reproduce their
contents.
> - The saved recipe records the main commit but omits nested commits
and local settings.
> - Reopening can therefore use newer nested code and lose merged user
settings.
> - This pull request saves and restores those contents before workspace
provisioning.
> - It also permits verified FETCH_HEAD metadata so ordinary fetches do
not block safe expiration.

## Linked Issues or Issue Description

Refs #16457, #16586, #16621. Related performance work: #16579.

**What happened?**

After a finished workspace expires, its nested project checkout can
reopen at the current upstream commit instead of its previous commit.
Cleanup also treats local Claude settings as disposable, although that
file can contain user settings. Ordinary linked-worktree FETCH_HEAD
metadata prevents otherwise safe cleanup.

**Expected behavior**

Restore the main and nested repositories at their saved commits.
Preserve local settings bytes and permissions. Skip files and Git state
that cannot be reproduced. Permit standard fetched metadata only when
its commits remain recoverable.

**Steps to reproduce**

1. Prepare a managed nested project repository at commit A and add
ignored local settings.
2. Advance upstream to B, finish the task tree, and expire its workspace
and base.
3. Reopen the workspace through the normal service path. Before this
fix, nested preparation selects B.

## What Changed

- Extend the existing restoration recipe with exact nested commits,
branches, source identity, and encrypted local settings. Keep the schema
compatible with older recipes.
- Restore nested repositories and settings before provisioning. Recheck
company/project source access and preserve conflicting files on retry.
- Recheck recorded contents under cleanup locks before deletion. Persist
lock ownership before atomic publication so process exits recover safely
and foreign locks stay intact. Preserve changed or unsupported contents,
including tracked settings.
- Keep saved project repository selections through normal preparation.
Checkpoint restored contents before provisioning so failed setup can
retry without overwriting later edits.
- Accept bounded linked-worktree FETCH_HEAD records only after local
ancestry or fresh upstream proof. Preserve local-only fetched commits.
- Add real Git/Postgres lifecycle cases and focused content, access,
publication-retry, and fetched-state controls. Document the behavior.

## Verification

- `pnpm -r typecheck` passed.
- `pnpm build` passed.
- The original real Git/Postgres experiment failed with nested B instead
of A. The same path passes with this change.
- The integrated lifecycle suite passed 42 cases in 91.66s. Related
preparation, credential, content, and Git metadata controls passed 46
cases in 24.41s. The final heartbeat checkpoint persistence case passed
separately in 8.17s.
- Coverage includes exact main/nested commits, encrypted settings and
modes, upstream failure and retry, provisioning order, concurrent
reopening, changed unique files, tracked settings, malformed metadata,
and company boundaries.
- Exact-head CI passed the implementation, typecheck, build, security,
and test matrices. One browser shard passed on a single rerun after an
unrelated fixture teardown raced an activity-log write; its feature
assertions passed in the initial attempt.
- The broad local test run was stopped after about 12 minutes of
unrelated serial tests. Full parallel CI is required before merge; no
local full-suite pass is claimed.
- Single-workspace deployment, real-backlog reclamation, and ordinary
product reopening remain required acceptance work after merge.

## Risks

- Expired Git repositories still depend on upstream access. A failed
restoration returns the existing retryable error and never substitutes
the default branch.
- Saved settings use the existing local encryption provider. Recovery
requires that encryption key to remain available.
- Unsupported files, tracked settings, unknown sources, and unverified
Git state remain on disk. This can reduce reclaimed space.
- No database migration, recovery-retention change, public API, or
fleet-default change is included.

## Model Used

OpenAI Codex, GPT-6 family, with tool use, code execution, and an
independent agent review. The runtime does not expose a more specific
model identifier.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-17 22:54:28 -05:00

229 lines
10 KiB
TypeScript

import { execFile as execFileCallback } from "node:child_process";
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { promisify } from "node:util";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import {
assertReproducibleGitFiles,
assertReproducibleManagedCheckout,
type ManagedCheckoutProver,
} from "../services/git-workspace-restoration.js";
const execFile = promisify(execFileCallback);
async function git(cwd: string, ...args: string[]) {
const { stdout } = await execFile("git", ["-C", cwd, "-c", "user.name=Test", "-c", "user.email=test@example.com",
"-c", "init.defaultBranch=main", "-c", "commit.gpgsign=false", ...args]);
return stdout;
}
async function write(file: string, content = "x\n") {
await fs.mkdir(path.dirname(file), { recursive: true });
await fs.writeFile(file, content);
}
async function commit(cwd: string, file: string, message: string) {
await write(path.join(cwd, file), `${message}\n`);
await git(cwd, "add", file);
await git(cwd, "commit", "-q", "-m", message);
await fs.rm(path.join(cwd, ".git", "COMMIT_EDITMSG"), { force: true });
return (await git(cwd, "rev-parse", "HEAD")).trim();
}
describe("assertReproducibleGitFiles with Paperclip-managed files", () => {
let root: string;
let work: string;
let projectSeed: string;
let projectRemote: string;
let prove: ManagedCheckoutProver;
async function upstream(name: string, files: Record<string, string>) {
const seed = path.join(root, `${name}-seed`);
await fs.mkdir(seed, { recursive: true });
await git(seed, "init", "-q");
for (const [file, content] of Object.entries(files)) await write(path.join(seed, file), content);
await git(seed, "add", "-A");
await git(seed, "commit", "-q", "-m", "seed");
const bare = path.join(root, `${name}.git`);
await execFile("git", ["clone", "-q", "--bare", seed, bare]);
await git(seed, "remote", "add", "origin", bare);
return { seed, bare };
}
beforeEach(async () => {
root = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-reproducible-")));
const main = await upstream("main", {
"README.md": "main\n",
".gitignore": "node_modules/\n.claude/settings.local.json\nsecret.txt\n",
});
({ seed: projectSeed, bare: projectRemote } = await upstream("project", { "README.md": "project\n", ".gitignore": "node_modules/\n" }));
work = path.join(root, "work");
await execFile("git", ["clone", "-q", main.bare, work]);
await fs.appendFile(path.join(work, ".git", "info", "exclude"), "\n/.paperclip-repositories/\n");
prove = (checkout, _name, signal) => assertReproducibleManagedCheckout(checkout, {
repoUrl: projectRemote, resolveGitAuth: async () => null, signal,
});
});
afterEach(async () => {
await fs.rm(root, { recursive: true, force: true });
});
async function managedCheckout(name = "project-d051a9c4e671") {
const checkout = path.join(work, ".paperclip-repositories", name);
await fs.mkdir(path.dirname(checkout), { recursive: true });
await execFile("git", ["clone", "-q", projectRemote, checkout]);
return checkout;
}
const check = () => assertReproducibleGitFiles(work, undefined, { proveManagedCheckout: prove });
const expectAccepted = () => expect(check()).resolves.toBeDefined();
const expectRejected = (reason: string | RegExp) => expect(check()).rejects.toThrow(reason);
it("preserves Claude settings without a restoration capture", async () => {
await write(path.join(work, ".claude", "settings.local.json"), "{}\n");
await expectRejected("unique_ignored_files");
});
it("accepts a clean managed checkout, including its regenerable directories", async () => {
const checkout = await managedCheckout();
await write(path.join(checkout, "node_modules", "pkg", "index.js"));
await write(path.join(work, "node_modules", "pkg", "index.js"));
await expectAccepted();
});
it("rejects managed checkouts unless the caller can prove them", async () => {
await managedCheckout();
await expect(assertReproducibleGitFiles(work)).rejects.toThrow("unique_ignored_files");
});
it("still rejects unknown ignored files", async () => {
await write(path.join(work, "secret.txt"));
await expectRejected("unique_ignored_files");
});
it("rejects changed, untracked, and flag-hidden edits", async () => {
const checkout = await managedCheckout();
await write(path.join(checkout, "README.md"), "edited\n");
await expectRejected("dirty_or_untracked_files");
await git(checkout, "checkout", "-q", "--", "README.md");
await write(path.join(checkout, "notes.txt"));
await expectRejected("dirty_or_untracked_files");
await fs.rm(path.join(checkout, "notes.txt"));
await git(checkout, "update-index", "--skip-worktree", "README.md");
await write(path.join(checkout, "README.md"), "hidden edit\n");
await expectRejected("unsupported_index_flags");
});
it("rejects commits kept by a branch, a tag, or only the reflog", async () => {
const checkout = await managedCheckout();
await commit(checkout, "branch.txt", "branch work");
await expectRejected("unpushed_managed_checkout");
await git(checkout, "tag", "kept");
await git(checkout, "pack-refs", "--all");
await git(checkout, "reset", "-q", "--hard", "origin/main");
await fs.rm(path.join(checkout, ".git", "ORIG_HEAD"), { force: true });
await git(checkout, "reflog", "expire", "--expire=now", "--all");
await expectRejected("unpushed_managed_checkout");
await git(checkout, "tag", "-d", "kept");
await commit(checkout, "amended.txt", "amended work");
await git(checkout, "reset", "-q", "--hard", "origin/main");
await fs.rm(path.join(checkout, ".git", "ORIG_HEAD"), { force: true });
await expectRejected(/unique_reflog_commits|unpushed_managed_checkout/);
});
it("rejects a detached HEAD commit after its reflog expires", async () => {
const checkout = await managedCheckout();
await git(checkout, "checkout", "-q", "--detach");
await commit(checkout, "detached.txt", "detached work");
await git(checkout, "reflog", "expire", "--expire=now", "--all");
await expectRejected("unpushed_managed_checkout");
});
it("accepts pushed history after upstream advances", async () => {
await managedCheckout();
await commit(projectSeed, "later.txt", "later upstream work");
await git(projectSeed, "push", "-q", "origin", "main");
await expectAccepted();
});
it("keeps a checkout whose upstream cannot be reached or authenticated", async () => {
await managedCheckout();
prove = (checkout, _name, signal) => assertReproducibleManagedCheckout(checkout, {
repoUrl: path.join(root, "missing.git"), resolveGitAuth: async () => null, signal,
});
await expectRejected("managed_checkout_upstream_unavailable");
prove = (checkout, _name, signal) => assertReproducibleManagedCheckout(checkout, {
repoUrl: projectRemote, resolveGitAuth: async () => { throw new Error("credentials refused"); }, signal,
});
await expectRejected("managed_checkout_upstream_unavailable");
});
it("reports a local proof failure as local, not as an upstream problem", async () => {
await managedCheckout();
const writeFile = fs.writeFile.bind(fs);
const spy = vi.spyOn(fs, "writeFile").mockImplementation(async (file, ...rest) => {
if (String(file).endsWith(path.join("objects", "info", "alternates"))) throw new Error("ENOSPC: no space left on device");
return writeFile(file, ...(rest as [never]));
});
try {
await expectRejected("ENOSPC");
} finally {
spy.mockRestore();
}
});
it("does not trust a stale remote-tracking ref", async () => {
await git(projectSeed, "checkout", "-q", "-b", "feature");
await commit(projectSeed, "feature.txt", "feature work");
await git(projectSeed, "push", "-q", "origin", "feature");
await managedCheckout();
await git(projectRemote, "branch", "-q", "-D", "feature");
await expectRejected("unpushed_managed_checkout");
});
it("rejects custom hooks, linked worktrees, stashes, and unreachable objects", async () => {
const checkout = await managedCheckout();
await write(path.join(checkout, ".git", "hooks", "pre-commit"), "#!/bin/sh\nexit 0\n");
await expectRejected("custom_git_metadata");
await fs.rm(path.join(checkout, ".git", "hooks", "pre-commit"));
const linked = path.join(root, "linked");
await git(checkout, "worktree", "add", "-q", "--detach", linked);
await expectRejected("managed_checkout_has_worktrees");
await git(checkout, "worktree", "remove", "--force", linked);
await write(path.join(checkout, "README.md"), "stashed\n");
await git(checkout, "stash", "-q");
await expectRejected(/unsupported_local_git_state|unsupported_managed_checkout_refs/);
await git(checkout, "stash", "clear");
await write(path.join(root, "orphan.txt"), "only copy\n");
await git(checkout, "hash-object", "-w", path.join(root, "orphan.txt"));
await expectRejected("unreachable_managed_checkout_objects");
});
it("never runs a configured fsmonitor command while inspecting", async () => {
const checkout = await managedCheckout();
const marker = path.join(root, "fsmonitor-ran");
const hook = path.join(root, "fsmonitor.sh");
await write(hook, `#!/bin/sh\ntouch '${marker}'\nexit 1\n`);
await fs.chmod(hook, 0o755);
for (const repository of [checkout, work]) {
await git(repository, "config", "core.fsmonitor", hook);
await check().catch(() => undefined);
await git(repository, "config", "--unset", "core.fsmonitor");
}
await expect(fs.access(marker)).rejects.toThrow();
});
it("never treats task repositories or plain directories as reproducible", async () => {
await managedCheckout("task-repo-0123456789abcdef01234567");
await expectRejected("unique_ignored_files");
await fs.rm(path.join(work, ".paperclip-repositories"), { recursive: true });
await write(path.join(work, ".paperclip-repositories", "project-d051a9c4e671", "README.md"));
await expectRejected("unproven_managed_checkout");
});
});