fix: save stale remote-tracking commits in nested restore points (#17289)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Git cache expiration removes closed issues' worktrees once their
remote, or a restore point, can reproduce them
> - A nested `.paperclip-repositories` checkout's remote-tracking refs
can point at commits the remote has since dropped, such as a branch
deleted after a squash merge
> - Strict expiration rightly keeps such a checkout, because it may hold
the only copy; but restore point mode also kept it forever, because
there was nothing else to save
> - This pull request saves those commits in the checkout's restore
point, so the workspace can be reclaimed without losing them
> - The benefit is that long-lived multi-repository workspaces are
reclaimed in restore point mode

## Linked Issues or Issue Description

Refs #17257. Bug report described in-PR (bug report template).

**What happened?**
With restore points enabled, four closed workspaces on one instance were
kept with `unpushed_managed_checkout`, after a dry run had counted them
as candidates. Each nested `paperclip` checkout was clean, on `main` at
a commit that is on GitHub `main`. But it held about 1,000
remote-tracking refs, and between 28 and 41 of them pointed at branches
the remote has since deleted or force-pushed (147 to 203 commits it no
longer has).

The relaxed check passed, so the checkout needed no restore point beyond
those commits. Capture saved only `HEAD` history, so it found nothing to
save and stored no restore point. The final check before deletion then
ran the strict check, which keeps stale remote-tracking commits, and
rolled back. The dry run never reaches that final check, so it
over-reported. Together these workspaces held about 2.4 GB.

**Expected behavior**
In restore point mode, commits that exist only in remote-tracking refs
are saved, and the workspace expires.

**Steps to reproduce**
1. In a nested checkout, fetch a remote branch, so
`refs/remotes/origin/<branch>` exists.
2. Delete that branch on the remote and garbage-collect it.
3. With restore points enabled, close the issue and run expiration: the
workspace is kept with `unpushed_managed_checkout`.

**Paperclip version or commit:** `master` at 412914cfa7.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).

## What Changed

- **Strict mode is unchanged.** An earlier revision of this pull request
ignored remote-tracking refs. Review pointed to `does not trust a stale
remote-tracking ref`, which deliberately protects those commits, so this
pull request keeps that behavior and the test is unchanged.
- **Relaxed nested check** (`assertReproducibleManagedCheckout(..., {
restorePoint: true })`):
- reads all reflogs in one bounded call (`--format=%H %gD`),
deduplicates them by commit, and splits them into local and
remote-tracking entries before applying the tip budget;
  - local entries still have to be covered or upstream, as before;
- remote-tracking tips and remote reflog entries that the remote no
longer has are returned as `preserve` (new `tipsMissingUpstream`,
sharing `commitsMissingUpstream` with the existing check).
- **Capture:** the proof is shared as `provenManagedCheckoutHistory`.
`storeGitRestorePoint({ resolvePreserveCommits })` calls it after taking
the history digest and before capture, then compares the digest again
after capture. A fetch before the digest is included in the preserved
commits, and one after it changes the digest and aborts capture. The
commits flow into `captureRestorePoint`. They are bundled as
history-only commits, within the existing limits (50 preserved tips,
1,000 commits, 256 MB bundle). The checkout therefore gets a restore
point, and the final check uses the relaxed check for it.
- **Final check:** `restorePointHistoryDigest` now also covers
remote-tracking refs and their reflogs, so a fetch, ref change or
reflog-only change after capture rolls expiration back.
- **Restore:** the commits are imported as objects; the stale
remote-tracking refs themselves are not recreated.
- **Docs:** `doc/recoverable-disk-cleanup.md`.

No code comments were added.

## Verification

- `pnpm exec vitest run src/__tests__/git-workspace-expiration.test.ts
src/__tests__/git-reproducible-managed-checkouts.test.ts -t "stale
remote-tracking|remote-tracking refs change"` (in `server/`): 5 passed.
  - The existing strict test passes unchanged.
- A clean nested checkout whose remote-tracking ref points at a deleted,
garbage-collected branch is kept strictly, expires with a restore point,
and reopens on its commit with the stale commit present.
- A remote-tracking ref created after capture rolls expiration back, and
so does a reflog-only change to one.
- 6,000 repeated reflog entries for one commit do not hit the tip
budget.
- **Mutation checks:** each of these fails a case:
- dropping the preserved commits reproduces the instance's symptom
(`unpushed_managed_checkout`);
  - dropping remote-tracking refs or their reflogs from the digest;
  - removing deduplication (`managed_checkout_local_tip_budget`).
- `pnpm exec vitest run git-restore-point git-workspace-expiration
git-workspace-contents git-workspace-restoration
git-reproducible-managed-checkouts execution-workspaces
heartbeat-git-restore-point-resume` (in `server/`): 261 passed.
- `pnpm run typecheck` (in `server/`): pass.
- Not run: server build and boot smoke.

## Risks

- **Only with restore points enabled.** Strict expiration is unchanged.
- **Retention:** saved stale commits are kept until the restore point
expires (90 days after the tree closes by default), then deleted with
it.
- **Limits:** checkouts with more than 50 stale tips, or bundles over
the limits, skip capture and stay kept, as before.
- **Spurious rollbacks:** a fetch during the short window between
capture and deletion rolls back; the next sweep retries.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, read-only inspection of a live
instance).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin FoleyandPaperclip authored and GitHub committed 2026-10-23 08:24:53 -07:00
1 parent 6877760904
commit 0fb152ee94
7 files changed
+171 -33

No files matched your search

+6
View File
@@ -141,6 +141,12 @@ remote lacks, when any reflog holds commits that are neither on the remote nor p
the checked-out history the restore point bundles, when it has objects reachable from
nothing, or when its `COMMIT_EDITMSG` holds a draft that matches no commit in its
history. A `COMMIT_EDITMSG` that matches a commit is accepted for every nested checkout.
A nested checkout's remote-tracking refs (`refs/remotes/origin/*`) and their reflogs can
point at commits the remote has since dropped, for example a branch deleted after a squash
merge. Strict expiration keeps such a checkout. In restore point mode those commits are
saved in the checkout's restore point instead, as history-only objects (the remote-tracking
refs themselves are not recreated), and any change to the remote-tracking refs after capture
rolls expiration back.
Each restore point also records a digest of its `HEAD` reflog and `ORIG_HEAD`; under the
expiration locks that history must be unchanged too, so a commit followed by a reset
after capture rolls expiration back. Reopening
@@ -199,6 +199,23 @@ describe("git restore points", () => {
.rejects.toMatchObject({ reason: "restore_point_history_too_large" });
});
it("refuses more preserved commits than the limit before checking each one", async () => {
const extra = Array.from({ length: 51 }, (_, index) => index.toString(16).padStart(40, "0"));
await expect(captureRestorePoint({ cwd: work, upstreamCommits: upstream(), preserveCommits: extra }))
.rejects.toMatchObject({ reason: "restore_point_history_too_large" });
});
it("saves preserved commits that exist and ignores ones that do not", async () => {
await write(work, "orphan.txt", "only reachable from a preserved commit\n");
git(work, "add", "orphan.txt"); git(work, "commit", "-q", "-m", "orphan");
const orphan = git(work, "rev-parse", "HEAD");
git(work, "reset", "-q", "--hard", "HEAD~1");
git(work, "reflog", "expire", "--expire=now", "--all");
const { manifest } = (await captureRestorePoint({ cwd: work, upstreamCommits: upstream(),
preserveCommits: [orphan, "f".repeat(40)] }))!;
expect(manifest.preserved).toEqual([orphan]);
});
it("keeps capture objects out of the repository", async () => {
await write(work, "notes/todo.md", "untracked work\n");
const before = git(work, "count-objects", "-v");
@@ -321,6 +321,74 @@ describeEmbeddedPostgres("reconstructable Git expiration", () => {
vi.unstubAllEnvs();
});
async function staleRemoteTrackingRef(f: Awaited<ReturnType<typeof nestedFixture>>) {
git(f.nestedSource, "checkout", "-q", "-b", "squashed-away");
await fs.writeFile(path.join(f.nestedSource, "feature.txt"), "merged by squash, then deleted\n");
git(f.nestedSource, "add", "."); git(f.nestedSource, "commit", "-m", "Feature");
const feature = git(f.nestedSource, "rev-parse", "HEAD");
git(f.nestedSource, "push", "-q", f.nestedRemote, "squashed-away");
git(f.nestedSource, "checkout", "-q", "main");
git(f.nested.cwd, "fetch", "-q", "origin", "+refs/heads/squashed-away:refs/remotes/origin/squashed-away");
git(f.nestedRemote, "branch", "-D", "squashed-away");
git(f.nestedRemote, "reflog", "expire", "--expire=now", "--all");
git(f.nestedRemote, "gc", "-q", "--prune=now");
expect(() => git(f.nestedRemote, "cat-file", "-e", `${feature}^{commit}`)).toThrow();
return feature;
}
it("saves a clean nested checkout's stale remote-tracking commits in its restore point", async () => {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32));
const f = await nestedFixture(false);
const feature = await staleRemoteTrackingRef(f);
expect(await f.expire({ expireBase: false })).toMatchObject({ expired: false, skipReason: "unpushed_managed_checkout" });
expect(await f.expire({ expireBase: false, restorePoint: restorePoint() })).toMatchObject({ expired: true, restorePoint: true, skipReason: null });
const recipe = (await f.row()).metadata?.gitCacheRestoration as Record<string, any>;
expect(recipe.contents.checkouts[0].restorePoint).toMatchObject({ head: f.nestedA, headOnRemote: true, worktree: null });
expect(await f.reopen()).toMatchObject({ ok: true, reopened: true });
expect(git(f.nested.cwd, "rev-parse", "HEAD")).toBe(f.nestedA);
expect(git(f.nested.cwd, "cat-file", "-t", feature)).toBe("commit");
vi.unstubAllEnvs();
});
it("rolls back when a nested checkout's remote-tracking refs change after capture", async () => {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32));
const f = await nestedFixture(false);
await staleRemoteTrackingRef(f);
const result = await f.expire({ expireBase: false, restorePoint: restorePoint(), beforeDelete: async () => {
const later = execFileSync("git", ["commit-tree", git(f.nested.cwd, "rev-parse", "HEAD^{tree}"), "-p", "HEAD", "-m", "Fetched after capture"],
{ cwd: f.nested.cwd, env: gitEnv, encoding: "utf8" }).trim();
git(f.nested.cwd, "update-ref", "refs/remotes/origin/fetched-later", later);
} });
expect(result).toMatchObject({ expired: false, skipReason: "restore_point_contents_changed" });
expect(await exists(f.cwd)).toBe(true);
vi.unstubAllEnvs();
});
it("rolls back when only a nested checkout's remote-tracking reflog changes after capture", async () => {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32));
const f = await nestedFixture(false);
await staleRemoteTrackingRef(f);
const result = await f.expire({ expireBase: false, restorePoint: restorePoint(), beforeDelete: async () => {
const tip = git(f.nested.cwd, "rev-parse", "refs/remotes/origin/squashed-away");
git(f.nested.cwd, "update-ref", "-m", "moved", "--create-reflog", "refs/remotes/origin/squashed-away", f.nestedA);
git(f.nested.cwd, "update-ref", "-m", "moved back", "refs/remotes/origin/squashed-away", tip);
} });
expect(result).toMatchObject({ expired: false, skipReason: "restore_point_contents_changed" });
vi.unstubAllEnvs();
});
it("counts distinct reflog commits, not repeated entries, against the tip budget", async () => {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32));
const f = await nestedFixture(false);
await unpushedNestedWork(f.nested.cwd);
const head = git(f.nested.cwd, "rev-parse", "HEAD");
const log = path.join(f.nested.cwd, ".git", "logs", "refs", "heads", "main");
const entry = `${head} ${head} Fixture <fixture@example.test> 1700000000 +0000\treset: moving to HEAD\n`;
await fs.appendFile(log, entry.repeat(6_000));
expect(await f.expire({ expireBase: false, restorePoint: restorePoint() })).toMatchObject({ expired: true, restorePoint: true, skipReason: null });
vi.unstubAllEnvs();
});
it("keeps a workspace whose nested checkout has unpushed work on another branch", async () => {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32));
const f = await nestedFixture(false);
@@ -85,11 +85,14 @@ export async function storeGitRestorePoint(input: {
checkout?: string;
upstreamCommits: readonly string[];
headOnRemote: boolean;
resolvePreserveCommits?: () => Promise<readonly string[]>;
limits?: Partial<RestorePointLimits>;
signal?: AbortSignal;
}): Promise<{ ref: GitRestorePointRef; ledgerId: string; manifest: RestorePointManifest } | null> {
const history = await restorePointHistoryDigest(input.cwd, input.signal);
const captured = await captureRestorePoint({ cwd: input.cwd, upstreamCommits: input.upstreamCommits, limits: input.limits, signal: input.signal });
const preserveCommits = await input.resolvePreserveCommits?.();
const captured = await captureRestorePoint({ cwd: input.cwd, upstreamCommits: input.upstreamCommits, preserveCommits,
limits: input.limits, signal: input.signal });
if (!captured) return null;
if (await restorePointHistoryDigest(input.cwd, input.signal) !== history) throw new RestorePointRestoreFailure("restore_point_history_changed");
const id = randomUUID();
+17 -4
View File
@@ -331,8 +331,16 @@ async function existingCommits(cwd: string, commits: readonly string[], signal?:
return present;
}
async function unpreservedHistory(cwd: string, head: string, boundary: readonly string[], signal?: AbortSignal) {
const candidates = new Set<string>();
async function presentCommits(cwd: string, commits: readonly string[], signal?: AbortSignal) {
const unique = [...new Set(commits.filter((commit) => SHA.test(commit)))];
if (unique.length === 0) return [];
const output = await gitText(cwd, ["cat-file", "--batch-check=%(objectname) %(objecttype)"], { signal, input: unique.join("\n") + "\n" });
return output.split("\n").map((line) => line.split(" ")).filter(([sha, type]) => SHA.test(sha ?? "") && type === "commit").map(([sha]) => sha!);
}
async function unpreservedHistory(cwd: string, head: string, boundary: readonly string[], signal?: AbortSignal, extra: readonly string[] = []) {
if (new Set(extra).size > MAX_PRESERVED_COMMITS) throw new RestorePointSkip("restore_point_history_too_large");
const candidates = new Set<string>(await presentCommits(cwd, extra, signal));
const reflog = await gitText(cwd, ["reflog", "show", "--format=%H", "HEAD"], { signal }).catch(() => "");
for (const line of reflog.split("\n")) if (SHA.test(line)) candidates.add(line);
const original = await gitText(cwd, ["rev-parse", "--verify", "-q", "ORIG_HEAD^{commit}"], { signal }).catch(() => "");
@@ -385,6 +393,7 @@ export function decodeRestorePointPayload(payload: Buffer): { manifest: RestoreP
export async function captureRestorePoint(input: {
cwd: string;
upstreamCommits: readonly string[];
preserveCommits?: readonly string[];
limits?: Partial<RestorePointLimits>;
signal?: AbortSignal;
}): Promise<{ payload: Buffer; manifest: RestorePointManifest } | null> {
@@ -396,7 +405,7 @@ export async function captureRestorePoint(input: {
const { store, env } = await snapshotStore(cwd, temporary);
const snapshot = await snapshotCommits(cwd, env, temporary, modes, input.signal);
const boundary = await existingCommits(cwd, input.upstreamCommits, input.signal);
const preserved = await unpreservedHistory(cwd, snapshot.head, boundary, input.signal);
const preserved = await unpreservedHistory(cwd, snapshot.head, boundary, input.signal, input.preserveCommits);
const headOnUpstream = await (async () => {
for (const tip of boundary) {
if (await git(cwd, ["merge-base", "--is-ancestor", snapshot.head, tip], { signal: input.signal }).then(() => true, () => false)) return true;
@@ -559,5 +568,9 @@ export async function restorePointHistoryDigest(input: string, signal?: AbortSig
const cwd = await fs.realpath(input);
const reflog = await gitText(cwd, ["reflog", "show", "--format=%H", "HEAD"], { signal }).catch(() => "");
const original = await gitText(cwd, ["rev-parse", "--verify", "-q", "ORIG_HEAD^{commit}"], { signal }).catch(() => "");
return createHash("sha256").update(JSON.stringify([reflog.split("\n").filter((line) => SHA.test(line)), SHA.test(original) ? original : null])).digest("hex");
const remotes = await gitText(cwd, ["for-each-ref", "--format=%(objectname) %(refname)", "refs/remotes"], { signal }).catch(() => "");
const remoteReflogs = (await gitText(cwd, ["reflog", "show", "--all", "--format=%H %gD"], { signal }).catch(() => ""))
.split("\n").filter((line) => / refs\/remotes\//.test(line));
return createHash("sha256").update(JSON.stringify([reflog.split("\n").filter((line) => SHA.test(line)), SHA.test(original) ? original : null,
remotes.split("\n").filter(Boolean), remoteReflogs])).digest("hex");
}
@@ -13,7 +13,7 @@ import { abandonGitRestorePoint, abandonGitRestorePointsByKey, activateGitRestor
import { withIsolatedRepositorySourceLock, verifiedRecordedIsolatedRemoteSource } from "./isolated-remote-workspace-source.js";
import { managedProjectCheckoutName } from "./managed-project-checkout-name.js";
import type { GitRemoteAuthProvider } from "./git-credentials.js";
import { assertReproducibleGitFiles, assertReproducibleBaseGitStorage, assertReproducibleManagedCheckout, inspectGitCacheRecipe, inspectGitCacheRecipeForRestorePoint, readGitCacheRestoration, GIT_CACHE_RESTORATION_SCHEMA_V2,
import { assertReproducibleGitFiles, assertReproducibleBaseGitStorage, assertReproducibleManagedCheckout, provenManagedCheckoutHistory, inspectGitCacheRecipe, inspectGitCacheRecipeForRestorePoint, readGitCacheRestoration, GIT_CACHE_RESTORATION_SCHEMA_V2,
verifyRemoteGitCommit, gitCacheExpirationLockLocations, recoverGitCacheExpirationLocks, GitCacheRestorationFailure, type ManagedCheckoutProver, gitCacheCommand, readGitCacheExpiration, finishGitCacheTrash, type GitCacheExpiration, type GitCacheRestoration, GIT_CACHE_RESTORATION_KEY, GIT_CACHE_EXPIRATION_REASON } from "./git-workspace-restoration.js";
async function allocatedBytes(target: string, shouldContinue: () => boolean): Promise<number> {
@@ -294,7 +294,9 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Worksp
} else if (!headOnRemote) throw new Error("upstream_commit_unverified");
for (const [name, need] of [...checkoutRestorePoints.entries()].sort(([a], [b]) => a.localeCompare(b))) {
const storedCheckout = await storeGitRestorePoint({ db, storage: options.restorePoint!.storage, workspace: owner,
cwd: need.path, checkout: name, generation, headOnRemote: need.headOnRemote, limits: options.restorePoint!.limits,
cwd: need.path, checkout: name, generation, headOnRemote: need.headOnRemote,
resolvePreserveCommits: async () => (await provenManagedCheckoutHistory(need.path, need.repoUrl, auth, options.signal)).preserve,
limits: options.restorePoint!.limits,
signal: options.signal, upstreamCommits: await restorePointBoundary(need.path, need.repoUrl, need.headOnRemote, auth, options.signal) });
if (!storedCheckout) {
if (!need.headOnRemote) throw new Error("unpushed_managed_checkout");
@@ -250,7 +250,7 @@ async function assertCommitMessageSaved(cwd: string, gitDir: string, signal?: Ab
export async function assertReproducibleManagedCheckout(checkout: string, input: {
repoUrl: string; resolveGitAuth: GitRemoteAuthProvider; signal?: AbortSignal; restorePoint?: boolean;
}): Promise<{ headOnRemote: boolean }> {
}): Promise<{ headOnRemote: boolean; preserve: string[] }> {
const { signal } = input;
const gitDir = await fs.lstat(path.join(checkout, ".git")).catch(() => null);
if (!gitDir?.isDirectory()) throw new Error("unproven_managed_checkout");
@@ -266,30 +266,10 @@ export async function assertReproducibleManagedCheckout(checkout: string, input:
const refs = (await gitCacheCommand(checkout, ["for-each-ref", "--format=%(refname)"], signal)).trim().split("\n").filter(Boolean);
if (refs.some(ref => !/^refs\/(heads|tags|remotes\/origin)\//.test(ref))) throw new Error("unsupported_managed_checkout_refs");
if (input.restorePoint) {
const head = (await gitCacheCommand(checkout, ["rev-parse", "--verify", "HEAD^{commit}"], signal)).trim();
const shas = (output: string) => output.split("\n").map(line => line.trim()).filter(sha => shaPattern.test(sha));
const covered = new Set([head,
...shas(await gitCacheCommand(checkout, ["reflog", "show", "--format=%H", "HEAD"], signal).catch(() => "")),
...shas(await gitCacheCommand(checkout, ["rev-parse", "--verify", "-q", "ORIG_HEAD^{commit}"], signal).catch(() => ""))]);
const recorded = new Set([
...shas(await gitCacheCommand(checkout, ["for-each-ref", "--format=%(objectname)", "refs/heads", "refs/tags"], signal)),
...shas(await gitCacheCommand(checkout, ["reflog", "show", "--all", "--format=%H"], signal).catch(() => "")),
]);
const current = (await gitCacheCommand(checkout, ["symbolic-ref", "--quiet", "HEAD"], signal).catch(() => "")).trim();
const otherTips = (await gitCacheCommand(checkout, ["for-each-ref", "--format=%(objectname) %(refname)", "refs/heads", "refs/tags"], signal))
.split("\n").map(line => line.trim().split(" ")).filter(([sha, ref]) => shaPattern.test(sha ?? "") && ref !== current).map(([sha]) => sha!);
if (otherTips.length && await hasCommitsMissingUpstream(checkout, otherTips, input.repoUrl, input.resolveGitAuth, signal, [head])) {
throw new Error("unpushed_managed_checkout_branches");
}
const uncovered = [...recorded].filter(sha => !covered.has(sha));
if (uncovered.length + covered.size > MANAGED_CHECKOUT_LOCAL_TIP_LIMIT) throw new Error("managed_checkout_local_tip_budget");
if (uncovered.length && await hasCommitsMissingUpstream(checkout, uncovered, input.repoUrl, input.resolveGitAuth, signal, [...covered])) {
throw new Error("unpushed_managed_checkout_branches");
}
const headOnRemote = !await hasCommitsMissingUpstream(checkout, [head], input.repoUrl, input.resolveGitAuth, signal);
const { headOnRemote, preserve } = await provenManagedCheckoutHistory(checkout, input.repoUrl, input.resolveGitAuth, signal);
const unreachable = await gitCacheCommand(checkout, ["fsck", "--connectivity-only", "--unreachable", "--no-progress"], signal);
if (unreachable.trim()) throw new Error("unreachable_managed_checkout_objects");
return { headOnRemote };
return { headOnRemote, preserve };
}
const localTips = new Set([
...(await gitCacheCommand(checkout, ["rev-parse", "--verify", "HEAD^{commit}"], signal)).split("\n"),
@@ -302,13 +282,61 @@ export async function assertReproducibleManagedCheckout(checkout: string, input:
}
const unreachable = await gitCacheCommand(checkout, ["fsck", "--connectivity-only", "--unreachable", "--no-progress"], signal);
if (unreachable.trim()) throw new Error("unreachable_managed_checkout_objects");
return { headOnRemote: true };
return { headOnRemote: true, preserve: [] };
}
const MANAGED_CHECKOUT_LOCAL_TIP_LIMIT = 5_000;
export async function provenManagedCheckoutHistory(checkout: string, repoUrl: string, resolveGitAuth: GitRemoteAuthProvider,
signal?: AbortSignal): Promise<{ headOnRemote: boolean; preserve: string[] }> {
const head = (await gitCacheCommand(checkout, ["rev-parse", "--verify", "HEAD^{commit}"], signal)).trim();
const shas = (output: string) => output.split("\n").map(line => line.trim()).filter(sha => shaPattern.test(sha));
const covered = new Set([head,
...shas(await gitCacheCommand(checkout, ["reflog", "show", "--format=%H", "HEAD"], signal).catch(() => "")),
...shas(await gitCacheCommand(checkout, ["rev-parse", "--verify", "-q", "ORIG_HEAD^{commit}"], signal).catch(() => ""))]);
const reflogs = [...new Set((await gitCacheCommand(checkout, ["reflog", "show", "--all", "--format=%H %gD"], signal).catch(() => ""))
.split("\n").map(line => line.trim().split(" ")).filter(([sha]) => shaPattern.test(sha ?? ""))
.map(([sha, selector]) => `${sha} ${(selector ?? "").startsWith("refs/remotes/") ? "remote" : "local"}`))].map(entry => entry.split(" "));
if (reflogs.length > MANAGED_CHECKOUT_LOCAL_TIP_LIMIT) throw new Error("managed_checkout_local_tip_budget");
const remote = (kind: string | undefined) => kind === "remote";
const recorded = new Set([
...shas(await gitCacheCommand(checkout, ["for-each-ref", "--format=%(objectname)", "refs/heads", "refs/tags"], signal)),
...reflogs.filter(([, selector]) => !remote(selector)).map(([sha]) => sha!),
]);
const remoteTracked = [...new Set([
...shas(await gitCacheCommand(checkout, ["for-each-ref", "--format=%(objectname)", "refs/remotes"], signal)),
...reflogs.filter(([, selector]) => remote(selector)).map(([sha]) => sha!),
])];
const current = (await gitCacheCommand(checkout, ["symbolic-ref", "--quiet", "HEAD"], signal).catch(() => "")).trim();
const otherTips = (await gitCacheCommand(checkout, ["for-each-ref", "--format=%(objectname) %(refname)", "refs/heads", "refs/tags"], signal))
.split("\n").map(line => line.trim().split(" ")).filter(([sha, ref]) => shaPattern.test(sha ?? "") && ref !== current).map(([sha]) => sha!);
if (otherTips.length && await hasCommitsMissingUpstream(checkout, otherTips, repoUrl, resolveGitAuth, signal, [head])) {
throw new Error("unpushed_managed_checkout_branches");
}
const uncovered = [...recorded].filter(sha => !covered.has(sha));
if (uncovered.length + covered.size + remoteTracked.length > MANAGED_CHECKOUT_LOCAL_TIP_LIMIT) throw new Error("managed_checkout_local_tip_budget");
if (uncovered.length && await hasCommitsMissingUpstream(checkout, uncovered, repoUrl, resolveGitAuth, signal, [...covered])) {
throw new Error("unpushed_managed_checkout_branches");
}
const headOnRemote = !await hasCommitsMissingUpstream(checkout, [head], repoUrl, resolveGitAuth, signal);
const preserve = remoteTracked.length
? await tipsMissingUpstream(checkout, remoteTracked, repoUrl, resolveGitAuth, signal, [...covered, ...recorded]) : [];
return { headOnRemote, preserve };
}
async function tipsMissingUpstream(checkout: string, tips: string[], repoUrl: string,
resolveGitAuth: GitRemoteAuthProvider, signal?: AbortSignal, covered: string[] = []) {
const missing = new Set(await commitsMissingUpstream(checkout, tips, repoUrl, resolveGitAuth, signal, covered));
return tips.filter(tip => missing.has(tip)).sort();
}
async function hasCommitsMissingUpstream(checkout: string, localTips: string[], repoUrl: string,
resolveGitAuth: GitRemoteAuthProvider, signal?: AbortSignal, covered: string[] = []) {
return (await commitsMissingUpstream(checkout, localTips, repoUrl, resolveGitAuth, signal, covered, 1)).length > 0;
}
async function commitsMissingUpstream(checkout: string, localTips: string[], repoUrl: string,
resolveGitAuth: GitRemoteAuthProvider, signal?: AbortSignal, covered: string[] = [], limit?: number) {
const store = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-git-checkout-proof-"));
try {
const options = { timeout: 120_000, signal, maxBuffer: 16 * 1024 * 1024, cwd: os.tmpdir(),
@@ -324,8 +352,9 @@ async function hasCommitsMissingUpstream(checkout: string, localTips: string[],
if (signal?.aborted) throw error;
throw new Error("managed_checkout_upstream_unavailable");
}
const missing = await execFile("git", ["-C", store, "rev-list", "--max-count=1", ...localTips, "--not", "--glob=refs/upstream/*", ...covered], options);
return missing.stdout.trim() !== "";
const missing = await execFile("git", ["-C", store, "rev-list", ...(limit ? [`--max-count=${limit}`] : []), ...localTips,
"--not", "--glob=refs/upstream/*", ...covered], options);
return missing.stdout.split("\n").map(line => line.trim()).filter(Boolean);
} finally {
await fs.rm(store, { recursive: true, force: true });
}