mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-24 18:19:05 +02:00
fix: save stale remote-tracking commits in nested restore points (#17289)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Git cache expiration removes closed issues' worktrees once their
remote, or a restore point, can reproduce them
> - A nested `.paperclip-repositories` checkout's remote-tracking refs
can point at commits the remote has since dropped, such as a branch
deleted after a squash merge
> - Strict expiration rightly keeps such a checkout, because it may hold
the only copy; but restore point mode also kept it forever, because
there was nothing else to save
> - This pull request saves those commits in the checkout's restore
point, so the workspace can be reclaimed without losing them
> - The benefit is that long-lived multi-repository workspaces are
reclaimed in restore point mode
## Linked Issues or Issue Description
Refs #17257. Bug report described in-PR (bug report template).
**What happened?**
With restore points enabled, four closed workspaces on one instance were
kept with `unpushed_managed_checkout`, after a dry run had counted them
as candidates. Each nested `paperclip` checkout was clean, on `main` at
a commit that is on GitHub `main`. But it held about 1,000
remote-tracking refs, and between 28 and 41 of them pointed at branches
the remote has since deleted or force-pushed (147 to 203 commits it no
longer has).
The relaxed check passed, so the checkout needed no restore point beyond
those commits. Capture saved only `HEAD` history, so it found nothing to
save and stored no restore point. The final check before deletion then
ran the strict check, which keeps stale remote-tracking commits, and
rolled back. The dry run never reaches that final check, so it
over-reported. Together these workspaces held about 2.4 GB.
**Expected behavior**
In restore point mode, commits that exist only in remote-tracking refs
are saved, and the workspace expires.
**Steps to reproduce**
1. In a nested checkout, fetch a remote branch, so
`refs/remotes/origin/<branch>` exists.
2. Delete that branch on the remote and garbage-collect it.
3. With restore points enabled, close the issue and run expiration: the
workspace is kept with `unpushed_managed_checkout`.
**Paperclip version or commit:** `master` at 412914cfa7.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
- **Strict mode is unchanged.** An earlier revision of this pull request
ignored remote-tracking refs. Review pointed to `does not trust a stale
remote-tracking ref`, which deliberately protects those commits, so this
pull request keeps that behavior and the test is unchanged.
- **Relaxed nested check** (`assertReproducibleManagedCheckout(..., {
restorePoint: true })`):
- reads all reflogs in one bounded call (`--format=%H %gD`),
deduplicates them by commit, and splits them into local and
remote-tracking entries before applying the tip budget;
- local entries still have to be covered or upstream, as before;
- remote-tracking tips and remote reflog entries that the remote no
longer has are returned as `preserve` (new `tipsMissingUpstream`,
sharing `commitsMissingUpstream` with the existing check).
- **Capture:** the proof is shared as `provenManagedCheckoutHistory`.
`storeGitRestorePoint({ resolvePreserveCommits })` calls it after taking
the history digest and before capture, then compares the digest again
after capture. A fetch before the digest is included in the preserved
commits, and one after it changes the digest and aborts capture. The
commits flow into `captureRestorePoint`. They are bundled as
history-only commits, within the existing limits (50 preserved tips,
1,000 commits, 256 MB bundle). The checkout therefore gets a restore
point, and the final check uses the relaxed check for it.
- **Final check:** `restorePointHistoryDigest` now also covers
remote-tracking refs and their reflogs, so a fetch, ref change or
reflog-only change after capture rolls expiration back.
- **Restore:** the commits are imported as objects; the stale
remote-tracking refs themselves are not recreated.
- **Docs:** `doc/recoverable-disk-cleanup.md`.
No code comments were added.
## Verification
- `pnpm exec vitest run src/__tests__/git-workspace-expiration.test.ts
src/__tests__/git-reproducible-managed-checkouts.test.ts -t "stale
remote-tracking|remote-tracking refs change"` (in `server/`): 5 passed.
- The existing strict test passes unchanged.
- A clean nested checkout whose remote-tracking ref points at a deleted,
garbage-collected branch is kept strictly, expires with a restore point,
and reopens on its commit with the stale commit present.
- A remote-tracking ref created after capture rolls expiration back, and
so does a reflog-only change to one.
- 6,000 repeated reflog entries for one commit do not hit the tip
budget.
- **Mutation checks:** each of these fails a case:
- dropping the preserved commits reproduces the instance's symptom
(`unpushed_managed_checkout`);
- dropping remote-tracking refs or their reflogs from the digest;
- removing deduplication (`managed_checkout_local_tip_budget`).
- `pnpm exec vitest run git-restore-point git-workspace-expiration
git-workspace-contents git-workspace-restoration
git-reproducible-managed-checkouts execution-workspaces
heartbeat-git-restore-point-resume` (in `server/`): 261 passed.
- `pnpm run typecheck` (in `server/`): pass.
- Not run: server build and boot smoke.
## Risks
- **Only with restore points enabled.** Strict expiration is unchanged.
- **Retention:** saved stale commits are kept until the restore point
expires (90 days after the tree closes by default), then deleted with
it.
- **Limits:** checkouts with more than 50 stale tips, or bundles over
the limits, skip capture and stay kept, as before.
- **Spurious rollbacks:** a fetch during the short window between
capture and deletion rolls back; the next sweep retries.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, read-only inspection of a live
instance).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
6877760904
commit
0fb152ee94
7 files changed
+171
-33
No files matched your search
@@ -141,6 +141,12 @@ remote lacks, when any reflog holds commits that are neither on the remote nor p
|
|||||||
the checked-out history the restore point bundles, when it has objects reachable from
|
the checked-out history the restore point bundles, when it has objects reachable from
|
||||||
nothing, or when its `COMMIT_EDITMSG` holds a draft that matches no commit in its
|
nothing, or when its `COMMIT_EDITMSG` holds a draft that matches no commit in its
|
||||||
history. A `COMMIT_EDITMSG` that matches a commit is accepted for every nested checkout.
|
history. A `COMMIT_EDITMSG` that matches a commit is accepted for every nested checkout.
|
||||||
|
A nested checkout's remote-tracking refs (`refs/remotes/origin/*`) and their reflogs can
|
||||||
|
point at commits the remote has since dropped, for example a branch deleted after a squash
|
||||||
|
merge. Strict expiration keeps such a checkout. In restore point mode those commits are
|
||||||
|
saved in the checkout's restore point instead, as history-only objects (the remote-tracking
|
||||||
|
refs themselves are not recreated), and any change to the remote-tracking refs after capture
|
||||||
|
rolls expiration back.
|
||||||
Each restore point also records a digest of its `HEAD` reflog and `ORIG_HEAD`; under the
|
Each restore point also records a digest of its `HEAD` reflog and `ORIG_HEAD`; under the
|
||||||
expiration locks that history must be unchanged too, so a commit followed by a reset
|
expiration locks that history must be unchanged too, so a commit followed by a reset
|
||||||
after capture rolls expiration back. Reopening
|
after capture rolls expiration back. Reopening
|
||||||
|
|||||||
@@ -199,6 +199,23 @@ describe("git restore points", () => {
|
|||||||
.rejects.toMatchObject({ reason: "restore_point_history_too_large" });
|
.rejects.toMatchObject({ reason: "restore_point_history_too_large" });
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("refuses more preserved commits than the limit before checking each one", async () => {
|
||||||
|
const extra = Array.from({ length: 51 }, (_, index) => index.toString(16).padStart(40, "0"));
|
||||||
|
await expect(captureRestorePoint({ cwd: work, upstreamCommits: upstream(), preserveCommits: extra }))
|
||||||
|
.rejects.toMatchObject({ reason: "restore_point_history_too_large" });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("saves preserved commits that exist and ignores ones that do not", async () => {
|
||||||
|
await write(work, "orphan.txt", "only reachable from a preserved commit\n");
|
||||||
|
git(work, "add", "orphan.txt"); git(work, "commit", "-q", "-m", "orphan");
|
||||||
|
const orphan = git(work, "rev-parse", "HEAD");
|
||||||
|
git(work, "reset", "-q", "--hard", "HEAD~1");
|
||||||
|
git(work, "reflog", "expire", "--expire=now", "--all");
|
||||||
|
const { manifest } = (await captureRestorePoint({ cwd: work, upstreamCommits: upstream(),
|
||||||
|
preserveCommits: [orphan, "f".repeat(40)] }))!;
|
||||||
|
expect(manifest.preserved).toEqual([orphan]);
|
||||||
|
});
|
||||||
|
|
||||||
it("keeps capture objects out of the repository", async () => {
|
it("keeps capture objects out of the repository", async () => {
|
||||||
await write(work, "notes/todo.md", "untracked work\n");
|
await write(work, "notes/todo.md", "untracked work\n");
|
||||||
const before = git(work, "count-objects", "-v");
|
const before = git(work, "count-objects", "-v");
|
||||||
|
|||||||
@@ -321,6 +321,74 @@ describeEmbeddedPostgres("reconstructable Git expiration", () => {
|
|||||||
vi.unstubAllEnvs();
|
vi.unstubAllEnvs();
|
||||||
});
|
});
|
||||||
|
|
||||||
|
async function staleRemoteTrackingRef(f: Awaited<ReturnType<typeof nestedFixture>>) {
|
||||||
|
git(f.nestedSource, "checkout", "-q", "-b", "squashed-away");
|
||||||
|
await fs.writeFile(path.join(f.nestedSource, "feature.txt"), "merged by squash, then deleted\n");
|
||||||
|
git(f.nestedSource, "add", "."); git(f.nestedSource, "commit", "-m", "Feature");
|
||||||
|
const feature = git(f.nestedSource, "rev-parse", "HEAD");
|
||||||
|
git(f.nestedSource, "push", "-q", f.nestedRemote, "squashed-away");
|
||||||
|
git(f.nestedSource, "checkout", "-q", "main");
|
||||||
|
git(f.nested.cwd, "fetch", "-q", "origin", "+refs/heads/squashed-away:refs/remotes/origin/squashed-away");
|
||||||
|
git(f.nestedRemote, "branch", "-D", "squashed-away");
|
||||||
|
git(f.nestedRemote, "reflog", "expire", "--expire=now", "--all");
|
||||||
|
git(f.nestedRemote, "gc", "-q", "--prune=now");
|
||||||
|
expect(() => git(f.nestedRemote, "cat-file", "-e", `${feature}^{commit}`)).toThrow();
|
||||||
|
return feature;
|
||||||
|
}
|
||||||
|
|
||||||
|
it("saves a clean nested checkout's stale remote-tracking commits in its restore point", async () => {
|
||||||
|
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32));
|
||||||
|
const f = await nestedFixture(false);
|
||||||
|
const feature = await staleRemoteTrackingRef(f);
|
||||||
|
expect(await f.expire({ expireBase: false })).toMatchObject({ expired: false, skipReason: "unpushed_managed_checkout" });
|
||||||
|
expect(await f.expire({ expireBase: false, restorePoint: restorePoint() })).toMatchObject({ expired: true, restorePoint: true, skipReason: null });
|
||||||
|
const recipe = (await f.row()).metadata?.gitCacheRestoration as Record<string, any>;
|
||||||
|
expect(recipe.contents.checkouts[0].restorePoint).toMatchObject({ head: f.nestedA, headOnRemote: true, worktree: null });
|
||||||
|
expect(await f.reopen()).toMatchObject({ ok: true, reopened: true });
|
||||||
|
expect(git(f.nested.cwd, "rev-parse", "HEAD")).toBe(f.nestedA);
|
||||||
|
expect(git(f.nested.cwd, "cat-file", "-t", feature)).toBe("commit");
|
||||||
|
vi.unstubAllEnvs();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("rolls back when a nested checkout's remote-tracking refs change after capture", async () => {
|
||||||
|
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32));
|
||||||
|
const f = await nestedFixture(false);
|
||||||
|
await staleRemoteTrackingRef(f);
|
||||||
|
const result = await f.expire({ expireBase: false, restorePoint: restorePoint(), beforeDelete: async () => {
|
||||||
|
const later = execFileSync("git", ["commit-tree", git(f.nested.cwd, "rev-parse", "HEAD^{tree}"), "-p", "HEAD", "-m", "Fetched after capture"],
|
||||||
|
{ cwd: f.nested.cwd, env: gitEnv, encoding: "utf8" }).trim();
|
||||||
|
git(f.nested.cwd, "update-ref", "refs/remotes/origin/fetched-later", later);
|
||||||
|
} });
|
||||||
|
expect(result).toMatchObject({ expired: false, skipReason: "restore_point_contents_changed" });
|
||||||
|
expect(await exists(f.cwd)).toBe(true);
|
||||||
|
vi.unstubAllEnvs();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("rolls back when only a nested checkout's remote-tracking reflog changes after capture", async () => {
|
||||||
|
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32));
|
||||||
|
const f = await nestedFixture(false);
|
||||||
|
await staleRemoteTrackingRef(f);
|
||||||
|
const result = await f.expire({ expireBase: false, restorePoint: restorePoint(), beforeDelete: async () => {
|
||||||
|
const tip = git(f.nested.cwd, "rev-parse", "refs/remotes/origin/squashed-away");
|
||||||
|
git(f.nested.cwd, "update-ref", "-m", "moved", "--create-reflog", "refs/remotes/origin/squashed-away", f.nestedA);
|
||||||
|
git(f.nested.cwd, "update-ref", "-m", "moved back", "refs/remotes/origin/squashed-away", tip);
|
||||||
|
} });
|
||||||
|
expect(result).toMatchObject({ expired: false, skipReason: "restore_point_contents_changed" });
|
||||||
|
vi.unstubAllEnvs();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("counts distinct reflog commits, not repeated entries, against the tip budget", async () => {
|
||||||
|
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32));
|
||||||
|
const f = await nestedFixture(false);
|
||||||
|
await unpushedNestedWork(f.nested.cwd);
|
||||||
|
const head = git(f.nested.cwd, "rev-parse", "HEAD");
|
||||||
|
const log = path.join(f.nested.cwd, ".git", "logs", "refs", "heads", "main");
|
||||||
|
const entry = `${head} ${head} Fixture <fixture@example.test> 1700000000 +0000\treset: moving to HEAD\n`;
|
||||||
|
await fs.appendFile(log, entry.repeat(6_000));
|
||||||
|
expect(await f.expire({ expireBase: false, restorePoint: restorePoint() })).toMatchObject({ expired: true, restorePoint: true, skipReason: null });
|
||||||
|
vi.unstubAllEnvs();
|
||||||
|
});
|
||||||
|
|
||||||
it("keeps a workspace whose nested checkout has unpushed work on another branch", async () => {
|
it("keeps a workspace whose nested checkout has unpushed work on another branch", async () => {
|
||||||
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32));
|
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32));
|
||||||
const f = await nestedFixture(false);
|
const f = await nestedFixture(false);
|
||||||
|
|||||||
@@ -85,11 +85,14 @@ export async function storeGitRestorePoint(input: {
|
|||||||
checkout?: string;
|
checkout?: string;
|
||||||
upstreamCommits: readonly string[];
|
upstreamCommits: readonly string[];
|
||||||
headOnRemote: boolean;
|
headOnRemote: boolean;
|
||||||
|
resolvePreserveCommits?: () => Promise<readonly string[]>;
|
||||||
limits?: Partial<RestorePointLimits>;
|
limits?: Partial<RestorePointLimits>;
|
||||||
signal?: AbortSignal;
|
signal?: AbortSignal;
|
||||||
}): Promise<{ ref: GitRestorePointRef; ledgerId: string; manifest: RestorePointManifest } | null> {
|
}): Promise<{ ref: GitRestorePointRef; ledgerId: string; manifest: RestorePointManifest } | null> {
|
||||||
const history = await restorePointHistoryDigest(input.cwd, input.signal);
|
const history = await restorePointHistoryDigest(input.cwd, input.signal);
|
||||||
const captured = await captureRestorePoint({ cwd: input.cwd, upstreamCommits: input.upstreamCommits, limits: input.limits, signal: input.signal });
|
const preserveCommits = await input.resolvePreserveCommits?.();
|
||||||
|
const captured = await captureRestorePoint({ cwd: input.cwd, upstreamCommits: input.upstreamCommits, preserveCommits,
|
||||||
|
limits: input.limits, signal: input.signal });
|
||||||
if (!captured) return null;
|
if (!captured) return null;
|
||||||
if (await restorePointHistoryDigest(input.cwd, input.signal) !== history) throw new RestorePointRestoreFailure("restore_point_history_changed");
|
if (await restorePointHistoryDigest(input.cwd, input.signal) !== history) throw new RestorePointRestoreFailure("restore_point_history_changed");
|
||||||
const id = randomUUID();
|
const id = randomUUID();
|
||||||
|
|||||||
@@ -331,8 +331,16 @@ async function existingCommits(cwd: string, commits: readonly string[], signal?:
|
|||||||
return present;
|
return present;
|
||||||
}
|
}
|
||||||
|
|
||||||
async function unpreservedHistory(cwd: string, head: string, boundary: readonly string[], signal?: AbortSignal) {
|
async function presentCommits(cwd: string, commits: readonly string[], signal?: AbortSignal) {
|
||||||
const candidates = new Set<string>();
|
const unique = [...new Set(commits.filter((commit) => SHA.test(commit)))];
|
||||||
|
if (unique.length === 0) return [];
|
||||||
|
const output = await gitText(cwd, ["cat-file", "--batch-check=%(objectname) %(objecttype)"], { signal, input: unique.join("\n") + "\n" });
|
||||||
|
return output.split("\n").map((line) => line.split(" ")).filter(([sha, type]) => SHA.test(sha ?? "") && type === "commit").map(([sha]) => sha!);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function unpreservedHistory(cwd: string, head: string, boundary: readonly string[], signal?: AbortSignal, extra: readonly string[] = []) {
|
||||||
|
if (new Set(extra).size > MAX_PRESERVED_COMMITS) throw new RestorePointSkip("restore_point_history_too_large");
|
||||||
|
const candidates = new Set<string>(await presentCommits(cwd, extra, signal));
|
||||||
const reflog = await gitText(cwd, ["reflog", "show", "--format=%H", "HEAD"], { signal }).catch(() => "");
|
const reflog = await gitText(cwd, ["reflog", "show", "--format=%H", "HEAD"], { signal }).catch(() => "");
|
||||||
for (const line of reflog.split("\n")) if (SHA.test(line)) candidates.add(line);
|
for (const line of reflog.split("\n")) if (SHA.test(line)) candidates.add(line);
|
||||||
const original = await gitText(cwd, ["rev-parse", "--verify", "-q", "ORIG_HEAD^{commit}"], { signal }).catch(() => "");
|
const original = await gitText(cwd, ["rev-parse", "--verify", "-q", "ORIG_HEAD^{commit}"], { signal }).catch(() => "");
|
||||||
@@ -385,6 +393,7 @@ export function decodeRestorePointPayload(payload: Buffer): { manifest: RestoreP
|
|||||||
export async function captureRestorePoint(input: {
|
export async function captureRestorePoint(input: {
|
||||||
cwd: string;
|
cwd: string;
|
||||||
upstreamCommits: readonly string[];
|
upstreamCommits: readonly string[];
|
||||||
|
preserveCommits?: readonly string[];
|
||||||
limits?: Partial<RestorePointLimits>;
|
limits?: Partial<RestorePointLimits>;
|
||||||
signal?: AbortSignal;
|
signal?: AbortSignal;
|
||||||
}): Promise<{ payload: Buffer; manifest: RestorePointManifest } | null> {
|
}): Promise<{ payload: Buffer; manifest: RestorePointManifest } | null> {
|
||||||
@@ -396,7 +405,7 @@ export async function captureRestorePoint(input: {
|
|||||||
const { store, env } = await snapshotStore(cwd, temporary);
|
const { store, env } = await snapshotStore(cwd, temporary);
|
||||||
const snapshot = await snapshotCommits(cwd, env, temporary, modes, input.signal);
|
const snapshot = await snapshotCommits(cwd, env, temporary, modes, input.signal);
|
||||||
const boundary = await existingCommits(cwd, input.upstreamCommits, input.signal);
|
const boundary = await existingCommits(cwd, input.upstreamCommits, input.signal);
|
||||||
const preserved = await unpreservedHistory(cwd, snapshot.head, boundary, input.signal);
|
const preserved = await unpreservedHistory(cwd, snapshot.head, boundary, input.signal, input.preserveCommits);
|
||||||
const headOnUpstream = await (async () => {
|
const headOnUpstream = await (async () => {
|
||||||
for (const tip of boundary) {
|
for (const tip of boundary) {
|
||||||
if (await git(cwd, ["merge-base", "--is-ancestor", snapshot.head, tip], { signal: input.signal }).then(() => true, () => false)) return true;
|
if (await git(cwd, ["merge-base", "--is-ancestor", snapshot.head, tip], { signal: input.signal }).then(() => true, () => false)) return true;
|
||||||
@@ -559,5 +568,9 @@ export async function restorePointHistoryDigest(input: string, signal?: AbortSig
|
|||||||
const cwd = await fs.realpath(input);
|
const cwd = await fs.realpath(input);
|
||||||
const reflog = await gitText(cwd, ["reflog", "show", "--format=%H", "HEAD"], { signal }).catch(() => "");
|
const reflog = await gitText(cwd, ["reflog", "show", "--format=%H", "HEAD"], { signal }).catch(() => "");
|
||||||
const original = await gitText(cwd, ["rev-parse", "--verify", "-q", "ORIG_HEAD^{commit}"], { signal }).catch(() => "");
|
const original = await gitText(cwd, ["rev-parse", "--verify", "-q", "ORIG_HEAD^{commit}"], { signal }).catch(() => "");
|
||||||
return createHash("sha256").update(JSON.stringify([reflog.split("\n").filter((line) => SHA.test(line)), SHA.test(original) ? original : null])).digest("hex");
|
const remotes = await gitText(cwd, ["for-each-ref", "--format=%(objectname) %(refname)", "refs/remotes"], { signal }).catch(() => "");
|
||||||
|
const remoteReflogs = (await gitText(cwd, ["reflog", "show", "--all", "--format=%H %gD"], { signal }).catch(() => ""))
|
||||||
|
.split("\n").filter((line) => / refs\/remotes\//.test(line));
|
||||||
|
return createHash("sha256").update(JSON.stringify([reflog.split("\n").filter((line) => SHA.test(line)), SHA.test(original) ? original : null,
|
||||||
|
remotes.split("\n").filter(Boolean), remoteReflogs])).digest("hex");
|
||||||
}
|
}
|
||||||
@@ -13,7 +13,7 @@ import { abandonGitRestorePoint, abandonGitRestorePointsByKey, activateGitRestor
|
|||||||
import { withIsolatedRepositorySourceLock, verifiedRecordedIsolatedRemoteSource } from "./isolated-remote-workspace-source.js";
|
import { withIsolatedRepositorySourceLock, verifiedRecordedIsolatedRemoteSource } from "./isolated-remote-workspace-source.js";
|
||||||
import { managedProjectCheckoutName } from "./managed-project-checkout-name.js";
|
import { managedProjectCheckoutName } from "./managed-project-checkout-name.js";
|
||||||
import type { GitRemoteAuthProvider } from "./git-credentials.js";
|
import type { GitRemoteAuthProvider } from "./git-credentials.js";
|
||||||
import { assertReproducibleGitFiles, assertReproducibleBaseGitStorage, assertReproducibleManagedCheckout, inspectGitCacheRecipe, inspectGitCacheRecipeForRestorePoint, readGitCacheRestoration, GIT_CACHE_RESTORATION_SCHEMA_V2,
|
import { assertReproducibleGitFiles, assertReproducibleBaseGitStorage, assertReproducibleManagedCheckout, provenManagedCheckoutHistory, inspectGitCacheRecipe, inspectGitCacheRecipeForRestorePoint, readGitCacheRestoration, GIT_CACHE_RESTORATION_SCHEMA_V2,
|
||||||
verifyRemoteGitCommit, gitCacheExpirationLockLocations, recoverGitCacheExpirationLocks, GitCacheRestorationFailure, type ManagedCheckoutProver, gitCacheCommand, readGitCacheExpiration, finishGitCacheTrash, type GitCacheExpiration, type GitCacheRestoration, GIT_CACHE_RESTORATION_KEY, GIT_CACHE_EXPIRATION_REASON } from "./git-workspace-restoration.js";
|
verifyRemoteGitCommit, gitCacheExpirationLockLocations, recoverGitCacheExpirationLocks, GitCacheRestorationFailure, type ManagedCheckoutProver, gitCacheCommand, readGitCacheExpiration, finishGitCacheTrash, type GitCacheExpiration, type GitCacheRestoration, GIT_CACHE_RESTORATION_KEY, GIT_CACHE_EXPIRATION_REASON } from "./git-workspace-restoration.js";
|
||||||
|
|
||||||
async function allocatedBytes(target: string, shouldContinue: () => boolean): Promise<number> {
|
async function allocatedBytes(target: string, shouldContinue: () => boolean): Promise<number> {
|
||||||
@@ -294,7 +294,9 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Worksp
|
|||||||
} else if (!headOnRemote) throw new Error("upstream_commit_unverified");
|
} else if (!headOnRemote) throw new Error("upstream_commit_unverified");
|
||||||
for (const [name, need] of [...checkoutRestorePoints.entries()].sort(([a], [b]) => a.localeCompare(b))) {
|
for (const [name, need] of [...checkoutRestorePoints.entries()].sort(([a], [b]) => a.localeCompare(b))) {
|
||||||
const storedCheckout = await storeGitRestorePoint({ db, storage: options.restorePoint!.storage, workspace: owner,
|
const storedCheckout = await storeGitRestorePoint({ db, storage: options.restorePoint!.storage, workspace: owner,
|
||||||
cwd: need.path, checkout: name, generation, headOnRemote: need.headOnRemote, limits: options.restorePoint!.limits,
|
cwd: need.path, checkout: name, generation, headOnRemote: need.headOnRemote,
|
||||||
|
resolvePreserveCommits: async () => (await provenManagedCheckoutHistory(need.path, need.repoUrl, auth, options.signal)).preserve,
|
||||||
|
limits: options.restorePoint!.limits,
|
||||||
signal: options.signal, upstreamCommits: await restorePointBoundary(need.path, need.repoUrl, need.headOnRemote, auth, options.signal) });
|
signal: options.signal, upstreamCommits: await restorePointBoundary(need.path, need.repoUrl, need.headOnRemote, auth, options.signal) });
|
||||||
if (!storedCheckout) {
|
if (!storedCheckout) {
|
||||||
if (!need.headOnRemote) throw new Error("unpushed_managed_checkout");
|
if (!need.headOnRemote) throw new Error("unpushed_managed_checkout");
|
||||||
|
|||||||
@@ -250,7 +250,7 @@ async function assertCommitMessageSaved(cwd: string, gitDir: string, signal?: Ab
|
|||||||
|
|
||||||
export async function assertReproducibleManagedCheckout(checkout: string, input: {
|
export async function assertReproducibleManagedCheckout(checkout: string, input: {
|
||||||
repoUrl: string; resolveGitAuth: GitRemoteAuthProvider; signal?: AbortSignal; restorePoint?: boolean;
|
repoUrl: string; resolveGitAuth: GitRemoteAuthProvider; signal?: AbortSignal; restorePoint?: boolean;
|
||||||
}): Promise<{ headOnRemote: boolean }> {
|
}): Promise<{ headOnRemote: boolean; preserve: string[] }> {
|
||||||
const { signal } = input;
|
const { signal } = input;
|
||||||
const gitDir = await fs.lstat(path.join(checkout, ".git")).catch(() => null);
|
const gitDir = await fs.lstat(path.join(checkout, ".git")).catch(() => null);
|
||||||
if (!gitDir?.isDirectory()) throw new Error("unproven_managed_checkout");
|
if (!gitDir?.isDirectory()) throw new Error("unproven_managed_checkout");
|
||||||
@@ -266,30 +266,10 @@ export async function assertReproducibleManagedCheckout(checkout: string, input:
|
|||||||
const refs = (await gitCacheCommand(checkout, ["for-each-ref", "--format=%(refname)"], signal)).trim().split("\n").filter(Boolean);
|
const refs = (await gitCacheCommand(checkout, ["for-each-ref", "--format=%(refname)"], signal)).trim().split("\n").filter(Boolean);
|
||||||
if (refs.some(ref => !/^refs\/(heads|tags|remotes\/origin)\//.test(ref))) throw new Error("unsupported_managed_checkout_refs");
|
if (refs.some(ref => !/^refs\/(heads|tags|remotes\/origin)\//.test(ref))) throw new Error("unsupported_managed_checkout_refs");
|
||||||
if (input.restorePoint) {
|
if (input.restorePoint) {
|
||||||
const head = (await gitCacheCommand(checkout, ["rev-parse", "--verify", "HEAD^{commit}"], signal)).trim();
|
const { headOnRemote, preserve } = await provenManagedCheckoutHistory(checkout, input.repoUrl, input.resolveGitAuth, signal);
|
||||||
const shas = (output: string) => output.split("\n").map(line => line.trim()).filter(sha => shaPattern.test(sha));
|
|
||||||
const covered = new Set([head,
|
|
||||||
...shas(await gitCacheCommand(checkout, ["reflog", "show", "--format=%H", "HEAD"], signal).catch(() => "")),
|
|
||||||
...shas(await gitCacheCommand(checkout, ["rev-parse", "--verify", "-q", "ORIG_HEAD^{commit}"], signal).catch(() => ""))]);
|
|
||||||
const recorded = new Set([
|
|
||||||
...shas(await gitCacheCommand(checkout, ["for-each-ref", "--format=%(objectname)", "refs/heads", "refs/tags"], signal)),
|
|
||||||
...shas(await gitCacheCommand(checkout, ["reflog", "show", "--all", "--format=%H"], signal).catch(() => "")),
|
|
||||||
]);
|
|
||||||
const current = (await gitCacheCommand(checkout, ["symbolic-ref", "--quiet", "HEAD"], signal).catch(() => "")).trim();
|
|
||||||
const otherTips = (await gitCacheCommand(checkout, ["for-each-ref", "--format=%(objectname) %(refname)", "refs/heads", "refs/tags"], signal))
|
|
||||||
.split("\n").map(line => line.trim().split(" ")).filter(([sha, ref]) => shaPattern.test(sha ?? "") && ref !== current).map(([sha]) => sha!);
|
|
||||||
if (otherTips.length && await hasCommitsMissingUpstream(checkout, otherTips, input.repoUrl, input.resolveGitAuth, signal, [head])) {
|
|
||||||
throw new Error("unpushed_managed_checkout_branches");
|
|
||||||
}
|
|
||||||
const uncovered = [...recorded].filter(sha => !covered.has(sha));
|
|
||||||
if (uncovered.length + covered.size > MANAGED_CHECKOUT_LOCAL_TIP_LIMIT) throw new Error("managed_checkout_local_tip_budget");
|
|
||||||
if (uncovered.length && await hasCommitsMissingUpstream(checkout, uncovered, input.repoUrl, input.resolveGitAuth, signal, [...covered])) {
|
|
||||||
throw new Error("unpushed_managed_checkout_branches");
|
|
||||||
}
|
|
||||||
const headOnRemote = !await hasCommitsMissingUpstream(checkout, [head], input.repoUrl, input.resolveGitAuth, signal);
|
|
||||||
const unreachable = await gitCacheCommand(checkout, ["fsck", "--connectivity-only", "--unreachable", "--no-progress"], signal);
|
const unreachable = await gitCacheCommand(checkout, ["fsck", "--connectivity-only", "--unreachable", "--no-progress"], signal);
|
||||||
if (unreachable.trim()) throw new Error("unreachable_managed_checkout_objects");
|
if (unreachable.trim()) throw new Error("unreachable_managed_checkout_objects");
|
||||||
return { headOnRemote };
|
return { headOnRemote, preserve };
|
||||||
}
|
}
|
||||||
const localTips = new Set([
|
const localTips = new Set([
|
||||||
...(await gitCacheCommand(checkout, ["rev-parse", "--verify", "HEAD^{commit}"], signal)).split("\n"),
|
...(await gitCacheCommand(checkout, ["rev-parse", "--verify", "HEAD^{commit}"], signal)).split("\n"),
|
||||||
@@ -302,13 +282,61 @@ export async function assertReproducibleManagedCheckout(checkout: string, input:
|
|||||||
}
|
}
|
||||||
const unreachable = await gitCacheCommand(checkout, ["fsck", "--connectivity-only", "--unreachable", "--no-progress"], signal);
|
const unreachable = await gitCacheCommand(checkout, ["fsck", "--connectivity-only", "--unreachable", "--no-progress"], signal);
|
||||||
if (unreachable.trim()) throw new Error("unreachable_managed_checkout_objects");
|
if (unreachable.trim()) throw new Error("unreachable_managed_checkout_objects");
|
||||||
return { headOnRemote: true };
|
return { headOnRemote: true, preserve: [] };
|
||||||
}
|
}
|
||||||
|
|
||||||
const MANAGED_CHECKOUT_LOCAL_TIP_LIMIT = 5_000;
|
const MANAGED_CHECKOUT_LOCAL_TIP_LIMIT = 5_000;
|
||||||
|
|
||||||
|
export async function provenManagedCheckoutHistory(checkout: string, repoUrl: string, resolveGitAuth: GitRemoteAuthProvider,
|
||||||
|
signal?: AbortSignal): Promise<{ headOnRemote: boolean; preserve: string[] }> {
|
||||||
|
const head = (await gitCacheCommand(checkout, ["rev-parse", "--verify", "HEAD^{commit}"], signal)).trim();
|
||||||
|
const shas = (output: string) => output.split("\n").map(line => line.trim()).filter(sha => shaPattern.test(sha));
|
||||||
|
const covered = new Set([head,
|
||||||
|
...shas(await gitCacheCommand(checkout, ["reflog", "show", "--format=%H", "HEAD"], signal).catch(() => "")),
|
||||||
|
...shas(await gitCacheCommand(checkout, ["rev-parse", "--verify", "-q", "ORIG_HEAD^{commit}"], signal).catch(() => ""))]);
|
||||||
|
const reflogs = [...new Set((await gitCacheCommand(checkout, ["reflog", "show", "--all", "--format=%H %gD"], signal).catch(() => ""))
|
||||||
|
.split("\n").map(line => line.trim().split(" ")).filter(([sha]) => shaPattern.test(sha ?? ""))
|
||||||
|
.map(([sha, selector]) => `${sha} ${(selector ?? "").startsWith("refs/remotes/") ? "remote" : "local"}`))].map(entry => entry.split(" "));
|
||||||
|
if (reflogs.length > MANAGED_CHECKOUT_LOCAL_TIP_LIMIT) throw new Error("managed_checkout_local_tip_budget");
|
||||||
|
const remote = (kind: string | undefined) => kind === "remote";
|
||||||
|
const recorded = new Set([
|
||||||
|
...shas(await gitCacheCommand(checkout, ["for-each-ref", "--format=%(objectname)", "refs/heads", "refs/tags"], signal)),
|
||||||
|
...reflogs.filter(([, selector]) => !remote(selector)).map(([sha]) => sha!),
|
||||||
|
]);
|
||||||
|
const remoteTracked = [...new Set([
|
||||||
|
...shas(await gitCacheCommand(checkout, ["for-each-ref", "--format=%(objectname)", "refs/remotes"], signal)),
|
||||||
|
...reflogs.filter(([, selector]) => remote(selector)).map(([sha]) => sha!),
|
||||||
|
])];
|
||||||
|
const current = (await gitCacheCommand(checkout, ["symbolic-ref", "--quiet", "HEAD"], signal).catch(() => "")).trim();
|
||||||
|
const otherTips = (await gitCacheCommand(checkout, ["for-each-ref", "--format=%(objectname) %(refname)", "refs/heads", "refs/tags"], signal))
|
||||||
|
.split("\n").map(line => line.trim().split(" ")).filter(([sha, ref]) => shaPattern.test(sha ?? "") && ref !== current).map(([sha]) => sha!);
|
||||||
|
if (otherTips.length && await hasCommitsMissingUpstream(checkout, otherTips, repoUrl, resolveGitAuth, signal, [head])) {
|
||||||
|
throw new Error("unpushed_managed_checkout_branches");
|
||||||
|
}
|
||||||
|
const uncovered = [...recorded].filter(sha => !covered.has(sha));
|
||||||
|
if (uncovered.length + covered.size + remoteTracked.length > MANAGED_CHECKOUT_LOCAL_TIP_LIMIT) throw new Error("managed_checkout_local_tip_budget");
|
||||||
|
if (uncovered.length && await hasCommitsMissingUpstream(checkout, uncovered, repoUrl, resolveGitAuth, signal, [...covered])) {
|
||||||
|
throw new Error("unpushed_managed_checkout_branches");
|
||||||
|
}
|
||||||
|
const headOnRemote = !await hasCommitsMissingUpstream(checkout, [head], repoUrl, resolveGitAuth, signal);
|
||||||
|
const preserve = remoteTracked.length
|
||||||
|
? await tipsMissingUpstream(checkout, remoteTracked, repoUrl, resolveGitAuth, signal, [...covered, ...recorded]) : [];
|
||||||
|
return { headOnRemote, preserve };
|
||||||
|
}
|
||||||
|
|
||||||
|
async function tipsMissingUpstream(checkout: string, tips: string[], repoUrl: string,
|
||||||
|
resolveGitAuth: GitRemoteAuthProvider, signal?: AbortSignal, covered: string[] = []) {
|
||||||
|
const missing = new Set(await commitsMissingUpstream(checkout, tips, repoUrl, resolveGitAuth, signal, covered));
|
||||||
|
return tips.filter(tip => missing.has(tip)).sort();
|
||||||
|
}
|
||||||
|
|
||||||
async function hasCommitsMissingUpstream(checkout: string, localTips: string[], repoUrl: string,
|
async function hasCommitsMissingUpstream(checkout: string, localTips: string[], repoUrl: string,
|
||||||
resolveGitAuth: GitRemoteAuthProvider, signal?: AbortSignal, covered: string[] = []) {
|
resolveGitAuth: GitRemoteAuthProvider, signal?: AbortSignal, covered: string[] = []) {
|
||||||
|
return (await commitsMissingUpstream(checkout, localTips, repoUrl, resolveGitAuth, signal, covered, 1)).length > 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function commitsMissingUpstream(checkout: string, localTips: string[], repoUrl: string,
|
||||||
|
resolveGitAuth: GitRemoteAuthProvider, signal?: AbortSignal, covered: string[] = [], limit?: number) {
|
||||||
const store = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-git-checkout-proof-"));
|
const store = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-git-checkout-proof-"));
|
||||||
try {
|
try {
|
||||||
const options = { timeout: 120_000, signal, maxBuffer: 16 * 1024 * 1024, cwd: os.tmpdir(),
|
const options = { timeout: 120_000, signal, maxBuffer: 16 * 1024 * 1024, cwd: os.tmpdir(),
|
||||||
@@ -324,8 +352,9 @@ async function hasCommitsMissingUpstream(checkout: string, localTips: string[],
|
|||||||
if (signal?.aborted) throw error;
|
if (signal?.aborted) throw error;
|
||||||
throw new Error("managed_checkout_upstream_unavailable");
|
throw new Error("managed_checkout_upstream_unavailable");
|
||||||
}
|
}
|
||||||
const missing = await execFile("git", ["-C", store, "rev-list", "--max-count=1", ...localTips, "--not", "--glob=refs/upstream/*", ...covered], options);
|
const missing = await execFile("git", ["-C", store, "rev-list", ...(limit ? [`--max-count=${limit}`] : []), ...localTips,
|
||||||
return missing.stdout.trim() !== "";
|
"--not", "--glob=refs/upstream/*", ...covered], options);
|
||||||
|
return missing.stdout.split("\n").map(line => line.trim()).filter(Boolean);
|
||||||
} finally {
|
} finally {
|
||||||
await fs.rm(store, { recursive: true, force: true });
|
await fs.rm(store, { recursive: true, force: true });
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in new issue
Block a user