mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-24 01:41:42 +02:00
feat: expire closed git worktrees with unsaved work behind restore points (#17248)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Git cache expiration removes a closed issue's worktree only when the
remote can reproduce it exactly
> - Most closed worktrees hold unpushed commits or uncommitted files, so
they are kept forever and disks fill up
> - #17238 added restore points (an encrypted save of exactly what the
remote cannot reproduce) and #17241 added their lifecycle
> - This pull request lets expiration remove those worktrees after
saving a restore point, and restores them exactly when the work resumes
> - The benefit is bounded disk use for closed work without losing
anything a user could need, behind a setting that is off by default
## Linked Issues or Issue Description
Refs #17238, #17241. Bug report described in-PR (bug report template).
**What happened?**
With `PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true` and dry run off, one
instance's last 111 expiration checks removed nothing:
| Skip reason | Checks |
|---|---|
| `dirty_or_untracked_files` | 39 |
| `unpushed_managed_checkout` | 32 |
| `live_or_unresolved_consumer` | 21 |
| `unique_ignored_files` | 9 |
| commit not on the remote | 10 |
Without expiration, the terminal workspace reaper archives only merged,
clean worktrees. Closed issues' worktrees therefore accumulate without
bound.
**Expected behavior**
A closed worktree whose only obstacle is unsaved work can be removed
after that work is saved, and restored exactly if the issue is reopened
or work resumes.
**Steps to reproduce**
1. Close an issue whose worktree has an unpushed commit or an
uncommitted file.
2. Enable git cache expiration with dry run off.
3. The worktree is never removed.
**Paperclip version or commit:** `master` at 12bb7e1976.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
**Expiration** (`server/src/services/git-workspace-expiration.ts`):
- **When it applies:** with `PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true`
(default `false`), a worktree that fails the reproducibility check for a
relaxable reason is reconsidered once its issue tree has been closed for
`PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS` (default `3`). The
relaxable reasons are uncommitted or untracked files, a commit not on
the remote, reflog or `ORIG_HEAD` history, and ignored `.env` files. A
leftover `COMMIT_EDITMSG` still blocks expiration unless it matches a
commit in the worktree's reflog, because restore points do not save
draft messages.
- **What still blocks:** a relaxed inspection
(`inspectGitCacheRecipeForRestorePoint`) keeps every other check. Nested
managed checkouts, submodules, special git state, local config and other
unique ignored files still prevent expiration.
- **Capture:** inside the existing source and lifecycle locks, a restore
point is captured with the #17238 library against a remote-verified
boundary (the `HEAD` itself, or its merge base with `origin/HEAD`), then
encrypted, uploaded, read back and hash-checked
(`git-restore-point-storage.ts`).
- **Commit before touching disk:** it is activated in the #17241 table
in the same transaction that archives the workspace with a
`paperclip.git-cache-restoration.v2` recipe referencing it, still before
anything on disk is touched.
- **Final check:** under the file locks, the worktree must still match
the captured snapshot (`HEAD`, index, working tree, modes). Otherwise
the existing rollback runs. The rollback now also drops the restore
point reference and returns the recipe to v1, so a later restore can
never apply a stale restore point over the live worktree; the stored
restore point is marked superseded.
- **Failure paths:** the `pending` row is inserted in a transaction that
first registers a `git-restore-points` notification. If the upload or
read-back fails, the row is marked `superseded` so retention deletes any
uploaded object. Nothing is deleted from disk.
- **Encryption key:** restore points are refused
(`restore_point_key_not_durable`) on `s3` storage unless the secrets
master key comes from `PAPERCLIP_SECRETS_MASTER_KEY`.
- **Dry run** reports `restorePoint: true` candidates without storing
anything.
**Restore** (`git-workspace-restoration.ts`, `workspace-runtime.ts`,
`execution-workspaces.ts`):
- **Recipe versions:** `readGitCacheRestoration` and
`savedGitWorkspaceRestoration` accept v1 without a restore point, or v2
with a valid one. Older servers reject v2 instead of restoring the
commit without the saved work.
- **Before the saved commit is needed:** reopening
(`restoreGitCacheBase`) and the runtime path import the bundle when the
saved commit is missing locally.
- **After the worktree is re-added:** the runtime always imports the
bundle, so history-only commits are present again as objects even when
the base was rebuilt (they are not recreated as branches), then applies
the restore point, verifies the result, and records `appliedAt`. Reopen
persists that through a callback, as it already does for saved contents.
- **Never overwriting:** a worktree that already matches is only marked
applied. A worktree that diverged from the saved commit's clean state is
never overwritten; the restore fails closed and can be retried. Missing,
corrupt or expired restore points also fail closed.
**Retention:** the #17241 worker's expiry hook now marks the recipe's
restore point expired before its object is deleted.
**Config and docs:**
- `server/src/config.ts` adds the two settings;
`doc/recoverable-disk-cleanup.md` and `doc/DEVELOPING.md` describe them.
- No code comments were added.
**Not in this PR (follow-ups):**
- workspaces with nested `.paperclip-repositories` checkouts;
- removing idle worktrees of issues that are still open;
- starting a fresh workspace with a warning when a restore point has
expired.
## Verification
- `pnpm exec vitest run
src/__tests__/git-restore-point-expiration.test.ts` (in `server/`): 12
passed, with real repositories and embedded Postgres. The cases:
- **Round trip:** an unpushed commit, staged and unstaged edits of one
file, a deletion, and an untracked file are restored exactly on reopen,
`.env` is dropped, and the restore point is active. Restoring through
the runtime path works the same way, and a later run never re-applies
over newer work.
- **Kept:** a worktree that is not yet due, one with the feature
disabled (today's skip reason), and one whose encryption key is not
durable.
- **Change after capture:** rolls back, keeps the worktree, returns the
recipe to v1 and supersedes the restore point.
- **Missing restore point:** fails closed, keeps the row archived and
unapplied, and fails closed again on retry.
- **Dry run:** stores nothing.
- **Reproducible worktrees:** expire as before, with a v1 recipe.
- **History-only commits:** reopen imports them even after they are
pruned from the base.
- **Draft messages:** a draft commit message keeps the worktree.
- **Upload failure:** a failed upload marks the pending restore point
for cleanup and keeps the worktree.
- **Mutation checks:** each of these fails at least one case:
- removing the pre-deletion snapshot check;
- keeping the restore point reference on rollback;
- removing the runtime apply;
- removing the re-apply guard (the divergence check then fails closed
instead of overwriting);
- skipping the import for a matching worktree, relaxing the draft
message guard, or not marking a failed upload's row.
- `pnpm exec vitest run
src/__tests__/git-restore-point-expiration-config.test.ts`: 5 passed.
- `pnpm exec vitest run git-workspace-expiration git-workspace-contents
git-restore-point workspace-runtime execution-workspaces` (in
`server/`): 529 passed on the first revision. Two
`workspace-runtime.test.ts` cases about remote base refs fail, and they
also fail on unmodified `master` on this machine. After the review
fixes, the expiration, contents, restore point and execution workspace
suites passed again: 222 tests.
- `pnpm run typecheck` (in `server/`): pass.
- The server builds, the runtime bundle builds, and
`scripts/server-boot-smoke.sh node server/dist-runtime/index.mjs`
reaches healthy.
## Risks
- **Off by default:** nothing changes unless both
`PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true` and
`PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true` are set. Roll out with
`PAPERCLIP_GIT_CACHE_EXPIRATION_DRY_RUN=true` first to see which
worktrees would receive restore points.
- **Data at rest:** restore points contain source and uncommitted files,
encrypted with keys wrapped by the instance secrets key. Losing that key
makes them unrecoverable, which is why `s3` storage requires an
environment-provided key. Untracked `.env` files are not stored.
- **Interrupted apply:** a crash in the middle of applying a restore
point leaves a worktree that neither matches the restore point nor is
clean. The restore then fails closed rather than overwriting it, and an
operator must reset that worktree to retry.
- **Rollback:** disable the setting. Already-expired worktrees keep
their v2 recipes and restore points, so reverting the code would leave
those recipes unreadable until restored by a server with this change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, local server builds and boots,
read-only measurements on live instances).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
9bf7dc0c8f
commit
6d44b80a5e
11 files changed
+760
-24
No files matched your search
@@ -0,0 +1,25 @@
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { loadConfig } from "../config.ts";
|
||||
|
||||
describe("git restore point expiration config parsing", () => {
|
||||
afterEach(() => {
|
||||
vi.unstubAllEnvs();
|
||||
});
|
||||
|
||||
it("is disabled by default with a three-day delay", () => {
|
||||
vi.stubEnv("PAPERCLIP_GIT_RESTORE_POINTS_ENABLED", undefined);
|
||||
vi.stubEnv("PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS", undefined);
|
||||
expect(loadConfig()).toMatchObject({ gitRestorePointsEnabled: false, gitRestorePointTerminalDelayDays: 3 });
|
||||
});
|
||||
|
||||
it("accepts explicit settings", () => {
|
||||
vi.stubEnv("PAPERCLIP_GIT_RESTORE_POINTS_ENABLED", "true");
|
||||
vi.stubEnv("PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS", "0");
|
||||
expect(loadConfig()).toMatchObject({ gitRestorePointsEnabled: true, gitRestorePointTerminalDelayDays: 0 });
|
||||
});
|
||||
|
||||
it.each(["", "-1", "many"])("falls back to three days for %j", (value) => {
|
||||
vi.stubEnv("PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS", value);
|
||||
expect(loadConfig()).toMatchObject({ gitRestorePointTerminalDelayDays: 3 });
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,244 @@
|
||||
import { promises as fs } from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { companies, createDb, executionWorkspaceRestorePoints, executionWorkspaces, issues } from "@paperclipai/db";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { materializeIsolatedRemoteWorkspaceSource } from "../services/isolated-remote-workspace-source.js";
|
||||
import { ensurePersistedExecutionWorkspaceAvailable, realizeExecutionWorkspace } from "../services/workspace-runtime.js";
|
||||
import { executionWorkspaceService } from "../services/execution-workspaces.js";
|
||||
import { expireReconstructableGitWorkspace } from "../services/git-workspace-expiration.js";
|
||||
import { getStorageService } from "../storage/index.js";
|
||||
|
||||
const gitEnv = { ...process.env, GIT_AUTHOR_NAME: "Fixture", GIT_AUTHOR_EMAIL: "fixture@example.test",
|
||||
GIT_COMMITTER_NAME: "Fixture", GIT_COMMITTER_EMAIL: "fixture@example.test", GIT_TERMINAL_PROMPT: "0" };
|
||||
const git = (cwd: string, ...args: string[]) => execFileSync("git", args, { cwd, env: gitEnv, encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }).trim();
|
||||
const exists = (target: string) => fs.lstat(target).then(() => true, () => false);
|
||||
const DAY_MS = 24 * 60 * 60 * 1000;
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = support.supported ? describe : describe.skip;
|
||||
describeEmbeddedPostgres("git restore point expiration", () => {
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let root: string;
|
||||
const previousHome = process.env.PAPERCLIP_HOME;
|
||||
const previousInstance = process.env.PAPERCLIP_INSTANCE_ID;
|
||||
beforeAll(async () => {
|
||||
database = await startEmbeddedPostgresTestDatabase("git-restore-point-expiration-");
|
||||
db = createDb(database.connectionString);
|
||||
root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-restore-point-expiration-"));
|
||||
process.env.PAPERCLIP_HOME = path.join(root, "home");
|
||||
process.env.PAPERCLIP_INSTANCE_ID = "default";
|
||||
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32));
|
||||
});
|
||||
afterEach(async () => {
|
||||
await db.delete(executionWorkspaceRestorePoints);
|
||||
await db.execute(sql`TRUNCATE TABLE companies CASCADE`);
|
||||
});
|
||||
afterAll(async () => {
|
||||
vi.unstubAllEnvs();
|
||||
if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome;
|
||||
if (previousInstance === undefined) delete process.env.PAPERCLIP_INSTANCE_ID; else process.env.PAPERCLIP_INSTANCE_ID = previousInstance;
|
||||
await db.$client.end({ timeout: 1 }); await database.cleanup(); await fs.rm(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
async function fixture() {
|
||||
const companyId = randomUUID(), issueId = randomUUID(), workspaceId = randomUUID();
|
||||
const location = path.join(root, companyId), source = path.join(location, "author"), remote = path.join(location, "remote.git");
|
||||
await fs.mkdir(source, { recursive: true });
|
||||
git(source, "init", "--initial-branch=main");
|
||||
await fs.writeFile(path.join(source, "marker.txt"), "commit A\n");
|
||||
await fs.writeFile(path.join(source, "remove-me.txt"), "tracked\n");
|
||||
await fs.writeFile(path.join(source, ".gitignore"), "node_modules/\n.env\n.claude/settings.local.json\n");
|
||||
git(source, "add", "."); git(source, "commit", "-m", "A");
|
||||
git(location, "clone", "--bare", source, remote); git(source, "remote", "add", "origin", remote);
|
||||
await db.insert(companies).values({ id: companyId, name: "Restore point fixture", issuePrefix: `RP${companyId.slice(0, 6)}` });
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Finished task", status: "done", completedAt: new Date(Date.now() - 4 * DAY_MS) });
|
||||
const repoUrl = pathToFileURL(remote).href;
|
||||
const baseCwd = await materializeIsolatedRemoteWorkspaceSource({ companyId, issueId, repoUrl, resolveGitAuth: async () => null });
|
||||
const base = { baseCwd, source: "task_session" as const, projectId: null, workspaceId: null, repoUrl, repoRef: "origin/main" };
|
||||
const workspace = await realizeExecutionWorkspace({ base, issue: { id: issueId, identifier: "RP-1", title: "Fixture" }, agent: { id: null, name: "Fixture", companyId },
|
||||
config: { workspaceStrategy: { type: "git_worktree", branchTemplate: "fixture-task", worktreeParentDir: path.join(location, "worktrees"), baseRef: "origin/main" } }, skipHostProvision: true });
|
||||
await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, sourceIssueId: issueId, mode: "isolated_workspace", strategyType: "git_worktree", name: "Fixture", status: "active",
|
||||
providerType: "local_fs", cwd: workspace.cwd, providerRef: workspace.cwd, repoUrl, baseRef: "origin/main", branchName: workspace.branchName,
|
||||
metadata: { createdByRuntime: workspace.branchCreatedByRuntime, gitBranchOwnershipVersion: 1, baseRefSha: workspace.baseRefSha,
|
||||
isolatedRemoteSource: { schema: "paperclip.isolated-remote-source.v1", companyId, issueId, repoUrl } } });
|
||||
await db.update(issues).set({ executionWorkspaceId: workspaceId }).where(eq(issues.id, issueId));
|
||||
const row = () => db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId)).then(rows => rows[0]!);
|
||||
const restorePoint = (overrides: Partial<{ notBefore: Date; keyDurable: boolean }> = {}) => ({
|
||||
storage: getStorageService(), notBefore: new Date(Date.now() - DAY_MS), keyDurable: true, ...overrides,
|
||||
});
|
||||
const expire = async (options: Partial<Parameters<typeof expireReconstructableGitWorkspace>[2]> = {}) =>
|
||||
expireReconstructableGitWorkspace(db, await row(), { dryRun: false, expireBase: false, now: new Date(), restorePoint: restorePoint(), ...options });
|
||||
const reopen = () => executionWorkspaceService(db).reopenClosedIsolatedExecutionWorkspaceForIssue({ workspaceId,
|
||||
issue: { id: issueId, companyId, projectId: null }, actor: { actorType: "user", agentId: null },
|
||||
authorizationActor: { type: "board", source: "local_implicit", isInstanceAdmin: true } });
|
||||
const ledger = () => db.select().from(executionWorkspaceRestorePoints).where(eq(executionWorkspaceRestorePoints.executionWorkspaceId, workspaceId));
|
||||
const runtimeRestore = async () => ensurePersistedExecutionWorkspaceAvailable({ db, base,
|
||||
workspace: (await executionWorkspaceService(db).getStoredById(companyId, workspaceId))!,
|
||||
issue: { id: issueId, identifier: "RP-1", title: "Resumed task" },
|
||||
agent: { id: null, name: "Fixture", companyId }, skipHostProvision: true });
|
||||
return { companyId, issueId, workspaceId, cwd: workspace.cwd, baseCwd, row, restorePoint, expire, reopen, ledger, runtimeRestore };
|
||||
}
|
||||
|
||||
async function makeUnsavedWork(cwd: string) {
|
||||
await fs.writeFile(path.join(cwd, "marker.txt"), "commit B (unpushed)\n");
|
||||
git(cwd, "commit", "-am", "B unpushed");
|
||||
const unpushed = git(cwd, "rev-parse", "HEAD");
|
||||
await fs.writeFile(path.join(cwd, "marker.txt"), "staged edit\n");
|
||||
git(cwd, "add", "marker.txt");
|
||||
await fs.writeFile(path.join(cwd, "marker.txt"), "unstaged edit\n");
|
||||
await fs.rm(path.join(cwd, "remove-me.txt"));
|
||||
await fs.mkdir(path.join(cwd, "notes"));
|
||||
await fs.writeFile(path.join(cwd, "notes", "todo.md"), "untracked work\n");
|
||||
await fs.writeFile(path.join(cwd, ".env"), "TOKEN=old\n");
|
||||
return unpushed;
|
||||
}
|
||||
|
||||
it("expires a worktree with unpushed and uncommitted work and restores it exactly on reopen", async () => {
|
||||
const f = await fixture();
|
||||
const unpushed = await makeUnsavedWork(f.cwd);
|
||||
expect(await f.expire()).toMatchObject({ candidate: true, expired: true, restorePoint: true, skipReason: null });
|
||||
expect(await exists(f.cwd)).toBe(false);
|
||||
const recipe = (await f.row()).metadata?.gitCacheRestoration as Record<string, any>;
|
||||
expect(recipe).toMatchObject({ schema: "paperclip.git-cache-restoration.v2", commit: unpushed,
|
||||
restorePoint: { head: unpushed, headOnRemote: false, appliedAt: null } });
|
||||
expect(await f.ledger()).toMatchObject([{ status: "active", objectKey: recipe.restorePoint.objectKey }]);
|
||||
|
||||
expect(await f.reopen()).toMatchObject({ ok: true, reopened: true });
|
||||
expect(git(f.cwd, "rev-parse", "HEAD")).toBe(unpushed);
|
||||
expect(await fs.readFile(path.join(f.cwd, "marker.txt"), "utf8")).toBe("unstaged edit\n");
|
||||
expect(git(f.cwd, "show", ":marker.txt")).toBe("staged edit");
|
||||
await expect(fs.access(path.join(f.cwd, "remove-me.txt"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect(await fs.readFile(path.join(f.cwd, "notes", "todo.md"), "utf8")).toBe("untracked work\n");
|
||||
await expect(fs.access(path.join(f.cwd, ".env"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
const restored = (await f.row()).metadata?.gitCacheRestoration as Record<string, any>;
|
||||
expect(restored.restorePoint.appliedAt).toEqual(expect.any(String));
|
||||
});
|
||||
|
||||
it("restores through the runtime path and never re-applies over later work", async () => {
|
||||
const f = await fixture();
|
||||
const unpushed = await makeUnsavedWork(f.cwd);
|
||||
expect(await f.expire()).toMatchObject({ expired: true });
|
||||
await db.update(executionWorkspaces).set({ status: "active", closedAt: null }).where(eq(executionWorkspaces.id, f.workspaceId));
|
||||
await f.runtimeRestore();
|
||||
expect(git(f.cwd, "rev-parse", "HEAD")).toBe(unpushed);
|
||||
expect(await fs.readFile(path.join(f.cwd, "notes", "todo.md"), "utf8")).toBe("untracked work\n");
|
||||
const applied = (await f.row()).metadata?.gitCacheRestoration as Record<string, any>;
|
||||
expect(applied.restorePoint.appliedAt).toEqual(expect.any(String));
|
||||
await fs.writeFile(path.join(f.cwd, "notes", "todo.md"), "later work\n");
|
||||
await f.runtimeRestore();
|
||||
expect(await fs.readFile(path.join(f.cwd, "notes", "todo.md"), "utf8")).toBe("later work\n");
|
||||
});
|
||||
|
||||
it("imports history-only commits on reopen even when the base no longer has them", async () => {
|
||||
const f = await fixture();
|
||||
await fs.writeFile(path.join(f.cwd, "lost.txt"), "rewritten away\n");
|
||||
git(f.cwd, "add", "lost.txt");
|
||||
git(f.cwd, "commit", "-m", "rewritten away");
|
||||
const lost = git(f.cwd, "rev-parse", "HEAD");
|
||||
git(f.cwd, "reset", "-q", "--hard", "HEAD~1");
|
||||
expect(await f.expire()).toMatchObject({ expired: true, restorePoint: true });
|
||||
git(f.baseCwd, "reflog", "expire", "--expire=now", "--all");
|
||||
git(f.baseCwd, "gc", "-q", "--prune=now");
|
||||
expect(() => git(f.baseCwd, "cat-file", "-e", `${lost}^{commit}`)).toThrow();
|
||||
expect(await f.reopen()).toMatchObject({ ok: true, reopened: true });
|
||||
expect(git(f.baseCwd, "cat-file", "-t", lost)).toBe("commit");
|
||||
});
|
||||
|
||||
it("keeps a worktree whose draft commit message a restore point cannot save", async () => {
|
||||
const f = await fixture();
|
||||
await fs.writeFile(path.join(f.cwd, "marker.txt"), "staged edit\n");
|
||||
git(f.cwd, "add", "marker.txt");
|
||||
const gitDir = git(f.cwd, "rev-parse", "--absolute-git-dir");
|
||||
await fs.writeFile(path.join(gitDir, "COMMIT_EDITMSG"), "Draft message from an aborted commit\n");
|
||||
expect(await f.expire()).toMatchObject({ expired: false, skipReason: "uncommitted_git_message" });
|
||||
expect(await exists(f.cwd)).toBe(true);
|
||||
expect(await f.ledger()).toEqual([]);
|
||||
});
|
||||
|
||||
it("marks a restore point for cleanup when its upload fails", async () => {
|
||||
const f = await fixture();
|
||||
await makeUnsavedWork(f.cwd);
|
||||
const storage = getStorageService();
|
||||
const failing = { ...storage, putFile: async () => { throw new Error("storage unavailable"); } };
|
||||
expect(await f.expire({ restorePoint: { ...f.restorePoint(), storage: failing as typeof storage } }))
|
||||
.toMatchObject({ expired: false });
|
||||
expect(await exists(f.cwd)).toBe(true);
|
||||
expect(await f.ledger()).toMatchObject([{ status: "superseded" }]);
|
||||
expect((await f.row()).metadata?.gitCacheRestoration).toBeUndefined();
|
||||
});
|
||||
|
||||
it("does not expire unsaved work before the restore point delay", async () => {
|
||||
const f = await fixture();
|
||||
await makeUnsavedWork(f.cwd);
|
||||
expect(await f.expire({ restorePoint: f.restorePoint({ notBefore: new Date(Date.now() + DAY_MS) }) }))
|
||||
.toMatchObject({ expired: false, skipReason: "restore_point_not_due" });
|
||||
expect(await exists(f.cwd)).toBe(true);
|
||||
expect(await f.ledger()).toEqual([]);
|
||||
});
|
||||
|
||||
it("keeps today's behavior when restore points are disabled", async () => {
|
||||
const f = await fixture();
|
||||
await makeUnsavedWork(f.cwd);
|
||||
expect(await f.expire({ restorePoint: undefined })).toMatchObject({ expired: false, skipReason: "dirty_or_untracked_files" });
|
||||
expect(await exists(f.cwd)).toBe(true);
|
||||
});
|
||||
|
||||
it("refuses to capture when the encryption key is not durable", async () => {
|
||||
const f = await fixture();
|
||||
await makeUnsavedWork(f.cwd);
|
||||
expect(await f.expire({ restorePoint: f.restorePoint({ keyDurable: false }) }))
|
||||
.toMatchObject({ expired: false, skipReason: "restore_point_key_not_durable" });
|
||||
expect(await exists(f.cwd)).toBe(true);
|
||||
});
|
||||
|
||||
it("keeps the worktree and drops the restore point when contents change after capture", async () => {
|
||||
const f = await fixture();
|
||||
await makeUnsavedWork(f.cwd);
|
||||
const result = await f.expire({ beforeDelete: async () => { await fs.writeFile(path.join(f.cwd, "notes", "todo.md"), "changed after capture\n"); } });
|
||||
expect(result).toMatchObject({ expired: false, skipReason: "restore_point_contents_changed" });
|
||||
expect(await exists(f.cwd)).toBe(true);
|
||||
expect(await fs.readFile(path.join(f.cwd, "notes", "todo.md"), "utf8")).toBe("changed after capture\n");
|
||||
const after = await f.row();
|
||||
expect(after.status).toBe("active");
|
||||
expect(after.metadata?.gitCacheRestoration).toMatchObject({ schema: "paperclip.git-cache-restoration.v1" });
|
||||
expect((after.metadata?.gitCacheRestoration as Record<string, unknown>).restorePoint).toBeUndefined();
|
||||
expect(await f.ledger()).toMatchObject([{ status: "superseded" }]);
|
||||
});
|
||||
|
||||
it("fails closed when the stored restore point is missing", async () => {
|
||||
const f = await fixture();
|
||||
await makeUnsavedWork(f.cwd);
|
||||
expect(await f.expire()).toMatchObject({ expired: true });
|
||||
const recipe = (await f.row()).metadata?.gitCacheRestoration as Record<string, any>;
|
||||
await getStorageService().deleteObject(f.companyId, recipe.restorePoint.objectKey);
|
||||
expect(await f.reopen()).toMatchObject({ ok: false, code: "rebuild_failed" });
|
||||
const failed = await f.row();
|
||||
expect(failed.status).toBe("archived");
|
||||
expect(failed.metadata?.gitCacheRestoration).toMatchObject({ restorePoint: { objectKey: recipe.restorePoint.objectKey, appliedAt: null } });
|
||||
await expect(fs.access(path.join(f.cwd, "notes", "todo.md"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
expect(await f.reopen()).toMatchObject({ ok: false, code: "rebuild_failed" });
|
||||
});
|
||||
|
||||
it("reports a restore point candidate in dry run without storing anything", async () => {
|
||||
const f = await fixture();
|
||||
await makeUnsavedWork(f.cwd);
|
||||
expect(await f.expire({ dryRun: true })).toMatchObject({ candidate: true, expired: false, restorePoint: true, skipReason: null });
|
||||
expect(await exists(f.cwd)).toBe(true);
|
||||
expect(await f.ledger()).toEqual([]);
|
||||
expect((await f.row()).metadata?.gitCacheRestoration).toBeUndefined();
|
||||
});
|
||||
|
||||
it("still expires a fully reproducible worktree without a restore point", async () => {
|
||||
const f = await fixture();
|
||||
expect(await f.expire()).toMatchObject({ expired: true, restorePoint: false });
|
||||
expect((await f.row()).metadata?.gitCacheRestoration).toMatchObject({ schema: "paperclip.git-cache-restoration.v1" });
|
||||
expect(await f.ledger()).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -92,6 +92,8 @@ export interface Config {
|
||||
nativeHarnessBackupExpirationDryRun: boolean;
|
||||
nativeHarnessBackupExpirationDays: number;
|
||||
gitRestorePointRetentionDays: number;
|
||||
gitRestorePointsEnabled: boolean;
|
||||
gitRestorePointTerminalDelayDays: number;
|
||||
serveUi: boolean;
|
||||
uiDevMiddleware: boolean;
|
||||
secretsProvider: SecretProvider;
|
||||
@@ -339,6 +341,7 @@ export function loadConfig(): Config {
|
||||
const skillVersionRetentionGraceMinutes = nonNegativeIntegerEnv("PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES", 60);
|
||||
const nativeHarnessBackupExpirationDays = Math.max(1, nonNegativeIntegerEnv("PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DAYS", 7));
|
||||
const gitRestorePointRetentionDays = Math.max(1, nonNegativeIntegerEnv("PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS", 90));
|
||||
const gitRestorePointTerminalDelayDays = nonNegativeIntegerEnv("PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS", 3);
|
||||
const bindValidationErrors = validateConfiguredBindMode({
|
||||
deploymentMode,
|
||||
deploymentExposure,
|
||||
@@ -397,6 +400,8 @@ export function loadConfig(): Config {
|
||||
nativeHarnessBackupExpirationDryRun: process.env.PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DRY_RUN === "true",
|
||||
nativeHarnessBackupExpirationDays,
|
||||
gitRestorePointRetentionDays,
|
||||
gitRestorePointsEnabled: process.env.PAPERCLIP_GIT_RESTORE_POINTS_ENABLED === "true",
|
||||
gitRestorePointTerminalDelayDays,
|
||||
serveUi:
|
||||
process.env.SERVE_UI !== undefined
|
||||
? process.env.SERVE_UI === "true"
|
||||
|
||||
@@ -13,6 +13,7 @@ import {
|
||||
nativeHarnessBackupExpirationService,
|
||||
} from "./services/native-runtime/native-harness-backup-expiration.js";
|
||||
import { gitRestorePointRetentionService, registerGitRestorePointWork } from "./services/git-restore-point-ledger.js";
|
||||
import { markGitRestorePointExpired } from "./services/git-restore-point-storage.js";
|
||||
import { subscribeAllCompanyLiveEvents } from "./services/live-events.js";
|
||||
import { DELIVERY_QUEUES } from "./services/delivery-work-notifications.js";
|
||||
import { registerBudgetEnforcementWork, registerCostAccountingWork } from "./services/budget-enforcement-worker.js";
|
||||
@@ -1479,6 +1480,11 @@ async function startServerWithDatabaseTeardown(
|
||||
gitCacheExpirationEnabled: config.gitCacheExpirationEnabled,
|
||||
gitBaseExpirationEnabled: config.gitBaseExpirationEnabled,
|
||||
gitCacheExpirationDryRun: config.gitCacheExpirationDryRun,
|
||||
gitRestorePoints: config.gitRestorePointsEnabled ? {
|
||||
storage: storageService,
|
||||
terminalDelayMs: config.gitRestorePointTerminalDelayDays * 24 * 60 * 60 * 1000,
|
||||
keyDurable: config.storageProvider !== "s3" || Boolean(process.env.PAPERCLIP_SECRETS_MASTER_KEY?.trim()),
|
||||
} : undefined,
|
||||
workspaceReaperCooldownDays: config.workspaceReaperCooldownDays,
|
||||
workspaceCompactionDelayMs: config.workspaceCompactionDelayHours === null
|
||||
? null
|
||||
@@ -1501,6 +1507,7 @@ async function startServerWithDatabaseTeardown(
|
||||
const gitRestorePointWorker = registerGitRestorePointWork(deliveryWork, gitRestorePointRetentionService(db as any, {
|
||||
storage: storageService,
|
||||
retentionDays: config.gitRestorePointRetentionDays,
|
||||
beforeExpire: (restorePoint) => markGitRestorePointExpired(db as any, restorePoint),
|
||||
}), (result) => {
|
||||
if (result.abandoned > 0 || result.expired > 0 || result.orphaned > 0 || result.deleted > 0 || result.failed > 0) {
|
||||
logger.info(result, "git restore point retention");
|
||||
|
||||
@@ -5,6 +5,7 @@ import { measureRunPhase } from "./execution-preparation-timing.js";
|
||||
import { assertExecutionWorkspaceRecordAccess, assertTaskWorkspaceExecutionAccess, assertTaskWorkspaceSourceProjectAccess } from "./task-workspace-access.js";
|
||||
import { verifiedRecordedIsolatedRemoteSource, withIsolatedRepositorySourceLock } from "./isolated-remote-workspace-source.js";
|
||||
import { expireReconstructableGitWorkspace, hasPendingGitExpiration } from "./git-workspace-expiration.js";
|
||||
import type { StorageService } from "../storage/types.js";
|
||||
import { readGitCacheRestoration, restoreGitCacheBase, recoverGitCacheExpirationLocks, GitCacheRestorationFailure } from "./git-workspace-restoration.js";
|
||||
import { parseObject } from "../adapters/utils.js";
|
||||
import { resolveCoreTrustPreset } from "./trust-preset-resolver.js";
|
||||
@@ -289,6 +290,7 @@ export type ExecutionWorkspaceServiceOptions = {
|
||||
gitCacheExpirationEnabled?: boolean;
|
||||
gitBaseExpirationEnabled?: boolean;
|
||||
gitCacheExpirationDryRun?: boolean;
|
||||
gitRestorePoints?: { storage: StorageService; terminalDelayMs: number; keyDurable: boolean };
|
||||
terminalWorkspaceInspectionBackoffMs?: number;
|
||||
resolvePullRequestDetails?: PullRequestMergeDetailsResolver;
|
||||
now?: () => Date;
|
||||
@@ -3343,6 +3345,11 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
|
||||
const expiration = await expireReconstructableGitWorkspace(db, workspace, {
|
||||
dryRun: opts.gitCacheExpirationDryRun ?? false, expireBase: opts.gitBaseExpirationEnabled ?? false,
|
||||
now: now(), signal, beforeDelete: opts.beforeTerminalWorkspaceCleanup,
|
||||
restorePoint: opts.gitRestorePoints ? {
|
||||
storage: opts.gitRestorePoints.storage,
|
||||
notBefore: new Date(latestTerminalAt + opts.gitRestorePoints.terminalDelayMs),
|
||||
keyDurable: opts.gitRestorePoints.keyDurable,
|
||||
} : undefined,
|
||||
});
|
||||
const expirationDueAt = now().getTime() + 60 * 60 * 1000;
|
||||
terminalWorkspaceDeadlines.set(workspace.id,
|
||||
@@ -3989,6 +3996,17 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
|
||||
if (updated.length !== 1) throw new GitCacheRestorationFailure();
|
||||
nextMetadata.gitCacheRestoration = { ...currentRecipe as Record<string, unknown>, contentsRestoredAt: at };
|
||||
},
|
||||
persistRestorePointAppliedAt: async at => {
|
||||
const currentRecipe = nextMetadata.gitCacheRestoration as Record<string, unknown>;
|
||||
const appliedRecipe = { ...currentRecipe, restorePoint: { ...currentRecipe.restorePoint as Record<string, unknown>, appliedAt: at } };
|
||||
const updated = await db.update(executionWorkspaces).set({
|
||||
metadata: sql`jsonb_set(${executionWorkspaces.metadata}, '{gitCacheRestoration}', ${JSON.stringify(appliedRecipe)}::jsonb)`,
|
||||
}).where(and(eq(executionWorkspaces.id, row.id), eq(executionWorkspaces.companyId, row.companyId),
|
||||
sql`${executionWorkspaces.metadata}->'gitCacheRestoration' = ${JSON.stringify(currentRecipe)}::jsonb`))
|
||||
.returning({ id: executionWorkspaces.id });
|
||||
if (updated.length !== 1) throw new GitCacheRestorationFailure();
|
||||
nextMetadata.gitCacheRestoration = appliedRecipe;
|
||||
},
|
||||
prepareRestoredLocalSource: async sourceCwd => {
|
||||
const sourceIdentity = await fs.realpath(sourceCwd);
|
||||
const targetIdentity = await canonicalRepositoryRoot(row.cwd!);
|
||||
|
||||
@@ -0,0 +1,251 @@
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { and, eq, inArray, ne, sql } from "drizzle-orm";
|
||||
import { executionWorkspaceRestorePoints, executionWorkspaces, type Db } from "@paperclipai/db";
|
||||
import type { StoredSecretVersionMaterial } from "../secrets/types.js";
|
||||
import type { StorageService } from "../storage/types.js";
|
||||
import {
|
||||
applyRestorePoint,
|
||||
captureRestorePoint,
|
||||
currentRestorePointSnapshot,
|
||||
decodeRestorePointPayload,
|
||||
importRestorePoint,
|
||||
openRestorePoint,
|
||||
RestorePointRestoreFailure,
|
||||
sealRestorePoint,
|
||||
type RestorePointLimits,
|
||||
type RestorePointManifest,
|
||||
} from "./git-restore-point.js";
|
||||
import { notifyGitRestorePointWork } from "./git-restore-point-ledger.js";
|
||||
import { gitCacheCommand } from "./git-workspace-restoration.js";
|
||||
|
||||
export const GIT_RESTORE_POINT_SCHEMA = "paperclip.git-restore-point.v1";
|
||||
const NAMESPACE = "git-restore-points";
|
||||
const MAX_RESTORE_POINT_BYTES = 512 * 1024 * 1024;
|
||||
|
||||
export type GitRestorePointRef = {
|
||||
schema: typeof GIT_RESTORE_POINT_SCHEMA;
|
||||
id: string;
|
||||
objectKey: string;
|
||||
sha256: string;
|
||||
byteSize: number;
|
||||
key: StoredSecretVersionMaterial;
|
||||
generation: number;
|
||||
head: string;
|
||||
index: string | null;
|
||||
worktree: string | null;
|
||||
modes: Array<{ path: string; mode: number }>;
|
||||
headOnRemote: boolean;
|
||||
appliedAt?: string | null;
|
||||
expiredAt?: string;
|
||||
};
|
||||
|
||||
type Tx = Parameters<Parameters<Db["transaction"]>[0]>[0];
|
||||
|
||||
function restorePointObjectKey(companyId: string, workspaceId: string, generation: number, id: string) {
|
||||
return `${companyId}/${NAMESPACE}/${workspaceId}/${generation}/${id}.pcrp`;
|
||||
}
|
||||
|
||||
export function validateGitRestorePointRef(value: unknown, workspace: { companyId: string; id: string }): GitRestorePointRef {
|
||||
const ref = value as GitRestorePointRef | null;
|
||||
const sha = /^[0-9a-f]{40}$/;
|
||||
if (!ref || ref.schema !== GIT_RESTORE_POINT_SCHEMA || typeof ref.id !== "string"
|
||||
|| ref.objectKey !== restorePointObjectKey(workspace.companyId, workspace.id, ref.generation, ref.id)
|
||||
|| !/^[0-9a-f]{64}$/.test(ref.sha256) || !Number.isSafeInteger(ref.byteSize) || ref.byteSize <= 0 || ref.byteSize > MAX_RESTORE_POINT_BYTES
|
||||
|| !ref.key || typeof ref.key !== "object" || Array.isArray(ref.key)
|
||||
|| !Number.isSafeInteger(ref.generation) || ref.generation < 1
|
||||
|| !sha.test(ref.head) || (ref.index !== null && !sha.test(ref.index)) || (ref.worktree !== null && !sha.test(ref.worktree))
|
||||
|| (ref.index === null) !== (ref.worktree === null) || typeof ref.headOnRemote !== "boolean"
|
||||
|| !Array.isArray(ref.modes) || ref.modes.length > 20_000 || ref.modes.some((entry) => !entry || typeof entry.path !== "string"
|
||||
|| entry.path.startsWith("/") || entry.path.split("/").includes("..") || !Number.isInteger(entry.mode) || entry.mode < 0 || entry.mode > 0o777)
|
||||
|| (ref.appliedAt != null && !Number.isFinite(Date.parse(ref.appliedAt)))
|
||||
|| (ref.expiredAt !== undefined && !Number.isFinite(Date.parse(ref.expiredAt)))) {
|
||||
throw new RestorePointRestoreFailure("invalid_restore_point_reference");
|
||||
}
|
||||
return ref;
|
||||
}
|
||||
|
||||
export async function storeGitRestorePoint(input: {
|
||||
db: Db;
|
||||
storage: StorageService;
|
||||
workspace: { companyId: string; id: string; sourceIssueId: string | null };
|
||||
cwd: string;
|
||||
generation: number;
|
||||
upstreamCommits: readonly string[];
|
||||
headOnRemote: boolean;
|
||||
limits?: Partial<RestorePointLimits>;
|
||||
signal?: AbortSignal;
|
||||
}): Promise<{ ref: GitRestorePointRef; ledgerId: string; manifest: RestorePointManifest } | null> {
|
||||
const captured = await captureRestorePoint({ cwd: input.cwd, upstreamCommits: input.upstreamCommits, limits: input.limits, signal: input.signal });
|
||||
if (!captured) return null;
|
||||
const id = randomUUID();
|
||||
const identity = { companyId: input.workspace.companyId, workspaceId: input.workspace.id, restorePointId: id, generation: input.generation };
|
||||
const sealed = await sealRestorePoint(captured.payload, identity);
|
||||
const objectKey = restorePointObjectKey(input.workspace.companyId, input.workspace.id, input.generation, id);
|
||||
const [ledger] = await input.db.transaction(async (tx) => {
|
||||
await notifyGitRestorePointWork(tx, `workspace:${input.workspace.id}`);
|
||||
return tx.insert(executionWorkspaceRestorePoints).values({
|
||||
companyId: input.workspace.companyId,
|
||||
executionWorkspaceId: input.workspace.id,
|
||||
sourceIssueId: input.workspace.sourceIssueId,
|
||||
generation: input.generation,
|
||||
objectKey,
|
||||
sha256: sealed.sha256,
|
||||
byteSize: sealed.ciphertext.length,
|
||||
status: "pending",
|
||||
}).returning({ id: executionWorkspaceRestorePoints.id });
|
||||
});
|
||||
try {
|
||||
await input.storage.putFile({
|
||||
objectKey,
|
||||
companyId: input.workspace.companyId,
|
||||
namespace: NAMESPACE,
|
||||
originalFilename: null,
|
||||
contentType: "application/octet-stream",
|
||||
body: sealed.ciphertext,
|
||||
});
|
||||
const stored = await readObject(input.storage, input.workspace.companyId, objectKey, input.signal);
|
||||
if (createHash("sha256").update(stored).digest("hex") !== sealed.sha256) {
|
||||
throw new RestorePointRestoreFailure("stored_restore_point_mismatch");
|
||||
}
|
||||
} catch (error) {
|
||||
await abandonGitRestorePoint(input.db, ledger!.id).catch(() => undefined);
|
||||
throw error;
|
||||
}
|
||||
const { manifest } = captured;
|
||||
return {
|
||||
ledgerId: ledger!.id,
|
||||
manifest,
|
||||
ref: {
|
||||
schema: GIT_RESTORE_POINT_SCHEMA,
|
||||
id,
|
||||
objectKey,
|
||||
sha256: sealed.sha256,
|
||||
byteSize: sealed.ciphertext.length,
|
||||
key: sealed.key,
|
||||
generation: input.generation,
|
||||
head: manifest.head,
|
||||
index: manifest.index,
|
||||
worktree: manifest.worktree,
|
||||
modes: manifest.modes,
|
||||
headOnRemote: input.headOnRemote,
|
||||
appliedAt: null,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export async function activateGitRestorePoint(tx: Tx, input: { ledgerId: string; workspaceId: string; at: Date }) {
|
||||
await notifyGitRestorePointWork(tx, `workspace:${input.workspaceId}`);
|
||||
await tx.update(executionWorkspaceRestorePoints)
|
||||
.set({ status: "superseded", updatedAt: input.at })
|
||||
.where(and(eq(executionWorkspaceRestorePoints.executionWorkspaceId, input.workspaceId),
|
||||
ne(executionWorkspaceRestorePoints.id, input.ledgerId),
|
||||
inArray(executionWorkspaceRestorePoints.status, ["pending", "active"])));
|
||||
const activated = await tx.update(executionWorkspaceRestorePoints)
|
||||
.set({ status: "active", activatedAt: input.at, deleteAfter: input.at, updatedAt: input.at })
|
||||
.where(and(eq(executionWorkspaceRestorePoints.id, input.ledgerId), eq(executionWorkspaceRestorePoints.status, "pending")))
|
||||
.returning({ id: executionWorkspaceRestorePoints.id });
|
||||
if (activated.length !== 1) throw new RestorePointRestoreFailure("restore_point_not_pending");
|
||||
}
|
||||
|
||||
export function restorePointSnapshotMatches(ref: GitRestorePointRef, snapshot: { head: string; index: string | null; worktree: string | null; modes: Array<{ path: string; mode: number }> }) {
|
||||
return snapshot.head === ref.head && snapshot.index === ref.index && snapshot.worktree === ref.worktree
|
||||
&& JSON.stringify(snapshot.modes) === JSON.stringify(ref.modes);
|
||||
}
|
||||
|
||||
export async function abandonGitRestorePointById(db: Db, workspaceId: string, objectKey: string) {
|
||||
await db.update(executionWorkspaceRestorePoints)
|
||||
.set({ status: "superseded", updatedAt: new Date() })
|
||||
.where(and(eq(executionWorkspaceRestorePoints.executionWorkspaceId, workspaceId), eq(executionWorkspaceRestorePoints.objectKey, objectKey),
|
||||
inArray(executionWorkspaceRestorePoints.status, ["pending", "active"])));
|
||||
}
|
||||
|
||||
export async function abandonGitRestorePoint(db: Db, ledgerId: string) {
|
||||
await db.update(executionWorkspaceRestorePoints)
|
||||
.set({ status: "superseded", updatedAt: new Date() })
|
||||
.where(and(eq(executionWorkspaceRestorePoints.id, ledgerId), inArray(executionWorkspaceRestorePoints.status, ["pending", "active"])));
|
||||
}
|
||||
|
||||
async function readObject(storage: StorageService, companyId: string, objectKey: string, signal?: AbortSignal) {
|
||||
const { stream } = await storage.getObject(companyId, objectKey);
|
||||
const chunks: Buffer[] = [];
|
||||
let total = 0;
|
||||
for await (const chunk of stream) {
|
||||
signal?.throwIfAborted();
|
||||
const buffer = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk);
|
||||
total += buffer.length;
|
||||
if (total > MAX_RESTORE_POINT_BYTES) throw new RestorePointRestoreFailure("restore_point_too_large");
|
||||
chunks.push(buffer);
|
||||
}
|
||||
return Buffer.concat(chunks);
|
||||
}
|
||||
|
||||
export async function loadGitRestorePoint(input: {
|
||||
storage: StorageService;
|
||||
workspace: { companyId: string; id: string };
|
||||
ref: GitRestorePointRef;
|
||||
signal?: AbortSignal;
|
||||
}): Promise<{ payload: Buffer; manifest: RestorePointManifest }> {
|
||||
if (input.ref.expiredAt) throw new RestorePointRestoreFailure("restore_point_expired");
|
||||
const ciphertext = await readObject(input.storage, input.workspace.companyId, input.ref.objectKey, input.signal)
|
||||
.catch((error) => {
|
||||
if (error instanceof RestorePointRestoreFailure) throw error;
|
||||
throw new RestorePointRestoreFailure("restore_point_unavailable");
|
||||
});
|
||||
if (ciphertext.length !== input.ref.byteSize) throw new RestorePointRestoreFailure("ciphertext_size_mismatch");
|
||||
const payload = await openRestorePoint({
|
||||
ciphertext,
|
||||
sha256: input.ref.sha256,
|
||||
key: input.ref.key,
|
||||
identity: { companyId: input.workspace.companyId, workspaceId: input.workspace.id, restorePointId: input.ref.id, generation: input.ref.generation },
|
||||
});
|
||||
const { manifest } = decodeRestorePointPayload(payload);
|
||||
if (manifest.head !== input.ref.head || manifest.index !== input.ref.index || manifest.worktree !== input.ref.worktree
|
||||
|| JSON.stringify(manifest.modes) !== JSON.stringify(input.ref.modes)) {
|
||||
throw new RestorePointRestoreFailure("restore_point_manifest_mismatch");
|
||||
}
|
||||
return { payload, manifest };
|
||||
}
|
||||
|
||||
export async function ensureGitRestorePointCommit(input: {
|
||||
storage: StorageService;
|
||||
workspace: { companyId: string; id: string };
|
||||
ref: GitRestorePointRef;
|
||||
repositoryCwd: string;
|
||||
signal?: AbortSignal;
|
||||
}) {
|
||||
if (input.ref.headOnRemote) return;
|
||||
const present = await importNeeded(input.repositoryCwd, input.ref.head);
|
||||
if (!present) return;
|
||||
const { payload } = await loadGitRestorePoint(input);
|
||||
await importRestorePoint({ repositoryCwd: input.repositoryCwd, payload, signal: input.signal });
|
||||
}
|
||||
|
||||
async function importNeeded(repositoryCwd: string, commit: string) {
|
||||
return !await gitCacheCommand(repositoryCwd, ["cat-file", "-e", `${commit}^{commit}`]).then(() => true, () => false);
|
||||
}
|
||||
|
||||
export async function applyGitRestorePointToWorktree(input: {
|
||||
storage: StorageService;
|
||||
workspace: { companyId: string; id: string };
|
||||
ref: GitRestorePointRef;
|
||||
repositoryCwd: string;
|
||||
worktreePath: string;
|
||||
signal?: AbortSignal;
|
||||
}) {
|
||||
const current = await currentRestorePointSnapshot(input.worktreePath, input.signal);
|
||||
const matches = restorePointSnapshotMatches(input.ref, current);
|
||||
if (!matches && (current.head !== input.ref.head || current.index !== null || current.worktree !== null || current.modes.length > 0)) {
|
||||
throw new RestorePointRestoreFailure("worktree_diverged_from_restore_point");
|
||||
}
|
||||
const { payload } = await loadGitRestorePoint(input);
|
||||
const manifest = await importRestorePoint({ repositoryCwd: input.repositoryCwd, payload, signal: input.signal });
|
||||
if (!matches) await applyRestorePoint({ cwd: input.worktreePath, manifest, signal: input.signal });
|
||||
}
|
||||
|
||||
export async function markGitRestorePointExpired(db: Db, restorePoint: { companyId: string; executionWorkspaceId: string; objectKey: string }) {
|
||||
await db.update(executionWorkspaces).set({
|
||||
metadata: sql`jsonb_set(${executionWorkspaces.metadata}, '{gitCacheRestoration,restorePoint,expiredAt}', ${JSON.stringify(new Date().toISOString())}::jsonb)`,
|
||||
updatedAt: new Date(),
|
||||
}).where(and(eq(executionWorkspaces.id, restorePoint.executionWorkspaceId), eq(executionWorkspaces.companyId, restorePoint.companyId),
|
||||
sql`${executionWorkspaces.metadata}->'gitCacheRestoration'->'restorePoint'->>'objectKey' = ${restorePoint.objectKey}`));
|
||||
}
|
||||
@@ -7,11 +7,14 @@ import { and, eq, ne, sql } from "drizzle-orm";
|
||||
import { executionWorkspaces, issues, heartbeatRuns, nativeRunFinalizations, environmentLeases,
|
||||
workspaceRuntimeServices, issueRecoveryActions, projectWorkspaces, withDedicatedDbConnection, type Db } from "@paperclipai/db";
|
||||
import { createGitRemoteAuthProvider } from "./git-credentials.js";
|
||||
import type { StorageService } from "../storage/types.js";
|
||||
import { currentRestorePointSnapshot, RestorePointSkip, type RestorePointLimits } from "./git-restore-point.js";
|
||||
import { abandonGitRestorePoint, abandonGitRestorePointById, activateGitRestorePoint, restorePointSnapshotMatches, storeGitRestorePoint } from "./git-restore-point-storage.js";
|
||||
import { withIsolatedRepositorySourceLock, verifiedRecordedIsolatedRemoteSource } from "./isolated-remote-workspace-source.js";
|
||||
import { managedProjectCheckoutName } from "./managed-project-checkout-name.js";
|
||||
import type { GitRemoteAuthProvider } from "./git-credentials.js";
|
||||
import { assertReproducibleGitFiles, assertReproducibleBaseGitStorage, assertReproducibleManagedCheckout, inspectGitCacheRecipe, readGitCacheRestoration,
|
||||
verifyRemoteGitCommit, gitCacheExpirationLockLocations, recoverGitCacheExpirationLocks, GitCacheRestorationFailure, type ManagedCheckoutProver, gitCacheCommand, readGitCacheExpiration, finishGitCacheTrash, type GitCacheExpiration, GIT_CACHE_RESTORATION_KEY, GIT_CACHE_EXPIRATION_REASON } from "./git-workspace-restoration.js";
|
||||
import { assertReproducibleGitFiles, assertReproducibleBaseGitStorage, assertReproducibleManagedCheckout, inspectGitCacheRecipe, inspectGitCacheRecipeForRestorePoint, readGitCacheRestoration, GIT_CACHE_RESTORATION_SCHEMA_V2,
|
||||
verifyRemoteGitCommit, gitCacheExpirationLockLocations, recoverGitCacheExpirationLocks, GitCacheRestorationFailure, type ManagedCheckoutProver, gitCacheCommand, readGitCacheExpiration, finishGitCacheTrash, type GitCacheExpiration, type GitCacheRestoration, GIT_CACHE_RESTORATION_KEY, GIT_CACHE_EXPIRATION_REASON } from "./git-workspace-restoration.js";
|
||||
|
||||
async function allocatedBytes(target: string, shouldContinue: () => boolean): Promise<number> {
|
||||
if (!shouldContinue()) throw new Error("scan_stopped");
|
||||
@@ -41,7 +44,29 @@ type Workspace = typeof executionWorkspaces.$inferSelect;
|
||||
type Tx = Parameters<Parameters<Db["transaction"]>[0]>[0];
|
||||
type Expiration = GitCacheExpiration;
|
||||
export type GitExpirationResult = { candidate: boolean; expired: boolean; reclaimedBytes: number;
|
||||
candidateBytes: number; baseExpired: boolean; skipReason: string | null };
|
||||
candidateBytes: number; baseExpired: boolean; skipReason: string | null; restorePoint: boolean };
|
||||
|
||||
export type GitRestorePointExpirationOptions = {
|
||||
storage: StorageService;
|
||||
notBefore: Date;
|
||||
keyDurable: boolean;
|
||||
limits?: Partial<RestorePointLimits>;
|
||||
};
|
||||
|
||||
const RESTORE_POINT_RELAXABLE = new Set(["dirty_or_untracked_files", "unique_ignored_files", "unique_original_head",
|
||||
"unique_reflog_commits", "upstream_commit_unverified"]);
|
||||
|
||||
async function restorePointBoundary(recipe: GitCacheRestoration, headOnRemote: boolean, auth: GitRemoteAuthProvider, signal?: AbortSignal) {
|
||||
if (headOnRemote) return [recipe.commit];
|
||||
const upstream = await gitCacheCommand(recipe.worktreePath, ["rev-parse", "--verify", "-q", "refs/remotes/origin/HEAD"], signal)
|
||||
.catch(() => "");
|
||||
const tip = upstream.trim();
|
||||
if (!tip) return [];
|
||||
const base = (await gitCacheCommand(recipe.worktreePath, ["merge-base", "HEAD", tip], signal).catch(() => "")).trim();
|
||||
if (!/^[a-f0-9]{40}$/.test(base)) return [];
|
||||
try { await verifyRemoteGitCommit(recipe.repoUrl, base, auth, signal); return [base]; }
|
||||
catch (error) { if (signal?.aborted) throw error; return []; }
|
||||
}
|
||||
|
||||
/** Re-read the whole tree and every runtime/recovery consumer immediately before deletion. */
|
||||
export async function isGitWorkspaceExpirationEligible(tx: Tx, workspace: Workspace, now: Date) {
|
||||
@@ -156,9 +181,11 @@ function managedCheckoutProver(db: Db, workspace: Workspace, auth: GitRemoteAuth
|
||||
export async function expireReconstructableGitWorkspace(db: Db, snapshot: Workspace, options: {
|
||||
dryRun: boolean; expireBase: boolean; now: Date; signal?: AbortSignal;
|
||||
beforeDelete?: (workspace: Workspace) => Promise<void>;
|
||||
restorePoint?: GitRestorePointExpirationOptions;
|
||||
}): Promise<GitExpirationResult> {
|
||||
const result: GitExpirationResult = { candidate: false, expired: false, reclaimedBytes: 0,
|
||||
candidateBytes: 0, baseExpired: false, skipReason: null };
|
||||
candidateBytes: 0, baseExpired: false, skipReason: null, restorePoint: false };
|
||||
let pendingRestorePoint: string | null = null;
|
||||
if (!snapshot.sourceIssueId) return { ...result, skipReason: "missing_source_issue" };
|
||||
try {
|
||||
return await withIsolatedRepositorySourceLock(db, snapshot.companyId, snapshot.sourceIssueId, () =>
|
||||
@@ -176,8 +203,24 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Worksp
|
||||
const checkouts = new Map<string, ManagedCheckoutContents>();
|
||||
const identity = { companyId: workspace.companyId, workspaceId: workspace.id };
|
||||
const proveManagedCheckout = managedCheckoutProver(db, workspace, auth, checkouts);
|
||||
let recipe = state ? readGitCacheRestoration(workspace)! : await inspectGitCacheRecipe(workspace,
|
||||
auth, options.signal, { proveManagedCheckout });
|
||||
let headOnRemote = true;
|
||||
let needsRestorePoint = false;
|
||||
let recipe: GitCacheRestoration;
|
||||
if (state) recipe = readGitCacheRestoration(workspace)!;
|
||||
else {
|
||||
try { recipe = await inspectGitCacheRecipe(workspace, auth, options.signal, { proveManagedCheckout }); }
|
||||
catch (error) {
|
||||
if (!options.restorePoint || !RESTORE_POINT_RELAXABLE.has(skipReason(error))) throw error;
|
||||
if (options.now.getTime() < options.restorePoint.notBefore.getTime()) throw new Error("restore_point_not_due");
|
||||
if (!options.restorePoint.keyDurable) throw new Error("restore_point_key_not_durable");
|
||||
checkouts.clear();
|
||||
const inspected = await inspectGitCacheRecipeForRestorePoint(workspace, auth, options.signal, { proveManagedCheckout });
|
||||
recipe = inspected.recipe;
|
||||
headOnRemote = inspected.headOnRemote;
|
||||
needsRestorePoint = true;
|
||||
result.restorePoint = true;
|
||||
}
|
||||
}
|
||||
if (!state) recipe.contents = await captureGitWorkspaceContents(recipe.worktreePath, identity, [...checkouts.values()]);
|
||||
const present = await exists(recipe.worktreePath);
|
||||
if (!state && !present) throw new Error("missing_unverified_worktree");
|
||||
@@ -199,16 +242,34 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Worksp
|
||||
if (await fs.realpath(worktreeAdmin) !== canonicalAdmin || await fs.realpath(rawAdmin) !== canonicalAdmin) {
|
||||
throw new Error("unproven_worktree_admin");
|
||||
}
|
||||
let activateLedger: string | null = null;
|
||||
if (needsRestorePoint) {
|
||||
const stored = await storeGitRestorePoint({ db, storage: options.restorePoint!.storage,
|
||||
workspace: { companyId: workspace.companyId, id: workspace.id, sourceIssueId: workspace.sourceIssueId },
|
||||
cwd: recipe.worktreePath, generation, headOnRemote, limits: options.restorePoint!.limits, signal: options.signal,
|
||||
upstreamCommits: await restorePointBoundary(recipe, headOnRemote, auth, options.signal) });
|
||||
if (stored) {
|
||||
pendingRestorePoint = stored.ledgerId;
|
||||
activateLedger = stored.ledgerId;
|
||||
recipe = { ...recipe, schema: GIT_CACHE_RESTORATION_SCHEMA_V2, restorePoint: stored.ref };
|
||||
} else if (!headOnRemote) throw new Error("upstream_commit_unverified");
|
||||
}
|
||||
state = { schema: "paperclip.git-cache-expiration.v1", generation, workspaceRemoved: false,
|
||||
worktreeTrash: path.join(path.dirname(recipe.worktreePath), `.expired-git-worktree-${workspace.id}-${generation}`),
|
||||
worktreeAdmin, baseTrash: null, completed: false, lockToken: randomBytes(32).toString("hex") };
|
||||
if (await exists(state.worktreeTrash)) throw new Error("unknown_worktree_trash");
|
||||
// Commit the immutable restoration recipe BEFORE touching disk. The separate
|
||||
// lifecycle-lock connection remains held through deletion and final state.
|
||||
[workspace] = await db.update(executionWorkspaces).set({ status: "archived", closedAt: options.now,
|
||||
const archive = { status: "archived", closedAt: options.now,
|
||||
cleanupEligibleAt: options.now, cleanupReason: GIT_CACHE_EXPIRATION_REASON, updatedAt: options.now,
|
||||
metadata: { ...workspace.metadata, lifecycleGeneration: generation, [GIT_CACHE_RESTORATION_KEY]: recipe, [EXPIRATION_KEY]: state },
|
||||
}).where(eq(executionWorkspaces.id, workspace.id)).returning();
|
||||
};
|
||||
[workspace] = await db.transaction(async (archiveTx) => {
|
||||
const rows = await archiveTx.update(executionWorkspaces).set(archive).where(eq(executionWorkspaces.id, workspace!.id)).returning();
|
||||
if (activateLedger) await activateGitRestorePoint(archiveTx, { ledgerId: activateLedger, workspaceId: workspace!.id, at: options.now });
|
||||
return rows;
|
||||
});
|
||||
pendingRestorePoint = null;
|
||||
}
|
||||
if (!workspace) throw new Error("workspace_changed");
|
||||
const persist = async () => {
|
||||
@@ -229,7 +290,10 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Worksp
|
||||
try {
|
||||
checkouts.clear();
|
||||
await assertReproducibleGitFiles(recipe.worktreePath, options.signal, { proveManagedCheckout, preserveSettings: true,
|
||||
restorePoint: Boolean(recipe.restorePoint),
|
||||
verifyFetchedCommit: (commit, signal) => verifyRemoteGitCommit(recipe.repoUrl, commit, auth, signal) });
|
||||
if (recipe.restorePoint && !restorePointSnapshotMatches(recipe.restorePoint,
|
||||
await currentRestorePointSnapshot(recipe.worktreePath, options.signal))) throw new Error("restore_point_contents_changed");
|
||||
for (const checkout of checkouts.values()) {
|
||||
const cwd = path.join(recipe.worktreePath, ".paperclip-repositories", checkout.name);
|
||||
locks.push(await acquireGitExpirationLocks(lockLocations.filter(location => location.checkout === checkout.name), state.lockToken));
|
||||
@@ -241,11 +305,13 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Worksp
|
||||
if ((await gitCacheCommand(recipe.worktreePath, ["rev-parse", "HEAD"], options.signal)).trim() !== recipe.commit) throw new Error("head_changed");
|
||||
} catch (error) {
|
||||
for (const lock of [...locks].reverse()) await lock.release();
|
||||
const { restorePoint: _abandoned, ...reproducible } = recipe;
|
||||
const metadata: Record<string, unknown> = { ...workspace.metadata, lifecycleGeneration: state.generation + 1,
|
||||
[GIT_CACHE_RESTORATION_KEY]: { ...recipe, restoredAt: new Date().toISOString() } };
|
||||
[GIT_CACHE_RESTORATION_KEY]: { ...reproducible, schema: "paperclip.git-cache-restoration.v1", restoredAt: new Date().toISOString() } };
|
||||
delete metadata[EXPIRATION_KEY];
|
||||
await db.update(executionWorkspaces).set({ status: "active", closedAt: null, cleanupEligibleAt: null,
|
||||
cleanupReason: null, metadata, updatedAt: new Date() }).where(eq(executionWorkspaces.id, workspace.id));
|
||||
if (recipe.restorePoint) await abandonGitRestorePointById(db, workspace.id, recipe.restorePoint.objectKey);
|
||||
throw error;
|
||||
}
|
||||
await fs.rename(recipe.worktreePath, state.worktreeTrash);
|
||||
@@ -284,8 +350,9 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Worksp
|
||||
return result;
|
||||
})), true);
|
||||
} catch (error) {
|
||||
if (pendingRestorePoint) await abandonGitRestorePoint(db, pendingRestorePoint).catch(() => undefined);
|
||||
options.signal?.throwIfAborted();
|
||||
result.skipReason = skipReason(error);
|
||||
result.skipReason = error instanceof RestorePointSkip ? error.reason : skipReason(error);
|
||||
return result;
|
||||
}
|
||||
}
|
||||
@@ -13,14 +13,18 @@ import type { GitRemoteAuthProvider } from "./git-credentials.js";
|
||||
import { isRuntimeOwnedGitBranch } from "./execution-workspace-branch-ownership.js";
|
||||
import { recordedIsolatedRemoteSource, verifiedRecordedIsolatedRemoteSource, materializeIsolatedRemoteWorkspaceSource } from "./isolated-remote-workspace-source.js";
|
||||
import { findRegenerableWorkspaceDirs } from "./workspace-compaction.js";
|
||||
import { ensureGitRestorePointCommit, validateGitRestorePointRef, type GitRestorePointRef } from "./git-restore-point-storage.js";
|
||||
import { getStorageService } from "../storage/index.js";
|
||||
import type { StorageService } from "../storage/types.js";
|
||||
import { PROJECT_REPOSITORIES_DIR } from "@paperclipai/adapter-utils/git-workspace-sync";
|
||||
|
||||
const execFile = promisify(execFileCallback);
|
||||
const shaPattern = /^[a-f0-9]{40}$/;
|
||||
export const GIT_CACHE_RESTORATION_KEY = "gitCacheRestoration";
|
||||
export const GIT_CACHE_EXPIRATION_REASON = "reproducible_git_cache_expired";
|
||||
export const GIT_CACHE_RESTORATION_SCHEMA_V2 = "paperclip.git-cache-restoration.v2";
|
||||
export type GitCacheRestoration = {
|
||||
schema: "paperclip.git-cache-restoration.v1";
|
||||
schema: "paperclip.git-cache-restoration.v1" | typeof GIT_CACHE_RESTORATION_SCHEMA_V2;
|
||||
companyId: string;
|
||||
workspaceId: string;
|
||||
sourceIssueId: string;
|
||||
@@ -36,6 +40,7 @@ export type GitCacheRestoration = {
|
||||
restoredAt?: string;
|
||||
contentsRestoredAt?: string | null;
|
||||
contents?: GitWorkspaceContents;
|
||||
restorePoint?: GitRestorePointRef;
|
||||
};
|
||||
export type GitCacheExpiration = {
|
||||
schema: "paperclip.git-cache-expiration.v1"; generation: number; workspaceRemoved: boolean;
|
||||
@@ -80,7 +85,7 @@ export function readGitCacheRestoration(workspace: Workspace): GitCacheRestorati
|
||||
const value = workspace.metadata?.[GIT_CACHE_RESTORATION_KEY];
|
||||
if (value === undefined) return null;
|
||||
const recipe = value as GitCacheRestoration | null;
|
||||
if (!recipe || recipe.schema !== "paperclip.git-cache-restoration.v1"
|
||||
if (!recipe || !restorationSchemaMatches(recipe)
|
||||
|| !isRuntimeOwnedGitBranch(workspace.metadata)
|
||||
|| recipe.gitBranchOwnershipVersion !== 1
|
||||
|| recipe.companyId !== workspace.companyId || recipe.workspaceId !== workspace.id
|
||||
@@ -96,9 +101,18 @@ export function readGitCacheRestoration(workspace: Workspace): GitCacheRestorati
|
||||
throw new GitCacheRestorationFailure();
|
||||
}
|
||||
validateGitWorkspaceContents(recipe.contents);
|
||||
if (recipe.restorePoint) {
|
||||
try { validateGitRestorePointRef(recipe.restorePoint, workspace); } catch { throw new GitCacheRestorationFailure(); }
|
||||
}
|
||||
return recipe;
|
||||
}
|
||||
|
||||
function restorationSchemaMatches(recipe: GitCacheRestoration) {
|
||||
return recipe.schema === "paperclip.git-cache-restoration.v1"
|
||||
? recipe.restorePoint === undefined
|
||||
: recipe.schema === GIT_CACHE_RESTORATION_SCHEMA_V2 && recipe.restorePoint !== undefined;
|
||||
}
|
||||
|
||||
export async function gitCacheCommand(cwd: string, args: string[], signal?: AbortSignal) {
|
||||
const result = await execFile("git", ["-c", "core.fsmonitor=false", "-C", cwd, ...args], {
|
||||
timeout: 60_000, signal, maxBuffer: 16 * 1024 * 1024,
|
||||
@@ -293,14 +307,16 @@ export async function assertReproducibleGitFiles(cwd: string, signal?: AbortSign
|
||||
proveManagedCheckout?: ManagedCheckoutProver;
|
||||
preserveSettings?: boolean;
|
||||
verifyFetchedCommit?: (commit: string, signal?: AbortSignal) => Promise<void>;
|
||||
restorePoint?: boolean;
|
||||
} = {}) {
|
||||
if ((await gitCacheCommand(cwd, ["status", "--porcelain=v1", "--untracked-files=all", "-z"], signal)).length) throw new Error("dirty_or_untracked_files");
|
||||
if (!options.restorePoint && (await gitCacheCommand(cwd, ["status", "--porcelain=v1", "--untracked-files=all", "-z"], signal)).length) throw new Error("dirty_or_untracked_files");
|
||||
if ((await gitCacheCommand(cwd, ["ls-files", "-v"], signal)).split("\n").some(line => /^[a-zS] /.test(line))) throw new Error("unsupported_index_flags");
|
||||
const localConfig = await gitCacheCommand(cwd, ["config", "--local", "--name-only", "--list"], signal);
|
||||
if (localConfig.split("\n").filter(Boolean).some(key => !/^(core\.(repositoryformatversion|filemode|bare|logallrefupdates|ignorecase|precomposeunicode)|remote\.origin\.(url|fetch)|branch\.[^.]+\.(remote|merge))$/.test(key))) throw new Error("unsupported_local_git_config");
|
||||
const regenerable = await findRegenerableWorkspaceDirs(cwd, { signal });
|
||||
const ignored = (await gitCacheCommand(cwd, ["ls-files", "--others", "--ignored", "--exclude-standard", "-z"], signal)).split("\0").filter(Boolean);
|
||||
const unexplained = ignored.filter(file => !(options.preserveSettings && file === PAPERCLIP_CLAUDE_SETTINGS_FILE) && !regenerable.some(dir => file.startsWith(`${dir}/`)));
|
||||
const unexplained = ignored.filter(file => !(options.preserveSettings && file === PAPERCLIP_CLAUDE_SETTINGS_FILE) && !regenerable.some(dir => file.startsWith(`${dir}/`))
|
||||
&& !(options.restorePoint && /^\.env(\..*)?$/.test(path.posix.basename(file))));
|
||||
const checkouts = new Set<string>();
|
||||
for (const file of unexplained) {
|
||||
const [namespace, name, rest] = file.split("/", 3);
|
||||
@@ -323,7 +339,7 @@ export async function assertReproducibleGitFiles(cwd: string, signal?: AbortSign
|
||||
if (await fs.lstat(path.join(root, name)).then(() => true, (e: NodeJS.ErrnoException) => { if(e.code === "ENOENT") return false; throw e; })) throw new Error("unsupported_local_git_state");
|
||||
}
|
||||
}
|
||||
const original = await gitCacheCommand(cwd, ["rev-parse", "--verify", "ORIG_HEAD"], signal).catch(() => null);
|
||||
const original = options.restorePoint ? null : await gitCacheCommand(cwd, ["rev-parse", "--verify", "ORIG_HEAD"], signal).catch(() => null);
|
||||
if (original) await gitCacheCommand(cwd, ["merge-base", "--is-ancestor", original.trim(), "HEAD"], signal)
|
||||
.catch(() => { throw new Error("unique_original_head"); });
|
||||
if (gitDir !== common) {
|
||||
@@ -343,7 +359,12 @@ export async function assertReproducibleGitFiles(cwd: string, signal?: AbortSign
|
||||
const message = await fs.readFile(path.join(gitDir, "COMMIT_EDITMSG"), "utf8").catch((error: NodeJS.ErrnoException) => {
|
||||
if (error.code === "ENOENT") return null; throw error;
|
||||
});
|
||||
if (message !== null && message.trim() !== (await gitCacheCommand(cwd, ["log", "-1", "--format=%B"], signal)).trim()) throw new Error("uncommitted_git_message");
|
||||
if (message !== null && message.trim() !== (await gitCacheCommand(cwd, ["log", "-1", "--format=%B"], signal)).trim()) {
|
||||
const saved = options.restorePoint
|
||||
? (await gitCacheCommand(cwd, ["log", "-g", "--format=%B%x00", "HEAD"], signal).catch(() => "")).split("\0").map(entry => entry.trim())
|
||||
: [];
|
||||
if (!saved.includes(message.trim())) throw new Error("uncommitted_git_message");
|
||||
}
|
||||
const logs = await fs.readdir(path.join(gitDir, "logs")).catch((error: NodeJS.ErrnoException) => {
|
||||
if (error.code === "ENOENT") return []; throw error;
|
||||
});
|
||||
@@ -352,7 +373,7 @@ export async function assertReproducibleGitFiles(cwd: string, signal?: AbortSign
|
||||
// Packed stashes/replacement refs are just as important as loose refs.
|
||||
if ((await gitCacheCommand(cwd, ["for-each-ref", "--format=%(refname)", "refs/stash", "refs/replace"], signal)).trim()) throw new Error("unique_git_refs");
|
||||
// Rewrites can leave commits reachable only from this worktree's reflog.
|
||||
for (const commit of new Set((await gitCacheCommand(cwd, ["reflog", "show", "--format=%H", "HEAD"], signal)).split("\n").filter(Boolean))) {
|
||||
for (const commit of options.restorePoint ? [] : new Set((await gitCacheCommand(cwd, ["reflog", "show", "--format=%H", "HEAD"], signal)).split("\n").filter(Boolean))) {
|
||||
await gitCacheCommand(cwd, ["merge-base", "--is-ancestor", commit, "HEAD"], signal).catch(() => { throw new Error("unique_reflog_commits"); });
|
||||
}
|
||||
return regenerable;
|
||||
@@ -360,6 +381,16 @@ export async function assertReproducibleGitFiles(cwd: string, signal?: AbortSign
|
||||
|
||||
export async function inspectGitCacheRecipe(workspace: Workspace, resolveGitAuth: GitRemoteAuthProvider, signal?: AbortSignal,
|
||||
options: { proveManagedCheckout?: ManagedCheckoutProver } = {}): Promise<GitCacheRestoration> {
|
||||
return (await inspectRecipe(workspace, resolveGitAuth, signal, options, false)).recipe;
|
||||
}
|
||||
|
||||
export async function inspectGitCacheRecipeForRestorePoint(workspace: Workspace, resolveGitAuth: GitRemoteAuthProvider, signal?: AbortSignal,
|
||||
options: { proveManagedCheckout?: ManagedCheckoutProver } = {}) {
|
||||
return inspectRecipe(workspace, resolveGitAuth, signal, options, true);
|
||||
}
|
||||
|
||||
async function inspectRecipe(workspace: Workspace, resolveGitAuth: GitRemoteAuthProvider, signal: AbortSignal | undefined,
|
||||
options: { proveManagedCheckout?: ManagedCheckoutProver }, restorePoint: boolean): Promise<{ recipe: GitCacheRestoration; headOnRemote: boolean }> {
|
||||
if (workspace.mode !== "isolated_workspace" || workspace.strategyType !== "git_worktree"
|
||||
|| !isRuntimeOwnedGitBranch(workspace.metadata) || !workspace.sourceIssueId || !workspace.repoUrl
|
||||
|| !workspace.cwd || !workspace.branchName || (workspace.providerRef && workspace.providerRef !== workspace.cwd)) throw new Error("unproven_workspace_ownership");
|
||||
@@ -378,14 +409,19 @@ export async function inspectGitCacheRecipe(workspace: Workspace, resolveGitAuth
|
||||
if (!(await fs.lstat(path.join(workspace.cwd, ".git"))).isFile()) throw new Error("not_linked_worktree");
|
||||
if ((await gitCacheCommand(workspace.cwd, ["rev-parse", "--path-format=absolute", "--git-common-dir"], signal)).trim() !== path.join(await fs.realpath(source), ".git")) throw new Error("source_identity_changed");
|
||||
if ((await gitCacheCommand(workspace.cwd, ["symbolic-ref", "--short", "HEAD"], signal)).trim() !== workspace.branchName) throw new Error("branch_identity_changed");
|
||||
await assertReproducibleGitFiles(workspace.cwd, signal, { ...options, preserveSettings: true,
|
||||
await assertReproducibleGitFiles(workspace.cwd, signal, { ...options, preserveSettings: true, restorePoint,
|
||||
verifyFetchedCommit: (commit, verificationSignal) => verifyRemoteGitCommit(workspace.repoUrl!, commit, resolveGitAuth, verificationSignal) });
|
||||
const commit = (await gitCacheCommand(workspace.cwd, ["rev-parse", "HEAD"], signal)).trim();
|
||||
await verifyRemoteGitCommit(workspace.repoUrl, commit, resolveGitAuth, signal);
|
||||
return { schema: "paperclip.git-cache-restoration.v1", companyId: workspace.companyId, workspaceId: workspace.id,
|
||||
let headOnRemote = true;
|
||||
try { await verifyRemoteGitCommit(workspace.repoUrl, commit, resolveGitAuth, signal); }
|
||||
catch (error) {
|
||||
if (!restorePoint || signal?.aborted) throw error;
|
||||
headOnRemote = false;
|
||||
}
|
||||
return { headOnRemote, recipe: { schema: "paperclip.git-cache-restoration.v1", companyId: workspace.companyId, workspaceId: workspace.id,
|
||||
sourceIssueId: workspace.sourceIssueId, repoUrl: workspace.repoUrl, commit, branchName: workspace.branchName,
|
||||
sourceKind: recordedIsolatedRemoteSource(workspace) ? "isolated_remote" : "managed_project", projectId: workspace.projectId,
|
||||
sourcePath: source, worktreePath: workspace.cwd, gitBranchOwnershipVersion: 1, verifiedAt: new Date().toISOString() };
|
||||
sourcePath: source, worktreePath: workspace.cwd, gitBranchOwnershipVersion: 1, verifiedAt: new Date().toISOString() } };
|
||||
}
|
||||
|
||||
export function readGitCacheExpiration(workspace: Workspace): GitCacheExpiration | null {
|
||||
@@ -466,7 +502,8 @@ export async function finishGitCacheTrash(workspace: Workspace) {
|
||||
}
|
||||
|
||||
/** Called only after authorization and while holding the source/lifecycle fences. */
|
||||
export async function restoreGitCacheBase(workspace: Workspace, recipe: GitCacheRestoration, resolveGitAuth: GitRemoteAuthProvider) {
|
||||
export async function restoreGitCacheBase(workspace: Workspace, recipe: GitCacheRestoration, resolveGitAuth: GitRemoteAuthProvider,
|
||||
storage?: StorageService) {
|
||||
if (readGitCacheRestoration(workspace)?.commit !== recipe.commit) throw new GitCacheRestorationFailure();
|
||||
try {
|
||||
await finishGitCacheTrash(workspace);
|
||||
@@ -491,6 +528,10 @@ export async function restoreGitCacheBase(workspace: Workspace, recipe: GitCache
|
||||
: (await materializeManagedProjectWorkspace(recipe.sourcePath, { repoUrl: recipe.repoUrl, resolveGitAuth })).cwd;
|
||||
await verifiedGitCacheSource(workspace, recipe.sourcePath);
|
||||
if (!cwd) throw new GitCacheRestorationFailure();
|
||||
if (recipe.restorePoint) {
|
||||
await ensureGitRestorePointCommit({ storage: storage ?? getStorageService(), workspace: { companyId: workspace.companyId, id: workspace.id },
|
||||
ref: recipe.restorePoint, repositoryCwd: cwd });
|
||||
}
|
||||
await ensureSavedGitCommit(cwd, recipe.repoUrl, recipe.commit, resolveGitAuth);
|
||||
return cwd;
|
||||
} catch { throw new GitCacheRestorationFailure(); }
|
||||
@@ -514,7 +555,7 @@ export function savedGitWorkspaceRestoration(input: { metadata?: Record<string,
|
||||
const raw = input.metadata?.[GIT_CACHE_RESTORATION_KEY];
|
||||
if (raw === undefined) return null;
|
||||
const value = raw as GitCacheRestoration | null;
|
||||
if (!value || value.schema !== "paperclip.git-cache-restoration.v1" || !shaPattern.test(value.commit)
|
||||
if (!value || !restorationSchemaMatches(value) || !shaPattern.test(value.commit)
|
||||
|| value.companyId !== input.companyId || value.workspaceId !== input.workspaceId
|
||||
|| value.worktreePath !== input.cwd || value.sourcePath !== input.baseCwd
|
||||
|| value.repoUrl !== input.repoUrl || value.branchName !== input.branchName
|
||||
@@ -522,6 +563,11 @@ export function savedGitWorkspaceRestoration(input: { metadata?: Record<string,
|
||||
|| (value.restoredAt !== undefined && !Number.isFinite(Date.parse(value.restoredAt)))
|
||||
|| (value.contentsRestoredAt != null && !Number.isFinite(Date.parse(value.contentsRestoredAt)))) throw new GitCacheRestorationFailure();
|
||||
validateGitWorkspaceContents(value.contents);
|
||||
return { commit: value.commit, restored: value.restoredAt !== undefined, contents: value.contents,
|
||||
let restorePoint: GitRestorePointRef | null = null;
|
||||
if (value.restorePoint) {
|
||||
try { restorePoint = validateGitRestorePointRef(value.restorePoint, { companyId: input.companyId, id: input.workspaceId ?? "" }); }
|
||||
catch { throw new GitCacheRestorationFailure(); }
|
||||
}
|
||||
return { commit: value.commit, restored: value.restoredAt !== undefined, contents: value.contents, restorePoint,
|
||||
contentsRestored: value.contentsRestoredAt === undefined ? value.restoredAt !== undefined : value.contentsRestoredAt !== null };
|
||||
}
|
||||
@@ -1,6 +1,9 @@
|
||||
import { insertIssueComment } from "./issue-comment-writes.js";
|
||||
import { restoreGitWorkspaceContents } from "./git-workspace-contents.js";
|
||||
import { savedGitWorkspaceRestoration, ensureSavedGitCommit, GitCacheRestorationFailure } from "./git-workspace-restoration.js";
|
||||
import { applyGitRestorePointToWorktree, ensureGitRestorePointCommit } from "./git-restore-point-storage.js";
|
||||
import { getStorageService } from "../storage/index.js";
|
||||
import type { StorageService } from "../storage/types.js";
|
||||
import { sanitizeRuntimeServiceBaseEnv } from "./runtime-service-env.js";
|
||||
import { verifiedRecordedIsolatedRemoteSource } from "./isolated-remote-workspace-source.js";
|
||||
export { sanitizeRuntimeServiceBaseEnv } from "./runtime-service-env.js";
|
||||
@@ -3779,6 +3782,8 @@ function isPersistedTaskDirectory(workspace: {
|
||||
export async function ensurePersistedExecutionWorkspaceAvailable(input: {
|
||||
prepareRestoredLocalSource?: (sourceCwd: string) => Promise<() => Promise<void>>;
|
||||
persistContentsRestoredAt?: (at: string | null) => Promise<void>;
|
||||
persistRestorePointAppliedAt?: (at: string) => Promise<void>;
|
||||
restorePointStorage?: StorageService;
|
||||
db?: Db | null;
|
||||
base: ExecutionWorkspaceInput;
|
||||
workspace: {
|
||||
@@ -3880,6 +3885,27 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: {
|
||||
currentRecipe = { ...currentRecipe as Record<string, unknown>, contentsRestoredAt: at };
|
||||
input.workspace.metadata!.gitCacheRestoration = currentRecipe;
|
||||
};
|
||||
const applyRestorePoint = async (repositoryCwd: string, worktreePath: string) => {
|
||||
const ref = restoration?.restorePoint;
|
||||
if (!ref || ref.appliedAt || !input.workspace.id) return;
|
||||
await applyGitRestorePointToWorktree({ storage: input.restorePointStorage ?? getStorageService(),
|
||||
workspace: { companyId: input.agent.companyId, id: input.workspace.id }, ref, repositoryCwd, worktreePath })
|
||||
.catch(() => { throw new GitCacheRestorationFailure(); });
|
||||
const at = new Date().toISOString();
|
||||
if (input.persistRestorePointAppliedAt) await input.persistRestorePointAppliedAt(at);
|
||||
else {
|
||||
if (!input.db) throw new GitCacheRestorationFailure();
|
||||
const updated = await input.db.update(executionWorkspaces).set({
|
||||
metadata: sql`jsonb_set(${executionWorkspaces.metadata}, '{gitCacheRestoration,restorePoint,appliedAt}', ${JSON.stringify(at)}::jsonb)`,
|
||||
}).where(and(eq(executionWorkspaces.id, input.workspace.id), eq(executionWorkspaces.companyId, input.agent.companyId),
|
||||
sql`${executionWorkspaces.metadata}->'gitCacheRestoration'->'restorePoint'->>'id' = ${ref.id}`))
|
||||
.returning({ id: executionWorkspaces.id });
|
||||
if (updated.length !== 1) throw new GitCacheRestorationFailure();
|
||||
}
|
||||
ref.appliedAt = at;
|
||||
currentRecipe = { ...currentRecipe as Record<string, unknown>, restorePoint: { ...ref } };
|
||||
input.workspace.metadata!.gitCacheRestoration = currentRecipe;
|
||||
};
|
||||
const restoreContents = async () => {
|
||||
if (!restoration?.contents || !input.workspace.id) return;
|
||||
await restoreGitWorkspaceContents({ cwd, companyId: input.agent.companyId, workspaceId: input.workspace.id,
|
||||
@@ -3980,6 +4006,7 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: {
|
||||
realized.warnings = [...repairWarnings, ...baseRefreshWarnings, ...baseDrift.warnings];
|
||||
realized.baseRefSha = refresh.baseRefSha ?? recordedBaseRefSha ?? baseDrift.branchBaseRefSha ?? baseDrift.currentBaseRefSha;
|
||||
await input.afterSourceCopy?.(repoRoot, realized.worktreePath ?? cwd);
|
||||
await applyRestorePoint(repoRoot, realized.worktreePath ?? cwd);
|
||||
if (restoration && !restoration.contentsRestored) await restoreContents();
|
||||
await provisionExecutionWorktree({
|
||||
skipHostProvision: input.skipHostProvision,
|
||||
@@ -4009,6 +4036,11 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: {
|
||||
await fs.mkdir(path.dirname(worktreePath), { recursive: true });
|
||||
await runGit(["worktree", "prune"], repoRoot).catch(() => {});
|
||||
if (savedCommit) {
|
||||
if (restoration?.restorePoint && input.workspace.id) {
|
||||
await ensureGitRestorePointCommit({ storage: input.restorePointStorage ?? getStorageService(),
|
||||
workspace: { companyId: input.agent.companyId, id: input.workspace.id }, ref: restoration.restorePoint, repositoryCwd: repoRoot })
|
||||
.catch(() => { throw new GitCacheRestorationFailure(); });
|
||||
}
|
||||
await ensureSavedGitCommit(repoRoot, input.workspace.repoUrl ?? input.base.repoUrl!, savedCommit, input.resolveGitAuth ?? (async () => null));
|
||||
const existingTip = await resolveBaseRefSha(repoRoot, `refs/heads/${branchName}`);
|
||||
// Once restored, subsequent task commits own this branch. The saved commit
|
||||
@@ -4083,6 +4115,7 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: {
|
||||
});
|
||||
|
||||
await input.afterSourceCopy?.(repoRoot, worktreePath);
|
||||
await applyRestorePoint(repoRoot, worktreePath);
|
||||
await restoreContents();
|
||||
await provisionExecutionWorktree({
|
||||
skipHostProvision: input.skipHostProvision,
|
||||
|
||||
Reference in new issue
Block a user