From 6d44b80a5edbcc9f69bfc3d520994e70aa9fab6d Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Thu, 22 Oct 2026 16:50:15 -0700 Subject: [PATCH] feat: expire closed git worktrees with unsaved work behind restore points (#17248) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Git cache expiration removes a closed issue's worktree only when the remote can reproduce it exactly > - Most closed worktrees hold unpushed commits or uncommitted files, so they are kept forever and disks fill up > - #17238 added restore points (an encrypted save of exactly what the remote cannot reproduce) and #17241 added their lifecycle > - This pull request lets expiration remove those worktrees after saving a restore point, and restores them exactly when the work resumes > - The benefit is bounded disk use for closed work without losing anything a user could need, behind a setting that is off by default ## Linked Issues or Issue Description Refs #17238, #17241. Bug report described in-PR (bug report template). **What happened?** With `PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true` and dry run off, one instance's last 111 expiration checks removed nothing: | Skip reason | Checks | |---|---| | `dirty_or_untracked_files` | 39 | | `unpushed_managed_checkout` | 32 | | `live_or_unresolved_consumer` | 21 | | `unique_ignored_files` | 9 | | commit not on the remote | 10 | Without expiration, the terminal workspace reaper archives only merged, clean worktrees. Closed issues' worktrees therefore accumulate without bound. **Expected behavior** A closed worktree whose only obstacle is unsaved work can be removed after that work is saved, and restored exactly if the issue is reopened or work resumes. **Steps to reproduce** 1. Close an issue whose worktree has an unpushed commit or an uncommitted file. 2. Enable git cache expiration with dry run off. 3. The worktree is never removed. **Paperclip version or commit:** `master` at 12bb7e1976c. **Deployment mode:** any. **Agent adapter(s) involved:** not adapter-specific (core). ## What Changed **Expiration** (`server/src/services/git-workspace-expiration.ts`): - **When it applies:** with `PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true` (default `false`), a worktree that fails the reproducibility check for a relaxable reason is reconsidered once its issue tree has been closed for `PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS` (default `3`). The relaxable reasons are uncommitted or untracked files, a commit not on the remote, reflog or `ORIG_HEAD` history, and ignored `.env` files. A leftover `COMMIT_EDITMSG` still blocks expiration unless it matches a commit in the worktree's reflog, because restore points do not save draft messages. - **What still blocks:** a relaxed inspection (`inspectGitCacheRecipeForRestorePoint`) keeps every other check. Nested managed checkouts, submodules, special git state, local config and other unique ignored files still prevent expiration. - **Capture:** inside the existing source and lifecycle locks, a restore point is captured with the #17238 library against a remote-verified boundary (the `HEAD` itself, or its merge base with `origin/HEAD`), then encrypted, uploaded, read back and hash-checked (`git-restore-point-storage.ts`). - **Commit before touching disk:** it is activated in the #17241 table in the same transaction that archives the workspace with a `paperclip.git-cache-restoration.v2` recipe referencing it, still before anything on disk is touched. - **Final check:** under the file locks, the worktree must still match the captured snapshot (`HEAD`, index, working tree, modes). Otherwise the existing rollback runs. The rollback now also drops the restore point reference and returns the recipe to v1, so a later restore can never apply a stale restore point over the live worktree; the stored restore point is marked superseded. - **Failure paths:** the `pending` row is inserted in a transaction that first registers a `git-restore-points` notification. If the upload or read-back fails, the row is marked `superseded` so retention deletes any uploaded object. Nothing is deleted from disk. - **Encryption key:** restore points are refused (`restore_point_key_not_durable`) on `s3` storage unless the secrets master key comes from `PAPERCLIP_SECRETS_MASTER_KEY`. - **Dry run** reports `restorePoint: true` candidates without storing anything. **Restore** (`git-workspace-restoration.ts`, `workspace-runtime.ts`, `execution-workspaces.ts`): - **Recipe versions:** `readGitCacheRestoration` and `savedGitWorkspaceRestoration` accept v1 without a restore point, or v2 with a valid one. Older servers reject v2 instead of restoring the commit without the saved work. - **Before the saved commit is needed:** reopening (`restoreGitCacheBase`) and the runtime path import the bundle when the saved commit is missing locally. - **After the worktree is re-added:** the runtime always imports the bundle, so history-only commits are present again as objects even when the base was rebuilt (they are not recreated as branches), then applies the restore point, verifies the result, and records `appliedAt`. Reopen persists that through a callback, as it already does for saved contents. - **Never overwriting:** a worktree that already matches is only marked applied. A worktree that diverged from the saved commit's clean state is never overwritten; the restore fails closed and can be retried. Missing, corrupt or expired restore points also fail closed. **Retention:** the #17241 worker's expiry hook now marks the recipe's restore point expired before its object is deleted. **Config and docs:** - `server/src/config.ts` adds the two settings; `doc/recoverable-disk-cleanup.md` and `doc/DEVELOPING.md` describe them. - No code comments were added. **Not in this PR (follow-ups):** - workspaces with nested `.paperclip-repositories` checkouts; - removing idle worktrees of issues that are still open; - starting a fresh workspace with a warning when a restore point has expired. ## Verification - `pnpm exec vitest run src/__tests__/git-restore-point-expiration.test.ts` (in `server/`): 12 passed, with real repositories and embedded Postgres. The cases: - **Round trip:** an unpushed commit, staged and unstaged edits of one file, a deletion, and an untracked file are restored exactly on reopen, `.env` is dropped, and the restore point is active. Restoring through the runtime path works the same way, and a later run never re-applies over newer work. - **Kept:** a worktree that is not yet due, one with the feature disabled (today's skip reason), and one whose encryption key is not durable. - **Change after capture:** rolls back, keeps the worktree, returns the recipe to v1 and supersedes the restore point. - **Missing restore point:** fails closed, keeps the row archived and unapplied, and fails closed again on retry. - **Dry run:** stores nothing. - **Reproducible worktrees:** expire as before, with a v1 recipe. - **History-only commits:** reopen imports them even after they are pruned from the base. - **Draft messages:** a draft commit message keeps the worktree. - **Upload failure:** a failed upload marks the pending restore point for cleanup and keeps the worktree. - **Mutation checks:** each of these fails at least one case: - removing the pre-deletion snapshot check; - keeping the restore point reference on rollback; - removing the runtime apply; - removing the re-apply guard (the divergence check then fails closed instead of overwriting); - skipping the import for a matching worktree, relaxing the draft message guard, or not marking a failed upload's row. - `pnpm exec vitest run src/__tests__/git-restore-point-expiration-config.test.ts`: 5 passed. - `pnpm exec vitest run git-workspace-expiration git-workspace-contents git-restore-point workspace-runtime execution-workspaces` (in `server/`): 529 passed on the first revision. Two `workspace-runtime.test.ts` cases about remote base refs fail, and they also fail on unmodified `master` on this machine. After the review fixes, the expiration, contents, restore point and execution workspace suites passed again: 222 tests. - `pnpm run typecheck` (in `server/`): pass. - The server builds, the runtime bundle builds, and `scripts/server-boot-smoke.sh node server/dist-runtime/index.mjs` reaches healthy. ## Risks - **Off by default:** nothing changes unless both `PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true` and `PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true` are set. Roll out with `PAPERCLIP_GIT_CACHE_EXPIRATION_DRY_RUN=true` first to see which worktrees would receive restore points. - **Data at rest:** restore points contain source and uncommitted files, encrypted with keys wrapped by the instance secrets key. Losing that key makes them unrecoverable, which is why `s3` storage requires an environment-provided key. Untracked `.env` files are not stored. - **Interrupted apply:** a crash in the middle of applying a restore point leaves a worktree that neither matches the restore point nor is clean. The restore then fails closed rather than overwriting it, and an operator must reset that worktree to retry. - **Rollback:** disable the setting. Already-expired worktrees keep their v2 recipes and restore points, so reverting the code would leave those recipes unreadable until restored by a server with this change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code with tool use (shell, file edits, tests, local server builds and boots, read-only measurements on live instances). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/DEVELOPING.md | 6 + doc/recoverable-disk-cleanup.md | 34 +++ ...it-restore-point-expiration-config.test.ts | 25 ++ .../git-restore-point-expiration.test.ts | 244 +++++++++++++++++ server/src/config.ts | 5 + server/src/index.ts | 7 + server/src/services/execution-workspaces.ts | 18 ++ .../src/services/git-restore-point-storage.ts | 251 ++++++++++++++++++ .../src/services/git-workspace-expiration.ts | 87 +++++- .../src/services/git-workspace-restoration.ts | 74 +++++- server/src/services/workspace-runtime.ts | 33 +++ 11 files changed, 760 insertions(+), 24 deletions(-) create mode 100644 server/src/__tests__/git-restore-point-expiration-config.test.ts create mode 100644 server/src/__tests__/git-restore-point-expiration.test.ts create mode 100644 server/src/services/git-restore-point-storage.ts diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 53504cd79e8..66d70867c58 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -2583,6 +2583,12 @@ Environment overrides: negative, empty, or non-numeric value falls back to the default. `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED=false` turns expiry off, and `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DRY_RUN=true` only reports candidates. +- `PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true` lets git cache expiration remove closed + worktrees with unpushed or uncommitted work after saving an encrypted restore point + (see `doc/recoverable-disk-cleanup.md`). It is off by default. + `PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS=` sets how long the issue tree + must be closed first. The default is `3`. A negative, empty, or non-numeric value falls + back to the default. - `PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS=` sets how long a git workspace restore point is kept after its issue tree closes (see `doc/recoverable-disk-cleanup.md`). The default is `90` and the minimum is `1`. A diff --git a/doc/recoverable-disk-cleanup.md b/doc/recoverable-disk-cleanup.md index 915c404e83c..42ba628c1bc 100644 --- a/doc/recoverable-disk-cleanup.md +++ b/doc/recoverable-disk-cleanup.md @@ -95,6 +95,40 @@ bytes, cleared task sessions, and skip reasons. ## Git workspace restore points +Git cache expiration (`PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true`) normally removes a +closed issue's worktree only when its remote can reproduce it exactly. With +`PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true`, it can also remove a worktree that has +unpushed commits, uncommitted or untracked files, or history-only commits, once its issue +tree has been closed for `PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS` (default `3`). +Before removal it saves a restore point: + +- a git bundle of commits the remote does not have, against a remote-verified base, + including commits only reachable from the reflog or `ORIG_HEAD`; +- deterministic snapshots of the index and working tree, including untracked files and + non-default file modes. Ignored files are left out, and so are untracked `.env` and + `.env.*` files, which provisioning recreates with current secrets. + +The restore point is encrypted with a per-restore-point key wrapped by the instance +secrets key, uploaded to file storage, read back and hash-checked, then activated in +the same transaction that archives the workspace with a versioned recipe +(`paperclip.git-cache-restoration.v2`). Older servers reject that recipe instead of +restoring without the saved work. Under the expiration locks, the worktree must still +match the captured snapshot exactly; otherwise expiration rolls back, keeps the +worktree, and drops the restore point. Nested managed checkouts, submodules, special +git state and ignored files other than `.env` still keep a worktree from expiring. +Restore points are refused when the file storage is `s3` and the secrets master key +does not come from `PAPERCLIP_SECRETS_MASTER_KEY`. + +Reopening the issue, or the next run that recreates the worktree, imports the bundle, +re-adds the worktree at the saved commit, applies the restore point and verifies the +result. The bundle is always imported, so history-only commits are present again as +objects in the repository; they are not recreated as branches or reflog entries, and the +restore point keeps them until it expires. A draft commit message left by an aborted +commit still keeps the worktree from expiring, because restore points do not save it. A worktree that already matches is only marked applied, and a worktree that +diverged is never overwritten: the restore fails closed and can be retried. A missing, +corrupt or expired restore point also fails closed. + + Restore points are encrypted saves of the parts of a git worktree that its remote cannot reproduce: unpushed commits and uncommitted files. They let a closed issue's worktree be removed and restored later. The table `execution_workspace_restore_points` diff --git a/server/src/__tests__/git-restore-point-expiration-config.test.ts b/server/src/__tests__/git-restore-point-expiration-config.test.ts new file mode 100644 index 00000000000..5d1aacc3999 --- /dev/null +++ b/server/src/__tests__/git-restore-point-expiration-config.test.ts @@ -0,0 +1,25 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; +import { loadConfig } from "../config.ts"; + +describe("git restore point expiration config parsing", () => { + afterEach(() => { + vi.unstubAllEnvs(); + }); + + it("is disabled by default with a three-day delay", () => { + vi.stubEnv("PAPERCLIP_GIT_RESTORE_POINTS_ENABLED", undefined); + vi.stubEnv("PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS", undefined); + expect(loadConfig()).toMatchObject({ gitRestorePointsEnabled: false, gitRestorePointTerminalDelayDays: 3 }); + }); + + it("accepts explicit settings", () => { + vi.stubEnv("PAPERCLIP_GIT_RESTORE_POINTS_ENABLED", "true"); + vi.stubEnv("PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS", "0"); + expect(loadConfig()).toMatchObject({ gitRestorePointsEnabled: true, gitRestorePointTerminalDelayDays: 0 }); + }); + + it.each(["", "-1", "many"])("falls back to three days for %j", (value) => { + vi.stubEnv("PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS", value); + expect(loadConfig()).toMatchObject({ gitRestorePointTerminalDelayDays: 3 }); + }); +}); diff --git a/server/src/__tests__/git-restore-point-expiration.test.ts b/server/src/__tests__/git-restore-point-expiration.test.ts new file mode 100644 index 00000000000..fd27b0a24ae --- /dev/null +++ b/server/src/__tests__/git-restore-point-expiration.test.ts @@ -0,0 +1,244 @@ +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { execFileSync } from "node:child_process"; +import { randomUUID } from "node:crypto"; +import { pathToFileURL } from "node:url"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { eq, sql } from "drizzle-orm"; +import { companies, createDb, executionWorkspaceRestorePoints, executionWorkspaces, issues } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { materializeIsolatedRemoteWorkspaceSource } from "../services/isolated-remote-workspace-source.js"; +import { ensurePersistedExecutionWorkspaceAvailable, realizeExecutionWorkspace } from "../services/workspace-runtime.js"; +import { executionWorkspaceService } from "../services/execution-workspaces.js"; +import { expireReconstructableGitWorkspace } from "../services/git-workspace-expiration.js"; +import { getStorageService } from "../storage/index.js"; + +const gitEnv = { ...process.env, GIT_AUTHOR_NAME: "Fixture", GIT_AUTHOR_EMAIL: "fixture@example.test", + GIT_COMMITTER_NAME: "Fixture", GIT_COMMITTER_EMAIL: "fixture@example.test", GIT_TERMINAL_PROMPT: "0" }; +const git = (cwd: string, ...args: string[]) => execFileSync("git", args, { cwd, env: gitEnv, encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }).trim(); +const exists = (target: string) => fs.lstat(target).then(() => true, () => false); +const DAY_MS = 24 * 60 * 60 * 1000; + +const support = await getEmbeddedPostgresTestSupport(); +const describeEmbeddedPostgres = support.supported ? describe : describe.skip; +describeEmbeddedPostgres("git restore point expiration", () => { + let db: ReturnType; + let database: Awaited>; + let root: string; + const previousHome = process.env.PAPERCLIP_HOME; + const previousInstance = process.env.PAPERCLIP_INSTANCE_ID; + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("git-restore-point-expiration-"); + db = createDb(database.connectionString); + root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-restore-point-expiration-")); + process.env.PAPERCLIP_HOME = path.join(root, "home"); + process.env.PAPERCLIP_INSTANCE_ID = "default"; + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32)); + }); + afterEach(async () => { + await db.delete(executionWorkspaceRestorePoints); + await db.execute(sql`TRUNCATE TABLE companies CASCADE`); + }); + afterAll(async () => { + vi.unstubAllEnvs(); + if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome; + if (previousInstance === undefined) delete process.env.PAPERCLIP_INSTANCE_ID; else process.env.PAPERCLIP_INSTANCE_ID = previousInstance; + await db.$client.end({ timeout: 1 }); await database.cleanup(); await fs.rm(root, { recursive: true, force: true }); + }); + + async function fixture() { + const companyId = randomUUID(), issueId = randomUUID(), workspaceId = randomUUID(); + const location = path.join(root, companyId), source = path.join(location, "author"), remote = path.join(location, "remote.git"); + await fs.mkdir(source, { recursive: true }); + git(source, "init", "--initial-branch=main"); + await fs.writeFile(path.join(source, "marker.txt"), "commit A\n"); + await fs.writeFile(path.join(source, "remove-me.txt"), "tracked\n"); + await fs.writeFile(path.join(source, ".gitignore"), "node_modules/\n.env\n.claude/settings.local.json\n"); + git(source, "add", "."); git(source, "commit", "-m", "A"); + git(location, "clone", "--bare", source, remote); git(source, "remote", "add", "origin", remote); + await db.insert(companies).values({ id: companyId, name: "Restore point fixture", issuePrefix: `RP${companyId.slice(0, 6)}` }); + await db.insert(issues).values({ id: issueId, companyId, title: "Finished task", status: "done", completedAt: new Date(Date.now() - 4 * DAY_MS) }); + const repoUrl = pathToFileURL(remote).href; + const baseCwd = await materializeIsolatedRemoteWorkspaceSource({ companyId, issueId, repoUrl, resolveGitAuth: async () => null }); + const base = { baseCwd, source: "task_session" as const, projectId: null, workspaceId: null, repoUrl, repoRef: "origin/main" }; + const workspace = await realizeExecutionWorkspace({ base, issue: { id: issueId, identifier: "RP-1", title: "Fixture" }, agent: { id: null, name: "Fixture", companyId }, + config: { workspaceStrategy: { type: "git_worktree", branchTemplate: "fixture-task", worktreeParentDir: path.join(location, "worktrees"), baseRef: "origin/main" } }, skipHostProvision: true }); + await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, sourceIssueId: issueId, mode: "isolated_workspace", strategyType: "git_worktree", name: "Fixture", status: "active", + providerType: "local_fs", cwd: workspace.cwd, providerRef: workspace.cwd, repoUrl, baseRef: "origin/main", branchName: workspace.branchName, + metadata: { createdByRuntime: workspace.branchCreatedByRuntime, gitBranchOwnershipVersion: 1, baseRefSha: workspace.baseRefSha, + isolatedRemoteSource: { schema: "paperclip.isolated-remote-source.v1", companyId, issueId, repoUrl } } }); + await db.update(issues).set({ executionWorkspaceId: workspaceId }).where(eq(issues.id, issueId)); + const row = () => db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId)).then(rows => rows[0]!); + const restorePoint = (overrides: Partial<{ notBefore: Date; keyDurable: boolean }> = {}) => ({ + storage: getStorageService(), notBefore: new Date(Date.now() - DAY_MS), keyDurable: true, ...overrides, + }); + const expire = async (options: Partial[2]> = {}) => + expireReconstructableGitWorkspace(db, await row(), { dryRun: false, expireBase: false, now: new Date(), restorePoint: restorePoint(), ...options }); + const reopen = () => executionWorkspaceService(db).reopenClosedIsolatedExecutionWorkspaceForIssue({ workspaceId, + issue: { id: issueId, companyId, projectId: null }, actor: { actorType: "user", agentId: null }, + authorizationActor: { type: "board", source: "local_implicit", isInstanceAdmin: true } }); + const ledger = () => db.select().from(executionWorkspaceRestorePoints).where(eq(executionWorkspaceRestorePoints.executionWorkspaceId, workspaceId)); + const runtimeRestore = async () => ensurePersistedExecutionWorkspaceAvailable({ db, base, + workspace: (await executionWorkspaceService(db).getStoredById(companyId, workspaceId))!, + issue: { id: issueId, identifier: "RP-1", title: "Resumed task" }, + agent: { id: null, name: "Fixture", companyId }, skipHostProvision: true }); + return { companyId, issueId, workspaceId, cwd: workspace.cwd, baseCwd, row, restorePoint, expire, reopen, ledger, runtimeRestore }; + } + + async function makeUnsavedWork(cwd: string) { + await fs.writeFile(path.join(cwd, "marker.txt"), "commit B (unpushed)\n"); + git(cwd, "commit", "-am", "B unpushed"); + const unpushed = git(cwd, "rev-parse", "HEAD"); + await fs.writeFile(path.join(cwd, "marker.txt"), "staged edit\n"); + git(cwd, "add", "marker.txt"); + await fs.writeFile(path.join(cwd, "marker.txt"), "unstaged edit\n"); + await fs.rm(path.join(cwd, "remove-me.txt")); + await fs.mkdir(path.join(cwd, "notes")); + await fs.writeFile(path.join(cwd, "notes", "todo.md"), "untracked work\n"); + await fs.writeFile(path.join(cwd, ".env"), "TOKEN=old\n"); + return unpushed; + } + + it("expires a worktree with unpushed and uncommitted work and restores it exactly on reopen", async () => { + const f = await fixture(); + const unpushed = await makeUnsavedWork(f.cwd); + expect(await f.expire()).toMatchObject({ candidate: true, expired: true, restorePoint: true, skipReason: null }); + expect(await exists(f.cwd)).toBe(false); + const recipe = (await f.row()).metadata?.gitCacheRestoration as Record; + expect(recipe).toMatchObject({ schema: "paperclip.git-cache-restoration.v2", commit: unpushed, + restorePoint: { head: unpushed, headOnRemote: false, appliedAt: null } }); + expect(await f.ledger()).toMatchObject([{ status: "active", objectKey: recipe.restorePoint.objectKey }]); + + expect(await f.reopen()).toMatchObject({ ok: true, reopened: true }); + expect(git(f.cwd, "rev-parse", "HEAD")).toBe(unpushed); + expect(await fs.readFile(path.join(f.cwd, "marker.txt"), "utf8")).toBe("unstaged edit\n"); + expect(git(f.cwd, "show", ":marker.txt")).toBe("staged edit"); + await expect(fs.access(path.join(f.cwd, "remove-me.txt"))).rejects.toMatchObject({ code: "ENOENT" }); + expect(await fs.readFile(path.join(f.cwd, "notes", "todo.md"), "utf8")).toBe("untracked work\n"); + await expect(fs.access(path.join(f.cwd, ".env"))).rejects.toMatchObject({ code: "ENOENT" }); + const restored = (await f.row()).metadata?.gitCacheRestoration as Record; + expect(restored.restorePoint.appliedAt).toEqual(expect.any(String)); + }); + + it("restores through the runtime path and never re-applies over later work", async () => { + const f = await fixture(); + const unpushed = await makeUnsavedWork(f.cwd); + expect(await f.expire()).toMatchObject({ expired: true }); + await db.update(executionWorkspaces).set({ status: "active", closedAt: null }).where(eq(executionWorkspaces.id, f.workspaceId)); + await f.runtimeRestore(); + expect(git(f.cwd, "rev-parse", "HEAD")).toBe(unpushed); + expect(await fs.readFile(path.join(f.cwd, "notes", "todo.md"), "utf8")).toBe("untracked work\n"); + const applied = (await f.row()).metadata?.gitCacheRestoration as Record; + expect(applied.restorePoint.appliedAt).toEqual(expect.any(String)); + await fs.writeFile(path.join(f.cwd, "notes", "todo.md"), "later work\n"); + await f.runtimeRestore(); + expect(await fs.readFile(path.join(f.cwd, "notes", "todo.md"), "utf8")).toBe("later work\n"); + }); + + it("imports history-only commits on reopen even when the base no longer has them", async () => { + const f = await fixture(); + await fs.writeFile(path.join(f.cwd, "lost.txt"), "rewritten away\n"); + git(f.cwd, "add", "lost.txt"); + git(f.cwd, "commit", "-m", "rewritten away"); + const lost = git(f.cwd, "rev-parse", "HEAD"); + git(f.cwd, "reset", "-q", "--hard", "HEAD~1"); + expect(await f.expire()).toMatchObject({ expired: true, restorePoint: true }); + git(f.baseCwd, "reflog", "expire", "--expire=now", "--all"); + git(f.baseCwd, "gc", "-q", "--prune=now"); + expect(() => git(f.baseCwd, "cat-file", "-e", `${lost}^{commit}`)).toThrow(); + expect(await f.reopen()).toMatchObject({ ok: true, reopened: true }); + expect(git(f.baseCwd, "cat-file", "-t", lost)).toBe("commit"); + }); + + it("keeps a worktree whose draft commit message a restore point cannot save", async () => { + const f = await fixture(); + await fs.writeFile(path.join(f.cwd, "marker.txt"), "staged edit\n"); + git(f.cwd, "add", "marker.txt"); + const gitDir = git(f.cwd, "rev-parse", "--absolute-git-dir"); + await fs.writeFile(path.join(gitDir, "COMMIT_EDITMSG"), "Draft message from an aborted commit\n"); + expect(await f.expire()).toMatchObject({ expired: false, skipReason: "uncommitted_git_message" }); + expect(await exists(f.cwd)).toBe(true); + expect(await f.ledger()).toEqual([]); + }); + + it("marks a restore point for cleanup when its upload fails", async () => { + const f = await fixture(); + await makeUnsavedWork(f.cwd); + const storage = getStorageService(); + const failing = { ...storage, putFile: async () => { throw new Error("storage unavailable"); } }; + expect(await f.expire({ restorePoint: { ...f.restorePoint(), storage: failing as typeof storage } })) + .toMatchObject({ expired: false }); + expect(await exists(f.cwd)).toBe(true); + expect(await f.ledger()).toMatchObject([{ status: "superseded" }]); + expect((await f.row()).metadata?.gitCacheRestoration).toBeUndefined(); + }); + + it("does not expire unsaved work before the restore point delay", async () => { + const f = await fixture(); + await makeUnsavedWork(f.cwd); + expect(await f.expire({ restorePoint: f.restorePoint({ notBefore: new Date(Date.now() + DAY_MS) }) })) + .toMatchObject({ expired: false, skipReason: "restore_point_not_due" }); + expect(await exists(f.cwd)).toBe(true); + expect(await f.ledger()).toEqual([]); + }); + + it("keeps today's behavior when restore points are disabled", async () => { + const f = await fixture(); + await makeUnsavedWork(f.cwd); + expect(await f.expire({ restorePoint: undefined })).toMatchObject({ expired: false, skipReason: "dirty_or_untracked_files" }); + expect(await exists(f.cwd)).toBe(true); + }); + + it("refuses to capture when the encryption key is not durable", async () => { + const f = await fixture(); + await makeUnsavedWork(f.cwd); + expect(await f.expire({ restorePoint: f.restorePoint({ keyDurable: false }) })) + .toMatchObject({ expired: false, skipReason: "restore_point_key_not_durable" }); + expect(await exists(f.cwd)).toBe(true); + }); + + it("keeps the worktree and drops the restore point when contents change after capture", async () => { + const f = await fixture(); + await makeUnsavedWork(f.cwd); + const result = await f.expire({ beforeDelete: async () => { await fs.writeFile(path.join(f.cwd, "notes", "todo.md"), "changed after capture\n"); } }); + expect(result).toMatchObject({ expired: false, skipReason: "restore_point_contents_changed" }); + expect(await exists(f.cwd)).toBe(true); + expect(await fs.readFile(path.join(f.cwd, "notes", "todo.md"), "utf8")).toBe("changed after capture\n"); + const after = await f.row(); + expect(after.status).toBe("active"); + expect(after.metadata?.gitCacheRestoration).toMatchObject({ schema: "paperclip.git-cache-restoration.v1" }); + expect((after.metadata?.gitCacheRestoration as Record).restorePoint).toBeUndefined(); + expect(await f.ledger()).toMatchObject([{ status: "superseded" }]); + }); + + it("fails closed when the stored restore point is missing", async () => { + const f = await fixture(); + await makeUnsavedWork(f.cwd); + expect(await f.expire()).toMatchObject({ expired: true }); + const recipe = (await f.row()).metadata?.gitCacheRestoration as Record; + await getStorageService().deleteObject(f.companyId, recipe.restorePoint.objectKey); + expect(await f.reopen()).toMatchObject({ ok: false, code: "rebuild_failed" }); + const failed = await f.row(); + expect(failed.status).toBe("archived"); + expect(failed.metadata?.gitCacheRestoration).toMatchObject({ restorePoint: { objectKey: recipe.restorePoint.objectKey, appliedAt: null } }); + await expect(fs.access(path.join(f.cwd, "notes", "todo.md"))).rejects.toMatchObject({ code: "ENOENT" }); + expect(await f.reopen()).toMatchObject({ ok: false, code: "rebuild_failed" }); + }); + + it("reports a restore point candidate in dry run without storing anything", async () => { + const f = await fixture(); + await makeUnsavedWork(f.cwd); + expect(await f.expire({ dryRun: true })).toMatchObject({ candidate: true, expired: false, restorePoint: true, skipReason: null }); + expect(await exists(f.cwd)).toBe(true); + expect(await f.ledger()).toEqual([]); + expect((await f.row()).metadata?.gitCacheRestoration).toBeUndefined(); + }); + + it("still expires a fully reproducible worktree without a restore point", async () => { + const f = await fixture(); + expect(await f.expire()).toMatchObject({ expired: true, restorePoint: false }); + expect((await f.row()).metadata?.gitCacheRestoration).toMatchObject({ schema: "paperclip.git-cache-restoration.v1" }); + expect(await f.ledger()).toEqual([]); + }); +}); diff --git a/server/src/config.ts b/server/src/config.ts index 23b1ee84c51..bc974203ec6 100644 --- a/server/src/config.ts +++ b/server/src/config.ts @@ -92,6 +92,8 @@ export interface Config { nativeHarnessBackupExpirationDryRun: boolean; nativeHarnessBackupExpirationDays: number; gitRestorePointRetentionDays: number; + gitRestorePointsEnabled: boolean; + gitRestorePointTerminalDelayDays: number; serveUi: boolean; uiDevMiddleware: boolean; secretsProvider: SecretProvider; @@ -339,6 +341,7 @@ export function loadConfig(): Config { const skillVersionRetentionGraceMinutes = nonNegativeIntegerEnv("PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES", 60); const nativeHarnessBackupExpirationDays = Math.max(1, nonNegativeIntegerEnv("PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DAYS", 7)); const gitRestorePointRetentionDays = Math.max(1, nonNegativeIntegerEnv("PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS", 90)); + const gitRestorePointTerminalDelayDays = nonNegativeIntegerEnv("PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS", 3); const bindValidationErrors = validateConfiguredBindMode({ deploymentMode, deploymentExposure, @@ -397,6 +400,8 @@ export function loadConfig(): Config { nativeHarnessBackupExpirationDryRun: process.env.PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DRY_RUN === "true", nativeHarnessBackupExpirationDays, gitRestorePointRetentionDays, + gitRestorePointsEnabled: process.env.PAPERCLIP_GIT_RESTORE_POINTS_ENABLED === "true", + gitRestorePointTerminalDelayDays, serveUi: process.env.SERVE_UI !== undefined ? process.env.SERVE_UI === "true" diff --git a/server/src/index.ts b/server/src/index.ts index 6b6de3cb71b..d8f26a3dbf9 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -13,6 +13,7 @@ import { nativeHarnessBackupExpirationService, } from "./services/native-runtime/native-harness-backup-expiration.js"; import { gitRestorePointRetentionService, registerGitRestorePointWork } from "./services/git-restore-point-ledger.js"; +import { markGitRestorePointExpired } from "./services/git-restore-point-storage.js"; import { subscribeAllCompanyLiveEvents } from "./services/live-events.js"; import { DELIVERY_QUEUES } from "./services/delivery-work-notifications.js"; import { registerBudgetEnforcementWork, registerCostAccountingWork } from "./services/budget-enforcement-worker.js"; @@ -1479,6 +1480,11 @@ async function startServerWithDatabaseTeardown( gitCacheExpirationEnabled: config.gitCacheExpirationEnabled, gitBaseExpirationEnabled: config.gitBaseExpirationEnabled, gitCacheExpirationDryRun: config.gitCacheExpirationDryRun, + gitRestorePoints: config.gitRestorePointsEnabled ? { + storage: storageService, + terminalDelayMs: config.gitRestorePointTerminalDelayDays * 24 * 60 * 60 * 1000, + keyDurable: config.storageProvider !== "s3" || Boolean(process.env.PAPERCLIP_SECRETS_MASTER_KEY?.trim()), + } : undefined, workspaceReaperCooldownDays: config.workspaceReaperCooldownDays, workspaceCompactionDelayMs: config.workspaceCompactionDelayHours === null ? null @@ -1501,6 +1507,7 @@ async function startServerWithDatabaseTeardown( const gitRestorePointWorker = registerGitRestorePointWork(deliveryWork, gitRestorePointRetentionService(db as any, { storage: storageService, retentionDays: config.gitRestorePointRetentionDays, + beforeExpire: (restorePoint) => markGitRestorePointExpired(db as any, restorePoint), }), (result) => { if (result.abandoned > 0 || result.expired > 0 || result.orphaned > 0 || result.deleted > 0 || result.failed > 0) { logger.info(result, "git restore point retention"); diff --git a/server/src/services/execution-workspaces.ts b/server/src/services/execution-workspaces.ts index 91f7a0d03d3..13830397cee 100644 --- a/server/src/services/execution-workspaces.ts +++ b/server/src/services/execution-workspaces.ts @@ -5,6 +5,7 @@ import { measureRunPhase } from "./execution-preparation-timing.js"; import { assertExecutionWorkspaceRecordAccess, assertTaskWorkspaceExecutionAccess, assertTaskWorkspaceSourceProjectAccess } from "./task-workspace-access.js"; import { verifiedRecordedIsolatedRemoteSource, withIsolatedRepositorySourceLock } from "./isolated-remote-workspace-source.js"; import { expireReconstructableGitWorkspace, hasPendingGitExpiration } from "./git-workspace-expiration.js"; +import type { StorageService } from "../storage/types.js"; import { readGitCacheRestoration, restoreGitCacheBase, recoverGitCacheExpirationLocks, GitCacheRestorationFailure } from "./git-workspace-restoration.js"; import { parseObject } from "../adapters/utils.js"; import { resolveCoreTrustPreset } from "./trust-preset-resolver.js"; @@ -289,6 +290,7 @@ export type ExecutionWorkspaceServiceOptions = { gitCacheExpirationEnabled?: boolean; gitBaseExpirationEnabled?: boolean; gitCacheExpirationDryRun?: boolean; + gitRestorePoints?: { storage: StorageService; terminalDelayMs: number; keyDurable: boolean }; terminalWorkspaceInspectionBackoffMs?: number; resolvePullRequestDetails?: PullRequestMergeDetailsResolver; now?: () => Date; @@ -3343,6 +3345,11 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic const expiration = await expireReconstructableGitWorkspace(db, workspace, { dryRun: opts.gitCacheExpirationDryRun ?? false, expireBase: opts.gitBaseExpirationEnabled ?? false, now: now(), signal, beforeDelete: opts.beforeTerminalWorkspaceCleanup, + restorePoint: opts.gitRestorePoints ? { + storage: opts.gitRestorePoints.storage, + notBefore: new Date(latestTerminalAt + opts.gitRestorePoints.terminalDelayMs), + keyDurable: opts.gitRestorePoints.keyDurable, + } : undefined, }); const expirationDueAt = now().getTime() + 60 * 60 * 1000; terminalWorkspaceDeadlines.set(workspace.id, @@ -3989,6 +3996,17 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic if (updated.length !== 1) throw new GitCacheRestorationFailure(); nextMetadata.gitCacheRestoration = { ...currentRecipe as Record, contentsRestoredAt: at }; }, + persistRestorePointAppliedAt: async at => { + const currentRecipe = nextMetadata.gitCacheRestoration as Record; + const appliedRecipe = { ...currentRecipe, restorePoint: { ...currentRecipe.restorePoint as Record, appliedAt: at } }; + const updated = await db.update(executionWorkspaces).set({ + metadata: sql`jsonb_set(${executionWorkspaces.metadata}, '{gitCacheRestoration}', ${JSON.stringify(appliedRecipe)}::jsonb)`, + }).where(and(eq(executionWorkspaces.id, row.id), eq(executionWorkspaces.companyId, row.companyId), + sql`${executionWorkspaces.metadata}->'gitCacheRestoration' = ${JSON.stringify(currentRecipe)}::jsonb`)) + .returning({ id: executionWorkspaces.id }); + if (updated.length !== 1) throw new GitCacheRestorationFailure(); + nextMetadata.gitCacheRestoration = appliedRecipe; + }, prepareRestoredLocalSource: async sourceCwd => { const sourceIdentity = await fs.realpath(sourceCwd); const targetIdentity = await canonicalRepositoryRoot(row.cwd!); diff --git a/server/src/services/git-restore-point-storage.ts b/server/src/services/git-restore-point-storage.ts new file mode 100644 index 00000000000..f0aaa5b2f56 --- /dev/null +++ b/server/src/services/git-restore-point-storage.ts @@ -0,0 +1,251 @@ +import { createHash, randomUUID } from "node:crypto"; +import { and, eq, inArray, ne, sql } from "drizzle-orm"; +import { executionWorkspaceRestorePoints, executionWorkspaces, type Db } from "@paperclipai/db"; +import type { StoredSecretVersionMaterial } from "../secrets/types.js"; +import type { StorageService } from "../storage/types.js"; +import { + applyRestorePoint, + captureRestorePoint, + currentRestorePointSnapshot, + decodeRestorePointPayload, + importRestorePoint, + openRestorePoint, + RestorePointRestoreFailure, + sealRestorePoint, + type RestorePointLimits, + type RestorePointManifest, +} from "./git-restore-point.js"; +import { notifyGitRestorePointWork } from "./git-restore-point-ledger.js"; +import { gitCacheCommand } from "./git-workspace-restoration.js"; + +export const GIT_RESTORE_POINT_SCHEMA = "paperclip.git-restore-point.v1"; +const NAMESPACE = "git-restore-points"; +const MAX_RESTORE_POINT_BYTES = 512 * 1024 * 1024; + +export type GitRestorePointRef = { + schema: typeof GIT_RESTORE_POINT_SCHEMA; + id: string; + objectKey: string; + sha256: string; + byteSize: number; + key: StoredSecretVersionMaterial; + generation: number; + head: string; + index: string | null; + worktree: string | null; + modes: Array<{ path: string; mode: number }>; + headOnRemote: boolean; + appliedAt?: string | null; + expiredAt?: string; +}; + +type Tx = Parameters[0]>[0]; + +function restorePointObjectKey(companyId: string, workspaceId: string, generation: number, id: string) { + return `${companyId}/${NAMESPACE}/${workspaceId}/${generation}/${id}.pcrp`; +} + +export function validateGitRestorePointRef(value: unknown, workspace: { companyId: string; id: string }): GitRestorePointRef { + const ref = value as GitRestorePointRef | null; + const sha = /^[0-9a-f]{40}$/; + if (!ref || ref.schema !== GIT_RESTORE_POINT_SCHEMA || typeof ref.id !== "string" + || ref.objectKey !== restorePointObjectKey(workspace.companyId, workspace.id, ref.generation, ref.id) + || !/^[0-9a-f]{64}$/.test(ref.sha256) || !Number.isSafeInteger(ref.byteSize) || ref.byteSize <= 0 || ref.byteSize > MAX_RESTORE_POINT_BYTES + || !ref.key || typeof ref.key !== "object" || Array.isArray(ref.key) + || !Number.isSafeInteger(ref.generation) || ref.generation < 1 + || !sha.test(ref.head) || (ref.index !== null && !sha.test(ref.index)) || (ref.worktree !== null && !sha.test(ref.worktree)) + || (ref.index === null) !== (ref.worktree === null) || typeof ref.headOnRemote !== "boolean" + || !Array.isArray(ref.modes) || ref.modes.length > 20_000 || ref.modes.some((entry) => !entry || typeof entry.path !== "string" + || entry.path.startsWith("/") || entry.path.split("/").includes("..") || !Number.isInteger(entry.mode) || entry.mode < 0 || entry.mode > 0o777) + || (ref.appliedAt != null && !Number.isFinite(Date.parse(ref.appliedAt))) + || (ref.expiredAt !== undefined && !Number.isFinite(Date.parse(ref.expiredAt)))) { + throw new RestorePointRestoreFailure("invalid_restore_point_reference"); + } + return ref; +} + +export async function storeGitRestorePoint(input: { + db: Db; + storage: StorageService; + workspace: { companyId: string; id: string; sourceIssueId: string | null }; + cwd: string; + generation: number; + upstreamCommits: readonly string[]; + headOnRemote: boolean; + limits?: Partial; + signal?: AbortSignal; +}): Promise<{ ref: GitRestorePointRef; ledgerId: string; manifest: RestorePointManifest } | null> { + const captured = await captureRestorePoint({ cwd: input.cwd, upstreamCommits: input.upstreamCommits, limits: input.limits, signal: input.signal }); + if (!captured) return null; + const id = randomUUID(); + const identity = { companyId: input.workspace.companyId, workspaceId: input.workspace.id, restorePointId: id, generation: input.generation }; + const sealed = await sealRestorePoint(captured.payload, identity); + const objectKey = restorePointObjectKey(input.workspace.companyId, input.workspace.id, input.generation, id); + const [ledger] = await input.db.transaction(async (tx) => { + await notifyGitRestorePointWork(tx, `workspace:${input.workspace.id}`); + return tx.insert(executionWorkspaceRestorePoints).values({ + companyId: input.workspace.companyId, + executionWorkspaceId: input.workspace.id, + sourceIssueId: input.workspace.sourceIssueId, + generation: input.generation, + objectKey, + sha256: sealed.sha256, + byteSize: sealed.ciphertext.length, + status: "pending", + }).returning({ id: executionWorkspaceRestorePoints.id }); + }); + try { + await input.storage.putFile({ + objectKey, + companyId: input.workspace.companyId, + namespace: NAMESPACE, + originalFilename: null, + contentType: "application/octet-stream", + body: sealed.ciphertext, + }); + const stored = await readObject(input.storage, input.workspace.companyId, objectKey, input.signal); + if (createHash("sha256").update(stored).digest("hex") !== sealed.sha256) { + throw new RestorePointRestoreFailure("stored_restore_point_mismatch"); + } + } catch (error) { + await abandonGitRestorePoint(input.db, ledger!.id).catch(() => undefined); + throw error; + } + const { manifest } = captured; + return { + ledgerId: ledger!.id, + manifest, + ref: { + schema: GIT_RESTORE_POINT_SCHEMA, + id, + objectKey, + sha256: sealed.sha256, + byteSize: sealed.ciphertext.length, + key: sealed.key, + generation: input.generation, + head: manifest.head, + index: manifest.index, + worktree: manifest.worktree, + modes: manifest.modes, + headOnRemote: input.headOnRemote, + appliedAt: null, + }, + }; +} + +export async function activateGitRestorePoint(tx: Tx, input: { ledgerId: string; workspaceId: string; at: Date }) { + await notifyGitRestorePointWork(tx, `workspace:${input.workspaceId}`); + await tx.update(executionWorkspaceRestorePoints) + .set({ status: "superseded", updatedAt: input.at }) + .where(and(eq(executionWorkspaceRestorePoints.executionWorkspaceId, input.workspaceId), + ne(executionWorkspaceRestorePoints.id, input.ledgerId), + inArray(executionWorkspaceRestorePoints.status, ["pending", "active"]))); + const activated = await tx.update(executionWorkspaceRestorePoints) + .set({ status: "active", activatedAt: input.at, deleteAfter: input.at, updatedAt: input.at }) + .where(and(eq(executionWorkspaceRestorePoints.id, input.ledgerId), eq(executionWorkspaceRestorePoints.status, "pending"))) + .returning({ id: executionWorkspaceRestorePoints.id }); + if (activated.length !== 1) throw new RestorePointRestoreFailure("restore_point_not_pending"); +} + +export function restorePointSnapshotMatches(ref: GitRestorePointRef, snapshot: { head: string; index: string | null; worktree: string | null; modes: Array<{ path: string; mode: number }> }) { + return snapshot.head === ref.head && snapshot.index === ref.index && snapshot.worktree === ref.worktree + && JSON.stringify(snapshot.modes) === JSON.stringify(ref.modes); +} + +export async function abandonGitRestorePointById(db: Db, workspaceId: string, objectKey: string) { + await db.update(executionWorkspaceRestorePoints) + .set({ status: "superseded", updatedAt: new Date() }) + .where(and(eq(executionWorkspaceRestorePoints.executionWorkspaceId, workspaceId), eq(executionWorkspaceRestorePoints.objectKey, objectKey), + inArray(executionWorkspaceRestorePoints.status, ["pending", "active"]))); +} + +export async function abandonGitRestorePoint(db: Db, ledgerId: string) { + await db.update(executionWorkspaceRestorePoints) + .set({ status: "superseded", updatedAt: new Date() }) + .where(and(eq(executionWorkspaceRestorePoints.id, ledgerId), inArray(executionWorkspaceRestorePoints.status, ["pending", "active"]))); +} + +async function readObject(storage: StorageService, companyId: string, objectKey: string, signal?: AbortSignal) { + const { stream } = await storage.getObject(companyId, objectKey); + const chunks: Buffer[] = []; + let total = 0; + for await (const chunk of stream) { + signal?.throwIfAborted(); + const buffer = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk); + total += buffer.length; + if (total > MAX_RESTORE_POINT_BYTES) throw new RestorePointRestoreFailure("restore_point_too_large"); + chunks.push(buffer); + } + return Buffer.concat(chunks); +} + +export async function loadGitRestorePoint(input: { + storage: StorageService; + workspace: { companyId: string; id: string }; + ref: GitRestorePointRef; + signal?: AbortSignal; +}): Promise<{ payload: Buffer; manifest: RestorePointManifest }> { + if (input.ref.expiredAt) throw new RestorePointRestoreFailure("restore_point_expired"); + const ciphertext = await readObject(input.storage, input.workspace.companyId, input.ref.objectKey, input.signal) + .catch((error) => { + if (error instanceof RestorePointRestoreFailure) throw error; + throw new RestorePointRestoreFailure("restore_point_unavailable"); + }); + if (ciphertext.length !== input.ref.byteSize) throw new RestorePointRestoreFailure("ciphertext_size_mismatch"); + const payload = await openRestorePoint({ + ciphertext, + sha256: input.ref.sha256, + key: input.ref.key, + identity: { companyId: input.workspace.companyId, workspaceId: input.workspace.id, restorePointId: input.ref.id, generation: input.ref.generation }, + }); + const { manifest } = decodeRestorePointPayload(payload); + if (manifest.head !== input.ref.head || manifest.index !== input.ref.index || manifest.worktree !== input.ref.worktree + || JSON.stringify(manifest.modes) !== JSON.stringify(input.ref.modes)) { + throw new RestorePointRestoreFailure("restore_point_manifest_mismatch"); + } + return { payload, manifest }; +} + +export async function ensureGitRestorePointCommit(input: { + storage: StorageService; + workspace: { companyId: string; id: string }; + ref: GitRestorePointRef; + repositoryCwd: string; + signal?: AbortSignal; +}) { + if (input.ref.headOnRemote) return; + const present = await importNeeded(input.repositoryCwd, input.ref.head); + if (!present) return; + const { payload } = await loadGitRestorePoint(input); + await importRestorePoint({ repositoryCwd: input.repositoryCwd, payload, signal: input.signal }); +} + +async function importNeeded(repositoryCwd: string, commit: string) { + return !await gitCacheCommand(repositoryCwd, ["cat-file", "-e", `${commit}^{commit}`]).then(() => true, () => false); +} + +export async function applyGitRestorePointToWorktree(input: { + storage: StorageService; + workspace: { companyId: string; id: string }; + ref: GitRestorePointRef; + repositoryCwd: string; + worktreePath: string; + signal?: AbortSignal; +}) { + const current = await currentRestorePointSnapshot(input.worktreePath, input.signal); + const matches = restorePointSnapshotMatches(input.ref, current); + if (!matches && (current.head !== input.ref.head || current.index !== null || current.worktree !== null || current.modes.length > 0)) { + throw new RestorePointRestoreFailure("worktree_diverged_from_restore_point"); + } + const { payload } = await loadGitRestorePoint(input); + const manifest = await importRestorePoint({ repositoryCwd: input.repositoryCwd, payload, signal: input.signal }); + if (!matches) await applyRestorePoint({ cwd: input.worktreePath, manifest, signal: input.signal }); +} + +export async function markGitRestorePointExpired(db: Db, restorePoint: { companyId: string; executionWorkspaceId: string; objectKey: string }) { + await db.update(executionWorkspaces).set({ + metadata: sql`jsonb_set(${executionWorkspaces.metadata}, '{gitCacheRestoration,restorePoint,expiredAt}', ${JSON.stringify(new Date().toISOString())}::jsonb)`, + updatedAt: new Date(), + }).where(and(eq(executionWorkspaces.id, restorePoint.executionWorkspaceId), eq(executionWorkspaces.companyId, restorePoint.companyId), + sql`${executionWorkspaces.metadata}->'gitCacheRestoration'->'restorePoint'->>'objectKey' = ${restorePoint.objectKey}`)); +} diff --git a/server/src/services/git-workspace-expiration.ts b/server/src/services/git-workspace-expiration.ts index 397b27b67e9..e55aab0988f 100644 --- a/server/src/services/git-workspace-expiration.ts +++ b/server/src/services/git-workspace-expiration.ts @@ -7,11 +7,14 @@ import { and, eq, ne, sql } from "drizzle-orm"; import { executionWorkspaces, issues, heartbeatRuns, nativeRunFinalizations, environmentLeases, workspaceRuntimeServices, issueRecoveryActions, projectWorkspaces, withDedicatedDbConnection, type Db } from "@paperclipai/db"; import { createGitRemoteAuthProvider } from "./git-credentials.js"; +import type { StorageService } from "../storage/types.js"; +import { currentRestorePointSnapshot, RestorePointSkip, type RestorePointLimits } from "./git-restore-point.js"; +import { abandonGitRestorePoint, abandonGitRestorePointById, activateGitRestorePoint, restorePointSnapshotMatches, storeGitRestorePoint } from "./git-restore-point-storage.js"; import { withIsolatedRepositorySourceLock, verifiedRecordedIsolatedRemoteSource } from "./isolated-remote-workspace-source.js"; import { managedProjectCheckoutName } from "./managed-project-checkout-name.js"; import type { GitRemoteAuthProvider } from "./git-credentials.js"; -import { assertReproducibleGitFiles, assertReproducibleBaseGitStorage, assertReproducibleManagedCheckout, inspectGitCacheRecipe, readGitCacheRestoration, - verifyRemoteGitCommit, gitCacheExpirationLockLocations, recoverGitCacheExpirationLocks, GitCacheRestorationFailure, type ManagedCheckoutProver, gitCacheCommand, readGitCacheExpiration, finishGitCacheTrash, type GitCacheExpiration, GIT_CACHE_RESTORATION_KEY, GIT_CACHE_EXPIRATION_REASON } from "./git-workspace-restoration.js"; +import { assertReproducibleGitFiles, assertReproducibleBaseGitStorage, assertReproducibleManagedCheckout, inspectGitCacheRecipe, inspectGitCacheRecipeForRestorePoint, readGitCacheRestoration, GIT_CACHE_RESTORATION_SCHEMA_V2, + verifyRemoteGitCommit, gitCacheExpirationLockLocations, recoverGitCacheExpirationLocks, GitCacheRestorationFailure, type ManagedCheckoutProver, gitCacheCommand, readGitCacheExpiration, finishGitCacheTrash, type GitCacheExpiration, type GitCacheRestoration, GIT_CACHE_RESTORATION_KEY, GIT_CACHE_EXPIRATION_REASON } from "./git-workspace-restoration.js"; async function allocatedBytes(target: string, shouldContinue: () => boolean): Promise { if (!shouldContinue()) throw new Error("scan_stopped"); @@ -41,7 +44,29 @@ type Workspace = typeof executionWorkspaces.$inferSelect; type Tx = Parameters[0]>[0]; type Expiration = GitCacheExpiration; export type GitExpirationResult = { candidate: boolean; expired: boolean; reclaimedBytes: number; - candidateBytes: number; baseExpired: boolean; skipReason: string | null }; + candidateBytes: number; baseExpired: boolean; skipReason: string | null; restorePoint: boolean }; + +export type GitRestorePointExpirationOptions = { + storage: StorageService; + notBefore: Date; + keyDurable: boolean; + limits?: Partial; +}; + +const RESTORE_POINT_RELAXABLE = new Set(["dirty_or_untracked_files", "unique_ignored_files", "unique_original_head", + "unique_reflog_commits", "upstream_commit_unverified"]); + +async function restorePointBoundary(recipe: GitCacheRestoration, headOnRemote: boolean, auth: GitRemoteAuthProvider, signal?: AbortSignal) { + if (headOnRemote) return [recipe.commit]; + const upstream = await gitCacheCommand(recipe.worktreePath, ["rev-parse", "--verify", "-q", "refs/remotes/origin/HEAD"], signal) + .catch(() => ""); + const tip = upstream.trim(); + if (!tip) return []; + const base = (await gitCacheCommand(recipe.worktreePath, ["merge-base", "HEAD", tip], signal).catch(() => "")).trim(); + if (!/^[a-f0-9]{40}$/.test(base)) return []; + try { await verifyRemoteGitCommit(recipe.repoUrl, base, auth, signal); return [base]; } + catch (error) { if (signal?.aborted) throw error; return []; } +} /** Re-read the whole tree and every runtime/recovery consumer immediately before deletion. */ export async function isGitWorkspaceExpirationEligible(tx: Tx, workspace: Workspace, now: Date) { @@ -156,9 +181,11 @@ function managedCheckoutProver(db: Db, workspace: Workspace, auth: GitRemoteAuth export async function expireReconstructableGitWorkspace(db: Db, snapshot: Workspace, options: { dryRun: boolean; expireBase: boolean; now: Date; signal?: AbortSignal; beforeDelete?: (workspace: Workspace) => Promise; + restorePoint?: GitRestorePointExpirationOptions; }): Promise { const result: GitExpirationResult = { candidate: false, expired: false, reclaimedBytes: 0, - candidateBytes: 0, baseExpired: false, skipReason: null }; + candidateBytes: 0, baseExpired: false, skipReason: null, restorePoint: false }; + let pendingRestorePoint: string | null = null; if (!snapshot.sourceIssueId) return { ...result, skipReason: "missing_source_issue" }; try { return await withIsolatedRepositorySourceLock(db, snapshot.companyId, snapshot.sourceIssueId, () => @@ -176,8 +203,24 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Worksp const checkouts = new Map(); const identity = { companyId: workspace.companyId, workspaceId: workspace.id }; const proveManagedCheckout = managedCheckoutProver(db, workspace, auth, checkouts); - let recipe = state ? readGitCacheRestoration(workspace)! : await inspectGitCacheRecipe(workspace, - auth, options.signal, { proveManagedCheckout }); + let headOnRemote = true; + let needsRestorePoint = false; + let recipe: GitCacheRestoration; + if (state) recipe = readGitCacheRestoration(workspace)!; + else { + try { recipe = await inspectGitCacheRecipe(workspace, auth, options.signal, { proveManagedCheckout }); } + catch (error) { + if (!options.restorePoint || !RESTORE_POINT_RELAXABLE.has(skipReason(error))) throw error; + if (options.now.getTime() < options.restorePoint.notBefore.getTime()) throw new Error("restore_point_not_due"); + if (!options.restorePoint.keyDurable) throw new Error("restore_point_key_not_durable"); + checkouts.clear(); + const inspected = await inspectGitCacheRecipeForRestorePoint(workspace, auth, options.signal, { proveManagedCheckout }); + recipe = inspected.recipe; + headOnRemote = inspected.headOnRemote; + needsRestorePoint = true; + result.restorePoint = true; + } + } if (!state) recipe.contents = await captureGitWorkspaceContents(recipe.worktreePath, identity, [...checkouts.values()]); const present = await exists(recipe.worktreePath); if (!state && !present) throw new Error("missing_unverified_worktree"); @@ -199,16 +242,34 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Worksp if (await fs.realpath(worktreeAdmin) !== canonicalAdmin || await fs.realpath(rawAdmin) !== canonicalAdmin) { throw new Error("unproven_worktree_admin"); } + let activateLedger: string | null = null; + if (needsRestorePoint) { + const stored = await storeGitRestorePoint({ db, storage: options.restorePoint!.storage, + workspace: { companyId: workspace.companyId, id: workspace.id, sourceIssueId: workspace.sourceIssueId }, + cwd: recipe.worktreePath, generation, headOnRemote, limits: options.restorePoint!.limits, signal: options.signal, + upstreamCommits: await restorePointBoundary(recipe, headOnRemote, auth, options.signal) }); + if (stored) { + pendingRestorePoint = stored.ledgerId; + activateLedger = stored.ledgerId; + recipe = { ...recipe, schema: GIT_CACHE_RESTORATION_SCHEMA_V2, restorePoint: stored.ref }; + } else if (!headOnRemote) throw new Error("upstream_commit_unverified"); + } state = { schema: "paperclip.git-cache-expiration.v1", generation, workspaceRemoved: false, worktreeTrash: path.join(path.dirname(recipe.worktreePath), `.expired-git-worktree-${workspace.id}-${generation}`), worktreeAdmin, baseTrash: null, completed: false, lockToken: randomBytes(32).toString("hex") }; if (await exists(state.worktreeTrash)) throw new Error("unknown_worktree_trash"); // Commit the immutable restoration recipe BEFORE touching disk. The separate // lifecycle-lock connection remains held through deletion and final state. - [workspace] = await db.update(executionWorkspaces).set({ status: "archived", closedAt: options.now, + const archive = { status: "archived", closedAt: options.now, cleanupEligibleAt: options.now, cleanupReason: GIT_CACHE_EXPIRATION_REASON, updatedAt: options.now, metadata: { ...workspace.metadata, lifecycleGeneration: generation, [GIT_CACHE_RESTORATION_KEY]: recipe, [EXPIRATION_KEY]: state }, - }).where(eq(executionWorkspaces.id, workspace.id)).returning(); + }; + [workspace] = await db.transaction(async (archiveTx) => { + const rows = await archiveTx.update(executionWorkspaces).set(archive).where(eq(executionWorkspaces.id, workspace!.id)).returning(); + if (activateLedger) await activateGitRestorePoint(archiveTx, { ledgerId: activateLedger, workspaceId: workspace!.id, at: options.now }); + return rows; + }); + pendingRestorePoint = null; } if (!workspace) throw new Error("workspace_changed"); const persist = async () => { @@ -229,7 +290,10 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Worksp try { checkouts.clear(); await assertReproducibleGitFiles(recipe.worktreePath, options.signal, { proveManagedCheckout, preserveSettings: true, + restorePoint: Boolean(recipe.restorePoint), verifyFetchedCommit: (commit, signal) => verifyRemoteGitCommit(recipe.repoUrl, commit, auth, signal) }); + if (recipe.restorePoint && !restorePointSnapshotMatches(recipe.restorePoint, + await currentRestorePointSnapshot(recipe.worktreePath, options.signal))) throw new Error("restore_point_contents_changed"); for (const checkout of checkouts.values()) { const cwd = path.join(recipe.worktreePath, ".paperclip-repositories", checkout.name); locks.push(await acquireGitExpirationLocks(lockLocations.filter(location => location.checkout === checkout.name), state.lockToken)); @@ -241,11 +305,13 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Worksp if ((await gitCacheCommand(recipe.worktreePath, ["rev-parse", "HEAD"], options.signal)).trim() !== recipe.commit) throw new Error("head_changed"); } catch (error) { for (const lock of [...locks].reverse()) await lock.release(); + const { restorePoint: _abandoned, ...reproducible } = recipe; const metadata: Record = { ...workspace.metadata, lifecycleGeneration: state.generation + 1, - [GIT_CACHE_RESTORATION_KEY]: { ...recipe, restoredAt: new Date().toISOString() } }; + [GIT_CACHE_RESTORATION_KEY]: { ...reproducible, schema: "paperclip.git-cache-restoration.v1", restoredAt: new Date().toISOString() } }; delete metadata[EXPIRATION_KEY]; await db.update(executionWorkspaces).set({ status: "active", closedAt: null, cleanupEligibleAt: null, cleanupReason: null, metadata, updatedAt: new Date() }).where(eq(executionWorkspaces.id, workspace.id)); + if (recipe.restorePoint) await abandonGitRestorePointById(db, workspace.id, recipe.restorePoint.objectKey); throw error; } await fs.rename(recipe.worktreePath, state.worktreeTrash); @@ -284,8 +350,9 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Worksp return result; })), true); } catch (error) { + if (pendingRestorePoint) await abandonGitRestorePoint(db, pendingRestorePoint).catch(() => undefined); options.signal?.throwIfAborted(); - result.skipReason = skipReason(error); + result.skipReason = error instanceof RestorePointSkip ? error.reason : skipReason(error); return result; } } diff --git a/server/src/services/git-workspace-restoration.ts b/server/src/services/git-workspace-restoration.ts index c05a14bbfc1..c5c3c2774f1 100644 --- a/server/src/services/git-workspace-restoration.ts +++ b/server/src/services/git-workspace-restoration.ts @@ -13,14 +13,18 @@ import type { GitRemoteAuthProvider } from "./git-credentials.js"; import { isRuntimeOwnedGitBranch } from "./execution-workspace-branch-ownership.js"; import { recordedIsolatedRemoteSource, verifiedRecordedIsolatedRemoteSource, materializeIsolatedRemoteWorkspaceSource } from "./isolated-remote-workspace-source.js"; import { findRegenerableWorkspaceDirs } from "./workspace-compaction.js"; +import { ensureGitRestorePointCommit, validateGitRestorePointRef, type GitRestorePointRef } from "./git-restore-point-storage.js"; +import { getStorageService } from "../storage/index.js"; +import type { StorageService } from "../storage/types.js"; import { PROJECT_REPOSITORIES_DIR } from "@paperclipai/adapter-utils/git-workspace-sync"; const execFile = promisify(execFileCallback); const shaPattern = /^[a-f0-9]{40}$/; export const GIT_CACHE_RESTORATION_KEY = "gitCacheRestoration"; export const GIT_CACHE_EXPIRATION_REASON = "reproducible_git_cache_expired"; +export const GIT_CACHE_RESTORATION_SCHEMA_V2 = "paperclip.git-cache-restoration.v2"; export type GitCacheRestoration = { - schema: "paperclip.git-cache-restoration.v1"; + schema: "paperclip.git-cache-restoration.v1" | typeof GIT_CACHE_RESTORATION_SCHEMA_V2; companyId: string; workspaceId: string; sourceIssueId: string; @@ -36,6 +40,7 @@ export type GitCacheRestoration = { restoredAt?: string; contentsRestoredAt?: string | null; contents?: GitWorkspaceContents; + restorePoint?: GitRestorePointRef; }; export type GitCacheExpiration = { schema: "paperclip.git-cache-expiration.v1"; generation: number; workspaceRemoved: boolean; @@ -80,7 +85,7 @@ export function readGitCacheRestoration(workspace: Workspace): GitCacheRestorati const value = workspace.metadata?.[GIT_CACHE_RESTORATION_KEY]; if (value === undefined) return null; const recipe = value as GitCacheRestoration | null; - if (!recipe || recipe.schema !== "paperclip.git-cache-restoration.v1" + if (!recipe || !restorationSchemaMatches(recipe) || !isRuntimeOwnedGitBranch(workspace.metadata) || recipe.gitBranchOwnershipVersion !== 1 || recipe.companyId !== workspace.companyId || recipe.workspaceId !== workspace.id @@ -96,9 +101,18 @@ export function readGitCacheRestoration(workspace: Workspace): GitCacheRestorati throw new GitCacheRestorationFailure(); } validateGitWorkspaceContents(recipe.contents); + if (recipe.restorePoint) { + try { validateGitRestorePointRef(recipe.restorePoint, workspace); } catch { throw new GitCacheRestorationFailure(); } + } return recipe; } +function restorationSchemaMatches(recipe: GitCacheRestoration) { + return recipe.schema === "paperclip.git-cache-restoration.v1" + ? recipe.restorePoint === undefined + : recipe.schema === GIT_CACHE_RESTORATION_SCHEMA_V2 && recipe.restorePoint !== undefined; +} + export async function gitCacheCommand(cwd: string, args: string[], signal?: AbortSignal) { const result = await execFile("git", ["-c", "core.fsmonitor=false", "-C", cwd, ...args], { timeout: 60_000, signal, maxBuffer: 16 * 1024 * 1024, @@ -293,14 +307,16 @@ export async function assertReproducibleGitFiles(cwd: string, signal?: AbortSign proveManagedCheckout?: ManagedCheckoutProver; preserveSettings?: boolean; verifyFetchedCommit?: (commit: string, signal?: AbortSignal) => Promise; + restorePoint?: boolean; } = {}) { - if ((await gitCacheCommand(cwd, ["status", "--porcelain=v1", "--untracked-files=all", "-z"], signal)).length) throw new Error("dirty_or_untracked_files"); + if (!options.restorePoint && (await gitCacheCommand(cwd, ["status", "--porcelain=v1", "--untracked-files=all", "-z"], signal)).length) throw new Error("dirty_or_untracked_files"); if ((await gitCacheCommand(cwd, ["ls-files", "-v"], signal)).split("\n").some(line => /^[a-zS] /.test(line))) throw new Error("unsupported_index_flags"); const localConfig = await gitCacheCommand(cwd, ["config", "--local", "--name-only", "--list"], signal); if (localConfig.split("\n").filter(Boolean).some(key => !/^(core\.(repositoryformatversion|filemode|bare|logallrefupdates|ignorecase|precomposeunicode)|remote\.origin\.(url|fetch)|branch\.[^.]+\.(remote|merge))$/.test(key))) throw new Error("unsupported_local_git_config"); const regenerable = await findRegenerableWorkspaceDirs(cwd, { signal }); const ignored = (await gitCacheCommand(cwd, ["ls-files", "--others", "--ignored", "--exclude-standard", "-z"], signal)).split("\0").filter(Boolean); - const unexplained = ignored.filter(file => !(options.preserveSettings && file === PAPERCLIP_CLAUDE_SETTINGS_FILE) && !regenerable.some(dir => file.startsWith(`${dir}/`))); + const unexplained = ignored.filter(file => !(options.preserveSettings && file === PAPERCLIP_CLAUDE_SETTINGS_FILE) && !regenerable.some(dir => file.startsWith(`${dir}/`)) + && !(options.restorePoint && /^\.env(\..*)?$/.test(path.posix.basename(file)))); const checkouts = new Set(); for (const file of unexplained) { const [namespace, name, rest] = file.split("/", 3); @@ -323,7 +339,7 @@ export async function assertReproducibleGitFiles(cwd: string, signal?: AbortSign if (await fs.lstat(path.join(root, name)).then(() => true, (e: NodeJS.ErrnoException) => { if(e.code === "ENOENT") return false; throw e; })) throw new Error("unsupported_local_git_state"); } } - const original = await gitCacheCommand(cwd, ["rev-parse", "--verify", "ORIG_HEAD"], signal).catch(() => null); + const original = options.restorePoint ? null : await gitCacheCommand(cwd, ["rev-parse", "--verify", "ORIG_HEAD"], signal).catch(() => null); if (original) await gitCacheCommand(cwd, ["merge-base", "--is-ancestor", original.trim(), "HEAD"], signal) .catch(() => { throw new Error("unique_original_head"); }); if (gitDir !== common) { @@ -343,7 +359,12 @@ export async function assertReproducibleGitFiles(cwd: string, signal?: AbortSign const message = await fs.readFile(path.join(gitDir, "COMMIT_EDITMSG"), "utf8").catch((error: NodeJS.ErrnoException) => { if (error.code === "ENOENT") return null; throw error; }); - if (message !== null && message.trim() !== (await gitCacheCommand(cwd, ["log", "-1", "--format=%B"], signal)).trim()) throw new Error("uncommitted_git_message"); + if (message !== null && message.trim() !== (await gitCacheCommand(cwd, ["log", "-1", "--format=%B"], signal)).trim()) { + const saved = options.restorePoint + ? (await gitCacheCommand(cwd, ["log", "-g", "--format=%B%x00", "HEAD"], signal).catch(() => "")).split("\0").map(entry => entry.trim()) + : []; + if (!saved.includes(message.trim())) throw new Error("uncommitted_git_message"); + } const logs = await fs.readdir(path.join(gitDir, "logs")).catch((error: NodeJS.ErrnoException) => { if (error.code === "ENOENT") return []; throw error; }); @@ -352,7 +373,7 @@ export async function assertReproducibleGitFiles(cwd: string, signal?: AbortSign // Packed stashes/replacement refs are just as important as loose refs. if ((await gitCacheCommand(cwd, ["for-each-ref", "--format=%(refname)", "refs/stash", "refs/replace"], signal)).trim()) throw new Error("unique_git_refs"); // Rewrites can leave commits reachable only from this worktree's reflog. - for (const commit of new Set((await gitCacheCommand(cwd, ["reflog", "show", "--format=%H", "HEAD"], signal)).split("\n").filter(Boolean))) { + for (const commit of options.restorePoint ? [] : new Set((await gitCacheCommand(cwd, ["reflog", "show", "--format=%H", "HEAD"], signal)).split("\n").filter(Boolean))) { await gitCacheCommand(cwd, ["merge-base", "--is-ancestor", commit, "HEAD"], signal).catch(() => { throw new Error("unique_reflog_commits"); }); } return regenerable; @@ -360,6 +381,16 @@ export async function assertReproducibleGitFiles(cwd: string, signal?: AbortSign export async function inspectGitCacheRecipe(workspace: Workspace, resolveGitAuth: GitRemoteAuthProvider, signal?: AbortSignal, options: { proveManagedCheckout?: ManagedCheckoutProver } = {}): Promise { + return (await inspectRecipe(workspace, resolveGitAuth, signal, options, false)).recipe; +} + +export async function inspectGitCacheRecipeForRestorePoint(workspace: Workspace, resolveGitAuth: GitRemoteAuthProvider, signal?: AbortSignal, + options: { proveManagedCheckout?: ManagedCheckoutProver } = {}) { + return inspectRecipe(workspace, resolveGitAuth, signal, options, true); +} + +async function inspectRecipe(workspace: Workspace, resolveGitAuth: GitRemoteAuthProvider, signal: AbortSignal | undefined, + options: { proveManagedCheckout?: ManagedCheckoutProver }, restorePoint: boolean): Promise<{ recipe: GitCacheRestoration; headOnRemote: boolean }> { if (workspace.mode !== "isolated_workspace" || workspace.strategyType !== "git_worktree" || !isRuntimeOwnedGitBranch(workspace.metadata) || !workspace.sourceIssueId || !workspace.repoUrl || !workspace.cwd || !workspace.branchName || (workspace.providerRef && workspace.providerRef !== workspace.cwd)) throw new Error("unproven_workspace_ownership"); @@ -378,14 +409,19 @@ export async function inspectGitCacheRecipe(workspace: Workspace, resolveGitAuth if (!(await fs.lstat(path.join(workspace.cwd, ".git"))).isFile()) throw new Error("not_linked_worktree"); if ((await gitCacheCommand(workspace.cwd, ["rev-parse", "--path-format=absolute", "--git-common-dir"], signal)).trim() !== path.join(await fs.realpath(source), ".git")) throw new Error("source_identity_changed"); if ((await gitCacheCommand(workspace.cwd, ["symbolic-ref", "--short", "HEAD"], signal)).trim() !== workspace.branchName) throw new Error("branch_identity_changed"); - await assertReproducibleGitFiles(workspace.cwd, signal, { ...options, preserveSettings: true, + await assertReproducibleGitFiles(workspace.cwd, signal, { ...options, preserveSettings: true, restorePoint, verifyFetchedCommit: (commit, verificationSignal) => verifyRemoteGitCommit(workspace.repoUrl!, commit, resolveGitAuth, verificationSignal) }); const commit = (await gitCacheCommand(workspace.cwd, ["rev-parse", "HEAD"], signal)).trim(); - await verifyRemoteGitCommit(workspace.repoUrl, commit, resolveGitAuth, signal); - return { schema: "paperclip.git-cache-restoration.v1", companyId: workspace.companyId, workspaceId: workspace.id, + let headOnRemote = true; + try { await verifyRemoteGitCommit(workspace.repoUrl, commit, resolveGitAuth, signal); } + catch (error) { + if (!restorePoint || signal?.aborted) throw error; + headOnRemote = false; + } + return { headOnRemote, recipe: { schema: "paperclip.git-cache-restoration.v1", companyId: workspace.companyId, workspaceId: workspace.id, sourceIssueId: workspace.sourceIssueId, repoUrl: workspace.repoUrl, commit, branchName: workspace.branchName, sourceKind: recordedIsolatedRemoteSource(workspace) ? "isolated_remote" : "managed_project", projectId: workspace.projectId, - sourcePath: source, worktreePath: workspace.cwd, gitBranchOwnershipVersion: 1, verifiedAt: new Date().toISOString() }; + sourcePath: source, worktreePath: workspace.cwd, gitBranchOwnershipVersion: 1, verifiedAt: new Date().toISOString() } }; } export function readGitCacheExpiration(workspace: Workspace): GitCacheExpiration | null { @@ -466,7 +502,8 @@ export async function finishGitCacheTrash(workspace: Workspace) { } /** Called only after authorization and while holding the source/lifecycle fences. */ -export async function restoreGitCacheBase(workspace: Workspace, recipe: GitCacheRestoration, resolveGitAuth: GitRemoteAuthProvider) { +export async function restoreGitCacheBase(workspace: Workspace, recipe: GitCacheRestoration, resolveGitAuth: GitRemoteAuthProvider, + storage?: StorageService) { if (readGitCacheRestoration(workspace)?.commit !== recipe.commit) throw new GitCacheRestorationFailure(); try { await finishGitCacheTrash(workspace); @@ -491,6 +528,10 @@ export async function restoreGitCacheBase(workspace: Workspace, recipe: GitCache : (await materializeManagedProjectWorkspace(recipe.sourcePath, { repoUrl: recipe.repoUrl, resolveGitAuth })).cwd; await verifiedGitCacheSource(workspace, recipe.sourcePath); if (!cwd) throw new GitCacheRestorationFailure(); + if (recipe.restorePoint) { + await ensureGitRestorePointCommit({ storage: storage ?? getStorageService(), workspace: { companyId: workspace.companyId, id: workspace.id }, + ref: recipe.restorePoint, repositoryCwd: cwd }); + } await ensureSavedGitCommit(cwd, recipe.repoUrl, recipe.commit, resolveGitAuth); return cwd; } catch { throw new GitCacheRestorationFailure(); } @@ -514,7 +555,7 @@ export function savedGitWorkspaceRestoration(input: { metadata?: Record Promise<() => Promise>; persistContentsRestoredAt?: (at: string | null) => Promise; + persistRestorePointAppliedAt?: (at: string) => Promise; + restorePointStorage?: StorageService; db?: Db | null; base: ExecutionWorkspaceInput; workspace: { @@ -3880,6 +3885,27 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: { currentRecipe = { ...currentRecipe as Record, contentsRestoredAt: at }; input.workspace.metadata!.gitCacheRestoration = currentRecipe; }; + const applyRestorePoint = async (repositoryCwd: string, worktreePath: string) => { + const ref = restoration?.restorePoint; + if (!ref || ref.appliedAt || !input.workspace.id) return; + await applyGitRestorePointToWorktree({ storage: input.restorePointStorage ?? getStorageService(), + workspace: { companyId: input.agent.companyId, id: input.workspace.id }, ref, repositoryCwd, worktreePath }) + .catch(() => { throw new GitCacheRestorationFailure(); }); + const at = new Date().toISOString(); + if (input.persistRestorePointAppliedAt) await input.persistRestorePointAppliedAt(at); + else { + if (!input.db) throw new GitCacheRestorationFailure(); + const updated = await input.db.update(executionWorkspaces).set({ + metadata: sql`jsonb_set(${executionWorkspaces.metadata}, '{gitCacheRestoration,restorePoint,appliedAt}', ${JSON.stringify(at)}::jsonb)`, + }).where(and(eq(executionWorkspaces.id, input.workspace.id), eq(executionWorkspaces.companyId, input.agent.companyId), + sql`${executionWorkspaces.metadata}->'gitCacheRestoration'->'restorePoint'->>'id' = ${ref.id}`)) + .returning({ id: executionWorkspaces.id }); + if (updated.length !== 1) throw new GitCacheRestorationFailure(); + } + ref.appliedAt = at; + currentRecipe = { ...currentRecipe as Record, restorePoint: { ...ref } }; + input.workspace.metadata!.gitCacheRestoration = currentRecipe; + }; const restoreContents = async () => { if (!restoration?.contents || !input.workspace.id) return; await restoreGitWorkspaceContents({ cwd, companyId: input.agent.companyId, workspaceId: input.workspace.id, @@ -3980,6 +4006,7 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: { realized.warnings = [...repairWarnings, ...baseRefreshWarnings, ...baseDrift.warnings]; realized.baseRefSha = refresh.baseRefSha ?? recordedBaseRefSha ?? baseDrift.branchBaseRefSha ?? baseDrift.currentBaseRefSha; await input.afterSourceCopy?.(repoRoot, realized.worktreePath ?? cwd); + await applyRestorePoint(repoRoot, realized.worktreePath ?? cwd); if (restoration && !restoration.contentsRestored) await restoreContents(); await provisionExecutionWorktree({ skipHostProvision: input.skipHostProvision, @@ -4009,6 +4036,11 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: { await fs.mkdir(path.dirname(worktreePath), { recursive: true }); await runGit(["worktree", "prune"], repoRoot).catch(() => {}); if (savedCommit) { + if (restoration?.restorePoint && input.workspace.id) { + await ensureGitRestorePointCommit({ storage: input.restorePointStorage ?? getStorageService(), + workspace: { companyId: input.agent.companyId, id: input.workspace.id }, ref: restoration.restorePoint, repositoryCwd: repoRoot }) + .catch(() => { throw new GitCacheRestorationFailure(); }); + } await ensureSavedGitCommit(repoRoot, input.workspace.repoUrl ?? input.base.repoUrl!, savedCommit, input.resolveGitAuth ?? (async () => null)); const existingTip = await resolveBaseRefSha(repoRoot, `refs/heads/${branchName}`); // Once restored, subsequent task commits own this branch. The saved commit @@ -4083,6 +4115,7 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: { }); await input.afterSourceCopy?.(repoRoot, worktreePath); + await applyRestorePoint(repoRoot, worktreePath); await restoreContents(); await provisionExecutionWorktree({ skipHostProvision: input.skipHostProvision,