Files
PaperClipAI/server/src/__tests__/git-restore-point-expiration.test.ts
T
Devin FoleyandPaperclip fe85fd7ac4 Expire closed workspaces that open descendants do not use (#17464)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Git cache expiration removes closed issues' worktrees and saves
restore points for unsaved work
> - It treated a workspace as closed only when the source issue and
every descendant were done or cancelled
> - A routine that closes a parent and leaves children open for days
therefore kept every parent worktree, although the children work in
their own workspaces
> - The idle path did not help, because the children are updated every
day
> - This pull request counts only the issues that use the workspace when
it decides that a closed workspace can expire
> - The benefit is that closed parent worktrees are removed while their
children work elsewhere, and a child that later binds to the workspace
gets it back from the restore point

## Linked Issues or Issue Description

Refs #17271. Bug report described in-PR (bug report template).

**What happened?**
On one instance, an hourly routine creates a parent issue, works it in
its own worktree, marks the parent done, and leaves child issues open
(`blocked`, `in_review` or `in_progress`) for days. The children work in
their own workspaces. They are not bound to the parent's workspace. Git
cache expiration kept every parent worktree (about 300 MB each), because
the closed-tree check and the workspace reaper required all descendants
to be closed. In one project, 18 of 19 long-closed parent worktrees were
kept only by open children that did not use them. The idle path never
ran, because the children were updated daily.

**Expected behavior**
A closed issue's worktree expires while descendants are open, as long as
no open descendant uses that workspace.

**Steps to reproduce**
1. Enable git cache expiration and restore points.
2. Close an issue that has an isolated git worktree, and leave a child
issue open that works in its own workspace (or has no workspace
binding).
3. Wait past the terminal delay and run the workspace reaper: the parent
worktree stays, and the reaper counts it as `skippedNonTerminalTree`.

**Paperclip version or commit:** `master` at 410e5c6eed.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).

## What Changed

- `isGitWorkspaceExpirationEligible` (closed path, not the idle path):
the "tree closed for one day" check now covers only the issues in the
tree that use the workspace. An issue uses the workspace when it is the
source issue, when its `execution_workspace_id` is the workspace, or
when its current `workspace_selection` or its
`workspace_pending_selection` selects it as the `existing` workspace.
Inheritance from a parent (`reuse_existing`) and `existing` selections
are both stored as `execution_workspace_id`, and heartbeat workspace
resolution reuses an existing workspace only through that binding or a
run's persisted native binding. An open descendant without one of these
references gets a new workspace on its next run, so it no longer keeps
the parent's worktree.
- All other checks stay the same: any open issue bound to the workspace,
live or unsettled runs of any issue in the tree or of the workspace
(including a run whose context names the workspace), finalizations,
leases, recovery actions, runtime services, shared cwd and the reopen
fence.
- Workspace reaper: when the source issue is closed, git cache
expiration applies, and no reopen is pending, the reaper routes the
workspace by the same rule. It takes the restore point due time from the
source issue and the descendants that use the workspace, also after
every descendant is closed. A recently closed child that used the
workspace still holds it for the terminal delay. The compaction delay
and the archive cooldown still use the whole tree when it is closed.
- Workspace reaper: compaction and the normal archive path keep their
behavior. While any descendant is open, the reaper does not compact the
worktree, and the expiration branch handles the workspace and continues.
The archive path still requires a closed subtree. With expiration off,
or a pending reopen, routing is unchanged.
- New exported helper `listIssuesUsingGitWorkspace` shares the SQL "uses
the workspace" predicate between eligibility and the reaper.
- Docs: `doc/recoverable-disk-cleanup.md` and the
`PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS` entry in
`doc/DEVELOPING.md`.
- The existing `preserves live descendants and pending reopening` case
now binds its open child to the workspace. An unbound open child no
longer keeps the worktree by design.

## Verification

- `pnpm exec vitest run
src/__tests__/git-restore-point-expiration.test.ts
src/__tests__/git-workspace-expiration.test.ts
src/__tests__/heartbeat-git-restore-point-resume.test.ts` in `server/`:
all pass.
- `pnpm exec vitest run src/__tests__/git-workspace-expiration
src/__tests__/git-restore-point src/__tests__/execution-workspace
src/__tests__/heartbeat-git-restore-point-resume
src/services/heartbeat/environment-selection
src/__tests__/isolated-repository` in `server/`: 15 of 17 files passed
in one parallel run. The other failures were the updated descendant case
(now fixed) and 45-second timeouts in restore point tests while 17
embedded Postgres files ran at the same time. Both files pass when run
on their own.
- `pnpm run typecheck` in `server/`: passes.
- New embedded Postgres cases with real git repositories:
- A closed parent with open `blocked`, `in_review`, `in_progress` and
`todo` children and a grandchild in their own workspaces (or with none):
the worktree expires with a restore point and the reopen restores it
exactly.
- The reaper archives it, timed from the issues that use the workspace.
A child closed one minute ago with its own workspace does not delay it.
- A recently closed child bound to the workspace holds it until its
terminal delay passes.
- When every child is closed and the most recent one used its own
workspace, the reaper does not restart the restore point delay.
- The worktree is kept when an open child is bound to it, when an open
grandchild under a child with its own workspace is bound to it, when a
child created later with `issueService.create` inherits it with
`reuse_existing`, when an open child selects it through
`workspace_selection` or `workspace_pending_selection`, when a child run
that references it is live, and when the parent was closed inside the
delay.
- The reaper never compacts the worktree while children are open, with
expiration on or off. A pending reopen keeps its current fence handling.
- After expiry, an open child that binds to the workspace reopens it
with the saved commits, index, working tree and untracked files.
- Heartbeat: a queued run for an open child that binds to the expired
workspace of a closed parent restores the workspace and runs the agent
in it with the saved state.
- Mutation checks (each run against the two restore point test files):
  - Closed-tree check restored to the whole tree: 7 tests fail.
  - `execution_workspace_id` removed from the rule: 3 fail.
  - `workspace_selection` removed from the rule: 1 fails.
  - `workspace_pending_selection` removed from the rule: 1 fails.
  - Reaper `latestTerminalAt` from the whole tree: 3 fail.
  - Reaper routing by the whole tree: 2 fail.
- Reaper restore point due time from the whole tree once it is closed: 1
fails.
  - Pending-reopen gate removed: 1 fails.
- Compaction gate (`!treeOpen`) removed: no test fails. Compaction's own
guard under the lifecycle lock still rejects open trees, so the gate
only skips work that cannot succeed.
- Not run: the full server suite and the UI suites. CI covers them.

## Risks

- Closed parent worktrees now expire while their children are open. A
child that later binds to the workspace depends on the restore point and
on the run-side reopen from #17271. The new tests cover this path. When
restore points are off, only exactly reproducible worktrees expire.
- An open child with no workspace binding no longer keeps the parent's
worktree. Heartbeat resolution never reuses a workspace without a
binding, so this does not change where the child runs.
- The rule is conservative. A workspace selected only by an issue's
current or pending selection is kept, and live runs of any issue in the
tree still block.
- The reaper runs one extra indexed query for each closed workspace when
git cache expiration applies. No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-24 12:43:57 -07:00

580 lines
37 KiB
TypeScript

import { promises as fs } from "node:fs";
import os from "node:os";
import path from "node:path";
import { execFileSync } from "node:child_process";
import { randomUUID } from "node:crypto";
import { pathToFileURL } from "node:url";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import { eq, sql } from "drizzle-orm";
import { agents, companies, createDb, executionWorkspaceRestorePoints, executionWorkspaces, heartbeatRuns, issueComments, issues } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { materializeIsolatedRemoteWorkspaceSource } from "../services/isolated-remote-workspace-source.js";
import { ensurePersistedExecutionWorkspaceAvailable, realizeExecutionWorkspace } from "../services/workspace-runtime.js";
import { executionWorkspaceService } from "../services/execution-workspaces.js";
import { expireReconstructableGitWorkspace, gitWorkspaceLastActivityAt } from "../services/git-workspace-expiration.js";
import { getStorageService } from "../storage/index.js";
import { markGitRestorePointExpired } from "../services/git-restore-point-storage.js";
import { GIT_RESTORE_POINT_EXPIRED_WARNING, setMetadataReopenPendingConsumption } from "../services/execution-workspaces.js";
import { issueService } from "../services/issues.js";
const gitEnv = { ...process.env, GIT_AUTHOR_NAME: "Fixture", GIT_AUTHOR_EMAIL: "fixture@example.test",
GIT_COMMITTER_NAME: "Fixture", GIT_COMMITTER_EMAIL: "fixture@example.test", GIT_TERMINAL_PROMPT: "0" };
const git = (cwd: string, ...args: string[]) => execFileSync("git", args, { cwd, env: gitEnv, encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }).trim();
const exists = (target: string) => fs.lstat(target).then(() => true, () => false);
const DAY_MS = 24 * 60 * 60 * 1000;
const support = await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = support.supported ? describe : describe.skip;
describeEmbeddedPostgres("git restore point expiration", () => {
let db: ReturnType<typeof createDb>;
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let root: string;
const previousHome = process.env.PAPERCLIP_HOME;
const previousInstance = process.env.PAPERCLIP_INSTANCE_ID;
beforeAll(async () => {
database = await startEmbeddedPostgresTestDatabase("git-restore-point-expiration-");
db = createDb(database.connectionString);
root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-restore-point-expiration-"));
process.env.PAPERCLIP_HOME = path.join(root, "home");
process.env.PAPERCLIP_INSTANCE_ID = "default";
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32));
});
afterEach(async () => {
await db.delete(executionWorkspaceRestorePoints);
await db.execute(sql`TRUNCATE TABLE companies CASCADE`);
});
afterAll(async () => {
vi.unstubAllEnvs();
if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome;
if (previousInstance === undefined) delete process.env.PAPERCLIP_INSTANCE_ID; else process.env.PAPERCLIP_INSTANCE_ID = previousInstance;
await db.$client.end({ timeout: 1 }); await database.cleanup(); await fs.rm(root, { recursive: true, force: true });
});
async function fixture() {
const companyId = randomUUID(), issueId = randomUUID(), workspaceId = randomUUID();
const location = path.join(root, companyId), source = path.join(location, "author"), remote = path.join(location, "remote.git");
await fs.mkdir(source, { recursive: true });
git(source, "init", "--initial-branch=main");
await fs.writeFile(path.join(source, "marker.txt"), "commit A\n");
await fs.writeFile(path.join(source, "remove-me.txt"), "tracked\n");
await fs.writeFile(path.join(source, ".gitignore"), "node_modules/\n.env\n.claude/settings.local.json\n*.tsbuildinfo\nevidence/\n");
git(source, "add", "."); git(source, "commit", "-m", "A");
git(location, "clone", "--bare", source, remote); git(source, "remote", "add", "origin", remote);
await db.insert(companies).values({ id: companyId, name: "Restore point fixture", issuePrefix: `RP${companyId.slice(0, 6)}` });
await db.insert(issues).values({ id: issueId, companyId, title: "Finished task", status: "done", completedAt: new Date(Date.now() - 4 * DAY_MS) });
const repoUrl = pathToFileURL(remote).href;
const baseCwd = await materializeIsolatedRemoteWorkspaceSource({ companyId, issueId, repoUrl, resolveGitAuth: async () => null });
const base = { baseCwd, source: "task_session" as const, projectId: null, workspaceId: null, repoUrl, repoRef: "origin/main" };
const workspace = await realizeExecutionWorkspace({ base, issue: { id: issueId, identifier: "RP-1", title: "Fixture" }, agent: { id: null, name: "Fixture", companyId },
config: { workspaceStrategy: { type: "git_worktree", branchTemplate: "fixture-task", worktreeParentDir: path.join(location, "worktrees"), baseRef: "origin/main" } }, skipHostProvision: true });
await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, sourceIssueId: issueId, mode: "isolated_workspace", strategyType: "git_worktree", name: "Fixture", status: "active",
providerType: "local_fs", cwd: workspace.cwd, providerRef: workspace.cwd, repoUrl, baseRef: "origin/main", branchName: workspace.branchName,
metadata: { createdByRuntime: workspace.branchCreatedByRuntime, gitBranchOwnershipVersion: 1, baseRefSha: workspace.baseRefSha,
isolatedRemoteSource: { schema: "paperclip.isolated-remote-source.v1", companyId, issueId, repoUrl } } });
await db.update(issues).set({ executionWorkspaceId: workspaceId }).where(eq(issues.id, issueId));
const row = () => db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId)).then(rows => rows[0]!);
const restorePoint = (overrides: Partial<{ notBefore: Date; keyDurable: boolean }> = {}) => ({
storage: getStorageService(), notBefore: new Date(Date.now() - DAY_MS), keyDurable: true, ...overrides,
});
const expire = async (options: Partial<Parameters<typeof expireReconstructableGitWorkspace>[2]> = {}) =>
expireReconstructableGitWorkspace(db, await row(), { dryRun: false, expireBase: false, now: new Date(), restorePoint: restorePoint(), ...options });
const reopen = () => executionWorkspaceService(db).reopenClosedIsolatedExecutionWorkspaceForIssue({ workspaceId,
issue: { id: issueId, companyId, projectId: null }, actor: { actorType: "user", agentId: null },
authorizationActor: { type: "board", source: "local_implicit", isInstanceAdmin: true } });
const ledger = () => db.select().from(executionWorkspaceRestorePoints).where(eq(executionWorkspaceRestorePoints.executionWorkspaceId, workspaceId));
const runtimeRestore = async () => ensurePersistedExecutionWorkspaceAvailable({ db, base,
workspace: (await executionWorkspaceService(db).getStoredById(companyId, workspaceId))!,
issue: { id: issueId, identifier: "RP-1", title: "Resumed task" },
agent: { id: null, name: "Fixture", companyId }, skipHostProvision: true });
return { companyId, issueId, workspaceId, cwd: workspace.cwd, baseCwd, row, restorePoint, expire, reopen, ledger, runtimeRestore };
}
async function makeUnsavedWork(cwd: string) {
await fs.writeFile(path.join(cwd, "marker.txt"), "commit B (unpushed)\n");
git(cwd, "commit", "-am", "B unpushed");
const unpushed = git(cwd, "rev-parse", "HEAD");
await fs.writeFile(path.join(cwd, "marker.txt"), "staged edit\n");
git(cwd, "add", "marker.txt");
await fs.writeFile(path.join(cwd, "marker.txt"), "unstaged edit\n");
await fs.rm(path.join(cwd, "remove-me.txt"));
await fs.mkdir(path.join(cwd, "notes"));
await fs.writeFile(path.join(cwd, "notes", "todo.md"), "untracked work\n");
await fs.writeFile(path.join(cwd, ".env"), "TOKEN=old\n");
return unpushed;
}
it("expires a worktree with unpushed and uncommitted work and restores it exactly on reopen", async () => {
const f = await fixture();
const unpushed = await makeUnsavedWork(f.cwd);
expect(await f.expire()).toMatchObject({ candidate: true, expired: true, restorePoint: true, skipReason: null });
expect(await exists(f.cwd)).toBe(false);
const recipe = (await f.row()).metadata?.gitCacheRestoration as Record<string, any>;
expect(recipe).toMatchObject({ schema: "paperclip.git-cache-restoration.v2", commit: unpushed,
restorePoint: { head: unpushed, headOnRemote: false, appliedAt: null } });
expect(await f.ledger()).toMatchObject([{ status: "active", objectKey: recipe.restorePoint.objectKey }]);
expect(await f.reopen()).toMatchObject({ ok: true, reopened: true });
expect(git(f.cwd, "rev-parse", "HEAD")).toBe(unpushed);
expect(await fs.readFile(path.join(f.cwd, "marker.txt"), "utf8")).toBe("unstaged edit\n");
expect(git(f.cwd, "show", ":marker.txt")).toBe("staged edit");
await expect(fs.access(path.join(f.cwd, "remove-me.txt"))).rejects.toMatchObject({ code: "ENOENT" });
expect(await fs.readFile(path.join(f.cwd, "notes", "todo.md"), "utf8")).toBe("untracked work\n");
await expect(fs.access(path.join(f.cwd, ".env"))).rejects.toMatchObject({ code: "ENOENT" });
const restored = (await f.row()).metadata?.gitCacheRestoration as Record<string, any>;
expect(restored.restorePoint.appliedAt).toEqual(expect.any(String));
});
it("restores through the runtime path and never re-applies over later work", async () => {
const f = await fixture();
const unpushed = await makeUnsavedWork(f.cwd);
expect(await f.expire()).toMatchObject({ expired: true });
await db.update(executionWorkspaces).set({ status: "active", closedAt: null }).where(eq(executionWorkspaces.id, f.workspaceId));
await f.runtimeRestore();
expect(git(f.cwd, "rev-parse", "HEAD")).toBe(unpushed);
expect(await fs.readFile(path.join(f.cwd, "notes", "todo.md"), "utf8")).toBe("untracked work\n");
const applied = (await f.row()).metadata?.gitCacheRestoration as Record<string, any>;
expect(applied.restorePoint.appliedAt).toEqual(expect.any(String));
await fs.writeFile(path.join(f.cwd, "notes", "todo.md"), "later work\n");
await f.runtimeRestore();
expect(await fs.readFile(path.join(f.cwd, "notes", "todo.md"), "utf8")).toBe("later work\n");
});
it("imports history-only commits on reopen even when the base no longer has them", async () => {
const f = await fixture();
await fs.writeFile(path.join(f.cwd, "lost.txt"), "rewritten away\n");
git(f.cwd, "add", "lost.txt");
git(f.cwd, "commit", "-m", "rewritten away");
const lost = git(f.cwd, "rev-parse", "HEAD");
git(f.cwd, "reset", "-q", "--hard", "HEAD~1");
expect(await f.expire()).toMatchObject({ expired: true, restorePoint: true });
git(f.baseCwd, "reflog", "expire", "--expire=now", "--all");
git(f.baseCwd, "gc", "-q", "--prune=now");
expect(() => git(f.baseCwd, "cat-file", "-e", `${lost}^{commit}`)).toThrow();
expect(await f.reopen()).toMatchObject({ ok: true, reopened: true });
expect(git(f.baseCwd, "cat-file", "-t", lost)).toBe("commit");
});
it("keeps a worktree whose draft commit message a restore point cannot save", async () => {
const f = await fixture();
await fs.writeFile(path.join(f.cwd, "marker.txt"), "staged edit\n");
git(f.cwd, "add", "marker.txt");
const gitDir = git(f.cwd, "rev-parse", "--absolute-git-dir");
await fs.writeFile(path.join(gitDir, "COMMIT_EDITMSG"), "Draft message from an aborted commit\n");
expect(await f.expire()).toMatchObject({ expired: false, skipReason: "uncommitted_git_message" });
expect(await exists(f.cwd)).toBe(true);
expect(await f.ledger()).toEqual([]);
});
it("marks a restore point for cleanup when its upload fails", async () => {
const f = await fixture();
await makeUnsavedWork(f.cwd);
const storage = getStorageService();
const failing = { ...storage, putFile: async () => { throw new Error("storage unavailable"); } };
expect(await f.expire({ restorePoint: { ...f.restorePoint(), storage: failing as typeof storage } }))
.toMatchObject({ expired: false });
expect(await exists(f.cwd)).toBe(true);
expect(await f.ledger()).toMatchObject([{ status: "superseded" }]);
expect((await f.row()).metadata?.gitCacheRestoration).toBeUndefined();
});
it("does not expire unsaved work before the restore point delay", async () => {
const f = await fixture();
await makeUnsavedWork(f.cwd);
expect(await f.expire({ restorePoint: f.restorePoint({ notBefore: new Date(Date.now() + DAY_MS) }) }))
.toMatchObject({ expired: false, skipReason: "restore_point_not_due" });
expect(await exists(f.cwd)).toBe(true);
expect(await f.ledger()).toEqual([]);
});
it("keeps today's behavior when restore points are disabled", async () => {
const f = await fixture();
await makeUnsavedWork(f.cwd);
expect(await f.expire({ restorePoint: undefined })).toMatchObject({ expired: false, skipReason: "dirty_or_untracked_files" });
expect(await exists(f.cwd)).toBe(true);
});
it("refuses to capture when the encryption key is not durable", async () => {
const f = await fixture();
await makeUnsavedWork(f.cwd);
expect(await f.expire({ restorePoint: f.restorePoint({ keyDurable: false }) }))
.toMatchObject({ expired: false, skipReason: "restore_point_key_not_durable" });
expect(await exists(f.cwd)).toBe(true);
});
it("keeps the worktree and drops the restore point when contents change after capture", async () => {
const f = await fixture();
await makeUnsavedWork(f.cwd);
const result = await f.expire({ beforeDelete: async () => { await fs.writeFile(path.join(f.cwd, "notes", "todo.md"), "changed after capture\n"); } });
expect(result).toMatchObject({ expired: false, skipReason: "restore_point_contents_changed" });
expect(await exists(f.cwd)).toBe(true);
expect(await fs.readFile(path.join(f.cwd, "notes", "todo.md"), "utf8")).toBe("changed after capture\n");
const after = await f.row();
expect(after.status).toBe("active");
expect(after.metadata?.gitCacheRestoration).toMatchObject({ schema: "paperclip.git-cache-restoration.v1" });
expect((after.metadata?.gitCacheRestoration as Record<string, unknown>).restorePoint).toBeUndefined();
expect(await f.ledger()).toMatchObject([{ status: "superseded" }]);
});
it("fails closed when the stored restore point is missing", async () => {
const f = await fixture();
await makeUnsavedWork(f.cwd);
expect(await f.expire()).toMatchObject({ expired: true });
const recipe = (await f.row()).metadata?.gitCacheRestoration as Record<string, any>;
await getStorageService().deleteObject(f.companyId, recipe.restorePoint.objectKey);
expect(await f.reopen()).toMatchObject({ ok: false, code: "rebuild_failed" });
const failed = await f.row();
expect(failed.status).toBe("archived");
expect(failed.metadata?.gitCacheRestoration).toMatchObject({ restorePoint: { objectKey: recipe.restorePoint.objectKey, appliedAt: null } });
await expect(fs.access(path.join(f.cwd, "notes", "todo.md"))).rejects.toMatchObject({ code: "ENOENT" });
expect(await f.reopen()).toMatchObject({ ok: false, code: "rebuild_failed" });
});
it("reopens a workspace whose restore point expired as a fresh workspace with a warning", async () => {
const f = await fixture();
const unpushed = await makeUnsavedWork(f.cwd);
expect(await f.expire()).toMatchObject({ expired: true });
const [ledgerRow] = await f.ledger();
await markGitRestorePointExpired(db, ledgerRow!);
await getStorageService().deleteObject(f.companyId, ledgerRow!.objectKey);
const branch = (await f.row()).branchName!;
git(f.baseCwd, "worktree", "prune");
git(f.baseCwd, "branch", "-D", branch);
git(f.baseCwd, "reflog", "expire", "--expire=now", "--all");
git(f.baseCwd, "gc", "-q", "--prune=now");
expect(() => git(f.baseCwd, "cat-file", "-e", `${unpushed}^{commit}`)).toThrow();
expect((await f.row()).metadata?.gitCacheRestoration).toMatchObject({ restorePoint: { expiredAt: expect.any(String) } });
await fs.writeFile(f.cwd, "blocks the worktree\n");
expect(await f.reopen()).toMatchObject({ ok: false, code: "rebuild_failed" });
expect((await f.row()).metadata?.gitCacheRestoration).toMatchObject({ restorePoint: { expiredAt: expect.any(String) } });
await fs.rm(f.cwd);
const reopened = await f.reopen();
expect(reopened).toMatchObject({ ok: true, reopened: true, warning: GIT_RESTORE_POINT_EXPIRED_WARNING });
expect(await exists(f.cwd)).toBe(true);
expect(git(f.cwd, "rev-parse", "--abbrev-ref", "HEAD")).toBe(branch);
expect(git(f.cwd, "rev-parse", "HEAD")).not.toBe(unpushed);
expect(git(f.cwd, "status", "--porcelain")).toBe("");
await expect(fs.access(path.join(f.cwd, "notes", "todo.md"))).rejects.toMatchObject({ code: "ENOENT" });
expect((await f.row()).metadata?.gitCacheRestoration).toBeUndefined();
const comments = await db.select().from(issueComments).where(eq(issueComments.issueId, f.issueId));
expect(comments.filter(comment => comment.body === GIT_RESTORE_POINT_EXPIRED_WARNING)).toMatchObject([{ authorType: "system" }]);
});
describe("idle open issues", () => {
async function openIdle(f: Awaited<ReturnType<typeof fixture>>, idleDays: number) {
await db.update(issues).set({ status: "in_progress", completedAt: null, updatedAt: new Date(Date.now() - idleDays * DAY_MS) })
.where(eq(issues.id, f.issueId));
}
const idle = (now = Date.now()) => ({ idleCutoff: new Date(now - 7 * DAY_MS) });
it("expires an open issue's idle worktree with a restore point and restores it on reopen", async () => {
const f = await fixture();
const unpushed = await makeUnsavedWork(f.cwd);
await openIdle(f, 8);
expect(await f.expire()).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" });
expect(await f.expire(idle())).toMatchObject({ expired: true, restorePoint: true, skipReason: null });
expect(await exists(f.cwd)).toBe(false);
expect((await f.row()).status).toBe("archived");
expect(await f.reopen()).toMatchObject({ ok: true, reopened: true });
expect(git(f.cwd, "rev-parse", "HEAD")).toBe(unpushed);
expect(await fs.readFile(path.join(f.cwd, "notes", "todo.md"), "utf8")).toBe("untracked work\n");
});
it("lets the workspace reaper expire idle open issues at their idle deadline", async () => {
const f = await fixture();
await makeUnsavedWork(f.cwd);
const updatedAt = new Date(Date.now() - 3 * DAY_MS);
await db.update(issues).set({ status: "in_progress", completedAt: null, updatedAt }).where(eq(issues.id, f.issueId));
const worker = (now: number) => executionWorkspaceService(db, { gitCacheExpirationEnabled: true, now: () => new Date(now),
gitRestorePoints: { storage: getStorageService(), terminalDelayMs: 3 * DAY_MS, keyDurable: true, idleMs: 7 * DAY_MS } });
const early = worker(Date.now());
expect(await early.sweepTerminalWorkspaces()).toMatchObject({ archived: 0, skippedNonTerminalTree: 1 });
expect(await early.nextTerminalWorkspaceSweepAt()).toBe(updatedAt.getTime() + 7 * DAY_MS);
expect(await exists(f.cwd)).toBe(true);
const due = worker(updatedAt.getTime() + 7 * DAY_MS + 60_000);
expect(await due.sweepTerminalWorkspaces()).toMatchObject({ archived: 1 });
expect(await exists(f.cwd)).toBe(false);
expect(await f.ledger()).toMatchObject([{ status: "active" }]);
});
it("lets the workspace reaper finish an interrupted idle expiration while the issue stays open", async () => {
const f = await fixture();
await makeUnsavedWork(f.cwd);
await openIdle(f, 8);
expect(await f.expire({ ...idle(), beforeDelete: async () => { throw new Error("interrupted_fixture"); } }))
.toMatchObject({ expired: false });
expect(await f.row()).toMatchObject({ status: "archived", metadata: { gitCacheExpiration: { completed: false } } });
expect(await exists(f.cwd)).toBe(true);
const worker = executionWorkspaceService(db, { gitCacheExpirationEnabled: true,
gitRestorePoints: { storage: getStorageService(), terminalDelayMs: 3 * DAY_MS, keyDurable: true, idleMs: 7 * DAY_MS } });
expect(await worker.sweepTerminalWorkspaces()).toMatchObject({ archived: 1 });
expect(await exists(f.cwd)).toBe(false);
expect(await f.reopen()).toMatchObject({ ok: true, reopened: true });
expect(await fs.readFile(path.join(f.cwd, "notes", "todo.md"), "utf8")).toBe("untracked work\n");
});
it("counts activity on another issue bound to the workspace", async () => {
const f = await fixture();
await makeUnsavedWork(f.cwd);
await openIdle(f, 8);
const other = randomUUID();
await db.insert(issues).values({ id: other, companyId: f.companyId, title: "Unrelated issue sharing the workspace", status: "todo",
executionWorkspaceId: f.workspaceId, updatedAt: new Date(Date.now() - DAY_MS) });
expect((await gitWorkspaceLastActivityAt(db, await f.row()))!.getTime()).toBeGreaterThan(Date.now() - 2 * DAY_MS);
expect(await f.expire(idle())).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" });
expect(await exists(f.cwd)).toBe(true);
});
it("never expires open issues when idle expiry is off", async () => {
const f = await fixture();
await makeUnsavedWork(f.cwd);
await openIdle(f, 30);
const worker = executionWorkspaceService(db, { gitCacheExpirationEnabled: true,
gitRestorePoints: { storage: getStorageService(), terminalDelayMs: 3 * DAY_MS, keyDurable: true, idleMs: null } });
expect(await worker.sweepTerminalWorkspaces()).toMatchObject({ archived: 0, skippedNonTerminalTree: 1 });
expect(await worker.nextTerminalWorkspaceSweepAt()).toBeNull();
expect(await exists(f.cwd)).toBe(true);
});
it("keeps an open issue's worktree that saw activity inside the idle window", async () => {
const f = await fixture();
await makeUnsavedWork(f.cwd);
await openIdle(f, 3);
expect(await f.expire(idle())).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" });
expect(await exists(f.cwd)).toBe(true);
});
it("keeps an open issue's worktree whose latest run finished inside the idle window", async () => {
const f = await fixture();
await makeUnsavedWork(f.cwd);
await openIdle(f, 8);
const agentId = randomUUID();
await db.insert(agents).values({ id: agentId, companyId: f.companyId, name: "Worker", adapterType: "process" });
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId, issueId: f.issueId, status: "succeeded",
createdAt: new Date(Date.now() - 2 * DAY_MS), startedAt: new Date(Date.now() - 2 * DAY_MS), finishedAt: new Date(Date.now() - 2 * DAY_MS) });
expect(await gitWorkspaceLastActivityAt(db, await f.row())).toEqual(expect.any(Date));
expect(await f.expire(idle())).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" });
expect(await exists(f.cwd)).toBe(true);
});
});
describe("closed source issue with open descendants", () => {
async function child(f: Awaited<ReturnType<typeof fixture>>, values: Partial<typeof issues.$inferInsert> = {}) {
const id = randomUUID();
await db.insert(issues).values({ id, companyId: f.companyId, parentId: f.issueId, title: "Follow-up", status: "blocked", ...values });
return id;
}
async function ownWorkspace(f: Awaited<ReturnType<typeof fixture>>, sourceIssueId: string) {
const id = randomUUID();
await db.insert(executionWorkspaces).values({ id, companyId: f.companyId, sourceIssueId, mode: "isolated_workspace", strategyType: "git_worktree",
name: "Child workspace", status: "active", providerType: "local_fs", cwd: path.join(root, "child-workspaces", id), providerRef: path.join(root, "child-workspaces", id) });
await db.update(issues).set({ executionWorkspaceId: id, executionWorkspacePreference: "reuse_existing" }).where(eq(issues.id, sourceIssueId));
return id;
}
const worker = (now = Date.now()) => executionWorkspaceService(db, { gitCacheExpirationEnabled: true, now: () => new Date(now),
gitRestorePoints: { storage: getStorageService(), terminalDelayMs: 3 * DAY_MS, keyDurable: true, idleMs: 7 * DAY_MS } });
it("expires the closed issue's worktree while open children work in their own workspaces", async () => {
const f = await fixture();
const unpushed = await makeUnsavedWork(f.cwd);
const blocked = await child(f, { status: "blocked" });
await ownWorkspace(f, blocked);
const reviewing = await child(f, { status: "in_review" });
await ownWorkspace(f, reviewing);
const grandchild = await child(f, { parentId: reviewing, status: "in_progress" });
await ownWorkspace(f, grandchild);
await child(f, { status: "todo" });
expect(await f.expire()).toMatchObject({ candidate: true, expired: true, restorePoint: true, skipReason: null });
expect(await exists(f.cwd)).toBe(false);
expect(await f.ledger()).toMatchObject([{ status: "active" }]);
expect(await f.reopen()).toMatchObject({ ok: true, reopened: true });
expect(git(f.cwd, "rev-parse", "HEAD")).toBe(unpushed);
expect(await fs.readFile(path.join(f.cwd, "notes", "todo.md"), "utf8")).toBe("untracked work\n");
});
it("lets the workspace reaper expire it, timed from the issues that use the workspace", async () => {
const f = await fixture();
await makeUnsavedWork(f.cwd);
const open = await child(f, { status: "in_progress", updatedAt: new Date() });
await ownWorkspace(f, open);
const closed = await child(f, { status: "done", completedAt: new Date(Date.now() - 60_000) });
await ownWorkspace(f, closed);
expect(await worker().sweepTerminalWorkspaces()).toMatchObject({ archived: 1 });
expect(await exists(f.cwd)).toBe(false);
expect((await f.row()).status).toBe("archived");
expect(await f.ledger()).toMatchObject([{ status: "active" }]);
});
it("times the restore point from the issues that use the workspace once every child is closed", async () => {
const f = await fixture();
await makeUnsavedWork(f.cwd);
const closed = await child(f, { status: "done", completedAt: new Date(Date.now() - 60_000) });
await ownWorkspace(f, closed);
expect(await worker().sweepTerminalWorkspaces()).toMatchObject({ archived: 1 });
expect(await exists(f.cwd)).toBe(false);
expect(await f.ledger()).toMatchObject([{ status: "active" }]);
});
it("waits for the terminal delay of a recently closed child that used the workspace", async () => {
const f = await fixture();
await makeUnsavedWork(f.cwd);
await child(f, { status: "in_progress" }).then(id => ownWorkspace(f, id));
const closedAt = new Date(Date.now() - DAY_MS);
await child(f, { status: "done", completedAt: closedAt, executionWorkspaceId: f.workspaceId });
const early = worker();
expect(await early.sweepTerminalWorkspaces()).toMatchObject({ archived: 0 });
expect(await exists(f.cwd)).toBe(true);
expect(await worker(closedAt.getTime() + 3 * DAY_MS + 60_000).sweepTerminalWorkspaces()).toMatchObject({ archived: 1 });
expect(await exists(f.cwd)).toBe(false);
});
it("never compacts a closed issue's worktree while children are open", async () => {
const f = await fixture();
await makeUnsavedWork(f.cwd);
await db.update(executionWorkspaces).set({ providerType: "git_worktree" }).where(eq(executionWorkspaces.id, f.workspaceId));
await db.update(issues).set({ completedAt: new Date(Date.now() - 2 * DAY_MS) }).where(eq(issues.id, f.issueId));
await fs.mkdir(path.join(f.cwd, "node_modules", "pkg"), { recursive: true });
await fs.writeFile(path.join(f.cwd, "node_modules", "pkg", "index.js"), "module.exports = 1;\n");
await child(f, { status: "in_progress" }).then(id => ownWorkspace(f, id));
for (const reaper of [executionWorkspaceService(db, { workspaceCompactionDelayMs: 0 }),
executionWorkspaceService(db, { workspaceCompactionDelayMs: 0, gitCacheExpirationEnabled: true,
gitRestorePoints: { storage: getStorageService(), terminalDelayMs: 3 * DAY_MS, keyDurable: true } })]) {
expect(await reaper.sweepTerminalWorkspaces()).toMatchObject({ archived: 0, compacted: 0 });
expect(await exists(path.join(f.cwd, "node_modules", "pkg", "index.js"))).toBe(true);
}
});
it("keeps today's reopen fence handling while a reopen is pending", async () => {
const f = await fixture();
await makeUnsavedWork(f.cwd);
await child(f, { status: "in_progress" }).then(id => ownWorkspace(f, id));
await db.update(executionWorkspaces).set({ metadata: setMetadataReopenPendingConsumption((await f.row()).metadata, new Date()) })
.where(eq(executionWorkspaces.id, f.workspaceId));
expect(await worker().sweepTerminalWorkspaces()).toMatchObject({ archived: 0 });
expect(await exists(f.cwd)).toBe(true);
expect((await f.row()).metadata?.reopenPendingConsumption).toBeUndefined();
});
it("keeps the worktree while an open child is bound to it", async () => {
const f = await fixture();
await makeUnsavedWork(f.cwd);
await child(f, { status: "blocked", executionWorkspaceId: f.workspaceId, executionWorkspacePreference: "reuse_existing" });
expect(await f.expire()).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" });
expect(await worker().sweepTerminalWorkspaces()).toMatchObject({ archived: 0, skippedNonTerminalTree: 1 });
expect(await exists(f.cwd)).toBe(true);
});
it("keeps the worktree while an open grandchild is bound to it under a child with its own workspace", async () => {
const f = await fixture();
const middle = await child(f, { status: "in_review" });
await ownWorkspace(f, middle);
await child(f, { parentId: middle, status: "todo", executionWorkspaceId: f.workspaceId });
expect(await f.expire()).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" });
expect(await exists(f.cwd)).toBe(true);
});
it("keeps the worktree for a child created later that inherits it with reuse_existing", async () => {
const f = await fixture();
const created = await issueService(db).create(f.companyId, { parentId: f.issueId, title: "Inherited follow-up", status: "todo" });
expect(created).toMatchObject({ executionWorkspaceId: f.workspaceId, executionWorkspacePreference: "reuse_existing" });
expect(await f.expire()).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" });
expect(await worker().sweepTerminalWorkspaces()).toMatchObject({ archived: 0, skippedNonTerminalTree: 1 });
expect(await exists(f.cwd)).toBe(true);
});
it("keeps the worktree while an open child selects it as its existing workspace", async () => {
const f = await fixture();
const selection = { kind: "existing" as const, workspaceId: f.workspaceId };
const selecting = await child(f, { status: "todo", workspaceSelection: { version: 1, selection, source: "explicit" } });
expect(await f.expire()).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" });
await db.update(issues).set({ workspaceSelection: null, workspacePendingSelection: { version: 1, requestKey: "fixture",
expectedBindingRevision: 0, intent: { version: 1, selection, source: "explicit" } } }).where(eq(issues.id, selecting));
expect(await f.expire()).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" });
await db.update(issues).set({ workspacePendingSelection: null }).where(eq(issues.id, selecting));
expect(await f.expire()).toMatchObject({ expired: true });
});
it("keeps the worktree while a child run referencing it is live", async () => {
const f = await fixture();
const running = await child(f, { status: "in_progress" });
await ownWorkspace(f, running);
const agentId = randomUUID();
await db.insert(agents).values({ id: agentId, companyId: f.companyId, name: "Worker", adapterType: "process" });
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId, status: "running", startedAt: new Date(),
contextSnapshot: { issueId: running, executionWorkspaceId: f.workspaceId } }).returning();
expect(await f.expire()).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" });
expect(await exists(f.cwd)).toBe(true);
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, run!.id));
expect(await f.expire()).toMatchObject({ expired: true });
});
it("keeps the worktree while the closed issue was closed inside the terminal delay", async () => {
const f = await fixture();
await db.update(issues).set({ completedAt: new Date(Date.now() - 60_000) }).where(eq(issues.id, f.issueId));
await child(f, { status: "in_progress" }).then(id => ownWorkspace(f, id));
expect(await f.expire()).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" });
expect(await exists(f.cwd)).toBe(true);
});
it("restores the saved state when an open child later binds to the expired workspace", async () => {
const f = await fixture();
const unpushed = await makeUnsavedWork(f.cwd);
const follower = await child(f, { status: "in_progress" });
expect(await f.expire()).toMatchObject({ expired: true, restorePoint: true });
await db.update(issues).set({ executionWorkspaceId: f.workspaceId, executionWorkspacePreference: "reuse_existing" }).where(eq(issues.id, follower));
const reopened = await executionWorkspaceService(db).reopenClosedIsolatedExecutionWorkspaceForIssue({ workspaceId: f.workspaceId,
issue: { id: follower, companyId: f.companyId, projectId: null }, actor: { actorType: "user", agentId: null },
authorizationActor: { type: "board", source: "local_implicit", isInstanceAdmin: true } });
expect(reopened).toMatchObject({ ok: true, reopened: true });
expect(git(f.cwd, "rev-parse", "HEAD")).toBe(unpushed);
expect(await fs.readFile(path.join(f.cwd, "marker.txt"), "utf8")).toBe("unstaged edit\n");
expect(git(f.cwd, "show", ":marker.txt")).toBe("staged edit");
expect(await fs.readFile(path.join(f.cwd, "notes", "todo.md"), "utf8")).toBe("untracked work\n");
});
});
it("expires a worktree whose only extra files are ignored, and restores them on reopen", async () => {
const f = await fixture();
await fs.mkdir(path.join(f.cwd, "ui"), { recursive: true });
await fs.writeFile(path.join(f.cwd, "ui", "tsconfig.tsbuildinfo"), "{\"version\":\"5\"}\n");
await fs.mkdir(path.join(f.cwd, "evidence"));
await fs.writeFile(path.join(f.cwd, "evidence", "run.log"), "agent evidence\n");
await fs.mkdir(path.join(f.cwd, "node_modules", "pkg"), { recursive: true });
await fs.writeFile(path.join(f.cwd, "node_modules", "pkg", "index.js"), "regenerable\n");
expect(git(f.cwd, "status", "--porcelain")).toBe("");
expect(await f.expire({ restorePoint: undefined })).toMatchObject({ expired: false, skipReason: "unique_ignored_files" });
expect(await f.expire()).toMatchObject({ expired: true, restorePoint: true, skipReason: null });
expect(await exists(f.cwd)).toBe(false);
expect(await f.reopen()).toMatchObject({ ok: true, reopened: true });
expect(await fs.readFile(path.join(f.cwd, "ui", "tsconfig.tsbuildinfo"), "utf8")).toBe("{\"version\":\"5\"}\n");
expect(await fs.readFile(path.join(f.cwd, "evidence", "run.log"), "utf8")).toBe("agent evidence\n");
expect(await exists(path.join(f.cwd, "node_modules", "pkg", "index.js"))).toBe(false);
});
it("rolls back when an ignored file changes after capture", async () => {
const f = await fixture();
await fs.mkdir(path.join(f.cwd, "evidence"));
await fs.writeFile(path.join(f.cwd, "evidence", "run.log"), "agent evidence\n");
const result = await f.expire({ beforeDelete: async () => {
await fs.writeFile(path.join(f.cwd, "evidence", "run.log"), "written after capture\n");
} });
expect(result).toMatchObject({ expired: false, skipReason: "restore_point_contents_changed" });
expect(await fs.readFile(path.join(f.cwd, "evidence", "run.log"), "utf8")).toBe("written after capture\n");
});
it("reports a restore point candidate in dry run without storing anything", async () => {
const f = await fixture();
await makeUnsavedWork(f.cwd);
expect(await f.expire({ dryRun: true })).toMatchObject({ candidate: true, expired: false, restorePoint: true, skipReason: null });
expect(await exists(f.cwd)).toBe(true);
expect(await f.ledger()).toEqual([]);
expect((await f.row()).metadata?.gitCacheRestoration).toBeUndefined();
});
it("still expires a fully reproducible worktree without a restore point", async () => {
const f = await fixture();
expect(await f.expire()).toMatchObject({ expired: true, restorePoint: false });
expect((await f.row()).metadata?.gitCacheRestoration).toMatchObject({ schema: "paperclip.git-cache-restoration.v1" });
expect(await f.ledger()).toEqual([]);
});
});