From fe85fd7ac45df92622b62205d94b2fbf52104a3c Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 24 Oct 2026 12:43:57 -0700 Subject: [PATCH] Expire closed workspaces that open descendants do not use (#17464) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Git cache expiration removes closed issues' worktrees and saves restore points for unsaved work > - It treated a workspace as closed only when the source issue and every descendant were done or cancelled > - A routine that closes a parent and leaves children open for days therefore kept every parent worktree, although the children work in their own workspaces > - The idle path did not help, because the children are updated every day > - This pull request counts only the issues that use the workspace when it decides that a closed workspace can expire > - The benefit is that closed parent worktrees are removed while their children work elsewhere, and a child that later binds to the workspace gets it back from the restore point ## Linked Issues or Issue Description Refs #17271. Bug report described in-PR (bug report template). **What happened?** On one instance, an hourly routine creates a parent issue, works it in its own worktree, marks the parent done, and leaves child issues open (`blocked`, `in_review` or `in_progress`) for days. The children work in their own workspaces. They are not bound to the parent's workspace. Git cache expiration kept every parent worktree (about 300 MB each), because the closed-tree check and the workspace reaper required all descendants to be closed. In one project, 18 of 19 long-closed parent worktrees were kept only by open children that did not use them. The idle path never ran, because the children were updated daily. **Expected behavior** A closed issue's worktree expires while descendants are open, as long as no open descendant uses that workspace. **Steps to reproduce** 1. Enable git cache expiration and restore points. 2. Close an issue that has an isolated git worktree, and leave a child issue open that works in its own workspace (or has no workspace binding). 3. Wait past the terminal delay and run the workspace reaper: the parent worktree stays, and the reaper counts it as `skippedNonTerminalTree`. **Paperclip version or commit:** `master` at 410e5c6eedb. **Deployment mode:** any. **Agent adapter(s) involved:** not adapter-specific (core). ## What Changed - `isGitWorkspaceExpirationEligible` (closed path, not the idle path): the "tree closed for one day" check now covers only the issues in the tree that use the workspace. An issue uses the workspace when it is the source issue, when its `execution_workspace_id` is the workspace, or when its current `workspace_selection` or its `workspace_pending_selection` selects it as the `existing` workspace. Inheritance from a parent (`reuse_existing`) and `existing` selections are both stored as `execution_workspace_id`, and heartbeat workspace resolution reuses an existing workspace only through that binding or a run's persisted native binding. An open descendant without one of these references gets a new workspace on its next run, so it no longer keeps the parent's worktree. - All other checks stay the same: any open issue bound to the workspace, live or unsettled runs of any issue in the tree or of the workspace (including a run whose context names the workspace), finalizations, leases, recovery actions, runtime services, shared cwd and the reopen fence. - Workspace reaper: when the source issue is closed, git cache expiration applies, and no reopen is pending, the reaper routes the workspace by the same rule. It takes the restore point due time from the source issue and the descendants that use the workspace, also after every descendant is closed. A recently closed child that used the workspace still holds it for the terminal delay. The compaction delay and the archive cooldown still use the whole tree when it is closed. - Workspace reaper: compaction and the normal archive path keep their behavior. While any descendant is open, the reaper does not compact the worktree, and the expiration branch handles the workspace and continues. The archive path still requires a closed subtree. With expiration off, or a pending reopen, routing is unchanged. - New exported helper `listIssuesUsingGitWorkspace` shares the SQL "uses the workspace" predicate between eligibility and the reaper. - Docs: `doc/recoverable-disk-cleanup.md` and the `PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS` entry in `doc/DEVELOPING.md`. - The existing `preserves live descendants and pending reopening` case now binds its open child to the workspace. An unbound open child no longer keeps the worktree by design. ## Verification - `pnpm exec vitest run src/__tests__/git-restore-point-expiration.test.ts src/__tests__/git-workspace-expiration.test.ts src/__tests__/heartbeat-git-restore-point-resume.test.ts` in `server/`: all pass. - `pnpm exec vitest run src/__tests__/git-workspace-expiration src/__tests__/git-restore-point src/__tests__/execution-workspace src/__tests__/heartbeat-git-restore-point-resume src/services/heartbeat/environment-selection src/__tests__/isolated-repository` in `server/`: 15 of 17 files passed in one parallel run. The other failures were the updated descendant case (now fixed) and 45-second timeouts in restore point tests while 17 embedded Postgres files ran at the same time. Both files pass when run on their own. - `pnpm run typecheck` in `server/`: passes. - New embedded Postgres cases with real git repositories: - A closed parent with open `blocked`, `in_review`, `in_progress` and `todo` children and a grandchild in their own workspaces (or with none): the worktree expires with a restore point and the reopen restores it exactly. - The reaper archives it, timed from the issues that use the workspace. A child closed one minute ago with its own workspace does not delay it. - A recently closed child bound to the workspace holds it until its terminal delay passes. - When every child is closed and the most recent one used its own workspace, the reaper does not restart the restore point delay. - The worktree is kept when an open child is bound to it, when an open grandchild under a child with its own workspace is bound to it, when a child created later with `issueService.create` inherits it with `reuse_existing`, when an open child selects it through `workspace_selection` or `workspace_pending_selection`, when a child run that references it is live, and when the parent was closed inside the delay. - The reaper never compacts the worktree while children are open, with expiration on or off. A pending reopen keeps its current fence handling. - After expiry, an open child that binds to the workspace reopens it with the saved commits, index, working tree and untracked files. - Heartbeat: a queued run for an open child that binds to the expired workspace of a closed parent restores the workspace and runs the agent in it with the saved state. - Mutation checks (each run against the two restore point test files): - Closed-tree check restored to the whole tree: 7 tests fail. - `execution_workspace_id` removed from the rule: 3 fail. - `workspace_selection` removed from the rule: 1 fails. - `workspace_pending_selection` removed from the rule: 1 fails. - Reaper `latestTerminalAt` from the whole tree: 3 fail. - Reaper routing by the whole tree: 2 fail. - Reaper restore point due time from the whole tree once it is closed: 1 fails. - Pending-reopen gate removed: 1 fails. - Compaction gate (`!treeOpen`) removed: no test fails. Compaction's own guard under the lifecycle lock still rejects open trees, so the gate only skips work that cannot succeed. - Not run: the full server suite and the UI suites. CI covers them. ## Risks - Closed parent worktrees now expire while their children are open. A child that later binds to the workspace depends on the restore point and on the run-side reopen from #17271. The new tests cover this path. When restore points are off, only exactly reproducible worktrees expire. - An open child with no workspace binding no longer keeps the parent's worktree. Heartbeat resolution never reuses a workspace without a binding, so this does not change where the child runs. - The rule is conservative. A workspace selected only by an issue's current or pending selection is kept, and live runs of any issue in the tree still block. - The reaper runs one extra indexed query for each closed workspace when git cache expiration applies. No migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/DEVELOPING.md | 4 +- doc/recoverable-disk-cleanup.md | 11 ++ .../git-restore-point-expiration.test.ts | 178 +++++++++++++++++- .../git-workspace-expiration.test.ts | 3 +- ...heartbeat-git-restore-point-resume.test.ts | 40 +++- server/src/services/execution-workspaces.ts | 20 +- .../src/services/git-workspace-expiration.ts | 23 ++- 7 files changed, 259 insertions(+), 20 deletions(-) diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 0e44d7feb81..fea6694f2e8 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -2618,8 +2618,8 @@ Environment overrides: - `PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true` lets git cache expiration remove closed worktrees with unpushed or uncommitted work after saving an encrypted restore point (see `doc/recoverable-disk-cleanup.md`). It is off by default. - `PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS=` sets how long the issue tree - must be closed first. The default is `3`. A negative, empty, or non-numeric value falls + `PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS=` sets how long the issues that + use the workspace (the source issue and descendants bound to it) must be closed first. The default is `3`. A negative, empty, or non-numeric value falls back to the default. `PAPERCLIP_GIT_RESTORE_POINT_IDLE_DAYS=` sets how long an open issue tree's worktree must go untouched before it is removed the same way; runs restore it diff --git a/doc/recoverable-disk-cleanup.md b/doc/recoverable-disk-cleanup.md index 1fa8a1a892e..ea4aa982001 100644 --- a/doc/recoverable-disk-cleanup.md +++ b/doc/recoverable-disk-cleanup.md @@ -158,6 +158,17 @@ closed issue's worktree only when its remote can reproduce it exactly. With `PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true`, it can also remove a worktree that has unpushed commits, uncommitted or untracked files, or history-only commits, once its issue tree has been closed for `PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS` (default `3`). +For this, only the issues in the tree that use the workspace count: the source issue, and +each descendant that is bound to it (`execution_workspace_id`, which is how a child that +inherits the parent's workspace with `reuse_existing` is recorded) or that selects it as +its current or pending existing workspace. The delay runs from the latest close time among +those issues. A closed issue's worktree can therefore be removed while descendants that +work in their own workspaces stay open. Any issue still bound to the workspace, and any +live run, finalization, lease, recovery action or runtime service of an issue in the tree +or of the workspace, still keeps it. If an open descendant later binds to the removed +workspace, its next run or a comment reopens it from the restore point, as below. The +workspace reaper does not compact or archive such a worktree by its normal rules while +descendants are open; only expiration removes it. It also removes the worktree of an issue tree that is still open once nothing has touched it for `PAPERCLIP_GIT_RESTORE_POINT_IDLE_DAYS` (default `7`, `0` turns this off): no issue in the tree or bound to the workspace was updated and no run for those issues or diff --git a/server/src/__tests__/git-restore-point-expiration.test.ts b/server/src/__tests__/git-restore-point-expiration.test.ts index 510c9a536ed..49a05169abc 100644 --- a/server/src/__tests__/git-restore-point-expiration.test.ts +++ b/server/src/__tests__/git-restore-point-expiration.test.ts @@ -14,7 +14,8 @@ import { executionWorkspaceService } from "../services/execution-workspaces.js"; import { expireReconstructableGitWorkspace, gitWorkspaceLastActivityAt } from "../services/git-workspace-expiration.js"; import { getStorageService } from "../storage/index.js"; import { markGitRestorePointExpired } from "../services/git-restore-point-storage.js"; -import { GIT_RESTORE_POINT_EXPIRED_WARNING } from "../services/execution-workspaces.js"; +import { GIT_RESTORE_POINT_EXPIRED_WARNING, setMetadataReopenPendingConsumption } from "../services/execution-workspaces.js"; +import { issueService } from "../services/issues.js"; const gitEnv = { ...process.env, GIT_AUTHOR_NAME: "Fixture", GIT_AUTHOR_EMAIL: "fixture@example.test", GIT_COMMITTER_NAME: "Fixture", GIT_COMMITTER_EMAIL: "fixture@example.test", GIT_TERMINAL_PROMPT: "0" }; @@ -356,6 +357,181 @@ describeEmbeddedPostgres("git restore point expiration", () => { }); }); + describe("closed source issue with open descendants", () => { + async function child(f: Awaited>, values: Partial = {}) { + const id = randomUUID(); + await db.insert(issues).values({ id, companyId: f.companyId, parentId: f.issueId, title: "Follow-up", status: "blocked", ...values }); + return id; + } + async function ownWorkspace(f: Awaited>, sourceIssueId: string) { + const id = randomUUID(); + await db.insert(executionWorkspaces).values({ id, companyId: f.companyId, sourceIssueId, mode: "isolated_workspace", strategyType: "git_worktree", + name: "Child workspace", status: "active", providerType: "local_fs", cwd: path.join(root, "child-workspaces", id), providerRef: path.join(root, "child-workspaces", id) }); + await db.update(issues).set({ executionWorkspaceId: id, executionWorkspacePreference: "reuse_existing" }).where(eq(issues.id, sourceIssueId)); + return id; + } + const worker = (now = Date.now()) => executionWorkspaceService(db, { gitCacheExpirationEnabled: true, now: () => new Date(now), + gitRestorePoints: { storage: getStorageService(), terminalDelayMs: 3 * DAY_MS, keyDurable: true, idleMs: 7 * DAY_MS } }); + + it("expires the closed issue's worktree while open children work in their own workspaces", async () => { + const f = await fixture(); + const unpushed = await makeUnsavedWork(f.cwd); + const blocked = await child(f, { status: "blocked" }); + await ownWorkspace(f, blocked); + const reviewing = await child(f, { status: "in_review" }); + await ownWorkspace(f, reviewing); + const grandchild = await child(f, { parentId: reviewing, status: "in_progress" }); + await ownWorkspace(f, grandchild); + await child(f, { status: "todo" }); + expect(await f.expire()).toMatchObject({ candidate: true, expired: true, restorePoint: true, skipReason: null }); + expect(await exists(f.cwd)).toBe(false); + expect(await f.ledger()).toMatchObject([{ status: "active" }]); + expect(await f.reopen()).toMatchObject({ ok: true, reopened: true }); + expect(git(f.cwd, "rev-parse", "HEAD")).toBe(unpushed); + expect(await fs.readFile(path.join(f.cwd, "notes", "todo.md"), "utf8")).toBe("untracked work\n"); + }); + + it("lets the workspace reaper expire it, timed from the issues that use the workspace", async () => { + const f = await fixture(); + await makeUnsavedWork(f.cwd); + const open = await child(f, { status: "in_progress", updatedAt: new Date() }); + await ownWorkspace(f, open); + const closed = await child(f, { status: "done", completedAt: new Date(Date.now() - 60_000) }); + await ownWorkspace(f, closed); + expect(await worker().sweepTerminalWorkspaces()).toMatchObject({ archived: 1 }); + expect(await exists(f.cwd)).toBe(false); + expect((await f.row()).status).toBe("archived"); + expect(await f.ledger()).toMatchObject([{ status: "active" }]); + }); + + it("times the restore point from the issues that use the workspace once every child is closed", async () => { + const f = await fixture(); + await makeUnsavedWork(f.cwd); + const closed = await child(f, { status: "done", completedAt: new Date(Date.now() - 60_000) }); + await ownWorkspace(f, closed); + expect(await worker().sweepTerminalWorkspaces()).toMatchObject({ archived: 1 }); + expect(await exists(f.cwd)).toBe(false); + expect(await f.ledger()).toMatchObject([{ status: "active" }]); + }); + + it("waits for the terminal delay of a recently closed child that used the workspace", async () => { + const f = await fixture(); + await makeUnsavedWork(f.cwd); + await child(f, { status: "in_progress" }).then(id => ownWorkspace(f, id)); + const closedAt = new Date(Date.now() - DAY_MS); + await child(f, { status: "done", completedAt: closedAt, executionWorkspaceId: f.workspaceId }); + const early = worker(); + expect(await early.sweepTerminalWorkspaces()).toMatchObject({ archived: 0 }); + expect(await exists(f.cwd)).toBe(true); + expect(await worker(closedAt.getTime() + 3 * DAY_MS + 60_000).sweepTerminalWorkspaces()).toMatchObject({ archived: 1 }); + expect(await exists(f.cwd)).toBe(false); + }); + + it("never compacts a closed issue's worktree while children are open", async () => { + const f = await fixture(); + await makeUnsavedWork(f.cwd); + await db.update(executionWorkspaces).set({ providerType: "git_worktree" }).where(eq(executionWorkspaces.id, f.workspaceId)); + await db.update(issues).set({ completedAt: new Date(Date.now() - 2 * DAY_MS) }).where(eq(issues.id, f.issueId)); + await fs.mkdir(path.join(f.cwd, "node_modules", "pkg"), { recursive: true }); + await fs.writeFile(path.join(f.cwd, "node_modules", "pkg", "index.js"), "module.exports = 1;\n"); + await child(f, { status: "in_progress" }).then(id => ownWorkspace(f, id)); + for (const reaper of [executionWorkspaceService(db, { workspaceCompactionDelayMs: 0 }), + executionWorkspaceService(db, { workspaceCompactionDelayMs: 0, gitCacheExpirationEnabled: true, + gitRestorePoints: { storage: getStorageService(), terminalDelayMs: 3 * DAY_MS, keyDurable: true } })]) { + expect(await reaper.sweepTerminalWorkspaces()).toMatchObject({ archived: 0, compacted: 0 }); + expect(await exists(path.join(f.cwd, "node_modules", "pkg", "index.js"))).toBe(true); + } + }); + + it("keeps today's reopen fence handling while a reopen is pending", async () => { + const f = await fixture(); + await makeUnsavedWork(f.cwd); + await child(f, { status: "in_progress" }).then(id => ownWorkspace(f, id)); + await db.update(executionWorkspaces).set({ metadata: setMetadataReopenPendingConsumption((await f.row()).metadata, new Date()) }) + .where(eq(executionWorkspaces.id, f.workspaceId)); + expect(await worker().sweepTerminalWorkspaces()).toMatchObject({ archived: 0 }); + expect(await exists(f.cwd)).toBe(true); + expect((await f.row()).metadata?.reopenPendingConsumption).toBeUndefined(); + }); + + it("keeps the worktree while an open child is bound to it", async () => { + const f = await fixture(); + await makeUnsavedWork(f.cwd); + await child(f, { status: "blocked", executionWorkspaceId: f.workspaceId, executionWorkspacePreference: "reuse_existing" }); + expect(await f.expire()).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" }); + expect(await worker().sweepTerminalWorkspaces()).toMatchObject({ archived: 0, skippedNonTerminalTree: 1 }); + expect(await exists(f.cwd)).toBe(true); + }); + + it("keeps the worktree while an open grandchild is bound to it under a child with its own workspace", async () => { + const f = await fixture(); + const middle = await child(f, { status: "in_review" }); + await ownWorkspace(f, middle); + await child(f, { parentId: middle, status: "todo", executionWorkspaceId: f.workspaceId }); + expect(await f.expire()).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" }); + expect(await exists(f.cwd)).toBe(true); + }); + + it("keeps the worktree for a child created later that inherits it with reuse_existing", async () => { + const f = await fixture(); + const created = await issueService(db).create(f.companyId, { parentId: f.issueId, title: "Inherited follow-up", status: "todo" }); + expect(created).toMatchObject({ executionWorkspaceId: f.workspaceId, executionWorkspacePreference: "reuse_existing" }); + expect(await f.expire()).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" }); + expect(await worker().sweepTerminalWorkspaces()).toMatchObject({ archived: 0, skippedNonTerminalTree: 1 }); + expect(await exists(f.cwd)).toBe(true); + }); + + it("keeps the worktree while an open child selects it as its existing workspace", async () => { + const f = await fixture(); + const selection = { kind: "existing" as const, workspaceId: f.workspaceId }; + const selecting = await child(f, { status: "todo", workspaceSelection: { version: 1, selection, source: "explicit" } }); + expect(await f.expire()).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" }); + await db.update(issues).set({ workspaceSelection: null, workspacePendingSelection: { version: 1, requestKey: "fixture", + expectedBindingRevision: 0, intent: { version: 1, selection, source: "explicit" } } }).where(eq(issues.id, selecting)); + expect(await f.expire()).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" }); + await db.update(issues).set({ workspacePendingSelection: null }).where(eq(issues.id, selecting)); + expect(await f.expire()).toMatchObject({ expired: true }); + }); + + it("keeps the worktree while a child run referencing it is live", async () => { + const f = await fixture(); + const running = await child(f, { status: "in_progress" }); + await ownWorkspace(f, running); + const agentId = randomUUID(); + await db.insert(agents).values({ id: agentId, companyId: f.companyId, name: "Worker", adapterType: "process" }); + const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId, status: "running", startedAt: new Date(), + contextSnapshot: { issueId: running, executionWorkspaceId: f.workspaceId } }).returning(); + expect(await f.expire()).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" }); + expect(await exists(f.cwd)).toBe(true); + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, run!.id)); + expect(await f.expire()).toMatchObject({ expired: true }); + }); + + it("keeps the worktree while the closed issue was closed inside the terminal delay", async () => { + const f = await fixture(); + await db.update(issues).set({ completedAt: new Date(Date.now() - 60_000) }).where(eq(issues.id, f.issueId)); + await child(f, { status: "in_progress" }).then(id => ownWorkspace(f, id)); + expect(await f.expire()).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" }); + expect(await exists(f.cwd)).toBe(true); + }); + + it("restores the saved state when an open child later binds to the expired workspace", async () => { + const f = await fixture(); + const unpushed = await makeUnsavedWork(f.cwd); + const follower = await child(f, { status: "in_progress" }); + expect(await f.expire()).toMatchObject({ expired: true, restorePoint: true }); + await db.update(issues).set({ executionWorkspaceId: f.workspaceId, executionWorkspacePreference: "reuse_existing" }).where(eq(issues.id, follower)); + const reopened = await executionWorkspaceService(db).reopenClosedIsolatedExecutionWorkspaceForIssue({ workspaceId: f.workspaceId, + issue: { id: follower, companyId: f.companyId, projectId: null }, actor: { actorType: "user", agentId: null }, + authorizationActor: { type: "board", source: "local_implicit", isInstanceAdmin: true } }); + expect(reopened).toMatchObject({ ok: true, reopened: true }); + expect(git(f.cwd, "rev-parse", "HEAD")).toBe(unpushed); + expect(await fs.readFile(path.join(f.cwd, "marker.txt"), "utf8")).toBe("unstaged edit\n"); + expect(git(f.cwd, "show", ":marker.txt")).toBe("staged edit"); + expect(await fs.readFile(path.join(f.cwd, "notes", "todo.md"), "utf8")).toBe("untracked work\n"); + }); + }); + it("expires a worktree whose only extra files are ignored, and restores them on reopen", async () => { const f = await fixture(); await fs.mkdir(path.join(f.cwd, "ui"), { recursive: true }); diff --git a/server/src/__tests__/git-workspace-expiration.test.ts b/server/src/__tests__/git-workspace-expiration.test.ts index 7055ede59d1..80a1b510058 100644 --- a/server/src/__tests__/git-workspace-expiration.test.ts +++ b/server/src/__tests__/git-workspace-expiration.test.ts @@ -816,7 +816,8 @@ describeEmbeddedPostgres("reconstructable Git expiration", () => { it("preserves live descendants and pending reopening", async () => { const f = await fixture(); - await db.insert(issues).values({ companyId: f.companyId, title: "Active child", status: "in_progress", parentId: f.issueId }); + await db.insert(issues).values({ companyId: f.companyId, title: "Active child", status: "in_progress", parentId: f.issueId, + executionWorkspaceId: f.workspaceId, executionWorkspacePreference: "reuse_existing" }); expect((await f.expire()).expired).toBe(false); await db.delete(issues).where(eq(issues.parentId, f.issueId)); const row = await f.row(); diff --git a/server/src/__tests__/heartbeat-git-restore-point-resume.test.ts b/server/src/__tests__/heartbeat-git-restore-point-resume.test.ts index dc5ebb6d3f2..9620083333c 100644 --- a/server/src/__tests__/heartbeat-git-restore-point-resume.test.ts +++ b/server/src/__tests__/heartbeat-git-restore-point-resume.test.ts @@ -68,7 +68,7 @@ describe.skipIf(!support.supported)("heartbeat resume of an idle-expired workspa await fs.rm(root, { recursive: true, force: true }); }, 60_000); - async function seed() { + async function seed(options: { closedParent?: boolean } = {}) { const home = path.join(root, randomUUID()); vi.stubEnv("PAPERCLIP_HOME", home); vi.stubEnv("PAPERCLIP_INSTANCE_ID", "default"); @@ -108,24 +108,31 @@ describe.skipIf(!support.supported)("heartbeat resume of an idle-expired workspa isolatedRemoteSource: { schema: "paperclip.isolated-remote-source.v1", companyId, issueId, repoUrl } } }); await db.update(issues).set({ executionWorkspaceId: workspaceId, executionWorkspacePreference: "reuse_existing", updatedAt: idleSince }) .where(eq(issues.id, issueId)); + const childId = randomUUID(); + if (options.closedParent) { + await db.update(issues).set({ status: "done", completedAt: new Date(Date.now() - 4 * DAY_MS) }).where(eq(issues.id, issueId)); + await db.insert(issues).values({ id: childId, companyId, projectId, parentId: issueId, title: "Open follow-up", status: "in_progress", + workMode: "standard", assigneeAgentId: agentId, responsibleUserId: "responsible-user", issueNumber: 2, identifier: `${issuePrefix}-2`, + executionWorkspaceSettings: { mode: "isolated_workspace" } }); + } await fs.writeFile(path.join(realized.cwd, "marker.txt"), "commit B (unpushed)\n"); git(realized.cwd, "commit", "-qam", "B"); const unpushed = git(realized.cwd, "rev-parse", "HEAD"); await fs.writeFile(path.join(realized.cwd, "notes.md"), "uncommitted notes\n"); const workspace = () => db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId)).then(rows => rows[0]!); const expiration = await expireReconstructableGitWorkspace(db, await workspace(), { - dryRun: false, expireBase: false, now: new Date(), idleCutoff: new Date(Date.now() - 7 * DAY_MS), + dryRun: false, expireBase: false, now: new Date(), idleCutoff: options.closedParent ? undefined : new Date(Date.now() - 7 * DAY_MS), restorePoint: { storage: getStorageService(), notBefore: new Date(Date.now() - DAY_MS), keyDurable: true }, }); - const queueRun = async () => { + const queueRun = async (target = issueId) => { await db.insert(agentWakeupRequests).values({ id: wakeupId, companyId, agentId, source: "assignment", triggerDetail: "system", - reason: "issue_assigned", payload: { issueId }, status: "queued", runId }); + reason: "issue_assigned", payload: { issueId: target }, status: "queued", runId }); await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "assignment", triggerDetail: "system", status: "queued", - wakeupRequestId: wakeupId, responsibleUserId: "responsible-user", contextSnapshot: { issueId, taskId: issueId, wakeReason: "issue_assigned" } }); + wakeupRequestId: wakeupId, responsibleUserId: "responsible-user", contextSnapshot: { issueId: target, taskId: target, wakeReason: "issue_assigned" } }); await db.update(issues).set({ checkoutRunId: runId, executionRunId: runId, executionAgentNameKey: "resumer", executionLockedAt: new Date() }) - .where(eq(issues.id, issueId)); + .where(eq(issues.id, target)); }; - return { issueId, workspaceId, runId, cwd: realized.cwd, unpushed, expiration, workspace, queueRun }; + return { issueId, childId, workspaceId, runId, cwd: realized.cwd, unpushed, expiration, workspace, queueRun }; } async function waitForRun(heartbeat: ReturnType, runId: string) { @@ -159,4 +166,23 @@ describe.skipIf(!support.supported)("heartbeat resume of an idle-expired workspa const [issue] = await db.select().from(issues).where(eq(issues.id, seeded.issueId)); expect(issue.executionWorkspaceId).toBe(seeded.workspaceId); }); + + it("restores a closed parent's expired workspace when an open child binds to it", async () => { + const seeded = await seed({ closedParent: true }); + expect(seeded.expiration).toMatchObject({ expired: true, restorePoint: true }); + await expect(fs.access(seeded.cwd)).rejects.toMatchObject({ code: "ENOENT" }); + await db.update(issues).set({ executionWorkspaceId: seeded.workspaceId, executionWorkspacePreference: "reuse_existing" }) + .where(eq(issues.id, seeded.childId)); + await seeded.queueRun(seeded.childId); + const heartbeat = heartbeatService(db); + await heartbeat.resumeQueuedRuns(); + const run = await waitForRun(heartbeat, seeded.runId); + await heartbeat.waitForRunExecutionDrain(seeded.runId); + expect(run).toMatchObject({ status: "succeeded" }); + expect(adapterExecute.mock.calls[0]?.[0]).toMatchObject({ context: { executionWorkspaceId: seeded.workspaceId, + paperclipWorkspace: { cwd: seeded.cwd } } }); + expect(git(seeded.cwd, "rev-parse", "HEAD")).toBe(seeded.unpushed); + expect(await fs.readFile(path.join(seeded.cwd, "notes.md"), "utf8")).toBe("uncommitted notes\n"); + expect((await seeded.workspace()).status).not.toBe("archived"); + }); }); diff --git a/server/src/services/execution-workspaces.ts b/server/src/services/execution-workspaces.ts index 9d42edcc6e1..caf9a90511c 100644 --- a/server/src/services/execution-workspaces.ts +++ b/server/src/services/execution-workspaces.ts @@ -5,7 +5,7 @@ import { measureRunTransaction } from "./execution-transaction-timing.js"; import { measureRunPhase } from "./execution-preparation-timing.js"; import { assertExecutionWorkspaceRecordAccess, assertTaskWorkspaceExecutionAccess, assertTaskWorkspaceSourceProjectAccess } from "./task-workspace-access.js"; import { verifiedRecordedIsolatedRemoteSource, withIsolatedRepositorySourceLock } from "./isolated-remote-workspace-source.js"; -import { expireReconstructableGitWorkspace, gitWorkspaceLastActivityAt, hasPendingGitExpiration } from "./git-workspace-expiration.js"; +import { expireReconstructableGitWorkspace, gitWorkspaceLastActivityAt, hasPendingGitExpiration, listIssuesUsingGitWorkspace } from "./git-workspace-expiration.js"; import { hasPendingIsolatedRepositoryRehomeCleanup, isIsolatedRepositoryRehomeCandidate, rehomeExpiredIsolatedRestoration, rehomeIsolatedRepositoryWorkspace, sweepOrphanedIsolatedRepositorySources } from "./isolated-repository-rehome.js"; import { hasPendingNestedCloneMigration, isNestedCloneMigrationCandidate, nextNestedCloneMigrationBatch, pendingNestedCloneMigrationNames, listNestedCloneMigrationNames, migrateNestedRepositoryClone, NESTED_CLONE_MIGRATION_RETRY_REASONS, type NestedCloneMigrationRotation } from "./nested-clone-migration.js"; @@ -3434,7 +3434,14 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic workspace.metadata as Record | null, ); if (reopenPending) terminalWorkspaceDeadlines.set(workspace.id, now().getTime()); - if (!sourceIssue || issueTree.some(issue => !TERMINAL_ISSUE_STATUSES.has(issue.status))) { + const treeOpen = issueTree.some(issue => !TERMINAL_ISSUE_STATUSES.has(issue.status)); + const gitExpirationCandidate = workspace.providerType !== "plugin" && Boolean(opts.gitCacheExpirationEnabled) + && (hasPendingGitExpiration(workspace) || (workspace.mode === "isolated_workspace" && workspace.strategyType === "git_worktree")); + const usingIssueIds = sourceIssue && gitExpirationCandidate && !reopenPending + && TERMINAL_ISSUE_STATUSES.has(sourceIssue.status) + ? await listIssuesUsingGitWorkspace(db, workspace, issueTree.map(issue => issue.id)) : null; + const expirationTree = usingIssueIds ? issueTree.filter(issue => usingIssueIds.has(issue.id)) : issueTree; + if (!sourceIssue || expirationTree.some(issue => !TERMINAL_ISSUE_STATUSES.has(issue.status))) { if (reopenPending) { await clearReopenPendingConsumptionUnderLock(workspace.id, { expectedGeneration: readExecutionWorkspaceLifecycleGeneration( @@ -3473,11 +3480,13 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic result.skippedNonTerminalTree += 1; continue; } - const latestTerminalAt = Math.max(...issueTree.map(issue => issueTerminalTimestamp(issue)!.getTime())); + const expirationTerminalAt = Math.max(...expirationTree.map(issue => issueTerminalTimestamp(issue)!.getTime())); + const latestTerminalAt = treeOpen ? expirationTerminalAt : Math.max(...issueTree.map(issue => issueTerminalTimestamp(issue)!.getTime())); // Compaction runs once per terminal cycle, after its own delay and // before the archive cooldown, so a workspace the reaper cannot archive // yet (or ever) stops holding regenerable files. const compactionPending = workspaceCompactionDelayMs !== null + && !treeOpen && !isClosedExecutionWorkspaceStatus(workspace.status) && !reopenPending && compactedTerminalAt.get(workspace.id) !== latestTerminalAt; @@ -3503,14 +3512,13 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic ); } } - if (workspace.providerType !== "plugin" && opts.gitCacheExpirationEnabled && (hasPendingGitExpiration(workspace) - || (workspace.mode === "isolated_workspace" && workspace.strategyType === "git_worktree"))) { + if (gitExpirationCandidate) { const expiration = await expireReconstructableGitWorkspace(db, workspace, { dryRun: opts.gitCacheExpirationDryRun ?? false, expireBase: opts.gitBaseExpirationEnabled ?? false, now: now(), signal, beforeDelete: opts.beforeTerminalWorkspaceCleanup, restorePoint: opts.gitRestorePoints ? { storage: opts.gitRestorePoints.storage, - notBefore: new Date(latestTerminalAt + opts.gitRestorePoints.terminalDelayMs), + notBefore: new Date(expirationTerminalAt + opts.gitRestorePoints.terminalDelayMs), keyDurable: opts.gitRestorePoints.keyDurable, } : undefined, }); diff --git a/server/src/services/git-workspace-expiration.ts b/server/src/services/git-workspace-expiration.ts index ff9e6c7ffc5..c551f7471a5 100644 --- a/server/src/services/git-workspace-expiration.ts +++ b/server/src/services/git-workspace-expiration.ts @@ -3,7 +3,7 @@ import fs from "node:fs/promises"; import path from "node:path"; import { randomBytes } from "node:crypto"; import { acquireGitExpirationLocks } from "./git-workspace-expiration-locks.js"; -import { and, eq, ne, sql } from "drizzle-orm"; +import { and, eq, inArray, ne, sql } from "drizzle-orm"; import { executionWorkspaces, issues, heartbeatRuns, nativeRunFinalizations, environmentLeases, workspaceRuntimeServices, issueRecoveryActions, projectWorkspaces, withDedicatedDbConnection, type Db } from "@paperclipai/db"; import { createGitRemoteAuthProvider } from "./git-credentials.js"; @@ -124,6 +124,21 @@ export function expirationScope(workspace: Workspace) { )`; } +function issueUsesWorkspace(workspace: Pick) { + return sql`(${issues.id} = ${workspace.sourceIssueId} OR ${issues.executionWorkspaceId} = ${workspace.id} + OR (${issues.workspaceSelection}->'selection'->>'kind' = 'existing' + AND ${issues.workspaceSelection}->'selection'->>'workspaceId' = ${workspace.id}) + OR (${issues.workspacePendingSelection}->'intent'->'selection'->>'kind' = 'existing' + AND ${issues.workspacePendingSelection}->'intent'->'selection'->>'workspaceId' = ${workspace.id}))`; +} + +export async function listIssuesUsingGitWorkspace(db: Db | Tx, workspace: Pick, issueIds: string[]) { + if (issueIds.length === 0) return new Set(); + const rows = await db.select({ id: issues.id }).from(issues) + .where(and(eq(issues.companyId, workspace.companyId), inArray(issues.id, issueIds), issueUsesWorkspace(workspace))); + return new Set(rows.map(row => row.id)); +} + export async function gitWorkspaceLastActivityAt(db: Db | Tx, workspace: Workspace): Promise { if (!workspace.sourceIssueId) return null; const rows = await db.execute<{ at: string | Date | null }>(sql`${expirationScope(workspace)} @@ -142,8 +157,10 @@ export async function isGitWorkspaceExpirationEligible(tx: Tx, workspace: Worksp NOT EXISTS (SELECT 1 FROM tree WHERE updated_at > ${idleCutoff.toISOString()}::timestamptz) AND NOT EXISTS (SELECT 1 FROM bound WHERE updated_at > ${idleCutoff.toISOString()}::timestamptz) AND NOT EXISTS (SELECT 1 FROM relevant_runs WHERE greatest(created_at, started_at, finished_at) > ${idleCutoff.toISOString()}::timestamptz)` : sql` - NOT EXISTS (SELECT 1 FROM tree WHERE status NOT IN ('done', 'cancelled') - OR coalesce(CASE WHEN status = 'done' THEN completed_at ELSE cancelled_at END, updated_at) > ${new Date(now.getTime() - DAY_MS).toISOString()}::timestamptz) + NOT EXISTS (SELECT 1 FROM ${issues} WHERE ${issues.companyId} = ${workspace.companyId} + AND ${issues.id} IN (SELECT id FROM tree) AND ${issueUsesWorkspace(workspace)} + AND (${issues.status} NOT IN ('done', 'cancelled') + OR coalesce(CASE WHEN ${issues.status} = 'done' THEN ${issues.completedAt} ELSE ${issues.cancelledAt} END, ${issues.updatedAt}) > ${new Date(now.getTime() - DAY_MS).toISOString()}::timestamptz)) AND NOT EXISTS (SELECT 1 FROM ${issues} WHERE company_id = ${workspace.companyId} AND execution_workspace_id = ${workspace.id} AND status NOT IN ('done', 'cancelled'))`; const recoveryActionBlocks = idleCutoff ? sql`true` : sql`(cause = 'native_workspace_finalization_owner_unverified'