mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-25 09:41:04 +01:00
Expire closed workspaces that open descendants do not use (#17464)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Git cache expiration removes closed issues' worktrees and saves
restore points for unsaved work
> - It treated a workspace as closed only when the source issue and
every descendant were done or cancelled
> - A routine that closes a parent and leaves children open for days
therefore kept every parent worktree, although the children work in
their own workspaces
> - The idle path did not help, because the children are updated every
day
> - This pull request counts only the issues that use the workspace when
it decides that a closed workspace can expire
> - The benefit is that closed parent worktrees are removed while their
children work elsewhere, and a child that later binds to the workspace
gets it back from the restore point
## Linked Issues or Issue Description
Refs #17271. Bug report described in-PR (bug report template).
**What happened?**
On one instance, an hourly routine creates a parent issue, works it in
its own worktree, marks the parent done, and leaves child issues open
(`blocked`, `in_review` or `in_progress`) for days. The children work in
their own workspaces. They are not bound to the parent's workspace. Git
cache expiration kept every parent worktree (about 300 MB each), because
the closed-tree check and the workspace reaper required all descendants
to be closed. In one project, 18 of 19 long-closed parent worktrees were
kept only by open children that did not use them. The idle path never
ran, because the children were updated daily.
**Expected behavior**
A closed issue's worktree expires while descendants are open, as long as
no open descendant uses that workspace.
**Steps to reproduce**
1. Enable git cache expiration and restore points.
2. Close an issue that has an isolated git worktree, and leave a child
issue open that works in its own workspace (or has no workspace
binding).
3. Wait past the terminal delay and run the workspace reaper: the parent
worktree stays, and the reaper counts it as `skippedNonTerminalTree`.
**Paperclip version or commit:** `master` at 410e5c6eed.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
- `isGitWorkspaceExpirationEligible` (closed path, not the idle path):
the "tree closed for one day" check now covers only the issues in the
tree that use the workspace. An issue uses the workspace when it is the
source issue, when its `execution_workspace_id` is the workspace, or
when its current `workspace_selection` or its
`workspace_pending_selection` selects it as the `existing` workspace.
Inheritance from a parent (`reuse_existing`) and `existing` selections
are both stored as `execution_workspace_id`, and heartbeat workspace
resolution reuses an existing workspace only through that binding or a
run's persisted native binding. An open descendant without one of these
references gets a new workspace on its next run, so it no longer keeps
the parent's worktree.
- All other checks stay the same: any open issue bound to the workspace,
live or unsettled runs of any issue in the tree or of the workspace
(including a run whose context names the workspace), finalizations,
leases, recovery actions, runtime services, shared cwd and the reopen
fence.
- Workspace reaper: when the source issue is closed, git cache
expiration applies, and no reopen is pending, the reaper routes the
workspace by the same rule. It takes the restore point due time from the
source issue and the descendants that use the workspace, also after
every descendant is closed. A recently closed child that used the
workspace still holds it for the terminal delay. The compaction delay
and the archive cooldown still use the whole tree when it is closed.
- Workspace reaper: compaction and the normal archive path keep their
behavior. While any descendant is open, the reaper does not compact the
worktree, and the expiration branch handles the workspace and continues.
The archive path still requires a closed subtree. With expiration off,
or a pending reopen, routing is unchanged.
- New exported helper `listIssuesUsingGitWorkspace` shares the SQL "uses
the workspace" predicate between eligibility and the reaper.
- Docs: `doc/recoverable-disk-cleanup.md` and the
`PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS` entry in
`doc/DEVELOPING.md`.
- The existing `preserves live descendants and pending reopening` case
now binds its open child to the workspace. An unbound open child no
longer keeps the worktree by design.
## Verification
- `pnpm exec vitest run
src/__tests__/git-restore-point-expiration.test.ts
src/__tests__/git-workspace-expiration.test.ts
src/__tests__/heartbeat-git-restore-point-resume.test.ts` in `server/`:
all pass.
- `pnpm exec vitest run src/__tests__/git-workspace-expiration
src/__tests__/git-restore-point src/__tests__/execution-workspace
src/__tests__/heartbeat-git-restore-point-resume
src/services/heartbeat/environment-selection
src/__tests__/isolated-repository` in `server/`: 15 of 17 files passed
in one parallel run. The other failures were the updated descendant case
(now fixed) and 45-second timeouts in restore point tests while 17
embedded Postgres files ran at the same time. Both files pass when run
on their own.
- `pnpm run typecheck` in `server/`: passes.
- New embedded Postgres cases with real git repositories:
- A closed parent with open `blocked`, `in_review`, `in_progress` and
`todo` children and a grandchild in their own workspaces (or with none):
the worktree expires with a restore point and the reopen restores it
exactly.
- The reaper archives it, timed from the issues that use the workspace.
A child closed one minute ago with its own workspace does not delay it.
- A recently closed child bound to the workspace holds it until its
terminal delay passes.
- When every child is closed and the most recent one used its own
workspace, the reaper does not restart the restore point delay.
- The worktree is kept when an open child is bound to it, when an open
grandchild under a child with its own workspace is bound to it, when a
child created later with `issueService.create` inherits it with
`reuse_existing`, when an open child selects it through
`workspace_selection` or `workspace_pending_selection`, when a child run
that references it is live, and when the parent was closed inside the
delay.
- The reaper never compacts the worktree while children are open, with
expiration on or off. A pending reopen keeps its current fence handling.
- After expiry, an open child that binds to the workspace reopens it
with the saved commits, index, working tree and untracked files.
- Heartbeat: a queued run for an open child that binds to the expired
workspace of a closed parent restores the workspace and runs the agent
in it with the saved state.
- Mutation checks (each run against the two restore point test files):
- Closed-tree check restored to the whole tree: 7 tests fail.
- `execution_workspace_id` removed from the rule: 3 fail.
- `workspace_selection` removed from the rule: 1 fails.
- `workspace_pending_selection` removed from the rule: 1 fails.
- Reaper `latestTerminalAt` from the whole tree: 3 fail.
- Reaper routing by the whole tree: 2 fail.
- Reaper restore point due time from the whole tree once it is closed: 1
fails.
- Pending-reopen gate removed: 1 fails.
- Compaction gate (`!treeOpen`) removed: no test fails. Compaction's own
guard under the lifecycle lock still rejects open trees, so the gate
only skips work that cannot succeed.
- Not run: the full server suite and the UI suites. CI covers them.
## Risks
- Closed parent worktrees now expire while their children are open. A
child that later binds to the workspace depends on the restore point and
on the run-side reopen from #17271. The new tests cover this path. When
restore points are off, only exactly reproducible worktrees expire.
- An open child with no workspace binding no longer keeps the parent's
worktree. Heartbeat resolution never reuses a workspace without a
binding, so this does not change where the child runs.
- The rule is conservative. A workspace selected only by an issue's
current or pending selection is kept, and live runs of any issue in the
tree still block.
- The reaper runs one extra indexed query for each closed workspace when
git cache expiration applies. No migration.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
3a2c56780c
commit
fe85fd7ac4
7 files changed
+259
-20
No files matched your search
+2
-2
@@ -2618,8 +2618,8 @@ Environment overrides:
|
||||
- `PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true` lets git cache expiration remove closed
|
||||
worktrees with unpushed or uncommitted work after saving an encrypted restore point
|
||||
(see `doc/recoverable-disk-cleanup.md`). It is off by default.
|
||||
`PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS=<days>` sets how long the issue tree
|
||||
must be closed first. The default is `3`. A negative, empty, or non-numeric value falls
|
||||
`PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS=<days>` sets how long the issues that
|
||||
use the workspace (the source issue and descendants bound to it) must be closed first. The default is `3`. A negative, empty, or non-numeric value falls
|
||||
back to the default.
|
||||
`PAPERCLIP_GIT_RESTORE_POINT_IDLE_DAYS=<days>` sets how long an open issue tree's
|
||||
worktree must go untouched before it is removed the same way; runs restore it
|
||||
|
||||
@@ -158,6 +158,17 @@ closed issue's worktree only when its remote can reproduce it exactly. With
|
||||
`PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true`, it can also remove a worktree that has
|
||||
unpushed commits, uncommitted or untracked files, or history-only commits, once its issue
|
||||
tree has been closed for `PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS` (default `3`).
|
||||
For this, only the issues in the tree that use the workspace count: the source issue, and
|
||||
each descendant that is bound to it (`execution_workspace_id`, which is how a child that
|
||||
inherits the parent's workspace with `reuse_existing` is recorded) or that selects it as
|
||||
its current or pending existing workspace. The delay runs from the latest close time among
|
||||
those issues. A closed issue's worktree can therefore be removed while descendants that
|
||||
work in their own workspaces stay open. Any issue still bound to the workspace, and any
|
||||
live run, finalization, lease, recovery action or runtime service of an issue in the tree
|
||||
or of the workspace, still keeps it. If an open descendant later binds to the removed
|
||||
workspace, its next run or a comment reopens it from the restore point, as below. The
|
||||
workspace reaper does not compact or archive such a worktree by its normal rules while
|
||||
descendants are open; only expiration removes it.
|
||||
It also removes the worktree of an issue tree that is still open once nothing has
|
||||
touched it for `PAPERCLIP_GIT_RESTORE_POINT_IDLE_DAYS` (default `7`, `0` turns this off):
|
||||
no issue in the tree or bound to the workspace was updated and no run for those issues or
|
||||
|
||||
@@ -14,7 +14,8 @@ import { executionWorkspaceService } from "../services/execution-workspaces.js";
|
||||
import { expireReconstructableGitWorkspace, gitWorkspaceLastActivityAt } from "../services/git-workspace-expiration.js";
|
||||
import { getStorageService } from "../storage/index.js";
|
||||
import { markGitRestorePointExpired } from "../services/git-restore-point-storage.js";
|
||||
import { GIT_RESTORE_POINT_EXPIRED_WARNING } from "../services/execution-workspaces.js";
|
||||
import { GIT_RESTORE_POINT_EXPIRED_WARNING, setMetadataReopenPendingConsumption } from "../services/execution-workspaces.js";
|
||||
import { issueService } from "../services/issues.js";
|
||||
|
||||
const gitEnv = { ...process.env, GIT_AUTHOR_NAME: "Fixture", GIT_AUTHOR_EMAIL: "fixture@example.test",
|
||||
GIT_COMMITTER_NAME: "Fixture", GIT_COMMITTER_EMAIL: "fixture@example.test", GIT_TERMINAL_PROMPT: "0" };
|
||||
@@ -356,6 +357,181 @@ describeEmbeddedPostgres("git restore point expiration", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("closed source issue with open descendants", () => {
|
||||
async function child(f: Awaited<ReturnType<typeof fixture>>, values: Partial<typeof issues.$inferInsert> = {}) {
|
||||
const id = randomUUID();
|
||||
await db.insert(issues).values({ id, companyId: f.companyId, parentId: f.issueId, title: "Follow-up", status: "blocked", ...values });
|
||||
return id;
|
||||
}
|
||||
async function ownWorkspace(f: Awaited<ReturnType<typeof fixture>>, sourceIssueId: string) {
|
||||
const id = randomUUID();
|
||||
await db.insert(executionWorkspaces).values({ id, companyId: f.companyId, sourceIssueId, mode: "isolated_workspace", strategyType: "git_worktree",
|
||||
name: "Child workspace", status: "active", providerType: "local_fs", cwd: path.join(root, "child-workspaces", id), providerRef: path.join(root, "child-workspaces", id) });
|
||||
await db.update(issues).set({ executionWorkspaceId: id, executionWorkspacePreference: "reuse_existing" }).where(eq(issues.id, sourceIssueId));
|
||||
return id;
|
||||
}
|
||||
const worker = (now = Date.now()) => executionWorkspaceService(db, { gitCacheExpirationEnabled: true, now: () => new Date(now),
|
||||
gitRestorePoints: { storage: getStorageService(), terminalDelayMs: 3 * DAY_MS, keyDurable: true, idleMs: 7 * DAY_MS } });
|
||||
|
||||
it("expires the closed issue's worktree while open children work in their own workspaces", async () => {
|
||||
const f = await fixture();
|
||||
const unpushed = await makeUnsavedWork(f.cwd);
|
||||
const blocked = await child(f, { status: "blocked" });
|
||||
await ownWorkspace(f, blocked);
|
||||
const reviewing = await child(f, { status: "in_review" });
|
||||
await ownWorkspace(f, reviewing);
|
||||
const grandchild = await child(f, { parentId: reviewing, status: "in_progress" });
|
||||
await ownWorkspace(f, grandchild);
|
||||
await child(f, { status: "todo" });
|
||||
expect(await f.expire()).toMatchObject({ candidate: true, expired: true, restorePoint: true, skipReason: null });
|
||||
expect(await exists(f.cwd)).toBe(false);
|
||||
expect(await f.ledger()).toMatchObject([{ status: "active" }]);
|
||||
expect(await f.reopen()).toMatchObject({ ok: true, reopened: true });
|
||||
expect(git(f.cwd, "rev-parse", "HEAD")).toBe(unpushed);
|
||||
expect(await fs.readFile(path.join(f.cwd, "notes", "todo.md"), "utf8")).toBe("untracked work\n");
|
||||
});
|
||||
|
||||
it("lets the workspace reaper expire it, timed from the issues that use the workspace", async () => {
|
||||
const f = await fixture();
|
||||
await makeUnsavedWork(f.cwd);
|
||||
const open = await child(f, { status: "in_progress", updatedAt: new Date() });
|
||||
await ownWorkspace(f, open);
|
||||
const closed = await child(f, { status: "done", completedAt: new Date(Date.now() - 60_000) });
|
||||
await ownWorkspace(f, closed);
|
||||
expect(await worker().sweepTerminalWorkspaces()).toMatchObject({ archived: 1 });
|
||||
expect(await exists(f.cwd)).toBe(false);
|
||||
expect((await f.row()).status).toBe("archived");
|
||||
expect(await f.ledger()).toMatchObject([{ status: "active" }]);
|
||||
});
|
||||
|
||||
it("times the restore point from the issues that use the workspace once every child is closed", async () => {
|
||||
const f = await fixture();
|
||||
await makeUnsavedWork(f.cwd);
|
||||
const closed = await child(f, { status: "done", completedAt: new Date(Date.now() - 60_000) });
|
||||
await ownWorkspace(f, closed);
|
||||
expect(await worker().sweepTerminalWorkspaces()).toMatchObject({ archived: 1 });
|
||||
expect(await exists(f.cwd)).toBe(false);
|
||||
expect(await f.ledger()).toMatchObject([{ status: "active" }]);
|
||||
});
|
||||
|
||||
it("waits for the terminal delay of a recently closed child that used the workspace", async () => {
|
||||
const f = await fixture();
|
||||
await makeUnsavedWork(f.cwd);
|
||||
await child(f, { status: "in_progress" }).then(id => ownWorkspace(f, id));
|
||||
const closedAt = new Date(Date.now() - DAY_MS);
|
||||
await child(f, { status: "done", completedAt: closedAt, executionWorkspaceId: f.workspaceId });
|
||||
const early = worker();
|
||||
expect(await early.sweepTerminalWorkspaces()).toMatchObject({ archived: 0 });
|
||||
expect(await exists(f.cwd)).toBe(true);
|
||||
expect(await worker(closedAt.getTime() + 3 * DAY_MS + 60_000).sweepTerminalWorkspaces()).toMatchObject({ archived: 1 });
|
||||
expect(await exists(f.cwd)).toBe(false);
|
||||
});
|
||||
|
||||
it("never compacts a closed issue's worktree while children are open", async () => {
|
||||
const f = await fixture();
|
||||
await makeUnsavedWork(f.cwd);
|
||||
await db.update(executionWorkspaces).set({ providerType: "git_worktree" }).where(eq(executionWorkspaces.id, f.workspaceId));
|
||||
await db.update(issues).set({ completedAt: new Date(Date.now() - 2 * DAY_MS) }).where(eq(issues.id, f.issueId));
|
||||
await fs.mkdir(path.join(f.cwd, "node_modules", "pkg"), { recursive: true });
|
||||
await fs.writeFile(path.join(f.cwd, "node_modules", "pkg", "index.js"), "module.exports = 1;\n");
|
||||
await child(f, { status: "in_progress" }).then(id => ownWorkspace(f, id));
|
||||
for (const reaper of [executionWorkspaceService(db, { workspaceCompactionDelayMs: 0 }),
|
||||
executionWorkspaceService(db, { workspaceCompactionDelayMs: 0, gitCacheExpirationEnabled: true,
|
||||
gitRestorePoints: { storage: getStorageService(), terminalDelayMs: 3 * DAY_MS, keyDurable: true } })]) {
|
||||
expect(await reaper.sweepTerminalWorkspaces()).toMatchObject({ archived: 0, compacted: 0 });
|
||||
expect(await exists(path.join(f.cwd, "node_modules", "pkg", "index.js"))).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps today's reopen fence handling while a reopen is pending", async () => {
|
||||
const f = await fixture();
|
||||
await makeUnsavedWork(f.cwd);
|
||||
await child(f, { status: "in_progress" }).then(id => ownWorkspace(f, id));
|
||||
await db.update(executionWorkspaces).set({ metadata: setMetadataReopenPendingConsumption((await f.row()).metadata, new Date()) })
|
||||
.where(eq(executionWorkspaces.id, f.workspaceId));
|
||||
expect(await worker().sweepTerminalWorkspaces()).toMatchObject({ archived: 0 });
|
||||
expect(await exists(f.cwd)).toBe(true);
|
||||
expect((await f.row()).metadata?.reopenPendingConsumption).toBeUndefined();
|
||||
});
|
||||
|
||||
it("keeps the worktree while an open child is bound to it", async () => {
|
||||
const f = await fixture();
|
||||
await makeUnsavedWork(f.cwd);
|
||||
await child(f, { status: "blocked", executionWorkspaceId: f.workspaceId, executionWorkspacePreference: "reuse_existing" });
|
||||
expect(await f.expire()).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" });
|
||||
expect(await worker().sweepTerminalWorkspaces()).toMatchObject({ archived: 0, skippedNonTerminalTree: 1 });
|
||||
expect(await exists(f.cwd)).toBe(true);
|
||||
});
|
||||
|
||||
it("keeps the worktree while an open grandchild is bound to it under a child with its own workspace", async () => {
|
||||
const f = await fixture();
|
||||
const middle = await child(f, { status: "in_review" });
|
||||
await ownWorkspace(f, middle);
|
||||
await child(f, { parentId: middle, status: "todo", executionWorkspaceId: f.workspaceId });
|
||||
expect(await f.expire()).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" });
|
||||
expect(await exists(f.cwd)).toBe(true);
|
||||
});
|
||||
|
||||
it("keeps the worktree for a child created later that inherits it with reuse_existing", async () => {
|
||||
const f = await fixture();
|
||||
const created = await issueService(db).create(f.companyId, { parentId: f.issueId, title: "Inherited follow-up", status: "todo" });
|
||||
expect(created).toMatchObject({ executionWorkspaceId: f.workspaceId, executionWorkspacePreference: "reuse_existing" });
|
||||
expect(await f.expire()).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" });
|
||||
expect(await worker().sweepTerminalWorkspaces()).toMatchObject({ archived: 0, skippedNonTerminalTree: 1 });
|
||||
expect(await exists(f.cwd)).toBe(true);
|
||||
});
|
||||
|
||||
it("keeps the worktree while an open child selects it as its existing workspace", async () => {
|
||||
const f = await fixture();
|
||||
const selection = { kind: "existing" as const, workspaceId: f.workspaceId };
|
||||
const selecting = await child(f, { status: "todo", workspaceSelection: { version: 1, selection, source: "explicit" } });
|
||||
expect(await f.expire()).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" });
|
||||
await db.update(issues).set({ workspaceSelection: null, workspacePendingSelection: { version: 1, requestKey: "fixture",
|
||||
expectedBindingRevision: 0, intent: { version: 1, selection, source: "explicit" } } }).where(eq(issues.id, selecting));
|
||||
expect(await f.expire()).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" });
|
||||
await db.update(issues).set({ workspacePendingSelection: null }).where(eq(issues.id, selecting));
|
||||
expect(await f.expire()).toMatchObject({ expired: true });
|
||||
});
|
||||
|
||||
it("keeps the worktree while a child run referencing it is live", async () => {
|
||||
const f = await fixture();
|
||||
const running = await child(f, { status: "in_progress" });
|
||||
await ownWorkspace(f, running);
|
||||
const agentId = randomUUID();
|
||||
await db.insert(agents).values({ id: agentId, companyId: f.companyId, name: "Worker", adapterType: "process" });
|
||||
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId, status: "running", startedAt: new Date(),
|
||||
contextSnapshot: { issueId: running, executionWorkspaceId: f.workspaceId } }).returning();
|
||||
expect(await f.expire()).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" });
|
||||
expect(await exists(f.cwd)).toBe(true);
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, run!.id));
|
||||
expect(await f.expire()).toMatchObject({ expired: true });
|
||||
});
|
||||
|
||||
it("keeps the worktree while the closed issue was closed inside the terminal delay", async () => {
|
||||
const f = await fixture();
|
||||
await db.update(issues).set({ completedAt: new Date(Date.now() - 60_000) }).where(eq(issues.id, f.issueId));
|
||||
await child(f, { status: "in_progress" }).then(id => ownWorkspace(f, id));
|
||||
expect(await f.expire()).toMatchObject({ expired: false, skipReason: "live_or_unresolved_consumer" });
|
||||
expect(await exists(f.cwd)).toBe(true);
|
||||
});
|
||||
|
||||
it("restores the saved state when an open child later binds to the expired workspace", async () => {
|
||||
const f = await fixture();
|
||||
const unpushed = await makeUnsavedWork(f.cwd);
|
||||
const follower = await child(f, { status: "in_progress" });
|
||||
expect(await f.expire()).toMatchObject({ expired: true, restorePoint: true });
|
||||
await db.update(issues).set({ executionWorkspaceId: f.workspaceId, executionWorkspacePreference: "reuse_existing" }).where(eq(issues.id, follower));
|
||||
const reopened = await executionWorkspaceService(db).reopenClosedIsolatedExecutionWorkspaceForIssue({ workspaceId: f.workspaceId,
|
||||
issue: { id: follower, companyId: f.companyId, projectId: null }, actor: { actorType: "user", agentId: null },
|
||||
authorizationActor: { type: "board", source: "local_implicit", isInstanceAdmin: true } });
|
||||
expect(reopened).toMatchObject({ ok: true, reopened: true });
|
||||
expect(git(f.cwd, "rev-parse", "HEAD")).toBe(unpushed);
|
||||
expect(await fs.readFile(path.join(f.cwd, "marker.txt"), "utf8")).toBe("unstaged edit\n");
|
||||
expect(git(f.cwd, "show", ":marker.txt")).toBe("staged edit");
|
||||
expect(await fs.readFile(path.join(f.cwd, "notes", "todo.md"), "utf8")).toBe("untracked work\n");
|
||||
});
|
||||
});
|
||||
|
||||
it("expires a worktree whose only extra files are ignored, and restores them on reopen", async () => {
|
||||
const f = await fixture();
|
||||
await fs.mkdir(path.join(f.cwd, "ui"), { recursive: true });
|
||||
|
||||
@@ -816,7 +816,8 @@ describeEmbeddedPostgres("reconstructable Git expiration", () => {
|
||||
|
||||
it("preserves live descendants and pending reopening", async () => {
|
||||
const f = await fixture();
|
||||
await db.insert(issues).values({ companyId: f.companyId, title: "Active child", status: "in_progress", parentId: f.issueId });
|
||||
await db.insert(issues).values({ companyId: f.companyId, title: "Active child", status: "in_progress", parentId: f.issueId,
|
||||
executionWorkspaceId: f.workspaceId, executionWorkspacePreference: "reuse_existing" });
|
||||
expect((await f.expire()).expired).toBe(false);
|
||||
await db.delete(issues).where(eq(issues.parentId, f.issueId));
|
||||
const row = await f.row();
|
||||
|
||||
@@ -68,7 +68,7 @@ describe.skipIf(!support.supported)("heartbeat resume of an idle-expired workspa
|
||||
await fs.rm(root, { recursive: true, force: true });
|
||||
}, 60_000);
|
||||
|
||||
async function seed() {
|
||||
async function seed(options: { closedParent?: boolean } = {}) {
|
||||
const home = path.join(root, randomUUID());
|
||||
vi.stubEnv("PAPERCLIP_HOME", home);
|
||||
vi.stubEnv("PAPERCLIP_INSTANCE_ID", "default");
|
||||
@@ -108,24 +108,31 @@ describe.skipIf(!support.supported)("heartbeat resume of an idle-expired workspa
|
||||
isolatedRemoteSource: { schema: "paperclip.isolated-remote-source.v1", companyId, issueId, repoUrl } } });
|
||||
await db.update(issues).set({ executionWorkspaceId: workspaceId, executionWorkspacePreference: "reuse_existing", updatedAt: idleSince })
|
||||
.where(eq(issues.id, issueId));
|
||||
const childId = randomUUID();
|
||||
if (options.closedParent) {
|
||||
await db.update(issues).set({ status: "done", completedAt: new Date(Date.now() - 4 * DAY_MS) }).where(eq(issues.id, issueId));
|
||||
await db.insert(issues).values({ id: childId, companyId, projectId, parentId: issueId, title: "Open follow-up", status: "in_progress",
|
||||
workMode: "standard", assigneeAgentId: agentId, responsibleUserId: "responsible-user", issueNumber: 2, identifier: `${issuePrefix}-2`,
|
||||
executionWorkspaceSettings: { mode: "isolated_workspace" } });
|
||||
}
|
||||
await fs.writeFile(path.join(realized.cwd, "marker.txt"), "commit B (unpushed)\n");
|
||||
git(realized.cwd, "commit", "-qam", "B");
|
||||
const unpushed = git(realized.cwd, "rev-parse", "HEAD");
|
||||
await fs.writeFile(path.join(realized.cwd, "notes.md"), "uncommitted notes\n");
|
||||
const workspace = () => db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId)).then(rows => rows[0]!);
|
||||
const expiration = await expireReconstructableGitWorkspace(db, await workspace(), {
|
||||
dryRun: false, expireBase: false, now: new Date(), idleCutoff: new Date(Date.now() - 7 * DAY_MS),
|
||||
dryRun: false, expireBase: false, now: new Date(), idleCutoff: options.closedParent ? undefined : new Date(Date.now() - 7 * DAY_MS),
|
||||
restorePoint: { storage: getStorageService(), notBefore: new Date(Date.now() - DAY_MS), keyDurable: true },
|
||||
});
|
||||
const queueRun = async () => {
|
||||
const queueRun = async (target = issueId) => {
|
||||
await db.insert(agentWakeupRequests).values({ id: wakeupId, companyId, agentId, source: "assignment", triggerDetail: "system",
|
||||
reason: "issue_assigned", payload: { issueId }, status: "queued", runId });
|
||||
reason: "issue_assigned", payload: { issueId: target }, status: "queued", runId });
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "assignment", triggerDetail: "system", status: "queued",
|
||||
wakeupRequestId: wakeupId, responsibleUserId: "responsible-user", contextSnapshot: { issueId, taskId: issueId, wakeReason: "issue_assigned" } });
|
||||
wakeupRequestId: wakeupId, responsibleUserId: "responsible-user", contextSnapshot: { issueId: target, taskId: target, wakeReason: "issue_assigned" } });
|
||||
await db.update(issues).set({ checkoutRunId: runId, executionRunId: runId, executionAgentNameKey: "resumer", executionLockedAt: new Date() })
|
||||
.where(eq(issues.id, issueId));
|
||||
.where(eq(issues.id, target));
|
||||
};
|
||||
return { issueId, workspaceId, runId, cwd: realized.cwd, unpushed, expiration, workspace, queueRun };
|
||||
return { issueId, childId, workspaceId, runId, cwd: realized.cwd, unpushed, expiration, workspace, queueRun };
|
||||
}
|
||||
|
||||
async function waitForRun(heartbeat: ReturnType<typeof heartbeatService>, runId: string) {
|
||||
@@ -159,4 +166,23 @@ describe.skipIf(!support.supported)("heartbeat resume of an idle-expired workspa
|
||||
const [issue] = await db.select().from(issues).where(eq(issues.id, seeded.issueId));
|
||||
expect(issue.executionWorkspaceId).toBe(seeded.workspaceId);
|
||||
});
|
||||
|
||||
it("restores a closed parent's expired workspace when an open child binds to it", async () => {
|
||||
const seeded = await seed({ closedParent: true });
|
||||
expect(seeded.expiration).toMatchObject({ expired: true, restorePoint: true });
|
||||
await expect(fs.access(seeded.cwd)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await db.update(issues).set({ executionWorkspaceId: seeded.workspaceId, executionWorkspacePreference: "reuse_existing" })
|
||||
.where(eq(issues.id, seeded.childId));
|
||||
await seeded.queueRun(seeded.childId);
|
||||
const heartbeat = heartbeatService(db);
|
||||
await heartbeat.resumeQueuedRuns();
|
||||
const run = await waitForRun(heartbeat, seeded.runId);
|
||||
await heartbeat.waitForRunExecutionDrain(seeded.runId);
|
||||
expect(run).toMatchObject({ status: "succeeded" });
|
||||
expect(adapterExecute.mock.calls[0]?.[0]).toMatchObject({ context: { executionWorkspaceId: seeded.workspaceId,
|
||||
paperclipWorkspace: { cwd: seeded.cwd } } });
|
||||
expect(git(seeded.cwd, "rev-parse", "HEAD")).toBe(seeded.unpushed);
|
||||
expect(await fs.readFile(path.join(seeded.cwd, "notes.md"), "utf8")).toBe("uncommitted notes\n");
|
||||
expect((await seeded.workspace()).status).not.toBe("archived");
|
||||
});
|
||||
});
|
||||
@@ -5,7 +5,7 @@ import { measureRunTransaction } from "./execution-transaction-timing.js";
|
||||
import { measureRunPhase } from "./execution-preparation-timing.js";
|
||||
import { assertExecutionWorkspaceRecordAccess, assertTaskWorkspaceExecutionAccess, assertTaskWorkspaceSourceProjectAccess } from "./task-workspace-access.js";
|
||||
import { verifiedRecordedIsolatedRemoteSource, withIsolatedRepositorySourceLock } from "./isolated-remote-workspace-source.js";
|
||||
import { expireReconstructableGitWorkspace, gitWorkspaceLastActivityAt, hasPendingGitExpiration } from "./git-workspace-expiration.js";
|
||||
import { expireReconstructableGitWorkspace, gitWorkspaceLastActivityAt, hasPendingGitExpiration, listIssuesUsingGitWorkspace } from "./git-workspace-expiration.js";
|
||||
import { hasPendingIsolatedRepositoryRehomeCleanup, isIsolatedRepositoryRehomeCandidate, rehomeExpiredIsolatedRestoration, rehomeIsolatedRepositoryWorkspace, sweepOrphanedIsolatedRepositorySources } from "./isolated-repository-rehome.js";
|
||||
import { hasPendingNestedCloneMigration, isNestedCloneMigrationCandidate, nextNestedCloneMigrationBatch, pendingNestedCloneMigrationNames, listNestedCloneMigrationNames, migrateNestedRepositoryClone,
|
||||
NESTED_CLONE_MIGRATION_RETRY_REASONS, type NestedCloneMigrationRotation } from "./nested-clone-migration.js";
|
||||
@@ -3434,7 +3434,14 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
|
||||
workspace.metadata as Record<string, unknown> | null,
|
||||
);
|
||||
if (reopenPending) terminalWorkspaceDeadlines.set(workspace.id, now().getTime());
|
||||
if (!sourceIssue || issueTree.some(issue => !TERMINAL_ISSUE_STATUSES.has(issue.status))) {
|
||||
const treeOpen = issueTree.some(issue => !TERMINAL_ISSUE_STATUSES.has(issue.status));
|
||||
const gitExpirationCandidate = workspace.providerType !== "plugin" && Boolean(opts.gitCacheExpirationEnabled)
|
||||
&& (hasPendingGitExpiration(workspace) || (workspace.mode === "isolated_workspace" && workspace.strategyType === "git_worktree"));
|
||||
const usingIssueIds = sourceIssue && gitExpirationCandidate && !reopenPending
|
||||
&& TERMINAL_ISSUE_STATUSES.has(sourceIssue.status)
|
||||
? await listIssuesUsingGitWorkspace(db, workspace, issueTree.map(issue => issue.id)) : null;
|
||||
const expirationTree = usingIssueIds ? issueTree.filter(issue => usingIssueIds.has(issue.id)) : issueTree;
|
||||
if (!sourceIssue || expirationTree.some(issue => !TERMINAL_ISSUE_STATUSES.has(issue.status))) {
|
||||
if (reopenPending) {
|
||||
await clearReopenPendingConsumptionUnderLock(workspace.id, {
|
||||
expectedGeneration: readExecutionWorkspaceLifecycleGeneration(
|
||||
@@ -3473,11 +3480,13 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
|
||||
result.skippedNonTerminalTree += 1;
|
||||
continue;
|
||||
}
|
||||
const latestTerminalAt = Math.max(...issueTree.map(issue => issueTerminalTimestamp(issue)!.getTime()));
|
||||
const expirationTerminalAt = Math.max(...expirationTree.map(issue => issueTerminalTimestamp(issue)!.getTime()));
|
||||
const latestTerminalAt = treeOpen ? expirationTerminalAt : Math.max(...issueTree.map(issue => issueTerminalTimestamp(issue)!.getTime()));
|
||||
// Compaction runs once per terminal cycle, after its own delay and
|
||||
// before the archive cooldown, so a workspace the reaper cannot archive
|
||||
// yet (or ever) stops holding regenerable files.
|
||||
const compactionPending = workspaceCompactionDelayMs !== null
|
||||
&& !treeOpen
|
||||
&& !isClosedExecutionWorkspaceStatus(workspace.status)
|
||||
&& !reopenPending
|
||||
&& compactedTerminalAt.get(workspace.id) !== latestTerminalAt;
|
||||
@@ -3503,14 +3512,13 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
|
||||
);
|
||||
}
|
||||
}
|
||||
if (workspace.providerType !== "plugin" && opts.gitCacheExpirationEnabled && (hasPendingGitExpiration(workspace)
|
||||
|| (workspace.mode === "isolated_workspace" && workspace.strategyType === "git_worktree"))) {
|
||||
if (gitExpirationCandidate) {
|
||||
const expiration = await expireReconstructableGitWorkspace(db, workspace, {
|
||||
dryRun: opts.gitCacheExpirationDryRun ?? false, expireBase: opts.gitBaseExpirationEnabled ?? false,
|
||||
now: now(), signal, beforeDelete: opts.beforeTerminalWorkspaceCleanup,
|
||||
restorePoint: opts.gitRestorePoints ? {
|
||||
storage: opts.gitRestorePoints.storage,
|
||||
notBefore: new Date(latestTerminalAt + opts.gitRestorePoints.terminalDelayMs),
|
||||
notBefore: new Date(expirationTerminalAt + opts.gitRestorePoints.terminalDelayMs),
|
||||
keyDurable: opts.gitRestorePoints.keyDurable,
|
||||
} : undefined,
|
||||
});
|
||||
|
||||
@@ -3,7 +3,7 @@ import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { randomBytes } from "node:crypto";
|
||||
import { acquireGitExpirationLocks } from "./git-workspace-expiration-locks.js";
|
||||
import { and, eq, ne, sql } from "drizzle-orm";
|
||||
import { and, eq, inArray, ne, sql } from "drizzle-orm";
|
||||
import { executionWorkspaces, issues, heartbeatRuns, nativeRunFinalizations, environmentLeases,
|
||||
workspaceRuntimeServices, issueRecoveryActions, projectWorkspaces, withDedicatedDbConnection, type Db } from "@paperclipai/db";
|
||||
import { createGitRemoteAuthProvider } from "./git-credentials.js";
|
||||
@@ -124,6 +124,21 @@ export function expirationScope(workspace: Workspace) {
|
||||
)`;
|
||||
}
|
||||
|
||||
function issueUsesWorkspace(workspace: Pick<Workspace, "id" | "sourceIssueId">) {
|
||||
return sql`(${issues.id} = ${workspace.sourceIssueId} OR ${issues.executionWorkspaceId} = ${workspace.id}
|
||||
OR (${issues.workspaceSelection}->'selection'->>'kind' = 'existing'
|
||||
AND ${issues.workspaceSelection}->'selection'->>'workspaceId' = ${workspace.id})
|
||||
OR (${issues.workspacePendingSelection}->'intent'->'selection'->>'kind' = 'existing'
|
||||
AND ${issues.workspacePendingSelection}->'intent'->'selection'->>'workspaceId' = ${workspace.id}))`;
|
||||
}
|
||||
|
||||
export async function listIssuesUsingGitWorkspace(db: Db | Tx, workspace: Pick<Workspace, "id" | "companyId" | "sourceIssueId">, issueIds: string[]) {
|
||||
if (issueIds.length === 0) return new Set<string>();
|
||||
const rows = await db.select({ id: issues.id }).from(issues)
|
||||
.where(and(eq(issues.companyId, workspace.companyId), inArray(issues.id, issueIds), issueUsesWorkspace(workspace)));
|
||||
return new Set(rows.map(row => row.id));
|
||||
}
|
||||
|
||||
export async function gitWorkspaceLastActivityAt(db: Db | Tx, workspace: Workspace): Promise<Date | null> {
|
||||
if (!workspace.sourceIssueId) return null;
|
||||
const rows = await db.execute<{ at: string | Date | null }>(sql`${expirationScope(workspace)}
|
||||
@@ -142,8 +157,10 @@ export async function isGitWorkspaceExpirationEligible(tx: Tx, workspace: Worksp
|
||||
NOT EXISTS (SELECT 1 FROM tree WHERE updated_at > ${idleCutoff.toISOString()}::timestamptz)
|
||||
AND NOT EXISTS (SELECT 1 FROM bound WHERE updated_at > ${idleCutoff.toISOString()}::timestamptz)
|
||||
AND NOT EXISTS (SELECT 1 FROM relevant_runs WHERE greatest(created_at, started_at, finished_at) > ${idleCutoff.toISOString()}::timestamptz)` : sql`
|
||||
NOT EXISTS (SELECT 1 FROM tree WHERE status NOT IN ('done', 'cancelled')
|
||||
OR coalesce(CASE WHEN status = 'done' THEN completed_at ELSE cancelled_at END, updated_at) > ${new Date(now.getTime() - DAY_MS).toISOString()}::timestamptz)
|
||||
NOT EXISTS (SELECT 1 FROM ${issues} WHERE ${issues.companyId} = ${workspace.companyId}
|
||||
AND ${issues.id} IN (SELECT id FROM tree) AND ${issueUsesWorkspace(workspace)}
|
||||
AND (${issues.status} NOT IN ('done', 'cancelled')
|
||||
OR coalesce(CASE WHEN ${issues.status} = 'done' THEN ${issues.completedAt} ELSE ${issues.cancelledAt} END, ${issues.updatedAt}) > ${new Date(now.getTime() - DAY_MS).toISOString()}::timestamptz))
|
||||
AND NOT EXISTS (SELECT 1 FROM ${issues} WHERE company_id = ${workspace.companyId}
|
||||
AND execution_workspace_id = ${workspace.id} AND status NOT IN ('done', 'cancelled'))`;
|
||||
const recoveryActionBlocks = idleCutoff ? sql`true` : sql`(cause = 'native_workspace_finalization_owner_unverified'
|
||||
|
||||
Reference in new issue
Block a user