Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-25 02:26:12 +02:00
Code Issues Packages Projects Releases Wiki Activity

Re-home isolated repository workspaces onto the shared project clone (#17369)

Browse Source 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each isolated task workspace is a Git worktree on the server disk,
and the terminal workspace reaper and git cache expiration keep that
disk usage bounded
> - For some days, workspace preparation gave each isolated
`git_worktree` workspace its own full clone under
`isolated-repository-sources/`, with the worktree inside it. #16963
removed that creation path, but the existing clones stay, and restoring
an expired workspace still recreates its private clone
> - On one instance, 30 such clones use about 23 GB: 12.1 GB of `.git`
data, 5.2 GB of unused default-branch checkouts and 5.75 GB of real
worktrees. 28 of them back open workspaces, and the disk is nearly full
> - This pull request moves each such open workspace onto the project's
shared managed clone with its exact state, deletes the private clone,
and makes restore use the shared clone
> - The benefit is that disk use drops to one shared clone per project
repository, with no lost work and no change for workspaces that cannot
move safely

## Linked Issues or Issue Description

Refs #16963. Bug report described in-PR (bug report template).

**What happened?**
On one instance, 30 per-issue clones under
`isolated-repository-sources/<companyId>/<issueId>/checkout` use about
23 GB. 28 of them back open workspaces. No process moves these
workspaces to the shared project clone that new workspaces use, and git
cache expiration does not remove a clone while its workspace is open.
Reopening an expired workspace whose recipe has `sourceKind:
"isolated_remote"` clones the full repository again. Separately, a
restore point with a non-default file mode cannot be restored: the
reference read back from jsonb has reordered `modes` keys, so
`loadGitRestorePoint` fails with `restore_point_manifest_mismatch`.

**Expected behavior**
An open workspace with a private clone moves to the shared project clone
with the same branch, HEAD, index, worktree, untracked and ignored
files, and file modes. The private clone is then deleted. Reopening an
expired workspace uses the shared clone when one exists. A restore point
with non-default file modes restores.

**Steps to reproduce**
1. Create an isolated `git_worktree` workspace with
`materializeIsolatedRemoteWorkspaceSource` and record
`isolatedRemoteSource` metadata, as preparation did before #16963.
2. Keep its issue open. The clone stays, because no sweep moves or
removes it.
3. Expire the workspace with a restore point that has a file with mode
`0600`, then reopen it. The private clone is created again, and the
restore fails with `restore_point_manifest_mismatch`.

**Paperclip version or commit:** `master` at ec2e0d6407.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).

## What Changed

- New `server/src/services/isolated-repository-rehome.ts` with
`rehomeIsolatedRepositoryWorkspace`. It holds the isolated source lock,
the `execution_workspace_lifecycle:<id>` advisory lock and
expiration-style Git lock files (`index.lock`, `HEAD.lock` and the
branch ref lock in the clone).
- Eligibility uses the safety checks of git cache expiration that matter
while a workspace stays open: no live or unfinished relevant runs, no
unsettled native finalizations, no unreleased environment leases, no
running runtime services, no pending reopen, and no other workspace on
the same path. It does not require a closed or idle issue tree. The
check runs again inside the update transaction.
- The target base is the project's existing shared managed clone for the
same `repoUrl` (the paths `ensureManagedProjectWorkspace` uses, verified
with the same origin and storage checks as restoration).
- The move captures a restore point of the old worktree. The upstream
boundary is commits that both repositories have, so the bundle is small.
It imports the bundle into the shared clone, creates
`refs/heads/<branch>` or fast-forwards it from an ancestor, and adds the
worktree at `<base>/.paperclip/worktrees/<branch>`. Then it applies the
restore point. It copies the `HEAD` reflog and `ORIG_HEAD`, copies the
branch's upstream config when it creates the branch, and moves
regenerable directories, untracked `.env`/`.env.*` files,
`.claude/settings.local.json` and nested `.paperclip-repositories/*`
checkouts by rename. Standalone nested clones move as is. Nested
checkouts that are worktrees of their own shared clone (#17382) move
too, followed by `git worktree repair` in that clone so the registration
follows the new path, and their `paperclip/<hash>/` branch is renamed to
the new path's prefix so ownership checks still pass. Entries inside a
moved nested checkout (for example its `node_modules`) move with it.
Rollback and interrupted-move recovery repair them again after moving
them back, and verification checks that each one still resolves to the
same shared clone.
- Verification: the restore point snapshot (HEAD, index, worktree,
ignored files, modes) of the new worktree must equal the old one before
and after the move. The reflog and `ORIG_HEAD` must match, and every
moved item must exist. Only then does one transaction set
`cwd`/`providerRef`, drop `isolatedRemoteSource` and
`gitCacheExpiration`, rewrite a saved active restoration recipe to
`managed_project`, and record `isolatedRepositoryRehome: { at, from, to,
cleanedAt }`. A failure before that removes the new worktree, moves
items back, restores the branch, and leaves the old worktree untouched.
- Safety of the checkout in the shared clone: the move skips with
`unsupported_base_git_config` when the shared clone's local config sets
filter drivers, includes, attribute files, worktree config or command
settings, or when it has `info/attributes`. It checks before planning
and again right before `git worktree add`.
- Crash safety before the update: the move records
`isolatedRepositoryRehomePending` (base, target, branch, head, previous
branch tip, planned moves) before the first change to the shared clone.
The next attempt undoes an interrupted move under the same locks: it
moves items back, removes the new worktree only when it is proven to be
this move's (a linked worktree of the shared clone on the issue branch
that no workspace row references), and restores the branch. A live
failure removes the target only after this attempt created it. When an
item exists in both places, rollback stops with
`rehome_rollback_conflict` and keeps the target and the record, so no
copy is lost.
- Cleanup after the update: remove the old worktree (rename to trash,
then delete, then `worktree prune`). Delete the clone (rename to trash,
then delete) only when no other workspace, project workspace or running
service uses it, it has no other worktree, its checkout has no changes
or non-regenerable files, its refs are only branches, tags and
remote-tracking refs, and every local commit is on a remote-tracking ref
or present in the shared clone. Cleanup reads at most 5,000 refs, reflog
entries and unique commits (`clone_history_budget`). `cleanedAt` records
completion. A crash after the update leaves a pending record that the
next pass finishes, also after the workspace is archived: the reaper
scan includes archived rows with unfinished re-home cleanup when the
feature is on.
- Reaper integration in `sweepTerminalWorkspaces`: behind
`PAPERCLIP_ISOLATED_SOURCE_REHOME_ENABLED` (default `false`) and
`PAPERCLIP_ISOLATED_SOURCE_REHOME_DRY_RUN`, only when git cache
expiration is enabled. It logs `event:
"execution_workspace.isolated_source_rehome"` with outcome, reason,
candidate bytes and reclaimed bytes. A skipped workspace gets a retry
deadline one hour later in the reaper's existing deadline set. No new
timer.
- Restore: `reopenClosedIsolatedExecutionWorkspaceForIssue` calls
`rehomeExpiredIsolatedRestoration` first. For an expired workspace with
an `isolated_remote` recipe, it uses the shared clone when one exists
and the branch there is absent or at the saved commit. It keeps the old
behavior for recipes with worktree-backed nested checkouts on branches,
because those branch names are bound to the old path. It also requires
that the shared clone has every base commit (prerequisite) of the saved
restore point bundle. It rewrites the row and recipe to the shared clone
and the standard worktree path, then the existing restore runs
(`ensureGitRestorePointCommit`, `ensureSavedGitCommit`, apply). Without
a shared clone, the old behavior stays.
- Fix: restore point `modes` comparison in
`git-restore-point-storage.ts` compares by path and mode, so jsonb key
order does not matter.
- Small exports: `allocatedBytes`, `expirationScope`,
`managedSourcePaths`, `verifiedGitCacheSource`.
- Docs: new "Isolated repository sources" section in
`doc/recoverable-disk-cleanup.md`, new settings in `doc/DEVELOPING.md`.

Deviations from the requested design:
- The sweep does not create a missing shared clone. When none exists, it
skips with `no_shared_base`, because a new network clone in a cleanup
path can use more disk. Restore follows the same rule and keeps the old
behavior.
- History moves through the restore point bundle, not a separate fetch
from the clone. The bundle contains HEAD and every reflog/`ORIG_HEAD`
commit that the shared clone does not have. The move then checks that
all of them are present.
- The standard location is `<base>/.paperclip/worktrees/<branch>`. A
worktree outside the clone's standard location is skipped
(`nonstandard_worktree_location`).
- Lock files use a token derived from the workspace id, so a crashed
attempt's locks are recovered without a database marker.
- A clone that another open workspace still uses is kept and marked
`cloneRetained: "base_has_consumers"`. That workspace's own move removes
it later.

Skip reasons: `live_or_unresolved_consumer`,
`unsupported_base_git_config`, `rehome_rollback_conflict`,
`nested_branch_conflict`, `invalid_rehome_record`,
`interrupted_rehome_pending` (dry run only), `lifecycle_busy`,
`source_busy`, `workspace_changed`, `no_shared_base`,
`shared_base_repo_mismatch`, `missing_worktree`,
`nonstandard_worktree_location`, `source_identity_changed`,
`branch_identity_changed`, `unproven_worktree_admin`,
`unsupported_worktree_git_state`, `unique_worktree_refs`,
`unknown_worktree_git_logs`, `unsupported_local_git_state`,
`unique_git_refs`, `unsupported_index_flags`, `unsupported_branch_name`,
`invalid_restoration_recipe`, `unsupported_restoration_recipe`,
`target_worktree_exists`, `base_branch_checked_out`,
`base_branch_diverged`, `untransferable_ignored_files`,
`untransferable_managed_checkouts`, `worktree_git_busy`, `head_changed`,
`history_not_transferred`, `transfer_conflict`, `snapshot_mismatch`,
`history_mismatch`, `transfer_incomplete`, and restore point reasons
such as `untracked_nested_repository`, `submodules_unsupported`,
`filtered_paths`, `restore_point_too_large`. Clone kept reasons:
`base_has_consumers`, `base_has_live_services`, `base_has_worktrees`,
`clone_checkout_has_files`, `unsupported_clone_refs`,
`unique_clone_history`, `clone_history_budget`.

## Verification

- `pnpm exec vitest run
src/__tests__/isolated-repository-rehome.test.ts` in `server/`: 18
passed. It uses real Git repositories and embedded Postgres. It covers:
- a move with an unpushed commit, staged and unstaged edits, a deletion,
an untracked file with mode `0600`, an ignored file, `node_modules`,
`.env`, `.claude/settings.local.json` and a nested
`.paperclip-repositories` checkout. Snapshots, branch, HEAD, `git
status` and reflog match, the row points to the new path without
`isolatedRemoteSource`, and the old worktree and clone are gone;
- skips with nothing changed on disk or in the database: live run,
branch in the shared clone at an unrelated commit, no shared clone, an
untracked nested repository, and a concurrent change between capture and
verify (hook). The new worktree is removed and the old one keeps its
moved items;
  - the clone is kept while another workspace references it;
- a worktree-backed nested checkout moves with the task worktree, stays
registered in its shared clone at the new path (not prunable), and keeps
its changes, with its branch renamed to the new path's prefix. On
rollback it returns to the old path with its registration repaired and
its branch name restored. A standalone nested checkout moves together
with its own ignored `node_modules`;
- a file recreated in the old worktree before verification stops the
rollback and keeps both copies, then the move completes once the
conflict is gone;
- a hostile `filter.*.smudge` in the shared clone's config makes the
move skip without running it;
- a clone with more than 5,000 refs is kept, and is deleted once the
refs are gone;
- a move interrupted before the database update is reported (not undone)
by a dry run, then undone and completed on the next attempt;
- a target that another party creates right before `git worktree add` is
never deleted;
- a crash after the database update; after the row is archived, a later
reaper sweep removes the old worktree and clone;
- restore keeps the private-clone fallback when the shared clone lacks
the restore point's base commits;
- disabled by default (both flags needed), dry run changes nothing, and
the retry deadline is registered;
- reopening an expired `isolated_remote` workspace rebuilds it in the
shared clone with the restore point applied and no clone created.
- After the round 4 fixes, the related suites
(`git-workspace-expiration`, `git-restore-point-expiration`,
`execution-workspace-reopen`, `execution-workspaces-service`,
`heartbeat-git-restore-point-resume`, `heartbeat-project-repositories`,
`isolated-remote-workspace-source`) with the new suite: 326 passed.
- After rebasing on #17382, the related suites (adding
`heartbeat-project-repositories`) with the new suite: 374 passed.
- After the first review fixes, the related suites
(`git-workspace-expiration`, `git-restore-point`,
`git-restore-point-expiration`, `execution-workspace-reopen`,
`execution-workspaces-service`, `execution-workspace-compaction`,
`heartbeat-git-restore-point-resume`,
`isolated-remote-workspace-source`) with the new suite: 301 passed.
- Related suites in `server/` (first commit): `pnpm exec vitest run` on
`git-workspace-expiration`, `git-workspace-expiration-eligibility`,
`git-restore-point`, `git-restore-point-expiration`,
`git-restore-point-expiration-config`, `git-restore-point-ledger`,
`git-restore-point-ledger-config`, `execution-workspace-reopen`,
`execution-workspaces-service`, `execution-workspace-compaction`,
`heartbeat-git-restore-point-resume`,
`isolated-remote-workspace-source`,
`isolated-remote-workspace-source-lock`, `git-workspace-base-storage`,
`git-reproducible-managed-checkouts`, `git-expiration-auth-fallback` and
`workspace-runtime`: 550 passed, 2 failed. The 2 failures are in
`workspace-runtime.test.ts` (base ref resolution) and also fail on
unmodified `master` on the same machine.
- `pnpm run typecheck` in `server/` passes (after `pnpm -r --filter
"./packages/**" build`).
- Mutation checks. Each reverted guard fails one test in the new suite:
  - eligibility always true: "keeps a workspace with a live run" fails;
- snapshot verification removed: "removes the new worktree and keeps the
old one when the workspace changes during the move" fails;
- clone reference check removed: "keeps the clone while another
workspace still references it" fails;
- restore re-home removed from reopen: "restores an expired isolated
workspace into the shared project clone" fails;
  - old `JSON.stringify` modes comparison: the same restore test fails;
  - interrupted-move recovery removed: the interrupted-move test fails;
- target existence check before `git worktree add` replaced by assumed
ownership: the foreign-target test fails;
- bundle prerequisite check removed: the stale shared clone restore test
fails;
  - archived-row cleanup scan removed: the crash test fails;
- dry run check before interrupted-move recovery removed: the
interrupted-move test fails;
- `git worktree repair` after the forward move removed: both nested
worktree tests fail. The repair after moving back removed: the rollback
test fails;
- nested branch rename forward or back removed: the matching nested test
fails;
  - rollback conflict check removed: the reappearing-file test fails;
  - covered-entry filter removed: the full re-home test fails;
  - shared clone config check removed: the hostile filter test fails;
  - ref budget removed: the many-ref test fails.
- Not run: the full server suite, and the move on a live instance.

## Risks

- Workspace paths change for re-homed workspaces. Agents that resume an
old session can remember the old path. The run's working directory comes
from the updated row, but text in a resumed conversation can still name
the old path.
- The feature is off by default. It runs only with
`PAPERCLIP_ISOLATED_SOURCE_REHOME_ENABLED=true` and git cache expiration
enabled. Dry run is available. The restore change is not behind the
flag: reopening an expired private-clone workspace now uses the shared
clone when one exists.
- A run can start between the last eligibility check and the database
update. The update transaction checks eligibility again, and the old
worktree is deleted only after another check. This is the same window
that git cache expiration accepts.
- Imported restore point objects stay in the shared clone as unreachable
objects after a failed attempt until Git garbage collection removes
them. Base expiration keeps a base with unreachable objects, so this
only delays base expiration.
- A workspace that cannot move (for example `no_shared_base`) is retried
each hour while the feature is on. Each retry runs a few Git commands
for that workspace.
- No migration. No schema change.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file editing, Git, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin FoleyandPaperclip authored and GitHub committed 2026-10-23 18:25:26 -07:00
1 parent 724fa6144d
commit 8549ecaf9a
10 files changed
+1253 -11
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

doc/DEVELOPING.md
+13
View File
@@ -3055,6 +3055,19 @@ Both enable switches default off. Rollback disables further expiration while
leaving saved recipes and restoration support intact. Recovery archives, runner
backups, non-Git workspaces, and immutable skill snapshots are unaffected.
### Isolated repository source re-home
Workspaces created with a private per-issue clone under
`isolated-repository-sources/` can be moved onto the project's shared managed
clone, after which the private clone is deleted. Enable it with
`PAPERCLIP_ISOLATED_SOURCE_REHOME_ENABLED=true`; it runs only when
`PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true` and defaults off. Set
`PAPERCLIP_ISOLATED_SOURCE_REHOME_DRY_RUN=true` to log candidates without changing
anything. Each attempt logs `execution_workspace.isolated_source_rehome` with its
outcome, skip reason and reclaimed bytes. See
[recoverable disk cleanup](recoverable-disk-cleanup.md#isolated-repository-sources)
for the conditions and the verification it performs.
### Terminal workspace compaction
A finished task often keeps its worktree for a long time: the reaper archives a
doc/recoverable-disk-cleanup.md
+91
View File
@@ -313,3 +313,94 @@ processed in batches of 50; a full batch continues immediately.
Rows have no foreign keys, so they outlive a deleted workspace, issue or company long
enough to remove the stored object. Each run that changes anything logs its counts.
## Isolated repository sources
For a while, workspace preparation gave some isolated `git_worktree` workspaces their
own full clone of the project repository under
`instances/<id>/isolated-repository-sources/<companyId>/<issueId>/checkout`, with the
issue's worktree inside it. Workspace metadata marks these with `isolatedRemoteSource`.
New workspaces no longer get one; they are worktrees of the project's shared managed
clone. Each old clone still holds a full `.git` and an unused default-branch checkout.
With `PAPERCLIP_ISOLATED_SOURCE_REHOME_ENABLED=true` (and git cache expiration enabled),
the terminal workspace reaper moves each such open workspace onto the shared clone and
then deletes its private clone. It runs under the same source lock, workspace lifecycle
lock and Git lock files as git cache expiration, so no run, expiration, reopen or
restore can interleave. A workspace is moved only when:
- no run, native finalization, environment lease or runtime service is still live for it,
and it is not waiting for a reopen to be consumed (the issue may be open);
- the project has a shared managed clone for the same `repoUrl`;
- its worktree sits at the standard `.paperclip/worktrees/<branch>` location, and the
branch is absent from the shared clone, points to an ancestor of the worktree's HEAD
there, and is not checked out in another of its worktrees;
- the shared clone's local config sets no filter drivers, includes, attribute files, worktree
config or command settings (`core.fsmonitor`, `core.hooksPath`, `core.sshCommand` and
similar), and it has no `info/attributes`, because the move checks out files there;
- its Git state is transferable: no stash or replacement refs, no in-progress merge,
rebase or other worktree-specific state, no hidden index flags, no shallow history, and
nothing a restore point refuses (submodules, filtered paths, special files, untracked
nested repositories, oversized contents).
The move captures a restore point of the old worktree against commits both repositories
share, imports it into the shared clone, creates or fast-forwards the issue branch, adds
the new worktree at the same HEAD and applies the restore point (index, worktree changes,
untracked and ignored files, file modes). The `HEAD` reflog and `ORIG_HEAD` are copied.
What restore points leave out is moved by rename on the same filesystem: regenerable
directories such as `node_modules`, untracked `.env` and `.env.*` files,
`.claude/settings.local.json`, and each nested `.paperclip-repositories/*` checkout as a
whole. A standalone nested clone moves as is. A nested checkout that is a worktree of its own
shared clone moves too, and the move runs `git worktree repair` in that clone so its
registration follows the new path. Its `paperclip/<hash>/` branch is renamed to the prefix of
the new path, because that prefix proves the checkout owns the branch (a rollback moves it
back, repairs it and renames the branch back). Items inside a moved nested checkout, such as
its `node_modules`, move with it. Any other
ignored file that a restore point does not save keeps the workspace in
place.
The new worktree must then match the old one exactly (HEAD, index, worktree, ignored files,
modes and reflog), and every moved item must exist. Only then does one database update
point the workspace at the new path, drop `isolatedRemoteSource`, rewrite a saved
restoration recipe to the shared clone, and record `isolatedRepositoryRehome`
(`{ at, from, to, cleanedAt }`). Any failure before that update removes the new worktree,
moves items back, restores the branch, and leaves the old worktree untouched. Before it
changes the shared clone, the move records `isolatedRepositoryRehomePending` (target, branch,
previous branch tip and planned moves). If the server stops before the update, the next
attempt uses that record to move items back, remove the new worktree and restore the
branch, then starts again. A dry run reports such a record without undoing it. If an item
exists in both places (for example a process recreated `.env` in the old worktree), the
rollback stops with `rehome_rollback_conflict` and keeps the new worktree and the record, so
neither copy is lost; later attempts retry the rollback once the conflict is resolved. A new worktree is removed only when the move created it: a
target that appears from elsewhere is never deleted.
After the update, the old worktree is removed and the clone is renamed to trash and
deleted when nothing else needs it: no other workspace or project workspace refers to it
(a workspace already expired with its worktree removed does not count), no runtime
service runs in it, it has no other worktree, its own checkout has no changes or files
other than regenerable ones, it has only branch, tag and remote-tracking refs, and every
commit its local refs and reflogs hold is either on a remote-tracking ref or present in the
shared clone. A clone with more than 5,000 refs, reflog entries or such commits is kept
(`clone_history_budget`). A clone another workspace still uses is kept (`cloneRetained`); that
workspace's own move removes it later. If the server stops after the update, a later sweep
finishes the removal and sets `cleanedAt`, also after the workspace has been archived.
Each attempt logs `event: "execution_workspace.isolated_source_rehome"` with the outcome,
skip reason, candidate bytes and reclaimed bytes. A workspace that is skipped, or whose
clone is kept for a reason that can change, is retried an hour later through the reaper's
existing deadlines; nothing runs while there is no candidate.
`PAPERCLIP_ISOLATED_SOURCE_REHOME_DRY_RUN=true` reports candidates without changing
anything.
Re-homed workspaces get a new path. An agent that resumes an older session may still
remember the old path; the run's working directory comes from the updated workspace.
Reopening or restoring an expired workspace whose recipe came from a private clone no
longer recreates that clone when the project has a shared managed clone for the same
repository, its branch there is absent or at the saved commit, and the shared clone has
every commit the saved restore point bundle needs as a base. A recipe with worktree-backed
nested checkouts on branches keeps the old behavior, because their branch names are bound to
the old path. The workspace is
rebuilt as a worktree of the shared clone at the standard location, with the saved commit
and restore point imported and applied as before. Without a shared clone, the old
behavior applies.
server/src/__tests__/isolated-repository-rehome.test.ts
+475
View File
@@ -0,0 +1,475 @@
import { promises as fs } from "node:fs";
import os from "node:os";
import path from "node:path";
import { execFileSync } from "node:child_process";
import { randomUUID } from "node:crypto";
import { pathToFileURL } from "node:url";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import { eq, sql } from "drizzle-orm";
import { agents, companies, createDb, executionWorkspaceRestorePoints, executionWorkspaces, heartbeatRuns, issues, projects } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { resolveManagedProjectWorkspaceDir } from "../home-paths.js";
import { materializeManagedProjectWorkspace, ensureManagedRepositoriesIgnored } from "../services/managed-repository-checkout.js";
import { isolatedRemoteSourcePath, materializeIsolatedRemoteWorkspaceSource } from "../services/isolated-remote-workspace-source.js";
import { realizeExecutionWorkspace } from "../services/workspace-runtime.js";
import { executionWorkspaceService } from "../services/execution-workspaces.js";
import { expireReconstructableGitWorkspace } from "../services/git-workspace-expiration.js";
import { currentRestorePointSnapshot } from "../services/git-restore-point.js";
import { rehomeIsolatedRepositoryWorkspace } from "../services/isolated-repository-rehome.js";
import { addNestedRepositoryWorktree, nestedRepositoryBaseCandidates, nestedWorktreeBranchPrefix, readLinkedNestedCheckout } from "../services/nested-repository-worktrees.js";
import { getStorageService } from "../storage/index.js";
const gitEnv = { ...process.env, GIT_AUTHOR_NAME: "Fixture", GIT_AUTHOR_EMAIL: "fixture@example.test",
GIT_COMMITTER_NAME: "Fixture", GIT_COMMITTER_EMAIL: "fixture@example.test", GIT_TERMINAL_PROMPT: "0" };
const git = (cwd: string, ...args: string[]) => execFileSync("git", args, { cwd, env: gitEnv, encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }).trim();
const exists = (target: string) => fs.lstat(target).then(() => true, () => false);
const DAY_MS = 24 * 60 * 60 * 1000;
const support = await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = support.supported ? describe : describe.skip;
describeEmbeddedPostgres("isolated repository source re-home", () => {
let db: ReturnType<typeof createDb>;
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let root: string;
const previousHome = process.env.PAPERCLIP_HOME;
const previousInstance = process.env.PAPERCLIP_INSTANCE_ID;
beforeAll(async () => {
database = await startEmbeddedPostgresTestDatabase("isolated-repository-rehome-");
db = createDb(database.connectionString);
root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-isolated-rehome-"));
process.env.PAPERCLIP_HOME = path.join(root, "home");
process.env.PAPERCLIP_INSTANCE_ID = "default";
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "41".repeat(32));
});
afterEach(async () => {
await db.delete(executionWorkspaceRestorePoints);
await db.execute(sql`TRUNCATE TABLE companies CASCADE`);
});
afterAll(async () => {
vi.unstubAllEnvs();
if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome;
if (previousInstance === undefined) delete process.env.PAPERCLIP_INSTANCE_ID; else process.env.PAPERCLIP_INSTANCE_ID = previousInstance;
await db.$client.end({ timeout: 1 }); await database.cleanup(); await fs.rm(root, { recursive: true, force: true });
});
async function fixture(options: { sharedBase?: boolean; issueStatus?: "in_progress" | "done" } = {}) {
const companyId = randomUUID(), issueId = randomUUID(), workspaceId = randomUUID(), projectId = randomUUID();
const location = path.join(root, companyId), author = path.join(location, "author"), remote = path.join(location, "remote.git");
await fs.mkdir(author, { recursive: true });
git(author, "init", "--initial-branch=main");
await fs.writeFile(path.join(author, "marker.txt"), "commit A\n");
await fs.writeFile(path.join(author, "remove-me.txt"), "tracked\n");
await fs.writeFile(path.join(author, ".gitignore"), "node_modules/\n.env\n.claude/settings.local.json\n*.tsbuildinfo\n");
git(author, "add", "."); git(author, "commit", "-m", "A");
git(location, "clone", "--bare", author, remote);
const done = options.issueStatus === "done";
await db.insert(companies).values({ id: companyId, name: "Re-home fixture", issuePrefix: `RH${companyId.slice(0, 6)}` });
await db.insert(projects).values({ id: projectId, companyId, name: "Fixture project" });
await db.insert(issues).values({ id: issueId, companyId, projectId, title: "Open task", status: done ? "done" : "in_progress",
...(done ? { completedAt: new Date(Date.now() - 4 * DAY_MS) } : {}) });
const repoUrl = pathToFileURL(remote).href;
const sharedPath = resolveManagedProjectWorkspaceDir({ companyId, projectId, repoName: "remote" });
if (options.sharedBase !== false) await materializeManagedProjectWorkspace(sharedPath, { repoUrl });
const clone = await materializeIsolatedRemoteWorkspaceSource({ companyId, issueId, repoUrl, resolveGitAuth: async () => null });
const realized = await realizeExecutionWorkspace({ base: { baseCwd: clone, source: "task_session", projectId: null, workspaceId: null, repoUrl, repoRef: "origin/main" },
issue: { id: issueId, identifier: "RH-1", title: "Fixture" }, agent: { id: null, name: "Fixture", companyId },
config: { workspaceStrategy: { type: "git_worktree", branchTemplate: "fixture-task", baseRef: "origin/main" } }, skipHostProvision: true });
await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, projectId, sourceIssueId: issueId, mode: "isolated_workspace", strategyType: "git_worktree",
name: "Fixture", status: "active", providerType: "local_fs", cwd: realized.cwd, providerRef: realized.cwd, repoUrl, baseRef: "origin/main",
branchName: realized.branchName, metadata: { createdByRuntime: realized.branchCreatedByRuntime, gitBranchOwnershipVersion: 1, baseRefSha: realized.baseRefSha,
isolatedRemoteSource: { schema: "paperclip.isolated-remote-source.v1", companyId, issueId, repoUrl } } });
await db.update(issues).set({ executionWorkspaceId: workspaceId }).where(eq(issues.id, issueId));
const row = () => db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId)).then(rows => rows[0]!);
const target = path.join(await fs.realpath(root), path.relative(root, sharedPath), ".paperclip", "worktrees", "fixture-task");
return { companyId, issueId, workspaceId, projectId, cwd: realized.cwd, clone, sharedPath, target, repoUrl, remote, author, row };
}
async function makeWork(cwd: string) {
await fs.writeFile(path.join(cwd, "marker.txt"), "commit B (unpushed)\n");
git(cwd, "commit", "-am", "B unpushed");
const unpushed = git(cwd, "rev-parse", "HEAD");
await fs.writeFile(path.join(cwd, "marker.txt"), "staged edit\n");
git(cwd, "add", "marker.txt");
await fs.writeFile(path.join(cwd, "marker.txt"), "unstaged edit\n");
await fs.rm(path.join(cwd, "remove-me.txt"));
await fs.mkdir(path.join(cwd, "notes"));
await fs.writeFile(path.join(cwd, "notes", "todo.md"), "untracked work\n");
await fs.chmod(path.join(cwd, "notes", "todo.md"), 0o600);
await fs.writeFile(path.join(cwd, "build.tsbuildinfo"), "ignored output\n");
await fs.mkdir(path.join(cwd, "node_modules", "pkg"), { recursive: true });
await fs.writeFile(path.join(cwd, "node_modules", "pkg", "index.js"), "module.exports = 1;\n");
await fs.writeFile(path.join(cwd, ".env"), "LOCAL_SETTING=1\n");
await fs.mkdir(path.join(cwd, ".claude"));
await fs.writeFile(path.join(cwd, ".claude", "settings.local.json"), "{\"local\":true}\n");
return unpushed;
}
async function addLinkedNested(f: Awaited<ReturnType<typeof fixture>>) {
const location = path.dirname(f.remote), author = path.join(location, "docs-author"), remote = path.join(location, "docs.git");
await fs.mkdir(author, { recursive: true });
git(author, "init", "--initial-branch=main");
await fs.writeFile(path.join(author, "guide.md"), "guide\n"); git(author, "add", "."); git(author, "commit", "-m", "docs");
git(location, "clone", "--bare", author, remote);
const repoUrl = pathToFileURL(remote).href;
const base = (await materializeManagedProjectWorkspace(nestedRepositoryBaseCandidates({ companyId: f.companyId, projectId: f.projectId, repoUrl })[0]!, { repoUrl })).cwd;
await ensureManagedRepositoriesIgnored(f.cwd);
const checkout = path.join(f.cwd, ".paperclip-repositories", "docs-linked");
await fs.mkdir(path.dirname(checkout), { recursive: true });
const { branchName } = await addNestedRepositoryWorktree({ base, cwd: checkout, repoUrl, repoRef: null });
await fs.writeFile(path.join(checkout, "guide.md"), "edited in the nested worktree\n");
return { base, checkout, branchName, realBase: await fs.realpath(base) };
}
const worktreePaths = (base: string) => git(base, "worktree", "list", "--porcelain").split("\n").filter(line => line.startsWith("worktree ")).map(line => line.slice(9));
const gitView = (cwd: string) => ({ branch: git(cwd, "rev-parse", "--abbrev-ref", "HEAD"), head: git(cwd, "rev-parse", "HEAD"),
status: git(cwd, "status", "--porcelain=v1", "--untracked-files=all"), reflog: git(cwd, "reflog", "show", "--format=%H", "HEAD") });
async function expectUntouched(f: Awaited<ReturnType<typeof fixture>>, before: Awaited<ReturnType<typeof currentRestorePointSnapshot>>, cwd = f.cwd) {
const current = await f.row();
expect(current.cwd).toBe(f.cwd);
expect(current.metadata?.isolatedRemoteSource).toBeDefined();
expect(current.metadata?.isolatedRepositoryRehome).toBeUndefined();
expect(await currentRestorePointSnapshot(cwd)).toEqual(before);
expect(await exists(f.target)).toBe(false);
if (await exists(f.sharedPath)) expect(() => git(f.sharedPath, "rev-parse", "--verify", "-q", "refs/heads/fixture-task")).toThrow();
}
it("re-homes an open workspace onto the shared project clone with its exact state", async () => {
const f = await fixture();
const unpushed = await makeWork(f.cwd);
await ensureManagedRepositoriesIgnored(f.cwd);
await fs.appendFile(path.join(f.cwd, ".gitignore"), "/.paperclip-repositories/\n");
const nested = path.join(f.cwd, ".paperclip-repositories", "docs");
await fs.mkdir(nested, { recursive: true });
git(nested, "init", "--initial-branch=main");
await fs.writeFile(path.join(nested, "readme.md"), "nested\n");
await fs.writeFile(path.join(nested, ".gitignore"), "node_modules/\n"); git(nested, "add", "."); git(nested, "commit", "-m", "nested");
await fs.mkdir(path.join(nested, "node_modules", "dep"), { recursive: true });
await fs.writeFile(path.join(nested, "node_modules", "dep", "index.js"), "nested dependency\n");
const before = await currentRestorePointSnapshot(f.cwd);
const view = gitView(f.cwd);
const result = await rehomeIsolatedRepositoryWorkspace(db, await f.row());
expect(result).toMatchObject({ outcome: "rehomed", rehomed: true, cloneRemoved: true, reason: null, cloneKeptReason: null, to: f.target });
expect(result.reclaimedBytes).toBeGreaterThan(0);
expect(await currentRestorePointSnapshot(f.target)).toEqual(before);
expect(gitView(f.target)).toEqual(view);
expect(view.head).toBe(unpushed);
expect(git(f.target, "rev-parse", "--path-format=absolute", "--git-common-dir")).toBe(path.join(await fs.realpath(f.sharedPath), ".git"));
expect(git(f.target, "show", ":marker.txt")).toBe("staged edit");
expect(await fs.readFile(path.join(f.target, "marker.txt"), "utf8")).toBe("unstaged edit\n");
expect(await exists(path.join(f.target, "remove-me.txt"))).toBe(false);
expect((await fs.stat(path.join(f.target, "notes", "todo.md"))).mode & 0o777).toBe(0o600);
expect(await fs.readFile(path.join(f.target, "build.tsbuildinfo"), "utf8")).toBe("ignored output\n");
expect(await fs.readFile(path.join(f.target, "node_modules", "pkg", "index.js"), "utf8")).toBe("module.exports = 1;\n");
expect(await fs.readFile(path.join(f.target, ".env"), "utf8")).toBe("LOCAL_SETTING=1\n");
expect(await fs.readFile(path.join(f.target, ".claude", "settings.local.json"), "utf8")).toBe("{\"local\":true}\n");
expect(git(path.join(f.target, ".paperclip-repositories", "docs"), "log", "-1", "--format=%s")).toBe("nested");
expect(await fs.readFile(path.join(f.target, ".paperclip-repositories", "docs", "node_modules", "dep", "index.js"), "utf8")).toBe("nested dependency\n");
const row = await f.row();
expect(row).toMatchObject({ cwd: f.target, providerRef: f.target, branchName: "fixture-task" });
expect(row.metadata?.isolatedRemoteSource).toBeUndefined();
expect(row.metadata?.isolatedRepositoryRehome).toMatchObject({ from: f.cwd, to: f.target, cleanedAt: expect.any(String) });
expect(await exists(f.cwd)).toBe(false);
expect(await exists(f.clone)).toBe(false);
expect(await exists(isolatedRemoteSourcePath(f.companyId, f.issueId))).toBe(false);
expect(await rehomeIsolatedRepositoryWorkspace(db, row)).toMatchObject({ rehomed: false, reason: "not_candidate" });
});
it("moves worktree-backed nested checkouts and repairs their registration in their shared clone", async () => {
const f = await fixture();
await makeWork(f.cwd);
const nested = await addLinkedNested(f);
const oldCheckout = await fs.realpath(nested.checkout);
const previousPrefix = await nestedWorktreeBranchPrefix(oldCheckout);
const before = await currentRestorePointSnapshot(f.cwd);
const result = await rehomeIsolatedRepositoryWorkspace(db, await f.row());
expect(result).toMatchObject({ rehomed: true, cloneRemoved: true, reason: null });
expect(await currentRestorePointSnapshot(f.target)).toEqual(before);
const moved = path.join(f.target, ".paperclip-repositories", "docs-linked");
expect(await readLinkedNestedCheckout(moved)).toMatchObject({ base: nested.realBase, checkout: moved });
expect(worktreePaths(nested.base)).toContain(moved);
expect(worktreePaths(nested.base)).not.toContain(oldCheckout);
expect(nested.branchName!.startsWith(previousPrefix)).toBe(true);
const renamed = `${await nestedWorktreeBranchPrefix(moved)}${nested.branchName!.slice(previousPrefix.length)}`;
expect(renamed).not.toBe(nested.branchName);
expect(git(moved, "symbolic-ref", "--short", "HEAD")).toBe(renamed);
expect(() => git(nested.base, "rev-parse", "--verify", "-q", `refs/heads/${nested.branchName}`)).toThrow();
expect(git(moved, "status", "--porcelain")).toBe("M guide.md");
expect(git(nested.base, "worktree", "list", "--porcelain")).not.toContain("prunable");
});
it("restores the registration of worktree-backed nested checkouts when a move rolls back", async () => {
const f = await fixture();
await makeWork(f.cwd);
const nested = await addLinkedNested(f);
const oldCheckout = await fs.realpath(nested.checkout);
const result = await rehomeIsolatedRepositoryWorkspace(db, await f.row(), { hooks: { beforeVerify: async ({ from, to }) => {
expect(worktreePaths(nested.base)).toContain(path.join(to, ".paperclip-repositories", "docs-linked"));
await fs.writeFile(path.join(from, "late.txt"), "written by a concurrent run\n");
} } });
expect(result).toMatchObject({ rehomed: false, reason: "snapshot_mismatch" });
expect(await readLinkedNestedCheckout(nested.checkout)).toMatchObject({ base: nested.realBase, checkout: oldCheckout });
expect(worktreePaths(nested.base)).toContain(oldCheckout);
expect(git(nested.checkout, "symbolic-ref", "--short", "HEAD")).toBe(nested.branchName);
expect(git(nested.checkout, "status", "--porcelain")).toBe("M guide.md");
expect(await exists(f.target)).toBe(false);
});
it("keeps a workspace with a live run", async () => {
const f = await fixture();
await makeWork(f.cwd);
const agentId = randomUUID();
await db.insert(agents).values({ id: agentId, companyId: f.companyId, name: "Fixture", role: "engineer", adapterType: "process" });
await db.insert(heartbeatRuns).values({ id: randomUUID(), companyId: f.companyId, agentId, issueId: f.issueId, status: "running" });
const before = await currentRestorePointSnapshot(f.cwd);
expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ rehomed: false, reason: "live_or_unresolved_consumer" });
await expectUntouched(f, before);
});
it("keeps a workspace whose branch already exists in the shared clone at an unrelated commit", async () => {
const f = await fixture();
await makeWork(f.cwd);
const foreign = git(f.sharedPath, "commit-tree", "HEAD^{tree}", "-p", "HEAD", "-m", "foreign");
git(f.sharedPath, "update-ref", "refs/heads/fixture-task", foreign);
const before = await currentRestorePointSnapshot(f.cwd);
expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ rehomed: false, reason: "base_branch_diverged" });
expect(git(f.sharedPath, "rev-parse", "refs/heads/fixture-task")).toBe(foreign);
git(f.sharedPath, "update-ref", "-d", "refs/heads/fixture-task");
await expectUntouched(f, before);
});
it("keeps a workspace when the project has no shared clone", async () => {
const f = await fixture({ sharedBase: false });
await makeWork(f.cwd);
const before = await currentRestorePointSnapshot(f.cwd);
expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ rehomed: false, reason: "no_shared_base" });
await expectUntouched(f, before);
});
it("keeps a workspace with state a restore point cannot transfer", async () => {
const f = await fixture();
await makeWork(f.cwd);
const nested = path.join(f.cwd, "vendor-checkout");
await fs.mkdir(nested);
git(nested, "init", "--initial-branch=main");
const view = gitView(f.cwd);
const result = await rehomeIsolatedRepositoryWorkspace(db, await f.row());
expect(result).toMatchObject({ rehomed: false, reason: "untracked_nested_repository" });
expect(gitView(f.cwd)).toEqual(view);
expect(await exists(path.join(f.cwd, "node_modules", "pkg", "index.js"))).toBe(true);
await fs.rm(nested, { recursive: true });
await expectUntouched(f, await currentRestorePointSnapshot(f.cwd));
});
it("removes the new worktree and keeps the old one when the workspace changes during the move", async () => {
const f = await fixture();
await makeWork(f.cwd);
const before = await currentRestorePointSnapshot(f.cwd);
const result = await rehomeIsolatedRepositoryWorkspace(db, await f.row(), { hooks: { beforeVerify: async ({ from, to }) => {
expect(await exists(path.join(to, "node_modules"))).toBe(true);
await fs.writeFile(path.join(from, "late.txt"), "written by a concurrent run\n");
} } });
expect(result).toMatchObject({ rehomed: false, reason: "snapshot_mismatch" });
await fs.rm(path.join(f.cwd, "late.txt"));
await expectUntouched(f, before);
expect(await fs.readFile(path.join(f.cwd, ".env"), "utf8")).toBe("LOCAL_SETTING=1\n");
expect(await fs.readFile(path.join(f.cwd, "node_modules", "pkg", "index.js"), "utf8")).toBe("module.exports = 1;\n");
expect(git(f.sharedPath, "worktree", "list", "--porcelain").split("\n").filter(line => line.startsWith("worktree "))).toHaveLength(1);
});
it("undoes a move interrupted before the database update and then completes it", async () => {
const f = await fixture();
await makeWork(f.cwd);
const before = await currentRestorePointSnapshot(f.cwd);
const from = await fs.realpath(f.cwd);
const head = git(f.cwd, "rev-parse", "HEAD");
git(f.sharedPath, "fetch", "-q", f.clone, "refs/heads/fixture-task:refs/heads/fixture-task");
git(f.sharedPath, "worktree", "add", "-q", f.target, "fixture-task");
await fs.rename(path.join(f.cwd, "node_modules"), path.join(f.target, "node_modules"));
await fs.rename(path.join(f.cwd, ".env"), path.join(f.target, ".env"));
const pending = { base: f.sharedPath, from, target: f.target, branch: "fixture-task", head, previousTip: null, moves: [".env", "node_modules"] };
const current = await f.row();
await db.update(executionWorkspaces).set({ metadata: { ...current.metadata, isolatedRepositoryRehomePending: pending } }).where(eq(executionWorkspaces.id, f.workspaceId));
expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row(), { dryRun: true }))
.toMatchObject({ outcome: "dry_run", rehomed: false, reason: "interrupted_rehome_pending" });
expect(await exists(path.join(f.target, "node_modules", "pkg", "index.js"))).toBe(true);
expect(await exists(path.join(f.cwd, ".env"))).toBe(false);
expect(git(f.sharedPath, "rev-parse", "refs/heads/fixture-task")).toBe(head);
expect((await f.row()).metadata?.isolatedRepositoryRehomePending).toEqual(pending);
const result = await rehomeIsolatedRepositoryWorkspace(db, await f.row());
expect(result).toMatchObject({ rehomed: true, cloneRemoved: true, reason: null });
expect(await currentRestorePointSnapshot(f.target)).toEqual(before);
expect(await fs.readFile(path.join(f.target, ".env"), "utf8")).toBe("LOCAL_SETTING=1\n");
expect(await fs.readFile(path.join(f.target, "node_modules", "pkg", "index.js"), "utf8")).toBe("module.exports = 1;\n");
expect((await f.row()).metadata?.isolatedRepositoryRehomePending).toBeUndefined();
});
it("never deletes a target worktree another workspace created during the move", async () => {
const f = await fixture();
await makeWork(f.cwd);
const before = await currentRestorePointSnapshot(f.cwd);
const result = await rehomeIsolatedRepositoryWorkspace(db, await f.row(), { hooks: { beforeWorktreeAdd: async ({ to }) => {
await fs.mkdir(to, { recursive: true });
await fs.writeFile(path.join(to, "other.txt"), "another workspace\n");
} } });
expect(result).toMatchObject({ rehomed: false, reason: "target_worktree_exists" });
expect(await fs.readFile(path.join(f.target, "other.txt"), "utf8")).toBe("another workspace\n");
await fs.rm(f.target, { recursive: true });
await expectUntouched(f, before);
expect((await f.row()).metadata?.isolatedRepositoryRehomePending).toBeUndefined();
});
it("keeps both copies when a file reappears in the old worktree before verification", async () => {
const f = await fixture();
await makeWork(f.cwd);
const result = await rehomeIsolatedRepositoryWorkspace(db, await f.row(), { hooks: { beforeVerify: async ({ from }) => {
await fs.writeFile(path.join(from, ".env"), "RECREATED=1\n");
} } });
expect(result).toMatchObject({ rehomed: false, reason: "rehome_rollback_conflict" });
expect(await fs.readFile(path.join(f.target, ".env"), "utf8")).toBe("LOCAL_SETTING=1\n");
expect(await fs.readFile(path.join(f.cwd, ".env"), "utf8")).toBe("RECREATED=1\n");
expect((await f.row()).metadata?.isolatedRepositoryRehomePending).toMatchObject({ target: f.target });
expect((await f.row()).cwd).toBe(f.cwd);
expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ rehomed: false, reason: "rehome_rollback_conflict" });
expect(await fs.readFile(path.join(f.target, ".env"), "utf8")).toBe("LOCAL_SETTING=1\n");
await fs.rm(path.join(f.cwd, ".env"));
expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ rehomed: true, reason: null });
expect(await fs.readFile(path.join(f.target, ".env"), "utf8")).toBe("LOCAL_SETTING=1\n");
});
it("refuses to check out from a shared clone whose config runs filters", async () => {
const f = await fixture();
await makeWork(f.cwd);
const marker = path.join(path.dirname(f.remote), "smudge-ran");
git(f.sharedPath, "config", "filter.evil.smudge", `sh -c 'touch ${marker}; cat'`);
git(f.sharedPath, "config", "filter.evil.required", "true");
const before = await currentRestorePointSnapshot(f.cwd);
expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ rehomed: false, reason: "unsupported_base_git_config" });
expect(await exists(marker)).toBe(false);
await expectUntouched(f, before);
});
it("keeps a clone with more refs than the cleanup budget", async () => {
const f = await fixture();
await makeWork(f.cwd);
await rehomeIsolatedRepositoryWorkspace(db, await f.row(), { hooks: { afterDatabaseUpdate: async () => { throw new Error("simulated crash"); } } });
const head = git(f.clone, "rev-parse", "HEAD");
execFileSync("git", ["-C", f.clone, "update-ref", "--stdin"], { input: Array.from({ length: 5_001 }, (_, index) => `create refs/tags/extra-${index} ${head}`).join("\n") + "\n" });
expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ cloneRemoved: false, cloneKeptReason: "clone_history_budget" });
expect(await exists(f.clone)).toBe(true);
expect((await f.row()).metadata?.isolatedRepositoryRehome).toMatchObject({ cleanedAt: null });
execFileSync("git", ["-C", f.clone, "update-ref", "--stdin"], { input: Array.from({ length: 5_001 }, (_, index) => `delete refs/tags/extra-${index}`).join("\n") + "\n" });
expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row())).toMatchObject({ outcome: "cleaned", cloneRemoved: true });
});
it("keeps the clone while another workspace still references it", async () => {
const f = await fixture();
await makeWork(f.cwd);
await db.insert(executionWorkspaces).values({ companyId: f.companyId, projectId: f.projectId, sourceIssueId: f.issueId, mode: "isolated_workspace",
strategyType: "git_worktree", name: "Sibling", status: "active", providerType: "local_fs", cwd: path.join(f.clone, "sibling"), repoUrl: f.repoUrl,
metadata: { isolatedRemoteSource: { schema: "paperclip.isolated-remote-source.v1", companyId: f.companyId, issueId: f.issueId, repoUrl: f.repoUrl } } });
const result = await rehomeIsolatedRepositoryWorkspace(db, await f.row());
expect(result).toMatchObject({ rehomed: true, cloneRemoved: false, cloneKeptReason: "base_has_consumers" });
expect(await exists(f.clone)).toBe(true);
expect(await exists(f.cwd)).toBe(false);
expect((await f.row()).metadata?.isolatedRepositoryRehome).toMatchObject({ cloneRetained: "base_has_consumers", cleanedAt: expect.any(String) });
});
it("finishes removing the clone on a later sweep after a crash following the database update", async () => {
const f = await fixture();
await makeWork(f.cwd);
const before = await currentRestorePointSnapshot(f.cwd);
const crashed = await rehomeIsolatedRepositoryWorkspace(db, await f.row(), { hooks: { afterDatabaseUpdate: async () => { throw new Error("simulated crash"); } } });
expect(crashed).toMatchObject({ rehomed: true, cloneRemoved: false });
expect(await f.row()).toMatchObject({ cwd: f.target, metadata: expect.objectContaining({ isolatedRepositoryRehome: expect.objectContaining({ cleanedAt: null }) }) });
expect(await exists(f.clone)).toBe(true);
expect(await currentRestorePointSnapshot(f.target)).toEqual(before);
await db.update(executionWorkspaces).set({ status: "archived", closedAt: new Date() }).where(eq(executionWorkspaces.id, f.workspaceId));
const service = executionWorkspaceService(db, { gitCacheExpirationEnabled: true, isolatedSourceRehomeEnabled: true });
await service.sweepTerminalWorkspaces();
expect(await exists(f.clone)).toBe(false);
expect(await exists(f.cwd)).toBe(false);
expect((await f.row()).metadata?.isolatedRepositoryRehome).toMatchObject({ cleanedAt: expect.any(String) });
expect(await currentRestorePointSnapshot(f.target)).toEqual(before);
expect(await service.nextTerminalWorkspaceSweepAt()).toBeNull();
});
it("is disabled by default and changes nothing in dry run", async () => {
const f = await fixture();
await makeWork(f.cwd);
const before = await currentRestorePointSnapshot(f.cwd);
await executionWorkspaceService(db, { gitCacheExpirationEnabled: true }).sweepTerminalWorkspaces();
await executionWorkspaceService(db, { isolatedSourceRehomeEnabled: true }).sweepTerminalWorkspaces();
await expectUntouched(f, before);
const dryRun = executionWorkspaceService(db, { gitCacheExpirationEnabled: true, isolatedSourceRehomeEnabled: true, isolatedSourceRehomeDryRun: true });
await dryRun.sweepTerminalWorkspaces();
await expectUntouched(f, before);
expect(await exists(f.clone)).toBe(true);
expect(await dryRun.nextTerminalWorkspaceSweepAt()).toBeGreaterThan(Date.now());
expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row(), { dryRun: true })).toMatchObject({ outcome: "dry_run", rehomed: false, to: f.target });
await expectUntouched(f, before);
});
it("restores an expired isolated workspace into the shared project clone without recreating its clone", async () => {
const f = await fixture({ issueStatus: "done" });
const unpushed = await makeWork(f.cwd);
await fs.rm(path.join(f.cwd, "node_modules"), { recursive: true });
const expired = await expireReconstructableGitWorkspace(db, await f.row(), { dryRun: false, expireBase: true, now: new Date(),
restorePoint: { storage: getStorageService(), notBefore: new Date(Date.now() - DAY_MS), keyDurable: true } });
expect(expired).toMatchObject({ expired: true, restorePoint: true });
await fs.rm(f.clone, { recursive: true, force: true });
const reopened = await executionWorkspaceService(db).reopenClosedIsolatedExecutionWorkspaceForIssue({ workspaceId: f.workspaceId,
issue: { id: f.issueId, companyId: f.companyId, projectId: f.projectId }, actor: { actorType: "user", agentId: null },
authorizationActor: { type: "board", source: "local_implicit", isInstanceAdmin: true } });
expect(reopened).toMatchObject({ ok: true, reopened: true });
expect(await exists(f.clone)).toBe(false);
const row = await f.row();
expect(row).toMatchObject({ status: "active", cwd: f.target, providerRef: f.target });
expect(row.metadata?.isolatedRemoteSource).toBeUndefined();
expect(row.metadata?.gitCacheRestoration).toMatchObject({ sourceKind: "managed_project", sourcePath: f.sharedPath, worktreePath: f.target,
restorePoint: { appliedAt: expect.any(String) } });
expect(git(f.target, "rev-parse", "HEAD")).toBe(unpushed);
expect(git(f.target, "rev-parse", "--path-format=absolute", "--git-common-dir")).toBe(path.join(await fs.realpath(f.sharedPath), ".git"));
expect(git(f.target, "show", ":marker.txt")).toBe("staged edit");
expect(await fs.readFile(path.join(f.target, "marker.txt"), "utf8")).toBe("unstaged edit\n");
expect(await fs.readFile(path.join(f.target, "notes", "todo.md"), "utf8")).toBe("untracked work\n");
expect(await fs.readFile(path.join(f.target, "build.tsbuildinfo"), "utf8")).toBe("ignored output\n");
});
it("restores through the private clone when the shared clone lacks the restore point's base commits", async () => {
const f = await fixture({ issueStatus: "done" });
await fs.writeFile(path.join(f.author, "marker.txt"), "commit C\n");
git(f.author, "commit", "-qam", "C");
git(f.author, "push", "-q", f.remote, "main");
const upstream = git(f.author, "rev-parse", "HEAD");
git(f.cwd, "fetch", "-q", "origin");
git(f.cwd, "reset", "-q", "--hard", "origin/main");
await fs.writeFile(path.join(f.cwd, "marker.txt"), "commit D (unpushed)\n");
git(f.cwd, "commit", "-qam", "D unpushed");
const unpushed = git(f.cwd, "rev-parse", "HEAD");
await fs.writeFile(path.join(f.cwd, "notes.txt"), "untracked work\n");
const expired = await expireReconstructableGitWorkspace(db, await f.row(), { dryRun: false, expireBase: true, now: new Date(),
restorePoint: { storage: getStorageService(), notBefore: new Date(Date.now() - DAY_MS), keyDurable: true } });
expect(expired).toMatchObject({ expired: true, restorePoint: true });
await fs.rm(f.clone, { recursive: true, force: true });
expect(() => git(f.sharedPath, "cat-file", "-e", `${upstream}^{commit}`)).toThrow();
const reopened = await executionWorkspaceService(db).reopenClosedIsolatedExecutionWorkspaceForIssue({ workspaceId: f.workspaceId,
issue: { id: f.issueId, companyId: f.companyId, projectId: f.projectId }, actor: { actorType: "user", agentId: null },
authorizationActor: { type: "board", source: "local_implicit", isInstanceAdmin: true } });
expect(reopened).toMatchObject({ ok: true, reopened: true });
const row = await f.row();
expect(row.cwd).toBe(f.cwd);
expect(row.metadata?.isolatedRemoteSource).toBeDefined();
expect(git(f.cwd, "rev-parse", "HEAD")).toBe(unpushed);
expect(await fs.readFile(path.join(f.cwd, "notes.txt"), "utf8")).toBe("untracked work\n");
});
});
server/src/config.ts
+4
View File
@@ -82,6 +82,8 @@ export interface Config {
gitCacheExpirationEnabled: boolean;
gitBaseExpirationEnabled: boolean;
gitCacheExpirationDryRun: boolean;
isolatedSourceRehomeEnabled: boolean;
isolatedSourceRehomeDryRun: boolean;
workspaceReaperCooldownDays: number;
workspaceCompactionDelayHours: number | null;
skillVersionRetentionEnabled: boolean;
@@ -396,6 +398,8 @@ export function loadConfig(): Config {
gitCacheExpirationEnabled: process.env.PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED === "true",
gitBaseExpirationEnabled: process.env.PAPERCLIP_GIT_BASE_EXPIRATION_ENABLED === "true",
gitCacheExpirationDryRun: process.env.PAPERCLIP_GIT_CACHE_EXPIRATION_DRY_RUN !== "false",
isolatedSourceRehomeEnabled: process.env.PAPERCLIP_ISOLATED_SOURCE_REHOME_ENABLED === "true",
isolatedSourceRehomeDryRun: process.env.PAPERCLIP_ISOLATED_SOURCE_REHOME_DRY_RUN === "true",
workspaceReaperCooldownDays,
workspaceCompactionDelayHours,
skillVersionRetentionEnabled: process.env.PAPERCLIP_SKILL_VERSION_RETENTION_ENABLED !== "false",
server/src/index.ts
+2
View File
@@ -1485,6 +1485,8 @@ async function startServerWithDatabaseTeardown(
gitCacheExpirationEnabled: config.gitCacheExpirationEnabled,
gitBaseExpirationEnabled: config.gitBaseExpirationEnabled,
gitCacheExpirationDryRun: config.gitCacheExpirationDryRun,
isolatedSourceRehomeEnabled: config.isolatedSourceRehomeEnabled,
isolatedSourceRehomeDryRun: config.isolatedSourceRehomeDryRun,
gitRestorePoints: config.gitRestorePointsEnabled ? {
storage: storageService,
terminalDelayMs: config.gitRestorePointTerminalDelayDays * 24 * 60 * 60 * 1000,
server/src/services/execution-workspaces.ts
+38 -5
View File
@@ -6,6 +6,7 @@ import { measureRunPhase } from "./execution-preparation-timing.js";
import { assertExecutionWorkspaceRecordAccess, assertTaskWorkspaceExecutionAccess, assertTaskWorkspaceSourceProjectAccess } from "./task-workspace-access.js";
import { verifiedRecordedIsolatedRemoteSource, withIsolatedRepositorySourceLock } from "./isolated-remote-workspace-source.js";
import { expireReconstructableGitWorkspace, gitWorkspaceLastActivityAt, hasPendingGitExpiration } from "./git-workspace-expiration.js";
import { hasPendingIsolatedRepositoryRehomeCleanup, isIsolatedRepositoryRehomeCandidate, rehomeExpiredIsolatedRestoration, rehomeIsolatedRepositoryWorkspace } from "./isolated-repository-rehome.js";
import type { StorageService } from "../storage/types.js";
import { readGitCacheRestoration, restoreGitCacheBase, recoverGitCacheExpirationLocks, GitCacheRestorationFailure, gitRestorePointExpired } from "./git-workspace-restoration.js";
@@ -301,6 +302,8 @@ export type ExecutionWorkspaceServiceOptions = {
gitCacheExpirationEnabled?: boolean;
gitBaseExpirationEnabled?: boolean;
gitCacheExpirationDryRun?: boolean;
isolatedSourceRehomeEnabled?: boolean;
isolatedSourceRehomeDryRun?: boolean;
gitRestorePoints?: { storage: StorageService; terminalDelayMs: number; keyDurable: boolean; idleMs?: number | null };
terminalWorkspaceInspectionBackoffMs?: number;
resolvePullRequestDetails?: PullRequestMergeDetailsResolver;
@@ -1742,6 +1745,9 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
let terminalSweepCursor: { updatedAt: Date; id: string } | null = null;
const terminalInspections = new Map<string, { revision: string; issueIds: Set<string>; nextAt: number }>();
const terminalWorkspaceDeadlines = new Map<string, number>();
const isolatedRehomeDeadlines = new Map<string, number>();
const isolatedRehomeNotBefore = new Map<string, number>();
const ISOLATED_REHOME_RETRY_MS = 60 * 60 * 1000;
let terminalSweepRescanRequested = false;
// The reaper freezes an upper bound on updatedAt at the start of each
// rotation. The scan only reads candidates at or below the bound, so a steady
@@ -3246,7 +3252,10 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
and(eq(executionWorkspaces.providerType, "plugin"), inArray(executionWorkspaces.status, ["archived", "cleanup_failed"]),
sql`(${executionWorkspaces.metadata}->>'workspaceProviderRemoved') IS DISTINCT FROM 'true'`),
opts.gitCacheExpirationEnabled ? and(inArray(executionWorkspaces.status, ["archived", "cleanup_failed"]),
sql`${executionWorkspaces.metadata}->'gitCacheExpiration'->>'completed' = 'false'`) : undefined),
sql`${executionWorkspaces.metadata}->'gitCacheExpiration'->>'completed' = 'false'`) : undefined,
opts.gitCacheExpirationEnabled && opts.isolatedSourceRehomeEnabled ? and(inArray(executionWorkspaces.status, ["archived", "cleanup_failed"]),
sql`(${executionWorkspaces.metadata}->'isolatedRepositoryRehome') IS NOT NULL`,
sql`(${executionWorkspaces.metadata}->'isolatedRepositoryRehome'->>'cleanedAt') IS NULL`) : undefined),
);
// Continue the scan after the previous sweep's last row. The keyset
// predicate uses the same (updatedAt, id) order as the query, so each
@@ -3261,6 +3270,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
if (!cursor) {
terminalSweepBoundary = now();
terminalWorkspaceDeadlines.clear();
isolatedRehomeDeadlines.clear();
terminalSweepRescanRequested = false;
}
const boundary = terminalSweepBoundary;
@@ -3311,12 +3321,35 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
};
const readinessByWorkingTree = new Map<string, ReturnType<typeof inspectGitCloseReadiness>>();
for (const workspace of candidates) {
for (let workspace of candidates) {
signal?.throwIfAborted();
// Trash deletion is part of compaction, so a sweep that skips compaction
// also leaves leftover trash for a later sweep.
if (options.compact !== false) await purgeCompactionTrashIfDue(workspace);
else deferCompactionTrashPurge(workspace);
if (opts.gitCacheExpirationEnabled && opts.isolatedSourceRehomeEnabled
&& (isIsolatedRepositoryRehomeCandidate(workspace) || hasPendingIsolatedRepositoryRehomeCleanup(workspace))) {
const dueAt = isolatedRehomeNotBefore.get(workspace.id) ?? 0;
if (dueAt > now().getTime()) isolatedRehomeDeadlines.set(workspace.id, dueAt);
else {
const dryRun = opts.isolatedSourceRehomeDryRun ?? false;
const rehome = await rehomeIsolatedRepositoryWorkspace(db, workspace, { dryRun, now: now(), signal });
logger.info({ event: "execution_workspace.isolated_source_rehome", executionWorkspaceId: workspace.id,
companyId: workspace.companyId, dryRun, ...rehome }, "Isolated repository source re-home inspected");
const [after] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace.id));
if (after && (isIsolatedRepositoryRehomeCandidate(after) || hasPendingIsolatedRepositoryRehomeCleanup(after))) {
if (isolatedRehomeNotBefore.size >= 4096) isolatedRehomeNotBefore.delete(isolatedRehomeNotBefore.keys().next().value!);
isolatedRehomeNotBefore.set(workspace.id, now().getTime() + ISOLATED_REHOME_RETRY_MS);
isolatedRehomeDeadlines.set(workspace.id, now().getTime() + ISOLATED_REHOME_RETRY_MS);
} else {
isolatedRehomeNotBefore.delete(workspace.id);
isolatedRehomeDeadlines.delete(workspace.id);
}
if (!after) continue;
workspace = after;
}
if (isClosedExecutionWorkspaceStatus(workspace.status) && !hasPendingGitExpiration(workspace)) continue;
}
const issueTree = await listWorkspaceIssueTree(workspace);
const sourceIssue = issueTree.find((issue) => issue.id === workspace.sourceIssueId);
const reopenPending = metadataHasReopenPendingConsumption(
@@ -3752,8 +3785,8 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
},
nextTerminalWorkspaceSweepAt: async () => {
if (terminalWorkspaceDeadlines.size === 0) return null;
const next = [...terminalWorkspaceDeadlines.values()].reduce((earliest, deadline) => Math.min(earliest, deadline), Infinity);
if (terminalWorkspaceDeadlines.size === 0 && isolatedRehomeDeadlines.size === 0) return null;
const next = [...terminalWorkspaceDeadlines.values(), ...isolatedRehomeDeadlines.values()].reduce((earliest, deadline) => Math.min(earliest, deadline), Infinity);
const nowMs = now().getTime();
if (next <= nowMs && opts.terminalWorkspaceInspectionBackoffMs) {
return Math.min(nowMs + opts.terminalWorkspaceInspectionBackoffMs,
@@ -3825,7 +3858,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic
if (!freshSnapshot || freshSnapshot.sourceIssueId !== sourceIssueId) {
return { ok: false, code: "not_reopenable", message: "Execution workspace source changed during reopen" };
}
const snapshot = freshSnapshot;
const snapshot = freshSnapshot.providerType === "plugin" ? freshSnapshot : await rehomeExpiredIsolatedRestoration(db, freshSnapshot);
if (snapshot.providerType === "plugin") {
return withExecutionWorkspaceLifecycleLock(db, snapshot.id, async tx => {
const [row] = await tx.select().from(executionWorkspaces).where(and(
server/src/services/git-restore-point-storage.ts
+4 -2
View File
@@ -178,10 +178,12 @@ export async function restorePointHistoryUnchanged(ref: GitRestorePointRef, cwd:
return ref.history === undefined || await restorePointHistoryDigest(cwd, signal) === ref.history;
}
const modesKey = (modes: Array<{ path: string; mode: number }>) => JSON.stringify(modes.map(entry => [entry.path, entry.mode]));
export function restorePointSnapshotMatches(ref: GitRestorePointRef, snapshot: { head: string; index: string | null; worktree: string | null;
ignored: string | null; modes: Array<{ path: string; mode: number }> }) {
return snapshot.head === ref.head && snapshot.index === ref.index && snapshot.worktree === ref.worktree && snapshot.ignored === (ref.ignored ?? null)
&& JSON.stringify(snapshot.modes) === JSON.stringify(ref.modes);
&& modesKey(snapshot.modes) === modesKey(ref.modes);
}
export async function abandonGitRestorePointsByKey(db: Db, workspaceId: string, objectKeys: string[]) {
@@ -234,7 +236,7 @@ export async function loadGitRestorePoint(input: {
const { manifest } = decodeRestorePointPayload(payload);
if (manifest.head !== input.ref.head || manifest.index !== input.ref.index || manifest.worktree !== input.ref.worktree
|| (manifest.ignored ?? null) !== (input.ref.ignored ?? null)
|| JSON.stringify(manifest.modes) !== JSON.stringify(input.ref.modes)) {
|| modesKey(manifest.modes) !== modesKey(input.ref.modes)) {
throw new RestorePointRestoreFailure("restore_point_manifest_mismatch");
}
return { payload, manifest };
server/src/services/git-workspace-expiration.ts
+2 -2
View File
@@ -17,7 +17,7 @@ import type { GitRemoteAuthProvider } from "./git-credentials.js";
import { assertReproducibleGitFiles, assertReproducibleBaseGitStorage, assertReproducibleManagedCheckout, provenManagedCheckoutHistory, inspectGitCacheRecipe, inspectGitCacheRecipeForRestorePoint, readGitCacheRestoration, GIT_CACHE_RESTORATION_SCHEMA_V2,
verifyRemoteGitCommit, gitCacheExpirationLockLocations, recoverGitCacheExpirationLocks, GitCacheRestorationFailure, type ManagedCheckoutProver, gitCacheCommand, readGitCacheExpiration, finishGitCacheTrash, type GitCacheExpiration, type GitCacheRestoration, GIT_CACHE_RESTORATION_KEY, GIT_CACHE_EXPIRATION_REASON } from "./git-workspace-restoration.js";
async function allocatedBytes(target: string, shouldContinue: () => boolean): Promise<number> {
export async function allocatedBytes(target: string, shouldContinue: () => boolean): Promise<number> {
if (!shouldContinue()) throw new Error("scan_stopped");
const entry = await fs.lstat(target);
if (!entry.isDirectory() && !entry.isFile() && !entry.isSymbolicLink()) throw new Error("unsupported_filesystem_entry");
@@ -100,7 +100,7 @@ async function restorePointBoundary(cwd: string, repoUrl: string, headOnRemote:
}
/** Re-read the whole tree and every runtime/recovery consumer immediately before deletion. */
function expirationScope(workspace: Workspace) {
export function expirationScope(workspace: Workspace) {
return sql`
WITH RECURSIVE tree AS (
SELECT id, status, completed_at, cancelled_at, updated_at FROM ${issues}
server/src/services/git-workspace-restoration.ts
+2 -2
View File
@@ -54,7 +54,7 @@ type Workspace = {
cwd: string | null; providerRef: string | null; status?: string; metadata: Record<string, unknown> | null;
};
function managedSourcePaths(workspace: Workspace) {
export function managedSourcePaths(workspace: Workspace) {
if (!workspace.projectId || !workspace.repoUrl) return [];
let repoName: string | null = null;
try { repoName = new URL(workspace.repoUrl).pathname.replace(/\/+$/, "").split("/").pop()?.replace(/\.git$/i, "") || null; } catch { return []; }
@@ -62,7 +62,7 @@ function managedSourcePaths(workspace: Workspace) {
return [root, `${root}-${createHash("sha256").update(workspace.repoUrl).digest("hex").slice(0, 12)}`];
}
async function verifiedGitCacheSource(workspace: Workspace, sourcePath: string) {
export async function verifiedGitCacheSource(workspace: Workspace, sourcePath: string) {
const privateSource = recordedIsolatedRemoteSource(workspace);
if (privateSource) {
if (sourcePath !== privateSource) throw new GitCacheRestorationFailure();
server/src/services/isolated-repository-rehome.ts
+622
View File
@@ -0,0 +1,622 @@
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { createHash } from "node:crypto";
import { execFile as execFileCallback } from "node:child_process";
import { promisify } from "node:util";
import { and, eq, ne, sql } from "drizzle-orm";
import { executionWorkspaces, projectWorkspaces, workspaceRuntimeServices, heartbeatRuns, nativeRunFinalizations, environmentLeases,
withDedicatedDbConnection, type Db } from "@paperclipai/db";
import { PROJECT_REPOSITORIES_DIR } from "@paperclipai/adapter-utils/git-workspace-sync";
import { IsolatedRepositorySourceBusy, isolatedRemoteSourcePath, verifiedRecordedIsolatedRemoteSource, withIsolatedRepositorySourceLock } from "./isolated-remote-workspace-source.js";
import { acquireGitExpirationLocks, recoverGitExpirationLocks, type GitExpirationLockLocation } from "./git-workspace-expiration-locks.js";
import { allocatedBytes, describeGitExpirationFailure, expirationScope, gitExpirationSkipReason, type GitExpirationFailure } from "./git-workspace-expiration.js";
import { finishGitCacheTrash, managedSourcePaths, readGitCacheExpiration, readGitCacheRestoration, verifiedGitCacheSource,
GIT_CACHE_RESTORATION_KEY, type GitCacheRestoration } from "./git-workspace-restoration.js";
import { applyRestorePoint, captureRestorePoint, currentRestorePointSnapshot, importRestorePoint, restorePointSavesIgnoredFile,
RestorePointRestoreFailure, RestorePointSkip, type RestorePointLimits, type RestorePointSnapshot } from "./git-restore-point.js";
import { findRegenerableWorkspaceDirs, REGENERABLE_WORKSPACE_DIR_NAMES } from "./workspace-compaction.js";
import { ensureManagedRepositoriesIgnored } from "./managed-repository-checkout.js";
import { withWorkspaceLockKeepalive } from "./native-runtime/native-workspace-lock-keepalive.js";
import { canonicalRepositoryRoot } from "./execution-workspace-source-roots.js";
import { withExecutionWorkspaceLifecycleLock } from "./execution-workspace-lifecycle-lock.js";
import { loadGitRestorePoint } from "./git-restore-point-storage.js";
import { nestedWorktreeBranchPrefix, readLinkedNestedCheckout, repairMovedNestedRepositoryWorktree, withNestedRepositoryBaseLock } from "./nested-repository-worktrees.js";
import { getStorageService } from "../storage/index.js";
const execFile = promisify(execFileCallback);
const SHA = /^[0-9a-f]{40}$/;
const OPEN_STATUSES = ["active", "idle", "in_review"];
const CLOSED_STATUSES = ["archived", "cleanup_failed"];
const ADMIN_ENTRIES = new Set(["HEAD", "ORIG_HEAD", "FETCH_HEAD", "index", "commondir", "gitdir", "logs", "refs", "COMMIT_EDITMSG"]);
const MAX_BOUNDARY_COMMITS = 64;
const CLONE_HISTORY_BUDGET = 5_000;
const UNSAFE_BASE_CONFIG = /^(filter\.|include\.|includeif\.|core\.(fsmonitor|hookspath|attributesfile|worktree|gitproxy|sshcommand|askpass|editor|pager|alternaterefscommand)$|extensions\.worktreeconfig$)/;
export const ISOLATED_REPOSITORY_REHOME_KEY = "isolatedRepositoryRehome";
export const ISOLATED_REPOSITORY_REHOME_PENDING_KEY = "isolatedRepositoryRehomePending";
type Workspace = typeof executionWorkspaces.$inferSelect;
type Tx = Parameters<Parameters<Db["transaction"]>[0]>[0];
export type IsolatedRepositoryRehomeRecord = { at: string; from: string; to: string; cleanedAt: string | null; cloneRetained?: string };
export type IsolatedRepositoryRehomePending = { base: string; from: string; target: string; branch: string; head: string; previousTip: string | null; moves: string[] };
export type IsolatedRepositoryRehomeHooks = {
beforeWorktreeAdd?: (input: { from: string; to: string }) => Promise<void>;
beforeVerify?: (input: { from: string; to: string }) => Promise<void>;
afterDatabaseUpdate?: (workspace: Workspace) => Promise<void>;
};
export type IsolatedRepositoryRehomeResult = {
outcome: "rehomed" | "dry_run" | "skipped" | "cleaned" | "clone_kept";
rehomed: boolean; cloneRemoved: boolean; reason: string | null; cloneKeptReason: string | null;
candidateBytes: number; reclaimedBytes: number; from: string | null; to: string | null; failure?: GitExpirationFailure;
};
export function isIsolatedRepositoryRehomeCandidate(workspace: Workspace) {
return workspace.providerType !== "plugin" && workspace.mode === "isolated_workspace" && workspace.strategyType === "git_worktree"
&& OPEN_STATUSES.includes(workspace.status) && !workspace.closedAt && workspace.metadata?.isolatedRemoteSource !== undefined;
}
function readRehome(workspace: Pick<Workspace, "metadata">) {
const value = workspace.metadata?.[ISOLATED_REPOSITORY_REHOME_KEY] as IsolatedRepositoryRehomeRecord | undefined;
return value && typeof value === "object" && typeof value.from === "string" && typeof value.to === "string" ? value : null;
}
export function hasPendingIsolatedRepositoryRehomeCleanup(workspace: Workspace) {
const rehome = readRehome(workspace);
return Boolean(rehome && !rehome.cleanedAt && workspace.providerType !== "plugin");
}
async function git(cwd: string, args: string[], options: { signal?: AbortSignal; input?: string; timeout?: number } = {}) {
const child = execFile("git", ["-c", "core.fsmonitor=false", "-c", `core.hooksPath=${os.devNull}`, "-C", cwd, ...args], {
timeout: options.timeout ?? 120_000, signal: options.signal, maxBuffer: 256 * 1024 * 1024,
env: { ...process.env, GIT_TERMINAL_PROMPT: "0", GIT_OPTIONAL_LOCKS: "0" },
});
if (options.input !== undefined) child.child.stdin?.end(options.input);
return (await child).stdout;
}
const shas = (output: string) => output.split("\n").map(line => line.trim()).filter(line => SHA.test(line));
const nul = (output: string) => output.split("\0").filter(Boolean);
async function exists(target: string) {
return fs.lstat(target).then(() => true, (error: NodeJS.ErrnoException) => { if (error.code === "ENOENT") return false; throw error; });
}
async function presentCommits(cwd: string, commits: Iterable<string>, signal?: AbortSignal) {
const unique = [...new Set(commits)].filter(sha => SHA.test(sha));
if (unique.length === 0) return new Set<string>();
const output = await git(cwd, ["cat-file", "--batch-check=%(objectname) %(objecttype)"], { signal, input: unique.join("\n") + "\n" });
return new Set(output.split("\n").map(line => line.split(" ")).filter(([sha, type]) => SHA.test(sha ?? "") && type === "commit").map(([sha]) => sha!));
}
function lockToken(workspaceId: string) {
return createHash("sha256").update(`paperclip-isolated-repository-rehome:${workspaceId}`).digest("hex");
}
async function recoverRehomeLocks(clone: string, branch: string | null, workspaceId: string) {
const worktrees = path.join(clone, ".git", "worktrees");
const admins = await fs.readdir(worktrees).catch(() => [] as string[]);
await recoverGitExpirationLocks(rehomeLockLocations(clone, admins, branch), lockToken(workspaceId)).catch(() => undefined);
}
function rehomeLockLocations(clone: string, admins: string[], branch: string | null): GitExpirationLockLocation[] {
const locations: GitExpirationLockLocation[] = admins.flatMap(admin => ["index.lock", "HEAD.lock"]
.map(name => ({ root: clone, relativePath: path.join(".git", "worktrees", admin, name), checkout: null })));
if (branch) locations.push({ root: clone, relativePath: path.join(".git", "refs", "heads", `${branch}.lock`), checkout: null });
return locations;
}
async function rehomeEligible(tx: Db | Tx, workspace: Workspace) {
if (workspace.metadata?.reopenPendingConsumption === true) return false;
const rows = await tx.execute<{ allowed: boolean }>(sql`${expirationScope(workspace)}
SELECT (
NOT EXISTS (SELECT 1 FROM ${executionWorkspaces} other WHERE other.id <> ${workspace.id}
AND (other.cwd = ${workspace.cwd} OR other.provider_ref = ${workspace.cwd}
OR left(other.cwd, length(${workspace.cwd}) + 1) = ${workspace.cwd} || '/'))
AND NOT EXISTS (SELECT 1 FROM relevant_runs WHERE status NOT IN ('succeeded', 'failed', 'timed_out', 'cancelled', 'interrupted')
OR finished_at IS NULL OR controller_lease_expires_at > now())
AND NOT EXISTS (SELECT 1 FROM ${nativeRunFinalizations} f WHERE f.company_id = ${workspace.companyId}
AND (f.issue_id IN (SELECT id FROM tree) OR f.run_id IN (SELECT id FROM relevant_runs))
AND (f.phase <> 'committed' OR f.lease_owner IS NOT NULL OR f.recovery_state IS NOT NULL))
AND NOT EXISTS (SELECT 1 FROM ${environmentLeases} l WHERE l.company_id = ${workspace.companyId}
AND (l.execution_workspace_id = ${workspace.id} OR l.heartbeat_run_id IN (SELECT id FROM relevant_runs))
AND (l.status = 'pending_cleanup' OR l.cleanup_status = 'failed' OR l.metadata ? 'nativeWorkspaceExportResume'
OR (l.released_at IS NULL AND NOT coalesce(l.status = 'retained' AND l.lease_policy = 'reuse_by_environment' AND l.cleanup_status = 'success', false))))
AND NOT EXISTS (SELECT 1 FROM ${workspaceRuntimeServices} WHERE company_id = ${workspace.companyId}
AND (execution_workspace_id = ${workspace.id} OR cwd = ${workspace.cwd}
OR left(cwd, length(${workspace.cwd}) + 1) = ${workspace.cwd} || '/') AND status <> 'stopped')
) AS allowed`);
return rows[0]?.allowed === true;
}
async function sharedManagedBase(workspace: Workspace) {
if (!workspace.projectId || !workspace.repoUrl) throw new Error("no_shared_base");
let mismatch = false;
for (const candidate of managedSourcePaths(workspace)) {
if (!await exists(path.join(candidate, ".git"))) continue;
try { return await verifiedGitCacheSource({ ...workspace, metadata: null }, candidate); } catch { mismatch = true; }
}
throw new Error(mismatch ? "shared_base_repo_mismatch" : "no_shared_base");
}
function standardWorktreeSuffix(branch: string) {
if (!branch || branch.split("/").some(part => !part || part === "." || part === "..") || branch.includes("\\")) throw new Error("unsupported_branch_name");
return path.join(".paperclip", "worktrees", branch);
}
async function targetWorktree(db: Db | Tx, workspace: Workspace, base: string, branch: string, signal?: AbortSignal) {
const target = path.join(await fs.realpath(base), standardWorktreeSuffix(branch));
if (await exists(target)) throw new Error("target_worktree_exists");
const [other] = await db.select({ id: executionWorkspaces.id }).from(executionWorkspaces).where(and(ne(executionWorkspaces.id, workspace.id),
sql`(${executionWorkspaces.cwd} = ${target} OR ${executionWorkspaces.providerRef} = ${target})`)).limit(1);
if (other) throw new Error("target_worktree_exists");
const checkedOut = (await git(base, ["worktree", "list", "--porcelain"], { signal })).split("\n").includes(`branch refs/heads/${branch}`);
if (checkedOut) throw new Error("base_branch_checked_out");
return target;
}
async function assertSafeBaseConfig(base: string, signal?: AbortSignal) {
const keys = (await git(base, ["config", "--local", "--name-only", "--list"], { signal })).split("\n").map(key => key.trim().toLowerCase()).filter(Boolean);
if (keys.some(key => UNSAFE_BASE_CONFIG.test(key))) throw new Error("unsupported_base_git_config");
if (await exists(path.join(base, ".git", "info", "attributes"))) throw new Error("unsupported_base_git_config");
}
async function inspectWorktree(workspace: Workspace, clone: string, signal?: AbortSignal) {
const cwd = workspace.cwd;
if (!cwd || !workspace.branchName || !workspace.repoUrl || (workspace.providerRef && workspace.providerRef !== cwd)) throw new Error("unproven_workspace_ownership");
const stat = await fs.lstat(cwd).catch(() => null);
if (!stat) throw new Error("missing_worktree");
if (!stat.isDirectory() || stat.isSymbolicLink()) throw new Error("unsafe_worktree_path");
const real = await fs.realpath(cwd);
const cloneReal = await fs.realpath(clone);
if (real !== path.join(cloneReal, standardWorktreeSuffix(workspace.branchName))) throw new Error("nonstandard_worktree_location");
if (!(await fs.lstat(path.join(real, ".git"))).isFile()) throw new Error("not_linked_worktree");
const common = (await git(real, ["rev-parse", "--path-format=absolute", "--git-common-dir"], { signal })).trim();
if (common !== path.join(cloneReal, ".git")) throw new Error("source_identity_changed");
if ((await git(real, ["symbolic-ref", "--short", "HEAD"], { signal })).trim() !== workspace.branchName) throw new Error("branch_identity_changed");
const admin = (await git(real, ["rev-parse", "--absolute-git-dir"], { signal })).trim();
if (path.dirname(admin) !== path.join(common, "worktrees")) throw new Error("unproven_worktree_admin");
if ((await fs.readdir(admin)).some(name => !ADMIN_ENTRIES.has(name))) throw new Error("unsupported_worktree_git_state");
const refs = async (dir: string): Promise<void> => {
for (const entry of await fs.readdir(dir, { withFileTypes: true }).catch(() => [])) {
if (!entry.isDirectory()) throw new Error("unique_worktree_refs");
await refs(path.join(dir, entry.name));
}
};
await refs(path.join(admin, "refs"));
if ((await fs.readdir(path.join(admin, "logs")).catch(() => [] as string[])).some(name => name !== "HEAD")) throw new Error("unknown_worktree_git_logs");
if (await exists(path.join(common, "shallow"))) throw new Error("unsupported_local_git_state");
if ((await git(real, ["for-each-ref", "--format=%(refname)", "refs/stash", "refs/replace"], { signal })).trim()) throw new Error("unique_git_refs");
if ((await git(real, ["ls-files", "-v"], { signal })).split("\n").some(line => /^[a-zS] /.test(line))) throw new Error("unsupported_index_flags");
return { cwd: real, admin, head: (await git(real, ["rev-parse", "--verify", "HEAD^{commit}"], { signal })).trim() };
}
async function transferPlan(cwd: string, signal?: AbortSignal) {
const regenerable = await findRegenerableWorkspaceDirs(cwd, { signal });
const nestedRoot = path.join(cwd, PROJECT_REPOSITORIES_DIR);
const nestedStat = await fs.lstat(nestedRoot).catch(() => null);
if (nestedStat && (!nestedStat.isDirectory() || nestedStat.isSymbolicLink())) throw new Error("untransferable_managed_checkouts");
const nested = nestedStat ? (await fs.readdir(nestedRoot)).map(name => path.posix.join(PROJECT_REPOSITORIES_DIR, name)) : [];
const covered = (file: string) => file === `${PROJECT_REPOSITORIES_DIR}/` || file.startsWith(`${PROJECT_REPOSITORIES_DIR}/`)
|| regenerable.some(dir => file.startsWith(`${dir}/`));
const env = (file: string) => /^\.env(\..*)?$/.test(path.posix.basename(file));
const untracked = nul(await git(cwd, ["ls-files", "-z", "--others", "--exclude-standard"], { signal }));
const ignored = nul(await git(cwd, ["ls-files", "-z", "--others", "--ignored", "--exclude-standard"], { signal }));
const files = [...new Set([...untracked.filter(file => !covered(file) && env(file)),
...ignored.filter(file => !covered(file) && (env(file) || file === ".claude/settings.local.json"))])].sort();
if (ignored.some(file => !covered(file) && !files.includes(file) && !restorePointSavesIgnoredFile(file))) throw new Error("untransferable_ignored_files");
if (files.some(file => file.endsWith("/"))) throw new Error("untransferable_ignored_files");
const plan = [...regenerable, ...nested, ...files];
return plan.filter(item => !plan.some(other => other !== item && item.startsWith(`${other}/`)));
}
async function boundary(cwd: string, base: string, head: string, signal?: AbortSignal) {
const remoteTips = async (repo: string) => shas(await git(repo, ["for-each-ref", "--format=%(objectname)", "--sort=-committerdate", "refs/remotes"], { signal }));
const candidates = [head, ...await remoteTips(base), ...await remoteTips(cwd)];
const inBase = await presentCommits(base, candidates, signal);
const inClone = await presentCommits(cwd, candidates, signal);
return [...new Set(candidates)].filter(sha => inBase.has(sha) && inClone.has(sha)).slice(0, MAX_BOUNDARY_COMMITS);
}
function snapshotKey(snapshot: RestorePointSnapshot) {
return JSON.stringify([snapshot.head, snapshot.index, snapshot.worktree, snapshot.ignored, snapshot.modes]);
}
async function historyKey(cwd: string, admin: string, signal?: AbortSignal) {
const reflog = await git(cwd, ["reflog", "show", "--format=%H", "HEAD"], { signal }).catch(() => "");
const original = await fs.readFile(path.join(admin, "ORIG_HEAD"), "utf8").catch(() => "");
return JSON.stringify([shas(reflog), original.trim()]);
}
async function repairLinkedCheckout(checkout: string) {
const marker = await fs.lstat(path.join(checkout, ".git")).catch(() => null);
if (!marker?.isFile()) return;
const common = (await git(checkout, ["rev-parse", "--path-format=absolute", "--git-common-dir"])).trim();
await repairMovedNestedRepositoryWorktree(path.dirname(common), checkout);
}
async function retargetNestedBranch(checkout: string, from: string) {
const marker = await fs.lstat(path.join(checkout, ".git")).catch(() => null);
if (!marker?.isFile()) return;
const head = (await git(checkout, ["symbolic-ref", "--quiet", "--short", "HEAD"]).catch(() => "")).trim();
const previous = await nestedWorktreeBranchPrefix(from);
if (!head.startsWith(previous)) return;
const next = `${await nestedWorktreeBranchPrefix(checkout)}${head.slice(previous.length)}`;
if (next === head) return;
const base = path.dirname((await git(checkout, ["rev-parse", "--path-format=absolute", "--git-common-dir"])).trim());
await withNestedRepositoryBaseLock(base, async () => {
if ((await git(base, ["rev-parse", "--verify", "-q", `refs/heads/${next}`]).catch(() => "")).trim()) throw new Error("nested_branch_conflict");
await git(checkout, ["branch", "-m", head, next]);
});
}
async function linkedCheckoutBases(cwd: string, moves: string[]) {
const bases = new Map<string, string>();
for (const relative of moves.filter(item => item.startsWith(`${PROJECT_REPOSITORIES_DIR}/`))) {
const linked = await readLinkedNestedCheckout(path.join(cwd, relative)).catch(() => { throw new Error("untransferable_managed_checkouts"); });
if (linked) bases.set(relative, linked.base);
}
return bases;
}
function readPending(workspace: Workspace) {
const value = workspace.metadata?.[ISOLATED_REPOSITORY_REHOME_PENDING_KEY] as IsolatedRepositoryRehomePending | undefined;
if (value === undefined) return null;
if (!value || typeof value !== "object" || ![value.base, value.from, value.target, value.branch, value.head].every(item => typeof item === "string")
|| !SHA.test(value.head) || (value.previousTip !== null && !SHA.test(value.previousTip)) || !Array.isArray(value.moves)
|| value.moves.some(item => typeof item !== "string" || !item || path.isAbsolute(item) || item.split("/").includes(".."))) throw new Error("invalid_rehome_record");
return value;
}
async function writePending(db: Db, workspaceId: string, value: IsolatedRepositoryRehomePending | null) {
await db.update(executionWorkspaces).set({ metadata: value
? sql`coalesce(${executionWorkspaces.metadata}, '{}'::jsonb) || jsonb_build_object(${ISOLATED_REPOSITORY_REHOME_PENDING_KEY}::text, ${JSON.stringify(value)}::jsonb)`
: sql`${executionWorkspaces.metadata} - ${ISOLATED_REPOSITORY_REHOME_PENDING_KEY}::text` }).where(eq(executionWorkspaces.id, workspaceId));
}
async function provenTarget(db: Db | Tx, workspaceId: string, pending: IsolatedRepositoryRehomePending) {
const stat = await fs.lstat(pending.target).catch(() => null);
if (!stat?.isDirectory() || stat.isSymbolicLink() || !(await fs.lstat(path.join(pending.target, ".git")).catch(() => null))?.isFile()) return false;
const common = (await git(pending.target, ["rev-parse", "--path-format=absolute", "--git-common-dir"]).catch(() => "")).trim();
if (common !== path.join(await fs.realpath(pending.base), ".git")) return false;
if ((await git(pending.target, ["symbolic-ref", "-q", "HEAD"]).catch(() => "")).trim() !== `refs/heads/${pending.branch}`) return false;
const [other] = await db.select({ id: executionWorkspaces.id }).from(executionWorkspaces).where(and(ne(executionWorkspaces.id, workspaceId),
sql`(${executionWorkspaces.cwd} = ${pending.target} OR ${executionWorkspaces.providerRef} = ${pending.target})`)).limit(1);
return !other;
}
async function undoMove(db: Db | Tx, workspaceId: string, pending: IsolatedRepositoryRehomePending, ownsTarget?: boolean) {
if (ownsTarget ?? await provenTarget(db, workspaceId, pending)) {
for (const relative of pending.moves) {
if (await exists(path.join(pending.target, relative)) && await exists(path.join(pending.from, relative))) throw new Error("rehome_rollback_conflict");
}
for (const relative of [...pending.moves].reverse()) {
const moved = path.join(pending.target, relative), original = path.join(pending.from, relative);
if (!await exists(moved) || await exists(original)) continue;
await fs.mkdir(path.dirname(original), { recursive: true });
await fs.rename(moved, original);
await repairLinkedCheckout(original).catch(() => undefined);
await retargetNestedBranch(original, moved).catch(() => undefined);
}
await git(pending.base, ["worktree", "remove", "--force", "--force", pending.target]).catch(() => undefined);
await fs.rm(pending.target, { recursive: true, force: true });
await git(pending.base, ["worktree", "prune"]).catch(() => undefined);
}
const branchRef = `refs/heads/${pending.branch}`;
const tip = (await git(pending.base, ["rev-parse", "--verify", "-q", `${branchRef}^{commit}`]).catch(() => "")).trim();
if (tip !== pending.head || (await git(pending.base, ["worktree", "list", "--porcelain"])).split("\n").includes(`branch ${branchRef}`)) return;
if (pending.previousTip) await git(pending.base, ["update-ref", branchRef, pending.previousTip, pending.head]);
else {
await git(pending.base, ["update-ref", "-d", branchRef, pending.head]);
for (const key of ["remote", "merge"]) await git(pending.base, ["config", "--local", "--unset-all", `branch.${pending.branch}.${key}`]).catch(() => undefined);
}
}
function withRehomeLocks<T>(db: Db, workspace: Workspace, work: (tx: Tx) => Promise<T>) {
return withIsolatedRepositorySourceLock(db, workspace.companyId, workspace.sourceIssueId!, () =>
withDedicatedDbConnection(db, connection => connection.transaction(async tx => {
const rows = await tx.execute<{ locked: boolean; idle_timeout_ms: string | number | null }>(sql`select
pg_try_advisory_xact_lock(hashtextextended(${`execution_workspace_lifecycle:${workspace.id}`}, 0)) as locked,
(select setting::bigint from pg_settings where name = 'idle_in_transaction_session_timeout') as idle_timeout_ms`);
if (!rows[0]?.locked) throw new Error("lifecycle_busy");
return withWorkspaceLockKeepalive(tx, Number(rows[0]?.idle_timeout_ms), () => work(tx));
})), true);
}
function emptyResult(): IsolatedRepositoryRehomeResult {
return { outcome: "skipped", rehomed: false, cloneRemoved: false, reason: null, cloneKeptReason: null,
candidateBytes: 0, reclaimedBytes: 0, from: null, to: null };
}
function skip(result: IsolatedRepositoryRehomeResult, error: unknown) {
const reason = error instanceof IsolatedRepositorySourceBusy ? "source_busy" : error instanceof RestorePointSkip ? error.reason
: error instanceof RestorePointRestoreFailure ? error.reason : gitExpirationSkipReason(error);
if (reason === "inspection_failed") result.failure = describeGitExpirationFailure(error);
return reason;
}
export async function rehomeIsolatedRepositoryWorkspace(db: Db, snapshot: Workspace, options: {
dryRun?: boolean; now?: Date; signal?: AbortSignal; limits?: Partial<RestorePointLimits>; hooks?: IsolatedRepositoryRehomeHooks;
} = {}): Promise<IsolatedRepositoryRehomeResult> {
const result = emptyResult();
const candidate = isIsolatedRepositoryRehomeCandidate(snapshot);
if (!snapshot.sourceIssueId || (!candidate && !hasPendingIsolatedRepositoryRehomeCleanup(snapshot))) return { ...result, reason: "not_candidate" };
try {
return await withRehomeLocks(db, snapshot, async tx => {
const [row] = await db.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, snapshot.id), eq(executionWorkspaces.companyId, snapshot.companyId)));
if (!row || row.updatedAt.getTime() !== snapshot.updatedAt.getTime()) throw new Error("workspace_changed");
if (!candidate) return await cleanupRehomedSource(db, tx, row, result, options);
if (!await rehomeEligible(tx, row)) throw new Error("live_or_unresolved_consumer");
let recipe: GitCacheRestoration | null = null;
try { recipe = row.metadata?.[GIT_CACHE_RESTORATION_KEY] === undefined ? null : readGitCacheRestoration(row); }
catch { throw new Error("invalid_restoration_recipe"); }
if (recipe && recipe.sourceKind !== "isolated_remote") throw new Error("unsupported_restoration_recipe");
const clone = (await verifiedRecordedIsolatedRemoteSource(row))!;
await recoverRehomeLocks(clone, row.branchName, row.id);
const old = await inspectWorktree(row, clone, options.signal);
const base = await sharedManagedBase(row);
const branch = row.branchName!;
const branchRef = `refs/heads/${branch}`;
const interrupted = readPending(row);
if (interrupted) {
if (interrupted.base !== base || interrupted.from !== old.cwd || interrupted.branch !== branch
|| interrupted.target !== path.join(await fs.realpath(base), standardWorktreeSuffix(branch))) throw new Error("invalid_rehome_record");
if (options.dryRun) return { ...result, outcome: "dry_run" as const, reason: "interrupted_rehome_pending", from: row.cwd, to: interrupted.target };
await undoMove(tx, row.id, interrupted);
await writePending(db, row.id, null);
}
const target = await targetWorktree(tx, row, base, branch, options.signal);
await assertSafeBaseConfig(base, options.signal);
const previousTip = (await git(base, ["rev-parse", "--verify", "-q", `${branchRef}^{commit}`], { signal: options.signal }).catch(() => "")).trim() || null;
if (previousTip && !await git(old.cwd, ["merge-base", "--is-ancestor", previousTip, "HEAD"], { signal: options.signal }).then(() => true, () => false)) {
throw new Error("base_branch_diverged");
}
const moves = await transferPlan(old.cwd, options.signal);
const linkedBases = await linkedCheckoutBases(old.cwd, moves);
result.from = row.cwd; result.to = target;
result.candidateBytes = await allocatedBytes(clone, () => { options.signal?.throwIfAborted(); return true; }).catch(() => 0);
if (options.dryRun) return { ...result, outcome: "dry_run" as const };
const admins = [path.basename(old.admin)];
const locks = await acquireGitExpirationLocks(rehomeLockLocations(clone, admins, branch), lockToken(row.id))
.catch(() => { throw new Error("worktree_git_busy"); });
const pending: IsolatedRepositoryRehomePending = { base, from: old.cwd, target, branch, head: old.head, previousTip, moves };
let recorded = false, ownsTarget = false, committed = false;
try {
const before = await currentRestorePointSnapshot(old.cwd, options.signal);
const beforeHistory = await historyKey(old.cwd, old.admin, options.signal);
if (before.head !== old.head) throw new Error("head_changed");
const captured = await captureRestorePoint({ cwd: old.cwd, upstreamCommits: await boundary(old.cwd, base, old.head, options.signal),
limits: options.limits, signal: options.signal });
const manifest = captured ? await importRestorePoint({ repositoryCwd: base, payload: captured.payload, signal: options.signal }) : null;
const reflog = shas(await git(old.cwd, ["reflog", "show", "--format=%H", "HEAD"], { signal: options.signal }).catch(() => ""));
const original = (await fs.readFile(path.join(old.admin, "ORIG_HEAD"), "utf8").catch(() => "")).trim();
const history = [old.head, ...reflog, ...(SHA.test(original) ? [original] : [])];
const present = await presentCommits(base, history, options.signal);
if (history.some(sha => !present.has(sha))) throw new Error("history_not_transferred");
await writePending(db, row.id, pending);
recorded = true;
if (previousTip) await git(base, ["update-ref", branchRef, old.head, previousTip], { signal: options.signal });
else {
await git(base, ["update-ref", branchRef, old.head, ""], { signal: options.signal });
for (const key of ["remote", "merge"]) {
const value = (await git(old.cwd, ["config", "--local", "--get", `branch.${branch}.${key}`]).catch(() => "")).trim();
if (value) await git(base, ["config", "--local", `branch.${branch}.${key}`, value], { signal: options.signal });
}
}
await fs.mkdir(path.dirname(target), { recursive: true });
await assertSafeBaseConfig(base, options.signal);
await options.hooks?.beforeWorktreeAdd?.({ from: old.cwd, to: target });
if (await exists(target)) throw new Error("target_worktree_exists");
await git(base, ["worktree", "add", target, branch], { signal: options.signal, timeout: 15 * 60_000 });
ownsTarget = true;
if (manifest) await applyRestorePoint({ cwd: target, manifest, signal: options.signal });
const newAdmin = (await git(target, ["rev-parse", "--absolute-git-dir"], { signal: options.signal })).trim();
await fs.mkdir(path.join(newAdmin, "logs"), { recursive: true });
await fs.copyFile(path.join(old.admin, "logs", "HEAD"), path.join(newAdmin, "logs", "HEAD")).catch(async (error: NodeJS.ErrnoException) => {
if (error.code !== "ENOENT") throw error;
await fs.rm(path.join(newAdmin, "logs", "HEAD"), { force: true });
});
if (original) await fs.copyFile(path.join(old.admin, "ORIG_HEAD"), path.join(newAdmin, "ORIG_HEAD"));
if (moves.some(item => item.startsWith(`${PROJECT_REPOSITORIES_DIR}/`))) await ensureManagedRepositoriesIgnored(target);
for (const relative of moves) {
const destination = path.join(target, relative);
if (await exists(destination)) throw new Error("transfer_conflict");
await fs.mkdir(path.dirname(destination), { recursive: true });
await fs.rename(path.join(old.cwd, relative), destination);
if (linkedBases.has(relative)) {
await repairLinkedCheckout(destination);
await retargetNestedBranch(destination, path.join(old.cwd, relative));
}
}
await options.hooks?.beforeVerify?.({ from: old.cwd, to: target });
const after = await currentRestorePointSnapshot(old.cwd, options.signal);
const moved = await currentRestorePointSnapshot(target, options.signal);
if (snapshotKey(after) !== snapshotKey(before) || snapshotKey(moved) !== snapshotKey(before)) throw new Error("snapshot_mismatch");
if (await historyKey(old.cwd, old.admin, options.signal) !== beforeHistory || await historyKey(target, newAdmin, options.signal) !== beforeHistory) {
throw new Error("history_mismatch");
}
if ((await git(target, ["symbolic-ref", "--short", "HEAD"], { signal: options.signal })).trim() !== branch) throw new Error("branch_identity_changed");
for (const relative of moves) {
if (!await exists(path.join(target, relative)) || await exists(path.join(old.cwd, relative))) throw new Error("transfer_incomplete");
const base = linkedBases.get(relative);
if (!base) continue;
const linked = await readLinkedNestedCheckout(path.join(target, relative)).catch(() => null);
const head = linked ? (await git(linked.checkout, ["symbolic-ref", "--quiet", "--short", "HEAD"]).catch(() => "")).trim() : "";
if (linked?.base !== base || (head.startsWith("paperclip/") && !head.startsWith(linked.branchPrefix))) throw new Error("transfer_incomplete");
}
const at = (options.now ?? new Date()).toISOString();
await db.transaction(async updateTx => {
const [current] = await updateTx.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, row.id)).for("update");
if (!current || current.updatedAt.getTime() !== row.updatedAt.getTime()) throw new Error("workspace_changed");
if (!await rehomeEligible(updateTx, current)) throw new Error("live_or_unresolved_consumer");
const metadata: Record<string, unknown> = { ...current.metadata };
delete metadata.isolatedRemoteSource;
delete metadata.gitCacheExpiration;
delete metadata[ISOLATED_REPOSITORY_REHOME_PENDING_KEY];
if (recipe) metadata[GIT_CACHE_RESTORATION_KEY] = { ...recipe, sourceKind: "managed_project", sourcePath: base, worktreePath: target };
metadata[ISOLATED_REPOSITORY_REHOME_KEY] = { at, from: current.cwd ?? old.cwd, to: target, cleanedAt: null } satisfies IsolatedRepositoryRehomeRecord;
await updateTx.update(executionWorkspaces).set({ cwd: target, providerRef: current.providerRef ? target : null, metadata, updatedAt: new Date() })
.where(eq(executionWorkspaces.id, row.id));
});
committed = true;
} finally {
try {
if (!committed && recorded) {
await undoMove(tx, row.id, pending, ownsTarget);
await writePending(db, row.id, null);
}
} finally { await locks.release().catch(() => undefined); }
}
result.rehomed = true;
result.outcome = "rehomed";
const [updated] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, row.id));
await options.hooks?.afterDatabaseUpdate?.(updated!);
return await cleanupRehomedSource(db, tx, updated!, result, options);
});
} catch (error) {
options.signal?.throwIfAborted();
result.reason = skip(result, error);
return result;
}
}
async function cloneConsumers(db: Db | Tx, workspace: Workspace, clone: string) {
const others = await db.select().from(executionWorkspaces).where(and(ne(executionWorkspaces.id, workspace.id),
sql`((${executionWorkspaces.metadata}->'isolatedRemoteSource'->>'companyId' = ${workspace.companyId}
AND ${executionWorkspaces.metadata}->'isolatedRemoteSource'->>'issueId' = ${workspace.sourceIssueId})
OR ${executionWorkspaces.cwd} = ${clone} OR ${executionWorkspaces.providerRef} = ${clone}
OR left(${executionWorkspaces.cwd}, length(${clone}) + 1) = ${clone} || '/')`));
const expired = (row: Workspace) => {
try { return CLOSED_STATUSES.includes(row.status) && Boolean(readGitCacheRestoration(row)) && Boolean(readGitCacheExpiration(row)?.workspaceRemoved); }
catch { return false; }
};
if (others.some(row => !expired(row))) return "base_has_consumers";
const [project] = await db.select({ id: projectWorkspaces.id }).from(projectWorkspaces).where(sql`${projectWorkspaces.cwd} = ${clone}
OR left(${projectWorkspaces.cwd}, length(${clone}) + 1) = ${clone} || '/'`).limit(1);
if (project) return "base_has_consumers";
const [service] = await db.select({ id: workspaceRuntimeServices.id }).from(workspaceRuntimeServices).where(sql`
${workspaceRuntimeServices.status} <> 'stopped' AND (${workspaceRuntimeServices.cwd} = ${clone}
OR left(${workspaceRuntimeServices.cwd}, length(${clone}) + 1) = ${clone} || '/')`).limit(1);
return service ? "base_has_live_services" : null;
}
async function cloneDisposable(clone: string, base: string, signal?: AbortSignal) {
const worktrees = (await git(clone, ["worktree", "list", "--porcelain"], { signal })).split("\n").filter(line => line.startsWith("worktree "));
if (worktrees.length !== 1) return "base_has_worktrees";
if ((await git(clone, ["status", "--porcelain=v1", "-z", "--untracked-files=no"], { signal })).length) return "clone_checkout_has_files";
const untracked = nul(await git(clone, ["ls-files", "-z", "--others", "--exclude-standard"], { signal }));
const ignored = nul(await git(clone, ["ls-files", "-z", "--others", "--ignored", "--exclude-standard", "--directory"], { signal }));
if (untracked.length || ignored.some(file => !file.split("/").slice(0, -1).some(part => REGENERABLE_WORKSPACE_DIR_NAMES.has(part)))) return "clone_checkout_has_files";
const refs = (await git(clone, ["for-each-ref", `--count=${CLONE_HISTORY_BUDGET + 1}`, "--format=%(objectname) %(refname)"], { signal })).split("\n").filter(Boolean);
if (refs.length > CLONE_HISTORY_BUDGET) return "clone_history_budget";
if (refs.some(ref => !/^[0-9a-f]{40} refs\/(heads|tags|remotes)\//.test(ref))) return "unsupported_clone_refs";
const reflog = (await git(clone, ["reflog", "show", "--all", "-n", String(CLONE_HISTORY_BUDGET + 1), "--format=%H %gD"], { signal }).catch(() => ""))
.split("\n").map(line => line.trim().split(" ")).filter(([sha]) => SHA.test(sha ?? ""));
if (reflog.length > CLONE_HISTORY_BUDGET) return "clone_history_budget";
const local = [...refs.filter(ref => !/ refs\/remotes\//.test(ref)).map(ref => ref.split(" ")[0]!),
...reflog.filter(([, selector]) => !(selector ?? "").startsWith("refs/remotes/")).map(([sha]) => sha!)];
if (local.length === 0) return null;
const unique = shas(await git(clone, ["rev-list", `--max-count=${CLONE_HISTORY_BUDGET + 1}`, "--stdin"], { signal, input: [...new Set(local)].join("\n") + "\n--not\n--remotes\n" }));
if (unique.length > CLONE_HISTORY_BUDGET) return "clone_history_budget";
const present = await presentCommits(base, unique, signal);
return unique.some(sha => !present.has(sha)) ? "unique_clone_history" : null;
}
async function cleanupRehomedSource(db: Db, tx: Tx, row: Workspace, result: IsolatedRepositoryRehomeResult,
options: { dryRun?: boolean; now?: Date; signal?: AbortSignal }): Promise<IsolatedRepositoryRehomeResult> {
const rehome = readRehome(row);
if (!rehome || rehome.cleanedAt) return result;
result.from ??= rehome.from; result.to ??= rehome.to;
if (row.cwd !== rehome.to || !row.sourceIssueId) throw new Error("workspace_changed");
const clone = isolatedRemoteSourcePath(row.companyId, row.sourceIssueId);
const parent = path.dirname(clone);
const worktreeTrash = path.join(parent, `.rehomed-worktree-${row.id}`);
const cloneTrash = path.join(parent, `.rehomed-checkout-${row.id}`);
const base = await sharedManagedBase(row);
const from = await canonicalRepositoryRoot(rehome.from);
const cloneReal = await exists(clone) ? await fs.realpath(clone) : null;
const keep = async (reason: string, retained: boolean) => {
result.cloneKeptReason = reason;
if (!result.rehomed) result.outcome = "clone_kept";
if (retained && !options.dryRun) await markCleaned(db, row.id, options.now, reason);
return result;
};
if (cloneReal && path.relative(cloneReal, from).startsWith("..")) return keep("unproven_worktree_admin", false);
if (options.dryRun) {
if (cloneReal) result.candidateBytes = await allocatedBytes(clone, () => { options.signal?.throwIfAborted(); return true; }).catch(() => 0);
result.outcome = "dry_run";
return result;
}
if (!await rehomeEligible(tx, row)) return keep("live_or_unresolved_consumer", false);
if (cloneReal) {
await recoverRehomeLocks(clone, row.branchName, row.id);
const fromStat = await fs.lstat(from).catch(() => null);
if (fromStat) {
if (!fromStat.isDirectory() || fromStat.isSymbolicLink()) return keep("unsafe_worktree_path", false);
const common = (await git(from, ["rev-parse", "--path-format=absolute", "--git-common-dir"], { signal: options.signal }).catch(() => "")).trim();
if (common !== path.join(cloneReal, ".git")) return keep("source_identity_changed", false);
result.reclaimedBytes += await allocatedBytes(from, () => true).catch(() => 0);
if (await exists(worktreeTrash)) await fs.rm(worktreeTrash, { recursive: true, force: true });
await fs.rename(from, worktreeTrash);
}
await git(clone, ["worktree", "prune"], { signal: options.signal });
}
if (await exists(worktreeTrash)) await fs.rm(worktreeTrash, { recursive: true, force: true });
if (cloneReal) {
const consumer = await cloneConsumers(tx, row, clone);
if (consumer) return keep(consumer, consumer === "base_has_consumers");
const disposable = await cloneDisposable(clone, base, options.signal);
if (disposable) return keep(disposable, false);
const bytes = await allocatedBytes(clone, () => true).catch(() => 0);
if (await exists(cloneTrash)) await fs.rm(cloneTrash, { recursive: true, force: true });
await fs.rename(clone, cloneTrash);
result.reclaimedBytes += bytes;
}
if (await exists(cloneTrash)) {
await fs.rm(cloneTrash, { recursive: true, force: true });
result.cloneRemoved = true;
}
await fs.rmdir(parent).catch(() => undefined);
if (!result.rehomed) result.outcome = "cleaned";
await markCleaned(db, row.id, options.now);
return result;
}
async function markCleaned(db: Db, workspaceId: string, now?: Date, retained?: string) {
const patch = { cleanedAt: (now ?? new Date()).toISOString(), ...(retained ? { cloneRetained: retained } : {}) };
await db.update(executionWorkspaces).set({
metadata: sql`jsonb_set(${executionWorkspaces.metadata}, ${`{${ISOLATED_REPOSITORY_REHOME_KEY}}`}::text[],
(${executionWorkspaces.metadata}->${ISOLATED_REPOSITORY_REHOME_KEY}) || ${JSON.stringify(patch)}::jsonb)`,
}).where(eq(executionWorkspaces.id, workspaceId));
}
export async function rehomeExpiredIsolatedRestoration(db: Db, snapshot: Workspace): Promise<Workspace> {
if (!CLOSED_STATUSES.includes(snapshot.status) || snapshot.metadata?.isolatedRemoteSource === undefined || !snapshot.branchName) return snapshot;
try {
const recipe = readGitCacheRestoration(snapshot);
if (!recipe || recipe.sourceKind !== "isolated_remote" || !readGitCacheExpiration(snapshot)?.workspaceRemoved || await exists(recipe.worktreePath)) return snapshot;
if (await canonicalRepositoryRoot(recipe.worktreePath) !== path.join(await canonicalRepositoryRoot(recipe.sourcePath), standardWorktreeSuffix(recipe.branchName))) return snapshot;
const base = await sharedManagedBase(snapshot);
const target = await targetWorktree(db, snapshot, base, recipe.branchName);
const tip = (await git(base, ["rev-parse", "--verify", "-q", `refs/heads/${recipe.branchName}^{commit}`]).catch(() => "")).trim();
if (tip && tip !== recipe.commit) return snapshot;
if ((recipe.contents?.checkouts ?? []).some(checkout => checkout.worktree && checkout.branchName)) return snapshot;
if (recipe.restorePoint && !recipe.restorePoint.headOnRemote && !recipe.restorePoint.expiredAt) {
const { manifest } = await loadGitRestorePoint({ storage: getStorageService(), workspace: { companyId: snapshot.companyId, id: snapshot.id }, ref: recipe.restorePoint });
const present = await presentCommits(base, manifest.prerequisites);
if (manifest.prerequisites.some(sha => !present.has(sha))) return snapshot;
}
await finishGitCacheTrash(snapshot);
return await withExecutionWorkspaceLifecycleLock(db, snapshot.id, async tx => {
const [row] = await tx.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, snapshot.id)).for("update");
if (!row || row.updatedAt.getTime() !== snapshot.updatedAt.getTime()) return row ?? snapshot;
const metadata: Record<string, unknown> = { ...row.metadata };
delete metadata.isolatedRemoteSource;
delete metadata.gitCacheExpiration;
metadata[GIT_CACHE_RESTORATION_KEY] = { ...recipe, sourceKind: "managed_project", sourcePath: base, worktreePath: target } satisfies GitCacheRestoration;
metadata[ISOLATED_REPOSITORY_REHOME_KEY] = { at: new Date().toISOString(), from: recipe.worktreePath, to: target, cleanedAt: null } satisfies IsolatedRepositoryRehomeRecord;
const [updated] = await tx.update(executionWorkspaces).set({ cwd: target, providerRef: row.providerRef ? target : null, metadata, updatedAt: new Date() })
.where(eq(executionWorkspaces.id, row.id)).returning();
return updated ?? row;
});
} catch { return snapshot; }
}