## Thinking Path
> - Paperclip manages agent work and its execution workspaces.
> - Workspace recovery scans candidates and retains deadlines for
unfinished work.
> - Rounded database timestamps can prevent a scan from advancing.
> - Separate clock reads can give one retry obligation different
deadlines.
> - Repeated inspection must preserve both scan progress and the saved
retry time.
> - This PR keeps full cursor precision and computes each retry deadline
once.
> - Recovery can finish its scan and wait until work is actually due.
## Linked Issues or Issue Description
**What happened?**
Terminal workspace recovery can repeatedly inspect the oldest skipped
candidate when its timestamp contains microseconds. Eligible workspaces
behind it never reach cleanup. Separately, nested-clone migration and
isolated-repository rehome recovery use separate clock reads for their
retry gate and advertised deadline. A later inspection can move the
advertised deadline backward.
**Expected behavior**
Each bounded scan advances using the complete timestamp and ID. The
retry gate and advertised deadline represent the same saved time,
including when the clock advances between calls.
**Steps to reproduce**
For cursor starvation, place two skipped workspaces with microsecond
timestamps before an eligible terminal workspace and use a one-row page.
For deadline drift, inject a clock that advances one millisecond per
read, run disabled migration or rehome recovery, then inspect it again
before the retry time.
Stacked on #17497. Related cursor fixes#17479 and #17482 have separate
owners. This PR owns workspace recovery cursor and retry precision.
## What Changed
- Preserve the database timestamp as text in the private timestamp/ID
cursor.
- Compute one retry time per migration or rehome outcome and reuse it in
the existing maps.
- Test the cursor with real PostgreSQL microsecond timestamps and the
retry boundary with a deterministic advancing clock.
- Support validated disposable external test databases in the affected
fixtures, retaining embedded fallback.
- Document the evidence in `doc/idle-terminal-workspace-pagination.md`.
## Verification
Current test revision: `9d846c84e76c90893ab62a09548c11bbdb627771`,
followed by docs-only head `a34dc50de3`, which received both 5/5 reviews
and passing CI. This candidate normally integrates landed approval
retention `553a004926`. Every runtime and acceptance-test source matches
that tested/reviewed tree; only four independent Runner E2E accounting
files and the ledger differ. Migration metadata is unchanged. Existing
local checks are reused for their unchanged source; fresh final-head CI
and reviews are required against master. This remains the
terminal-precision duty.
- **136 cases pass** in four complete suites: execution workspace
service, terminal worker, isolated repository rehome and nested-clone
migration. Server typecheck and production build pass with the current
Runner vendor.
- The preceding `f0f0de1f6f` run also passed all 136 on three newly
created, unmigrated PostgreSQL databases. The suites themselves applied
migrations. The current run reuses those clean fixtures after
integrating the parent's approval wake repair; it is not presented as a
fresh-database run.
- The regressions retain real PostgreSQL microsecond timestamps and an
advancing clock. Before repair, skipped rows starved eligible cleanup
and early inspection moved a saved retry by one millisecond. Current
assertions preserve exact progress/deadline behavior; no timeout or
business policy is weakened.
- Earlier external-fixture setup tests failed with missing
company/restore-point schema. Their setup repair and red/green record
remain in `doc/idle-terminal-workspace-pagination.md`. This PR adds no
migration or new queue.
- Earlier stack integration receipts and failures remain in the stack
ledger and `doc/idle-approved-tool-recovery.md`; they are historical
parent evidence, not this head's acceptance. No full local repository
suite, deployed suspension, fleet cost or live provider qualification is
claimed.
Commands: server Vitest for `execution-workspaces-service`,
`terminal-workspace-work`, `isolated-repository-rehome` and
`nested-clone-migration`; server typecheck/build. Fresh exact-head CI
and Greptile/Ponytail 5/5 remain required before merge in dependency
order.
## Risks
The cursor remains private query state. Retry intervals, eligibility,
cleanup authority, and shutdown behavior are unchanged. No schema,
queue, external alarm, or additional polling loop is introduced. The
tests establish workspace recovery behavior, not host suspension or cost
savings.
## Model Used
OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
serving model identifier and context window were not exposed.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Nested project repositories are now worktrees of one shared clone
per project repository, and existing standalone clones are converted in
place
> - Conversion keeps the clone's remote-tracking refs as task-owned
branches (`paperclip/<id>/remotes/...`) so commits only they hold
survive
> - Expiration then required every task-owned branch, including those,
to be on the remote
> - Remote-tracking refs for branches deleted or force-pushed upstream
never are, so converted checkouts were kept forever
> - This pull request treats those refs the way the standalone-clone
path already does: commits missing upstream go into the restore point
instead of blocking
> - The benefit is that converted workspaces can expire and free disk
without losing any history
## Linked Issues or Issue Description
Refs #17407, #17382, #17289. Bug report described in-PR (bug report
template).
**What happened?**
On one instance, 17 closed workspaces in a two-repository project were
kept on every expiration sweep with
`unpushed_managed_checkout_branches`. Each nested checkout is a
converted worktree whose task-owned refs include
`paperclip/<id>/remotes/origin/<branch>` for upstream branches since
deleted or force-pushed (dependabot, squashed feature branches). Their
commits are on no remote, so `provenNestedWorktreeHistory` refused. The
task's own branch was fully pushed.
**Expected behavior**
As in the standalone-clone path (#17289), remote-tracking commits
missing upstream are preserved in the restore point, and only genuinely
local branches block expiration.
**Steps to reproduce**
1. Fetch an upstream branch into a standalone nested clone, then delete
that branch upstream.
2. Convert the clone into a worktree (nested clone migration).
3. Close the issue and let expiration inspect it: kept with
`unpushed_managed_checkout_branches`.
**Paperclip version or commit:** `master` at cb47dbb2da.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
- `provenNestedWorktreeHistory` in
`server/src/services/git-workspace-restoration.ts`:
- splits task-owned refs into those under `<prefix>remotes/` and the
rest;
- still requires the rest (and their reflogs) to be on the remote;
- returns commits under `remotes/` that are missing upstream, and their
reflogs, as `preserve`, which the restore point bundles. This matches
the standalone-clone path.
- `restorePointHistoryDigest` in
`server/src/services/git-restore-point.ts`: for a linked worktree, it
fingerprints the task's own refs and their reflogs instead of the shared
clone's `refs/remotes`. Other tasks change those whenever they fetch,
and they never covered the task-owned remote refs. It loads
`nested-repository-worktrees` lazily to avoid an import cycle.
No code comments were added.
## Verification
- `pnpm exec vitest run nested-clone-migration` (in `server/`). New
cases:
- A converted checkout whose only extra history is a remote-tracking
commit deleted upstream expires with a restore point. After its refs are
removed and the shared clone is garbage-collected, reopening restores
the commit.
- The digest ignores the shared clone's remote refs and changes when the
task's own refs move.
- **Mutation checks**, each failing its test:
- reverting the prover split;
- dropping the preserve list;
- reverting the digest change.
- The existing case where a converted checkout with a genuinely unpushed
branch is kept still passes.
- `pnpm exec vitest run git-restore-point nested-repository-worktrees
nested-clone-migration git-workspace-expiration
git-reproducible-managed-checkouts isolated-task-directory
isolated-repository-rehome git-credentials execution-workspace` (in
`server/`): 591 passed.
- `pnpm run typecheck` (in `server/`): pass.
- Not run: server build and boot smoke.
## Risks
- **Misclassified branches:** a local branch literally named
`remotes/...` in a standalone clone becomes `<prefix>remotes/...` after
conversion and is now preserved in the restore point instead of blocking
expiration. Its commits are still saved, not lost.
- **Leftover refs:** after expiry, the task's other owned refs stay in
the shared clone, as they did before this change, so their commits stay
in two places.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, read-only diagnosis on a live
instance).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Each task runs in a git worktree. Extra project repositories sit in
the task under `.paperclip-repositories/`.
> - #17382 made new nested checkouts git worktrees of one shared clone
per project repository. Existing standalone nested clones stayed as full
clones.
> - Long-lived tasks keep those clones for a long time. Many tasks hold
the same large repository, so most of the disk is duplicate `.git` data.
> - #17369 added a locked and verified move of isolated workspaces onto
the shared clone, with a reaper sweep behind a setting.
> - This pull request adds the same kind of locked and verified
conversion for standalone nested clones in open tasks, and runs it from
the same reaper path behind a new setting.
> - The benefit is that the duplicate clones go away without losing any
state or local history, and the disk stops filling.
## Linked Issues or Issue Description
Refs #17382#17369. Bug report described in-PR (bug report template).
**What happened?**
On one instance, 44 open task worktrees in one project each hold a
standalone nested clone of the same large repository. Together they use
about 25 GB. About 16 GB of that is duplicate `.git` data. Another
project has 33 smaller clones. The clones are removed only when their
tasks expire, and many tasks stay open for a long time. The disk keeps
filling.
**Expected behavior**
Existing standalone nested clones in open tasks become worktrees of the
project's shared clone, with the same state and local history, and the
standalone clones are deleted.
**Steps to reproduce**
1. Prepare a task with a nested project repository before #17382, or
without a shared clone, so that `.paperclip-repositories/<name>` is a
standalone clone.
2. Create the project's shared clone, as #17382 does for new tasks.
3. Observe that the old checkout stays a full clone until the task
expires.
**Paperclip version or commit:** `master` at 8549ecaf9a.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
- New `server/src/services/nested-clone-migration.ts`. It converts one
standalone nested clone in an open execution workspace into a worktree
of the shared clone:
- It holds the source lock and workspace lifecycle lock (as git cache
expiration does), the per-shared-clone lock from #17382, and the
expiration lock files (`index.lock`, `HEAD.lock`, current branch ref
lock) in the clone.
- It requires the #17369 no-live-consumer eligibility (no live runs,
leased finalizations, unreleased leases, running services, or reopen
flag). It checks again before the swap.
- It identifies the repository by managed checkout name, project
workspace rows and the origin URL, and it requires a verified shared
clone.
- It brings local history into the shared clone under the checkout's own
`paperclip/<id>/` prefix. The current branch becomes
`paperclip/<id>/<branch>`, tracking `origin/<branch>`. Unpushed local
branches become `paperclip/<id>/<branch>`. Unpushed tags become
`paperclip/<id>/tags/<tag>`. A restore point carries the commits, the
`HEAD` reflog and `ORIG_HEAD`.
- It stages the worktree at `<name>.clone-migrating`, applies the
restore point, copies the `HEAD` reflog and `ORIG_HEAD`, and moves
regenerable dirs, untracked `.env` files and
`.claude/settings.local.json` by rename. Old and new must then match
exactly (snapshot, history, branch, created refs, moved items).
- Swap: it renames the clone to `<name>.clone-standalone`, runs `git
worktree move` and `git worktree repair`, updates a saved restoration
recipe that lists the checkout, and then renames the old clone to trash
and deletes it.
- A `nestedCloneMigrationPending.<name>` metadata record makes recovery
possible. A crash before the swap rolls back (moves items back, removes
the staged worktree and the created branches). A crash after the swap
rolls forward.
- Skip reasons (nothing changes): `live_or_unresolved_consumer`,
`unknown_managed_checkout`, `local_source_checkout`, `foreign_origin`,
`no_shared_base`, `shared_base_repo_mismatch`,
`unsupported_base_git_config`, `nested_clone_has_stash` (stashes are
skipped, because `refs/stash` would be shared by every worktree),
`nested_clone_has_worktrees`, `nested_clone_has_hooks`,
`unsupported_local_git_config`, `unsupported_local_git_state`,
`unsupported_info_exclude`, `unsupported_nested_clone_refs`,
`unsupported_index_flags`, `unique_branch_reflog_commits`,
`nested_clone_ref_conflict`, `snapshot_mismatch`, `history_mismatch`,
restore point limits, and `nested_migration_recovery_conflict`.
- Reaper: `sweepTerminalWorkspaces` runs the conversion for open
workspaces when `PAPERCLIP_NESTED_CLONE_MIGRATION_ENABLED=true` (default
off). `PAPERCLIP_NESTED_CLONE_MIGRATION_DRY_RUN=true` only reports. Each
attempt logs `event: "execution_workspace.nested_clone_migration"` with
outcome, reason, branch, candidate bytes and reclaimed bytes. There is
no new timer. A retry deadline is registered on the existing reaper
deadlines only for a pending record or a transient skip (live consumer,
busy lock, changed row). A workspace is attempted again at most hourly.
A workspace without standalone clones costs one directory listing.
- `isolated-repository-rehome.ts`: I exported the shared pieces
(eligibility, locks, safe base config check, transfer plan, history
boundary, snapshot and history keys, item moves, rollback and
verification) instead of copying them. Re-home behavior does not change.
- `nested-repository-worktrees.ts`: I exported the stale registration
cleanup and added the `moveNestedRepositoryWorktree` (move and repair)
and `trackNestedWorktreeUpstream` helpers.
- Docs: `doc/recoverable-disk-cleanup.md` (new section about the
conversion) and `doc/DEVELOPING.md` (settings).
## Verification
- `cd server && pnpm exec vitest run
src/__tests__/nested-clone-migration.test.ts`: 19 passed. It uses
embedded Postgres and real git repositories, and covers:
- Conversion with an unpushed commit, a second branch with unpushed
commits, a lightweight tag and an annotated tag, staged and unstaged
edits, a deleted file, an untracked file with mode 0600, an ignored
file, `node_modules`, `.env` and the settings file. The result is a
worktree of the shared clone (`--git-common-dir`), with the same HEAD,
status, staged content and reflog, the owned branch tracking
`origin/main`, and the history under the task's prefix. No other branch
in the shared clone changes. The old clone is gone and reclaimed bytes
are reported.
- Skips with nothing changed: stash, live run, foreign origin, unknown
checkout, no shared clone, local excludes, and a change between capture
and verification (hook).
- Two tasks convert concurrently against one shared clone. The test
holds the shared-clone lock and shows that neither conversion advances.
Then they run one at a time (peak 1), and each keeps its own branches.
- Crash recovery at `afterAside`, `afterMove` and `afterFinalize` (roll
forward), and before the swap (roll back, then convert). A dry run
reports the pending record and changes nothing.
- Expiration and restore of a converted checkout work afterwards.
Unpushed branches in a converted checkout block expiration
(`unpushed_managed_checkout_branches`).
- Reaper: disabled by default; dry run changes nothing and registers no
deadline; a live run registers a retry deadline that survives a second
sweep; enabled conversion leaves no deadline.
- Commits that only the clone's `origin` remote-tracking refs hold are
kept as `paperclip/<id>/remotes/origin/<branch>`.
- A pending conversion is finished after the workspace is archived, and
forgotten (keeping unpushed branches) when the worktree is gone.
- A filter planted in the shared clone right before the worktree add is
refused (`unsupported_base_git_config`) and never runs.
- Only checkouts named after configured project repositories are
considered; at most 8 per sweep, pending first, rotating (unit test).
- Related suites (run before review fixes; the nested, re-home and
execution workspace service suites were rerun after them): `pnpm exec
vitest run` on execution-workspaces-routes,
execution-workspaces-service, git-reproducible-managed-checkouts,
git-restore-point (all 5 files), git-workspace-contents,
git-workspace-expiration, git-workspace-expiration-eligibility,
heartbeat-git-restore-point-resume, heartbeat-project-repositories,
isolated-repository-rehome, nested-clone-migration: 15 files, 393 tests
passed.
- `cd server && pnpm run typecheck` (`tsc --noEmit`): passes.
- Mutation checks. Each mutation makes the targeted test fail, and I
reverted each one:
- Remove both eligibility checks: the live run test fails.
- Remove the snapshot comparison: the capture/verify change test fails.
- Disable roll forward after the swap: the recovery tests fail.
- Drop the `paperclip/<id>/` prefix from transferred branches: the
conversion and concurrency tests fail.
- Remove the shared-clone lock: the concurrency test fails.
- Drop remote-tracking refs from the plan, the deadline restore during
backoff, archived recovery, or the config check before the worktree add:
each targeted test fails.
- Not run: the full server suite and a run on a live instance.
## Risks
- The setting is off by default. Enable a dry run first and read the
`execution_workspace.nested_clone_migration` logs.
- A clone that a run writes to between verification and the swap rename
can lose those writes. This window is the same as in #17369. Locks, the
eligibility check and the git lock files make it small.
- A crash in the short time between the rename aside and `git worktree
move` can collide with a run that recreates `<name>`. In that case the
conversion stops with `nested_migration_recovery_conflict` and keeps
both copies. Expiration keeps such a workspace because of the `.clone-`
names.
- Stashes, annotated tag messages (the tagged commit is kept as a
branch), custom hooks, custom local config and per-branch reflog-only
history are not converted. Clones with them are skipped with a named
reason. Only unpushed branches and tags get a name in the shared clone.
A pushed local branch name is not kept, but its commits stay on the
remote.
- The conversion fetches the shared clone first. A failed fetch is
ignored, and then the restore point can be larger.
- Pushing from a converted checkout needs an explicit destination, as
for all #17382 nested worktrees.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file editing, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
