Files
PaperClipAI/server
DottaandClaude Opus 5.5 6b631dd477 fix(recovery): check held retained orphans only for candidate leases (#21542)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The ownership reconciliation pass lists in-progress records of each
kind every 30 seconds, and settles the records whose owner is gone
> - One kind, the retained orphan sandbox, lists leases with a large SQL
filter that has correlated subqueries on `heartbeat_runs`, `issues`,
`issue_recovery_actions` and `environment_leases`
> - The planner ran those subqueries for every released, failed and
expired lease in history, before the cheap orphan predicates. The cost
grew with lease history, not with the number of orphans
> - On a busy hosted database one page of this listing ran for 17 to 138
seconds as one statement. It held an AccessShareLock on `heartbeat_runs`
for all of that time, so an online `ALTER TABLE heartbeat_runs ADD
COLUMN` with a 5 second lock timeout failed on every try
> - This pull request runs a cheap candidate filter first and evaluates
the held checks only for candidate rows
> - The benefit is that the listing cost now follows the number of
candidates. Schema changes on `heartbeat_runs` no longer wait behind it,
and the pass reads far fewer pages

## Linked Issues or Issue Description

No public issue exists. This describes the bug in the format of the bug
report template.

**What happened?**
A hosted instance could not apply a pending migration that adds a column
to `heartbeat_runs`. The deployer sets `lock_timeout = 5s` and tries
three times. All three tries timed out. The database showed one session
that ran `select "id", "company_id", "heartbeat_run_id", "issue_id",
"metadata", "updated_at" from "environment_leases" ...` in state
`active` for up to 138 seconds. Its transaction start and query start
differed by less than 0.1 seconds, so it was one statement. It held an
AccessShareLock on `heartbeat_runs`. That instance had about 18,600
lease rows: 13,240 expired, 4,695 released, 610 failed, 69 retained, 17
active and 9 pending cleanup. Each failed try also stalled all
statements on `heartbeat_runs` for up to 5 seconds.

**Expected behavior**
The ownership listing must finish in a short time that does not grow
with lease history. It must not keep a lock on `heartbeat_runs` long
enough to block an online schema change.

**Steps to reproduce**
1. Use a database with many terminal environment leases (expired,
released or failed) and few or no retained orphan sandboxes.
2. Run the `environment_retained_orphan_sandbox` listing
(`OWNABLE_KINDS`, `listInProgress`), or wait for the ownership
reconciliation pass.
3. Run `EXPLAIN (ANALYZE, BUFFERS)` on the listing. The `heartbeat_runs`
primary key subplans run once for each terminal lease, not only for
candidates.
4. While the listing runs, run `ALTER TABLE heartbeat_runs ADD COLUMN
probe jsonb` with `lock_timeout = '5s'`. The ALTER waits until the
listing ends.

**Paperclip version or commit**
`master` at 0a96f972e1.

**Database mode**
Hosted PostgreSQL. Reproduced on embedded PostgreSQL 18.

**Additional context**
The 138 second session was not a transaction held open across remote
work. The call path is `registerOwnershipReconciliationWork` →
`runOwnershipReconciliationPass(db)` → `kind.listInProgress(db, 200,
offset)` on the pool, with no transaction. Only the `leaseKind` and
`operationKind` listings send this exact unqualified column list. Of
these, only this kind's filter reads `heartbeat_runs`.

## What Changed

- `releasableRetainedOrphanCondition` and `nextRetainedOrphanDueAt`
first apply `retainedOrphanCandidateCondition`. This is a filter with no
subqueries and no NULL results: cleanup status, run, provider, lease
status, lease policy, and the metadata keys of the three retained orphan
shapes. Expired, active and pending-cleanup leases fail it at once.
- The held checks (always held and softly held) now run inside `CASE
WHEN <retained orphan condition> THEN ... ELSE false END`. PostgreSQL
evaluates a CASE in order, so the correlated subqueries run only for
candidates. The planner can no longer split the `NOT (a OR b OR ...)`
into separate top-level subplans and run them before the candidate
predicates.
- The two `heartbeat_runs` checks of the always-held part are now one
scalar subquery with a primary key lookup. It reads
`runner_profile_json->'nativeWorkspaceSync'` once through a fenced
lateral, instead of about 12 `#>>` path reads. A scalar subquery cannot
become a hashed subplan, so the planner cannot replace it with a full
scan of `heartbeat_runs`.
- The listed rows, the order and the paging do not change.
`softlyHeldSql` is now exported so that the test can rebuild the
previous condition.
- The paging stays OFFSET-based. The `listInProgress(db, limit, offset)`
contract is shared by all ownership kinds and by the I2 rule, which
compares consecutive passes. Keyset paging is a change to that shared
contract, so this pull request does not make it.

## Verification

- `cd server && npx vitest run
src/__tests__/environment-retained-orphan-listing-cost.test.ts
src/__tests__/environment-retained-orphan-sandbox.test.ts
src/__tests__/abandoned-workspace-files-hold.test.ts`: 3 files, 55 tests
pass.
- New test `environment-retained-orphan-listing-cost.test.ts`:
- Equivalence: four seeded random fixtures of 600 leases each, across
two companies. The fixtures cover every lease status and policy, runs of
each status and runtime mode, valid and invalid native workspace
descriptors, every always-held metadata key, sibling leases on the same
sandbox, open and resolved recovery actions, open runs on the issue, and
past, future and absent expiry. Every page (limit 7 and 200) equals the
previous SQL, which is kept only in the test. `nextRetainedOrphanDueAt`
also equals the previous query.
- Red/green: over 2,000 history leases with the hosted status mix (71%
expired, 25% released, 4% failed), the test counts the `heartbeat_runs`
rows that the listing reads, from `EXPLAIN (ANALYZE, FORMAT JSON)`. On
`master` it reads 1,760 rows for 1 candidate, so the test fails. With
this change the count is within the bound of 10 per candidate.
- Incident regression: while listings run in a loop over 5,000 history
leases, `ALTER TABLE heartbeat_runs ADD COLUMN ...` and `DROP COLUMN`
with `lock_timeout = '5s'` complete in less than 5 seconds.
- Local measurement (embedded PostgreSQL 18, Apple M-series, warm
cache), not in CI:
- 18,640 leases with the hosted status mix and a 19 KB runner profile
per run. `master`: 422 ms, 296,771 buffers (12,357 read). This change:
26 ms, 36,004 buffers. The `heartbeat_runs` subplans run 13 times (once
for each candidate), not once for each lease.
- 300,000 history leases. `master`: one listing page takes 2,822 ms, and
the ALTER under load waits 2,448 ms. This change: 212 ms, and the ALTER
takes 96 ms.
- `pnpm check:module-boundaries` passes.

## Risks

- Low risk. The listing and the due-time query return the same rows, and
the equivalence test compares them. The settle is unchanged and still
re-checks every hold under the row lock.
- The candidate filter must stay a superset of
`retainedOrphanSandboxCondition`. If a later change adds a retained
orphan shape and does not update the candidate filter, the listing skips
that shape. The equivalence test catches this only for the shapes in the
fixture.
- The listing still reads each lease row once with cheap predicates. A
partial index on the candidate filter could make it scale only with
candidates. This pull request does not add a migration.
- The ownership pass still lists all kinds every 30 seconds when its
fingerprint changes. This pull request does not change that cadence.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edit, embedded PostgreSQL runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:48:21 -06:00
..