feat(recovery): settle stale issue execution and checkout locks on the shared ownership path (#20006)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task is locked to the run that executes it, and the lock must be
released when that run ends
> - A restart between the run's terminal write and the lock release left
the task locked to a dead run until a periodic sweep noticed
> - The durable ownership plan moves each in-progress record onto the
shared reconciler, which settles a record in its first pass after its
owner is proven gone
> - This pull request registers the task execution and checkout locks
with the reconciler, and makes the lock release promote any messages
saved behind it
> - The benefit is that a task stuck behind a dead run is freed in the
first reconciler pass, and its waiting messages go out at once

## Linked Issues or Issue Description

Refs #19831 (plan, Phase 2 sections A and C), #19838 (inventory: "Issue
execution lock" and "Issue checkout lock"), #19956 (work-intent outbox),
#20004 (lost-process reapers).

**What happened?**
The task locks were report-only kinds in the ownership registry. A lock
whose run had already ended was released only by `sweepStaleIssueLocks`
on its own schedule. Messages saved behind the lock then still waited
out their 30-second refusal backoff.

**Expected behavior**
A lock whose owner run is proven ended (a terminal or missing run row)
is settled by the reconciler on that proof, never by age. A lock that a
same-run resume, an explicit-retry claim, a manual admission hold or a
newly live run still needs is kept.

## What Changed

- New `server/src/services/recovery/issue-lock-settlement.ts`:
`settleIssueLock` runs the stale-lock sweep for exactly that task, so
its existing compare-and-swap clear and all of its guards decide. It
loads the sweep lazily.
- `recovery/ownership-registry.ts`: `issue_execution_lock` and
`issue_checkout_lock` register `settleIssueLock`.
- `recovery/service.ts`: clearing a stale lock now calls
`promoteDeferredWorkIntents(trigger: "lock_released")` before its write,
in the same transaction.
- The periodic sweep stays, because its pre-pass also hands lost-process
runs to the single reaper (#20004). The reconciler is now the faster
backstop for the lock itself.

## Verification

Each focused file ran alone, in one process, on embedded Postgres:
- `ownership-reconciliation-report.test.ts` (25 pass). A new test covers
dead, live and resuming locks: a read-only pass never settles; the dead
run's lock is cleared with an `issue.stale_lock_cleared` row; the live
and same-run-resume locks are kept. It fails against `master`'s registry
and passes here. The test that lists settling kinds now includes both
lock kinds.
- `recovery-stale-issue-lock-sweep.test.ts` (49 pass). A new test checks
that a message saved behind a stale lock is made due when the lock is
cleared. It fails with `master`'s `service.ts` and passes here.
- The server package typecheck passes.

## Risks

- The reconciler can now clear a lock that only the periodic sweep
cleared before. It goes through the same compare-and-swap and guards, so
the outcome is the same, only sooner.
- More passes call the sweep for a single task. Each call is scoped to
one issue.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-08 03:51:18 -06:00
1 parent 7c508d3e42
commit ada44a669a
5 files changed
+113 -4

No files matched your search

@@ -0,0 +1,28 @@
import type { Db } from "@paperclipai/db";
import type { OwnerProof } from "../owner-proof.js";
import type { InProgressRecord, SettleOutcome } from "./ownership-registry.js";
/**
* Settle of the `issue_execution_lock` and `issue_checkout_lock` ownership
* kinds (plan Phase 2, taken over by Phase 3 for non-wake recovery).
*
* A task's lock is owned by the run it names. That run's terminal row, or a
* missing row, is the proof the reconciler acts on; age never is. Settling
* runs the stale-lock sweep for exactly this task, so the one compare-and-swap
* clear decides: it re-reads both lock columns under the task row lock, keeps
* a lock that a same-run resume, an explicit-retry claim, a manual admission
* hold or a newly live run still needs, and releases messages waiting behind
* the lock. The sweep loads lazily so the registry stays light.
*/
export async function settleIssueLock(db: Db, record: InProgressRecord,
_context: { now: Date; proof: OwnerProof }): Promise<SettleOutcome> {
if (record.owner.kind !== "run" || !record.companyId || !record.issueId) return { settled: false, reason: "owner_not_a_run" };
const { recoveryService } = await import("./service.js");
const result = await recoveryService(db, {
// Clearing a stale lock never starts work itself; the release drain does.
enqueueWakeup: async () => null,
}).sweepStaleIssueLocks({ companyId: record.companyId, issueId: record.issueId });
return result.issueIds.includes(record.issueId)
? { settled: true, reason: "stale_lock_cleared" }
: { settled: false, reason: "lock_still_needed" };
}
@@ -34,6 +34,7 @@ import {
import { settleClaimedWakeRequest } from "./wake-request-claimed-settlement.js";
import { settleAbandonedWorkspaceFinalizeAttempt } from "./workspace-finalize-attempt-settlement.js";
import { nativeControllerOwnerStamp, runProcessOwnerStamp } from "../native-runtime/native-owner-stamps.js";
import { settleIssueLock } from "./issue-lock-settlement.js";
import { legacyControllerOwnerStamp, settleLegacyControllerLease } from "../legacy-controller-lease.js";
import {
nativeLeaseHolderStamp,
@@ -892,7 +893,7 @@ export const OWNABLE_KINDS: readonly OwnableKind[] = [
[2]),
{
id: "issue_execution_lock", title: "Issue execution lock", obligation: "ownership", storage: "database",
settle: null, invariantChecks: [2, 3, 7],
settle: settleIssueLock, invariantChecks: [2, 3, 7],
listInProgress: async (db, limit, offset) => {
const rows = await db.select({ id: issues.id, companyId: issues.companyId, runId: issues.executionRunId,
updatedAt: issues.updatedAt }).from(issues).where(isNotNull(issues.executionRunId)).orderBy(asc(issues.id)).limit(limit + 1).offset(offset);
@@ -905,7 +906,7 @@ export const OWNABLE_KINDS: readonly OwnableKind[] = [
},
{
id: "issue_checkout_lock", title: "Issue checkout lock", obligation: "ownership", storage: "database",
settle: null, invariantChecks: [2, 7],
settle: settleIssueLock, invariantChecks: [2, 7],
listInProgress: async (db, limit, offset) => {
const rows = await db.select({ id: issues.id, companyId: issues.companyId, runId: issues.checkoutRunId,
updatedAt: issues.updatedAt }).from(issues).where(isNotNull(issues.checkoutRunId)).orderBy(asc(issues.id)).limit(limit + 1).offset(offset);
+8 -1
View File
@@ -1,4 +1,5 @@
import { notifyScheduledRetryDispatch } from "../run-dispatch-notifications.js";
import { promoteDeferredWorkIntents } from "../work-intent-outbox.js";
import { isContextCompactionRun } from "../context-compaction.js";
import { notifyNativeEnvironmentSettlement } from "../native-environment-settlement-notifications.js";
import { notifyExternalAgentSettlement } from "../../modules/external-agents/index.js";
@@ -6544,7 +6545,7 @@ export function recoveryService(
}
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId: issue.id });
await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id });
return tx
const cleared = await tx
.update(issues)
.set({
checkoutRunId: null,
@@ -6570,6 +6571,12 @@ export function recoveryService(
)
.returning({ id: issues.id })
.then((rows) => rows[0] ?? null);
// Releasing the lock is a hold-release event: messages saved behind it
// are promoted through the one outbox path (section E), only when the
// guarded clear actually released it. The promotion registers its own
// work signals before its write.
if (cleared) await promoteDeferredWorkIntents(tx, { companyId: issue.companyId, trigger: "lock_released", issueId: issue.id });
return cleared;
});
if (!updated) continue;