feat(recovery): settle workspace runtime leases of proven-gone owners on the shared path (#20015)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that starts, stops or repairs a workspace's runtime services
takes a runtime lease, so two tasks never drive the same workspace at
once
> - A lease whose owner task or run has ended must be released. Until
now that happened only when the next claimant arrived and checked, or
when 30 minutes of inactivity passed
> - The durable ownership plan releases a record when its owner is
proven gone, by identity and never by age, through the shared reconciler
> - This pull request registers the runtime lease with the reconciler.
Proof releases a lease; the 30-minute idle rule stays only as a fairness
policy at claim time
> - The benefit is that a workspace locked by a finished task frees
itself at once, and a live task's lease is never taken by age

## Linked Issues or Issue Description

Refs #19831 (plan, Phase 2 sections A and C), #19838 (inventory:
"Execution workspace runtime lease", and the "Released only by TTL/age"
list).

**What happened?**
`execution_workspace_runtime_lease` was a report-only kind. A lease
whose issue or run had ended stayed held until another claim came, or
until its 30-minute TTL expired. The claim path also mixed the TTL with
the ownership check.

**Expected behavior**
An issue- or run-owned lease is released when its owner is proven gone:
the issue is missing, hidden or terminal, or the run is missing or
terminal. Age is not proof. An agent-scoped owner has no identity to
prove, so the lease's expiry plus the next claim's locked takeover is
the fenced handoff.

## What Changed

- New `workspace-runtime-lease-settlement.ts`:
`settleExecutionWorkspaceRuntimeLease` locks the row, requires the same
owner key, re-proves that owner gone, and only then deletes the lease
(compare-and-swap).
- `workspace-runtime-leases.ts`: the new
`proveWorkspaceRuntimeLeaseOwnerGone` holds the identity proof. The
claim path checks it first, then applies the 30-minute idle reclaim as a
separate fairness policy.
- `recovery/ownership-registry.ts`: the kind registers the settle.
Owners are now `issue`, `run`, or a fence on `expires_at` for
agent-scoped keys. `RecordOwner` gains an `issue` kind.
- `recovery/ownership-invariant-report.ts`: issue owners are classified
(missing, hidden or terminal means gone).

Salvaged from the stopped Phase 2 workspace agent's uncommitted work. I
resolved it against current `master` and verified it.

## Verification

- New `workspace-runtime-lease-ownership.test.ts`: 8 pass. Six fail on
`master` and pass here:
  - a crash after the claim;
  - hidden or missing issues, and terminal or missing runs;
  - an agent-scoped lease never settles;
  - a read-only pass changes nothing;
  - a lease taken over after the proof is kept;
  - the release reason names the ownership proof, not the idle policy.

Two guards pass on both: a live owner is never released by age, and the
idle reclaim still works at claim time.
- `ownership-reconciliation-report.test.ts` (25),
`workspace-runtime-leases.test.ts` (10) and
`execution-workspace-runtime-lease-route.test.ts` (4) pass.
- The server package typecheck passes.

## Risks

- The reconciler now deletes leases whose owner is proven gone. It
always re-proves under the row lock against the exact owner key.
- The idle reclaim behaves as before at claim time.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-08 04:45:48 -06:00
1 parent a8ab1d3d94
commit 2aac7d29ff
6 files changed
+324 -19

No files matched your search

@@ -145,7 +145,7 @@ describe("report-only ownership reconciliation", () => {
expect(OWNABLE_KINDS.filter(kind => kind.settle !== null).map(kind => kind.id))
.toEqual(["native_accepted_result_lease", "native_finalizer_lease", "native_cleanup_lease", "sandbox_creation_intent",
"environment_run_lease_active", "environment_teardown_claim", "environment_retained_orphan_sandbox",
"native_workspace_finalize_attempt", "wake_request_claimed", "legacy_controller_lease",
"native_workspace_finalize_attempt", "execution_workspace_runtime_lease", "wake_request_claimed", "legacy_controller_lease",
"issue_execution_lock", "issue_checkout_lock"]);
expect(OWNABLE_KINDS.filter(kind => !kind.listInProgress).every(kind => kind.coverageGap)).toBe(true);
});
@@ -0,0 +1,196 @@
import { randomUUID } from "node:crypto";
import { eq, sql } from "drizzle-orm";
import { afterAll, beforeAll, beforeEach, describe, expect, it } from "vitest";
import {
activityLog,
agents,
companies,
createDb,
executionWorkspaceRuntimeLeases,
executionWorkspaces,
heartbeatRuns,
issues,
projects,
} from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import {
createOwnershipReportState,
runOwnershipReconciliationPass,
settleOwnableKind,
} from "../services/recovery/ownership-invariant-report.js";
import { OWNABLE_KINDS } from "../services/recovery/ownership-registry.js";
import { settleExecutionWorkspaceRuntimeLease } from "../services/workspace-runtime-lease-settlement.js";
import {
WORKSPACE_RUNTIME_LEASE_TTL_MS,
workspaceRuntimeLeaseService,
} from "../services/workspace-runtime-leases.js";
const support = await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = support.supported ? describe : describe.skip;
const KIND = "execution_workspace_runtime_lease";
const kind = OWNABLE_KINDS.find(entry => entry.id === KIND)!;
/**
* The execution workspace runtime lease on the shared ownership path: a lease
* whose issue or run owner is proven gone is released by identity, never by
* age; live and agent-scoped owners are never released by the reconciler.
*/
describeEmbeddedPostgres("execution workspace runtime lease ownership", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let db!: ReturnType<typeof createDb>;
let companyId: string;
let projectId: string;
let agentId: string;
beforeAll(async () => {
database = await startEmbeddedPostgresTestDatabase("paperclip-runtime-lease-ownership-");
db = createDb(database.connectionString);
}, 60_000);
afterAll(async () => {
await database?.cleanup();
});
beforeEach(async () => {
await db.execute(sql`truncate activity_log, execution_workspace_runtime_leases, issues, execution_workspaces,
projects, heartbeat_runs, agents, companies cascade`);
companyId = randomUUID(); projectId = randomUUID(); agentId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Lease ownership", issuePrefix: "LOW",
requireBoardApprovalForNewAgents: false });
await db.insert(projects).values({ id: projectId, companyId, name: "Lane", status: "in_progress" });
await db.insert(agents).values({ id: agentId, companyId, name: "Engineer", role: "engineer" });
});
async function workspace() {
const id = randomUUID();
await db.insert(executionWorkspaces).values({ id, companyId, projectId, mode: "shared_workspace",
strategyType: "git_worktree", name: "Shared", status: "active", cwd: `/tmp/lease-${id}` });
return id;
}
async function issue(status = "in_progress") {
const id = randomUUID();
await db.insert(issues).values({ id, companyId, projectId, title: "Lane issue", status, priority: "high",
assigneeAgentId: agentId });
return id;
}
async function run(status = "running") {
const id = randomUUID();
await db.insert(heartbeatRuns).values({ id, companyId, agentId, status });
return id;
}
const owner = (input: { issueId?: string; runId?: string }) =>
({ actorType: "agent", agentId, runId: input.runId ?? null, issueId: input.issueId ?? null });
async function claim(executionWorkspaceId: string, input: { issueId?: string; runId?: string }, now?: Date) {
return workspaceRuntimeLeaseService(db).claim({ companyId, executionWorkspaceId, action: "start",
owner: owner(input), now });
}
const leaseRows = () => db.select().from(executionWorkspaceRuntimeLeases);
const pass = (record = true) => runOwnershipReconciliationPass(db, { state: createOwnershipReportState(),
kinds: [kind], record });
it("settles a lease whose owner issue finished without releasing it (crash after claim)", async () => {
const workspaceId = await workspace();
const issueId = await issue();
await claim(workspaceId, { issueId });
// The owner died between entering the lease and leaving it: the issue is
// done, and nothing released the lane.
await db.update(issues).set({ status: "done" }).where(eq(issues.id, issueId));
const report = await pass();
expect(report.kinds[0]).toMatchObject({ kind: KIND, inProgress: 1, settled: 1 });
expect(report.kinds[0]!.reasons).toEqual({ owner_issue_terminal: 1 });
expect(await leaseRows()).toHaveLength(0);
// The release is logged with the previous owner and the proof.
expect(await db.select().from(activityLog).where(eq(activityLog.action, "execution_workspace.runtime_lease_released")))
.toMatchObject([{ actorId: "ownership-reconciler", entityType: "execution_workspace", entityId: workspaceId,
details: { previousOwnerKey: `issue:${issueId}`, proof: "owner_issue_terminal" } }]);
});
it("settles leases of hidden or missing issues and of terminal or missing runs", async () => {
const hiddenIssue = await issue();
const missingIssue = await issue();
const terminalRun = await run();
const missingRun = await run();
await claim(await workspace(), { issueId: hiddenIssue });
await claim(await workspace(), { issueId: missingIssue });
await claim(await workspace(), { runId: terminalRun });
await claim(await workspace(), { runId: missingRun });
await db.update(issues).set({ hiddenAt: new Date() }).where(eq(issues.id, hiddenIssue));
await db.delete(issues).where(eq(issues.id, missingIssue));
await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, terminalRun));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, missingRun));
const settled = await settleOwnableKind(db, KIND);
expect(settled.map(entry => entry.reason).sort()).toEqual(
["owner_issue_hidden", "owner_issue_missing", "owner_run_missing", "owner_run_terminal"]);
expect(await leaseRows()).toHaveLength(0);
});
it("never releases a live issue's or run's lease by age, even long past the idle window", async () => {
const longAgo = new Date(Date.now() - 10 * WORKSPACE_RUNTIME_LEASE_TTL_MS);
await claim(await workspace(), { issueId: await issue("blocked") }, longAgo);
await claim(await workspace(), { runId: await run("running") }, longAgo);
const report = await pass();
expect(report.kinds[0]).toMatchObject({ inProgress: 2, settled: 0, verdicts: { gone: 0, alive: 2, unprovable: 0 } });
expect(await leaseRows()).toHaveLength(2);
});
it("never settles an agent-scoped lease: alive while held, unprovable once expired", async () => {
const held = await workspace();
const expired = await workspace();
await claim(held, {});
await claim(expired, {}, new Date(Date.now() - 2 * WORKSPACE_RUNTIME_LEASE_TTL_MS));
const report = await pass();
expect(report.kinds[0]).toMatchObject({ inProgress: 2, settled: 0, verdicts: { gone: 0, alive: 1, unprovable: 1 } });
expect(report.kinds[0]!.reasons).toEqual({ fence_held: 1, fence_expired_without_provider_proof: 1 });
expect(await leaseRows()).toHaveLength(2);
});
it("changes nothing on a read-only pass", async () => {
const issueId = await issue();
await claim(await workspace(), { issueId });
await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, issueId));
const report = await pass(false);
expect(report.kinds[0]).toMatchObject({ settled: 0, verdicts: { gone: 1, alive: 0, unprovable: 0 } });
expect(await leaseRows()).toHaveLength(1);
});
it("keeps a lease that was taken over after the proof (compare-and-swap on the proven owner)", async () => {
const workspaceId = await workspace();
const doneIssue = await issue();
const nextIssue = await issue();
await claim(workspaceId, { issueId: doneIssue });
await db.update(issues).set({ status: "done" }).where(eq(issues.id, doneIssue));
const listed = (await kind.listInProgress!(db, 10, 0)).records;
expect(listed).toHaveLength(1);
// A competing claim reclaims the lane between the proof and the settle.
await claim(workspaceId, { issueId: nextIssue });
const outcome = await settleExecutionWorkspaceRuntimeLease(db, listed[0]!,
{ now: new Date(), proof: { verdict: "gone", reason: "pid_dead" } });
expect(outcome).toEqual({ settled: false, reason: "lease_no_longer_held" });
expect((await leaseRows())[0]?.ownerIssueId).toBe(nextIssue);
});
it("names the ownership proof, not the idle policy, when a gone owner's lease also expired", async () => {
const workspaceId = await workspace();
const doneIssue = await issue();
await claim(workspaceId, { issueId: doneIssue }, new Date(Date.now() - 2 * WORKSPACE_RUNTIME_LEASE_TTL_MS));
await db.update(issues).set({ status: "done" }).where(eq(issues.id, doneIssue));
const reclaimed = await claim(workspaceId, { issueId: await issue() });
expect(reclaimed.reclaimedFrom).toEqual({ ownerKey: `issue:${doneIssue}`, reason: "owner_issue_terminal" });
});
it("keeps the idle-reclaim policy at claim time for a live but idle owner", async () => {
const workspaceId = await workspace();
const idleIssue = await issue();
await claim(workspaceId, { issueId: idleIssue }, new Date(Date.now() - WORKSPACE_RUNTIME_LEASE_TTL_MS - 60_000));
const reclaimed = await claim(workspaceId, { issueId: await issue() });
expect(reclaimed.reclaimedFrom).toEqual({ ownerKey: `issue:${idleIssue}`, reason: "lease_expired" });
});
});
@@ -128,6 +128,7 @@ export function createOwnershipReportState(): OwnershipReportState {
}
const TERMINAL_RUN_STATUSES = ["succeeded", "failed", "cancelled", "timed_out", "interrupted"];
const TERMINAL_ISSUE_STATUSES = ["done", "cancelled"];
const ACTIVE_RUN_STATUSES = ["queued", "running", "scheduled_retry"];
/** An in-progress issue must show a next action within this time. */
const NEXT_ACTION_GRACE_MS = 5 * 60_000;
@@ -156,6 +157,21 @@ async function classifyRunOwners(db: Db, records: InProgressRecord[]): Promise<M
: { verdict: "alive", reason: "owner_run_active" });
}
}
// Issue owners share the map under an `issue:` key (run ids are bare UUIDs).
const issueIds = [...new Set(records.flatMap(record => record.owner.kind === "issue" ? [record.owner.issueId] : []))];
for (let index = 0; index < issueIds.length; index += 500) {
const batch = issueIds.slice(index, index + 500);
const rows = await db.select({ id: issues.id, status: issues.status, hiddenAt: issues.hiddenAt }).from(issues)
.where(inArray(issues.id, batch));
const byId = new Map(rows.map(row => [row.id, row]));
for (const issueId of batch) {
const row = byId.get(issueId);
proofs.set(`issue:${issueId}`, !row ? { verdict: "gone", reason: "owner_issue_missing" }
: row.hiddenAt ? { verdict: "gone", reason: "owner_issue_hidden" }
: TERMINAL_ISSUE_STATUSES.includes(row.status) ? { verdict: "gone", reason: "owner_issue_terminal" }
: { verdict: "alive", reason: "owner_issue_active" });
}
}
return proofs;
}
@@ -165,6 +181,7 @@ async function classify(owner: RecordOwner, runProofs: Map<string, Classificatio
case "process": return proveOwnerGone(owner.stamp, probe);
case "fence": return proveOwnerGone(owner.stamp, owner.fence, now);
case "run": return runProofs.get(owner.runId) ?? { verdict: "unprovable", reason: "owner_run_unread" };
case "issue": return runProofs.get(`issue:${owner.issueId}`) ?? { verdict: "unprovable", reason: "owner_issue_unread" };
case "operator": return { verdict: "alive", reason: "operator_owned" };
// A queue is not a process: it is never proven gone. A deadline long past
// means the queue is not acting, which is a gap to report, not a release.
@@ -57,6 +57,8 @@ import {
releasableRetainedOrphanCondition,
settleRetainedOrphanSandbox,
} from "../environment-retained-orphan-settlement.js";
import { settleExecutionWorkspaceRuntimeLease } from "../workspace-runtime-lease-settlement.js";
import { parseWorkspaceRuntimeLeaseOwnerKey } from "../workspace-runtime-leases.js";
/**
* Registry of ownable record kinds (plan section C).
@@ -82,6 +84,8 @@ export type RecordOwner =
| { kind: "fence"; stamp: RemoteOwnerStamp; fence: RemoteFenceState }
/** A run id only. The record's owner is that run's execution. */
| { kind: "run"; runId: string }
/** An issue id only. Gone when the issue is missing, hidden or terminal. */
| { kind: "issue"; issueId: string }
/** A person or agent assignment, such as an operator-owned recovery action. */
| { kind: "operator"; ownerType: string | null }
/** A durable work queue that acts on the record by its saved deadline. The
@@ -511,6 +515,21 @@ function spoolKind(id: string, title: string, obligation: Obligation, directory:
};
}
// ----------------------------------------------------------------------------
// execution_workspace_runtime_leases
/** The lease's owner key names its owner: an issue or run is proven by
* identity; an agent-scoped (or unparseable) owner is a fence on the lease
* expiry that only a claim's locked takeover moves. */
function runtimeLeaseOwner(row: typeof executionWorkspaceRuntimeLeases.$inferSelect): RecordOwner {
const owner = parseWorkspaceRuntimeLeaseOwnerKey(row.ownerKey);
if (owner?.kind === "issue") return { kind: "issue", issueId: owner.id };
if (owner?.kind === "run") return { kind: "run", runId: owner.id };
return { kind: "fence", stamp: { kind: "remote", scope: "execution_workspace_runtime_lease", generation: null,
fenceToken: row.ownerKey }, fence: { currentFenceToken: row.ownerKey, currentGeneration: null,
leaseExpiresAt: row.expiresAt } };
}
// ----------------------------------------------------------------------------
// The registry
@@ -715,15 +734,18 @@ export const OWNABLE_KINDS: readonly OwnableKind[] = [
and(eq(workspaceOperations.phase, "remote_workspace_migration"), eq(workspaceOperations.status, "running")),
() => ({ kind: "none" })),
{
// Shared path since Phase 2: a lease whose issue or run owner is proven
// gone (by identity, never age) is released here and at the next claim.
// An agent-scoped owner is a fence on `expires_at`: alive while held,
// unprovable after; only the next claim's locked takeover moves it.
id: "execution_workspace_runtime_lease", title: "Execution workspace runtime lease", obligation: "ownership",
storage: "database", settle: null, invariantChecks: [1, 2],
storage: "database", settle: settleExecutionWorkspaceRuntimeLease, invariantChecks: [1, 2],
listInProgress: async (db, limit, offset) => {
const rows = await db.select().from(executionWorkspaceRuntimeLeases).orderBy(asc(executionWorkspaceRuntimeLeases.id)).limit(limit + 1).offset(offset);
const result = page(rows, limit);
return { truncated: result.truncated, records: result.rows.map(row => ({
recordId: row.id, companyId: row.companyId, runId: row.ownerRunId, issueId: row.ownerIssueId,
owner: row.ownerRunId ? runOwner(row.ownerRunId) : { kind: "operator", ownerType: row.ownerKey.split(":")[0] ?? null },
updatedAt: row.updatedAt,
owner: runtimeLeaseOwner(row), updatedAt: row.updatedAt,
})) };
},
},
@@ -0,0 +1,49 @@
import { and, eq } from "drizzle-orm";
import { executionWorkspaceRuntimeLeases, type Db } from "@paperclipai/db";
import { persistActivity, publishActivity, type ActivityPublication } from "./activity-log.js";
import type { OwnerProof } from "./owner-proof.js";
import type { InProgressRecord, SettleOutcome } from "./recovery/ownership-registry.js";
import { proveWorkspaceRuntimeLeaseOwnerGone } from "./workspace-runtime-leases.js";
/**
* Settle of the `execution_workspace_runtime_lease` ownership kind (plan Phase 2).
*
* The shared reconciler calls this only after it proved the lease's issue or run
* owner gone (issue terminal, hidden or missing; run terminal or missing). Settle
* releases exactly that lease: under the row lock it re-reads the row, requires
* the same owner key, and re-proves the owner gone, so a lease renewed or taken
* over since the proof is never released. Age plays no part. Agent-scoped leases
* are never settled here: no identity proves such an owner gone, and their expiry
* plus the next claim's locked takeover is the fenced handoff.
*/
export async function settleExecutionWorkspaceRuntimeLease(db: Db, record: InProgressRecord,
_context: { now: Date; proof: OwnerProof }): Promise<SettleOutcome> {
const ownerKey = record.owner.kind === "issue" ? `issue:${record.owner.issueId}`
: record.owner.kind === "run" ? `run:${record.owner.runId}` : null;
if (!ownerKey) return { settled: false, reason: "owner_not_provable" };
let publication: ActivityPublication | undefined;
const outcome = await db.transaction(async transaction => {
const tx = transaction as unknown as Db;
const [row] = await tx.select().from(executionWorkspaceRuntimeLeases)
.where(and(eq(executionWorkspaceRuntimeLeases.id, record.recordId),
eq(executionWorkspaceRuntimeLeases.ownerKey, ownerKey)))
.for("update");
if (!row) return { settled: false, reason: "lease_no_longer_held" };
const gone = await proveWorkspaceRuntimeLeaseOwnerGone(tx, row);
if (!gone) return { settled: false, reason: "owner_changed_since_proof" };
const deleted = await tx.delete(executionWorkspaceRuntimeLeases)
.where(and(eq(executionWorkspaceRuntimeLeases.id, row.id), eq(executionWorkspaceRuntimeLeases.ownerKey, ownerKey)))
.returning({ id: executionWorkspaceRuntimeLeases.id });
if (!deleted.length) return { settled: false, reason: "lease_no_longer_held" };
// The lease row is gone; the activity entry keeps who held it and why it was released.
publication = (await persistActivity(tx, {
companyId: row.companyId, actorType: "system", actorId: "ownership-reconciler",
action: "execution_workspace.runtime_lease_released", entityType: "execution_workspace",
entityId: row.executionWorkspaceId, runId: row.ownerRunId, issueId: row.ownerIssueId,
details: { leaseId: row.id, previousOwnerKey: row.ownerKey, lastAction: row.lastAction, proof: gone },
})).publication;
return { settled: true, reason: gone };
});
if (outcome.settled && publication) publishActivity(publication);
return outcome;
}
+36 -15
View File
@@ -31,10 +31,12 @@ const TERMINAL_HEARTBEAT_RUN_STATUSES = new Set([
export const LEASED_WORKSPACE_RUNTIME_ACTIONS: readonly string[] = ["start", "stop", "restart", "repair"];
/**
* Upper bound on how long a lease survives without the owner touching it. Recovery
* from a stale owner is otherwise driven by owner eligibility, which is cheaper and
* more precise; the TTL only bounds the cases eligibility cannot see (for example an
* agent-scoped owner with no run or issue identity).
* Idle-reclaim policy window. A lease untouched this long may be taken over by the
* next claimant at claim time. This is a fairness policy, not an ownership proof:
* dead owners (terminal, hidden or missing issue; terminal or missing run) are
* released by identity, with no age, both at claim time and by the shared
* ownership reconciler. For agent-scoped owners, which no identity can prove gone,
* expiry plus the locked takeover is the fenced handoff.
*/
export const WORKSPACE_RUNTIME_LEASE_TTL_MS = 30 * 60 * 1000;
@@ -72,7 +74,7 @@ export function buildWorkspaceRuntimeLeaseOwnerKey(owner: WorkspaceRuntimeLeaseO
return null;
}
function parseOwnerKey(ownerKey: string): { kind: "issue" | "run" | "agent"; id: string } | null {
export function parseWorkspaceRuntimeLeaseOwnerKey(ownerKey: string): { kind: "issue" | "run" | "agent"; id: string } | null {
const separator = ownerKey.indexOf(":");
if (separator <= 0) return null;
const kind = ownerKey.slice(0, separator);
@@ -82,14 +84,18 @@ function parseOwnerKey(ownerKey: string): { kind: "issue" | "run" | "agent"; id:
return { kind, id };
}
async function evaluateStaleOwner(
tx: LeaseTx,
lease: ExecutionWorkspaceRuntimeLeaseRow,
now: Date,
): Promise<WorkspaceRuntimeLeaseStaleReason | null> {
if (lease.expiresAt.getTime() <= now.getTime()) return "lease_expired";
const owner = parseOwnerKey(lease.ownerKey);
/**
* Proves by identity that an issue- or run-owned lease's owner is gone: the issue
* is missing, hidden or terminal, or the run is missing or terminal. Returns null
* while the owner lives, and for agent-scoped or unparseable owners, which have no
* identity to prove. The shared ownership reconciler's settle and the claim path
* both use it, so the lane is released on the same proof wherever it is checked.
*/
export async function proveWorkspaceRuntimeLeaseOwnerGone(
tx: Pick<Db, "select">,
lease: Pick<ExecutionWorkspaceRuntimeLeaseRow, "ownerKey" | "companyId">,
): Promise<Exclude<WorkspaceRuntimeLeaseStaleReason, "lease_expired"> | null> {
const owner = parseWorkspaceRuntimeLeaseOwnerKey(lease.ownerKey);
if (owner?.kind === "issue") {
const ownerIssue = await tx
.select({ status: issues.status, hiddenAt: issues.hiddenAt })
@@ -113,11 +119,26 @@ async function evaluateStaleOwner(
return null;
}
// Agent-scoped and unparseable owners have no lifecycle to inspect, so the TTL
// checked above is their only recovery path.
return null;
}
async function evaluateStaleOwner(
tx: LeaseTx,
lease: ExecutionWorkspaceRuntimeLeaseRow,
now: Date,
): Promise<WorkspaceRuntimeLeaseStaleReason | null> {
// Ownership proof first: a gone issue or run releases the lane with no age
// involved (the same proof the shared reconciler settles on).
const gone = await proveWorkspaceRuntimeLeaseOwnerGone(tx, lease);
if (gone) return gone;
// Idle reclaim: a POLICY, not an ownership proof. A lease untouched for the
// TTL is handed to the next claimant even when its issue or run still lives
// (fairness between issues sharing a workspace). For agent-scoped owners,
// which no identity can prove gone, expiry plus this locked takeover is the
// fenced handoff. The reconciler never releases a lease on this rule.
return lease.expiresAt.getTime() <= now.getTime() ? "lease_expired" : null;
}
function throwLeaseConflict(input: {
lease: ExecutionWorkspaceRuntimeLeaseRow;
requestedAction: string;