mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 12:37:55 +01:00
feat(recovery): settle workspace runtime leases of proven-gone owners on the shared path (#20015)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task that starts, stops or repairs a workspace's runtime services takes a runtime lease, so two tasks never drive the same workspace at once > - A lease whose owner task or run has ended must be released. Until now that happened only when the next claimant arrived and checked, or when 30 minutes of inactivity passed > - The durable ownership plan releases a record when its owner is proven gone, by identity and never by age, through the shared reconciler > - This pull request registers the runtime lease with the reconciler. Proof releases a lease; the 30-minute idle rule stays only as a fairness policy at claim time > - The benefit is that a workspace locked by a finished task frees itself at once, and a live task's lease is never taken by age ## Linked Issues or Issue Description Refs #19831 (plan, Phase 2 sections A and C), #19838 (inventory: "Execution workspace runtime lease", and the "Released only by TTL/age" list). **What happened?** `execution_workspace_runtime_lease` was a report-only kind. A lease whose issue or run had ended stayed held until another claim came, or until its 30-minute TTL expired. The claim path also mixed the TTL with the ownership check. **Expected behavior** An issue- or run-owned lease is released when its owner is proven gone: the issue is missing, hidden or terminal, or the run is missing or terminal. Age is not proof. An agent-scoped owner has no identity to prove, so the lease's expiry plus the next claim's locked takeover is the fenced handoff. ## What Changed - New `workspace-runtime-lease-settlement.ts`: `settleExecutionWorkspaceRuntimeLease` locks the row, requires the same owner key, re-proves that owner gone, and only then deletes the lease (compare-and-swap). - `workspace-runtime-leases.ts`: the new `proveWorkspaceRuntimeLeaseOwnerGone` holds the identity proof. The claim path checks it first, then applies the 30-minute idle reclaim as a separate fairness policy. - `recovery/ownership-registry.ts`: the kind registers the settle. Owners are now `issue`, `run`, or a fence on `expires_at` for agent-scoped keys. `RecordOwner` gains an `issue` kind. - `recovery/ownership-invariant-report.ts`: issue owners are classified (missing, hidden or terminal means gone). Salvaged from the stopped Phase 2 workspace agent's uncommitted work. I resolved it against current `master` and verified it. ## Verification - New `workspace-runtime-lease-ownership.test.ts`: 8 pass. Six fail on `master` and pass here: - a crash after the claim; - hidden or missing issues, and terminal or missing runs; - an agent-scoped lease never settles; - a read-only pass changes nothing; - a lease taken over after the proof is kept; - the release reason names the ownership proof, not the idle policy. Two guards pass on both: a live owner is never released by age, and the idle reclaim still works at claim time. - `ownership-reconciliation-report.test.ts` (25), `workspace-runtime-leases.test.ts` (10) and `execution-workspace-runtime-lease-route.test.ts` (4) pass. - The server package typecheck passes. ## Risks - The reconciler now deletes leases whose owner is proven gone. It always re-proves under the row lock against the exact owner key. - The idle reclaim behaves as before at claim time. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
a8ab1d3d94
commit
2aac7d29ff
6 files changed
+324
-19
No files matched your search
@@ -145,7 +145,7 @@ describe("report-only ownership reconciliation", () => {
|
||||
expect(OWNABLE_KINDS.filter(kind => kind.settle !== null).map(kind => kind.id))
|
||||
.toEqual(["native_accepted_result_lease", "native_finalizer_lease", "native_cleanup_lease", "sandbox_creation_intent",
|
||||
"environment_run_lease_active", "environment_teardown_claim", "environment_retained_orphan_sandbox",
|
||||
"native_workspace_finalize_attempt", "wake_request_claimed", "legacy_controller_lease",
|
||||
"native_workspace_finalize_attempt", "execution_workspace_runtime_lease", "wake_request_claimed", "legacy_controller_lease",
|
||||
"issue_execution_lock", "issue_checkout_lock"]);
|
||||
expect(OWNABLE_KINDS.filter(kind => !kind.listInProgress).every(kind => kind.coverageGap)).toBe(true);
|
||||
});
|
||||
|
||||
@@ -0,0 +1,196 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it } from "vitest";
|
||||
import {
|
||||
activityLog,
|
||||
agents,
|
||||
companies,
|
||||
createDb,
|
||||
executionWorkspaceRuntimeLeases,
|
||||
executionWorkspaces,
|
||||
heartbeatRuns,
|
||||
issues,
|
||||
projects,
|
||||
} from "@paperclipai/db";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import {
|
||||
createOwnershipReportState,
|
||||
runOwnershipReconciliationPass,
|
||||
settleOwnableKind,
|
||||
} from "../services/recovery/ownership-invariant-report.js";
|
||||
import { OWNABLE_KINDS } from "../services/recovery/ownership-registry.js";
|
||||
import { settleExecutionWorkspaceRuntimeLease } from "../services/workspace-runtime-lease-settlement.js";
|
||||
import {
|
||||
WORKSPACE_RUNTIME_LEASE_TTL_MS,
|
||||
workspaceRuntimeLeaseService,
|
||||
} from "../services/workspace-runtime-leases.js";
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = support.supported ? describe : describe.skip;
|
||||
|
||||
const KIND = "execution_workspace_runtime_lease";
|
||||
const kind = OWNABLE_KINDS.find(entry => entry.id === KIND)!;
|
||||
|
||||
/**
|
||||
* The execution workspace runtime lease on the shared ownership path: a lease
|
||||
* whose issue or run owner is proven gone is released by identity, never by
|
||||
* age; live and agent-scoped owners are never released by the reconciler.
|
||||
*/
|
||||
describeEmbeddedPostgres("execution workspace runtime lease ownership", () => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let db!: ReturnType<typeof createDb>;
|
||||
let companyId: string;
|
||||
let projectId: string;
|
||||
let agentId: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
database = await startEmbeddedPostgresTestDatabase("paperclip-runtime-lease-ownership-");
|
||||
db = createDb(database.connectionString);
|
||||
}, 60_000);
|
||||
afterAll(async () => {
|
||||
await database?.cleanup();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await db.execute(sql`truncate activity_log, execution_workspace_runtime_leases, issues, execution_workspaces,
|
||||
projects, heartbeat_runs, agents, companies cascade`);
|
||||
companyId = randomUUID(); projectId = randomUUID(); agentId = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name: "Lease ownership", issuePrefix: "LOW",
|
||||
requireBoardApprovalForNewAgents: false });
|
||||
await db.insert(projects).values({ id: projectId, companyId, name: "Lane", status: "in_progress" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Engineer", role: "engineer" });
|
||||
});
|
||||
|
||||
async function workspace() {
|
||||
const id = randomUUID();
|
||||
await db.insert(executionWorkspaces).values({ id, companyId, projectId, mode: "shared_workspace",
|
||||
strategyType: "git_worktree", name: "Shared", status: "active", cwd: `/tmp/lease-${id}` });
|
||||
return id;
|
||||
}
|
||||
async function issue(status = "in_progress") {
|
||||
const id = randomUUID();
|
||||
await db.insert(issues).values({ id, companyId, projectId, title: "Lane issue", status, priority: "high",
|
||||
assigneeAgentId: agentId });
|
||||
return id;
|
||||
}
|
||||
async function run(status = "running") {
|
||||
const id = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id, companyId, agentId, status });
|
||||
return id;
|
||||
}
|
||||
const owner = (input: { issueId?: string; runId?: string }) =>
|
||||
({ actorType: "agent", agentId, runId: input.runId ?? null, issueId: input.issueId ?? null });
|
||||
async function claim(executionWorkspaceId: string, input: { issueId?: string; runId?: string }, now?: Date) {
|
||||
return workspaceRuntimeLeaseService(db).claim({ companyId, executionWorkspaceId, action: "start",
|
||||
owner: owner(input), now });
|
||||
}
|
||||
const leaseRows = () => db.select().from(executionWorkspaceRuntimeLeases);
|
||||
const pass = (record = true) => runOwnershipReconciliationPass(db, { state: createOwnershipReportState(),
|
||||
kinds: [kind], record });
|
||||
|
||||
it("settles a lease whose owner issue finished without releasing it (crash after claim)", async () => {
|
||||
const workspaceId = await workspace();
|
||||
const issueId = await issue();
|
||||
await claim(workspaceId, { issueId });
|
||||
// The owner died between entering the lease and leaving it: the issue is
|
||||
// done, and nothing released the lane.
|
||||
await db.update(issues).set({ status: "done" }).where(eq(issues.id, issueId));
|
||||
|
||||
const report = await pass();
|
||||
expect(report.kinds[0]).toMatchObject({ kind: KIND, inProgress: 1, settled: 1 });
|
||||
expect(report.kinds[0]!.reasons).toEqual({ owner_issue_terminal: 1 });
|
||||
expect(await leaseRows()).toHaveLength(0);
|
||||
// The release is logged with the previous owner and the proof.
|
||||
expect(await db.select().from(activityLog).where(eq(activityLog.action, "execution_workspace.runtime_lease_released")))
|
||||
.toMatchObject([{ actorId: "ownership-reconciler", entityType: "execution_workspace", entityId: workspaceId,
|
||||
details: { previousOwnerKey: `issue:${issueId}`, proof: "owner_issue_terminal" } }]);
|
||||
});
|
||||
|
||||
it("settles leases of hidden or missing issues and of terminal or missing runs", async () => {
|
||||
const hiddenIssue = await issue();
|
||||
const missingIssue = await issue();
|
||||
const terminalRun = await run();
|
||||
const missingRun = await run();
|
||||
await claim(await workspace(), { issueId: hiddenIssue });
|
||||
await claim(await workspace(), { issueId: missingIssue });
|
||||
await claim(await workspace(), { runId: terminalRun });
|
||||
await claim(await workspace(), { runId: missingRun });
|
||||
await db.update(issues).set({ hiddenAt: new Date() }).where(eq(issues.id, hiddenIssue));
|
||||
await db.delete(issues).where(eq(issues.id, missingIssue));
|
||||
await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, terminalRun));
|
||||
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, missingRun));
|
||||
|
||||
const settled = await settleOwnableKind(db, KIND);
|
||||
expect(settled.map(entry => entry.reason).sort()).toEqual(
|
||||
["owner_issue_hidden", "owner_issue_missing", "owner_run_missing", "owner_run_terminal"]);
|
||||
expect(await leaseRows()).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("never releases a live issue's or run's lease by age, even long past the idle window", async () => {
|
||||
const longAgo = new Date(Date.now() - 10 * WORKSPACE_RUNTIME_LEASE_TTL_MS);
|
||||
await claim(await workspace(), { issueId: await issue("blocked") }, longAgo);
|
||||
await claim(await workspace(), { runId: await run("running") }, longAgo);
|
||||
|
||||
const report = await pass();
|
||||
expect(report.kinds[0]).toMatchObject({ inProgress: 2, settled: 0, verdicts: { gone: 0, alive: 2, unprovable: 0 } });
|
||||
expect(await leaseRows()).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("never settles an agent-scoped lease: alive while held, unprovable once expired", async () => {
|
||||
const held = await workspace();
|
||||
const expired = await workspace();
|
||||
await claim(held, {});
|
||||
await claim(expired, {}, new Date(Date.now() - 2 * WORKSPACE_RUNTIME_LEASE_TTL_MS));
|
||||
|
||||
const report = await pass();
|
||||
expect(report.kinds[0]).toMatchObject({ inProgress: 2, settled: 0, verdicts: { gone: 0, alive: 1, unprovable: 1 } });
|
||||
expect(report.kinds[0]!.reasons).toEqual({ fence_held: 1, fence_expired_without_provider_proof: 1 });
|
||||
expect(await leaseRows()).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("changes nothing on a read-only pass", async () => {
|
||||
const issueId = await issue();
|
||||
await claim(await workspace(), { issueId });
|
||||
await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, issueId));
|
||||
|
||||
const report = await pass(false);
|
||||
expect(report.kinds[0]).toMatchObject({ settled: 0, verdicts: { gone: 1, alive: 0, unprovable: 0 } });
|
||||
expect(await leaseRows()).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("keeps a lease that was taken over after the proof (compare-and-swap on the proven owner)", async () => {
|
||||
const workspaceId = await workspace();
|
||||
const doneIssue = await issue();
|
||||
const nextIssue = await issue();
|
||||
await claim(workspaceId, { issueId: doneIssue });
|
||||
await db.update(issues).set({ status: "done" }).where(eq(issues.id, doneIssue));
|
||||
const listed = (await kind.listInProgress!(db, 10, 0)).records;
|
||||
expect(listed).toHaveLength(1);
|
||||
// A competing claim reclaims the lane between the proof and the settle.
|
||||
await claim(workspaceId, { issueId: nextIssue });
|
||||
|
||||
const outcome = await settleExecutionWorkspaceRuntimeLease(db, listed[0]!,
|
||||
{ now: new Date(), proof: { verdict: "gone", reason: "pid_dead" } });
|
||||
expect(outcome).toEqual({ settled: false, reason: "lease_no_longer_held" });
|
||||
expect((await leaseRows())[0]?.ownerIssueId).toBe(nextIssue);
|
||||
});
|
||||
|
||||
it("names the ownership proof, not the idle policy, when a gone owner's lease also expired", async () => {
|
||||
const workspaceId = await workspace();
|
||||
const doneIssue = await issue();
|
||||
await claim(workspaceId, { issueId: doneIssue }, new Date(Date.now() - 2 * WORKSPACE_RUNTIME_LEASE_TTL_MS));
|
||||
await db.update(issues).set({ status: "done" }).where(eq(issues.id, doneIssue));
|
||||
|
||||
const reclaimed = await claim(workspaceId, { issueId: await issue() });
|
||||
expect(reclaimed.reclaimedFrom).toEqual({ ownerKey: `issue:${doneIssue}`, reason: "owner_issue_terminal" });
|
||||
});
|
||||
|
||||
it("keeps the idle-reclaim policy at claim time for a live but idle owner", async () => {
|
||||
const workspaceId = await workspace();
|
||||
const idleIssue = await issue();
|
||||
await claim(workspaceId, { issueId: idleIssue }, new Date(Date.now() - WORKSPACE_RUNTIME_LEASE_TTL_MS - 60_000));
|
||||
|
||||
const reclaimed = await claim(workspaceId, { issueId: await issue() });
|
||||
expect(reclaimed.reclaimedFrom).toEqual({ ownerKey: `issue:${idleIssue}`, reason: "lease_expired" });
|
||||
});
|
||||
});
|
||||
@@ -128,6 +128,7 @@ export function createOwnershipReportState(): OwnershipReportState {
|
||||
}
|
||||
|
||||
const TERMINAL_RUN_STATUSES = ["succeeded", "failed", "cancelled", "timed_out", "interrupted"];
|
||||
const TERMINAL_ISSUE_STATUSES = ["done", "cancelled"];
|
||||
const ACTIVE_RUN_STATUSES = ["queued", "running", "scheduled_retry"];
|
||||
/** An in-progress issue must show a next action within this time. */
|
||||
const NEXT_ACTION_GRACE_MS = 5 * 60_000;
|
||||
@@ -156,6 +157,21 @@ async function classifyRunOwners(db: Db, records: InProgressRecord[]): Promise<M
|
||||
: { verdict: "alive", reason: "owner_run_active" });
|
||||
}
|
||||
}
|
||||
// Issue owners share the map under an `issue:` key (run ids are bare UUIDs).
|
||||
const issueIds = [...new Set(records.flatMap(record => record.owner.kind === "issue" ? [record.owner.issueId] : []))];
|
||||
for (let index = 0; index < issueIds.length; index += 500) {
|
||||
const batch = issueIds.slice(index, index + 500);
|
||||
const rows = await db.select({ id: issues.id, status: issues.status, hiddenAt: issues.hiddenAt }).from(issues)
|
||||
.where(inArray(issues.id, batch));
|
||||
const byId = new Map(rows.map(row => [row.id, row]));
|
||||
for (const issueId of batch) {
|
||||
const row = byId.get(issueId);
|
||||
proofs.set(`issue:${issueId}`, !row ? { verdict: "gone", reason: "owner_issue_missing" }
|
||||
: row.hiddenAt ? { verdict: "gone", reason: "owner_issue_hidden" }
|
||||
: TERMINAL_ISSUE_STATUSES.includes(row.status) ? { verdict: "gone", reason: "owner_issue_terminal" }
|
||||
: { verdict: "alive", reason: "owner_issue_active" });
|
||||
}
|
||||
}
|
||||
return proofs;
|
||||
}
|
||||
|
||||
@@ -165,6 +181,7 @@ async function classify(owner: RecordOwner, runProofs: Map<string, Classificatio
|
||||
case "process": return proveOwnerGone(owner.stamp, probe);
|
||||
case "fence": return proveOwnerGone(owner.stamp, owner.fence, now);
|
||||
case "run": return runProofs.get(owner.runId) ?? { verdict: "unprovable", reason: "owner_run_unread" };
|
||||
case "issue": return runProofs.get(`issue:${owner.issueId}`) ?? { verdict: "unprovable", reason: "owner_issue_unread" };
|
||||
case "operator": return { verdict: "alive", reason: "operator_owned" };
|
||||
// A queue is not a process: it is never proven gone. A deadline long past
|
||||
// means the queue is not acting, which is a gap to report, not a release.
|
||||
|
||||
@@ -57,6 +57,8 @@ import {
|
||||
releasableRetainedOrphanCondition,
|
||||
settleRetainedOrphanSandbox,
|
||||
} from "../environment-retained-orphan-settlement.js";
|
||||
import { settleExecutionWorkspaceRuntimeLease } from "../workspace-runtime-lease-settlement.js";
|
||||
import { parseWorkspaceRuntimeLeaseOwnerKey } from "../workspace-runtime-leases.js";
|
||||
|
||||
/**
|
||||
* Registry of ownable record kinds (plan section C).
|
||||
@@ -82,6 +84,8 @@ export type RecordOwner =
|
||||
| { kind: "fence"; stamp: RemoteOwnerStamp; fence: RemoteFenceState }
|
||||
/** A run id only. The record's owner is that run's execution. */
|
||||
| { kind: "run"; runId: string }
|
||||
/** An issue id only. Gone when the issue is missing, hidden or terminal. */
|
||||
| { kind: "issue"; issueId: string }
|
||||
/** A person or agent assignment, such as an operator-owned recovery action. */
|
||||
| { kind: "operator"; ownerType: string | null }
|
||||
/** A durable work queue that acts on the record by its saved deadline. The
|
||||
@@ -511,6 +515,21 @@ function spoolKind(id: string, title: string, obligation: Obligation, directory:
|
||||
};
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// execution_workspace_runtime_leases
|
||||
|
||||
/** The lease's owner key names its owner: an issue or run is proven by
|
||||
* identity; an agent-scoped (or unparseable) owner is a fence on the lease
|
||||
* expiry that only a claim's locked takeover moves. */
|
||||
function runtimeLeaseOwner(row: typeof executionWorkspaceRuntimeLeases.$inferSelect): RecordOwner {
|
||||
const owner = parseWorkspaceRuntimeLeaseOwnerKey(row.ownerKey);
|
||||
if (owner?.kind === "issue") return { kind: "issue", issueId: owner.id };
|
||||
if (owner?.kind === "run") return { kind: "run", runId: owner.id };
|
||||
return { kind: "fence", stamp: { kind: "remote", scope: "execution_workspace_runtime_lease", generation: null,
|
||||
fenceToken: row.ownerKey }, fence: { currentFenceToken: row.ownerKey, currentGeneration: null,
|
||||
leaseExpiresAt: row.expiresAt } };
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// The registry
|
||||
|
||||
@@ -715,15 +734,18 @@ export const OWNABLE_KINDS: readonly OwnableKind[] = [
|
||||
and(eq(workspaceOperations.phase, "remote_workspace_migration"), eq(workspaceOperations.status, "running")),
|
||||
() => ({ kind: "none" })),
|
||||
{
|
||||
// Shared path since Phase 2: a lease whose issue or run owner is proven
|
||||
// gone (by identity, never age) is released here and at the next claim.
|
||||
// An agent-scoped owner is a fence on `expires_at`: alive while held,
|
||||
// unprovable after; only the next claim's locked takeover moves it.
|
||||
id: "execution_workspace_runtime_lease", title: "Execution workspace runtime lease", obligation: "ownership",
|
||||
storage: "database", settle: null, invariantChecks: [1, 2],
|
||||
storage: "database", settle: settleExecutionWorkspaceRuntimeLease, invariantChecks: [1, 2],
|
||||
listInProgress: async (db, limit, offset) => {
|
||||
const rows = await db.select().from(executionWorkspaceRuntimeLeases).orderBy(asc(executionWorkspaceRuntimeLeases.id)).limit(limit + 1).offset(offset);
|
||||
const result = page(rows, limit);
|
||||
return { truncated: result.truncated, records: result.rows.map(row => ({
|
||||
recordId: row.id, companyId: row.companyId, runId: row.ownerRunId, issueId: row.ownerIssueId,
|
||||
owner: row.ownerRunId ? runOwner(row.ownerRunId) : { kind: "operator", ownerType: row.ownerKey.split(":")[0] ?? null },
|
||||
updatedAt: row.updatedAt,
|
||||
owner: runtimeLeaseOwner(row), updatedAt: row.updatedAt,
|
||||
})) };
|
||||
},
|
||||
},
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { executionWorkspaceRuntimeLeases, type Db } from "@paperclipai/db";
|
||||
import { persistActivity, publishActivity, type ActivityPublication } from "./activity-log.js";
|
||||
import type { OwnerProof } from "./owner-proof.js";
|
||||
import type { InProgressRecord, SettleOutcome } from "./recovery/ownership-registry.js";
|
||||
import { proveWorkspaceRuntimeLeaseOwnerGone } from "./workspace-runtime-leases.js";
|
||||
|
||||
/**
|
||||
* Settle of the `execution_workspace_runtime_lease` ownership kind (plan Phase 2).
|
||||
*
|
||||
* The shared reconciler calls this only after it proved the lease's issue or run
|
||||
* owner gone (issue terminal, hidden or missing; run terminal or missing). Settle
|
||||
* releases exactly that lease: under the row lock it re-reads the row, requires
|
||||
* the same owner key, and re-proves the owner gone, so a lease renewed or taken
|
||||
* over since the proof is never released. Age plays no part. Agent-scoped leases
|
||||
* are never settled here: no identity proves such an owner gone, and their expiry
|
||||
* plus the next claim's locked takeover is the fenced handoff.
|
||||
*/
|
||||
export async function settleExecutionWorkspaceRuntimeLease(db: Db, record: InProgressRecord,
|
||||
_context: { now: Date; proof: OwnerProof }): Promise<SettleOutcome> {
|
||||
const ownerKey = record.owner.kind === "issue" ? `issue:${record.owner.issueId}`
|
||||
: record.owner.kind === "run" ? `run:${record.owner.runId}` : null;
|
||||
if (!ownerKey) return { settled: false, reason: "owner_not_provable" };
|
||||
let publication: ActivityPublication | undefined;
|
||||
const outcome = await db.transaction(async transaction => {
|
||||
const tx = transaction as unknown as Db;
|
||||
const [row] = await tx.select().from(executionWorkspaceRuntimeLeases)
|
||||
.where(and(eq(executionWorkspaceRuntimeLeases.id, record.recordId),
|
||||
eq(executionWorkspaceRuntimeLeases.ownerKey, ownerKey)))
|
||||
.for("update");
|
||||
if (!row) return { settled: false, reason: "lease_no_longer_held" };
|
||||
const gone = await proveWorkspaceRuntimeLeaseOwnerGone(tx, row);
|
||||
if (!gone) return { settled: false, reason: "owner_changed_since_proof" };
|
||||
const deleted = await tx.delete(executionWorkspaceRuntimeLeases)
|
||||
.where(and(eq(executionWorkspaceRuntimeLeases.id, row.id), eq(executionWorkspaceRuntimeLeases.ownerKey, ownerKey)))
|
||||
.returning({ id: executionWorkspaceRuntimeLeases.id });
|
||||
if (!deleted.length) return { settled: false, reason: "lease_no_longer_held" };
|
||||
// The lease row is gone; the activity entry keeps who held it and why it was released.
|
||||
publication = (await persistActivity(tx, {
|
||||
companyId: row.companyId, actorType: "system", actorId: "ownership-reconciler",
|
||||
action: "execution_workspace.runtime_lease_released", entityType: "execution_workspace",
|
||||
entityId: row.executionWorkspaceId, runId: row.ownerRunId, issueId: row.ownerIssueId,
|
||||
details: { leaseId: row.id, previousOwnerKey: row.ownerKey, lastAction: row.lastAction, proof: gone },
|
||||
})).publication;
|
||||
return { settled: true, reason: gone };
|
||||
});
|
||||
if (outcome.settled && publication) publishActivity(publication);
|
||||
return outcome;
|
||||
}
|
||||
@@ -31,10 +31,12 @@ const TERMINAL_HEARTBEAT_RUN_STATUSES = new Set([
|
||||
export const LEASED_WORKSPACE_RUNTIME_ACTIONS: readonly string[] = ["start", "stop", "restart", "repair"];
|
||||
|
||||
/**
|
||||
* Upper bound on how long a lease survives without the owner touching it. Recovery
|
||||
* from a stale owner is otherwise driven by owner eligibility, which is cheaper and
|
||||
* more precise; the TTL only bounds the cases eligibility cannot see (for example an
|
||||
* agent-scoped owner with no run or issue identity).
|
||||
* Idle-reclaim policy window. A lease untouched this long may be taken over by the
|
||||
* next claimant at claim time. This is a fairness policy, not an ownership proof:
|
||||
* dead owners (terminal, hidden or missing issue; terminal or missing run) are
|
||||
* released by identity, with no age, both at claim time and by the shared
|
||||
* ownership reconciler. For agent-scoped owners, which no identity can prove gone,
|
||||
* expiry plus the locked takeover is the fenced handoff.
|
||||
*/
|
||||
export const WORKSPACE_RUNTIME_LEASE_TTL_MS = 30 * 60 * 1000;
|
||||
|
||||
@@ -72,7 +74,7 @@ export function buildWorkspaceRuntimeLeaseOwnerKey(owner: WorkspaceRuntimeLeaseO
|
||||
return null;
|
||||
}
|
||||
|
||||
function parseOwnerKey(ownerKey: string): { kind: "issue" | "run" | "agent"; id: string } | null {
|
||||
export function parseWorkspaceRuntimeLeaseOwnerKey(ownerKey: string): { kind: "issue" | "run" | "agent"; id: string } | null {
|
||||
const separator = ownerKey.indexOf(":");
|
||||
if (separator <= 0) return null;
|
||||
const kind = ownerKey.slice(0, separator);
|
||||
@@ -82,14 +84,18 @@ function parseOwnerKey(ownerKey: string): { kind: "issue" | "run" | "agent"; id:
|
||||
return { kind, id };
|
||||
}
|
||||
|
||||
async function evaluateStaleOwner(
|
||||
tx: LeaseTx,
|
||||
lease: ExecutionWorkspaceRuntimeLeaseRow,
|
||||
now: Date,
|
||||
): Promise<WorkspaceRuntimeLeaseStaleReason | null> {
|
||||
if (lease.expiresAt.getTime() <= now.getTime()) return "lease_expired";
|
||||
|
||||
const owner = parseOwnerKey(lease.ownerKey);
|
||||
/**
|
||||
* Proves by identity that an issue- or run-owned lease's owner is gone: the issue
|
||||
* is missing, hidden or terminal, or the run is missing or terminal. Returns null
|
||||
* while the owner lives, and for agent-scoped or unparseable owners, which have no
|
||||
* identity to prove. The shared ownership reconciler's settle and the claim path
|
||||
* both use it, so the lane is released on the same proof wherever it is checked.
|
||||
*/
|
||||
export async function proveWorkspaceRuntimeLeaseOwnerGone(
|
||||
tx: Pick<Db, "select">,
|
||||
lease: Pick<ExecutionWorkspaceRuntimeLeaseRow, "ownerKey" | "companyId">,
|
||||
): Promise<Exclude<WorkspaceRuntimeLeaseStaleReason, "lease_expired"> | null> {
|
||||
const owner = parseWorkspaceRuntimeLeaseOwnerKey(lease.ownerKey);
|
||||
if (owner?.kind === "issue") {
|
||||
const ownerIssue = await tx
|
||||
.select({ status: issues.status, hiddenAt: issues.hiddenAt })
|
||||
@@ -113,11 +119,26 @@ async function evaluateStaleOwner(
|
||||
return null;
|
||||
}
|
||||
|
||||
// Agent-scoped and unparseable owners have no lifecycle to inspect, so the TTL
|
||||
// checked above is their only recovery path.
|
||||
return null;
|
||||
}
|
||||
|
||||
async function evaluateStaleOwner(
|
||||
tx: LeaseTx,
|
||||
lease: ExecutionWorkspaceRuntimeLeaseRow,
|
||||
now: Date,
|
||||
): Promise<WorkspaceRuntimeLeaseStaleReason | null> {
|
||||
// Ownership proof first: a gone issue or run releases the lane with no age
|
||||
// involved (the same proof the shared reconciler settles on).
|
||||
const gone = await proveWorkspaceRuntimeLeaseOwnerGone(tx, lease);
|
||||
if (gone) return gone;
|
||||
// Idle reclaim: a POLICY, not an ownership proof. A lease untouched for the
|
||||
// TTL is handed to the next claimant even when its issue or run still lives
|
||||
// (fairness between issues sharing a workspace). For agent-scoped owners,
|
||||
// which no identity can prove gone, expiry plus this locked takeover is the
|
||||
// fenced handoff. The reconciler never releases a lease on this rule.
|
||||
return lease.expiresAt.getTime() <= now.getTime() ? "lease_expired" : null;
|
||||
}
|
||||
|
||||
function throwLeaseConflict(input: {
|
||||
lease: ExecutionWorkspaceRuntimeLeaseRow;
|
||||
requestedAction: string;
|
||||
|
||||
Reference in new issue
Block a user